The only situation in which to leave the buttons on a collar point unbuttoned
Collar buttons exist purely for utilitarian reasons, and the rules for buttoning couldn't be simpler.
The first thing you must realize is that button-down collars are inherently more casual; you'll typically find a button-down collar on an Oxford-cloth or broadcloth shirt.
But, casual or not, collar buttons are there for a reason - and more often than not they should buttoned.
To button or not to button.
Aside from keeping your collar from flapping around, the ensuing rigidity provides a perfect collar arch between the collar and the shirt. This is called the collar roll: a huge feature of button-down sport shirts made in the 20th Century.
Leaving your collar buttons undone - with the collar flapping around and the buttons just sitting there, sad and unused - will make your outfit look undone, or as if you hurried out of the house before you could finish dressing.
The only exception to the rule.
That said, some men do choose to leave collar buttons unbuttoned for an artfully casual and carefree style. That kind of aesthetic can work if you're wearing shorts or headed to the beach, but, for the most part, you're better off keeping the collar buttoned and the roll firmly in place.
And if you decide to wear the shirt with a tie, you should never even consider leaving the collar buttons unbuttoned.