The concept of native dress has been around for literally centuries, and for most of that time it has been perfectly acceptable at the English court. That means garb ranging from saris to buckskin and feathered headresses legitimately cohabits with dinner jackets, morning coats and lounge suits depending on the occasion.
Dress on the street in San Francisco tends to be khakis and shirts, but the financial district sees the occasional Savile Row suit. And to my mind, both are correct. The polo-wearing engineer wears the native dress of his set, and the management consultant does the same. Appropriateness is all in the mind.
To my way of thinking, confusion sets in when a member of a group where, for example, no-one wears pocket squares begins appearing with silk foulards in his jackets. He may raise an eyebrow. He will cause comment if he escalates again and moves to ascots or bow ties. For he has left the native dress of his circle behind.
At this point a man has a couple of choices. He can regress to the normal dress of his friends and acquaintances. He may choose a new social circle where his dress is considered normal. Or he may begin dressing situationally, where his appearance varies depending on who he is seeing that day.
My own clothing devolved from lounge suits to the knitwear, flannels and leather jackets of the photograph over the course of a career in Silicon Valley. And then things progressed to the point where I was in New York and London much of the time, and my clothing changed back again. I stuck with that, and today I am usually wearing a lounge suit.
It does not matter what people are wearing around me. For that is my native dress.