I like the more-casual-than-a-vest look of a sweater under a suit, but the sweater had better not have sleeves or it'll be too hot to wear indoors. Fortunately, there are plenty of sources for armless v-necks, including this very refined cashmere pullover from Parisian designer Marc Guyot.
Guyot's knitwear is made to measure and the vest in the photo below is constructed much like a conventional waistcoat, down to the strap across the back.
Men who wear belted trousers should consider spending their money on sweaters rather than waistcoats. The unattractive sight of the belt buckle under a vest is covered by sweaters with round bottoms. Which is as it should be.