It is a curious but well established fact that a man does not feel the warmth of heavy trousers to the extent that he does when he wears a jacket made from the same cloth. And, since trousers are all about drape - we want them hanging in a straight line from the waist - heavier cloth makes for a better-looking pair. That means it is a good idea to wear trousers that are a few ounces heavier than the accompanying jacket.
Of course, this practice is not very practical with suits but it does make for better looking odd jacket pairings. Reserve twelve ounce/360 gram flannels to wear with a mid-weight blazer. Don the 16 ounce/480 gram stuff (Minnis offers it and Holland & Sherry has some worsteds that are even heavier) to look like the man to the left in the Apparel Arts illustration. The pipe is optional.
Wear heavier trousers.