Casual trousers for cooler weather? We need only remember two words: corduroy and moleskin. Pair them with tweed jackets, navy blazers or sweaters in colors of the season.
Corduroy of course is velour côtelé, or ribbed cotton velvet. It takes its name from the material used on the jackets of hunting livery worn by the servants of the Kings of France. Both corduroy and moleskin, a brushed cotton, are hard wearing members of the fustian class of cotton fabrics that also includes cotton velvet.
The ridges in corduroy are known as wales and the cloth is made with as few as seven (jumbo) to as many as 16 (needlecord) wales per inch. Usually, though not always, more wales means lighter cloth. Corduroy is generally available in cloth as light as nine ounces and as heavy as 23 ounces and a wardrobe might include different weights for different temperatures. I find needle cord comfortable in temperatures as warm as 75 °F (24 °C) whereas a walk in the 23 ounce jumbo stuff will get me sweating at 50 °F (10 °C).
Try them tieless with long sleeved knitted polo shirts worn with a neckerchief or buttoned at the neck and a pair of suede chukkas.