When human ancestors left the constant equatorial climate of Africa ~70,000 years ago to spread around the planet, the first challenge some of those groups faced was cold seasonal climate - no gathering of ready-to-hand plants in the winter months.
This put a premium on a number of new behaviours: mandatory hunting in the winter; enhanced collaborative-working; tighter male-female bonding (women were now reliant on their hunting menfolk in the winter months); conscientiousness, preparedness and forethought as more things could lethally go wrong; generalised problem-solving and technological skills. These IQ and personality traits therefore underwent a round of selection.
Around 12,000 years ago in the Middle-East agriculture developed and spread. Another round of challenges involving the ability to live placidly in much larger groups (less interpersonal aggression conferred an advantage) plus the ability to live in a society of abstractions (laws, personal planning, markets and money, education, bureaucracy). So more selection, again for IQ and also for the traits of 'domestication' such as greater 'agreeableness'. This was analysed in Cochran and Harpending's "The 10,000 Year Explosion" reviewed here.
So which selective force had the greater influence on IQ? We have the natural experiment of different human groups where extensive measurements have been made (reviewed here).
** The Chinese have lived in agricultural societies for more than 6,000 years and evolved in a cold climate with cold adaptations.
** The Inuit have even more extreme cold adaptations but lived a pre-agricultural hunting lifestyle until the last century.
** Hot climate Sub-Saharan Africans have practiced agriculture/herding for perhaps 2,000 years.
** Australian Aborigines live in a hot climate and are hunter-gatherers.
So here's the matrix. It looks like climate (cold and seasonal) is the main driver with agriculture/pastoralism less important.
Contribution of climate and agriculture to intelligence increasesNote: agriculture (farming crops) is sedentary, scalable and leads to "civilizations". Pastoralism (herding animals) is nomadic, creates violent "honour cultures" in defence of easily-stolen goods and leads to "barbarians". However, in history there has been much gene flow between the two kinds of post-Neolithic societies and the numbers have always been with the farmers.
Note: surely you don't think I consider this any kind of substantive analysis? With so many confounding variables? No, I think the evolutionary account is plausible but the two-by-two matrix is only suggestive at best.