Heavy alcohol use has numerous negative impacts on cognition. In many cases, it appears that excess drinking influences working memory, problem solving, attention, patterns of regional brain activation, and even gray and white matter volume. Due to these effects, most significant conclusions related to cognitive decline are limited to the assumption that “more is worse.” However, heavy alcohol use is a very complex psychosocial behavior and is subject to a problematic number of variables. As a result, it is very difficult to extend generalizations about drinking-related consequences to the overall population since many environmental and biological differences come into play for every individual. Three of these variables shown to have significant impact on cognitive sensitivity to alcohol are age of first alcohol use (age of drinking initiation), specific pattern of alcohol consumption and gender. Potential insight can be gained into how individual drinking scenarios differ in risk for cognitive decline by assessing how each of these important factors influences cognition independently. Although some overlap exists between categories, each still appears to contribute unique influence on likelihood and presentation of cognition-related effects. Based on existing and current research, age and gender tend to augment baseline sensitivity to alcohol, with patterns of alcohol intake also influencing how changes appear. Given these categorical differences, it is important to consider personal alcohol drinking history rather than just aggregate alcohol intake as risk factors in the cognitive impacts of drinking. The objective of this brief review is to examine the role of these factors on the effect of alcohol on cognition.