Cognitive function and testosterone level of men decline concurrently with age. Low testosterone levels are associated with higher risk of Alzheimer’s disease and mild cognitive impairment in men. There are continuous debates on whether this relationship is casual. This paper aims to summarize the current evidence on the association between testosterone level and cognitive function in elderly men. The presence of testosterone, androgen receptor and its responsive genes indicates that testosterone has biological functions in the central nervous system. The ability of the body to convert testosterone into estrogen suggests that part of the actions of testosterone could be mediated by estrogen. Observational studies generally showed that low endogenous testosterone levels were associated with poor cognitive performance in healthy elderly men. Testosterone substitution exerted positive effects on certain cognitive domains in normal and hypogonadal elderly men. In conclusion, testosterone may influence cognitive function in elderly men and its substitution may be considered in men with cognitive impairment and testosterone deficiency.