Findings with experimental rodent models reveal that exposures to dietary factors during the in utero and pubertal periods when the mammary gland is undergoing extensive modeling and re-modeling, alter susceptibility to develop mammary tumors. Similar observations have been made in humans: childhood exposure to genistein in soy or to some other bioactive food components reduces later breast cancer risk, although they may have no effect if consumed during adulthood. Thus, food components may be more effective in affecting cancer risk in some periods of life than others. Many of these dietary exposures modify fetal and postnatal hormonal environment, including changing the concentrations of estrogens and leptin. The hormonal alterations then may induce persistent epigenetic changes by affecting gene promoter regions or by inducing histone modifications that affect chromatin transcription. The targets of epigenetic changes are likely to be the terminal end buds (TEBs), the structures where carcinogen-induced mammary tumors in rats and mice are initiated. More specifically, the site of these changes in TEBs may be the stem cells and their niche; this might explain how an exposure early in life affects the risk of breast cancer decades later. Similar structures in women, called terminal ductal lobular units, are the sites where most human breast cancers rise. According to this hypothetical model, cancer is initiated only when the epigenetically altered cells are exposed to carcinogens/radiation, etc. during adult life. In a “normal” stem cell or its niche, cancer initiating exposures do not necessarily cause cancer, because the cells can either repair the damage or undergo apoptosis. Thus, the most likely molecular targets of early life dietary exposures are genes that regulate DNA adduct formation, repair DNA damage or induce apoptosis, such as genes affecting cellular metabolism, tumor suppressor genes or genes promoting cell survival. It is possible that some of these epigenetic changes also explain why the number of TEBs generally, but not always, correlates with breast cancer risk. This hypothesis may imply that adult intake of some bioactive dietary components reduces cancer risk increased by early life dietary exposures or inhibits tumor growth by reversing epigenetic changes in various molecular targets.