This study investigated the diversity of carotenoids and phenolics in germplasm from three Capsicum (chile pepper) species, Capsicum annuum, Capsicum baccatum, and Capsicum chinense. Lutein, a yellow-pigmented carotenoid, and phenolics, a group of secondary metabolites, are reported to have health-promoting properties. The germplasm studied matured to a yellow color. The hypothesis was that all yellow fruits would contain either the carotenoid lutein, a yellow pigment, or a large amount of phenolics, a group of secondary metabolites that may be yellow among other colors. Thirty-one Capsicum accessions were grown in the field over a period of two seasons. On a dry weight (DW) basis, lutein ranged from 0.14 to 94.2 μg·g−1, and total phenolics ranged from 5.79 to 15.01 mg·g−1. No lutein was detected in one accession and β-carotene, another health-promoting compound, was lacking in four accessions. Accessions were grouped into four groups according to a principal component analysis plot. Results from this study indicate that in only nine accessions, lutein represented at least 50% of the total carotenoid amounts in each accession. These accessions are desirable not only as a source of dietary lutein, a natural yellow pigment, but also as genetic material that can be used to breed for higher lutein Capsicum. Therefore, yellow color is not a good indicator of lutein content and phytochemical analysis is required to determine the content of health-promoting compounds.