Aromatic hot peppers (Capsicum chinense Jacq.) are an important agricultural commodity for many small-scale farmers in the Caribbean because it is a commercially viable crop and one that is integral to the cuisines of the region. The large variation in fruit shape, size, color, pungency, and aroma of this species facilitates a diverse range of uses. Using 264 accessions from a Caribbean germplasm collection (representing primarily the Caribbean Basin, Central and South America), this study investigated 1) morphological variation in 13 fruit descriptors of agro-economic importance; and 2) morphological groups based on geographic origin. All 13 fruit descriptors showed significant variation. Fruit color [immature (six states) and mature (12 states)] was the most diverse qualitative fruit trait. Among the quantitative traits, fruit weight and fruit width showed the highest broad-sense heritability (0.81), and fruit weight was highly correlated with fruit width and placenta size. Cluster analysis revealed four main clusters, which did not show a clear separation of accessions based on major geographic regions, but there was a highly significant association (P < 0.0001) between geographic subgroups and the clusters to which they were assigned. Most accessions of the Northern Caribbean (particularly the Bahamas and Puerto Rico) separated quite distinctly from most accessions of the Southern Caribbean and clustered with most accessions of Central and South America. Accessions of the Southern Caribbean (Lesser Antilles, Trinidad & Tobago) were substantially more similar to each other than they were to most accessions of Central and South America, thereby suggesting genetic differences between accessions of the Southern Caribbean islands and the mainland populations. Collectively, the results show that this germplasm collection contains useful accessions with desired fruit quality traits and a level of genetic variation that can be used to encourage its active conservation and use for further evaluation trials and crop improvement as well as guide ongoing complementary germplasm introductions to augment the collection’s diversity.