The rich and varied Harvard University botanical collections trace back to the activities of Asa Gray, who came to Harvard in 1842. Gray arrived at a particularly opportune time in American botanical history. Private collectors accompanying various survey expeditions in the expanding American West were sending back an abundance of specimens to scholars in the East. Gray, who had been collaborating with John Torrey in New York on a Flora of North America, was the ideal person to receive, describe and catalogue them, many of which proved new to science. At the same time Gray used duplicates from those collections to exchange with his colleagues in other parts of the world, who themselves were reaping the fruits of an expanding period of exploration in the Old World. The results of those efforts, which continue to this day, have resulted in a collection cosmopolitan in scope, unique in much of its representation and, with well over five million specimens, among the largest in the world.

Gray not only studied and published on the specimens sent to him, but also trained many American botanists, several of whom founded or developed separate botanical institutions at Harvard. They included Charles Sprague Sargent, first director of the Arnold Arboretum, George Lincoln Goodale, first director of the Botanical Museum; and William Gilson Farlow, founder of the Farlow Reference Library and Herbarium of Cryptogamic Botany.  The Economic Botany Collections and the Oakes Ames Orchid Herbarium would later follow.

Until the 1950s, all but the Arnold Arboretum herbarium, which was located in the Jamaica Plain section of Boston, were housed separately in Cambridge. The Gray Herbarium was on Garden Street, within the Botanical Gardens, but blocks from the other biology collections. The Farlow Herbarium was adjacent to the Biological Laboratories on Divinity Avenue and the Botanical Museum was in the Museum of Comparative Zoology building. The herbarium of the New England Botanical Club, although housed at the Gray Herbarium, was a separate unit.  Upon construction of a new herbarium building on Divinity Avenue in 1954, the collections of the Gray Herbarium and Arnold Arboretum (except for cultivated plants of the Arnold Arboretum, which remained in Jamaica Plain) were brought together and integrated. The Arnold, Gray, Farlow and the Botanical Museum herbaria were thus close at hand.

Despite their contiguity, each unit was separately administered until the mid-1970s when administrative functions began to be merged. The collections themselves were finally and fully merged in 1996 when the previously separate NEBC collection was integrated. Although the specimens are now interfiled and administered as a unit, the institution to which a specimen belongs is stamped on each sheet. The acronym for each herbarium should be indicated when citing specimens in the Harvard Herbaria.