Edward Rawson, the secretary of the Colony of Massachusetts, is recorded as the first European settler of Caryville and North Bellingham in the mid-1600s who owned a farm in the area. The land was given to him as a land grant by the Massachusetts General Court in Boston as a form of compensation for his work. The land which became the village of Caryville was passed on as an 800-acre parcel to William Rawson, his heir, who broke up the land and sold it in 1701. Thomas Burch bought 200 acres of the land along the Country Road, an early name for the modern day Hartford Avenue through the center of Caryville. Burch sold the land to John Metcalf in 1735. Stephen Metcalf, his grandson, sold the land to Joseph Fairbanks in 1800.
As Bellingham took shape as a town, the community divided itself up into school districts. Caryville was included in the West Parish of Medway in 1747. Caryville emerged as a full fledged village beginning with the opening of the Caryville Mill in 1813 by Joseph Fairbanks. The site became the second active mill in Bellingham. Property rights were split between Metcalf and Fairbanks with provisions for a dam, a small water channel, a grist mill and a cotton mill.
In 1830, William White purchased the Caryville mill. Upon his death, his widow remarried William H. Cary, the namesake of the village, who had moved to Medway from Attleboro to work in cotton mills in 1818. Cary rebuilt and enlarged the mill after a fire and constructed three tenement houses. Cary lived past the age of 80, owning the mill for 16 years. The residents of the small village named the post office after him when it was built in 1866.
Electric streetcars began running between Milford, Caryville and Medway beginning in 1897, with electric lighting in the village by 1907. Throughout its history, Caryville was dominated by the Metcalf family. Edwin and William Fairbanks, the grandsons of Joseph Fairbanks, operated a shoe factory until it burned down in 1876.
Caryville was a large and prosperous mill village that at one point in the 1800s petitioned to be set off as a separate town. Into the early 20th century, the village had an active baseball rivalry with the Dean College baseball team in the center of Franklin.
