He was born at Shirley Plantation to the former Elizabeth Hill, who had been born to the last of three generations of men named Edward Hill to operate that plantation and represent surrounding Charles City County in the House of Burgesses. His great-grandfather had also served as its Speaker. Around the time that her brother, also Edward Hill, had died of tuberculosis at age 16, Elizabeth had married John Carter, the colony's secretary of state and son of King Carter, probably the colony's wealthiest and most powerful man. Her father selected her (and her husband) to inherit Shirley plantation, instead of either of her two elder sisters. His elder brother would be named John (and was alive in 1728 and mentioned in King Carter's will) and his younger brother named Edward would also follow the family's planter and politician traditions, serving in both the House of Burgesses and later in the Virginia House of Delegates representing Albemarle County. Their sister Elizabeth became the wife of nearby burgess William Byrd III (1728–1777), and among their children was powerful politician William Byrd IV.
Charles Hill Carter married twice. His first wife, Mary Carter, was his cousin, daughter of Charles Carter of Cleve. Before she died in 1770, they had four sons who reached adulthood: John Hill Carter (1757 until after 1777), George (1761–1788, who married Lelia, the daughter of Sir Peyton Skipwith), Charles B. Carter (1766–1807, who married his cousin Nancy Beale Carter and lived at "Mt. Atlas"), and Edward Carter (1767–1806, of "Cloverland" in Prince William County who married his cousin Janet Carter (daughter of John Carter of "Sudley") and served in the House of Delegates representing Prince William County).[3] Charles and Mary also had at least three sons who died as infants (Charles William, Landon, and Robert) and two daughters who married and bore many descendants: Mary Walker Carter (1763-1836) married her cousin George Braxton of "Hybla" (1762-1801; son of founding father Carter Braxton), and Elizabeth Hill Carter (1767–1832) married Col. Robert Randolph (1760–1825) of Eastern View plantation in Fauquier County).[4] After a suitable mourning period, Charles Hill Carter remarried, to Ann Butler Moore, daughter of burgess Bernard Moore of Chelsea plantation in King William County, and wife Ann Catherine Spotswood. Their son Dr. Robert Carter (1774–1805 married Mary Nelson of York County who bore Hill Carter (1796-1870 ) who inherited Shirley plantation as would his son Charles Carter (1827-1906), although the most famous descendant would be of daughter Anne Hill Carter (1773-1829), who became the second wife of General and Virginia governor Lighthorse Henry Lee and now most famous as mother of CSA General Robert E. Lee and grandmother of future general and Virginia governor Fitzhugh Lee[5]