On 15 December 1763, at the church of St Andrew in Chichester, he married Lucy Hart (1739–1774).[7] She died on 9 March 1774 and of their children only one son reached adulthood:
- John Drew (1764–1800), who married Lady Susan (née Murray) Tharp (1768–1826), the daughter of John Murray, 4th Earl of Dunmore, and his wife Lady Charlotte Stewart. Her sister, Lady Augusta Murray, was married to the King's son, Prince Augustus Frederick, Duke of Sussex. Lady Susan was the widow of Joseph Tharp (1767–1797), heir to a huge sugar fortune amassed in Jamaica by his father John Tharp of Chippenham Park, and had two children. After his death, Lady Susan married an Irish clergyman, the Rev. Archibald Edward Douglas, and had a further child.
On 25 October 1781, at the church of All Saints in Chichester, he married Agnes, Lady Frankland (1726–1783),[8] the American widow of Sir Charles Henry Frankland, 4th Baronet. She died on 23 April 1783 and was buried at the church of Church of St Pancras, Chichester [it].
He made a third marriage on 23 April 1804 at St Luke's Church, Chelsea to a spinster, Mary Powlett Drinkwater (1744–1827), daughter of Woodroffe Drinkwater and his wife Ann Costellow.[9]
Drew died in 1808 and his will was proved in London on 22 June 1808.[10] She survived him, dying on 4 June 1827. In her will of 4 November 1825, proved in London on 16 August 1827, the first beneficiary was Charles Peckham Peckham, grandson of her sister Elizabeth who had married John Peckham of Nyton, and she also left bequests to her husband's two grandchildren.[11]
Through his only surviving son John, he was a grandfather of Lucy Ann Drew, born 1798,[12] and Georgiana Drew, born posthumously in 1801, neither of whom married.[13]