He was the eldest son of Evan Fraser (1826–1906), a Scottish doctor from Duddingston; and Sarah Hewat (born 1829) from Portobello.[4] Evan Fraser and Sarah Hewat moved to Hull shortly after their marriage in 1858 – and he became chairman of the Hull Health committee. The Evan Fraser hospital in Hull bore his name. The hospital specialised in infectious diseases; notably smallpox.[7] James was one of five children the couple had.
James Fraser married Rose Thorney in 1883. Miss Thorney was the daughter of the Hull city coroner.[6] They had a daughter, Dorothy, in 1885. James outlived his wife, who died in 1927, and his daughter, who died in 1941. He died in the Victoria Nursing Home in 1943, leaving £7,511 and 2 shillings in his estate.[8]