Graves moved to Sheffield to become an apprentice watchmaker. He was apprenticed to W. Wichman of Gibraltar Street. He then set up one of Britain's first mail order businesses, selling first watches and then a wide range of goods. The company employed, at its peak, 3,000 people in Sheffield and had an annual turnover of £1 million. The company was absorbed by Great Universal Stores after Graves' death.
He married Lucy Dawson on 28 June 1891 in Crookes, Sheffield. They had one daughter, Ruth Julia.
A keen art collector, Graves donated nearly £60,000 towards the development of art galleries in Sheffield, including Sheffield Central Library, the Graves Art Gallery and the Mappin Art Gallery. He also contributed hundreds of pictures from his private collection. Between 1902 and his death in 1945, Graves lived at Riverdale House in the Ranmoor area of Sheffield. In the 1911 census he was living with his wife Lucy and daughter at The Folds, off Abbey Lane, Beauchief, Sheffield.
Graves donated over £1 million to Sheffield, including the establishment of Sheffield University's Student Union. Part of the Union building is known as the "Graves Building". He also made gifts of land to the city, including Ecclesall Woods, Tinsley playing fields, Concord Park, Blacka Moor and one of the largest plots of land he donated was named after him, Graves Park. He also gifted land known as Cliffe Park to the town of Dronfield, Derbyshire, which is roughly 2.5 miles south of Graves Park in Sheffield.
The J. G. Graves Trust, a charitable trust set up in his name, exists to the present day.