James B. Elliott was born in 1849, in Philadelphia, where he was raised by his mother. He found work as a carpenter and later became a freethinker, giving lectures on rationalism at the Philadelphia Ethical Society and Friendship Liberal League, and writing articles for The Truth Seeker. Through one freethought newspaper, The Progressive Age, Elliott encountered the writings of Voltairine de Cleyre, noting her expressive anger towards the Catholic Church.
In June 1888, he greeted de Cleyre at the Philadelphia train station, having invited her to give a speech at a meeting of the Friendship Liberal League. The following year, she settled in the city and moved into Elliott's rooming house. The two were quickly attracted to each other, although historian Margaret Marsh notes that Elliott was "intellectually inferior" to de Cleyre and had an "eccentric" character, which left her to wonder why de Cleyre found him so appealing. They struck up a brief romantic relationship, but as de Cleyre refused to marry Elliott, it quickly broke down and ended. They remained friends and continued to live in the same house, with Elliott caring for de Cleyre's plants and pets while she was on lecture tours.
On June 12, 1890, de Cleyre gave birth to their son Harry, but she felt emotionally and physically unable to raise him, so left him in Elliott's care. De Cleyre's sister Adelaide D. Thayer, herself childless, asked if she could take care of Harry, but Elliott refused. In 1894, de Cleyre moved out of the Elliott household, finding a new home with the couple Mary and George Brown. In 1896, Elliott visited de Cleyre's mother Harriet De Claire and built an extension for her house in St. Johns, Michigan. Around this time, Elliott was made secretary of both the Paine Memorial Association and the Paine Historical Association. Elliott idolised Thomas Paine, going on a pilgrimage to all of Paine's homes in both North America and England and collecting his memorabilia.
By the turn of the 20th century, Elliott had become increasingly mentally imbalanced and left his son at the age of ten to fend for himself. In 1900, he managed to find de Cleyre's estranged father Hector De Claire, and together they sent Harriet a number of messages that worried her. For this, de Cleyre called Elliott a "devil" and said he was "not right in his intellects", telling her sister Adelaide that he had "no self-respect whatever, but insists on intruding himself constantly everywhere."
In 1910, de Cleyre moved to Chicago and Elliott fell out of touch with her. After her death, her anarchist comrades claimed that Elliott had deliberately prevented de Cleyre from seeing their child, but Margaret Marsh notes that this depiction is inaccurate. Elliott himself died in 1931, at the age of 82. Elliott was posthumously denounced in anarchist newspapers for neglecting his son Harry and preventing de Cleyre from seeing her child; according to historian Margaret Marsh, this depiction was "not strictly true, but it preserved an unsullied image" of de Cleyre.
