Stearns worked as an overseer for a Georgia railroad in 1836 and the next year moved to Richmond, Virginia, to work in a quarry for the James River and Kanawha Canal. By 1843 he was a contractor helping to build the canal, and switched to more mercantile pursuits. He operated a distillery and wine bottling plant, and became a leading member of the Chamber of Commerce and one of the state's largest land owners. The 1860 U.S. census listed Stearns as owning $155,000 in real estate and $200,000 in personal property. His distillery stood on 15th St. between Main and Cary streets in downtown Richmond. His country estate "Tree Hill Farm" in Henrico County outside the city limits overlooked both the city and the James River and stood near the intersection of the important Osborne turnpike and New Market road.
During the American Civil War, Confederate provost John H. Winder considered Stearns a traitor to the Confederacy, and placed him in jail with John Minor Botts, or later house arrest in his Richmond warehouse (where his family could care for him). Evidence that Stearns was disloyal to the Confederate cause was collected after the arrest of the Universalist Church's Reverend Alden Bosserman who admitted to authorities that Stearns had advised him against opening his church on the day after the Manassas battle which was seen as not supportive of the cause. Stearns was also the largest donor to Bosserman's church.[5] Stearns' younger brother William moved to DuPage County, Illinois (where their brother Daniel lived) to avoid the conflict, and Stearns sent his two older sons out of the country so they would not be conscripted into the Confederate army.[6] However, Stearns let the Confederate Chimborazo Hospital use Tree Hill farm to graze its livestock.
On the night of April 1, 1865, as downtown Richmond burned due to fires set by evacuating Confederate troops, Richmond mayor Joseph Mayo traveled to Tree Hill farm to deliver a surrender note to two Union majors he found encamped there, with Union troops. They conveyed the note to Major General Godfrey Weitzel, who traveled to City Hall to accept the Confederate Capitol's formal surrender document at 8:15 a.m. Union troops also stopped the fires and restored order).[7]
After the war, the U.S. Marshall ordered Stearns' Richmond bank account frozen, which prompted a note from Virginia's Provisional Governor Francis H. Pierpont to President Andrew Johnson, attached to a formal pardon application on Stearns' behalf. On July 18, 1865, Stearns received a pardon from President Andrew Johnson for his activities which aided the Confederacy, conditioned upon him never acquiring any slaves or using slave labor.[8] He also was elected to the Virginia General Assembly to represent Henrico County, along with J.J. White. He served in the first session (December 4, 1865 – March 3, 1866) but was replaced by Z. S. McGruder for the session which began on December 3, 1866.[9]
Stearns acquired what had once been the Planter's Bank Building on Richmond's Main Street, and in 1868 erected rental housing and commercial office space, which was nicknamed the Stearns block. The city's circuit court was held there beginning in 1870. His grandchildren's estate sold the property in 1923; the remaining iron front was noted in the Historic American Buildings Survey,[10] and remains today. Stearns was also a director of several banks, and of the James River and Kanawha Canal Company, the Richmond and York River Railroad and the Chesapeake and Ohio Railway.[11] According to family accounts, Stearns refused to become involved in politics, declining to run in 1867 to become a delegate to Virginia's constitutional convention, and later when offered a position as U.S. Senator and even Governor.[11]
In 1869, Stearns traveled to Washington, D.C., to meet with General Grant, provisional governor Henry H. Wells, fellow Conservative Republican Gilbert C. Walker of Norfolk and the Committee of Nine to discuss the future political status of Confederate veterans.[12] Many with more Confederate involvement than Stearns had also received pardons and restoration of their civil rights after surrendering and signing loyalty oaths. However, Virginia needed to adopt a new state Constitution (without the slavery provisions of the 1851 constitution) in order to be readmitted to the Union. The Constitutional Convention of 1868-69 drafted a new constitution pending voter approval, which contained a highly controversial provision barring former Confederates from holding office. Upon General Grant's recommendation after the meetings, occupying General John Schofield permitted Virginia's voters (of all races) to vote separately on the constitution (which passed overwhelmingly) and the disenfranchisement provision (which failed). However, over the next decades, the former Confederates then enacted Jim Crow laws which disenfranchised the former slaves.