Willoughby was admitted to the New York bar at Oswego. He then began his private legal practice back home in Groton.
During the American Civil War, Willoughby helped recruit soldiers for the 137th New York Volunteer Regiment and received an officer's commission on November 1, 1862. At the Battle of Chancellorsville on May 3, 1863, he was severely wounded in an artillery barrage. Willoughby recovered enough for release from the hospital but resigned from his commission due to his injuries in September.
Willoughby remained in Union-occupied Alexandria, Virginia, and was admitted to the Virginia bar. In 1864, he won the election and on July 5, 1864, assumed office as Alexandria's Commonwealth Attorney (a/k/a prosecutor, including for the portions which later became Arlington County). In May 1866, local judge H. W. Thomas refused to allow an African American woman to testify as a witness against a white man, citing old state law. Willoughby appealed to the federal court and secured an indictment against Judge Thomas (who had reported years earlier that barred the Fairfax courthouse doors to intimidate Unionist voters) for violating the witness' civil rights. Willoughby also allowed African Americans to sit on juries.
During Congressional Reconstruction, Major General John Schofield dismissed Judge Thomas and other judges pursuant to a new federal statute that forbade all former Confederates from holding state offices. Willoughby was appointed to replace Judge Thomas on March 22, 1869, as judge of the Ninth Circuit, which encompassed Alexandria, Fairfax, Fauquier, Loudoun, Prince William, Rappahannock, and Stafford counties. On June 3, Schofield dismissed all three members of the Court of Appeals of Virginia (which later became the Virginia Supreme Court) due to that new law. Schofield appointed Willoughby and two other former Union Officers to that body. When the new Virginia Constitution of 1869 was adopted, he was allowed to remain in office until the General Assembly reorganized the state judiciary. The Court on which he sat met from June 22 to 28, October 12, 1869, and finally from January 11 to February 25, 1870. Judge Willoughby wrote 6 of the 8 opinions it issued.[2]
Willoughby then became a director of the Virginia Peat Company and also resumed his private legal practice in Alexandria. In 1871 he became the Republican candidate for state senate but lost.[3] Later that year, he and Charles Whittlesey formed a partnership specializing in seeking reimbursement from the recently created Southern Claims Commission for Union supporters. The federal government also hired the firm to defend seizures of property from Confederate supporters, including Arlington House, previously owned by Washington descendant George Washington Custis Lee, whose daughter married Confederate General Robert E. Lee, and which had become Arlington National Cemetery. Willoughby also ran to become a member of Alexandria's city council in 1873 as a Radical Republican and lost.[2]
Willoughby became a director of the East Fairmont Gas, Coal, and Coke Company of West Virginia in 1876. He was nominated again in 1881 as the Republican nominee for Attorney General but withdrew from consideration.[3] In 1884, he campaigned for Republican Presidential candidate James G. Blaine, who lost.
He had previously conducted some of his legal practice across the Potomac River in Washington, D.C. He moved there in 1887. Willoughby also taught real property law in 1889 and 1890 at the National University Law School (that later became part of George Washington University). In 1884, he became vice president of the Arlington-Breckenridge Mining Company, and in 1897 president of the Columbia Dredging and Construction Company.[2]
United States