After being selected twice by the Toronto Blue Jays in the 1993 and 1994 amateur drafts, he was drafted by the Atlanta Braves in the fourth round of the 1996. Nelson made it to the majors in 2001, seeing two stints of an inning apiece in mop-up duty, where his ERA was 36.00.
Released by the Braves the following summer, he was signed by the Boston Red Sox and quickly released after four appearances for Double-A Trenton. Nelson re-signed with the Red Sox in the spring of 2004, and was promoted to the majors in June as bullpen help. After recording a scoreless inning against Texas, he gave up 5 runs over 12⁄3 innings in his next two appearances. He was optioned back to Triple-A when Mark Malaska was activated off the disabled list.
After spending spring training with the New York Mets in 2005, he signed to play with the Devil Rays' Triple-A team. Despite good peripheral numbers of 61 strikeouts in 46 innings and a .229 opponents' batting average, he went 0–3 with a 4.11 ERA and was released in July. Nelson then had a short 13-inning stint with the Cardinals' Double-A club, allowing just four hits over that span.
He signed with the Kansas City Royals organization in November. Nelson bounced between Omaha and the major-league club before being called up for good on July 19, 2006. After Royals closer Ambiorix Burgos was demoted on August 14, Nelson threw a perfect ninth inning the following day. Manager Buddy Bell used the 31-year-old journeyman in the closer's role for 2006. In 2007, he was hurt. At the end of the 2007 season, he was designated for assignment to Triple-A Omaha, which he refused and opted out becoming a free agent.
In December 2007, Nelson was signed by the Florida Marlins to a minor league contract. Joe Nelson earned a victory over the Mets in his final appearance with the Marlins on September 28, 2008, becoming the last recorded winning pitcher at Shea Stadium.[3] Nelson was non-tendered by the Marlins following the 2008 season.[4]
On December 30, 2008, Nelson signed with the Tampa Bay Rays. In October 2009 Nelson was granted free agency.[5]
On February 1, 2010, Nelson signed a minor league contract with the Boston Red Sox. When Josh Beckett was placed on the disabled list on May 19, Nelson was called up to take his roster spot. He elected to become a free agent after he was designated for assignment by the Red Sox last week after opening the season with a 9.72 ERA over his first eight appearances.
On June 24, 2010, Nelson signed a minor league contract with the Seattle Mariners, but was released on July 28, 2010.
After his active career, Nelson served as a professional scout for the Red Sox, based in West Palm Beach, Florida. As of 2014, he holds a similar position with the Seattle Mariners. In February 2017, the Chicago Cubs hired him as a scout.
