Caldwell's first professional boxing match was on June 9, 1969, defeating Allen Heard. He defeated Heard in two consecutive rematches between June and July of that year, all fought in Caldwell's home state of Louisiana.
Caldwell went on to become a journeyman fighter, competing against a range of heavyweights. His first notable challenge was fought on December 9, 1969, against Cleveland "Big Cat" Williams. Two years later on July 24, 1971, Caldwell fought Ron Lyle, losing by a unanimous decision. Two months later on September 21, 1971, Caldwell lost a fight against George Foreman via knockout.
On October 25, 1972, Caldwell lost a fight against heavyweight contender Earnie Shavers. In later years, Caldwell described this fight as being hit with a shotgun and remembering little more than feeling the force of Shavers' punch down to his toes. He also stated by the time he woke up, he was in the dressing room and his cornermen were in the process of removing his gloves from his hands.[3] On July 23, 1973, Caldwell lost a fight against Oscar Bonavena via technical knockout.
Caldwell's first title fight was held on July 16, 1979, against Eddie Brooks to crown the new Wisconsin State Heavyweight Champion. The match ended in a draw.
On October 6, 1981, Caldwell fought his second title fight defeating John Williams for the Nevada State Heavyweight Championship. His last fight was held on June 30, 1985, losing to Henry Tillman.