In 1956, Spencer wrote the popular song "My Boy Lollipop".[1][2] Robert would lose the rights to the song in a card game. The winner of the rights was then offered $100 by Morris Levy, a record executive, to have the rights to himself. Levy then showed the lyrics to Gaetano Vastola, a mobster who also worked as a concert promoter, who got a fourteen-year old girl, Barbie Gaye, to record it.[3]
Levy removed Spencer's name from the credits to avoid any royalties going to him and even if "Robert Spencer" was credited to any "My Boy Lollipop" singles, Levy would claim that it was his pseudonym.[1][4] Originally sang by Barbie Gaye, the cover best associated with "My Boy Lollipop" would be that of Millie Small's version in 1964, whose version went to number two in the US in 1964.[5]
In 1957, Spencer joined The Cadillacs originally as a baritone backing vocalist before he became the groups lead singer when Earl Carroll left in late 1958.[6][7] Also during this time, Robert was acting as a fill-in member for another doo-wop group, The Tornados, who he also wrote songs for.[8][9] Spencer became a session singer after the Cadillacs diabanded in 1962. He reformed the Cadillacs with other members of the group in the 1970s.[2]
In 1969, Jerry Kasenetz and Jeffry Katz of Super K Productions, trying to cash in on the growing bubblegum pop genre, assembled a new pop group consisting of session musicians, including Spencer on lead vocals, to record the song "Gimme Gimme Good Lovin'",[10] whose recording was released under the name Crazy Elephant. The song became a transatlantic hit, and got to number 12 in both the US and UK. When the song got popular, an album was made still featuring Spencer on vocals, and a touring group was crafted to perform the songs of Crazy Elephant, the touring group however did not feature Spencer in the lineup.