Born in Stepney, London, Groves started his career at east London non-league clubs Leytonstone and Walthamstow Avenue, and then briefly as an amateur for Tottenham Hotspur before signing professional forms with Leyton Orient in 1954.
He scored twice on his Tottenham debut in a 3–1 victory over Liverpool at White Hart Lane in September 1952.[2] He made four appearances for the 'Lilywhites'.
Playing as an inside forward, his regular goalscoring for Orient soon attracted the attention of Arsenal, who signed him in November 1955 for £23,000, at the time a very high figure.
He scored on his debut, against Sheffield United on 12 November 1955, in a 2–1 First Division win (Don Roper grabbing the other Arsenal goal).
Despite an eye-catching start, Groves initially struggled to live up to expectations at Arsenal, his first season at the club hampered by injury, first to a knee and then to his back. It wasn't until the 1958–59 season, that he had established a regular place. Groves scored ten goals that season and formed a useful attacking partnership with David Herd.
The season after that, he was switched to wing-half, where he played for the rest of his career; Groves was also made Arsenal captain, following the departure of Dave Bowen. His spell at Arsenal was continually hampered by injury; although when he did play, he was impressive enough so that he was selected for the London XI that played (and lost to) Barcelona in the first Inter-Cities Fairs Cup final in 1958.
Groves lost his first-team place in the 1961–62 season, although he stayed at Arsenal as a bit-part player until the summer of 1964, when he left to sign for Southern League side Canterbury City. In all he played 201 times for Arsenal, scoring 37 goals. Although he represented England at amateur and youth level, he never won a full cap.
Whilst a player, Groves ran a garage, just off Blackstock Road, close to the Arsenal ground. After retiring from playing Groves left the game completely, running a pub and working in insurance. His great nephew, Perry Groves, played for Arsenal in the 1980s and 1990s, helping them win two league titles and a Football League Cup.