Patterson was signed by Queens manager Jimmy McKinnell, Jr. in 1949. At the time, Queens had been in the top division of Scottish football since 1933, a club that already had McKinnell as manager and Roy Henderson, Dougie Sharpe and Billy Houliston at Palmerston Park as players. Jimmy Binning joined in 1951 and goal scoring outside right Bobby Black in 1952. All would represent Scotland at some level except Henderson despite being selected six times as reserve goalkeeper for Scotland.[2]
Patterson made his Queens debut versus Dundee at Palmerston on 12 November 1949. Although Queens were relegated in 1950 this was also the only season in the 20th century when the club would reach the Scottish Cup semi finals. Queens were promoted straight back to the top flight as Scottish B Division champions in 1950–51. The success of that season was enhanced by a run to the semi-final of the Scottish League Cup. The early and mid-1950s saw the club's most successful spell to date, achieving consistent mid table finishes in the Scottish A Division, as the top flight was then called. This peaked with a sixth-place finish in 1956, a finish surpassed only once in the club's history.[2] In one game against the current champions Hibernian at that time, Patterson scored four goals in Queens 5-2 victory versus Hibernian's Famous Five forward line during the 1951-52 season.[2]
Queens were subsequently relegated from the top division in 1959. The following year, the former Scotland and long-time Blackpool FA Cup winning goalkeeper George Farm signed for Queens. The Doonhamers would make it to the League Cup semi final in 1960–61 for the second time. Farm was made player-manager in 1961 and with Jim Patterson still playing the club were promoted back to the top division in 1962. Thus Patterson finished at Queens playing top division football. With Patterson at Queens at this time were future Scotland centre forward Neil Martin, right winger Ernie Hannigan who like Martin went on to play in England's top division and long servants Allan Ball and Iain McChesney.[2]
Patterson's greatest scoring feat was when he scored six goals in a 7–0 win versus Cowdenbeath during the 1961-62 season.[2] Patterson was one of the few full-time players at the Doonhamers during his time at Palmerston.
Patterson is the club's all-time record goalscorer on 252 goals, 86 goals ahead of Stephen Dobbie on 166 goals. The only other three players to score more than one hundred goals for the Doonhamers are Bobby Black with 120 goals, Derek Lyle with 117 goals and Andy Thomson with 114 goals.
Patterson made 462 appearances for Queens between 1949 and 1963 and is the third highest in the club's all-time appearances list behind Allan Ball and Iain McChesney. Patterson's final match for the Doonhamers was at Shawfield Stadium versus Clyde on 4 May 1963.[2]
Patterson's scoring exploits were honoured with one game for Scotland, when he led the Scottish attack versus The Army in 1953 at Hampden Park. Patterson can consider himself unlucky to have played his best years at a time when there was an abundance of quality Scottish centre forwards like Lawrie Reilly who played for Hibernian, part of the Famous Five.[2]
Patterson died on 16 December 2012, at the age of 84.[4][5]