Olsen progressed through the youth academies of Copenhagen and Nordsjælland, before joining AB's under-17 team in 2011, coached at the time by Thomas Nørgaard.[2]
In July 2014, he was promoted to the first team, who were competing in the second-tier Danish 1st Division.[2] He made his league debut on 17 August 2014, starting in a 1–0 away loss to Horsens.[3][4] On 10 June 2015, Olsen signed a contract extension with AB, keeping him at the club until 2017.[5] In his first season as part of the first team, he had made 18 appearances of which 14 were as a starter, as the club had suffered relegation to the third-tier Danish 2nd Division.[6][7]
AB returned to the 1st Division after one season down, but the 2016–17 saw them relegate to the third tier again. On 9 June 2017, Olsen extended his contract by another year, following the club down a division.[8] This time, AB would stay down, with Olsen being a regular starter for the club through the following two seasons.[6] On 23 June 2018, he signed another one-year contract extension with the club.[9]
On 20 June 2019, Olsen signed a one-year contract with 1st Division club Hvidovre, where he was set to replace Malte Kiilerich who had moved to Horsens.[10] He immediately became a starter for the club, debuting in central defense on the first matchday of the season, in a 2–1 away loss against Kolding.[11] On 11 August, he scored his first goal for the club to save a point late in a 2–2 draw against Nykøbing.[12]
Olsen reached promotion to the Danish Superliga with Hvidovre in the 2022–23 season, and extended his contract with the club shortly after.[13] The previous seasons he had formed a defensive core alongside fellow centre-back Daniel Stenderup and goalkeeper Filip Đukić.[14] On 21 July 2023, he made his Superliga debut in a 1–0 loss to Midtjylland, where he played 90 minutes.[15]
On January 29, 2025, it was confirmed that Olsen had joined Danish Superliga side Sønderjyske on a deal until June 2026.[16]
On 1 September 2025, Olsen's contract with the club was terminated by mutual agreement.[17]
On 18 September 2025, Olsen joined Danish 2nd Division club Fremad Amager.[18]