Holst was born at Røyken in Buskerud, Norway. He was the son of parish priest Christian Holst (1743–1824) and Marthe Grønbech (1743–1786). He studied both theology and law at the University of Copenhagen from which he graduated in 1798.
In 1806 as, he was bailiff at Akershus Castle. In 1813, he was appointed acting district governor in Akershus.
He was the functioning head of the First Ministry (finance and taxes) in 1814, state secretary 1814–1822, member of the Council of State Division in Stockholm in various periods between 1823–1844, Minister of the Navy 1824–1825, Minister of Justice 1825–1826, 1827–1832, 1833–1836 and 1837–1838, and Minister of Education and Church Affairs 1822–1823, 1835, 1836–1839, 1840–1843 and 1844–1848.[2][3]