On 23 August 1830, he married Charlotte Louise Elisabeth Olsen (1808–66). She was the daughter of military prosecutor (Generalauditør) Ulrik Christian Olsen and Henriette Zinn.[1]
He owned Fuglsang Manor and Priorskov on Lolland.[3] In the same year, Neergaard bought Trøjborg Castle [da] at Rønder. In 1842, he replaced Fuglsang's main building with a modest new house.[1]
He died on 10 April 1849. He is buried at Gammelholm, an old castle bank just north of Fuglsang's main building.[1]
Some time after Louise Neergaard had become a widow, probably in around 1852, she purchased the county house Olufshøj in Søllerød, north of Copenhagen. The property had belonged to her father. Her city home was an apartment at Amaliegade 7.[1]
Johan Ferdinand Neergaard and his wife were the parents of six children:[1]
- Carl Peter Ulrich de Neergaard (1831–1863)
- Johan Thomas Oluf de Neergaard (1834–1921), married to Anna Maria Elisabeth Antoinette Castenschiold
- Anna Henriette Elisabeth de Neergaard (1835–1891), married to Ole Christian Kahrs
- Rolf Viggo de Neergaard(1837–1915), married to Bodil Neergaard, inherited Fuglsang Manor and Priorskov
- Louise Oline Julie Charlotte de Neergaard (1840–1900), narried Frederikm Count Ahlefeldt-Laurvigen (1834-1888)
- Johanne Marie Lucie de Neergaard (1841–1932), married