The land was first farmed by a tenant farmer named Thomas Davenport in the early 18th century, when it was still part of the vast holdings of Frederick Philipse. Most were later confiscated due to the Philipse family's support for the Loyalist cause during the Revolutionary War; however the future de Rham property was not among these as it was owned by the heirs of Philip Philipse, who had died in the late 1760s. Davenport and his family lived in a small frame house, probably on the present property, whose exact site is unknown.[1]
Davenport's grandson William appears to have left the area around 1802 and sold his lease to a family named Wheelock. They bought the property outright from the Philipse heirs in 1827. The deed for this transaction mentions the main house and a few of the other structures, so they were apparently built during their ownership.[1]
Three years later, in 1830, the Wheelocks sold the property for $6,900 ($217,000 in 2009 dollars[3]) to brothers Adolphus and John Watson, who bought out a third partner the following year. The Watsons operated the Highland School in the main house until 1835, the year after they had sold to Henry Casimir de Rham.
In 1834, the Watsons sold the farm to Henry Casimir de Rham, a Swiss immigrant who had become a successful New York merchant and banker, for $11,000 ($346,000 in 2009 dollars).[1] He owned and operated the farm as a summer and weekend residence, renaming it Giez, after the town of his birth in Switzerland, until his death in 1874 (after which it passed to his son Charles de Rham).[1]
He was responsible for three major renovations to the main house in the 19th century, which added the Greek Revival porch, the enlargement of the east wing and the addition of a west wing and the mansard roofs on the whole house. The orchard house was remodeled and expanded in 1929. The de Rham family continued to own the farm until 1949, when it was subdivided into the current four parcels, one of which the family kept. The following year, the main house's east wing was demolished. There have been no major changes since then.[1]