Properties on the site were owned by the Rossi family since the 13th-century, and combined in the 18th-century in to the present building. The Rossi family branched out through Italy, and Porzia de Rossi, daughter from a Rossi living in the Kingdom of Naples was the mother of the author Torquato Tasso. Bocchino di Girolamo Rossi was a mathematician in Siena in 1593, and Andrea d'Antonio Rossi was dean of the University of Pisa in 1608. Giulio Rossi was the bishop of Pistoia in 1804.
The palace on the northwest of the site was designed circa 1749 by P. Raffaello Ulivi and completed by Salvadore Piccioli by 1795. It was situated on the family's ancient quarters and tower, located adjacent to the gate of Sant'Andrea and fortress of San Jacopo in Castellare located in the first circle of walls. The palace once displayed a stone head supposedly depicting the infamous Filippo Tredici, much like the bust outside the Palazzo degli Anziani.
The interiors are frescoed and decorated with stucco. The stairwells (1760) were stuccoed by Tommaso Cremano and frescoed by Meucci. Some of the rooms were frescoed by Bartolomeo Valiani, with landscapes by Fini. The ballroom has a fresco depicting the opening dance of the Decameron, painted in 1838 by Giuseppe Bezzuoli. The adjacent palace was designed in 1830 by Alessandro Gherardeschi. It was frescoed by Niccolo Monti, and stuccoed by Ferdinando Marini and sculptures by Francesco Carradori.[2]