Palazzo Pandolfini was one of the first Renaissance palaces to be built in a part of the city that was considered partly for agricultural use. It was built in the 15th century and had a small "garden of delights" and a large cultivated vegetable garden. According to what Vasari tells us, it was Raffaello Sanzio who designed the building, probably in 1514, for Bishop Giannozzo Pandolfini. The garden, with its lush plants, had several fountains with water features, gifts from Pope Leo X.
At the end of the 19th century, the palace enjoyed a considerable period of splendour: the exterior was transformed by Eleonora Pandolfini into a romantic English-style garden. However, it was thanks to Count Alessio's wife, Sophronia Stibbert, Eleonora's daugther-in-law and heir, that the garden became famous for its collections of camellias and cineraria plants, some of which are botanical rarities that received awards at the end of the 19th century from the Botanical Society of Horticulture. Sophronia Stibbert was the sister of Frederick Stibbert, a great art collector who gave the city of Florence the museum of the same name. Her son, Roberto Pandolfini, built a greenhouse for orchids, above the winter garden, for his wife Beatrice Corsini. This was the last modification made to the splendid residence of Palazzo Pandolfini.