The Pitti family, along with the Strozzi family, was one of the most prestigious in Florence, and both of them were competing with the Medici family for supremacy over the city. Cosimo the Elder had increased his economic and cultural power to the point of creating a great many enemies and considerable envy. The wealthy banker, Luca Pitti, wanted to own a palace that was more impressive and lavish than that belonging to the Medici family in Via Larga, which is now via Cavour. He bought large areas of land in front of the church of Santa Felicita in the part beyond the Arno river, and large gardens called "Bogoli", above which there was an ancient rock quarry of pietraforte, where the material used to build the new palace was extracted. It was years before the work began and, in Vasari's accounts, there is evidence that Luca Pitti chose the project presented by Brunelleschi to Cosimo il Vecchio, that had been rejected by himself because it was too "pretentious".
Palazzo Pitti was built under the direction of Luca Fancelli, one of Brunelleschi's collaborators: an impressive, huge two-storey parallelepiped with seven windows per floor, with a huge open space that extended before the entrance door. The Pitti family was so intent upon competing with the Medici family that they soon found themselves in debt and had to stop the building work. It was not until 1549 that Buonaccorso Pitti sold all the property and land to the family that his ancestors had so strongly opposed: the Medici family. It was the beautiful and cultured Eleonora of Toledo, along with her husband Cosimo I, who bought the Palace, extended the land for the garden and entrusted the design of the new Pitti Palace to Niccolò Tribolo. Everything was built in a short time, with great, expert judgement, making Pitti the official residence of the Signori, or Lords, of Florence in the centuries to come.