Everyone knows Manhattan is all about high-rise condos, tall apartment buildings, and any other kind of building in which people live above other people. But it wasn’t always that way. A hundred years ago, there was still room on this small island for the ultra-rich to build mansions all to themselves, single-family homes with the square footage of a castle. Today many of these buildings, all “Millionaire’s Row” mansions in the Upper East Side, belong to museums and schools, but the question remains: What are the biggest buildings in Manhattan today that were built as single-family homes?

Warburg House (Jewish Museum of New York) – 82,000 square feet

C. P. H. Gilbert designed this house for Felix Warburg, a German-born banker, in 1908, and it has remained the largest mansion in Manhattan for over a hundred years. This is partly thanks to expansions added in 1963 and 1993, but the Warburg House was undoubtedly a tremendously large building for a single family in New York. Warburg’s widow donated the building to the Jewish Museum in 1944, who still occupy the building today.

Carnegie Mansion (Cooper-Hewitt National Design Museum) – 56,368 square feet

The famous Scottish-born steel man Andrew Carnegie lived in the second-largest mansion in Manhattan, built in 1903. The building is so prominent that the surrounding Upper East Side neighborhood is called Carnegie Hill.

Carnegie reportedly asked for the mansion to be “the most modest, plainest, and roomiest house in New York,” which seems ridiculous when you look at the gorgeous Georgian facade. The building now houses the National Design Museum, part of the Smithsonian family of museums, currently under renovation.

Otto Kahn House (Convent of the Sacred Heart) – 50,316 square feet

Modeled after the Palazzo della Cancelleria in Rome, the Otto and Addie Kahn Mansion was finished in 1918. Kahn was a German-Jewish banker like Warburg, but after his death, the house was sold to the Convent of the Sacred Heart, who still use the building as a Catholic girl’s school today.

As you might expect from a school housed in a historic Upper East Side mansion (actually two historic mansions, with the neighboring James Burden House being the other), school tuition is sky-high: Sending your child there for a year in third grade would cost $42,810. In “A Very Gaga Thanksgiving,” Lady Gaga talked at length about her education at Sacred Heart on national TV, a big advertisement for a school with about 700 total students.

James Duke House (New York University) – 31,089 square feet

James Buchanan Duke (as in Duke Power) had this mansion built in 1912, and it’s a relatively modest 30,000 square feet. The building was inspired by the Hotel Labottiere in Bordeaux.

The building was donated in 1952 to New York University’s Institute of Fine Arts, who still occupy both the Duke House and the Stephen Chan house across the street.

Image © PropertyShark

Harkness Mansion – 21,700 square feet

The Harkness Mansion is the only entry on this list that isn’t currently occupied by something other than a tycoon’s family — though its days in that regard may be numbered. Gallery owner Larry Gagosian bought the building a few years ago for $36.5 million, with a lot of renovation to do to turn the building into much of anything, whether that will be a private residence, a gallery space, or something else entirely.

The Harkness Family, the building’s namesake, got their money from founding Standard Oil. This Harkness Mansion is at 4 East 75th Street, but he also owned another smaller building just across the street at 1 East 75th Street which is called the Harkness House, now the home of the Commonwealth Fund.

All images via Wikicommons unless otherwise noted