When the Whitney Museum of American Art announced last fall it would expand into the former Dia Art Foundation site at the base of the High Line, all talk was on the future of downtown. Back on the Upper East Side, meanwhile, observers were amazed that the venerable museum’s third attempt to expand uptown had fallen flat.

The real story of the Whitney’s failed bid to expand uptown has to do with the valuable properties it owns in the neighborhood. The four historic brownstones on Madison Avenue and two town houses on East 74th Street have been the targets of failed efforts by architects Michael Graves in 1985, Rem Koolhaas in 2001, and, most recently, Renzo Piano.

Mr. Piano navigated the city’s approvals process last year, succeeding despite significant neighborhood opposition and a series of compromises. But the Whitney was disappointed with the small amount of gallery space that the resulting plan would have afforded — just 30,000 square feet — and ended up rejecting the plan worth $200 million. Instead, the museum has decided to buy two sites on Gansevoort Street in the meatpacking district. The new downtown expansion, also designed by Mr. Piano, is expected to cost less and contain more than twice the gallery space of the uptown plan.

The Whitney is now left with several valuable buildings on the Upper East Side that it had planned to use for its expansion and now no longer needs. The 47,496 square feet of space and 7,635 square feet of retail could sell for upward of $65.5 million, according to a broker at Prudential Douglas Elliman, Jeff Tanenbaum. The buildings are on “one of the most celebrated and sought after streets in the entire world,” he said of Madison Avenue.

The frontage along the avenue, which dates almost entirely from the 1870s, has “stunning neo-Grecian style,” and he estimated that the buildings cost roughly $1 million each when the museum purchased them in 1983. The fact that the buildings had been owned by the Whitney could increase their value, he said.

A museum spokesman, Stephen Soba, wouldn’t speak about the buildings’ futures, but confirmed the museum is currently deciding their fate.

The buildings are also valuable because of the Whitney’s long habit of offering short-term leases at below-market rates to its Madison Avenue retail tenants, always preparing for the possibility that an expansion could lead it to suddenly terminate the leases. This arrangement increases the buildings’ values because new landlords would not be stuck with long-term tenants and could remarket the spaces at higher rents. If the Whitney decides not to sell these properties, it too could entice luxury tenants to sign long-term leases at considerably higher rents than it currently receives.

There are some questions surrounding the fate of these Upper East Side buildings. Namely, the city granted permits to the Whitney for its Renzo Piano-designed additions. These permits are still valid and could be tempting to a future buyer.

The Landmarks Preservation Commission issued a Certificate of Appropriateness permitting the Whitney to build a 178-foot, nine-story tower and sever the back portions of four brownstones. The permit, emphasizing the “institutional campus” of the museum’s plan, could potentially be executable by future owners who demonstrated similar cultural uses — or even related educational ones, such as an art school. At the least, it would be significantly easier for such a buyer to build on these sites because Landmarks spent years vetting the Whitney’s plan.

The existence of these permits could be expected to push prices “way higher than I even considered,” Mr. Tanenbaum said.

There is a catch, however. Landmarks’ Elisabeth de Bourbon cautioned that the permit expires six years after the date of approval, or January 2012, so the clock is ticking. Also, the Board of Standards and Appeals granted seven variances last year at the site, which would presumably have to be obtained anew. This might be difficult, as no petitioner would be able to boast ownership of the adjacent museum. Despite these problems, there are unique legal prospects for building on the Upper East Side, a neighborhood that is notoriously unfriendly to new development.

Don Gringer, president of the Coalition of Concerned Whitney Neighbors, the residents’ association that sued last year to challenge the Piano plan’s approval, is concerned about the future use of these permits. Mr. Gringer, an engineer by profession, also takes issue with the Whitney covering up the brownstones with black plastic netting, concerned that the netting may have damaged the buildings’ facades.

The executive director of the Upper East Side Historic District, Seri Worden, said she considers new construction on the site of Whitney’s brownstones to be a long shot, but may support such a move. “We’re not opposed to all contemporary architecture,” she said.