Upper East Side residents who want to stop the Whitney Museum of American Art’s planned expansion have filed suit against the city, hoping to rewind the approval process and force the museum to scale back its plan.

The plaintiffs — two associations of residents and the owners of the Carlyle Hotel — claim that the Board of Standards and Appeals erred in granting the museum variances to zoning regulations, in order to allow the Whitney to go forward with its expansion, which includes a 178-foot stainless-steel-clad tower designed by the award-winning architect Renzo Piano.

“They granted seven variances; it’s almost as if they rewrote the zoning law just to fit the Whitney,” a member of the Coalition of Concerned Whitney Neighbors, a party to the lawsuit, Edward Klimerman, said.

The Whitney has undergone a long process to get approval for its expansion, which it claims it needs in order to create more gallery space, as well as an auditorium, classrooms, and a conservation lab. Mr. Piano’s tower will be set behind six brownstones on Madison Avenue between 74th and 75th streets, which are owned by the Whitney, and connected by glass bridges to the original 1966 Marcel Breuer building. One brownstone, which, within the Upper East Side Historic District, is not a “contributing” building — that is, not deemed by the city as artistically important to the character of the neighborhood — will be demolished; the Whitney will reduce the five others in depth to make room for the expansion. The Whitney’s plan was approved by the New York City Landmarks Preservation Commission in 2005 and by the Upper East Side Community Board in January. The BSA granted the variances on July 25.

In interviews, individuals involved in the suit were most vociferous in their objections to the height and appearance of Mr. Piano’s tower. “This is an 18-story filing cabinet, a metal building with basically no windows whatsoever, standing amid beautiful townhouses,” the president of the Coalition of Concerned Whitney Neighbors, Donald Gringer, said. “A steel-paneled building really belongs in an industrial park,” he added.”I wouldn’t up a building like that in the Bronx,” where he owns a factory. “He’s putting it up in a beautiful landmarked district.”

“The scale and design of this does not fit into the historic district,” a cochairwoman of the Defenders of the Historic Upper East Side, Elizabeth Ashby, said. “It’s a windowless, metal-clad tower.” She added, “One of the things that we suggested is, since they don’t have any windows, they could put that underground.”

The suit devised by the groups’ lawyers, however, relies not on aesthetic arguments but on legal ones. The plaintiffs claim that the BSA gave the museum undue special treatment and failed to adequately scrutinize its variance application.

They argue that the BSA gave the museum a level of deference due only to educational and religious institutions, by accepting the museum’s claim that its programs, to which much of the new space would be dedicated, made it an educational institution.

The plaintiffs also claim that that the museum’s description of its space needs was “incomplete, vague, and unsubstantiated,” and that the BSA didn’t have a reasonable basis to evaluate whether these needs could be met without violating zoning regulations. Lastly, they argue that the museum took advantage of its nonprofit status to build an expansion that will include for-profit components.

A spokeswoman for the city’s legal department said, “We have not yet received the papers, but we will evaluate them thoroughly.” A spokeswoman for the Whitney, Jan Rothschild, said the museum was also not ready to comment on the lawsuit, as no one there had seen the papers, either.

The plaintiffs hope that the Supreme Court of the State of New York will annul the BSA’s July 25 decision, and that the museum will be forced to design a smaller expansion. “We really see no reason why it cannot be reduced in size,” a co-chairwoman of the Defenders of the Historic Upper East Side, Teri Slater, said.

“I would like to see them build a smaller addition, go further underground if possible,” Mr. Klimerman said.

Both Ms. Slater and Mr. Klimerman saw the Whitney as an example of a citywide trend of institutions expanding and compromising historically residential areas. Ms. Slater cited NYU’s expansion of its facilities in Greenwich Village, medical institutions on the Upper East Side, and Columbia University in Harlem. “The city should ensure its residents that the residential areas are going to remain just as they are,” Ms. Slater said. “They’re areas where people live and rest and recharge. They’re not supposed to be developed the way the rest of the city is, the more commercial areas.”

“I am very concerned as a citizen of New York with this institution expansion,” Mr. Klimerman said. “Where do you draw the line, with these schools and religious institutions and hospitals and museums? Soon they’ll be no place to live.”

Some Upper East Side residents and preservationists now support the Whitney’s expansion, since the museum abandoned an earlier plan that would have required demolishing a second, more stylistically significant brownstone. “It will be a startling building, there’s no question about it, but I think Piano is a distinguished museum architect,” a board member of the Friends of the Upper East Side Historic District, Franny Eberhart, said. “It has its approvals,” she added, “so it will be interesting to see if they have any success with this lawsuit.