The Museum of Modern Art has occupied five buildings in its history. From its 1929 founding until 1932 the museum occupied space in the Heckscher Building, at the southwest corner of Fifth Avenue and 57th Street, a Beaux-Arts skyscraper by Warren & Wetmore. In 1932, MoMA moved to a limestone townhouse, courtesy of John D. Rockefeller Jr., at 11 West 53rd Street. When MoMA decided to replace the townhouse with its first purposebuilt building, the museum moved for two years into the Time & Life Building at Rockefeller Plaza and 49th Street. In 1939, the new building opened on 53rd Street. (The fifth building was MoMA QNS.)

The townhouse of 1932 was much like countless row houses that once pervaded the west side of Midtown. A huge number of these were torn down in the 1930s to make way for Rockefeller Center. Alfred H. Barr Jr., MoMA’s first director and presiding genius, wished for his museum not merely to reflect the currents of Modern art and design but to direct those currents. To that end, he desired not only that MoMA have a permanent home but that the building itself express MoMA’s mission. Most people know Barr wanted Ludwig Mies van der Rohe, late of the Bauhaus in Germany, to design the building. MoMA’s trustees, however, wished to keep the job in the family – though they equally desired a Bauhaus-inspired structure. The job went to Philip Lippincott Goodwin, himself a MoMA trustee, and Edward Durell Stone, who had worked for Wallace K. Harrison, the Rockefellers’ favorite architect, who was then engaged on Rockefeller Center.

Goodwin was known as a traditionalist architect, and had even attended the Ecole des Beaux-Arts. He was the son of the banker James Junius Goodwin and had grown up in his father’s spectacular McKim, Mead & White-designed house, which still stands on the north side of 54th between Fifth and Sixth Avenues, right across the street from where Philip’s contemporary, John D. Rockefeller Jr., grew up. If Goodwin began as a traditionalist but turned Modern, Stone was the opposite. He started out as an International Style Modernist, but in mid-career developed a personal, romantic style that diverged sharply from orthodox modernism. In the 1960s, Stone designed the Gallery of Modern Art at Columbus Circle, founded by Huntington Hartford specifically to counter the influence of MoMA.

Barr may have considered it a defeat that he could not get his board to award the job to Mies. But Modernist critics, such as Lewis Mumford in the New Yorker, lavishly praised Goodwin’s and Stone’s building. And Hilton Kramer has pointed out that it may have been a blessing that Mies did not get the job: “the two most conspicuous museum facilities which Mies was given the opportunity to design in later years – the Brown Pavilion addition to the Museum of Fine Arts in Houston and the National Gallery in West Berlin – have proved, as spaces in which to exhibit works of art for public viewing, to be among the very worst on the international museum scene.”

If you look at a Manhattan map that has building lots marked on it, you will see something you may not have noticed as you have walked the streets of Midtown. NewYorkers know that when Rockefeller Center rose in the 1930s, it included a new north-south street, called Rockefeller Plaza, extending from 49th to 51st Streets about a third of the way from Fifth to Sixth Avenues. Now locate MoMA, the second building in from the east on 53rd Street between Fifth and Sixth. You’ll see that it is right on axis with Rockefeller Plaza; only two blocks intervene between it and the northern terminus of Rockefeller Plaza.

The tall building on the north side of 51st opposite Rockefeller Plaza was originally the Esso Building and later the Time Warner Building. Clad in the same striated Indiana limestone as Rockefeller Center’s surrounding buildings, it looks at first as though it had been built at the same time as the others. In fact, it postdates the last of the original Rockefeller group by seven years. Nelson A. Rockefeller, working with his father on developing the center, wished to keep open the possibility that Rockefeller Plaza might be extended northward by two blocks to form an axial pathway to MoMA’s front door. In time, he thought he might be able to acquire the intervening block, 52nd to 53rd Streets. When he did so, he thought he might build homes along Rockefeller Plaza for the Metropolitan Opera Company, Solomon Guggenheim’s Museum of Non-Objective Art, and the Columbia Broadcasting System – a cultural component of Rockefeller Center prefiguring Lincoln Center, which Nelson’s brother John would help to develop some years later.

The Rockefellers did acquire the south side of 53rd, where the Donnell Library now stands. All Nelson then needed was the north side of 52nd, where the 21 Club blocked his path. This he was never were able to acquire. The famed watering-hole started out on 49th Street and had already been uprooted once by the Rockefellers. The owners, Jack Kriendler and Charlie Berns, did not wish to move again. The Rockefellers may have been the most powerful family in the city, but Jack and Charlie were not slouches in the influence department: Their devoted clientele included many of the city’s top businessmen and politicians, if not any of the abstemious Rockefellers. When finally the Rockefellers jettisoned the idea, they erected the Esso Building, which extends north to 52nd Street.

In 1939, MoMA, with its white marble, gleaming stainless steel, and translucent Thermolux facade, could not have seemed more incongruous on its brownstone block. While the brown stones were set back behind stoops and small front gardens, MoMA went out to the street line. It must have been a startling sight.

When MoMA opened in 1939, the New York Times called the building “an example of advanced modern architectural design.” MoMA expanded in 1951, 1954 (the Abby Aldrich Rockefeller Sculpture Garden), and 1964 (including an enlarged Sculpture Garden), all under the direction of Philip Johnson, who, before he became an architect, had served as MoMA’s curator of architecture. (When Mr. Johnson and the architectural historian Henry-Russell Hitchcock presented MoMA’s “Modern Architecture: International Exhibition,” whence derived Alfred Barr’s coinage “International Style,” in 1932, MoMA still resided in the town house on 53rd.) In 1984, the museum underwent a massive renovation, together with the construction of the 52-story Museum Tower apartment building, designed by Cesar Pelli.

Yoshio Taniguchi’s current expansion marks the fifth major revision of MoMA’s fourth home. MoMA, which in 1939 had stood out so glaringly and gleamingly from its old surroundings, has by now well-nigh put those surroundings to rout.