When Mayor Bloomberg first mused last month about apartment buildings at the World Trade Center site, City Hall aides privately whispered that their boss was fantasizing about a perfect world rather than previewing a plan of action for his second term. But since the election, Mr. Bloomberg has ramped up the apartment-talk and he now seems determined to exert control over the rebuilding process. Apartments are likely after all.

At first glance, Mr. Bloomberg is on a collision course with Governor Pataki – who has been pushing to rebuild the entire 10 million square feet of office space lost on September 11. Mr. Pataki initially said he was “perplexed” by Mr. Bloomberg’s desire for apartments, and just last week told me he told me he doesn’t see “any need for any public discussion beyond what occurs in the normal circumstance.” But the two leaders agree about much more than they disagree about.

Mr. Pataki is concerned primarily with the three most symbolic parts of the redevelopment process: the Freedom Tower, the transit hub, and the memorial. Mr. Bloomberg is firmly on board with these iconic structures. Their dispute involves only the eastern strip of the site along Church Street, which is most easily described as the long block across from the Century 21 department store.

A year ago this month, Mr. Bloomberg signed on to the master plan calling for three office buildings – Towers Two, Three, and Four – to line the street. Mr. Bloomberg now objects to two of those buildings, totaling about a third of the office space planned for the entire site. Mr. Bloomberg argues that Lower Manhattan should be a 24-hour community that doesn’t revolve solely around the financial industry. He also argues that Lower Manhattan simply can’t absorb all the office space the World Trade Center complex once provided.

Notably, none of those objections involves any of the projects nearest and dearest to Mr. Pataki’s personal and political priorities. In fact, the governor’s aides concede Mr. Pataki’s interest in using Towers Two, Three, and Four as office space is based on his concern that the site’s developer, Larry Silverstein, won’t meet other commitments if those office towers – which will theoretically generate substantial revenue some day – go away.

But Mr. Silverstein’s track record inspires something short of 100% confidence. He does deserve incalculable credit for quickly building World Trade Center Seven, on the northern part of the site. While government-led projects stumble and stall, the one building entirely under Mr. Silverstein’s control is virtually finished. That’s a fantastic accomplishment, and Mr. Silverstein has definitively proven that he can build quickly and efficiently. But buildings need tenants, and Mr. Silverstein has announced none – despite generous taxpayer funded cash incentives for any business willing to move in.

Just last weekend, Mr. Silverstein distributed a glossy newspaper insert practically begging businesses to move into his state-of-the art building. Most office buildings line up tenants before construction begins. Mr. Silverstein instead relied on the wishful “Field of Dreams” doctrine – “If you build it, they will come” – that is better suited for the movies than billion-dollar construction projects.

Most ominously, even the Port Authority – which owns the trade center land – won’t commit to moving into any of the new buildings. The Freedom Tower, like Tower Seven, is being built “on spec” with no guaranteed tenants.

Mr. Silverstein finds himself in a difficult situation that is largely not his fault. The developer took over the 16-acre complex just weeks before the September 11 attacks and still pays $10 million a month in rent. But his inability to find any tenants proves Mr. Bloomberg’s point that downtown needs more than office towers.

Mr. Silverstein’s most powerful ally is Assembly Speaker Sheldon Silver, whose district includes the entire World Trade Center complex. Mr. Silver lashed out at the mayor’s demand for apartments and also a hotel by labeling the idea “absurd” and vowing to block any proposal that would jeopardize his district’s status as the world’s financial capital. While Mr. Silver wields enormous power, he might not be able to save Mr. Silverstein.

Earlier this year, Mr. Pataki lost patience with Mr. Silverstein when the developer asked Albany to help pay for security enhancements to the Freedom Tower. When Mr. Pataki retaliated by exploring how the state’s eminent domain power could condemn Mr. Silverstein’s lease, Mr. Silverstein quickly got the hint and stopped complaining.

Now the mayor is the one itching to strip Mr. Silverstein’s control. He has suggested the Port Authority could sell the land under Towers Three and Four to another developer more willing to consider housing options, and the mayor has also accused of Mr. Silverstein of scaring off tenants by charging too much rent. Mr. Silverstein’s alliance with Mr. Silver puts an even bigger bull’s eye on the target.

Mr. Bloomberg is still furious at Mr. Silver for blocking his dream of a West Side stadium over the summer. The stadium may be off the table, but Mr. Bloomberg still envisions a new commercial district on the Far West Side that would be served by an expansion of the No. 7 subway line. Mr. Bloomberg would be quite content to shift office space uptown.

Mr. Silverstein needs to lock up tenants fast for Seven World Trade Center because an empty office tower is the mayor’s strongest argument for shifting from office space to apartments. The mayor just won re-election by a big margin, and the governor is gearing up for presidential politics. If the Freedom Tower, transit hub, and memorial stay on track, look for Mr. Pataki to give Mr. Bloomberg a win on this fight. After all, apartments are a lot less controversial than the commuter tax.

Mr. Goldin is a host of NY1’s “Road to City Hall,” which airs weeknights at 7 and 10:30 p.m.