New York Times Co. shareholders, led by Morgan Stanley, withheld 42% of their votes from directors to protest the Sulzberger family’s control over the company.

An average of 52.5 million of the 124.2 million shares voted declined to support the directors’s re-election, the company announced on its Web site following the annual shareholder meeting in New York.

The withhold tally compared with 28% at last year’s meeting, which marked the beginning of a year-long campaign by Morgan Stanley. The firm and it supporters, concerned about falling profits and a slumping share price, complained that the New York Times’s two classes of stock give shareholders too little say in the company. The chairman and publisher, Arthur Sulzberger Jr., should give up one of his roles, they said.

The vote “is a clear mandate for meaningful change,” Morgan Stanley said in a statement. “The withhold vote this year is significantly higher than last year and is an emphatic call for accountability.”

The Sulzberger and Ochs families own Class B shares, which allow them to appoint nine of the company’s 13 directors. Class A shareholders voted on the remaining four yesterday.

Class A shares of the New York Times, which owns the namesake newspaper and the Boston Globe, fell 21 cents to $23.77 at 4 p.m. in New York Stock Exchange composite trading.

“We understand shareholder frustration as reflected in today’s vote,” Mr. Sulzberger said in a statement released after the meeting. “At the same time, many shareholders have expressed to us that we are pursuing the key actions needed to improve performance and returns to shareholders.”

Shares of the New York Times have fallen 48% in three years. Net income fell 3% in 2004 and 13% in 2005, and the company posted a $543 million loss in 2006 after writing down the value of its New England publications. In the first quarter of 2007, profit dropped 32%.

“Rather than dismantling the dual-class structure, shareholders may be pushing instead for changes at the board and among top executives,” the vice president at Glass Lewis & Co., a shareholder advisory firm in San Francisco, Robert McCormick, said. “A more reasonable expectation is that New York Times separates the roles of publisher and chairman.”

The vote brought to a head a campaign by Hassan Elmasry, managing director of Morgan Stanley Investment Management. He first made his complaints public in April last year, sending a letter to the New York Times board criticizing its reliance on two classes of stock to maintain family control.

Morgan Stanley, with 7.1% of the company, convinced T. Rowe Price Group Inc. and Private Capital Management to withhold their votes last year. The three holders combined now have about 31% of the Class A shares.

Shareholder advisory firms Glass Lewis and Institutional Shareholder Services joined the dissidents this year, helping increase the withhold tally.

Private Capital Management spokesman Chad Atkins didn’t return calls for comment yesterday. Monday he declined to discuss how the firm would vote. Neither would T. Rowe Price spokesman Brian Lewbart nor a spokesman for Mr. Elmasry, Steve Frankel.

Sulzberger said his family is “firmly and unanimously committed” to the company’s two classes of shares.

“Anyone who purchased New York Times Company stock did so with full knowledge and understanding of how the company was structured,” Mr. Sulzberger said in a speech at the meeting.

Filling the role of chairman and publisher “allows me to balance the financial and journalistic needs of this institution,” Mr. Sulzberger said.

Mr. Sulzberger defended the company’s spending, which had been criticized by Mr. Elmasry. The $410 million purchase of About.com in 2005 helped buoy revenue growth and the company’s $500 million investment on a new headquarters is now worth more than $1 billion, Mr. Sulzberger said.

Directors up for re-election by Class A shareholders yesterday were Commercial Worldwide LLC managing partner, Raul Cesan; former Federal Communications Commission chairman, William Kennard; former Gillette Co. chief executive officer, James Kilts; and Verizon Communications Inc. chief financial officer, Doreen Toben.

Dow Jones & Co., Washington Post Co. and McClatchy Co. also have two classes of stock.

The chief executive officer of the Washington Post, Donald Graham, defended the dual-class structure in an opinion piece yesterday in the Wall Street Journal. Supporting Morgan Stanley’s campaign against the New York Times “is to run crazy risks with the future” of the New York Times newspaper, Mr. Graham wrote.

The New York Times won support from Emigrant Bancorp, which bought its 4.4% stake in the fourth quarter and will vote for the directors, the vice chairman of New York Private Bank & Trust, Emigrant’s parent company, John Hart, said.

A Chicago-based manager of the $635 million ABN Amro Mid-Cap Fund, which owns 1.04 million shares, Thyra Zerhusen, said she supports management.

“I did not buy the stock expecting the dual-class structure would be eliminated,” Ms. Zerhusen said.

The Sulzbergers have controlled the New York Times for more than 110 years. Arthur Sulzberger Jr.’s great grandfather, Arthur Ochs, bought the New York Times in 1896.

“The Sulzbergers as a family are united, and if they’re united about anything, it’s that they’re not going to let some folks at Morgan Stanley push them around,” the author of “The Trust: The Private and Powerful Family Behind The New York Times,” Alex Jones, said.

[The pension fund for the State of New York had about 458,000 shares in the New York Times Company stock, as of earlier this month, worth more than $10 million. A spokesman for the state comptroller’s office, which oversees the pension fund, told the New York Sun that the fund voted in favor of all of the directors on the board.]