NEW YORK (AP) – The Ochs-Sulzberger family is pulling most of the assets they own out of Morgan Stanley after one of the investment bank’s fund managers sharply criticized their financial oversight of The New York Times Co.

Last year Hassan Elmasry, an investment manager for Morgan Stanley, withheld votes for several directors at the Times’ annual meeting in protest of the company’s dual-class share structure, which allows the Sulzberger family to maintain control of the company through a special class of shares with powerful voting rights.

Criticizing the poor performance of the company’s shares, Elmasry said at the time that the dual-class structure fostered a “lack of accountability” to the company’s public shareholders. The company says the structure is necessary to preserve the editorial integrity of The New York Times.

Elmasry also tried to get a shareholder proposal on the company’s proxy statement for this year that would have recommended putting the dual-class share structure to a shareholder vote.

But the company, with the blessing of the Securities and Exchange Commission, declined to allow the proposal, noting that the only way to change the share structure is by a vote of the Sulzberger family members, and they have indicated they don’t plan to do so.

New York Times spokeswoman Catherine Mathis said in an e-mailed statement that custody of the “majority of the assets” belonging to the Ochs-Sulzberger family was being moved from Morgan Stanley to another institution, but she declined to give the size of those assets or say which institution they were being moved to.

Fortune magazine reported the news earlier Friday of the Sulzbergers move, saying the value of the Sulzbergers’ assets was close to $640 million.

Christy Pollack, a spokeswoman for Morgan Stanley, declined to comment.

Like several other newspaper companies, the Times is publicly traded but controlled by a family through a special class of stock. Other family-controlled and publicly traded newspaper companies include Dow Jones & Co., which publishes The Wall Street Journal, The Washington Post Co. and McClatchy Co.

Shares of many newspaper publishers have slumped over the past few years as more advertising dollars and readers move to the Internet. The Times’ shares, which traded in the high $40s in early 2004, rose 66 cents, or 2.81 percent, to close at $24.17 on the New York Stock Exchange.

Last year the second-largest newspaper publisher in the country, Knight Ridder Inc., was forced to sell itself under pressure from shareholders, and Chicago-based Tribune Co., the third-largest newspaper company, is also considering a sale following shareholder protests.

Janet Robinson, the CEO of the Times, told an investor conference last December that the Sulzberger family “has no intention of opening our doors to the kind of action that is tearing at the heart of some of the other great journalistic institutions in our country.”