Since arriving in Manhattan to attend Columbia Business School in 1972, Mallory Factor has been solidly ensconced in New York City. In 1999, he bought a $6 million, 8,750-square-foot townhouse on East 71st Street. Governor Pataki has appointed him to some of the most powerful public boards in the state, including a stint as chairman of the New York Public Asset Fund, overseeing billions of dollars in state funds and lucrative contracts to Wall Street firms. He’s a trustee of the Brooklyn Academy of Music and a member of the administration committee of Broadway’s Tony Awards. And he’s a founder of the “Monday Meeting,” the invitation-only monthly gathering of the city’s conservative elite and Washington power players that’s been profiled in New York magazine, the Washington Post, and the New York Observer.

From his base in New York, he emerged as a force in national politics, a conduit between Manhattan money and national Republican politicians. The more than 50 fund-raising dinners he hosted at his townhouse drew dozens of senators and top officials at the White House.The Bush campaign’s Web site listed him as a “ranger,” one of the top-level fund-raisers who collected $200,000 or more in political donations. White House officials held out the possibility that he could join the administration as an ambassador if he met multimillion-dollar fund-raising goals in 2002 and 2004, according to a former colleague of Mr. Factor’s.

So it took many by surprise when Mr. Factor this summer decided to pack up and leave town.With his family, Mr. Factor has decamped to Charleston, S.C., where he has his eye on a prime piece of real estate, an 11-bedroom mansion known as “The Governor’s House” that was once the home of a signer of the Declaration of Independence, Edward Rutledge. It’s possible he won’t ever move back to New York, he acknowledges.

In a recent interview with The New York Sun, Mr. Factor, 56, dismissed widespread speculation that he’s moving to South Carolina as a result of financial pressure. Although his closest associates say they don’t know for sure how Mr. Factor earns his income, Mr. Factor says he isn’t in a cash crunch. Wearing alligator-skin loafers and a handmade suit from Kiton, an exclusive fashion house in Naples, Mr. Factor during the interview took a call on his cell phone to confirm his dinner plans at Nobu 57 that night.

Mr. Factor offers a simple reason for his departure: Taxes and the cost of living are too high in New York, he says.”I don’t have to pay double for a loaf of bread for the privilege of living in New York,” he said. He and his wife Elizabeth, a tax attorney and Yale Law School graduate from Alabama who encouraged Mr. Factor to become a fundraiser five years ago, agreed that Charleston would be a safer and more comfortable place to raise their three children. Mr. Factor said he is also friendly with South Carolina’s senators and governor, who all slept over at his Manhattan townhouse.

While bread may be cheaper in Charleston, Mr. Factor has left behind many of the trappings of his life in Manhattan. Gone is his Madison Avenue office, a monochrome black suite (“That’s my personality,” he explained) where he worked as a merchant banker and conducted much of his political operations. Gone is his townhouse, which he sold for $9.5 million, about $3.5 million more than his purchase price. Gone is the upper-floor guest area in the townhouse — dubbed “the Norquist suite” after his friend and frequent guest Grover Norquist — which doubled as an overnight shelter for a large chunk of Congress over the years, according to Mr. Factor. Senator DeMint of South Carolina, Senator Graham of South Carolina, Senator Coleman of Minnesota, Senator Talent of Missouri, Senator Smith of Oregon, Senator Bunning of Kentucky, Senator Allard of Colorado, and Senator Crapo of Idaho all stayed over at the townhouse. Mr. Factor has fond memories of Governor Sanford of South Carolina raiding his freezer for ice cream and of testing knowledge of sports trivia with Mr. Pataki.

Mr. Factor, who visited New York last week, conducted this interview in the lobby of the Grand Hyatt in Manhattan.

Mr. Factor’s move is the latest twist in the life of a man who grew up in a low-income housing project in a tough neighborhood in Bridgeport, Conn., called Success Park and was the first person in his family to graduate from college.

Mr. Factor said most of his income comes from his work as a merchant banker. He invested in a data collection company called Acsis that was sold to Safeguard Scientifics last December for $26 million. He advised a company called Eclipse Surgical Technologies that specialized in using lasers to drill tiny holes in heart tissue. In another deal, he amassed 17 medical office buildings on the East Coast and refinanced them for a profit.

Although some of his closest associates, including Mr. Norquist and Mr. Factor’s Monday Meeting co-founder, James Higgins, take him at his word, the question on the minds of many who know him is why a man who has so diligently cultivated a reputation as a Manhattan power-player is taking his show on the road.

“I told Mallory I thought it would potentially diminish his political impact. He assured me it’s not going to happen,” said Stephen Moore, the founder of the supply-side Club for Growth PAC who last year recruited Mr. Factor to serve as chairman of the Free Enterprise Fund, an advocacy group hatched by Mr. Moore after his PAC splintered in a power struggle.

Others suspect that Mr. Factor, in choosing to set up a new base in South Carolina, is looking for greener political pastures at a point when his political influence in New York may have had hit its peak.

Polls show that if an election were held now, a Democratic administration led by Eliot Spitzer would take over in Albany. Mr. Factor speaks of Mr. Pataki, whom he calls a “wonderful person,”as a politician who led a state government that disappointed the expectations of fiscal conservatives. “You have a governor who was absolutely brilliant during his first term. As the years went on things started to change a bit,” Mr. Factor said. He blames the state Senate and Assembly for overriding the governor’s vetoes. New York, Mr. Factor says, is one economic downturn away from fiscal disaster.

Asked about Mr. Factor’s bleak view of the state’s fiscal health, a spokesman for the governor, David Catalfamo, said in an e-mail, “Even the most cynical self-serving critic can’t help but acknowledge the transformation of a city and state from the brink of catastrophe and despair to a bastion of renewed opportunity and hope.”

While the Monday Meetings are as large as ever — attendance is now at around 500 — there’s a growing sense of disillusionment among participants and organizers about the meeting’s ability to affect policy changes in Washington.The disappointment, Mr. Factor says, speaks to the broader disenchantment with the record of the Bush White House and Congress. “We controlled the House, the Senate, the executive branch,” Mr. Factor said. “We should have been able to do more to create a lot more personal freedoms.”

The disappointment with the Bush administration may also be personal for Mr. Factor, according to Mr. Moore. He says Mr. Factor approached senior White House officials about his interest in an appointment. “He was talking about trying to get something in Bush’s first term,” Mr. Moore said. White House officials suggested to Mr. Factor that he “go out and raise $2 million for Senate candidates” in 2002 and two years later asked him to raise $4 million, Mr. Moore said.

“The Bush people totally screwed him,”Mr. Moore said.”Both in 2002 and 2004, he did all the things they told him to. He was straightforward that he wanted to be part of this administration. I never understood why they would stab him in the back. The Bush administration is a very inbred kind of culture. It’s very difficult for an outsider like Mallory to be part of that.”

Mr. Factor denies that he ever actively sought out an ambassadorship and says White House officials never linked Mr. Factor’s fund-raising and the potential for an appointment. “That’s illegal. There is not a quid pro quo. That is absolutely illegal,” Mr. Factor said. “You can’t do it. You may not like what this White House does, but they’re not stupid. They’re not going to do something like that for a couple of bucks.”

“The president has a consistent record of nominating and appointing the most qualified people to serve in his administration,” said a White House spokesman, Peter Watkins.

Mr. Factor said he didn’t become a fund-raiser to become an ambassador but out of a desire to help the Republican Party and “change the world a little bit.” When his daughter was born days after the September 11 attacks, the first words he said to the infant were a promise to make the world a better place, he said.

Mr. Factor said he’s on good terms with the Bush administration, pointing to Karl Rove’s attendance at a fundraising dinner he gave in June for Senate candidate Mark Kennedy.

He insists that the fund-raising empire and political network he built in New York will remain intact, despite his absence. He says he plans to fly up to New York once a month for oneweek visits to emcee the Monday Meetings with his friend James Higgins, a hedge fund manager and former chairman of the College Republicans who co-founded the event with Mr. Factor under the guidance of Mr. Norquist.The visits will allow Mr. Factor to attend another one of his policy roundtable creations,”the Conspiracy Dinner,” a much smaller and lesserknown meeting of conservative minds whose regular guest list includes an Opinionjournal.com columnist, John Fund, and an economist and CNBC television host, Lawrence Kudlow.

“New York is not going to be a foreign place to me.They didn’t revoke my visa,” Mr. Factor said.