Sir David Freud has said he left his position as a government adviser because the Conservatives made him an offer he could not refuse.
He told the BBC that his decision was not based on party politics, and that he saw himself as "apolitical".
Sir David was a key adviser on welfare reform to work and pensions secretary James Purnell, but will now join the Conservative frontbench in the Lords.
A former investment banker, he advised Purnell on how to use the private and voluntary sector to help reduce unemployment.
But he will now help any future Cameron administration deal with unemployment levels which are expected to exceed three million.
Sir David has been credited as the leading architect of Labour's plans to shake up the benefit system.
A Conservative source claimed: "He feels very much that through this recession, welfare is going to be one of the big challenges in the next few years and he would like to be part of the response to it, hopefully in a Cameron administration."
A spokesman for the Department for Work and Pensions confirmed that Sir David had quit as an adviser.
"We can confirm we discussed this matter in advance with David Freud and he has left his role as an adviser to the department by mutual consent," said the spokesman.
"We thanked him for his important work in building consensus around the Welfare Reform Green Paper in July and wished him well for the future."
Sir David told the BBC: "I did it because the opportunity that David Cameron offered seemed to me to be exceptionally interesting and challenging at an extraordinarily difficult time. I thought I should take it."
He denied that the move was politically motivated. "I have been an independent, apolitical adviser," he explained.
But he admitted: "I suppose I was an adviser to two Labour parties – first it was the Blairite side, and then I joined the Brownite side.
"I was working for James Purnell. I said I would do that for a year and help him get the white paper out. With that completed, I looked for the next thing."
But Sir David noted that opinion polls were suggesting a Conservative government at the next election.
"The opinion polls would suggest that they have got an extremely good chance of being the next government," he told the BBC.
