As Olympic inspectors begin their tour of London today, Ken Livingstone is still refusing to apologise for an allegedly anti-Semitic remark.

The Times says the mayor's refusal to say sorry is distracting attention from the four-day visit which concludes with a dinner hosted by the Queen at Buckingham Palace.

At a press conference yesterday Livingstone repeatedly refused to apologise for telling a Jewish reporter that he was behaving like a concentration camp guard.

Tessa Jowell, the culture secretary, is said to be among a growing number of politicians dismayed by Livingstone's conduct.

A senior official in her department said last night: "This is carrying on much longer than we would have hoped. We would not want this sort of distraction in such an important week."

Sir Gerald Kaufman, chairman of the Commons culture select committee, also voiced his concerns.

"With Madrid disfigured by the recent Eta terrorist atrocity and Paris potentially handicapped by what Muslims regard as France's anti-Islamic prejudice, London really has a chance to shine. Will Ken's lippy remarks tarnish that sheen?" he said.

The Olympic inspectors are due to meet Tony Blair and Livingstone in Downing Street on Friday.

Lord Moynihan, the former sports minister, claimed that the mayor had harmed Britain's chances of staging the Games.

"This is a serious matter which will do the bid no good particularly as the Olympic committee is here. The mayor should apologise. He should have done so already," he said.

Kate Hoey, the former Labour sports minister, added: "The mayor has behaved in a typically arrogant way. He should realise that if he wants London to win the bid he should swallow his pride and say sorry."