A Labour peer has warned the prime minister he is leaving it too late to campaign for a "yes" vote in the EU constitution referendum.