Cabinet ministers arrive at Downing Street