Barker is leaving after 24 years while team captains Matt Dawson and Phil Tufnell will also depart as part of a major shake-up. BBC bosses are said to be looking at making the panel more diverse – with Alex Scott rumoured to be in the frame to take over from Barker.
But Carson, 77, who appeared with Bill Beaumont as a team captain in 1982, questioned the decision to replace the whole panel.
He said: “The BBC are on dangerous ground because these are big shoes to fill. Very big shoes to fill.
“It seems a little bit harsh in a way, because it seems to be a very successful team. I’m sure most people were very happy the way the format is at the moment. I certainly am.
“They have great chemistry between the three of them – it seems a shame to lose that.”
On rumours BBC bosses want to have a more diverse panel, he said: “Oh my God, we’ve got to have Black Lives Matter…so do white.”
But he added: “I always say: it doesn’t matter what colour, what creed, whatever…I am not racist in any way…always the best man for the job is who you should get, it doesn’t matter what they are. Or female, male… it doesn’t matter. You put the best person for the job, always.”
Carson was particularly scathing of the BBC middle management for the decision to axe Barker, 64.
He said: “People sitting at a desk think they can do better.
“The racing term is: people in the stands can ride the horse better than the fellow on top, the jockey. That’s what’s happening. A fellow sitting in a comfy chair in an office thinks they can do better.”
Carson’s comments come as Alex Scott thanked fans for their support following online abuse after she was linked to the programme.
The pundit and former footballer, 35, played down suggestions she had already signed up to replace Barker.
Her comments came after Match Of The Day host Gary Lineker congratulated Scott on the role, before deleting his message on Twitter. The BBC insists no decision has been made over the new presenting team, saying “no conversations have been had about the future line-up”.
Scott wrote on Twitter: “The love and support shown to me yesterday has me smiling through the weekend, thank you.”
Referring to the Arsenal captain signing a new deal on Tuesday, she added: “The only thing that is signed sealed delivered that I am buzzing about right now though is Pierre-Emerick Aubameyang.”