They call it snowy peak syndrome: no matter how colourful the foothills and middle reaches of the average British organisation are, the top is nearly always very white.

A research project conducted by Operation Black Vote and the Guardian has established that barely 3% of the most powerful, prominent 1,000 people in Britain are from ethnic minorities.

And for the few prominent black, Asian and minority ethnic (BAME) leaders who have made it to the top, the causes of this strikingly unfair social imbalance are manifold, stubborn, shameful and urgently need addressing.

“Racism has never been about calling people names. It’s a system of power,” says Marvin Rees, the Labour mayor of Bristol.

“In the junior and middle ranks, there’s a good representation of colour,” says Sam Gyimah, a junior minister in Theresa May’s government, “but at the metaphorical tree-line, diversity disappears.”

The Guardian project, conducted in partnership with Operation Black Vote, established that some sectors fare better than others: politics has actually improved significantly, while the judiciary, military and publishing lag behind.

Sharon White, head of Ofcom, blames it on an organisational cliff edge which prevents good people from advancing higher. “There is a disproportionate exodus of people from a minority background not getting promoted. It’s a very common problem.”

Other public sector leaders from minorities concur. “It’s quite striking in the NHS. There are significantly more doctors [from BAME backgrounds] but at senior level there are very few chairs of NHS trusts and CEOs,” says Karamjit Singh CBE, chair of University Hospitals of Leicester NHS trust.

The latest findings amplify those of a report earlier this year, which detected a range of factors holding back minorities in the workplace.

Gyimah recalls how the Conservative party had to “change the game so people like me were interviewed”. He doesn’t believe this was racism, just the well-documented tendency of people to promote people who are like them.

“If you feel that someone might not fit into your community because they look different, that’s not racism – it’s just sticking to what you’re used to,” he says.

Affinity bias – the technical term for sticking to what you’re used to – is frequently used to explain the phenomenon of overly male, overly white boardrooms.

Jas Athwal, elected the first BAME leader of Redbridge council in 2014, said that when in opposition he often wondered why senior officers didn’t reflect the diversity of the London borough.

Now, he says, “having gone through a huge recruitment drive … the majority have been white”. He cites potential cultural barriers for the lack of diversity in politics.

Rees’s views are informed by five years he spent as a journalist. “My work was extensively edited where that of my white counterparts wasn’t … then I would see younger people come in and be given training and exposure to all these different roles while I was doing the same old shift. You get demoralised and by the end of it I felt humiliated every day just going in.”

A faltering confidence and the isolated, ambiguous nature of the incidents made him question himself rather than the structure.

“I didn’t want to make excuses or make myself vulnerable by calling it out,” Rees says. “It’s called a permanent state of imbalance – was that me? Or was that them? Each individual instance feels like it can fly, that it was your own imperfection – but when you line them up you see something’s not right.”

Gyimah concurs: “A lot of BAME professionals will get to review, have done everything that’s asked of them but they’ll be hearing: ‘You’re not quite there yet.’”

And yet there is huge reluctance to challenge such discrimination for fear of being accused of “playing the race card”.

So what do BAME leaders feel is the best way forward? Some stress the importance of performing well in their prominent roles to advance the cause. As Athwal says: “When it gets to the next election you want them to say: ‘Oh, we had a Sikh guy last time. He was all right.’”

Quotas aren’t popular. Though they may have had an impact elsewhere in the world – South African sport, for example – some are uncomfortable with them.

“You don’t want to be forced to employ people to tick a box,” says Perminder Mann, chief executive of Bonnier Publishing UK. “I want to employ those who are right for the job.”

White, the Ofcom chief, believes workplace culture has to change. “It’s not just about supporting people to be confident, but letting the dominant culture become aware of how they are inhibiting the success of others around the table,” she says.

She references meetings at senior level where non-participation is used as a tool to measure who should progress, saying: “Someone might chair a meeting and genuinely think they’re opening up the floor for discussion. But the way that you open the conversation can implicitly hold back the person who thinks their voice is worth less.”

White also explains how minorities adopting the behaviour of the predominant group can fear penalties. “For women who adopt equally articulate and clear presentation of issues, it’s being ‘bossy’ … the few black men that I worked with as a civil servant … were also very conscious about being the first or loudest voice in case they were seen as aggressive or pushy.”

“That’s white privilege,” says Rees. “We have to speak about privilege as well as racism so that we can begin to dismantle it.”

The approach could be attacked for holding back the white people in the room, but recent research by McKinsey shows that more ethnically diverse companies are also 35% more likely to earn above the industry median. For White, the case is simpler. “I think it’s a basic morality point. Everybody, regardless of background, has got to have the same opportunities.”

She believes the different approaches are cultural, rather than raw signifiers of talent: “It’s really important that it’s not about ‘we’re teaching you confidence to speak at a meeting’, because that’s enforcing the idea that there is a dominant culture and that person has a gap that they need to overcome.”

Instead, she believes that the burden of progress rests on all shoulders, not just those from BAME backgrounds. “Progress doesn’t simply happen,” she says. “We need to ensure that the dominant voice isn’t squeezing out the one that’s different to theirs.”

What can be done?

BAME high-flyers suggest a range of approaches to tackle the problem:

1. Spread the word

If you want to encourage a broad and talented team, you might have to go looking for it.

Mann explains that her parents had limited knowledge of what they thought of as high status careers when they came to England. “It’s important to create more awareness if you want the best people. That might mean going into schools in areas with high numbers of BAME students and talking about careers.”

Having a BAME person at the top of your own organisation can help. “I want to champion that – it’s natural that you see someone from your own background and think you can get there too,” says Mann.

2. Beat the bias

Unconscious bias about what leadership and talent look like still skew recruitment practices.

“We need to be very honest and unflinching about where we’re still failing as a nation to access people from all backgrounds,” says Gyimah. But the very fact that these biases are subconscious rather than deliberate means that mining data is key to correcting them. “It’s only when data is collected that organisations are forced to change. What gets measured is what gets done,” he adds.

White has a further suggestion: “Exit interviews can be a massive wake-up call if people feel their progression is being stunted.”

3. Open up

Meritocracy is essential for a fair workplace, but at present progression opportunities are unequal. Equal access to progression requires everybody to have an understanding of how to progress. “Progress requires those at the top to signal, very clearly, what they want to see for people to do well,” says Gyimah.

4. Targets and quotas

The debate over targets and quotas is a heated one. Some argue that only by mandating quotas can in-built biases in hiring practice be corrected. Others, like Athwal, say: “We should be trying to get the best brains regardless of colour. Quotas drive quality down.”

5. Leadership

Ultimately, without a leadership team who care about diversity, it’s unlikely to happen, Gyimah says. “It needs to come from the very top, otherwise it’s at the mercy of others to make their own rules.”

White adds: “You want change to be sustained. That means ensuring that it’s not just those at the very top who are leading on this, but senior leadership too.”

• Have your opportunities for career advancement been restricted because of your ethnicity? Share your thoughts using the form below. We’ll use a selection in our reporting.