The European Central Bank has fanned fears that London could lose its status as Europe’s financial capital after warning that the Brexit vote might sever the City’s trade relationship with the EU.

A top ECB official said banks in the City of London risked being stripped of their lucrative EU “passports” that allow them to sell services to the rest of the union. François Villeroy de Galhau said keeping the so-called “passport” would not be possible if the UK leaves the single market of trade in goods and services.

The ECB governing council member also echoed calls from EU leaders for quick Brexit talks and he warned that Britain would bear the brunt of the economic consequences of leaving the EU, while other European countries would be less hard hit.

The warning came as pressure built on the UK economy in the wake of the vote to leave the EU, with credit ratings agency Moody’s lowering the outlook for the UK from stable to negative in a move that threatens higher borrowing costs for the British government. Citing the uncertainty created by the vote, Moody’s said lower economic growth could outweigh any savings from ending contributions to the EU budget. Investors have also marked concerns that global growth will be hit, wiping $2tn (£1.5tn) off share values on Friday.

However, the ECB intervention on Saturday signalled that London’s status as a major financial centre could be in doubt. The passport issue is a key concern for big non-EU banks working out of London as the system has given access to the EU’s market of 500 million people.

Villeroy de Galhau, who is also the governor of France’s central bank, told radio station France Inter: “As long as Britain is in the single market the City can remain a big European financial centre. If tomorrow, Britain is not part of the internal market, the City cannot keep its European passport.”

He added: “What happened on Thursday is bad news, first of all for Britain. Of course there will be negative consequences for the European economy but there will be much more limited than the negative consequences all experts forecast for the UK economy.”

In the run-up to Britain’s in/out referendum this week, some investment banks had already warned that a vote to leave the EU would prompt them to move from the City to new bases in cities such as Dublin, Frankfurt and Paris.

Such moves would mark a major shift for the UK economy. The country accounts for more than 2m of the EU’s 11m financial services jobs, according to trade body TheCityUK. Leaving the EU could put at risk up to 100,000 financial services jobs by 2020, according to a report that the lobby group commissioned from consultancy PwC.

The report said the financial sector was more vulnerable to leaving the EU than the rest of the UK economy because Britain’s place as a financial hub relies partly on access to European markets. The financial sector is also closely linked to the health of the economy and responds sharply to slowing economic growth.

But it also predicted financial services would still grow under a Brexit scenario – though not as quickly as if Britain remained in the EU.

There are broader worries about the impact on the UK economy and tax receipts from Brexit’s potential impact on financial services.

The sector is a significant part of Britain’s economy and big contributor to tax receipts. Financial services account for about 12% of GDP – more than manufacturing.

However, Villeroy de Galhau said the UK could retain trade links by adopting a Norway-style relationship with the EU. “There is a precedent, it is the Norwegian model of European Economic Area, that would allow Britain to keep access to the single market but by committing to implement all EU rules.” He added: “It would be a bit paradoxical to leave the EU and apply all EU rules but that is one solution if Britain wants to keep access to the single market.”