I believe it has to do with their colonial period.
Becoming enmeshed in wars and occupations in India, Africa, the Middle East and Asia over the past two (+?) centuries and mingling with the native populations gave those native populations the right to petition the British government to allow their individuals with a sterling war record fighting alongside the British, to settle in the UK.
This was resisted to a greater degree earlier than later, when the fair minded and uninformed British spoke up for fairness to reward the noble Muslims and other natives from all over their colonies for their friendship and loyalty.
And eventually their argument won the day.
And at first there was little problem with the influx of foreigners as they DID mostly prove to be loyal friends. But they also brought with them a set of “luggage” which would become more apparent with each successive generation until in 1968 there were so many instances of discrimination against people of color, that a law was passed preventing racial discrimination.
In April 1968 a member of Parliament (MP) who was thought to be a leading contender to become the next Prime Minister of Great Britain, Enoch Powell delivered an historic and incendiary address known as The River of Blood speech.
This brief video marks the 50th Anniversary of it.
Powell was called the worst kinds of names for saying the new law would give the Black man the whip hand over the White man.
(All of this is subject to factual edit, but is generally close to being spot on, iinm)