For several decades, racists who thought it politically incorrect to attack people for being black or brown instead protested about “immigration”, and wanted to send immigrants “back where they came from”. This desire to repatriate applied irrespective of whether the black or brown person in question was actually an immigrant or someone who had been born in Britain. Such details did not matter, since what the racists were really against was the idea that Britons could be anything other than people with white skins, preferably of Anglo-Saxon origin.
While racists can be found in all political parties, it is a historical fact that every significant piece of legislation promoting racial equality was enacted by Labour governments, usually over Conservative opposition. Meanwhile from time to time some Conservative politicians would come out with speeches making it clear that they would rather people like me left the country, if necessary with the assistance of a repatriation grant. While such racist policies never became official Conservative policy, we had enough such people within our party to ruin our brand with most people from an ethnic minority background. That is why even today if someone is from an ethnic minority, even if they are highly educated or rich, they are much more likely to vote Labour than vote Conservative. If you have not already read it, I recommend Lord Ashcroft's report "Degrees of Separation."