Essentially, racism is just one of many counterproductive definitions given to a facet of human nature that demands the classification of things and then references that classification in any extrapolations involving those concerned. As with any classification built on 'good or bad' logic, 'good or bad' evidence, the mere existence of the existing classification makes it easier to refer to it as a cause or reason for addenda. Such classifications do not need a difference in color or place of origin to exist, and will be forced into existence by any who through whatever reasoning feel a classification is good or necessary.
The irony of contemporary attitudes is quite extraordinary. If we say 'people were wrong to classify all x as possessing y, b and z natures' See: Sexism or Racism. That is one thing.
One does not 'not be racist' by saying instead that 'all x are not y, but all c are d.' Where x and c are 'races.'
Europe is not a race, neither is africa. The question makes a great many false assumptions. Slave traders did not '..focus their slave-taking on the poor kind hearted souls of africa.. ' until white slaves began to rebel.
From PBS.org - " this explains how one-half to two-thirds of the immigrants who came to the American colonies arrived as indentured servants."
From Wikipedia - "In the 17th century, the islands became known as death traps, as between 33 and 50 percent of indentured servants died before they were freed, many from yellow fever, malaria and other diseases."
"In many countries, systems of indentured labor have now been outlawed, and are banned by the Universal Declaration of Human Rights as a form of slavery."
"by which 3.5 million Indians were transported to various colonies of European powers to provide labour for the (mainly sugar) plantations. It started from the end of slavery in 1833 and continued until 1920. "
"The vast majority of those who were enslaved and transported in the transatlantic slave trade were Africans from central and western Africa, who had been sold by other West Africans to Western European slave traders (with a small number being captured directly by the slave traders in coastal raids), who brought them to the Americas. "
That is to say that the slaves transported to the americas were already slaves of other africans.
The issue is at any rate impossible to disentangle, as every act of a slave by proponent and opponent of slavery was attributed to their status as a slave, attacking a white man was because he hated being a slave, or was a terrible human being in general and thus should rightly be a slave. It's the same today whereby people refuse to account individual acts to individuals. White folks grab on to their specialness as history's bad guys, when slavery existed all over the world and was practised by nigh every culture under the sun that ever had the opportunity.
When one tries to propose that racism is a bad thing and has no basis in rightful thinking and in the same breath that one race is more guilty of it than another...
For those who can't connect the dots:
Europeans enslaved 'lower class' europeans for labor in the Americas, which was one of the primary reasons why the American colonies were successful enough to warrant importing African slaves later. The attitudes of the wealthy (and the not so wealthy, but free) towards those slaves was all but indistinguishable from their attitudes to non-white slaves.
Every nation today that has been extant for any length of time has significant recorded issues of it's industrialists, aristocrats, landowners etc taking advantage of those with lesser social protections(be that independent wealth, belonging to a lower caste or class, lack of social organisation etc..), when it's between two people of an apparent different 'race' we term the attitudes of the one toward the 'rights' and 'nature' of the other 'racism.' But otherwise indistinguishable acts are taken and attitudes presented between people based on any number of different criteria.
People are naturally protectionist and exclusionary, 'this money is my money,' 'this house is my families house' 'this land is my people's land,' 'this is a club for conservatives/progressives,' 'this stackexchange is for worldbuilding,' frames of reference and sets of criteria to be met before inclusion can be allowed.
The development of sets of criteria to rationalize and offer the basis for definitions is ubiquitous in human thought, the introduction of the new must therefore not cause undue dissonance with existing sets.
If laws (and laws by and large are made such that they appear reasonable to those with the power to enforce and as such will be informed by and inform non-legal thought) are made to protect differing classes to differing degrees and a trend develops wherein all the classes extant increase their ability to threaten one another, we have egalitarianism. But that egalitarianism is still bound initially by nationality or place of birth, Americans may still be press-ganged by British Naval forces after it becomes illegal or contrary to policy re: britons) for example.
(In the meta)The proposition that changing the colors of peoples would change their nature as peoples would, if shown to be accurate, justify and accelerate racism, as it would show that to classify behavioral trends based on the color of one's skin was justifiable. (That is, the question seems to suppose that white people were racist slavers because they were white)
On the assumption that the people involve do not witness the change.. to assume that people's views of 'black' people was justified primarily by that particular color of skin is not justified, when all of the evidence points to the fact that 'people' treated 'people' in similar ways no matter their color.