Not to buck at this, but I think you have put the cart before the buck. Surely, the metaphor is offensive to anyone it is applied to in as far as it is held to be derived from the disagreeable characteristics of the non-metaphorical animal the word first signifies. If one finds the sexual prowess of a he-goat vulgar, then it is vulgar to call a black man a buck, or, as it were, a stud or a stallion; if one finds the wildness and rebeliousness of a male onager or deer objectionable, then it is objectionable to call an American Indian a buck, though here there might also be an influence of buckskin (during the Revolutionary War this being the (positive) slang for Continental troops, who wore leathers the Indians either traded them or taught them how to tan to a texture superior to that available from Europe). There is no need to spin these "citation needed" just-so stories about how a singular offensive usage hopped and skipped from one race to another. I am sure you can certainly imagine an offensive term for aboriginals getting transferred to black men a hundred years later (than what?), and a great deal besides, but it is neither here nor there. What is given is sufficient: the variety of male animals referred to as "bucks" have good qualities and bad qualities, and the bad qualities are always the ones the powerless get stuck with by the powerful, without any historical prestidigitation of meaning needed. Else, what further gyrations would you propose to explain how these buckish slurs then jumped the Atlantic to stick to disagreeable English fellows who were decidedly white (and probably had buck teeth)?

BUCK. A blind horse; also a gay debauchee.
[...]
A BUCK OF THE FIRST HEAD. One who in debauchery
surpasses the rest of his companions, a blood or choice
spirit. There are in London divers lodges or societies of
Bucks, formed in imitation of the Free Masons: one was
held at the Rose, in Monkwell-street, about the year
1705. The president is styled the Grand Buck. A buck
sometimes signifies a cuckold.
BUCK'S FACE. A cuckold.
BUCK FITCH. A lecherous old fellow.

(from 1811 Dictionary Of The Vulgar Tongue).