Speaking as a non-Jewish person (and more broadly as a white middle-class heterosexual English man), personally I've come to accept that I can just never fully understand what it feels like to be persecuted for some part of my identity, and therefore that I also can’t fully understand what it feels like to hear particular words that have historically been used to persecute (e.g. yid), even if they’re now being used in a different and more positive way (as they are among Spurs fans).
So in the end my simple position is that if a majority of Jewish Spurs fans want us to stop using the term, we should. If a majority of Jewish Spurs fans don’t want us to stop using it, then we shouldn’t. Maybe it's controversial, but I don't think non-Spurs Jews or non-Jewish Spurs fans should sway the decision much much at all.
(Even though on a personal level I'd be gutted to stop using the term, because for me it is 400% associated with being a Spurs fan and almost 0% associated with being Jewish - and for some reason I love the term and the songs that go with it).
The 'nigga' example is also interesting - rightly or wrongly, my understanding is that the majority of black people aren't comfortable with a white person using the term at all.