The same reason chavs wear Burberry because they think it makes them look classy.
Chelsea had a hardcore of racists back in the day, these people usually builder trades people( I am not looking down on them as I did that work myself) a lot grew up with outdated racist views, then they had children and their children grew up with similar views. They think it is a way of sticking things up to the political elite. They are wrong, as much as I think the working class may have been screwed over the answer is not to be racist.
I also believe these people probably have low intelligence and education levels, I think people like to stick to their own so when someone who is getting into football and holds racist views they would probably be attracted to a club where fans have similar views, I would hazard a guess they might even meet up at EDL marches.
Me, my personality is one created from a very working class background with also low education level, the difference is I had an amazing father who instilled a hard work ethic in me and told me to not blame others for my failings but to work to achieve for myself, I also inherited and absolute hatred of government from him, hence why I hate the EU.
So I imagine it stems from the 70's and 80's when they had a lot of racists and those guys childrens now go plus other racist scum look around and think it is fun to act like Neanderthals and start supporting them.
If I were not a Tottenham fan, I would start supporting us because I love how we moan even when we win, I love how we are a negative bunch who when things are going well we wait for the blip. Tottenham suit my personality. Chelsea as much as they might try and get the glory chasers and middle class parents taking their children to a game, will always be the club of scummy racists.