It is wrong for two reasons: firstly, it singles out Islam and Muslims for special discrimination, and secondly, it is deeply counter-productive. The first point relates to your own views on the subject, which largely echo with mine: I'm not a religious man, and save for some vague notions regarding Deism which I'd like to eventually explore when I've got the time to do so, I don't think much of spiritual pursuits and generally see them as geopolitical tools more than anything else. However, Islamophobia singles out that particular religion, and that's just wrong in my view: I've seen enough religions in action during my travels and in my place of ethnic origin (India) to know that Hindus, Buddhists, Christians and even Jews can at times be as violent and barbaric as the very worst Islamists when it comes to defending their own imaginary friends, and all of the holy books written a thousand years ago seem deeply flawed today in one form or another (Sikhism and Jainism are rare exceptions to this general rule, but even they have flaws). Islam has over a billion adherents with hugely different views and values, but its birthplace is in the ever-roiling Middle East, center of the West's media attention: its own flaws and problems are magnified a thousand fold due to that unfortunate circumstance, while gay men being necklaced in deeply Christian Uganda, Buddhists massacring Muslims in Myanmar, and Hindu extremists killing Christian missionaries in India are overlooked because they don't happen to be the Western media's targets for ire. I'd be happy with all religions being equally disdained by the populace (while still being allowed to exist, of course), but picking on Islam because the lunatics in the religion get so much screen time compared to the rest is just wrong from my logical standpoint.
The second reason Islamophobia isn't a good thing is, as mentioned before, that it is deeply counter- productive. If we in the West alienate Muslims already here, they'll only withdraw further into their own communities, which will then create endless opportunities for power-hungry mullahs to keep them scared of the wider society outside their mosques and communities and thus radicalize the previously moderate populations. And any actions loonies drunk on Islamophobia take against the Muslim community in particular (again, discriminating against them somewhat unfairly) will create endless fuel for the ISISes and Al-Qaedas to use in their recruitment videos and war against the 'Crusaders'. It serves no practical purpose, and only harms the West.
Look, religions are all largely bunkum in some form or another. Oppose them all, or oppose none: discriminating when it comes to judging one imaginary friend worse than another won't help us one bit, from both a moral and practical standpoint.
