Very much this. Lack of consistency is a massive issue that refs need to address. There's also referees who seemingly love to put themselves in the spotlight, instead of quietly running the game according to the rules.
I'll give the referees this, though: It's hard to fully focus on your job when players surround you to try to influence (correct) decisions, pundits dissect every 50/50 decision you make, managers directly or indirectly criticise you on camera and there is massive media scrutiny for every minor detail in your officiating. It's not always the refs fault alone that they're the talking point.
I think the solution is two fold. Ideally much more respect for match officials from players, managers, media and pundits, as well as an acceptance that refs are human and make mistakes - plus of course an overhaul of the refereeing itself.