We have had this discussion/argument before and whilst a 'rent boy' is by definition a young male sex worker the street market has always been dominated by male customers so the term has, for many people, homosexual connections - young males selling their bodies to older men. This is where the homophobic element comes in because it is throwing that issue of homosexuality as an insult.
Females, especially historically, rarely picked up sex workers off the street, particularly for their own safety and due to societal norms and restrictions I guess, so to try and salvage the use of the term by comparing it to escorts and gigolos is somewhat disingenuous in my opinion.
The insult is the homophobic slur element and it is rightly being called out as such - even if the majority of people using it don't have that intention.