Difficult to quantify, really. My concern is partly losing headers, absolutely - it's also more abstract, in that a taller right-back frees up space for a shorter player elsewhere where defensive solidity is less important than agility and trickery - viz, in attacking midfield. Really, I just don't want to end up like Arsenal after their stadium move - (to generalize a bit) a team of midgets and physically suspect players, playing beautiful stuff but being knocked about by the more solid, relentless teams.
I concede that yes, there are more factors that can offset a lack of height, like strength, positioning, willingness to get stuck in and so on. But height is important too - it is not an insignificant concern in these matters, is all.