there are other things that happen when one gets on the pitch and into the game - a surge of adrenaline and sudden high level of energy that changes the ability for finesse, an unfamiliar atmosphere that disturbs concentration and getting into the zone, competitors already fired up and in overdrive ready to take knocks with you - all these adds up to "nerves" and inhibits one's ability to get into "the flow" where calm and instinct takes over.
so maybe not physical fitness, but even being there helps with mental fitness and experience to deal with oneself.
I'd argue that at the highest level of any game that excellent physicial attributes and skill is a given; however mental strength and dexterity makes the difference between champions, world-class and the rest who make up the supporting cast.