The point isn't the quality of the architecture but rather the legacy of the building.
Spurs proposed to send British Athletics packing back to the inaccessible and limiting backwater that is Crystal Palace. And that would have been totally unacceptable to the British people after the stunning success of both the Olympics and the Paralympics.
P.S. The London Olympic stadium was very deliberately designed not to be a massively expensive architectural statement. For starters, unlike Beijing, London had nothing to prove to the world. Unlike Beijing, it was already one of the world's alpha cities (not to mention the fact that it already has more than enough big stadiums). Additionally, one of London 2012's core themes was legacy and sustainability. The Bird's Nest might be mightily impressive (though it isn't anything like as intimate and good an experience for spectators) but it is now nothing more than a visitor attraction. The London Olympic stadium, by contrast, was designed to be two thirds temporary, with only one third remaining as an athletics stadium.
It was only after politicians performed a u-turn that it was decided to maintain the stadium more or less as is.