I think that's precisely why he left, and at the end of the day both club and manager have to be working how they are comfortable, otherwise it's best they part ways.
I just think the trust between club and head coach was not there enough. For example when the contract is signed and AVB is hired, they can say to each other: 'the target is top 4, we want you to make good use of the club's youth academy and we want you to develop the undervalued potential that we sign'.
AVB then thinks 'ok, no problem'. But to both parties, achieving top 4, using the youth and developing potential can all mean different things. He can point to his use of Caulker and say he was happy to use a youth product. He can point to Bale and show that he helped a player achieve their massive potential. But if the club wants top 4 and he says that in his opinion he would prefer fewer signings but top quality, rather than more signings but lesser quality, then there is a disparity.
So they can agree on objectives, but in practice they can take it to mean different things. I don't blame AVB for it, and I don't blame Levy and the board for having their own point of view. I blame them for not aligning their own objectives with reality. I imagine how it went down is AVB says 'get me Hulk, Soldado and Willian and we will make a top 4 challenge'. The club says 'We can't get you Hulk, Willian has gone to Chelsea, but we can get you Eriksen, Chadli and Lamela'. AVB probably will have said 'fine, I will work with them, I will do my best but they are not as proven quality at the top level and may take longer to settle.' But expectations weren't realigned, they still expected top 4 as a minimum this year and thought that because they let 100M go out the door this summer they were justified in that. Again, I think it's differences of opinion that lead to a parting of the ways needing to be required. I don't blame AVB, because agreeing upon objectives and giving him the backing to make it happen are 2 different things.
AVB strikes me as someone that prefers a smaller squad of quality rather than a bigger squad where quality is in thinner supply. He didn't rotate much at Porto and I think his methods are designed to allow a smaller group of players the chance to perform consistently over a season. (Kind of like Ramos, which is why he didn't even register something like 5 good players into the squad in 2008). When he's given a bigger squad, his methods and his reason for being highly rated (to make a quality squad perform effectively) are negated because he is juggling more players that don't need to play in the way he prefers because more rotation is possible.
So I think he would have rather money was spent on proven top quality rather than the majority on potential. When that didn't happen for him he knew that he probably shouldn't be here anymore, and that's where I think stories that he was glad it was over come from. He knew he wasn't backed by the club and they didn't really trust him.