I haven't read the books, no, and have avoided checking to see what happens. It's just my feeling that there's a bit too many strands going on and it's starting to lack a central focus.
Like DHSF said, it's the human story that attracted me to the show, and dragons aside (because they used to exist before St George killed them) the magical/fantasy stuff leaves me a bit cold. Random Giant people suddenly appearing, The Lord of Light business, now the "Children", random people like Mckenzie Crook who see through a bird's eyes, the White Walkers who appeared to be a major part of the show then disappeared.
It's obviously based on different periods of English History, like the period of seven Anglo-Saxon kingdoms that united into England, the Wars of the Roses, and the wall is obviously based on Hadrian's wall. The best part of it is the power politics.