Why did Sandor 'The Hound' Clegane abandoned King's Landing in the middle of the battle?

I believe, Sandor was basically a good man and he was obeying Joffrey, only because he was bound by his sense of duty. Besides, Sandor had a past with fire. In one of the episodes from season 1, we learn that when Sandor was a kid, his elder brother shoved his face into fire and hence the burn marks on his face.
Now, during the battle with Stanis, initially he goes for fighting but in the middle of it, he faces fire and this disturbs him deeply. His memories of childhood were back and he became disillusioned. That's why he returns back, drinks wine and abandons King's Landing.