Beyond leaving and discussing why we were leaving, I didn't punish him further. He was very upset, and I suspect the girl may have grabbed him or pulled on his arm to get him to move. He doesn't care to be touched by strangers at all and would not react in a good way to this under the best circumstances. He was obviously overwhelmed by the heat, noise, and crowd, so taking him away from that was the best thing to do.

I'm not angry at him, just worried. This is an indication that he may still have some underlying issues we may have to work around.