Norm Munday, 75, says he couldn't believe his eyes when he turned to face the opposition coaches box after the final siren on Saturday and was confronted with the gesture.
Far from initiating the exchange, Mr Munday claims he was only responding to Ayres and his coaching staff when he returned the finger salute.
``I was watching the game, the siren had just gone and I stood up to watch the umpires come off the ground because I knew they were going to get a raspberry and there was a bang and we looked around and there was Ayres,'' Mr Munday said.
``I couldn't believe what I saw and that's the honest truth.
``He took it out on us because we were the closest to him. When someone starts doing it to you you do it back to them. That's fair enough isn't it?''
Ayres claimed after the match that Cat fans in front of the coaches box continually banged on the glass and questioned his parentage.
But Mr Munday says no one in the back row of the Charles Brownlow Stand below the coach's box provoked Ayres, suggesting the former Cats coach held a grudge against his old club.
``I don't know where this banging of the window comes from. Only they banged the window at the end, that's why I looked around,'' Mr Munday said.
``I don't know whether a fine is good enough for him.
``I think he needs to go to an anger management class and learn to grow up and act like an adult.
``He's carrying a grudge against Geelong. Why else would anyone carry on like that.''
A friend of Mr Munday who was at the football with him backed up his claims, saying the pair shook their heads in amazement after they'd left the ground.
``We were right in front of the box and he (Ayres) was just screaming. I looked at (assistant coach, Darren) Jarman who was laughing and I laughed back and then Ayres gave the two-finger salute and Norm gave it back,'' Bill Collins said.
``There was definitely no banging on the glass all day.''