No I decided that walking off would be the better option as I didn't want him to get hurt
That is a very sensible attitude young man! If only more people adopted your attitude in town/city centres on a night out, there wouldn't be so much pointless drunken violence!
My mate got abused on his stag night as we had dressed him up as Harry Potter! This guy was giving him loads of stick about it! 'Why you dressed like a cunt' he asked. 'Cos it's my stag night' my mate replied. 'that's nae excuse'. He clearly was looking for a fight, but we just walked away! What's the point in these people?

to be quite honest, a good fight can be quite an exciting experience