+1 on the switch. There is a fair amount of electrons going through those contacts. As things get funky the resistance increases which causes the contacts to heat up which accelerates the whack. Since there is very little time on the Hi Beam flasher contact, that one stays pretty clean.
I would look at the possibility of cleaning the contacts. Start out by shooting some Deoxit in there and working the switch a bunch. If that doesn't get it, you will have to go in there and clean things up manually. If all that fails, there are a bunch of switches on eBay for reasonable money.
Before you go much further, get a relay harness from Eastern Beaver. Not only will it keep you from burning up your handlebar switch, it will make you headlight a bit brighter than when all the electrons were running through the handlebar switch.