If the relay is good I second the starter button. Especially considering how the starter motor is sporadically running. Does it also stop randomly or do you need to shut it down? Could you have a worn wire; possibly grounding itself as you ride and triggering the relay? That's a real longshot, and until I have some coffee I have even less of a clue than normal.
These bikes start up so easily I would gladly trade the starter button, load shed relay, starter relay, starter motor, feet of connections to the headlight, battery, "control units," etc etc for a piece of steel and cog. And my foot.
Now I'm curious to add up the cost to fill a shoebox with everything related to the electric start. Dealer prices would be good for a laugh.
good luck. My money is still on the button.
p/s
I forgot to mention the motor itself but second what RT Keith posted. I replaced my brushes when I had issues but think my problem was the relay. I am a somewhat ignorant of the inner workings of the starter. I'm not sure if it can fire itself up without the juice from the relay.