Does the starter continue to run even after you've started the bike? When you turn the bike OFF is the starter still running, turning the engine over? If so, then you need to fix the starter relay.
You can try whacking the front left corner of the relay box to jolt the starter relay. Otherwise you'll need to disconnect the battery to turn the bike off.
Have your battery load tested. If it's more than 3 or 4 years old, just go buy a new one.