Should the lights come on when I do this with the key off?
No, that does not sound normal. Admittedly I can't say that I have tried to do the same thing with my bike, but if the key is off, all should be off.
I'm thinking along the lines of Johnny, go ahead and pull the starter and clean it good, probably also may have to replace the starter brushes. Clean the brush contact area too.
The circuitry of the bike uses a "load shed relay" that turns off all extra unnecessary load when you press the start switch so all battery power is available to turn the starter. The engineers connect the part of the relay coil circuit THRU the starter. When the starter relay is energized, with 12v power applied to the starter post, the lsr is "de-energized" and extra load is turned off. When the starter relay is then turned off, bike running etc..... the lsr is then "energized" to turn all the extra stuff back on.
If the current pathway thru the starter is excessively dirty(high resistance) it cannot pass the "low" current required to keep the lsr energized.
Hope I explained it well enough to help you without confusing you.