Yes, it would, depending on how fast you're moving and how much cool air is passing over the engine. If you're not moving on a hot day, it'll heat up pretty quick.
If the light comes on, pull over and wait till you have a clear road so the air can cool the engine. Don't ride with the light on - you can damage old and brittle rubber bits on the throttle bodies not to mention other hazards of overheating a gas engine.