Ah, that's good information. What's happening is that the Yellow/Blue wire is open circuit. The gear position switch will make contact with ground in a particular pattern as the gears are selected. In your case, the pattern is being disrupted.
The problem might be in the switch, but more likely it is in the connectors. there is one connector on the right side of the frame tube where the switch joins the harness. There is another connector on the back of the speedometer and finally there are circuit card connectors inside of the speedometer (I am assuming your bike has all original equipment).
My first port of call if I were you would be to inspect, clean and make certain the plug on the right side of the frame is making good connection, especially on the Yellow/Blue conductor.
If that fails to solve the problem, check and clean the connector on the back of the speedometer.
If that fails to solve the problem, then it might be time to get at the switch itself and clean it's insides.