Odd, I tried to recreate this myself with 4.0.1, but it doesn't happen at all. Works fine regardless of how many times I rotate back and forth.
I wonder if the issue is possibly due to your trackpad occasionally registering a right-click. Some trackpads have options where you can right-click just by placing 2 fingers on the trackpad, which could cause these inconsistent movements.
What happens if you plug in a USB mouse and use that?