Then just for troubleshooting purposes, try setting the outer perimeter speed to 100% of the printing speed and see if your problem goes away. In fact, set all speeds the same for the moment to eliminate the same problem when moving from shell to infill.
See if that proves any improvement.