Update for the sake of documentation:
I just found out something very interesting this moment. In my desperation, I downgraded the firmware even more, to version 1.0.9 155 VC1.1.1 20190809 from this page
linked on the official German Flashforge website and this one actually worked! No clunk, no layer shift. This one just spews out filament like crazy on every tool change, but that's another issue I guess. The actual print did work, the layers were perfectly aligned.
There is also version 1.1.0 in between the two I tried, but this seems to be an incomplete file on the website as it's only 2MB and the printer didn't recognize it.
So there seems to be something that changed between version 1.0.9 and 1.2.2 (and is still present in 1.2.3) that causes this issue. Finding out what that could be is obviously way out of my league though.
I'll send that insight to the support people from S3D and FF I'm dealing with though, I hope they can do something with it.