I did try to go through all modes one by one and also changed the dma mode just to see if it makes a difference, without success.
In the end I grabbed a different SSD, wiped it and only put that single game on it.
Since that worked I thought it must be related to the partition. And indeed, now...