See https://youtu.be/nq00He4oz_4

I did notice that navigation keypresses in the 3D view also "reset" the panning. It may be that the mouse scroll wheel up/down action are treated more like keypresses than mouse button actions.