I should clarify, the application returns to zooming on middle-click drag. It is like in certain situations the application re-asserts the normal mouse handling from before your plugin inserted itself.