Note that contrary to you:

Interestingly the 3d view is perfectly smooth with all textures in it.

On my system (Ryzen 3 5800X/32GB/RTX 2060/Winddows 10 Pro/SweetHome3D 6.6a), the visitor 3D view was heavily lagging too (had to make very small incremental movements with the mouse) UNTIL I switched plan room rendering to monochrome !
However, this doesn't change anything to the visitor 3D view rendering (except that the lagging disappear) !!!
This is really puzzling to me, but I will use it as a workaround.