I have a similar system: duo E6750, 4 Gb RAM, Ubuntu 12.04 32-bits, Sun Java. Looking at the same model house, it doesn't move very quickly, but the delay isn't more than maybe 1 second. When I make all the trees invisible, the movement is very smooth. (SH3D doesn't like trees.)

If you are using SH3D v4.0 you should use Sun Java; the other Java's may cause problems.

Hans