to speed up rendering avoid these as much as possible:
-transparency
-mirrors
-(number of) lights

You already noticed that the size of objects makes a difference. Using Blender to reduce vertice & face counts does help but you will have to be aggressive. If you have objects that are used in high numbers than these are especially interesting to optimize as far as possible. One face less for a hundred objects means 100 less faces to render.

The Sweet Home 3D renderers (Sunflow & Yafaray) both use the CPU for rendering so a faster graphics card changes nothing.