With Sweet Home 3D application, you can't get a smaller SH3D compressed file without removing some items from it.
But there could some other ways to reduce this size (but there's a long way from 180 MB to 20 MB). Using the Export to HTML5 plug-in will create a version of your home in a ZIP file with textures at a 256x256 pixels size max.
If you imported some 3D models from Archive 3D, your SH3D file might contain some useless GSM files that you could remove with a ZIP tool.