Sweet Home 3D Forum
»  List all forums
» »  Forum: 3D problems
» » »  Thread: Suddenly, huge file size...
Print at Dec 18, 2025, 8:26:06 AM
Posted by mazoola at Nov 18, 2015, 12:26:59 AM
Suddenly, huge file size...
Today, I noticed sh3d seemed to be taking an awfully long time to save my current home plan -- and when I checked my data directory, I discovered the file was at 660 Mbytes and counting. (For the past month or so, the plan typically hovered around 145 - 170 Mb; my most recent offline backup -- from 4 November, alas -- was around 70 Mb, compressed.) Even though the file still appaeared to load OK, I was reluctant to continue using it, as I feared it would eventually fail completely.

When I saved the .sh3d file compressed and opened it using an archive client, I discovered two oof the entries weren't as one would expect: Items 300 and 316 in the ContentDigests were both listed as "###/123/untitled/texture_3.jpg" -- but the 300 and 316 folders actually held seemingly complete copies of the entire plan. (More correctly, they appeared to hold complete copies of slightly earlier versions of the plan.) And in case that wasn't enough, inside folder 300 there was another copy of folder 316 (numbered '315' in this instance) for a third level of recursion. Folder 300 weighed in at 314 Mb, with 316 contributing another 156 Mb.

I tried hand-editing the compressed plan file, but could never get it clean enough for sh3d to accept it. (I'd get the erroneous file warning asking me to choose whether I wanted to replace or remove the missing objects; no matter what I chose, though, sh3d would quit, leaving me to kill the java process.)

In case the problem was caused by a corrupted model, I went through and deleted everything I'd added in the past two weeks, resaving after each deletion -- but to no avail. Since the failure didn't seem to be object-linked, I decided to take it the other direction -- starting with my two-week-old backup plan, I cut-and-pasted only the changes I'd made from the corrupted plan to the original. Unfortunately, when I went to save the updated plan, it again ran greater than 650 Mb. (I later discovered copying the doors and windows from one floor of the corrupted file to the back-up would cause the expansion -- but deleting them afterwards only brought the file down from 650+ to 335 Mb.)

Finally I returned to the backup plan and added all of my recent changes from scratch, re-importing the models. When I was done, the uncompressed .sh3d file was 172 Mb.

I also noticed there were two 'UNRECOVERABLE' files in the sh3d 'recovery' folder under 'Application Data' -- one 120 Mb and the other 327 Mb. These are near enough the sizes of folders 300 and 316 to suggest some link -- especially as this plan has never legitimately hit 200 Mb, let alone 300. Deleting those files, though, seemingly had no effect.

I understand this was, more than likely, a purely happenstance occurrence; given I have no idea what caused it -- or, for that matter, at what point in the past two weeks it happened -- I realize there's virtually no chance of reproducing it. I did want to pass along what little I was able to find, though, on the off chance it might help you resolve a known issue.

Thanks!

Posted by Puybaret at Nov 18, 2015, 9:35:36 AM
Re: Suddenly, huge file size...
Sorry for this problem. It's not the first I see it and wonder how it can happen. A copy/paste operation between files could be a possible track, but I didn't succeed to reproduce this "recursion" issue after new attempts.

For your information,
- The recovery folder is the place where a copy of the edited home is automatically saved in a file with a .recovered extension, each time the delay set in Save home data for recovery every preference expires. When you quit normally Sweet Home 3D, the .recovered files in that folder are deleted. So these files are kept only if the program crashes.
Each time the program is launched, it reads and opens the .recovered files in that folder to let the user recover his recent changes after a crash. But sometimes the automatically saved file is incorrect and it can't be reopened. To avoid trying to open a wrong .recovered file at each launch, it's renamed with the .unrecoverable extension. There are very few changes that this file can be used, but who knows?
- The ContentDigests entry in a .sh3d file gives the SHA-1-Digest value of every entry of the .sh3d file used by Sweet Home 3D. This value is used to quickly check whether an entry is correct or not. So if you edit an entry in the .sh3d zipped file, Sweet Home 3D will think the file isn't correct anymore. I didn't program this feature to avoid users to manually edit an .sh3d file of course, but to check if an error on the disk didn't happen and inform the user accordingly. If you want to bypass this data integrity check, just remove the ContentDigests entry of a .sh3d file before opening it with Sweet Home 3D.

As you can see, I pay a lot of attention to users .sh3d files. wink
----------------------------------------
Emmanuel Puybaret, Sweet Home 3D creator

Posted by mazoola at Nov 19, 2015, 6:59:23 PM
Re: Suddenly, huge file size...
If you want to bypass this data integrity check, just remove the ContentDigests entry of a .sh3d file before opening it with Sweet Home 3D.


Good to know! When I get back home, I'll see if I can correct the corrupted file this way. (I should also metion I'm running 25? 26? 27? levels in this plan... wink )

Posted by mazoola at Nov 25, 2015, 6:33:18 AM
Re: Suddenly, huge file size...
I returned home late last night, and one of the first things on my list for the morning was to see if I could simply delete the corrupted directories and the ContentDigests file from the .sh3d archive and have the plan load correctly. First, though, I wanted to check out a few models I'd built while I was away. (In advance of some podiatric surgery, I'd been helping a friend reorganize her home so all critical living functions would now take place on the ground level. In the evenings, I'd occasionally commandeer one of her PCs and refine some Sketchup modeling on which I've been working...)

The new designs appeared to work well, and after a couple of hours spent tweaking colors and textures, I thought I had the items reasonably well integrated into the home design. I decided I'd break for lunch and tackle the question of hand-editing .sh3d files afterwards. Before I stepped away from the keyboard, though, I hit 'save'...

...which suddenly seemed to be taking an awfully long time to handle the request. Sure enough, when the app finally returned control, my 78 Mb (compressed) plan had suddenly grown to 373 Mb (uncompressed). Inside the .sw3d file, I found item 315 now pointed to a folder containing an almost-complete copy of the plan, weighing in at 195 Mb.

Based on what I hope was an accurate reading of your previous post, I removed the offending file from the zipped plan, along with the entire ContentDigests. As expected, when I attempted to load the edited plan, sh3d responded with a warning the data file was invalid; while I could continue to load the plan, one of its furniture items would have to be removed or replaced with a placeholder object.

Unfortunately, as was the case last week as well, regardless of whether I choose to remove or replace the offending object, sh3d hangs. Last week I thought the process suspended and had to be cleared manually. This week, though, I saw it continued to perform at least some calculations, although the process's CPU utilization dropped to below 0.20%. I couldn't determine if the threads being executed were typical or correct for the application at that stage of invocation, or if they indicated the system had entered into an irrecoverable loop. In any case *I* didn't have the patience to wait a few hundred minutes to see if the system would eventually give me a welcome screen.

Once again, I couldn't identify the moment the corruption took place;. nor could I recall any unusual action I'd taken that could have introduced the error. I'm disappointed I still can't get the edited plan to load, as it means I most likely will have to hand-assemble a replacement plan by taking my most recent known-good file -- which is now three weeks' old -- and manually repeating each change since made. (Obviously, such an approach can only continue to work for me for a limited time.)

Again, any pointers or corrections most gratefully accepted!

Posted by HawkDawg at Nov 25, 2015, 6:44:02 AM
Re: Suddenly, huge file size...
I'm having the same thing happen to me. I'd be interested in why SH3D creates this duplicate folder containing all other duplicated folders and files inside of it, and why - when deleted - SH3D won't load.

Edit 1: Puybaret - I can make my 70 MB file available on one of my websites if you'd like to have a look at it.
The folder with more folders inside is about 25 MB.
I zipped it up using 7Zip. As a zip file it's about 30.5 MB. As a 7z file it's about 19 MB.

Just let me know if you'd like to take a look at it, to try and figure out what's going on, and I'll upload it and give you a link.
----------------------------------------
Hawk

Posted by HawkDawg at Nov 25, 2015, 8:37:43 AM
Re: Suddenly, huge file size...
A copy/paste operation between files could be a possible track,

I think this might be what is creating the folder with the sub-folders and files.
I looked through mine and it looks like it is filled with models I've copied/pasted within the plan, although I didn't look through every folder.
----------------------------------------
Hawk

Posted by mazoola at Nov 28, 2015, 5:14:20 AM
Re: Suddenly, huge file size...
OK, it's not solely linked to copy/paste between files.

The past few days, I've been compressing and checking each save, being careful not to copy/paste from file to file -- and until a few moments ago, it looked as if that might be the answer. Unfortunately, I just saved a plan that should have weighed in around 190 Mb uncompressed, and it ended up over twice that size. Compressing and opening the zipped file shows it contains a recursively saved plan.

The only oddity I noticed was that during the save process the UI seemingly "jumped" -- shifted over a few pixels , with the scroll bar on the object pane disappearing. Whether that's of any help, I don't know....

I have a suspicion or two to investigate, but wanted to pass along the little I've learned.

Thanks,
Maz

Posted by HawkDawg at Nov 28, 2015, 6:28:43 AM
Re: Suddenly, huge file size...
It will be interesting to see what your investigation turns up.
----------------------------------------
Hawk

Posted by mazoola at Nov 29, 2015, 9:54:53 PM
Re: Suddenly, huge file size...
I'd worked for most of a day with no problems, with the plan working its way to about 200 Mb. (I added several complex models built in Sketchup, so the sudden 50 - 60 Mb jump is normal.) However, after making a few tweaks to fix corrections I'd subsequently lost when reverting to an earlier save, I hit 'save' one last time -- and ended up with a 400 Mb file. (Fortunately, I actually hit 'save as,' so I only lost my final half-hour of fiddling about.)

Once again, I *didn't* copy/paste from file to file; however, I *did* perform a number of copy/paste operations from level to level.

I also deleted several [empty] levels I'd mistakenly added by hitting the '+' tab instead of my target.

In addition, I copy/pasted several *walls* -- not just furniture -- from one level to another. (For each of the 6 floors in my plan, I'm using as many as 7 co-located levels, for a current total of 24? 25? levels.)

I also experienced a previously unseen glitch earlier, when, after modifying an object's materials, its normal onscreen display was replaced by a featureless box with a texture applied. (As the texture happened to be the JPEG used as a diffusion map for the standard mirror object, after I'd edited a handful of objects, my plan looked like a mid-1990s showcase kitchen: shiny stainless steel as far as one could see.) Objects rendered without a hitch; only their 2D representations were corrupted.

I don't know if this suggests the recursive bug had occurred before my actual attempt to save, but while the program was laboriously writing its 400 Mb plan to disk, I took a look at sh3d's working directories. Inside the contemporary 'work' folder, I discovered two [auto?] saved plans, *each* weighing in at 400+ Mb. (Unfortunately, I failed to make a note of their respective timestamps, but IIRC, at the time of the save, they were both a dozen or more minutes old.) During the actual save, it appeared that both my .sh3d data file and the .RECOVERY file were being incrementally saved in tandem. Once the save was complete and I exited the program, both the .RECOVERY file and the current work folder were deleted automatically. (However, I still have four work folders dating from the 24th and 27th, totaling 1.1 Gb. I assume these are the result of a program crash -- quite possibly during my unsuccessful attempts to load a .sh3d file from which I'd removed a recursively saved plan -- and can safely be deleted.)

Unsurprisingly, every suspicion I had (primarily involving multitasking during the save process) has proven unfounded, and the problem occurs *just* infrequently enough that when it does, I typically can't reconstruct exactly what it was I'd done since the last problem-free save. When I return to sh3d later tonight or tomorrow, though, I'm planning to log each action, in hopes of nailing this sucker.

Maz

Posted by HawkDawg at Nov 29, 2015, 11:03:15 PM
Re: Suddenly, huge file size...
This is of no help to this particular issue but I thought I'd mention that all folders and files in the work folder can be deleted if they start getting to be too big, with no negative issues, as per Puybaret stating so in this post:

http://www.sweethome3d.com/support/forum/viewthread_thread,5683#24718
----------------------------------------
Hawk

Posted by Puybaret at Dec 2, 2015, 2:44:27 PM
Re: Suddenly, huge file size...
[...] all folders and files in the work folder can be deleted
Yes, but be sure to quit Sweet Home 3D first.

HawkDawg, thanks for proposing a file that shows this issue. I've got already one but yours might reveal something else...

mazoola, thanks for your patience to search a way to reproduce this issue. I don't think that saving the .recovery and .sh3d files at the same time is a problem, because the tasks that handle these operations work on copies of the edited home, so they don't share any data.
----------------------------------------
Emmanuel Puybaret, Sweet Home 3D creator

Posted by HawkDawg at Dec 2, 2015, 4:30:43 PM
Re: Suddenly, huge file size...
HawkDawg, thanks for proposing a file that shows this issue. I've got already one but yours might reveal something else...

I think I sent you a Private Message. I'm not sure if it went through OK or not.
----------------------------------------
Hawk

Posted by mazoola at Dec 3, 2015, 9:06:52 AM
Re: Suddenly, huge file size...
mazoola, thanks for your patience to search a way to reproduce this issue. I don't think that saving the .recovery and .sh3d files at the same time is a problem, because the tasks that handle these operations work on copies of the edited home, so they don't share any data.


Sorry; I should've made it clear I wasn't reporting this because i thought there was a potential link, but simply in case what I was seeing wasn't what you expected. It's been 15? 16? years since I last did any serious Java development -- and, frankly, the only serious thing about it was that I charged someone for doing it; it was embarrassingly simple work -- but in subsequent years and using subsequent languages, I've occasionally run into problems when the language didn't execute concurrent processes the way I expected or did so in differing orders under differing conditions. Since I managed to catch the save processes in the act this time, I figured I'd pass it along.

I realize this is likely to be extremely difficult to reproduce, and likely even harder to correct -- I seem currently to be experiencing it more frequently than most, and even I can't get it to misbehave on demand. (I slapped sh3d around all evening without ever getting it to fail -- and that was after importing by far the ugliest furniture item to date: 50k+ faces, 230+ materials.) Right now I think I have procedural work-arounds in place -- namely, frequent and liberal saves -- so I'm no longer all that inconvenienced by it. I'm going to keep trying to see if I can reproduce the problem consistently, and as I try things I'll continue to post what will likely be completely irrelevant updates and documentation from time to time, just on the off chance something helps you troubleshoot this problem. If (when wink ) what I post is pointless, feel free to ignore it!

Maz

Posted by mazoola at Jan 5, 2016, 8:07:43 AM
Re: Suddenly, huge file size...
So, after more than a month without this problem repeating,* I just now went to save a model that's been weighing in around 255 Mb -- and ended up with a 507 Mb model. Since the previous save, I'd performed no cuts-and-pastes between files or levels; instead, all I did was tweak a half-dozen or so light sources, duplicate another half-dozen or so, resize a couple of doors, and make an object invisible.

Emmanuel, could the problem be as simple as requesting a save while (or right as) the application is auto-saving the model? As best I recall, before I went to click 'save,' I heard my drive laboring as if under an automatic save. That's about the only pertinent potential linkage I noticed.

Maz
__________

* Admittedly, I've not worked with SH3D daily during that period; 1 day out of 3 is probably a good estimate.

Posted by Puybaret at Jan 5, 2016, 10:24:34 AM
Re: Suddenly, huge file size...
Thanks for keeping posting your reports about this issue. I'm sure all your comments will finish to give me the idea that will help to fix this bug.
Autosave and save tasks work on copies of a home and on different files, so even if they run at the same moment, I don't think this should cause any problem.
----------------------------------------
Emmanuel Puybaret, Sweet Home 3D creator

Posted by Puybaret at Jan 5, 2016, 6:51:39 PM
Re: Suddenly, huge file size...
I may have found a new track: exploring the data in two files showing this bug, I discovered that the error could come from a material that references a texture image of an other 3D model in a home file. A simple way to check this issue in your sh3d file is to search an occurence of ".jpg", ".bmp" or ".png" in its ContentDigests entry. In a clean file, no occurence shouldn't be found. In a buggy sh3d file, the folder entry that contains a duplicate home will appear at the line where ".jpg", ".bmp" or ".png" is found. For example, if that line contains:
Name: 117/110/texture.jpg
the buggy folder is 117.
Having such references isn't a blocking issue, since Sweet Home 3D is able to open those files. The problem comes from the saving routine that save such an image with all the home data from which it comes, instead of saving just the texture image itself. I could try to change this routine to avoid this behavior, but I would prefer to avoid such references.
Unfortunately, I didn't find any way to produce such a reference and then a buggy file yet. I tried to copy/paste objects with modified materials from a file to another one, to paste style between such objects, to export/import such objects. Every time, the saved file was correct.
Hope you can now help me to reproduce the bug more easily with this track in mind.
----------------------------------------
Emmanuel Puybaret, Sweet Home 3D creator

Posted by Puybaret at Jan 6, 2016, 12:18:38 PM
Re: Suddenly, huge file size...
No need to search further. I found how to reproduce this bug and as it's actually easy enough to provoke it, I fear it could happen often enough sad
Here's the simplest way to provoke it:
- In a new file, add a piece that contains at least a texture
- Save the file and quit Sweet Home 3D
- Reopen the file and edit the materials of the piece it contains
- Select the material that references a texture and click on the Texture radio button
- Confirm and save. The sh3d file gets buggy and will contain the file hierarchy a second time.
If you want to fix the error in your sh3d file by yourself, it's quite easy once you found the piece that uses its own texture in one of its materials: edit its materials and select the Unchanged radio button for this material in the Furniture materials dialog box.

Expect soon a beta version that will avoid this bug to happen again (but I don't think it will be able to fix existing files). It shouldn't be too difficult to fix smile
----------------------------------------
Emmanuel Puybaret, Sweet Home 3D creator

Posted by mazoola at Jan 6, 2016, 12:46:34 PM
Re: Suddenly, huge file size...
Well, that description certainly applies to my current 'double-stuffed' project: ContentsDigests entry 61 contains a reference to texture.png -- and directory /61 contains the recursively saved house plan.

Is there a way to determine which object(s) contain the offending texture? In mentally retracing the steps leading to the corrupted save, I can't recall adding any texture-containing objects to the plan. (As best I can remember, the only new objects since the previous, non-corrupt save were light sources I copy/pasted from the save level.)

The only seeming constant I've found so far has had to do with time since last save -- which is partly why I asked about autosave. In every instance I can recall, a corrupted save occurred only when I'd been working on a unsaved plan for longer than, say, 20 minutes. Not all prolonged work times resulted in a bum save, but I've not yet saved a file open less than 15 - 20 minutes and had it double in size.

Also, in an earlier reply you said

The ContentDigests entry in a .sh3d file gives the SHA-1-Digest value of every entry of the .sh3d file used by Sweet Home 3D. This value is used to quickly check whether an entry is correct or not. So if you edit an entry in the .sh3d zipped file, Sweet Home 3D will think the file isn't correct anymore. I didn't program this feature to avoid users to manually edit an .sh3d file of course, but to check if an error on the disk didn't happen and inform the user accordingly. If you want to bypass this data integrity check, just remove the ContentDigests entry of a .sh3d file before opening it with Sweet Home 3D.


---which suggests to me that I should be able to save-and-compress a corrupted design, open the compressed .sh3d file in an archiver, delete the spurious directory, delete ContentDigests, and load the modified .sh3d file. When I try this, SH3D starts to load the plan but then pauses with a 'remove, replace, or halt' warning. If I respond with either remove or replace, the app simply stops the load. This occurs with the offending directory removed or merely left empty and with ContentsDigests deleted or merely edited to remove any reference to the directory. Am I missing something here? Being able to hack the plan back to size and not lose the time worked since the previous save would be of great benefit -- even if I had to reimport and position an object.

Thanks,
Maz

Posted by mazoola at Jan 6, 2016, 1:21:18 PM
Re: Suddenly, huge file size...
Ah, the joy of simultaneous entry! smile Feel free to ignore my previous reply.

I can easily understand why I occasionally trigger this bug: At times I want to apply a texture to an object to match an existing object. Rather than go to the trouble of hunting through my texture list, I've gotten into the habit of opening an object with the desired texture, going into materials, reselecting the existing texture, and saving the [unchanged] object. This adds the texture thumbnail to the quick-pick array across the bottom of the texture selecter, making it easy to apply the texture elsewhere. I use the same technique with colors, as well.

This also explains why the trouble only kicked in with a vengeance for me with v 5.0: After I installed the latest version and loaded your predefined texture libraries, it still left me without a number of textures I'd used under 4.x. Although I still had the original images, in many cases I didn't recall at what resolution they'd originally been imported. Rather than go through a trial-and-error process to identify the correct size mapping,* I discovered I could make even textures not present in the current libraries available through the above steps. Unsurprisingly, that's when the trouble started.

My recent month+ without save errors most likely reflects I've been working primarily with newly created objects (and, accordingly, standard or newly loaded textures).

Given this latest information, I now recall that one of the changes I made prior to my latest bum save was an attempt to address the undefined face problem with some Sketchup models: I modified the materials in a long-existing object in hopes of correcting a photo texture that would occasionally disappear depending upon its orientation in relation to the camera. The texture in question no longer exists in the 5.0 library, so I picked it up from the existing object definition. Since the object had been present in the plan since May, it never occurred to me it could have contributed to the problem -- and, in fact, I'd completely forgotten I'd changed it.

I'm currently not at home, and for some reason I can no longer run SH3D on my system here. As soon as I can, I'll see if I can duplicate the problem. (Do I understand your last reply to mean I can open my 1/2-gig plan, either delete the offending object *or* change its material definition to remove references to the texture, and re-save the file once again in its 250-meg glory?)

Posted by mazoola at Jan 6, 2016, 1:26:06 PM
Re: Suddenly, huge file size...
Oops. The asterisk in the previous entry should be footnoted as such:

* It would be nice if the dimensions at which a texture had been imported was visible in the texture selecter. It'd be even better if the dimensions were editable and thus could be changed without having to re-import the image.

Posted by mazoola at Jan 8, 2016, 1:40:32 AM
Re: Suddenly, huge file size...
FYI, I opened the 500+ Mbyte .sh3d file with the recursive save and replaced the furniture item containing a reference to a texture used in an object (that is, as opposed to textures accessed through the texture picker window) with a corrected item, and re-saved the file -- which suddenly shrank to 253 Mb. biggrin

So, yeah -- I think you've got this one figured out.

Many, many thanks!

Posted by Puybaret at May 11, 2016, 9:29:04 AM
Re: Suddenly, huge file size...
Hi,

I recently was in contact with a user of Sweet Home 3D who uses very often the list of recent textures in the textures choice dialog box to copy a texture from a material of an object to a different one. She used this tip so often with versions of Sweet Home 3D older than 5.2 that some of her files were larger than 50 GB! This is not a typo, you correctly read GB and not MB! rolling eyes
Beside the amazing fact that Sweet Home 3D was still able to read and write such huge files (with a lot of patience), I couldn't let her continue to lose her time on this bug. There was no problem with brand new files made with version 5.2, but as soon as she copied/pasted some objects coming from sh3d files designed with older versions, there was a risk that the final file get uselessly huge.
As she wasn't able to fix the .sh3d files by herself, I finally programmed a small tool named Huge File Cleaner that removes useless entries in her .sh3d files. She successfully tested this tool on different files (which are now always smaller than 100 MB), that's why I propose this tool here (7 KB - source code included) too, in case you encounter the same problem.
The tool is a Jar executable file, so once downloaded, just double click on it to launch it if Java is installed on your computer. The tool will request you to select a .sh3d file to fix, will analyze the file, and if it founds a potential issue, it will request to select another .sh3d file where the fixed file will be saved (files must be different). Then, starts the operation that copies only necessary files from the buggy .sh3d to the destination file. At the end, just check the saved file in Sweet Home 3D.
Note that the destination file is always compressed, so it may be much smaller just because the original file wasn't compressed.
----------------------------------------
Emmanuel Puybaret, Sweet Home 3D creator


RSS feeds
Powered by mvnForum 1.2.2 GA
© Copyrights 2002-2010 by MyVietnam.net
Current timezone is GMT
Dec 18, 2025, 8:26:06 AM