Save File Bloat, and UI Cleanup

Hey Folks! A bit more behind-the-scenes work today, and I tried to tackle bloat in the save file, and did some minor UI cleanup.

Yesterday, while working on the IMB build, I noticed that my save file was 40MB. This is a plain text, json file, with over 1.3 million lines of text! As I was checking-in the file to version control, I thought to myself, "this is going to be a problem."

In case you're wondering why, today's screenshot tries to illustrate. That grid at the top is a space station. The grid lines are for ship tiles, and each tile can have 0 or more items stacked in it. (E.g. a single tile might contain a floor + wall + conduit + light, while another might just be a floor). This station has thousands of items in it.

In the final game, these stations will likely need to be bigger and more complex.

Similarly, ships are about as complex as this station right now. Hundreds or thousands of tiles per. These will be more varied in the final game, but also more numerous.

And speaking of numerous, the bottom right of the screenshot shows most of the ships and stations currently loaded in the System. 4-digit callsigns are stations, the rest are ships. That's just the major stations added so far, plus a cluster of ships in the K-Leg junkyard, and the player's tiny salvage pod (docked to the top right airlock of this grid). Each of these is like one of those grids above.

It's only going to get worse from here.

So I did a little tinkering to see if I could reduce that file size. There was a lot of "property = null" type stuff in there, so that was the first to go. No need to save data that doesn't exist! Ditto for arrays and lists that were effectively empty sets.

I also noticed I was saving message logs for things which probably didn't need them. We didn't need to know the history of ItmWall1x1, which consisted of "ItmWall1x1 is a wall." over and over as it gets saved/loaded each time. Ditto for floors, conduits, and the like.

Removing the above cut the file size down to 26MB. Still a huge file, but a big improvement.

I started looking into trimming it further, but I think it gets more complex from here. One possible improvement is replacing GUIDs with something smaller. GUIDs are fairly long strings, and they appear in the data at least 3 times for each item instance. (Mainly as a way to link data in separate tables together.)

I could probably replace each of these with a unique integer and save a ton of space. Especially if I could get the deep references, too. (Stuff in AI memory and user-defined settings in UIs.) However, it means remapping values on save and load, and that gets complicated. Doable, but complicated.

I also noticed that the vast majority of the data is composed of a repeated block of data with slightly different values each time. I could probably trim a bunch of fat by clumping similar data together, and just saving the diffs between each, and do away with all the block formatting, whitespace, and braces.

This, however, is also a bit more complicated. It'd require me to divert the save/load code in some places with special case stuff, and that gets tricky when we're dealing with ships that are fully-loaded vs. those that are partially loaded.

In the end, I decided to stop here with the partial improvement.

Apart from that, I also did a bit of clean-up on the dialogue UI. There were occasions where blank data appeared, leaving "blah blah, , , blah blah" type strings. So I found the causes of those (mostly hidden conditions that were accidentally set to visible), and fixed them. It's starting to look much cleaner.

From there, I got started thinking about what to do next. Part of me wants to overhaul the UI to make personal status and interactions easier to view, and more engaging. Part of me wants to flesh-out more technical systems. Part of me wants to start implementing audio.

There's a ton left to do, and I could really use this weekend to think it over. So I'll go do that :) Have a good one, all!

Tags: Ostranauts


ra1's picture

For large data-sets there are much more compact/efficient formats than json. If you like the flexability of json, you can still support it, but set the default to be a different format.

If you don't like that option, then at least get rid of guids, and create a dictionary for storing common key names/values in json.

dcfedor's picture

What other formats did you have in mind? Are any of them as intuitive for modders as json?

As for GUIDs, that was my thinking, too. I could remap GUIDs to ints or something at save time, then reverse the process at load time. I'd expect a ~30% savings, just eyeballing the data.

I actually started doing this, but there's a bit more refactoring than I was anticipating. So I'll likely revisit again later, when I have the time to do it right.

Dan Fedor - Founder, Blue Bottle Games

jenden's picture

If your objective is mainly to just reduce storage space and IO overhead for load/save you might want to think about streaming the file through a compression library before writing to disk. It sounds like the json file will be highly compressible, so even on a fast/low compression setting you'd probably get a large size reduction (at a bit more CPU cost, but the CPU would be idle during the IO anyway)

dcfedor's picture

I think this might end up in there one way or the other. The compression I saw with a typical zip format was down to 3MB, which is a lot more reasonable. And I hear Unity has one or more compression libs I could use.

Dan Fedor - Founder, Blue Bottle Games