Saving the world!…to disk?

The problem:

Since the inception of Zombox almost 5 years ago, one thing I’ve never addressed is how the save system will work.

For a long time in gaming, it was fairly common for game designers to provide players with some kind of manual ‘save’ menu (whether in a user interface, or within gameplay itself). But modern players — especially on mobile — just assume that their progress will be saved when they close the game, and resumed when they re-open it. Implementing an auto-save feature which provides that functionality can come with many challenges…especially for Zombox. To list a few:

  • Zombox is a massive open-world game with a city spanning over 20,000 individual ground tiles, and over 100,000 individual interactive objects per map. Since players can change the state of these objects at any time (destroy, repair, put items in, take items out, etc), their states need to be saved between plays.
  • Zombox needs to run smoothly on mobile devices, and can’t freeze every time the player needs to save.
  • Due to the constantly changing nature of the game world, saves need to happen often.
  • Save data needs to be small in size, because nobody wants their mobile device storage space eaten up by huge save files

Originally I had brainstormed several solutions to these problems. For example, since Zombox maps are generated procedurally from a random seed, what if I just saved the seed of the map and re-generated the world from that seed when the player reloads the game? Ignoring the fact that this wouldn’t retain any information about what objects the player has changed in the world, the problem with this solution is that it means I have to be very rigid with future updates to the game’s internal city tiles (the building blocks of each map). For example, what if a player builds his shelter in the middle of a grass tile, and then in a future update I place a tree in the middle of grass tiles. Now when the player reloads his game, there will be a tree intersecting the center of his shelter! And that’s only a basic example — it could end up much worse. The idea that a player’s save data could be ruined if the game’s internal city tiles don’t remain perfectly static is a bit of a dealbreaker for that idea.

As mentioned, that solution also ignores the problem of a player’s changes to objects in the game world needing to be saved too. For example, if I just re-generate a city from a saved seed, all of the objects that the player constructed, destroyed, or used as containers, etc…will be lost. I could just limit my save data to only the objects that the player has changed in some way, but there’s no guarantee that such data wouldn’t eventually encompass all objects in the game world, if the player plays long enough.

With those issues in mind, it was obvious that save data was going to have to include virtually all data pertaining to a particular city, because using cheats to skip over some data could lead to big problems down the road.

At this point I began looking into the serialization of raw city data, as a way to retain all important information about a particular Zombox world. For those of you who don’t know: serializing data involves converting in-memory classes, properties, fields, variables, etc, into a format which can then be saved to disk. The nice thing about serializing data, is that deserializing it allows you to convert that data back into the original objects in memory. So saving an entire city’s worth of data could be as simple as the following pseudocode:

Serialize(cityArray, cityFile);

…and loading it from disk could be as simple as:

cityArray = Deserialize(cityFile)

I got this working pretty quickly using built-in BinaryFormatter/MemoryStream classes….but it had some issues:

  • It’s very slow and could never perform in realtime on mobile. If any kind of auto-save function was implemented into Zombox using this method, players could expect 10-30second in-game freezes every time it activates.
  • It generates a ton of garbage. “Garbage”, in C# terms, is any unreferenced object that was allocated to the managed heap. Explanations about how it works are not necessary here, suffice to say a single serialization of my city data generates about 450mb of garbage — which is more RAM usage than a lot of mobile devices can handle for a single app. Using this method would result in the app crashing as soon as it tries to save. That’s not even mentioning the amount of time it would take C#’s garbage collector to clean up the memory afterwards.
  • Marking all of my city data classes as ‘serializable’ generates a ton of Unity warnings/errors. For some reason, Unity really doesn’t like the hierarchy that my city object classes exist in (unity doesn’t like when serializable classes have self-same members beyond a certain depth). While this didn’t prevent the game from compiling, it meant that I’d get about 650 error popups in the Unity log each time it reloads my scripts….very annoying to say the least! The errors go away when the ‘serializable’ attribute is removed, but I would need that attribute enabled in order to use the built-in serializers.
  • The resulting save data file is about 20mb in size. That’s far too large and would cause a pretty severe hiccup in gameplay while writing to a device’s storage during gameplay.

After the failure of this method, I started looking into BinaryFormatter/MemoryStream alternatives. Are there more performance-optimized ways to serialize data? I looked into Protobuf, FlatBuffers, UnitySerializer…all of them came with their own issues that didn’t solve all of the problems mentioned above. The main issue is that they all generate lots of garbage, cannot be easily converted to coroutines in order to run them in realtime, and the resulting save files are still too bloated.

Then, while studying the way FlatBuffers worked, I had an epiphany:

Instead of trying to find a way to serialize/deserialize my city data in a single step (a process that is undoubtedly slow and memory-inefficient), why don’t I manually recurse through all of the city data, saving out only the parts I need in a simple binary format? Maybe to some this would seem like an obvious solution, but with the vast majority of developers on the internet recommending serialization methods as a way to save game data, it was something I hadn’t considered while doing my research. Here are the benefits of manually exporting binary data, as opposed to using an external serialization library:

  • You save the step of converting your serialized data to a byte[] array (for file export), and avoid doing any string conversions, thereby minimizing garbage generated by the conversion process.
  • You have precise control over what gets saved, thereby reducing memory/filesize bloat
  • Implementing a custom, granular save system allows you to do things like run it inside of a coroutine, so a single save can happen over multiple frames. This can give a huge boost to overall performance when saving during active gameplay.

But how do you convert in-memory objects to raw binary data? The process is actually quite simple. Imagine the following class:

class foo
{
	public int bar;
}

We can create an instance of it in memory, by calling:

foo theFoo = new foo();

And we can change its bar variable by calling:

theFoo.bar = 10;

Now, if we want to save that instance to disk and load it later, we only need to know three pieces of information:

  1. theFoo’s class type
  2. bar’s value type
  3. bar’s value.

We can store the first two pieces of information as single bytes, and since bar is an integer type, we can store it’s value with 4 bytes. So the entire object can be efficiently saved to disk using only 6 bytes of storage!

In the Zombox save system, all information pertaining to object/value types is categorized inside a single enum, whose values do not exceeed 255 (hence the reason why they only require 1 byte of storage). For the above example, I might write the enum out like this:

    public enum dataTypes
    {
        none = 0,
        class_foo = 1,
        value_integer = 2
    }

Enums are integers by default (so they are 4 bytes in size), but as long as none of the values inside of our dataTypes enum exceed 255, we can easily convert it to/from a single byte. So the actual save function pseudocode would look like this:

public class foo
{
    ...
    //pseudocode
    public void Save (FileStream file)
    {
        file.Write((byte)dataTypes.class_foo);
        file.Write((byte)dataTypes.value_integer);
        file.Write(System.BitConverter.GetBytes(bar));
    }
}

In the above psuedocode, ‘GetBytes’ would actually generate a small amount of garbage each call, since it’s returning a new byte array (byte[]) that it created…but you can avoid this by caching and re-using your own byte arrays and passing them as an argument to that function (which is what I do in Zombox).

So, we’ve got our basic save system implemented….how would we load that data? Well, we’d simply parse through our save file and perform actions based on the bytes we read. Given the above example, here would be the corresponding load function pseudocode:

public class theLoader
{
    public void Load(FileStream file)
    {
        //psuedocode
        byte readType;
        byte[] readData;
        while (file.readByte(readType))
        {
            if ((dataTypes)readType == dataTypes.class_foo)
            {
                foo newFoo = new foo();
                file.readByte(readType); //get the type of the next data
                if ((dataTypes)readType == dataTypes.value_integer) //next value in the file is an integer...ie, our bar variable!
                {
                    file.readBytes(readData, 4); //read 4 bytes for integer
                    newFoo.bar = System.BitConverter.ToInt32(readData); //convert those 4 bytes to an integer, and assign to bar
                }
            }
        }
    }
}

Obviously in that function we don’t actually do anything with the new foo object we instanced, but that’s not relevant for this illustration to work. The fact that we’ve managed to successfully save and load our class, while incurring a minimal performance hit (and with the right caching structures in place, zero garbage), is what we want!

There are also a few other subtleties to mention. For example, what if our class has 2 ints? Then our dataTypes marker would need some more information to tell us which int we should assign the next piece of data to. Also, if you are sure your class will never change, you don’t need the dataTypes markers for the internal variables at all…you could just read all the data in the order it was saved (but I wanted more flexibility in the way Zombox saves/loads data, so peoples’ save files wouldn’t instantly become corrupt if I changed the structure of any saveable classes in the future).

So, basically that’s what I do in Zombox. My save/load code is much more complex, but the gist of it is that I recurse through all of my city data, saving out the necessary information required to recreate it all again at load time, while minimizing the total number of storage bytes and memory allocations required to do so. As for performance, here is the result:

  • Due to some creative caching, my method incurs zero garbage from start to finish during a save.
  • My method runs inside of a coroutine, which means auto-saves can occur during gameplay with no real impact on performance. A full save happens once every 90ish seconds during gameplay (approximately 2,000 city objects are processed per frame), as well as whenever the game is paused or exited (meaning that data should never be lost if the game functions normally, and only data changed in the last 90 seconds will be lost if a crash occurs or you force-quit the app).
  • Direct file access in my method is double-buffered. All save data is written to a temp file until the save is complete, at which point the real save file is quickly overwritten with the temp file by the OS. This ensures that if a game crash happens mid-save, only the temp file will be corrupted — not your actual save file.
  • The total filesize for an average Zombox save using this method, which contains all city data, is just 1.5 megabytes (which zlib can further compress to just 100kb!). When left uncompressed, that’s still over 10x smaller than files created with other serialization methods I tried!

So there you have it. Zombox now features a robust auto-save system that runs fast on mobile! Thanks for reading!

34 Responses to “Saving the world!…to disk?”

  1. Fadi98n says:

    Awesome, I really love all these clever solutions you find for problems in the game, and the detailed description of how it works is really appreciated.

    Keep up the good work :)

  2. KristopherPeterson says:

    While my knowledge of programming and Unity isn’t quite at this level, I still appreciate all the specific methods and detail that you put into this post, and will probably refer back to it at a later time.

    Thank you!

  3. wokste says:

    Very interesting and nice you found a working solution.

    I also know some things about loading / saving. One thing that I know about many open-world games is that they use a kind of chunk mechanic. Basically you split the map in chunks and each chunk is loaded/saved seperately. This some advantages:
    1. First of all you don’t need to load or save everything
    2. Because of (1) you could often use much bigger maps.
    The disadvantage is of course that you need to somehow save the map in variable chunk-sized objects.

    I hope this helps.
    Wokste

    • Tyson Ibele says:

      Yea I had started looking into a chunk-based approach, but once I realized the speed and ease of the method detailed in this post, the extra effort required to manage the world with chunks wasn’t worth it. But for those making a traditional voxel game with a higher density of data points (thousands of voxels per chunk as opposed to dozens of objects per tile in Zombox), it is a must.

  4. Fadi98n says:

    Hey Tyson, i noticed something in the gif above that i wanted to ask you about: in the gif there’s a mark on the mini-map that seems to indicate that you can go down to the sewer below, but i remember you saying that NPC sewer hideouts are marked with blue markings on the hatch, does that mean we will be able to enter non-npc hideout sewers? maybe ones overrun by zombies or completely empty.

    And another question, can npc hideouts be taken over by zombies/bandits (are there even bandits in the game?)

    One last thing, does this blog use markdown or only html tags?

    • Tyson Ibele says:

      Hey Fadi,

      You can go down all sewers in the game. Currently there are 3 sewer types: NPC bases (large sewers that have lots of NPCs and shops), NPC one-off shops (small sewers that have a single shop), zombie sewers (sewers overrun with zombies).

      Right now all the map markers for sewers are the same if I remember correctly, even though NPC bases should be marked with a blue x on the manhole. At some point I should color-code the map markings better, probably.

      As for zombies taking over things, currently NPC sewers are protected and will never be overrun with zombies. I wanted players to have a safe place they know they can always go to without having to worry about zombies being there to. Maybe in the future I’ll implement a zombie take over system of some sort….

      And currently the blog is only html.

      • Fadi98n says:

        Thanks for the reply, i think keeping them all the same color is better, like you’re running from a horde and see a marker for a sewer and start heading towards it hoping that it’s an NPC hideout, only for it to turn out to be a zombie sewer so you’re now on the run again, that feels awesome just thinking about it.

  5. rene0121 says:

    gahhh I am so excited. despite all the coding (or is programming) jargon. I always love to see these updates. Also I would just like to mention I appreciate that you’ve written out detailed descriptions of your intuitive solutions, it’s generous and contributive to the community of fans and your colleagues alike. If you started a patreon or if you currently have one I’d love to contribute to it :D .

  6. Chris says:

    Hey, don’t know if you respond to these, but I used to check this blog daily when it first went live (essentially) and made some YouTube videos about this that got a couple thousand views… Headed to college now 4 years later. Something made me think of this game. Hope you are doing well and wanted to say keep doing what you are doing. And btw a PC release w coop is still a great idea (especially these days).

    • Tyson Ibele says:

      Hey Chris,

      Yup! I read these comments.

      I always feel bad when I think about the people who’ve been following the game for so many years, without being able to play it yet. Unfortunately a massive undertaking like this is something that takes a huge effort and time commitment, especially when I’m the only person working on it. As you may have seen I also took some breaks from it over the years (developing other games and things) just because I needed to debrief a bit after chugging on it for so long.

      I’ve made big strides in the last few months, and besides the quest system the game is nearly complete. With the save system added, and some other things that I haven’t posted about, it finally feels like a game and not just a procedural city engine with some bells and whistles.

      Good luck at college, and thanks for stopping by!

      • Fadi98n says:

        no need to feel bad, i’ve been following the game for 4-5 years and even though i’m itching to play the game, it’s awesome to see how far the game has come, and i’m still as excited when i see a new update as i was all those years ago, take your time, a working game is better than a broken one, and i’ll still be here when the game comes out.

    • Fadi98n says:

      hey can i get a link to your channel/mentioned video, i was introduced to this game via a YT video and want to know if was yours

  7. Chris says:

    I went through your YouTube videos and One quick thing: Is there any way that zombie killing can be a bit more visceral? As it stands now enemies just sort of “pop” into two or 3 squares and a blood splatter. I remember in the first few posts zombie deaths had a satisfying look to them (post 7 on YouTube for example), but now it looks a little anticlimactic haha. Of course I don’t know what goes on behind the scenes with that or why it was changed, but personally I liked how it was.

    • Tyson Ibele says:

      Could you link to the clip you saw that you feel is disappointing, just for reference? I watched clip#7 on youtube again and I think zombie deaths have actually improved quite a bit since that one. There’s a better decal/splat system being used now, better zombie dismembering (you can visible remove limbs, break skulls, etc)…more blood and better blood animation system (it’s now directional and proportional to the attack)…

      It’s possible I just haven’t really shown those things in a while though, since most of the clips I’ve posted have been for stuff other than zombie killing.

  8. Yuriq says:

    Ha, one thing to consider is PZS – when you quit a zone (chunk of map), save that chunk, without the Zeds but with Zed graves that weren’t destroyed. Use thin mist to indicate that you left a chunk behind and it will be saved over 5 seconds once you’re sufficiently far.

  9. Evler says:

    How is it that no game that exists today has anything on this title? NPCS, destructin, loot, driving, this game has it all. And it’s a freaking IOS title. I’m about to throw my steam machine away when this releases. I’ve been watching this device for 3 years, hurry the fudge up on a beta we can purchase or something.

    • Spracky says:

      I as well have waited 3 years, not sure how much longer I can wait D: I wish we could at least get another update, Even if nothing has gone on… Scares me to think this project is dying.

  10. Ben Cornell says:

    Anyone saying this project is dying its not. Hes almost done with it and Im super pumped. I could wait indefinitely. Ive been following for three years and never decided to comment. But now i am :)

  11. danny says:

    Hello,

    I just wanted to let you know that i am checking this place almost once a day!
    Keep up your amazing and hard work!
    Zombox looks gorgeous in style and gameplay :)

    Thank you.

  12. iRYO400 says:

    Hey, Tyson!
    How it’s going?

  13. Wine The Poo says:

    Dear Tyson please please give us an update for this game I have been following for 3 years and I hope that it’s almost done but don’t rush it because rushed games are never good just please release something telling us what’s happening

  14. Spracky says:

    Are we going to get an update soon? Even if it’s just an update to tell us you haven’t done much to zombox, I’m fine with that.

    • Tyson Ibele says:

      Hey Spracky,

      I’ve actually got a ton of new things done and added to the game….I may hold off posting about them until the beta is ready or I might spill the beans sooner…we’ll see :)

  15. i’m able to find numerous great solutions if i have any difficulty!

  16. Moises Alvarado says:

    Awesome solution to your problem! It’s been a while since the last update how are things going?

  17. sean says:

    still looking forward to this game – any chance it’ll see the light of day?

    You should repurpose it as a vr game w/ 3rd person controls!

  18. Dan_patate says:

    Ive been watching since i was in highschool, and im happy to see that a lot of people keep looking for an update eaven after à few years ( im one of them :p ). Trust me Tyson, the day you will finish à bêta ready for the people, your game will be know like minecraft&Notch and unturned&Nelson. Il be there to see you shine! (By that time, il keep playing Quest keeper !!)

  19. Evaldas says:

    It’s so fun to read the comments – I’m following this page for years as well, never posting but every time silently hoping for any news, even those with “have been busy lately, did not do much”.

    Tyson, keep on doing what you’re doing, I think we all – the silent lurkers as well as participants – have proved to ourselves that we have the patience :)

Log in or Register

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>