{"id":589,"date":"2015-12-04T15:15:46","date_gmt":"2015-12-04T22:15:46","guid":{"rendered":"http:\/\/www.zombox.net\/?p=589"},"modified":"2015-12-04T17:42:49","modified_gmt":"2015-12-05T00:42:49","slug":"saving-the-world-to-disk","status":"publish","type":"post","link":"http:\/\/www.zombox.net\/?p=589","title":{"rendered":"Saving the world!&#8230;to disk?"},"content":{"rendered":"<p><strong>The problem:<\/strong><\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone\" title=\"Zombox save system\" src=\"http:\/\/apps.tysonibele.com\/zombox\/screencaps\/zombox_cap_155\/zombox_cap_155.gif\" alt=\"\" width=\"480\" height=\"320\" \/><\/p>\n<p>Since the inception of Zombox almost 5 years ago, one thing I&#8217;ve never addressed is how the save system will work.<\/p>\n<p>For a long time in gaming, it was fairly common for game designers to provide players with some kind of manual &#8216;save&#8217; menu (whether in a user interface, or within gameplay itself). But modern players &#8212; especially on mobile &#8212; just assume that their progress will be saved when they close the game, and resumed when they re-open it. Implementing an auto-save feature which provides that functionality can come with many challenges&#8230;especially for Zombox. To list a few:<\/p>\n<ul>\n<li><strong>Zombox is a massive open-world game<\/strong> with a city spanning over 20,000 individual ground tiles, and over 100,000 individual interactive objects per map. Since players can change the state of these objects at any time (destroy, repair, put items in, take items out, etc), their states need to be saved between plays.<\/li>\n<li><strong>Zombox needs to run smoothly on mobile devices<\/strong>, and can&#8217;t freeze every time the player needs to save.<\/li>\n<li>Due to the constantly changing nature of the game world, <strong>saves need to happen often<\/strong>.<\/li>\n<li><strong>Save data needs to be small in size<\/strong>, because nobody wants their mobile device storage space eaten up by huge save files<\/li>\n<\/ul>\n<p><!--more--><\/p>\n<p>Originally I had brainstormed several solutions to these problems. For example, since Zombox maps are generated procedurally from a random seed, what if I just saved the seed of the map and re-generated the world from that seed when the player reloads the game? Ignoring the fact that this wouldn&#8217;t retain any information about what objects the player has changed in the world, the problem with this solution is that it means I have to be very rigid with future updates to the game&#8217;s internal city tiles (the building blocks of each map). For example, what if a player builds his shelter in the middle of a grass tile, and then in a future update I place a tree in the middle of grass tiles. Now when the player reloads his game, there will be a tree intersecting the center of his shelter! And that&#8217;s only a basic example &#8212; it could end up much worse. The idea that a player&#8217;s save data could be ruined if the game&#8217;s internal city tiles don&#8217;t remain perfectly static is a bit of a dealbreaker for that idea.<\/p>\n<p>As mentioned, that solution also ignores the problem of a player&#8217;s changes to objects in the game world needing to be saved too. For example, if I just re-generate a city from a saved seed, all of the objects that the player constructed, destroyed, or used as containers, etc&#8230;will be lost. I could just limit my save data to only the objects that the player has changed in some way, but there&#8217;s no guarantee that such data wouldn&#8217;t eventually encompass all objects in the game world, if the player plays long enough.<\/p>\n<p>With those issues in mind, it was obvious that save data was going to have to include virtually all data pertaining to a particular city, because using cheats to skip over some data could lead to big problems down the road.<\/p>\n<p>At this point I began looking into the <strong>serialization<\/strong> of raw city data, as a way to retain all important information about a particular Zombox world. For those of you who don&#8217;t know: serializing data involves converting in-memory classes, properties, fields, variables, etc, into a format which can then be saved to disk. The nice thing about serializing data, is that <em>deserializing<\/em> it allows you to convert that data back into the original objects in memory. So saving an entire city&#8217;s worth of data could be as simple as the following pseudocode:<\/p>\n<pre class=\"brush: csharp; title: ; notranslate\" title=\"\">\r\nSerialize(cityArray, cityFile);\r\n<\/pre>\n<p>&#8230;and loading it from disk could be as simple as:<\/p>\n<pre class=\"brush: csharp; title: ; notranslate\" title=\"\">\r\ncityArray = Deserialize(cityFile)\r\n<\/pre>\n<p>I got this working pretty quickly using built-in BinaryFormatter\/MemoryStream classes&#8230;.but it had some issues:<\/p>\n<ul>\n<li><strong>It&#8217;s very slow and could never perform in realtime on mobile<\/strong>. If any kind of auto-save function was implemented into Zombox using this method, players could expect 10-30second in-game freezes every time it activates.<\/li>\n<li><strong>It generates a ton of garbage<\/strong>. &#8220;Garbage&#8221;, in C# terms, is any unreferenced object that was allocated to the managed heap. Explanations about how it works are not necessary here, suffice to say a single serialization of my city data generates about 450mb of garbage &#8212; which is more RAM usage than a lot of mobile devices can handle for a single app. Using this method would result in the app crashing as soon as it tries to save. That&#8217;s not even mentioning the amount of time it would take C#&#8217;s garbage collector to clean up the memory afterwards.<\/li>\n<li><strong>Marking all of my city data classes as &#8216;serializable&#8217; generates a ton of Unity warnings\/errors<\/strong>. For some reason, Unity really doesn&#8217;t like the hierarchy that my city object classes exist in (unity doesn&#8217;t like when serializable classes have self-same members beyond a certain depth). While this didn&#8217;t prevent the game from compiling, it meant that I&#8217;d get about 650 error popups in the Unity log each time it reloads my scripts&#8230;.very annoying to say the least! The errors go away when the &#8216;serializable&#8217; attribute is removed, but I would need that attribute enabled in order to use the built-in serializers.<\/li>\n<li><strong>The resulting save data file is about 20mb in size.<\/strong> That&#8217;s far too large and would cause a pretty severe hiccup in gameplay while writing to a device&#8217;s storage during gameplay.<\/li>\n<\/ul>\n<p>After the failure of this method, I started looking into BinaryFormatter\/MemoryStream alternatives. Are there more performance-optimized ways to serialize data? I looked into Protobuf, FlatBuffers, UnitySerializer&#8230;all of them came with their own issues that didn&#8217;t solve all of the problems mentioned above. The main issue is that they all generate lots of garbage, cannot be easily converted to coroutines in order to run them in realtime, and the resulting save files are still too bloated.<\/p>\n<p>Then, while studying the way FlatBuffers worked, <strong>I had an epiphany<\/strong>:<\/p>\n<p>Instead of trying to find a way to serialize\/deserialize my city data in a single step (a process that is undoubtedly slow and memory-inefficient), why don&#8217;t I manually recurse through all of the city data, saving out only the parts I need in a simple binary format? Maybe to some this would seem like an obvious solution, but with the vast majority of developers on the internet recommending serialization methods as a way to save game data, it was something I hadn&#8217;t considered while doing my research. Here are the benefits of manually exporting binary data, as opposed to using an external serialization library:<\/p>\n<ul>\n<li><strong>You save the step of converting your serialized data to a byte[] array (for file export), and avoid doing any string conversions<\/strong>, thereby minimizing garbage generated by the conversion process.<\/li>\n<li><strong>You have precise control over what gets saved<\/strong>, thereby reducing memory\/filesize bloat<\/li>\n<li><strong>Implementing a custom, granular save system allows you to do things like run it inside of a coroutine<\/strong>, so a single save can happen over multiple frames. This can give a huge boost to overall performance when saving during active gameplay.<\/li>\n<\/ul>\n<p>But how do you convert in-memory objects to raw binary data? The process is actually quite simple. Imagine the following class:<\/p>\n<pre class=\"brush: csharp; title: ; notranslate\" title=\"\">\r\nclass foo\r\n{\r\n\tpublic int bar;\r\n}\r\n<\/pre>\n<p>We can create an instance of it in memory, by calling:<\/p>\n<pre class=\"brush: csharp; title: ; notranslate\" title=\"\">\r\nfoo theFoo = new foo();\r\n<\/pre>\n<p>And we can change its bar variable by calling:<\/p>\n<pre class=\"brush: csharp; title: ; notranslate\" title=\"\">\r\ntheFoo.bar = 10;\r\n<\/pre>\n<p>Now, if we want to save that instance to disk and load it later, we only need to know three pieces of information:<\/p>\n<ol>\n<li>theFoo&#8217;s class type<\/li>\n<li>bar&#8217;s value type<\/li>\n<li>bar&#8217;s value.<\/li>\n<\/ol>\n<p>We can store the first two pieces of information as single bytes, and since bar is an integer type, we can store it&#8217;s value with 4 bytes. So the entire object can be efficiently saved to disk using only 6 bytes of storage!<\/p>\n<p>In the Zombox save system, all information pertaining to object\/value types is categorized inside a single enum, whose values do not exceeed 255 (hence the reason why they only require 1 byte of storage). For the above example, I might write the enum out like this:<\/p>\n<pre class=\"brush: csharp; title: ; notranslate\" title=\"\">\r\n    public enum dataTypes\r\n    {\r\n        none = 0,\r\n        class_foo = 1,\r\n        value_integer = 2\r\n    }\r\n<\/pre>\n<p>Enums are integers by default (so they are 4 bytes in size), but as long as none of the values inside of our dataTypes enum exceed 255, we can easily convert it to\/from a <em>single<\/em> byte. So the actual save function pseudocode would look like this:<\/p>\n<pre class=\"brush: csharp; title: ; notranslate\" title=\"\">\r\npublic class foo\r\n{\r\n    ...\r\n    \/\/pseudocode\r\n    public void Save (FileStream file)\r\n    {\r\n        file.Write((byte)dataTypes.class_foo);\r\n        file.Write((byte)dataTypes.value_integer);\r\n        file.Write(System.BitConverter.GetBytes(bar));\r\n    }\r\n}\r\n<\/pre>\n<p>In the above psuedocode, &#8216;GetBytes&#8217; would actually generate a small amount of garbage each call, since it&#8217;s returning a new byte array (byte[]) that it created&#8230;but you can avoid this by caching and re-using your own byte arrays and passing them as an argument to that function (which is what I do in Zombox).<\/p>\n<p>So, we&#8217;ve got our basic save system implemented&#8230;.how would we load that data? Well, we&#8217;d simply parse through our save file and perform actions based on the bytes we read. Given the above example, here would be the corresponding load function pseudocode:<\/p>\n<pre class=\"brush: csharp; title: ; notranslate\" title=\"\">\r\npublic class theLoader\r\n{\r\n    public void Load(FileStream file)\r\n    {\r\n        \/\/psuedocode\r\n        byte readType;\r\n        byte[] readData;\r\n        while (file.readByte(readType))\r\n        {\r\n            if ((dataTypes)readType == dataTypes.class_foo)\r\n            {\r\n                foo newFoo = new foo();\r\n                file.readByte(readType); \/\/get the type of the next data\r\n                if ((dataTypes)readType == dataTypes.value_integer) \/\/next value in the file is an integer...ie, our bar variable!\r\n                {\r\n                    file.readBytes(readData, 4); \/\/read 4 bytes for integer\r\n                    newFoo.bar = System.BitConverter.ToInt32(readData); \/\/convert those 4 bytes to an integer, and assign to bar\r\n                }\r\n            }\r\n        }\r\n    }\r\n}\r\n<\/pre>\n<p>Obviously in that function we don&#8217;t actually do anything with the new foo object we instanced, but that&#8217;s not relevant for this illustration to work. The fact that we&#8217;ve managed to successfully save and load our class, while incurring a minimal performance hit (and with the right caching structures in place, zero garbage), is what we want!<\/p>\n<p>There are also a few other subtleties to mention. For example, what if our class has <em>2<\/em> ints? Then our dataTypes marker would need some more information to tell us <em>which<\/em> int we should assign the next piece of data to. Also, if you are sure your class will never change, you don&#8217;t need the dataTypes markers for the internal variables at all&#8230;you could just read all the data in the order it was saved (but I wanted more flexibility in the way Zombox saves\/loads data, so peoples&#8217; save files wouldn&#8217;t instantly become corrupt if I changed the structure of any saveable classes in the future).<\/p>\n<p>So, basically that&#8217;s what I do in Zombox. My save\/load code is much more complex, but the gist of it is that I recurse through all of my city data, saving out the necessary information required to recreate it all again at load time, while minimizing the total number of storage bytes and memory allocations required to do so. As for performance, <strong>here is the result:<\/strong><\/p>\n<ul>\n<li>Due to some creative caching, my method incurs <strong>zero<\/strong> garbage from start to finish during a save.<\/li>\n<li><strong>My method runs inside of a coroutine<\/strong>, which means auto-saves can occur during gameplay with no real impact on performance. A full save happens once every 90ish seconds during gameplay (approximately 2,000 city objects are processed per frame), as well as whenever the game is paused or exited (meaning that data should never be lost if the game functions normally, and only data changed in the last 90 seconds will be lost if a crash occurs or you force-quit the app).<\/li>\n<li><strong>Direct file access in my method is double-buffered<\/strong>. All save data is written to a temp file until the save is complete, at which point the real save file is quickly overwritten with the temp file by the OS. This ensures that if a game crash happens mid-save, only the temp file will be corrupted &#8212; not your actual save file.<\/li>\n<li>The total filesize for an average Zombox save using this method, which contains all city data, is just <strong> 1.5 megabytes<\/strong> (which zlib can further compress to just 100kb!). When left uncompressed, that&#8217;s still over 10x smaller than files created with other serialization methods I tried!<\/li>\n<\/ul>\n<p>So there you have it. <strong>Zombox now features a robust auto-save system that runs fast on mobile!<\/strong> Thanks for reading!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The problem: Since the inception of Zombox almost 5 years ago, one thing I&#8217;ve never addressed is how the save system will work. For a long time in gaming, it was fairly common for game designers to provide players with some kind of manual &#8216;save&#8217; menu (whether in a user interface, or within gameplay itself). [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[7,3],"tags":[],"_links":{"self":[{"href":"http:\/\/www.zombox.net\/index.php?rest_route=\/wp\/v2\/posts\/589"}],"collection":[{"href":"http:\/\/www.zombox.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.zombox.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.zombox.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.zombox.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=589"}],"version-history":[{"count":62,"href":"http:\/\/www.zombox.net\/index.php?rest_route=\/wp\/v2\/posts\/589\/revisions"}],"predecessor-version":[{"id":645,"href":"http:\/\/www.zombox.net\/index.php?rest_route=\/wp\/v2\/posts\/589\/revisions\/645"}],"wp:attachment":[{"href":"http:\/\/www.zombox.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=589"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.zombox.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=589"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.zombox.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=589"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}