While working on Fondusi’s I came up with a great way to store the map data (or any binary data, really) in such a way that previous files could be loaded using new versions of the software. It has some great capabilities and, after answering a question about it on gamedev.StackExchange, I thought it might be helpful to write a post about it here.
Let’s start out with a simple Map class with a string property called Name that we want to save and load from a file.
The Options
Let’s look at what options we have:
XML
Pros:
- Widely supported
- Easy to edit by hand
- Easy to change schema
Cons:
- Potentially large file size (compression can greatly help, but then you have to decompress it to use it)
- The larger it gets, the slower it is. Ultimately, you’re parsing text and that’s never going to be faster than loading from binary.
- You have to convert between types to get data out of (or into) it.
SQLite
Pros:
- Ability to query for different sets of data really easily.
- Many external tools for editing data.
- Fast*
- Depending on your structure, it could be easy to change the data format.
Cons:
- *While it is fast, it still requires extra overhead for handling the database and running queries.
- Larger file – The file won’t just be storing your data, it will also contain the schema information and other data related to handling the database.
- Depending on your structure, it could be difficult to change the data format.
- It’s probably overkill if you’re not already using it.
Custom Binary Format
Pros:
- Very fast.
- Easily supports old versions.
- File size is exactly the same size as the data you’re saving.
- Precise control over what gets saved and how it gets saved. — Saving a short instead of an int can save a lot of space if it’s in an array or list.
- You don’t have to convert between types when loading.
Cons:
- Very difficult to edit by hand.
As you probably guessed based on the title of this post, I’m going to describe the custom binary file format. So, without further ado, here we go!
Setting Up Our Class
Let’s get back to that Map class from before. It currently looks like this (C#):
|
1 2 3 |
public class Map { public string Name { get; set; } } |
Pretty simple right? So, let’s get to saving this thing.
Saving/Loading – The Usual Way
The first thing we want to do is create a Save method in our Map class.
|
1 2 3 4 5 6 |
public void Save(string fileName) { using (BinaryWriter bw = new BinaryWriter(File.Open(fileName, FileMode.Create))) { bw.Write(Name); } } |
Ok, that was no problem. Now, let’s take a look at how we’d load that file.
|
1 2 3 4 5 6 |
public void Load(string fileName) { using (BinaryReader br = new BinaryReader(File.Open(fileName, FileMode.Open))) { Name = br.ReadString(); } } |
Great, that was easy!
Making a Change
Now, let’s say we want to add something to this file, perhaps a List of Platform objects. We’ll just add another property after Name:
|
1 |
public List<Platforms> { get; set; } |
Ok, now we need to update our Save method.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
public void Save(string fileName) { using (BinaryWriter bw = new BinaryWriter(File.Open(fileName, FileMode.Create))) { bw.Write(Name); bw.Write((short) Platforms.Count); //The amount of platforms foreach(Platform p in Platforms) { //For the sake of simplicity, we'll assume the Platform // class has a Write() method that accepts a BinaryWriter // as a parameter. p.Write(bw); } } } |
That takes care of saving. Nothing really fancy is being done. We’re just writing the amount of Platform objects that are in our list, so we can loop through and read them in the Load method. You may have noticed that I sneaked one of my other pros into it. I know that it’s unlikely that my map will ever have more than 1,000 platforms let alone 2,147,483,647 (maximum int value), so why save the count as an int? By converting it to short, we save two bytes! That may not seem like much, but when you’re saving two bytes instead of four for each of 160,000 tiles in a 2D array, it really matters. Just keep in mind that whatever you save it as, you have to read it as the same type and convert it back when you load it.
Anyways, let’s get to loading of this file.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
public void Load(string fileName) { using (BinaryReader br = new BinaryReader(File.Open(fileName, FileMode.Open))) { Name = br.ReadString(); short platformCount = br.ReadInt16(); Platforms = new List<Platform>(); for (int i = 0; i < platformCount; i++) { //Again, for the sake of simplicity, we're going to assume // that the platform has a constructor that takes a // BinaryReader as a parameter. Platforms.Add(new Platform(br)); } } } |
That’s not so tough either. So, we’re doing pretty good now. We’ve got our updated file format. But, whoa there, hang on a minute. Your best testing buddy, Billy, just called and sounded irate: “Hey <insert your name here>, you monkey lover! You changed the file format on me and now that great map I made with the awesome name won’t load anymore! It just crashes and burns.”
“Great scott,” you exclaim. “How can I fix this?”
How do we make it better?
Here are some of your options:
- You can take the easy way out and throw a try catch around the using statement (you probably should have had one anyways, tsk tsk!) that just alerts the user that something went wrong and it couldn’t load. Not very user friendly if you ask me!
- You can add an if statement to see if the binary reader is at the end of the file. However, what if you wanted to add an item in the future in between Name and Platforms? That check won’t work then.
- You can go back in time (in your memory) and remember that you read this and do it properly -or- you can, not have made the mistake above in the first place. 😛
As you know, I’m gonna go with option number 3 here.
Back to the Roots
To make this better, we have to go back to our first version of the class. Let’s take a look at it:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
public class Map { public string Name { get; set; } public void Save(string fileName) { using (BinaryWriter bw = new BinaryWriter(File.Open(fileName, FileMode.Create))) { bw.Write(Name); } } public void Load(string fileName) { using (BinaryReader br = new BinaryReader(File.Open(fileName, FileMode.Open))) { Name = br.ReadString(); } } } |
Ok, so how can we improve upon this? Well, we’re going to make one slight change that will alleviate all of the headaches in the future, allow all map files to be loaded regardless of how old they are AND help you sleep at night (well, hopefully) and that change is: *drum roll* to add a version number! Yep, that’s the whole trick. Since the code is pretty short, I’m just going to repost it:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
public class Map { //By all means, if you think you'll have more than ~32,000 // versions of your map file structure, make this an int. // For most situations, a byte would probably suffice, but // this is the ONE thing in the file that's difficult to // change while keeping backwards structure compatibility, // so choose wisely at version 1. const short Version = 1; public string Name { get; set; } public void Save(string fileName) { using (BinaryWriter bw = new BinaryWriter(File.Open(fileName, FileMode.Create))) { bw.Write(Version); bw.Write(Name); } } public void Load(string fileName) { using (BinaryReader br = new BinaryReader(File.Open(fileName, FileMode.Open))) { short vers = br.ReadInt16(); Name = br.ReadString(); } } } |
That’s pretty easy. Just creating a short variable and writing it to the file. Note that we don’t read it into any class-scoped variables in the Load method, we just read it into a locally-scoped variable for use later.
Making a Change++
Now we want to add that pesky List of Platform objects to the file structure. Let’s step through it.
Step 1. Add the new property
|
1 |
public List<Platform> { get; set; } |
Add this below the Name property declaration.
Step 2. Increment the version number
|
1 |
const short Version = 2; |
I don’t think I need to explain this.
Step 3. Update the Save method
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
public void Save(string fileName) { using (BinaryWriter bw = new BinaryWriter(File.Open(fileName, FileMode.Create))) { bw.Write(Version); bw.Write(Name); bw.Write((short) Platforms.Count); //The amount of platforms foreach(Platform p in Platforms) { //For the sake of simplicity, we'll assume the Platform // class has a Write() method that accepts a BinaryWriter // as a parameter. p.Write(bw); } } } |
Whoa! It’s exactly the same as the other method (seriously, I copy & pasted), except this time the first thing we write to the file is the version number. Ok, that was easy, now what about loading?
Step 4. Update the Load method
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
public void Load(string fileName) { using (BinaryReader br = new BinaryReader(File.Open(fileName, FileMode.Open))) { short vers = br.ReadInt16(); Name = br.ReadString(); Platforms = new List<Platform>(); if (vers < 2) { //There were no platforms since the file is older // than version 2 so we'll add a default platform. Platforms.Add(new Platform(100,200)); } else { short platformCount = br.ReadInt16(); for (int i = 0; i < platformCount; i++) { //Again, for the sake of simplicity, we're // going to assume that the platform has a // constructor that takes a BinaryReader // as a parameter. Platforms.Add(new Platform(br)); } } } } |
Ok, this one may take a bit more explaining, but this is all there is to it. Basically, we read in the version number first and store it in a variable we can reference (vers). We then read in the string Name. That was in our first version so it’s fine as is. The next step is to check what version of file we’re loading. If the version is lower than 2 (our new version) then it won’t have the Platform data in it. So we set that to a default value (clear the list). If, however, the version is greater than 2, we get the platform count (as we did before) and then load each platform.
Conclusion
We’ve looked at some of the options for saving data to a file and focused on the custom binary format. We looked at how binary files are commonly saved and saw some of the pitfalls of creating and updating them. We then saw how easy it was to get around those pitfalls simply by adding a version number to our initial file structure. Great, we’ve created a file structure that allows for modification and supports previous versions seamlessly.
As always, if you have any questions or concerns, drop a comment!
Sorry, someone abused my PC to write this comment while I was away. Thank you very much for sharing this helpful source code.
No worries. It actually made me realize that my syntax highlighter wasn’t handling escaped HTML entities properly, so in a roundabout way, it was helpful. 🙂