In the last post I said that I would write a dedicated post about the asset system once it was finished. The rewrite was finished at the end of December, followed by a couple of weeks of work on tags in January, and then the engine sat untouched for most of the year. Since then a lot has changed, so I’m splitting the catch-up into three posts. This one closes the asset system chapter, the next one is about the new platform layer, and the last one is about how games are built on top of the engine now.

A Quick Recap

The asset pipeline has three files. KESA (Kaban Engine Stored Asset file) is the editor side: it knows where every asset came from, how it was cut, and which tags it has. KET (Kaban Engine Tag file) stores all tags the user created, each one a key with a list of values. KEA (Kaban Engine Asset file) is what a game actually loads, and its version always matches the KESA version it was built from. I described the versioning rules in the previous post, so here I want to focus on the part that was missing back then: building the KEA and finding assets in it.

Building the KEA

The KEA builder takes the stored assets and the tag file and turns them into a single file that is laid out for loading, not for editing. Every asset gets a type: bitmap, sound, sprite sheet, sprite, tileset, tile, text, binary file, or SSWM.

Sprite sheets and tilesets are worth mentioning separately. In the editor they are one asset each, a single image with information on how to cut it. In the KEA every sprite and every tile becomes its own asset with its own GUID, and the sprite sheet or tileset only keeps the list of GUIDs of its pieces. A game never has to cut anything at runtime; it asks for a tile the same way it asks for any other bitmap.

The file itself starts with a header that is mostly offsets. Each offset points at a section that is already sorted the way it will be searched:

  • the tag maps, sorted by tag GUID,
  • the tags actually used by assets,
  • a hash table for tag lookups,
  • an array of asset indices that the hash table points into,
  • an asset type table, sorted by type,
  • and the assets themselves, sorted by GUID.

Since asset GUIDs are generated from source file names, sorting by GUID means the order is the same every time the file is built from the same sources. Finding an asset by GUID is a binary search, and finding all assets of a type is a single lookup in the type table.

I also removed fonts as an asset type during this pass. The old font asset was a leftover from the Handmade Hero days, and keeping it alive through the rewrite wasn’t worth it. Fonts came back later in a different form, which I’ll get to in the third post.

Tags, Again

The tag format from the previous post stayed the same: a key, a value, and GUIDs generated from them. What changed is the editing side. Creating and saving tags was reworked, and tags can now be edited and removed, not only added. Every change to the tags writes a new version of the KET file, so the tag history follows the same explicit versioning rules as the rest of the pipeline.

The Assets Mode also got a search box, and assets can be searched by tag. Once a project has a few hundred assets, scrolling through a list stops being an option. The Arkham project that I started at the same time has close to two hundred card images alone, so this was needed right away.

Finding Assets by Tags

The whole point of tags is that a game doesn’t ask for card_Guardian_Investigator_01001a.bmp. It asks for “the front of card 01001” and lets the asset system figure out which asset fits.

On the game side this is a tag vector, a short list of key and value pairs:

asset_tag_vector Vector = {};
TagVectorAdd(&Vector, "Tag_CardCode", "01001");
TagVectorAdd(&Vector, "Tag_CardSide", "Front");
asset_best_match_result Match = GetBestMatchAssets(Assets, &Vector);

When the KEA is built, every key and value pair that is used by at least one asset gets an entry in the hash table. The entry stores the GUID of that pair and a range in the asset index array, which lists every asset that has that tag. The table uses open addressing with linear probing and is sized to the next power of two above the entry count divided by 0.7, so lookups stay short.

GetBestMatchAssets hashes each pair in the vector, collects the asset index ranges of every entry it finds, sorts them, and keeps the indices that show up the most times. In other words, the asset that matches the most tags wins. If exactly one asset wins, the result is just its index. If several assets match equally well, the result is a tie, and the caller can pass an arena to get all of them back. Without an arena it only gets the count, which is exactly what I want most of the time: in Arkham, a tie means two cards have the same tags, which is a data mistake, so the game asserts on it.

Modules (and Why They Were Rolled Back)

The last thing I did in December was a first attempt at loading game code as a module. The idea was to compile Arkham as its own DLL, list it in editor_metadata.json together with its asset file, and let the editor load it and hot reload it the same way it hot reloads the engine. A small engine_api table gave the game access to rendering functions.

It worked, but only barely. The engine knew Arkham by name, the module list in the metadata only had one possible entry, and the API was a handful of render functions with no real plan behind it. In January I removed it again and compiled Arkham directly into the engine. The module design wasn’t worked out, and I wanted to work on the game rather than on the module system. That decision bought me a few weeks of progress on Arkham, but it left the engine with a game living inside it, which is the opposite of what I wanted.

That problem was solved properly only this October, and it is the topic of the third post.

Where This Leaves Things

The asset system is finished in the sense that it does what I need: assets are edited and tagged in the editor, built into a KEA, and found by tags at runtime. It is not finished in the sense that there is nothing left to do. SSWM files are not built yet, there is no font type in the KEA, and some of the editor code is still rough. But it’s stable enough that I stopped thinking about it, which for a system like this is the real goal.

Next up is the platform layer, which turned out to be the biggest change the engine has gone through so far.