A New Renderer: OpenGL 3.3, a Pixel World, and a Very Stubborn Wobble

The last engine post ended with games living outside the engine as projects. Since then Spellweaver has started asking the engine for real things, and most of the work of the past two and a half weeks came from those requests. This is the first of three posts about it: the renderer here, then the map editor and entities, then navigation meshes.

A Note on the Pace

Three posts in two and a half weeks is a lot for this blog, so I should explain how it happened. Since early October I’ve been working with Claude Code, an AI coding assistant, on the engine and on Spellweaver. It helps me talk through designs, prototype ideas, and write a lot of the code.

The engine is still mine. I decide what gets built and how it should work, I test every change myself before it’s committed, and plenty of what we build gets reworked or thrown out once I’ve tried it. The next two posts have good examples of that. None of this is work I couldn’t do on my own, but it would take an awful amount of time that I don’t have right now. The speed is what changed, not who’s steering.

Why Now

The renderer was still fixed-function OpenGL, a compatibility context with glBegin calls from the Handmade Hero days. It drew what it needed to, but Spellweaver is a pixel art game, and I wanted a colour grade, lights at night, and later a shader for the Hush, the grey stillness between its islands. None of that fits a fixed-function pipeline.

I split the work into five phases and tested each one before starting the next: OpenGL 3.3 core with the same picture as before, the pixel world, sprites, game shaders, and lighting. All five landed on the same day, although the second one tried hard to prevent that.

OpenGL 3.3 Core

The first phase was meant to change nothing you could see. The editor now asks for a 3.3 core context, and every glBegin is gone.

One shader program draws everything: position, UV and colour in, texture times colour out, with a 1x1 white texture for anything untextured. Render commands go into one batched vertex buffer, and the batch is drawn whenever the texture, the primitive type, the scissor, the render target, or the blend mode changes. Nuklear moved to vertex and index buffers and uses the same program.

The one surprise came from the second window. The debug window’s OpenGL context shares objects with the main one, but vertex array objects aren’t shared between contexts, so each window now makes its own the first time it draws.

Internal builds also log the first few OpenGL debug messages as errors, which goes straight to the log panel on the debug window.

The Pixel World

Pixel art wants to be drawn at a low resolution and scaled up by a whole number. A game now does this:

pixel_world PixelWorld = BeginPixelWorld(&Group, 640, 360);
// draw the world
EndPixelWorld(&Group);
// draw the HUD and text

The two numbers are the minimum size of the image. The engine picks the biggest whole-number scale at which the image is still at least that big, so a wider or taller screen just sees more of the world. Everything drawn between the two calls lands on whole pixels of the small image. Everything after them is drawn at full resolution on top, so text and the HUD stay sharp.

That part worked right away. Scrolling was another story.

The Wobble

The first version rounded the camera to whole image pixels. At a scale of three, the world moved 6 or 9 screen pixels from one frame to the next, and it looked like it was shaking.

The usual fix is a sub-pixel camera. The game rounds its camera and passes the part it rounded off to SetPixelWorldScroll, and the renderer shifts the finished image by that many screen pixels. The image has one extra pixel on each side to cover the gap. Inside the image everything still sits on whole pixels, but the world as a whole now scrolls one screen pixel at a time.

The camera became smooth and the hero started to wobble. Whatever the camera follows no longer lands on the same screen position every frame, because the hero is rounded to image pixels while the camera isn’t.

So I split the world in two. Things that stand upright in the world, like characters, trees and props, are now drawn on top of the image, directly to screen pixels, with their texels still exactly Scale wide. They look like part of the pixel world, but they can sit at any screen position.

The hero still wobbled. I got fed up at this point, parked the whole thing on a branch, and decided to go back to no pixel image at all.

Then I found out that the test I had just run never included the fix. My text editor had saved an older copy of spellweaver.cpp over the new one, so I’d been testing the old code the whole time. The work came back from the branch, I retested it, and the hero still wobbled, but this time for a real reason: upright things were placed relative to the image’s position on screen, and that position is rounded. The hero’s leftover offset changed every frame, by up to half a screen pixel.

The fix was one change. Upright things now use the unrounded scroll, and only the ground image is rounded. Something the camera follows lands on the same screen spot every frame, and the wobble was finally gone.

Sprites

PNGs now go through stb_image in the asset builder and in the editor, next to the old BMP loader, and the image is told apart by its first bytes. GetSprite gives a game one sprite of a sprite sheet that it found by tags.

A related bug came up a few days later, when the tile editors showed pixel art slightly off their grids. PushBitmap was drawing every bitmap one texel in from its edges. That was a leftover from Handmade Hero, where bitmaps were padded and bilinear filtered. My textures are nearest filtered, so all it did was cut off the edge pixels and stretch the rest. Removing it fixed the editors.

Game Shaders

A game shader is a text asset, a .glsl file with the body of a fragment shader, built into the game’s asset file like any other asset. SetPixelWorldShader hands it to the renderer, which puts a header of declarations in front of it and compiles it the first time it sees it.

Shaders are kept by a hash of their source, so an edited shader simply counts as a new one. A shader that doesn’t compile is logged once, and the world is drawn without it. A typo in a shader shouldn’t take the game down, and it shouldn’t fill the log either.

The first test was a colour grade, and the ground graded fine while the hero, the trees and the people stayed exactly as they were. This was the price of the wobble fix: upright things went directly to the screen after the image, so the shader never saw them. Now, when a game has a shader, the image and its upright things are drawn into a screen-sized image first, and the shader draws that. The shader still gets the scale and origin of the pixel grid, so it can work per image pixel if it wants to.

Lighting

Lighting builds on the same passes. A sprite sheet or tile set with a <name>_n file next to it gets a normal map for each sprite or tile. The normal map’s GUID is derived from the image’s GUID, so the asset file’s layout didn’t have to change.

The world is drawn with its normals into a second colour buffer, and everything without a normal map gets a flat one. A light pass then works out the colour of each world pixel:

albedo * (ambient + sum of (facing * falloff^2 * light colour))

A game calls SetPixelWorldLighting with the ambient colour and PushLight for each light, up to 64 per frame. The game’s shader runs after that, on the lit picture.

I should be honest about how this turned out. Spellweaver decided against normal maps three days later, when I rethought its art (more on that in the next post). The shading now lives in the pictures themselves, and lights only tint and brighten them. So the game uses the lights for its nights, and the normal maps are a feature with no users for now. That’s fine, because the engine is meant to serve more than one game.

What’s Next

Glowing things are still missing. Fire is dimmed like any other lit object, which is wrong for fire, so bloom or something like it is on the list. The shader for the Hush, distortion and water are also still ahead.

But the renderer no longer stands in the way. The engine went from glBegin to shaders, render targets and lighting, and everything a game needs goes through a few calls in engine_api.h. The next post is about where the pictures it draws come from, the map editor.