An Unreal Continuous Integration Quest

game dev, unreal engine, and devops
a movie still from The Lord of the Rings - The Fellowship of the Ring, 2001
The Lord of the Rings - The Fellowship of the Ring, 2001

[Gandalf reading] The year 3434 of the Second Age. Here follows the account of Isildur, the High King of Gondor, at the finding of the Ring of Power. “It has come to me, the One Ring. (…)

TL;DR Summary

Introduction

I’ve been enamored with game development as early as I’ve been in love with games. I remember fondly when myself, one of my brothers and a few friends would lose ourselves over tools like the Age of Empires scenario editor or later on with the Source SDK and the Hammer Editor. At some point even the long since surpassed Unreal Development Kit (UDK).

In this spirit, I will share in this post something that eluded me for a long time. For the last 2 to 3 years I’ve been tinkering at times with the Unreal Engine 4 5, and it goes a bit like this:

Locally I am able to set up Visual Studio. I can also burn some rainforests download and install the 100+ GB required Unreal Engine libraries. Now look at me, I’m doing stuff on Unreal Engine! Hooray! But…

(a few moments later)

But… How on earth do I bake this thing on a pipeline somewhere?

When working on experiments in other non-game “ecosystems“… it feels less impossible. I can think of regular projects using Java, Python, Javascript, …, and I feel there are always straightforward ways to make CI pipelines for these. Regardless of the CI platform used at the time, be it Jenkins, or TeamCity, or Gitlab CI, or GitHub Actions, … , the gist of going from local development to a full fledged CI pipeline is usually just about answering questions like:

What’s the command to build/compile?

What’s the command to test?

What’s the command to package the code into a shareable and runnable artifact?

It’s all about incantations made up of tools and CLI commands. The syntax of the chosen CI platform and development ecosystem might be different. But, at the end of the day, the mental building blocks are always the same.

So… what’s the catch for doing the same but in Unreal Engine 5 development?

The Problem(s)

In an ecosystem of Unreal Engine development, to answer the previous questions, even at a small hobbyist scale, we face (at least) two major challenges:

These challenges are hard, but they are not impossible to overcome. What follows is a recollection of notes on the approach I took to solve these.

As you (the reader) might suspect, there is no single perfect approach. This one has its frailties. Regardless of that, I’m sharing it here so it might help other folks in search of solutions to similar problems.

Proposed solution

All of the example code you’ll see here is also available at https://github.com/filfreire/coop-game-fleep.

“The Runners”

Let’s start “where (and how) the CI pipeline will run”.

First, we need to choose a CI platform. I wanted to avoid having to maintain or run CI in a separate platform than the one I host and version my code. Since I already hosted my experiments repository on Github, GitHub Actions was my first choice for platform.

GitHub provides runners of their own for free.

These are incredibly useful for most non-demanding opensource projects. The issue though is that the specs of those runners are insufficient for Unreal-based projects.

There’s the possibility of paying GitHub some dolla dolla bills for more powerful runners. We want to avoid that at this stage. Plus, the GitHub provided runners, paid or not, don’t have the needed dependencies to build Unreal projects readily installed and configured. That would involve extra work to setup, and is a rabbit hole of its own, which we can look at in a separate post.

Luckily, GitHub provides a way for us to use our own hardware with their concept of Self-hosted Runners.

This is the first piece of the puzzle’s solution.

Take a Windows host machine which we can spare and set it up as self-hosted runner.

The jobs that will run in that runner will have access to any libraries and programs we have installed on that machine. That is including any Unreal setup work we may have done. This is the key that turns a local development machine into a CI runner.

Example of a self hosted GitHub runner
Example of a self hosted GitHub runner

So let’s say we have installed Unreal Engine 5.2 on C:\EpicGames\UE_5.2.

That means that in a GitHub actions workflow we can in practice run Unreal’s Automation Tool commands in our CI pipeline.

If we look into the output logs when we run Build or Packaging of our Unreal project from within the Unreal Editor, it’s usually running Automation Tool commands under the hood.

Once we have our host configured as GitHub self-hosted runner, these same commands are available to be run in a pipeline:

name: Example

on: [push, pull_request]

jobs:
  example:
    runs-on: self-hosted

    steps:
    - name: Clone repository
      uses: actions/checkout@v3

    - name: Show UAT help menu
      run: "C:\EpicGames\UE_5.2\Engine\Build\BatchFiles\RunUAT.bat" -Help

Side-note: There are a few security issues from this. We are in practice providing access to a host machine on GitHub. They warn about it on their documentation. Be sure to setup the configs of a project to be private. If it is public and opensource, there are some settings also which we can set to prevent pull-requests from using the runners without the project owners’ approval. Be careful with this stuff. Or not… after all, why not, why shouldn’t we keep it?

Building

The next piece of the puzzle is figuring out how to build and compile our project. If we run Unreal Automation tool locally with the -List argument, we’ll see a list of commands we can use:

PS C:\EpicGames\UE_5.2\Engine\Build\BatchFiles> .\RunUAT.bat -List
Running AutomationTool...
Using bundled DotNet SDK version: 6.0.302
Starting AutomationTool...
Parsing command line: -List
Initializing script modules...
Total script module initialization time: 0.31 s.

Available commands:
  AutomationScripts.Automation:
    (...)
    BuildCookRun
    (...)

The command we are interested in this case is BuildCookRun. The arguments that would just build/compile the project in my personal case were:

"%ueLocation%\Engine\Build\BatchFiles\RunUAT.bat" BuildCookRun -project="%projectLocation%\%projectName%" -noP4 -platform=Win64 -clientconfig=Development -build

I’ve abstracted away this command and the follow-up commands for testing and packaging in such a way that in the CI job I can call a batch file with a few arguments instead of writing the whole command verbatim:

# Build.bat

set ueLocation=%~1
set projectLocation=%~2
set projectName=%~3

"%ueLocation%\Engine\Build\BatchFiles\RunUAT.bat" BuildCookRun -project="%projectLocation%\%projectName%" -noP4 -platform=Win64 -clientconfig=Development -build

And here’s what calling that batch file would look like in the GitHub Actions workflow code:

# GitHub actions build step
# Define UNREAL_PATH and PROJECT_NAME environment variables
- name: Build
  run: .\scripts\Build.bat $env:UNREAL_PATH (Get-Location).Path $env:PROJECT_NAME

Testing / Running automated checks

A quick detour before I go into how to implement a CI step to run tests.

I have to be honest, my first read of the abdominal upsetting diarrhea inducing official documentation on Test Tooling that comes with Unreal felt… problematic… to say the least.

In terms of “lingo” there’s confusing spots depending on if we are more of a Programmer or a Tester. For example, starting with how the documentation defines which types of automated checks we can build and how to even set those up.

Side-note: I’m not even going to dwell in what hot takes members of different software testing cults and some charismatic testing influencers would spew upon pretending to read reading the docs.

Thankfully, there are folks in the Unreal Development community that have poured over this so we don’t have to go to Mordor alone and fend for ourselves in the official docs.

After going through some pretty awesome community presentations and posts, like this one and this other one, I feel moderately confident to share the mental model I arrived at on Unreal Testing frameworks/tooling. It might be missing some bits, but we can split it in 3 parts, something like this:

FAutomationTestBase

There are unit-like tests we can build with their FAutomationTestBase library. And in theory, with it we can check bits of our game logic without having to “boot up” a running instance of the game. The simplest form of what it could look like is something like this:

#include "Misc/AutomationTest.h"

IMPLEMENT_SIMPLE_AUTOMATION_TEST(FBasicExampleTest, "CoopGameFleepTests.Basic", EAutomationTestFlags::EditorContext | EAutomationTestFlags::ProductFilter)

bool FBasicExampleTest::RunTest(const FString &Parameters)
{
    return TestTrue("math still works", 1 < 2);
}

I’ve seen examples where folks instantiate and/or mock parts of what we’d have in a live instance of a game, but I believe the key area of these is for checking stuff that doesn’t need the game to run, regardless of the complexity of the test script or the scenario being checked.

Example of an Unreal test run through the Unreal Editor
Example of an Unreal test run through the Unreal Editor

Functional Framework

There’s the Functional Framework for Testing. The gist of it is we can build any kind of tests, using Blueprints.

These will execute against (partial) “live instances” of the game. Think of small test levels where in each we are only checking a bit of some game design mechanic. Folks seem to be using these for both unit and integration-like tests (see this example and this other example).

There is also support for complex intricate scenarios where we want to check some over-arching behavior of our game. Mileage may vary depending on what folks choose to do with these, but pretty much I could see these being useful for example for adding checks for regressions or testing core designs and mechanics of the game that always “need to work” on every build with the game running, not with mocks or just the logic of a mechanic.

Gauntlet

There’s also something called Gauntlet - I don’t know a lot about it yet, but it appeared to be a distant equivalent in form to, say, running Playwright against a web app or an Electron app. The catch being - the test script code doesn’t need to know in theory about the internals of the target code its checking against.

So unlike using FAutomationTestBase or the Functional Framework, one would be able to spin up a handful of different instances of the game using Gauntlet Controllers and then run whichever scenarios (even trigger Functional Framework tests) we might need at scale. Theres a video that explains this way better at depth here.

Detour over.

For the purposes of my experiment, for now I just wanted to be able to have a Test stage in my CI pipeline and trigger tests, even if it’s just a dummy test like the one example above that checks if… *checks notes*… basic math still works and 1 is lesser than 2.

After some setup and some research, I narrowed down the command I needed to trigger the test above to a sub-command of UnrealEditor-Cmd.exe. When we run tests through the Unreal Editor, it’s basically using that command under the hood. We can find more about the setup with this example. Again, I’ve abstracted away the command I needed into a batch file like the one bellow:

# RunTests.bat

set ueLocation=%~1
set projectLocation=%~2
set projectName=%~3
set testSuiteToRun=%~4
set testReportFolder=%~5
set testLogName=%~6


"%ueLocation%\Engine\Binaries\Win64\UnrealEditor-Cmd.exe" "%projectLocation%\%projectName%" -nosplash -Unattended -nopause -nosound -NullRHI -nocontentbrowser -ExecCmds="Automation RunTests %testSuiteToRun%;quit" -TestExit="Automation Test Queue Empty" -ReportOutputPath="%projectLocation%\%testReportFolder%" -Log=%testLogName%

Calling that batch file in the GitHub Actions workflow code will look something like this:

# GitHub actions run tests step
# Define UNREAL_PATH, PROJECT_NAME, TEST_SUITE_TO_RUN, TEST_REPORT_FOLDER, TEST_LOGNAME environment variables
- name: Test
  run: .\scripts\RunTests.bat $env:UNREAL_PATH (Get-Location).Path $env:PROJECT_NAME $env:TEST_SUITE_TO_RUN $env:TEST_REPORT_FOLDER $env:TEST_LOGNAME

After some back and forth, I finally managed to have the test step running, and even saving test results into artifacts on GitHub.

Packaging the game

Finally, after building and then running automated checks, we want to pre-package a build of the game and make it available to download.

From running Packaging from the Unreal Editor we can spot the Unreal Automation Tool that runs under the hood and we can abstract that into a reusable batch file:

# Package.bat

set ueLocation=%~1
set projectLocation=%~2
set projectName=%~3
set target=%~4
set packageFolder=%~5

"%ueLocation%\Engine\Build\BatchFiles\RunUAT.bat" BuildCookRun -project="%projectLocation%\%projectName%" -nop4 -utf8output -nocompileeditor -skipbuildeditor -cook -project="%projectLocation%\%projectName%" -target=%target% -platform=Win64 -installed -stage -archive -package -build -pak -iostore -compressed -prereqs -archivedirectory="%projectLocation%\%packageFolder%" -clientconfig=Development -nocompile -nocompileuat


# GitHub actions package step
# Define UNREAL_PATH, PROJECT_NAME, TARGET_NAME and PACKAGE_FOLDER environment variables

- name: Package
  run: .\scripts\Package.bat $env:UNREAL_PATH (Get-Location).Path $env:PROJECT_NAME $env:TARGET_NAME $env:PACKAGE_FOLDER

- name: Archive package
  uses: actions/upload-artifact@v3
  if: always()
  with:
    if-no-files-found: ignore
    name: win64-build
    path: |
      \$\{\{env.PACKAGE_FOLDER\}\}

In the end we have a working CI pipeline for our Unreal Engine based game, which compiles, runs tests and bakes a build of the game:

Example of the GitHub workflow after running, with test results and a build of the game as available artifacts
Example of the GitHub workflow after running, with test results and a build of the game as available artifacts

We can now download that build, and voila, we can boot up our own version of the Red Dead 3 - Tears of the Burger Kingdom:

Example screenshot of our game built via our new CI pipeline
Example screenshot of our game built via our new CI pipeline

Wrapping up

There are a number of improvements that I could draw from the current solution. I might tackle those in other explorations and blog posts in the future. What I have in mind right now:

On a serious note, there were also some problems just with this small experiment I experienced first-hand:

I might write about these as well in the future.

Such is life. I’m looking forward to my next experiments and learning more. These things look like weird magic oftentimes, but they are not impossible to pick apart and learn.

References

Credit where it’s due, here are some references of interest that have helped a lot while working on this post:

If you read this far, thank you. Feel free to reach out to me with comments, ideas, grammar errors, and suggestions via any of my social media. Until next time, stay safe, take care! If you are up for it, you can also buy me a coffee ☕