An Unreal Continuous Integration Quest
[Gandalf reading] The year 3434 of the Second Age. Here follows the account of Isildur, the High King of Gondor, at the finding of the Ring of Power. “It has come to me, the One Ring. (…)
- Needed a way to have a Continuous Integration pipeline for building, running tests and packaging an Unreal Engine based game.
- Used GitHub self-hosted runners with Unreal Engine and necessary dependencies installed;
- Used Unreal Automation Tool for running Builds and Packaging in GitHub Actions workflow;
- Automation in Test tooling in Unreal Engine ecosystem can be confusing, more on that in future blog posts;
- Example implementation can be found here: https://github.com/filfreire/coop-game-fleep.
I’ve been enamored with game development as early as I’ve been in love with games. I remember fondly when myself, one of my brothers and a few friends would lose ourselves over tools like the Age of Empires scenario editor or later on with the Source SDK and the Hammer Editor. At some point even the long since surpassed Unreal Development Kit (UDK).
In this spirit, I will share in this post something that eluded me for a long time. For the last 2 to 3 years I’ve been tinkering at times with the Unreal Engine
4 5, and it goes a bit like this:
Locally I am able to set up Visual Studio. I can also
burn some rainforestsdownload and install the 100+ GB required Unreal Engine libraries. Now look at me, I’m doing stuff on Unreal Engine! Hooray! But…
(a few moments later)
But… How on earth do I bake this thing on a pipeline somewhere?
What’s the command to build/compile?
What’s the command to test?
What’s the command to package the code into a shareable and runnable artifact?
It’s all about incantations made up of tools and CLI commands. The syntax of the chosen CI platform and development ecosystem might be different. But, at the end of the day, the mental building blocks are always the same.
So… what’s the catch for doing the same but in Unreal Engine 5 development?
In an ecosystem of Unreal Engine development, to answer the previous questions, even at a small hobbyist scale, we face (at least) two major challenges:
- 1) Regardless of CI runner/executor used, it needs to hold 100+ GB baggage of context to build/test/package “the thing”
- 2) The incantations to “compile”, or “run tests” or “package” are not easy to find. The information is supposedly out there in the official
ancient scrollsdocumentation… but it can feel terse and hard to grasp for non wizardregular folks like myself.
These challenges are hard, but they are not impossible to overcome. What follows is a recollection of notes on the approach I took to solve these.
As you (the reader) might suspect, there is no single perfect approach. This one has its frailties. Regardless of that, I’m sharing it here so it might help other folks in search of solutions to similar problems.
All of the example code you’ll see here is also available at https://github.com/filfreire/coop-game-fleep.
Let’s start “where (and how) the CI pipeline will run”.
First, we need to choose a CI platform. I wanted to avoid having to maintain or run CI in a separate platform than the one I host and version my code. Since I already hosted my experiments repository on Github, GitHub Actions was my first choice for platform.
GitHub provides runners of their own for free.
These are incredibly useful for most non-demanding opensource projects. The issue though is that the specs of those runners are insufficient for Unreal-based projects.
There’s the possibility of paying GitHub some dolla dolla bills for more powerful runners. We want to avoid that at this stage. Plus, the GitHub provided runners, paid or not, don’t have the needed dependencies to build Unreal projects readily installed and configured. That would involve extra work to setup, and is a rabbit hole of its own, which we can look at in a separate post.
Luckily, GitHub provides a way for us to use our own hardware with their concept of Self-hosted Runners.
This is the first piece of the puzzle’s solution.
Take a Windows host machine which we can spare and set it up as self-hosted runner.
The jobs that will run in that runner will have access to any libraries and programs we have installed on that machine. That is including any Unreal setup work we may have done. This is the key that turns a local development machine into a CI runner.
So let’s say we have installed Unreal Engine 5.2 on
That means that in a GitHub actions workflow we can in practice run Unreal’s Automation Tool commands in our CI pipeline.
If we look into the output logs when we run Build or Packaging of our Unreal project from within the Unreal Editor, it’s usually running Automation Tool commands under the hood.
Once we have our host configured as GitHub self-hosted runner, these same commands are available to be run in a pipeline:
on: [push, pull_request]
- name: Clone repository
- name: Show UAT help menu
run: "C:\EpicGames\UE_5.2\Engine\Build\BatchFiles\RunUAT.bat" -Help
Side-note: There are a few security issues from this. We are in practice providing access to a host machine on GitHub. They warn about it on their documentation. Be sure to setup the configs of a project to be private. If it is public and opensource, there are some settings also which we can set to prevent pull-requests from using the runners without the project owners’ approval. Be careful with this stuff. Or not… after all, why not, why shouldn’t we keep it?
The next piece of the puzzle is figuring out how to build and compile our project. If we run Unreal Automation tool locally with the
-List argument, we’ll see a list of commands we can use:
PS C:\EpicGames\UE_5.2\Engine\Build\BatchFiles> .\RunUAT.bat -List
Using bundled DotNet SDK version: 6.0.302
Parsing command line: -List
Initializing script modules...
Total script module initialization time: 0.31 s.
The command we are interested in this case is
BuildCookRun. The arguments that would just build/compile the project in my personal case were:
"%ueLocation%\Engine\Build\BatchFiles\RunUAT.bat" BuildCookRun -project="%projectLocation%\%projectName%" -noP4 -platform=Win64 -clientconfig=Development -build
I’ve abstracted away this command and the follow-up commands for testing and packaging in such a way that in the CI job I can call a batch file with a few arguments instead of writing the whole command verbatim:
"%ueLocation%\Engine\Build\BatchFiles\RunUAT.bat" BuildCookRun -project="%projectLocation%\%projectName%" -noP4 -platform=Win64 -clientconfig=Development -build
And here’s what calling that batch file would look like in the GitHub Actions workflow code:
# GitHub actions build step
# Define UNREAL_PATH and PROJECT_NAME environment variables
- name: Build
run: .\scripts\Build.bat $env:UNREAL_PATH (Get-Location).Path $env:PROJECT_NAME
Testing / Running automated checks
A quick detour before I go into how to implement a CI step to run tests.
I have to be honest, my first read of the
abdominal upsetting diarrhea inducing official documentation on Test Tooling that comes with Unreal felt… problematic… to say the least.
In terms of “lingo” there’s confusing spots depending on if we are more of a Programmer or a Tester. For example, starting with how the documentation defines which types of automated checks we can build and how to even set those up.
Side-note: I’m not even going to dwell in what hot takes members of different software testing cults and some charismatic testing influencers would spew upon
pretending to readreading the docs.
Thankfully, there are folks in the Unreal Development community that have poured over this so we don’t have to go to Mordor alone and fend for ourselves in the official docs.
After going through some pretty awesome community presentations and posts, like this one and this other one, I feel moderately confident to share the mental model I arrived at on Unreal Testing frameworks/tooling. It might be missing some bits, but we can split it in 3 parts, something like this:
There are unit-like tests we can build with their FAutomationTestBase library. And in theory, with it we can check bits of our game logic without having to “boot up” a running instance of the game. The simplest form of what it could look like is something like this:
IMPLEMENT_SIMPLE_AUTOMATION_TEST(FBasicExampleTest, "CoopGameFleepTests.Basic", EAutomationTestFlags::EditorContext | EAutomationTestFlags::ProductFilter)
bool FBasicExampleTest::RunTest(const FString &Parameters)
return TestTrue("math still works", 1 < 2);
I’ve seen examples where folks instantiate and/or mock parts of what we’d have in a live instance of a game, but I believe the key area of these is for checking stuff that doesn’t need the game to run, regardless of the complexity of the test script or the scenario being checked.
There’s the Functional Framework for Testing. The gist of it is we can build any kind of tests, using Blueprints.
These will execute against (partial) “live instances” of the game. Think of small test levels where in each we are only checking a bit of some game design mechanic. Folks seem to be using these for both unit and integration-like tests (see this example and this other example).
There is also support for complex intricate scenarios where we want to check some over-arching behavior of our game. Mileage may vary depending on what folks choose to do with these, but pretty much I could see these being useful for example for adding checks for regressions or testing core designs and mechanics of the game that always “need to work” on every build with the game running, not with mocks or just the logic of a mechanic.
There’s also something called Gauntlet - I don’t know a lot about it yet, but it appeared to be a distant equivalent in form to, say, running Playwright against a web app or an Electron app. The catch being - the test script code doesn’t need to know in theory about the internals of the target code its checking against.
So unlike using
FAutomationTestBase or the Functional Framework, one would be able to spin up a handful of different instances of the game using Gauntlet Controllers and then run whichever scenarios (even trigger Functional Framework tests) we might need at scale. Theres a video that explains this way better at depth here.
For the purposes of my experiment, for now I just wanted to be able to have a Test stage in my CI pipeline and trigger tests, even if it’s just a dummy test like the one example above that checks if… *checks notes*… basic math still works and 1 is lesser than 2.
After some setup and some research, I narrowed down the command I needed to trigger the test above to a sub-command of
UnrealEditor-Cmd.exe. When we run tests through the Unreal Editor, it’s basically using that command under the hood. We can find more about the setup with this example. Again, I’ve abstracted away the command I needed into a batch file like the one bellow:
"%ueLocation%\Engine\Binaries\Win64\UnrealEditor-Cmd.exe" "%projectLocation%\%projectName%" -nosplash -Unattended -nopause -nosound -NullRHI -nocontentbrowser -ExecCmds="Automation RunTests %testSuiteToRun%;quit" -TestExit="Automation Test Queue Empty" -ReportOutputPath="%projectLocation%\%testReportFolder%" -Log=%testLogName%
Calling that batch file in the GitHub Actions workflow code will look something like this:
# GitHub actions run tests step
# Define UNREAL_PATH, PROJECT_NAME, TEST_SUITE_TO_RUN, TEST_REPORT_FOLDER, TEST_LOGNAME environment variables
- name: Test
run: .\scripts\RunTests.bat $env:UNREAL_PATH (Get-Location).Path $env:PROJECT_NAME $env:TEST_SUITE_TO_RUN $env:TEST_REPORT_FOLDER $env:TEST_LOGNAME
After some back and forth, I finally managed to have the test step running, and even saving test results into artifacts on GitHub.
Packaging the game
Finally, after building and then running automated checks, we want to pre-package a build of the game and make it available to download.
From running Packaging from the Unreal Editor we can spot the Unreal Automation Tool that runs under the hood and we can abstract that into a reusable batch file:
"%ueLocation%\Engine\Build\BatchFiles\RunUAT.bat" BuildCookRun -project="%projectLocation%\%projectName%" -nop4 -utf8output -nocompileeditor -skipbuildeditor -cook -project="%projectLocation%\%projectName%" -target=%target% -platform=Win64 -installed -stage -archive -package -build -pak -iostore -compressed -prereqs -archivedirectory="%projectLocation%\%packageFolder%" -clientconfig=Development -nocompile -nocompileuat
# GitHub actions package step
# Define UNREAL_PATH, PROJECT_NAME, TARGET_NAME and PACKAGE_FOLDER environment variables
- name: Package
run: .\scripts\Package.bat $env:UNREAL_PATH (Get-Location).Path $env:PROJECT_NAME $env:TARGET_NAME $env:PACKAGE_FOLDER
- name: Archive package
In the end we have a working CI pipeline for our Unreal Engine based game, which compiles, runs tests and bakes a build of the game:
We can now download that build, and voila, we can boot up our own version of the Red Dead 3 - Tears of the Burger Kingdom:
There are a number of improvements that I could draw from the current solution. I might tackle those in other explorations and blog posts in the future. What I have in mind right now:
- In the current solution I’m using a personal Windows machine that I keep at home as the self-hosted GitHub runner. It was a deeply manual process to configure it. This won’t work at scale. There needs to be a way to automate that part in an infrastructure-as-code way, provisioning the github runners with the needed unreal dependencies and spawning more of those where needed;
- The test stage is only running a dummy test at the moment. I wanted to get that out of the way so the skeleton of the pipeline would be formed. Definitely will want to add proper checks and try out the different Testing related tooling Unreal provides;
- I’ve seen in other Unreal CI example implementations some folks setting up code coverage, which is also probably just a few lines of code away from the current implementation I have. Something to look into.
- I’ve done all of this assuming Windows as the main and sole platform for building and running the game. I think a good follow-up experiment is to see how the same Unreal tooling fares on Linux and Mac CI runners. Although let’s be honest… Linux and Mac have no games 😏 (*bait successful, tips fedora and leaves the room*).
On a serious note, there were also some problems just with this small experiment I experienced first-hand:
- Numerous crashes while using the Unreal Editor;
- The Editor stuttering and not working properly if it’s in full screen!?
- Intellisense not working properly both on Visual Studio and on VS Code… no code suggestions for the rest of us!
- The overall brittle experience of trying to uncover which Unreal Automation Tool commands are running in the background when we build or package the game via the Unreal Editor;
- The terse official documentation;
I might write about these as well in the future.
Such is life. I’m looking forward to my next experiments and learning more. These things look like weird magic oftentimes, but they are not impossible to pick apart and learn.
Credit where it’s due, here are some references of interest that have helped a lot while working on this post:
- Self hosted runners: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners
- Unreal Automation Tool: https://docs.unrealengine.com/5.2/en-US/unreal-automation-tool-for-unreal-engine/
- Article by Ben Humphreys on Unreal AutomationTest lib https://benui.ca/unreal/unreal-testing-introduction/
- Automated Testing at Scale in Sea of Thieves presentation by Jessica Baker https://www.youtube.com/watch?v=KmaGxprTUfI
- Alberto Mikulan’s alternative example of Unreal CI using Jenkins: https://github.com/Floating-Island/ProjectR and Paolo Galeone’s example using Gitlab CI: https://pgaleone.eu/cicd/unreal-engine/2020/09/30/continuous-integration-with-unreal-engine-4/
- Great presentations on Functional Test framework of Unreal https://www.youtube.com/watch?v=X673tOi8pU8, https://www.youtube.com/watch?v=528XSNTfxX8 and another one about Gauntlet https://www.youtube.com/watch?v=K8gNN0FyIw4
If you read this far, thank you. Feel free to reach out to me with comments, ideas, grammar errors, and suggestions via any of my social media. Until next time, stay safe, take care! If you are up for it, you can also buy me a coffee ☕