What's behind a ‘run tests’ button

And so, does the destination matter? Or is it the path we take? I declare that no accomplishment has substance nearly as great as the road used to achieve it. We are not creatures of destinations. It is the journey that shapes us. Our callused feet, our backs strong from carrying the weight of our travels, our eyes open with the fresh delight of experiences lived.

Building a custom learning platform for my university sounded fairly straightforward at first. The problem was that the client also wanted to be able to reliably run testing for hundreds of untrusted student code submissions in configurable environments.

Client brief #

Avon was a project we built as part of Software Engineering Project, a year-long unit where our team was tasked with working with a real client to design a software system that would solve their problem. Our client was the School of Computer Science at the University of Bristol, and the pitch was as follows:

A custom VLE¹ that can integrate with GitLab, automatically provision Git repos for students, and run tests on code submitted for coursework.

The rough sketch that we came up with was that lecturers would create courseworks using the UI, providing a template repository.² Then, our system would provision repositories on a Git forge (GitLab in our case) for each student, using that template. Lecturers would be able to select students and start a test batch, and they would be able to see the results of running the test script on each student’s repository. Simple enough, right?

The suffering begins #

Initially, we thought that this would be fairly simple to implement. This is because there are already tools designed explicitly for running user-defined tasks on code stored in a Git forge: CI systems. Sadly, most of these tools require you to configure them within the repository itself, which wouldn’t have worked when students have access to that.³ Also, the tools that we evaluated didn’t work with our security model that the code in the repository could be overtly malicious, and this is problematic in our case as the CI runner would need to have credentials to be able to report results back to Avon. No such luck.

Because I suspected that designing a system that could satisfy these constraints would be a nightmare, I did quite a bit of research to find existing platforms that might be more aligned with our use case.⁴ One promising candidate was a Kubernetes-based platform called Argo Workflows, which caught my eye because it would have handled things like cloning the student’s repository into the container automatically, so we would need less custom logic. Unfortunately, we weren’t able to use it, primarily because it was quite heavy and complicated to use, in addition to having an opinionated API that makes sense for some use cases, but not for ours. We also couldn’t come up with a way to connect it to our backend and have it update the state of the test run asynchronously, rather than needing polling.

I also considered just having a simple service that accepts jobs, then locally clones the repo and runs the test command inside it, collecting the results. But because these submissions will come from students, we need some isolation to ensure that enterprising individuals can’t run a Bitcoin miner on our testing system, or gain access to data they shouldn’t. This began to point me towards using Docker.

Docker was also appealing because it would give us a way to run test jobs in parallel easily; some of the test suites that we were working with are quite heavy, and often containers took about thirty to forty seconds to start up, prepare the environment, run the tests, and report the results. With Docker, we can leverage AWS ECS to handle as many jobs as we need, and it was convenient (mostly) to hook up to the rest of our infrastructure on AWS.

Something else that we had to consider was that each coursework is different, and requires different dependencies, testing and language tools, et cetera. This means that lecturers need a way to be able to define the environment that their tests will run in, as well as the specific commands that will be run to perform the tests. An initial design involved giving lecturers access to the Dockerfile that would be used, but after some time thinking about it, I came to the conclusion that it wouldn’t be feasible. This is because we need to inject instructions to clone the repo, collect the results, and report them, and this wouldn’t be possible with a lecturer-controlled file. The design we ended up with was one where there are multiple pre-set environment options that a lecturer can choose from.⁵

Saying goodbye to our majestic monolith #

At the point when a team member and I began working on the testing engine, the architecture of our platform looked roughly like this:

After the implementation of the testing engine, the architecture of the backend looked like this:

A queue-centric architecture emerged naturally here, because it elegantly solved several problems that I anticipated. If the backend ever died, any results would simply wait in the queue until it came back online to process any results. In addition, multiple backend copies could run at the same time, and it would be safe for any of them to process results and persist them to the same database. This is because even if two backends try to process the same result, or one dies halfway through processing one, we can just discard duplicates.

Unfortunately, even though this architecture is elegant and decoupled, it introduced significant challenges when debugging. Now, in addition to all the usual places where our code can fail, we introduced ten more!⁶

What actually happens when you click ‘run tests’? #

To help understand why debugging this system was so difficult, it might help if I explain what the ‘run tests’ button in the UI kicks off. Here goes:

1. Lecturer starts the test batch. This is the part that you see - you select student groups in the UI to run tests for, and you click ‘run’.
2. Backend starts up jobs. The backend (FastAPI) connects to AWS and makes an API call to start up the testing jobs in the testing cluster. We start up one job for every repo in the test batch; they all run in parallel. The test jobs are now stored as ‘started’ in the database, and we attach the ARNs that we receive from the AWS API.
3. Containers start. At this point, the FastAPI backend has already moved on. In the background, the containers have all started running. We have some code written that logs into GitLab using a Service Account, and clones the student repo into the container so the tests can access it. Then, we run the test command that the lecturer provided in a subprocess with the parent process’ environment variables removed.⁷
4. Tests execute. As the tests run, we collect any logs that are produced and store them to a file.
5. Tests finish. When the tests finish, we take the log file and upload it to S3. The URI of the S3 resource and the exit code of the command that the lecturer provided is then sent in an SQS message.
6. Backend persists results. A worker consumes the SQS queue. It receives the message and persists the results to the PostgreSQL database, with the appropriate status and log file URI. This status will then appear in the UI, which will poll for updates while viewing the test run.⁸ The worker is designed to be idempotent, and discard any duplicates that may be received.

Sadly, none of this project (except perhaps the initial CRUD routes) could be said to have been plain sailing, and that’s what I’ll talk about next.

Nothing ever works #

As you can probably imagine from that workflow, there are lots of moving parts here, and each one can fail in unique and fun ways that are difficult to observe. The hardest part of debugging a distributed system like this is that failures become distributed too. When you run something locally, or as part of a monolith, problems are (usually) visible immediately, and in one place. With something like this, often the most you’ll get is just nothing, and we just have to work from there: hunting through AWS container logs, and hoping that we had the foresight to instrument things.

For example, when testing an early version of the system, we clicked go, but it just got stuck and nothing happened. I checked the AWS console, and the container did start, and even exited as well. In the logs, the tests completed, and it looked like everything was okay. Eventually, I worked out that the problem was that the container didn’t have permission to upload to S3, or send results to the queue, because I had incorrectly configured the IAM role’s permissions. This particular issue was especially difficult to debug because every change required the registration of a new Task Definition Revision. To avoid this kind of issue in the future, I added a step to the container startup that tested the AWS connection before doing the tests, with helpful log output.

This was the first exposure that I (and indeed the rest of the team) had to queue-driven systems like this, and it was a great introduction to how with distributed systems, it’s less about getting it working, and more so about reasoning through the often complex ways in which it can fail.

Conclusion #

In the end, I did manage to get it working. We were able to demonstrate starting test runs, looking at test run logs in the UI, and seeing it automatically refresh when the run completes. It shipped, and the client was happy. Even though this project was only a proof-of-concept, and it might never be used, I’m still really glad I picked it at the start of the year. In the end, the user interface still just has a ‘run tests’ button, but the machinery underneath it turned out to be far more complicated than I had expected.

Despite being frustrating at times, this project taught me better than any other I’ve built how to ruthlessly separate ‘nice-to-have’ and ‘need-to-have’. In the beginning, we had a lot of ideas and features that we wanted to implement properly, like sandboxing and real-time log streaming, but when deadlines started approaching, it became necessary for us to sit down with the client, understand what it is that they really want, and ship something that works, even if it isn’t perfect. This is something that has changed how I approach planning projects today, and I’m grateful for it.

Footnotes #

1. Virtual learning environment. ↩

2. This would contain, for example, the skeleton code for a programming coursework. ↩

3. It may have been possible to get around this by injecting the CI config automatically when tests are run, and keeping it out of the repo otherwise, but that would be really complicated. ↩

4. Some systems that we explored included E2B, Deno Sandboxes, and several others. Unfortunately, none of these platforms were really what we were looking for, as they seemed more focussed on different use-cases, such as providing a workspace for AI agents. ↩

5. These correspond to individual Dockerfiles that we set up, that each installed a different set of dependencies. ↩

6. Much time was spent debugging this. Especially AWS. This was our first time using it, and ‘into the deep end’ is an understatement. It is an engineering marvel, and exceptionally powerful, but I think we had to re-configure everything at least four times before it worked. ↩

7. I did attempt some level of hardening here. Using tini, we handle signal forwarding, and prevent leftover zombie processes. Unfortunately, due to deadline constraints, I wasn’t able to explore proper sandbox hardening as much as I would have liked. In particular, something like Firecracker would be a much more secure sandboxing primitive to use here, and it was something that I did consider, but it is far more complicated to get started with, and due to time constraints it wasn’t possible. ↩

8. This was another compromise: we decided that for this project, polling would certainly be acceptable. We considered WebSockets or SSE but it would have been very complicated for little benefit. ↩