As I was putting together this website and getting it ready for publication, there were a couple of things I wanted to add that were inherently dynamic: a hit counter, and a contact form. Today, a lot of people faced with this situation would reach for a web framework like NextJS, perhaps combined with ‘serverless’ functions.² However, as my site was already nearing completion and I didn’t want to re-write it, I went looking for alternatives, and I found one: Perl CGI scripts.

CGI (Common Gateway Interface) is a web standard first developed in the 1990s that lets web servers (like the one hosting the static site you’re now viewing) call into external programs to process web requests. When a web request comes in for the CGI script, the web server calls the script, giving it information about the request like the parameters, and then responds to the website viewer with whatever output the script produced. Because this protocol is so simple, writing CGI scripts is very straightforward.

Instead of needing a complicated web framework or serverless functions runtime that I’d then be locked into, I can write simple one-file scripts that I can fully understand, with no special execution environment needed. It meant that I could keep the existing static site that I’d already built exactly the same, and add just those few bits of reactivity that I wanted. For example, the visitor counter is implemented using a bit of Perl that increments the counter in the database, and responds with an image rendering the count:

my $dbh = DBI->connect("dbi:SQLite:dbname=$db", '', '', {
    # connection args omitted...
});
my ($count) = $dbh->selectrow_array(
	'UPDATE hits SET n = n + 1 WHERE id = 1 RETURNING n'
);

$cgi->set_response_type('image/png');
$cgi->add_response_header('Cache-Control' => 'no-store');
$cgi->render(data => SegmentedCounter::render($count));

That’s it! The code is maybe 50 lines in total. I did use a simple wrapper library around common CGI tasks, CGI::Tiny, but it’s even possible to just print HTTP output to stdout to use it as a response.

Of course, CGI scripts do have their own downsides, which is why they are rarely used anymore for new projects. The main problem is latency: as the web server has to spawn a new instance of the Perl interpreter every time a request comes in, request responses are slower than with modern paradigms that use a prewarmed worker pool to handle requests.

One of the biggest timesinks here can be Perl needing to load all the included modules over and over again, so I would recommend keeping your scripts as simple as possible and avoiding heavy frameworks. For example, I cut startup time from 94ms to 21ms by pruning the modules used. What we pay for in raw performance though, we get back in readability and simplicity. Highly recommended!

On this site, I use a custom Figure component, that sets the inline property on svgs. At build time, Lume’s inline plugin replaces the <img> element with the SVG’s code.

In your CSS, you can add a rule for the svg:

.figure {
  color: var(--my-amazing-colour);
}

and you can have --my-amazing-colour change depending on the theme.

Enjoy your newly flashbang-proof diagrams!

And so, does the destination matter? Or is it the path we take? I declare that no accomplishment has substance nearly as great as the road used to achieve it. We are not creatures of destinations. It is the journey that shapes us. Our callused feet, our backs strong from carrying the weight of our travels, our eyes open with the fresh delight of experiences lived.

Building a custom learning platform for my university sounded fairly straightforward at first. The problem was that the client also wanted to be able to reliably run testing for hundreds of untrusted student code submissions in configurable environments.

Not production-ready

The sandboxing setup that I describe in this post was built under time constraints for an MVP, and is not suitable for running untrusted code in production. I talk about the tradeoffs further on.

Client brief

Avon was a project we built as part of Software Engineering Project, a year-long unit where our team was tasked with working with a real client to design a software system that would solve their problem. Our client was the School of Computer Science at the University of Bristol, and the pitch was as follows:

The rough sketch that we came up with was that lecturers would create courseworks using the UI, providing a template repository.² Then, our system would provision repositories on a Git forge (GitLab in our case) for each student, using that template. Lecturers would be able to select students and start a test batch, and they would be able to see the results of running the test script on each student’s repository. Simple enough, right?

The suffering begins

Initially, we thought that this would be fairly simple to implement. This is because there are already tools designed explicitly for running user-defined tasks on code stored in a Git forge: CI systems. Sadly, most of these tools require you to configure them within the repository itself, which wouldn’t have worked when students have access to that.³ Also, the tools that we evaluated didn’t work with our security model that the code in the repository could be overtly malicious, and this is problematic in our case as the CI runner would need to have credentials to be able to report results back to Avon. No such luck.

Because I suspected that designing a system that could satisfy these constraints would be a nightmare, I did quite a bit of research to find existing platforms that might be more aligned with our use case.⁴ One promising candidate was a Kubernetes-based platform called Argo Workflows, which caught my eye because it would have handled things like cloning the student’s repository into the container automatically, so we would need less custom logic. Unfortunately, we weren’t able to use it, primarily because it was quite heavy and complicated to use, in addition to having an opinionated API that makes sense for some use cases, but not for ours. We also couldn’t come up with a way to connect it to our backend and have it update the state of the test run asynchronously, rather than needing polling.

I also considered just having a simple service that accepts jobs, then locally clones the repo and runs the test command inside it, collecting the results. But because these submissions will come from students, we need some isolation to ensure that enterprising individuals can’t run a Bitcoin miner on our testing system, or gain access to data they shouldn’t. This began to point me towards using Docker.

Docker was also appealing because it would give us a way to run test jobs in parallel easily; some of the test suites that we were working with are quite heavy, and often containers took about thirty to forty seconds to start up, prepare the environment, run the tests, and report the results. With Docker, we can leverage AWS ECS to handle as many jobs as we need, and it was convenient (mostly) to hook up to the rest of our infrastructure on AWS.

Something else that we had to consider was that each coursework is different, and requires different dependencies, testing and language tools, et cetera. This means that lecturers need a way to be able to define the environment that their tests will run in, as well as the specific commands that will be run to perform the tests. An initial design involved giving lecturers access to the Dockerfile that would be used, but after some time thinking about it, I came to the conclusion that it wouldn’t be feasible. This is because we need to inject instructions to clone the repo, collect the results, and report them, and this wouldn’t be possible with a lecturer-controlled file. The design we ended up with was one where there are multiple pre-set environment options that a lecturer can choose from.⁵

Saying goodbye to our majestic monolith

At the point when a team member and I began working on the testing engine, the architecture of our platform looked roughly like this:

After the implementation of the testing engine, the architecture of the backend looked like this:

A queue-centric architecture emerged naturally here, because it elegantly solved several problems that I anticipated. If the backend ever died, any results would simply wait in the queue until it came back online to process any results. In addition, multiple backend copies could run at the same time, and it would be safe for any of them to process results and persist them to the same database. This is because even if two backends try to process the same result, or one dies halfway through processing one, we can just discard duplicates.

Unfortunately, even though this architecture is elegant and decoupled, it introduced significant challenges when debugging. Now, in addition to all the usual places where our code can fail, we introduced ten more!⁶

What actually happens when you click ‘run tests’?

To help understand why debugging this system was so difficult, it might help if I explain what the ‘run tests’ button in the UI kicks off. Here goes:

1. Lecturer starts the test batch. This is the part that you see - you select student groups in the UI to run tests for, and you click ‘run’.
2. Backend starts up jobs. The backend (FastAPI) connects to AWS and makes an API call to start up the testing jobs in the testing cluster. We start up one job for every repo in the test batch; they all run in parallel. The test jobs are now stored as ‘started’ in the database, and we attach the ARNs that we receive from the AWS API.
3. Containers start. At this point, the FastAPI backend has already moved on. In the background, the containers have all started running. We have some code written that logs into GitLab using a Service Account, and clones the student repo into the container so the tests can access it. Then, we run the test command that the lecturer provided in a subprocess with the parent process’ environment variables removed.⁷
4. Tests execute. As the tests run, we collect any logs that are produced and store them to a file.
5. Tests finish. When the tests finish, we take the log file and upload it to S3. The URI of the S3 resource and the exit code of the command that the lecturer provided is then sent in an SQS message.
6. Backend persists results. A worker consumes the SQS queue. It receives the message and persists the results to the PostgreSQL database, with the appropriate status and log file URI. This status will then appear in the UI, which will poll for updates while viewing the test run.⁸ The worker is designed to be idempotent, and discard any duplicates that may be received.

Sadly, none of this project (except perhaps the initial CRUD routes) could be said to have been plain sailing, and that’s what I’ll talk about next.

Nothing ever works

As you can probably imagine from that workflow, there are lots of moving parts here, and each one can fail in unique and fun ways that are difficult to observe. The hardest part of debugging a distributed system like this is that failures become distributed too. When you run something locally, or as part of a monolith, problems are (usually) visible immediately, and in one place. With something like this, often the most you’ll get is just nothing, and we just have to work from there: hunting through AWS container logs, and hoping that we had the foresight to instrument things.

For example, when testing an early version of the system, we clicked go, but it just got stuck and nothing happened. I checked the AWS console, and the container did start, and even exited as well. In the logs, the tests completed, and it looked like everything was okay. Eventually, I worked out that the problem was that the container didn’t have permission to upload to S3, or send results to the queue, because I had incorrectly configured the IAM role’s permissions. This particular issue was especially difficult to debug because every change required the registration of a new Task Definition Revision. To avoid this kind of issue in the future, I added a step to the container startup that tested the AWS connection before doing the tests, with helpful log output.

This was the first exposure that I (and indeed the rest of the team) had to queue-driven systems like this, and it was a great introduction to how with distributed systems, it’s less about getting it working, and more so about reasoning through the often complex ways in which it can fail.

Conclusion

In the end, I did manage to get it working. We were able to demonstrate starting test runs, looking at test run logs in the UI, and seeing it automatically refresh when the run completes. It shipped, and the client was happy. Even though this project was only a proof-of-concept, and it might never be used, I’m still really glad I picked it at the start of the year. In the end, the user interface still just has a ‘run tests’ button, but the machinery underneath it turned out to be far more complicated than I had expected.

Despite being frustrating at times, this project taught me better than any other I’ve built how to ruthlessly separate ‘nice-to-have’ and ‘need-to-have’. In the beginning, we had a lot of ideas and features that we wanted to implement properly, like sandboxing and real-time log streaming, but when deadlines started approaching, it became necessary for us to sit down with the client, understand what it is that they really want, and ship something that works, even if it isn’t perfect. This is something that has changed how I approach planning projects today, and I’m grateful for it.