Hosting the Compete With Team Europe CTF
The Compete With Team Europe CTF was a multi-national capture the flag contest organized by ENISA and the Team Europe Trainers. It was created in order to give Team Europe a chance to refine their skills under pressure.
It consisted of two events: a Jeopardy and Attack-Defense CTF.
2023-07-04: Right-size the VMs 2023-07-05: Cloud done 2023-07-06: Source NAT / Networking done 2023-07-07: All but 1 checker green 2023-07-07: Recover from cluster downtime, learn why m6i instances are better
We chose the ECSC2022 / Faust CTF engine early, to align with the tools that the ECSC 2022 and ICC 2022 competitions used. This would be familiar to the players.
Ideating the challenges
The challenge ideas were discussed by the organizers early on, and validated against the format of the game. Challenge authors were CTF players from Team Europe.
Authoring the challenges
The challenges were built in a manner of a few weeks by the authors working
closely together. The deliverable format was a
Since the vulnbox was deployed by us, the easiest way to set it up was to use Packer and generate an AMI from that. Then, it’s trivial to roll out the same AMI for 25 teams.
For repeated / iterative work, we generated a new AMI and replaced the running test copies of the vulnbox with the new image.
Since in most aspects, the 25 VMs are identical, they can be spawned from the same AMI and that would be all. However, the instances will need some individualized configuration as well. This is where user data comes in. Both AWS Windows and Linux instances support user data, which is essentially a text field in the AWS database that the VM can access. Official AMIs are configured to read the user-data as an EC2Launch or cloud-init config file, for Windows or Linux respectively.
User data is read from the EC2 Instance Metadata Service (IMDS).
EC2Launch is a system feature integrated into AWS, that essentially performs first-boot setup on Windows VMs. The default configuration for EC2Launch sets the admin password and uploads it to EC2.
The syntax of EC2Launch is either a PowerShell file in XML-like brackets, or a
YAML file detailing the steps to perform. Critically for us, EC2Launch includes
a feature to reboot the instance and continue the script where it left off. In
order to trigger that feature, the same exit code is used as when triggering a
reboot in SSM System Manager Run Commands:
exit 3010. Exiting with that code
will reboot the VM and simply restart the script.
We found that for the most part, creating marker files (stage1.txt, stage2.txt) and testing for their existence is sufficient to restart the script at an appropriate position.
Unfortunately for us, user data also creates a bunch of logs on the system, under Windows Event Viewer, C:\ProgramData\Amazon\EC2Launch\logs and %TEMP%. The files are saved to disk after the PowerShell script exits.
So, this way we can configure 25 Windows domains automatically. Nice!
cloud-init is a system feature built into Linux AWS AMIs to also initialize the VM on first boot. There’s many more options, but the main ones we’re interested in are in the user data.
Building the checkers
ENISA’s AWS account was used. Since they deploy Control Tower, the default VPC is deleted, so a VPC and subnets need to be created.
We opted to go for the following AWS architecture:
VPC | +--- VPN subnet 10.20.150.0/24 | | | +---- VPN (m6i.2xlarge) | +--- Gameserver subnet 10.20.151.0/24 | | | +---- Gameservers (x 11) | | | +---- Windows Checker VMs (x 6) | +--- Player 1 subnet 10.20.1.0/24 | | | +---- Player 1 dc1 10.20.1.4 (t3.medium) | +---- Player 1 srv1 10.20.1.5 (t3.medium) | +---- Player 1 lin1 10.20.1.6 (t3.medium) | +--- Player 2 subnet 10.20.2.0/24 ...
The natural tool for the job is Terraform, due to its modules support and simple iteration features. It also naturally fit into the model, where one person could prepare the changes and another could apply them to the cluster.
The setup was quite simple: a module was used for Player subnets, and using a hashmap, it was possible to iterate over player subnets to avoid repeating the player block over and over.
One of the biggest goals was to offload to the cloud provider as much as possible. This way, there are less single points of failure. However, there’s a core part of the Faust CTF architecture that doesn’t lend itself to resiliency: the router.
In the CTF, we hosted the router on a single EC2 instance (“VPN”) and scaled the instance up to sufficient size. This way the chance of the router quitting on us was reduced.
In the subnet configuration, the route for
0.0.0.0/0 was pointed to the ENI of
the VPN instance. We disabled source/dest checking on the router, and made sure
that the router had two network interfaces. (In hindsight, that might have been
The AWS reachability analyser was crucial to help traverse the nesting doll of security groups, routes etc.
One of my main learnings here was: no matter how much you want it, AWS will not route in-subnet traffic out based on Subnet Route Tables.
Why a Kubernetes cluster
Initializing the database
Deploying the gameserver
Deploying the checkers
Deploying the Windows checker
Deploying the TCP submission server
Deploying the network
Deploying the reverse proxy
Closing the network
Deploying the VPN
Deploying ticketer (again)
Fixing the checkers
Updating the scoreboard design
Fixing the VMs
Scoping-in Jeopardy server
Scaling up to 25 teams
We reset all the VMs on the night before the CTF so there are no logs or similar available.
Resetting VMs was simple: terminate in AWS and re-deploy. At that point we had not figured out how to clear EC2Launch metadata from the VMs, so instead we opted for the manual approach of having @JaGoTu RDP into each box and delete the EC2Launch artifacts.
We started the CTF by simply setting the gamecontrol times correctly. The first 5 ticks were failing for basically everyone, because of “usual Faust problems”.