gRust Why making your own source server is a great idea
gKarts is built to be a scale-able and high performance gamemode. With that in mind, it means we need our servers to run with minimal resource usage. Here's how we're going to achieve just that.
This month, we restarted the gKarts project. And so we set some goals in mind for the back-end of the gamemode:
- Optimization - We should be able to run multiple game instances on a single core without impact on user experience or any performance detriments.
- Efficient/Effective Networking - The client should be able to drive without any de-syncing issues even with 250+ms ping.
- Security - The players cheating shouldn't be interrupted by any cheaters within games.
- Scalability - the server should automatically spin up and down depending on demands on the network.
Optimization is the one thing that people either dread or make it their entire life's focus. Even if we move any Lua work we do on the server to a module with C++ we'd be hitting a hard boundary, the Source SDK. The source SDK is the game engine that Garry's Mod runs on. The Source SDK is great for regular DarkRP or TTT servers and does everything you want; features in non-performance critical contexts, however when you have the goal of scalability this really starts to add up to the cost of hosting those instances. And thus, I decided to embark on quite possibly the most difficult and challenging project I have worked on in my programming career so far, writing our own server. Alright, so with this set we went about deciding a suitable language to write this project in. We needed memory safety, with no impact on the application.
Yeah, we chose rust, hence the project name Rust. Thanks to Billy for his suggestion when we were figuring out how and what we wanted to write this in. Some benefits to rust as a programming language is it's ability to generate extremely optimized LLVM code, Type and Memory safety along with a plethora of other extremely useful traits.
Admittedly Rust was an alien language to me and so, I set about writing the application, refactoring as I went and learnt new dynamics of the language compared to what I was use to, being more strictly OOP languages like Java and C#. Rust has it all, optional OOP, direct memory management via the unsafe keyword; Now you're wondering, "you said earlier we needed memory safety", well surprisingly when porting some of C++ code from the opensource aspects of Source SDK we needed to access memory to achieve some of it. I expect later on down the line we'll refactor this to completely remove the use of direct memory management.
Networking was our next issue we needed to solve. Writing efficient networking is the quintessential aspect to this gamemode, otherwise when using the kart people will start to lag behind, our solution for this? Well, it's a simple (idea, with a complex implementation in practice) one, implement vehicles purely client-side (Blog post out on this) and just let the server network less often and then interpolating between different positions so you have smooth performance on our servers and Digital Ocean won't scream at us for using too much bandwidth.
Security, this is one of the core aspects of our gamemode and as a result we follow the rule of "Don't trust the client", usually. However in this circumstance, we follow the mantra "Fact check the client". What I mean by this is, because our networking is so light, we need to make sure our client is sending reasonable data, within what we expect and so, even if someone was to abuse this, they wouldn't have a noticeable improvement over other players. This coupled with offensive checking of the client, which should mean we won't have to worry about cheaters within our games.
As hinted at earlier, we we settled on Digital Ocean for our hosting of the game services. Usually, this would be absolutely abhorrent, but because we're writing our own server, there's no reason not to do it. We're specifically targeting the server towards use on very low end servers, which means we should be able to run 128 players on half a core and 2gb of ram, easily. In early tests of gRust it barely uses 15mb's of ram per user, however I expect this early prediction will change during our project, as the code base grows. So how are we going to scale with Digital Ocean, Well, the plan is to create a docker image and publish the ip address it's allocated to an internal API, so we can start controlling and using it to serve our clients trying to play gKarts.
Some issues I've faced so far with gRust.
Source 1's networking protocol, oh boy this is a pain in the ass. Being relatively restricted in what tools I'm using to figure out what we're meant to be responding to client, I've turned to some amazing work carried out by the Garry's Mod community, I'd like to thank Leystryku, and Galico for their work within the space for proving some great projects on GitHub which have aided our development as they've done a lot of the legwork in reverse engineering the source protocol.
Another issue, again with networking, porting a C++ code-base which uses some quirks with C++ to a Rust code base. Encoding packets that the client can understand, I've started to use some tools provided by the Rust community for this specific purpose, however I've had issues with them so I decided to bite the bullet and translate it myself. I've had great success with this approach and although slow and tedious I've been able to create a pretty performant and reliable buffer decoder and encoder, however I'll more than likely have to refactor this later on.
Some cool things I've learned/found out
Working within the networking side of gRust I've been able to make some pretty useful things that client emulators (often used to perform DoS attacks). Something interesting is that the regular Source SDK Servers don't validate packets all that well (well enough, but not good enough in most cases), the luxury of only targeting Garry's Mod is that we can validate packets slightly better. And so, the clients that don't follow it are presumed to be bots and are kicked.
As we can see here, it works pretty well.
Well, It's a long way up from here, I've got to finalize our implementation server side of networking, have a functional map loader (BSP fun) which populates the server with the correct props from the map, a system to handle entities and much more.