Network Concepts: The Stack

Building out a multiplayer framework, and even working on an existing codebase, requires good knowledge of the overall stack in play by your server and your client. By the term “stack”, we discuss layers within the application that eventually support gameplay systems. Each layer supports the next. This can sometimes be confused with server stacks, where various servers are operating with eachother in distribution. This is not that - we’re focusing on the application that our server and client will sit on.

We have to think specifically on the relationship between clients and servers. Each client is going to be connecting to a remote peer. How we accomplish this requires us knowing in advance how we are structuring our network model. Is this a peer-to-peer game, or is this a dedicated server?

A lot of developers will immediately jump to wanting to build out a client-server model because of the reputation of peer-to-peer being extremely exploitable. This intuition is solid, but this is where business either overrides this choice, or you may want to actively decentralise your game for longevity. Here are the pros and cons of both approaches.

Dedicated Server Model

Dedicated servers are ran as a separate installation alongside your primary game client. They most often share parts of the game code base, or in some cases, share only the same messaging protocol. Usually though, there is a lot of overlap in the game client and server implementations to utilise shared concepts.

In most cases, these are servers you are deploying either to the cloud, or to bare metal. Alternatively some developers offer players the ability to host their own dedicated servers, but the practice of this is dwindling as companies move towards live operational games. A developer may also consider this for a hobby project to decentralise hosting towards tech savvy players.

Pros

  1. Hardened network model. There are less opportunities for hackers to manipulate authoritative data, however not completely protected.
  2. Controlling the deployment of IP - this is a business choice more than anything.
  3. Simulation quality of your game remains consistent to your server hardware. Player’s individual performance doesn’t degrade the quality of the match.
  4. Gameplay synchronization is a lot more easier to rationalize with because the server is authoritative on all matters of gameplay. This can make payment or reward systems as a blackbox, and away from immediate exploitation.

Cons

  1. Significant expense tied to the performance of your game. Profiling your game’s CPU now becomes an important factor before and during your game’s launch.
  2. Scaling problem. How does your game handle launch, and how does it handle concurrent players scaling down without burning money on empty servers? How does your game load balance?
  3. Creates a surface for denial of service. Likely your gameplay server will be utilizing UDP, so you’ll be inheriting problems surrounding UDP flooding vectors.
  4. Requires a live operation team to support 24/7 availability, or moves this responsibility to players hosting servers for communities. Minecraft, SA:MP (San Andreas: Multiplayer), FiveM, and Left 4 Dead are examples of community hubs taking the responsibility of non-official hosted platforms.

Peer-to-peer Model

Peer-to-peer (often abbreviated as P2P) is a distributed network model. Take a lobby of 10 players, with peer-to-peer each player is either connected to each other, or to one single host. The principle idea is that the game clients drive other game clients.

In a match where one host controls the game, this acts similar to a server model. There is usually some backend service that is determining grouping of players and driving host migration. However some games now lean on the idea of distributed P2P, where each player is connected to every other player. The game breaks down concepts such as: objects with physics, metagame systems, gameplay systems, with a new layer of individual ownership and migration on player machines.

For example (crudely), a player approaching a field of cows and being the closest player to those cows will initiate a migration to the previous owners’s machine. This is so that they can simulate them with better experienced fidelity because of their newfound proximity (and now responsibility).

And maybe your game is set up so that if you approach a field that is empty, your client may consider spawning cows in so that it remains populated if others approach too. Then you’d be passing on the (cow) torch to others if you leave.

Pros

  1. No server costs involved. Your entire simulation, responsibility, and upkeep is limited to just connecting players to eachothers machines.
  2. Easier for players to establish self hosted servers through “listen servers”, where the player can host the simulation directly to other players.
  3. Engineering efforts can be focused within the game client, no server variant or application needs to be deployed or maintained.

Cons

  1. Significant challenges on object and gameplay system migrations. Passing over data to another player can be a hard problem to solve, and doing this seamlessly without visual disturbance is where the larger portion of the problem lies - and where your tech debt starts to pile up.
  2. NAT traversal issues. Some players who have closed NATs may never be able to host gameplay to others, or obtain partial ownership in a shared object ownership model.
  3. IP exposure between clients. A client connected to a session host can be made aware of that host’s IP address, opening up the possibility of denial-of-service, or threatening behavior. The same problem happens with the host - they receive every connected player’s IP address in their session. Some integration platforms (such as Steam, EOS) have built around this through relay servers, but that is a concept you have to tie yourself to a platform for.

Genre will define your model

Picking your model may be completely dependent on your game’s genre. The underlying implementations at its core shouldn’t change much, the concepts should be the same (bit serialization, compression, packet acks, etc), but peer-to-peer or server models will change how your higher level network implementations will operate.

Consider a competitive FPS. This is a high stake, fast pace environment - that also may be dealing with rewards and player progression. Peer-to-peer doesn’t make much sense here because your competitive integrity is going to be compromised. Rewards and progression is compromised additionally.

An open world game may lean into peer-to-peer, or a dedicated model that has players simulate objects to other players through the dedicated server. We’ll touch object replication concepts in a later post, but for now start to consider what objects in your game are going to be synchronised - and what constraints do you need to set early.

Layering the Stack

Now that both client and server models are explained, we have two paths our stack can be built upon.

At it’s core, both models implement the follow stack - lowest to highest:

  1. Transport Layer (traditional OSI level 4) - sockets in UDP, or TCP (hopefully not) will manage your peer-to-peer or peer-to-server relationship. At this layer, we are just working with sockets - or connections. Some libraries that back this layer are: ENET, RakNet, Steamworks Networking API, or traditional winsock/BSD sockets.
  2. Player (or the server) - accepting a socket requies you to reinterpret what responsibility it has. For a server, you will be accepting sockets that represent players. You want to handshake with these connections and accept some form of authenticated request, and reject requests that do not seem right. You would do your versioning here too so that you are rejecting connections that are either outdated clients, or do not seem to be from the same ecosystem. Magic tags are great for this.
  3. Serialization Utility - transport layers send blobs. How we serialize and deserialize those blobs is based on our serialization utilities we build to write bits, and read those bits on the other side - importantly in the same sequence.
  4. The Manager - you want both clients and servers to be distributing messages from the remote connection and processing them locally, whilst also pumping messages out to the remote connection. The manager should be updating regularly.

The next parts of the stack are molded based on your network model.

  1. Message Rx + Dispatcher - part of your server and client needs to understand that incoming packets are actually messages destined for a target system. You’d aim to identify these packets by checking a unique flag at the start of the payload, and then an identifier of what message type this is. The dispatcher part of this layer then ferries this part of the message to the appropriate deserializer and gameplay system.
  2. Gameplay Sync Rx - both server and client need this layer. It’s obtusely complicated, and will scale up work the more your game depends on it being right. Similar to the Message Rx layer, we are not tagging messages with unique identifiers. Instead, we are tagging them with destination objects. This depends on your game having a good grasp on unique entity IDs, and the ability for these IDs to be replicated from an authoratative source. Gameplay Sync Rx creates three main implementations: what objects to create, what objects to destroy (or prune), and what state needs to be applied.
  3. Gameplay Sync Tx - both server and client also need this layer. Client responsibility is usually at a much lower scale, but if you know you are building a peer-to-peer game, vs. a dedicated server model, you may be able to get away with only developing this tech on the server - and have your clients send only inputs to control their player. In Unreal, that concept is called Pawns - and we’ll cover that more in a later post. For servers, this is a large performance challenge to get right. An MVP might look like iterating over all of your synchronised objects on the server and sending out each object state in full. You’ll find quite quickly that you are resending data repeatedly for properties that haven’t changed. You’ll also find out the hard way that not every object should be sent to players, and you cannot afford to iterate over every object every tick. Two concepts that solve this are: Potentially-visible-sets, Delta Compression, and scoped rules. Glenn Fielder covers this extensively in his networking for video games series.

The Gameplay Sync Rx layer is extremely hard to get right, and is very fragile to latency, corruption, and missing synchronisation components your game may depend on being synced. However, if done right, this can be a very powerful full loop that can sync state immeediately to a target object - giving you the sense of a replicated world from server (or host) to client.

The reward at the end of this is liberating.