Runes Reforged: A Technical Retro

Hey folks! We’re going to take a trip back in time. The year is now 20xx, and we’ve decided that it’s finally time to send one of League of Legends’ most long standing, revered or reviled features (depending on who you ask) to the meme graveyard.

That’s right - it’s time to retire runes and masteries.

I’m Dave Le, a member of the League of Legends Gameplay Systems engineering team. My team was tasked with integrating the many technical systems required to make the runes and masteries replacement - Runes Reforged - a reality. We’ve previously talked about our goals in changing the design of our pre-game experience to become more streamlined and impactful. And now that the dust has settled after an entire rune-filled season, we’d like to take a look at what changed in the systems responsible for getting your runes choices from pre-game to in-game.

So, brace yourself - The Reforging is coming.

Runes Preforged

Before any game of League even begins, every player is already making pre-game decisions - selecting their champions, picking summoner spells, and even banning the potential options of their opponents. The runes and masteries system was designed to give players additional interesting pre-game decisions to make, in the form of pure stat bonuses via runes and situational bonuses via masteries. While they did do their job for the past many years, the technical systems powering them had aged poorly to the point where they had become too inflexible to support anything new beyond simple quality of life changes.

Runes and masteries are so old, they predate the Game Data Server, our preferred system for managing game content. Instead, runes and masteries content exists as loose configuration files that define names and descriptions for each item. A mastery involves logic beyond just stat adjustment, and are also accompanied by a script file that executes the conditional behavior during the game. For a rune, the configuration file contains mod values for every possible stat value that can be modified by the runes system. If you recall, runes only modified a single stat. So, yes, there’s going to be a whole lot of zeroes.
 

DisplayName=game_item_displayname_10001
Description=game_item_description_10001
FlatCritDamageMod=0.022314
rFlatCritDamageModPerLevel=0
PercentCritDamageMod=0
FlatMovementSpeedMod=0
…

“Repeat 50+ more zeroes for 50+ more files. Fun.”

Each of these runes and masteries were designated with a unique ID value, stored in a SQL database accessed by “the Platform” - a large, monolithic service that houses a large number of components, including ones responsible for starting games.

The Platform contains business logic to validate the player’s choice of runes and masteries to prevent illegal combinations, identifies which runes and masteries are disabled or deprecated (in the case of certain masteries in between seasons), and confirms access to individual items (ownership for runes, level restrictions for masteries). Unfortunately for us, all this business logic snaked across a large number of code files as the Platform gained functionality and bloat over the many seasons of League. Runes and masteries related logic is attached to 100+ files in the Platform alone, weighing in at a hefty 1.5% of the total codebase. Scary stuff!

Finally, as players make their pre-game selections in the League client, we need a way to collect and communicate them to the Game Server which ultimately hosts the match. Game Server instances are allocated by a service running on Game Server hosts called the Local Server Manager, or “LSM.” The LSM starts a new Game Server process when it receives a GAME_START message from its counterpart: the Global Server Manager, or “GSM.”

The GSM is yet another component of the Platform. When the Platform deems itself ready to start a game, the GSM component will update a pending game queue. The LSM is continually watching this queue and will spin up a new Game Server process when a new item appears.

“Hello middle management”

As a player makes updates to selections in the League client, the client sends those updates to the Platform and the GSM. When the players are ready to go and the GSM sends the GAME_START message to the LSM, it includes a blob of data required to start the game. This blob, known as the “Megapacket,” contains all of the players’ said selections as well as other information needed to start the game, such as which map (e.g. Summoner’s Rift) and what type of game mode (e.g. Normal Summoner’s Rift vs URF).

The Megapacket starts out as just a JSON document for ease of reading and ease of modification, and is ultimately serialized into a binary format for transportation and parsing downstream.

{
 "participants" : [
   {
     "championId" : 1,
     "runes" : [ARRAY_OF_RUNE_IDS],
     "summoner" :
     {
       "level" : 30,
      "masteries" : [ARRAY_OF_MASTERY_IDS],
       "spells" : ["SummonerHeal", "SummonerHaste"]
     },
     "summonerId" : 1,
     "summonerName" : "Player1",
     "teamId" : 100,
     "teamParticipantId" : 0
   }
 ],
 "platformId" : "NA1",
 "mapId" : 21
 ...
}

If we decide to alter the runes or masteries selection data beyond simple arrays of IDs to support a new design paradigm, it would require changing the Megapacket data structure to support it. This also means downstream systems that parse the Megapacket would also need to be updated every time - something we’d prefer to avoid if we’d like the flexibility of modifying it often to enable new design spaces.

Runes Reforged

While our ultimate goal is to retire runes and masteries once and for all, we’d be remiss not to consider how we approach future feature development. As we rip the floorboards out of the house that runes and masteries built, we have the opportunity to lay a foundation for enabling new features that customize the game without having to start from scratch each time.

Game Customization

For getting rune selections to the Game Server, we can reuse the existing transport pipeline of the League client to Platform GSM to LSM to Game Server. However, we’ll still need to alter the Megapacket payload in some way to include the selections.

The existing Megapacket structure contains a section dedicated specifically to runes and masteries. While removing that deprecated section, we introduce a completely new section to the Megapacket: Game Customization. This section functions similarly to our previous system: an array of selected rune identifiers. But rather than being tied to a specific content system, we add a “content identifier” field which the Game Server can use to determine how to parse and apply the selected content.

{
  "participants" : [
    {
      "championId" : 1,
      "gameCustomizationObjects" : [
        {
          "category" : "Runes",
          "content" : {
                        "runes": [8437,8446,8444,8451,8224,8237],
                        "primaryPath": 8400,
                        "secondaryPath": 8200
                        }
        },
 	  {
          "category" : "summonerEmotes",
          "content" : { "summonerEmoteIds": [0,0,0,0,0,0,0,0,0] }
        },
      ],
      "summoner" :
      {
        "level" : 30,
        "spells" : ["SummonerHeal", "SummonerHaste"]
      },
      "summonerId" : 1,
      "summonerName" : "Player1",
      "teamId" : 100,
      "teamParticipantId" : 0
    }
  ],
  "platformId" : "NA1",
  "mapId" : 21
  ...
}

This small but important change gives us a generic system that is feature-agnostic. We now define a runes content section to support Runes Reforged, and in the future we can define additional content types for new features without having to revisit the Megapacket generation system.

The Microservice

We’ve decided on the structure of how the Game Server receives Runes Reforged data via the Megapacket. But there’s still the question of how it receives that data. We decided previously that adding more code to the Platform monolith may put us in a bad spot down the road. Luckily, we have an entire microservices platform at our disposal, specifically designed for running online services. We just have to build a microservice to fill this role.

When dealing with a microservice-based implementation, we have to consider a few new details. First, we need to modify how we’ve orchestrated our communication path. The Platform monolith is still the League client’s first point of contact to any microservice, and it’s also ultimately responsible for getting the Megapacket to the GSM. For Runes Reforged, our microservice is now involved in updating the contents of the Megapacket, receiving a request from the League client, and forwarding the data to the Platform.

“Runes Talk”

Keeping the microservice functionally simple as possible is definitely in our favor. This means much lower development costs, a more easy-to-understand implementation for those maintaining it, and a considerably easier process to deploy and horizontally scale. The last point is especially worth noting; League is a game supporting millions of players, and new systems must have the flexibility to scale with them.

To that end, we make the microservice’s job simple: Validate the runes choices of a player and forward those validated choices to the Platform monolith for constructing the Megapacket. We explicitly avoid having the microservice maintain any state - such as storing pending selections during champ select - and try not to have it depend on other services aside from the Platform monolith.

Having a stateless service gets us a few free wins. Scaling to millions of players is as easy as spinning up more instances, and there’s no coordination involved with making sure you’re talking to the “right” instance of the microservice. This means any instance can handle your request, and recovery from a critical failure can be handled by a restart without needing to save and restore the state.

Going dependency-free from other microservices does leave us with an interesting problem - how do we validate whether or not a player owns a rune they’ve chosen? Involving the player’s inventory is certainly at odds with our microservice technical design goals. However, revisiting our product design goals presents us with a very bold opportunity: make runes free.

Having all runes available to all players gets us some wins across the board. Players get a full suite of runes from day 1 without having to grind IP to purchase them. The game designers are free to balance the power of runes under a stronger assumption that all players have equal access. On the implementation side, the microservice no longer needs access to a separate inventory datastore. That’s less work for us, less complicated service coordination, and less points of failure in the whole chain.

“Easy peasy.”

It’s a pretty scary picture, but what we finally have is an end-to-end picture of a player selecting runes and the Game Server receiving the selections to make those runes pop in-game. As we get deeper and deeper into the implementation and begin touching more systems, the requirement of knowing and understanding all the systems involved becomes a real challenge.

In this case, we have the unique opportunity to turn that challenge into direct player value - making new runes available to every player - with the side benefit of making our work of implementing it a little easier.

Runes Retrospected

Congrats! It was no easy task, but after a bit of effort and a brief 90s movie hard-work montage we’ve finally reached the present - runes have been reforged, and the original runes and masteries system is no more. Aside from giving players new pre-game toys to play with, we have plenty of other victories to celebrate.

We sanitized and improved our aging tech. The amount of tech debt in the code that drove runes and masteries was a huge deterrent from altering the system, leaving us stuck in a cramped game design space. Whereas previously we limited our changes to shuffling around some masteries and introducing keystones, now we have the flexibility for more dramatic system changes with much less effort.

We reduced the footprint of our Platform monolith. While the Platform monolith is still a critical part of getting players into game, every step towards reduction brings us closer to our goals of reaching a cleaner, more reasonably sized codebase.

And we introduced a new way to bring features into the game with a consistent, standardized system. A more straightforward way to develope new features means they’ll be in players’ hands sooner rather than later. Having a proven path for customization has been incredibly beneficial as the team explores different ways to bring fun new twists to the current experience. Emotes were our first test of the system (which technically launched before Runes Reforged!), and its success has paved the way for even more highly impactful customizations, such as the Augment system of the Odyssey event.

But was it worth?

This isn’t to say that Runes Reforged was all freelo. Every project has its trade-offs and this was certainly no exception.

We were stuck to the League client. We decided to pass the Air Client, and that meant we weren’t just binding ourselves to the League client release date, which was already becoming challenging to get out the door. We also had to make sure everyone had migrated over to it and Air Client was dead in the ground. Fun fact: We had hoped to get Runes Reforged out for preseason 2017, but the League client’s ship schedule didn’t give us much hope of landing on the tiny preseason runway. We instead used the opportunity of an additional season of development to make sure Runes Reforged was delivered the way we wanted.

We delivered free runes, but not for free. Deciding to make runes free for all players was fistbumps all around for design, but it certainly gave more than just engineering pause. There were lots of discussions between several teams - design folks, engineering folks, finance folks - about what free runes looked like to us and how we could deliver the best player value possible. In the end, we had to spin up an entire subproject dedicated to rewarding players who invested in the old runes economy. This meant collating the purchase history of millions of players worldwide, constructing a system that would deliver their rewards in time for preseason start, and footing the bill for the large hole old runes used to fill. That bill was a pretty substantial cost for the company to eat - easily four dollar signs on Yelp. But ultimately we decided it was the right decision for the right player experience.

We were playing service telephone. Introducing a new microservice is good for addressing the separation of concerns and decreasing code complexity, but it resulted in increased DevOps complexity. Deploying the game every patch has now added an additional microservice dependency to account for, and our core game loop now has a new potential point of failure. Microservice subject matter experts in addition to Platform subject matter experts are now necessary to involve in issues with game starts, of which there are fewer. It certainly makes juggling on-call rotations interesting.

And we often found ourselves in unfamiliar territory. The core Runes Reforged engineering team was just a small handful of LoL engineering folks, and as one Rioter once put it, we were “changing code that we had no business changing.” This was not your typical preseason update, and we were dipping into code in unfamiliar spaces, owned by completely different groups. Riot’s tech stack is a big place and this is a solid case where teaming up with many other teams, disciplines, and subject matter experts added overhead. However, teamwork here is certainly better than solo queuing.

Despite the challenges and costs, we’re in a happier tech state than we were before, we’ve got much more space for the game to grow, and we managed to give players the spiciest preseason in League history. Worth.