Hello! My name is Alex Sherrell, and I’m the quality owner for Teamfight Tactics. I’m responsible for making sure TFT reaches our quality standards, and I work closely with individual teams to establish quality bars, holding them accountable and helping smash bugs and improve the overall player experience. 

I’ve been part of TFT since early days - I was one of the original 6 members of the team, and it was the first time Riot had really attempted this kind of idea, where a side project became a full game on a short timeline. We needed to have excellent communication and trust in each other to make it work at a quality we were willing to ship. 

In this article, I’ll be describing how we handle patches across PC and mobile for TFT, and how this relates to quality assurance. Later, I’ll tag in my engineering counterpart, Gavin Jenkins, to give a super techy point of view on patches, and we’ll dive into two use cases that demonstrate different types of patches and how we deploy them. 

Welcome to the TFT edition of the Bug Blog!

Game Data and Patches

Server vs Client

To understand patches in TFT, we first have to explain how we handle game data and where it’s stored. 

There are two locations for information - server and client. Server information is shared with players from Riot’s servers. Client information means the player is getting new data to store on their client (a PC or mobile device).

Server changes are significantly easier for players because they don’t affect their machines directly. But client changes require data changing in the local environment. A player’s computer needs to download the graphics files for the sick new skins (client change) but it’s easy to adjust damage ratios (server change). 

Another important note - server data is stored in two places, Lua and our Game Data Server. Lua holds logic content, like how abilities happen and when to apply damage, while GDS holds raw numbers. 

To illustrate the difference between these types of information even further - art assets are stored on a player’s client, because otherwise downloading them during the game would be slow. And although a savvy user can manipulate their own client, changing client data like art assets shouldn’t negatively impact other players. 

On the other hand, game data isn’t manipulated by other players, so we store it on the server. 

The Different Kinds of Patches

Now that we have a general idea of the two kinds of changes - server and client - we can take a closer look at the patch options we have. 

Full patch: These are for client changes, the more extensive kind of download. We release at least one of these every 2 weeks to patch and balance the game. This is required for any kind of large change such as art assets or champion additions. 

Micropatch: These are for server changes, the easier kind of adjustment. These changes override existing data on the server. This is great for things like changing tooltips or any ability-related number (damage, health regen, mana, etc) which, as mentioned above, are based on data coming from the server.

Mobile Submissions: If you had asked me 3 years ago, “How hard could mobile deploys be?” I’d be like… “There are what, 10 types of phones out there? Those are the ones I’ve seen ads for, anyway.” In today’s reality, though, there are hundreds of phones with wildly ranging specs, which can make a huge difference when it comes to app submissions, which we do through third party stores, like the Google Store or the Apple Store. 

These have their own requirements and submission processes that we need to keep in mind while building out patches for mobile. And because TFT has cross-play, we need to consider how these different versions of applications all interact with each other, especially when we’re changing version or game data numbers with micropatches. Luckily, our mobile game does pull from the same servers as the PC versions, so micropatches - which are based on the server - do impact mobile as well. 

Ultimately, when we think about these kinds of deploys, we have to keep a global perspective to fully understand how we’re impacting our players. A full patch - which requires downloading new information - may be pretty simple for someone playing from the PC at home… but what about PC bangs? Or countries with slower download speeds? These kinds of considerations pile up, and heavily impact any bug-related decision making.

The Story Behind Micropatching

Fun fact - micropatches are actually a pretty recent feature that we’ve been able to leverage in the past, oh, four years or so. The ability to make small adjustments that affect any device with limited player impact - PC, Apple, Android, it doesn’t matter - is a total game-changer.

But micropatching in Summoner’s Rift can be pretty different from micropatching in TFT - a simple port of the systems in our initial builds highlighted the ways these game modes process data differently. 

To tell this story, I’d like to introduce you to Gavin Jenkins, TFT engineering manager extraordinaire. I’ll see you again in a few paragraphs for bug-catching time.

From SR To TFT

Hey everyone, I’m Gavin Jenkins, and I’m an engineering manager on TFT. The story of how we translated League micropatches to TFT patches is pretty interesting, from both a technical and social POV. 

So, remember, we can micropatch two things - Lua scripts and GDS data. Micropatches take that binary data and base 64 encode it, and create a long string which is put in a file on the server. When the game server runs, it loads that file, decodes the string, and checks to see if there’s a micropatch key for the property it’s loading. If so, it overrides the data from the original installation with the micropatched data. 

Ultimately a pretty straightforward override system ported directly from League. Or, it should have been, but here’s the catch - data within TFT interacts with other data in TFT in totally new ways. 

For example, a change to the carousel would impact multiple systems at the same time. When you upload micropatches for scripts, you have to upload the entire file, effectively locking out any other changes to that script file. Now consider the fact these systems were all originally created by a fleet of designers defining a web of interconnected game systems in script files to accomplish quick iteration to build out TFT. That means that unlike League, there are a large number of systems in script, so a change to the carousel - which would include multiple champions and items - ends up being a pretty huge set of script file changes that are now locked to any other changes. 

The carousel from the Fates set. That’s a lot of items and champs!

With this kind of system, you have to base 64-encode the entire file for micropatches. If you wanted to make a second micropatch that involves a change to that file, you must again base 64-encode the entire file. But which base 64-encoded representation of that Lua file do you want the game to use as the override? The answer is, well, neither. In order to get both fixes you must base 64-encode the version of the file that has both fixes in it, and have the game use that as the override.

In other words, if there are two competing micropatches affecting the same file… who wins? What order is it loaded in? What is overwriting what?

Building Awareness

This whole new world of data interactions meant we had to really investigate to figure out what was wrong. At first, it would look like new bugs would be fixed while also bringing back old bugs. We realized pretty quickly that we’d need to overhaul scripts by treating them more like how engineers structure code - using computer science principles like abstraction and encapsulation, data separation, and staying DRY, to optimize the code structure and how we handle scripts to prevent bug regressions. 

And equally important - we needed to socialize the problem among all designers and engineers on the team, because it would require a pretty drastic shift in how we saw patches and data structure in scripts.

This resulted in some pretty funny lunchtime conversations between TFT and League engineers, where TFT devs would be discussing the complexity and problem with micropatches, and League devs would be pretty confused, considering how simple and straightforward League micropatches are. 

Sustainable Structure

There’s a computer science principle called the separation of concerns which states the need to create distinct concern-based sections to avoid monolithic structural dilemmas. The well-defined core gameplay loop of SR means there’s already a pretty clear separation of concerns with champion scripting, which doesn’t exactly work for a system like the TFT carousel. So a critical step in our port of the micropatching tool included reframing what concerns we were basing our separation on. 

Once we reached this understanding, it was a pretty straightforward path to extracting bits that didn’t need to be monolithic and separating the Lua files for ease of micropatching. All of this led us to our current micropatching process, which lets us make informed decisions around different kinds of patches - and also improved maintainability of scripts and TFT game systems in general. 

Passing this back to you, Alex, to finish up with a deep dive into two bugs that demonstrate the power of patches… and the effectiveness of our micropatching tool.

Bug Catching

Alex here! Now that we’ve got an understanding of some of the backend considerations and a bit of a history of our micropatching tool, let’s see these solutions in action with two recent bugs.

Bug 1: Micropatching Mor-evil-lonomicon

Decimals are the bane of all existence. When Mor-evil-lonomicon was patched recently, the burn damage wasn’t correctly tracking the ticks per second. Instead of dealing 4% damage every 1.0 second, it tried to deal 4% damage every 100.0 seconds. Battles never last that long, so the damage was just never applied.

The bug as described to players.

The Solution

Remember earlier when we mentioned that game data like damage ratios are stored on the server? That means this bug is the perfect candidate for a micropatch. We used our sweet micropatching tool to quickly swap around decimals so the damage ratios would be correct. It only took about an hour for us to get that change going, so many players didn’t even encounter the issue.

Changing Things in Script

Back in the day, this kind of mistake would require a ticker message explaining the broken item until a fix could be deployed. With our micropatching tool, a fix is a matter of hours, not days. 

Bug 2: Mobile Art Assets

Now let’s take a look at something that can’t be micropatched, and some of the considerations we have to take into account for mobile deploys.

With the new radiant items in 11.15, the armory is getting restocked with new items to replace the shadow thematic from the first half of the set. An art asset change ended up being applied to both our live patch and a future patch. This meant a double overlay of items, with both the shadow and radiant art assets layered on top of each other on mobile apps.

Well, that’s not right.

The Solution

In this case, we do have to do a full patch because assets are stored on player clients and not in our servers. Because this issue only affected mobile, we only had to redeploy the patch for Apple and Android.

Game Versions with Mobile

The sneaky bit here is that we have to make sure that server and game clients can still communicate. In this case, mobile apps got a new version number, but PC didn’t. To ensure cross-play would still function, we did several tests to validate that the game clients could still communicate properly. By setting up a 1v1 in our testing environment, my testing partner and I were able to run a suite of tests to confirm cross-play was still possible.

Wrapping Up

TFT’s development timeline required extensive communication, validation, and a testing mindset across all teams. We continue to think about how we can improve the player experience, especially in ways that the game mode diverges from League loops.

Patching is something we take really seriously. It can make or break the game… literally. By focusing on customizing tools like the micropatch tool to increase usability and collaboration between designers, QA, engineering, and art, we can hit our quality standards without restricting creativity. 

Thanks for reading! Post any comments or questions below.