The Riot Games API: Transforms
Hello all, Leigh Estes, aka RiotSchmick, here. I’m a software engineer at Riot Games working on the Riot Developer Experience team. Our responsibilities include providing the edge infrastructure that supports both internal and external developers. I previously wrote a series on the infrastructure that supports our public API product. I’m excited to revisit this series to tell you more about a new part of our infrastructure - the feature we call transforms. I’ll outline the reasons that we felt transforms were a valuable feature to invest in and how we implemented them. I’ll also dive into specific use cases for transforms that have added significant value.
Building a Scalable API
As mentioned in our previous articles, one of the original reasons we decided to build a public API was to prevent scraping of platform services. We needed to get it up and running as quickly as possible, because the scraping of the platform was putting a heavy load on the platform servers. That additional load was affecting players’ ability to play games in certain regions.
To build a public API product that external parties would use in lieu of scraping, we had to pull data from various unrelated services in order to provide the same data that players could scrape themselves. We found that all of these unrelated services used different conventions for request parameters, response data, paths, etc. For example, across these services, we found the same concept referred to variably as “platform,” “shard,” “platformId,” “location,” and “region.” In addition, some services used REST conventions, some used RPC conventions, and some used a blend of the two.
We wanted our API to be a best-in-class product, which meant it was important to provide a consistent, easy to understand, and easy to use experience for our third-party developers. We needed to make the public API product a cohesive experience that used the same nomenclature everywhere and followed REST best practices. That meant we had to find a way to translate all of these different conventions into a single consistent one.
Our First Solution: Shims
In the interest of speed, we ham-fisted a solution together that involved writing “shim” services that sat between the public API proxy and the underlying services. Most of the shims hit the underlying service, did some restructuring and renaming on the response to make it consistent with our conventions, cached that result, and then returned it to the proxy. By the time we had our fourth shim written and operational, we realized that this solution was not going to scale.
The Problems
We would have to update and redeploy each shim every time the underlying service changed something. Sometimes we wouldn’t find out about those changes until after something was broken, and we were hearing about it from our third-party community users! In an ideal world, we would always be the first to know when there is an issue, long before consumers of our product. In addition, these shim layers meant there was one more dependency and hop in the flow for each API call that could contribute to downtime and latency. We realized that we were going to have to find a more sustainable and scalable solution than adding a shim for each new underlying service that we wanted to add to the public API product.
Getting every service-owning team at Riot to adopt the conventions we wanted to use for the public product was not particularly feasible either, as it would be a constant battle of education. We would have to spend a lot of time and resources educating teams before they ever built their services, as most teams didn’t have the time to go back and change them after they were already built. Alternately, we could have enforced that no service could be put into the public API unless it already met our conventions, but that would run the risk of excluding valuable data and also seemed overly punitive.
Gaining Riot-level adoption would also mean a lot of back and forth with teams that had already completed and deployed a service and wanted to add it to the API after the fact. These issues seemed like huge hurdles for adoption, as well as a lot of overhead for the API team. Thus, we hunkered down to come up with a technical solution for deprecating the shims, rather than a procedural or cultural one. We knew a good solution would fulfill the needs of both our third-party community and our internal customers. Transforms fit the bill for all of these requirements.
Our Second Solution: Transforms
Since all of the endpoints in our public API product return JSON, it occurred to us that we could use JavaScript to easily make changes to response objects in a generic way. We did some prototyping to test that concept, and we found that it worked quite well. These results showed us that we could use the solution of executing JavaScript to transform responses in the edge proxy itself to replace some of the functionality of the shims.
Once we had this solution implemented to transform response objects, we realized it could easily be expanded. We could use this same solution to replace other shim functionality, including transforming URIs and request parameters. Once fully implemented, transforms would allow us to move any logic that the shims were executing for a given endpoint into JavaScript configured for that endpoint and executed in the proxy. In other words, we could ultimately deprecate the shims altogether.
Transforms: The Details
We already had a robust configuration engine for each individual API endpoint, so adding the ability to define and execute JavaScript on a per endpoint basis was fairly trivial. We did some research and discovered Nashorn, a JavaScript engine that can execute JavaScript code within a JVM. We set about adding the ability to define transforms in JavaScript for any endpoint.
Transforms Use Cases
Transforms allowed us to modify any request parameters in the cookies, headers, body, or query string, the request path, or the response body.
Some examples of how transforms are used in the public API product today include:
-
Renaming fields in the response body to be consistent. For example, if one service refers to `platformId` in its response and another refers to `shard`, we can rename them both to `platform` to be consistent before sending the response to the caller.
-
Removing fields from the response body that we don’t want to expose before sending the response to the caller.
-
Reworking fields that may be overly complex before sending the response to the caller. For example, a response body may include an enumeration that has only two values - `GAME_MODE_ON` or `GAME_MODE_OFF` - and we could convert that to a boolean `gameModeOn` with a value of `true` or `false`.
-
Extracting values from the RSO token in an incoming request and adding them as headers in the request to the underlying service.
-
Converting the path convention used by the public API to the path expected by the underlying service.
-
Moving parameters from one place to another before sending the request to the underlying service. For example, moving a parameter from the query string to a header.
-
Changing the format of the entire response to match the conventions used by the public API. For example, the public API convention if there is no data matching the request is to return a 404 with a standard error response body. However, some services return a 204 with an empty response body instead. We use transforms to convert the 204/empty body to a 404/error body before sending the response to the caller.
Benefits of Transforms
The previous section covers just a few examples of the common use cases we have run into. Since we added the transforms feature to our public API product, we have been able to entirely deprecate the shims we used to operate, with no visible changes in functionality for the end users of our product. In addition, we have been able to solve numerous new use cases that have come up since we added the feature that we hadn’t even anticipated, which are described in the next few sections.
Handling Feature Requests
We’re able to use transforms to solve many of the feature requests we get from internal service teams. For example, extracting values from the RSO token and passing them along to the underlying services was not one of our original use cases, but a service team asked if we could do that for them, and transforms made it easy to do.
Before transforms existed, we generally had to push back on these kinds of requests because it would have meant adding custom logic for every service into our edge code. We would have had to implement code that said if the requesting path equals some value, then look for an RSO token, and, if it exists, extract some hard-coded values from it and put them in some hard-coded header names. Over time, our edge proxy’s code would have become littered with these special cases, making it prone to errors and hard to debug. With transforms, all this custom logic can be isolated as configuration tied to specific endpoints, keeping it out of the generic edge code, and isolating its execution to only those requests to the given endpoint.
Integrating Services
If a service owner comes to us with a completed service, and says that they want to integrate it into the API product, we can quickly and easily do so. We don’t have to write a shim, ask them to change their service to conform to our standards, or compromise those standards. We simply throw a transform on the endpoints to convert the service to the API product’s standards, which means the time-to-market for adding a new service can be minutes instead of weeks or months.
Fixing Issues Quickly
Since the logic is in configuration, changes can be made in real-time if issues are found or changes to the underlying service are made. We don’t have to redeploy the edge proxy or anything else that the API team owns.
Assigning Ownership
We assign ownership of the configuration for each endpoint to the service-owning team, rather than having the API team own the transform logic for every service, as was the case with the shims. This solution, then, is scalable and maintainable for the API team.
Implementation
For our implementation, we decided to create two new objects - core transforms and transforms. There is a `type` field on both objects that represents the three areas where we execute transforms - the path, the request parameters, and the response. There is also a `script` field that contains the JavaScript. For each of the three types, there is a single core transform that contains all the shared code needed to support associated transforms. We wanted core transforms to be a separate object simply so we could enforce only one of each type existing. Having a single core transform that contains all the shared supporting code not only reduces code duplication, but also allows us to ensure that we are enforcing specific call patterns in each transform execution to limit the possibility and scope of errors.
The specific transforms for a given set of data are captured in the transform objects. When a transform of a given type executes inside the proxy, its JavaScript is embedded inside the JavaScript from the core transform of the same type, which together define valid executable code.
Deterministic Ordering
We decided that we should allow any number of transform objects to be attached to each endpoint using deterministic ordering, which provided a few benefits.
-
Debugging a specific transform is much simpler. We use separation of concerns on the transforms, keeping each JavaScript generic and self-contained, doing only one job (e.g., rename `shardId` to `platform`).
-
Debugging the execution chain of a specific endpoint is simpler and code duplication is reduced. We attach multiple generic transforms to a given endpoint, rather than having to write one giant transform for each endpoint that covered everything it needed. This strategy reduces code duplication because a single transform that does X (e.g., renames `shardId` to `platform`) can be attached to every endpoint that needs it.
-
Chaining transforms allows for complex logic. We chain transforms in a specific order, letting the logic of a previous transform play into the logic of a later one (e.g., transform 1 decrypts a parameter, transform 2 puts the value of the parameter into the path).
Transforms Example
Let’s take a look at an example that highlights some of the varied use cases that transforms can serve. It will also show how the core transforms define shared functionality, inside which the chained transforms for an endpoint can be executed.
For this example, we’ll examine the getLeagueEntries endpoint in the public API, which has the following URI.
https://<shard>.api.riotgames.com/lol/league/v4/entries/by-summoner/{encryptedSummonerId}. Transform #1: Consistent Casing
The first transform that executes for this endpoint is a request transform. Its purpose is to ensure that the casing of the shard value passed in matches what is expected by the service.
variables.shard[0] = variables.shard[0].toUpperCase();This transform upper cases the shard, which was specified by the requester in the host portion of the URI. The underlying leagues service expects its shard parameters to be upper case, and will not return proper data if they are lower case.
We don’t expect the users of the API to know which services want upper case and which want lower case. We enforce that everything is lower case on the edge, and then use transforms to convert when needed.
Transform #2: Deobfuscate IDs
The next transform that executes for this endpoint is another request transform. The purpose of this request transform is to deobfuscate IDs before they are sent to the service, as the service expects the raw ID values.
try {
if (variables.encryptedSummonerId) {
variables.summonerId = [decrypt(salts.summonerId, variables.encryptedSummonerId[0])];
}
if (variables.encryptedAccountId) {
variables.accountId = [decrypt(salts.accountId, variables.encryptedAccountId[0])];
}
if (variables.encryptedPUUID) {
variables.puuid = [decrypt(salts.puuid, variables.encryptedPUUID[0])];
}
} catch (err) {
if (!(err instanceof com.riotgames.zuul.exception.CryptographicException)) {
throw err;
}
var result = {};
result.status = {};
result.status.message = "Bad Request - " + err.getMessage();
result.status.status_code = 400;
response = {
'httpStatusCode': 400,
'result': JSON.stringify(result),
'headers': {
"Content-Type": [
"application/json;charset=utf-8"
]
}
};
}Privacy law requires that we individually encrypt any relevant IDs. We made the decision to handle that at the edge. This way, individual service owners across Riot won’t have to implement their own solutions, which would result in potentially different implementations and redundant effort. We only encrypt these values when exposing them to third parties, which means that it’s a lot of overhead for individual service owners to implement solutions when their service may mainly be consumed internally.
Transforms provided us with an easy solution to encrypt all relevant values in responses and decrypt all relevant request parameters in the edge, without service owners having to care or know about it. The transform shown above handles the latter. Note that the decrypt() method shown here is defined in the core transform for requests.
Transform #3: Filtering Out Internal Data
The next transform executed for this endpoint is a response transform. Its purpose is to filter out data that would be irrelevant or inappropriate for third-party use.
if (httpStatusCode == 200) {
result = JSON.parse(result);
var response = [];
var positions;
if ("summonerLeagues" in result) {
positions = result['summonerLeagues'];
}
if (positions) {
for (var i = 0; i < positions.length; i++) {
var position = {};
position.leagueId = positions[i].leagueId;
position.queueType = positions[i].queueType;
position.tier = positions[i].tier;
position.rank = positions[i].rank;
position.summonerId = positions[i].playerOrTeamId;
position.summonerName = positions[i].playerOrTeamName;
position.leaguePoints = positions[i].leaguePoints;
position.wins = positions[i].wins;
position.losses = positions[i].losses;
position.veteran = positions[i].isVeteran;
position.inactive = positions[i].isInactive;
position.freshBlood = positions[i].isFreshBlood;
position.hotStreak = positions[i].isHotStreak;
if (positions[i].miniSeries) {
position.miniSeries = {};
position.miniSeries.target = positions[i].miniSeries.target;
position.miniSeries.wins = positions[i].miniSeries.wins;
position.miniSeries.losses = positions[i].miniSeries.losses;
position.miniSeries.progress = positions[i].miniSeries.progress;
}
response[i] = position;
}
}
result = JSON.stringify(response);
} else if (httpStatusCode == 404) {
result = '[]';
httpStatusCode = 200;
}The leagues service returns some parameters that are only intended for internal consumption. This transform constructs a new response that contains only the fields that are intended for external consumption.
Transform #4: Encrypting Summoner IDs
The last transform that runs for this endpoint is another response transform. Its purpose is to encrypt summoner IDs that were returned in the service response before passing the response on to the requester.
if (httpStatusCode == 200) {
result = JSON.parse(result);
var positions = result;
for (var i = 0; i < positions.length; i++) {
var position = positions[i];
position.summonerId = encrypt(salts.summonerId, position.summonerId);
}
result = JSON.stringify(positions);
} else if (httpStatusCode == 404) {
result = '[]';
httpStatusCode = 200;
}The encrypt() method shown here is defined in the response core transform. To fully illustrate how this works, here are the full request and response transforms for this endpoint, with all of the transforms of each type chained together and injected into the core transform JavaScript for the given type.
Request Transform
function transform(input) {
var json = JSON.parse(input);
var variables = json.variables;
var response = json.response;
input = {
'variables': variables,
'response': response
};
var obfuscationKey = json.obfuscationKey;
var encrypter = function(salt, message) {
return encrypt(obfuscationKey, salt, message);
};
var decrypter = function(salt, message) {
return decrypt(obfuscationKey, salt, message);
};
var transformedVariables = customTransform(input, encrypter, decrypter);
json.variables = transformedVariables.variables;
json.response = transformedVariables.response;
return JSON.stringify(json);
}
function customTransform(input, encrypt, decrypt) {
var variables = input.variables;
var response = input.response;
var salts = getSalts();
variables.shard[0] = variables.shard[0].toUpperCase();
try {
if (variables.encryptedSummonerId) {
variables.summonerId = [decrypt(salts.summonerId, variables.encryptedSummonerId[0])];
}
if (variables.encryptedAccountId) {
variables.accountId = [decrypt(salts.accountId, variables.encryptedAccountId[0])];
}
if (variables.encryptedPUUID) {
variables.puuid = [decrypt(salts.puuid, variables.encryptedPUUID[0])];
}
} catch (err) {
if (!(err instanceof CryptographicException)) {
throw err;
}
var result = {};
result.status = {};
result.status.message = "Bad Request - " + err.getMessage();
result.status.status_code = 400;
response = {
'httpStatusCode': 400,
'result': JSON.stringify(result),
'headers': {
"Content-Type": [
"application/json;charset=utf-8"
]
}
};
}
input.variables = variables;
input.response = response;
return input;
}
function encrypt(key, salt, message) {
return CryptoUtils.encrypt(key, salt, message)
}
function decrypt(key, salt, message) {
return CryptoUtils.decryptAndValidateSalt(key, salt, message);
}
function getSalts() {
var salts = {};
// salts are defined here, but obfuscated for security
Object.freeze(salts);
return salts;
}Response Transform
function transform(input) {
input = JSON.parse(input);
var customTransformInput = {};
customTransformInput.result = input.responseTransform.result;
customTransformInput.httpStatusCode = input.responseTransform.httpStatusCode;
customTransformInput.headers = input.responseTransform.headers;
customTransformInput.uriTransform = input.uriTransform;
customTransformInput.variables = input.variables || {};
var obfuscationKey = input.obfuscationKey;
var encrypter = function(salt, message) {
return encrypt(obfuscationKey, salt, message);
};
var decrypter = function(salt, message) {
return decrypt(obfuscationKey, salt, message);
};
var transformed = customTransform(customTransformInput, encrypter, decrypter);
input.responseTransform.httpStatusCode = transformed.httpStatusCode;
input.responseTransform.headers = transformed.headers;
input.responseTransform.result = transformed.result;
return JSON.stringify(input);
}
function customTransform(input, encrypt, decrypt) {
var result = input.result;
var httpStatusCode = input.httpStatusCode;
var headers = input.headers;
var uriTransform = input.uriTransform;
var variables = input.variables;
var salts = getSalts();
if (httpStatusCode == 200) {
result = JSON.parse(result);
var response = [];
var positions;
if ("summonerLeagues" in result) {
positions = result['summonerLeagues'];
}
if (positions) {
for (var i = 0; i < positions.length; i++) {
var position = {};
position.leagueId = positions[i].leagueId;
position.queueType = positions[i].queueType;
position.tier = positions[i].tier;
position.rank = positions[i].rank;
position.summonerId = positions[i].playerOrTeamId;
position.summonerName = positions[i].playerOrTeamName;
position.leaguePoints = positions[i].leaguePoints;
position.wins = positions[i].wins;
position.losses = positions[i].losses;
position.veteran = positions[i].isVeteran;
position.inactive = positions[i].isInactive;
position.freshBlood = positions[i].isFreshBlood;
position.hotStreak = positions[i].isHotStreak;
if (positions[i].miniSeries) {
position.miniSeries = {};
position.miniSeries.target = positions[i].miniSeries.target;
position.miniSeries.wins = positions[i].miniSeries.wins;
position.miniSeries.losses = positions[i].miniSeries.losses;
position.miniSeries.progress = positions[i].miniSeries.progress;
}
response[i] = position;
}
}
result = JSON.stringify(response);
} else if (httpStatusCode == 404) {
result = '[]';
httpStatusCode = 200;
}
// ---- core code from Zuul transform filter
input.httpStatusCode = httpStatusCode;
input.headers = headers;
input.result = result;
return input;
}
function encrypt(key, salt, message) {
return com.riotgames.zuul.util.CryptoUtils.encrypt(key, salt, message);
}
function decrypt(key, salt, message) {
return CryptoUtils.decryptAndValidateSalt(key, salt, message);
}
function getSalts() {
var salts = {};
// salts are defined here, but obfuscated for security
Object.freeze(salts);
return salts;
}The Unexpected Power of Transforms
As mentioned in the previous section, after implementing transforms, we were delighted to realize that it solved even more problems than we had originally envisioned. As we began to explore all the features that transforms unlocked for us, we had a realization. Through the use of transforms, we could theoretically implement a “serverless” API.
Serverless APIs
A serverless API is an API that takes requests and returns responses, all without any actual underlying service to process requests. Transforms can inspect all parts of the request and can generate a complete response body regardless of what the service sent back. So why couldn’t we take a request and generate a response body without ever making a call to a service?
It turns out, we could!
The API April Fools Event
The April Fools event is one example of the value that serverless APIs can provide thanks to their ease of implementation and low maintenance cost.
We knew theoretically that this type of serverless configuration was a possibility, but we never had an occasion to use it until April Fools of this year. Our 2019 API April Fools event was an escape room. We realized that this was the opportunity we had been waiting for to write a serverless API.
All we needed was to have the webpage post whatever the user typed to an API, parse the input in a big if-statement, and return relevant responses based on what the user input. Based on the user’s input, we could return valid instructions for the escape room, helpful responses, and some special interactions with certain incorrect inputs. We were able to add variety to the special interactions by making arrays of responses that mapped to each input and randomly picking one of the entries.
Easter Egg Extras
We included some easter egg inputs, which we called “Extras,” and which, when entered, resulted in the escape room printing out an ASCII art easter egg. The easter egg inputs mostly made references to pop culture, our favorite video games, and memes from our third-party API community.
Transforming April Fools
The entire experience was easily implementable in JavaScript and there was no real reason that we needed to write, deploy, and operate a service to support the experience. Implementing it as a serverless API meant that we could update the JavaScript during the event to address issues and feedback that came up, or to add/modify responses.
The API team was on a really tight deadline around the time that the idea for the April Fools event came about. If we had needed to write, deploy, and maintain a service to support it, we would have had to pass on doing the event. Missing out on the opportunity to connect with our third-party developer community and give them a fun experience would’ve been a real disappointment to both the team and the community. We’ve done an April Fools event for several years in a row. We didn’t want to have to skip providing that fun for the community this year, or miss out on the great experience the team has interacting with the community through the event.
Due to the power of the transforms feature, we were able to complete the entire implementation in less than a day, keep it updated during the event, and provide a great experience for the team and the community. We were also able to prove that serverless APIs are possible and tenable - all thanks to transforms.
Third-Party Community Response
The 2019 April Fools event was a huge success, with a great response from the third-party API community. Engagement with the event was high, with the channel for the event in our API Discord server seeing constant activity throughout the day.
There were several requests made by our third-party API community around the event. They requested that we write up all the interactions and easter eggs after the event because they wanted to know what they were missing. They also requested that we write a tech blog post explaining how it was implemented, which was the impetus for this very article.
Looking Back
We initially added the transforms feature so that we could deprecate shims in favor of a more sustainable solution without giving up the ability to provide a consistent API to third-party developers. We also wanted to make sure that the solution wouldn’t require service- or endpoint-specific logic living in our generic edge proxy code. Transforms not only hit upon these points, but have provided value above and beyond what we initially envisioned.
We created such a generic, robust, and extensible system with transforms that we have been able to easily and quickly address most new service-owner requests using this system. Transforms have even allowed us to implement APIs that live entirely in the edge proxy and don’t require requests to be sent to a service.
Transforms have exceeded all of our expectations. Today they stand as one of the most powerful features our edge layer provides to both service-owner teams as well as customers that make requests via the edge. They are adding value, saving time, and reducing operational and adoption burden for teams across the company.
Thanks for reading! If you have any comments or questions please post them below.
For more information, check out the rest of this series:
Part I: Goals and Design
Part II: Deep Dive
Part III: Fulfilling Zuul’s Destiny
Part IV: Transforms (this article)