Debugging Titles: Part II

In my last article, we discussed a problem with titles: they poorly describe the many ways an engineer can develop their career. In response, we developed a system we call Masteries comprised of six attributes that, when combined, create a more well-rounded perspective on an engineer’s capabilities and growth path. In this second article, we’ll explore how we designed the levels and attributes, including the “minimum bar” we expect for engineers at Riot. We hope the results deliver compelling and aspirational paths in each mastery.

For some background in this conversation, consider two major challenges we encountered when constructing the leveling system. The first was determining what the levels measured in exact terms - our efforts to provide better career clarity would fail if a mastery is just as ambiguous as a title. The second was delivering a system that, in short, doesn’t feel bad. It was essential to avoid creating unhealthy comparisons, labeling, silos, and other unhealthy “management” behaviors. We felt strongly that this system could be effective, but we also had to combat the baggage that many of us (myself included) had from other, similar systems that were poorly used at companies we’d previously worked at. I’ll address both of these challenges by telling the story of the leveling system’s evolution, while talking about where we went wrong, and where we landed.

Like most ideas, we started by brainstorming and throwing some ideas on paper. We reached out to people across engineering who were known for being subject matter experts and gathered feedback on the attributes we thought defined our careers as engineers. As a reminder, those attributes are:

  • Programming

  • Subject matter expertise

  • People leadership

  • Craft leadership

  • Product sense

  • Delivery methods

Because of our affinity for games, and all of us being passionate League of Legends players, we playfully modeled the original version of the levels after ranked levels. We started at “Unranked” and included six levels, from Bronze to Diamond. It’s worth a brief sidebar here about the Unranked level. As I mentioned in the introduction, one of the goals of the levels was to define the minimum bar for being a Riot engineer. We knew that aspiring engineers might want to understand the difference between being at that bar and being below it, so we made sure to invest effort here. This has yielded significant benefits and clarity for us when we’re interviewing entry-level candidates or when transferring Rioters from other disciplines into Engineering.

Having the levels labeled, we then threw together a Google doc and shared it with everyone interested. We started adding bullet points to each level of the attributes, which created a giant 6x6 grid. We iterated until we thought we had a solid rough draft, and then shared the document with all of Engineering using our RFC process. We quickly learned through RFC feedback that we’d made some mistakes in the original design. I’ll list them here, and then touch on how we approached each one in turn:

  1. The use of Bronze through Diamond as labels felt awful to most engineers

  2. The intended distribution across levels wasn’t clear

  3. What the levels were measuring wasn’t consistent

  4. The bullets drove people to “check boxes” rather than being more subjective

  5. The time horizon for measuring yourself against them wasn’t defined

  6. The intended use of the whole system still wasn’t clear

Starting at the top: the most fascinating lesson-learned I personally took away from this project was in labeling the levels. While I love playing League, let’s just say I’m not turning pro anytime soon - I hang out mostly in Bronze knowing that the echelons of Platinum and Diamond are beyond my reach. As a result, I had little appreciation for the distress caused by thinking of your programming or leadership skills as “Bronze” or “Silver.” In fact, after several other attempts, including labeling them 0 - 6 and other such discrete hierarchies, we stumbled on the major issue: the comparative language. If you look at the six attributes carefully, you should arrive at the conclusion that no single person could possibly master all six. You simply can’t be the best people leader and programmer and subject matter expert, and so on. That meant you had to make choices about where to invest your personal growth time. If I decide to focus on leveling up my people leadership, to some extent I’m doing so at the expense of growing in technical leadership - and of course that’s totally fine. We needed labels that didn’t make people feel bad about those choices. We landed on a belt system that colored each level, with White being the lowest, then Yellow, Orange, Red, Brown, and Black. This resonated with most people and eliminated the concern about being “Bronze” or a “Zero.”

The second issue dealt with the expected distribution of Rioters across the masteries. Was a Black belt intended to be fairly easy to achieve, or was it highly aspirational? We wanted the levels to stand the test of time. If we had to redefine them every year, we’d lose the ability to track progression or measure anything meaningful - not to mention the amount of work that would go into such an effort. We also wanted to allow our engineers to aspire to an extremely high bar. Therefore, we decided that the Black belt should be highly aspirational, and we intentionally set each level such that the distribution among the belts would ideally be a bell curve (more on that in our next article). We essentially wanted most of our engineers to fall into the Orange/Red belt with few, if any (currently) Black belts.

The third and fourth issues were related, and they dealt with how we defined the contents of each level. As I mentioned, we originally put a list of bullets together based on the collective experience of our subject matter experts. But, some bullets talked about things you needed to know, while some talked about things you needed to be able to do. That created a feeling of inconsistency across the different attributes. What we really needed was to align on what we were measuring. At Riot, player value is paramount - everything we do is about generating value for players. Adding value as engineers on product teams needed to be the focal point of what we were measuring. Skills and knowledge are important, but much less so in a vacuum - they need to be applied to the efforts of a team. Ultimately, it’s the value an engineer adds that creates the opportunity for growth, taking on new challenges, and being entrusted with larger responsibilities. With this in mind, we had a new problem: how do we actually measure that value?

We consulted Riot’s research team and decided the best approach would be interviewing a large sample of our Engineering discipline and asking a series of questions to help determine what value looked like at a low, medium, and high benchmark. We selected 65 engineers across junior, mid, and senior ranks of each attribute and spent several weeks conducting hours and hours of interviews. We then analyzed the results, looked for themes in each attribute, and wrote a descriptive paragraph that we call the tl;dr. Here’s what they look like:

You’ll notice there are only three, but each mastery has six levels. This was a happy accident that helped address the fourth problem: box-checking. In our early tests, engineers wanted to treat each bullet as a check box and would add up how many bullets they had in each level to determine where they were. The problem here was that our lists of bullets weren’t exhaustive, and there was no way they ever could be. They were intended to be representative accomplishments that helped you make a determination. The tl;dr’s allowed you to narrow in on low, mid, high and then use the bullets as tie-breakers.

The fifth and sixth issues involved the system’s intended use. Specifically, what was the time horizon for “measuring” masteries, and how would that affect how we viewed the system? We first considered using the levels to measure all the skills and capabilities you’ve gained over your entire career. However, we felt this wasn’t useful if we wanted to really measure value and impact, and drive development conversations between engineers and their mentors. Having a list of all my capabilities might be useful in some contexts, but most of the conversations about performance or development had to do with what value you were adding in your current role, and how you develop the skills needed for the next role. As a result, we decided the system would only measure the value you actually demonstrated over roughly the last year.

That decision had one implication, which was the ability to “decay” in an attribute. As an example, let’s say you were an amazing programmer, so you were a Brown belt as a lead engineer on a feature team. Then, you spent two years in the Product Owner role on a new team. During that period, you spent most of your time setting product vision, and very little, if any, time actually programming. Using the mastery system, after your tenure as a Product Owner, your Programming mastery would likely be a Yellow belt and Product Sense would increase to something like Red or Brown depending on the scope of your role:

Keeping the system timeboxed allows us to look at the current state of our teams, and also track changes over time. This also keeps conversations focused on the value being added in the present.

With these questions answered, we had what we felt was a comprehensive set of attributes and levels. Now we had to apply it to our entire engineering discipline. In the next article, I’ll talk about how we rolled this out across our organization, had everyone in Engineering develop their personal masteries “profile,” what we learned, and how we plan to maintain the system for the future.

For more information, check out the rest of this series:

Part I: Titles 
Part II: Levels and Attributes (this article)
Part III: The Future of Masteries

Posted by Mike Seavers