Hey folks! We’re Mark Hillick, Jason Clark, and David Rook from the Riot Security team and we’re here to discuss how the security program at Riot has evolved over the last 4 years. This is our first article in a series of security articles so it’ll be more high-level, but keep an eye on the tech blog in the future for all the juicy techy bits.

In this article, we’ll talk about how our approach has involved listening to our fellow Rioters, leveraging Riot’s feedback culture to raise the security bar, and finally creating a set of customized security best practices. From introducing security standards to securing our offices and building tools that help protect our products, we’ll explore how Riot approaches security and share a few stories from the battlefield.

Challenges

Like many security teams, we were originally a team that said “no.” You know the type - the gatekeeper, the team in the corner. We were the security team with which no one wanted to engage (read The Phoenix Project for another example). Our processes often resulted in blocking the delivery of player value, which hurts for a company full of players with a mission to become the most player-focused game company in the world. As you can guess, our philosophy wasn’t exactly embraced by other teams within Riot and so teams often bypassed us because they didn’t see the value we could add.

In addition to cultural challenges, we struggled with firefighting - much of our time was consumed by incident response. We also experienced some very public security challenges such as losing data from our NA password database and Marc Merrill’s Twitter being hacked.

Reactive security sucks and we needed to move on from past security incidents. We wanted to support our Product and Engineering teams in building kick ass stuff for players, and that meant it was time to reevaluate how we worked. The traditional security gatekeeper role wouldn’t allow us to drive the necessary changes and build an effective security program at Riot; in fact, it would only further alienate us from the rest of Riot.

Our Approach

Our philosophical tenets can be broken down into three main strategies:

• Security should be a guardian of your company's culture. We are driven by that culture, not the drivers of it.
• Security teams should be feedback-driven and audience-focused. Your message has to resonate with the people it impacts, because if they don’t buy into your overarching goals, they won’t trust or cooperate with your processes.
• Security teams should provide options, not roadblocks.

Figuring out what good security would look like at Riot was the first step. We knew that we had to align with our culture to have our plans adopted. There were some critical factors we needed to consider:

• We analysed where and why we had failed in the past without falling into blamestorming or “but Riot’s different” arguments.
• What changes would have the biggest impact on players and Rioters?
• How would we engage the less security-minded Rioters without making them feel singled out?
• How much should we try to change? How much is too much?
• How could we scale security without hiring hundreds of new folks? Did we have teams that contained potential champions who would advocate for security?

It’s easy to make a list and say “go do that thing,” but adjusting our approach and strategies to be feedback-driven and audience-focused was not an overnight process. Our culture is based on putting players first, and trust is a huge part of that. We trust Rioters’ intentions and that allows them the autonomy and singular focus to build cool stuff for players (e.g. Runes Reforged, Riot Direct, Lag Report, and fun support videos). The trust to do what you do best with minimal interference is a huge reason people come to work at Riot.

But trust doesn’t come down from on high. If we want Rioters to trust us as a team, it means we need to go where they are, learn what they do, and speak to them in a language they understand. The days of being security oracles are dead and gone; we needed to become the newest member on dev teams, Riot Direct, compliance, the IT help desk, and anywhere else security should have a seat at the table.

Temporarily embedding dedicated security engineers onto a team produces some value, but it’s not a long-term solution. The embed model alone doesn’t scale and any knowledge gained on a project loses its relevance as team members shift. We needed a way to keep up with our products, which meant prioritising where to engage, finding the individual Rioters involved in relevant situations, and making it easy for them to communicate with us. We needed to learn more about how Rioters had been burnt by security in the past, so we asked. This is what we heard:

• Being told “no” for security reasons with no reasoning or context
• Receiving conflicting guidance
• Feeling disengaged from the Security team

Continuous improvement is a top-level value in Riot Engineering, so our fellow engineers weren’t shy about telling us what they wanted. We needed to stop trying to force solutions from a security point of view and pay attention to what Rioters actually needed. During our one-on-one meetings, we started asking a simple question:

“When you’re doing your job, what do you wish you had from security?”

These answers would drive how the Security team engaged with engineers and non-engineers across Riot for the next several years. Their answers gave us a bit of a reality check. The list of improvement points was beginning to take shape.

• Stop using security jargon
• Don’t roll out platform-wide password changes without notifying us or giving us context
• Provide us with advice and guidelines instead of just saying “no”
• Offer solutions instead of listing problems
• Build relationships so that you understand challenges from the product perspective

We had our work cut out for us and no shortage of places to start. Our conversations had armed us with a timely list of requirements and scalable tools we hadn’t considered. We weren’t limited to just Wireshark, Metasploit, Nmap, or Burp. By giving developers the capabilities we needed them to have and opening the lines of communication, we could make real progress and become a first-class citizen at Riot. All we had to do was ask.

Breaking out of our Echo Chamber

The Security industry isn’t known for being great at communication with folks outside the industry. We tend to blame others for being insecure or tell people how cool we are because we smashed the stack. We’re usually regarded as the team in the corner that comes in to clean up the mess. Pivoting to focus on good communication meant more than just a slogan on a poster. For our AppSec team, it meant understanding how to actually achieve our stated goal:

“To arm every software engineer with the tools and knowledge they need to build secure products for players and Rioters.”

In other words, our AppSec team should make it easy for engineers to write code that’s secure from the beginning. That doesn’t have to mean mandatory tools and training. If we’d followed the traditional approach, we would have implemented mandatory code reviews, awareness training, and static analysis, but we did none of that. Instead, we grabbed any engineer that wasn’t fast enough to run away and sat them down for a conversation, we attended other teams’ stand-ups, and we ultimately reconfigured our roadmap around the needs of the organization. In our one-on-ones, we consistently asked questions like: “Do you like what we’re doing? What else can we provide?” Additionally, we gave teams visibility into things like our security vulnerability tracking and our bug bounty program so they could see the impact of their own efforts, whether positive or negative.

After a year of soliciting feedback and iterating on a roadmap that would add value for teams around Riot, we delivered Riot’s first public security talk which goes into more detail about the early efforts. You can see where those early roadmaps ended up taking us by checking out this talk from earlier this year. We have also presented externally on our approach to network security and how security leverages our feedback-driven culture at BruCon and Security Fest. In summary, our engineers didn’t want heavy technical lifting or to hear about zero-days. They wanted us to keep up with them as we reviewed their code or products, provide data on how their code fits into Riot’s security goals, and write tools that made their lives easier. And they learned to love checklists.

The culmination of our research and collaboration went into our RFC library (similar to the IETF internet standards), described in more detail here by our colleague Cam Dunn. This is a tech design process where we discuss tech challenges, share knowledge, agree on approaches, and ultimately come up with standards through our adoption process.

Putting it All Together with RFCs

RFCs were a major factor in changing and guiding the security outlook at Riot. Through RFCs, we can gain bottom-up alignment with other teams, solicit feedback, and ensure that our solutions resonate with the global Riot Engineering audience. Three RFCs in particular have made a significant impact on the evolution of security at Riot and the Security team itself.

RFC0026: AWS Security

Amazon Web Services (AWS) is an easy and low-friction way to build a product and deliver player value quickly. But this same lack of friction typically results in sleepless nights for the Security team. There were many rules we could have mandated and ways we could’ve forced teams to change, but we didn’t want to cause decision paralysis or block their ability to deliver new player features. As a result, we focused our efforts on ownership of EC2 instances as our first major AWS security challenge to tackle. Ownership attribution greatly facilitates our security incident response and ensures that every product, system, and service is actively used, which the Finance team also appreciates.

Using the tagging feature in AWS, we had a solution that would provide clear ownership of infrastructure and products within AWS. To build alignment and get adoption of tags, we ended up with two RFCs, one covering the philosophical “what and why” of tags (RFC0026) and the second describing the implementation details (RFC0026a). In order to implement the philosophies of ownership attribution, we wrote and implemented a tool to automate its enforcement. We went through some trials and tribulations on the first implementation that led to degraded service for production features in League.

Inevitably, this resulted in some player pain and considerable feedback from our colleagues. We listened to this feedback and performed a transparent Root Cause Analysis (RCA) where we explored where we failed and how we could improve. We iterated on our solution, over-indexed on our communication strategies, and re-implemented the tool with much-improved alignment and visibility. We now had a solution that met the goal of “know what you have” and also fit our goals of being player focused and moving quickly.

The tool is written in a modular framework style using Python on Flask and an AngularJS front-end with a MySQL back-end. It runs within a security AWS account, using AssumeRole access to read the configuration of all objects within an AWS account. Automated emails notify AWS account owners and, where relevant, instance owners of a lack of compliance and when their instance will be shut down. After repeated notification over an agreed timeframe, the tool shuts down and eventually terminates any instance of a service in AWS that isn’t tagged properly. It also detects DNS misconfiguration that causes sub-domain hijacking, which you can learn more about here and here.

The tool has been so successful that we’ve received requests from engineers on other teams to include checks for EBS volumes. We also have both RFCs up for readoption to enforce tagging across all taggable AWS objects. It’s important to note that, like our technical changes, these philosophical and cultural changes have been staggered. You can’t change everything at once, regardless of how great your ideas are. If you’d like to learn more about our AWS journey, please check out our talk at re:Invent 2017, where we announced that we open-sourced this tool, called Cloud Inquisitor.

RFC0242: Office Security

An often overlooked aspect of security is making sure you have the same capabilities everywhere you play. Our ability to respond should be the same whether it’s in a data center or a small office halfway across the world. We laid out what those capabilities should look like in RFC0242, which describes how to build a defensible office that enables Rioters to deliver awesome player experiences in a secure and scalable fashion. We assume every network is compromised, and while prevention is important, visibility, detection, and containment are critical for success.

The original RFC itself was controversial. We spent a fair amount of time taking feedback, making tweaks and, perhaps most importantly, being radically transparent with Rioters. We took great pains to never come back with answers like “trust me” or “because hackers” when the inevitable questions around Big Brother popped up. By the time the RFC was adopted at the top-level scope (all of Riot), there were six times as many lines of comments as lines of content. While we may have only kept 70-80% of our original RFC, we had engaged the hearts and minds of our fellow Rioters and had company-wide alignment.

Of course none of this happened in a vacuum. We already had a sprawling deployment of endpoint tools, millions of dollars in blinky boxes and renewals, and all the trappings of a vendor brochure-driven security program. Operating with those tools inside of Riot’s creative and curiosity-driven environment was possible, but not very effective. While those tools gave us some of the capabilities we needed, we spent a good portion of every incident wishing we had better options. It turns out that having the wrong tools can be an asset - you’re forced to find or build the right ones.

Looking critically at our security stack, we learned a few things.

Capabilities should reflect your threats.

If you’re spending 90% of your time dealing with phishing, the focus shouldn’t be on network forensic gear - even if ripping packets apart is cool.

Standardize your logs.
This was an immutable requirement for any vendor we talked to. If they couldn’t give us standard log formats (syslog, json, etc) they were immediately off the list.

You are BYOD whether you choose to believe it or not.

Work increasingly happens off-network and we need to be flexible so Rioters can securely work from anywhere.

Evangelising

The three of us are engineers and, as many of you know, it’s easy to get caught up in all the cool tech. Early on we realized that to be truly successful we needed to put the cool security stuff aside and get non-security folks to buy into our goals. We can’t sit with every product team at Riot, so it’s extremely valuable to have Rioters who are willing to advocate for good security practices within their teams. We have a variety of strategies to help us accomplish this. From information-packed Secure Code cheat sheets to sweet t-shirts, we decided to apply our focus on communication to the real world. We summoned our artistic side and created this cool content to help engage our fellow Rioters.

Secure Code Cards

Thanks to the feedback from our engineers, one of our first products was a “Definition of Done” checklist on a postcard. This was our response to the request for “advice and guidelines” - putting a checklist on the desk of each engineer put security best practices literally within reach.

We (okay, someone more artistic than us) created a postcard with the secure code basics our engineers had asked for including links for further reading. We have Heimerdinger in the background because he’s our Riot Engineering culture icon and has the best hair on the Rift (though Draven may disagree). We sent these cards out globally and received tons of positive feedback. Three years later, we still see these cards on desks.

T-Shirts

While the cards were meant to go to anyone and everyone, we wanted to create something special for Rioters that went above and beyond, “security champions” if you will. Gnar was our champion of choice because we were looking for folks that had a greater impact than they might have anticipated. T-shirts might not work in your company - and, to be honest, it’s hard to measure their success in terms of overall improving sentiment towards the Security team - but we feel that they’ve helped increase positive engagement with our team and security in general.

The existence of a security champion is clearly evident when a team asks for their products to be reviewed and we only find one or two minor issues. We love it when Rioters answer questions on security mailing lists or in chat rooms before a member of the Security team can. It’s especially encouraging when security champions come to us and ask for replacement t-shirts when theirs get old.

What’s next?

When we look back at the last few years, we’re very happy with what we’ve done and the direction we’re heading. We still have a significant amount of work ahead of us as Riot is constantly changing, but we’ve set a tone that has allowed security to become an integral part of our culture and our teams.

Looking forward, we’ve created a new RFC (RFC0650) to help engineers, product owners, and development managers understand how to best secure their products.

It outlines traditional guidelines like:

• Restrict access to what’s absolutely required
• Store a minimum amount of data
• Patch consistently
• Leverage existing infrastructure and solutions for logging, authentication, and authorisation

As well as some less traditional guidelines, like:

• Engage us - we want to help and like to talk. Honestly, we don’t bite.

This may seem pretty simple, but 4 years ago that advice would’ve fallen on deaf ears. We were gatekeepers with a reputation for saying “no.” By building relationships across Riot, we’ve gained that trust back and made security a fundamental part of our culture, allowing us to adopt more disruptive tech and tools at Riot. We work in an environment of understanding and cooperation rather than begrudging acceptance.

Where do we go from here?

Over the next year, we’ll continue supporting our product teams in delivering awesome player experiences while further leveling up our security efforts at Riot. We now have a solid platform to roll out ownership attribution across all AWS objects. We’re looking at how to scale out our office security tech. We’ll provide teams and engineers with more visibility into the security posture of their code and arm them with the knowledge they need to create secure products and services for our players.

It’s been a long journey but watching the security team grow and evolve into a crucial part of Riot’s culture has been something we’re hugely proud of. That said, we’re not finished and we’ve got an exciting roadmap ahead of us.

We plan on diving into some of the stories that we’ve introduced here in future articles. Thanks for reading and please let us know your thoughts and questions.