Hello!

I’m Byron Dover, engineering manager for information technology at Riot, and I lead the team responsible for developing enterprise software at Riot - or as we sometimes call it, Riot’s Operating System. I’m excited to share a look at how Riot integrates with Slack to support the game development lifecycle.

I recently spoke at the Slack Frontiers conference on this topic - the video is available on YouTube.

IN THE BEGINNING

Slack adoption at Riot goes back several years, and started with small pockets of organic usage by Rioters in 2014. Interest quickly grew, and we officially adopted Slack as our primary real-time communications platform in 2015, replacing Atlassian HipChat and IRC at the time.

In 2017, we became one of Slack’s very first Enterprise Grid customers, after an extensive early adopter alpha and beta testing period. Since then, our Slack usage has grown dramatically in both raw numbers and complexity, and today we’ll provide a little more insight into some of the ways we’re using Slack to accelerate game development at Riot.

SLACK @ SCALE

Riot supports more than 6,000 Slack users globally, and nearly 1,000 installed apps and integrations. The majority of Riot employees and external partners rely on Slack communication to do their jobs every day.

Our Slack app and integrations ecosystem is vast. We rely heavily on custom integrations which meet our strict information security requirements, along with a handful of heavily vetted marketplace apps (like Google Drive).

Slack is also a key part of Riot’s enterprise strategy, in support of our player-focused company mission. It helps us streamline game development through a rich ecosystem of apps and integrations, and expedites incident response, triaging, and issue resolution.

BUILD & DEPLOY

Slack plays an important role in engineering productivity at Riot. Investing in deeper and richer integrations has reduced mean time to resolution (MTTR) times and increased developer velocity.

Let’s dive into how engineering teams at Riot use Slack to increase build pipeline speed and visibility.

JENKINS PIPELINE INTEGRATIONS

Our first developer highlight comes from our Legends of Runeterra team. Legends of Runeterra is a free-to-play digital collectible card game that supports PC, Android, and iOS cross-play. Internally we’ve developed a sophisticated build and release pipeline for this game to ensure new versions are available to players simultaneously across all platforms.

Behind the scenes, Legends of Runeterra build pipelines are powered by Docker and Jenkins. Riot engineers rely on custom Slack apps to provide insight and visibility into game builds and deploys in real time.

When a Legends of Runeterra build starts, a new message is posted to a shared Slack channel. When a build fails, an alert is automatically posted to Slack which includes a detailed accounting of what went wrong. It even @ mentions the developer responsible for the latest code changes to begin immediate triage.

These Slack applications and integrations can be highly interactive, with specific actions available to developers at every step in the form of buttons, dialogs, screens, and workflows.

In this way, Slack provides a single pane of glass into the build and deployment lifecycle for our engineers, reducing the need to context switch between a myriad of different continuous integration tools just to understand the status of game builds and releases.

LEVERAGING DIRECT MESSAGES

Next up, a developer highlight from our Teamfight Tactics team. Teamfight Tactics is a free-to-play auto-battler game. Like Legends of Runeterra, Teamfight Tactics also supports PC, Android, and iOS cross-play, and benefits from sharing much of the same infrastructure and build/release technology as League of Legends.

To reduce noise - and possibly some developer embarrassment - in public channels, we’ve developed a Slack app named Game Build and Deploy “GBaD” Bot, which sends engineers direct messages whenever they break a game build, and lets them know once they’ve fixed the issue.

This implementation is a favorite among our R&D teams, who prefer to keep their build notifications out of public channels, and can still reap the benefits of automated build and deploy notifications and pipeline integrations via direct messages.

We love the flexibility Slack provides when it comes to build and deploy pipeline integrations. Slack’s real-team communication platform allows Riot to mirror our organization in digital channels and workspaces, while the robust APIs and SDKs allow us to deeply integrate these channels with our source code repositories and work management tools to accelerate the game development lifecycle.

OPERATIONS & SUPPORT

Just as important as developing games is ensuring they’re up and running when players log on to play

Let’s take a look at how Riot engineers use Slack to stay on top of operations and on-call support.

MONITORING & ALERTING

Many of our game teams integrate with Sentry for robust crash analytics and diagnostics. Sentry provides a feature-rich marketplace application which integrates directly with Riot’s game operations and monitoring channels.

Sentry automatically captures an enormous amount of valuable information whenever a player experiences a game crash, including screenshots and the actions that occurred directly before a crash. A summary of these events is automatically posted to actively monitored Slack channels. On-call engineers can assign crashes to colleagues directly via Slack, or click through to learn more.

Another Slack marketplace app we use heavily is PagerDuty. PagerDuty provides excellent incident, service, and escalation policy management, which we ingest directly into Slack for central control and visibility.

We route PagerDuty alerts to dedicated notification channels created for each team at Riot. From there, teams can escalate as needed to a universal channel shared across all of our workspaces, which is actively monitored by Riot’s global Network Operations Center.

PINNED BUTTONS

Pinned Buttons is a custom Slack app we built to help Rioters surface information and request help from other teams. Pinned Buttons attach themselves just above the message bar, giving Rioters contextual links and references before they send a message.

We want to give support teams an easy way to highlight the most common information that Rioters need to do their jobs. Whether it’s a link to a FAQ, a runbook, or a way to submit a ticket, teams can configure Pinned Buttons to highlight the most useful contextual information for their channels.

Pinned Buttons works by ensuring it’s always the latest message in the channel. The app listens to each channel in which it’s installed, and guarantees it’s always the bottom-most message by instantly posting itself again every time someone sends a new message (cleaning up its previous message each time).

Pinned Buttons is configurable per channel, with a custom banner and up to three buttons. Simply inviting the app into a channel will present users with a configuration model. After the initial setup, Rioters can use the /pinned_buttons slash command to update their channel configurations. Lastly, if Pinned Buttons are no longer of use or teams want to pause the functionality, they can simply kick the bot out the channel.

GOING CUSTOM

We’ve built hundreds of custom applications at Riot to augment everything from peer recognition to on-call support for engineering teams.

One of our most popular custom apps is Riot’s dev team support app. This app works by listening for emoji reactions on messages in support channels, and then kicking off a triage workflow whenever the appropriate reactji is used.

Reporters are prompted to categorize the issue based on their original message - information which the Slack app uses to automatically notify an on-call engineer from the appropriate team. The app @ mentions the on-call engineer in a threaded reply, and the conversation continues between the two Rioters in that thread as normal.

On-call engineers don’t need to monitor or read every message in the support channel. They can trust the support app to @ mention them in any thread that requires their attention.

Once an issue is resolved, the app captures the entire threaded conversation in the appropriate incident management tool, and it even provides a link to the Slack thread for future reference.

APP DEVELOPMENT

Let’s take a closer look at how we approach building custom Slack apps at Riot with PoroBot, one of our newest innovations in support request intake and management.

PoroBot is a custom-built Slack AI that can be trained to answer questions in Slack channels, triage and classify support tickets, and provide useful information based on previously answered questions by that channel's on-call team.

Adding PoroBot AI to a channel is self-service via /invite @porobot, and the app can be configured to create and manage tickets in either Jira and ServiceNow (via a pluggable work systems architecture). PoroBot uses extensible workflows behind the scenes, and leverages natural language processing (NLP) and machine learning (ML) models for pattern recognition and classification.

DEVELOPER FRAMEWORK

PoroBot uses @slack/bolt, which provides a great abstraction layer and SDK for quickly building Slack apps and integrations.

Bolt allows for quick bootstrapping while still leaving room for a high degree of customization and extensibility when we need to go deeper. The Bolt framework uses Express.js to abstract Slack events into API routes. In doing so, it also provides a heap of handy and familiar functions for interacting with Slack. We typically organize the code according to a consistent design pattern which abstracts away and separates the listener service and event workflows from the commands and business logic they invoke.

Using this approach with Bolt allows us to dive right into developing command and listener logic, without having to reinvent the wheel on event subscription mechanics or worrying about where we’re going to deploy the Slack app itself.

listeners.ts

import { app } from '@core/app';
 
import { exportMessagesListener } from '@commands/export_messages';
import { notesListener } from '@commands/notes';
import { wordcloudListener } from '@commands/wordcloud';
import { authorizeListener, silentAckListener } from '@core/listeners';
 
// Register listeners
app.command('/export_messages', authorizeListener, exportMessagesListener);
app.command('/notes', silentAckListener, notesListener);
app.command('/wordcloud', authorizeListener, wordcloudListener);
 
// Export the completed app with listeners
export const server = app;

CODE DEPLOYMENT

Slack’s Bolt framework provides us with the flexibility to deploy Slack apps in a number of locations and contexts, including serverless compute, depending on the use case. Here are a few examples of how we can extract and extend Bolt’s built-in Express.js server to suit a variety of different deployment environments.

Local Development (Docker)

// Start the app
(async () => {
  await initEnv();
  const { server } = await import ('../listeners');
  await server.start(process.env.PORT || 3000);
  console.log('⚡️ Bolt app is running!');
})();

import { App, ExpressReceiver } from '@slack/bolt'
import { APIGatewayEvent, Context, Callback } from 'aws-lambda'
import { createServer, proxy } from 'aws-serverless-express'

const expressReceiver = new ExpressReceiver({
  signingSecret: process.env['SLACK_SIGNING_SECRET']
})

export const app = (event, context, callback) => {
  const server = createServer(expressReceiver.app)

  context.succeed = (response) => {
    server.close()
    callback(undefined, response)
 }

  return proxy(server, event, context)
}

Google Cloud Functions (Firebase)

import { ExpressReceiver } from '@slack/bolt'
import { config, https } from 'firebase-functions'

const expressReceiver = new ExpressReceiver({
  signingSecret: config().slack_signing_secret
})

export const slackapp = https.onRequest(expressReceiver.app)

MANAGING STATE

Bolt also handy provides contextual hooks for managing state. We take advantage of this by injecting custom middleware functions into Bolt, which PoroBot uses to gather data about the requesting user and their internal account details.

• Conversation Store: We extend this to persist Slack conversation state beyond the original request. We do this with a Redis cache for short-lived workflows, and PostgreSQL for long-term storage.

• Context: A shallow object which we can read/write to as needed between function calls.

PoroBot hydrates state via middleware hooks, and provides that data as appropriate to the specific workflows being initialized. Workflows are built using a simple finite state machine triggered by Slack events (e.g. messages, button clicks, etc.) and containing metadata attached during middleware hydration (auto-response classifications, work system IDs, etc.).

PUSHING THE BOUNDARIES WITH MACHINE LEARNING

When a message enters a channel, PoroBot automatically attempts to classify it to determine whether or not to trigger an auto-response. Improving the classification model involves two distinct components: Patterns and Annotations.

1. Patterns: Analyze channel content and identify popular or common patterns of speech, then use this to create a list of source messages which are distilled into a generic syntax for the way questions are being asked. This syntax can be translated into patterns for deep learning with SpaCy PatternMatcher.
   Essentially, patterns are Slack messages that have been reduced to only the words that create meaning. We typically analyze and store 10–50 patterns per topic to make sure we have adequate coverage.
   Using patterns is adequate for most use cases.

2. Annotations: Once patterns have been battle-tested, annotation tools help label the data which we then use to train a convolutional neural network (CNN) text classifier using SpaCy.
   This method is generally used when there are similar question classifications that aren’t easily targeted with patterns. We use Prodigy as an aid in annotating the data, which allows us to train and correct the model in real time.

PoroBot and similar Slack AIs are still relatively new at Riot, and we’re excited to continue to push the envelope of what’s possible with Slack integrations to improve the game development lifecycle for Riot designers and engineers.

FINAL THOUGHTS

As we’ve seen, Slack enables a rich ecosystem of integrations which allows Rioters to spend more time focused on delivering delightful player experiences.

Here are our key learnings and takeaways from managing the Slack ecosystem at Riot:

1. Adopt best-in-class Slack apps like Sentry and PagerDuty to reduce incident resolution time.

2. Give your developers a single pane of glass into their build pipelines to reduce context switching.

3. Automate support intake and on-call assignments so the rest of the dev team stays focused on feature delivery.

4. Leverage buttons, dialogs and workflows to enable your developers take action directly from Slack.

5. Balance use of public channel and direct message integrations to improve focus and reduce noise.

COMMUNITY AND ENGAGEMENT

Developing video games doesn’t just involve code, it requires humans working together on teams to design and create incredible player experiences. These human systems by nature involve a lot of complexity, which Slack helps us navigate by providing convenient and accessible locations to have the important conversations.

We have a discussion channel where our Diversity & Inclusion teams share insights into what they're working on, and encourage open dialogue about diversity at Riot. Our GG Bot Slack app allows Rioters to recognize their peers, highlighting the specific Riot values exemplified by colleagues. We also have a vibrant social workspace where Rioters bond over thousands of different interests and hobbies - from pets and plants to boba and motorcycles, and everything in between.

To help keep our communication (and memes) strong, we also have a rich custom emoji culture. There are currently over 17,500 emojis at Riot! It’s a language all on its own, with reactions on messages being used in our development processes to initiate support requests and acknowledge when something is done. Fun fact: in the early days of Slack, we managed to accidentally DoS crash Slack’s emoji upload API while attempting a bulk emoji transfer between workspaces.

Slack has scaled up a lot since then, meanwhile our emoji game remains as strong as ever.

Thanks for reading! If you have any questions or comments, please post them below.