Chat Service Architecture: Protocol

Some games of League of Legends I know I’ll love even before the loading screen appears. Take last night, for instance. I logged in to find a couple of fellow Rioters available to play, and while we queued together in a pre-game lobby we had a great discussion that led to smart role choices and an intentional team composition. That connection and strategic communication doesn’t guarantee victory — we lost — but win or lose that’s my favorite way to start.

LoL chat services support that pre-game experience. Every day players rely on chat to catch up with friends, arrange and plan games, celebrate victories, and mourn defeats. Because chat holds a significant role in the player experience, over the last few years we’ve exerted significant effort to make it more scalable, performant, and secure.

I’m excited to share our journey developing chat by diving into the relevant engineering through three posts on our most important backend components. This will also be a chance to open a dialogue for your questions and comments. I’ll deliver one post per component, filled with stories of our decision-making, successes, and failures on the road to providing players with an enjoyable chat experience. I hope these articles provide interesting engineering insight into one aspect of the game, inform the work of anyone working on a similar project, and spark useful conversation here and elsewhere.

The three backend components are the following:

  • The Extensible Messaging and Presence Protocol (XMPP) used for client/server communications and extended internally to meet League of Legends’ specific needs.
  • Chat servers written in Erlang and C used to communicate with clients, initially based on the open source version of ejabberd and rewritten over the past few years.
  • A persistent data store used for the social graph, ignore lists, offline messages, message history, and other features.

We’ll start off this series with a dive into XMPP protocol and how we use it!

XMPP

In League, chat plays a crucial role in keeping players in contact. Players need to see who else is online (and signal their own availability), exchange messages, add notes on friends, and sometimes block future interactions. To accomplish this, chat clients must speak a command language with the servers.

We took six factors into consideration when choosing a protocol:

  • Support for 1:1 messages, group chats, and offline messages. (These fundamental concepts support the development of player experiences like private messages, game lobby conversations, and pre- and post-game chats without the need to reinvent the wheel.)
  • Built-in presence mechanisms. (These notify players of their friends’ availability.)
  • Open specifications for transparency and auditability; a wide review scope;and, allowance for Riot-specific tuning.
  • A proven security model. (This ensures we can protect the privacy of players.)
  • An extensibility model. (This allows for custom behaviors.)
  • Availability of open source projects. (This accelerates development and delivery of chat to players.)

We evaluated several alternatives and XMPP best satisfied all of the above factors. XMPP is a XML-based protocol designed primarily for messaging and presence propagation, defined by three main RFCs: RFC 6120 (XMPP Core), RFC 6121 (XMPP Instant Messaging and Presence) and RFC 6122 (XMPP Address Format). Specifications and reference implementations are publicly available at http://xmpp.org/.

Each time I check a friend’s availability or status, discuss preferred roles during the champ select phase, or express satisfaction and humility after a decisive victory in the post-game chat, my client sends and receives a number of so-called XMPP “stanzas.” The type, layout, and content of each stanza depend on the context, but you can think of them as commands sent to the server.

For instance, in order to broadcast availability to friends, the client sends:

<presence type=’available’/>

To ask the server to fetch a full friends list:

<iq type=’get’ id=’roster1234’>
 <query xmlns=’jabber:iq:roster’/>
</iq>

And to message a friend:

<message to=’[email protected]’>
  <body>Hey there, wanna play?</body>
</message>

These three stanza types (presence, iq, and message, respectively) are the building blocks of any XMPP-based application protocol. But for League, our needs go beyond these types. One main feature of XMPP is how it gives developers freedom to define custom extensions in order to support deployment-specific needs. I’ll explain the extension mechanism and philosophy in detail in the following section.

XMPP extensions

Apart from the official RFCs, the XMPP Standards Foundation provides a framework for introducing official extensions to the protocol by means of XEPs (XMPP Enhancement Proposals). These define custom behaviors that expose new functionality, alter existing functionality, or suggest best practices when using the protocol. A fuller description and a comprehensive list of XEPs can be found here: http://xmpp.org/xmpp-protocols/xmpp-extensions/.

From the very beginning, we were aware that the core XMPP features and existing XEPs might not be enough for us to reach our goals with regards to game-specific features, scale, and performance. Since League of Legends chat has some properties that are not part of any of the existing XMPP extensions and has its own proprietary client, we introduced a number of protocol extensions that alter both core and existing XEPs. We maintain these extensions on an internal wiki page called the RXEP Library (Riot XMPP Enhancement Proposals) and use it to document any changes.

Let’s review two of the extensions we introduced to the RXEP Library: friend roster notes and incremental privacy list updates.

XMPP extension #1: friend roster notes

Often times I send friend invites to people I’ve met in game that I’d love to play with again in the future. I enjoy their clever and fun summoner names, but days later I often have trouble distinguishing who is who, and what their preferred play style is. To solve this, every time I add someone to my friends list I attach a short note to their name explaining their best position, favorite game mode (ranked 5s!), and sometimes the context in which I met them. “Great Darius player, Summoner’s Rift only.” “Friendly plat guy, met him at PAX.” “Mom.”

To enable this type of functionality, we designed a very simple extension to the roster protocol. According to the RFC 6121, an item element represents each friend on the roster. Contacts can have multiple parameters, such as a summoner name (name attribute), internal Jabber identifier or JID (jid attribute) used for internal routing, and their group (group sub-element):

<item jid="[email protected]" name="0xDEADB33F">
  <group>General</group>
</item>

We introduced an additional optional sub-element note that holds short, contact-specific text written by the player:

<item jid="[email protected]" name="0xDEADB33F">
  <group>General</group>
  <note>Versatile dude, plays only ranked, not a big fan of ARAMs</note>
</item>

The note itself is only visible to the contact owner (my friend cannot see it) and can contain any content I wish: comments, notes, the player’s real name, etc. In order to protect players’ privacy, we ensure that the notes stay encrypted on disk using strong cryptography algorithms. We encrypt payloads — before persisting in a data store — with AES-CTR-128, while we use HMAC-SHA1 for signing the data. We also implemented support for rotating encryption keys should we decide to update them in the future.

XMPP extension #2: incremental ignore list updates

Unfortunately, during some games on the Rift I encounter players with whom I’m not interested in having any future interaction. I can block these people using the ignore feature, stopping any sort of out-of-game communication from them to me (including sending regular messages or friend invites).

A fairly complex XEP describing so-called privacy lists management is found here: http://xmpp.org/extensions/xep-0016.html. However, it only provides get/set semantics. According to the extension, we can’t modify the existing privacy list by just adding or removing an item. Every change I make requires uploading the whole list to the chat server. Since ignore lists can grow relatively large, transmitting them back and forth between server and client is both costly and useless. To address that issue we added two new commands to the privacy lists API.

In order to add an item to the privacy list, clients issue:

<add name=’LOL’>
  <item value=’[email protected]’ action=’deny’ order=’1’ type=’jid’ />
</add>

And to unblock someone:

<remove name=’LOL’>
  <item value=’[email protected]’ />
</remove>

Protocol compatibility

I might want to stay in touch with League of Legends friends and use chat to communicate with them outside of the game. A number of community-driven projects bring the Riot chat experience to desktop or mobile devices using standard XMPP channels, such as LoL Connect for iOS or LoL Chat for Android.

We want to make sure any changes made to the protocol are backward compatible and that third party clients aren’t kicked out of the ecosystem. Certain features (such as roster notes) might not be available using standard XMPP libraries. However, with a relatively low effort developers can add them to the client libraries.

When designing new extensions we also have to make sure that any future potential engagement channels such as web, mobile, or Riot’s next game will also fit into our chat world and allow players to seamlessly communicate with each other.

Summary

I hope this post provides insight into how we use XMPP for Riot chat to deliver an awesome experience in and out of the game. At the same time, I am fully aware that we just scratched the surface here, and that’s where we need your thoughts. Please post comments and questions, and I’ll do my best to provide answers from my team.

Also, if you’d like to learn more about chat services, I recently gave a talk entitled “Scaling LoL Chat to 70 Million Players.” A summary of that talk is also available at highscalability.com.

But wait, there’s more! In the coming months I’ll be back with the next part in this series on Riot chat, which will examine the server-side architecture and persistency layer for the social graph. Stay tuned!

For more information, check out the rest of this series:

Part I: Protocol (this article)
Part II: Servers
Part III: Persistence

Posted by Michal Ptaszek