Moving Mountains of Data to AWS

Hi, I’m Sean Maloney, and I’m part of Insights Tech. We’re within the Insights group, which helps teams at Riot make better decisions in serving players. At Riot, we’re data-informed rather than data-driven—we always incorporate data in our decision-making process, but never allow data to dictate decisions. Insights Tech provides all Riot teams with a data platform to capture, store, and serve up all the player data that they need.

League of Legends' international community generates an enormous amount of data. Since the game launched in 2009, we’ve collected 26 petabytes of how players interact with the game and our websites, which we use to improve the player experience through projects like game balance and bot detection. Ingesting and storing all of that data is quite the challenge, and, in 2011, we ran into serious scaling problems when our existing data warehouse ran out of space.

This October, I had the opportunity to speak at Amazon's re:Invent conference and tell the story of how Insights Tech tackled this problem, embarking on an epic journey to move the entire data warehouse into AWS. It was no simple task, and we made mistakes along the way. However, in the end, we succeeded (and avoided a disaster I discuss in the talk).

You can watch the talk below. We’ve learned many lessons from our successes and failures throughout the process, and I hope it’ll prove helpful to anyone else facing a similar situation.

Some key topics covered are:

• Why we decided to move from a datacenter to AWS
• Important considerations if you are planning to make a similar move
• Pitfalls to avoid when using Hive
• How we use EMR to support ad hoc compute power for our analysts
• What our AWS data warehouse platform looks like today

I'm more than happy to answer any questions, so please leave comments below.