A Path Towards Light and Possibilities

Brushing Up On Part 1

Since a lot of information was covered in the first part, here’s a quick refresher! The “Postponed spell” system is a quality of life spell cast buffering feature that we want all champions to have, but Hwei’s unique way of spell casting made it so his actual spell casts didn’t get buffered properly.

Various implementation methods were rapidly attempted to get his spell casting up to speed, including extending the “Postponed spell” system, but they all ended in failure one way or another. 

The most promising implementation we had was the server side spell slot queue, which we were able to start tests on quickly. It had access to the source of truth, and increased the spell cast buffer size using spell slots so that Hwei could have at least one “complete” spell cast on the buffer, which was the initial goal.

But while it felt better initially, getting the tech into a shippable state was a whole other can of worms.

Trouble With Spell Slot Queuing

Tormented by Targeting

What plagues this implementation greatly is the mismatch between client and server information. The source of strength of this implementation actually becomes its achilles heel. What ends up happening is the client sends spell cast packets that are generated oblivious to what the source of truth is. The client has its own copy of what spells Hwei currently has, and uses those spells to determine how targeting information is generated, which can be different than what the server says Hwei truly has.

Let’s concoct a goofy hypothetical example to explain the issue here. Say Hwei has “Subject: Jhin” as a spell that swaps his spells to Jhin’s spells. Hwei casts “Subject: Jhin” and then Jhin’s Q spell, “Dancing Grenade”, which is a targeted spell. The subject spell has to do a round trip before it actually changes the spells on the client, sending the spell cast packet to the server which then replies with the spell changes on the client. If a player client casts Jhin’s Q before the round trip finishes, we would actually still have the subject spell in the spell slot, which doesn’t require a target with its spell cast. This means if we try to queue up Jhin’s Q before the round trip returns, the spell cast wouldn’t have a target, which would inevitably cause issues once the malformed spell cast packet gets sent to the server.

Jhin’s targeted Q spell indicator example (Jhindicator)

 

“Subject: Jhin” example of targeting troubles

Thankfully, Hwei didn’t have any spells that required targeting data. We could get around this issue because the essential information we needed to cast Hwei’s spells on the server were just the spell slot and the cast location, which were already being sent with the spell cast packet to the server. But short term, if another champion wanted to use this tech this meant that we were either heavily constrained on how that champion could cast their spells, or would have to commit to a large system update to get this to work with targeting. This was far from ideal, but not a deal breaker just yet.

Normal Cast Nightmares

If targeting was the only issue, we could have worked with it. But different casting paradigms, like normal cast, quick cast, quick cast with indicator, clicking the HUD and casting a spell with mouse only (you are a gigachad if you do this), all had their special different behaviors that had to be supported with whatever implementation we ended up with. The problem with different casting paradigms is that these are all handled client side only; the server only receives necessary data for the attempted spell cast which doesn’t include the cast paradigm. Which meant that casting paradigms and targeting issues both stem from the same root problem, client and server information mismatch.

With normal cast, you have to click to confirm your spell cast after selecting your spell. A spell cast for Ezreal would then look like Q -> mouse click, to cast “Mystic Shot”, while a spell cast for Hwei would look like Q -> Q -> mouse click, to cast “Devastating Fire”. But with quick cast, no mouse click is needed since the confirmation step is done with a button press. A spell cast for Ezreal’s “Mystic Shot” is just pressing Q, and Hwei’s “Devastating Fire” is just Q -> Q. 

Ezreal casting “Mystic Shot” with Normal Cast

 

Ezreal casting “Mystic Shot” with Quick Cast

This difference means we couldn’t just have spells fire based on the keypress input chain, but we would also have to introduce a mouse click confirmation into the queue. And not just that, since players can choose either option for each spell in their settings, we would also have to track whether a specific spell was a normal cast spell or quick cast spell in their settings.

This was actually a big issue. Options were stored client side only, and we couldn’t just store extra information in the spell cast packet on whether or not it was normal cast due to the information mismatch problem, so we would have to introduce the concept of server side settings that the server had to keep track of. This was a large body of extra work that would have to be done, and while it was achievable it felt suboptimal and a little hacky.

Flubbing, the Final Nail in the Coffin

While the normal cast issue felt like it could have been solved with a bit more time, there was yet one more problem. Deeply ingrained in a server side queue solution was the concept of flubbing (moniker coined by Emizery), which was the act of pressing an input incorrectly, human error. It’s common for players to fat finger a button, or spam inputs on a spell just coming off cooldown, and League actually has some behind the scenes flub prevention magic that helps with this. For certain spells, you can queue up casting a spell a little bit before it comes off cooldown, which is a form of flub prevention.

Ezreal casting “Essence Flux” before it comes off cooldown

The initial design side implementation had concepts of flub prevention baked into it, having timers that would help with accidental inputs wherever it could. The server side spell slot queue would also need similar concepts of flub handling, which required a bunch of new script calls to code to handle various flub cases.

The problem here was that different folks had different strokes, but we had to come up with a default that felt good for a majority of players. Having these invisible flub prevention mechanics could be a detriment to an experienced Hwei player who is precise and wouldn’t want their inputs to be modified behind the scenes. Meanwhile, if we tune the prevention to cater to more precise players or remove it completely, we could devastate the experience of a newer Hwei player due to the system being too responsive to any input. Players with less precise inputs would commonly queue up flubbed inputs meaning a lot of unexpected spell casts and a lot of ending up in unexpected states. Because of the various precision levels across all players, we ideally wanted to have a setting that players could mess with, which in turn added more work to implementing Hwei options.

We had reached the limits of a server side queue system, and incremental progress was taking more and more effort. Since a lot of issues were with the system feel and subjective, we would have to implement tools and logging to actually find out what was causing playtester pain to improve upon the system. I had already implemented a nifty debugger as a baseline to find out simple issues, but without intensive game logging requiring minimal playtest overhead, we wouldn’t be able to find out what specific scenarios caused player pain points, which meant even more work for optimization and amelioration of the queue.

Spell Slot Queue Debugger Demo (with the queue having unrealistic settings)

Flipping the Canvas

It was around this time where we started to question whether or not this was the right approach. Cliff "1kFeathers" Zhu had joined the pod to help with Hwei since it seemed like there was going to be a significant amount of engineering work and not a lot of time, and I am eternally grateful for him hopping on the project and helping out. One of the things he asked me after joining changed the trajectory of Hwei’s spell casting tech forever. “Why can’t we implement a client side solution?”. 

I had thought about it before, Hwei’s first input for a “complete” spell cast is just a spell swap and only after the second input do we cast the intended “complete” spell. If we could implement something that fakes everything the subject selection spell cast does, we would only need to send a spell cast packet on that second input. This would mean that spell casting for Hwei would be handled on the server in line with other champions. One spell cast packet meant one spell cast for server side systems, which would resolve issues stemming from two spells for one. Things like the “Postponed spell” system would be able to work as intended.

On the outside it’s simple idea, but if we were to do all the faking client side, we would need to do it all in code which was not ideal, since we would have to fake animations, VFX, SFX, HUD icons, and everything else needed when he swaps to a subject and off of one newly in code. This would require a gargantuan amount of work primarily because the ability to fake all of these client side would require new systems that allow for the client to go ahead and preemptively do things that normally require server confirmation before server confirmation. Faking being in code also meant that even after doing the work to expose these systems to a designer, it wouldn’t be as flexible as having it in script. Specific requests to change swapping logic would be at the mercy of an engineer.

However, one fateful day after some discussion, a key insight that 1kFeathers brought up was that we didn’t need to do all of this faking in code. We could have a script side event that we call from the client to do all the visuals of swapping for us. The script being on the server meant that this would cost us a round trip packet worth of latency before our client visual state matches the logic handling state of the client. But this idea also meant we would get rid of some of the biggest costs of faking since we could use scripts for handling the visual aspects of it now.

Visual representation of theoretical ideal visual change speed vs. compromise

While it wasn’t optimal, if we interpret this “faking” as a spell cast (which it was previously), it was very much in line with what any other champion would do. For any champion, after a player’s initial input for a spell cast, a round trip packet would have to happen before their champion plays associated spell cast visuals. If we make comparisons with this in mind, rather than with the optimal solution, the tradeoff cost was next to negligible. 

After weighing out the pros and cons and the workload of optimizing each solution, we both agreed that this was THE implementation we had to shoot for. This had the capacity to better spell casting feel significantly and completely blow the server side queue implementation out of the water. But with little time to spare before shipping, we were cutting it extraordinarily close and had yet to prove our hypothesis that this was better than the server side queue.

So how did we go about turning this idea into a reality? For that I’ll hand it over to 1kFeathers, who’ll explain our journey to completion and ultimately why Hwei feels so fluid to spell cast with!

Contemporary Spell Casting system

Fancy seeing you here of all places, Cliff “1kFeathers” Zhu here, and I’m going to talk about the solution we ended up rolling with. Before we get into the weeds, let’s fill our palette with how spellcasting works in league of legends. 

When the player presses a button, let’s say Q, the client receives an input and looks up a dictionary to find the spell slot (0) associated with that button/input. Here a really important concept of the spell casting system comes up: “Spell Slot”.

What is a Spell Slot?
As Quasmic touched upon lightly, a spell slot is the primary medium in the spell casting system that tells the game which spell to cast. Every champion has ability slots, which map to their Q/W/E/R abilities. Slot 0 means Q, 1 is W, so on and so forth. We always pass slots around in the system and rely on the slot to retrieve metadata of a spell.  

Spell to spell slot mapping

Now that we know what a spell slot is, we get the slot and start a cast request from the client, client sends this request to the server, and server locates the spell object (Such as Ezreal Q, “Mystic Shot”) using the slot and validates the request: Are you casting your own spell? Is it off-cooldown? Do you have the mana? If the spell passes validation, the game server sends it back to the client. And boom, Ezreal casts his “Mystic Shot”. 

For shapeshifters like Jayce or Nidalee, we would swap spells onto and off a slot as they transform, so they can change spells. You might wonder, can’t Hwei do something similar? If we open spellbook Q (“Subject: Disaster”), we swap QQ/QW/QE onto slot 0, and so on and so forth. 

That is the right idea, but would need a bit more work to make sure this works well with Multi-spell Spell Queueing and instant client side HUD feedback. Everything Hwei casts should function the same as any other spell cast in the game. For that, we needed a full on “Spellbook Override” system.

Postmodernism SpellbookOverride prototype

Now that we’re equipped with the knowledge of how the spellcasting system works. Let’s talk about the MVP (Minimum Viable Product) for the client-side driven solution Hwei uses to cast his spells, or as we call it, the “Spellbook Override” system.

“Spellbook Override” starts on the client. We receive one of the inputs for Q/W/E and move the spellbook to an override state. Once in the override state, we listen for new inputs and send a spell request to the server according to the additional input and the spellbook we’re in. For example, if we are in the Q override state (“Subject: Disaster”) and a player inputs another Q, we would send the data that the server needs to locate the metadata for QQ (“Devastating Fire”). The server would receive this packet, locate QQ, and put it on slot 0 (Q slot). Then the server will cast QQ and let the client know that a spellcast has been successfully casted. Client will receive the acknowledgement and reset the override state back to default. 

Visualization of the "Spellbook Override" prototype

That was a quick TL;DR of the system, but a couple interesting things you might have noticed:

Fake it till you make it

“Moving the spellbook to an override state” is a fake spell cast. Instead of going with the “one button one spell cast request” pattern in our Contemporary Spell Casting system, we are doing “two buttons one spell cast” and thus the first button of any spell cast sequence (QQ/QW/QE/WQ/WW/WE/EQ/EW/EE) becomes a “fake spell cast” as Quasmic mentioned briefly. The fakeness means we’re doing everything to make it look like a spell cast: The Cast Animation, VFX, SFX, etc. Unlike any other spell casts, we’re not asking the server to get permission to cast the spell since players are always allowed to open/close spellbooks.

Nine Spells, Three slots

You might notice that unlike most champions who have one spell statically located on a spell slot, Hwei does not do that! In the “Spellbook Override” system, the client will request to put a specific spell onto the spell slot. Since this change is dynamic, managing when to move a spell into and out of a slot becomes very important. If managed incorrectly, we would get bugs like we intended to cast QE (Molten Fissure), but Hwei casted something completely different. This happened in some of our early day playtests, oops!

Erasing Cast Times

With the new system, opening and closing spellbooks do not have a cast time anymore. This is a combination of:

• Since the implementation of client-side spellbook override, we no longer require a cast time to hide the latency between client and server.

• Design has aligned that having no cast time for open spellbooks feels better/smoother.

• Implementing cast time would require additional work in the client side implementation due to the “fake spell cast” nature of it.

Hwei, All Out Edition

It’s only MVP, not all things are working perfectly! One thing we did not implement is that the server had no validations. It trusts the client and assumes that all spell requests sent from the client were valid. While this is not a concern during development, we do not want this in the release build. The reason is hackers can technically send fake spell request packets and the server will accept it, causing the painter to go all out and spam spells.

Harmony and Discord

Here you go, we now have a client side implementation. But how does it compare to the server side solution? Let’s chat about its strengths and weaknesses as the artist seeks Golden Ratio in his spell casts.

The Golden Artist (Pros)

• When determining which spell to cast and switching the HUD state, latency between client and server is avoided. 

• Input handling is client side so we would have more metadata about the cast, whether it’s a normal cast, quick cast, etc and don’t need special handling for those.

• Since the server only receives the information of one spell, we get some server-side systems for free and also save us from consuming server throughput. 

• Flubbing altogether does not need to be implemented since there is no cast time when we are opening spellbooks, which is a huge blessing

A Mind Lost (Cons)

• Since the spellbook system is on the client, we don’t have the source of truth that the server has. Since we don’t want client to wait for server confirmation, we’d need to predict whether Hwei has the resources to cast a spell or if he needs to move into range

• Engineering overhead will be required in the future to maintain Hwei since debugging issues with his spell casting will require code fixes.

We believe the good outweighs the bad, the client-side solution will be more robust, latency-tolerant and most importantly, improve the player experience for Hwei. So we decided to move forward with this approach.

Polish Pass

Have we solved all the problems for the painter? Not quite, there’s still some loose ends we need to tie up. This section will cover the remaining problems with the client-side solution and how we approached them.

You Can (Not) Undo

The nemesis of every multiplayer game developer is latency. In Hwei’s case, when the client receives the input of a (Q->Q), it needs to wait for the server to get back to it to really know if that spell is successfully cast, or failed since we don’t have enough resources.

During high latency circumstances, it could take a while (a whole client->server->client roundtrip) for the client to really know the outcome of the spellcast. 

This is where client-side prediction comes into place. When the client is waiting for the server's response, it will predict the success/failure of the spellcast and autocorrect itself when the server sends back the validation. This would allow us to instantly swap HUD states on the client side so every input the player makes feels responsive and fluid.

However, client-side prediction could be a nightmare to manage as well, as we need to handle the edge case where client thinks we are doing A, but server thinks we are doing B. Client needs to undo the changes and correct itself based on the validation from the server. We are doing this in a couple of places and one place is mana management. Imagine you’re on the verge of running out of mana as Hwei and casted a spell. Note that on client it would think that the spellcast will fail since you don’t have enough mana. But if you gained mana (from honeyfruits let’s say) during the latency between client and server. He could now have enough mana to cast the previously predicted-failed spell!

Hwei only has mana to casts one of the spell; after casting,
all his spells on the HUD says “out of mana”

Proper Postponing

When players press multiple inputs at the same time, they expect spells to be queued up like Quasmic mentioned in the "Postponed Spell" section. When you press E->Q on Ezreal, you expect “Mystic Shot” to fire as soon as “Arcane Shift” finishes. We want the same thing for Hwei, except that in his case, two inputs form a spell. This is not too big of a problem for us since the “Spellbook Override” system already takes care of most of this. We’re bundling up both inputs into one spell so when the player presses “Q->Q->W->W”, only spell1 “Devastating Fire (QQ)” and spell2 “Fleeting Current (WW)” are sent to the server. This lets the server put “WW” onto the postponed queue while we’re casting “QQ” and cast “WW” immediately after. We didn’t exactly get the “Postponed Spell” system for free and had to do some refactoring. One problem that came up was that postponed spells can be cleared due to a variety of reasons: movement, caster getting crowd-controlled, maximum time has passed, or caster issued a new spell, etc. In these cases, the client needs to be notified that the postponed spell is no longer queued, and should stop expecting its cast.

Hwei casting multiple spells in quick succession

“I’m moving as fast as I can”

When champions try to cast abilities outside of their cast range, sometimes it makes sense for them to walk into range and then cast the ability. Some spells on Hwei (“Severing Bolt (QW)”, “Pool of Reflection (WW)”, “Gaze of the Abyss (EW)”, “ Crushing Maw (EE)”) would appreciate if we could do something like this. However, the nature of the “Spellbook Override” system makes it hard due to these reasons:

• For our MVP, we instantly put a spell on cooldown after two button presses, which doesn’t work for move-to-cast since we haven’t really started a spellcast yet until we walked in range.

• We want to keep the spellbook open as he’s walking to cast, and the player should have the ability to stop the move to cast or cast a different spell in the spellbook or even use “Wash Brush (R)” to exit out a spellbook.

• Range check happens only on the server, so it’s hard for the client to know if a spell will be a move-to-cast or not.

To solve all of these problems, we implemented a client-side range prediction mechanism to predict if a spell requires walking. It will correct its state as the server sends the validation to prevent edge cases where client thinks it’s move to cast but server thinks otherwise, or vice versa. And when we’re walking to cast, we would keep the pending spell onto the slot, and swap it off either when we are casting the spell or the player issued another order that is not a move to cast.

Hwei moving to cast a spell EW, cancels it and
casts a different spell EQ in the same spell book

On our Hwei to the future

Now that Hwei is shipped and players are enjoying playing (with/against) him on the rift, what’s next? There are a bunch of things that Quasmic and I think would be cool for the future of the game and players if we could implement them. 

Shifting Patterns

The “Spellbook Override” system is designed to be reused by future champions and creatures in the game. The idea is designers just need to specify the data of the spellbook , for example:

And the system will automatically handle swapping spells onto/off of spell slots instead of having bespoke handling logic in champion specific scripts. So potentially we could convert Shapeshifters like Jayce/Nidalee/Elise/Rell/Gnar to use the new system, as well as Aphelios, as he technically swaps spells all the time depending on which weapon combination he has. The conversion does need some legwork, but the benefit would be that shapeshifters could buffer additional spells so their combos get even smoother. 

Nidalee doing Nidalee things

Settings Standards

One thing that came up during the development process of Hwei is the custom settings for “Move To Cast”, which currently exists as an in-game option “Clamp cast target location within max range”! When enabled, spells will auto cast at the maximum range if your cursor is out of range of the spell, avoiding walking to cast. The reason why Emizery pushed for this setting is we were getting feedback on the cast paradigm for some of the spells (“Pool of Reflection (WW)”, “Gaze of the Abyss (EW)”, “ Crushing Maw (EE)”): Some players preferred walk to cast while others enjoyed clamp casts. In order to address these feedbacks, we implemented this toggle which would hopefully be helpful for more than just the artist himself. We also have some ideas to augment this setting in the future, but no promises whether it will be implemented at the current time.

Farewell, Good Painter

You made it! That’s the end of the article and thank you for making this far. It was quite a journey for Quasmic and I to come up with and implement the tech and we could not do it without the rest of the team. We’d like to shout out our very own design mastermind Riot Emizery, Riot Etlios, who was responsible for the quality assurance of the system, Riot Entquine and Riot Chibattabun, for system architecture consultations, and Riot DarthBatman, for networking consultations.