It was like "finding a needle in a haystack."
Six different characters from the game Apex Legends in various poses and holding various weapons.

For months now, Apex Legends players have been complaining about issues where various weapon sound effects and visual effects just flat out stopped playing during a match. The issue seemed to start for most of those impacted around the start of Season 16, which began in February 2023.

The story of this bug and how Respawn arrived at the fix is quite interesting. Thankfully for all of us, they shared the entire story in a new developer team update that was posted to Reddit.

Apex Legends' developer, Respawn Entertainment, says that players were experiencing a variety of related issues. Sometimes grenades would visually fail to explode but they still caused damage. Sometimes shots would not seem to fire from a weapon but the damage on targets was registered just fine. Respawn says that despite the numerous reports from players, the cause of the issue was very difficult to pin down.

After a preliminary investigation, the primary suspect was found to be the system our servers use to dispatch “start”/”stop” commands for various effects (e.g. certain sounds, particle systems, physics impacts, bullet tracers, explosions). Because our servers simulate entities (such as players, weapons, abilities, loot, etc.), they will emit sound/visual effects. Every server frame compiles an effects list of a maximum of 128 entries - any additional effects above the limit were getting dropped. This list is sent to any players who need the effects for the specific server frame.
Using the limitation of 128 entries per server frame, the team started to narrow their search further. Something, somewhere was flooding the servers and causing this limitation to be exceeded, which led to the missing effects for players. The challenge here is that every season there are "thousands of changes to assets, code, script, and levels." For Respawn, this was a real equivalent to "finding a needle in a haystack."

Their usual telemetry wasn't helpful here because it wasn't showing any issues with the system itself. Everything from their perspective was working as intended. The team went on to conduct a series of internal tests to try to push the limits of their systems. They were finally able to reproduce the problem in an artificial situation when they had a squad of 50 legends all firing the same weapon at the same time or spawning a bunch of invincible players that would auto-run while spamming their abilities and weapons as much as possible. This allowed the team to replicate the missing effects, but finding what caused this to happen in real games was yet to be discovered.

Using the information they gathered from those artificial tests, they released a server update that added in additional metrics to be monitored. Just as that update was being finalized, they found the culprit: It was a single line of code tied to a new weapon, the Nemesis, that was added in Season 16.

A single line of code was identified to be the root cause of the issue. Season 16’s new weapon.

The Nemesis has a particle effect that ramps up when heating up, but when it’s uncharged we don’t need to waste resources playing it as the effects are concealed inside of the weapon. So, when the weapon gets updated we would simply stop this particle effect if the weapon had no charge.

Every time the server simulates an input from the weapon's owner, this particular line of code is executed. Players send in their inputs for every single frame that is run on their client, and it’s the server’s job to simulate all of these inputs. This means that every single player with an uncharged Nemesis would create a “stop particle” "effect" on the server every frame, and this line of code was being called even when the weapon was holstered.​
Respawn says that this explains why they saw the bug reported more frequently at high-level play. They say that if there were 14 clients running around with a Nemesis and those clients were running at 180FPS, that would be enough to cause the effects to be dropped. They also note that this explains why they weren't able to see the issue during initial internal testing. They simply didn't have enough people in those internal builds running around with holstered Nemesis weapons, or have enough clients at those high framerates.

With this discovery in mind, Respawn deployed a server update last Tuesday that seemingly fixed this specific issue that plagued players for several months. The server update also includes some of those aforementioned additional metrics that should, in theory, help the team more easily narrow down any future issues like this.