Nearly 20 years ago, on 8 November 2003, Andrew Gower wrote the same thing. I always thought it was a bit of a strange decision for them to not roll back the servers a day or so. As they stated in their initial reaction to the problem
They had database backups, and could have used them to go back a few days, but chose not to. Instead, 20 years later, still today in RuneScape 3, "Purple" party hats, which were once the most rare of all party hats, were duplicated to the point of being the least valuable party hat. It seems like in hindsight, they should have just taken the hit and rolled back the servers, instead of having this permanent effect on the economy.Luckily we take regular backups of the RuneScape database which are securely stored here on DVD, so it will be possible for us to run a comparison against these backups and identify anyone who has suddenly gained a large amount of wealth in the last few days. In this way we expect to identify and remove the items which have been introduced to the game, and take action against those involved.
Having now been through such an event, even on a smaller scale with less players involved, I understand better why they reacted the way they did. Essentially, we are reacting to our own duplication bug the same way.
Our duplication bug
We first became aware of duplication bugs being exploited on our server in the evening of 25 January 2023. On the RSC Cabbage server, a user Niikasd reported the very strange behaviour of someone dropping a Dragon 2-handed Sword, likely the most valuable item in that game, on the ground and letting it despawn.
Mod Aurora escalated this concern to the admin team, and when Ryan investigated who had done this, we found that they had a couple dozen Dragon 2-handed Swords, and that they were distributing them freely to lots of players. None of this made much sense, since the Dragon 2-handed Sword is a very rare item. Only 30 of them existed in the game two weeks prior, and now there were 65. This prompted us to look for commonly duplicated items in our other servers as well, and we found that Uranium and Preservation had increased numbers of (crackers + party hats), which is a number that should only ever go down without duplication bugs.
After a night's sleep and much deliberation later, we chose to shut down all of our servers, despite the fact that we were currently enjoying unprecedented popularity, with a concurrent player count on RSC Preservation of 190 at its highest, that I saw. We were not sure when the duplication bug had first been abused, or how far we would have to roll back, but we knew that it was still being actively exploited and that the amount of clean-up would only get worse. All servers were shut down at 11:25 AM EST, 26 January 2023.
Compounding the bad timing of this happening while hosting an order of magnitude more players than usual, this duplication bug hit us at a time when all of our regular contributors were unable to attend to the problem. In fact, none (including myself) would be available for at least 2 days.
On the same day that the servers were shut down, we promoted Ken to administrator in order to help investigate the significance of the damage done. Ken is a long-time contributor and was in fact the third person to join the Open RSC project, after Kenix and Marwolf. What he found is that the amount of party hats on RSC Preservation had only increased by about 20% compared to a database backup from two weeks ago. He also found that other items, like Dragon bone certificates, were not duplicated.
Having a hunch that an old duping bug known to exist a long time ago involving dropping an item once and picking it up twice had re-emerged, we also found in the database records that a user named "pcap" did seem to be duplicating on RSC Cabbage using a drop method:
We would have continued investigating this, but it was at about this time that we were notified by Chomp in the #general-chat channel of our discord of what was going on.
Unfortunately, the full records of what was said and communicated here are lost as a result of the conversation being bulk deleted, but from my memory, there were two anonymous users that joined the discord. One was Ryn, who posted the above image as part of an imgur album proving their identity and that members of the redacted Discord held vile opinions about LGBT+ people. The other anonymous user was more abrasive and said something to the effect of "These bugs are all old why shut down now? They were reported months ago back in April, you didn't fix it then, why would you now? Just go put the server back online! Lolololol"
Well, we had no idea what they meant by any of that, but after looking into it, it did seem to be true that there was an open bug report in our tracker from Chomp. #3327, which until it was resolved, had been a hidden issue, due to its severity. It completely detailed how the dupe bug worked, and how to solve it.
I've thought hard about why such a serious bug would have been ignored and allowed to exist for so long after being reported. I believe there are several contributing factors:
- This was only one of many bug reports that Chomp had filed. He actually was kind enough to not only find and report them, but also to resolve them. In this same timeframe, there was #3329, concerning being able to sleep multiple times simultaneously to restore fatigue faster. Resolved by Chomp. #3319 Trade/Duel bug, which allowed any user to reset another player in the middle of a trade or duel. Resolved by Chomp. #3337, which sounds like it allows infinite prayer usage. Resolved by Chomp.
There were likely other serious historic bugs that I haven't mentioned here. But the point here is that there was a pattern of him both reporting and resolving bugs, which may have contributed to us ignoring the dupe bug in question today, since we believed that he had already handled it. I can say for sure that I would have read the issue when it was filed, but that I didn't remember it being still open at the time that it was exploited on our live servers.
- Due to how helpful he was, we trusted Chomp as a good actor. Even if there were a dupe bug, he was trusted to not take advantage of it, which means we would not have taken the bug as seriously. He is the one that reported it, so why would he then go and abuse an issue he had voluntarily made us aware of?
- Our player base at the time of these bugs being reported was much smaller, and I believe there was no one in the community as talented at finding bugs as Chomp. An issue like this could have persisted years without being discovered without his intervention, which decreased the priority of ensuring it was fixed, in our minds.
We were also made aware of when exactly Chomp had publicly released these duping scripts.
We contemplated both a full rollback to just before these scripts were publicly released, which would have been a 2 day rollback, as well as a partial-rollback of just the items related table. The partial rollback when more carefully considered turned out to be a very bad idea, since it would affect any player who had trained a skill that consumes items, by giving them essentially double the experience in skills like Smithing. A rollback would have been all-or-nothing.
We still weren't sure of the total severity of the dupe. We needed more data to check more thoroughly what was duplicated and by who and how much. Writing SQL queries manually for this type of thing, comparing against historic backups, was laborious and not as thorough as we would have liked. They give a great overview on the increase of specific items, but some actions (like deploying a cannon, which duplicated Coins as part of the glitch) are not logged to the database.
Ace in the hole
Finally, on Saturday the 28th, I was available to help with the investigation. There is a little-known feature of the Open RSC framework, which I implemented two years ago. It records all network data sent both to and from a player's client for their entire play session, in a standard network data logging format known as a "pcap". We have used pcaps in the past during more involved Botting investigations. In 2019, 2020, and 2021, I wrote a protocol dissector for the RSC235 protocol in wireshark (massive undertaking), so that we can view individual packets in a graphical user interface which explains what each byte means in English.
Here's an example of what the cracker dupe looks like:
This is great, because it lets us see exactly how the packets were structured by malicious actors to execute the dupe, confirming what Chomp reported in #3327. But until now, we have only ever used pcaps in this way: looking at individual packets in individual play sessions, one at a time. We have never parsed all pcaps at the same time, which to find all instances of this behaviour, we would now have to do.
We had no tool to do this, so I built it on Saturday. Piggybacking off of the rscminus project, which we have used to parse authentic era RSC+ replays to find and aggregate historical data. I ripped out all of the code which loads the complicated and storied RSC+ replay format and got to work implementing PCAP format loading instead. I then programmed the packet parsing loop to find instances of duping like the one in the screenshot above. Any time that a user does any action AND logs out in the same tick was able to found, and I loaded all PCAPs for the month of January into the modified rscminus scraper. Over 111,000 play sessions in January were searched.
Here is the result, a list of all cheaters who abused this glitch on the RSC Preservation server:
We of course also got a list of every single duping attempt conducted on the server. The duping began at 3:50 AM EST on 20 January 2023, about 4 days before the scripts were publicly released. It can also be seen the development of the "ready-to-go" aposbot script. Leading up to successful attempts at duping a cracker with the script, there were many failed attempts at using item on player and then logging out, including using Cooked Swordfish on a player instead of a cracker.
From here, with all the names of players who duped on RSC Preservation, we can look at what they did with their wealth in the standard Trade and generic_logs tables.
For example, here is every time Byte dropped coins (we also have traced who picked up the coins):
And here is every time a duper traded with another player on the server:
We have the data to undo each instance that a party hat was distributed or that a trade with illegitimate goods was transacted. However, we have mostly chosen not to do this.
Frankly, it is just a lot of work that would keep our servers offline for longer. Some of the trades are legitimate (e.g. Rin legitimately sold coal to STS9), which complicates the matter. We don't currently have any tool to automatically parse the trades logged or roll them back. Additionally, the amounts introduced are small enough to not worry about having a lasting effect on the economy. A maximum of 10M GP was duped by STS9 and a maximum of 54M GP was duped by Byte, of which we can see only 2M distributed by STS9 and around 10M distributed by Byte. The rest of the wealth is on banned accounts.
Lastly, we suspect that the innocent users who received party hats and gold during the drop parties would generally prefer to keep them. Making people most happy is what has led to our decision to not roll back the server in any way, further than just banning the accounts of those who directly abused the glitch.
If you are concerned that the purity of your account might be ruined by these illegitimate gains, you can contact us of your own free will and I should be able to tell you how your specific account benefitted, so that you can voluntarily give up your contraband. But if you are not worried about it, we aren't worried about it either. We have evaluated that the economic impact of the dupes that occurred on all worlds was not significant.
The accounts of all the dupers were all banned, and their alts were banned too, on all servers. That has been our policy whenever someone either bug abuses or bots on RSC Preservation. Banning these accounts has also removed significant amounts of wealth and rares from the server.
Of course, we also fixed the underlying bug detailed in #3322, in pull request !3574. You can read the full changelog there, but essentially, it is just doing exactly what Chomp advised: only processing a logout at the end of a tick. Doing that fixes all duping related bugs, including the party hat dupe, the cannon base dupe, the arbitrary item batch-drop dupe on Cabbage, as well as countless other ways to abuse the same bug.
As a result of all this, we will be better prepared in the future for critical bugs. First of all, we do have the PCAP parsing tool now, which I imagine can be used for more general anti-cheat as well. Ken, our newest administrator, created wealth monitoring scripts that will alert us if rare or valuable items are duplicated much faster.
As for the economic impact of not rolling anything back, we have meticulously inspected historical quantity records of over 70 most desired items, and determined that Chomp's estimated damage on the RSC Uranium server was not actually accurate. Month-to-month going back to April 2022 when the dupe was discovered, the only items that were duplicated were Party hats. The quantity of (crackers + party hats) increased by 47% on that server, only really starting around 24 January 2023, when the duping scripts were released publicly. Since the timespan during which the dupe was exploited was actually quite small, we were actually able to clean it all up, and there is no longer any increase in duped party hats on that server. Additionally, gold is not significantly inflated from months prior. No other items were duplicated.
The RSC Preservation server also did not have anything duplicated except for the relatively small amount gold coins and party hats. We are actually going to allow the party hats to continue to exist in game, since most of the ones that still exist are ones that were given away at drop parties, which new players really enjoyed. The total number of (crackers + party hats) on the server increased by a modest 20%.
RSC Cabbage got cleaned up, with the extra Dragon 2 Handers being confiscated by Mod Aurora. RSC Coleslaw and 2001scape were determined to be unaffected.
684 accounts were banned on the RSC Uranium server. 37 accounts were banned on the RSC Preservation server. 3 were banned from Cabbage. 2 were banned from Coleslaw. 4 were banned from 2001scape.
Thank you all for your patience and understanding over the past few days. I personally do not hold any hard feelings against any of the dupers, (other than insisting their accounts are banned for their actions), so please do not harass them.
If you're reading Chomp, do poke in again and help audit our LoginServer when we finally get it released. I have a feeling that a LoginServer is quite error-prone, and that your expertise would help. It would be a great way to make up for the trouble you caused us...!
We're relaunching the servers now. All of us on the Open RSC team are looking forward to seeing players in-game again soon!
— Logg, aka Mod L