moot:

Waking up to a string of missed calls is rarely a harbinger of good, and this time would prove no different. Upon returning the calls I was greeted with a simple “You’ve been hacked.” Great.

It turned out someone with my username had spent the morning causing a ruckus on 4chan. At first other users and moderators assumed I’d been drinking—at 7:00AM no less—but quickly concluded it was the work an intruder.

I immediately hopped out of bed and onto a call with our developer and a moderator to establish a timeline of events, and set about digging through error logs. It didn’t take long to piece together what had happened.

First, the intruder was able to enumerate files on a domain hosting moderator tools. He used an off-the-shelf script, and had the benefit of having leaked source code in his possession to build a site-specific wordlist.

Mistake #1: No rate limiting or HTTP auth dialog was present on the domain.

Most of the enumerated files did him no good since they made use of a PHP auth check, but one file was different: instead of displaying an error message, it simply bounced him to our front page. He manually set a 4chan admin username cookie (the cookie name gleaned from the source code), and voila, he was no longer redirected.

Mistake #2: The PHP auth check for this particular file was broken.

The page in question was a once-off used to generate statistics about reported posts. “0 days” was displayed on the page, and he guessed it might be a query parameter. He was correct, and as it turned out this one parameter was vulnerable to SQL injection. After some more testing, he realized he was able to return information from our database using the error messages displayed.

Mistake #3: Unescaped SQL query, and not disabling MySQL errors in production.

Using the injection vulnerability, he was able to exfiltrate information from our backend database. Thankfully we have a record of all information accessed (thanks to those handy error messages being logged), and it appears he mostly poked around before settling on accessing moderator credentials. He took my username and password, set them in his cookies, confirmed they worked by visiting an admin panel on the main site, and began lurking.

Mistake #4: Boneheaded cookie auth—we simply stored the bcrypted password from the database in a cookie, which was all that was required to pass PHP auth.

After lurking for a few days, the intruder decided to make himself known by publicly shaming a user under my username. He later stated this was his motivation for gaining entry all along—to out a user he disliked. Using admin panels, he was able to obtain and leak information about other users, as well as moderator names and IP addresses.

We quickly patched the vulnerability, forced a password reset for all moderator accounts, and have spent the past two weeks reviewing our code and servers to apply fixes and make improvements where possible.

The day didn’t end there though. This next bit concerns DrawQuest.

I loaded my e-mail for the first time that morning, and at the top of the list was a message from Amazon: “Unauthorized activity in your AWS account”

To make an already long story short, when leaving the company a few months ago, one of our developers asked to open source a project he had worked on. You can probably already guess where this is going, but suffice it to say he did so by flipping the existing repository to public.

Despite having overwritten our AWS keys, they were still available in the commit history and likely recovered by a bot. An intruder was able to use them to create an administrator account, giving them access to our AWS dashboard, and used it to spin up a hundred extra-large instances—probably for Bitcoin mining. They also would have had access to all of our S3 buckets, EC2 and RDS instances, DNS, etc.

Mistake #5: Not creating a fresh repo for a newly open sourced project, or at least scrubbing commit history.

Mistake #6: Using a highly-privileged key where a lesser-privileged one would have sufficed.

For DrawQuest, this was catastrophic. Despite our team going their separate ways a few months ago, we’d hoped to continue operating the app using the company’s remaining capital. However without any full-time employees left on staff, after the breach we felt the only responsible path forward was to shutter the service completely since we lack the time and resources to do a complete audit in order to ensure the integrity of our servers and Amazon account.

And so last night, with a lump in my throat, I announced that we’d be shutting down DrawQuest—a beloved drawing community that still boasted 100,000 monthly active users.

It was a long day to say the least.

On one hand I’m frustrated we made such simple mistakes that resulted in very real consequences, but also grateful that it provided us an opportunity to learn from those mistakes, and share them with the world.

There’s no silver bullet when it comes to security, and the only way to stay ahead of it is constant vigilance. Don’t rely on any one method to protect your service, assume the methods you already have in place don’t work, adhere to best practices, and make it a point to revisit security on a regular basis—not just when something goes terribly wrong.

To that end and in keeping with our ongoing commitment to security, I’m pleased to announce the launch of 4chan’s Vulnerability Disclosure Program. It’s my hope that by embracing responsible disclosure and providing an officially sanctioned way for security researchers to submit such reports, we’ll be in a better position to avoid or at least mitigate future incidents.

In the end, I accept full responsibility for both breaches. I wasn’t ever involved on the technical side for DrawQuest and don’t actively write code for 4chan any longer, but know it was ultimately my responsibility as founder and CEO to ensure the security of both. I’m very sorry to both communities that I failed in that capacity.

If I’ve learned anything from this experience, it’s that if you don’t treat security as a top priority, it will bite you. It’s not a matter of if—only when.