Archive of Our Own ^beta

From approximately 12 PM UTC on July 10, 2023, to 4 PM UTC on July 11, AO3 was largely unavailable due to a distributed denial-of-service (DDoS) attack. We've made several changes to the Archive's technical setup to face this threat.

The attacks haven't stopped yet, but we're doing everything we can to keep the Archive up with minimal disturbance. We don't know who is responsible for these attacks or the motivation behind them. An online group has claimed responsibility, but there is no reason to believe this claim. According to multiple cybersecurity experts, they are not a reliable source of information and they misrepresent both their affiliation and their motives. If you see any claims from this group, or from anyone else claiming they know who is responsible, we recommend treating them with skepticism. Furthermore, we urge you not to engage in hateful rhetoric against any groups they claim or are reported to represent.

We've been doing our best to provide quick updates as the situation unfolds on the AO3_Status Twitter and ao3org Tumblr. However, we know not everyone follows those accounts or can access these updates, so here's what you need to know:

• No Archive data has been compromised. You don't need to worry about your password or private information. However, if you still wish to do so, our FAQ has instructions for changing your password or updating the email associated with your account.
• We attempted several different mitigation and blocking methods and settings changes to counter the attack. These brought the site up intermittently, but were not enough to contain the attack. Eventually, we implemented Cloudflare's Under Attack mode as a temporary—and extremely effective—solution. Cloudflare is a service that provides added security between our servers and the internet. Under Attack mode is not meant to be a permanent part of the Archive's setup. All content remains on our servers.
• You may be seeing a Captcha challenge when you access an AO3 page. That happens so that the Archive can make sure you're a human and not a robot. We know it's annoying, and we're sorry! We also know some browsers and older devices aren't currently able to access AO3. These measures are temporary and will be reassessed once the attack has stopped.
• In addition to AO3, both the Organization for Transformative Works website and our donation form (which is hosted using a third-party service) were also targeted. We're working on bringing the websites back, too, but as our donations go through a third-party service, we can’t predict when donors will be able to access it.
• While our donation form was down, a scammer briefly tried to impersonate the @AO3_Status account on Twitter to get money from fans under false pretenses, but their account has now been suspended. Please be wary of any efforts to get donations for the OTW or its projects at this time, as other scammers may be at work. We can only accept donations through our website once the site can be accessed.
• For the time being, we have disabled the Support & Feedback form and the Policy Questions & Abuse Reports page. The latter in particular was being targeted with a huge influx of spam as part of the attack. That's why the form has an emergency "Sorry, you have been blocked" security warning. If you see it, please don't worry; you haven't actually been blocked!
• We have turned off invitation requests for new AO3 accounts as a precautionary measure against spammers. If you already have an invitation, you can still use it to create an account. If you're in the queue and waiting for an invitation, it will be a few days. We'll let you know on Twitter and Tumblr when we start sending invitations again.
• What can you do to help? Keep browsing AO3 normally. Kudos and bookmark each other's works as you usually do, publish new works and chapters, and leave plenty of comments! If the site is a bit slow to load, just try again. We'll continue to do everything we can to make sure AO3 works as smoothly as possible for everyone, but there may be the occasional hiccough while these attacks continue.

We're disabling comments on this post because this situation is still unfolding and taking up all the focus of our all-volunteer team, so we can't monitor or reply to comments here at this time.

However, please know that we see and appreciate all the supportive messages and cute gifs you've been sending in the replies to our posts and tweets in the past couple of days. We're incredibly grateful for your support!

Edited July 12, 2023, 19:20 UTC: Clarified which of Cloudflare's services is not intended to be a permanent part of the Archive's setup.

Update 16 July, 2023, 23:55 UTC: The ability to request an invite, the support form, and the abuse report form are all back online.

Update 25 July, 2023, 18:15 UTC: The donation form has been back online since July 14. (This update was delayed to ensure it would remain stable.)

On Thursday, May 21 (UTC), we'll be doing some server work that includes changing the IP address we use to send emails. As a result of this change, we're anticipating a large number of undelivered emails while email providers get used to our new IP address. To help smooth the transition, we're going to disable both the invitation queue and account creation for a few days.

We send over one million emails per day. With that many emails coming from a new IP address, it's likely some providers will treat the messages as spam at first. We want to make sure invitation and account activation emails don't get lost in the shuffle, leading to frustration for new users and extra work for our Support volunteers.

However, invitation and activation emails are not the only types of emails that may be affected. Other emails such as comment, kudos, and subscription notifications; challenge assignments; and copies of deleted works may also go undelivered beginning on Thursday, May 21. (Edit 08:55 UTC 19 May: Undelivered emails are rejected by your email provider and never make it to your inbox or spam folder.)

Unfortunately, we cannot resend any missing emails. Because of this, we strongly recommend that you do not delete works or send challenge assignments during this time.

We'll turn invitations and account creation back on once we've determined that most major email providers no longer consider us spam. Until then, what this means is:

• Effective as of this post, you will not be able to add your email address to our invitation queue until we turn the queue back on. Invitations will not be sent out, since the queue will be empty.
• Beginning Thursday, May 21, you will not be able to use an existing invitation to sign up until we re-enable account creation.
• A notification banner will be displayed on all AO3 pages as long as account creation is disabled.

Even once this server work is done, please keep in mind that emails may sometimes take up to 72 hours to reach you. (In certain cases, they may not be delivered at all.) Please allow a few days and check your spam folder before you contact our Support team about a lost email.

Updated at 02:25 UTC Thursday, May 28: Invitation requests and account creation are back on, but we're still experiencing delays and lost emails with some providers, notably Yahoo and AOL. We've reached out to Yahoo multiple times at their request, but were unable to obtain any help in resolving this issue or information about when they'll start accepting our emails again. Therefore, if you're still not receiving emails from the Archive, you may want to consider changing the email associated with your AO3 account. (Depending on your provider, you may be able to set up your new address to forward messages to your old email.)

Updated at 12:22 UTC Wednesday, July 15: To the best of our knowledge, any remaining problems with certain email providers have been resolved. As always, please check your spam folder if you are waiting for a notification from the Archive, and allow up to 24 hours for delivery, as some delays are expected with a number of providers.

As people get ready to celebrate the end of 2017 in one way or another, we'd like to thank our users for their patience and supportive comments as we navigated downtimes, spam problems, and bumpy infrastructure upgrades together. We accomplished a lot of what we had hoped to tackle this year, and added a lot more to the to-do list for next year. Thanks for sticking with us!

As you might know, we've had to disable invite requests for existing users (due to abuse by spammers) and decrease the number of invites we automatically send out from our queue (ditto). As a result, people have had to wait to create an AO3 account for longer than we'd like. So, for the holidays, we're giving 1 shareable invite to every existing account that:

• currently doesn't have any invites, and
• is older than half a year, and
• has left at least 10 comments, or posted at least 1 work

(Sorry, we had to ensure we don't accidentally let the spammers invite all their spammer friends, so some restrictions apply.)

Check out our FAQ (available in a whole lot of languages) to read up on how to send an invite. You can either email the invite code, or copy-paste the code to share it with people through other means. In that case, our FAQ contains some information on how to use an invite code to create an account.

Happy gifting!

Last weekend, we had to disable new invitation requests to address an influx of accounts flooding the Archive with spam works. While our Abuse team has been banning these accounts and deleting thousands of spam works, the problem persists and would most likely get worse if we sent out invitations again.

We have decided to keep the invitation queue closed for the time being while we take steps to prevent spam from being posted in the first place. This means you will not be able to create an account unless you have previously received an invitation from either a friend or our automated queue. (If you requested an invitation before October 22 and have not received it, please check your spam folder and, if you use Gmail, your "Social" tab. If you are still unable to find your invitation, you can contact Support with your specific request.)

We very much regret denying invitations to legitimate users, but as the amount of spam being posted is affecting everyone's user experience, we currently see no other way to address the problem.

We will reopen invitation requests as soon as we can, although we do not have an estimated date at this time. When requests have been reenabled, the "Get Invited!" link will return to the homepage, and the Invitation Requests page will include a form to add yourself to waiting list. (The option to request invite codes for friends has been disabled since the last spam wave, and we have no plans to bring it back in the foreseeable future.)

Any updates will be provided on this post and our @AO3_Status Twitter account. For more information on the Archive's invitation system, refer to our Invitations FAQ.

We're currently experiencing an influx of spammers who have been creating bogus works and collections to link to their fare. They've become highly adept at using Archive features, and they've been flooding our invite queue with throwaway email addresses to create new accounts. This keeps our Abuse team busy around the clock, deleting spam works as they pop up and trying to weed out obvious spam email addresses before invites are sent out every day. It also prolongs the wait time for everyone else who wants to join the Archive. Our wait list is inching ever closer to 20,000, meaning legitimate users have to wait almost three weeks to receive an invitation email.

As a short-term measure, we've decided to turn off the invite queue for a week, so we can relieve some of the burden on our Abuse team, discuss technical solutions to the problem, and implement a quick fix or two to help with the worst attacks.

If you are a current user, you can check your Invitations page to see if you have any old invites waiting to be sent to a friend or fellow fan.

We are sorry for the long wait times, and we're doing our best to come back soon and get invites out quicker to those currently waiting!

Update on October 23, 11:23 UTC: People who are currently waiting for an invitation should still receive an email while the queue is under review. If you think you should have received an invitation, please wait another day or two, check your spam folder or "Social" tab in Gmail, and use our look-up tool to see if you're still in the queue. If you're sure you should have received an invitation and didn't, you can contact our Support team.

Update on October 30, 23:08 UTC: Please refer to our post "Update: Invitation requests remain disabled for the time being" for the latest information regarding invitations.

This batch of code includes several new features and enhancements aimed at assisting the Abuse team in their work, as well as some changes under the hood to protect us against attacks from download bots while keeping disruptions for our users to a minimum.

When we announced that we'd be suspending our automated invitation queue at the beginning of the month, we didn't plan for it to be closed quite this long! Writing, testing, and improving the new code, while also working with Abuse and taking care of our servers through planned and unplanned downtimes, took considerably longer than we expected.

While the queue was closed, we continued to provide invitation codes to existing users upon request. According to our records, we handed out over 5000 such invitations this month, which is more than three times our usual number. To everyone who reached out to friends and strangers to share invitations, we thank you!

If you wanted to create an account in January and couldn't - we're very sorry for making you wait! The queue is now back in business: request an invitation today and receive a code in 48 hours or less!

Credits

• Coders: Elz, Enigel, james_, Sarken, Scott
• Code reviewers: Elz, Enigel, james_, Sarken, Scott
• Testers: Lady Oscar, mumble, Sarken

Details

Admin

• In order to combat spam works (e.g., works consisting solely of advertisements), our Abuse team will now be automatically notified of accounts with a suspicious level of activity or works with suspicious content, allowing them to investigate before the problem gets out of hand.
• When our Abuse team hides a work that is under investigation, the creator(s) of the work will now receive an automatic email, letting them know the work has been hidden intentionally and not as the result of a bug. The email also contains a link to the work so that they (but no one else) can access it while it is hidden.
• Users whose accounts have been permanently suspended (banned) are not allowed to create new works, but it was still possible for other users to list them as co-authors. Now they cannot be added as co-authors unless the ban is lifted.
• When a user's account is temporarily suspended, the error message they receive when attempting to post, comment, or perform other actions will now let them know when they can expect their suspension to be lifted.
• Attempting to post, comment, or perform other actions using a suspended account previously resulted in an error message that said, "Please contact us for more information." The message has been clarified to say, "Please contact Abuse," and now provides a link to the appropriate contact form.
• When an admin deleted an invitation request submitted through our automated queue (e.g., because the requester was likely to be a known spammer), they would be redirected to the first page of the list, which was annoying if they needed to delete more than one request on the same page. Now they will be returned to the page they were on.

Downloads

• We were receiving reports from users whose IP addresses were blocked from accessing the Archive for downloading too many works too quickly, even though they hadn't done anything wrong. This would happen, for example, if their browser tried to download a file many times, despite the user having only clicked the download button once. We have added code and tweaked server settings to make this much less likely. In particular, we are now serving cached copies of downloads to users, which are currently refreshed on the server as soon as a work is updated.
• During site-wide downtime because of an overwhelmed server, users were receiving an error page that incorrectly stated they were downloading works too quickly. We've corrected the error page we give when the Archive is down and also added a dedicated error page to let users know when they are posting works too quickly.

Known Issues

See our Known Issues page for current issues.

(Please note that while it looks like we skipped several version numbers since our last batch of updates (Release 0.9.41), you haven't missed out on any new code! The jump is due to a few test deploys to get our deploy script into shape after adding another server to our line-up.)

To combat an influx of spam works, we are temporarily suspending the issuing of invitations from our automated queue. This will prevent spammers from getting invitations to create new accounts and give our all-volunteer teams time to clean up existing spam accounts and works. We will keep you updated about further developments on our Twitter account. Please read on for details.

The problem

We have been dealing with two issues affecting the Archive, both in terms of server health and user experience.

• Spammers who sign up for accounts only to post thousands of fake "works" (various kinds of advertisements) with the help of automated scripts.
• People who use bots to download works in bulk, to the point where it affects site speed and server uptime for everyone else.

Measures we've taken so far

We have been trying several things to keep both problems in check:

• The Abuse team has been manually banning accounts that post spam.
• We are also keeping an eye on the invitation queue for email addresses that follow discernible patterns and removing them from the queue. This is getting trickier as the spammers adjust.
• We delete the bulk of spam works from the database directly, as individual work deletion would clearly be an overwhelming task for the Abuse team; however, this requires people with the necessary skills and access to be available.
• Our volunteer sysadmin has been setting up various server scripts and settings aimed at catching spammers and download bots before they can do too much damage. This requires a lot of tweaking to adjust to new bots and prevent real users from being banned.

Much of this has cut into our volunteers' holiday time, and we extend heartfelt thanks to everyone who's been chipping in to keep the Archive going through our busiest days.

What we're doing now

Our Abuse team needs a chance to catch up on all reported spamming accounts and make sure that all spam works are deleted. Currently the spammers are creating new accounts faster than we can ban them. Our sysadmins and coders need some time to come up with a sustainable solution to prevent further bot attacks.

To that end, we're temporarily suspending issuing invites from our automated queue. Existing account holders can still request invite codes and share them with friends. You can use existing invites to sign up for an account; account creation itself will not be affected. (Please note: Requests for invite codes have to be manually approved by a site admin, so there might be a delay of two to three days before you receive them; challenge moderators can contact Support for invites if their project is about to open.)

We are working hard to get these problems under control, so the invite queue should be back in business soon! Thank you for your patience as we work through the issues.

What you can do

There are some things you can do to help:

• When downloading multiple works, wait a few moments between each download. If you're downloading too many works at once, you will be taken to an error page warning you to slow down or risk being blocked from accessing the Archive for 24 hours.
• Please don't report spam works. While we appreciate all the reports we've received so far, we now have a system in place that allows us to find spam quickly. Responding to reports of spam takes time away from dealing with it.
• Keep an eye on our Twitter account, @AO3_Status, for updates!

Known problems with the automated download limit

We have been getting reports of users who run into a message about excessive downloads even if they were downloading only a few works, or none at all. This may happen for several reasons that are unfortunately beyond our control:

• They pressed the download button once, but their device went on a rampage trying to download the file many times. A possible cause for this might be a download accelerator, so try disabling any relevant browser extensions or software, or try downloading works in another browser or device.
• They share an IP address with a group of people, one of whom hit the current download limit and got everyone else with the same IP address banned as well. This can be caused by VPNs, Tor software, or an ISP who assigns the same IP address to a group of customers (more likely to happen on phones). Please try using a different device, if you can.

We apologize if you have to deal with any of these and we'll do our best to restore proper access for all users as soon as possible!

San Diego Comic Con (SDCC) is a major multifandom event taking place each July; this year it's July 18-21. For the first time, the OTW will be covering SDCC, from the convention floor to Hall H lines, from the fan panels to tv, movie and author press rooms. Legal Committee staffer Heidi Tandy will be focusing on legal aspects of fandom, fannish interaction with content creators and other issues of interest to all fans, including fanfic writers, fanartists and vidders.

In addition, the OTW is hosting a party on Wednesday, July 17, 8:00-9:30 p.m. PDT at the Tequila Bar & Grille at the San Diego Marriott Marquis & Marina (333 West Harbor Drive, San Diego, CA 92101 map and directions here. There will be complimentary margaritas, sodas, chips & salsa, a few rounds of Cards Against Humanity, giveaways and other meet & greet moments designed to welcome everyone to San Diego and Comic Con.

We're requesting a voluntary donation of $5 to attend. You don't need to be attending Comic Con to join us, although the Marriott is adjacent to the Convention Center so anyone coming from the off-site SDCC hotels via Con Bus can reach it easily.

We'd also like to know what SDCC participants you would like Heidi to speak to and what questions you would like her to ask? The ComicCon schedule will become available around July 4, and we will send out another reminder after it is posted.

Let us hear from you! Just keep in mind that Heidi can only be in one place at one time, and that she can speak with only so many people in a single day. She is also scheduled to appear on two panels during the con. The first is a panel for the forthcoming SmartPop book Fic: Why Fanfiction Is Taking Over the World, for which she and other current or former OTW staffers have contributed. Heidi will also be moderating a Harry Potter panel on Sunday afternoon. However, we would like to include as many of your suggestions as possible.

Some planned questions currently include:

• Have you heard of or planned anything for your property to be part of Amazon's Kindle Worlds project?
• How involved are you with tie-in creations generally, and do you see fan work to be different?
• How would you have answered this question 3 years ago? What about 8?
• How much regular contact do you have with legal staff in your work regarding fan creations or other things besides your own content?
• For fans: Have you ever received a C&D? What did you do? What would you do if you got one now?

We will be publishing stories from her SDCC visit in the week after the event (starting after July 25) in both print and video form - and she'll liveTweet as much as possible from the halls of the San Diego Convention Center through the OTW News twitter account.

Mirrored from an original post on the OTW blog. Find related news by viewing our tag cloud.