AO3 performance issues

As pretty much all users of the Archive of Our Own have no doubt noticed, we’ve been experiencing some problems with Archive loads: slowdowns and the appearance of the dreaded 502 page have become a regular occurrence. We’re working on addressing these issues, but it’s taking longer than we’d like, so we wanted to update you on what’s going on.

Why the slowdowns?

Mostly because there’s so much demand! The number of people reading and posting now is overwhelming – we’re glad so many people want to be here, but sorry that the rapid expansion of the site is making it less functional than it should be.

We now get over a million and a half pageviews on an average day, often clustered at peak times in the evening (particularly when folks in the Western Hemisphere are home from work and school) – we were using a self-hosted analytics system to monitor site traffic, and we had to disable it because it was too overloaded to keep up. The traffic places high demands on our servers, and you see the 502 errors when the systems are getting more requests than they can handle. Ultimately we’ll need to buy more servers to cope with rising demand, but there’s ongoing work that we’ve done and need to continue to do to make our code more efficient. We’ve been working on long-term plans to improve our work and bookmark searching and browsing, since those are the pages that get the most traffic; right now, they present some challenges because they were designed and built when the site was much smaller. We’ve learned a lot about scaling over the years, but rewriting different areas of the code takes some time!

What are you doing to fix it?

Our Systems team are making some adjustments to our server setup and databases. Their first action was to increase the amount of tmp space for our MySQL database on the server – this has alleviated some of the worst problems, but doesn’t really get at the underlying issues. They’re continuing to investigate to see if there are additional adjustments we can make to the servers to help with the problems.

We’re also actively working on the searching and browsing code: that’s been a big project, and it will hopefully make a significant impact. Because it affects a lot of crucial areas of the site, we want to make sure we get everything right and do as much testing as we can to ensure that performance is where it needs to be before we release it. We’re switching from the Sphinx search engine to elasticsearch, which can index new records more rapidly, allowing us to use that for filtering. That will offer us more flexibility, get rid of some of our slower SQL queries, and take some pressure off our main database, and it also has some nice sharding/scaling capabilities built in.

We also try to cache as much data as we can, and that’s something we’re always looking to improve on. Systems and AD&T have discussed different options there, and we’ll be continuing to work on small improvements and see what larger ones we may be able to incorporate.

When will it be fixed?

It’s going to take us a few weeks to get through all the changes that we need to make. Our next code deploy will probably be within the next week – that will include email bundling of subscription and kudos notifications, so that we can scale our sending of emails better as well. After that, we’ll be able to dedicate our resources to testing the search and browsing changes, and we’re hoping to have that out to everyone by the end of June. We rely on volunteer time for coding and testing, so we need to schedule our work for evenings and weekends for the most part, but we’re highly motivated to resolve the current problems, and we’ll do our best to get the changes out to you as soon as we can.

Improving the Archive is an ongoing task, and after we’ve made the changes to search and browse we’ll be continuing to work on other areas of the site to enable better scalability. We’re currently investigating the best options for developing the site going forward, including the possibility of paying for some training and/or expert advice to cover areas our existing volunteers don’t have much experience with. (If you have experience in these areas and time to work closely with our teams, we’d also welcome more volunteers!)

Thanks for your patience!

We know it’s really annoying and frustrating when the site isn’t working properly. We are working hard to fix it! We really appreciate the support of all our users. ♥

Mirrored from an original post on the Archive of Our Own.

30 thoughts to “AO3 performance issues”

  1. It sucks that the demand is having such an impact, but it is amazing that the demand is there. I hope you know it’s because AO3 is a much better platform than certain other unnamed fanfiction websites! I think more and more people are realizing they are better off here.
    Really appreciate the work you guys do. I know this is expensive stuff. I’ll keep encouraging my fandom friends to become financial supporters because we really appreciate you. We doooo!

    1. Thank you! We really appreciate your support.

      Lucy
      OTW Communications

  2. I know that I am personally responsible for a portion of the 502s, as I am trying to read every single Clint/Coulson story on the archive. I’d promise to stop, but I’d be lying. Your efforts are appreciated.

      1. LOL, and every other fangirl who’s just seen Avengers, the TeenWolf 2nd season premier tonight is not going to improve the sitution either XD

    1. Thanks! We appreciate your support (and we very much understand the drive to read ALL THE FICS).

      Lucy
      OTW Communications

  3. I love Ao3, despite it’s server issues. Any other site that gave me this much trouble , I would probably curse and never come back to. But you have such quality fic, I just keep coming back. If I had any money, I’d donate towards getting a new server. Maybe in a few years. Good luck

  4. All your hard work and effort is appreciated by all of us! Thank you so much for working so diligently! <3

  5. No one likes slow-downs, but it’s good to know that it’s being addressed. This page is awesome, and it’s worth waiting for the updates to get finished. I’m a financial supporter, and I know it’s worth the money 😉

  6. Yeah, apparently ffn is starting to be stupid about explicit fics so you are probably going to get even more people jumping ship over to AO3. I think AO3 is wonderful and definitely worth it.
    -Ellipsis aka …

    1. Very true, I was just thinking about that. It’s absolutely revolting if ffn does indeed do that (I have such a thing against censorship- it makes my blood boil!) but AO3 is infinitely better in any case, so good riddance!

    2. Thanks, we appreciate your support!

      It’s certainly true that the recent changes at Fanfiction.net have resulted in more demand for accounts here – we’re only sorry that the expansion is causing a few logistical challenges! We’re glad to be able to offer people an alternative, though.

      Lucy
      OTW Communications

  7. Hey – would donating help/is money for making the fixes go faster part of the issue? And if so, is there an off-AO3 legit place for folks to do that if they want to help support you guys?

    Thanks for all the hard work! Seriously, the AO3 is a wonderful place and we know how much y’all put into it.

    1. We have a few more details on http://archiveofourown.org/donate. You can donate at https://transformativeworks.org/how-you-can-help/support (The OTW is the parent organisation of the AO3), but the main thing we need is more volunteers. Our volunteering form is https://transformativeworks.org/volunteer/volunteer-interest-form (closed at the moment, but do join the waiting list), or if you’re interested in programming, our code is at github.com/otwcode/otwarchive/, bugs are at code.google.com/p/otwarchive/issues/list, and you can learn the basics at tryruby.org and railsforzombies.org while you’re waiting for the volunteers form to reopen.

  8. Thanks for all your hard work! I really glad to see that the team is taking this seriously and doing their best to solved the problems. I love AO3 and all it adds to my fannish life. But, I’m a little surprised and curious about how code deploys are being prioritized? Because as far as I’m aware, the 502 errors have been being a problem increasingly for quite a while now, as the archive continues to grow with leaps and bounds! (That is seriously awesome! \o/) So, IDK, it seems that fixing this underlying instability / problem might be better as a code priority rather than something like stats which would just put more load on the servers? They are very cool, but they do seem a bit like chocolate sprinkes on my fannish sundae, while all the ice-cream is gone. 🙁

    I’m really excited to see the new Searching Function you guys whip up, though! Thanks once again for your hard work, but heh, I’m just a bit curious about the… rough timeline or order, rather, AO3 is looking at, wrt its issues. I’m getting a little interested in volunteering as I might be up for some free-time soon, but I like to know what I’m getting into. 😉

    1. Stats are actually decreasing the load on the servers – putting it all onto one page and grouping the emails together has helped performance. Before this change, the number of emails being sent out every hour was adding a fair amount of load to the servers.

      Timelines are always very much a draft/work-in-progress, as everything is dependent on volunteers, so we try not to commit to any dates in public. Our volunteering form is https://transformativeworks.org/volunteer/volunteer-interest-form (closed at the moment, but do join the waiting list), or if you’re interested in programming, our code is at github.com/otwcode/otwarchive/, bugs are at code.google.com/p/otwarchive/issues/list, and you can learn the basics at tryruby.org and railsforzombies.org while you’re waiting for the volunteers form to reopen. The bugs are labelled by priority, but our top priority is performance issues at the moment.

      1. *goes to get familiar*
        Here’s hoping the volunteer spots open up soon-ish. Looking forward to that! 😀

  9. I saw another comment to this effect too. Is there some way we can make donations to help with buying those servers? because you guys continually rock my socks right off, and I would love to help.

    1. We have a few more details on http://archiveofourown.org/donate. You can donate at https://transformativeworks.org/how-you-can-help/support (The OTW is the parent organisation of the AO3), but the main thing we need is more volunteers. Our volunteering form is https://transformativeworks.org/volunteer/volunteer-interest-form (closed at the moment, but do join the waiting list), or if you’re interested in programming, our code is at github.com/otwcode/otwarchive/, bugs are at code.google.com/p/otwarchive/issues/list, and you can learn the basics at tryruby.org and railsforzombies.org while you’re waiting for the volunteers form to reopen.

      Basically, more servers will help a little, and we have enough money from the last drive to buy those already, but code changes will make a much bigger difference.

  10. Thanks so much for all your hard work, guys. This is such a great site, and while the 502s are frustrating, I know you’re doing everything possible to keep up with the increasing demand. Thanks also for letting us know exactly what’s going on, and what you’re doing to address the issues. As a couple other people have said, I would be very happy to toss a few bucks in the hat to pay for the new servers. Where can I do that? I can’t seem to find the donate-button on the site anymore…

    1. Oh for heaven’s sake…you’d think I’d have noticed the huge read “donate” button up above, wouldn’t you? Awkward! Disregard that part of my comment, please. It’s just been a long day!

  11. Just wanted to add my voice to everyone else and say “thanks” for not just the hard work that you guys put into this, but your support as well! Every time I see the dreaded 502 I swear, but then realize that it’s just because there’s so many people out there who are involved in the same things I am and that’s FANTASTIC.
    Seriously, keep up the good work.
    Ganbatte!

  12. Thank you for explaining the problem in such a detailed way. I guess it’s a double-edged sword, but what an amazing problem to have!

  13. Thanks for the hard work…I totally understand and will be patient as you guys work this out. Great job so far. The site is awesome, even with the 502’s! XD

  14. I just wanted to add to everyone who’s thanking you for the great work you do keeping the archive up and running. Obviously slowdowns and 502s are annoying, but that’s not your fault in any way. I love AO3 as an archive, and I htink it’s great that it’s becoming more popular. And we all know you’re trying your hardest to get this fixed quickly 🙂 Keep up the good work guys! <3

  15. If you are over 18, AdultFanFiction(dot)Net would love to invite any authors or readers to join! We have the largest adult (and even some non adult) fan fiction archive on the internet! We’ve been around for over 10 years and aren’t experiencing the same growing pains that AO3 is apparently having with the huge rush from FanFiction wiping their adult content! We’ve had over 2000 members from FF already start moving their stories over with out any server errors, lag time or other issues 🙂

    Feel free to stop on by, we’d love to see you! Come find your favorite stories and authors as they settle in to their new home 🙂

Comments are closed.