What we did on our holidays: the AO3 and Yuletide

As most of our users will know, the Archive of Our Own has been playing host to the popular multifandom challenge Yuletide Treasure, one of the highlights of the fannish party season. There’ve been a few metaphorical spilled drinks and broken glasses along the way, but we’ve been having a blast! There’s been a lot of hard work from everyone on our team, and a lot of great new stories in the Archive.

Yuletide was our test case for our new Collections code, and was also an opportunity for us to give the servers a real workout. We’d already had the opportunity to see how the Archive performed under high demand during the first days of Open Beta, but the high volume and short time period of the Yuletide challenge – c. 2000 users joining, posting and browsing in the space of less than a week – took things to a whole new level. The exercise highlighted a lot of the things we needed to work on – although we knew we needed to optimise the code and improve some processes, having the real world usage made it possible for us to see exactly where the fixes needed to go. This did result in quite a few hiccups and slowdowns – at peak usage browsing the Archive has been an extremely slow process – and we appreciate our users’ patience while we work on these issues.

Our coders and Systems team have worked miracles in the last week, tracking and fixing bugs and working to improve performance. You may have noticed the Archive going down a few times over the past week – while one of these outages was due to server overload, all the rest were maintenance periods for us to deploy performance fixes. Some of the key things we deployed included:

  • A host of bugfixes for Collections code, including some work strengthening anonymity and ensuring gift recipients got their gifts.
  • Lots of refactoring and improvements for the ‘Post New’ form to make it load much, much faster. If you’ve experienced problems with the form freezing up or crashing your browser in the past, this should now be fixed.
  • Some caching for the Archive so that loading pages doesn’t make a whole lot of database calls every time. This is something we’ve been working on for a while, but we were having problems perfecting it. It’s still not perfect, but we decided the performance benefits WAY outweighed the remaining costs.
  • Behind the scenes, we added lots of indexes and other database improvements to speed everything up.

In the past week we’ve gone from revision 1980 to revision 2020 – a BIG jump! Massive, massive thanks go to our coders Elz and Naomi, and our Systems admin Sidra, who have worked on through sleepless nights and holiday commitments and flu to fix bugs, make improvements and keep things working. Sidra has appeared like a ministering angel at every server hiccup and coaxed the Archive back to life – she is truly amazing.

In the course of all this activity, we’ve also received a huge amount of feedback from users. In the past week, we’ve received more than 200 support requests, and our Support team have done an amazing job of keeping up with the volume. It’s testament to their hard work and dedication that at the time of writing, almost 90% of all support requests have been resolved, and most of the rest are in progress. Special mentions go to Cesy and Cal, who have done the work of ten women. We do our best to answer support requests within a few days – and much faster for urgent requests – but please bear with us if there’s a delay.

The Archive still needs a lot of refactoring and refining: at peak usage times (typically the evening hours in North America) it’s still running very slowly indeed. The great news is that our servers themselves can take the strain – we just need to improve our code and our setup so that they’re used to their full capacity. We ask for your patience while we work to make it even better – please refrain from refreshing multiple times if a page is slow, and if you’re able to browse at quieter times then you’ll be helping us out (and getting a nice fast Archive *g*). Remember that you can keep track of site status via our Twitter feed AO3_Status, which you can also follow on the front page of transformativeworks.org or via the feeds on AO3_status @ Livejournal, AO3_status @ Dreamwidth or AO3_status @ Insanejournal.

Some stats for the period 18th Dec – 26th Dec 2009:

  • Number of new users added: 1,566
  • Number of new fandoms added: 1,188
  • Number of new works posted: 8,012
  • Number of code issues committed: 29
  • Number of deploys (new code pushed to the Archive): 7
  • Number of support requests raised: 260
  • Number of support requests solved: 229

As you can see from the above, a phenomenal amount of work has gone on. The Archive of Our Own is entirely run and supported by volunteers. If you’ve had a great experience, or you have ideas you’d like to contribute, or you’re being driven mad by the imperfections, please consider volunteering! We would love to welcome more people to work on support, coding and testing – no experience required in any of these areas, just enthusiasm and a willingness to learn. If you’re interested, get in touch via our Volunteers and recruiting webform letting us know what you think you might enjoy working on. If you would like to support the Archive for the future, you might also consider making a donation to the Organization for Transformative Works – keeping us on a sound financial footing is important so we can continue to purchase the hardware and technical support that we need.

We’d like to say a huge thank you to:

  • Sidra and the Systems team for keeping the servers running.
  • All the AO3 coders for their immense hard work and ingenuity.
  • All the AO3 testers who worked against the clock to test out new code and allow it to go to the live site.
  • The AO3 Support team for working like crazy to ensure users were helped as quickly as humanly possible.
  • All the AO3 tag wranglers who have worked so hard in such a limited amount of time to make sure all the new fandoms show up in the filters and listings.
  • Maia, AD&T chair, whose amazing leadership formed the team that made all this possible.
  • Yuletide mods astolat and elynross for their support, encouragement and great communication.
  • All the Yuletide participants for being so patient and encouraging during the upheaval of a move to a new site and the subsequent bugs (nice new bugs instead of the boring old ones on the original Yuletide site *g*)
  • All of the other Archive users who have been so patient and supportive while the site slowed to a crawl.

2009 has been a great year, and Yuletide has been an exciting and crazy way of rounding it off. We hope that this year’s experience will lay the foundation for an even better, stronger Archive, and that this will be the first of many challenges on the AO3. Although it’s been crazy hard work, we’ve had a lot of fun! All of us are fans, so when we hit problems we have a BIG incentive to work on them! Thanks again to everyone who joined our party!

Mirrored from an original post on the Archive of Our Own.