Archive of Our Own

  • AO3 Connection Issues

    By .Lucy Pearson on Thursday, 23 August 2012 - 7:08pm
    Message type:
    Tags:

    We've been receiving a small number of reports of people unable to access the Archive of Our Own - if you've been affected by this issue, this post will give you a bit more information about what's going on. We'd also like to appeal for your help as we work to fix it!

    What's wrong

    We recently upgraded our firewall to improve the security of our servers. Unfortunately, it seems that we haven't got the configuration absolutely right and it's causing connection problems for some users. This problem is only affecting a small number of users, and it's not completely consistent. However, if you've received an Error 404, a warning saying 'Secure Connection Failed', or you've been redirected to a url with 8080 in it, then this is what's causing it.

    While we worked on the issue, we temporarily disabled https on the site, as that was causing some additional problems. This means that if you have a browser extension such as HTTPS Everywhere enabled, or you use a browser which enforces https by default, then the site will not load - apologies for this. If the site has been consistently timing out for you, it's worth checking if this applies in your case - if the url defaults to https://archiveofourown.org then you have been affected by this issue.

    How you can workaround

    If you're being affected by the https issue, you can work around by adding an exception to HTTPS Everywhere, or using a different browser.

    If you're getting errors at random, then clearing your browser cache and refreshing should help. You may also find it helps to use another browser.

    How you can help us

    We're working to get to the bottom of this problem, and we know we've already reduced the number of errors which are occurring. However, it would be enormously helpful for our Systems team to have a little more information. If you encounter an error, please submit a Support request giving the following information:

    Your IP address You can find this out by going to http://www.whatsmyip.org/.
    The url of the page where you got the error:
    The exact error you got: You may find it easiest to copy and paste the error. If you didn't get an error but the page just never loads, tell us that.
    What time (UTC) you got the error: Please check the current time in UTC when you get the error - this will make it easier for us to keep track, since we're dealing with users in lots of different timezones.
    Is the error intermittent or constant?
    What browser are you using? It would be extra helpful if you can tell us your user agent string, which you can find out by going to http://whatsmyuseragent.com/".

    If you know how to view the source of a page in the browser, it would also be very helpful if you could could copy and paste the source code of the page that throws up the problem.

    If you're comfortable working on the command line, then it would also be helpful if you could provide us with some additional information (if you're already wondering what we're talking about, don't worry, you can ignore this bit). Open up a command line window and type nslookup ao3.org. Copy whatever pops up in your console and paste it into your Support message.

    If you can't access the Archive at all (and thus can't submit a Support request there) you can send us this information via our backup Support form.

    A note on https

    We know that many people prefer to use https connection to provide additional security on the web, and we will be reenabling this option as soon as we can. Because the AO3 doesn't handle data such as credit card information or similar, browsing without https doesn't expose our users to any significant security risks. However, it is always a good policy to use a unique password (i.e. don't use the same username and password combo for the AO3 and your email account) in order to ensure that if for any reason someone else obtains your AO3 credentials, they can't use them to access other data). Apologies for the inconvenience to users while this option is disabled.

    ETA for a fix

    We're hoping to resolve these lingering problems asap; however, our Systems team have limited time, so we may not be able to track down the root of the problem as fast as we'd like. We'll keep you updated, and in the meantime apologise for the inconvenience.

    Mirrored from an original post on the Archive of Our Own.

  • Update: Site problems and our firewall upgrade

    By .Lucy Pearson on Friday, 17 August 2012 - 1:28pm
    Message type:
    Tags:

    Our Systems team have been doing some behind-the-scenes maintenance over the past week or so to improve the Archive of Our Own's firewalls. This has mostly been invisible to users, but last night it briefly gave everyone a fright when a typo introduced during maintenance caused some people to be redirected to some weird pages when trying to access the AO3. We also had a few additional problems today which caused a bit of site downtime. We've fixed the problems and the site should now be back to normal, but we wanted to give you all an explanation of what we've been working on and what caused the issues.

    Please note: We will be doing some more maintenance relating to these issues at c. 22:00 UTC today (see when this is in your timezone). The site should remain up, but will run slowly for a while.

    Upgrading our firewall

    The AO3's servers have some built-in firewalls which stop outside IP services accessing bits of the servers they shouldn't, in the same way that the firewall on your home computer protects you from malicious programmes modifying your computer. Until recently, we were using these firewalls, which meant that each server was behind its own firewall, and data passed between servers was unencrypted. However, now that we have a lot more machines (with different levels of firewall), this setup is not as secure as it could be. It also makes it difficult for us to do some of the Systems work we need to, since the firewalls get in the way. We've therefore been upgrading our firewall setup: it's better to put all the machines behind the same firewall so that data passing between different servers is always protected by the firewall.

    We've been slowly moving all our servers behind the new firewall. We're almost done with this work, which will put all the main servers for the Archive (that is the ones all on the same site together) behind the firewall. In addition, our remote servers (which can't go behind the firewall) will be connected to the firewall so that they can be sure they're talking to the right machine, and all the data sent to them is properly encrypted. (The remote servers are used for data backups - they are at a different location so that if one site is hit by a meteor, we'll still have our data.) This means that everything is more secure and that we can do further Systems maintenance without our own firewalls getting in the way.

    What went wrong - redirects

    Last night, some users started getting redirected to a different site when trying to access the AO3. The redirect site was serving up various types of spammy content, so we know this was very alarming for everyone who experienced it. The problem was caused by an error introduced during our maintenance. It was fixed very quickly, but we're very sorry to everyone who was affected.

    In order to understand what caused the bug, it's necessary to understand a little bit about DNS. Every address on the internet is actually a string of numbers (an IP address), but you usually access it via a much friendlier address like http://archiveofourown.org. DNS is a bit like a phonebook for the internet: when you go to http://archiveofourown.org, your Domain Name Service goes to look and see what number is listed for that address, then sends you to the right place. In the case of the AO3, we actually have several servers, so there are several 'phone numbers' listed and you can get sent to any one of those.

    As part of our maintenance, we had to make changes to our DNS configuration. Unfortunately, during one of those changes, we accidentally introduced a typo into one of our names (actually into the delegation of the domain, for those of you who are systems savvy). This meant that some people were being sent to the wrong place when they tried to access our address - it's as if the phone book had a misprint and you suddenly found yourself calling the laundry instead of a taxi service. Initially this was just sending people to a non-existent place, but a spammer noticed the error and registered that IP address so they would get the redirected traffic. (In the phone book analogy, the laundry noticed the misprint and quickly registered to use that phone number so they could take advantage of it.) It didn't affect everyone since some people were still being sent to the other, valid IP addresses.

    We fixed the typo as soon as the problem was reported. However, Domain Name Services don't update immediately, so some users were still getting sent to the wrong address for a few hours after we introduced the fix. To continue the phone book analogy, it's as if the misprinted phone book was still in circulation at the same time as the new, updated one.

    If you were affected by this issue, then it should be completely resolved now. Provided you didn't click any links on the site you were redirected to, you shouldn't have anything to worry about. However, it's a a good idea to run your antivirus programme just to be absolutely sure.

    Downtime today

    It turned out one bit of the firewall configuration was a little overenthusiastic and was blocking some users from getting to the site at all. We rolled back part of the changes, which caused a little bit of downtime. Because this involved changing our DNS configuration again the change took a while to take effect and the downtime was different for different users (effectively we changed our phone number, and the phonebook had to update).

    The site should be back up for everyone now. We'll be completing the last bits of work on the firewall upgrade today at roughly 22:00 UTC. At present we don't expect any downtime, but the site will be running more slowly than usual.

    Thank you

    We'd like to say a massive thank you to James_, who has done almost all of the work upgrading the firewall. He's done a sterling job and the site is much more secure because of his work. This glitch reminds us just how high pressure Systems' work is - for most of us, a tiny typo does not have such noticeable effects! We really appreciate all the work James_ has put in, and the speed at which he identified and fixed the problem when it went wrong.

    We'd also like to thank our other staff who swung into action to keep people informed on Twitter, our news sites, and via Support, and who provided moral support while the issues were being dealt with.

    Finally, thanks to all our users: you guys were super understanding while we were dealing with these problems and gave us lots of useful info which helped us track down the source of the bug.

    Reminder: site status information

    The first place to be updated when we have problems with the site is our Twitter AO3_Status. We try to answer questions addressed to us there as well as putting out general tweets, but it can be hard for us to keep up with direct conversations in busy periods, so apologies if you sent us a message and we didn't respond directly. If you see a problem, it's a good idea to check our timeline first to see if we already tweeted about it. For problems other than site status issues, the best place to go for help is AO3 Support.

    Mirrored from an original post on the Archive of Our Own. Feel free to comment there or here!

  • Status Update: AO3 is A-OK, Be Back Soon!!

    By .fcoppa on Thursday, 16 August 2012 - 9:15pm
    Message type:
    Tags:

    This is just a quick message to say that, due to a typo in our DNS, the Archive of Our Own is currently offline for some users. The error has already been fixed but it may take as long as 7 hours to propagate through the system. However we were not hacked or anything and we hope to be up as soon as possible. (Systems is telling us that this was a test of our communication and response teams...oh, wily wily Systems! Also typing is hard!)

    We are sorry for the inconvenience and for any alarm. Also, it would be best not to click any links or submit any personal information to any unfamiliar sites.

    Systems will have a fuller report on the problem soon for those who want the dirty details. Otherwise stay tuned and thank you for your patience!

  • Tag filtering and an apology

    By .Lucy Pearson on Wednesday, 15 August 2012 - 10:17am
    Message type:
    Tags:

    We'd like to apologise for the delay in returning tag filtering on the Archive of Our Own. We had hoped to have tag filtering updated, run through our testers, and back out to you by the end of July. However, it's taking us much longer than we expected to finish that code and get it all tested. We hope to have all the work finished and the tag filters returned by the end of September. However, we can't promise we won't run into more unexpected snags, so right now this is a very rough timetable and we may have to revise it. We'll keep people updated as the work progresses.

    We've been able to tackle a lot of other performance-related issues since the problems started in May/June. However, sadly this isn't enough to allow us to restore the old filter code, which just isn't able to cope with the number of users the site has now. It's been necessary to completely remove the old filter code and rewrite it from the ground up, which is a major piece of coding.

    We already knew that this code was nearing breaking point and were actively working on new tag filters before the performance problems hit, which is why we were optimistic about introducing the new tag filters quickly. However, coding is always an uncertain business, and it's taken longer than we expected to finish the new code. In writing a big piece of functionality like this, it's not too unusual to run up against challenges while coding that you didn't anticipate up front. In addition, we only have a limited number of coders who are able to take on something this complex: the coder who is taking the lead on this project has been working on it all available hours, but sadly we've yet to figure out how to clone her.

    In the meantime, if you've not already seen the Disabling filters: information and search tips post, check it out for alternate ways of finding works on the Archive.

    Once again, many apologies for the inconvenience. We thank you for your patience while we work hard to bring back tag filtering!

    Mirrored from an original post on the Archive of Our Own.

  • Archive of Our Own Newsletter - July 2012

    By .Lucy Pearson on Tuesday, 31 July 2012 - 3:54pm
    Message type:

    *coughs* *taps microphone* Is this thing on?

    Ahem, after something of a pause, this is (hopefully) the return of our regular AO3 newsletter! Your friendly AO3 news poster (that's me, Lucy!) started off the year with good intentions and a shiny new format. However, between writing various other special edition posts, co-chairing the Communications committee, and dealing with real life, somehow the newsletter kept slipping down the to-do list. The good news is that new Communications staffer Camden has volunteered to take the baton, so hopefully these newsletters will be making a more frequent appearance in the future. So, without further ado, a few updates on the major happenings in recent times.

    Major doings: performance, performance, performance

    For the last few months, all the AO3 teams have been working really hard dealing with the various performance issues on the Archive. These started to appear in around May and got really acute in June, before we got them under control with a lot of hard work and some emergency measures (which involved 5 code deploys in the space of a few weeks). There's been some heroic work from Coders, Testers and Systems to get the situation under control quickly: we're really grateful to everyone who pitched in and helped out. We're continuing to work on performance, most recently with a RAM upgrade and upgrades to our server software, and we'll keep users posted as new developments arise.

    Fanstravaganza!

    Our performance crunch was largely down to a gigantic increase in traffic: we had over 1.4 MILLION visitors in the month of June alone! Our invitations queue also continues to grow apace: we recently increased the number of invitations being issued each day to 300, but this is still not keeping up with demand. We're doing our best to keep expanding, but we can only increase accounts so fast while preserving site stability, so we're afraid the queue will probably remain long for a while. :(

    Tag filters are coming back - we promise!

    As an emergency performance measure, we disabled our tag filters. We know they're much missed, but their absence has enabled the site to run without 502 errors. We'd hoped to have the replacement filters finished by the end of July, but unfortunately they're taking a bit longer than we'd hoped (turns out we have to let coders leave the house now and again). Everyone is working super hard to get them coded, tested, and up on the site as soon as possible - we apologise for the inconvenience in the meantime.

    Tags and media categories

    Speaking of tags, the new Category Change workgroup recently got off the ground with their discussion of how we organise the media categories on our Fandoms page. This is an issue which we've been chewing over in-house for a really, really long time - we know lots of users find the current categories confusing and/or problematic, and we'd like to come up with a better way of handling them. However, this is a really big can of worms: for example, we know that "Anime & Manga" is problematic as a catch-all category including manhua, manhwa, etc., but we also know that lots of fans are used to looking for anime and manga and would be confused if we scrapped that terminology altogether. So, we're looking to find ways of handling the different categories which are understandable but also less problematic.

    The Category Change workgroup includes members from lots of relevant committees: Internationalization and Outreach, Tag Wranglers, Support, and Accessibility, Design and Technology. They will be discussing the various issues and a range of possible solutions.

    As part of this process they'll be seeking input from various groups inside and outside the organisation and using that feedback to help identify some possible solutions. They'll also be looking into the technical aspects – for example, improvements in searching and browsing on the Archive in general will open up more possibilities for how we handle categories specifically - so we don't wind up with a great theoretical solution that’s impossible to implement. Tag wranglers have already given some great ideas and thoughts on this issue - they've been discussing it on and off for at least two years - and the workgroup is really looking forward to moving ahead with it. Stay posted for more news on this!

    Tag wrangling - a new view!

    We've had lots of feedback from users who've said that they'd love to be able to get more information about how tags are wrangled and how they all relate to one another. This has also been one of the most frequent feature requests from tag wranglers, who put lots of work into making the tags link up in a meaningful way and would like their work to be more useful to users. Our awesome coder sarken took up the challenge and did some work to make the tags pages which are currently only visible to tag wranglers visible to any user. This will be an 'alpha' feature when it goes live - we expect we'll get lots of feedback about how it could be better - but we hope this news will be welcome to lots of people who have been wondering what's going on behind the scenes!

    For those who are really curious about tag wrangling, the Tag Wrangling Committee are also currently preparing a public version of the tag wrangling guidelines. We hope to have these posted shortly after the tags pages are made public.

    What else is up in the world of tags?

    The Archive welcomes all types of fanworks! We have plans to make it much more useful for different types of fanworks by adding some more browsing and posting features. However, right now, people use the "Additional Tags" field to make it clear what type of creative endeavour their particular work consists of. Tag Wranglers have wrangled the more common tags like podfic, fanart, fanvids and AMV, and even things like knitting and crochet patterns, and we're always excited to see people tagging for a diverse range of fanworks.

    But you might not be aware that the tags we've wrangled go further than just describing the media type and content. Once the filters are reenabled, you'll be able to use the additional tags we've made canonical to further filter those media types by format or length - click on one of the fanwork types linked above, and browse the list, or use the advanced search.

    For Fanart, there are tags that describe medium, such as Crayons, or Digital Art. For podfic and video, there are tags which describe the audio and video formats that it is available in, such as Video Format: AVI or Audio Format: MP3. For podfic specifically, there are tags which indicate the length of the recorded work, such as Podfic Length: 0-10 Minutes, through to Podfic Length: 15-20 hours.

    Wranglers will add more to the canonical tag lists as we find them, but if there's a media related tag that you'd find useful to be filterable, let us know through a support request or by tweeting us at @ao3_wranglers!

    Streaming podfic

    We hit a snag recently with our podfic provision when Google disabled their Audio Player, which we were using to enable people to stream podfic. The good news is that we've snagged a copy of the audio player code to host on our own servers, so this will return in our next code deploy. When the deploy has taken place, you'll once more be able to embed podfic, using the following embed code:

    <embed type="application/x-shockwave-flash" flashvars="audioUrl=MP3_FILE_URL" src="http://archiveofourown.org/static/audio-player.swf" width="400" height="270" quality="best" > </embed>

    (Essentially, if you were previously using the Google player, you'll just be replacing the Google url with http://archiveofourown.org/static/audio-player.swf.) Apologies for the inconvenience in the meantime!

    Support superstars

    The recent increase in users on the site has resulted in a corresponding increase in Support tickets. The fantastic Support team have been amazing at keeping up with all the tickets and giving users the help they need. Their turnaround is really quick and we know that people appreciate their hard work - thanks Support!

    Support staffer Yshyn wrote a great post on Support ticket stats which gives an insight into what kinds of questions people ask and how many tickets we receive. One of the things which has been particularly nice for Support recently is the number of tickets which have included some positive feedback for the site, even when it's been experiencing problems. They always pass this onto the other teams and it makes everyone's day brighter - thank you lovely users!

    Roadmap, feature requests, and plans

    We try to keep users informed on what's going on with the site, although it can be lots of hard work! You can see the details of all our code updates to date in our release notes. Awesome AD&T staffer mumble is currently working really hard to update our very, very outdated Roadmap: this is important as a way of helping us focus our work, and a useful way of giving users an idea of what we have planned. We've also recently started using a new tool, Trello, to manage feature requests from users. Like our codebase on Github and our issues list on Google Code, this is open to the public, so you can now see what's already been requested and vote for the things you'd like! You can also see proposals that were rejected, and why. Support staffer Sam wrote up a handy introduction to our internal tools, and there's also a new AO3 Internal Tools FAQ.

    We have limited resources, so we can't always introduce requested features as fast as we'd like (right now, we're focusing on site stability and trying to avoid too many new things). However, we hope that these tools will help people see what's in the works, and we're excited to have found a tool (Trello) which makes it easier for people to let us know which features they're most interested in.

    Questions, comments?

    We welcome feedback from users! If you have questions or comments, feel free to leave them in the comments of the latest news post, or send in a Support request (if you're reporting a bug, please send that to Support, as they're super efficient - comments on our news posts sometimes get overlooked).

  • Spotlight on Support: AO3 ticket stats

    By .Lucy Pearson on Wednesday, 25 July 2012 - 9:15am
    Message type:

    This is going to be a very boring post. It's going to be full of numbers, and graphs, those things that I may or may not have spent many years at school colouring in with lovely coloured pencil without understanding them much (because I was apparently too much of an innocent mind to turn them into rude, crude approximations of things not related to mathematics except in the most abstract sense), and yet, these will be very easy to understand numbers. I am not a statistician, nor are the levels of data I have access to very deep. What I am is a member of the Support Committee with a curiosity about the numbers and types of tickets that pass through our hands, and who decided to add up the numbers one day and turn them into graphs. That was last year, and somehow the lure of the bar chart means that I have continued to collate information through to where we are right now, having just finished the second quarter of 2012.

    In this post, I'm going to summarise the types of tickets received, what categories they fall under, and the general trends we witness.

    But first, some explanation of the process.

    Collecting The Numbers

    I'm sure that the method I have used is going to come under some degree of criticism for being inefficient; however, our Support software, provided by 16bugs (see Sam's spotlight post for more information) was not designed for data export. This means that the only way to extract numbers of tickets is to do it manually. And by manually, I mean I go through the email duplicates of each ticket one by one, assigning them a category, then add up the numbers for each month and enter them into an Excel spreadsheet.

    What this method is, for all its faults, is quick, which means that I can rapidly pull up a given time period to see what sort of tickets were received between those dates. These graphs were originally created as an informal overview of ticket stats (which is a position they remain in – production of these stats is not an official Support Committee duty). They are simply counts of the original tickets, what they are about, and when they were received. They are not a count of how quickly they were responded to, who responded to what, or what follow ups were conducted with the users.

    Categories

    I'm going to leave direct explanations of the categories until the sections for the respective quarters, as these change on a quarter-by-quarter basis. This is due to the simple fact that new features are added, which generates new issues, and old issues are resolved. For example, squid caching was not implemented until June of this year, so prior to that, it was not shown in the graphs because issues relating to it did not occur. Here I'll instead explain the process by which tickets are categorised.

    If you've ever submitted a comment or query to Support you will notice that on our form is a drop down menu.

     Bug Report, Feedback/Suggestions, General/Other, Help Using the Archive, Languages/Translation, Tags

    These categories are not the ones I have used to sort tickets. Since the categories in the menu are so few and so broad, I felt it necessary to granulate them further, and count tickets as they related to specific archive functions and features.

    If a new category is created in my sorting, it's because an issue got a large number of tickets and wasn't a transient bug. For example, if a ticket is related to subscription emails, it is categorised under "Subscriptions", not "Email" because it is related to a specific Archive function (in this case subscriptions) that has an existing category. If it were related to invitation emails, it would go under "Accounts/Invitations/Login". However, if it's related to kudos batching, it goes under general "Emails", because there is no category for kudos.

    The Stats

    2011, In Brief

    I won't linger on 2011 too much (see Q1 2012 for an explanation of categories), since this information was a little more awkwardly hacked together than for 2012 – by which time I had sorted out my process for quickly organising tickets.

    bar chart with different colors for every month in 2011, representing absolute ticket numbers for each in 15 different categories
    (full size)

    Prior to August, tickets were collated by the Support Chair, using slightly different categories than I did. I attempted to meld the two sets of information as best I could to produce the above year overview.

    What is easily and clearly visible is the spike in tickets in November, resulting from a change to the front-end presentation of the AO3. The biggest spike is split between Interface/CSS tickets and Feedback. While many of the tickets sorted under Feedback were directly related to the changes to the AO3's interface, they did not contain bug reports or requests for information, and therefore fell under the heading of Feedback.

    Q1 2012

    Categories for Q1 2012:

    • Error 502 - the 'server busy' messages
    • 1000 Works - queries related to why we have a 1000 work limit on the fandom landing pages
    • Activation/Invitation/Login - problems activating accounts, getting invitations, or logging in
    • Admin/Abuse - issues that need to be examined by Admin or Abuse teams
    • Bug Report - Reports of transient bugs that aren't separately categorised
    • Collection/Challenges/Prompts - any problems/queries about these
    • Downloads - errors, bugs, queries related to downloading
    • Feature Request - any 'can I have/I would like/will you implement' queries
    • Feedback - any complaints, or any positive feedback (alone with no other feature-related issue)
    • Help/Information - any questions about AO3/OTW in general, or how to use specific features
    • Interface/CSS/Display - problems/queries relating to how the archive appears on screen, i.e. interface
    • Imports - issues with importing from LiveJournal/Fanfiction.net/other
    • Open Doors - questions related to fics imported through OD
    • Search/Browse/Filter - Problem or queries about sorting through archive contents
    • Tag Wrangling - any tag related questions

    bar chart with different colors for Jan, Feb, and Mar 2012, representing absolute ticket numbers for each in 14 different categories
    (full size)

    Possibly due to the fact that the holidays are still going on at the beginning of January (and thus, people have more time to spend on fandom sites) we saw more tickets in general than during the following two months.

    Q2 2012

    Categories added for Q2 2012:

    • Embedding – queries/problems with embedding media (images/audio/video) into Works pages
    • Bookmarks – queries/problems involving bookmarking
    • Caching – bug reports that are actually caching issues (e.g., reporting 0 works in a fandom as a bug – this is a caching issue, or appearing as logged in as another user). The kind of caching which causes these particular bugs was only implemented in June.
    • Email – email issues unrelated to other categories (e.g., kudos email batching)
    • Subscriptions – issues/queries to do with the subscribe feature

    bar chart with different colors for Jan, Feb, and Mar 2012, representing absolute ticket numbers for each in 14 different categories
    (full size)

    To break down the invitations emails, in June we received 140 tickets related to Invitations.

    • How Do I Use This Invite: 22
    • Did Not Receive Invitation Email: 31
    • Fell Off Invite List (unaware of security changes): 41
    • (of those, who admitted to re-adding themselves: 6)
    • General Invite Queue Unhappiness: 10
    • Can I have An Invite?: 9
    • I Requested Invites, Where Are They?: 12
    • Paid Accounts: 3
    • My friend on FF.net needs an invite: 12
    • Need Invites for a Challenge: 5
    • Please Remove Me From Queue: 1

    The remaining 37 tickets in that category were related to account activation or login issues.

    bar chart with different colors for each week of June 2012, representing ticket numbers for each in 20 categories
    (full size)

    This graph shows how the tickets were distributed during the weeks that span the month of June. In week 23 (commencing 4th June) we received the greatest number of queries regarding invites, as this was the point at which the invitations queue started growing at the rate of nearly 1000 new additions per day (a rate since slowed to around 300-odd per day). This coincided with the point at which the AO3 servers started creaking under the strain of lots more visitors and a filtering system that was originally designed with a smaller user base in mind.

    When squid caching was implemented to help ease the strain (around week 24) we saw an increased number of tickets related to this change. In week 25, when filtering was disabled, we began to see an increased number of tickets related to that. (Originally, the message was ill-worded, appearing to be an error message, rather than an admin message – this has since been altered, and tickets regarding the filtering being 'down' have disappeared.)

    And This All Means...

    I always have fun posting these stats to the support committee. Everyone already knows more-or-less how things have gone, but sometimes looking at the numbers surprises us. When I originally created them, one frequent question was "what's the most common ticket you get" to which we would generally reply "queries regarding the 1000 work limit". I was curious as to whether this was actually the case. As it turned out, Feature Requests came in more often. Questions about the 1000 Works came lower down the list.

    If you are wondering how many tickets we answer altogether, I can tell you that at the time of writing there are no unanswered tickets in our support software (except for one bugged ticket, which we are attempting to resolve with 16 Bugs). Every single ticket we receive is read and personally answered by a member of our staff, usually within a day or two. So, the answer is: we answer all of them.

    graph showing the number of tickets for each month from Jan 2011 (170) to May 2012 (590)
    (full size)

    This post by Support staffer Yshyn. If you find a bug, have a question about the site, or want to request a feature, you can submit a Support request.

    Mirrored from an original post on the Archive of Our Own.

  • Planned Archive downtime: Server software upgrade

    By .Lucy Pearson on Monday, 23 July 2012 - 11:09am
    Message type:
    Tags:

    The Archive of our Own will be down for planned maintenance for approximately 90 minutes from 07.00 UTC on Thursday 26 July (see what time this is in your timezone). We'll be upgrading our server software during this time (more details below for the curious!).

    We'll keep users updated on our Twitter AO3_Status as the work progresses. Thanks for your patience while we complete this work!

    Server software upgrades

    This downtime will allow us to upgrade Nginx and MySQL on our servers. It's important for us to keep this software up-to-date in order to avoid bugs and get better performance.

    Nginx is web server software which everyone's browser communicates with - when you come to the Archive and make a request for a work, Nginx does the job of communicating with the application and getting the data you wanted. It handles some information itself and passes requests on which are too complex for it.

    MySQL is the database which handles all the persistent data in the Archive - that's things like works. We're updating this to a much more recent version of the software, which will bring us some performance gains. We're also moving from the Oracle branch to Percona, which will bring us some additional benefits: it should give better performance than Oracle, and will also give us some additional instrumentation to monitor the database and identify problem areas. In addition, we hope to draw on the support of the company who produce it (also called Percona).

    Users shouldn't see any changes after this update. However, we wanted to keep this work separate from our recent RAM upgrade so that if any problems do arise, we will find it easier to identify the cause.

    Mirrored from an original post on the Archive of Our Own.

  • Links roundup for 21 July 2012

    By Claudia Rebaza on Saturday, 21 July 2012 - 4:33pm
    Message type:

    Here's a roundup of troubling issues that might be of interest to fans:

    • Last week, the OTW posted a news alert to fanfic writers and podfic makers about a contest for fanfic recordings being held at ComicCon. Our post pointed out troubling aspects of the contract fans would be required to sign, which led to additional discussion of the terms and contest by other fans. One was semaphore-drivethru on Tumblr who concluded "This, guys, is why you should always, always read a contract/release before signing. There is no length of contract on this, so I’m assuming it’s in perpetuity. There’s no language at all to protect you, either. Just an agreement for you to give them everything for a chance at a twenty minute recording. If you feel it’s a worthwhile trade, an opportunity with[sic] taking, then go for it. But be aware that in no reputable publishing circles would a contract like this be considered reasonable." Since then, it's been announced that Random House will be extending the contest to the upcoming Star Wars Celebration VI in Orlando next month. We urge those fans to also do a careful read through if they're considering entering their material.
    • Fanwork contests in general have proliferated wildly through many fandoms and media properties. In many cases the contests are just a form of spotlight on fan work and there is an absence of contracts or, for that matter, prizes. However the fact that legal rights and financial rewards are now on offer in many places sheds a particularly troubling light on the longstanding problem of fanworks plagiarism. Many fans have at some time found their fanworks reposted with credit to them removed or left unclear, or have had their fanworks slightly altered and presented as someone else's work. The rise of frequent contests has now also led to fans having their work entered in those contests without their knowledge. Given that professional publications of all kinds have had plagiarism scandals of their own makes it seem unlikely that the plagiarists will come to light due to careful research by the contest hosts.
    • Also on a front closer to home, a recent complaint was circulating on Tumblr involving an ad being shown to an AO3 user who was reading at the archive. The reader assumed that the ads were coming from the Archive of Our Own. We want to clarify that this is not the case, as the AO3 does not host ads. Rather the problem likely stemmed from the user's own browser, which may have been infected with malware to produce the ad content. If users encounter a problem like this, please report it to the AO3's Support team. We would appreciate it if fans could signal boost this information.

    If you've experienced plagiarism or have perspectives to share about fans and fandom, why not write about it in Fanlore? Additions are welcome from all fans.

    We want your suggestions! If you know of an essay, video, article, event, or link you think we should know about, comment on the most recent Links Roundup. Links are welcome in all languages! Submitting a link doesn't guarantee that it will be included in a roundup post, and inclusion of a link doesn't mean that it is endorsed by the OTW.

  • Planned Archive downtime: RAM upgrade

    By .Lucy Pearson on Tuesday, 17 July 2012 - 8:44am
    Message type:
    Tags:

    The Archive of our Own will be down for planned maintenance for approximately three hours from 15.00 UTC on Friday 20 July (see what time this is in your timezone). During this period we'll be installing some new RAM and performing some other maintenance (more details below for the curious!).

    We'll keep users updated on our Twitter AO3_Status as the work progresses. Thanks for your patience while we complete this work!

    New RAM

     cartoon style image of server
    Our database server looking grumpy about having too little RAM!

    We're doubling the RAM in our database server and in our two application servers. Increasing RAM will help our system cope with more users: for example, it will allow us to run more unicorn workers, which serve up the content you're trying to access. This should help site performance as the site expands.

    You can imagine the unicorns lining up in the hall of RAM to fetch you things from the treasure trove of fanworks: if there aren't many unicorns, you have to wait till one can serve you, which sometimes means you get a 502 error. We can increase the number of unicorns to make things go faster for you, but if the hall is too small (there isn't enough RAM) then things get crowded and inefficient and everything slows down again. More RAM allows us to increase the number of unicorns without slowing things down. (For the interested, this more technical explanation of Unicorn isn't exactly the way things are set up on the AO3, but will give you an idea.)

    New drives

    We're also installing some new drives in our two oldest machines. Both these machines have room for six drives; currently they each have four installed. Information is mirrored on the drives so that if one goes down, the system continues to work. At the moment, one machine has a broken drive. We'll be replacing the broken drive, and at the same time adding two spares to both machines so that we have more backups if anything else breaks.

    Our two original machines preparing to nom their new drives
     cartoon style image of server cartoon style image of server

    Mirrored from an original post on the Archive of Our Own.

  • My, how we've grown! A few AO3 stats

    By .Lucy Pearson on Monday, 16 July 2012 - 4:12pm
    Message type:
    Tags:

    We've been talking a lot recently about how much the AO3 has expanded over the last few months. One easy statistic for us to lay our hands on is the number of registered accounts, but this only represents a tiny portion of site activity. Our awesome sys-admin James_ has been doing some number crunching with our server logs to establish just how much we've grown, and provided us with the following stats (numbers for June not yet available). Thanks to hele for making them into pretty graphs!

    Visitors to the AO3

    Line graph showing the number of visitors to the AO3 per month, December 2010 to May 2012. The line progresses steadily upwards with a significant spike from 1,197,637 in April 2012 to 1,409,265 in May 2012.

    The number of unique visitors to the site has increased almost every month since December 2010 (each unique IP address is counted as one visitor). There are a few points where the rate of increase gets more dramatic: there was a jump of 244,587 across December 2011 and January 2012, compared to one of 137,917 over the two months before that. This can probably be accounted for by the fact that during December and January, holiday challenges such as Yuletide bring more people to the site. This theory is borne out by the fact there was a slight dip in the number of visitors during February 2012, indicating that some of the extra traffic in the previous two months were 'drive by' visitors who didn't stick around.

    May 2012 saw a steep increase in the number of visitors: there were 211,628 more visitors to the site than there had been the month before! The rapid increase in visitors was not without its price: this was the month of many 502 errors!

    Traffic to the AO3

    Line graph showing AO3 traffic in GB per month, December 2010 to May 2012. The line progresses steadily upwards with a significant spike from 2192 GB in April 2012 to  2758 GB in May 2012.

    The increase in the number of visitors to the site has also been accompanied by an increase in overall site traffic (how much data we're serving up). Again, there's a significant spike during December/January. Interestingly, there's no dip in traffic for February 2012, showing that even though there were some 'one time' visitors over the holiday period, there were also plenty of people who stayed and continued to enjoy fanworks on the site.

    The increase in traffic to the site clearly accelerated in 2012. Between January and May 2011 traffic increased by just 159.92 GB; the same period in 2012 saw an increase of 1,870.26 GB! In fact, with an increase of 566 GB during May 2012, that month alone saw almost as big a jump in traffic as the whole of the previous year (595.63GB)!

    And the other stuff

    With these kinds of numbers, it's not surprising that there've been a few bumps along the way. For information on how we're dealing with the growth in the site you can check out our posts on performance and growth and accounts and invitations.

    Many thanks to our dedicated volunteers for their hard work dealing with the growth of the site, and to our fabulous users for their patience with our growing pains - and for creating the awesome fanworks so many people are flocking here to see!

    Mirrored from an original post on the Archive of Our Own.

Pages

Subscribe to Archive of Our Own