Archive News #4 – Colocation and Open Beta

In this, our fourth Archive news post, the OTW is delighted and excited to announce Server Colocation and the Coming Of Open Beta! Yes, this extra special Archive post is brought to you by ‘We own the goddamn servers!’ Thanks to the generous donations of fandom and the hard work of our team, we’ve bought two shiny new servers! \0/ The Archive of Our Own will soon be moved onto these servers, which will mean that we will have enough capacity to increase our number of users and move into Open Beta. \0/

We know that people will have questions about the details of what we’ve selected and the path we take from here. This post will deal with some of the main issues, but please do leave us your questions in comments if there’s anything else you’d like to know. (Specific comments about the design or usability of the Archive can, as always, be left as feedback on the Archive site itself, so it goes into our bugfix and design process).

Why is it important for us to own the servers?

Owning our own servers is at the heart of the OTW’s mission as a nonprofit fan organization. We believe that fanworks are transformative and that transformative works are legitimate. Historically, many sites that have hosted fanworks have not been interested in or capable of defending the legitimacy of fanworks and have therefore been quick to take down or delete fan content when challenged for whatever reason (copyright concerns, concerns over explicit material, business or advertising concerns etc). The OTW’s goal has always been to host fanworks on nonprofit servers owned and maintained by fellow fans.

What’s the basic technical setup?

We have colocated servers. \0/ In brief, this means that we own the actual machines and we pay a colocation company to provide the physical space they’re housed in. We also have to pay recurring costs for bandwidth and power. Two major advantages of this set-up are that the machines are ours, so no one can mess with them for any reason, and there is much more potential to scale up to support a bigger site if and when we ever need to. If you’re interested in a more detailed explanation of colocation and why it is the best option for a site like ours, then we recommend you check out Synecdochic’s awesome post on hosting options and colocation.

What are the specs on your servers?

After extensive research and comparison shopping, we purchased two shiny new Hewlett Packard ProLiant DL360 G5 rack servers, each powered by 2 Quad-Core Intel Xeon E5420 processors. We wanted to make sure we had plenty of memory to support simultaneous users and fast drives to ensure speedy response from the database to retrieve stories, so each server will be loaded with 16GB RAM, and 4×72GB 15k RPM hard drives arranged in a RAID10.

Each server will house both our Web application and database, but one will be primary for Web, one will be primary for the database, and they will each provide failover for the other.

How much is all this costing?

The two servers with extra RAM (they come standard with 4GB each, which we upped to 16GB each) cost US$6,330 and the hard drives cost an additional US$1,835 for a grand total of US$8,165 (including shipping.)

We will pay for initial installation/set-up, and then colocation hosting will be an ongoing cost of US$228.00 per month.

What happens now?

The colocation hosts will rack and install our servers and set things up so that our Systems team can access and administer the servers remotely. As noted above, they’ll also to do our initial OS installation for us. We have to pay for this initial set-up, but it’s worth it to lighten the load for our Systems team. (Our wonderful Systems volunteers are in heavy demand across the entire OTW, so we like to spare this small but incredibly hard-working team when we can.)

Once the basic set-up is complete, our Systems team will install any remaining software and code libraries necessary to run the Archive of Our Own. They’ll then begin the process of moving the Archive itself over to the new servers!

What’s the plan for moving the Archive?

  1. Our Systems team will deploy our stable Archive code on the new server. We’ll be deploying code which has already been used on the existing server for a little while so that we can easily spot any bugs that occur during the move. We’ll also be porting the existing data in the AO3 onto the new server.
  2. Our team of Testers and anyone else we can rope in will spend some time extensively testing the code, banging on every aspect of the site and generally making sure that everything is working as it should.
  3. We will refresh the data on the site to bring it in line with the existing beta site and delete any crud generated during testing, and redirect so that it points to our new home.
  4. We’ll move to Open Beta \0/

What is Open Beta?

Open Beta is the time when we start expanding our user numbers so that fandom can use the Archive for real! It won’t be completely open for sign-ups – we have to expand gradually, so we’ll be using an invitation system – but it will be many, many more than our current tiny testing userbase. We’re looking at moving from 277 users to c. 2000 users over the course of the first month or so, and we’ll keep ramping up from there.

Closed Beta was the period in which we did major code developments and tested them out on our willing victims volunteers. By the time we go into Open Beta, we will have built much of the major functionality for the Archive and tested it out a bit. Open Beta will let us see how the Archive runs with a lot of users. However, we’ll still be actively coding – building new features and refining the existing ones based on user feedback – and monitoring the site closely to see if and how it works. Contents may shift in transit!

For those of you who are currently using Dreamwidth, you’ll be familiar with how the Open Beta process works – managed user numbers and fairly frequent code pushes.

When will you move to Open Beta?

We’re hoping to move to Open Beta in late 2009! This is provisional, and is dependent on our move going smoothly, our coders not dying of exhaustion, etc. But our brilliant AD&T chair Maia has been creating detailed time plans, and things are looking promising for us achieving our goals.

How can I get an account once you’re in Open Beta?

It’s really important to us to ensure that everyone who wants one has the chance to get an account. We can’t just have open sign-ups, because we need to increase our numbers gradually, but we will have an open-access invitations request system. There will therefore be two ways of getting an account:

  1. Anyone who is interested will be able to add their email address to a queue, and we’ll periodically send out invitations to that pool depending on how many new users the site can take at any given time. This will be first-come, first-served. This way, you won’t have to know someone who already has an account in order to have a chance of getting one.
  2. Users who have an account on the Archive will be given a certain number of invitations to hand out to their friends. We know that people do like to share the fannish glee and to build their networks on new sites, and we want to give people a chance to do that. Extra invitation codes will be issued to users periodically depending on how many new users the site can manage.

We’re really, really excited about finally owning our own servers, and we hope you are too! AD&T committee member Amelia deserves a massive, massive shout-out for doing all the legwork on this – she has worked like a crazy person making technical comparisons, interfacing between all the people involved, and generally wrangling the whole affair. We’re also very grateful to PRK and JP, two non-fannish friends of the OTW who gave us lots of support and advice. Finally, we’re VERY grateful to all of YOU, our awesome fannish community who donated the money to make this possible. We’re really jazzed about seeing more and more wonderful fanworks filling up the Archive in the months to come!

We hope this post answers a few of your questions! Please leave other questions and comments here. We won’t always answer comments on this post directly – we’ll put your feedback into our pool of things to answer in future posts.

Archive of Our Own
  1. Zooey Glass commented:

    I’m so proud of fandom!

  2. Laura Shapiro commented:

    So exciting! I’m thrilled for us!

  3. Jessica Hekman commented:

    Congratulations! That is great. I’m curious about the ongoing costs — I am sure you have a plan to ensure that you are able to pay colo fees every month, and would be interested to hear what it is. Obviously the nightmare scenario is “we don’t have the money to pay the colo fee, so we have to shut down the server, and will no longer be able to host the content on it.” This organization is so well run that I’m pretty sure that isn’t ever going to happen, but it would be reassuring to hear specifics. Good luck with your new servers!

    • rbarenblat commented:

      Thanks for your comment, Jessica! I’m the chair of the OTW’s Development/Membership committee; it’s our job to make sure that we raise enough money to support the OTW’s ongoing needs, colo fees among them.

      Fortunately, I don’t think the nightmare scenario that you mention is likely to ever arise. :-) I can offer three reasons why I think we’re in good shape. One is that we keep a “safety net” of money in our bank account at all times. This is good nonprofit fiscal practice.

      The second reason is, we hold fundraising drives twice a year. Twelve months’ worth of colo fees at $248/month comes to $2,976, and one option is to sequester the first 3K that we earn in our next donations drive (scheduled for October) as colo fees for the next 12 months.

      And the third reason is, some of our members also donate on a monthly basis. One of DevMem’s goals for the coming year (now that we know what our colo fees will be) is to line up more monthly donors — ideally, enough that the colo fees just flow in to our coffers naturally each month! If we can get 25 people to commit to giving $10/month, that would cover our current colo costs.

      Given fandom’s incredible generosity so far (especially during what have been tough economic times for most of us), I feel confident that we’ll be able to continue to raise the money we need to pay our colo fees, most likely without ever needing to dip into our “safety net.”

      Does this answer your questions? If not, please feel free to ask more!

      • Stultiloquentia commented:

        Thanks for laying this out; I had the same concern.

        Meanwhile: SQUEE!!!

  4. xkatjafx commented:

    //Anyone who is interested will be able to add their email address to a queue//

    Am I the only one who remembers that when this whole thing started the members – who paid for “our servers” – were promised to receive archive accounts first?

    To be honest I’m not looking forward to go begging for an account that was promised me two years ago. I don’t really like the idea of adding my name to a queue that will quite probably go up when we Europeans are still asleep, meaning I will be no. 6000 and something in this queue.

    And yeah, please forward this message to the appropriate committee, because I’m not the only one who feels kind of betrayed.

    Katja, a rather disappointed OTW member.

    • elizabethyalkut commented:

      Thanks for coming to us with your concerns, Katja!

      Membership in the OTW and having an account on the Archive of Our Own are in no way related, both to protect the privacy of our donors and members and to ensure that access to the Archive of the Our Own is not a financial matter for any fan (who has internet access, at any rate). Membership confers the right to vote in OTW elections and supports the organization’s work financially. This has always been the case.

      The OTW firmly supports the right of users to separate fannish and nonfannish identities. Donating to the OTW and using the OTW’s services are entirely separate — if you choose to donate or become a member (and we’re so thrilled you have!), you do not need to tell us your fan identity, and we will not have any connection between your fan identity and your financial information or real-life identity.

      The FAQs on this site discuss this, as in the following: “We hope to move to open beta in late 2009. At this time we’ll be able to welcome lots more users to the Archive; we’ll still need to manage the load on our servers, so we’ll be using an invitations system. You’ll be able to add your name to a queue, and we’ll periodically send out invitations to people on that list depending on how many new users the site can take. We’ll let people know more information on how this will work when it becomes available.” ( Further, we will never charge a fee to use any of our services (

      In fact, I just ran the numbers as of today (we’ve run them exhaustively in the past few months, as I’m sure you can imagine), and even if every OTW member signed up to be added to the queue of account-holders immediately, we would still be able to handle the volume and more; that’s why we’ve waited until we had our own co-located servers to go to Open Beta. I honestly wouldn’t be too concerned about being too far down on the list!

      Elizabeth Yalkut, Development and Membership staff

      • xkatjafx commented:

        //Membership confers the right to vote in OTW elections and supports the organization’s work financially. This has always been the case//

        No, it hasn’t. When this organization did the first official drive for money, you promised all the members an account in the Fanfic Archive *before* all others. (Meaning we will not be the only ones with accounts, but we would get them sooner.) I’m not the only one who remembers that.

        Which now you happily seem to ignore. And while I get that this might be complicated, you could still simply send out an email to the members and invite them first to the queue or whatever. Obviously you don’t even try.

        Anyway, I still stand behind the cause of OTW, but if this is how it goes, you completely lost my trust. Because obviously your promises don’t mean squat.

        //and even if every OTW member signed up to be added to the queue of account-holders immediately, we would still be able to handle the volume and more; that’s why we’ve waited until we had our own co-located servers to go to Open Beta. I honestly wouldn’t be too concerned about being too far down on the list!//

        I saw how “well” the Dreamwidth invitation thing worked for fandom, and I know how many people are waiting for OTW-Archive accounts, so 2000 accounts are really nothing compared to the number of people who will try to get one, once you set up this queue.

        • fcoppa commented:

          E-mailing the members isn’t the same thing as guaranteeing an invite to members: as Elizabeth explained, to guarantee an invite, we’d have to know who you are, which goes against our privacy policy connecting fan and legal identities. That said, emailing the membership to make sure that OTW members are aware of the coming of open beta is a good idea, and we’ll make sure its on DevMem’s radar if it wasn’t already!

          • xkatjafx commented:

            //to guarantee an invite, we’d have to know who you are, which goes against our privacy policy connecting fan and legal identities.//

            How does opening up the queue for members first equal connecting fan and legal identity? If there are 400 members and 350 people are signing up, how do you know which of those 350 people I am?

            Also, it’s not like there’s a list of the real names of all members out there, so nobody can compare that list to the people who would get the archive accounts. And fan activity and OTW membership has been obviously connected by the members themselves for a long time.

          • fcoppa commented:

            Members will certainly be told when Archive accounts are available. We will make sure everyone’s aware of the opportunity to get an archive account, and then they’ll be able to sign up with whatever fan names/email addresses they like.

            That being said, membership in the OTW has *never* equaled membership in one of our services (AO3, Fanlore, etc). Unlike LJ, DW, etc, we are not a business but a nonprofit organization: we don’t sell levels of service (in fact, we don’t sell any service at all!) Moreover, we don’t want to imply that richer fans (who can afford OTW membership) are in any way advantaged over poorer fans who can’t. Rather, we’re raising funds to build tools that any fan can use, OTW member or not; what we hope is that people who like our tools or like us and support our mission will donate to us if they can (and help to support those fans who can’t, in the model of public television or radio.)

            I’m sorry if this wasn’t clear to you; we’ve always tried to be explicit that all membership gets you is our thanks and a vote in any elections we hold. :( But the AO3 is there for any fan who wants it.

            All that being said, I really wouldn’t worry about not getting an account! Also, you don’t have to get an invite from someone with an account: the queue system will allow people to sign up with us directly. (And we will be expanding quite radically in the move from Closed to Open Beta!)

          • xkatjafx commented:

            //That being said, membership in the OTW has *never* equaled membership in one of our services //

            I so wished I had saved that stuff (whereever it was and whoever said it), but boy, I didn’t think I had to prove it years later.

            //Moreover, we don’t want to imply that richer fans (who can afford OTW membership) are in any way advantaged over poorer fans who can’t.//

            I never implied that you did. I realize that I pay for other fans and I’m totally fine with that. I also want everyone to get a free account.
            But you what you are actually implying is that paying for the archive is not worth as much as working on the archive. Which I think is not fair.
            And yes, I’ve talked to other people about this and this might be culture clash thing, because around here we have this saying that “time is money”. So I’m sorry that can’t spend hours on this project – that’s why I decided to give money *instead*. Now I realize money obviously doesn’t mean anything to you. As do your members.

            That said, I’m not really looking forward to this queue. As I mentioned before this queue will quite probably go up when I’m asleep, so there will be tons of people before me and this will end up to be just as annoying as getting the DW account. There is no way to start a fair queue if you have to do it globally!!

            So yeah, still utterly disappointed.

          • Heather Cook commented:

            I think what you’re missing here is is that even if AO3 accounts had been promised to donors, there is no way the OTW can connect your REAL NAME which you used when you made the donations to your FAN NAME which you would presumably use when requesting an AO3 account. Because there is no way to make that connection (and because many people do not want that connection), the best (and most fair) way to give out accounts is to have a sign-up queue. This way no one has to out themselves, and the OTW doesn’t have to match real name to fan name.

            What it boils down to is this: Susie Jones, real person, makes a donation to the OTW. In the meantime, SJFangirl (who is really Susie Jones, but not out about it) wants an AO3 account. The folks at OTW have a record of Susie Jones’ donation, and they have a request from SJFangirl for an AO3 account. While they may know who is who unofficially, officially, there is no way to tie them together, and thus no way to give SJFangirl an AO3 account in exchange for the donation made by Susie Jones.

          • xkatjafx commented:

            Actually I think this is no problem at all.

            Because my guess is you have the email addresses of your members. (I mean really, how can I vote later on if you don’t know if I am allowed to vote?) You send out an email containing a link to the “queue” a day or a week before it gets announced to the rest of the word.

            Even better: You send out a link to the page where I can create an archive account.
            Maybe you add a special password to this link, just to make sure, nobody wrong stumbles over it by accident.

            Everyone who gets the email can create an account. They do not have to create the account using their real name, they just go there and create an account with the password and claim their usual fan identity name. So there’s no way to connect them to their RL names. You have 400 members – of those 320 will create an account. How will you tell which of those 400 members I am? Or if I even really did create an account with the first wave? How will anybody later even find out if I belonged to those in the front row of the queue if you don’t show that (membership) queue/list to anyone?

            How does this *not* work? Please explain it to me?

            I’m very sure you could make it work – if you just wanted to. Also, people who feel paranoid about this don’t have to use the first wave and can just create their account whenever.

          • Heather Cook commented:

            I suppose your solution makes sense (although I’m sure the logistics of that would be a nightmare for the AO3 staff), but then you’re back to an instance where wealthier fans are getting a perk that’s meant to be for everyone regardless of financial status, which again, is not something the OTW wants to do, nor is it something that was ever promised.

            Asking people to sign up in a queue regardless of their membership status has nothing to do with whether or not we “could make it work – if you just wanted to.” It has to do with being fair and unbiased toward everyone in fandom, regardless of their financial means. I get that you’re worried the sign-ups will happen at a time when some part of the world is asleep, and yes, that’s always a possibility, but I’m sure the AO3 committees will do their best to be accommodating to everyone.

            Honestly? I think you’re worrying for no reason — you might or might not get an account right away, but you will get one, so why all the stress?

          • xkatjafx commented:

            //but then you’re back to an instance where wealthier fans are getting a perk that’s meant to be for everyone regardless of financial status,//

            Wow, I wish I was that wealthy as you paint me to be.
            But yes, to be honest I want that *one* small perk. Because I put my money in this archive, so why do I have to come *begging* to get an account?

            See, the thing is I get that you want to be “unbiased toward everyone in fandom, regardless of their financial means.” But this situation boils down the fact that my contribution is not worth more than no contribution at all. So why contribute at all, if it obviously doesn’t mean anything to you?

            And I get that this is probably a culture clash thing, but sorry, from my point of view your behavior is insulting. It’s fairly incomprehensible that we even have to discuss this.

            Also, if you plan to open 2000 accounts and there are probably about 300 members who still don’t have an account, why don’t you just give them one? It’s not like there’s nothing left afterwards.

            //Honestly? I think you’re worrying for no reason — you might or might not get an account right away, but you will get one, so why all the stress?//

            Because I remember the DW thing very well? Because I really don’t want to go around having to find somebody to spare me an invite? It’s work, time-consuming and basically it’s annoying. And I didn’t even put any money into DW. But I did so with this project, because I actually believe that it’s a good thing and I’d love to stay involved in some way – but having to come here to be put on an a list and maybe get an account by *accident* is just wrong.

          • fcoppa commented:

            Ah! I feel like I understand better where you’re coming from. A couple of quick things–well, aside from “sorry”: I’m sorry you’re disappointed in us.

            1) Vis a vis the invite system: honestly, I’m not building it and so I don’t know exactly what it can or can’t do. I asked and was told that we can’t close signups to members without busting privacy protection. I’ll see if I can get more info for you from the folks actually building the thing.

            2) I really really REALLY do believe that the timezone won’t make a difference with the numbers we’re talking about, but I will totally bring this up with ADT to make sure that the queue doesn’t open at a time that disadvantages fans not in North America. (FWIW, we’ve been making a conscious effort to use UTC/GMT and to resist North-American and northern hemisphere regionalisms when we notice them. I realize that might be too little, too late, but I wanted you to know that it is something we think about.)

            3) While I understand why it might feel like time was more rewarded or valued than money, that’s not why we gave accounts to users who left feedback during Closed Beta, many of whom are also members. It’s just that we were doing major coding work on the Archive and it was very buggy, so our users were essentially a testing group. While we’re glad people are enjoying their Archive accounts , Closed Beta users are always testers first and foremost.

          • xkatjafx commented:

            1) Thanks, that would be nice.

            2) I think you underestimate the number of people that will come once they realize what AoOO actually is. It’s not that widely known at the moment, but it will be, once you will offer accounts and people will start talking. And really, 2000 people? That’s doesn’t even cover *one* of the bigger fandoms out there. Okay, I don’t know where you want to go after that and maybe you’re going to amp it up to 20,000 and everything will be fine (if you only invite HP fans ; )
            Nevertheless if I were you I would not do the queue thing – to make this work you would have to offer a at least a queue per continent. If you want to be fair, open the list up for a day and do a lottery later. (And yes, this will probably mean that I won’t get an account until the end of days, because I’m extremely unlucky with these things *g*)
            Also, I’d be very careful in which way you start spreading the invites after that. Looking back at the DW thing, you might not want to give the impression that this a BNF/elite thing. (If you ask me this is the reason why DW never really took off. It had too much “bad press” before it even really started. There were several weeks were fandom was literally seething.)

            3) Yeah, well, your members could also be a testing group ; )

          • fcoppa commented:

            Just to assure you–I hope!

            There IS a plan to make sure our members know about the queue before we post broadly or do any wide outreach–just the queue itself AFAIK isn’t designed to be members-only for any number of reasons (privacy among them). I hope you’re right and people WILL come when they realize what the AO3 really is, but members (and people paying close attention to the OTW) will know that accounts are open before that.

            The open queue was meant, fwiw, to provide a counter to the DW invite thing where you had to know someone (though of course, everyone’s swimming in invites now.) Its actually not the only way we’re going to be giving A03 accounts–I believe that current account holders are also to be given invites to share–but open sign-up means that even if a fan doesn’t know anybody currently with an account, they’re not disenfranchised.

            FWIW, the archive is being built on a scale to rival; (not that we necessarily want that, but we know that scale is a possibility.)

            Lastly, the queue a continent is a really interesting thought: I like it because it aids in our mission of NOT giving anyone a particular advantage based on cash, age, or geography. *g* I’m not sure if its possible, but I’ll pass it on to the coders–hey, I’m not Technological Girl, I’m the pretty one! ;)

          • Maia Bobrowicz commented:

            I’ll have you know that AD&T is composed of people of such startling personal beauty that we are frequently approached by people crawling on their knees to offer poems of praise. I personally, have inspired an entire chapbook of dirty limericks ;)

          • xkatjafx commented:

            //There IS a plan to make sure our members know about the queue before we
            post broadly or do any wide outreach-//

            Thank you, that’s all I wanted : )

            //I hope you’re right and people WILL come when they realize what the AO3
            really is,//

            I’m fairly sure they will. If any archive ever had a chance to get as big as
            (and hopefully way better than), it’s this one. I’m also hoping the
            newbie fans will realize how great it is to have a big working archive on
            your hands – I mean by now there’s a whole generation who has obviously no
            idea, because they completely “grew up” on LJ.

            //but open sign-up means that even if a fan doesn’t know anybody currently
            with an account, they’re not disenfranchised.//

            I really hope it works out. It’s certainly better than the pure invite-through-friends thing.

            //Lastly, the queue a continent is a really interesting thought://

            Yeah, I’m not really sure it’s possible, though. I mean it’s easy to set up 5 queues at different times, but there’s probably no way to control who goes into which queue (well, at least not without a lot of additional programming.). So you have to hope that people will play fair and not hop onto queues that are obviously less full. (On the other hand it will probably still be better than one queue – because one queue will always make sure that half the people are asleep or at work when it goes up.)

            Anyway, thanks for everything. When I started posting here I wasn’t really sure this would go anywhere (besides annoying you), but it’s nice to see you care. So yes, rest assured I’m happy now and I will renew the membership : )

          • Zooey Glass commented:

            Hi, I’m Zooey from AD&T. I just wanted to let you know that Francesca passed on your concerns to us. Our Chair talks a little more about invitations in her weekly update, which will be posted here later on today. We’re still in the process of building the code for invitations (it’s been slowed down because one of our senior coders has had to step back from the project for a while), so we can’t give you specific technical details just yet. But I wanted to let you know that your comments haven’t gone into a black hole and that we’ll be posting more details about exactly how the invitations system will work in the near future.

          • xkatjafx commented:

            I’m looking forward to read more : ) Anyway, thanks for hearing me out – and keep up the good work! Even though I bet this stunt of mine got on your nerves, I *really* appreciate what you’re doing.

          • Zooey Glass commented:

            Oh dude, not annoying! We’d so much rather hear our members’ concerns and address them. Thanks for supporting us!