Announcing Survey Sunday!

To everyone following news on the OTW’s Survey we conducted last month, we wanted to not only provide you with some more results, but also lay out our plan going forward in terms of

1) What we can release;
2) How quickly we can release it;
3) What sort of schedule you can expect.

Last week’s post, OTW Community Survey: First results, was only a stopgap release while we were putting together a plan for not only dealing with the data, but also figuring out a sustainable schedule for releasing the information as this involves volunteers from various committees as well as coordination between Internationalization & Outreach, which developed the survey, and Communications which will be working on the public posts. In addition to releasing the information publicly, the survey workgroup is working on creating data reports for the internal use of its various committees. And since, as with all OTW projects, we are dependent on the time our volunteers and staffers can make available, the work is more likely to progress in fits and starts than in a consistent way every week.

To that end, we wanted to introduce what we’re calling…

Survey Sunday

Survey Sunday will take place every two weeks now through December, and will allow anyone interested to ask their own questions which can be answered with our big pile of data!

We’ve had a few suggestions to release the survey data set publicly so that the survey analysis can be crowdsourced. However, one of our primary considerations when putting together this survey was ensuring the privacy of our respondents; as such, we decided to release only aggregate data this time around (those of you who took the survey may remember that this was part of the commitment we made in our disclosure statement). Specifically, the statement promised to survey takers that information would only be released in the aggregate (with the exception of specific, anonymous quotes) and that individual responses would be available only to OTW staff and volunteers. (All those people who have access to the data have signed a confidentially agreement which restricts them from discussing the survey contents with anyone outside of the workgroup, including anyone else in the OTW.)

However, we also realize that people are curious to find out specific data given the large number of responses we received, and we want to fulfill those wishes. So to that end, we are going to be offering a regular forum where people can make those requests, and we will turn around and attempt to answer them by the next posting date. We have to include a few disclaimers though, such as:

1) Some results may be preliminary, due to the text answers not yet being included in the overall results (we will specify when this is the case).
2) We may already be planning to answer the inquiry in an upcoming update post: in that case we will let you know once it goes up.
3) Depending upon the volume of requests, we can’t guarantee an answer within the two week period before the next Survey Sunday.
4) Depending upon which questions are being cross-tabulated, we may not be able to come up with a clear response.

And since we are still just starting to go through data, we may have to add to the above disclaimers as we go along and figure out what we can and can’t do.

A look at our process

In the interests of offering regular updates on the survey data, we won’t be publishing data in the order it originally appeared in the survey. Instead, we’ll publish the results out of order, based on how quickly they are processed, and we’ll sometimes give you some preliminary data (for example, data which doesn’t factor in written-in results) while the more complex data is being processed. We think that this will ensure you can be more involved in the process, and can ask questions and discuss the results as they emerge instead of waiting for us to produce one big report.

Not only will different types of questions take different time & effort to evaluate, but different people from the survey workgroup volunteered for different sets of questions, depending on interest and skill set. This spreads out work, but it also means not everyone is available at the same time or can work at the same speed. (In other words, welcome to volunteering at the OTW!)

Here’s an example from the first section of the survey, which covered “demographics & fandom-related questions.”

1. What country/countries do you consider your home?

You might think these answers would be simple to sort but that isn’t always the case. Geographic information isn’t viewed by all people in the same way, so there isn’t any single way to report it, although the linked graph above was put together to give us at least an initial look.

To analyse this question, the first step was to filter out duplicate results that were clearly typos – “United Kindgom” and “UK” are fairly clearly the same place. The next step was the tricky cases – grouping certain countries together can seem more controversial to one person than another, and the names used may also be important. Kudos to the person who justified writing Texas as their home country by giving the years it was independent, but sorry, we’re counting that as part of the USA for these stats. In most other cases, for the purpose of the graph countries were grouped together where there were doubts for the purposes of creating an initial top 10 (e.g. 8% for Canada includes Quebec).

But this data could also be put in graphs in different ways which yield other problems: what would be considered a continental area? And this could separate data — La Reunion is in a different continent from France. We’d like to do more analysis of the second and third countries people listed, and we’ve got a world map that we’re going to update with the full data at some point, but that requires us to re-do that analysis to match the countries the mapping software uses.

2. What is/are your native language/s?

This was another write-in answer, and its 6844 replies need more filtering & categorizing.

7. Do you consume (watch, play, read…) fanworks?

This is relatively simple at first glance (that’s why we picked this for inclusion in our first post), but there were a number of write-in responses that will have to be added into the final results.

9. What general fannish community/communities, if any, do you feel part of?

About the most challenging question from an analsyis perspective, apart from the big essay-style ones. Because this question predictably yielded extremely disparate answers, we need to figure out how to group and classify them meaningfully. While this is perhaps one of the most interesting answers to see the results to, unfortunately it’s not likely to be finished very soon.

How will the survey results affect the OTW’s future?

The survey was designed for two reasons. First, it would provide us information on our members, users, and people aware of the OTW. Second, this information would help us develop policies and procedures for the organization and its projects. For this reason, we are simultaneously creating survey results for the use of our committees and workgroups. However, this work is likely to proceed at the same pace as the public survey results, so it is likely to be many months before committees have all the information pertinent to them. We see the survey as one of the OTW’s major projects for this term, and its data likely won’t be incorporated and implemented until next year.

However, we expect to communicate these effects publicly once the project concludes and all parts of the org have a chance to provide input and reactions to the data. This may be done in conjunction with the strategic plan. The simple answer is that we don’t have a set date for this yet, but it will be part of the project’s conclusion.

Sharing data

We know that people will probably be keen to reblog and share the information we put out. The aggregate data that we publish through OTW news channels may be freely reblogged, shared, linked to, etc. If you post about the data, please provide a link back to the OTW website (survey reports can be found through: https://transformativeworks.org/news-topic-tags/otw-sections/survey-research) so that people can find more information about it, and ask their own questions.

What happens next?

Our first Survey Sunday will kick off this weekend, on May 20 (or 21 for some of our readers). We will begin by posting the full list of questions in the survey. This will enable anyone who either did or didn’t take the survey to see what interests them and start coming up with suggestions for results they’d like to see.

We would like to ask for comments to be made at our website posts (no account needed) which you’ll be able to find through https://transformativeworks.org/news-topic-tags/otw-sections/survey-research. We realize that people read OTW News through other sites and sources as well and it would be more convenient to comment through those sources. However there are several reasons why we’d prefer not to gather comments through them for this project. The first is that not all sites that receive our news feed are being actively posted to, rather the posting has been automated. The OTW currently has eight different outlets just for OTW-News (not counting additional sites/sources for AO3, Fanlore news, OTW Tag Wranglers, etc.) While most of these sites are set up to deliver responses to our Communications Committee we have been having technical problems caused by spam blacklisting that mean that some messages do not get delivered from all sites, and some sites like Twitter are currently not getting delivered at all. Our Systems team is working on solutions for this but in the meantime, to be certain your comment gets seen, please make it at our website. This also saves time for the survey workgroup team in rounding up your comments.

Thank you for your patience as we work on this project. We welcome your input and look forward to sharing exciting developments from the results with you soon.

In the meantime, here are a few more graphs we have pulled together from data where we have complete results:

Results: Graphs for questions #54,#56,#70,#71