NatCen Social Media Research: What Users Want

fryAt the beginning of October 2013, there were reportedly 1.26 billion Facebook users worldwide. The number of Tweets sent per day is over 500 million. That’s a lot of communication happening every day! Importantly for researchers, it’s also being recorded, and because social media websites offer rich, naturally-occurring data, it’s no wonder researchers are increasingly turning to such websites to observe human behaviour, recruit participants, and interview online.

As technology constantly evolves, researchers must re-think their ethical practices. Existing guidelines could be adapted ad-hoc, but wouldn’t it be better to rethink the guidelines for this new paradigm? And what do social media users think about research that utilises social media? The work of the “New Social Media, New Social Science?” network in reviewing existing studies suggests that these questions have not yet been adequately answered.

In response, a group of NatCen researchers are soon to report data from a recent study on how social media users feel about their posts being used in research, and offer recommendations about how to approach ethical issues.

What do participants want?

A key ethical issue participants talked about was consent: participants wanted researchers to ask them before using their posts and information. Although it was acknowledged that “scraping” a large number of Tweets would pose practical problems for the researcher trying to gain consent, users would still like to be asked. Consent was seen as particularly important when the post contained sensitive or personal information (including photographs that pictured the user). An alternative view was that social media users shouldn’t expect researchers to gain consent because, by posting online, you automatically waive your right to ownership.

Participants’ views about consent were affected by other factors, including the platform being used. Twitter, for example, was seen as more public than Facebook so researchers wouldn’t necessarily need to ask for the user’s permission to incorporate a Tweet in a report.

Views about anonymity were less varied. Users felt anonymity should be afforded to all, especially if posts had been taken without consent. Users wanted to remain anonymous so that their posts wouldn’t be negatively judged, or because they were protecting identities they had developed in other contexts, such as at work.

Our participants were also concerned about the quality of information posted on social media. There was confusion about why researchers would want to use social media posts because participants felt that people didn’t always present a true reflection of themselves or their views. Participants noted, for example, how users post pictures of themselves drinking alcohol (which omits any mention of them having jobs or other, more serious things!), and that ”people either have more bravado, and ‘acting up’ which doesn’t reflect their real world self”. They expressed concern over this partial ‘self’ that can be presented on social media.

What does it mean?

Later this year, NatCen will publish a full report of our findings, so stay tuned! If you can’t wait, here’s a preview:

  • Consider that users’ posts and profiles may not be a reflection of their offline personality but an online creation or redefinition;
  •  Even if users are not utilizing privacy settings they still might expect you to ask permission to use their post(s);
  • Afford anonymity. Even if someone has let you know you can quote their username, you should learn how ‘traceable’ this is and let the user know (i.e. can you type their username into Google and be presented with a number of their social media profiles?). It’s our responsibility as researchers that the consent we get is informed consent.

Let us know at NatCen if you would like to receive an electronic copy of the report, or if you have any questions about the study.

Survey: What’s in a Word?

As those of us in the survey research field are aware, survey response rates in the United States and other countries have been in decline over the last couple decades.  The Pew Research Center sums up the concerning* state of affairs with a pretty eye-popping table showing response rates to their telephone surveys from 1997 (around 36%) to 2012 (around 9%).  Others have noted, and studied the same phenomenon.

So what’s really going on here?  There are plenty of explanations, including over-surveying**, controlled access, and a disinterested public.  But what else has changed about sampled survey respondents or their views towards surveys in recent years that might contribute to such a drop?  As a survey methodologist, my first instinct is to carry out a survey to find the answer.  But conducting a survey to ask people why they won’t do a survey can be like going fishing in a swimming pool.

One place many people*** are talking these days is on social media.  In the past decade, the percentage of Americans using social media has increased from 0 to “most.”  I was curious to see how the terms survey and surveys were being portrayed on online and social media.  Do those who use (or are exposed) these terms have the same things in mind as we “noble” researchers?  When we ask someone to take a survey, what thoughts might pop into his or her mind?  Social media is by no means the only place to look****, but there is a wealth of data out there and you can learn some interesting things pretty quickly.

Using Crimson Hexagon’s ForSight platform, I pulled social media posts that included the word survey or surveys from 2008 (the earliest data available) to today (January 8, 2014).  First I looked to see just how often the terms showed up by source.  Here’s what I found:

Survey

In sheer volume, Twitter seems to dominate the social media conversation about surveys, which is surprising given that only about 1 in 6 U.S. adults use it. Of course, just because the volume is so high doesn’t mean everyone is seeing these posts.  The surge in volume is quite dramatic late in 2012!  Maybe this had to do with the presidential election?  We’ll see… keep reading!  My next question was what are they saying when it comes to surveys?  I took a closer look at the period before that huge spike in 2012, focusing just on those co-occurring terms that pop up most frequently with survey(s).  I also split it out by Twitter and non-Twitter to see what comes up.

clouds1

We see according, each, and online for Twitter posts and according, found, and new for all other social media.  Hmm, what could this mean?  Drilling down into each term, we can look at individual posts for each term.  I include just one example for each here just to give a flavor of what the data show:

Twitter 5/08-7/12

  • According to one national survey on drug use, each day…
  • D-I-Y Alert! Nor your usual survey job $3 each – Research: This job….
  • We want you 2 do online survey and research for us. Easy…

Other online media 5/08-7/12

  • Nonresidential construction expected to lag in 2010 according to survey…
  • Surprise! Survey found young users protect their privacy online
  • New survey-Oil dips on demand worry, consumer view supports

Among these sample posts, we survey results being disseminated from several kinds of surveys on both Twitter and other online media.  The Twitter posts, though, seem to have more to do with earning money online than other social media.  Next, I looked at August 2012 to today (January 8, 2014):

clouds2

Among the other online media, there’s not much change here from the previous period.  People replaces found among top co-occurring terms, but the focus is still on survey results.  For Twitter, we see a new top 3 terms co-occurring with survey(s): earned, far, and taking.  Here’s what some of the Tweets from the more recent period look like:

Twitter 8/12-1/14

  • Awesomest week ever! I earned $281.24 just doing surveys this week :)
  • Cool! I got paid $112.29 so far from like surveys! Can’t wait for more!
  • What the heck – I got a free pizza for taking a survey!

Now, I know that most of this is pure Twitter spam***** and not every Tweet is read or even seen by the same number of people, but I do think the increasing predominance of ploys to sign up people for paid surveys on networks like Twitter is a sign that term survey is being corrupted in a way that, if it does not contribute to declining response rates, surely doesn’t help matters.  They leave an impression and if these are the messages some of our prospective respondents have in mind when we contact them with a survey request, we are facing an even steeper uphill battle that we might have thought.

So, this leads us back to the classic survey methods question: what should we do?  How do we differentiate the “good” surveys from the “bad” surveys among a public who likely finds the distinction less than salient and won’t bother to read a lead letter, let alone open a message that mentions the word survey? Should we come up with a new term?  Does study get across the task at hand for the respondent?  Would adding scientific before survey help keep our invitations out of trash cans, both physical and virtual?

What are your thoughts on the term survey? Leave a comment here, or discuss on your favorite listserv or social media platform.  If you do, I promise not to send you a survey about it!

*scary=the degree to which lower response rates equate to lower accuracy, which isn’t always the case

**Personally, I sympathize with respondents when I get a survey request on my receipt every time I buy a coffee or sign up for a webinar.  “Enough already with the surveys!  I’ve got surveys to write!”

***not all people, and not all kinds of people, but still many!

****A few years ago, Sara Zuckerbraun and I looked at the portrayal of surveys in a few select print news media.

*****Late 2012 appears to have been a golden age for Twitter spam about paid surveys.

Social Media, Sociality, and Survey Research: Broadcast Communication

In our new book, Social Media, Sociality, and Survey Research, Craig Hill, Joe Murphy, and I define the sociality hierarchy, a framework we’ve used to conceptualize how individuals use digital media to communicate with each other. The sociality hierarchy presents three levels of communication: broadcast (one-way), conversation (two-way), and community (within groups). This post is about broadcast communication. Broadcast communication describes the behaviors of expressing thoughts, opinions, and status statements in social media and directing them at an audience. In the broadcast level, individuals send a one-way message to any friends or followers who happen to be tuning in. Broadcast social media communication includes things such as Tweets, status updates, check-ins and blogs, as well as YouTube videos, book reviews on Goodreads and restaurant reviews on Yelp.

Our book features two chapters presenting analytic techniques for broadcast data. In “Sentiment Analysis: Providing Categorical Insight into Unstructured Data,” Carol Haney describes a sentiment analysis methodology applied to data scraped from a vast range of publicly available websites, including Twitter, Facebook, blogs, message boards and wikis. She describes the steps of preparing a framework for data capture (what data to include and what not to include), harvesting the data, cleaning and organizing the data, and analyzing the data, with both machine coding and human coding. In her research, Haney finds sentiment analysis to be a valuable tool in supplementing surveys, particularly in market research.

Sentiment analysis has challenges too. The complexities of analyzing statements expressing sarcasm and irony and decoding words with multiple meanings can make the process more difficult or the results more open to interpretation. Despite the challenges, sentiment analysis of unstructured text yields insights into organic expressions of opinion by consumers that may never be captured through surveys. Two examples Haney provides:

  • A swimsuit company learned from blog content that they were not providing enough bikinis in large sizes. A subsequent market research survey confirmed that more than 25% of 18-34 year-olds could not find their size in the brand’s swimsuit line.
  • A public messaging campaign sought out factors that motivate young people to avoid smoking by scraping publicly available Twitter and Facebook data. The campaign determined that watching loved ones live with chronic illness or die because of the side effects were painful events. The campaign referenced this in the messaging and validated the effectiveness of the messages with subsequent testing with research subjects.

In “Can Tweets Replace Polls? A U.S. Health-Care Reform Case Study,” Annice Kim and coauthors analyze Tweets on the topic of health care reform as a case study of whether Twitter analyses could be a viable replacement for opinion polls.  Kim’s chapter describes the process of searching for and capturing Tweets with health-care reform content, coding Tweets using a provider of crowdsourced labor, and the comparison of analyses of Twitter sentiment to the results of a probability-based opinion poll on health-care reform. Ultimately, Kim and her coauthors found that sentiment expressed on Twitter was more negative than sentiment expressed in the opinion poll. This does not correspond with earlier studies on other topics, which demonstrated correlation between Twitter sentiment and opinion poll data on consumer confidence and presidential job approval. For certain types of sentiment, Twitter results correlate with public opinion polls, but for other types, Twitter is not currently a viable replacement for polling research.

These two chapters in our book represent a subset of the types of analyses that can be done with broadcast social media data. Are your findings similar? Have you worked with other types of online content? Or do you think of one-way communication differently than the broadcast model?

A Birds-eye View of the 2013 AAPOR Twittersphere

As Joe Murphy mentioned in the preceding post, Twitter can provide a unique and efficient glimpse into conference dialogue that may be difficult to capture in other ways. We can measure when conversations occur between tweeters, the subject of those conversations, and other interesting patterns such as the most frequent words appearing in Tweets or the most popular Tweet (as indicated via retweets). There’s one analytic possibility, though, that’s frequently overlooked: social network analysis (SNA).

As I discussed in my poster presented at this year’s AAPOR conference, something inherent and potentially very useful to researchers is the network aspect of social networking sites. Inspired by a growing Twitter presence among the AAPOR community, and Joe’s analysis of the AAPOR Twittersphere, I decided to put some of the SNA concepts I discussed in my poster to practice. Specifically, in my attempt to examine public opinion in a different light, I asked the question “who is talking to whom?”

For those of you familiar with SNA, or are just curious as to how the measures I discuss are actually quantified, I’ve pulled the equations and explanations from my poster and inserted them for reference. In the first image below, a birds-eye view captures what the AAPOR Twitter conversation looked like. By conversation, I’m specifically referring only to Tweets where one user Tweeted to another using the hashtag #AAPOR. For instance, if (hypothetically speaking) Joe (@joejohnmurphy) Tweets to me “hey @AdamSage, your poster was A-M-A-Z-I-N-G #AAPOR,” the link connecting Joe to me would be directed from Joe to me.

At first glance, this birds-eye view looks kind of cool (or like a big mess depending on your perspective) – but what does it mean? Each dot, or node, represents a Twitter user that at one point used the hashtag #AAPOR and mentioned another Twitter user in a Tweet. Each line represents the association that’s created when such a Tweet occurred. The degree of each node is number of different connections made as a result of these Tweets. In the AAPOR Twittersphere, the average degree, or the average number of people to whom a given Twitter user is linked via these dialogues, is slightly over 2 (2.017). This is important because an average degree of 1 would mean we’re not a community, but rather a group of 1 on 1 dyadic cliques, which makes ideas difficult to spread. While these did occur, I excluded them in these graphs and focused what is called a giant component in SNA, or the largest group of connected individuals in a network.

Excluding those conversations occurring outside of the giant component, the diameter of the entire conference Twitter conversation was 8. In other words, the furthest any two people were away from each other during this Twitter conversation was 8 steps. Imagine yourself at the largest Tweet-up in AAPOR history consisting of those AAPOR attendees participating in a dialogue on Twitter (using #AAPOR of course). As people traverse from conversation to conversation, everything you say would be no more than 8 conversations away from anyone in the room. So that rumor about my awesome poster could theoretically spread quite quickly. Hopefully it’s starting to become clear how great papers and keynote addresses can become popular Twitter topics in a given year. Great papers resonate and travel quickly; great keynotes are viewed by a lot of people and travel even faster.

But wait, it gets better. Ideas spreading among the AAPOR community wouldn’t traverse the network as you might think. Looking at the overall graph below you will notice several color-coded groups of tweeters. Each color represents a community, which is essentially defined as those tweeters who are mentioning each other in their Tweets more than others. For instance, community around @AAPOR (the green colored node with several links in the top middle) does not include many of the users with high betweenness and closeness. Betweenness measures the strength of one’s connection to the entire network (i.e., how interconnected or embedded one is to the entire network), and closeness measures one’s distance to the overall network (an average distance). They are measured as:

Betweenness:

Closeness:

As you might expect, these people tend to drive conversations because they have many connections, which when analyzed using certain measures of network centrality, can reveal likely members of such conversations. For instance, just by measuring mere mentions within Tweets, clustering coefficients (defined as a given groups number of connections divided by the number of possible connections) allow us to distinguish “communities” from the rest of the network. Because this graph is directed, meaning I can tell if someone is doing the Tweeting or being Tweeted at, I know a majority of @AAPOR links were created by people mentioning @AAPOR in a Tweet, rather than @AAPOR mentioning others. The community around @AAPOR (the green colored node with several links in the top middle circled in red) does not include many of the users with high betweenness and closeness because @AAPOR’s propensity to draw Tweets to it naturally steals the thunder of everyone in close proximity.

While I haven’t analyzed the content of the conversations occurring among these communities, my guess is there are general topics (e.g., polling, elections, survey methods, statistics etc). After all, we tend to Tweet about what we’re interested in. In fact, just looking at the categories identified in Joe Murphy’s post, Tweets coming from @AAPOR and the highest frequency Tweeter (who I’ll leave unnamed, but I have circled in yellow) were generally categorized differently with the former much more often covering general conference information and the latter results from paper sessions.

While this is a quick analysis of the Twittersphere, these analyses can reveal characteristics of conversations and conversation participants that can potentially lead to other discoveries, such as the propensity for ideas to spread (or fade) among certain groups, identifying which individuals (and their behaviors) are key to group cohesiveness, and how we can eventually focus efforts to create a more robust environment for innovation, e.g., creating a hashtag to link people and ideas in the survey and public opinion research Twittersphere, much like market researchers use #MRX.

AAPOR 2013 – The view from the Twittersphere

Last week, more than 1,000 public opinion researchers convened in Boston for the annual meeting of the American Association for Public Opinion Research (AAPOR). The four day conference was centered on the theme “Asking Critical Questions” and included papers, posters, courses, and addresses from top researchers in the field.

As with each of the last several conferences, some attendees (and non-attendees) took to Twitter to discuss the results being shared and connect with colleagues old and new.  Our analysis of the #AAPOR hashtag (the “official” hashtag for the conference) shows more than 1,500 Tweets during the conference by more than 250 Tweeters.  About three quarters of these were original Tweets and the remaining quarter reposts (retweets) of #AAPOR posts made by others.  This is up about 500% from Tweeting at the 2010 conference. Five Tweeters posted more than 50 times during the 2013 conference, but more than half only posted once.

The conference activities kicked off on Wednesday 5/15 with afternoon short courses and wrapped up on Sunday 5/19.  Overall Tweet volume peaked on Saturday, though the most original Tweets (total minus retweets) were posted on Friday.

By hour, Tweet volume peaked during AAPOR events such as the plenary, President’s address, and award ceremony.  There was also a peak during the Saturday morning paper sessions with many Tweeters relaying thoughts shared by high-profile researchers like Jon Krosnik and Tom Smith.  In the chart below, the solid lines show volume excluding and the dotted lines including retweets.  Noon is indicated by the vertical line above each day.

We sorted original Tweets by time and looked at every fourth one to get a sense of the popular topics of discussion.  Content and research findings from presentations topped the list, making up about a third of all original tweets.

The most retweeted post was from @AAPOR announcing the release of the Non-Probability Task Force Report.  In the spirit of transparency though, it should be said that some (including me) were asked to retweet this announcement.  The post was retweeted 26 times.

Here is a quick word cloud of #AAPOR tweet content during the conference (the phrase “AAPOR” removed, for obvious reasons).

Interestingly (to us, anyway), the #AAPORbuzz experiment was more of an #AAPORbust… Few attendees were interested in replying to our Tweeted survey items, despite endorsement from @AAPOR itself.  It may be that attendees were more interested in sharing just what they wanted when they wanted and weren’t looking to respond to a survey/poll/vote simultaneous with discussing the topic of surveys itself at the conference.  It would be interesting to find out more about why people did not respond to these items.  Were they lost in the sea of Tweets?  Would another outreach approach be more effective? Would this work with a different population?  Further experimentation should help answer these questions.

Also interesting was the increased focus on Twitter itself at the conference.  In addition to the short course I gave with @carolsuehaney (Carol Haney), there were papers focused on Twitter analysis and at least one company (Evaluating Effectiveness), downloading #AAPOR Tweets and posting a dataset on their website. Popular bloggers like @mysterypollster (Mark Blumenthal) used Twitter to announce and link to their blog posts covering the conference.

Is Twitter here to stay as a mode of communication and topic of research for public opinion research?  Time will tell, but adoption and utility for those in the field seems to be on the rise.

 

Tracking 2013 AAPOR Buzz on Twitter – Just for Fun!

For those of you who have attended an AAPOR conference since 2010, you may have seen the Twitter Board that RTI sponsors annually (see below). Over the past few years, many of us at SurveyPost have focused on discovering and exploring the potential of social networking sites (e.g., Facebook and Twitter) in research. Although we will be presenting on these topics once again this year, we will also be putting some of what we’ve learned about analyzing Twitter data to good use.

In cooperation with (and great thanks to!) AAPOR planning and executive committee members, we have developed a series of fun questions that we will field via Twitter. Aware of the limitations of using Twitter data, and knowing this is the first effort of its kind at an AAPOR conference, we hope many of you will find the results and analyses we do interesting and, well, fun!

If you’re excited to participate (which we know you all are), simply follow the #AAPOR hashtag. For these particular questions, we are also using the hashtag #AAPORbuzz. You’ll see questions coming from our Twitter handle @SurveyPost, so be sure to follow us. We look forward to your Tweets!

 

10 Things Survey Researchers Should Know About Twitter (Part 2)

Yesterday I shared the first half of the ten things every survey researcher should know about Twitter.  Today I give you things 6-10:

6.      There are research applications beyond trying to supplant survey estimates.

Think about the survey lifecycle and where there may be needs for a large, cheap, timely source of data on behaviors and opinions or a standing network of users to provide information.  In the design phase of a survey, can we use Twitter to help identify items to include?  Can we identify and recruit subjects for a study using Twitter?  How about a diary study when we need a more continuous data collection and want to let people work with a system they know instead of trying to train them to do something unfamiliar?  Can Twitter be used to disseminate study results? What about network analysis?  Is there information that can be gleaned from someone’s network of friends and followers, or the spread of Tweets from one (or few) users to many? We often think of public opinion as characterizing sentiment at a specific place and time, but are there insights to be had from Twitter on opinion formation and influence?

7.      Twitter is cheap and fast, but making sense of it may not be. 

What’s the unit of analysis? Can we apply or adapt the total survey error framework when looking at Twitter?  What does it mean when someone Tweets as opposed to gives a response in a survey? Beyond demographics, how do Twitter users differ from other populations?  How can we account for Twitter’s exponential growth when analyzing the data?  The best answer to each right now is “it depends” or “more research is needed.”  We need a more solid understanding and some common metrics as we look to use Twitter for research.  Work on this front is beginning but has a long way to go.

8.      Naïve and general text mining methods for tweets can be severely lacking in quality.

The brevity of Tweets, inclusion of misnomers, misspellings, slang, and sarcasm make sentiment analysis a real challenge.  We’ve found the off-the-shelf systems pretty bad and inconsistent when coding sentiment on Tweets.  If you’re going to do automated sentiment analysis, be sure to account for nuances of your topic or population as much as possible and have a human coding component for validation. One approach we’ve found to be promising is to use crowdsourcing for human coding of Tweet content.

9.      Beware of the curse of big data and the file cabinet effect. 

Searching for patterns in trillions of data points, you’re bound to find coincidences with no predictive power or that can’t be replicated. The file cabinet effect is when researchers publish exciting results about Twitter but hide away their null or negative findings.

10.      Surveys aren’t perfect either.

Surveys are getting harder to complete with issues like declining response rates and reduced landline coverage.  Twitter isn’t a fix-all but it may be able to fill some gaps.  It’ll take some focused study and creative thinking to get there.

10 Things Survey Researchers Should Know About Twitter (Part 1)

Over the last couple years, I’ve been looking at Twitter’s potential in survey research.  Why Twitter?  Because it’s vast, it’s fast, and it’s cheap.  Recently, at the 2013 FedCASIC Workshops, I presented ten things survey researchers should know about Twitter. Below are five of the ten (I’ll present the other five tomorrow!).

1.       Twitter is like a giant opt-in survey with one question.

Twitter started in 2006 with a simple prompt for its users: “what are you doing?” From a survey methodologist’s perspective, this isn’t really optimal question design.  How people actually use twitter is so varied, there might as well be no question at all.  We aren’t used to working with answers to a question no one asked, and Twitter is a good example of what has been described as “organic” data – It just appears without our having designed for it. Tweets are limited to one hundred and forty characters in length. Pretty short, but a Tweet can capture a lot of information, and include links to other websites, photos, videos, and conversations.

2.       Twitter is massive.

Every day, half a billion Tweets are posted.  Half a billion!  That means by the time you finish reading this, there will be approximately one million new Tweets. And the pace is only growing. With Twitter’s application programming interface (API) you can pull from a random one percent of Tweets.  To get at all Tweets, or the Firehose (one hundred percent of Tweets), you need to go through one of a few vendors and for a fee, though the library of congress is working on providing access in the future.

3.       Twitter is increasingly popular on mobile devices like smartphones and tablets.

You’ll see people Tweeting at events, as news is happening right in front of them, or where you don’t really expect or want to see them Tweeting, like while they’re driving. Many use Twitter on mobile devices with another screen on at the same time.  That’s called multiscreening. Like when people Tweet while watching television in a backchannel discussion with friends and fans of their favorite shows.

4.       The user-base is large, but it doesn’t exactly reflect the general population. 

It would be kind of weird it if did, honestly.  There are surely many factors that influence the likelihood of adoption and wouldn’t it be surprising if we saw no differences by demographics? The Pew Research Center estimates sixteen percent of online Americans now use Twitter, and about half of those do so on a typical day.  Users are younger, more urban, and disproportionately black non-Hispanic compared to the general population.  This is interesting when thinking about new approaches for sometimes hard to reach populations.

5.       It is made up of more than just people. 

Twitter is not cleanly defined with one account per person or even just one person behind every account.  Some people have multiple accounts and some accounts are inactive. Groups and organizations use Twitter to promote products and inform followers.  They can purchase “promoted Tweets” that show up in users’ streams like a commercial.  And watch out for robots! Some software applications run automated tasks to query or Retweet content making it extra challenging when trying to interpret the data.

Social media for social science: The imperfect window

Much of the world, it seems, has been atwitter about social media in recent years. Researchers are no exception. Rather than needing to solicit insight from people with telephone calls during dinner or mailing surveys that largely end up in the trash, social scientists now have readily available tools to observe people’s thoughts and ideas, posted publicly. We also can now easily track, at least in the aggregate, what information people are seeking. As Google has emerged as close confidante to many of us, we collectively can track concerns about the flu, interest in political party conventions, and what questions people have about nutrition. All of these developments suggest a veritable gold mine for social science.

Researchers have responded in earnest. As Senior Editor for Health Communication, I have noticed a distinct uptick in the percentage of submitted papers that rely in some fashion on electronic surveillance rather than formal solicitation of survey respondents. I have even joined the party myself a number of times.  A few years ago, for example, former graduate student Brian Weeks and I looked at search interest in (completely unsubstantiated) rumors about Barack Obama, as measured by Google search data, and its direct (if ephemeral) correspondence to television and print news coverage. Whether we should rush headlong toward this research approach without caveats, though, is an open question.

Mounting empirical evidence suggests that we vary substantially in our engagement with social media and yet the exact nature of that variation is not fully understood or appreciated by researchers. Much has been made of the so-called digital divide, which suggests the role of socioeconomic factors in explaining Internet use and a gap between those with access to technology and those without. The electronic media landscape has changed since the 1990s, however, and economic factors may not be the most powerful predictor of social media technology any longer. Spokespeople from IBM have forecast the imminent closure of the digital divide as more and more people from a range of socioeconomic backgrounds adopt mobile technology that allows ready access to the Internet. Despite these changes, we cannot say that people do not differ fundamentally in using social media. A recent paper suggests that our basic personality is evident in our pattern of engagement with Facebook, for example.

What we also know is that the public display of information and information sharing between people vary as a function of topic, circumstance, and even available social network ties. A few years ago, collaborators and I found that viral marketing for a free mammography program was constrained by the social ties available in one’s immediate community. In a different example, colleagues and I recently found in a study of household energy tip sharing between people that relatively few people opted to post such information via social media (as opposed to other means of interpersonal communication). As I outline in a new book – Sharing Disparities: Social Networks and Popular Understanding of Science and Health – to be published this year by RTI Press, information itself is not equal in its tendency to be shared. Emotionally provocative information or information that addresses a pressing situation of uncertainty, for example, seem more prone to sharing than other types of information (hence the proliferation of rumors relative to dry expository information). Moreover, Pew recently reported substantial discrepancy between Twitter sentiment and that assessed through other public opinion measurement.

What does all of this mean for social scientists interested in leveraging our electronic forays as evidence of generalizable thoughts and sentiments? It does not suggest that there is no utility in such data; far from it. Research using such datasets is noteworthy and has proven useful in detecting the emergence of urgent concern, e.g., searches for flu symptoms. Nonetheless, we need to be cautious in suggesting that the only generalizability limitation for Internet-based research involves socioeconomic disparity. Who publicly posts, what and when they post, who forwards content and to whom they forward, and even who searches are all constrained by fluctuation in individual circumstance, topical salience, social norms, and the availability of technology and social network resources. We need more research regarding these constraints to better understand when, and how much of, the glittering mine of big data from social media is actually valuable and what and whom it represents.

The Advantages of Crowdsourcing Through Twitter

A colleague of mine recently shared an NPR story on Women Under Siege, a project using crowdmapping to gather real time data on rape and other forms of sexualized violence in Syria. Women Under Siege collects reports from survivors, witnesses, and first-responders via a web form, email, SMS, and Twitter (#RapeinSyria). The data are then analyzed by public health researchers and reports are plotted on a crowdmap using the open source Ushahidi Platform.  The map provides a visual reminder of the prevalence of this violence, further emphasizing the importance of this public health research.

I was initially surprised to read that Women Under Siege collects these data on Twitter. I assumed that accounts of sexualized violence are rare on Twitter for the same reasons this violence tends to be underreported in surveys (shame, stigma, fear of retaliation, etc.). And it turns out that this reporting method is underutilized. Even though the project seeks reports of violence via Twitter and other methods, so far all 137 reports submitted have been submitted via the web form. Twitter issues aside, I remained skeptical about whether crowdsourcing was actually beneficial for such a sensitive topic.

However, as I read more about the project, I started to see the benefits of crowdsourcing to collect these data. Compared to traditional survey data collection, this crowdsourced approach has several benefits. First, crowdsourcing is much cheaper than conducting a survey. Second, data are collected and available much quicker. The crowdsourced data could be available within hours of the violence taking place, compared to potentially months for survey data. Third, crowdsourcing enables anonymous reporting through Women Under Siege’s web form. Anonymity is assured in legitimate surveys, but respondents may question how their information will be protected and they may hesitate to reveal sensitive information to an interviewer. Fourth, crowdsourcing has the potential to reach out to a broader group of people, including those with only second or third hand knowledge of the violence. Although these respondents may have fewer details and perhaps some inaccurate details of the violence, they may be more inclined to report all the information they have. Perhaps it is worth accepting less accuracy in the details of the incidents to gain this perspective on the scope.

Overall, is reaching out to more people through crowdsourcing better? Normally I’d say no, that it’s more important to draw a representative sample to make inferences to the target population, as is done in scientifically rigorous surveys. In this case, however, I can see the benefits of collecting as many reports as possible via crowdsourcing. For instance, collecting these reports in real time draws more attention to the prevalence and seriousness of this violence in Syria. Also, the method provides what is perhaps a more accurate snapshot by reducing the likelihood of underreporting by reaching out to more people who may have encountered sexualized violence and offering them a more anonymous way to report it. As with a survey, the findings should be interpreted with a critical eye and the shortcomings of the methods made clear. But even if these methods may not provide the probability-based assurance a survey may, they provide a relatively efficient and timely glimpse where traditional surveys may not provide the best cost/benefit for the job.  Where else might these methods be appropriate in addition to or in place of a survey?  We welcome your thoughts!