You Are What You Tweet: An Exploration of Tweets as an Auxiliary Data Source

Last fall at MAPOR , Joe Murphy presented the findings of a fun study he did with our colleague, Justin Landwehr, and me. We asked survey respondents if we could look at their recent Tweets and combine them with their survey data. We took a subset of those respondents and masked their responses on six categorical variables. We then had three human coders and a machine algorithm try to predict the masked responses by reviewing the respondents’ Tweets and guessing how they would have responded on the survey. The coders looked for any clues in the Tweets, while the algorithm used a subset of Tweets and survey responses to find patterns in the way words were used. We found that both the humans and machine were better than random in predicting values of most of the variables.

We recently took this research a step further and compared the accuracy of these approaches to multiple imputation, with the help of our colleague Darryl Creel. Imputation is the approach traditionally used to account for missing data and we wanted to see how the nontraditional approaches stack up. Furthermore, we wanted to check out these approaches because imputation cannot be used in the case where survey questions are not asked. This commonly occurs because of space limitations, the desire to reduce respondent burden, or other factors. I will be presenting on this research at the upcoming Joint Statistical Meetings (JSM), in early August. I’ll give a brief summary here, but if you’d like more details on it please check out my presentation or email me for a copy of the paper.

Income was the only variable for which imputation was the most accurate approach, but the differences between imputation and the other approaches were not statistically significant. Imputation correctly predicted income 32% of the time, compared to 25% for human coders and 26% for the machine algorithm. Considering that there were four income categories and a person would have a 25% chance of randomly selecting the correct response, I am unimpressed with these success rates of 25%-32%.

Human coders outperformed imputation on the other demographic items (age and sex), but imputation was more accurate than the machine algorithm. For these variables, the human coders picked up on clues in respondents’ Tweets. I was one of the coders and found myself jumping to conclusions, but I did so with a pretty good rate of success. For instance, if a Tweeter said “haha” a lot or used smiley faces, I was more likely to guess the person was young and/or female. These are tendencies that I’ve observed personally but I’ve read about them too.

As a coder I struggled to predict respondents’ health and depression statuses, and this was evident in the results. Imputation was better than humans at predicting these, but the machine algorithm was even more accurate. The machine was also best at predicting who respondents voted for in the previous presidential election, with human coders in second place and imputation in last place. As a coder I found that predicting voting was fairly simple among the subset of respondents who Tweeted about politics. Many Tweeters avoided the subject altogether, but those who Tweeted about politics tended to make it obvious who they supported.


So what does this all mean? We found that even with a small set of respondents, Tweets can be used to produce estimates with accuracy in the same range or better[1] as imputation procedures. There is quite a bit of room for improvement in our methods that could make them even more accurate. For example, we could use a larger sample of Tweets to train the machine algorithm and we could select human coders who are especially perceptive and detail-oriented. The finding that Tweets are as good or better as imputation is important because imputation cannot be used in the case where survey questions were not asked.

As interesting as these findings may be, they need to be taken with a grain of salt, especially because of our small sample size (n=29).[2] Relying on Twitter data is challenging because many respondents are not on Twitter, and those who are on Twitter are not representative of the general population and may not be willing to share their Tweets for these purposes. Another challenge is the variation in Tweet content. For example, as I mentioned earlier, some people Tweet their political views while others stay away from the topic on Twitter.

Despite these limitations, Twitter may represent an important resource for estimating values that are desired but not asked for in a survey. Many of our survey respondents are dropping clues about these values across the Internet, and now it’s time to decide if and how to use them. How many clues have you dropped about yourself online? Is your online identity revealing of your true characteristics?!?


[1] Even if approaches using Tweets may be more accurate than imputation, they require more time and money and in many cases may not be worth the tradeoff. As discussed later, these findings need to be taken with a grain of salt.

[2] We had more than 2,000 respondents, but our sample size for this portion of the study was greatly reduced after excluding respondents who don’t use Twitter, respondents who did not authorize our use of their Tweets, and respondents whose Tweets were not in English. Furthermore, half of the remaining respondents’ Tweets were used to train the machine algorithm.


Ashley will be presenting this research at the 2014 Joint Statistical Meetings in Boston, MA.

Session: 105

Date: 8/4/2014

Time: 8:30am

Location: CC-213

Watching the Fireworks Explosion on Google and Twitter

When I woke up this morning, I remembered I have the day off tomorrow.  Independence Day (July 4th) brings many images to mind in the United States, but one of the most common, and potentially dangerous, is “fireworks.”  The Nationwide Emergency Department Sample reports that between 2006 and 2010 fireworks-related injuries in the U.S. were most common in July (68.1%), followed by June (8.3%), January (6.6%), December (3.4%), and August (3.1%).  I was interested to see if others were thinking and/or talking about fireworks leading up to the holiday.  Perhaps this would suggest a “population at risk.”

Without a budget and still in my pajamas, I turned to a couple go-to sources for this kind of very cursory look – Google Trends and Twitter.  Google Trends allows you to see the relative volume of search activity on different terms over time and by geography.  To me, it is a really rough proxy of what people are thinking about.  Of course, not all people use Google or even have easy access to it.  Just because they are thinking about something doesn’t mean they’ll be searching for information on it. Even when they are searching on it, there’s no guarantee they are spelling it like I do or even using the same terms. Even if they are searching on the same term with the same spelling, maybe they’re looking for something else.  Still, in about 5 seconds, I can get a glimpse of some interesting trends, and I still haven’t changed out of my pajamas.  If Google might be a rough proxy of what people are thinking about, Twitter may be an equally rough proxy of what people are talking about, with some of the same and some of its own caveats.  To get those results, I go to Crimson Hexagon’s Forsight tool.

Here’s the Google search volume for “fireworks” over the last several years:

“Fireworks” Google Search Volume

Fireworks_Google Search Volume

The big spikes are in July, as I expected.  What about those secondary bumps?  On November 5, the U.K. celebrates Guy Fawkes Night.  Repeating this by country confirms the association:

U.S. “Fireworks” Google Searches Spike on July 4

Fireworks Google Search Spike

U.K. “Fireworks” Google Searches Spike on November 5

UK Fireworks Search Volume

Here are the raw volume numbers from Twitter.  Keep in mind that some of the overall increase here is due to the increase is popularity of Twitter itself over time.

“Fireworks” Total Posts on Twitter

Fireworks Twitter Volume

That second little bump is New Year’s Eve, another big fireworks night and also high on the emergency-room visit list.

And just what are people in the U.S. saying on Twitter about fireworks leading up to July 4?  Forsight’s “Clusters” gives some clues:

“Fireworks” Twitter Post Clusters, 6/27-7/2/2014

4th of July Word Cloud

Digging into a few of those terms makes it clear what many are saying or sharing.  For example, the Boston fireworks show has been moved from the 4th to the 3rd, there are methods to keep your pets calm during the fireworks, and the Onion is still a go to source for some holiday satire.

I’m tempted to dig further into these data, but its time to change into my day clothes and do some survey work.  Stay safe and have a Happy 4th!


RTP180 Series Explores Social Media in the Triangle

On the third Thursday of each month, the Research Triangle Park Foundation hosts panels of local speakers for a community event called RTP180. This month, the topic was Social Media, and speakers included representatives from local institutions and organizations, ranging from private startups to major academic universities to beer breweries.

The atmosphere at these events is always informal and fun, and the 5-minute presentations are just long enough to convey key points, but short enough to maintain the crowd’s interest. Social media is certainly a hot button topic in recent years, and it was interesting to see these organizations share how they utilize social media to convey unique and usable content to their audiences.  Common themes across the presentations focused on how best to brand yourself and your company, and how to reach the greatest number of users, how to select content that is usable to your audience, and how to determine which platforms are the best to use to accomplish your specific goals.


RTP180 attendees network and mingle in advance of the event.

Matthew Royse of Forsyth Tech gave some very specific pointers, gleaned from social media research, indicating the ideal character length for tweets (100), Facebook posts (40), and domain names (8), and suggested that a balance of 60% curated content and 40% unique user content would work best to attract and maintain the attention of an audience. He also noted that timing of posts is critical in reaching your targets, given that 80% of the country is in the Central and Eastern time zones, and many users are online during weekends, a time that businesses often don’t consider to be prime for posting new content.

Amanda Peralta, a Social Media fellow at Duke University, also gave an informative presentation on determining the most suitable social media platforms for a particular organization’s goals. For instance, Facebook is the best match for those who host frequent events or need to provide customer service information, while Twitter is better for conveying current information on events or for those who have niche areas of expertise. Instagram is the best choice for reaching younger demographics when you have a lot of visual content, and especially in cases where users are already Instagramming photos relevant to your location or events.

Several other speakers reiterated some of these themes and discussed how social media has impacted their own business growth. It was clear that social media engagement can play a critical factor in the dissemination of information, in branding and marketing, and in generating interest for organizations. Because it is still a new area that most are just becoming acclimated to, one speaker indicated that social media is much like the “Wild West,” meaning there are few solid rules, and still plenty of time and room for individuals to pave their own way.


Justin Miller from WedPics discusses how his photo social sharing startup has achieved such success.

However, Chris Cohen from Bands to Fans noted that one must be careful of spreading their self too thin across the many available social media platforms. While it may seem beneficial to be present on all forms of social media, he suggest that if you find one or two platforms that are suitable for you, and you consistently and frequently post relevant content, that many times your audience will actually spread that content to other platforms on your behalf, increasing your visibility while still limiting your time and effort. This sentiment was echoed by Peralta, who also suggested to “limit yourself to what you can do well.”

There was such a positive response to this particular session, that some suggested that RTP180 host a social media “bootcamp” so this conversation can continue and grow in depth. The RTP180 events continue on a monthly basis throughout the year and cover various topics that could be relevant to survey researchers and those interested in new technologies. Upcoming events include topics on Big Data and Health; all are free and open to the public, though an RSVP is required and most events fill to capacity quickly. Free refreshments and a post-session meal are offered to all attendees. To find out more about RTP180 and their schedule of events, visit

NatCen Social Media Research: What Users Want

fryAt the beginning of October 2013, there were reportedly 1.26 billion Facebook users worldwide. The number of Tweets sent per day is over 500 million. That’s a lot of communication happening every day! Importantly for researchers, it’s also being recorded, and because social media websites offer rich, naturally-occurring data, it’s no wonder researchers are increasingly turning to such websites to observe human behaviour, recruit participants, and interview online.

As technology constantly evolves, researchers must re-think their ethical practices. Existing guidelines could be adapted ad-hoc, but wouldn’t it be better to rethink the guidelines for this new paradigm? And what do social media users think about research that utilises social media? The work of the “New Social Media, New Social Science?” network in reviewing existing studies suggests that these questions have not yet been adequately answered.

In response, a group of NatCen researchers are soon to report data from a recent study on how social media users feel about their posts being used in research, and offer recommendations about how to approach ethical issues.

What do participants want?

A key ethical issue participants talked about was consent: participants wanted researchers to ask them before using their posts and information. Although it was acknowledged that “scraping” a large number of Tweets would pose practical problems for the researcher trying to gain consent, users would still like to be asked. Consent was seen as particularly important when the post contained sensitive or personal information (including photographs that pictured the user). An alternative view was that social media users shouldn’t expect researchers to gain consent because, by posting online, you automatically waive your right to ownership.

Participants’ views about consent were affected by other factors, including the platform being used. Twitter, for example, was seen as more public than Facebook so researchers wouldn’t necessarily need to ask for the user’s permission to incorporate a Tweet in a report.

Views about anonymity were less varied. Users felt anonymity should be afforded to all, especially if posts had been taken without consent. Users wanted to remain anonymous so that their posts wouldn’t be negatively judged, or because they were protecting identities they had developed in other contexts, such as at work.

Our participants were also concerned about the quality of information posted on social media. There was confusion about why researchers would want to use social media posts because participants felt that people didn’t always present a true reflection of themselves or their views. Participants noted, for example, how users post pictures of themselves drinking alcohol (which omits any mention of them having jobs or other, more serious things!), and that ”people either have more bravado, and ‘acting up’ which doesn’t reflect their real world self”. They expressed concern over this partial ‘self’ that can be presented on social media.

What does it mean?

Later this year, NatCen will publish a full report of our findings, so stay tuned! If you can’t wait, here’s a preview:

  • Consider that users’ posts and profiles may not be a reflection of their offline personality but an online creation or redefinition;
  •  Even if users are not utilizing privacy settings they still might expect you to ask permission to use their post(s);
  • Afford anonymity. Even if someone has let you know you can quote their username, you should learn how ‘traceable’ this is and let the user know (i.e. can you type their username into Google and be presented with a number of their social media profiles?). It’s our responsibility as researchers that the consent we get is informed consent.

Let us know at NatCen if you would like to receive an electronic copy of the report, or if you have any questions about the study.

Survey: What’s in a Word?

As those of us in the survey research field are aware, survey response rates in the United States and other countries have been in decline over the last couple decades.  The Pew Research Center sums up the concerning* state of affairs with a pretty eye-popping table showing response rates to their telephone surveys from 1997 (around 36%) to 2012 (around 9%).  Others have noted, and studied the same phenomenon.

So what’s really going on here?  There are plenty of explanations, including over-surveying**, controlled access, and a disinterested public.  But what else has changed about sampled survey respondents or their views towards surveys in recent years that might contribute to such a drop?  As a survey methodologist, my first instinct is to carry out a survey to find the answer.  But conducting a survey to ask people why they won’t do a survey can be like going fishing in a swimming pool.

One place many people*** are talking these days is on social media.  In the past decade, the percentage of Americans using social media has increased from 0 to “most.”  I was curious to see how the terms survey and surveys were being portrayed on online and social media.  Do those who use (or are exposed) these terms have the same things in mind as we “noble” researchers?  When we ask someone to take a survey, what thoughts might pop into his or her mind?  Social media is by no means the only place to look****, but there is a wealth of data out there and you can learn some interesting things pretty quickly.

Using Crimson Hexagon’s ForSight platform, I pulled social media posts that included the word survey or surveys from 2008 (the earliest data available) to today (January 8, 2014).  First I looked to see just how often the terms showed up by source.  Here’s what I found:


In sheer volume, Twitter seems to dominate the social media conversation about surveys, which is surprising given that only about 1 in 6 U.S. adults use it. Of course, just because the volume is so high doesn’t mean everyone is seeing these posts.  The surge in volume is quite dramatic late in 2012!  Maybe this had to do with the presidential election?  We’ll see… keep reading!  My next question was what are they saying when it comes to surveys?  I took a closer look at the period before that huge spike in 2012, focusing just on those co-occurring terms that pop up most frequently with survey(s).  I also split it out by Twitter and non-Twitter to see what comes up.


We see according, each, and online for Twitter posts and according, found, and new for all other social media.  Hmm, what could this mean?  Drilling down into each term, we can look at individual posts for each term.  I include just one example for each here just to give a flavor of what the data show:

Twitter 5/08-7/12

  • According to one national survey on drug use, each day…
  • D-I-Y Alert! Nor your usual survey job $3 each – Research: This job….
  • We want you 2 do online survey and research for us. Easy…

Other online media 5/08-7/12

  • Nonresidential construction expected to lag in 2010 according to survey…
  • Surprise! Survey found young users protect their privacy online
  • New survey-Oil dips on demand worry, consumer view supports

Among these sample posts, we survey results being disseminated from several kinds of surveys on both Twitter and other online media.  The Twitter posts, though, seem to have more to do with earning money online than other social media.  Next, I looked at August 2012 to today (January 8, 2014):


Among the other online media, there’s not much change here from the previous period.  People replaces found among top co-occurring terms, but the focus is still on survey results.  For Twitter, we see a new top 3 terms co-occurring with survey(s): earned, far, and taking.  Here’s what some of the Tweets from the more recent period look like:

Twitter 8/12-1/14

  • Awesomest week ever! I earned $281.24 just doing surveys this week :)
  • Cool! I got paid $112.29 so far from like surveys! Can’t wait for more!
  • What the heck – I got a free pizza for taking a survey!

Now, I know that most of this is pure Twitter spam***** and not every Tweet is read or even seen by the same number of people, but I do think the increasing predominance of ploys to sign up people for paid surveys on networks like Twitter is a sign that term survey is being corrupted in a way that, if it does not contribute to declining response rates, surely doesn’t help matters.  They leave an impression and if these are the messages some of our prospective respondents have in mind when we contact them with a survey request, we are facing an even steeper uphill battle that we might have thought.

So, this leads us back to the classic survey methods question: what should we do?  How do we differentiate the “good” surveys from the “bad” surveys among a public who likely finds the distinction less than salient and won’t bother to read a lead letter, let alone open a message that mentions the word survey? Should we come up with a new term?  Does study get across the task at hand for the respondent?  Would adding scientific before survey help keep our invitations out of trash cans, both physical and virtual?

What are your thoughts on the term survey? Leave a comment here, or discuss on your favorite listserv or social media platform.  If you do, I promise not to send you a survey about it!

*scary=the degree to which lower response rates equate to lower accuracy, which isn’t always the case

**Personally, I sympathize with respondents when I get a survey request on my receipt every time I buy a coffee or sign up for a webinar.  “Enough already with the surveys!  I’ve got surveys to write!”

***not all people, and not all kinds of people, but still many!

****A few years ago, Sara Zuckerbraun and I looked at the portrayal of surveys in a few select print news media.

*****Late 2012 appears to have been a golden age for Twitter spam about paid surveys.

Social Media, Sociality, and Survey Research: Broadcast Communication

In our new book, Social Media, Sociality, and Survey Research, Craig Hill, Joe Murphy, and I define the sociality hierarchy, a framework we’ve used to conceptualize how individuals use digital media to communicate with each other. The sociality hierarchy presents three levels of communication: broadcast (one-way), conversation (two-way), and community (within groups). This post is about broadcast communication. Broadcast communication describes the behaviors of expressing thoughts, opinions, and status statements in social media and directing them at an audience. In the broadcast level, individuals send a one-way message to any friends or followers who happen to be tuning in. Broadcast social media communication includes things such as Tweets, status updates, check-ins and blogs, as well as YouTube videos, book reviews on Goodreads and restaurant reviews on Yelp.

Our book features two chapters presenting analytic techniques for broadcast data. In “Sentiment Analysis: Providing Categorical Insight into Unstructured Data,” Carol Haney describes a sentiment analysis methodology applied to data scraped from a vast range of publicly available websites, including Twitter, Facebook, blogs, message boards and wikis. She describes the steps of preparing a framework for data capture (what data to include and what not to include), harvesting the data, cleaning and organizing the data, and analyzing the data, with both machine coding and human coding. In her research, Haney finds sentiment analysis to be a valuable tool in supplementing surveys, particularly in market research.

Sentiment analysis has challenges too. The complexities of analyzing statements expressing sarcasm and irony and decoding words with multiple meanings can make the process more difficult or the results more open to interpretation. Despite the challenges, sentiment analysis of unstructured text yields insights into organic expressions of opinion by consumers that may never be captured through surveys. Two examples Haney provides:

  • A swimsuit company learned from blog content that they were not providing enough bikinis in large sizes. A subsequent market research survey confirmed that more than 25% of 18-34 year-olds could not find their size in the brand’s swimsuit line.
  • A public messaging campaign sought out factors that motivate young people to avoid smoking by scraping publicly available Twitter and Facebook data. The campaign determined that watching loved ones live with chronic illness or die because of the side effects were painful events. The campaign referenced this in the messaging and validated the effectiveness of the messages with subsequent testing with research subjects.

In “Can Tweets Replace Polls? A U.S. Health-Care Reform Case Study,” Annice Kim and coauthors analyze Tweets on the topic of health care reform as a case study of whether Twitter analyses could be a viable replacement for opinion polls.  Kim’s chapter describes the process of searching for and capturing Tweets with health-care reform content, coding Tweets using a provider of crowdsourced labor, and the comparison of analyses of Twitter sentiment to the results of a probability-based opinion poll on health-care reform. Ultimately, Kim and her coauthors found that sentiment expressed on Twitter was more negative than sentiment expressed in the opinion poll. This does not correspond with earlier studies on other topics, which demonstrated correlation between Twitter sentiment and opinion poll data on consumer confidence and presidential job approval. For certain types of sentiment, Twitter results correlate with public opinion polls, but for other types, Twitter is not currently a viable replacement for polling research.

These two chapters in our book represent a subset of the types of analyses that can be done with broadcast social media data. Are your findings similar? Have you worked with other types of online content? Or do you think of one-way communication differently than the broadcast model?

A Birds-eye View of the 2013 AAPOR Twittersphere

As Joe Murphy mentioned in the preceding post, Twitter can provide a unique and efficient glimpse into conference dialogue that may be difficult to capture in other ways. We can measure when conversations occur between tweeters, the subject of those conversations, and other interesting patterns such as the most frequent words appearing in Tweets or the most popular Tweet (as indicated via retweets). There’s one analytic possibility, though, that’s frequently overlooked: social network analysis (SNA).

As I discussed in my poster presented at this year’s AAPOR conference, something inherent and potentially very useful to researchers is the network aspect of social networking sites. Inspired by a growing Twitter presence among the AAPOR community, and Joe’s analysis of the AAPOR Twittersphere, I decided to put some of the SNA concepts I discussed in my poster to practice. Specifically, in my attempt to examine public opinion in a different light, I asked the question “who is talking to whom?”

For those of you familiar with SNA, or are just curious as to how the measures I discuss are actually quantified, I’ve pulled the equations and explanations from my poster and inserted them for reference. In the first image below, a birds-eye view captures what the AAPOR Twitter conversation looked like. By conversation, I’m specifically referring only to Tweets where one user Tweeted to another using the hashtag #AAPOR. For instance, if (hypothetically speaking) Joe (@joejohnmurphy) Tweets to me “hey @AdamSage, your poster was A-M-A-Z-I-N-G #AAPOR,” the link connecting Joe to me would be directed from Joe to me.

At first glance, this birds-eye view looks kind of cool (or like a big mess depending on your perspective) – but what does it mean? Each dot, or node, represents a Twitter user that at one point used the hashtag #AAPOR and mentioned another Twitter user in a Tweet. Each line represents the association that’s created when such a Tweet occurred. The degree of each node is number of different connections made as a result of these Tweets. In the AAPOR Twittersphere, the average degree, or the average number of people to whom a given Twitter user is linked via these dialogues, is slightly over 2 (2.017). This is important because an average degree of 1 would mean we’re not a community, but rather a group of 1 on 1 dyadic cliques, which makes ideas difficult to spread. While these did occur, I excluded them in these graphs and focused what is called a giant component in SNA, or the largest group of connected individuals in a network.

Excluding those conversations occurring outside of the giant component, the diameter of the entire conference Twitter conversation was 8. In other words, the furthest any two people were away from each other during this Twitter conversation was 8 steps. Imagine yourself at the largest Tweet-up in AAPOR history consisting of those AAPOR attendees participating in a dialogue on Twitter (using #AAPOR of course). As people traverse from conversation to conversation, everything you say would be no more than 8 conversations away from anyone in the room. So that rumor about my awesome poster could theoretically spread quite quickly. Hopefully it’s starting to become clear how great papers and keynote addresses can become popular Twitter topics in a given year. Great papers resonate and travel quickly; great keynotes are viewed by a lot of people and travel even faster.

But wait, it gets better. Ideas spreading among the AAPOR community wouldn’t traverse the network as you might think. Looking at the overall graph below you will notice several color-coded groups of tweeters. Each color represents a community, which is essentially defined as those tweeters who are mentioning each other in their Tweets more than others. For instance, community around @AAPOR (the green colored node with several links in the top middle) does not include many of the users with high betweenness and closeness. Betweenness measures the strength of one’s connection to the entire network (i.e., how interconnected or embedded one is to the entire network), and closeness measures one’s distance to the overall network (an average distance). They are measured as:



As you might expect, these people tend to drive conversations because they have many connections, which when analyzed using certain measures of network centrality, can reveal likely members of such conversations. For instance, just by measuring mere mentions within Tweets, clustering coefficients (defined as a given groups number of connections divided by the number of possible connections) allow us to distinguish “communities” from the rest of the network. Because this graph is directed, meaning I can tell if someone is doing the Tweeting or being Tweeted at, I know a majority of @AAPOR links were created by people mentioning @AAPOR in a Tweet, rather than @AAPOR mentioning others. The community around @AAPOR (the green colored node with several links in the top middle circled in red) does not include many of the users with high betweenness and closeness because @AAPOR’s propensity to draw Tweets to it naturally steals the thunder of everyone in close proximity.

While I haven’t analyzed the content of the conversations occurring among these communities, my guess is there are general topics (e.g., polling, elections, survey methods, statistics etc). After all, we tend to Tweet about what we’re interested in. In fact, just looking at the categories identified in Joe Murphy’s post, Tweets coming from @AAPOR and the highest frequency Tweeter (who I’ll leave unnamed, but I have circled in yellow) were generally categorized differently with the former much more often covering general conference information and the latter results from paper sessions.

While this is a quick analysis of the Twittersphere, these analyses can reveal characteristics of conversations and conversation participants that can potentially lead to other discoveries, such as the propensity for ideas to spread (or fade) among certain groups, identifying which individuals (and their behaviors) are key to group cohesiveness, and how we can eventually focus efforts to create a more robust environment for innovation, e.g., creating a hashtag to link people and ideas in the survey and public opinion research Twittersphere, much like market researchers use #MRX.

AAPOR 2013 – The view from the Twittersphere

Last week, more than 1,000 public opinion researchers convened in Boston for the annual meeting of the American Association for Public Opinion Research (AAPOR). The four day conference was centered on the theme “Asking Critical Questions” and included papers, posters, courses, and addresses from top researchers in the field.

As with each of the last several conferences, some attendees (and non-attendees) took to Twitter to discuss the results being shared and connect with colleagues old and new.  Our analysis of the #AAPOR hashtag (the “official” hashtag for the conference) shows more than 1,500 Tweets during the conference by more than 250 Tweeters.  About three quarters of these were original Tweets and the remaining quarter reposts (retweets) of #AAPOR posts made by others.  This is up about 500% from Tweeting at the 2010 conference. Five Tweeters posted more than 50 times during the 2013 conference, but more than half only posted once.

The conference activities kicked off on Wednesday 5/15 with afternoon short courses and wrapped up on Sunday 5/19.  Overall Tweet volume peaked on Saturday, though the most original Tweets (total minus retweets) were posted on Friday.

By hour, Tweet volume peaked during AAPOR events such as the plenary, President’s address, and award ceremony.  There was also a peak during the Saturday morning paper sessions with many Tweeters relaying thoughts shared by high-profile researchers like Jon Krosnik and Tom Smith.  In the chart below, the solid lines show volume excluding and the dotted lines including retweets.  Noon is indicated by the vertical line above each day.

We sorted original Tweets by time and looked at every fourth one to get a sense of the popular topics of discussion.  Content and research findings from presentations topped the list, making up about a third of all original tweets.

The most retweeted post was from @AAPOR announcing the release of the Non-Probability Task Force Report.  In the spirit of transparency though, it should be said that some (including me) were asked to retweet this announcement.  The post was retweeted 26 times.

Here is a quick word cloud of #AAPOR tweet content during the conference (the phrase “AAPOR” removed, for obvious reasons).

Interestingly (to us, anyway), the #AAPORbuzz experiment was more of an #AAPORbust… Few attendees were interested in replying to our Tweeted survey items, despite endorsement from @AAPOR itself.  It may be that attendees were more interested in sharing just what they wanted when they wanted and weren’t looking to respond to a survey/poll/vote simultaneous with discussing the topic of surveys itself at the conference.  It would be interesting to find out more about why people did not respond to these items.  Were they lost in the sea of Tweets?  Would another outreach approach be more effective? Would this work with a different population?  Further experimentation should help answer these questions.

Also interesting was the increased focus on Twitter itself at the conference.  In addition to the short course I gave with @carolsuehaney (Carol Haney), there were papers focused on Twitter analysis and at least one company (Evaluating Effectiveness), downloading #AAPOR Tweets and posting a dataset on their website. Popular bloggers like @mysterypollster (Mark Blumenthal) used Twitter to announce and link to their blog posts covering the conference.

Is Twitter here to stay as a mode of communication and topic of research for public opinion research?  Time will tell, but adoption and utility for those in the field seems to be on the rise.


Tracking 2013 AAPOR Buzz on Twitter – Just for Fun!

For those of you who have attended an AAPOR conference since 2010, you may have seen the Twitter Board that RTI sponsors annually (see below). Over the past few years, many of us at SurveyPost have focused on discovering and exploring the potential of social networking sites (e.g., Facebook and Twitter) in research. Although we will be presenting on these topics once again this year, we will also be putting some of what we’ve learned about analyzing Twitter data to good use.

In cooperation with (and great thanks to!) AAPOR planning and executive committee members, we have developed a series of fun questions that we will field via Twitter. Aware of the limitations of using Twitter data, and knowing this is the first effort of its kind at an AAPOR conference, we hope many of you will find the results and analyses we do interesting and, well, fun!

If you’re excited to participate (which we know you all are), simply follow the #AAPOR hashtag. For these particular questions, we are also using the hashtag #AAPORbuzz. You’ll see questions coming from our Twitter handle @SurveyPost, so be sure to follow us. We look forward to your Tweets!


10 Things Survey Researchers Should Know About Twitter (Part 2)

Yesterday I shared the first half of the ten things every survey researcher should know about Twitter.  Today I give you things 6-10:

6.      There are research applications beyond trying to supplant survey estimates.

Think about the survey lifecycle and where there may be needs for a large, cheap, timely source of data on behaviors and opinions or a standing network of users to provide information.  In the design phase of a survey, can we use Twitter to help identify items to include?  Can we identify and recruit subjects for a study using Twitter?  How about a diary study when we need a more continuous data collection and want to let people work with a system they know instead of trying to train them to do something unfamiliar?  Can Twitter be used to disseminate study results? What about network analysis?  Is there information that can be gleaned from someone’s network of friends and followers, or the spread of Tweets from one (or few) users to many? We often think of public opinion as characterizing sentiment at a specific place and time, but are there insights to be had from Twitter on opinion formation and influence?

7.      Twitter is cheap and fast, but making sense of it may not be. 

What’s the unit of analysis? Can we apply or adapt the total survey error framework when looking at Twitter?  What does it mean when someone Tweets as opposed to gives a response in a survey? Beyond demographics, how do Twitter users differ from other populations?  How can we account for Twitter’s exponential growth when analyzing the data?  The best answer to each right now is “it depends” or “more research is needed.”  We need a more solid understanding and some common metrics as we look to use Twitter for research.  Work on this front is beginning but has a long way to go.

8.      Naïve and general text mining methods for tweets can be severely lacking in quality.

The brevity of Tweets, inclusion of misnomers, misspellings, slang, and sarcasm make sentiment analysis a real challenge.  We’ve found the off-the-shelf systems pretty bad and inconsistent when coding sentiment on Tweets.  If you’re going to do automated sentiment analysis, be sure to account for nuances of your topic or population as much as possible and have a human coding component for validation. One approach we’ve found to be promising is to use crowdsourcing for human coding of Tweet content.

9.      Beware of the curse of big data and the file cabinet effect. 

Searching for patterns in trillions of data points, you’re bound to find coincidences with no predictive power or that can’t be replicated. The file cabinet effect is when researchers publish exciting results about Twitter but hide away their null or negative findings.

10.      Surveys aren’t perfect either.

Surveys are getting harder to complete with issues like declining response rates and reduced landline coverage.  Twitter isn’t a fix-all but it may be able to fill some gaps.  It’ll take some focused study and creative thinking to get there.