Big Data and Primary Market Research

One of the phrases we have started to hear a lot of lately is “Big Data”.  First, a definition for those who aren’t up on this trend.

From Wikipedia: Big data usually includes data sets with sizes beyond the ability of commonly-used software tools to capturecurate, manage, and process the data within a tolerable elapsed time.  Big data sizes are a constantly moving target.

Big data type analysis is used now in a lot of different venues.  One use of big data that has been getting a lot of attention is when people pull together data from disparate sources such as credit reports, on-line browsing behavior, car ownership records, and other sources to create individual profiles of people which can be used for targeting purposes.  Social media is generating a lot of data, which may be folded into this, as well as locatoin data from cell phones which now allows delivery of coupons based on where your phone is   Big data is finding uses in health care, with the advent of electronic health records.  It has been touted as bringing the ability to affect changes in health behavior such as reducing hospitalizations.  In essence, it is an extreme form of segmentation – we are now talking about segments of one.

Big data is giving rise to new privacy concerns.  I used to dismiss concerns about the safety of giving out information to Market Research both because I knew from experience (at that time) that the CASRO guidelines meant the data wouldn’t be used to sell plus that marketers weren’t interested in individual behavior and couldn’t deal with the volume of information on an individual basis. At that time in the past, detailed data like that had no value for them.  Well, the decrease in computing cost and the development of more sophisticated software has changed that.

Implication:  This may be another factor contributing to the decrease in cooperation rates for market research surveys. We may know that we are adhering to CASRO guidelines or HIPPA regulations, but our audience doesn’t know what those guidelines are, and can’t discern when something is true market research and when it is data collection to enable direct marketing.  Recognizing this factor could help us identify a potential solution to declining response rates. As I have noted before, if the decline doesn’t stop, we won’t be able to conduct quantitative market research for much longer See my post here for more details:

Beyond these, there are some other implications of Big Data for Primary Market Researchers.

  1. Unless your corporate market research department shows that it is capable of dealing with Big Data, another  department will.  Find out how to make use of it fast!  or else you will lose an opportunity.  (It may already be too late.)  
  2. Learn how to integrate Big Data with primary market research.  You should use it before doing any primary study to inform what you do, and you should interpret your results in light of Big Data findings.
  3. There may be some projects that you don’t need to do with primary research because you can do them instead with Big Data.  Do it!  Don’t waste respondents time and energy providing estimates when you can get actual data.  Every unnecessary use of a respondent may contribute to declining response rates.
  4.  You should know which of your vendors can deal with Big Data, so you are ready to have them help you integrate Big Data with Primary Market Research.

However, in my opinion, Big Data should never substitute for Primary Market Research or for the insights that people can bring.   This article from the NY Times makes a great case for how Big Data can’t substitute for human reasoning and intuition.

As always, I appreciate your comments and suggestions.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s