Growing research with data science

I’m sure you’ve spotted this already, but ‘data science’ is disrupting the Market Research business. It has triggered growth in new solutions and methodologies, but also creativity as the discipline rises to the challenge of standing out against data overload.

The start of this disruption is most evident in the new research opportunities, such as machine learning or the ‘mainstreaming’ of methods like eye tracking. But so far we’ve seen little of how data science will change the way we work day to day. This is about to change.

I have three predictions for the next 10 years:

1. The majority of market research jobs will require a working knowledge of at least one programming language.

Rare is a research position that does not require competent use of Excel, or solid maths skills. They are currencies of the job. Computers are enabling us to work more quickly, automate manual tasks, reduce potential human error, spot ‘bad data’, and distribute insights more impactfully. We need code to tell them how to do so.

At the moment, the languages of greatest use to researchers are R, Python, SQL or VBA, all of which are free-to-use, and allow us to write and automate procedures for opening, transforming, extracting meaning from, and reporting data. R and Python is where things are heading in the world of data: ‘universal’ open-source data-specific languages which allow us to write one language, but get the benefits of all of them.

2. We will halve our spend on specialist desktop software tools in order to analyse data.

In order to be able to analyse some types of data, we have to buy specialist tools. SPSS is an example, there are many others. They allow access to bigger datasets than Excel can handle, and provide simple user interfaces to run complicated queries on the data.

R and Python, described above, do away with the need for specialist tools for big or multi-dimensional data. They access data from any source, can handle data many orders of magnitude bigger than traditional desktop software, and use a ‘plain english’ form of language that any researcher can master. They are also free, so allowing us to save money on data access.

3. The alignment and merger of data science/analytics teams and research teams will be complete.

Big Data is not about to kill market research. According to a study run by Dataiku and Carruthers & Jackson (see last month’s MRS Impact Magazine), only 8% of Chief Data Officers are happy with the data they have access to. One reason for this is that they don’t put market research in their data warehouses!

The practical barriers are that ‘big data’ can just be plugged in for quick access, and has a predictable shape. Market Research on the other hand, has all sorts of dimensions, and is usually stuck in the proprietary tools described above.

With code this is no longer a problem. You can turn a brand tracker into an API and merge it with your web transaction data, to put more ‘why?’ behind the ‘what?’ of big data.


Holborn Studios
49-50 Eagle Wharf Rd

London N1 7ED​


T: +44 (0)20 7502 5921
M: +44 (0)7717 605116


© 2019 by Culture of Insight. 

  • w-tbird
  • LinkedIn - White Circle