THE CHALLENGES OF IMPLEMENTING AND USING BIG DATA

Introduction

There is certainly a large amount of noise at the moment regarding Big Data around what it can do, its challenges, and how it could change the world for a better place.

However, like most new concepts, one has to maintain a certain amount of suspicion around any new technology idea.   This is because (a) new ideas have a large amount of hype and therefore under-deliver (b) people cannot see anything wrong with new ideas and overlook its shortfalls and (c) people often jump on the bang wagon and ‘re-badge’ other ideas as the new idea.

What is Big Data?

There are hundreds of different definitions of Big Data but the following three points provide a good description

• There is a massive volume of data. While size and volume are often relative, we are talking in the range of millions of data items with hundreds of data variables within each data item.

• The data is constantly changing often at a rapid pace with new items are being added, updated, and removed constantly.

• The data is stored in a variety of different formats.  This will cover the more ‘traditional’ pre-defined structured database formats but also a wide range of unstructured formats such as videos, audio recordings, free format text, images, social media comments, etc.

Once this data is collected then it is possible to undertake various forms of analysis such as finding patterns, trends, themes, and correlations/   This analysis can then be used to explain historical behaviors as well as to predict and shape future behaviors.

However, what are the challenges and issues with using Big Data?

• Big Data is not a silver bullet and there are challenges with implementing it successfully.  

• Before an organization attempts to implement Big Data then (like any change) it needs to have a clear business reason linked to the organization’s strategy.  This will ensure senior management buy-in and a clear focus on what needs to be implemented.

• It would also be advisable to perform m some sort of cost/benefits analysis to understand whether the benefits outweigh the costs, stress, and challenges of implementation.  Six of the main implementation challenges are detailed below:

1. The first challenge of Big Data is that the term is often misunderstood and misused.  While it is often easy to be skeptical but firms selling business will often use Big Data to cover a wide range of data analysis techniques because they feel the ‘more trendy’ term will generate business for them.
2. The second item relates to the sheer challenge of processing a vast amount of constantly changing data across many differing and incompatible formats.    A complex (and expensive) technology stack will be required to retrieve the data, interpret it, store it, and then analyze it.
3. Thirdly data security is a massive challenge.  Data security and its set of legal rules is a complex issue at the best of times.  However, its complexity will increase dramatically when used with Big Data especially if data is gathered and processed across international boundaries.
4. Like any new discipline or specialty, there is a large shortage of genuinely skilled individuals in Big Data.  There are many people who will pass themselves off as data scientists, data miners, or Big Data specialists but care needs to be taken when employing people to ensure they have the skills and experiences required.
5. (Like all data analysis or research techniques) there is the risk of inaccurate data.  This could be due to (a) the data sources being separate and not linked together properly (such as purchasing habits not being linked to geographical locations) (b) the data being of poor quality (c) the data is gathered over a poor sample size which means the results could be bias and/or (d) the data being gathered is misunderstood by the data analysis team.    These problems are exaggerated by the size of the data, its constantly changing nature, and the differing formats. 
6. Finally, there could also be issues when processing or analyzing the data.  There could be errors in the algorithms employed, the wrong variables could be measured or people simply misinterpret the outcomes provided.  Again this will be exaggerated by the size of the data, its constantly changing nature, and the differing formats.

The Dark Side of Big Data

Finally, there is a dark side of big data.  As mentioned earlier, Big Data techniques allow one to predict and change people’s behaviors.   While this is not necessarily a bad thing (because it could help with disease prevention) but this technique could be used to change people’s behaviors for somebody else’s own personal needs.  For example, there have been various documented examples where Big Data techniques have been used to change people’s voting intentions.

What is the future of big data?

Big data definitely has a massive future going forward and will no doubt provide a great benefit to society.   However, it is important that one does not underestimate the implementation challenges posed, the regulatory risks as well as the dark side of Big Data.

Name: Paul Taylor Title: Management Consultant London, UK