My issues with Big Data: Distraction

While reading the economist on the plane from Switzerland every other ad was somehow Big Data related. After a while it would not have surprised me if Lexus advertised for a car powered by Big Data. This made me think about the enormous amounts of effort and resources that goes into this concept. With Cloud being a household name, Big Data is perceived as the next thing to drive growth. Venture capital is funding Big Data start-ups while existing companies are re-branding or extending their product lines into the Big Data space. With money and time being scarce resources they have to be reallocated from somewhere else.



Looking at this from a information management perspective there are still big unsolved challenges and untapped opportunities that deserve all the attention they can get. Here are some examples:

  • Data quality is an issue in almost every organization. Big Data will only make this worse when attention is shifted to integrating vast amounts of noisy data with data of already poor quality.
  • Organizations are still introvert in their reporting and analysis. External data sources to benchmark and enrich internal information are underutilized. Big Data may be one such source, but there are far more mature and less costly alternatives from market research providers, governmental agencies and so forth. Some are calling this Wide Data.
  • Companies are not effectively utilizing their existing data let alone Big Data. Every consulting company worth their salt has some kind of BI or Information Management strategy offering. The logical reason is that there must be a big market for helping companies become more mature in this space.
  • Big Data is a solution looking for a problem. A lot of effort is going into finding this problem both among providers and customers. Good to know there are other avenues to follow.

On the positive side, Big Data is associated with Business Intelligence and related fields. Some of the effort put into it will surely trickle down into better offerings for the good old “small data” solutions. I just hope we do not get too distracted from the main purpose of our field: Help customers make better decisions.

Disagree? Feel free to discuss!


One thought on “My issues with Big Data: Distraction

  1. Good points, it had been one of my fear that once the hype dies down business will walk away from not just Big Data, but the data analysis and data mining all together.

    Your bullet list basically sums it all. To me, the buzz of Big Data is almost like back in the days when human had just discovered fire. They feel the warmth from it, got burned when get too engaged into it, and have yet to find out what they can do with it (Imagine it took them more than 100000 years from the moment they controlled the fire till they discover combustion). Overall, fascinated about what it can do but yet skeptical about the actual value gain vs the investment which is mostly a lack of good and economical solutions going around.

    In front of the project board or customer, it is much easier to justify the investment in quantifiable
    results (How much, how fast, what to deliver). Even if someone is open mind enough to trust the reports to deliver key information for decision making it is still limited to displaying trivial information, I still have yet to meet someone who trusts a system to outright suggests the decision from data mining.

    There are many consulting company with a BI department, but they seems to be limited to setting up data storage, data transfer and basic reporting. When not enough supplier and even less consumer sees the true value in Big Data, it slows down the development of the field.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s