6 Big Data Challenges and How to Overcome Them

breaking barriers 3 300x242 6 Big Data Challenges and How to Overcome ThemBig data is here to stay.

And that’s a good thing for companies that understand how to capture and analyze the massive amounts of data generated every day to make the right business decisions, improve operations, become more competitive, and increase profits.

But putting big data to good use is about more than Hadoop and a “bunch of fancy technology,” according to IT consultant Graham Oakes. For big data to do its job, companies have to overcome the “very real” organizational silos that have grown around that data.

In a recent survey, 56% of respondents cite organizational silos as the biggest obstacle to effective decision making that’s driven by big data.

Oakes shares six barriers that will get in the way of big data in a typical organization:

  1. Infrastructure: Big data uses a lot of technical infrastructure, storage, bandwidth, CPU, etc., all of which generate highly variable workloads. That means the amount of infrastructure you need varies as well – sometimes you need a lot, sometimes you need a little. The answer to this challenge – the cloud. But not so much as it pertains to the technical side of things. Rather it’s about selecting the right cloud vendor for your company’s needs and ensuring you don’t break the bank doing it.
  2. Applications: Behind big data is a complex application stack, some of it not very mature. As an example, Oakes points to the Cloudera Hadoop distribution, which contains a dozen applications, some of which are pretty new. To overcome this barrier, you have to “get up several learning curves at once,” integrate a number of tools with your existing application stack, and build a stable operating environment out of these different pieces.
  3. Skills: Not only do you need business analysts to ask the right questions, and technologists to “tame the infrastructure and applications,” you also need data scientists. You know, the people who can do things like understand the statistical algorithms, and drive the visualization tools. They’re out there, but they’re hard to find. But when you do find them, be sure to make them part of your team, Oakes says.
  4. Attitude: When it comes to big data projects, you have to “experiment, learn and evolve,” rather than “plan then do,” according to Oakes. But it’s important to balance that research with the objectives of the business.
  5. Fragmentation: Most organizational data is highly fragmented. That’s because every business unit seems to own a different piece of the data, which creates data quality issues. No one department is responsible for all the data and no one ensures that it’s “correct, consistent and up to date,” Oakes says.
  6. Valuation: Not very many companies can accurately value their current data, “let alone on the fuzzy web that Big Data exposes,” Oakes says.

Oakes says the most important thing for organizations to do to over come these challenges is to build multi-skilled teams with the right attitudes.

“Right now, many big data projects are merely playing with the data, exploring the tools and shifting data around within its silos,” he says. “If we could build some stable, cross-functional teams and focus them on business-led experimentation, then we’d probably begin to find real value in the data we have stashed away.”

Next Steps:

  • Subscribe to our blog to stay up to date on the latest insights and trends in big data and data analytics.
  • Check out our complimentary “5-Minute Guide to Business Analytics” to find out how user-driven “analytic” or “data discovery” technologies help business and technology users more quickly uncover insights and speed action.

Linda Rosencrance
Spotfire Blogging Team

Read this post in source website.

No comments yet.

Leave a Reply