Critical.
Authoritative.
Strategic.
Subscribe to CIO Magazine »

Big data skills crisis needs vendor action: IDC

Enterprises are struggling with big data tech

Enterprise IT departments are struggling to take advantage of the benefits big data can bring due to a lack of skills in the marketplace, according to IDC research VP Susan Feldman.

At a big data roundtable hosted by Autonomy this week, Feldman explained that the complexity of big data technology requires an advanced skillset that is quite rare amongst IT professionals.

"There aren't a lot of people who are very skilled in these new technologies. How are enterprises supposed to hire people if they aren't there?" asked Feldman.

The most common technology used by companies to analyse hundreds of terabytes, or even petabytes, of unstructured data is an open-source tool called Hadoop.

Hadoop uses a process called parallel programming, which allows analytics to be run on hundreds of servers, with lots of disk drives, all at the same time. It stores this data in a file system called HDFS (Hadoop distributed file system), in effect a flat file system that can spread data across multiple disk drives and servers.

However, it is widely agreed in the industry that Hadoop is an extremely complex system to master and requires intensive developer skills.There is also a lack of an effective ecosystem and standards around the open-source offering.

"There are very few Hadoop experts around, and there are only very poor tools available for using it. You don't only need experts that know how to master an Hadoop file system, but experts that know how to master an Hadoop file system using bad tools," said Feldman.

Feldman urged the likes of Autonomy, EMC, Teradata and IBM to improve the tools on offer to reduce the impact of the skills crisis.

"If vendors could supply the expertise, if they could keep the software updated, then this would lessen the burden on IT departments in the enterprise," she said.

"Integrating all of the big data pieces into a well formed architecture, so that everything can interact with everything else, that's very difficult. Most people doing this could probably happily hire ten extra people, but they just aren't around."

IDC predicts a compound annual growth rate of 39.4 percent in big data hardware, software and service sales between 2010 and 2015. However, this doesn't take into account open-source offerings, which accounts for a significant amount of enterprise use, but is difficult to measure.

Comments are now closed.
Related Whitepapers
Latest Stories
Community Comments
Whitepapers
  • Delphix and Pure Storage Team to Super-Charge Database Deployments
    This webcast presentation, prepared by Delphix and Pure storage, explores super-charge database deployments and how they can aid business strategy. The presentation details the main features of a new flash solution – high performance, inline data reduction, resilience and scalability, and the value of simplicity. Viewers can learn how to put an end to inefficient or delayed QA, Sharing DB environments, using DB subsets and slow environment builds.
    Learn more »
  • Case Study: ETEL Limited
    Read how ETEL Limited, a pioneering design and manufacture business in New Zealand, managed to perfect their expansion into new markets by utilising an ERP system to support growth and provide “one source for truth” accessible to the entire organisation.
    Learn more »
  • Avoiding Common Pitfalls of Evaluating and Implementing DCIM Solutions
    While many who invest in Data Centre Infrastructure Management (DCIM) software benefit greatly, some do not. Research has revealed a number of pitfalls that end users should avoid when evaluating and implementing DCIM solutions. Choosing an inappropriate solution, relying on inadequate processes, and a lack of commitment / ownership / knowledge can each undermine a chosen toolset’s ability to deliver the value it was designed to provide. This paper describes these common pitfalls and provides practical guidance on how to avoid them.
    Learn more »
All whitepapers
rhs_login_lockGet exclusive access to Invitation only events CIO, reports & analysis.
Recent comments