Companies are increasingly collecting and analyzing vast amounts of data to improve their services and decision making. However, the automation of data management inside organizations comes with multiple challenges and security is viewed as the top one, according to a new survey.
Data management automation, also known as DataOps, involves the use of multiple technologies and processes to simplify the collection, storage, access and analysis of data to enable agile data-driven business decisions and product development. The problem is that without a strategy in place that takes security into consideration, businesses are at risk of experiencing serious data breaches that can result in financial losses and negative impact to their reputations.
Over the past several years many large organizations have experienced data breaches not because of hacking, but because they failed to properly configure and secure various data-related technologies such as Amazon S3 buckets, Elasticsearch nodes, MongoDB databases, Hadoop deployments and others. In a survey conducted by 415 Research that was released today, two-thirds of respondents cited security and compliance as the top challenge associated with data management. This was followed by concerns related to data quality, the governance of distributed data, access to internal data and data pipeline reliability.
Understanding the compliance requirements related to data privacy was also viewed as a challenge by 40 percent of respondents.
The survey questioned 150 representatives from U.S.-based organizations with over 1,000 employees and a minimum of 2 petabytes of managed data, and who had a good understanding of their organization's data management strategy. The organizations were from various industry segments, including banking, healthcare, technology, manufacturing, retail, government, insurance and telecom. Around a fifth considered DataOps already ingrained in their culture, 37 percent were in the course of DataOps adoption across multiple departments and 34 percent were still defining a DataOps strategy.
Data distribution and volume growth complicate the security picture
"Data security is a perennial concern, of course, while addressing data quality is essential if organizations are to ensure they are making decision based on accurate data," 415 Research said in its report. "Additionally, the fact that data is increasingly distributed across multiple data platforms (relational, nonrelational and Apache Hadoop/Spark), and multiple locations (on-premises and multiple clouds) complicates the understanding of what data organizations truly have at their disposal."
The push of moving data into the cloud is only expected to accelerate. Nearly a third of respondents said that their organization's data grows at a rate of 100 gigabytes to 500 gigabytes per day. Nineteen percent reported growth rates between 500 gigabytes and 1 terabyte per day, 12 percent between 1 terabyte and 2 terabytes and 13 percent over two terabytes per day. This means that's it's becoming harder and more expensive for companies to store and analyze data on-premise.
According to 415 Research, 41 percent of surveyed organizations currently store more than half of their data in the cloud and over 70 percent of them are expected to reach that milestone over the next two years. Furthermore, 60 percent are already using NoSQL databases and almost half have distributed data clusters based on Apache Hadoop/Spark.
The increased popularity of these technologies and the fact that some of them were designed for internal use and lack strong access controls or secure default configurations could explain why security researchers are finding tens of thousands of deployments that are publicly accessible from the internet without authentication.
The good news is that many respondents whose organizations are well on their way into DataOps adoption are citing greater security and compliance as a benefit (66 percent). Other benefits according to the survey results include increased business agility and faster time to market (63 percent), improved decision making and business insight (55 percent), the introduction of application-driven business capabilities (54 percent) and cost reduction (49 percent).
"While the concept of DataOps is perhaps most associated with operational efficiencies, the survey results indicate that those efficiency improvements are not just related to agility, but also to security and transformational change," 415 Research said. "Companies that are already engaged with DataOps overwhelmingly agree that it is having a positive impact on their organization, and while improved agility and efficiency are closely associated with DataOps, the biggest driver, priority and benefit is actually related to security and compliance."
Join the CIO New Zealand group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.