There’s gold in your organisation: 7 steps to access your data gold
Originally published on LinkedIn
Your organisation is probably sitting on a big-data gold mine: but are you struggling to mine it?
Do you know the true value of the data sitting in your organisation? Or are you focusing solely on certain data types, leaving large portions of data you already own untapped?
How do you manage the inflow of data at rates that continue to grow?
How do you know where to direct your data analysis resources, and are those resources equipped to respond fast enough?
These are complex questions – and they are ones we asked of 639 companies, in partnership with SAP and with the expertise of Forrester Group, late in 2015. Our start point was a simple (and perhaps obvious) hypothesis: that businesses were either analysing and deriving value from only small amounts of the data they already held, or weren’t extracting value at all.
The survey talked to organisations using Hadoop (the open-source software framework for storing and processing large amounts of data on distributed computer systems) to ensure the organisations being surveyed really did have large amounts of data.
That original hypothesis was quickly confirmed: data are being stored in large amounts (up to 500 TB in some cases), but as much as 70% are not being used. The reasons for this are varied: complexity, mixed data structures, too few resources, unclear objectives, and issues managing the sheer volume of data.
Although perhaps not a surprise, this is a sobering result. What’s exciting is that it also highlights, for many companies, the opportunity to mine data gold very quickly with a high probability of finding valuable nuggets.
Understand, assess, implement
We can summarise the challenges faced by most companies seeking to mine value from big data in the following chart:
Accessing and integrating data
One challenge that organisations surveyed seek to overcome is the variety of systems and inputs that characterise and feed their big data systems, many of them transactional. Many mine data from discrete silos, missing the complete picture, or worse being misdirected by spurious conclusions. As with any picture, the higher the resolution, the more accurate it will be.
The objective therefore must be to manage and analyse the data as a unified set – and the first task is to remove, as much as possible, any silos around your data. With silos in place, it’s difficult even to see that data – and value – exist. Our survey with Forrester indicated that as many as 80% of those questioned have more than one data source, that only 21% are using their internal data to the full, and as few as 14% are using their external data in the same way. That leaves a large amount of data left unmined.
Our survey with Forrester indicated that as many as 80% of those questioned have more than one data source, that only 21% are using their internal data to the full, and as few as 14% are using their external data in the same way. That leaves a large amount of data left unmined.
The second task is making data access universal to everyone within the organisation. This need not necessarily be a problem of volume: it can be a problem of data structure, with no two sets of data seemingly sharing anything that makes merging them easy. Different data formats are the bane of many organisations. A top-down, strategic approach to data capture and data storage, with a focus on formats and structures, will start to fix this.
Data preparation and analysis
When half the organisations surveyed confirmed that data requests can take months to answer, it’s clear that new methods and systems for accessing data are needed. Data analysts are well-regarded experts who know how to “think data” in the same way that programmers and developers know how to “think code.” It’s unlikely that organisations would trust analysts to write applications, yet our survey showed that IT departments are often given the task of pulling and analysing data. The irony of receiving analyses of data that were once real-time but are now weeks out of date is compelling.
Budgets, technology and skills
Budget constraints were top of the list of challenges for 45% of the companies surveyed. When placed alongside the earlier result – that 70% of data are not being used, and that internal and external data are left unmined to varying degrees – the argument for investment seems clear.
Alongside budgets sit technology. The solution to any technology constraint is clever deployment of budget on the right technology. Aim for solutions that let users and business managers find, analyse and process their own data. By definition, your systems are in place to store data: your investment needs to be in the technology tools to extract value.
Finally, to skills. Our research indicates that familiar skills and tools support self-service data access fastest. Employees can be trained on new tools, but budgets of course are always finite. Our recommendation is to use familiar tools to access large amounts of data in new ways.
Time to get mining: a data mining action plan
Imagine your organisation in the near-future: you’ve removed the silos around your data, added more inputs, trained and equipped your workforce, and made data a cornerstone of your strategic planning – all for an investment that’s affordable and justified.
You know and understand more about your business, your customers, even your employees. Data are constantly refreshed, so your insights are up-to-date. Risk is reduced because the information is timely. Data are available to everyone who needs access. Your IT infrastructure supports these needs, and the cost is justified.
Now you can move into new territory: you can explore your own data to find new patterns, and respond more effectively to opportunities. The business owners within your organisation can access their own data, in a timely way. Decisions are better and faster.
7 steps to access your data gold:
- Consider how budget, integration, data preparation, data access, technology and skills currently constrain your business. Analyse the benefits of removing any that do apply, and model the case for adopting and using big data.
- Create a big data strategy that includes ultra-fast access across your organisation.
- If that’s too large a step to take in one go, run a pilot with consultants on projects to help you confirm that data and value are hidden within your organisation. Demonstrate the likely, even actual, increases in revenue or profit that better access to existing data can deliver.
- If your data access is still legacy in style, discard these processes. You’re arguably worse off because you’re using data that’s outdated.
- Gain leverage from open source data access technology (such as Hadoop or Spark).
- If budgets are an impediment to investment in skills, tools or technology, there’s little option but to create an argument – using data – to demonstrate the detailed ROI you believe remains unlocked in the data you cannot currently access.
- Train your employees on new tools that give them the skills they need to access data in new ways.The solution to having data in time to make good business use of the results is investment – in analysts, the data gold mining experts; and in tools, the data gold mining implements.
Unlike a real gold rush, there’s no uncertainty about the existence of data gold on your organisation: you simply need to mine it.