Big Data, Business Intelligence and KPIs

What are you keeping all of that business data for?

Every year enterprises, mid-market and smaller growing businesses are generating vast amounts of data. The exponential increase has been likened to a flood. Besides the minimum retention of business data for financial compliance, which is typically from 5 to 7 years, there is an awful lot of other non-financial information being generated, collected and stored by the systems that power your business operations.

One area of tech where there have been significant developments over the last decade is that of Big Data. Although many people may have a top-level idea of what this means, there are many side-shoots that lead us into different areas.

In its purest sense, Big Data is about collecting and crunching massive sets of data to identify patterns. Some large enterprises have been accumulating data for at least two-and-a-half decades. It has also been growing at a rate that is reckoned to be exponential, because each year generates far more than the previous one. This has led to massive amounts of data being stockpiled.

Some Chief Financial Officers may have questioned the large amounts of money that they have been investing in enterprise data storage and archive systems. Quite naturally, when told that it is to maintain data that may be up to 25 years old, one response might be: “But what are we keeping it for?”

And that is a very good question!

In some ways, it’s a bit like cryonics, where people choose to have their bodies frozen after they die in the hope that medical advances will be able to cure them of the terminal illness that led to their death. Essentially, this is preserving the body while waiting for technology to catch up.

And that’s similar to what was happening with the accumulating mass of enterprise data. There was a suspicion that it would be of use one day: “We just have to wait for technology to catch up.” And that is just what has happened. Now, massive amounts of enterprise data is crunched with the ultimate aim of releasing value.

It’s worth noting that the science of cryonics has some catching up to do all of its own. Currently the cryopreservation process cannot be successfully reversed! If you cannot successfully thaw a body out without damaging it, then whether you can cure the original disease that killed the person is pretty much irrelevant! Luckily with Big Data there was no such problem…

The science of Big Data makes for big numbers!

One strand of the Big Data story is the science. There have been a number of breakthroughs that have led to the creation of Big Data solutions which deliver business value. One of the key innovations by data scientists was in developing adaptive pattern recognition techniques. This led to the rise of companies specializing in a variety of enterprise search and knowledge management solutions.

The most notable was Autonomy, sold to HP in October 2011. In a deal that has since become the focus of attention for all the wrong reasons, Autonomy capitalized on the growing trend of releasing value from data, in a deal worth $11.7 billion.

High-level Big Data solutions are sophisticated in that they can handle complexity. Such tools are able to crunch multiple data sets of historical and newly gathered, real time information. It is also able to analyze all the different types of information files to extract intelligence.

To better understand what this means, here is an example. Consider a lap top seized from a suspected criminal. Big Data analysis with an appropriate solution would be able to look at every item of data to see which files are related.

Images, video, and audio. Word-processing, spreadsheets and email. Website cache, browser history and meta. All this and more could be analyzed to look for patterns, and uncover behavior, activity or intent, supporting innocence or guilt.

When it comes to enterprise data sets that have been piling up for a quarter century, Big Data analysis enables deep insight. In some cases, it may reveal that some data is of no value, suggesting the business might be better off without it.

Business Intelligence

As a concept that was applied for commercial advantage, business intelligence was first documented in Richard Millar Devens' Cyclopædia of Commercial and Business Anecdotes published in 1865. Devens pinpointed that business intelligence was the ability to collect and react accordingly to the information retrieved.

In the late 1950s and 60s Business Intelligence (BI) as an area of IT practice evolved from the decision support systems (DSS) which assisted with decision making and planning. Today, Leading IT analyst organization Gartner defines:

Despite being in the Gartner glossary, some think BI is a bit of cheat. They believe it is a marketing buzzword term that piggy backs on the Big Data trend. This view holds that it is just an evolution of a more refined form of business reporting enabled by powerful and easy-to-use data analysis tools.

However, to others, the case for BI is clearly differentiated from the limited scope of standard reporting. This view holds that BI is more highly developed and that it is an enterprise-wide project, that unifies information gathering and analysis right across the business, end-to-end.

Key Performance Indicators

So far, this discussion has focused on the enterprise level and how big businesses realize value from data. But what of mid-market and smaller growing businesses? How can they benefit from the trend for Big Data?

Well in many ways this is the same as it is for many technologies. Typically, the first wave of solutions are out of reach of smaller businesses because of the big ticket price tag. Initially, the solution might be consultant-led, and it is also likely to be bespoke and need significant capital investment.

Following waves see solutions become less consultative, more productized and refined, with competitive market forces taking hold. Prices become realistic and affordable to companies working with smaller budgets.

Today, service providers specializing in KPIs are able to deliver affordable solutions. Their products are relatively simple. They are dealing with known data types, and generally, it is current data to see what’s happening now rather than masses of historical information to uncover what happened before.

It’s possible to view KPIs as the practical side-shoot of Big Data. A KPI report is defined as the final output format from a process of gathering information for sharing with audiences inside or outside of a business.

With systems for a variety of operational needs now deeply embedded throughout your business, it is within reach of every company to obtain data points that can be combined to produce management-level KPI reports as the final output.

This unlocks intelligence about how well your business is performing. You can see what’s not right and understand what’s good and how to make the performance of the company better.

What can KPI dashboards tell you?

There are a lot of metrics which feed into the overall performance of your business. Choosing the right KPIs to report is an important decision. When you get it right, you’ll find it focuses your team to bring improvements to any areas where the business is underperforming.

Here are some easy examples from the area of sales and marketing. We ask, “What are the most meaningful metrics?” followed by questions that suggest how things can be improved.

• Is it the cost of acquiring customers? Or is it the cost of retaining them? - Can we reduce costs for either, or both?
• Is it how many shares your marketing content gets on social media? - Are we wasting money on content marketing?
• Is it the Annual Recurring Revenue (ARR)? - Should we focus more on cross and upselling?
• Is it the number of sales meetings we need to close deals? - Can we bring in deals with less meetings?

One of the really great things about KPIs is that over time a series of reports is generated. This lets you see the current performance of your business against its previous results. There are also many publicly available data sets. If suitable data is available for your industry, you can benchmark the performance of your company against the market.

Where else is Big Data and data science driving tech innovation?

Big Data and the discipline of data science is also driving innovation in a number of other areas of technology. Some examples of how this is being applied include:

• Machine Learning - algorithms for speech recognition power digital assistants like Apple Siri, Microsoft Cortana and Amazon Alexa
• Online behavior prediction - website tools that can predict when online shoppers are about to leave a webpage and pitch special offers
• Medical decision support - enables medical technicians without high level qualifications to diagnose patient symptoms or escalate by referring to qualified medics
• Financial fraud detection - predictive analytics identifies claims that need to be scrutinized by loss adjusters to identify fraudulent insurance claim