Big Data – The Reality beyond the Hype

I was having a coffee with a friend last week and the conversation turned to the latest trends in technology – as it often does.  His view was that ‘Big Data’ was just another in a long line of over-hyped technologies, aimed more at selling the shiniest new product than solving some real-world problem.

I think that the Big Data term is really a shorthand way of describing the escalating amount of data being generated by the actions of people and their devices as they interact with each other and the world at large.  Every time we use a web-site, smartphone or other electronic service data is created and collected – to understand our behaviour, predict what we’d like to buy or where we’ll go, or perhaps show a relevant advertisement. 

An even larger amount of data is beginning to be created by the ‘internet of things’ – a term used to describe the invisible devices and sensors all around us in our vehicles and transport systems, communications and power grids which collect and report on the health of these environments.  For example, engines in the latest commercial aircraft capture a large volume of performance data and report any abnormal operation in real-time via satellite links.  Current car models can already report back if they are involved in a crash or require roadside assistance, collecting engine and performance data can’t be too far in the future.

As the cost of collecting and storing this data continues to drop, it doesn’t take too much imagination to see the value in being able to analyse more fine-grained data on power consumption, real-time traffic, when and where we buy products, or whatever we can imagine being sensed and measured.  Having this newly available data can lead to discovery of previously unknown patterns of behaviour or relationships – telling us about a new artist, restaurant or author, a nearby bargain or a group who share our passion.

So, even if you think Big Data is just an over-hyped buzzword, a tremendous and ever-growing variety and volume of data is being created by our use of web-sites devices and sensors. I don’t think this trend is likely to slow down in the foreseeable future, as ever more of our interactions move into the digital realm.

The era of Big Data is with us, no matter what we call it.