Should we be concerned about Big Data?

Rate this item
(6 votes)

Big Data seems to be one of the latest buzzword to appear in the ITC dialog. I still wonder if most of us understand its meaning or impact.

It appears that various groups believe they have a handle on it the Big Data phenomenon. The interesting thing is that if you comb the internet you’ll see the term being sponsored by companies such as NetApp, EMC and IBM as vendors that manage big or large amounts of data. I wonder if this means that these companies are positioning themselves to capitalise on this slice of the pie or are they still trying to ascertain what it really means in a commercial sense before it’s more widely adopted by the market.

If we start with Wikipedia then they say it's "a collection of data sets so large and complex that it becomes awkward to work with using on-hand database management tools". As its early days, Big Data sizes are a constantly moving target, but as of 2012 it is thought that they range from a few dozen terabytes to a few petabytes or even an exabyte of data in a single data set. To explain this further, one terabyte is 1000 GB. A petabyte is 1000 times that, and an exabyte is 1000 petabytes. The reality is that most SME businesses in Australia appear to have databases that are not much larger than between 100-500GB. This on the grander scale is quite tiny as a 100GB dataset is just 0.00001% of a petabyte

Apparently some of the Big Data candidates are in social data collation via social networks, web logs, Internet search indexing, call detail records and large scale e-commerce. Quoting a recent Cisco forecast, they claim that by the end of 2015, global data centre IP traffic will reach 4.8 million zettabytes and traffic will reach 402 exabytes per month. If we refer to the above terminology then that relates to 4.8 million terabytes worth of potential untapped data out there. Naturally companies within these sectors stand to benefit from having efficient methods in place to collect and process large amounts of data.

From a retail perspective If we look at one of the largest retailers in the world Walmart, who handles more than 1 million transactions per hour they have databases which are estimated to be more than 2.5 petabytes in size. Obviously this is growing and much bigger than our 100-500GB SME reference above. It is at this size that managing the data effectively represents massive opportunities for the business - if they had the tools to process it! There are real benefits from analysing the buying habits of their customers across the wider enterprise which could be used for benchmarking, marketing activities or supplier purchasing negotiations.

In Australia it appears that there are probably only a handful of retailers collecting enough data in what would be considered big. Here we are referring to the data collected via Coles’ FlyBuys, Myer’s Myer One and Woolworths’ Everyday Rewards card systems, Ebay and the Telco’s. That said most retailers I speak to agree they want to understand more about their customers via analytic tools but are struggling to make sense of their most basic current data, including customer details and purchase history.

In October 2012 a Big Data conference is being held in Australia by CeBIT which indicates to me that there’s a need to analyse its impact and potential solutions from a technology perspective. I can only ascertain that the underlying message for businesses is to explore methods now on how to capture more data, store it and keep abreast of the developments around processing it, even if your data is’t considered ‘Big’

Stephen Duncan

Stephen Duncan

Stephen Duncan is a Technology Retail Specialist.

Post a Reply Comment

Make sure you enter the (*) required information where indicated.
Basic HTML code is allowed.

You are here Should we be concerned about Big Data?