If we start with Wikipedia then they say it's "a collection of data sets so large and complex that it becomes awkward to work with using on-hand database management tools". As its early days, Big Data sizes are a constantly moving target, but as of 2012 it is thought that they range from a few dozen terabytes to a few petabytes or even an exabyte of data in a single data set. To explain this further, one terabyte is 1000 GB. A petabyte is 1000 times that, and an exabyte is 1000 petabytes. The reality is that most SME businesses in Australia appear to have databases that are not much larger than between 100-500GB. This on the grander scale is quite tiny as a 100GB dataset is just 0.00001% of a petabyte
Apparently some of the Big Data candidates are in social data collation via social networks, web logs, Internet search indexing, call detail records and large scale e-commerce. Quoting a recent Cisco forecast, they claim that by the end of 2015, global data centre IP traffic will reach 4.8 million zettabytes and traffic will reach 402 exabytes per month. If we refer to the above terminology then that relates to 4.8 million terabytes worth of potential untapped data out there. Naturally companies within these sectors stand to benefit from having efficient methods in place to collect and process large amounts of data.
From a retail perspective If we look at one of the largest retailers in the world Walmart, who handles more than 1 million transactions per hour they have databases which are estimated to be more than 2.5 petabytes in size. Obviously this is growing and much bigger than our 100-500GB SME reference above. It is at this size that managing the data effectively represents massive opportunities for the business - if they had the tools to process it! There are real benefits from analysing the buying habits of their customers across the wider enterprise which could be used for benchmarking, marketing activities or supplier purchasing negotiations.
In Australia it appears that there are probably only a handful of retailers collecting enough data in what would be considered big. Here we are referring to the data collected via Coles’ FlyBuys, Myer’s Myer One and Woolworths’ Everyday Rewards card systems, Ebay and the Telco’s. That said most retailers I speak to agree they want to understand more about their customers via analytic tools but are struggling to make sense of their most basic current data, including customer details and purchase history.
In October 2012 a Big Data conference is being held in Australia by CeBIT which indicates to me that there’s a need to analyse its impact and potential solutions from a technology perspective. I can only ascertain that the underlying message for businesses is to explore methods now on how to capture more data, store it and keep abreast of the developments around processing it, even if your data is’t considered ‘Big’