The technology marketplace has recently seen an influx of cloud storage solutions. This is no surprise, considering the sheer volume of data that we generate and disseminate every day. The simplicity with which these solutions are branded, however, belies the complexity of big data management; it’s very tempting to believe that any level of granular data can be accessed instantly – almost as though by sheer magic. Is this really the case?

It’s important to manage expectations about the ‘how’ and the ‘when’ of big data management – and equally important (although perhaps less common) to ask ourselves what we want to store – and why. Do we really need to store everything for retrieval at a later date? Why? It could be that the fear of missing something drives us to keep every piece of information we deem to be significant, but at what point does data storage become data hoarding?

There’s no denying that big data presents us with big potential – knowledge is power after all. For example, the data captured by millions of ‘wearables’ in a relatively short time can yield valuable insight into things like obesity prevalence, activity and fitness levels, age-related heart disease etc. Some data is clearly worth storing. Other information, however, becomes meaningless quite swiftly. In forensic IT, new information stays pertinent for around 6 months, although legal storage requirements are closer to 3 years. 6 months may seem short in this context, but in the context of Networking Systems and network monitoring, could 6 months be too long? What do you think?

The question of why we store certain information is difficult to answer objectively. “One man’s trash is another man’s treasure”… That said, countless packets of data have been created and moved in the time it’s taken to read this. So while we ponder these questions, with the Big Data Buffet growing steadily in the background, let’s chew on this too: Will our ability to manage big data keep up with the amount of data we’re storing?

In 1965 co-founder of Intel Gordon Moore made an interesting observation that speaks to this question. His theory has been developed into what is now known as Moore’s Law. Click here to read about it.