Big Data Puzzle

The world of data is exploding in our “always on” connected culture. “Every day, we create 2.5 quintillion bytes of data — so much that 90% of the data in the world today has been created in the last two years alone” according to IBM.

The term Big Data is commonplace; many believe that it will transform business and government, but its real meaning remains vague. In a nutshell Big Data refers to both structured data (sensor data, sales data, etc.) and unstructured data (social media, text documents, video, etc.). Often it is real time data offering the Holy Grail for any company looking to predict future trends. Not all of it is new, but data is now available faster, its coverage and scope is wider, and includes new types of observations and measurements that were previously unavailable.

Companies and policymakers are realising the potential of their data; however, making sense of it and finding meaningful and real information that can be used to help us improve our lives and businesses is still a challenge. Imagine a box containing jigsaw puzzle pieces; it’s simply overwhelming: how do you know where to start? You empty the box and space every piece out. A complex algorithm in your brain searches and analyses pieces that fit together. However, finding these pieces and placing them together doesn’t give you the information you need until the picture is complete. With several puzzles mixed together, the information they give you does not become apparent until you finally look at all the images.

90% of the data in the world today has been created in the last two years alone


Of course, this is an oversimplification; analysing data can often highlight pieces of information that you would never have thought about, some insights into new trends and potential opportunities. Consider the Google Flu trend a few years back. Google tracked the outbreak by finding a correlation between what people searched for online; it saw the patterns and was able to watch the spread far quicker than medical professionals.

Moving beyond simply tracking the outbreak, imagine being able to predict where it’ll hit next or its potential economic impact. Predictive models created by the ACRC are utilised in critical transactional systems and support decisions and actions in near real time. Planners and managers need to understand how complex environments will work in practice and understand potential problems before they occur. Urban planning, healthcare, train stations and airports all need to understand factors such as pedestrian flow and the impact of unexpected events. By focusing on understanding business challenges and delivering action-orientated solutions, these models can analyse multiple aspects of individual behaviour in differing conditions and scenarios. Analysing multiple instances of a given decision to identify the most effective action to take can provide valuable insights that help reduce cost and risk.

The ACRC specialises in the latest research in data analytics and complex simulation to bring these puzzle pieces together. We combine these technologies to produce powerful predictive models often using supercomputers to provide advanced decision support and analysis. Our state of the art research provides real information and knowledge from data that can be acted upon to maximise opportunities.

Published in the September version of Modern Gov Magazine

Read More