Data is everywhere – what you eat, where you work, what you buy. Paraphrasing Forrester Research, the aggregation of massive amounts of data such that its volume, velocity, and variety make it difficult to store, process, and access, is known as “big data”. And big data is being used in some pretty remarkable ways.
Traditionally, big data is used to find trends and relationships in datasets to help businesses operate more efficiently and effectively. Most notably, big data helped Nate Silver accurately predict all 50 state outcomes during the 2012 presidential election. Maybe he got lucky? Well considering he was 49 of 50 in the 2008 election, I’m siding with data and statistics.
So then, is big data the 21st Century’s Nostradamus? Not quite. Silver missed the notions of “heart” and “determination” (among other non-quantifiable human variables) when he based an uncharacteristically inaccurate prediction of this year’s Super Bowl winner on big data alone. But big data, according to David Selinger, will still be necessary for corporations to “stay relevant, and, frankly, to stay in business” moving forward.
So how do you get there? Succeeding in the big data era will require a friendly blend between human and machine. As Derrick Harris puts it, “What makes the big data era so significant isn’t that people are using data to inform their decisions, but that there’s just too much data of too many types… Like the purpose built robots that have revolutionized manufacturing, today’s methods for processing and analyzing data are fast, scalable and precise, but they don’t yet (in most cases) make our decisions.” Machines and algorithms are great for doing the heavy lifting, but making sound business decisions requires human intuition. Remember how IBM’s Watson goofed several times during its celebrated appearance on Jeopardy – when given the clue of “It was the anatomical oddity of US gymnast George Eyser,” Watson answered “leg” when the correct answer was “missing leg.” Watson’s engineering team can update its algorithm to more accurately understand and answer questions, but as Derrick Harris puts it, “once the heavy lifting is done, humans come in and use their subject-matter expertise and logic to prune off bad connections, add context and maybe even inject a little serendipity into the final algorithms.”
Important business decisions will always require human instincts and intuition, and the notion of data-driven decisions is by no means a new one, but with big data we can get to those decisions faster and have a lot more confidence in them. Can you rely on big data alone? No. But the important takeaways for corporations are to 1) use it or get left behind and 2) let it serve as a really good guide, but not as your CIO.
