The simple definition of Big Data refers to any analysis that consumes data that exceeds a single systems capacity to collect and analyze the data. Admittedly simplistic, but a great starting point. Dr. Attila Barta, one of the major Architecture leads at BMO, advises that we (managers) must all take the time to understand how we got to this point, in order to prepare to move forward. Sage advice, but will enough of us take the time to understand this before it takes us over? Okay, so how did we get here? Dr. Barta observes some major events in 2001 that contributed to the rapid growth in the ensuing decade. These examples show the increase in Volume,Velocity, Variety and Variability of data which drove the development of big data architectures, such as Hadoop and advanced analytics.
1. Sloan Sky Survey began collecting astronomical data in the realm of 200GB /night requiring systems that could handle volumes of data.
2. Sensor networks, RFIDs, and streaming data bases drove the messaging architectures that can handle the velocity of data.
3. Semi structured databases, xml and unstructured data such as video and voice drives the need for systems to integrate variety and variability of data.
Immediately following this, we saw the rise of search engines, social media, cloud computing all while the cost and availability of high performance systems, memory and analytics plummeted.
Eleven years later, we are in the midst of the big data revolution, which is a global phenomena, not some localized or academic anomaly. Think about the amount of data and analytics that are required the next time your grocery store sends you a ridiculously accurate sales incentive, such as the ones coming from Metro and Sobey’s.
Unfortunately, Corporate Canada is a bit behind. Only 10% of professionals know about Hadoop and related advances in processing models. 75% of the worlds enterprises are working with Big Data, but in Canada, only about 50% of companies have formal big data projects. A key accelerator of the effectiveness of big data projects is the integration of external data but the majority of Canadian firms limit themselves to directly collected data, not even capitalizing on the “exhaust data” from their systems. These are the symptoms we see in the technology aware community, but the real challenge is that companies still relegate technology strategy and decisions below the corporate leadership, leaving it to middle managers to determine the fate of the business, slightly left of the core strategy. World class companies like Google and our own Airmiles have figured out that data and technology driven creativity is the future and is therefore the center of their strategies.
I won’t try to educate the audience on Hadoop, realtime processing, location based mobility, NFC and RFID potential, parallel processing or advanced analytics, but I would ask that as a manager in any business other than ??? (I can’t think of one), you should get educated and start paying attention. For an eye opener, start by reading SuperCrunchers by Ian Ayers.
Dr. Barta is a wise man with a great perspective, so I will quote him here.
1. Business leaders must take the initiative, they know how to get things funded and moving. McKinsey estimates that we have as many as 1.5 Million jobs open for data savvy managers and technologists. This skill and resource gap will favour the first movers and the creative managers.
2. CIO’s must become Holistic, well beyond the confines of their organization and time frame, they must look to create the future.
3. Specifically, organization information management strategy must build new data point for social data, device data and sensor data.
4. The BI strategy must adopt commoditization and massive parallel processing to drive greater use of these capabilities across the organization.
5. Invest now in skills, see item 1 for rationale.
6. Most important - Augment every initiative in the organization with a big data filter. Big data is an evolution, as well as technology disruptor.
So it Big Data really Borg Data? I think so, to a point. Big Data will consume everything, but it is not self aware and will require creative managers to guide and drive the realization of Big Data opportunities in every company. If there is not someone core to your corporate strategy team with a big data perspective, your strategy is probably in question.
Comments