We have huge quantities of Big Data about customer preferences and behavior pouring in every minute, and no end of fancy tools for analyzing those data. And yet, for every Sony Walkman or Apple iPad to rapidly dominate its respective sui generis market, there are hundreds of less successful innovations piled up in the bargain bins at the Dollar Store. With all these Big Data, then, why aren't we that much better at innovating?
The answer: data analysis is much better suited for optimization than innovation. As we leverage data to get better and better at marketing, carving out a percent of market share here and a percent of conversion there, we're actually making decisions that end up stifling innovation. Our incessant march toward "better-faster-cheaper" is actually making us more brittle, not more innovative.
In fact, innovation depends more upon disruption than optimization. Disruption -- both internal and external -- can lead to periods of intense creativity and intuition. And yet, without data-driven optimization, disruption-driven innovation is nothing more than a crapshoot. Leveraging data to properly balance optimization and disruption, therefore, is essential for achieving market-leading levels of innovation.