Why is IBM involved with Apache Spark?
How Emerging Technologies Works Within IBM,
and How It Led to Spark.
Recently, IBM announced its IBM Spark initiative, detailing how the company would move forward with the open-source compute cluster framework, including the creation of a Spark Technology Center based in San Francisco, California. This initiative, one of IBM’s largest, already includes over a dozen IBM labs, as well as thousands of IBM researchers and developers. IBM Emerging Technologies played a key role in exploring Spark with real world business challenges with its customers, and then advocating Spark internally to IBM. Here’s the story of how Spark came to be within IBM…
So what caused IBM Emerging Technologies to get interested in Apache Spark? Recently I chatted with the folks at the Cube about what caught my attention regarding Spark, what I was hearing in the marketplace that lead me to the technology, as well as the business need and challenges which were not only interesting but addressed a significant business need. Finally, I share my thoughts on how Spark figures into IBM’s plans moving forward.
Highlights from that Conversation:
- What drove the initial interest? IBM Emerging Technologies gleaned from it’s clients and customers a need to do large scale analytics faster–within seconds instead of hours. They couldn’t tell us how they wanted it implemented, just that there was business rationale behind the need.
- What was the business driver for this interest in rapid analytics? In essence, the more our clients learned, the more questions they had. And the more questions, the more pressing their need to get results quickly and enable the iterations needed to drive towards valuable business insights and opportunity identification.
- Why Spark? Created as an analytics framework, Spark combined that capability with significant advancements in speed, reducing time-to-value by an order of magnitude. In addition, the flexibility of the framework lent itself to creating business solutions that could exist within existing infrastructures, key to helping businesses transition to a data driven model.
- How committed is IBM to Spark? As I noted earlier, IBM has committed significant resources to Spark–including the Spark Technology Center, enabling a Spark-As-A-Service model on it’s Cloud Platform, Bluemix, as well as dedicating thousands of IBMers to building solutions and engaging the Spark community. That doesn’t happen if an initiative doesn’t have support at the highest levels of the company. But it’s not just executive support–in fact, at a recent Hackathon, 28,000 IBMers participated, illustrating the interest within the company for the technology.
- What’s the future of Spark and IBM? IBM is committed to helping the Spark community grow by contributing and participating in it and to help provide solutions which enable that growth. The future of Spark is not only in it’s evolution as an integration technology, but it also lends itself to something I call “portability analytics”–the ability to easily and transparently move analytics models from platform to platform or service to service, enabling the movement of the analytics to wherever it’s needed.
Curious to learn more about what I’ve learned about Spark, and the future of Spark in IBM?Feel free to check out the video of the entire chat.