Cloud Event Processing - Analyze, Sense, Respond

Colin Clark

Subscribe to Colin Clark: eMailAlertsEmail Alerts
Get Colin Clark via: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn

Blog Feed Post

Predictions for 2011

Some predictions for 2011.  In no particular order or importance.

1. CEP – The Feature

There’s a couple of things going on here.  The most important being that Mark Palmer is writing blog posts about Richard Tibbetts writing blog posts on the Tabb Group’s site about writing better software on Wall Street (because software startups write better code and deal with bigger problems than firms on Wall Street, especially exchanges, do, right?…).  No customer win.  No ‘yet another use for CEP.’  Just good old fashion buzz-word copy designed to remind people that there’s still one stand-alone CEP vendor. (click click click, is this link working?)

CEP will become a feature of larger, more established, horizontal offerings.  Because once the opportunities in Canada and South America dry up (you know those hotbeds for financial engineering, right? Canada and South America?  You don’t?) the reality will sink in.  What’s that reality?  That no one in NYC is buying CEP engines for HFT trading anymore.  Why?  The CEP vendors don’t know.  Even Apama has seen the light.  You can now process events one at a time, or within the context of their CEP engine.  Stunning.  They’re pushing the ‘platform.’  Good.  It’s about time Tibco got some competition.

So this coming year won’t be the year of CEP, it will be the yesteryear of CEP, like, “Remember yesteryear, when we all thought CEP was going to be really hot?”  CEP will become a feature found in Event Processing Platforms.  And we’ll finally start to see adoption of those platforms in large, house hold names.

2. Hadoop & Analytical Databases

People are going to begin realizing that databases that incorporate map/reduce into their architecture will be *no faster* than Hadoop.  Why?  Go buy a book about Hadoop and then sit down with a piece of paper and pencil.  Databases are designed to support multitudes of users all asking different questions.  And vendors would like to have us believe that they’re also all running long lasting jobs taking advantage of their shiny map/reduce implementations.  Long lasting.  As in, not interactive.  As in, batch jobs running on a map/reduce framework.  The bigger the job, the less increase in throughput the analytical database will be able to offer.  It’s physics.  Save your money.  So this coming year, we’ll probably see further consolidation and some unexpected exits in this area.

3.  And Speaking About Hadoop

Batch is dead as competitive advantage.  As Jeff Jonas loves to point out, and points out really well, data velocity is growing.  And the rate at which data velocity is growing is increasing.  And companies can’t process the data they have today.  And many companies are actually making bad decisions with more data, not better decisions, why?  Because the data has lost most of its value once it’s been crunched.  You can only take a batch system like Hadoop so far.  But right now (at least for the near future) you actually still need to incorporate some ideas from the batch world for everything to come together.  So this coming year, we’ll see more people start treating Hadoop as either a must have to compete (minimal cost of entry) or “How f(*)(* much does that cluster cost to run?  There’s got to be a better way!”  It’s time to outsource your Hadoop cluster.

4. Real Time & Batch

I’m not saying that Hadoop or Map/Reduce is a waste of time and money like some vendors who make outrageous claims like, “Google has stopped using map/reduce.”  That’s idiotic.  Please put the kool-aid down, you’ve had enough to drink.  What’s important is the ability to analyze data in flight, to make decisions while there’s still the opportunity to have an impact.  How does one accomplish this?  By having a context in which events are analyzed.  How is context built?  Via the constant processing of events in flight, constructing and augmenting context, and supplementing that context with the result of monster jobs run on gazilla-bytes of data (like that? gazilla- it’s mine).  So this coming year, we’ll see a focus on moving analytics to real time.  (deeper analytics than VWAP-please!)

5. The Big Picture

There have been some really neato-keeno entries in the visualization space.  Things like Tableau, Spotfire, etc.  But they’re great for analysis of relatively static data sources.  They’re not for real time stuff.  Even offerings from vendors like Panopticon, which can provide some insight into multidimensional data sources updating in real time, really offer quite limited analysis tools for big data.  So in the coming year, we should see more focus on real time data mining.

6.  Did He Say, “Big Data?”

Yup.  I said it.  I’ve heard people define big data as more data than will fit on one machine.  Those people haven’t worked on the machines I’ve worked on.  I define big data as “when you can’t turn your data into actionable intelligence fast enough to have an impact during the window of opportunity.”  Or something like that.  I’m not in marketing.  Common patterns for analyzing big data are emerging that tie as-it-happens analysis with context and historical data.  The lines are blurring.  It’s all just becoming data.  And business wants it all.  Even my father-in-law, in Germany, asked, “Ja, SAP wants me to store my data in ze cloud.”  Everyone knows about big data and 2011 is going to be all about it.  Getting it.  Storing it.  Analyzing it.  Visualizing it.  And then we’ll see the real emergence of privacy issues, like what happens when our respective governments start using simple tools like the ‘People You May Know’ from Linkedin during child pornography investigations?  It’s going to happen.  Our government isn’t ready.  The legal infrstructure isn’t there.  There will be the formation of chaos in this regard in 2011.

7.  Computer Network Attack Platforms (CNA)

In 2011, something important is going to happen.  The general populace will be made aware that, in addition to all the traditional, ground-based, we can’t win engagements in places like Afghanistan and Iraq, that we’re also involved in a different kind of war.  One that rages on every day, one that runs 24×7 and involves facets of technology present both on earth and in space and in the Internet.  And that’s cyber warfare.  World War III is here already – China has already ‘stolen’ the Internet from the United States for about 15 minutes, diverting the majority of our most important Internet based traffic through their country for storage and analysis.  Doesn’t that raise an eyebrow?  It should.  One of the biggest things that happened in 2010 went off without a hitch and without a great deal of coverage.  Iran’s nuclear power plant, the one everyone was afraid of, was rendered inoperable by a virus.  Because that plant is inoperable, Russia is going to continue to make big money.  And Israel is going to sleep easier.  Odd bedfellows, wouldn’t you say?  We’re going to learn more about CNA’s in 2011.

7.  What about NoSQL?

Yawn.  It’s going to be a TWO BILLION DOLLAR market.  Just like CEP was.  Really.

Happy New Year!

Read the original blog entry...

More Stories By Colin Clark

Colin Clark is the CTO for Cloud Event Processing, Inc. and is widely regarded as a thought leader and pioneer in both Complex Event Processing and its application within Capital Markets.

Follow Colin on Twitter at http:\\twitter.com\EventCloudPro to learn more about cloud based event processing using map/reduce, complex event processing, and event driven pattern matching agents. You can also send topic suggestions or questions to colin@cloudeventprocessing.com