Welcome!

Cloud Event Processing - Analyze, Sense, Respond

Colin Clark

Subscribe to Colin Clark: eMailAlertsEmail Alerts
Get Colin Clark via: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


Blog Feed Post

It’s Deja Vu – All Over Again

I was recently asked “What problems does CEP solve that cannot be solved with smart coding, a columnar database or a whopping great grid?” via a Linkedin group for Complex Event Processing.  Here’s the link.  I think I understand the question, and if I do, it’s really the wrong question.  Which means of course that my answer is probably going to cause a bit of a ruckus.

WHAT IS CEP ANYWAY?

There are a couple of things that a system needs to say that it incorporates CEP – the ability to continously query an event stream, pattern matching semantics, and sliding windows over data.  For example:

SELECT PHASORID, AVG(FREQUENCY)  FROM SMARTGRID.WIN:TIME(30 MINUTES) GROUP BY PHASORID;

This sets up a continuous query – it listens for events that are named ‘SMARTGRID’ and calculates the average frequency by phasorid.  This is an example of a time based window.  The window always refers to the last 30 minutes.  Another example:

SELECT A.* FROM PATTERN(A=SMARTGRID(FREQUENCY>130) -> (TIME.INTERVAL=30 MINUTES) AND B=SMARTGRID(FREQUENCY<110 AND A.PHASORID=B.PHASORID));

This statement says, “I want to know when a phasor measures a  high frequency followed by a lower frequency.  This is an example of pattern matching.  Both examples demonstrate a continuous query – those queries just keep running, returning a continuous stream of results.

THAT’S A LOT OF CEP

I’ve been focused primarily on solving problems using CEP for the last 6 years, and using event processing for the last 20+.  So, I’ve seen lots of examples of the right use of CEP and some wrong uses of CEP as well.  We’ll talk about the right uses here.  Mark Piper asks about, “smart coding.”  Let’s answer that one first.

SMART CODERS

Sure, smart coders can get along without CEP.  Just like they can get along without a database.  Or a messaging bus.  And if you’re a really smart coder, you don’t even need an operating system.  You’ll just write all of those using your copious amount of spare time that your employer provides you because you don’t have any deliverables that are time sensitive.  And since you don’t have any customers, you can do whatever you want anyway.  Right?  Wrong!

There’s a difference between an implementation of CEP and CEP itself – maybe you want the whole development environment ala StreamBase, Apama or Sybase.  Maybe you want an API.  That’s most likely up to your individual/team coding style and whether or not you actually buy into the, “Use our platform for everything – it will save you time and money!” argument.

But the gist of this argument is that if you want to have a CEP source of functionality to use when solving problems by writing code, you probably want a general library, or system, that you can use to do so instead of writing it yourself.  And I’ve had the privilege of working with some very smart coders over the years and it took even them slightly longer than a month or two to build something that could be used in a number of different scenarios.

CHEAPER CODE

So that’s the tech side.  What about the business side?  That’s even easier – I don’t want to hire a bunch of C++ coders to write infrastructure – unless that’s my business.  And most businesses don’t fall into that category.  A lot of big financial firms do get a bigger bang for buck by writing their own stuff – they’ve got the staff (because they can afford the really smart coders it takes to build this stuff), they’ve got performance requirements that are extremely stringent and it’s actually cheaper for them to build vs buy given scope and breadth of deployment.  So, unless it’s your business to build infrastructure by using very expensive coders, it’s cheaper and just as effective to use third party libraries or systems.  And that’s just not for CEP but generally applicable across the board.

COLUMNAR DATABASES AND GRIDS?

Not sure if I understand the last part of Mark’s question, but I’ll take a stab anyway.  Mark mentions grid – and the use of grid vs stream processing typically finds itself used in different parts of the same problem.  For example, if I’m running a big credit derivatives trading operation, I’ve probably got a grid running some fairly heavy compute and updating a shared cache of stuff that the bank needs for a variety of applications.  Then let’s imagine that I’m receiving client orders and want to do something based upon data in that grid, I might just use a system that incorporates components of CEP (notice that I did not say a CEP system here – that’s important) to streamline the event stream processing characteristics of that application using the grid cache to grab parameters for me as they change.  So the use of grid and CEP here is complimentary – not dynamically opposed.  Two different compute problems, two different technologies applied to a very common pattern.  There are similar patterns involving the use of CEP for data-in-flight and columnar, or any database really, for historical data.

SO IN CONCLUSION

I recommend the use of existing code over writing new code just because you’ve got smart coders if it makes sense and is economically attractive.  And, just like in the 90′s when we were selling EAI applications, all the C++ guys whined back then, complaining that, “We can do it better and faster.”  While that may have been true (very few ever actually proved that to me), it was usually far more expensive and very brittle.  Tools exist for a reason – it’s what separates us from the animals.

AND AS ALWAYS

Thanks for reading!

Read the original blog entry...

More Stories By Colin Clark

Colin Clark is the CTO for Cloud Event Processing, Inc. and is widely regarded as a thought leader and pioneer in both Complex Event Processing and its application within Capital Markets.

Follow Colin on Twitter at http:\\twitter.com\EventCloudPro to learn more about cloud based event processing using map/reduce, complex event processing, and event driven pattern matching agents. You can also send topic suggestions or questions to colin@cloudeventprocessing.com