Welcome!

Cloud Event Processing - Analyze, Sense, Respond

Colin Clark

Subscribe to Colin Clark: eMailAlertsEmail Alerts
Get Colin Clark via: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


Top Stories by Colin Clark

Every year, I like to decompress a bit and take a break. Usually, I like to go scuba diving – the dive sites I like are usually far removed from email, Twitter, Facebook, etc. and it gives me a chance to actually unplug, defrag, and think a bit. This year, the family went to Grand Cayman to experience some of the world’s best diving. Within Grand Cayman is Hell, a small township dedicated to tourism and aptly named given the attached photo. We visited Hell in between dives, and rather than make my ex-wife room reservations or send out postcards, I thought I’d amplify a few predictions I’d made earlier after the benefit of some quality thinking time. So here they are: Cloud based services will finally hit Wall Street in 2011. It makes a lot of sense for firms to offer shared access to things like market data and to provide elastic resource for its manipulation. It d... (more)

Let’s Get Esper Up & Running

For our CEP in the Cloud example, I’ve briefly outlined a stream based load balancing idea.  In this example, RuleBots (pieces of code that do something – think of the procedural extensions built into some vendor’s products) send utilization statistics to the CEP Load Balancer via RabbitMQ.  The CEP Load Balancer in this case is written in Java and uses Esper to create a stream containing all available processes (destinations) in a particular service pool. What’s a Service Pool? For this example, we define a service pool as available compute resources in the cloud available to ... (more)

Event Processing in the Cloud – DataSift is a Big Proof Point

In the past year or so, I’ve heard from many skeptics – people who didn’t believe that Event Processing could be successfully deployed in the cloud.  Granted, most of these folks represented firms actively engaged in providing the High Frequency Trading (Algo Trading) industry with tools.  And in that arena, cloud deployment probably doesn’t make sense.  Yet. Close to Home Though Ask people in Capital Markets about Twitter and the most common response you’ll get is, “What do people use it for?”  This is because most of the people in Capital Markets can’t use things like Twitter, i... (more)

Just When I Thought I Was Done… Bring on the Visualization!

“They pulled me back in.” – The God Father. I’ve received some interest/emails about TwitURL – our Map/Reduce as it applies to CEP (cloud event processing) project.  Seems that people would like to see the results of these processes visually.  Who can blame them, right?  So, I was thinking, how can I add a little sizzle to TwitURL? Panopticon Panopticon offers some pretty slick visualization capabilities – you can check them out here.  And based upon some feedback/requests, I’m going to hook up a heat map to the output of TwitURL.  The heat map will show which URL’s are the hott... (more)

Cloud Event Processing: CEP in the Cloud

Over the past few weeks, I’ve implemented map/reduce using techniques commonly found in Complex Event Processing.  Here’s a summary of what was involved, and what tools would make such a deployment easier. Getting the Data One of the first tasks accomplished was the creation of an OnRamp – we use OnRamps to get data into our cloud for processing.  The specific OnRamp used in this learning exercise subscribed to Twitter and fed the resulting JSON objects onto the service bus, RabbitMQ in this case.  We had to correctly configure RabbitMQ for this, and the OnRamp needed to be specif... (more)