Cloud Event Processing - Analyze, Sense, Respond

Colin Clark

Subscribe to Colin Clark: eMailAlertsEmail Alerts
Get Colin Clark via: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn

Related Topics: Cloud Computing, Intel Virtualization Journal, Rackspace Journal, Infrastructure On Demand, Intel SOA Journal, Twitter on Ulitzer, Ubuntu Linux Journal, Infrastructure 2.0 Journal, CEP on Ulitzer, Big Data on Ulitzer

BigData: Blog Feed Post

IaaS and Building a Private Cloud

Cloud Defined (IaaS or Infrastructure as a Service)

Cloud Expo on Ulitzer

I was speaking with a company last week that would like to get into cloud event processing.  For the sake of discussion, I’m going to define ‘cloud’ as the ability to dynamically, and upon demand, use more or less compute, storage, or network capacity. This is often referred to as ‘elasticity.’  I’m also going to add a few other requirements; this cloud must be able to:

  1. Host operations for a number of customers, all accessing the same data, but in different ways,
  2. Provide access to a lot of big data (see #3 – it’s all stored too),
  3. Able to handle messages rates approach 1MM per second (yes, that’s a million messages per second),
  4. Do something with those messages before storing them (in fact, we’re not even worried about storing them),
  5. And while we’re doing something with those messages, relate them to previously stored messages.

Cloud Event Processing Defined
And I’m going to define cloud event processing as, ‘using many varieties of event processing; including complex event processing (CEP), map/reduce, and event, or message, driven agents to process events.’  Using DarkStar, we’re able to deploy these various forms of event processing without regard for underlying storage, compute or network – the event processing services are simply deployed and begin processing; if a particular service needs more resources, it asks for them and when it receives them, it uses them.

Private Cloud Defined
It’s a private cloud when you deploy the above (Iaas) internally.  It’s a public cloud when it’s deployed publicly, and it’s a hybrid cloud when it’s deployed both privately and publicly.

Data Affinity
The large amount of inbound network bandwidth that would be required to get this particular cloud off the ground and into the sky already makes hosting with Amazon or Rackspace more than a ’sign up with a credit card’ process.  Having the elastic resources available to each customer of our customer makes things even more difficult.  See, our customer has 100’s of customers who want to do something with this data.  And each one of those customers will need access to both the old data, and the streaming 1MM messages per second.  Without creating a separate physical footprint in a cloud provider, it would be very difficult to solve this problem.

Buy vs Build
The more things change, the more they stay the same.  So, now we’re involved in the old, ‘how much to build vs buy?’ process.  The thing is, this company has the skills to run their own cloud so we’ll most likely start with a proof of concept and then grow it.  We’re cloud seeding at Cloud Event Processing!

The Test Platform
Our recent work with Twitter will be our first deployment in this cloud infant.  Using DarkStar, we’re able to analyze the Twitter feed to sense and respond to opportunities and threats.  This relatively simple project includes components of event driven agents, NoSQL storage, CEP sliding windows & aggregation, visualization, map/reduce and analytics.  (For those of you who’ve been reading all along, you’re asking, ‘analytics?’-we didn’t see any of those Colin! – and you’re right, you didn’t – we’ll share those when we’re ready.)

Kickin’ Tires & Lightin’ Fires
We’ll be starting off with some bog standard Intel hardware, Eucalyptus, and the Ubuntu Enterprise Cloud.  There’s a white paper available on Eucalyptus’s site that details our initial approach.

Thanks for reading!

Read the original blog entry...

More Stories By Colin Clark

Colin Clark is the CTO for Cloud Event Processing, Inc. and is widely regarded as a thought leader and pioneer in both Complex Event Processing and its application within Capital Markets.

Follow Colin on Twitter at http:\\twitter.com\EventCloudPro to learn more about cloud based event processing using map/reduce, complex event processing, and event driven pattern matching agents. You can also send topic suggestions or questions to colin@cloudeventprocessing.com

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.