Over the past few weeks, I’ve implemented map/reduce using techniques
commonly found in Complex Event Processing. Here’s a summary of what was
involved, and what tools would make such a deployment easier.
Getting the Data
One of the first tasks accomplished was the creation of an OnRamp – we use
OnRamps to get data into our cloud for processing. The specific OnRamp used
in this learning exercise subscribed to Twitter and fed the resulting JSON
objects onto the service bus, RabbitMQ in this case. We had to correctly
configure RabbitMQ for this, and the OnRamp needed to be specifically aware
of and implement semantics required to publish on this bus. It would be
easier and more portable if this were abstracted in some type of OnRamp api;
we had abstracted this at Kaskad. In Korrelera, the bus didn’t matter –
we could just as easily use direct sockets, JMS, Tibco o... (more)
Lately, I’ve been working on some interesting projects involving not just
the usual suspects of stream processing, but data mining within high velocity
time series. In conjunction with that effort, I’ve been doing a lot of
research in the areas of symbolic representation, dimension reduction,
clustering, indexing, classification, and anomaly detection. A prolific
researcher in this area is Dr. Eamonn Keogh – I’ll be applying some of
his team’s ideas so some interesting customer problems and telling you all
about it here. Let’s get started!
TOO MUCH DATA
In dealing with real... (more)
Every year, I like to decompress a bit and take a break. Usually, I like to
go scuba diving – the dive sites I like are usually far removed from email,
Twitter, Facebook, etc. and it gives me a chance to actually unplug, defrag,
and think a bit.
This year, the family went to Grand Cayman to experience some of the
world’s best diving. Within Grand Cayman is Hell, a small township
dedicated to tourism and aptly named given the attached photo.
We visited Hell in between dives, and rather than make my ex-wife room
reservations or send out postcards, I thought I’d amplify a few predic... (more)
VMWare has been on a buying spree lately.
In the last month, they’ve announced both Redis and RabbitMQ.
Here’s VMware’s take on Redis, and spring source’s on RabbitMQ.
RabbitMQ is built with Erlang.
Much Rejoicing in the Village
We use both of these technologies at Cloud Event Processing. And we love
Erlang too. VMware’s acquisition of these technologies not only validates
our decisions, which we are very selfishly pleased about, but also sends an
The Message Please
Everyone’s busy abstracting resources in the cloud – making resources
like compute, stora... (more)
“They pulled me back in.” – The God Father.
I’ve received some interest/emails about TwitURL – our Map/Reduce as it
applies to CEP (cloud event processing) project. Seems that people would
like to see the results of these processes visually. Who can blame them,
right? So, I was thinking, how can I add a little sizzle to TwitURL?
Panopticon offers some pretty slick visualization capabilities – you can
check them out here. And based upon some feedback/requests, I’m going to
hook up a heat map to the output of TwitURL. The heat map will show which
URL’s are the hott... (more)