Lately, I’ve been working on some interesting projects involving not just
the usual suspects of stream processing, but data mining within high velocity
time series. In conjunction with that effort, I’ve been doing a lot of
research in the areas of symbolic representation, dimension reduction,
clustering, indexing, classification, and anomaly detection. A prolific
researcher in this area is Dr. Eamonn Keogh – I’ll be applying some of
his team’s ideas so some interesting customer problems and telling you all
about it here. Let’s get started!
TOO MUCH DATA
In dealing with real time streaming numerical data, there is just too much of
it sometimes to do anything meaningful with it in real time. For example,
in pattern recognition, trying to compute nearest neighbors using continuous,
highly dimensional data is a compute nightmare. Or, once you’ve
identified a pat... (more)
In the past year or so, I’ve heard from many skeptics – people who
didn’t believe that Event Processing could be successfully deployed in the
cloud. Granted, most of these folks represented firms actively engaged in
providing the High Frequency Trading (Algo Trading) industry with tools.
And in that arena, cloud deployment probably doesn’t make sense. Yet.
Close to Home Though
Ask people in Capital Markets about Twitter and the most common response
you’ll get is, “What do people use it for?” This is because most of
the people in Capital Markets can’t use things like Twitter, i... (more)
On this continuing series, I am examining thoughts and specific
implementation details around building a back-testing platform for algo
trading. Eventually, we’ll see where complex event processing plays and
how to implement it.
Appendix to Part One – The Data Format
Rather than looking at various database solutions first and then trying to
define the problem in terms of those solutions, let’s first examine what
market data looks like. In its most simple form, market data looks like
this (there’s usually a little more, but this is fine for our purposes):
Date: The date of the ... (more)
Every year, I like to decompress a bit and take a break. Usually, I like to
go scuba diving – the dive sites I like are usually far removed from email,
Twitter, Facebook, etc. and it gives me a chance to actually unplug, defrag,
and think a bit.
This year, the family went to Grand Cayman to experience some of the
world’s best diving. Within Grand Cayman is Hell, a small township
dedicated to tourism and aptly named given the attached photo.
We visited Hell in between dives, and rather than make my ex-wife room
reservations or send out postcards, I thought I’d amplify a few predic... (more)
VMWare has been on a buying spree lately.
In the last month, they’ve announced both Redis and RabbitMQ.
Here’s VMware’s take on Redis, and spring source’s on RabbitMQ.
RabbitMQ is built with Erlang.
Much Rejoicing in the Village
We use both of these technologies at Cloud Event Processing. And we love
Erlang too. VMware’s acquisition of these technologies not only validates
our decisions, which we are very selfishly pleased about, but also sends an
The Message Please
Everyone’s busy abstracting resources in the cloud – making resources
like compute, stora... (more)