Simple Event Processing != Complex Event Processing

One of the brillant minds in the CEP community, Claudio Paniagua Macia, recently posted, Event Stream Processing != Complex Event Processing.   In his post, Claudi draws a bold conclusion:

(1) SQL-based approaches to ESP might have a hard time doing CEP.

(2) No real CEP engine exists today in the marketplace, perhaps not even “off” the marketplace.

Friend, colleague, and co-chair Opher Etzion replied, On Event Stream Processing:

 “CEP engines do exist today, none is perfect, but probably sufficient for big majority of the existing applications today.”

Respectfully, I find it necessary to agree with Claudi and disagree with Opher.   Most of the so-called CEP engines today are solving quite simple event processing problems.   If the CEP engines on the market were truly solving a “majority of the exisiting applications today” then sales would be orders of magnitudes larger.

The fact-of-the-matter is that the current “simple rules-based approach” dominate in today’s marketplace are used to solve problems where rules-based approaches are useful.   Unfortunately, this is just a small fraction of the true potential of the CEP market.

For example (just one example of many), the vast majority of intrusion or fraud detection systems available today use rule-based approaches, and their detection capability, and the confidence in the detection, is quite elementary (poor quality).   If these systems worked well, cyberspace would be a very different and much safer place.  

Yes, it is useful to add another layer of rules, but rules alone will not solve the vast majority of CEP-domain classes of problems.   In addition, the CEP applications that have made the press recently are quite simple, certainly nothing scientifically earth shattering.

So, the sad truth of the matter, from an architectural, scientific and solutions perspective, is exactly as Claudi boldly offered, no real CEP engine exists today.    Furthermore, the vast majority, if not all, CEP applications sold today are used in very simple event processing (SEP) applications.  This is not very “advanced,” but it is a good start.  

What is holding the CEP market back is quite straight forward; the current “engines” are quite elementary (We should call them SEP engines.), relatively speaking, and SEP engines do not have the capability to solve difficult detection-oriented CEP problems in cyberspace.   These difficult problems compose the vast majority of the applications where “true complex event processing” is required.


5 Responses to Simple Event Processing != Complex Event Processing

  1. Opher Etzion says:

    Hi Tim. Good that you disagree with me, otherwise life will be dull :-).

    But I will have to disappoint you, by agreeing with most of what you have said, first I agree that Claudi is a very bright person and has a very original mind, second I agree that CEP engines are not solving very complex problems merely by the fact that they are deterministic, and complex problems are not deterministic (cannot be phrased by patterns/rules/queries – each one can use his/her favorite API), and since the pattern/rule/query style has been already “captured” by the name CEP, and all these products are associated with the CEP name, I prefer to call the predictive/stochastic type of event processing in the name IEP (Intelligent Event Processing) — actually I am not the copyrighter, one of my other IBM colleagues coined this term, and it is being used inside IBM now. As for “does CEP engine exists” – I have answered my young and energetic colleage Claudi, that the fact that a CEP engine does not support transitive closure does not mean that it is not a CEP engine, and there is a very big space for applications that are being supported today by the current capabilities of CEP; even current CEP are far from getting to their market potential, but this is another discussion about the current market inhibitros.

    Have fun in Bangkok (the traffic in Bangkok is impossible, but Thailand is amazing).

  2. Mark Proctor says:

    Our Temporal Reasoning support is starting to take shape in Drools, we would very much love feedback on the best syntax; which is an orthogonal extension to our existing rule language. What we will deliver initially will be quite simple, although it will allow multiple entry points to be defined within a rule, with full correlation between t he entry point and it’s patterns and the rest of the rule and working memory. Over time we want to add more complex things, like interval based semantics, and we also recognise the need to start to include some “soft” reasoning sub systems, like bayesian networks, fuzzy logic etc.
    Recent blog, showing where we are now:
    wiki page with different syntax proposals:
    Javapolos presentation showing a vision for a Business Logic Platform with unified and integrated rules, processes and cep:

    Lots of feedback please 🙂

    Mark The Drools Blog

  3. Tim Bass says:

    Hi Mark,

    Thank you for visiting and for your comment.

    It is encouraging to know that Drools has recognized the need to enhance your event processing platform with Bayesian Networks or Fuzzy Logic capabilities.

    Could you detail how you plan to include or integrate theee capabilities within the Drools platform?

    Yours faithfully, Tim

  4. Mark Proctor says:

    heh, haven’t thought that far ahead 🙂 One part is simply having evaluators that are handled by uncertainty sub systems – that is mentioned at the back of the presentation. If we go for a bayesian network we’ll probably have that as a totally separate sub-sytem and simple reason over the output. For now we have more basic things to fry, expect we’ll get round to those in another 12 months or so – depending on community feedback.

    We will probably make a streaming sql parser too, which will map to our drl, for those that need it.

  5. peter lin says:

    There are some old research papers exploring the intersection of bayesian analysis with case base reasoning that might be useful. I know that some vendors have used case base reasoning to build adaptive customer support systems. The basic idea is to start with a set of training data. The system learns which facts correlate with other facts. Once the system is deployed, it takes the users feedback and continues to learn. This way, over time, the system learns and adapts based on feedback.

    I know that within the education agent field, a lot of research has gone into adaptive techniques. The common example is an agent driven tutor. As the user reads the material and answers questions, the system monitors how long it took to read the material and which questions the users gets wrong. Using that data, the system tries to guess why the user got a particular question wrong. Say for example, if the reader flipped past the page and didn’t bother to read it, the system may respond with “review pages x-x”. If the user spent a lot of time reaading a page and missed all the questions related to an important concept, the system makes a suggestion. Many of these systems combine a rule engine with fuzzy logic or statistical techniques to make “intelligent guesses”.

    From my limited knowledge, many of these kinds of systems have metamodels and metarules, which drive how the system learns and adapts.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: