What is Complex Event Processing? (Part 8)

Posted by Tim Bass

In What is Complex Event Processing? (Part 7), we introduced process refinement – the feedback loop, resource management and work flow components of event processing architectures. Today we review another critical area in the functional CEP reference architecture – visualization, which includes both user interfaces and scientific visualizations.

Solving complex distributed problems require visualization at all levels of the inference model we have been describing in our blog series, What is Complex Event Processing?; and all of us have heard anecdotal stories about how today’s computers have the equivalence in intelligence of an earthworm. Alas, it is true! Present day high-speed computers, operating at a million operations a second still cannot compete with the human brain in many areas – especially in the area of pattern matching.

Event Processing Reference Architecture

This brings to memory “the good old days” back around 1992-1993 when I was consulting for Sprint. Sprint (Sprintlink) had the contract from the NSF to transition an academic-oriented NSFNet to a commercial Internet backbone. I was leading efforts to develop and build the network and security management for Sprintlink. It is interesting to note that good friends David Luckham and John Bates, two leaders in event processing, also have their roots in network management and security.

Well, to make a long story short, I had installed HP OpenView to help manage the network and was monitoring the Internet traffic at the major backbone routers, including FIXEAST and FIXWEST. Looking at the graph, the traffic looked odd; it would peak very high and then stop to almost zero, over and over again at regular intervals. I called everyone over, excited like a kid with a new toy (the Internet backbone is a nice toy, BTW) and exclaimed, “The Internet has a heartbeat! – It’s alive!!

Well, as an electrical engineer, it started to make perfect sense. The Internet is based on a queuing model for communications, the packets transmitted, queued and retransmitted across the Internet, just like a heartbeat. It was the visualization that brought theory and practical application into sharp focus. There is nothing in the known universe that compares to the human mind and the impact visualization has to help us understand and solve complex problems.

Complex event processing requires visualization at every level of the event processing model (above).

Examples of visualization in CEP are:

* Event Pre-processing: using an XPATH tool to map raw sensor input data to a JMS message format;

* Event Refinement: tracking and graphing a stock price, foreign exchange, or other event object.;

* Situational Refinement: providing a visual list in a network management center of the top 20 detected threat-related situations in an on-line banking application, with a estimated “name” and conditional probability;

* Impact Assessment: providing a visual list of the top 10 detected equity trading opportunities with estimated profit, along with with a risk KPI, if a trade is executed;

* Process Refinement: providing a visual graphic of alternative routes for commercial aircraft during a snowstorm; or,

* User Interfaces (UIs): tools to model and design event processing scenarios, rules and other analytics.

Business Activity Monitoring, or BAM, is another example of a current buzzword (another over-hyped subject!) for visualization in business applications.

So, What is Complex Event Processing?

I hope this brief eight part series on CEP was useful to readers interested in event processing and how to apply CEP to their area of expertise.

Copyright © 2007 by Tim Bass, All Rights Reserved.


3 Responses to What is Complex Event Processing? (Part 8)

  1. Thomas Owens says:

    After reading the entire 8 part series, one thing jumps out at me –

    I’m just curious as to how many of the levels (including event preprocessing and database management) a system must be capable of to be considered a complex event processing engine. I’m also curious as to how many of the levels must be applicable to a specific problem to make CEP an answer that should be considered (as I’m sure not every evel can be applied to every problem).

    I guess my question can be boiled down to something like – With your reference architecture, where do you draw the line between a system that can perform CEP and a system that can not? Is there even a measure of how well a system conforms to this architecture?

  2. Tim Bass says:

    Hi Thomas,

    Your question is a really good one. First of all Dr. Luckham has stated that the overall goals of CEP are to model, simulate, and process a wide range of complex problems, including partially ordered event data (posets), where relationship between events in the set are not obvious. In addition, noise and uncertainty tends to dominate these problems, hence the term “event cloud” by Dr. Luckham.

    On the other hand, since totally ordered sets (tosets), like many data streams, are a subset of a poset set, there must be “levels’ of compliance, which we have not yet defined as a CEP community.

    So, to answer your question, I would suggest we need, as a community, to define compliance levels, based on the level of inference in the system. These would roughly correspond to the “levels” in the reference architecture.

    You are completely right in your statement that all classes of CEP problems do not require all the functionality illustrated in the reference diagram.

    Does that answer your question?

    Yours faithfully, Tim

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: