On the Maturity of CEP

May 31, 2008

Deciphering the Myths Around Complex Event Processing  by Ivy Schmerken stimulated a recent flurry of blog posts about the maturity of CEP, including; Mark Palmer’s CEP Myths: Mature or Not? and Opher Etzion’s On Maturity.

I agree with Ivy.  CEP is not yet a mature technology by any stretch of the imagination.  In fact, I agree with all three of Ivy’s main points about CEP.

In 1998 David C. Luckham and Brian Frasca published a paper, Complex Event Processing in Distributed Systems on a new technology called complex event processing, or CEP (Postscript Version).  In that seminal paper on CEP, the authors said, precisely:

“Complex event processing is a new technology for extracting information from message-based systems.”

Ten years later there are niche players, mostly self-proclaimed CEP vendors,  whom do very little in the way of extracting critical, undiscovered, information from message-based, or event-based, systems.  

A handful of these niche players have informally redefined CEP as “performing low latency calculations across streaming market data.”  The calculations they perform are still relatively straight forward and they focus on how to promote white-box algo trading with commercial-off-the-shelf (COTS) software.  In this domain, we might be better off not using the term CEP at all, as this appears to be simply a type of new-fangled COTS algo trading engine.

The real domain of CEP, we thought, was in detecting complex events, sometime referred to as situations, from your digital event-driven infrastructure – the “event soup” for a lack of a better term.    In this domain, CEP, as COTS software, is still relatively immature and the current self-styled COTS CEP software on the market today is not yet tooled to perform complex situational analysis.

This perspective naturally leads to more energy flowing in-and-around the blogosphere, as folks “dumb down” CEP to be redefined as it benefits their marketing strategy, causing more confusion with customers who want CEP capabilties that have zero to do with low latency, high throughput algo trading, streaming market data processing, which maybe we should call “Capital Market Event Stream Processing” or CESP – but wait we don’t really need more acronyms!

Hold on just a minute!  Wasn’t it just a short couple of years ago that folks were arguing that, in capital markets, it was really ESP, not CEP, remember?  Now folks are saying that it is really CEP and that CEP is mature?   

CEP is mature?  CEP is really not ESP?  CEP is really event-driven SOA?  CEP is really real-time BI?  CEP is really low latency, high throughput, white-box COTs algo trading?  CEP is really not a type of BPM?  CEP is not really for detecting complex events?   Complex does not really  mean complex? 

Come on guys, give us a break! 

(Anyway, no one is going to give us a break….  so stay tuned!)

  


Clouding and Confusing the CEP Community

April 20, 2008

Ironically, our favorite software vendors have decided, in a nutshell, to redefine Dr. David Luckham’s definition of “event cloud” to match the lack-of-capabilities in their products.  

This is really funny, if you think about it. 

The definition of “event cloud” was coordinated over a long (over two year) period with the leading vendors in the event processing community and is based on the same concepts in David’s book, The Power of Events. 

But, since the stream-processing oriented vendors do not yet have the analytical capability to discover unknown causal relationship in contextually complex data sets, they have chosen to reduce and redefine the term “event cloud” to match their product’s lack-of-capability.  Why not simply admit they can only process a subdomain of the CEP space as defined by both Dr. Luckham and the CEP community-at-large? 

What’s the big deal?   Stream processing is a perfectly respectable profession!

David, along with the “event processing community,” defined the term “event cloud” as follows:

Event cloud: a partially ordered set of events (poset), either bounded or unbounded, where the partial orderings are imposed by the causal, timing and other relationships between the events.

Notes: Typically an event cloud is created by the events produced by one or more distributed systems. An event cloud may contain many event types, event streams and event channels. The difference between a cloud and a stream is that there is no event relationship that totally orders the events in a cloud. A stream is a cloud, but the converse is not necessarily true.

Note: CEP usually refers to event processing that assumes an event cloud as input, and thereby can make no assumptions about the arrival order of events.

Oddly enough, quite a few event processing vendors seem to have succeeded at confusing their customers, as evident in this post, Abstracting the CEP Engine, where a customer has seemingly been convinced by the (disinformational) marketing pitches  – “there are no clouds of events, only ordered streams.”

I think the problem is that folks are not comfortable with uncertainty and hidden causal relationships, so they give the standard “let’s run a calculation over a stream” example and state “that is all there is…” confusing the customers who know there is more to solving complex event processing problems.

So, let’s make this simple (we hope). referencing the invited keynote at DEBS 2007, Mythbusters: Event Stream Processing Versus Complex Event Processing.

In a nutshell…. (these examples are in the PDF above, BTW)

The set of market data from Citigroup (C) is an example of multiple “event streams.”

The set of all events that influence the NASDAQ is an “event cloud”.

Why?

Because a stream  of market data is a linear ordered set of data related by the timestamp of each transaction linked (relatively speaking) in context because it is Citigroup market data.    So, event processing software can process a stream of market data, perform a VWAP if they chose, and estimate a good time to enter and exit the market.  This is “good”.

However, the same software, at this point in time, cannot process many market data feeds in NASDAQ and provide a reasonable estimate of why the market moved a certain direction based on a statistical analysis of a large set of event data where the cause-and-effect features (in this case, relationships) are difficult to extract.  (BTW, this is generally called “feature extraction” in the scientific community.)

Why?

Because the current-state-of-the-art of stream-processing oriented event processing software do not perform the required backwards chaining to infer causality from large sets of data where causality is unknown, undiscovered and uncertain.

Forward chaining, continuous query, time series analytics across sliding time windows of streaming data can only perform a subset of the overall CEP domain as defined by Dr. Luckham et al.

It is really that simple.   Why cloud and confuse the community?

We like forward chaining using continuous queries and time series analysis across sliding time windows of streaming data. 

  • There is nothing dishonorable about forward chaining using continuous queries and time series analysis across sliding time windows of streaming data.   
  • There is nothing wrong with forward chaining using continuous queries and time series analysis across sliding time windows of streaming data. 
  • There is nothing embarrassing about forward chaining using continuous queries and time series analysis across sliding time windows of streaming data. 

Forward chaining using continuous queries and time series analysis across sliding time windows of streaming data is a subset of the CEP space, just like the definition above, repeated below:

The difference between a cloud and a stream is that there is no event relationship that totally orders the events in a cloud. A stream is a cloud, but the converse is not necessarily true.

It is really simple.   Why cloud a concept so simple and so accurate?


Aite Group Finds Huge Gains for CEP

February 6, 2008

A report by the Aite Group, Capital Markets Firms to Spend $41.8 Billion on IT in 2008,  finds that twice as many IT executives are planning to spend money on SOA compared to CEP.

“46% of respondents plan to implement a SOA this year.  [and ….] 23% anticipate adopting a complex event processing solution […]. “

Considering the many years of hype about SOA, this is a stunning gain for CEP.

SOA has been around for many years, so long in fact, folks often refer to SOA as “Same Old Architecture”.    Therefore, given the fact that the market for CEP is only just now starting to come into play, it is quite impressive that an emerging technology such as CEP is now in the funding radar at levels that are only half of the planned SOA implementations.

The Aite Group writes that “in spite of CEP vendor hype, only 23% anticipate adopting a complex event processing solution this year”; but what Aite failed to say was that in spite of years and years of SOA hype, and countless millions of dollars of SOA marketing, SOA only commands twice the buzz in new project implements.

This is really tremendous news for CEP.


Key Indicators (KIs) Versus Key Performance Indicators (KPIs)

January 31, 2008

SL‘s new web page, Solutions for CEP Engine Users, discusses how CEP is a “technology that is used to help companies detect both opportunities and threats in real-time with minimal coding and reusable key performance indicators (KPIs) and business models.”

I agree with SL, but would like to suggest my friends at SL expand the notion of KPIs in CEP to include the idea of KIs.  In my opinion, the SL phrase should read,  “technology that is used to help companies detect both opportunities and threats in real-time with minimal coding and reusable key indicators (KIs) and business models.”  

The reason for my suggestion is that KPIs are a subset of KIs.   KIs designate, in my mind, more than just performance.  

CEP is used to both detect opportunities and threats in real-time which may, or may not be, performance related.  For example, when a CEP engine detects evidence of fraudulent behavior, this is a KI.  The knowledge, or pattern, used to estimate this situation is a KI not a KPI, per se.   Also, when a CEP application is processing market data and indicates that it is the right time to purchase an equity and enter the market,  the knowledge used in this decision support application is a KI, not a KPI.

Therefore, I recommend when folks think about the notion of  “key performance indicators” (KPIs) in CEP and BAM, they should also think in terms of “key indicators” (KIs).   Detecting opportunities and threats in real-time are much broader than the traditional notion of KPIs. 


Events are the Heart of the COSO ERM Framework

January 24, 2008

COSO was originally formed in 1985 to sponsor the National Commission on Fraudulent Financial Reporting, an independent private sector initiative which studied the cause-and-effects that can lead to fraudulent financial reporting. 

COSO developed enterprise risk management (ERM) recommendations for public companies and their independent auditors, and also for the SEC, other regulators, and for educational institutions.

At the heart of COSO is events and how events, both opportunity and threat-related events, in context, effect enterprise risk management.

Detecting opportunity and threats in real-time, both mentioned in COSO, is a core CEP concept; so I will be blogging on how CEP relates to COSO and ERM (and also Basel II ORM) in a future blog post.

Please stay tuned …


IBM Will Acquire AptSoft

January 23, 2008

I was wondering when IBM would actually jump into the event processing market.  

Well, it was announced today that IBM will acquire Aptsoft, adding an event processing platform to the IBM WebSphere porfolio.  IBM will also gain AptSoft’s event processing reference customers.  This was a smart move by IBM.

Oracle is acquiring BEA which uses Esper under the hood, another stream processing engine. 

However, AptSoft has a more advanced user interface and graphical design-time environment when compared to the Oracle/BEA/Esper platform,; so the IBM/WebSphere/AptSoft offering will propel IBM to the top of the event processing market.

Now, it’s TIBCO’s turn to acquire an event stream (time series) processing platform to compliment their process-driven event processing offering.


The ART of Event Processing: Agility, Reuse, Transparency

January 18, 2008

The other day I discussed CEP in Layman’s Terms: Reuse and Agility. Today, our topic is CEP and transparency. One of the major benefits of “white box” event processing solutions is transparency, something not readily available or obvious in black-box solutions.

Friend and colleague John Bates, Progress Apama, often discusses the benefits of white-box algorithmic trading platforms in terms of increased time-to-market and other competitive advantages. I agree with John and would like to point out that there is another key benefit, in simple layman’s terms, transparency.

For example, let’s say you have designed an event processing solution for operational risk management (ORM). It is time for your favorite auditors to come by and they wish to take a look at what is going on with that proprietary black-box ORM application running quietly in the server room.

The nice auditors ask you, “What does that application do?” and you reply “Well, it looks for evidence of insider trading,” and they ask “Do you mind if we ask how?” and you respond “Good question, do you mind to wait a moment while I get you the contact info for the vendor because we don’t have access to the source code or the actual key indicators (KIs)?”

Now, let’s look at the white-box scenario:

Again, the nice auditors ask you, “What does that application do?” and you reply “Well, it looks for evidence of insider trading,” and they ask “Do you mind if we ask how?” and you respond “Yes, sit down and we will pull up our insider trading key indicator models. These models are stored in XML format and viewable in our graphical KI design studio. We can print out the KI models for insider trading if you like!” and the smiling auditor says “Thank you, your system is much more transparent than the last place we visited!”

This scenario also applies in looking for why certain KIs were not detected that should have been; or when performing a root cause analysis to see why the KI you used in your wrong business decision was inaccurate.

So, CEP in layman’s terms is what we might refer to as the ART of event processing:

  • Agility
  • Reuse
  • Transparency

Please feel free to reuse these idea, but please don’t forget to reference the author and this blog 🙂

Kindly share and reuse by reference, because all content in The CEP Blog is ©2007-2008 Tim Bass – All Rights Reserved. Thank you!