The Grammar of Complex and Intelligent Events

Folks defining CEP, and now this new term IEP, have been very passionate over the past few years that “Complex Event Processing” means the Processing of “Complex Events” not the “Complex Processing” of Events.   

Grammatically speaking, it follows that Complex is an adjective describing a noun, Event; and Processing is a verb. 

Complex events are defined by the same community as composite events, or events that are composed of two or more “contributing” events.

To be consistent, I think we should follow the same logic and grammar in the discussion of “Intelligent Event Processing”. 

It follows that Intelligent should be an adjective describing a noun, Event; and Processing is a verb.  It also follows that “Intelligent Event Processing” means the Processing of “Intelligent Events” not the “Intelligent Processing” of Events.   

This is precisely the problem that folks are creating a new CEP term, “Intelligent Event Processing” to describe processing capabilitities that are missing from the current suite of self-described CEP software products.   What people really mean to describe is the Intelligent Processing of Complex Events.   However, based on the same grammer used in defining CEP, they have created the Processing of Intelligent Events.

The use of inconsistent grammar and logic is not good for the CEP community, in my opinion.   Just because the current generation of self-described CEP vendors do not rise to the capability required by the vast majority of business event applications, we should not create new terms just to make marketing folks happy.

I think I am in a good position to speak about this, because some of my best friends work for software companies selling self-described CEP software and they have seemingly lost patience  because I refuse to support inconsistent illogical positioning and repositioning of the CEP market.

Why is the grammar between the terms “Complex Event Processing” and “Intelligent Event Processing” inconsistent?.   Folks can only spin and reposition CEP so much before all the spin, hype, and repositioning begins to catch up with the community.   

Dr. David Luckham’s original papers and single book on CEP was clear enough about CEP; and CEP covers the entire space that Opher Etzion would like to reposition as IEP.    The Grammar of Complex and Intelligent Events are, at best, misleading and inconsistent.

I think the main problem is that what Opher has been describing is the Intelligent Processing of Complex Events – however, to say this would affirm what I have been evangelizing for over two years.

The Infant, the Elephant and the Intelligent Event

Fellow blogger Opher Etzion, replies to  On Elephants and Analytics with On Unicorn, Professor and Infant.   Opher is kindly giving us another metaphor to consider, the Infant and the Profession, since we are both big fans of big gentle elephants, babies and our universities.  

Opher and I agree that Infants are not Professors, and we also agree that CEP is in its Infancy and there is overhype by folks often implying CEP is a Professor.     So it seems we all have a huge elephant in the room with an Infant Professor hanging on the end of a wildly swinging Elephant’s trunk!

To keep the blogopoints interesting, I should point out that with all this agreement and Kumbaya campfire singing, there are a couple of things I do disagree with in Opher’s amusing counterpoint. 

First of all, Opher uses the well know debate technique of falsely attributing some easily refutable discussion point and then offering a slam dunk counterpoint.   He does this in this clever, but completely inaccurate Opher quote,

 “I [Opher] respectfully disagree with Tim … in his claim that what has been done until today is just hype and hence totally worthless…”

Folks reading my blog know that I have never said “what has been done until today is … totally worthless.”    This is a misfortunate misquote.  Shame on you Opher!  

What I said, easily read in the blog, was that CEP is overhyped and that most of the self-described CEP software on the market today does not live up to the inflated claims we read and hear from CEP software vendors, the analysts and reporters they influence.

The second counterpoint that I find interesting is Opher’s consistent attempt to redress the dramatic lack of capability and analytics in current generation self-described CEP software by repositioning CEP as “intelligent event processing” (IEP) as he is continues in On Intelligent Event Processing.   

Perhaps Opher will be successful in repositioning the vast majority of the original CEP problem space as IEP.   This is a interesting slippery slope, in my opinion.   The new positioning that Opher is offering is that when “event processing” has advanced analytics, it is not CEP anymore, it becomes IEP because CEP is really “Simple Event Processing” (SEP) – event processing with little to no analytical capability.

I don’t know about most of our readers, but all this positioning and repositioning to match the capabilities, or lack of capabilities, in the current portfolio of self-described CEP software vendors is fascinating.

Here is the next logical question is:

What is the difference between a “Complex Event” and an “Intelligent Event” ?

This could get quite interesting, so stay tuned!

TIBCO Leaps Ahead in CEP with Insightful Acquisition

TIBCO Software shows, yet again, why the team in Palo Alto far outpaces the rest of the field with their announced acquisition of Insightful.  

Everyone who follows The CEP Blog and my vision for the business use of CEP understands how much energy and passion I have put into explaining why the crude time-series analysis of streaming data cannot possibly solve the vast majority of complex business problems CEP must address. 

TIBCO’s acquisition of Insightful shows just how serious TIBCO is about working to make the vision of “Predictive Business” a reality.    TIBCO means business, and a large part of what that means is helping customers solve their most challenging business integration problems, which can be summarized in CEP-speak as detecting opportunities and threats, in near real-time, as a core corporate competency. 

If you spend a few moments on the Insightful web site, you will find a treasure of documentation that discusses a gold mine of advanced statistical analytics that can be used in a number of mission critical applications.

This is the class of analytics that form the backbone of complex event processing.  In fact, as I have often pointed out (to the dismay of some of my CEP colleagues), any software company that discusses CEP and does not support or advocate advanced analytics are selling snake oil.      TIBCO obviously understands the difference between snake oil, smoke-and-mirrors marketing, and the technology it takes to solve real operational problems.

My hats off and warm congratulations to the team in Palo Alto for demonstrating, yet again, why TIBCO is committed to solving real customer problems with realistic solutions.

Maybe TIBCO will evolve to mean “The Insightful Business Company”   versus the tired and stale “The Information Bus Company” of yesteryears?

Disclaimer:  I have not been an employee of TIBCO for over a year. 

The Predictive Battlespace

Friend and colleague Don Adams, CTO World Wide Public Sector, TIBCO Software, explains how CEP can be used to sense, adapt and respond to complex situations in The “Predictive” Battlespace: Leveraging the Power of Event-Driven Architecture in Defense. 

Probabilistic Complex Event Triggering

Here is an interesting paper, Probabilistic Complex Event Triggering, Daisy Zhe Wang, Eirinaios Michelakis, and Liviu Tancau, Computer Science Division, University of California at Berkeley, circa 2005.

One of the first things I noticed about the paper was the discussion of probability in the content of complex event processing, including Hidden Markov processes, Bayesian Belief Networks, and inference models.  

The second thing I noticed was that David Luckham’s work on CEP at Stanford was not referenced anywhere in the Berkeley paper.

 

Epilogue on CEP Maturity

In On the Maturity of Complex Event Processing, the author concludes:

“I think [… the. …] comment at the end of [… the. …] post “we shouldn’t feel compelled to thwart that growth with a claim that the products are not ‘mature’ when they actually are in a lot of ways” is quite revealing. The fact that such a level of debate about CEP’s maturity is taking place, and the fact that [… someone …] is concerned that the debate might stifle growth, is itself indicative of an immature market segment in my opinion.”

This quote is compelling.  When vendors disagree with the direction and tone a debate is going and they call to end the debate, labelling the discussion “a distraction” – it tends to prove the premise of the original post Deciphering the Myths Around Complex Event Processing  by Ivy Schmerken;  the CEP market, both exciting and promising, is today, mostly immature and brittle. 

For more conclusive evidence, I turn our readers attention to this post, An Overture to the 2007 CEP Blog Awards,  That analysis was based, in part, on CEP/EP Reference Customers 2005-2007 where we documented 18 public “CEP reference clients” in 2007 (25 for the entire period 2005 – 2007).

Twenty five public reference clients over a three year period with 18 last year (2007) do not demonstrate a mature market or technology domain.

————————

Footnote:

Here were the results of the CEP/EP Reference Customers Survey for 2005-2007:

Apama	5
TIBCO	5
StreamBase	4
AptSoft  (purchased by IBM)	4
Coral8	2
Aleri	2
Agent Logic	1
BEA	1
Total CEP/EP Reference Customers (2005-2007)	25

~~~

Looking only at 2007, the total CEP/EP reference customers available in the public domain were as follows:

~~~

Apama	4
StreamBase	4
TIBCO	2
AptSoft (purchased by IBM)	2
Coral8	2
Aleri	2
Agent Logic	1
BEA	1
Total CEP/EP Reference Customers (2007)	18

More on CEP Maturity: Capability Versus Reliability

Louis Lovas of Progress Apama wrote a complimentary blog entry on the topic at hand, CEP Maturity Models.   In his post, Louis says:

“What a CEP platform has tracks independently of what it is capable of doing. ….. What CEP does, is likely what Tim is referring to when he states we’re in the Technology Trigger phase.”

Peter Lin’s comment, in reply to Louis, concurs:

“Given that COTS CEP has only been around a few years, I think it is safe to say it’s still in the early phase. If we compare it to messaging middleware, which has been around for more than 15 years, CEP isn’t as mature. Another comparison is business rule engines and expert systems. The earliest business rule engines date back to late 80’s. All things considered, I would agree with Tim. COTS CEP still has a lot of time to mature.”

Louis was spot on when he said that I was focused on overall CEP functionality; not individual product reliability.

Independent of how reliable a particular CEP-type application might appear; the overall state-of-the-art of CEP is really quite immature.