The Infant, the Elephant and the Intelligent Event

Fellow blogger Opher Etzion, replies to  On Elephants and Analytics with On Unicorn, Professor and Infant.   Opher is kindly giving us another metaphor to consider, the Infant and the Profession, since we are both big fans of big gentle elephants, babies and our universities.  

Opher and I agree that Infants are not Professors, and we also agree that CEP is in its Infancy and there is overhype by folks often implying CEP is a Professor.     So it seems we all have a huge elephant in the room with an Infant Professor hanging on the end of a wildly swinging Elephant’s trunk!

To keep the blogopoints interesting, I should point out that with all this agreement and Kumbaya campfire singing, there are a couple of things I do disagree with in Opher’s amusing counterpoint. 

First of all, Opher uses the well know debate technique of falsely attributing some easily refutable discussion point and then offering a slam dunk counterpoint.   He does this in this clever, but completely inaccurate Opher quote,

 “I [Opher] respectfully disagree with Tim … in his claim that what has been done until today is just hype and hence totally worthless…”

Folks reading my blog know that I have never said “what has been done until today is … totally worthless.”    This is a misfortunate misquote.  Shame on you Opher!  

What I said, easily read in the blog, was that CEP is overhyped and that most of the self-described CEP software on the market today does not live up to the inflated claims we read and hear from CEP software vendors, the analysts and reporters they influence.

The second counterpoint that I find interesting is Opher’s consistent attempt to redress the dramatic lack of capability and analytics in current generation self-described CEP software by repositioning CEP as “intelligent event processing” (IEP) as he is continues in On Intelligent Event Processing.   

Perhaps Opher will be successful in repositioning the vast majority of the original CEP problem space as IEP.   This is a interesting slippery slope, in my opinion.   The new positioning that Opher is offering is that when “event processing” has advanced analytics, it is not CEP anymore, it becomes IEP because CEP is really “Simple Event Processing” (SEP) – event processing with little to no analytical capability.

I don’t know about most of our readers, but all this positioning and repositioning to match the capabilities, or lack of capabilities, in the current portfolio of self-described CEP software vendors is fascinating.

Here is the next logical question is:

What is the difference between a “Complex Event” and an “Intelligent Event” ?

This could get quite interesting, so stay tuned!

TIBCO Leaps Ahead in CEP with Insightful Acquisition

TIBCO Software shows, yet again, why the team in Palo Alto far outpaces the rest of the field with their announced acquisition of Insightful.  

Everyone who follows The CEP Blog and my vision for the business use of CEP understands how much energy and passion I have put into explaining why the crude time-series analysis of streaming data cannot possibly solve the vast majority of complex business problems CEP must address. 

TIBCO’s acquisition of Insightful shows just how serious TIBCO is about working to make the vision of “Predictive Business” a reality.    TIBCO means business, and a large part of what that means is helping customers solve their most challenging business integration problems, which can be summarized in CEP-speak as detecting opportunities and threats, in near real-time, as a core corporate competency. 

If you spend a few moments on the Insightful web site, you will find a treasure of documentation that discusses a gold mine of advanced statistical analytics that can be used in a number of mission critical applications.

This is the class of analytics that form the backbone of complex event processing.  In fact, as I have often pointed out (to the dismay of some of my CEP colleagues), any software company that discusses CEP and does not support or advocate advanced analytics are selling snake oil.      TIBCO obviously understands the difference between snake oil, smoke-and-mirrors marketing, and the technology it takes to solve real operational problems.

My hats off and warm congratulations to the team in Palo Alto for demonstrating, yet again, why TIBCO is committed to solving real customer problems with realistic solutions.

Maybe TIBCO will evolve to mean “The Insightful Business Company”   versus the tired and stale “The Information Bus Company” of yesteryears?

Disclaimer:  I have not been an employee of TIBCO for over a year. 

Probabilistic Complex Event Triggering

Here is an interesting paper, Probabilistic Complex Event Triggering, Daisy Zhe Wang, Eirinaios Michelakis, and Liviu Tancau, Computer Science Division, University of California at Berkeley, circa 2005.

One of the first things I noticed about the paper was the discussion of probability in the content of complex event processing, including Hidden Markov processes, Bayesian Belief Networks, and inference models.  

The second thing I noticed was that David Luckham’s work on CEP at Stanford was not referenced anywhere in the Berkeley paper.

 

Scheduling Agents with Rules Engines

Paul Vincent of TIBCO talks about agents in his post, CEP and Agents…

At the core, TIBCO’s BusinessEvents is RETE-based rules engine and rules engines are well suited for scheduling problems.  This makes perfect sense, since many of TIBCO’s customers deploy BusinessEvents in scheduling-oriented, not detection-oriented, solutions.

It begs to be pointed out, however, that scheduling is only one component of a CEP architecture. 

Normally, the scheduling component of a distributed event processing architecture manages the intelligent scheduling of the sharing of data between distributed agents that are running a variety of analytics.

Simply stated, all agents are not rules engines; however, rules engines are often used to schedule the cooperation between analytical agents in a distributed agent-based architecture.

Military Event Processing Requirements and COTS CEP Software

In Q&A from BCS SPA meeting on CEP,  friend and colleague Paul Vincent says:

 “AFAIK there are no current military systems (as opposed to government intelligence systems) using Commercial Off The Shelf CEP systems, although I recall one commercial product being developed with US military money (your tax $ at work, etc etc).”

Actually, Paul’s statement is slightly misleading.   Companies like StreamBase and AgentLogic have their roots in supporting the military.  In addition, IBM has a number of event processing related solutions in the military.   (There are also others, we suspect.)

It is true, however, that current generation COTS CEP engines do not have the advanced event processing capabilities required for most CEP applications  in the military; but as CEP engines advance, this should change.

Please Welcome Dr. Rainer von Ammon to The CEP Blog

Today is an especially joyful occasion on The CEP Blog.    I am pleased to announce that one of the world’s top experts on CEP, Dr. Rainer von Ammon, has joined the blog.

Dr. Rainer von Ammon is managing director of the Centrum für Informations-Technology Transfer (CITT) in Regensburg. Until October 2005 he was Professor for Software Engineering, specializing in E-Business infrastructures and distributed systems, at the University of Applied Sciences Upper Austria. Rainer is still teaching there and at the University of Applied Sciences of Regensburg. From 1998 to 2002, he worked as Principal Consultant and Manager for R+D Cooperations at BEA Systems (Central and Eastern Europe). Prior to this, he was Professor for Software Engineering in Dresden with a focus on development of applications with event driven object oriented user interfaces and component based application development. Before this Rainer was acting as manager of the field Basic Systems at the Mummert + Partner Unternehmensberatung, Hamburg. After finishing his studies of Information Sciences at the University of Regensburg, he started as project leader of Computer Based Office Systems (COBIS) from 1978 to 1983 and afterward founded a start up company with some of his colleagues.

Some of you may recall my recent musings, A Bitter Pill To Swallow: First Generation CEP Software Needs To Evolve.   When you read Rainer’s excellent reply, you will quickly see why we are very pleased to have his thought leadership here at The CEP Blog.  Dr. von Ammon and his team are leading experts in CEP and related business integration domains.  Not only does he provide thought leadership, his team  researches, develops, implements and tests CEP solutions.   

In another example of  his thought leadership, some of you might recall this post, Brandl and Guschakowski Deliver Excellent CEP/BAM Report, where Hans-Martin Brandl and David Guschakowski of the University of Applied Sciences Regensburg, Faculty of Information Technology/Mathematics, advised by Dr. von Ammon, completed an excellent CEP thesis, Complex Event Processing in the context of Business Activity Monitoring. 

Please join me in extending a warm welcome for Dr. Rainer von Ammon to The CEP Blog.

A Bitter Pill To Swallow: First Generation CEP Software Needs To Evolve

Frankly speaking, the CEP market is now saturated with hype about all the great things CEP can do, detecting opportunities and threats in real time and supporting the decision cycle.  However, in my opinion, it is time for the software vendors and analysts to move beyond the marketing hype and demonstrate real operational value with strong end user success, something seriously lacking today.

I have advocated this evolution for two years, including the notion of expanding CEP capabilities with proven techniques for event processing that have worked well long before current “Not yet CEP but called CEP” software hit the marketplace and airwaves.

For example, in my first CEP/EP presentation in New York in 1Q 2006, I presented Processing Patterns for Predictive Business and talked about how the US military has implemented high performance detection-oriented systems for many years (in the art-and-science of multisensor data fusion, MSDF), and how every day, when we sit at home (or at work or in transit), we are comforted to know we are safe from missile attacks because of what I would also call “complex event processing.”   There is a very rich history of “CEP but not called CEP” behind the scenes keeping people safe and warm. (The same thing can be said with many similar examples of complex event processing in use today, but not called “CEP” by CEP software vendors.)

This is one reason, when I read the “CEP history lessons,” I am amused at how, at times, the lessons appear self-serving, not end user serving.  There is so much rich event processing history and proven architectures in “CEP but not called CEP” (CEP that actually works, in practice everyday, long before it was called CEP).  It continues to puzzle me that a few people the CEP/EP community continue to take the “we invented EP” view.  Quite frankly, the history we read is missing most, if not all, of the history and practice of MSDF.

When we take the current CEP COTS software offerings and apply it to these working “CEP but not called CEP” applications, the folks with real operational “CEP but not called CEP” detection-oriented experience quickly cut through the hype because they are, based on their state-of-the-practice, now seeking self-learning, self-healing “real CEP type” systems.  They are not so excited about first generation technologies full of promises from software vendors with only a few years of experience in solving detection-oriented problems and very few real success stories.

The same is true for advanced fraud detection and other state-of-the-art detection-oriented processing of “complex events” and situations.  The state-of-the-art of complex event processing, in practice, is far beyond the first generation CEP engines on the market today. 

This is one of the reasons I have agreed with the IBM folks who are calling these first generation “CEP orchestration engines” BEP engines, because that view is closer to fact than fiction.  Frankly speaking again, process orchestration is much easier than complex detection with high situation detection confidence and also low false alarms.

Customers who are detection-savvy also know this, and I have blogged about a few of these meetings and customer concerns.  For example, please read my blog entry about a banker who was very sceptical in a recent wealth management conference in Bangkok.  I see this reaction all the time, in practice. 

Complex problems are not new and they still cry out for solutions.  Furthermore, many current-generation event processing solutions are already more advanced that the first generation CEP engines on the “call it CEP” market today.  This is a very real inhibitor, in my opinion, to growth in the “call it CEP” software space today – and credibility may ultimately be “at risk.”  Caution is advised.

Candidly speaking again, there are too many red-herring CEP-related discussions and not enough solid results given the time software vendors have been promoting CEP/EP (again, this is simply my opinion).  The market is in danger of eventually losing credibility, at least in the circles I travel and complex problems I enjoy solving, because the capabilities of the (so called) CEP technologies by software vendors in the (so called) CEP space have been over sold; and, frankly speaking, I have yet to see tangible proof of “real CEP capabilities” in the road maps and plans of the current CEP software vendors.  This is dissappointing.

This pill is bitter and difficult to swallow, but most of my life’s work has been advising, formulating and architecting real-time solutions for the end user (the C-level executives and the operational experts with the complex problems to solve).   CEP software must evolve and there needs to be more tangible results, not more marketing hype.