The Grammar of Complex and Intelligent Events

Folks defining CEP, and now this new term IEP, have been very passionate over the past few years that “Complex Event Processing” means the Processing of “Complex Events” not the “Complex Processing” of Events.   

Grammatically speaking, it follows that Complex is an adjective describing a noun, Event; and Processing is a verb. 

Complex events are defined by the same community as composite events, or events that are composed of two or more “contributing” events.

To be consistent, I think we should follow the same logic and grammar in the discussion of “Intelligent Event Processing”. 

It follows that Intelligent should be an adjective describing a noun, Event; and Processing is a verb.  It also follows that “Intelligent Event Processing” means the Processing of “Intelligent Events” not the “Intelligent Processing” of Events.   

This is precisely the problem that folks are creating a new CEP term, “Intelligent Event Processing” to describe processing capabilitities that are missing from the current suite of self-described CEP software products.   What people really mean to describe is the Intelligent Processing of Complex Events.   However, based on the same grammer used in defining CEP, they have created the Processing of Intelligent Events.

The use of inconsistent grammar and logic is not good for the CEP community, in my opinion.   Just because the current generation of self-described CEP vendors do not rise to the capability required by the vast majority of business event applications, we should not create new terms just to make marketing folks happy.

I think I am in a good position to speak about this, because some of my best friends work for software companies selling self-described CEP software and they have seemingly lost patience  because I refuse to support inconsistent illogical positioning and repositioning of the CEP market.

Why is the grammar between the terms “Complex Event Processing” and “Intelligent Event Processing” inconsistent?.   Folks can only spin and reposition CEP so much before all the spin, hype, and repositioning begins to catch up with the community.   

Dr. David Luckham’s original papers and single book on CEP was clear enough about CEP; and CEP covers the entire space that Opher Etzion would like to reposition as IEP.    The Grammar of Complex and Intelligent Events are, at best, misleading and inconsistent.

I think the main problem is that what Opher has been describing is the Intelligent Processing of Complex Events – however, to say this would affirm what I have been evangelizing for over two years.

A Page from Greg’s Diary: Nerwana Software

I started my career in IT many years ago and since that year have worked in enterprise IT for year and years.     Almost all of my odd career story evolves around working with end users, often advising, architecting and managing the complexity of large systems integration projects, from hands on implementation to strategic vision development.  My deep background is with Techrotech in network systems engineering.

A few years ago, years after I started my career at Techrotech, I grew a bit dismayed at enterprise software companies.   They would, for the most part, always come to us, the end users, and try to sell us large software packages.  Their sales and technical teams had very little domain knowledge of the problems they claimed they could solve – and they had little doubt that if we purchased their wares, our problems would be solved,

These software companies were keen on buzzwords and technology jargon but somewhat clueless on operational solutions or the challenges of implementation across a large federated organization with many powerful business units and “in name only” CIOs.  We often referred to these software sales guys, and their favorite systems integrators, as “drive by (or fly by) implementations” where they dump the software (and hardware) at your door and run like crazy!

So, I joined a very cool Silicon Valley company,  Nerwana Software, hoping to change all of that, or so I thought 🙂

Naturally, when I first came on board Nerwana , the entire organization, from executives to recent new hires out of school, heaped praise-upon-praise on my years of operational experience at Techrotech and elsewhere.   They cheered me on as I wrote papers and created slides on operational use cases and event processing solutions that the sales and solutions teams could take to market.   They sang my praises as I spoke to large audiences and evangelized their most innovative software and solutions.  They were pleased with the great reviews from customers.

As one would expect, I was destined to learn the face of the problems I experienced as an end-user “outsider,” now from an “insider’s” perspective. 

One of the interesting challenges that surfaced at Nerwana was the “let’s export our culture and business model to the world” mantra, maybe better referred to as “if it sells in New York, then we must sell it the same way in Tokyo or Bejing!”

Also, I really was surprised to find out how dependent Nerwana was on the opinion of analysts.   When I worked for the customers and end users, we rarely paid any special attention to the analyst’s opinions.   Sure, analysts provides a good data point, but that is all it was (or is), simply another data point.   

I soon found that software companies are often held hostage by “analyst chasing” which really was an eye opener for me, because we end-users, the people who actually buy the software, view analysts as mere mortals reading from the same foggy crystal ball as everyone else. 

Another one of the fasinating challenges I experienced at Nerwana was what some would call  “The Hero Culture.”  

I’ll elaborate on some these, hopefully interesting, observations and experiences in a future Page from Greg’s Diary.

Military Event Processing Requirements and COTS CEP Software

In Q&A from BCS SPA meeting on CEP,  friend and colleague Paul Vincent says:

 “AFAIK there are no current military systems (as opposed to government intelligence systems) using Commercial Off The Shelf CEP systems, although I recall one commercial product being developed with US military money (your tax $ at work, etc etc).”

Actually, Paul’s statement is slightly misleading.   Companies like StreamBase and AgentLogic have their roots in supporting the military.  In addition, IBM has a number of event processing related solutions in the military.   (There are also others, we suspect.)

It is true, however, that current generation COTS CEP engines do not have the advanced event processing capabilities required for most CEP applications  in the military; but as CEP engines advance, this should change.

Please Welcome Dr. Rainer von Ammon to The CEP Blog

Today is an especially joyful occasion on The CEP Blog.    I am pleased to announce that one of the world’s top experts on CEP, Dr. Rainer von Ammon, has joined the blog.

Dr. Rainer von Ammon is managing director of the Centrum für Informations-Technology Transfer (CITT) in Regensburg. Until October 2005 he was Professor for Software Engineering, specializing in E-Business infrastructures and distributed systems, at the University of Applied Sciences Upper Austria. Rainer is still teaching there and at the University of Applied Sciences of Regensburg. From 1998 to 2002, he worked as Principal Consultant and Manager for R+D Cooperations at BEA Systems (Central and Eastern Europe). Prior to this, he was Professor for Software Engineering in Dresden with a focus on development of applications with event driven object oriented user interfaces and component based application development. Before this Rainer was acting as manager of the field Basic Systems at the Mummert + Partner Unternehmensberatung, Hamburg. After finishing his studies of Information Sciences at the University of Regensburg, he started as project leader of Computer Based Office Systems (COBIS) from 1978 to 1983 and afterward founded a start up company with some of his colleagues.

Some of you may recall my recent musings, A Bitter Pill To Swallow: First Generation CEP Software Needs To Evolve.   When you read Rainer’s excellent reply, you will quickly see why we are very pleased to have his thought leadership here at The CEP Blog.  Dr. von Ammon and his team are leading experts in CEP and related business integration domains.  Not only does he provide thought leadership, his team  researches, develops, implements and tests CEP solutions.   

In another example of  his thought leadership, some of you might recall this post, Brandl and Guschakowski Deliver Excellent CEP/BAM Report, where Hans-Martin Brandl and David Guschakowski of the University of Applied Sciences Regensburg, Faculty of Information Technology/Mathematics, advised by Dr. von Ammon, completed an excellent CEP thesis, Complex Event Processing in the context of Business Activity Monitoring. 

Please join me in extending a warm welcome for Dr. Rainer von Ammon to The CEP Blog.

A Bitter Pill To Swallow: First Generation CEP Software Needs To Evolve

Frankly speaking, the CEP market is now saturated with hype about all the great things CEP can do, detecting opportunities and threats in real time and supporting the decision cycle.  However, in my opinion, it is time for the software vendors and analysts to move beyond the marketing hype and demonstrate real operational value with strong end user success, something seriously lacking today.

I have advocated this evolution for two years, including the notion of expanding CEP capabilities with proven techniques for event processing that have worked well long before current “Not yet CEP but called CEP” software hit the marketplace and airwaves.

For example, in my first CEP/EP presentation in New York in 1Q 2006, I presented Processing Patterns for Predictive Business and talked about how the US military has implemented high performance detection-oriented systems for many years (in the art-and-science of multisensor data fusion, MSDF), and how every day, when we sit at home (or at work or in transit), we are comforted to know we are safe from missile attacks because of what I would also call “complex event processing.”   There is a very rich history of “CEP but not called CEP” behind the scenes keeping people safe and warm. (The same thing can be said with many similar examples of complex event processing in use today, but not called “CEP” by CEP software vendors.)

This is one reason, when I read the “CEP history lessons,” I am amused at how, at times, the lessons appear self-serving, not end user serving.  There is so much rich event processing history and proven architectures in “CEP but not called CEP” (CEP that actually works, in practice everyday, long before it was called CEP).  It continues to puzzle me that a few people the CEP/EP community continue to take the “we invented EP” view.  Quite frankly, the history we read is missing most, if not all, of the history and practice of MSDF.

When we take the current CEP COTS software offerings and apply it to these working “CEP but not called CEP” applications, the folks with real operational “CEP but not called CEP” detection-oriented experience quickly cut through the hype because they are, based on their state-of-the-practice, now seeking self-learning, self-healing “real CEP type” systems.  They are not so excited about first generation technologies full of promises from software vendors with only a few years of experience in solving detection-oriented problems and very few real success stories.

The same is true for advanced fraud detection and other state-of-the-art detection-oriented processing of “complex events” and situations.  The state-of-the-art of complex event processing, in practice, is far beyond the first generation CEP engines on the market today. 

This is one of the reasons I have agreed with the IBM folks who are calling these first generation “CEP orchestration engines” BEP engines, because that view is closer to fact than fiction.  Frankly speaking again, process orchestration is much easier than complex detection with high situation detection confidence and also low false alarms.

Customers who are detection-savvy also know this, and I have blogged about a few of these meetings and customer concerns.  For example, please read my blog entry about a banker who was very sceptical in a recent wealth management conference in Bangkok.  I see this reaction all the time, in practice. 

Complex problems are not new and they still cry out for solutions.  Furthermore, many current-generation event processing solutions are already more advanced that the first generation CEP engines on the “call it CEP” market today.  This is a very real inhibitor, in my opinion, to growth in the “call it CEP” software space today – and credibility may ultimately be “at risk.”  Caution is advised.

Candidly speaking again, there are too many red-herring CEP-related discussions and not enough solid results given the time software vendors have been promoting CEP/EP (again, this is simply my opinion).  The market is in danger of eventually losing credibility, at least in the circles I travel and complex problems I enjoy solving, because the capabilities of the (so called) CEP technologies by software vendors in the (so called) CEP space have been over sold; and, frankly speaking, I have yet to see tangible proof of “real CEP capabilities” in the road maps and plans of the current CEP software vendors.  This is dissappointing.

This pill is bitter and difficult to swallow, but most of my life’s work has been advising, formulating and architecting real-time solutions for the end user (the C-level executives and the operational experts with the complex problems to solve).   CEP software must evolve and there needs to be more tangible results, not more marketing hype.

End Users Should Define the CEP Market.

My friend Opher mistakenly thought I was thinking of him when I related the story of the fish, as he replied, CEP and the Story of the Captured Traveller.

I must not have related the fish story very well, because to understood the story of the fish, is to know that we are all like the fish, in certain aspects of life, and there is nothing negative to be gleaned from the story.

However, to Opher’s point on CEP, I disagree.   Just because the marketing people (not the market) has misdefined CEP and therefore the vendors are drifting from the technology described in Dr. Luckham’s original CEP work, including his CEP book, we should not change the context of CEP.    Therefore, I don’t agree we should redefine CEP, as David envisioned, as Intelligent Event Processing (IEP) because CEP, as today’s software vendors sell it, is really SEP (or whatever!)  Please recall that David’s background at Stanford was AI and he did not define CEP as the software vendors have defined it either!

The fact of the matter is that the software marketing folks have decided they are going to use Dr. Luckham’s book to sell software that does not perform as Dr. Luckham described or envisioned!   I make no apologies for being on the side of end users who actually need to solve complex problems, not sell software that underperforms.

As I mentioned, this positioning and repositioning does not help solve complex problems.   At the end of the day, we have problems to solve and the software community is not very helpful when they place form over substance, consistently. 

Furthermore, as most customers are saying, time and time again, “so what?” … “these COTS event processing platforms with simple joins, selects and rules do not solve my complex event processing problems.”  “We already have similar approaches, where we have spent millions of dollars, and they do not work well.”

In other words, the market is crying out for true COTS CEP solutions, but the software community is not yet delivering.  OBTW, this is nothing new.  In my first briefing to the EP community in January of 2006, I mentioned that CEP required stating the business problem, or domain problem, and then selecting the method or methods that best solve the problem or problems.

To date, the CEP community has not done this because they have no COTS tool set other than SEP engines (marketed as either ESP engines or CEP engines – and at least ESP was closer to being technically accurate.) 

Experienced end users are very intelligent. 

These end users know the complex event processing problems they need to solve; and they know the limitations of the current COTS approaches marketed by the CEP community.  Even in Thailand, a country many of you might mistakenly think is not very advanced technologically, there are experts in telecommunications (who run large networks) who are working on very difficult fraud detection applications, and they use neural networks and say the results are very good.   However, there is not one CEP vendor, that I know of, who offers true CEP capability in the form of neural nets.  

Almost every major bank, telco, etc. has the same opinion, and the same problem. They need much more capability than streaming joins, selects and rules to solve their complex event processing problems that Dr. Luckham outlined in his book.   The software vendors are attempting to define the CEP market to match their capability; unfortunately, their capabilities do not meet the requirements of the vast majority of end users who have CEP problems to solve.

If the current CEP platforms were truely solving complex event processing problems, annual sales would be orders of magnitudes higher.  Hence, the users have already voted.   The problem is that the CEP community is not listening.

Coral8 Developer’s License

I agree with Marc Adler regarding Coral8 and Transparency.   

Marc discusses the fact that Coral8 encourages you download their full development environment without any requirement to get a license key.   A Coral8 license is only needed when you move into production.  This is really great for kicking the tires and looking under the hood.

There is something I wish Coral8 would change, however.

Coral8’s development license, as I read it, is only valid for a single CPU machine.   However, most folks today run multiple CPUs even in their development environment, especially if they are running Linux.  If you want to do any development on a multiple CPU machine you have to contact Coral8.

In my opinion, it would be good if Coral8 would open their “easy” development license up to multiple CPU Linux servers since most of us run multiple CPU Linux servers even in the development environment.