On Elephants and Analytics

In On EP and Analytics, good friend and respected colleague Opher Etzion applies the well known metaphor of the big elephant to describe how, if you are observing certain specific domains of a subject, like fraud detection, then your view of the whole elephant is biased by your lack of perspective of the entire big elephant.

I am pleased that dear Opher continues to use this metaphor in counterpoint because the same metaphor can be used to describe the carefully selected group of vendors that have banded together to called themselves CEP Vendors.  This group, many founding members of the EPTS, have formed a merry band of well-intended event processing “specialists” and the same lovely elephant causes this group of bonded colleagues to make elephant-blinded statements, as Opher has made in his quoted post:

“Currently most CEP applications do not require analytics.” 

The reason, I believe, that Opher makes the statement above is because the group of software vendors calling themselves “CEP vendors” represent a very small part of the overall event processing elephant;  and hence, since these self-described CEP applications appear to require very little or no analytics, then, by the same logic, CEP requires no analytics. 

(I should outline the boolean logic in a future post!)

For example, one friend and colleague in Thailand is the CTO of True Internet, a leading telecommunications, voice, Video and Internet service provider in Thailand.   True processes myriad events on their network using a dynamic, self-learning neural networking technology.    The US company providing this very clever and highly recommended event processing application does not call themselves a “CEP vendor”; however, they process complex events better and more interesting than the band of merry self-described “CEP players”.

Again,  visualize the gentle giant elephant metaphor that Opher likes to use as a basis for his comments in CEP counterpoint.

When folks define the term “complex event processing” to match a technology marketing campaign that is primarily driven by software running rules against time-series data streaming in a sliding-time windows, and then go on to take the same software capabilities and apply these capabilities to problems that are suitable for that domain, then you match Opher’s elegant description of “a small view of the overall elephant”.

The fact of the matter is that the overall domain of event processing is at least two orders of magnitude larger (maybe more) than the combined annual revenue of the self-described companies marketing what they call “CEP engines.”  The very large “rest of the big elephant” is doing what is also “complex event processing” in everyday operations that are somehow overlooked in “other” analysis and counterplay.

Therefore,  I kindly remain unmoved from my view  that the self-described CEP community, as currently organized, is not immune to counterpoint using the same gentle giant elephant metaphor.  I like this metaphor and hope well-respected colleagues will continue to use this metaphor; because we can easily apply this elegant manner of discussion to explain why the current group of self-described CEP vendors are, in a manner of speaking, selling Capital Market Snake Oil because they are making outrageous claims about the capabilities of their products, as if they can solve the entire “elephant” of event processing problems.   Recently, in this article, CEP was positioned as a technology to mitigate against corporate megadisasters like the subprime meltdown.

Advice:  Tone down the hype.

Furthermore, the noise in the counter arguments marginalize most of the real event processing challenges faced by customers.

In consistant and well respected rebuttal, Opher likes to use the “glass half-full, half-empty” metaphor.   Opher’s point is a valid attempt to paint my operational realism as “half empty” negativism; while at the same time positioning the promotion of the (narrow) event processing capabilities of the self-described CEP rules community as “half-full” thinking. 

For the record, I do see my worldview as “half full” or “half empty”; but an unbiased pragmatic view based on day-to-day interaction with customers with what they would call “complex event processing” problems. 

These same customers would fall over laughing if we tried to bolt one of these rule-based, time-series streaming data processing engines on their network and told them they can detect anything other than trival business events, business opportunities and threats, in near real-time. 

Is it “half empty” thinking to caution people that a “glass” of software that is being touted as the answer to a wide range of complex (even going so far in a recent news article to imply CEP would have magically stopped the subprime crisis!) tangible business problems is not really as that it is hyped to be?  

If so, then I plead guilty to honesty and realism, with the added offense of a sense of fiscal responsibility to customers and end users.

Congrats to Coral8 and Marc Adler at Citigroup

In Coral8 is Our Choice or “How the Hell Did We Get Here?”, Marc Adler does his normal (and now expected) fantastic job of cutting past the CEP marketing hype and getting to the meat of the issues, from an actual user’s perspective.  Marc is spot on in his evaluation of the various so-called CEP vendors.   I highly recommend you read Marc’s post above.

The bottom line, today, is that CEP software products have a long way to go to live up to the current CEP hype and none are really doing what we would call “CEP”.   So, in the current market, the intangibles, as Marc points out, are critically important.  

Coral8 has recently demonstrated to the event processing community that they are above-and-beyond the competition in that category. 

Coral8 has an open software evaluation and licensing model, one you would expect in the year 2003-3005 (this is 2008). 

Coral8 has significant white papers, thought leadership papers and documentation, all freely and readily available. 

Coral8 is standing by to support you in your event processing efforts, from Marc at the big and powerful Citigroup (be careful of your subprime portfolio) to consultants in Asia (be careful of mosquitos), you can count on Coral8’s leadership to support you.

As Marc keenly pointed out, it is not the final imaginary number in low latency that is important; nor is it important that you call yourself the “top leader” and the “creator of the standards” that makes you important; nor is it how innovative or smart you are (or think you are).  What is important is your customer service model.

Coral8 has demonstrated to many of us that they take the customer service model very seriously and this is the reason that Coral8 has caught our attention in the past 6 months.

Congrats to both Coral8 and Marc.   We look forward to hearing more about the results of your teamwork and event processing solutions at Citigroup.

 

A Page from Greg’s Diary: Nerwana Software

I started my career in IT many years ago and since that year have worked in enterprise IT for year and years.     Almost all of my odd career story evolves around working with end users, often advising, architecting and managing the complexity of large systems integration projects, from hands on implementation to strategic vision development.  My deep background is with Techrotech in network systems engineering.

A few years ago, years after I started my career at Techrotech, I grew a bit dismayed at enterprise software companies.   They would, for the most part, always come to us, the end users, and try to sell us large software packages.  Their sales and technical teams had very little domain knowledge of the problems they claimed they could solve – and they had little doubt that if we purchased their wares, our problems would be solved,

These software companies were keen on buzzwords and technology jargon but somewhat clueless on operational solutions or the challenges of implementation across a large federated organization with many powerful business units and “in name only” CIOs.  We often referred to these software sales guys, and their favorite systems integrators, as “drive by (or fly by) implementations” where they dump the software (and hardware) at your door and run like crazy!

So, I joined a very cool Silicon Valley company,  Nerwana Software, hoping to change all of that, or so I thought 🙂

Naturally, when I first came on board Nerwana , the entire organization, from executives to recent new hires out of school, heaped praise-upon-praise on my years of operational experience at Techrotech and elsewhere.   They cheered me on as I wrote papers and created slides on operational use cases and event processing solutions that the sales and solutions teams could take to market.   They sang my praises as I spoke to large audiences and evangelized their most innovative software and solutions.  They were pleased with the great reviews from customers.

As one would expect, I was destined to learn the face of the problems I experienced as an end-user “outsider,” now from an “insider’s” perspective. 

One of the interesting challenges that surfaced at Nerwana was the “let’s export our culture and business model to the world” mantra, maybe better referred to as “if it sells in New York, then we must sell it the same way in Tokyo or Bejing!”

Also, I really was surprised to find out how dependent Nerwana was on the opinion of analysts.   When I worked for the customers and end users, we rarely paid any special attention to the analyst’s opinions.   Sure, analysts provides a good data point, but that is all it was (or is), simply another data point.   

I soon found that software companies are often held hostage by “analyst chasing” which really was an eye opener for me, because we end-users, the people who actually buy the software, view analysts as mere mortals reading from the same foggy crystal ball as everyone else. 

Another one of the fasinating challenges I experienced at Nerwana was what some would call  “The Hero Culture.”  

I’ll elaborate on some these, hopefully interesting, observations and experiences in a future Page from Greg’s Diary.

Please Welcome Dr. Rainer von Ammon to The CEP Blog

Today is an especially joyful occasion on The CEP Blog.    I am pleased to announce that one of the world’s top experts on CEP, Dr. Rainer von Ammon, has joined the blog.

Dr. Rainer von Ammon is managing director of the Centrum für Informations-Technology Transfer (CITT) in Regensburg. Until October 2005 he was Professor for Software Engineering, specializing in E-Business infrastructures and distributed systems, at the University of Applied Sciences Upper Austria. Rainer is still teaching there and at the University of Applied Sciences of Regensburg. From 1998 to 2002, he worked as Principal Consultant and Manager for R+D Cooperations at BEA Systems (Central and Eastern Europe). Prior to this, he was Professor for Software Engineering in Dresden with a focus on development of applications with event driven object oriented user interfaces and component based application development. Before this Rainer was acting as manager of the field Basic Systems at the Mummert + Partner Unternehmensberatung, Hamburg. After finishing his studies of Information Sciences at the University of Regensburg, he started as project leader of Computer Based Office Systems (COBIS) from 1978 to 1983 and afterward founded a start up company with some of his colleagues.

Some of you may recall my recent musings, A Bitter Pill To Swallow: First Generation CEP Software Needs To Evolve.   When you read Rainer’s excellent reply, you will quickly see why we are very pleased to have his thought leadership here at The CEP Blog.  Dr. von Ammon and his team are leading experts in CEP and related business integration domains.  Not only does he provide thought leadership, his team  researches, develops, implements and tests CEP solutions.   

In another example of  his thought leadership, some of you might recall this post, Brandl and Guschakowski Deliver Excellent CEP/BAM Report, where Hans-Martin Brandl and David Guschakowski of the University of Applied Sciences Regensburg, Faculty of Information Technology/Mathematics, advised by Dr. von Ammon, completed an excellent CEP thesis, Complex Event Processing in the context of Business Activity Monitoring. 

Please join me in extending a warm welcome for Dr. Rainer von Ammon to The CEP Blog.

A Bitter Pill To Swallow: First Generation CEP Software Needs To Evolve

Frankly speaking, the CEP market is now saturated with hype about all the great things CEP can do, detecting opportunities and threats in real time and supporting the decision cycle.  However, in my opinion, it is time for the software vendors and analysts to move beyond the marketing hype and demonstrate real operational value with strong end user success, something seriously lacking today.

I have advocated this evolution for two years, including the notion of expanding CEP capabilities with proven techniques for event processing that have worked well long before current “Not yet CEP but called CEP” software hit the marketplace and airwaves.

For example, in my first CEP/EP presentation in New York in 1Q 2006, I presented Processing Patterns for Predictive Business and talked about how the US military has implemented high performance detection-oriented systems for many years (in the art-and-science of multisensor data fusion, MSDF), and how every day, when we sit at home (or at work or in transit), we are comforted to know we are safe from missile attacks because of what I would also call “complex event processing.”   There is a very rich history of “CEP but not called CEP” behind the scenes keeping people safe and warm. (The same thing can be said with many similar examples of complex event processing in use today, but not called “CEP” by CEP software vendors.)

This is one reason, when I read the “CEP history lessons,” I am amused at how, at times, the lessons appear self-serving, not end user serving.  There is so much rich event processing history and proven architectures in “CEP but not called CEP” (CEP that actually works, in practice everyday, long before it was called CEP).  It continues to puzzle me that a few people the CEP/EP community continue to take the “we invented EP” view.  Quite frankly, the history we read is missing most, if not all, of the history and practice of MSDF.

When we take the current CEP COTS software offerings and apply it to these working “CEP but not called CEP” applications, the folks with real operational “CEP but not called CEP” detection-oriented experience quickly cut through the hype because they are, based on their state-of-the-practice, now seeking self-learning, self-healing “real CEP type” systems.  They are not so excited about first generation technologies full of promises from software vendors with only a few years of experience in solving detection-oriented problems and very few real success stories.

The same is true for advanced fraud detection and other state-of-the-art detection-oriented processing of “complex events” and situations.  The state-of-the-art of complex event processing, in practice, is far beyond the first generation CEP engines on the market today. 

This is one of the reasons I have agreed with the IBM folks who are calling these first generation “CEP orchestration engines” BEP engines, because that view is closer to fact than fiction.  Frankly speaking again, process orchestration is much easier than complex detection with high situation detection confidence and also low false alarms.

Customers who are detection-savvy also know this, and I have blogged about a few of these meetings and customer concerns.  For example, please read my blog entry about a banker who was very sceptical in a recent wealth management conference in Bangkok.  I see this reaction all the time, in practice. 

Complex problems are not new and they still cry out for solutions.  Furthermore, many current-generation event processing solutions are already more advanced that the first generation CEP engines on the “call it CEP” market today.  This is a very real inhibitor, in my opinion, to growth in the “call it CEP” software space today – and credibility may ultimately be “at risk.”  Caution is advised.

Candidly speaking again, there are too many red-herring CEP-related discussions and not enough solid results given the time software vendors have been promoting CEP/EP (again, this is simply my opinion).  The market is in danger of eventually losing credibility, at least in the circles I travel and complex problems I enjoy solving, because the capabilities of the (so called) CEP technologies by software vendors in the (so called) CEP space have been over sold; and, frankly speaking, I have yet to see tangible proof of “real CEP capabilities” in the road maps and plans of the current CEP software vendors.  This is dissappointing.

This pill is bitter and difficult to swallow, but most of my life’s work has been advising, formulating and architecting real-time solutions for the end user (the C-level executives and the operational experts with the complex problems to solve).   CEP software must evolve and there needs to be more tangible results, not more marketing hype.

Orthogonal Blogging at the SOA Horse Races

Dear friend Opher Etzion responds to my post Betting on the SOA Horse with a discussion on how SOA, EDA and CEP are technically orthogonal, concluding:

“Event Processing can have different interactions with SOA, and when IBM’s announcements in this area will be available you’ll realize that there are different entry points. Event processing can also work in legacy and non-SOA environment.”

Richard Veryard, who also kindly reads my blog (and Opher’s blog) replies with Technological Perfecta where he opines,

“I think there are some mutual dependencies between these technologies, but they are what I call soft dependencies.”

Opher, Richard, you guys are technically right, but you are blogging orthogonally to the message in Betting on the SOA Horse.

First of all, my post was not a technical discussion, it was a discussion about business, marketing, timing positioning and the software industry in general. Therefore, it is a bit humorously orthogonal to reply to a marketing metaphor about investments, competition, software postioning and horse racing with architectual posts about technology and how they are related or interdependent.

In a nutshell, here is why….

Candidly speaking, despite what many analysts want you to believe, end users rarely build “SOAs” “EDAs” or CEPs”. End users have IT budgets to solve business problems with the most cost effective technology they can find; and they do not care (if they have a clue) what cute three letter acronyms have been created by analysts to describe momentum in the software market. Sorry, it is true really.

For example, I remember when I was in Tokyo where the very capable and conservatively risk adverse Japanese executives told me time and time again, “We don’t care about SOA we simply want to integrate our systems.” They were quick to remind me, “You guys in America must realize we don’t care what the western analysts, supported by software companies, say. They have a conflict-of-interest anyway and they are not end users. What we care about are mature technologies with solid reference clients and proven implementations.”

By the way, this is one reason I admire Japanese business so much. They are not impressed with handwaving hyperbole. They just want to see results. In other words, “Prove it, don’t just say it.” The devil is in the details, as they say. The Japanese are highly skillful at cutting through the smoke-and-mirrors. I think this is one reason the Japanese are among the leaders in so many industry sectors, but that is a blog story for another day.

To this point, if you are in front of customers and you are pushing SOA because your software company has “bet the farm” on positioning themselves as an SOA company, you are making a mistake. Three letter acronyms and technology jargon do not solve business problems. In fact, for the most part, they are a red-herring. The same is true of EDA and CEP. This was the main message in my post Betting on the SOA Horse.

How do I make such a statement?

Because for over 20 years I have worked as a consultant working on the opposite side of the table of hungry software vendors who come into our house (organization) tossing out buzzwords, acronyms, and jargon. My job was solving real business problems, not selling software. We used to wonder when all the scrabble and babble the software companies were tossing at us was going to turn into a business language that solves real business problems easily, rapidly and economically. That day never came.

Then, I made a conscious decision to take a break from a long career of consulting to get an insiders perspective on, and perhaps even transform, the software industry (What was I thinking?). This experience, working for a software company, was an eye-opener.  

Candidly speaking again, some software companies tend to live in “La La Land”.

They create go-to-market strategies based on jargon, buzzwords and three letter acronyms that have little to do with understanding their customer’s business problems, risks, and culture. They spin, position and reposition in a market of smoke-and-mirrors happy to sell you a gold disk of “the-answers-to-all-your-problems.”  Then, they leave you the gold disk, and your business problem, as they drive away, looking at your business in the rear view mirror as they count the revenue from their victorious campaign.

These same companies bet on jargon like SOA, EDA, CEP, BAM and they hedge their bets with different combinations of the above, the theme of my post Betting on the SOA Horse, which was not a technology, architectural nor implementation discussion, in any way.

Is it any real wonder why SOA has become, for the most part, complex, vendor-driven jargon barely making a dent in the practical real-world, whereas social-networking and other grass-roots user-driven technologies, most without trendy jargon, has left SOA in the dust for the past few years?

Coral8: Event Stream Processing and Intrusion Detection

Not quite ready for prime-time, we have been testing our home-grown UNIX domain socket adapter using Coral8 Java APIs.   We are using this adapter to evaluate and demonstrate stream processing with intrusion detection systems (IDS) using event stream processing to reduce false alarms, detect derived situations from the raw intrusion event data, and feed a security management visualization dashboard.

You can click on the teaser image below to see more of our first IDS screenshots from Coral8’s Studio stream visualization tool.

If you click on the image above, you will four additional event stream properties.  For this part of the demo, there are 14 total IDS properties in the event stream, but we only show 5 properties in this cropped screen capture.

I am quite sure that we could do similar integration with other event stream processing engines, but fortunately Coral8 makes it easy to download, start developing and testing.