Capital Market CEP Fantasy Land

In Tech Spending Hit by Subprime Mess, Jeffery Schwartz says,

“According to Tabb, spending on development is being refocused on projects that can help firms improve their margins and, not surprisingly, do a better job at risk management. As such, investments in capabilities such as algorithmic trading and complex event processing (CEP) are likely to be pivotal in some firms’ efforts to become more competitive and improve their efforts at mitigating risks.”

“But for some banks that have deployed such technologies — the now-defunct Bear Stearns, Lehman Brothers, Citigroup and Merrill Lynch — the question is: How did these companies fail to mitigate the risks that have slammed their businesses if their development teams were developing and deploying sophisticated systems?

“There is definitely an awareness that perhaps the systems that existed in place to assess the value of portfolios or judge risk [are being scrutinized],” said Stevan Vidich, an industry architect in Microsoft’s financial services group. “

He added that there is strong interest in CEP and other risk management methodologies. A growing number of shops have started deploying such solutions based on the .NET Framework, Vidich said, and he believes such investments will continue.

“Clearly, there’s a lot of need to deal with the immense influx of data and being able to analyze data in a timely manner,” Vidich said. “It also drives need for systems like business intelligence, or BI, applied to a near-real-time scenario, which is a very attractive proposition.”

What are these guys on Wall Street smoking? 

This is the precise “over hyping” problem I have warned about repeatedly.   Folks selling rule engines that perform basic calculations over a time window of streaming data have been marketing their wares as “superbrains” that can solve very complicated problems and, at the same time, save Wall Street and The Planet.

Let me be perfectly clear here Wall Street.  Listen very carefully.

There is nothing in any of the so called CEP products in the market place that is going to stop losses related to the subprime meltdown effecting the “now-defunct Bear Stearns, Lehman Brothers, Citigroup and Merrill Lynch,” as Jeffery Schwartz implies.

To imply that the risk management (and corporate governance) required to mitigate the current crisis on Wall Street can be foreseen, solved, or even mitigated, by a rules engine (or any software) is complete and absolute fantasy.   

I think the fever created by the subprime flu is putting folks on Wall Street, or at least the vendors and the analysts pandering to them, in a Capital Market CEP Fantasy Land.

 

The Predictive Battlespace

Friend and colleague Don Adams, CTO World Wide Public Sector, TIBCO Software, explains how CEP can be used to sense, adapt and respond to complex situations in The “Predictive” Battlespace: Leveraging the Power of Event-Driven Architecture in Defense. 

Probabilistic Complex Event Triggering

Here is an interesting paper, Probabilistic Complex Event Triggering, Daisy Zhe Wang, Eirinaios Michelakis, and Liviu Tancau, Computer Science Division, University of California at Berkeley, circa 2005.

One of the first things I noticed about the paper was the discussion of probability in the content of complex event processing, including Hidden Markov processes, Bayesian Belief Networks, and inference models.  

The second thing I noticed was that David Luckham’s work on CEP at Stanford was not referenced anywhere in the Berkeley paper.

 

More on CEP Maturity: Capability Versus Reliability

Louis Lovas of Progress Apama wrote a complimentary blog entry on the topic at hand, CEP Maturity Models.   In his post, Louis says:

“What a CEP platform has tracks independently of what it is capable of doing. ….. What CEP does, is likely what Tim is referring to when he states we’re in the Technology Trigger phase.”

Peter Lin’s comment, in reply to Louis, concurs:

“Given that COTS CEP has only been around a few years, I think it is safe to say it’s still in the early phase. If we compare it to messaging middleware, which has been around for more than 15 years, CEP isn’t as mature. Another comparison is business rule engines and expert systems. The earliest business rule engines date back to late 80’s. All things considered, I would agree with Tim. COTS CEP still has a lot of time to mature.”

Louis was spot on when he said that I was focused on overall CEP functionality; not individual product reliability.

Independent of how reliable a particular CEP-type application might appear; the overall state-of-the-art of CEP is really quite immature.

 

 

More on CEP: Process, Service or Reference Architecture?

In reply to Paul Vincent’s post Is CEP a Service or a Process? I posted Is CEP a Service or a Process? Reloaded.  This post is a follow-up to my dialog with Paul and the CEP community, as a whole.

Some of the more remarkable critical comments on the book “The Power of Events” was that the book did not (for the most part) discuss architecture. 

As we all know, there are many definitions of “architecture;” however, one definition that is easy to discuss, in this context, is that an IT systems “architecture” represents the components of an IT system and the relationships between the various components in the architecture. 

An architecture can be “technical” or “functional” or “operational” or “data” centric.  For example, an architecture can be based on an orchestration of service-components, like an SOA.  In another example, an architecture can be represented by the semantics of the data.  In yet another example, an architecture can be represented by the functionality of the components.

Because David’s book on CEP did not address architecture, folks have been free to use any “tool” or “technique” they like, and call it “CEP”.   My focus has been on overall CEP functionality and reference architectures that depict this functionality for solving CEP classes of problems.

This was one of the first topics (issues) with CEP we identified a few years ago; and is why we, including me at my good ole’ days at TIBCO until now, created a functional reference architecture for CEP (also in this blog and the TIBCO CEP blog).

In that functional reference architecture, we discussed and illustrated how CEP should operate as a cooperative (distributed) functional reference architecture to solve most “real” CEP classes of problems.

Therefore,  CEP should not be, generally speaking, considered as a “process” or a “service”,  per se,  because CEP, as a functional reference architecture, depicts the methodologies (functionaility) required to solve complex detection-oriented problems.  This abstract permits CEP to have meaning in a broad context of event processing applications.

Naturally, a functional reference architecture can be viewed as a “service” if all the components in the architecture cooperate to solve a problem and are encapsulated as a service.  In addition, a functional reference architecture can be viewed as a “process” when solving problems in a specific domain.  So, a “process,” in this case, is an instance of the functional reference architecture; and if the instance is packaged as a solution, this solution can be encapsulated as a service.

So, it is misleading, at least in my opinion, to reduce CEP to a “process” or a “service” unless we are discussing a particular solution to a domain problem within a (functional) reference architecture (functional context).

This confusion also manifests itself in the lively debate between Mark Palmer and the blogosphere regarding the maturity of CEP.   Mark and others have created an instance of event processing in capital markets and call it “CEP,” when in fact, what they are doing is COTS algo trading and using one or more functional components of CEP to realize their solution.

The is an important distinction, in my opinion.

On CEP Maturity and the Gartner Hype Cycle

In reply to Mark Palmer’s rebuttal, What Does it Mean to be Mature?, the figure below illustrates the popular Gartner Hype Cycle.  You can click on the illustration to get a clearer image.

In context to the Gartner Hype Cycle, CEP is closer to the “Technology Trigger” phase than anywhere else in the hype cycle.  CEP has not yet reached the “Peak of Inflated Expectations”, but is inching closer and closer.

In addition, as a correlating reference point, if you look at a recent Gartner Hype Cycle that covers EDA, for example, you will find that EDA  (Event Driven Architecture) is at a similar phase, the “Technology Trigger” phase. 

On the Maturity of CEP

Deciphering the Myths Around Complex Event Processing  by Ivy Schmerken stimulated a recent flurry of blog posts about the maturity of CEP, including; Mark Palmer’s CEP Myths: Mature or Not? and Opher Etzion’s On Maturity.

I agree with Ivy.  CEP is not yet a mature technology by any stretch of the imagination.  In fact, I agree with all three of Ivy’s main points about CEP.

In 1998 David C. Luckham and Brian Frasca published a paper, Complex Event Processing in Distributed Systems on a new technology called complex event processing, or CEP (Postscript Version).  In that seminal paper on CEP, the authors said, precisely:

“Complex event processing is a new technology for extracting information from message-based systems.”

Ten years later there are niche players, mostly self-proclaimed CEP vendors,  whom do very little in the way of extracting critical, undiscovered, information from message-based, or event-based, systems.  

A handful of these niche players have informally redefined CEP as “performing low latency calculations across streaming market data.”  The calculations they perform are still relatively straight forward and they focus on how to promote white-box algo trading with commercial-off-the-shelf (COTS) software.  In this domain, we might be better off not using the term CEP at all, as this appears to be simply a type of new-fangled COTS algo trading engine.

The real domain of CEP, we thought, was in detecting complex events, sometime referred to as situations, from your digital event-driven infrastructure – the “event soup” for a lack of a better term.    In this domain, CEP, as COTS software, is still relatively immature and the current self-styled COTS CEP software on the market today is not yet tooled to perform complex situational analysis.

This perspective naturally leads to more energy flowing in-and-around the blogosphere, as folks “dumb down” CEP to be redefined as it benefits their marketing strategy, causing more confusion with customers who want CEP capabilties that have zero to do with low latency, high throughput algo trading, streaming market data processing, which maybe we should call “Capital Market Event Stream Processing” or CESP – but wait we don’t really need more acronyms!

Hold on just a minute!  Wasn’t it just a short couple of years ago that folks were arguing that, in capital markets, it was really ESP, not CEP, remember?  Now folks are saying that it is really CEP and that CEP is mature?   

CEP is mature?  CEP is really not ESP?  CEP is really event-driven SOA?  CEP is really real-time BI?  CEP is really low latency, high throughput, white-box COTs algo trading?  CEP is really not a type of BPM?  CEP is not really for detecting complex events?   Complex does not really  mean complex? 

Come on guys, give us a break! 

(Anyway, no one is going to give us a break….  so stay tuned!)