An Overture to the 2007 CEP Blog Awards

January 9, 2008

Before announcing the winners of the 2007 CEP Blog Awards I thought it would be helpful to introduce the award categories to our readers.

I have given considerable thought to how to structure The CEP Blog Awards. This was not an easy task, as you might imagine, given the confusion in the event processing marketspace. So here goes.

For the 2007 CEP Blog Awards I have created three event processing categories. Here are the categories and a brief description of each one:

The CEP Blog Award for Rule-Based Event Processing

Preface: I was also inclined to call this category “process-based event processing” or “control-based event processing” and might actually do so in the future. As always, your comments and feedback are important and appreciated.

Rule-based (or process-based) event processing is a major subcategory of event processing. Rule-based approaches to event processing are very useful for stateful event-driven process control, track and trace, dynamic resource management and basic pattern detection (see slide 12 of this presentation). Rule-based approaches are optimal for a wide-range of production-related event processing systems.

However, just like any system, there are engineering trade-offs using this approach. Rule-based systems tend not to scale well when the number of rules (facts) are large. Rule-based approaches can also be difficult to manage in a distributed multi-designer environment. Moreover, rule-based approaches are suboptimal for self-learning and tend not to process uncertainty very well. Never the less, rule-based event processing is a very important CEP category.

The CEP Blog Award for Event Stream Processing

Stream-centric approaches to event processing are also a very important overall category of event processing. Unlike a stateful, process-driven rule-based approach, event stream processing optimizes high performance continuous queries over sliding time windows. High performance, low latency event processing is one of the main design goals for many stream processing engines.

Continuous queries over event streams are genenerally designed to be executed in milliseconds, seconds and perhaps a bit longer time intervals. Process-driven event processing, on the other hand, can manage processes, resources, states and patterns over long time intervals, for example, hours and days, not just milliseconds and seconds.

Therefore, event stream processing tends to be optimized for a different set of problems than process-based (which I am calling rule-based this year) event processing. Similar to rule or process-based approaches, most current stream processing engines do not manage or deal with probability, likelihood and uncertainty very well (if at all).

The CEP Blog Award for Advanced Event Processing

For a lack of a better term, I call this category advanced event processing. Advanced event processing will more-than-likely have a rule-based and/or a stream-based event processing component. However, to be categorized as advanced event processing software the software platform must also be able to perform more advanced event processing that can deal with probability, fuzzy logic and/or uncertainty. Event processing software in this category should also have the capability to automatically learn, or be trained, similar to artificial neural networks (ANNs).

Some of my good colleagues might prefer to call this category AI-capable event processing (or intelligent event processing), but I prefer to call this award category advanced event processing for the 2007 awards. If you like the term intelligent event processing, let’s talk about this in 2008!

Ideally, advanced event processing software should have plug-in modules that permit the event processing architect, or systems programmer, to select and configure one or more different analytical methods at design-time. The results from one method should be available to other methods, for example the output of a stream processing module might be the input to a neural network (NN) or Bayesian Belief (BN) module. In another example pipeline operation, the output of a Bayesian classifier could be the input to a process or rule-based event processing module within the same run-time environment.

For all three categories for 2007, there should be a graphical user interface for design-time construction and modeling. There should also be a robust run-time environment and most, if not all, of the other “goodies” that we expect from event processing platforms.

Most importantly, there should be reference customers for the software and the company. The CEP Blog Awards will be only given to companies with a proven and public customer base.

In my next post on this topic, I’ll name the Awardees for 2007. Thank you for standing by. If you have any questions or comments, please contact me directly.


Cyberattack! Manipulation and Subversion of Financial Markets!

January 8, 2008

In The Top Ten Cybersecurity Threats for 2008 I mentioned one of most critical cybersecurity threats of 2008 will be the manipulation and subversion of financial markets.

Well, folks in New York have barely finished cleaning the colorful New Years confetti off Wall Street and a vivid example of manipulating the market appears in the news.  We have yet to see the second week of January and the cyberattacks are upon us.

Notice how a direct competitor to E*Trade Bank, Citigroup, used the power of cyberspace, rumors and (mis)information to manipulate the market price of  E*Trade (ETFC).  This might have not been such an eyebrow raising event if the rumor (cyberattack) was by a disinterested third party.  The attack was by a direct competitor with their own subprime balance sheet problems!

On or about November 12, 2006, Citigroup Investment Research analyst Prashant Bhatia, a longtime critic of E*Trade (ETFC), sent E*Trade stocks into a free fall with his cyberattack, “[E*Trade] Bankruptcy risk cannot be ruled out.”  This rumor, by a competing financial institution, basically wiped out countless millions of dollars in investor equity in a few short minutes.  Amazing!

Of course, E*Trade is not going bankrupt any more than Citigroup is going bankrupt, but the information warfare by Citigroup’s Prashant Bhatia continues.  Here is the lastest:

On January 7th, 2008, Bhatia changes his cyberattack slightly and reiterates his “prediction”  that E*Trade will not be profitable for at least three years.  This statement was quite different than his earlier attack where he “predicted” bankruptcy for a competing E*Trade.    

Is Prashant Bhatia or Citigroup accountable for these “predictions”?   When investors are wiped out in seconds based on a cyberattack by a competing financial institution, who is responsible for these actions and investor losses?  Or, are these attacks simply “capitalism” and “the free market” where cyberattacks are commonplace and investors are no more than pawns in the market battlesphere?

Some on the net have said that Prashant Bhatia’s actions are not much different than the owner of a competing movie theater, who stands up and yells, FIRE!! FIRE!! FIRE!!   Then,  after he clears the theater, he then yells, RATS!! COCKROACHES!!  POISON POPCORN!!  to insure no one comes back to the movie house. 

To me, this is an amazingly obvious cyberattack with a direct purpose to manipulate the market and cause damage to a competing financial services institution.   

How can this attack be allowed to happen?

I think anyone can easily see how this is a very serious cybersecurity threat in 2008 and beyond.  Unfortunately, these types of attacks will certainly get worse before it gets better.  Welcome to “the real world”….   In this vividly real example, an analyst from a competing bank yells FIRE! and basically subverts the market, causing countless of investors to lose many millions upon millions of dollars in the blink of an eye.   Wow.


E*Trade was vulnerable (as were other banks) due to subprime issues.  E*Trade Financial is a critical competitor (and threat) to Citibank.    This is a textbook cyberattack in today’s world.   

Dr. David Luckham, discussing financial news and complex event processing (CEP) recently asked, How well does Elementizing work? and reminds us” News Moves Markets.”   He is certainly correct, obviously.

The cybersecurity issue  that I am highlighing here is how easy it is for news to be manipulated and, in turn, subvert markets, as easily observable with Citigroup’s Prashant Bhatia and his commercial cyberattacks against E*Trade Financial and, most importantly, their investors.

Apama: Fraud Detection and Heat Maps

January 2, 2008

A few days ago in Visualization Reloaded I touched upon the subject of heat maps.  In that post the application context was monitoring a massively parallel online gaming platform using a combination of event processing technologies by StreamBase and SL

Today, I was reminded of another heat map created by Progress Apama during a leisurely morning viewing of a Fox Business New video interview with John Bates.  This time the context is the detection of patterns of insider trading.  

Apama Heat Map

In this graphic above (click the image for a larger view) Apama uses a heat map to visualize suspicious trading activity in real time.   Also, you might be interested to know that the cool heat map in this use case is based on the event processing visualization platform by SL Corporation, similar to the heat map in this use case by StreamBase, Simultronics and SL.

Amazingly, in the Fox Business interview John mentions an interesting statistic.   During certain business situations, like mergers and acquisitions, experts have estimated that up to 30 percent of trading activity can be linked to insider trading.   The event processing goal, of course, is to detect fraud sooner than later, minimizing fraudulent market transactions and their influence on the market.

Aleri is Not the Leading CEP Vendor, Sorry.

December 7, 2007

Similar to the outrageous marketing post we exposed in, Agent Logic is Not the Leading CEP Vendor, Sorry, a similar press release just came across my desk.   This time the exorbitance is from Aleri, containing the inordinate statement:

“Aleri Inc., the leading provider of enterprise-class complex event processing (CEP) technology, ….”

Come on guys!   Get control of your marketing department!  

Aleri is not the leading provider of enterprise-class CEP technology.

Type I and Type II Errors – The Heart of Event Processing

December 5, 2007

Opher Etzion begins to discuss one of the topics I consider to be the heart of event processing in his post, On False Positives and False Negatives.

Statistically speaking, false positives are called Type I errors (α errors) and false negatives are called Type II errors (β errors).

If you are interested in “getting to the heart” of complex event processing, visit this link.   I’ll discuss detection theory in more detail in a future post.

The Top Ten Security Threats for 2008 (Part 10)

December 3, 2007

Starting on my next five top cyber security threats for 2008, here is another serious threat:

      — Criminal manipulation and subversion of financial markets.

In one of our earlier posts, I mentioned the subversion of democratic political processes as a top ten cybersecurity threat for 2008, which included information warfare techniques applied to elections and other political processes.  

The top ten cybersecurity threat in this post, criminal manipulation and subversion of financial markets, is very similar to the threat of the subversion of democratic processes.  Criminals can easily use powerful botnets, blogs, social networking, spam, RSS and other cyber technologies to target and manipulate financial markets with pump-and-dump and similar fraudulent schemes.

Not only do fraudsters, unethical brokers and corrupt analysts inject good news into cyberspace to illegally inflate the market value of equities; they also poison cyberspace with doom-and-glum messages to deflate market values.   This threat poses a serious threat to investors as well as the overall integrity of financial markets.    Indeed, this threat may be used to subvert the democratic process, indirectly attacking our of the most critical infrastructures globally, the financial markets.

Bad news travels fast and the effects are furious.  Just look at E*Trade and the effect each analyst has had on their stock price recently.   One analyst talks about how good a new development is for E*Trade, stocks rise.  Another analyst talks about how bad the same development is for E*Trade, the stocks plunges.   Fortunes are made and lost on news blurbs that rise, fall and fade away, like the light of fireflys, in cyberspace.  This is a very real cybersecurity threat now, in 2008 and beyond.

COTS Software Versus (Hard) Coding in EP Applications

November 21, 2007

Opher Etzion has kindly asked me to write a paragraph or two on commercial-off-the-shelf  (COTS) software versus (hard) coding software in event processing applications. 

My thoughts on this topic are similar to my earlier blog musings, Latency Takes a Back Seat to Accuracy in CEP Applications.

If you buy a EP engine (today) because it permits you run some quick-and-dirty (rule-based) analytics against a stream of incoming events, and you can do this quickly without spending considerable software development costs, and the learning curve and implementation curve for the COTS is relatively low, this could be a good business decision, obviously.   Having a software license for an EP engine that permits you to quickly develop and change analytics, variables and parameters on-the-fly is useful. 

On the other hand, the current state of many CEP platforms, and their declarative programming modelling capabilities, is that they focus on If-Then-Else, Event-Condition-Action (ECA), rule-based analytics.  Sophisticated processing requires more functionality that just ECA rules, because most advanced detection-oriented applications are not just ECA solutions.

For many classes of EP applications today, writing code may still be the best way to achieve the results (accuracy, confidence) you are looking for, because CEP software platforms have not yet evolved to plug-and-play analytical platforms, providing a wide range of sophisticated analytics in combination with quality modelling tools for the all business users and their advanced detection requirements.

For this reason, and others which I don’t have time to write about today, I don’t think that we can say blanket statements that “CEP is about using engines versus writing programs or hard coding procedures.”   Event processing, in context to specific business problems, is the “what” and using a CEP/EP modelling tool and execution engine is only one of the possible “hows” in an event processing architecture.  

As we know, CEP/EP engines, and the marketplace for using them, are still evolving and maturing; hence, there are many CEP/EP applications, today, and in the foreseeable future, that require hard coding to meet performance objectives, when performance is measured by actual business-decision results (accuracy). 

Furthermore, as many of our friends point out, if you truely want the fastest, lowest latency possible, you need to be as close to the “metal” as possible, so C and C++ will always be faster than Java byte code running in a sandbox written in C or C++.   

And, as you (Opher) correctly point out, along with most of the end users we talk to, they do not process megaevents per second on a single platform, so this is a marketing red herring.  This brings us back to our discussions on the future of distributed object caching, grid computing and virtualization – so I’ll stop and go out for some fried rice and shrimp, and some cold Singha beer.