Webinar: BAM: The Killer App for CEP

Just recently found out that the folks at SL have confirmed an eBiz webinar where I’ll be speaking with colleague Ted Wilson about BAM: The Killer App for CEP.

Key Indicators (KIs) Versus Key Performance Indicators (KPIs)

SL‘s new web page, Solutions for CEP Engine Users, discusses how CEP is a “technology that is used to help companies detect both opportunities and threats in real-time with minimal coding and reusable key performance indicators (KPIs) and business models.”

I agree with SL, but would like to suggest my friends at SL expand the notion of KPIs in CEP to include the idea of KIs.  In my opinion, the SL phrase should read,  “technology that is used to help companies detect both opportunities and threats in real-time with minimal coding and reusable key indicators (KIs) and business models.”  

The reason for my suggestion is that KPIs are a subset of KIs.   KIs designate, in my mind, more than just performance.  

CEP is used to both detect opportunities and threats in real-time which may, or may not be, performance related.  For example, when a CEP engine detects evidence of fraudulent behavior, this is a KI.  The knowledge, or pattern, used to estimate this situation is a KI not a KPI, per se.   Also, when a CEP application is processing market data and indicates that it is the right time to purchase an equity and enter the market,  the knowledge used in this decision support application is a KI, not a KPI.

Therefore, I recommend when folks think about the notion of  “key performance indicators” (KPIs) in CEP and BAM, they should also think in terms of “key indicators” (KIs).   Detecting opportunities and threats in real-time are much broader than the traditional notion of KPIs. 

BAM Solutions for CEP Engine Users

Today I noticed that SL Corporation has revamped their website with a new page, Solutions for CEP Engine Users.    The page is well written, reinforcing some of my earlier posts on the value proposition for CEP; so I hope the folks at SL don’t mind if I repost their excellent thoughts on BAM and CEP here. 

Solutions for CEP Engine Users by SL Corporation

© 1999-2008 Sherrill-Lubinski Corporation. All rights reserved.

Complex Event Processing (CEP) is a relatively new technology that is used to help companies detect both opportunities and threats in real-time with minimal coding and reusable key performance indicators (KPIs) and business models. Just as services are shared and reused in a SOA, CEP permits the sharing and reuse of KPIs in business activity monitoring while efficiently processing events so businesses can act on situations that impact business and take advantage of real-time processing.

Business activity monitoring, often referred to as BAM, is the capability that Gartner and other distinguished analysts use to describe this visualization capability in the business world. BAM introduces a human element to CEP. It is well-established that the human mind is, today and for the foreseeable future, far superior to machine intelligence in making sense out of complicated situations and events. Therefore, BAM is critical to the success of any complex event processing (CEP) solution.

Depending on an organization’s mission, BAM can be used in various levels within an event processing solution to help users visualize and understand the dynamics behind rapidly changing situations and critical business events. In other words, BAM plays a key role wherever there is a need for better insight into the myriad events that effect your business operations.

BAM provides real-time visualization and alerting capabilities for users to better understand how business events impact their organization. BAM software permits users to quickly prototype, build and deploy event processing business solutions. For example, a telecommunications company would find BAM useful to achieve event-driven SLA monitoring and management; and a large retailer would find BAM important as they stay on top of business-critical events in their supply chain.

Insight gained from BAM, in concert with event processing solutions, enable organizations to make better and faster business decisions so they can rapidly sense and respond to threats, problems and opportunities. BAM solutions permit applications to be designed, deployed and modified rapidly with minimal or no coding resulting in significantly lower development costs. Therefore, a key benefit of BAM in real-time event processing solutions is that KPIs can be deployed, monitored, revised, reused and utilized, economically and rapidly.

Depending on the business application, BAM-enabled visualization is required at numerous levels in an event processing architecture. For example, events from across the enterprise are typically processed by a CEP software platforms from companies such as TIBCO, BEA (soon to be Oracle), Progress Apama, StreamBase, Aleri, and Coral8.

Long before KPIs are displayed to the business users, BAM tools can be configured to assist application developers to monitor and visualize the raw event stream. For the developer, their business is developing applications, and BAM can be very useful when designing KPIs for event processing applications.

Fine-tuned KPIs that have been derived from an event processing application are displayed to the business user. These KPIs can indicate risks, threats, problems, opportunities and other emerging business situations that impact the business.

BAM, in concert with state-of-the-art event processing software, provides the framework for a complete sense-and-respond capability for businesses. Processing raw events and event streams for business opportunities and threats requires robust and rapidly deployable visualization solutions. This is the reason that many distinguished analysts believe that BAM and CEP are complementary and critically interdependent core business capabilities. We at SL Corporation agree, and are pleased to be the leading BAM visualization platform in the event processing/CEP ecosystem today.

© 1999-2008 Sherrill-Lubinski Corporation. All rights reserved.

Coral8: Event Stream Processing and Intrusion Detection

Not quite ready for prime-time, we have been testing our home-grown UNIX domain socket adapter using Coral8 Java APIs.   We are using this adapter to evaluate and demonstrate stream processing with intrusion detection systems (IDS) using event stream processing to reduce false alarms, detect derived situations from the raw intrusion event data, and feed a security management visualization dashboard.

You can click on the teaser image below to see more of our first IDS screenshots from Coral8’s Studio stream visualization tool.

If you click on the image above, you will four additional event stream properties.  For this part of the demo, there are 14 total IDS properties in the event stream, but we only show 5 properties in this cropped screen capture.

I am quite sure that we could do similar integration with other event stream processing engines, but fortunately Coral8 makes it easy to download, start developing and testing. 

Apama: Fraud Detection and Heat Maps

A few days ago in Visualization Reloaded I touched upon the subject of heat maps.  In that post the application context was monitoring a massively parallel online gaming platform using a combination of event processing technologies by StreamBase and SL

Today, I was reminded of another heat map created by Progress Apama during a leisurely morning viewing of a Fox Business New video interview with John Bates.  This time the context is the detection of patterns of insider trading.  

In this graphic above (click the image for a larger view) Apama uses a heat map to visualize suspicious trading activity in real time.   Also, you might be interested to know that the cool heat map in this use case is based on the event processing visualization platform by SL Corporation, similar to the heat map in this use case by StreamBase, Simultronics and SL.

Amazingly, in the Fox Business interview John mentions an interesting statistic.   During certain business situations, like mergers and acquisitions, experts have estimated that up to 30 percent of trading activity can be linked to insider trading.   The event processing goal, of course, is to detect fraud sooner than later, minimizing fraudulent market transactions and their influence on the market.

OpenCourseWare: Get Smart for Complex Event Processing!

Ready to move beyond the basics of event processing?   Perhaps you would like to beef up your Java skills?   The Basics of Signal Processing?  Or maybe you are interested in Advanced Complexity Theory?   Artificial Intelligence?  Computer Language Engineering?  Queueing Theory?

Well then, put your feet up, relax and click on over to the Department of Electrical Engineering and Computer Science at MIT OpenCourseWare  (OCW) and enjoy their courses, freely available to anyone, anywhere. 

MIT’s OCW program freely shares their lecture notes, exams, and other resources from more than 1800 courses spanning MIT’s entire curriculum, including many fields related to event processing.     There are even RSS feeds on new courses as they hit the wire, so you don’t have to miss a thing!  Also, check out the OSC Consortium.

Complex event processing is a multi-discipline approach for detecting both opportunities and threats in real-time cyberspace.   Make it your New Years Resolution to review a few OCW lectures and help advance the state-of-the-art of CEP!

Complex Event Processing with Esphion Neural Agents

Detection-oriented technologies generally fall into two broad areas, signature-based detection and anomaly-based detection.    Complex event processing (CEP) is also a detection-oriented technology, so we can readily understand that CEP applications must also fall within the same two general areas.

Signature-based detection is sometime referred to as static detection because the technology relies on pre-defined rules, filters, and signatures to match known patterns.  At the most fundamental level, a virus checking program is an example of a signature-based system. 

On the other hand, anomaly-based detection systems strive to maintain a baseline of what is considered normal and then matches patterns outside normal operating parameters, often usings adaptive or artifical intelligence techniques.

Experts know that both anomaly and signature-based detection methods are important and each have their unique challenges and engineering tradeoffs.  For example, signature-based systems tend to generate false negatives because it is not possible to write all possible rules and filters to match every pattern, especially in dynamic real-time environments. Anomaly-based detection, on the other hand, tends to generate false positives because it is quite difficult to create a perfect profile of normal behavior. 

The challenge in most, if not all, detection-oriented systems is finding the right balance between false positives and false negatives.  In some situations, a system should error toward false positives.  In other applications, the system should error toward false negatives. 

CEP is, by defination, a technology to detect both opportunities and threats in distributed networks, in real-time, so it goes without saying that CEP is challenged by the same engineering tradeoffs that affect other detection-oriented systems.

A few weeks ago, I was discussing CEP with a CTO of one of Thailand’s largest telecommunications companies and he was very bullish on neural-based anomaly detection and from Esphion.

First generation detection systems rely on determinism, which is generally rule-based, and known to be insufficient for more complex real-time problems.  Esphion uses neural agents to gathering information on network activity and then creates a unifying situational infrastructure to protect against previously unknown threats.   For example, a fast spreading threat, such as the SQL/Slammer worm, will have reached all possible targets faster than any signature can be published or rule can be written, as mentioned in Worm detection – You need to do it yourself.

Since CEP is designed and marketed as a technology that brings real-time advantages to the detection of both opportunties and threats, we must ask ourselves the question why do all the current CEP software vendors fail to provide non-deterministic methods that are proven to adapt to a rapidly changing world?  

In Anomaly Detection 101, Esphion does a great job of describing how they do not rely on any pre-specified rules, baselines, models, signatures, or any other apriori knowledge.   They claim, and my highly respected telecommunications CTO colleague confirms, that there is no prior knowledge required and their customers are no longer adversely affected by zero-day anomalies or changing network conditions.

The technology behind Esphion does is what I would call complex event processing.