Complex Event Processing with Esphion Neural Agents

December 19, 2007

Detection-oriented technologies generally fall into two broad areas, signature-based detection and anomaly-based detection.    Complex event processing (CEP) is also a detection-oriented technology, so we can readily understand that CEP applications must also fall within the same two general areas.

Signature-based detection is sometime referred to as static detection because the technology relies on pre-defined rules, filters, and signatures to match known patterns.  At the most fundamental level, a virus checking program is an example of a signature-based system. 

On the other hand, anomaly-based detection systems strive to maintain a baseline of what is considered normal and then matches patterns outside normal operating parameters, often usings adaptive or artifical intelligence techniques.

Experts know that both anomaly and signature-based detection methods are important and each have their unique challenges and engineering tradeoffs.  For example, signature-based systems tend to generate false negatives because it is not possible to write all possible rules and filters to match every pattern, especially in dynamic real-time environments. Anomaly-based detection, on the other hand, tends to generate false positives because it is quite difficult to create a perfect profile of normal behavior. 

The challenge in most, if not all, detection-oriented systems is finding the right balance between false positives and false negatives.  In some situations, a system should error toward false positives.  In other applications, the system should error toward false negatives. 

CEP is, by defination, a technology to detect both opportunities and threats in distributed networks, in real-time, so it goes without saying that CEP is challenged by the same engineering tradeoffs that affect other detection-oriented systems.

A few weeks ago, I was discussing CEP with a CTO of one of Thailand’s largest telecommunications companies and he was very bullish on neural-based anomaly detection and from Esphion.

First generation detection systems rely on determinism, which is generally rule-based, and known to be insufficient for more complex real-time problems.  Esphion uses neural agents to gathering information on network activity and then creates a unifying situational infrastructure to protect against previously unknown threats.   For example, a fast spreading threat, such as the SQL/Slammer worm, will have reached all possible targets faster than any signature can be published or rule can be written, as mentioned in Worm detection – You need to do it yourself.

Since CEP is designed and marketed as a technology that brings real-time advantages to the detection of both opportunties and threats, we must ask ourselves the question why do all the current CEP software vendors fail to provide non-deterministic methods that are proven to adapt to a rapidly changing world?  

In Anomaly Detection 101, Esphion does a great job of describing how they do not rely on any pre-specified rules, baselines, models, signatures, or any other apriori knowledge.   They claim, and my highly respected telecommunications CTO colleague confirms, that there is no prior knowledge required and their customers are no longer adversely affected by zero-day anomalies or changing network conditions.

The technology behind Esphion does is what I would call complex event processing.

Advertisements

End Users Should Define the CEP Market.

December 17, 2007

My friend Opher mistakenly thought I was thinking of him when I related the story of the fish, as he replied, CEP and the Story of the Captured Traveller.

I must not have related the fish story very well, because to understood the story of the fish, is to know that we are all like the fish, in certain aspects of life, and there is nothing negative to be gleaned from the story.

However, to Opher’s point on CEP, I disagree.   Just because the marketing people (not the market) has misdefined CEP and therefore the vendors are drifting from the technology described in Dr. Luckham’s original CEP work, including his CEP book, we should not change the context of CEP.    Therefore, I don’t agree we should redefine CEP, as David envisioned, as Intelligent Event Processing (IEP) because CEP, as today’s software vendors sell it, is really SEP (or whatever!)  Please recall that David’s background at Stanford was AI and he did not define CEP as the software vendors have defined it either!

The fact of the matter is that the software marketing folks have decided they are going to use Dr. Luckham’s book to sell software that does not perform as Dr. Luckham described or envisioned!   I make no apologies for being on the side of end users who actually need to solve complex problems, not sell software that underperforms.

As I mentioned, this positioning and repositioning does not help solve complex problems.   At the end of the day, we have problems to solve and the software community is not very helpful when they place form over substance, consistently. 

Furthermore, as most customers are saying, time and time again, “so what?” … “these COTS event processing platforms with simple joins, selects and rules do not solve my complex event processing problems.”  “We already have similar approaches, where we have spent millions of dollars, and they do not work well.”

In other words, the market is crying out for true COTS CEP solutions, but the software community is not yet delivering.  OBTW, this is nothing new.  In my first briefing to the EP community in January of 2006, I mentioned that CEP required stating the business problem, or domain problem, and then selecting the method or methods that best solve the problem or problems.

To date, the CEP community has not done this because they have no COTS tool set other than SEP engines (marketed as either ESP engines or CEP engines – and at least ESP was closer to being technically accurate.) 

Experienced end users are very intelligent. 

These end users know the complex event processing problems they need to solve; and they know the limitations of the current COTS approaches marketed by the CEP community.  Even in Thailand, a country many of you might mistakenly think is not very advanced technologically, there are experts in telecommunications (who run large networks) who are working on very difficult fraud detection applications, and they use neural networks and say the results are very good.   However, there is not one CEP vendor, that I know of, who offers true CEP capability in the form of neural nets.  

Almost every major bank, telco, etc. has the same opinion, and the same problem. They need much more capability than streaming joins, selects and rules to solve their complex event processing problems that Dr. Luckham outlined in his book.   The software vendors are attempting to define the CEP market to match their capability; unfortunately, their capabilities do not meet the requirements of the vast majority of end users who have CEP problems to solve.

If the current CEP platforms were truely solving complex event processing problems, annual sales would be orders of magnitudes higher.  Hence, the users have already voted.   The problem is that the CEP community is not listening.


CEP Center of Excellence for Cybersecurity at Software Park Thailand

December 16, 2007

In July 2007, at InformationSecurityAsia2007,  I unveiled an idea to create a cybersecurity CEP Center of Excellence (COE) in Thailand.  Under the collaborative guidance of Dr. Rom Hiranpruk, Deputy Director, Technology Management Center, National Science and Technology Development Agency (NSTDA), Dr. Prinya Hom-anek, President and Founder, ACIS Professional Center, and Dr. Komain Pipulyarojana, Chief National Security Section, National Electronics and Computer Technology Center (NECTEC), this idea continues to move forward.

Today, in a meeting with Mrs. Suwipa Wanasathop, Director, Software Park Thailand, and her executive team, we reached a tentative agreement to host the CEP COE at Software Park.   

The mission of Software Park Thailand is to be the region’s premier agency supporting entrepreneurs to help create a strong world-class software industry that will enhance the strength and competitiveness of the Thai economy.

Since 2001, Thailand’s software industry has experienced approximately 20% year-over-year (YOY) growth.  Presently, Software Parks Thailand supports a business-technology ecosystem with over 300 active participants employing over 40,000 qualified software engineers across a wide range of technology domains.

I am very pleased that Software Park Thailand is excited about the potential benefits of CEP in the area of cybersecurity and detection-oriented approaches to cyberdefense. The COE will be working with best-of-breed CEP vendors to build, test and refine rule-based (RBS), neural network (NN) based and Bayesian network (BN) based approaches (as well as other detection methods) for cybersecurity.

I will be announcing more details in the future, so stay tuned.  Please feel free to contact me if you have any questions.


Thai Hospitality Makes It All Worth While

December 13, 2007

Many of you know that I do a lot of public speaking, events, meetings and briefings on a variety of technology-related topics.  

Yesterday I had the honor and pleasure of presenting to 60 banking executives and their team about a security topic very important to their bank. 

Normally, I’m quite confident speaking to any group; but I have to admit that I was quite nervous as I waited to present to 60 people, knowing that I was expected to speak in both Thai and English.   To make matters even more challenging,  I was following an excellent presentation by a very good speaker, and very funny man, who is a cybersecurity expert with the Royal Thai Police.    

When my turn to speak arrived, I opened my presentation in Thai and spoke for a few minutes only in Thai, talking about my experience in Thai culture and how happy I was to be there, thanking my hosts for inviting me.

To my surprise, the entire group warmly applauded me after my introduction!

It is really challenging moving from being a fluent, confident speaker in English to presenting to Thai executives and their team; but the kindness and warm hospitality of the Thai people make it all worth while.

My personal goal is to advance from presenting about 25 percent in Thai and 75 percent in English to 50 percent in Thai and 50 percent in English by this time next year.     

So, when you are presenting topics such as CEP or cybersecurity in your native English language, spare a few moment to think about your expatriate colleague in Thailand, presenting the same advanced technical concepts in Thai.