Middleware and Event Processing Expenditures

Paul Vincent of TIBCO Software in his post, Outside CEP: the infrastructure stack, makes a statement that “every $ a CEP vendor spends on middleware integration is a $ less on interesting CEP functionality.”

Opher Etzion of IBM, in turn, agrees with Paul in his post, On the envelope for CEP, and discusses how there is much overlap between the capabilities in middleware and CEP.

I agree with both Paul and Opher, from a purely technical perspective.

On the other hand, if we sail a different tact and look deeper into the business aspects, we will see that, more often than not, EAI and CEP projects are (and will be) funded out of different part of the business organization.   There indeed is an overlap with CEP in the pure middleware applications, as Paul and Opher mentioned; but there is also a quite specific business domain aspect of CEP that is not traditionally (or purely) middleware related.

Recall that most detection-oriented CEP applications (the reason that CEP exists – detecting opportunities and threats in real-time) are domain specific applications.   These applications may, or may not, be funded out of the middleware side-of-the-house (often the CIO).  

Middleware is sometimes a difficult pill for organizations to swallow. 

Organizations know they need to integrate, but many have glued together their business processes with seemingly free protocols such as SMTP, FTP, SQL and HTTP for so long, that it hard to justify a large capital expenditure to rebuild the infrastructure.    Most of these organizations realize that it costs a lot of money to maintain this glueware, but getting everyone on board to paddle the ship in the same direction for integration is easier said than done.

CEP applications do not necessarily require such a broad coalition of people to work together because CEP is not purely about integration, it is about event processing – detecting opportunties and threats in real-time.    

The security team, working with the web team, can work together on an event processing project.  

The marketing team, working with the web team, can work together on an event processing project.

In other words, it may be easier to make business cases for specific event processing applications versus making a business case for a “revolutionize the enterprise” integration effort.

CEP is not purely a middleware technology.  Indeed, CEP can certainly be used in the middleware space,  but it can also be used in domain specific applications that are typically not funded from the same pot of gold as middleware.

Event Cloud Computing – IBM Turning Data Centers Into ‘Computing Cloud’

 I predict we may experience less debates on the use of the term “event cloud” related to CEP in the future, now that both IBM and Google  have made announcements about “cloud computing” and “computing cloud”, IBM Turning Data Centers Into ‘Computing Cloud’

“The initiative also builds on IBM’s announcement with Google last  month that they are developing cloud computing environments for  academic use.  But today’s announcement is aimed more widely at corporations and government users that want the extreme scale made possibly by lashing together pools of computers.”

This also furthers the notion and architecture we have been advocating, that event processing will move toward a distributed, cooperative computing model..

XASAX Launches Into CEP with Virtual Cyber Trading Hosting

XASAX has put together a virtual machine CEP hosting solution, at the source of the majority of stock exchanges in the US, using VMWare and clustered hardware. They have implemented a 5 POP financial backbone of market data with proximity hosting / co-location at each exchange throughout the country, which they claim, encompasses 95% of all publicly available market liquidity.

XASAX is currently using a StreamBase CEP engine with Studio seats. However, the XASAX network is touted as agnostic, so users can use any CEP or cyber trading software platform.

Each ticker plant on the XASAX network uses a proprietary message bus for machine to machine connectivity. The CEP engines are connected via Infiniband to a ticker plant. The ticker plants receive raw multicast market data direct from the exchanges.

There are 3 hosted ticker plants on the XASAX network.

1. Exegy (http://www.exegy.com). Hardware accelerated feed handlers providing consolidated market data feeds. They are parsing OPRA (~200 mbps of streaming market data) in ~80 microseconds in one machine using a combination of FPGAs and CPU.

2. InfoDyne (http://www.infodyne.com). Software based parsing. This solution parses OPRA in ~500 microseconds in 4 machines. It is very similar to Wombat.

3. XASAX custom parsers – http://www.opentick.com. XAXAS told us that they have spent the past 4 years developing ticker plant technology, entitlement systems, distribution methods, and feed handlers for exchanges. They are using lightweight versions of this software to drive CEP engines.

StreamBase, at the same time, incorporated opentick APIs this month independent of XASAX. Therefore, StreamBase works with the XASAX software in the initial launch period.

Under the XASAX teaming strategy with CEP vendors, XASAX hopes to bring down the overall cost of deploying a CEP algo trading engine dramatically.

XASAX plans to have ongoing beta tests for new feeds, engines & products. In addition, they plan to maintain beta test trial periods for all hosted applications and virtual servers. In the latter part of November, XASAX plans to go live with Nasdaq feeds & StreamBase in a production cyber trading environment.

Standard Data Sets for CEP/IDS Evaluation

We have been discussing standard data sets for CEP on CEP-Interest lately and have introduced the topic of “event cloud generation” here.    For those interested in applying CEP to intrusion detection, there is an evaluation dataset available from MIT.  

“These evaluations measured probability of detection and probability of false-alarm for each system under test.  These evaluations contributed significantly to the intrusion detection research field by providing direction for research efforts and an objective calibration of the technical state-of-the-art.  They are of interest to all researchers working on the general problem of workstation and network intrusion detection.  The evaluation was designed to be simple, to focus on core technology issues, and to encourage the widest possible participation by eliminating security and privacy concerns, and by providing data types that were used commonly by the majority of intrusion detection systems.”

Two data sets are the result of the DARPA Intrusion Detection Evaluations.

• 1998 DARPA Intrusion Detection Evaluation Data Sets
• 1999 DARPA Intrusion Detection Evaluation Data Sets

Three additional data sets are the result of experiments run in 2000 to
  address specific scenarios.

• 2000 DARPA Intrusion Detection Scenario Specific Data Sets

For folks seeking standard traces or datasets to evaluate CEP solutions for intrusion or fraud detection, the DARPA dataset is an excellent place to start.

An Event Cloud Generator for CEP Testing

Alexander Widder (Centrum für Informations-Technologie Transfer GmbH, Germany),  and I have been discussing the need for an event cloud generator that could be used for generating CEP scenarios for testing and evaluation purposes.   For example, integrating  flat files from UNIX/Linux syslog generators  (if one exists, need to check) with an open source ESB like Mule (from MuleSource) might be interesting. 

Another possibility (there are many possibilities, naturally) would be to use open source ESP software, for example Esper, to generate a stream of events that would be published to an open ESB like MuleSource to model and simulate (or generate) an event cloud. 

We all read a lot of posts, marketing material and blog entries about numerous CEP and ESP topics related to event processing; but we don’t have an open, standards based CEP event generator that can be used to model and simulate CEP scenarios.   I think it is time for a group of “vendor neutral” folks to work together in this important area.  Anyone interested?  Please comment.

Extrusion Detection is Ripe for CEP

The afternoon sessions of InformationSecurityAsia2007 were exceptional.   Dr. Keith White, APAC Security Services Director of Alcatel-Lucent, Australia described how they partnered with Cloudshield to process security events in a distributed SEM environment.   Topics covered included edge processing, content/context based routing and event processing.   After Keith’s excellent presentation I had a chance to speak with him about white-box event processing engines and strategic partnerships.

The next session was really interesting (session info), highlighting a similar situation – the criminals are far ahead of black-box SEM processing engines; and this is readily demonstrated in the emerging domain of extrusion detection.    For those not familiar with this term, extrusion detection is the network traffic inverse of intrusion detection.   In intrusion detection systems the focus is on the detection of threats from the outside of the network, to the inside of the network. 

However, what happens when criminals implant malware, covert tunnels (for example HTTP tunnels or ICMP tunnels), and malicious bot networks inside of organizations, and the detection challenge shifts to detecting outbound traffic from malicious users, malware, and botnets?    This form of criminal activity is evolving so fast that the models to detect extrusions are being formulated and tested in near real-time.   This is where CEP can help.

Imagine a high performance, declarative programming framework that can be used to implement extrusion detection models created by experts, like the cybersecurity experts gathered together at InformationSecurityAsia2007.   On top of that, visualize a design time studio environment that allows these same experts to graphically express their extrusion models in design time, avoiding most of the overhead of code development.   CEP and ESP engines are ripe for assisting security engineers detect the exploding commercialization of criminal extrusions, where, for example,  bot hearders can rent their botnets from $350 to $1000 USD per day.

I spoke to a number experts at InformationSecurityAsia2007 about CEP and I was pleased to learn that they have been considering CEP and ESP engines, including open source software (i.e. Esper) as well as commercial offerings.    We are considering collaborating on a new Center-of-Excellence that combines CEP/ESP engines with extrusion detection models.  Please contact me directly if you would like to participate.

We live in complex times.   Complex times require complex event processing.

More coming from InformationSecurityAsia2007 ….