A Closer Look at Apama’s Economic News Alerts

August 31, 2007

If you haven’t noticed, Apama is doing some interesting things in the CEP space toward future event correlation.   I have been closely following their partnership with Dow Jones to support their elementized news feed. Let’s take a closer look.

The Apama screenshot, Economic News Alerts, highlights three or four top level tabs:

  • Welcome;
  • News based momentum trading;
  • CPI news trade launcher; and
  • All Economic Indicators.

Apama Economic News Alerts

In the sample screenshot, we are looking at examples (test data) of economic indicators  from 2004, 2005, 2006 and 2007, timestamped on April 30th, 2007, 17:16:15 – 17:16:23, specifically:

  • The 1/2006 Employment Cost Index (ECI);
  • The 5-8/2005 Producer Price Index (PCI);
  • The 6-11/2004 Consumer Price Index(CPI);
  • The 8-11/2004 Consumer Price Index, Average Weekly Earnings (CPI);
  • The 4-5/2004 Consumer Price Index, Energy and Food prices index;
  • The 3-4/2006 Consumer Price Index, Energy and Food prices index; and
  • The 10-11/2004 Weekly Jobless Claims.

I believe the quantities and indicators (from the screenshot) are from test data because, believe-it-or-not, I checked a few of the indicators in the screenshot and they were different than the actual published values.  Plus, in my mind, it does not make much sense to be trading in April 2007 based on indicators in 2004 (for example), please comment if you have a different opinion on this.

None-the-less, the examples in the test data show how traders are beginning to think about executing trades based, in part, on the differences between the expected values of economic indicators v. the actual values, hot-off-the-press, in real-time. 

An article by Wall Street & Technology, Trading on the News, has an interesting quote on this topic:

“The Holy Grail is to do in-depth semantic analysis of news, and I think that’s going to be helped by the proliferation more of XML-based news feeds and research feeds, and popular Web XML feeds like RSS and Atom — and then there’s XBRL and RIXML,” says Carrie. “If you’re XML-based, your content can be classified,” he continues. “That reduces the complexity of parsing and transferring the information into a format that can be read by the computer.”

There are folks who doubt whether news, by itself, could actually be used to trade. However, many believe that traders will certainly augment their strategies with news events.

EDA Visualized

August 28, 2007

Below is a graphic I put together for my September Gartner presentation, Getting Started in CEP: How to Build an Event Processing Application, called “EDA Visualized.”

EDA Visualized

In this graphical representation of EDA, we show the sense-and-respond model around the functional CEP/EP reference architecture. I have slightly modified the traditional view of “Sense – Analyze – Respond” to “Sense – Analyze and Visualize – Respond” because visualization tools are critically important.

Key EDA Concepts

Sense – Detect events across the enterprise in real-time. Normalize and contextualize.

Analyze & Visualize – Aggregate events across multiple sources; correlate with historical data, refine.

Respond – Manage resources, processes; Invoke actions in real-time.

The Sense – Analyze and Visualize – Respond (decision loop) model illustrates the high level concepts of EDA (the “what”) and the CEP/EP functional reference architecture (adapted from the JDL model) represents “the how”, from a high level functional components perspective.

CEP and Rules Reloaded

August 28, 2007

Wait a few moments for the file to download and see what Neo-Architects have to say about CEP….

The Matrix

I know you’re out there. I can feel you now. I know that you’re afraid… you’re afraid of CEP. You’re afraid of change. I don’t know the future. I didn’t come here to tell you how this is going to end. I came here to tell you how it’s going to begin. I’m going to write this blog, and then I’m going to show these people what you don’t want them to see. I’m going to show them a world without you. A world without rules and controls, without borders or boundaries. A world where anything is possible. Where we go from there is a choice I leave to you.

Brandl and Guschakowski Deliver Excellent CEP/BAM Report

August 27, 2007

Hans-Martin Brandl and David Guschakowski of the University of Applied Sciences Regensburg, Faculty of Information Technology/Mathematics, have delivered an excellent CEP thesis, Complex Event Processing in the context of Business Activity Monitoring, to their advisor Dr. Rainer v. Ammon.   In their report they examined a BAM application using CEP concepts to implement what they referred to as a “next generation easyCredit” activity, using CEP software from Coral8, Business Bridge (Systar France), StreamBase, and TIBCO.

Here is one of the noteworthy quotes from the CITT report:

“TIBCO offers a huge portfolio for implementing CEP-based BAM. According to the requirements of this thesis, four components were utilized; TIBCO EMS, TIBCO BE, TIBCO BW and TIBCO RTView. TIBCO BE correlated all events correctly, based on the requirements. Due to the implementation of the TIBCO specific visualization tool RTView, a presentation of the correlated data is possible. RTView provides a variety of charts, dashboards and other visualization options in order to display the relevant metrics like “loss by cancelled credit applications” or performance and status of the deployed business logic components respectively JSPs. The messaging application TIBCO EMS provides multiple adaptors for all incoming and outgoing event-types. When increasing the amount of incoming CBEs, TIBCO´s BE processed well, according to the applied rules and concepts. With TIBCO BW it was possible to create, orchestrate and execute all relevant processes.”

I highly recommend everyone who is interested in BAM applications of CEP to download and read this report (240 page  thesis), look at the screenshots and learn more about CEP and BAM.

Rules Engines and Bayes’ Theorem

August 27, 2007

Charles Young kindly calls out my blog post, Bending Rules for CEP, and the discussion on rules and Bayesian analytics in his post, Rules Engines and Bayes’ Theorem.

FWIW, implementing a very simple Bayes network with a rules engine, as in Charles interesting example, does little to counter the argument that rules engines are not an efficient implementation for larger Bayes nets and more complex data sets.  It is not convincing to implement somewhat trivial solutions to make a point that does not hold up to the state-of-the-art in Bayesian networks for complex problems.   The complexity and inefficiency (with rule-based systems) comes with larger data sets; a long standing problem with rules.

If rule-based systems were efficient for large data sets, complex spam filters would use rule-based systems, but they don’t.    Most modern fraud detection algorithms are implemented with Bayesian algorithms (not rules).  Few, if any, credible large companies uses rules for these classes of problems anymore – they all use specially formulated Bayesian algorithms.

In fact, in the late 90s at Langley Air Force Base we soon discovered the same problem dealing with massive distributed email bomb attacks on the Internet (one example reference, also see Popular Science article, WAR.COM, by Frank Vizard).  After documenting our rule-based approach, subsequent researchers and implementations all commented that a rules-based approach is primitive (paraphrased) compared to modern Bayesian techniques.

These comments are not designed, BTW, to disparage rules or  rule-based systems.  Rules are great; but they are not very efficient in large, complex problems.

I can provide more historical and current references on this topic if anyone is interested.


See, Introduction to Bayesian Belief Nets by Russ Greiner, University of Alberta and note the basic compexity  in the example,  Forecasting Oil Prices in slide 31.

Event Transformation Services

August 24, 2007

In my last post, CEP Event Sources, I started a (short) list of event sources, both in theory and in practice, for complex event processing. There were some good comments and the list is evolving and improving.

Event transformations are used to convert inbound event objects to an outbound object type required by a transport layer, such as a JMS Message, or a stream, such as a market data format, or other event or feed handler.

Event transformations typically happen in the event preprocessing stage, of an application; but in a generalized architecture, events can be preprocessed throughout the event processing lifecycle. Multiple transformations can be combined, chained or piped together as required.

Below is a list of commonly used event transformations. Please feel free to comment as we add (or subtract) to (or from) the list.

Other Possible Transformations (not exhaustive)

  • Cryptographic Transformations
  • Encoding Transformations
  • CVS Transformations
  • Vendor Specific Transformations

Please contribute to the list with your comments.

CEP Event Sources

August 23, 2007

In this post we start to list different event sources for CEP applications. I will start with a broad classification, following the probe architecture by IBM’s NetCool OMNIbus, provide examples, expand and elaborate adding market data feeds, and other event sources that come to mind.

We will expand the list based on comments. Also, we might find it necessary to reclassify the top level sources at some point in the future:

  • Devices as Event Sources
    • Routers, hubs, switches, and similar network or telecommunication devices
    • Network appliances designed for alerting, like a firewall, sniffer or an IDS
    • Sensors and sensor networks including RFID readers
  • Log Files as Event Sources
    • Application log files including syslog ()
    • Other flat file event logs
  • Databases as Event Sources
    • Change (insert, update, or delete) to a row of the source table events
    • Events captured in real time during the transaction
  • Data Feeds as Event Sources
    • Market data feeds
    • News feeds (RSS / Atom)
    • Other feeds from feed handlers
  • Message-Oriented Middleware (MOM) as Event Sources
    • JMS
    • TIBCO RV
    • IBM MQ
  • Analytics as Event Sources
    • Rules-engines
    • Statistical analytics (e.g. Bayesian inference engines)
    • Artificial neural networks
    • Filters and signal processors (e.g. Kalman filters)
    • CEP and ESP engines (could also be or include one or more of the above)
  • Process and Resource Management Systems as Event Sources
    • BPM systems
    • CRM systems
    • Tealeaf adapter for Customer Experience Monitoring/Analytics.
  • Users as Event Sources
    • E-mail, SMTP
    • IM
  • Timers as Event Sources

Please comment to add more event sources. This list is by no means complete, it is just a beginning. I’ll update as time permits.