On Time-Series Analysis with Strict Determinism

Like the predictable ebb and flow of ocean tides, we see the rise, falling and passing away of lively debates about event processing languages (EPLs).   For example, you might recall that Louis Lovas, Progress Apama,  did an excellent job in his post, Bending the Nail, where he described why SQL or Extended SQL is not the optimal EPL for event processing.  

A few of us in the “SQL is not necessarily the best EPL” choir started singing with Louis which motivated a counter voice the choir with the post, Fair and unfair criticism of an SQL EP approach only to have the same author counter that post with, One down side to an SQL EP approach.   

Many technologists, including some of my team members at Techrotech, enjoy focusing on linear event processing problems with strict determinism, for example, processing a stream of market data and looking for opportunities to enter or exit the market (algo trading).    These same technologists tend champion event processing problems that are basic transformations of simple streams of time-series data.  

Many of the so-called CEP cybertrading examples (I would argue that these are simple event processing, not complex event processing examples) are not rooted in event processing scenarios that require looking for causal linkages between seemingly unrelated events; for example, debugging complex distributed systems or detecting fraud over long periods of time where sliding time windows on continuous streaming data are only a part of the solution in the uncertain world of  cloudy event-causality relationships.

Time-series analysis with strict determinism are interesting, but I would not go so far at to call this processing “complex event processing” relative to the myriad challenging complex problems in the real-world.

A Page from Greg’s Diary: Nerwana Software

I started my career in IT many years ago and since that year have worked in enterprise IT for year and years.     Almost all of my odd career story evolves around working with end users, often advising, architecting and managing the complexity of large systems integration projects, from hands on implementation to strategic vision development.  My deep background is with Techrotech in network systems engineering.

A few years ago, years after I started my career at Techrotech, I grew a bit dismayed at enterprise software companies.   They would, for the most part, always come to us, the end users, and try to sell us large software packages.  Their sales and technical teams had very little domain knowledge of the problems they claimed they could solve – and they had little doubt that if we purchased their wares, our problems would be solved,

These software companies were keen on buzzwords and technology jargon but somewhat clueless on operational solutions or the challenges of implementation across a large federated organization with many powerful business units and “in name only” CIOs.  We often referred to these software sales guys, and their favorite systems integrators, as “drive by (or fly by) implementations” where they dump the software (and hardware) at your door and run like crazy!

So, I joined a very cool Silicon Valley company,  Nerwana Software, hoping to change all of that, or so I thought 🙂

Naturally, when I first came on board Nerwana , the entire organization, from executives to recent new hires out of school, heaped praise-upon-praise on my years of operational experience at Techrotech and elsewhere.   They cheered me on as I wrote papers and created slides on operational use cases and event processing solutions that the sales and solutions teams could take to market.   They sang my praises as I spoke to large audiences and evangelized their most innovative software and solutions.  They were pleased with the great reviews from customers.

As one would expect, I was destined to learn the face of the problems I experienced as an end-user “outsider,” now from an “insider’s” perspective. 

One of the interesting challenges that surfaced at Nerwana was the “let’s export our culture and business model to the world” mantra, maybe better referred to as “if it sells in New York, then we must sell it the same way in Tokyo or Bejing!”

Also, I really was surprised to find out how dependent Nerwana was on the opinion of analysts.   When I worked for the customers and end users, we rarely paid any special attention to the analyst’s opinions.   Sure, analysts provides a good data point, but that is all it was (or is), simply another data point.   

I soon found that software companies are often held hostage by “analyst chasing” which really was an eye opener for me, because we end-users, the people who actually buy the software, view analysts as mere mortals reading from the same foggy crystal ball as everyone else. 

Another one of the fasinating challenges I experienced at Nerwana was what some would call  “The Hero Culture.”  

I’ll elaborate on some these, hopefully interesting, observations and experiences in a future Page from Greg’s Diary.

Military Event Processing Requirements and COTS CEP Software

In Q&A from BCS SPA meeting on CEP,  friend and colleague Paul Vincent says:

 “AFAIK there are no current military systems (as opposed to government intelligence systems) using Commercial Off The Shelf CEP systems, although I recall one commercial product being developed with US military money (your tax $ at work, etc etc).”

Actually, Paul’s statement is slightly misleading.   Companies like StreamBase and AgentLogic have their roots in supporting the military.  In addition, IBM has a number of event processing related solutions in the military.   (There are also others, we suspect.)

It is true, however, that current generation COTS CEP engines do not have the advanced event processing capabilities required for most CEP applications  in the military; but as CEP engines advance, this should change.

Event-Driven Business Process Management and the Example of the Deutsche Post AG

Christoph Emmersberger and Florian Springer have finished their thesis which was written onsite at Oracle Headquarters in Redwood Shores, CA, USA (Note: the link to this paper is not working now):

Event-Driven Business Process Management taking the Example of Deutsche Post AG:  An evaluation of the Approach of Oracle and the SOPERA Open Source SOA Framework

The topic of this thesis was the prototypical integration of the Oracle products

• ·Oracle BPEL (Business Process Management),
• ·Oracle BAM (Business Activity Monitoring), and
• ·Oracle CEP (Complex Event Processing),

within the SOPERA system environment, with the focus on CEP.

For evaluating the capabilities of the components, a business process regarding to shipment, investigation and claim was modelled and implemented.

Different approaches were discussed, evaluated and implemented as prototypes.

The focus of the implementation was to use events for the purpose of monitoring a business process.