TIBCO Leaps Ahead in CEP with Insightful Acquisition

June 24, 2008

TIBCO Software shows, yet again, why the team in Palo Alto far outpaces the rest of the field with their announced acquisition of Insightful.  

Everyone who follows The CEP Blog and my vision for the business use of CEP understands how much energy and passion I have put into explaining why the crude time-series analysis of streaming data cannot possibly solve the vast majority of complex business problems CEP must address. 

TIBCO’s acquisition of Insightful shows just how serious TIBCO is about working to make the vision of “Predictive Business” a reality.    TIBCO means business, and a large part of what that means is helping customers solve their most challenging business integration problems, which can be summarized in CEP-speak as detecting opportunities and threats, in near real-time, as a core corporate competency. 

If you spend a few moments on the Insightful web site, you will find a treasure of documentation that discusses a gold mine of advanced statistical analytics that can be used in a number of mission critical applications.

This is the class of analytics that form the backbone of complex event processing.  In fact, as I have often pointed out (to the dismay of some of my CEP colleagues), any software company that discusses CEP and does not support or advocate advanced analytics are selling snake oil.      TIBCO obviously understands the difference between snake oil, smoke-and-mirrors marketing, and the technology it takes to solve real operational problems.

My hats off and warm congratulations to the team in Palo Alto for demonstrating, yet again, why TIBCO is committed to solving real customer problems with realistic solutions.

Maybe TIBCO will evolve to mean “The Insightful Business Company”   versus the tired and stale “The Information Bus Company” of yesteryears?

Disclaimer:  I have not been an employee of TIBCO for over a year. 


Capital Market CEP Fantasy Land

June 23, 2008

In Tech Spending Hit by Subprime Mess, Jeffery Schwartz says,

“According to Tabb, spending on development is being refocused on projects that can help firms improve their margins and, not surprisingly, do a better job at risk management. As such, investments in capabilities such as algorithmic trading and complex event processing (CEP) are likely to be pivotal in some firms’ efforts to become more competitive and improve their efforts at mitigating risks.”

“But for some banks that have deployed such technologies — the now-defunct Bear Stearns, Lehman Brothers, Citigroup and Merrill Lynch — the question is: How did these companies fail to mitigate the risks that have slammed their businesses if their development teams were developing and deploying sophisticated systems?

“There is definitely an awareness that perhaps the systems that existed in place to assess the value of portfolios or judge risk [are being scrutinized],” said Stevan Vidich, an industry architect in Microsoft’s financial services group. “

He added that there is strong interest in CEP and other risk management methodologies. A growing number of shops have started deploying such solutions based on the .NET Framework, Vidich said, and he believes such investments will continue.

“Clearly, there’s a lot of need to deal with the immense influx of data and being able to analyze data in a timely manner,” Vidich said. “It also drives need for systems like business intelligence, or BI, applied to a near-real-time scenario, which is a very attractive proposition.”

What are these guys on Wall Street smoking? 

This is the precise “over hyping” problem I have warned about repeatedly.   Folks selling rule engines that perform basic calculations over a time window of streaming data have been marketing their wares as “superbrains” that can solve very complicated problems and, at the same time, save Wall Street and The Planet.

Let me be perfectly clear here Wall Street.  Listen very carefully.

There is nothing in any of the so called CEP products in the market place that is going to stop losses related to the subprime meltdown effecting the “now-defunct Bear Stearns, Lehman Brothers, Citigroup and Merrill Lynch,” as Jeffery Schwartz implies.

To imply that the risk management (and corporate governance) required to mitigate the current crisis on Wall Street can be foreseen, solved, or even mitigated, by a rules engine (or any software) is complete and absolute fantasy.   

I think the fever created by the subprime flu is putting folks on Wall Street, or at least the vendors and the analysts pandering to them, in a Capital Market CEP Fantasy Land.

 


The (ISC)2 Blog

May 28, 2008

(ISC)²® is globally recognized for certifying information security professionals throughout their careers and the kind folks there have asked me to blog in information security topics.  If interested, here is the link to my posts over at the (ISC)2 blog.  


A Bitter Pill To Swallow: First Generation CEP Software Needs To Evolve

February 8, 2008

Frankly speaking, the CEP market is now saturated with hype about all the great things CEP can do, detecting opportunities and threats in real time and supporting the decision cycle.  However, in my opinion, it is time for the software vendors and analysts to move beyond the marketing hype and demonstrate real operational value with strong end user success, something seriously lacking today.

I have advocated this evolution for two years, including the notion of expanding CEP capabilities with proven techniques for event processing that have worked well long before current “Not yet CEP but called CEP” software hit the marketplace and airwaves.

For example, in my first CEP/EP presentation in New York in 1Q 2006, I presented Processing Patterns for Predictive Business and talked about how the US military has implemented high performance detection-oriented systems for many years (in the art-and-science of multisensor data fusion, MSDF), and how every day, when we sit at home (or at work or in transit), we are comforted to know we are safe from missile attacks because of what I would also call “complex event processing.”   There is a very rich history of “CEP but not called CEP” behind the scenes keeping people safe and warm. (The same thing can be said with many similar examples of complex event processing in use today, but not called “CEP” by CEP software vendors.)

This is one reason, when I read the “CEP history lessons,” I am amused at how, at times, the lessons appear self-serving, not end user serving.  There is so much rich event processing history and proven architectures in “CEP but not called CEP” (CEP that actually works, in practice everyday, long before it was called CEP).  It continues to puzzle me that a few people the CEP/EP community continue to take the “we invented EP” view.  Quite frankly, the history we read is missing most, if not all, of the history and practice of MSDF.

When we take the current CEP COTS software offerings and apply it to these working “CEP but not called CEP” applications, the folks with real operational “CEP but not called CEP” detection-oriented experience quickly cut through the hype because they are, based on their state-of-the-practice, now seeking self-learning, self-healing “real CEP type” systems.  They are not so excited about first generation technologies full of promises from software vendors with only a few years of experience in solving detection-oriented problems and very few real success stories.

The same is true for advanced fraud detection and other state-of-the-art detection-oriented processing of “complex events” and situations.  The state-of-the-art of complex event processing, in practice, is far beyond the first generation CEP engines on the market today. 

This is one of the reasons I have agreed with the IBM folks who are calling these first generation “CEP orchestration engines” BEP engines, because that view is closer to fact than fiction.  Frankly speaking again, process orchestration is much easier than complex detection with high situation detection confidence and also low false alarms.

Customers who are detection-savvy also know this, and I have blogged about a few of these meetings and customer concerns.  For example, please read my blog entry about a banker who was very sceptical in a recent wealth management conference in Bangkok.  I see this reaction all the time, in practice. 

Complex problems are not new and they still cry out for solutions.  Furthermore, many current-generation event processing solutions are already more advanced that the first generation CEP engines on the “call it CEP” market today.  This is a very real inhibitor, in my opinion, to growth in the “call it CEP” software space today – and credibility may ultimately be “at risk.”  Caution is advised.

Candidly speaking again, there are too many red-herring CEP-related discussions and not enough solid results given the time software vendors have been promoting CEP/EP (again, this is simply my opinion).  The market is in danger of eventually losing credibility, at least in the circles I travel and complex problems I enjoy solving, because the capabilities of the (so called) CEP technologies by software vendors in the (so called) CEP space have been over sold; and, frankly speaking, I have yet to see tangible proof of “real CEP capabilities” in the road maps and plans of the current CEP software vendors.  This is dissappointing.

This pill is bitter and difficult to swallow, but most of my life’s work has been advising, formulating and architecting real-time solutions for the end user (the C-level executives and the operational experts with the complex problems to solve).   CEP software must evolve and there needs to be more tangible results, not more marketing hype.


Events are the Heart of the COSO ERM Framework

January 24, 2008

COSO was originally formed in 1985 to sponsor the National Commission on Fraudulent Financial Reporting, an independent private sector initiative which studied the cause-and-effects that can lead to fraudulent financial reporting. 

COSO developed enterprise risk management (ERM) recommendations for public companies and their independent auditors, and also for the SEC, other regulators, and for educational institutions.

At the heart of COSO is events and how events, both opportunity and threat-related events, in context, effect enterprise risk management.

Detecting opportunity and threats in real-time, both mentioned in COSO, is a core CEP concept; so I will be blogging on how CEP relates to COSO and ERM (and also Basel II ORM) in a future blog post.

Please stay tuned …


The ART of Event Processing: Agility, Reuse, Transparency

January 18, 2008

The other day I discussed CEP in Layman’s Terms: Reuse and Agility. Today, our topic is CEP and transparency. One of the major benefits of “white box” event processing solutions is transparency, something not readily available or obvious in black-box solutions.

Friend and colleague John Bates, Progress Apama, often discusses the benefits of white-box algorithmic trading platforms in terms of increased time-to-market and other competitive advantages. I agree with John and would like to point out that there is another key benefit, in simple layman’s terms, transparency.

For example, let’s say you have designed an event processing solution for operational risk management (ORM). It is time for your favorite auditors to come by and they wish to take a look at what is going on with that proprietary black-box ORM application running quietly in the server room.

The nice auditors ask you, “What does that application do?” and you reply “Well, it looks for evidence of insider trading,” and they ask “Do you mind if we ask how?” and you respond “Good question, do you mind to wait a moment while I get you the contact info for the vendor because we don’t have access to the source code or the actual key indicators (KIs)?”

Now, let’s look at the white-box scenario:

Again, the nice auditors ask you, “What does that application do?” and you reply “Well, it looks for evidence of insider trading,” and they ask “Do you mind if we ask how?” and you respond “Yes, sit down and we will pull up our insider trading key indicator models. These models are stored in XML format and viewable in our graphical KI design studio. We can print out the KI models for insider trading if you like!” and the smiling auditor says “Thank you, your system is much more transparent than the last place we visited!”

This scenario also applies in looking for why certain KIs were not detected that should have been; or when performing a root cause analysis to see why the KI you used in your wrong business decision was inaccurate.

So, CEP in layman’s terms is what we might refer to as the ART of event processing:

  • Agility
  • Reuse
  • Transparency

Please feel free to reuse these idea, but please don’t forget to reference the author and this blog 🙂

Kindly share and reuse by reference, because all content in The CEP Blog is ©2007-2008 Tim Bass – All Rights Reserved. Thank you!


The Top Information Security Risks for 2008

January 15, 2008

Blogging has it’s rewards.

I recently published a list of the Top Ten Cybersecurity Threats for 2008.

This list motivated another collaborative list for 2008, organized by Dr. Gary HinsonThe Top Information Security Risks for 2008.