TIBCO Leaps Ahead in CEP with Insightful Acquisition

TIBCO Software shows, yet again, why the team in Palo Alto far outpaces the rest of the field with their announced acquisition of Insightful.  

Everyone who follows The CEP Blog and my vision for the business use of CEP understands how much energy and passion I have put into explaining why the crude time-series analysis of streaming data cannot possibly solve the vast majority of complex business problems CEP must address. 

TIBCO’s acquisition of Insightful shows just how serious TIBCO is about working to make the vision of “Predictive Business” a reality.    TIBCO means business, and a large part of what that means is helping customers solve their most challenging business integration problems, which can be summarized in CEP-speak as detecting opportunities and threats, in near real-time, as a core corporate competency. 

If you spend a few moments on the Insightful web site, you will find a treasure of documentation that discusses a gold mine of advanced statistical analytics that can be used in a number of mission critical applications.

This is the class of analytics that form the backbone of complex event processing.  In fact, as I have often pointed out (to the dismay of some of my CEP colleagues), any software company that discusses CEP and does not support or advocate advanced analytics are selling snake oil.      TIBCO obviously understands the difference between snake oil, smoke-and-mirrors marketing, and the technology it takes to solve real operational problems.

My hats off and warm congratulations to the team in Palo Alto for demonstrating, yet again, why TIBCO is committed to solving real customer problems with realistic solutions.

Maybe TIBCO will evolve to mean “The Insightful Business Company”   versus the tired and stale “The Information Bus Company” of yesteryears?

Disclaimer:  I have not been an employee of TIBCO for over a year. 

Capital Market CEP Fantasy Land

In Tech Spending Hit by Subprime Mess, Jeffery Schwartz says,

“According to Tabb, spending on development is being refocused on projects that can help firms improve their margins and, not surprisingly, do a better job at risk management. As such, investments in capabilities such as algorithmic trading and complex event processing (CEP) are likely to be pivotal in some firms’ efforts to become more competitive and improve their efforts at mitigating risks.”

“But for some banks that have deployed such technologies — the now-defunct Bear Stearns, Lehman Brothers, Citigroup and Merrill Lynch — the question is: How did these companies fail to mitigate the risks that have slammed their businesses if their development teams were developing and deploying sophisticated systems?

“There is definitely an awareness that perhaps the systems that existed in place to assess the value of portfolios or judge risk [are being scrutinized],” said Stevan Vidich, an industry architect in Microsoft’s financial services group. “

He added that there is strong interest in CEP and other risk management methodologies. A growing number of shops have started deploying such solutions based on the .NET Framework, Vidich said, and he believes such investments will continue.

“Clearly, there’s a lot of need to deal with the immense influx of data and being able to analyze data in a timely manner,” Vidich said. “It also drives need for systems like business intelligence, or BI, applied to a near-real-time scenario, which is a very attractive proposition.”

What are these guys on Wall Street smoking? 

This is the precise “over hyping” problem I have warned about repeatedly.   Folks selling rule engines that perform basic calculations over a time window of streaming data have been marketing their wares as “superbrains” that can solve very complicated problems and, at the same time, save Wall Street and The Planet.

Let me be perfectly clear here Wall Street.  Listen very carefully.

There is nothing in any of the so called CEP products in the market place that is going to stop losses related to the subprime meltdown effecting the “now-defunct Bear Stearns, Lehman Brothers, Citigroup and Merrill Lynch,” as Jeffery Schwartz implies.

To imply that the risk management (and corporate governance) required to mitigate the current crisis on Wall Street can be foreseen, solved, or even mitigated, by a rules engine (or any software) is complete and absolute fantasy.   

I think the fever created by the subprime flu is putting folks on Wall Street, or at least the vendors and the analysts pandering to them, in a Capital Market CEP Fantasy Land.

 

ICT Cmte: Thailand’s Cyber Law Compliance Seminar

ICT Cmte: Thailand’s Cyber Law Compliance Seminar

American Chamber of Commerce in Thailand

Date & Time: 17-Jun-2008

Details: This month You are invited to attend a Computer Crime Act Compliance Seminar. Find out what the Thai “Cyber Law” requires, when it will start to be enforced and how you can comply. If your business or hotel offers Internet access to customers, employees or end users, this will be a practical session for you to gain a better understanding of the Thai Computer Crime Act.

The Predictive Battlespace

Friend and colleague Don Adams, CTO World Wide Public Sector, TIBCO Software, explains how CEP can be used to sense, adapt and respond to complex situations in The “Predictive” Battlespace: Leveraging the Power of Event-Driven Architecture in Defense. 

The (ISC)2 Blog

(ISC)²^® is globally recognized for certifying information security professionals throughout their careers and the kind folks there have asked me to blog in information security topics.  If interested, here is the link to my posts over at the (ISC)2 blog.  

Spam Filtering: Understanding SEP and CEP

In order to help folks further understand the differences between CEP and SEP, prompted by Marc’s reply in the blogosphere, More Cloudy Thoughts, here is the scoop.

In the early days of spam filtering, let’s go back around 10 years, detecting spam was performed with rule-based systems.  In fact, here is a link to one of the first papers that documented rule-based approaches in spam filtering, E-Mail Bombs and Countermeasures: Cyber Attacks on Availability and Brand Integrity published in IEEE Network Magazine, Volume 12, Issue 2, p.10-17 (1998).   At the time, rule-based approaches were common (the state-of-the-art) in antispam filtering.

Over time, however, the spammers get more clever and they find many ways to poke holes in rule-based detection approaches.  They learn to write with spaces between the letters in the words, they change the subject and message text frequently, they randomize their originating IP addresses, they use IP addresses of your best friends, they changed the timing and frequency of the spam, etc. ad infinitium.

Not to sound like an elitist for speaking the truth,  but the more operational experience you have with detection-oriented solutions, the more you will understand that rule-based approaches (alone) are not scalable nor efficient.  If you followed a rules-based approach (only), against heavy, complex spam (the type of spam we see in cyberspace today), you would spend much of your time writing rules and still not stop very much of the spam!

The same is true for the security situation-detection example in Marc’s example.

Like Google’s Gmail spam filter, and Microsoft’s old Mr Clippy (the goofy help algorithm of the past), you need detection techiques that use advanced statistical methods to detect complex situations as they emerge.  With rules, you can only detect simple situations unless you have a tremendous amount of resources to build a maintain very complex rule bases (and even then rules have limitations for real-time analytics).

We did not make this up at Techrotech, BTW.   Neither did our favorite search engine and leading free email provider, Google!   

This is precisely why Gmail has a great spam filter.   Google detects spam with a Bayesian Classifer, not a rule-based system.    If they used (only) a rule-based approach, your Gmail inbox would be full of spam!!! 

The same is true for search and retrieval algorithms, but that is a topic for another day.  However, you can bet your annual paycheck that Google uses a Bayesian type of classifer in their highly confidential search and retreival (and – hint – classification) algorithms.

In closing, don’t let the folks selling software and analysts promoting three-letter-acronyms (TLAs) cloud your thinking. 

What we are seeing in the market place, the so-called CEP market place, are simple event processing engines.  CEP is already happening in the operations of Google, a company that needs real-time CEP for spam filtering and also for search-and-retrieval.  We also see real-time CEP in top quality security products that use advanced neural networks, and Bayesian networks, to detect problems (fraud, abuse, denial-of-service attacks, phishing, identity theft) in cyberspace.

Threats to the Democratic Process

Our readers might recall that this post by Tim Bass, The Top Ten Cybersecurity Threats for 2008.  One of the top ten threats to cybersecurity in 2008, according to this post, was:

    — Subversion of democratic political processes.

Interestingly enough, Electoral-Vote.com, a site maintained by Dr. Andrew Tanenbaum, Professor of Computer Science at the Vrige University in Amsterdam, reports (a link to a news story) that the US presidental election can be hacked.

This is not science fiction folks, it is simply the political and social realities of our brave new electronic world.