CEP Event Sources

In this post we start to list different event sources for CEP applications. I will start with a broad classification, following the probe architecture by IBM’s NetCool OMNIbus, provide examples, expand and elaborate adding market data feeds, and other event sources that come to mind.

We will expand the list based on comments. Also, we might find it necessary to reclassify the top level sources at some point in the future:

  • Devices as Event Sources
    • Routers, hubs, switches, and similar network or telecommunication devices
    • Network appliances designed for alerting, like a firewall, sniffer or an IDS
    • Sensors and sensor networks including RFID readers
  • Log Files as Event Sources
    • Application log files including syslog ()
    • Other flat file event logs
  • Databases as Event Sources
    • Change (insert, update, or delete) to a row of the source table events
    • Events captured in real time during the transaction
  • Data Feeds as Event Sources
    • Market data feeds
    • News feeds (RSS / Atom)
    • Other feeds from feed handlers
  • Message-Oriented Middleware (MOM) as Event Sources
    • JMS
    • TIBCO RV
    • IBM MQ
  • Analytics as Event Sources
    • Rules-engines
    • Statistical analytics (e.g. Bayesian inference engines)
    • Artificial neural networks
    • Filters and signal processors (e.g. Kalman filters)
    • CEP and ESP engines (could also be or include one or more of the above)
  • Process and Resource Management Systems as Event Sources
    • BPM systems
    • CRM systems
    • Tealeaf adapter for Customer Experience Monitoring/Analytics.
  • Users as Event Sources
    • E-mail, SMTP
    • IM
  • Timers as Event Sources

Please comment to add more event sources. This list is by no means complete, it is just a beginning. I’ll update as time permits.

Advertisements

18 Responses to CEP Event Sources

  1. What about BPM and Rules Engines as event sources.

  2. Tim Bass says:

    Thanks for the reminder York.

    Of course!

    (added)

  3. Jason Brome says:

    Here are some initial thoughts:

    Applications as Event Sources
    – Network-bound Instrumented management/monitoring alerts (SNMP traps etc.)

    Application Servers as Event Sources
    – Java – JMX/MBean attribute/change notifications

    Web Servers/Containers as Event Sources
    – access_log / error_log monitoring (these would also apply under the ‘Log Files as Event Sources’ section)
    – in-line filters ‘teeing’ copies of request/response metadata and/or content (e.g. Java Servlet filters)

  4. Most of our work takes place in environments where the infrastructure and applications are already in place (very little green-field stuff), so we’re mostly trying to create events in a way that doesn’t have a measurable impact (doesn’t involve instrumentation, doesn’t affect application/business logic, doesn’t add noticable latency to transactions, etc. We’ve used the following as event sources – some have already been mentioned but I thought I’d include them to move them from the “theory” pile to the “reality” pile:

    Web Servers: JSP/ASP filters
    CORBA: Portable Interceptors (with injected context propagation for full execution-path correlation of events for track and trace or transaction tracking)
    Java Applications: Aspect-based sensors
    Packet Inspection: Interpreting specific network traffic protocols e.g. IIOP, MQSeries
    Web Services: JAX-RPC, .NET assemblies (No ESB’s yet, but expect news soon!)
    Messaging: TIBCO

  5. Tim Bass says:

    Hi Brian and Jason,

    Thanks for posting.

    Yours faithfully, Tim

  6. Anil Datt says:

    What about the most fundamental one – A Timer Based Event?

  7. Tim Bass says:

    Hi Anil,

    Please explain the event source of your “Timer Based Event”.

    All event sources can be time based, so I am not following your comment, sorry.

    Thanks,

    Yours faithfully, Tim

  8. Damien Joguet says:

    What about ERP in general (SAP, Oracle, PeopleSoft, …) ?

  9. What I’m missing here is some way of capturing events in the “real world” and transforming them (see your next post) into system events. For example: customer buys product, customer phones helpdesk, customer returns product, customer complains. Passenger passes security check, passenger fails initial security check, passenger falls ill (before take-off), passenger falls ill (midflight). Are you assuming that these are all captured by some “application” or other?

  10. Tim Bass says:

    Hi Richard,

    Yes, EDA and CEP are not generally based on stand-alone architectures; they are generally built on integration platforms. When “customer buys product” the event could be generated by the application at the point-of-sale (POS), or the inventory control system, or the CRM system, depending on the architecture and application (or where the event-object is publshable). The same is true of “passenger passes security check”, where in this case, the IT application that controls the checkpoint publishes the event.

    Yours faithfully, Tim

  11. I’d like to be able to apply event-driven architecture to the application level as well, rather than having events emerge from a black hole called “applications”. It would be useful to have a classification scheme that will permit us (but not force us) to deconstruct the applications as appropriate.

  12. Anil Datt says:

    There could be processes kicked off by a Timer (it could be releated to a Calendar,Schedular etc). In this case the Timer itself is the Event Source. I do not agree with your comment that all the Event Sources are time based. They could be related to the Timer, but may not always be based of timer.

    -Anil Datt

  13. Tim Bass says:

    Hi Anil,

    I do not necessarily agree with you that “the timer is the event source” in the scenario you describe above. A strong counter argument can be made that the Calendar, Scheduler, etc. is the event source and the timer is simply a mechanism of measurement (of time, relative to something).

    For example, if a UNIX cron runs a process and the process kicks off an event, the event source is not the crontab, it was the process that kicked off the events.

    On the other hand, I agree that time is critical. Moreover, time is relative. I agree that timers can generate events, but the events must be relative to a source, otherwise, we have reduced the scenario to simply “a clock.” If so, using your argument, then we can say that clocks are event sources. I’ll have to think about that one.

    Also, I did not say in my comment above that “all events are time based”. I said, “all events can be time based”.

    Time exists, regardless of our perspective. Where does time stand still?

    Yours faithfully, Tim

  14. Anil Datt says:

    Tim,

    What I meant was that the “Timer” is different from “Time”.
    Time has existing forever and it is not stand still. Time is directly related to the clock.
    Calendar is one form of Timer. Timer I think is the most primitive type of Event.
    I agree that the Calendar can be argued as the Event source and not the Timer.
    Calendar can kick off a process at a certain date and Time.

    For e.g.:
    An email that has to sent at specific Date and time of the day is a combination of Email event and the Calendar event (which inherently has Timer event)

    An email which does not necessarily have to be sent at any specific date and time will simply be an email event.

    A process which has a Timer (not associated with the Calendar) at start can wait for the Timer to expire before executing the process.
    In this case the process is kicked off by the Timer Event (an alarm on the clock).

    Here are some of the links which talk about timer Events.

    http://www.oasis-open.org/committees/download.php/23964/wsbpel-v2.0-primer.htm#_Toc166509710

    http://e-docs.bea.com/wli/docs81/manage/evntgen.html#1063936

    http://www.californiasw.com/Knowledge-center/whitepaper/webmethods.html

    http://cpan.uwinnipeg.ca/htdocs/Tibco-Rv/

    Anil Datt

  15. Tim Bass says:

    Hi Anil,

    I agree with you but would like to clarify a few points.

    First, an email is transmitted with numerous time references (time sent, time received), so I am not sure if your example of email is disassociated with time. In fact, most events occur outside of a specified time, but they, none-the-less, occur relative to a clock. Well….

    This line of thinking reminds me of the “chicken or the egg” argument, but in a different way.

    If a timer sits and waits to fire and, when it fires, kicks off an event (like a bomb), it is a philosophical question as to “is the event source the timer or the bomb”. Without the bomb, there is no event. The source of the event is the bomb, the timer simply triggers it; or at least that is one way to look at it. Your perspective is equally valid.

    The references you site show timer events, so I will add it to the list. Thanks!

    Yours faithfully, Tim

  16. Anil Datt says:

    Tim,

    My email example came up from my practical experience, where a Process A can send several emails at different points in its execution to a Process B (Event sink). Let us say the SLA at the Event sink dictates to process the Emails from Process A after a specific time of the day. In this case the Process A is configured to accumulate all the messages till the time of the day (may be merge them) and release it at that specific time. This is the only time reference that is important in this case that would be associated with the Email Event. The time sent and received is important here because of the SLA.

    If the SLA does not enforce processing constraint based on time then Process B can receive emails at any point of time and process them individually. Here the Time is not important. I would say these email events are not time related.

    The first case is a time sensitive email event and the second one is not.

    Thanks,
    Anil

  17. Tim Bass says:

    Hi Anil,

    Nice example.

    In general, most people and systems associate e-mail with time. Your application may be a specific case where you don’t care about time. We can always find exceptions to any situation or example.

    Generally, if a criminal sends a threatening email to someone, the time of the email is critically important in a court of law.

    More genearlly all the email in our inbox(s) and outbox(s) are ordered by time.

    Also, technically speaking, all SMTP headers contain time stamp info.

    BTW, when we send email who / what do you consider as the “event source” ?

    Yours faithfully, Tim

  18. Anil Datt says:

    Tim,

    There could be different types of notifications sent from the running process. It could E-mail, SMS,Message,could be a recored audio message etc…In each case the process is using a specific type of Event generator to send the message. In this case it would be obviously the Email event generator (or a email adapter). From the receiving end perpective , the source it has received the event from, is a email event source.

    Anil

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: