Event Processing and Streaming Query Engines

I  just went over a nice article in the January issue of the Communications of the ACM, Data In Flight by Julian Hyde (he maintains a great blog by the way). I think that it is a good read for anyone questioning the added value of Event Processing software in general and CEP (Complex Event Processing) in particular.

The author argues successfully that Streaming Query Engines are at the heart of Event Processing applications and quickly recaps their advantages over a relational database, for example, when processing events:

• Event processing is rarely concerned with transactions, as such streaming query engines store queries only, not data, which makes them very efficient
• Streaming query engines are concerned with the latest data and keep very little history in memory: This varies from a few seconds to at most a business day worth of data, which relieves the engine from the need to maintain a persistent storage and support ad-hoc querying  – of course this is just the typical usage, not a hard rule
• Streaming query engines process data asynchronously to achieve the highest level of performance

It is worth noting here that stream-oriented languages are just one flavor of event processing languages. IBM WebSphere, for example, uses a rule-oriented language (WebSphere Business Events). Understandably the JBoss offering in this space (DROOLS Fusion) uses a rule-oriented language too. Just to be clear, these two products are event processing software, not BRMS (Business Rule Management System) software. They may use a rule-oriented language to act on events, but they differ from a BRMS product where a) rules are invoked by requests made from within the application (as opposed to event processing where the processing is triggered by the occurrence of events or situations) and where b) rules operate on the various states (see for example http://www.jboss.org/drools/drools-flow.html).

I did not conduct an exhaustive survey but from my own experience  I would venture to say that stream-oriented languages are the more popular ones out there to express event processing logic. That’s because they naturally lend themselves to express time constructs as a fundamental dimension of event processing. Stream-oriented languages do that by typically introducing time operators to SQL-like languages. Here is a list of a few vendors that went down this route:

• SQLStream
• Aleri/Coral8 CCL
• Esper EPL
• Oracle CQL
• Streambase StreamSQL
• Open ESB IEP CQL

The list above gives a flavor of the various Streaming Query Engines out there. There is enough value in the expressiveness of the languages alone to warrant  their adoption as opposed to rolling your own half-baked event processing solution based on, say, pattern matching of messages on a bus.  As soon as the rate of arrival/dissemination of the messages increases and the need to correlate between various out-of-order messages over variable windows becomes pressing I think that it makes sense to consider an event driven application.

WebSphere Business Events

About these ads