A stream refers to data that is incrementally made available over time with an unbounded input. Basically makes use of the Publisher Subscriber Model, where messages notify consumers of events.
One challenge that needs to be solved is the issue in which messages are being produced faster than they are consumed.
- Use a buffer (Message Queue?
- Drop messages?
- Apply backpressure?
- What if a node crashes? How do we remain durable?
It is common for messages to be send using a broker or a Message Queue
Analytics
Complex event processing is an approach that is the opposite of typical database querying, where the query is held long-term and the stream (which is transient), is examined for a pattern that matches the query.
- Esper
- IVM InfoSphere Streams
- Apama
- TIBCO StreamBase
- SQLstream
- Samza Moreover, typically statistics are computed over a fixed time interval, with analytics, these are computed over a window or analytics use a probabilistic algorithm like a bloom filter. You can also run search on streams which is like applications that let you be notified when something that matches a search filter pops up (Zillow).
Joins
Stream-stream
- Both input streams consist of events and the join operator searches both events that occur within some window of time. Stream-table
- One input stream consists of events while the other is a database change log. The changelog keeps a local copy of the database up to date. For each event, the join operator queries the database and outputs an enriched event. Table-table
- Both input streams are database changelogs. In this case, every change on one side is joined with the latest state of the other side. The result is a stream of changes to the materialized view of the join between the two tables.
Achieving integrity in stream processing systems without the need for distributed transactions and Atomic Commits uses:
- Representing the write operation as a single message that can be atomically committed
- Deriving all other state updates from that single message using a pure function
- Using client generated request IDs for duplication suppression and Idempotence
- Making messages immutable and allowing derived data to be reprocessed to recover from bugs This allows coordination-avoiding data systems to have comparable correctness for better performance and operational robustness (we can use things like multi-leader Replication across different data centers asynchronously replicating since we don’t need strict Linearity).