Our Firehose can stream your realtime data published within the Ably platform directly to another streaming or queueing service. For example, all messages published by any device on a channel could be immediately streamed to Amazon Kinesis allowing you to process this data in realtime. Firehose is offered as part of our Ably Reactor service but is only available to Enterprise customers.
Using configurable rules, you can stream any of messages, presence events, and channel lifecycle events. Unlike our channels which follow a pub/sub pattern where each message is delivered to any number of subscribers, our Firehose operates on the basis that each message is delivered once to your streaming or queueing server.
As each message is delivered once to your streaming or queueing server, this design is commonly used to process realtime data published by Ably asynchronously. For example, using workers consuming from your stream or queue, you could persist each message of a live chat to your own database, start publishing updates once a channel becomes active, or trigger an event if a device has submitted a location that indicates that it has reached its destination. Find out why we think streams and message queues help solve many of the challenges associated with consuming pub/sub data server-side in our article: Message queues — the right way to process and work with realtime data on your servers.
Please note that if you want to consume realtime data from a queue, you should take a look at our Reactor Queues. They provide a straightforward and robust way to consume realtime from your worker servers without having to worry about queueing infrastructure.
Ably currently supports three types of data that can be delivered via our Firehose:
- Messages – messages are streamed as soon as they are published on a channel
- Presence events – when clients enter, update their data, or leave channels, the presence event is streamed
- Channel lifecycle events – when a channel is created (following the first client attaching to this channel) or discarded (when there are no more clients attached to the channel), the lifecycle event is streamed
Streaming server support
We can support the following streaming servers:
- Amazon Kinesis
- Apache Spark
- Apache Storm
- Google DataFlow
Queue server support
We can support the following queueing servers:
- Amazon SQS
- Apache Kafka