Realtime Notifications at Redfin
Redfin is a leading full-service real estate brokerage that uses modern technology to help people buy and sell homes. Notification is the critical feature to communicate with Redfin’s customers, notification includes recommendations, instant emails, scheduled digests and push notifications. Thousands of emails are delivered to customers every minute at peak.
The notification system used to be a monolithic system, which served the company well. However, as the business grew and requirements evolved, it became harder and harder to maintain and scale.
The engineering team at Redfin decided to replace the existing system with Samza primarily for Samza’s performance, scalability, support for stateful processing and Kafka-integration. A multi-stage stream processing pipeline was developed. At the Identify stage, external events such as new listings are identified as candidates for sending a new notification; Then potential recipients of notifications are determined by analyzing data in the events and customer profiles. The results are grouped by customer at the end of each time window during the Match Stage. Once notifications and recipients are identified, the Organize stage further joins them with additional data-sources (eg: notification settings, customer profiles) leveraging Samza’s support for local state. It makes heavy use of RocksDB to store and merge individual notifications before sending them to customers. Finally, the notifications are formatted at the Format stage and sent to the delivery system at the Notify stage.
With the new notification system based on Apache Samza, Redfin observed that
- It is now easier to add support for new use cases
- The new system is more performant and horizontally scalable
- Reduced pressure on downstream services due to the use of local RocksDB state store
- Processing stages can be scaled individually since they are isolated
In addition to the notifications platform, other engineering teams at Redfin also use Samza for calculating business metrics, document processing, event scheduling etc.,
Key Samza Features: Stateful processing, Windowing, Kafka-integration