Building event streaming pipelines using Apache Pulsar
•
1 like•304 views
This document introduces Sijie Guo and StreamNative. Sijie Guo is the co-founder and CEO of StreamNative, which was founded by original developers of Apache Pulsar and Apache BookKeeper. StreamNative offers a cloud-native event streaming platform powered by Pulsar that provides fully managed Pulsar services. The document then discusses Apache Pulsar's capabilities for connecting, storing, and processing event streams in real-time at scale and StreamNative's extensibilities including Kafka-on-Pulsar and integrations with Spark and Flink.
Building event streaming pipelines using Apache Pulsar
- 1. streamnative.io Who am I? ● Sijie Guo (@sijieg) ● Co-Founder & CEO, StreamNative ● PMC Member of Pulsar/BookKeeper ● Ex Co-Founder, Streamlio ● Ex-Twitter, Ex-Yahoo ● Work on messaging and streaming data technologies for many years
- 2. streamnative.io About StreamNative ● San Francisco based company ● Founded by the original developers of Apache Pulsar and Apache BookKeeper ● Offer a cloud-native event streaming platform powered by Pulsar ● Provide fully managed Pulsar services from public cloud to on-premises
- 3. streamnative.io Event Streaming with Apache Pulsar Sijie Guo (@sijie) streamnative.io
- 4. streamnative.io Event Streaming Platform ● Connect ○ Publish and subscribe to event streams ● Store ○ Store infinite event streams reliably ● Process ○ Process event streams in real-time at scale
- 6. streamnative.io Apache Pulsar ● Publish-subscribe: unified messaging model (streaming + queueing) ● Infinite event stream storage: Apache BookKeeper + Tiered Storage ● Connectors: ingest events without writing code ● Process events in real-time ○ Pulsar Functions for serverless / lightweight computation ○ Spark / Flink for unified data processing ○ Presto for interactive queries
- 8. streamnative.io ● Connect ○ Protocol Handler ■ Kafka-on-Pulsar ■ AMQP-on-Pulsar ● Process ○ Unified data processing ■ Spark-Pulsar Integration ■ Flink-Pulsar Integration StreamNative Extensibilities
- 9. streamnative.io Why Kafka-on-Pulsar ● Kafka - Rich ecosystem of connectors and integrations ● Pulsar ○ Infinite stream storage ○ Pulsar Functions ● Adoption & Migration ● Real-time data pipelines ○ Ingesting data using Kafka Connect ○ Processing data using Pulsar Functions
- 11. streamnative.io Kafka-on-Pulsar ● Support Kafka 2.0+ protocols ● Support official java client, librdkafka, and non-java clients ● Multi-tenancy support ● SSL + SASL support ● It is a plugin and you can install it to your existing Pulsar cluster!
- 13. streamnative.io Demo ● K/P-Producer -> K/P-Consumer ○ TLS & SASL-PLAIN ● Kafka Connect -> Pulsar Functions * All demos run with TLS and SASL-PLAIN https://hackmd.io/nLj5M9BEQIacKcZsNrDxmQ
- 14. streamnative.io Demo 1: K-Producer -> K-Consumer
- 15. streamnative.io Demo 2: P-Producer -> K-Consumer
- 16. streamnative.io Demo 3: K-Producer -> P-Consumer
- 17. streamnative.io Demo 4: Kafka Connect
- 18. streamnative.io Demo 5: Pulsar Functions
- 19. streamnative.io 2020 Pulsar User Survey Report https://streamnative.io/whitepaper/sn-apache-pulsar-user-survey-report-2020/
- 20. streamnative.io TGI Pulsar Weekly Live Stream https://www.youtube.com/channel/UCywxUI5HlIyc0VEKYR4X9Pg/live
- 21. streamnative.io Follow us! ● KoP - https://github.com/streamnative/kop ● Webinar - https://www.youtube.com/watch?v=gL6hzRtij8M ● Follow us at Twitter ○ Sijie Guo (@sijieg) ○ Apache Pulsar (@apache_pulsar) ○ StreamNative (@streamnativeio) ● Join us at Pulsar slack!