Serverless Event Streaming with Pulsar Functions
•
0 likes•393 views
The last few years have seen the emergence of Serverless as a paradigm for event streaming. Its very simple programming model has attracted developers in droves. At the same time, its ability to elastically scale has simplified operations significantly. Combined together with the ubiquity of their presence across all cloud providers, serverless today has become the leading choice to do event processing at scale for a lot of companies. In this talk, Sijie Guo from StreamNative will explore how the serverless paradigm is applied to event streaming in Apache Pulsar, a next-generation event streaming system. Pulsar provides native support for serverless functions where the events are processed as soon as they arrive in a streaming manner and that provides flexible deployment options (thread, process, container). He will describe how these serverless functions make data engineering easier and share the real world usage of Pulsar Functions.
Serverless Event Streaming with Pulsar Functions
- 1. Serverless Event Streaming with Pulsar Functions Sijie Guo (@sijieg) 2019.06.22
- 2. ● Apache Pulsar PMC Member ● Apache BookKeeper PMC Chair ● Twitter, Yahoo Alumni ● Founder of StreamNative ● Interested in technologies around Event Streaming Who am I
- 3. ● What is Apache Pulsar? ● Event Stream - Pulsar view on Data ● When Event Streaming meets serverless ○ Programming Model ○ Architecture ○ Use cases Agenda
- 4. What is Apache Pulsar?
- 5. “Flexible pub/sub messaging backed by a durable stream storage” What is Apache Pulsar?
- 7. Pulsar - Multi Tenancy
- 8. Pulsar - Flexible Messaging
- 9. ● One data, different ways to consume ● Queuing (aka stateless messaging) ○ Shared (* RabbitMQ) ● Streaming (aka stateful messaging) ○ Exclusive ○ Failover (* Kafka) ○ Key_Shared Pulsar - Flexible Messaging
- 10. Pulsar - Cloud Native Architecture Layered Architecture ❏ Independent scalability ❏ Instance failure recovery ❏ Balance-free on cluster expansions
- 11. A Pulsar view on Data
- 12. Pulsar View - Topic
- 13. Pulsar View - Partition
- 14. Pulsar View - Segment
- 15. Pulsar View - Event Stream
- 16. Event Stream is the right foundation for your data
- 18. MSP - Interactive Queries
- 19. MSP - Stream & Batch Processing
- 20. MSP - What is next? What is next?
- 21. When Event Streaming meets Serverless
- 23. Pulsar Functions ● A serverless event streaming framework ● Lightweight computation ● Event-first, Stream-first ● Multi languages ● Multi runtimes ● SDK-less & SDK
- 24. Function Elements ● Input Topics ● Output Topics ● Function ● State ● Log Topics
- 25. API - Native Java / Python / Go Function Golang Python Java
- 26. API - Function Context ● Logger ● State ● Metrics ● Security / Secrets ● ...
- 27. Context - Logger
- 28. Context - State
- 29. Context - State ● Global Managed State * ● Mutable by functions and admin-cli ● Queryable by functions and admin-cli ● State are stored at storage layer ● State are implemented using streams + snapshot
- 30. Context - State API ● Key/Value State API ○ putState ○ getState ● Counter State API ○ getCounter ○ incrCounter
- 31. Context - Metrics ● API - recordMetric(String metricName, double value) ● Exposed in prometheus format ● Collected by prometheus
- 32. Flexible Runtime ● Colocate with Broker - Thread & Process ● Managed Function Workers - Thread & Process ● External Schedulers - Container ○ Kubernetes
- 35. External Schedulers - Kubernetes
- 36. Event Routing ● Events are routed to different partitions ● Leverage Pulsar’s MessageRouter ● Existing MessageRouters ○ Round-Robin ○ SinglePartition ○ Hash (Murmur32) ● Customize MessageRouter
- 37. Auto load balancing ● Pulsar Functions use Pulsar’s auto-balancing mechanism on consumers ● Shared Subscription ○ Load is distributed among function instances (consumers) evenly ○ More function instances provides more processing capability ● Failover Subscription ○ Load is distributed among function instances (consumers) by partitions ○ The number of function instances is limited by the number of partitions
- 38. Integrated with pulsar-admin CLI
- 40. Pulsar Functions Architecture ● Stream-First Design ● Leverage existing infrastructure, no more external dependencies ● Core components ○ Function metadata management ○ Function Worker membership and coordination ○ Function Assignment
- 41. Function metadata management ● Store function metadata ● Key/value store backed by a Pulsar topic ○ Function FQFN as the store key ● Use compaction to compact function metadata store
- 42. Function Worker Membership ● Manage the memberships of function workers ● Every function worker subscribes to a coordination topic ● Pulsar broker tracks the alive consumers for a subscription ● Pulsar queries the list of function workers by querying the alive consumers of the subscription to coordination topic
- 43. Function Assignment ● Function workers elect a leader as the scheduler manager by using “Failover” subscription ● The scheduler manager computes the assignment using function metadata and membership ● The assignments are published to Assignment Topic ● Each function worker receives assignment by subscribing to Assignment Topic
- 44. Function Runtime Manager ● Function Runtime Manager manages the running function instances ● It receives assignment from Assignment Topic ● It compares its current running state with the assignments and react to the assignments ○ Start function instances to invoke functions ○ Stop function instances
- 45. Use Cases
- 46. Content Routing
- 48. Transformation
- 50. Complex Event Processing Pipelines
- 51. Pulsar Functions Summary ● A serverless approach to do event streaming ● Flexible, lightweight, easy to understand and use ● Event-first, Stream-first ● Stateless + Stateful (*) ● Flexible runtime and data locality ● Functions can be orchestrated to do complex processing ○ Workflow, DAG, Iterations, Graph, and ...
- 52. Pulsar Functions Roadmap ● More languages support ● Function Orchestration ● Managed state vs Local state ● Large state ● Transactional Processing ● RL on Functions ● ...
- 53. Community ● Twitter: @apache_pulsar , @streamnativeio ● Wechat: ApachePulsar, StreamNative ● Mailing Lists: [email protected] [email protected] ● Slack: https://apache-pulsar.slack.com/ ● Github: ○ https://github.com/apache/pulsar ○ https://github.com/apache/bookkeeper ● Documentation: https://pulsar.apache.org
- 54. Thanks!