SlideShare a Scribd company logo
Serverless Event Streaming
with Pulsar Functions
Xiaolong Ran (@wolf4j1)
09/12/2019
● Go programmer
● PingCAP > Bitmain > StreamNative
● Pulsar committer
○ pulsar-client-go
○ Go Functions
○ …
● https://github.com/wolfstudy
Who am I
● What is Apache Pulsar?
● Event Stream - Pulsar view on Data
● When Event Streaming meets serverless
○ Programming Model
○ Architecture
○ Use cases
Agenda
● What is Apache Pulsar?
● Event Stream - Pulsar view on Data
● When Event Streaming meets serverless
○ Programming Model
○ Architecture
○ Use cases
Agenda
A Pulsar view on Data
Pulsar View - Topic
Pulsar View - Partition
Pulsar View - Segment
Pulsar View - Event Stream
Event Stream is the right foundation for your data
M(essaging), S(torage), P(rocessing)
MSP - Interactive Queries
MSP - Stream & Batch Processing
MSP - What is next?
What is next?
When Event Streaming
meets Serverless
Serverless Event Streaming with Pulsar Functions
Introduce Pulsar Functions
Pulsar Functions
● A serverless framework for data processing
● Lightweight computation
● Event-first, Stream-first
● Support both stateless and stateful computation
● Multi languages
● Multi runtimes
● SDK-less & SDK
Function Elements
● Input Topics
● Output Topics
● Function
● State
● Log Topics
API - Native Java / Python / Go Function
Golang
Python
Java
API - Function Context
● Logger
● State
● Metrics
● Security / Secrets
● ...
Context - Logger
Context - State
Context - State
● Global Managed State *
● Mutable by functions and admin-cli
● Queryable by functions and admin-cli
● State are stored at storage layer
● State are implemented using streams + snapshot
Context - State API
● Key/Value State API
○ putState
○ getState
● Counter State API
○ getCounter
○ incrCounter
Context - Metrics
● API - recordMetric(String metricName, double value)
● Exposed in prometheus format
● Collected by prometheus
Flexible Runtime
● Colocate with Broker - Thread & Process
● Managed Function Workers - Thread & Process
● External Schedulers - Container
○ Kubernetes
Colocate with Brokers
Managed Function Workers
External Schedulers - Kubernetes
Event Routing
● Events are routed to different partitions
● Leverage Pulsar’s MessageRouter
● Existing MessageRouters
○ Round-Robin
○ SinglePartition
○ Hash (Murmur32)
● Customize MessageRouter
Auto load balancing
● Pulsar Functions use Pulsar’s auto-balancing mechanism on consumers
● Shared Subscription
○ Load is distributed among function instances (consumers) evenly
○ More function instances provides more processing capability
● Failover Subscription
○ Load is distributed among function instances (consumers) by
partitions
○ The number of function instances is limited by the number of
partitions
Integrated with pulsar-admin CLI
Pulsar Functions Architecture
Usage Pattern
Usage Pattern
Simple Event Processing with Pulsar Functions (by David Kjerrumgaard)
Content Routing
Message Filtering
Transformation
Alert and thresholds
Complex Event Processing Pipelines
Pulsar Functions at EMQ
EMQ
● IoT Messaging Provider
● Open source software company
● Users
○ HPE
○ Ericsson
○ Huawei
○ China Mobile
○ China UnionPay
○ ...
ActorCloud
● Open Source IoT Platform
ActorCloud Streaming
Summary
Pulsar Functions Summary
● A serverless approach to do event streaming
● Flexible, lightweight, easy to understand and use
● Event-first, Stream-first
● Stateless + Stateful (*)
● Flexible runtime and data locality
● Functions can be orchestrated to do complex processing
○ Workflow, DAG, Iterations, Graph, and ...
Pulsar Functions Roadmap
● More languages support
● Function Orchestration
● Managed state vs Local state
● Large state
● Transactional Processing
● ...
Community
● Twitter: @apache_pulsar , @streamnativeio
● Mailing Lists: dev@pulsar.apache.org users@pulsar.apache.org
● Slack: https://apache-pulsar.slack.com/
● Github:
○ https://github.com/apache/pulsar
○ https://github.com/apache/bookkeeper
● Documentation: https://pulsar.apache.org
● Medium
https://medium.com/streamnative
Pulsar in Action
https://www.manning.com/books/
pulsar-in-action
The first Pulsar book by
David Kjerrumgaard
Thanks!

More Related Content

Serverless Event Streaming with Pulsar Functions