Securing your Pulsar Cluster with Vault_Chris Kellogg

@cckellogg #PulsarSummit
Securing Your Pulsar Cluster
with Vault
&

Chris Kellogg
Software Engineer at Splunk

Contributor to Apache Pulsar and Apache Heron committer

cckellogg
You can find me on:
cckellogg

Agenda
• Vault Overview

• Why Pulsar and Vault

• Pulsar Authentication/Authorization Model

• Creating Custom Plugins

• Packaging Custom Plugins

• Kubernetes Integration

• Demo

https://www.vaultproject.io
“Vault is a tool for securely accessing secrets. A secret is anything
that you want to tightly control access to, such as API keys,
passwords, or certificates. Vault provides a unified interface to any
secret, while providing tight access control and recording a detailed
audit log.”
What is Vault?

Vault Features
Secret Management
Authentication and Identity
Data Encryption

Why Vault
• Single source to manage secrets and tokens

• Dynamic and Revokable tokens and secrets

• Audit tracking for secrets and token

• Merges identities across providers

- LDAP, Okta, Kubernetes, AWS, GCP

• Cloud friendly

Why Pulsar and Vault
• No more forever tokens

• Revokable tokens

• Secure secret management for functions and connectors

• Supports authenticating against many trusted sources of identity

- LDAP, Okta, Kubernetes, AWS, GCP, GitHub

• Central location for all security

Default is No Security
• Produce and consume from any topic

• Modify any tenant, namespace, topic or function

• Function/Connector secrets stored as plain text in conﬁgs

• No auditing of actions

Pulsar Security Features
• TLS Encryption for traﬃc

• Authentication - validate identity

• Authorization - can user perform an action

• Data encryption between producers and consumers

Pulsar Authentication
• Responsible for determining identity of clients

• Plugin System

• Built-in Plugins

- TLS

- JWT

- Authenz

- Kerberos

Pulsar Authorization
• Determines if a client has permission to perform an action

• Plugin System

• Built-in Plugin - Role based system backed by Zookeeper

- SuperUsers

- Tenant Admins

- Actions: produce/consume/functions

Building Plugins Best Practices
• Minimize third party dependencies

• Use your own executor and threads for remote requests

• Cache responses

public class VaultAuthenticationProvider implements AuthenticationProvider {
void initialize(ServiceConfiguration config) throws IOException {};
String getAuthMethodName() { return "token" };
boolean authenticateHttpRequest(HttpServletRequest req, HttpServletResponse resp)
throws Exception {
throw new AuthenticationException("Not supported");
}
String authenticate(AuthenticationDataSource authData) throws AuthenticationException {
// Implement code to authenticate the client with vault
}
AuthenticationState newAuthState(AuthData authData,
SocketAddress remoteAddress, SSLSession sslSession) throws AuthenticationException {
// Implement code to authenticate the client with vault -
// Used in binary connections for challenges
}
}
Vault Authentication Plugin

### --- Authentication --- ###
# Enable authentication
authenticationEnabled=true
# Autentication provider name list, which is comma separated list of class names
authenticationProviders=org.apache.pulsar.vault.authentication.VaultAuthentictionProvider
# Interval of time for checking for expired authentication credentials
authenticationRefreshCheckSeconds=60
Configuring Auth Plugin
broker.conf

Client
BROKER
Vault Authentication Provider
Authentication
Service
1
5 2
4
6
3
1. Client Request with Vault Token
2. Authenticate Client
3. Token pass to Vault for Authentication
4. Vault token info returned
5. Return user identity
6. Return result to client
Pulsar Vault Authentication

Pulsar Secret Plugins
Secrets Provider
• Run in the instance

• Provides secrets through the function context api
Secrets Conﬁgurator
• Runs on the server (Broker or Function Worker)

• Determines the Secret Provider the instance should use

public interface SecretsProvider {
// Initialize the SecretsProvider.
default void init(Map<String, String> config) {}
// Fetches a secret
String provideSecret(String secretName, Object pathToSecret);
}
public class MySecretFunction implements Function<String, Void> {
@Override
public Void process(String input, Context context) throws Exception {
final String password = context.getSecret("password");
context.getLogger().info("read secret password=" + password);
return null;
}
}
Example code
SecretsProvider - Client Side Plugin

public interface SecretsProviderConfigurator {
default void init(Map<String, String> config) {}
void configureKubernetesRuntimeSecretsProvider(V1PodSpec ps, String container,
Function.FunctionDetails details;
void configureProcessRuntimeSecretsProvider(ProcessBuilder pb,
Function.FunctionDetails detailsetails);
Type getSecretObjectType();
default void doAdmissionChecks(AppsV1Api appsV1Api, CoreV1Api coreV1Api,
String ns, Function.FunctionDetails details) {}
String getSecretsProviderClassName(Function.FunctionDetails details);
Map<String, String> getSecretsProviderConfig(Function.FunctionDetails details);
}
SecretsProviderConﬁgurator - Server Side Plugin
Highlighted methods are used to setup secrets plugins on the instances

########################
# Secrets
########################
secretsProviderConfiguratorClassName: org.apache.pulsar.vault.secrets.VaultSecretsProviderConfigurator
secretsProviderConfiguratorConfig:
vaultAddress: http://localhost:8200
tokenPath: /etc/auth/token
Configuring Secret Plugins
Secrets Conﬁgurator
functions_worker.yml

Secrets Provider
public class VaultSecretsProviderConfigurator implements SecretsProviderConfigurator {
@Override
public String getSecretsProviderClassName(Function.FunctionDetails details) {
if (!isEmpty(functionDetails.getSecretsMap())) {
if (Function.FunctionDetails.Runtime.JAVA == details.getRuntime()) {
return "org.apache.pulsar.vault.secrets.VaultSecretsProvider";
} else if (Function.FunctionDetails.Runtime.PYTHON == details.getRuntime()) {
return "python_secret_provider";
}
}
return null;
}
@Override
public Map<String, String> getSecretsProviderConfig(Function.FunctionDetails details) {
final Map<String, String> secrets = new HashMap<>();
secrets.put("vaultAddress", "http://localhost:8200");
secrets.put("tokenPath", "/var/auth/token");
return secrets;
}
Configuring Secret Plugins

Java Function Instance
User Code
final String password =
context.getSecret("password");
Vault Secret Provider
1
2
3
4
1. Request secret from code
2. Secret request with token
3. Secret returned to plugin
4. Return secret value

Pulsar Kubernetes Plugins
Kubernetes Manifest Customizer

• Enables customization to the K8s function specs
Kubernetes Function Auth Provider

• Determines the auth params passed to the instances

public interface KubernetesManifestCustomizer extends RuntimeCustomizer {
default V1StatefulSet customizeStatefulSet(Function.FunctionDetails funcDetails,
V1StatefulSet statefulSet) {
return statefulSet;
}
default V1Service customizeService(Function.FunctionDetails funcDetails,
V1Service service) {
return service;
}
default String customizeNamespace(Function.FunctionDetails funcDetails,
String currentNamespace) {
return currentNamespace;
}
}
KubernetesManifestCustomizer - Server Side Plugin

public interface KubernetesFunctionAuthProvider extends FunctionAuthProvider {
public void configureAuthDataStatefulSet(V1StatefulSet sts, Optional<FunctionAuthData> o) {}
public void configureAuthenticationConfig(AuthenticationConfig config,
Optional<FunctionAuthData> o) {
** configures the client auth for the function instances
}
public Optional<FunctionAuthData> cacheAuthData(Function.FunctionDetails details,
AuthenticationDataSource s) throws Exception {
** Optional<FunctionAuthData> returned is used in configureAuthenticationConfig
}
public Optional<FunctionAuthData> updateAuthData(Function.FunctionDetails details,
Optional<FunctionAuthData> o, AuthenticationDataSource s) throws Exception {
** Optional<FunctionAuthData> returned is used in configureAuthenticationConfig
}
public void cleanUpAuthData(Function.FunctionDetails details, Optional<FunctionAuthData> o)
throws Exception {}
}
KubernetesFunctionAuthProvider - Server Side Plugin

Java Function Instance
User Code
final String password =
context.getSecret("password");
1
2
3
1. Request secret from code
2. Read secret from ﬁle
3. Return secret value
Vault Secret Provider with Vault Agent

Packaging Plugins
Where do my plugins go?

pulsar/
instances/
lib/
authentication.jar
secret-conﬁgurator.jar
secret-provider.jar
deps/
kubernetes-plugins.jar

pulsar functions
vault
zookeeper brokers bookies
proxy

Pulsar Kubernetes Pod
Pulsar Process
Vault Agent
Kubernetes JWT
4
3
2
1
1. Service Account JWT passed to Vault
for Authentication
2. Vault Token auth returned
3. Write token to ﬁle
4. Pulsar process reads token from ﬁle
Pulsar Vault Kubernetes Integration

Function Secret Configuration
tenant: "public"
namespace: "default"
name: “secrets-printer"
className: “secrets_printer”
inputs: ["public/default/secrets-trigger"]
autoAck: true
parallelism: 1
resources:
cpu: 0.5
ram: 536870912
disk: 536870912
secrets:
username:
path: "internal/data/database/config"
key: username
password:
path: "internal/data/database/config"
key: password
customRuntimeOptions: >-
{
"serviceAccountName": "pf-secrets-printer"
}
Used by the VaultKubernetesCustomizer to
add annotations for vault token and secret
injection

Function Vault Annotations
vault.hashicorp.com/role: pf-secrets-printer
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/agent-inject-token: 'true'
vault.hashicorp.com/agent-inject-secret-password: secret-path
vault.hashicorp.com/agent-inject-template-password: |
'{{- with secret "secret-path"
}}{{ .Data.data.password }}{{ end }}'

Future Enhancements
• Vault for certiﬁcate management

• Pulsar Vault authorization plugin

• Vault for data encryption

Resources
• https://pulsar.apache.org/docs/en/security-overview/

• https://pulsar.apache.org/docs/en/security-authorization/

• https://pulsar.apache.org/docs/en/security-extending/
• https://github.com/hashicorp/vault-k8s

• https://www.vaultproject.io/docs/platform/k8s/helm

• https://www.vaultproject.io/docs/platform/k8s/injector

• https://learn.hashicorp.com/vault/kubernetes/k8s-reference-architecture
Pulsar
Vault

Thank You
cckellogg
cckellogg
#PulsarSummit
Code: https://github.com/cckellogg/pulsar-vault

Securing your Pulsar Cluster with Vault_Chris Kellogg

More Related Content

Securing your Pulsar Cluster with Vault_Chris Kellogg