GDPR Compliance: Transparent Handing of Personally Identifiable Information in Event-Driven Systems (Masih Derkani, SolarWinds) Kafka Summit 2020

1@solarwinds
GDPR Compliance: Transparent Handing of
PII in Event-Driven Systems
Masih H. Derkani
© 2020 SolarWinds Worldwide, LLC. All rights reserved.

2@solarwinds
GDPR* Compliance
Right to be forgotten
* https://bit.ly/39w1LhR
≈ within 28 Days

3@solarwinds
Personally Identifiable
Information
Email, IP Address, Name, Location, Cookies, etc.
Information relating to natural persons
who can be directly or indirectly
identified.*
* https://bit.ly/3303uuv

4@solarwinds© 2020 SolarWinds Worldwide, LLC. All rights reserved.
Enter…
Immutable, Append-only, Replicated
Logs.
Great for fault-tolerance;
not so much for GDPR Compliance.

6@solarwinds 6@solarwinds
retention.ms<2419200000

Simple to “forget” data.
Effective.
Minimal Effort.
Retention duration cannot be longer than 28 days.
Data is not removed on demand.
Cannot have per “natural person” access control.
High risk of leftover data.
But…
Retention duration
less than 28 days

9@solarwinds 9@solarwinds
Crypto-shredding
“…the practice of 'deleting' data by deliberately deleting or overwriting the encryption keys.” *
* https://bit.ly/39waDEb

Simple to “forget” data.
On demand data removal.
Fine-grained access control over data.
Lower risk of leftover data.
Medium to high engineering effort.
Extra COGS.
Requires careful key management.
Will impact throughput.
But…
Crypto-shredding

Encryption key per:
Cluster
Topic
User
Crypto-shredding

Per User
Encryption Key
6. Decrypted
Key Management System
2. Encrypted
1. Encrypt
3. SendProducer
5. Decrypt
Consumer4. Poll
Encrypted PII Payload Plain PII Payload TLS/SSL Communication Flow

Key Management
System
Producer Encrypt

Key Management
System
Consumer Decrypt

Per User
Encryption Key

Messages typically have a
schema.
Not all fields are PII.
Not all consumers care about
PII.
Per User
Encryption Key

18@solarwinds 18@solarwinds© 2020 SolarWinds Worldwide, LLC. All rights reserved.
Identify PII fields.
Build it into Message Schema.
Partially encrypt Message.

Per User
Encryption Key
ProtocolBuffer Message Schema.
syntax = "proto3";
message MyMessage {
string email = 1;
string first_name = 2;
string last_name = 3;
double purchase_value = 4;
}

Identify PII fields
…using Custom Options*
built into Message Schema
* https://bit.ly/2WZ3GGC

Identify PII fields
syntax = "proto3";
import "google/protobuf/descriptor.proto";
extend google.protobuf.FieldOptions { GDPRCompliance gdpr = 51234; }
message GDPRCompliance { bool key = 1; bool pii = 2; }
message MyMessage {
string email = 1 [(gdpr) = {pii: true, key: true}];
string first_name = 2 [(gdpr) = {pii: true}];
string last_name = 3 [(gdpr) = {pii: true}];
double purchase_value = 4;
}

Identify PII fields
private Message processPIIFields(Message message) {
message.getAllFields().forEach((fieldDescriptor, value) -> {
final FieldOptions fieldOptions = fieldDescriptor.getOptions();
final MyProtoFile.GDPRCompliance gdpr = fieldOptions.getExtension(MyProtoFile.gdpr);
if (gdpr.getKey()) {
final String fieldName = fieldDescriptor.getName();
final Type fieldType = fieldDescriptor.getType();
...
}
});
...
}

Partially encrypt Message
…using interceptor.classes*
built into Kafka Client Library
* https://bit.ly/30Uwnpc
* https://bit.ly/2X0pRw7

class MyProducerInterceptor implements ProducerInterceptor<String, Message> {
@Override
public ProducerRecord<String, Message> onSend(final ProducerRecord<String, Message> record) {
final Message value = record.value();
final Message complyingValue = processPIIFields(value);
return new ProducerRecord<>(
record.topic(),
record.partition(),
record.timestamp(),
record.key(),
complyingValue,
record.headers());
}
...
}

final Properties config = new Properties();
config.put(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG, MyProducerInterceptor.class.getName());
...
final KafkaProducer<String, MyProtoFile.MyMessage> producer = new KafkaProducer<>(config);
...

+
Contract-first PII specification.
Message Schema backward compatibility.
Transparent Kafka record interception.
Access to non-PII fields while remaining compliant;
only decrypt when needed.

-
Non-string field type not supported.
Only ProtocolBuffer Message Schema.

Future Work
Schema-agnostic partial encryption.
Utilising Kafka record Headers.
Nested messages with multiple PII Key.
POC non-Java clients.
Load test and throughput optimisation.

29@solarwinds
THANK
YOU
🍻

30@solarwinds
Q&A
?

31@solarwinds
This presentation contains forward-looking statements regarding future
product plans and development efforts. SolarWinds considers various
features and functionality prior to any final generally available release.
Information in this presentation regarding future features and
functionality is not and should not be interpreted as a commitment from
SolarWinds that it will deliver any specific feature or functionality in the
future or, if it delivers such feature or functionality, any time frame when
that feature or functionality will be delivered. All information is based
upon current product interests, and product plans and priorities can
change at any time. SolarWinds undertakes no obligation to update any
forward-looking statements regarding future product plans and
development efforts if product plans or priorities change.

32@solarwinds
The SolarWinds, SolarWinds & Design, Orion, and THWACK
trademarks are the exclusive property of SolarWinds Worldwide,
LLC or its affiliates, are registered with the U.S. Patent and
Trademark Office, and may be registered or pending registration
in other countries. All other SolarWinds trademarks, service
marks, and logos may be common law marks or are registered or
pending registration. All other trademarks mentioned herein are
used for identification purposes only and are trademarks of (and
may be registered trademarks) of their respective companies.

GDPR Compliance: Transparent Handing of Personally Identifiable Information in Event-Driven Systems (Masih Derkani, SolarWinds) Kafka Summit 2020

Recommended

Recommended

More Related Content

Similar to GDPR Compliance: Transparent Handing of Personally Identifiable Information in Event-Driven Systems (Masih Derkani, SolarWinds) Kafka Summit 2020

Similar to GDPR Compliance: Transparent Handing of Personally Identifiable Information in Event-Driven Systems (Masih Derkani, SolarWinds) Kafka Summit 2020 (20)

More from confluent

More from confluent (20)

Recently uploaded

Recently uploaded (20)

GDPR Compliance: Transparent Handing of Personally Identifiable Information in Event-Driven Systems (Masih Derkani, SolarWinds) Kafka Summit 2020