The community for Apache Atlas and Apache Ranger, which are foundational components for security and governance across the Hadoop stack, has spawned a robust partner ecosystem of tools and platforms. Such partner solutions build upon the extensibility offered in these platforms via open and robust APIs via integration patterns to provide innovative “better-together” capabilities. In this talk, we will showcase how the ecosystem of partners is building value-added capabilities to address GDPR based on Apache Ranger and Apache Atlas frameworks to complement the Hadoop ecosystem. The talk will showcase multiple ecosystem partner demonstrations that will include how to identify, map, and classify personal data, harvest and maintain metadata, track and map the movement of data through your enterprise, and enforce appropriate controls to monitor access and usage of personal data to help organizations address GDPR. We will also provide a short overview of Gov Ready and Sec Ready programs and how partners can benefit from the certification process as part of this program. Speakers Ali Bajwa, Principal Solutions Engineer, Hortonworks Srikanth Venkat, Senior Director Product Management, Hortonworks

1. 1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Hortonworks Inc. Dataguise Syncsort Synerscope
Ali Bajwa, Partner Solutions Subra Ramesh, SVP Marco de Jong, Director Jan-Kees Buenen, CEO
Srikanth Venkat, Product Management
DataWorks Summit - Berlin
April 2018
Partner Ecosystem Showcase For
Apache Ranger And Apache Atlas

2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Apache Ranger & Apache Atlas
Journey, Ecosystem & Partners
Hortonworks Partner Certification Program
SEC Ready & GOV Ready program
Partner Technology Showcase

3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Community Snapshot
May 2014
XASecure
Acquisition
July 2014
Enters Apache
Incubation
Nov 2014
Ranger 0.4.0
Release
July 2015
Ranger 0.5/
HDP2.3
Aug 2016
Ranger 0.6/
HDP2.5
Nov 2016
Ranger 0.6.2/
HDP2.5.3
Jan 2017
Ranger TLP
graduation!
Jun 2017
Ranger 0.7.1
/HDP2.6.1
1.0.0
Q3 2018
• Committers: 27
• Contributors from:
Ebay, MSFT, Huawei,
Pandora, Accenture,
ING, Talend, ZTE
Ranger 0.7.1/
HDP2.6.1- HDP 2.6.4
Ranger 0.7/HDP2.6
• Export/import of Policies
• $User and macros
• Plugin status tab
• “Show columns” and “describe
extended support”
• Incremental LDAP Sync
• SmartSense Metrics
• User Sync Nested LDAP Support
• Tag based Masking
• Tag Attribute Based Policy
• Hive Replication Authorization
• Hive kill query Authorization
• Default Admin Group Mapping
Apr 2017
Ranger 0.7
/HDP2.6
Oct 2017
Ranger 0.7.1++
/HDP2.6.3
Aug 2017
Ranger 0.7.1+
/HDP2.6.2

4. 4 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Apache Ranger: Ecosystem
PartnerPartner Integrations
Apache Ranger
Apache
Kafka
Native Hadoop
Service Authorizers
Azure Data Lake
Store (ADLS)*
(Future)
Authorizer
Extensions
for Non-
Hadoop
Filesystems
& Stores

5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Community Snapshot
May
2015
Apache
Atlas
Incubation
DGI group
Kickoff
Dec
2014
Apr
2017
Apache 0.8
Release
Global Financial
Company
Aug
2016
Apache 0.7
Foundation
Release
Apache Atlas 0.8.2/HDP2.6.1-2.6.4
• Business User Friendly Search &
Filtering
• Knox Token Based Auth. Support
• Knox Proxying of Atlas UI
• Tag Deletion
Apache Atlas 0.8/HDP2.6.0
• Simplified Search UI
• Simplified API
• Classification-based security for
HDFS, Kafka, HBase
• Knox SSO
• Performance/scalability
improvements
• Committers – 37
• Code contributors:
Hortonworks, IBM, Aetna, Merck, Target
Jun
2017
Atlas
Becomes
TLP!
Q4
2017
Apache 0.8.1
Release
Apache 1.0
Release
Q3
2018

6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Atlas: Current Connectors and Ecosystem
Custom
Integration
PartnerPartner
Apache Atlas
RDBMS
Apache
Kafka
Pending:

7. Dataguise, Hortonworks, and GDPR
Subra Ramesh,
SVP Products, Dataguise, Inc.
7

8. ©2017 Dataguise, Inc.
Confidential and Proprietary
Dataguise Company Background
8
 Based in Silicon Valley
 10 years since founding
 Company Focus is on Sensitive, Personal Data
 Pioneers in Data-Centric Security in Big Data (first deployed Hadoop
customer 2012)
 Wide coverage of Cloud Technologies over the last 3-4 years
 Product: DgSecure, Service: DgSecure On-Demand

9. ©2017 Dataguise, Inc.
Confidential and Proprietary
Proven technology, trusted by the world’s largest brands
9

10. ©2017 Dataguise, Inc.
Confidential and Proprietary
Dataguise Technology
10
Where does DgSecure work? On-premises Cloud
What does DgSecure do?
What Hadoop Distributions does it support?
What Databases can it support?
What Cloud Stores does it support?
What other Repositories does it support?
Detection Mask/Encrypt/Access
Control Integration
Right of Access Right of Erasure ReportingBreach Detection
SQL Server
Right to Restrict
Processing

11. ©2017 Dataguise, Inc.
Confidential and Proprietary
Dataguise has the widest automated coverage of key back-
end GDPR requirements
11
 Finding and cataloging personal data – structured and unstructured – in a wide
range of data stores
 Pseudonymizing – and where needed, anonymizing – data
 Breach detection with a focus on personal data access
 Right of access
 Right to be forgotten
 Right to restrict processing

12. ©2017 Dataguise, Inc.
Confidential and Proprietary
Dataguise support of the Hortonworks Ecosystem
12
 Support for Detection of Personal Data in all the latest Hortonworks
distributions
 Support for Masking (35+ different options), AES, Format Preserving
Encryption in HDP
 Same level of support via direct HDFS and Hive APIs
 Support for YARN, Tez, Spark, and classical MapReduce execution engines
 Integration with Kerberos, Hortonworks TDE
 Monitoring of sensitive data access in both Hive and Hadoop
 HDInsight support
 Integration with Atlas and Ranger

13. ©2017 Dataguise, Inc.
Confidential and Proprietary
Access Control Integration: DgSecure -> Atlas -> Ranger
13
 DgSecure does detection on a continuous basis
 DgSecure pipes its results to Atlas, marking elements or columns containing
personal data via Tags
 Ranger Tag-Based Policies can be used to protect access at the level of
personal data types
 Optionally, Ranger Tag-based Masking can be used to hide
data selectively.

14. ©2017 Dataguise, Inc.
Confidential and Proprietary
DgSecure -> Atlas -> Ranger Integration
14
DgSecure Detection
Atlas Populated with
Personal Data Tags
Ranger Policies based
on tags
Access Control based
on type of Personal
Data

15. Demo

16. Thank You

17. Back-up Slides

18. ©2017 Dataguise, Inc.
Confidential and Proprietary
DgSecure Policies
18

19. ©2017 Dataguise, Inc.
Confidential and Proprietary
DgSecure Sensitive Types
19

20. ©2017 Dataguise, Inc.
Confidential and Proprietary
DgSecure Dashboard
20

21. ©2017 Dataguise, Inc.
Confidential and Proprietary
DgSecure Hadoop Results
21

22. ©2017 Dataguise, Inc.
Confidential and Proprietary
DgSecure Hive Results
22

23. ©2017 Dataguise, Inc.
Confidential and Proprietary
DgSecure Monitor Overview
23

24. ©2017 Dataguise, Inc.
Confidential and Proprietary
DgSecure GDPR DSAR Overview
24

25. ©2017 Dataguise, Inc.
Confidential and Proprietary
DgSecure Atlas Tags
25

26. ©2017 Dataguise, Inc.
Confidential and Proprietary
DgSecure Atlas Tags - Detail
26

27. ©2017 Dataguise, Inc.
Confidential and Proprietary
Ranger Tag-Based Policy (Disabled)
27

28. ©2017 Dataguise, Inc.
Confidential and Proprietary
Hive Query – fails, user doesn’t have Ranger permission
28

29. ©2017 Dataguise, Inc.
Confidential and Proprietary
Hive Query – failure details
29

30. ©2017 Dataguise, Inc.
Confidential and Proprietary
Ranger Policy – enabled
30

31. ©2017 Dataguise, Inc.
Confidential and Proprietary
Hive Query Executes Successfully
31

32. ©2017 Dataguise, Inc.
Confidential and Proprietary
Hive Query Results
32

33. Lineage in DMX-h – ingestion to the cluster
DMX-h job executes
• In the cluster Sources/Targets: HDFS, Hive, S3
• Out of the cluster Sources/Targets: Mainframe, DBMSs, local and remote FS –
Syncsort External Datasets
DMX-h job collects lineage information
• Source/Target File or Table level
DMX-h job lineage is published into Apache Atlas
• Connect with lineage published from other tools (REST)
33Syncsort Confidential and Proprietary - do not copy or distribute

34. Syncsort DMX-h Atlas Integration
34Syncsort Confidential and Proprietary - do not copy or distribute

35. Govern and Track Everything for Compliance
• Metadata and data lineage for Hive, Avro and
Parquet through HCatalog
• Metadata lineage export and API from DMX/DMX-h
– Simplify audits, analytics dashboards, metrics
– Integrate with enterprise metadata repositories
• Apache Ambari integration
– Native LDAP and Kerberos support
– Secure mainframe data access through FTPS and
Connect:Direct
• Apache Atlas ingestion lineage integration
– Audit and track data from source to cluster
– Lineage & tagging of Metadata for GDPR
Compliance
35Syncsort Confidential and Proprietary - do not copy or distribute

36. End-to-End Data Lineage in Apache Atlas
36Syncsort Confidential and Proprietary - do not copy or distribute
Data Sources

37. End-to-End Data Lineage in Apache Atlas
37Syncsort Confidential and Proprietary - do not copy or distribute
Data Sources
Syncsort accesses
data from sources
outside cluster.

38. End-to-End Data Lineage in Apache Atlas
38Syncsort Confidential and Proprietary - do not copy or distribute
Syncsort onboards
data, modifies
on-the-fly to match
Hadoop storage
model.
Data Sources
Syncsort accesses
data from sources
outside cluster.

39. End-to-End Data Lineage in Apache Atlas
39Syncsort Confidential and Proprietary - do not copy or distribute
Syncsort onboards
data, modifies
on-the-fly to match
Hadoop storage
model.
Data Sources
Syncsort accesses
data from sources
outside cluster.
Syncsort changes,
enhances, joins
data in cluster with
MapReduce or
Spark.
Data Hub

40. End-to-End Data Lineage in Apache Atlas
40Syncsort Confidential and Proprietary - do not copy or distribute
Syncsort onboards
data, modifies
on-the-fly to match
Hadoop storage
model.
Data Sources
Syncsort accesses
data from sources
outside cluster.
Syncsort changes,
enhances, joins
data in cluster with
MapReduce or
Spark.
Syncsort passes
source-to-cluster
data lineage info
to Atlas.
Data Hub

41. End-to-End Data Lineage in Apache Atlas
41Syncsort Confidential and Proprietary - do not copy or distribute
Syncsort onboards
data, modifies
on-the-fly to match
Hadoop storage
model.
Data Sources
Syncsort accesses
data from sources
outside cluster.
Syncsort changes,
enhances, joins
data in cluster with
MapReduce or
Spark.
Analytics and
visualizations
get complete
data.
Data analyst
gets end-to-
end data
lineage info
from Atlas
Syncsort passes
source-to-cluster
data lineage info
to Atlas.
Data Hub
Analytics,
Visualization

42. Syncsort: High Performance Import from Existing Databases
42
• Connect to virtually any data source, including
mainframe and MPP databases.
• Move data into and out of Hadoop up to 6x
faster without the need for manual scripts.
• Develop ETL processes without writing code.
• Seamlessly accelerate Hadoop performance and
scalability for ETL operations in both
MapReduce and Spark.
Benefits
Syncsort Confidential and Proprietary - do not copy or distribute

43. Syncsort + Hortonworks Advantages
• Apache Ambari Integration
• Deploy DMX-h across cluster
• Monitor DMX-h jobs
• Process in MapReduce or Spark
• Source relational and non relational data
(including mainframes)
• Out-of-the-box integration, interoperability &
certifications
• Kerberos-secured clusters
• Apache Ranger security certified
• Early beta, release certification
• Metadata lineage export from DMX
• Supports easy identification and management
of GDPR relevant Metadata
Technical Benefits
43Syncsort Confidential and Proprietary - do not copy or distribute

44. Demo: Apache Atlas
44Syncsort Confidential and Proprietary - do not copy or distribute

45. Demo: Apache Atlas
45Syncsort Confidential and Proprietary - do not copy or distribute

46. Demo: Apache Atlas
46Syncsort Confidential and Proprietary - do not copy or distribute

47. Demo: Apache Atlas
47Syncsort Confidential and Proprietary - do not copy or distribute

48. Demo: Apache Atlas
48Syncsort Confidential and Proprietary - do not copy or distribute

49. Demo: Apache Atlas
49Syncsort Confidential and Proprietary - do not copy or distribute

50. Demo: Apache Atlas
50Syncsort Confidential and Proprietary - do not copy or distribute

51. Demo: Apache Atlas
51Syncsort Confidential and Proprietary - do not copy or distribute

52. Demo: Apache Atlas
52Syncsort Confidential and Proprietary - do not copy or distribute

53. Demo: Apache Atlas
53Syncsort Confidential and Proprietary - do not copy or distribute

54. Demo: Apache Atlas
54Syncsort Confidential and Proprietary - do not copy or distribute

55. Demo: Apache Atlas
55Syncsort Confidential and Proprietary - do not copy or distribute

56. Demo: Apache Atlas
56Syncsort Confidential and Proprietary - do not copy or distribute

57. Demo: Apache Atlas
57Syncsort Confidential and Proprietary - do not copy or distribute

58. Demo: Apache Atlas
58Syncsort Confidential and Proprietary - do not copy or distribute

59. Demo: Apache Atlas
59Syncsort Confidential and Proprietary - do not copy or distribute

60. Demo: Apache Atlas
60Syncsort Confidential and Proprietary - do not copy or distribute

61. Demo: Apache Atlas
61Syncsort Confidential and Proprietary - do not copy or distribute

62. GDPR
Be transparent with all Pii data
Why not turn GDPR into a new
customer experience?
Dataworks Summit Berlin 2018
Jan-Kees Buenen, CEO
62(C) 2018 SynerScope

63. Discover
and classify
data
content in
full context
Know the
entire data
infrastructur
e
Know the
entire data
flows
patterns
Establish
and
execute
remediation
policies
Apply same
governance
to
processing
Monitor
through
certified
audits
“6 Steps for GDPR” expanded to unstructured enterprise data
63
Know who and what application produces and uses
Pii Data
Know the Pii data that rests in your unstructured data
Know its exact location, expiry date, consent status
Set and execute your policies based on your granular
knowledge of the content
Log every event touching your data  Atlas and
Ranger are integrated in fully automated processes in
SynerScope
Have the data instantly available at individual record
level for external (certified) audit purposes (Big4 love
sampling)

64. GDPR compliance for all content
Transparency for governance
• Data Discovery
• Data Search
• Data Matching
• Data Context
• Data Quality
• Data Use patterns
• Audit Ready (Big Four
endorsed)
64(C) 2018 SynerScope
Numbers Text
IoT Video, Audio
Eco-
system
Include the “other“ 80% of
the enterprise data

65. SynerScope Product position
for GDPR and IFRS
FAST, FLEXIBLE AND TRANSPARENT
o Fast and flexible with raw data, no cleaning, no
upfront modeling
o Fast and flexible for complex new combinations of
data brought in from many different silos
o Transparency at individual cell record level, with data
presented in full context allows for certified audits
o The big audit firms will play an important role
between the enterprise, regulators and supervisories
o Eco system demands independent certification of
data operations
Unstructured data is the Achilles Heel for true
GDPR compliance
…… “SynerScope’s Intelligence Augmentation (IA)
can handle the most complex data situations fast
and reliably” (Big 4 accounting firm)
(C) 2018 SynerScope

66. How SynerScope Ixiwa uses Atlas
ATLAS
MongoDB
IXIWA API
Push
RANGER
ATLAS
GraphDB
YARN
Launch
SPARK
content scan
TAGS
SPARK
HIVE
HDFS
(C) 2018 SynerScope

67. LIVE DEMO
(C) 2018 SynerScope

68. CONTACTS
WWW.SYNERSCOPE.COM
CEO jan-kees.Buenen@synerscope.com
FS Lead David.de.jong@synerscope.com
Thank
You
(C) 2018 SynerScope

69. 69 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
HDP SEC READY & GOV READY Programs
✔ Choice: Customers choose features that they want to deploy—a la carte
✔ Curated & Fast: Partners to provide rich, complimentary and complete features ready to
deploy
✔ Agile: Faster deployment and accelerate innovation
✔ Centralized : Open metadata/governance and security infrastructure
✔ Flexibility: Portfolio of partner reference architectures and integration patterns
✔ Safe: HDP at core to provide stability and interoperability

70. 70 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Hortonworks Certified Technology Program
HDP YARN Ready
Integrates with YARN
(native, Tez, Slider) or
uses/runs on a YARN
Ready engine
HDP Operations Ready
Integrates with Ambari
APIs, Stacks, Blueprints,
or Views
HDP Governance Ready
Integrates with Atlas
HDP Security Ready
Integrates with
Ranger, Knox, or other
security features
Sign up to be a partner and request certification kit!
http://hortonworks.com/partners/product-integration-certification/

71. 71 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Questions