G-Research - Privacera - Dataworks Summit 2018
•Download as PPTX, PDF•
1 like•306 views
Data governance and security with Atlas, Ranger and Privacera.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to G-Research - Privacera - Dataworks Summit 2018
Similar to G-Research - Privacera - Dataworks Summit 2018 (20)
Recently uploaded
Recently uploaded (20)
G-Research - Privacera - Dataworks Summit 2018
- 1. Fintech case study on Automated Discovery, Control and Monitoring Data
- 3. AGENDA ▸Introduction ▸Challenges with big data ▸G-Research overview ▸Journey with HDP ▸Challenges and learnings ▸Recommendation
- 4. PARTNE RSGLOBAL SOLUTIONS FOR DISCOVERING AND MANAGING SENSITIVE DATA ABOUT PRIVACERA
- 5. DETECT MALICIOUS OR ACCIDENTAL USE CONTROL ANONYMIZE DATA/ RESTRICT ACCESS DISCOVER WHAT TYPE OF DATA STORED AND WHERE? REPORT SECURITY AND COMPLIANCE REPORTING SOLUTIONS FOR MANAGING SENSITIVE DATA
- 6. CHALLENGES WITH BIG DATA ▸High volume of data ▸Stringent security and privacy requirements ▸Multi-tenant environments ▸Multiple methods for accessing data ▸Business users using newer tools ▸Traditional tools for security and governance “retrofitted” for the modern data architecture
- 7. DATA DISCOVERY ACCESS CONTROL ANONYMIZATION MONITORING 4 STEPS FOR MANAGING SENSITIVE DATA
- 9. ABOUT G-RESEARCH ▸Quantitative research technology company ▸Statistical analysis and Big Data pipelines to recognise patterns and extract insights from very large market datasets ▸Forecasting analytics to predict variances in financial markets ▸Clients operate in capital markets globally ▸Undergoing very aggressive growth and adoption of cutting edge technology
- 10. DATASETS ▸Market Data ▸Level 1 (top of book) ▸Basic market data (instrument, bid price, bid size, ask price, ask size) ▸Level 2 (order book or market depth) ▸Richer data (highest bid prices, lowest ask prices)
- 11. DATASETS ▸Market Data ▸Level 1 (top of book) ▸Level 2 (order book or market depth) ▸Often represented as incremental updates at nanosecond granularity ▸HUGE dataset! Time Quantity Bid Ask Quantity Time 12:32:16 120 12.25 12.25 150 12:32:55 12:31:01 50 12.26 12.27 60 12:31:19 12:33:45 150 12.25 12.27 100 12:34:27
- 12. DATASETS ▸Other datasets include ▸Datapacks for additional enrichment ▸Risk analysis on positions, portfolios and strategies ▸Reference data about markets structures and corporate entities ▸News feeds ▸…
- 13. HADOOP DATA PLATFORM – REASONS High Volume Fast Processing Multi Tenant Flexibility
- 14. TECHNOLOGY STACK
- 15. CORE REQUIREMENTS FOR DATA PLATFORM Security ▸Protect intellectual property of the company ▸Datasets processed through the pipeline ▸...but also code Integration ▸Integration with existing security systems and policies ▸Authentication ▸Encryption ▸Strict and flexible authorization
- 16. CORE REQUIREMENTS.. CONTD.. Governance ▸Governance ▸Ability to find data easily enabling collaboration ▸Track data changes, impact, lineage and maintain consistency Multi Tenancy and Scalability ▸Multi-tenancy ▸Variety of development teams need to work on the the same platform and share data and resources ▸Scalability ▸Explosive data growth
- 17. IMPORTANCE OF GOVERNANCE - Someone who’s been quoted about a billion times “Metadata is the new Data”
- 18. GOVERNANCE AND SECURITY - SOLUTION DESIGN
- 19. GOVERNANCE AND SECURITY – SECURITY FOUNDATION Authentication Authorization Auditing Data Protection Kerberos + Knox Ranger + Knox Ranger HDFS Encryption
- 20. GOVERNANCE AND SECURITY – MANAGING RESTRICTED DATA Data Discovery Access Control Anonymization Monitoring Privacera + Atlas Ranger tag based policies Ranger Dynamic Masking Privacera Custom Spark Lineage
- 21. ARCHITECTURE
- 22. CUSTOM METADATA IN ATLAS ▸Custom datasets Atlas metamodel definitions Type spark_job id cardinality indexable operations cardinality indexable input_data cardinality indexable output_data cardinality indexable Entity Type spark_join_ operation Type string Type spark_dataset Type Dataset Type Process Type spark_operation Attributes Type spark_entity Type Referenceable
- 23. OUR JOURNEY IN DATA SECURITY ▸Datazone definition to capture data movement ▸Advanced data discovery and tagging ▸Custom lineage applied on our own data types Standard Ranger Policies Tag-based Ranger Policies Comprehensive Tag-based Ranger Policies Privacera Advanced Security Policies Atlas OoB Custom Atlas metamodels Privacera ▸Access management through data tags ▸Basic Ranger policies t=0 t=1 t=2 t=3
- 24. OUR JOURNEY IN SECURITY MANAGEMENT - MONITORING PUBLIC DATAZONE RESTRICTED DATAZONE TABLE A TABLE B FOLDER A FOLDER B TABLE C TABLE D FOLDER C FOLDER D “Sensitive tags not allowed” “Sensitive tags are OK”
- 25. KEY CHALLENGES AND LEARNINGS ▸Technical challenges ▸GraphDB does not scale as metastore ▸~100k’s entities tagged per week ▸Back-end rewritten to only use HBase + Solr ▸Open source flexibility can be a 2-sided coin ▸Business challenges ▸Business process integration
- 26. SUMMARY ▸Understand your data before expanding your data lake ▸Invest in automated classification and centralized metadata ▸Manage user access through data classification ▸Anonymise data to reduce exposure ▸Monitor the use of data - “trust but verify”
- 27. WE’RE