This document discusses data encryption in Hadoop. It describes two common cases for encrypting data: using a Crypto API to encrypt/decrypt with an AES key stored in a keystore, and encrypting MapReduce outputs using a CryptoContext. It also covers the Hadoop Encryption Framework APIs, HBase encryption via HBASE-7544, and related JIRAs around Hive and Pig encryption. Key management tools like keytool and potential future improvements like Knox gateway integration are also mentioned.
Securing Big Data at rest with encryption for Hadoop, Cassandra and MongoDB o...Big Data Spain
This session shows how to secure different Big Data sensitive data items such as log files, metastore databases, control files, config files, data directories or data files for different Big Data technologies.
As Hadoop, MongoDB, Cassandra and other massively distributed Big Data stores grow in popularity, so too does the volume of sensitive regulatory data that gets captured for analysis. Cloudera Navigator Encrypt gives peace of mind, knowing the sensitive information used to run massive-scale queries and analytics is secure. Navigator Encrypt works as a last line of defense for protecting data, by providing a transparent layer between the application and file system and securing information as it gets written to disk, ensuring minimal performance lag in the encryption or decryption process. The solution also includes robust key management and process-based access controls, while simultaneously preventing admins or super users like root from accessing data that they don’t need to see allowing users to store their cryptographic keys separate from the encrypted data.
Session presented at Big Data Spain 2015 Conference
15th Oct 2015
Kinépolis Madrid
http://www.bigdataspain.org
Event promoted by: http://www.paradigmatecnologico.com
Abstract: http://www.bigdataspain.org/program/thu/slot-13.html
An overview of securing Hadoop. Content primarily by Balaji Ganesan, one of the leaders of the Apache Argus project. Presented on Sept 4, 2014 at the Toronto Hadoop User Group by Adam Muise.
Overview of Hadoop security (revise from presentation in Hadoop in Taiwan, 2012). Detail configuration of security infrastructure leveraging kerberos and also extensive integration with LDAP aiming for fast exchange of cluster information. Introduction also Etu Appliance end of the slide.
Securing Big Data at rest with encryption for Hadoop, Cassandra and MongoDB o...Big Data Spain
This session shows how to secure different Big Data sensitive data items such as log files, metastore databases, control files, config files, data directories or data files for different Big Data technologies.
As Hadoop, MongoDB, Cassandra and other massively distributed Big Data stores grow in popularity, so too does the volume of sensitive regulatory data that gets captured for analysis. Cloudera Navigator Encrypt gives peace of mind, knowing the sensitive information used to run massive-scale queries and analytics is secure. Navigator Encrypt works as a last line of defense for protecting data, by providing a transparent layer between the application and file system and securing information as it gets written to disk, ensuring minimal performance lag in the encryption or decryption process. The solution also includes robust key management and process-based access controls, while simultaneously preventing admins or super users like root from accessing data that they don’t need to see allowing users to store their cryptographic keys separate from the encrypted data.
Session presented at Big Data Spain 2015 Conference
15th Oct 2015
Kinépolis Madrid
http://www.bigdataspain.org
Event promoted by: http://www.paradigmatecnologico.com
Abstract: http://www.bigdataspain.org/program/thu/slot-13.html
An overview of securing Hadoop. Content primarily by Balaji Ganesan, one of the leaders of the Apache Argus project. Presented on Sept 4, 2014 at the Toronto Hadoop User Group by Adam Muise.
Overview of Hadoop security (revise from presentation in Hadoop in Taiwan, 2012). Detail configuration of security infrastructure leveraging kerberos and also extensive integration with LDAP aiming for fast exchange of cluster information. Introduction also Etu Appliance end of the slide.
Securing Hadoop's REST APIs with Apache Knox Gateway Hadoop Summit June 6th, ...Kevin Minder
Securing Hadoop's REST APIs with Apache Knox Gateway
Presented at Hadoop Summit on June 6th, 2014
Describes the overall roles the Apache Knox Gateway plays in Hadoop security and briefly covers its primary features.
Hadoop and Kerberos: the Madness Beyond the Gate: January 2016 editionSteve Loughran
An update of the "Hadoop and Kerberos: the Madness Beyond the Gate" talk, covering recent work "the Fix Kerberos" JIRA and its first deliverable: KDiag
As Hadoop becomes a critical part of Enterprise data infrastructure, securing Hadoop has become critically important. Enterprises want assurance that all their data is protected and that only authorized users have access to the relevant bits of information. In this session we will cover all aspects of Hadoop security including authentication, authorization, audit and data protection. We will also provide demonstration and detailed instructions for implementing comprehensive Hadoop security.
A comprehensive overview of the security concepts in the open source Hadoop stack in mid 2015 with a look back into the "old days" and an outlook into future developments.
Securing your Pulsar Cluster with Vault_Chris KelloggStreamNative
Learn how to secure a Pulsar cluster with Hashicorp Vault and deploy it on Kubernetes. Vault provides a secure way to generate tokens and store sensitive data and Pulsar has a pluggable architecture for authentication, authorization and secret management. This talk will walk through how to create custom plugins for Vault, integrate them with Pulsar and then deploy a Pulsar cluster on Kubernetes.
Deploying enterprise grade security for Hadoop with Apache Sentry (incubating).
Apache Hive is deployed in the vast majority of Hadoop use cases despite the major practical flaws in it's most secure operational mode (Kerberos + User Impersonation).
In this talk we will discuss these flaws and how Apache Sentry addresses them. We will then enable Apache Sentry on a existing cluster. Additional topics will include Hadoop security and Role Based Access Control (RBAC).
In a dynamic infrastructure world, let's stop pretending credentials aren't public knowledge in an organization and just assume that they have already been leaked, now what?
Securing Hadoop's REST APIs with Apache Knox Gateway Hadoop Summit June 6th, ...Kevin Minder
Securing Hadoop's REST APIs with Apache Knox Gateway
Presented at Hadoop Summit on June 6th, 2014
Describes the overall roles the Apache Knox Gateway plays in Hadoop security and briefly covers its primary features.
Hadoop and Kerberos: the Madness Beyond the Gate: January 2016 editionSteve Loughran
An update of the "Hadoop and Kerberos: the Madness Beyond the Gate" talk, covering recent work "the Fix Kerberos" JIRA and its first deliverable: KDiag
As Hadoop becomes a critical part of Enterprise data infrastructure, securing Hadoop has become critically important. Enterprises want assurance that all their data is protected and that only authorized users have access to the relevant bits of information. In this session we will cover all aspects of Hadoop security including authentication, authorization, audit and data protection. We will also provide demonstration and detailed instructions for implementing comprehensive Hadoop security.
A comprehensive overview of the security concepts in the open source Hadoop stack in mid 2015 with a look back into the "old days" and an outlook into future developments.
Securing your Pulsar Cluster with Vault_Chris KelloggStreamNative
Learn how to secure a Pulsar cluster with Hashicorp Vault and deploy it on Kubernetes. Vault provides a secure way to generate tokens and store sensitive data and Pulsar has a pluggable architecture for authentication, authorization and secret management. This talk will walk through how to create custom plugins for Vault, integrate them with Pulsar and then deploy a Pulsar cluster on Kubernetes.
Deploying enterprise grade security for Hadoop with Apache Sentry (incubating).
Apache Hive is deployed in the vast majority of Hadoop use cases despite the major practical flaws in it's most secure operational mode (Kerberos + User Impersonation).
In this talk we will discuss these flaws and how Apache Sentry addresses them. We will then enable Apache Sentry on a existing cluster. Additional topics will include Hadoop security and Role Based Access Control (RBAC).
In a dynamic infrastructure world, let's stop pretending credentials aren't public knowledge in an organization and just assume that they have already been leaked, now what?
HDP Advanced Security: Comprehensive Security for Enterprise HadoopHortonworks
With the introduction of YARN, Hadoop has emerged as a first class citizen in the data center as a single Hadoop cluster can now be used to power multiple applications and hold more data. This advance has also put a spotlight on a need for more comprehensive approach to Hadoop security.
Hortonworks recently acquired Hadoop security company XA Secure to provide a common interface for central administration of security policy and coordinated enforcement across authentication, authorization, audit and data protection for the entire Hadoop stack.
In this presentation, Balaji Ganesan and Bosco Durai (previously with XA Secure, now with Hortonworks) introduce HDP Advanced Security, review a comprehensive set of Hadoop security requirements and demonstrate how HDP Advanced Security addresses them.
Reference architecture for Internet of ThingsSujee Maniyam
What kind of a data infrastructure is needed, to support Internet of Things?
This talk presents a reference architecture.
We are actually building this architecture as open source project. See here : bit.ly / iotxyz
Protecting Enterprise Data in Apache HadoopOwen O'Malley
From Hadoop Summit 2015, San Jose
From Apache BigData 2016, Vancouver
Hadoop has long had strong authentication via integration with Kerberos, authorization via User/Group/Other HDFS permissions, and auditing via the audit log. Recent developments in Hadoop have added HDFS file access control lists, pluggable encryption key provider APIs, HDFS snapshots, and HDFS encryption zones. These features combine to give important new data protection features that every company should be using to protect their data. This talk will cover what the new features are and when and how to use them in enterprise production environments. Upcoming features including columnar encryption in the ORC columnar format will also be covered.
Hadoop has long had strong authentication via integration with Kerberos, authorization via user/group/other HDFS permissions and auditing via the audit log. Recent developments in Hadoop have added HDFS file access control lists, pluggable encryption key provider APIs, HDFS snapshots, and HDFS encryption zones. These features combine to given important new data protection features that every company should be using to protect their data. This talk will cover what the new features are and when and how to use them in enterprise production environments. Upcoming features including columnar encryption in the ORC file format will also be covered.
Risk Management for Data: Secured and GovernedCloudera, Inc.
Cloudera Tech Day Presentation by Eddie Garcia, Chief Security Architect, Cloudera. Protecting enterprise data is an increasingly complex challenge given the diversity and sophistication of threat actors and their cyber-tactics. In this session, participants will hear a comprehensive introduction to Hadoop Security, including the “three A’s” for secure operating environments: Authentication, Authorization, and Audit. In addition, the presenter will cover strategies to orchestrate data security, encryption, and compliance, and will explain the Cloudera Security Maturity Model for Hadoop. Attendees will leave with a greater understanding of how effective INFOSEC relies on an enterprise big data governance and risk management approach.
Data in Hadoop is getting bigger every day, consumers of the data are growing, organizations are now looking at making their Hadoop cluster compliant to federal regulations and commercial demands. Apache Ranger simplifies the management of security policies across all components in Hadoop. Ranger provides granular access controls to data.
The deck describes what security tools are available in Hadoop and their purpose then it moves on to discuss in detail Apache Ranger.
Discover HDP 2.2: Comprehensive Hadoop Security with Apache Ranger and Apache...Hortonworks
This presentation was included in a 30-minute webinar Balaji Ganesan, Hortonworks senior director for enterprise security strategy and Vinay Shukla, director of product management.
They discussed Hortonworks Data Platform 2.2’s features for delivering comprehensive security in HDP.
Balaji and Vinay discussed Apache Ranger and Apache Knox and how they are integrated in HDP 2.2 to provide fine grain authorization, auditing and API security that can be centrally administered.
Technical introduction into Apache Spark - the Swiss Army Knife of Big Data analytics tools.
The talk was held at the Big Data User Group Mannheim, Germany at 24.11.2014.
The Future of Hadoop Security - Hadoop Summit 2014Cloudera, Inc.
Hadoop deployments are rapidly moving from pilots to production, enabling unprecedented opportunity to build big data applications that deliver faster access to more information to more users than ever before possible. Yet without the ability to address data security and compliance regulations, Hadoop will be limited to another data silo.
In this talk, Matt Brandwein and David Tishgart discuss the requirements for securing Hadoop and how Cloudera (now with Gazzang) and Intel are collaborating in the open to deliver comprehensive, transparent, compliance-ready security to unlock the potential of the Hadoop ecosystem and enable innovation without compromise.
The only way to get where we need to be in security analysis is if we use Security Intelligence. This means working harder and understanding the big picture of your data.
Chickens & Eggs: Managing secrets in AWS with Hashicorp VaultJeff Horwitz
Presented to the Philly DevOps Meetup November 29, 2016.
Managing secrets is hard. It’s even harder in the cloud. At Jornaya (formerly LeadiD), we chose Hashicorp Vault to manage our secrets in AWS, and I’d like to share our experience with everyone.
Securing Microservices using Play and Akka HTTPRafal Gancarz
Going down the microservices route makes a lot of things around creating and maintaining large systems easier but it comes at a cost too, particularly associated with challenges around security. While securing monolithic applications was a relatively well understood area, the same can't be said about microservice based architectures.
This presentation covers how implementing microservices affects the security of distributed systems, outlines pros and cons of several standards and common practices and offers practical suggestions for securing microservice based systems using Play and Akka HTTP.
Every enterprise system has tons of sensitive data like database passwords or third-party API keys. Quite often people store this data openly in internal repositories, continuous integration pipeline or configuration managements systems. The bigger company the stricter security rules. It is more complex and important when you have thousands of different applications and each one has its own secrets. In this talk I am giving an overview of my personal experience on Vault technology and will show by example how you can build your own policies and move your secrets to the Vault.
(Stephane Maarek, DataCumulus) Kafka Summit SF 2018
Security in Kafka is a cornerstone of true enterprise production-ready deployment: It enables companies to control access to the cluster and limit risks in data corruption and unwanted operations. Understanding how to use security in Kafka and exploiting its capabilities can be complex, especially as the documentation that is available is aimed at people with substantial existing knowledge on the matter.
This talk will be delivered in a “hero journey” fashion, tracing the experience of an engineer with basic understanding of Kafka who is tasked with securing a Kafka cluster. Along the way, I will illustrate the benefits and implications of various mechanisms and provide some real-world tips on how users can simplify security management.
Attendees of this talk will learn about aspects of security in Kafka, including:
-Encryption: What is SSL, what problems it solves and how Kafka leverages it. We’ll discuss encryption in flight vs. encryption at rest.
-Authentication: Without authentication, anyone would be able to write to any topic in a Kafka cluster, do anything and remain anonymous. We’ll explore the available authentication mechanisms and their suitability for different types of deployment, including mutual SSL authentication, SASL/GSSAPI, SASL/SCRAM and SASL/PLAIN.
-Authorization: How ACLs work in Kafka, ZooKeeper security (risks and mitigations) and how to manage ACLs at scale
Cryptography 101 for Java Developers - JavaZone2019Michel Schudel
So you're logging in to your favorite crypto currency exchange over https using a username and password, executing some transactions, and you're not at all surprised that, security wise, everything's hunky dory...
Ever wondered about the amount of cryptography begin used here? No? Let's dive into the key concepts of cryptography then, and see how the JDK supports this using the standard cryptography API's: JCA (Java Cryptography Architecture) and JCE (Java Cryptography Extension)! We'll be exploring message digests, encryption, and digital signatures, and see how they'are used in password checks, https, and block chain technology.
After this session, you'll have a better understanding of basic cryptography, its applications, and how to use the cryptography APIs in Java.
AWS offers you the ability to add additional layers of security to your data at rest in the cloud, providing access control as well scalable and efficient encryption features. Flexible key management options allow you to choose whether to have AWS manage the encryption keys or to keep complete control over the keys yourself. In this session, you will learn how to secure data when using AWS services. We will discuss data encryption using Key Management Service, S3 access controls, edge and host access security, and database platform security features.
Infrastructure as Code: Manage your Architecture with GitDanilo Poccia
With the AWS Cloud you have an on-demand, programmable infrastructure that you can manage using tools and practices from software development. You can create resources when you need and dispose of them when you don’t. Using Amazon CloudFormation you can describe your architecture in text files. To change your infrastructure, you edit those files. Having application and infrastructure code in a single, robust, versioned repository like Git gives a lot of advantages. Using AWS Elastic Beanstalk you can link your Git branches to different infrastructure environments (e.g. test, production) and automate deployments. You can create test environments on-demand, even for a short time. Instead of continuously update your resources, you can recreate them quickly from scratch, simplifying lifecycle management and making deployments immutable. As a result, you have more time to focus on the unique features of your application.
Packer and TerraForm are fundamental components of Infrastructure as Code. I recently gave a talk at a DevOps meetup, which allowed me the opportunity to discuss the basics of these two tools, and how DevOps teams should be using them
Toni de la Fuente - Automate or die! How to survive to an attack in the Cloud...RootedCON
Los procedimientos relacionados con Respuesta a Incidentes y Análisis Forense son diferentes en la nube respecto a cuando se realizan en entornos tradicionales, locales. Veremos las diferencias entre el análisis forense digital tradicional y el relacionado con sistemas en la nube de AWS, Azure o Google Compute Platform. Cuando se trata de la nube y nos movemos en un entorno totalmente virtual nos enfrentamos a desafíos que son diferentes al mundo tradicional. Lo que antes era hardware, ahora es software. Con los proveedores de infraestructura en la nube trabajamos con APIs, creamos, eliminamos o modificamos cualquier recurso con una llamada a su API. Disponemos de balanceadores, servidores, routers, firewalls, bases de datos, WAFs, sistemas de cifrado y muchos recursos más a sin abrir una caja y sin tocar un cable. A golpe de comando. Es lo que conocemos como Infraestructura como código. Si lo puedes programar, lo puedes automatizar. ¿Como podemos aprovecharnos de ello desde el punto de vista de la respuesta a incidentes, análisis forense o incluso hardening automatizado?
Security in IaaS, attacks, hardening, incident response, forensics and all about its automation. Despite I will talk about general concept related to AWS, Azure and GCP, I will show specific demos and threats in AWS and I will go in detail with some caveats and hazards in AWS.
MongoDB .local Bengaluru 2019: New Encryption Capabilities in MongoDB 4.2: A ...MongoDB
Many applications with high-sensitivity workloads require enhanced technical options to control and limit access to confidential and regulated data. In some cases, system requirements or compliance obligations dictate a separation of duties for staff operating the database and those who maintain the application layer. In cloud-hosted environments, certain data are sometimes deemed too sensitive to store on third-party infrastructure. This is a common pain for system architects in the healthcare, finance, and consumer tech sectors — the benefits of managed, easily expanded compute and storage have been considered unavailable because of data confidentiality and privacy concerns.
This session will take a deep dive into new security capabilities in MongoDB 4.2 that address these scenarios, by enabling native client-side field-level encryption, using customer-managed keys. We will review how confidential data can be securely stored and easily accessed by applications running on MongoDB. Common query design patterns will be presented, with example code demonstrating strong end-to-end encryption in Atlas or on-premise. Implications for developers and others designing systems in regulated environments will be discussed, followed by a Q&A with senior MongoDB security engineers.
Intelligent Cloud Conference 2018 - Building secure cloud applications with A...Tom Kerkhove
It is not a secret that it is hard to manage sensitive information. Azure Key Vault allows you to securely store this kind of information ranging from secrets & certificates to cryptographic keys.
Great! But how do you use it? How do I authenticate with it and how do I build robust applications with it?
Come join me and I'll walk you through the challenges and give you some recommendations.
Tcloud Computing Hadoop Family and Ecosystem Service 2013.Q2tcloudcomputing-tw
The presentation is designed for those interested in Hadoop technology, and can enhance your knowledge in Hadoop, such as community history, current development status, features of services, distributed computing framework and scenario of big data development in Enterprise.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
GridMate - End to end testing is a critical piece to ensure quality and avoid...ThomasParaiso2
End to end testing is a critical piece to ensure quality and avoid regressions. In this session, we share our journey building an E2E testing pipeline for GridMate components (LWC and Aura) using Cypress, JSForce, FakerJS…
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
4. WITHOUT KERBEROS
• Authorization
Ensuring the user can only do things that they are allowed to do
• Yes: Owner/Group Permission
• Authentication
Ensuring the user is who they claim to be
• NO
10. HADOOP GATEWAY - NOW
• Webhdfs
• Rest: curl "http://GATEWAYHOST/webhdfs/v1/PATH?[user.name=USER&]op=…”
• Hadoop: hadoop fs -fs webhdfs://GATEWAYHOST:14000 -cat FILe_PATH
• Oozie
• REST API , supports direct submission of MapReduce, Pig, and Hive jobs
• Steps
• Use webhdfs to upload your files and jars
• create an oozie workflow
• Hbase
• Hbase Stargate Rest Gateway
• Hbase Thrift server
11. HADOOP GATEWAY - FUTURE
• Apache Knox Gateway
Provides a single point of authentication and access for Apache™ Hadoop® services in
a cluster
12. HADOOP GATEWAY - FUTURE
• Apache Knox Gateway
• Integrate with the existing frameworks for Active Directory /LDAP
• Shell and Rest Interface support
• Currently working on kerberized cluster support
13.
14. HADOOP DATA ENCRYPTION
• Disk Encryption
• Partition Encryption dm-crypt
• File System Encryption
• Folder Encryption encryptfs
• Hadoop Encryption Framework
• Just encrypt what it should be
16. HADOOP ENCRYPTION FRAMEWORK - MR
File Map File Reduce
HDFS
HDFS
File
Encryption/Decryption All the Path(Stages)
17. JIRAS
• hadoop-9331: Hadoop crypto codec framework and crypto codec implementations
• hadoop-9332: Crypto codec implementations for AES
• hadoop-9333: Hadoop crypto codec framework based on compression codec
• mapreduce-5025: Key Distribution and Management for supporting crypto codec in
Map Reduce
• hbase-7544: Transparent table/CF encryption
18. Brief
• Two Crypto Typical Case in Hadoop
• Crypto API Case: Using AES Key (Store in KeyStore) to Encrypt/Decrypt Data
• MR CryptoContext Case: Encrypt the MR output
• Tool – Distcrypto
• Hbase Encryption
• Other Related JIRAs and Security Key Store(Manager)
• TODOs
19. KEY STORE TOOL - KEYTOOL
A key and certificate management utility.
• Create & Store an AES key
• keytool -keystore /tmp/hbase.jks -storetype jceks -storepass 123456 -genseckey -
keyalg AES -keysize 256 -alias hbase
• Create & Store an RSA Private Key
• keytool -genkey -keyalg RSA -keysize 2048 -storetype jceks -storepass 123456 -
keystore privateKeyStore.jks -alias testPrivate
• Export Certificate from KeyStore to a cert file
• keytool -export -keystore privateKeyStore.jks -storetype jceks -storepass 123456 -
alias testPrivate -file publicKey.crt
• Import a cert file to a KeyStore
• keytool -import -trustcacerts -file publicKey.crt -storetype jceks -storepass 123456 -
alias testPublic -keystore publicKeyStore.jks
21. CRYPTO API CASE: USING AES KEY (STORE IN
KEYSTORE) TO ENCRYPT/DECRYPT DATA
Use Crypto API to retrieve AES secret key from a key store file and use the key to
encrypt/decrypt data
• KeyProvider
• CryptoContext
• CryptoCodec
• Sample Code
23. CryptoContext
• To store key related info
• Key Attributes
• Raw Key Data
• Key Type: SYMMETRIC_KEY, PUBLIC_KEY, PRIVATE_KEY, CERTIFICATE
• Cryptographic Algorithm: e.g AES
• Cryptographic Length
24. CryptoCodec
• A wrap, contain CryptoContext and provide Crypto IO Stream
• Major member
• CryptoContext
• Crypto IO Stream Method
• createOutputStream(……)
• createInputStream(……)
30. CryptoContextProvider
Provide several static helper methods to update Crypto related Job Configurations. For
example, store the following Parameters and Secrets to the Job Credential in the secret key
list
• mapred.[[[STAGE]]].crypto.context.provider.parameters
• mapred.[[[STAGE]]].crypto.context.secrets
[[[STAGE]]]: input, output, map.output
AbstractCryptoContextProvider
FileMatchCryptoContextProvider
KeyProviderCryptoContextProvider
Credentials credentials = jobConf.getCredentials();
credentials.addSecretKey(new Text("mapred.map.output.crypto.context.provider.parameters"), parameters);
credentials.addSecretKey(new Text("mapred.map.output.crypto.context.secrets"), secrets);
32. FileMatchCryptoContextProvider
Provides the ability to select the appropriate CryptoContext according to the file path
FileMatches fileMatches = new FileMatches(KeyContext.derive("12345678"));
fileMatches.addMatch("^.*/input1.intel_aes$", KeyContext.derive("1234"));
fileMatches.addMatch("^.*/input2.intel_aes$", KeyContext.derive("5678"));
FileMatchCryptoContextProvider.setInputCryptoContextProvider(jobConf,
fileMatches, null);
33. KeyProviderCryptoContextProvider
Not only include the capability of FileMatchCryptoContextProvider also provide the ability to
retrieve the Key from Key Store
FileMatches fileMatches = new FileMatches(KeyContext.refer("KEY00",
Key.KeyType.SYMMETRIC_KEY, "AES", 128));
String keyStoreFile = "file:///" + KEYSTORE_HOME + "/mr.jks";
String keyStorePassword = "12345678";
KeyProviderConfig keyProviderConfig =
KeyProviderCryptoContextProvider.getKeyStoreKeyProviderConfig(
keyStoreFile, "JCEKS", keyStorePassword, null, true);
KeyProviderCryptoContextProvider.setInputCryptoContextProvider(jobConf, fileMatches,
true, keyProviderConfig);
34. SAMPLE CODE - ENCRYPT THE MR OUTPUT
Configuration conf = new Configuration();
Job job = new Job(conf, "wordcount");
JobConf jobConf = (JobConf)job.getConfiguration();
35. SAMPLE CODE - ENCRYPT THE MR OUTPUT
Configuration conf = new Configuration();
Job job = new Job(conf, "wordcount");
JobConf jobConf = (JobConf)job.getConfiguration();
FileOutputFormat.setOutputCompressorClass(job, AESCodec.class);
jobConf.set(AESCodec.CRYPTO_COMPRESSOR,
org.apache.hadoop.io.compress.SnappyCodec);
36. SAMPLE CODE - ENCRYPT THE MR OUTPUT
- Conti
FileMatches fileMatches = new FileMatches(KeyContext.refer("KEY00",
Key.KeyType.SYMMETRIC_KEY, "AES", 256));
40. MORE IN KeyProviderCryptoContextProvider
• Using asymmetric key (RSA) to protect Parameters & Secrets
CredentialProtection credentialProtection = new CredentialProtection(jobConf,
RSACredentialProtectionCodec.class,
encryptionKeyProviderConfig, encryptionKeyName,
decryptionKeyProviderConfig, decryptionKeyName);
KeyProviderCryptoContextProvider.setInputCryptoContextProvider(
jobConf,
fileMatches,
false,
keyProviderConfig,
credentialProtection);
41. MORE IN KeyProviderCryptoContextProvider - Conti
• How to use Customized KeyProvider in KeyProviderCryptoContextProvider
String keyProviderParameters = KeyStoreKeyProvider.getKeyStoreParameterString(
keyStoreFile, keyStoreType,
keyStorePassword,
keyStorePasswordFile,
sharedPassword);
KeyProviderConfig keyProviderConfig = new KeyProviderConfig(
CustomizeKeyStoreKeyProvider.class.getName(),
keyProviderParameters);
43. TOOL – DISTCRYPTO - conti
• Source Definition File (XML format)
• src
• path
• format:
• raw
• Sequence
• the full class name of a class which implement CryptoHandler for
customized format.
• includeFilter & excludeFilter
• stripSuffix & appendSuffix
• keyClassName & valueClassName.
44. TOOL – DISTCRYPTO - conti
• Encryption Sample
• command
• hadoop distcrypto -op encrypt -ek
21EF7D7487F69A19E552C1274A9FCAC721EF7D7487F69A19E552C1274A9F
CAC7 -log /tmp/log.distcrypto.encrypt -src file:///working/crypto_encrypt.xml
• Source Definition File (crypto_encrypt.xml)
• TODO: Not support retrieve keys from key store --- Not Good
<configuration><src>
<path>/tmp/install.log</path>
<format>raw</format>
<appendSuffix>.encrypted</appendSuffix>
</src></configuration>
46. HBASE ENCRYPTION – HBASE-7544
• Introduce transparent encryption of HBase on disk data.
• Transparent encryption at the CF level
• Two-tier key architecture for consistency with best practices for this feature in the
RDBMS world
• Flexible and non-intrusive key rotation
48. HBASE ENCRYPTION – HBASE-7544
HFile
Block0
……
Block N
Meta Block0
……
Meta Block N
File Info
Data Block Index
Mwta Block Index
Fixed File Trailer
Key block data
format
1 byte ordinal
4 bytes key data length
encrypted key
data
Encryption
KeyBlock
Offset
49. HBASE-7544 SETTINGS
1. Set up the keystore with a secret key
Create a secret key of appropriate length for AES.
$ keytool -keystore /path/to/hbase/conf/hbase.jks
-storetype jceks -storepass password
-genseckey -keyalg AES -keysize 256
-alias ${USER}
Press RETURN to store the key with the same password as the store
50. HBASE-7544 SETTINGS
2. Configure HBase to use the keystore
Add this to the hbase-site.xml file:
<property>
<name>hbase.crypto.keyprovider</name>
<value>org.apache.hadoop.io.crypto.KeyStoreKeyProvider</value>
</property>
<property>
<name>hbase.crypto.keyprovider.parameters</name>
<value><![CDATA[keyStoreUrl=file:///path/to/hbase/conf/
hbase.jks&keyStoreType=JCEKS&password=password]]></value>
</property>
52. HBASE-7544
• CF key rotation
• CF key is changed by modifying the column descriptor via
HBaseAdmin.
• Then, major compaction is triggered either on the table at once or region by
region.
• Performance
• Using this AES-NI codec, HFile read and write code paths introduces an overhead
roughly on par with GZIP compression for reads, and half that as for writes.
53. OTHER RELATED JIRAS
• MAPREDUCE-4491: Encryption and Key Protection
• 4550: Key Protection : Define Encryption and Key Protection interfaces and default
implementation
• 4551: Key Protection : Add ability to read keys and protect keys in JobClient and
TTS/NodeManagers
• 4552: Encryption: Add support for PGP Encryption
• 4553: Key Protection : Implement KeyProvider to read key from a WebService Based
KeyStore
• 5025: Key Distribution and Management for supporting crypto codec in Map Reduce
54. SECURITY WEB KEYSTORE SERVER
safe (http://benoyantony.github.com/safe/)
Web service based keystore
Support ACL Per Key
Authenticates the user using SPNego
Base on Cloudera Alfredo, a Java library consisting of a client and a server components
to enable Kerberos SPNEGO authentication for HTTP.
WEB Server
(safe(alfredo))
KDC user
authorization
authentication
MR/Hbase +
WebStoreKeyProvider
55. OTHER TODOs
• Hive support
• https://issues.apache.org/jira/browse/HIVE-5207
• Support data encryption for Hive tables
• https://issues.apache.org/jira/browse/HIVE-4227
• Add column level encryption to ORC files (Created: 25/Mar/13 17:14)
• Pig support
• https://issues.apache.org/jira/browse/PIG-3289
• Encryption aware load and store functions