Securing and governing a multi-tenant data lake within the financial industry
•Download as PPTX, PDF•
2 likes•1,412 views
Standard Bank South Africa is a Hortonworks client, with several multi-node clusters hosting Hortonworks Data Platform (HDP) and Hortonworks Data Flow (HDF). This presentation will discuss the technical detail of implementing security, governance and multi-tenancy on a "Data Lake" within the finance industry. The talk will address the team's experiences, challenges, failures and learnings that we took away from this behemoth of an adventure. After introducing Standard Bank and the Hadoop admin team, the presentation will describe the security and governance journey Standard Bank has undergone since the project's inception in 2015, as well as the roadmap for the future ahead. Presentation structure: 1. Team introduction with background information 2. Environment overview (Where we are - Current) -----Security ---------Authentication through Kerberos and LDAP/ AD ---------Authorization through Ranger and Centrify ---------Transparent Data Encryption (TDE) at rest -----Governance ---------Centralized auditing ---------Ranger policies and data steward ownership -----Multi-Tenancy ---------Data lake Vs. data analytics platform ---------Edge nodes Vs. API framework through Knox 3. How did we get to this stage? (Past) -----Challenges faced (Kerberos, AD integration, SSL) -----How we overcame these challenges 4. Future challenges we foresee (Future) -----How we are planning to prepare for them Speakers Ian Pillay, Hadoop Administrator, Standard Bank Brad Smith, Hadoop Administrator, Standard Bank
Report
Share
Report
Share
Recommended
The Convergence of Reporting and Interactive BI on Hadoop
Since the early days of Hive, SQL on Hadoop has evolved from being a SQL wrapper on top of MapReduce to a viable replacement for the traditional EDW. In the meantime, while SQL on Hadoop vendors were busy adding enterprise capabilities and comparing their TPC-DS prowess against Hive, a niche industry emerged on the side for OLAP (a.k.a. "Interactive BI") on Hadoop data. Unlike general-purpose SQL on Hadoop engines, which deal with the multiple aspects of warehousing, including reporting, OLAP on Hadoop engines focus almost exclusively on answering OLAP queries fast by using implementation techniques that had not been part of the SQL on Hadoop toolbox so far.
But SQL on Hadoop engines are not standing still. After having made huge progress in catching up to traditional EDWs for reporting workloads, SQL on Hadoop engines are now setting their sights on Interactive BI. This is great news for enterprises: as the line between reporting and OLAP gets blurred, enterprises can now start considering using a single engine for both reporting and interactive BI on their Hadoop data, as opposed to having to host, manage and license two separate products.
Can a single engine satisfy both your reporting and Interactive BI needs? This may be a hard question to answer. Vendors use inconsistent terminology to describe their products and make ambitious and sometimes conflicting claims. This makes it very hard for enterprises to compare products, let alone decide which is the product that best matches their needs.
In this presentation, we'll provide an overview of the different approaches to OLAP on Hadoop, and explain the key technologies behind each of them. We'll use consistent terminology to describe what you get from multiple proprietary and open source products, and outline advantages and disadvantages. You'll come out equipped with the knowledge you need to read past marketing and sales pitches; you'll be able to compare products and make an informed decision on whether a single engine for both reporting and Interactive BI on Hadoop is right for you.
Speaker
Gustavo Arocena, Big Data Architect, IBM
Not Just a necessary evil, it’s good for business: implementing PCI DSS contr...
For firms in the financial industry, especially within regulated organizations such as credit card processors and banks, PCI DSS compliance has become a business and operational necessity. Although the blueprint of a PCI-compliant architecture varies from organization to organization, the mixture of modern Hadoop-based data lakes and legacy systems are a common theme.
In this talk, we will discuss recent updates to PCI DSS and how significant portions of PCI DSS compliance controls can be achieved using open source Hadoop security stack and technologies for the Hadoop ecosystem. We will provide a broad overview of implementing key aspects of PCI DSS standards at WorldPay such as encryption management, data protection with anonymization, separation of duties, and deployment considerations regarding securing the Hadoop clusters at the network layer from a practitioner’s perspective. The talk will provide patterns and practices map current Hadoop security capabilities to security controls that a PCI-compliant environment requires.
Speaker
David Walker, Enterprise Data Platform Programme Director, Worldpay
Srikanth Venkat, Senior Director Product Management, Hortonworks
Reaching scale limits on a Hadoop platform: issues and errors created by spee...
Santander UK’s Big Data journey began in 2014, using Hadoop to make the most of our data and generate value for customers. Within 9 months, we created a highly available real-time customer facing application for customer analytics. We currently have 500 different people doing their own analysis and projects with this data, spanning a total of 50 different use cases. This data, (consisting of over 40 million customer records with billions of transactions), provides our business new insights that were inaccessible before.
Our business moves quickly, with several products and 20 use cases currently in production. We currently have a customer data lake and a technical data lake. Having a platform with very different workloads has proven to be challenging.
Our success in generating value created such growth in terms of data, use cases, analysts and usage patterns that 3 years later we find issues with scalability in HDFS, Hive metastore and Hadoop operations and challenges with highly available architectures with Hbase, Flume and Kafka. Going forward we are exploring alternative architectures including a hybrid cloud model, and moving towards streaming.
Our goal with this session is to assist people in the early part of their journey by building a solid foundation. We hope that others can benefit from us sharing our experiences and lessons learned during our journey.
Speaker
Nicolette Bullivant, Head of Data Engineering at Santander UK Technology, Santander UK Technology
Inside open metadata—the deep dive
This document discusses metadata and the importance of metadata management. It introduces Apache Atlas as an open source platform for metadata management and governance. Key points include:
- Metadata is important for data reuse, analytics, and governance. It provides context and meaning about data.
- Current reality is that metadata is often not well supported or integrated across tools. Apache Atlas aims to provide an open, unified approach.
- Apache Atlas has graduated to a top-level Apache project. It provides a type-agnostic metadata store and interfaces that can be accessed by various tools.
- The vision is for an open ecosystem where metadata is shared and federated across repositories from different vendors and tools.
Multi-tenant Hadoop - the challenge of maintaining high SLAS
In shared configuration, the same Hadoop environment supports many applications. Each has
specific requirements and criticality (SLA). Yet they all rely on an assembly of shared application
bricks.
At the same time, the life cycle of a cluster is not static in time. It evolves horizontally, with the
arrival of new applications, but also vertically, as the applications grow in load or evolve in
terms of functionality.
With this in mind, a multi-tenant production cluster presents several challenges including and
not limited to:
- Maintain a high level of SLA for a set of use cases with heterogeneous needs
- Plan and implement the architecture evolution of a cluster in production to ensure the
maintenance of SLA throughout the integration of new use cases on it
EDF will present how it manages this heterogeneity of SLA inherent of any Big Data cluster. EDF
is focusing on how it is renovating its cluster, its organization, its processes and its approach in
order to deliver a platform with strong SLA throughout its life cycle.
Speaker
Edouard Rousseaux, Tech Lead, EDF
Data Mesh in Practice: How Europe’s Leading Online Platform for Fashion Goes ...
Zalando transitioned from a centralized data platform to a data mesh architecture. This decentralized their data infrastructure by having individual domains own datasets and pipelines rather than a central team. It provided self-service data infrastructure tools and governance to enable domains to operate independently while maintaining global interoperability. This improved data quality by making domains responsible for their data and empowering them through the data mesh approach.
Verizon Centralizes Data into a Data Lake in Real Time for Analytics
Verizon – Global Technology Services (GTS) was challenged by a multi-tier, labor-intensive process when trying to migrate data from disparate sources into a data lake to create financial reports and business insights. Join this session to learn more about how Verizon:
• Easily accessed data from multiple sources including SAP data
• Ingested data into major targets including Hadoop
• Achieved real-time insights from data leveraging change data capture (CDC) technology
• Reduced costs and labor
From an experiment to a real production environment
Rabobank is a worldwide food- and agri-bank from the Netherlands. Rabobank wants to make a substantial contribution to welfare and prosperity in the Netherlands and to feeding the world sustainably. Rabobank Group operates through Rabobank and its subsidiaries in 40 countries.
Rabobank is active in both retail and wholesale banking. For our wholesale clients we provide real-time business insight information by making use of Cloudera and Hortonworks technology. An example is our recently launched service that gives insight in market performance of Rabobank customers, starting with the dairy farmers market segment, by making use of benchmark information. Our current technology stack contains Hortonworks Data Flow (HDF) and Cloudera Hadoop (CDH). Our real-time data stream is implemented by making use of Kafka and Nifi from HDF. Cloudera is used to store the data needed for the business insight information, mainly in HDFS and HBase.
During our presentation we will provides insight about the project approach, the architecture and actual implementation.
Speaker
Jeroen Wolffensperger, Solution Architect Data, Rabobank
Martijn Groen, Delivery Manager Data , Rabobank Netherlands
Recommended
The Convergence of Reporting and Interactive BI on Hadoop
Since the early days of Hive, SQL on Hadoop has evolved from being a SQL wrapper on top of MapReduce to a viable replacement for the traditional EDW. In the meantime, while SQL on Hadoop vendors were busy adding enterprise capabilities and comparing their TPC-DS prowess against Hive, a niche industry emerged on the side for OLAP (a.k.a. "Interactive BI") on Hadoop data. Unlike general-purpose SQL on Hadoop engines, which deal with the multiple aspects of warehousing, including reporting, OLAP on Hadoop engines focus almost exclusively on answering OLAP queries fast by using implementation techniques that had not been part of the SQL on Hadoop toolbox so far.
But SQL on Hadoop engines are not standing still. After having made huge progress in catching up to traditional EDWs for reporting workloads, SQL on Hadoop engines are now setting their sights on Interactive BI. This is great news for enterprises: as the line between reporting and OLAP gets blurred, enterprises can now start considering using a single engine for both reporting and interactive BI on their Hadoop data, as opposed to having to host, manage and license two separate products.
Can a single engine satisfy both your reporting and Interactive BI needs? This may be a hard question to answer. Vendors use inconsistent terminology to describe their products and make ambitious and sometimes conflicting claims. This makes it very hard for enterprises to compare products, let alone decide which is the product that best matches their needs.
In this presentation, we'll provide an overview of the different approaches to OLAP on Hadoop, and explain the key technologies behind each of them. We'll use consistent terminology to describe what you get from multiple proprietary and open source products, and outline advantages and disadvantages. You'll come out equipped with the knowledge you need to read past marketing and sales pitches; you'll be able to compare products and make an informed decision on whether a single engine for both reporting and Interactive BI on Hadoop is right for you.
Speaker
Gustavo Arocena, Big Data Architect, IBM
Not Just a necessary evil, it’s good for business: implementing PCI DSS contr...
For firms in the financial industry, especially within regulated organizations such as credit card processors and banks, PCI DSS compliance has become a business and operational necessity. Although the blueprint of a PCI-compliant architecture varies from organization to organization, the mixture of modern Hadoop-based data lakes and legacy systems are a common theme.
In this talk, we will discuss recent updates to PCI DSS and how significant portions of PCI DSS compliance controls can be achieved using open source Hadoop security stack and technologies for the Hadoop ecosystem. We will provide a broad overview of implementing key aspects of PCI DSS standards at WorldPay such as encryption management, data protection with anonymization, separation of duties, and deployment considerations regarding securing the Hadoop clusters at the network layer from a practitioner’s perspective. The talk will provide patterns and practices map current Hadoop security capabilities to security controls that a PCI-compliant environment requires.
Speaker
David Walker, Enterprise Data Platform Programme Director, Worldpay
Srikanth Venkat, Senior Director Product Management, Hortonworks
Reaching scale limits on a Hadoop platform: issues and errors created by spee...
Santander UK’s Big Data journey began in 2014, using Hadoop to make the most of our data and generate value for customers. Within 9 months, we created a highly available real-time customer facing application for customer analytics. We currently have 500 different people doing their own analysis and projects with this data, spanning a total of 50 different use cases. This data, (consisting of over 40 million customer records with billions of transactions), provides our business new insights that were inaccessible before.
Our business moves quickly, with several products and 20 use cases currently in production. We currently have a customer data lake and a technical data lake. Having a platform with very different workloads has proven to be challenging.
Our success in generating value created such growth in terms of data, use cases, analysts and usage patterns that 3 years later we find issues with scalability in HDFS, Hive metastore and Hadoop operations and challenges with highly available architectures with Hbase, Flume and Kafka. Going forward we are exploring alternative architectures including a hybrid cloud model, and moving towards streaming.
Our goal with this session is to assist people in the early part of their journey by building a solid foundation. We hope that others can benefit from us sharing our experiences and lessons learned during our journey.
Speaker
Nicolette Bullivant, Head of Data Engineering at Santander UK Technology, Santander UK Technology
Inside open metadata—the deep dive
This document discusses metadata and the importance of metadata management. It introduces Apache Atlas as an open source platform for metadata management and governance. Key points include:
- Metadata is important for data reuse, analytics, and governance. It provides context and meaning about data.
- Current reality is that metadata is often not well supported or integrated across tools. Apache Atlas aims to provide an open, unified approach.
- Apache Atlas has graduated to a top-level Apache project. It provides a type-agnostic metadata store and interfaces that can be accessed by various tools.
- The vision is for an open ecosystem where metadata is shared and federated across repositories from different vendors and tools.
Multi-tenant Hadoop - the challenge of maintaining high SLAS
In shared configuration, the same Hadoop environment supports many applications. Each has
specific requirements and criticality (SLA). Yet they all rely on an assembly of shared application
bricks.
At the same time, the life cycle of a cluster is not static in time. It evolves horizontally, with the
arrival of new applications, but also vertically, as the applications grow in load or evolve in
terms of functionality.
With this in mind, a multi-tenant production cluster presents several challenges including and
not limited to:
- Maintain a high level of SLA for a set of use cases with heterogeneous needs
- Plan and implement the architecture evolution of a cluster in production to ensure the
maintenance of SLA throughout the integration of new use cases on it
EDF will present how it manages this heterogeneity of SLA inherent of any Big Data cluster. EDF
is focusing on how it is renovating its cluster, its organization, its processes and its approach in
order to deliver a platform with strong SLA throughout its life cycle.
Speaker
Edouard Rousseaux, Tech Lead, EDF
Data Mesh in Practice: How Europe’s Leading Online Platform for Fashion Goes ...
Zalando transitioned from a centralized data platform to a data mesh architecture. This decentralized their data infrastructure by having individual domains own datasets and pipelines rather than a central team. It provided self-service data infrastructure tools and governance to enable domains to operate independently while maintaining global interoperability. This improved data quality by making domains responsible for their data and empowering them through the data mesh approach.
Verizon Centralizes Data into a Data Lake in Real Time for Analytics
Verizon – Global Technology Services (GTS) was challenged by a multi-tier, labor-intensive process when trying to migrate data from disparate sources into a data lake to create financial reports and business insights. Join this session to learn more about how Verizon:
• Easily accessed data from multiple sources including SAP data
• Ingested data into major targets including Hadoop
• Achieved real-time insights from data leveraging change data capture (CDC) technology
• Reduced costs and labor
From an experiment to a real production environment
Rabobank is a worldwide food- and agri-bank from the Netherlands. Rabobank wants to make a substantial contribution to welfare and prosperity in the Netherlands and to feeding the world sustainably. Rabobank Group operates through Rabobank and its subsidiaries in 40 countries.
Rabobank is active in both retail and wholesale banking. For our wholesale clients we provide real-time business insight information by making use of Cloudera and Hortonworks technology. An example is our recently launched service that gives insight in market performance of Rabobank customers, starting with the dairy farmers market segment, by making use of benchmark information. Our current technology stack contains Hortonworks Data Flow (HDF) and Cloudera Hadoop (CDH). Our real-time data stream is implemented by making use of Kafka and Nifi from HDF. Cloudera is used to store the data needed for the business insight information, mainly in HDFS and HBase.
During our presentation we will provides insight about the project approach, the architecture and actual implementation.
Speaker
Jeroen Wolffensperger, Solution Architect Data, Rabobank
Martijn Groen, Delivery Manager Data , Rabobank Netherlands
Ford
This document discusses Ford's data analytics strategy. It notes that the volume of data Ford collects is increasing significantly from connected vehicles and other sources. This includes up to 25 gigabytes per hour from individual vehicles. Ford is working to build applications and drive adoption of analytics across the company through education and training programs to democratize access to tools and infrastructure while ensuring privacy, security, and governance of customer data. The goal is to provide the right data, tools, and support to analysts and data scientists to improve products and services.
Synchronicity of a distributed financial system
In this presentation we look at the use of the Hortonworks Data Flow (HDF) platform application in the finance sector operations. We define the process of deployment and use of the distribution as part of the operations and how the different components are integrated to support real-time payments and banking functionality. Documenting the challenges faced to allow a high performance system that promotes data integrity and real-time visualisation using the Hortonworks Data Platform (HDP). Focusing specially on the use of Apache Zeppelin workbooks across the business as they main information management tool. Our specific approach focus on creating a flexible system that allows fast prototyping and integrated visualisation, monitoring and audit. Demonstrating the use of the HDF distribution on the creation of business domain abstractions to in a real life application of a Domain Driven Design. We implement a cross platform use of Avro, as future prof model and language that is understood by the business and IT areas. Due to the scalability of the platform we can execute payments operations at high rate even across countries at the same time reusing the same architecture to monitor the business operations.
Speaker
Luis Caldeira, Chief Architect, Orwell Group
Gian Marco Cabiato, Head of Engineering, Orwell Group
Munich Re: Driving a Big Data Transformation
Data and analytics are at the heart of the digital transformation. Implementing a modern data platform can be challenging; moreover, success requires a shift in culture. Andreas will discuss the ways Munich Re drives cultural and technological change within their company, focusing on three key elements: people, processes, and technology. What does it mean to be a data-driven organization? How can we provide self-service analytics to our internal and external customers in an agile way? How do we get the most value out of our big data lake? How does Munich Re balance technology and culture to meet the data demands of their business?
Speaker
Andreas Kohlmaier, Head of Data Engineering, Munich Re
O2’s Financial Data Hub: going beyond IFRS compliance to support digital tran...
This document summarizes the key aspects of a project by O2, a UK telecommunications company, to build a Finance Data Hub to improve financial reporting and comply with new IFRS 15 accounting standards. The hub centralized financial data from various sources using Hortonworks Data Platform for loading and Ab Initio for ETL and lineage. It delivered daily reconciled reports, enabled IFRS 15 compliance, and provided a foundation for future analytics. Lessons included building on defined business value, utilizing experts like Hortonworks, and considering cloud hosting costs and capabilities. The project created new analytics opportunities and supported accurate monthly revenue reporting.
Artificial Intelligence and Analytic Ops to Continuously Improve Business Out...
Analytic Ops is an approach that focuses on continuously improving business outcomes through artificial intelligence by getting AI solutions into production quickly while ensuring regulatory compliance. It addresses typical challenges where only 15% of advanced analytics projects reach production due to underestimating complexity and lack of agility. Analytic Ops prioritizes production, focuses on business value, and allows for iterative changes through agile processes and best practices from software development. This enables the creation of sustainable data products and models in a fraction of the usual time.
Stream Processing as Game Changer for Big Data and Internet of Things by Kai ...
https://www.bigdataspain.org/2016/program/thu-big-data-analytics-machine-learning-to-real-time-processing.html
https://www.youtube.com/watch?v=QiQ6LKKvd9k&t=3s&index=19&list=PL6O3g23-p8Tr5eqnIIPdBD_8eE5JBDBik
Ultralight Data Movement for IoT with SDC Edge
Edge computing and the Internet of Things bring great promise, but often just getting data from the edge requires moving mountains. Let's learn how to make edge data ingestion and analytics easier using StreamSets Data Collector edge, an ultralight, platform independent and small-footprint Open Source solution written in Go for streaming data from resource-constrained sensors and personal devices (like medical equipment or smartphones) to Apache Kafka, Amazon Kinesis and many others. This talk includes an overview of the SDC Edge main features, supported protocols and available processors for data transformation, insights on how it solves some challenges of traditional approaches to data ingestion, pipeline design basics, a walk-through some practical applications (Android devices and Raspberry Pi) and its integration with other technologies such as Streamsets Data Collector, Apache Kafka, Apache Hadoop, InfluxDB and Grafana. The goal here is to make attendees ready to quickly become IoT data intake and SDC Edge Ninjas.
Speaker
Guglielmo Iozzia, Big Data Delivery Manager, Optum (United Health)
Presentacin webinar move_up_to_power8_with_scale_out_servers_final
IBM Power SystemsMove up to Ibm POWER8
With POWER8 technology, your business will be ready:
For data
In the cloud
In the open world
Use Cases from Batch to Streaming, MapReduce to Spark, Mainframe to Cloud: To...
This document discusses how Syncsort helps companies access and integrate data from various sources to power analytics. It provides examples of how Syncsort has helped companies in insurance, media, and hotels easily onboard and integrate both historical and streaming data from multiple sources like mainframes, databases, and IoT devices. This allows for faster insights, increased productivity, cost savings, and helps future proof applications.
Destroying Data Silos
This document discusses ING NL's efforts to create a data lake architecture using Hadoop to integrate all of the bank's data sources onto a single processing platform. The data lake aims to collect data in a unified format, securely store it to prevent manipulation and unauthorized access, and make it available for analytical applications. Some of the challenges discussed include managing security, aligning with legacy systems, and facilitating interdepartmental cooperation on agile delivery. The presentation focuses on one part of the data lake, the archive, and how a Hadoop cluster can effectively address the goals of collecting, storing, and accessing data for business intelligence and data science purposes.
It Takes a Village: Organizational Alignment to Deliver Big Data Value in Hea...
The business and technology teams within a health insurer must align the company’s central data platform with its data strategy. That requires substantial organizational alignment. Hear the firsthand perspective from Health Care Service Corporation (HCSC), the largest customer-owned health insurance company in the United States. The speaker will cover how they integrated membership information, regulatory compliance, and the general ledger, to improve overall healthcare management. At HCSC, the strong alignment between executive leadership, business portfolio direction, architectural strategy, technology delivery, and program management have helped create leading-edge capabilities which help the company respond nimbly to a quickly evolving healthcare industry.
Data Acquisition Automation for NiFi in a Hybrid Cloud environment – the Path...
Liberty Global is one of the world’s largest international TV and broadband company, operating in multiple European countries, and with tens of millions of TV, broadband internet, telephony and mobile subscribers.
The Data Solutions team's journey started last year with a strategic project that aimed to implement a state of the art Hybrid Cloud Big Data platform. In this talk, the Manager and the Platform Architect are presenting the team’s data acquisition journey which begins with implementing NiFi flows with simple Get-Put pattern and, in its the final iteration, produces a solution capable of generating complex flows automatically, leading the path to the DataOps way of working.
Georgia Azure Event - Scalable cloud games using Microsoft Azure
The document discusses using Microsoft Azure cloud services for game development and operations. It provides examples of how games use Azure for scalable storage, global load balancing for multiplayer games, predictive analytics using big data, and DevOps approaches for deployment, monitoring, and development. Key Azure services highlighted include Storage, SQL Database, Virtual Machines, Mobile Services, HDInsight, and Application Insights.
Airline reservations and routing: a graph use case
We've all been there before... you hear the announcement that your flight is canceled. Fellow passengers race to the gate agent to rebook on the next available flight. How do they quickly determine the best route from Berlin to San Francisco? Ultimately the flight route network is best solved as a graph problem. We will discuss our lessons learned from working with a major airline to solve this problem using JanusGraph database. JanusGraph is an open source graph database designed for massive scale. It is compatible with several pieces of the open source big data stack: Apache TinkerPop (graph computing framework), HBase, Cassandra, and Solr. We will go into depth about our approach to benchmarking graph performance and discuss the utilities we developed. We will share our comparison results for evaluating which storage backend use with JanusGraph. Whether you are productizing a new database or you are a frustrated traveler, a fast resolution is needed to satisfy everybody involved.
Speaker
Jason Plurad, Open Source Developer and Advocate, IBM
Chin Huang, Software Engineer, IBM
Hadoop for the Masses
The document discusses rolling out a Hadoop-based data lake for self-service analytics within a corporate environment. It describes the background and motivation for implementing the data lake. Key challenges addressed include security, governance, and change management. Lessons learned include the importance of guidelines, reusable components, integration testing, and understanding users' diverse needs.
MongoDB IoT City Tour STUTTGART: Hadoop and future data management. By, Cloudera
Bernard Doering, Senior Slaes Director DACH, Cloudera.
Hadoop and the Future of Data Management. As Hadoop takes the data management market by storm, organisations are evolving the role it plays in the modern data centre. Explore how this disruptive technology is quickly transforming an industry and how you can leverage it today, in combination with MongoDB, to drive meaningful change in your business.
Webinar future dataintegration-datamesh-and-goldengatekafka
The Future of Data Integration: Data Mesh, and a Special Deep Dive into Stream Processing with GoldenGate, Apache Kafka and Apache Spark. This video is a replay of a Live Webinar hosted on 03/19/2020.
Join us for a timely 45min webinar to see our take on the future of Data Integration. As the global industry shift towards the “Fourth Industrial Revolution” continues, outmoded styles of centralized batch processing and ETL tooling continue to be replaced by realtime, streaming, microservices and distributed data architecture patterns.
This webinar will start with a brief look at the macro-trends happening around distributed data management and how that affects Data Integration. Next, we’ll discuss the event-driven integrations provided by GoldenGate Big Data, and continue with a deep-dive into some essential patterns we see when replicating Database change events into Apache Kafka. In this deep-dive we will explain how to effectively deal with issues like Transaction Consistency, Table/Topic Mappings, managing the DB Change Stream, and various Deployment Topologies to consider. Finally, we’ll wrap up with a brief look into how Stream Processing will help to empower modern Data Integration by supplying realtime data transformations, time-series analytics, and embedded Machine Learning from within data pipelines.
GoldenGate: https://www.oracle.com/middleware/tec...
Webinar Speaker: Jeff Pollock, VP Product (https://www.linkedin.com/in/jtpollock/)
Security, ETL, BI & Analytics, and Software Integration
This document discusses how Liberty Mutual Insurance is using a Hadoop data lake to power analytics. It provides examples of how the data lake is used to integrate data from various sources and support various use cases. Specifically, it discusses how the data lake enables:
- Storage of structured and unstructured data from multiple sources in a centralized and secure location
- Analytics and machine learning by data scientists and analysts accessing the stored data
- Integrations with tools like Elasticsearch, Spark, and PowerBI for querying, analyzing and visualizing the data
- Archiving of log and sensor data from systems into hot, warm and cold storage tiers based on age and access frequency
Big Data Analytics: Reference Architectures and Case Studies by Serhiy Haziye...
BI architecture drivers have to change to satisfy new requirements in format, volume, latency, hosting, analysis, reporting, and visualization. In this presentation delivered at the 2014 SATURN conference, SoftServe`s Serhiy and Olha showcased a number of reference architectures that address these challenges and speed up the design and implementation process, making it more predictable and economical:
- Traditional architecture based on an RDMBS data warehouse but modernized with column-based storage to handle a high load and capacity
- NoSQL-based architectures that address Big Data batch and stream-based processing and use popular NoSQL and complex event-processing solutions
- Hybrid architecture that combines traditional and NoSQL approaches to achieve completeness that would not be possible with either alone
The architectures are accompanied by real-life projects and case studies that the presenters have performed for multiple companies, including Fortune 100 and start-ups.
Oracle Stream Analytics - Developer Introduction
This is a brief technology introduction to Oracle Stream Analytics, and how to use the platform to develop streaming data pipelines that support a wide variety of industry use cases
A Key to Real-time Insights in a Post-COVID World (ASEAN)
Watch full webinar here: https://bit.ly/2EpHGyd
Presented at Data Champions, Online Asia 2020
Businesses and individuals around the world are experiencing the impact of a global pandemic. With many workers and potential shoppers still sequestered, COVID-19 is proving to have a momentous impact on the global economy. Regardless of the current situation and post-pandemic era, real-time data becomes even more critical to healthcare practitioners, business owners, government officials, and the public at large where holistic and timely information are important to make quick decisions. It enables doctors to make quick decisions about where to focus the care, business owners to alter production schedules to meet the demand, government agencies to contain the epidemic, and the public to be informed about prevention.
In this on-demand session, you will learn about the capabilities of data virtualization as a modern data integration technique and how can organisations:
- Rapidly unify information from disparate data sources to make accurate decisions and analyse data in real-time
- Build a single engine for security that provides audit and control by geographies
- Accelerate delivery of insights from your advanced analytics project
Improve IT Security and Compliance with Mainframe Data in Splunk
Avoid security blind spots with an enterprise-wide view.
If your organization relies on Splunk as its security nerve center, you can’t afford to leave out your mainframes.
They work with the rest of your IT infrastructure to support critical business applications–and they need to be
viewed in that wider context to address potential security blind spots.
Although the importance of including mainframe data in Splunk is undeniable, many organizations have left it out
because Splunk doesn’t natively support IBM Z® environments. Learn how Precisely Ironstream can help with a
straight-forward, powerful approach for integrating your mainframe security data into Splunk, and making it actionable
once it’s there.
More Related Content
What's hot
Ford
This document discusses Ford's data analytics strategy. It notes that the volume of data Ford collects is increasing significantly from connected vehicles and other sources. This includes up to 25 gigabytes per hour from individual vehicles. Ford is working to build applications and drive adoption of analytics across the company through education and training programs to democratize access to tools and infrastructure while ensuring privacy, security, and governance of customer data. The goal is to provide the right data, tools, and support to analysts and data scientists to improve products and services.
Synchronicity of a distributed financial system
In this presentation we look at the use of the Hortonworks Data Flow (HDF) platform application in the finance sector operations. We define the process of deployment and use of the distribution as part of the operations and how the different components are integrated to support real-time payments and banking functionality. Documenting the challenges faced to allow a high performance system that promotes data integrity and real-time visualisation using the Hortonworks Data Platform (HDP). Focusing specially on the use of Apache Zeppelin workbooks across the business as they main information management tool. Our specific approach focus on creating a flexible system that allows fast prototyping and integrated visualisation, monitoring and audit. Demonstrating the use of the HDF distribution on the creation of business domain abstractions to in a real life application of a Domain Driven Design. We implement a cross platform use of Avro, as future prof model and language that is understood by the business and IT areas. Due to the scalability of the platform we can execute payments operations at high rate even across countries at the same time reusing the same architecture to monitor the business operations.
Speaker
Luis Caldeira, Chief Architect, Orwell Group
Gian Marco Cabiato, Head of Engineering, Orwell Group
Munich Re: Driving a Big Data Transformation
Data and analytics are at the heart of the digital transformation. Implementing a modern data platform can be challenging; moreover, success requires a shift in culture. Andreas will discuss the ways Munich Re drives cultural and technological change within their company, focusing on three key elements: people, processes, and technology. What does it mean to be a data-driven organization? How can we provide self-service analytics to our internal and external customers in an agile way? How do we get the most value out of our big data lake? How does Munich Re balance technology and culture to meet the data demands of their business?
Speaker
Andreas Kohlmaier, Head of Data Engineering, Munich Re
O2’s Financial Data Hub: going beyond IFRS compliance to support digital tran...
This document summarizes the key aspects of a project by O2, a UK telecommunications company, to build a Finance Data Hub to improve financial reporting and comply with new IFRS 15 accounting standards. The hub centralized financial data from various sources using Hortonworks Data Platform for loading and Ab Initio for ETL and lineage. It delivered daily reconciled reports, enabled IFRS 15 compliance, and provided a foundation for future analytics. Lessons included building on defined business value, utilizing experts like Hortonworks, and considering cloud hosting costs and capabilities. The project created new analytics opportunities and supported accurate monthly revenue reporting.
Artificial Intelligence and Analytic Ops to Continuously Improve Business Out...
Analytic Ops is an approach that focuses on continuously improving business outcomes through artificial intelligence by getting AI solutions into production quickly while ensuring regulatory compliance. It addresses typical challenges where only 15% of advanced analytics projects reach production due to underestimating complexity and lack of agility. Analytic Ops prioritizes production, focuses on business value, and allows for iterative changes through agile processes and best practices from software development. This enables the creation of sustainable data products and models in a fraction of the usual time.
Stream Processing as Game Changer for Big Data and Internet of Things by Kai ...
https://www.bigdataspain.org/2016/program/thu-big-data-analytics-machine-learning-to-real-time-processing.html
https://www.youtube.com/watch?v=QiQ6LKKvd9k&t=3s&index=19&list=PL6O3g23-p8Tr5eqnIIPdBD_8eE5JBDBik
Ultralight Data Movement for IoT with SDC Edge
Edge computing and the Internet of Things bring great promise, but often just getting data from the edge requires moving mountains. Let's learn how to make edge data ingestion and analytics easier using StreamSets Data Collector edge, an ultralight, platform independent and small-footprint Open Source solution written in Go for streaming data from resource-constrained sensors and personal devices (like medical equipment or smartphones) to Apache Kafka, Amazon Kinesis and many others. This talk includes an overview of the SDC Edge main features, supported protocols and available processors for data transformation, insights on how it solves some challenges of traditional approaches to data ingestion, pipeline design basics, a walk-through some practical applications (Android devices and Raspberry Pi) and its integration with other technologies such as Streamsets Data Collector, Apache Kafka, Apache Hadoop, InfluxDB and Grafana. The goal here is to make attendees ready to quickly become IoT data intake and SDC Edge Ninjas.
Speaker
Guglielmo Iozzia, Big Data Delivery Manager, Optum (United Health)
Presentacin webinar move_up_to_power8_with_scale_out_servers_final
IBM Power SystemsMove up to Ibm POWER8
With POWER8 technology, your business will be ready:
For data
In the cloud
In the open world
Use Cases from Batch to Streaming, MapReduce to Spark, Mainframe to Cloud: To...
This document discusses how Syncsort helps companies access and integrate data from various sources to power analytics. It provides examples of how Syncsort has helped companies in insurance, media, and hotels easily onboard and integrate both historical and streaming data from multiple sources like mainframes, databases, and IoT devices. This allows for faster insights, increased productivity, cost savings, and helps future proof applications.
Destroying Data Silos
This document discusses ING NL's efforts to create a data lake architecture using Hadoop to integrate all of the bank's data sources onto a single processing platform. The data lake aims to collect data in a unified format, securely store it to prevent manipulation and unauthorized access, and make it available for analytical applications. Some of the challenges discussed include managing security, aligning with legacy systems, and facilitating interdepartmental cooperation on agile delivery. The presentation focuses on one part of the data lake, the archive, and how a Hadoop cluster can effectively address the goals of collecting, storing, and accessing data for business intelligence and data science purposes.
It Takes a Village: Organizational Alignment to Deliver Big Data Value in Hea...
The business and technology teams within a health insurer must align the company’s central data platform with its data strategy. That requires substantial organizational alignment. Hear the firsthand perspective from Health Care Service Corporation (HCSC), the largest customer-owned health insurance company in the United States. The speaker will cover how they integrated membership information, regulatory compliance, and the general ledger, to improve overall healthcare management. At HCSC, the strong alignment between executive leadership, business portfolio direction, architectural strategy, technology delivery, and program management have helped create leading-edge capabilities which help the company respond nimbly to a quickly evolving healthcare industry.
Data Acquisition Automation for NiFi in a Hybrid Cloud environment – the Path...
Liberty Global is one of the world’s largest international TV and broadband company, operating in multiple European countries, and with tens of millions of TV, broadband internet, telephony and mobile subscribers.
The Data Solutions team's journey started last year with a strategic project that aimed to implement a state of the art Hybrid Cloud Big Data platform. In this talk, the Manager and the Platform Architect are presenting the team’s data acquisition journey which begins with implementing NiFi flows with simple Get-Put pattern and, in its the final iteration, produces a solution capable of generating complex flows automatically, leading the path to the DataOps way of working.
Georgia Azure Event - Scalable cloud games using Microsoft Azure
The document discusses using Microsoft Azure cloud services for game development and operations. It provides examples of how games use Azure for scalable storage, global load balancing for multiplayer games, predictive analytics using big data, and DevOps approaches for deployment, monitoring, and development. Key Azure services highlighted include Storage, SQL Database, Virtual Machines, Mobile Services, HDInsight, and Application Insights.
Airline reservations and routing: a graph use case
We've all been there before... you hear the announcement that your flight is canceled. Fellow passengers race to the gate agent to rebook on the next available flight. How do they quickly determine the best route from Berlin to San Francisco? Ultimately the flight route network is best solved as a graph problem. We will discuss our lessons learned from working with a major airline to solve this problem using JanusGraph database. JanusGraph is an open source graph database designed for massive scale. It is compatible with several pieces of the open source big data stack: Apache TinkerPop (graph computing framework), HBase, Cassandra, and Solr. We will go into depth about our approach to benchmarking graph performance and discuss the utilities we developed. We will share our comparison results for evaluating which storage backend use with JanusGraph. Whether you are productizing a new database or you are a frustrated traveler, a fast resolution is needed to satisfy everybody involved.
Speaker
Jason Plurad, Open Source Developer and Advocate, IBM
Chin Huang, Software Engineer, IBM
Hadoop for the Masses
The document discusses rolling out a Hadoop-based data lake for self-service analytics within a corporate environment. It describes the background and motivation for implementing the data lake. Key challenges addressed include security, governance, and change management. Lessons learned include the importance of guidelines, reusable components, integration testing, and understanding users' diverse needs.
MongoDB IoT City Tour STUTTGART: Hadoop and future data management. By, Cloudera
Bernard Doering, Senior Slaes Director DACH, Cloudera.
Hadoop and the Future of Data Management. As Hadoop takes the data management market by storm, organisations are evolving the role it plays in the modern data centre. Explore how this disruptive technology is quickly transforming an industry and how you can leverage it today, in combination with MongoDB, to drive meaningful change in your business.
Webinar future dataintegration-datamesh-and-goldengatekafka
The Future of Data Integration: Data Mesh, and a Special Deep Dive into Stream Processing with GoldenGate, Apache Kafka and Apache Spark. This video is a replay of a Live Webinar hosted on 03/19/2020.
Join us for a timely 45min webinar to see our take on the future of Data Integration. As the global industry shift towards the “Fourth Industrial Revolution” continues, outmoded styles of centralized batch processing and ETL tooling continue to be replaced by realtime, streaming, microservices and distributed data architecture patterns.
This webinar will start with a brief look at the macro-trends happening around distributed data management and how that affects Data Integration. Next, we’ll discuss the event-driven integrations provided by GoldenGate Big Data, and continue with a deep-dive into some essential patterns we see when replicating Database change events into Apache Kafka. In this deep-dive we will explain how to effectively deal with issues like Transaction Consistency, Table/Topic Mappings, managing the DB Change Stream, and various Deployment Topologies to consider. Finally, we’ll wrap up with a brief look into how Stream Processing will help to empower modern Data Integration by supplying realtime data transformations, time-series analytics, and embedded Machine Learning from within data pipelines.
GoldenGate: https://www.oracle.com/middleware/tec...
Webinar Speaker: Jeff Pollock, VP Product (https://www.linkedin.com/in/jtpollock/)
Security, ETL, BI & Analytics, and Software Integration
This document discusses how Liberty Mutual Insurance is using a Hadoop data lake to power analytics. It provides examples of how the data lake is used to integrate data from various sources and support various use cases. Specifically, it discusses how the data lake enables:
- Storage of structured and unstructured data from multiple sources in a centralized and secure location
- Analytics and machine learning by data scientists and analysts accessing the stored data
- Integrations with tools like Elasticsearch, Spark, and PowerBI for querying, analyzing and visualizing the data
- Archiving of log and sensor data from systems into hot, warm and cold storage tiers based on age and access frequency
Big Data Analytics: Reference Architectures and Case Studies by Serhiy Haziye...
BI architecture drivers have to change to satisfy new requirements in format, volume, latency, hosting, analysis, reporting, and visualization. In this presentation delivered at the 2014 SATURN conference, SoftServe`s Serhiy and Olha showcased a number of reference architectures that address these challenges and speed up the design and implementation process, making it more predictable and economical:
- Traditional architecture based on an RDMBS data warehouse but modernized with column-based storage to handle a high load and capacity
- NoSQL-based architectures that address Big Data batch and stream-based processing and use popular NoSQL and complex event-processing solutions
- Hybrid architecture that combines traditional and NoSQL approaches to achieve completeness that would not be possible with either alone
The architectures are accompanied by real-life projects and case studies that the presenters have performed for multiple companies, including Fortune 100 and start-ups.
Oracle Stream Analytics - Developer Introduction
This is a brief technology introduction to Oracle Stream Analytics, and how to use the platform to develop streaming data pipelines that support a wide variety of industry use cases
What's hot (20)
O2’s Financial Data Hub: going beyond IFRS compliance to support digital tran...
O2’s Financial Data Hub: going beyond IFRS compliance to support digital tran...
Artificial Intelligence and Analytic Ops to Continuously Improve Business Out...
Artificial Intelligence and Analytic Ops to Continuously Improve Business Out...
Stream Processing as Game Changer for Big Data and Internet of Things by Kai ...
Stream Processing as Game Changer for Big Data and Internet of Things by Kai ...
Presentacin webinar move_up_to_power8_with_scale_out_servers_final
Presentacin webinar move_up_to_power8_with_scale_out_servers_final
Use Cases from Batch to Streaming, MapReduce to Spark, Mainframe to Cloud: To...
Use Cases from Batch to Streaming, MapReduce to Spark, Mainframe to Cloud: To...
It Takes a Village: Organizational Alignment to Deliver Big Data Value in Hea...
It Takes a Village: Organizational Alignment to Deliver Big Data Value in Hea...
Data Acquisition Automation for NiFi in a Hybrid Cloud environment – the Path...
Data Acquisition Automation for NiFi in a Hybrid Cloud environment – the Path...
Georgia Azure Event - Scalable cloud games using Microsoft Azure
Georgia Azure Event - Scalable cloud games using Microsoft Azure
Airline reservations and routing: a graph use case
Airline reservations and routing: a graph use case
MongoDB IoT City Tour STUTTGART: Hadoop and future data management. By, Cloudera
MongoDB IoT City Tour STUTTGART: Hadoop and future data management. By, Cloudera
Webinar future dataintegration-datamesh-and-goldengatekafka
Webinar future dataintegration-datamesh-and-goldengatekafka
Security, ETL, BI & Analytics, and Software Integration
Security, ETL, BI & Analytics, and Software Integration
Big Data Analytics: Reference Architectures and Case Studies by Serhiy Haziye...
Big Data Analytics: Reference Architectures and Case Studies by Serhiy Haziye...
Similar to Securing and governing a multi-tenant data lake within the financial industry
A Key to Real-time Insights in a Post-COVID World (ASEAN)
Watch full webinar here: https://bit.ly/2EpHGyd
Presented at Data Champions, Online Asia 2020
Businesses and individuals around the world are experiencing the impact of a global pandemic. With many workers and potential shoppers still sequestered, COVID-19 is proving to have a momentous impact on the global economy. Regardless of the current situation and post-pandemic era, real-time data becomes even more critical to healthcare practitioners, business owners, government officials, and the public at large where holistic and timely information are important to make quick decisions. It enables doctors to make quick decisions about where to focus the care, business owners to alter production schedules to meet the demand, government agencies to contain the epidemic, and the public to be informed about prevention.
In this on-demand session, you will learn about the capabilities of data virtualization as a modern data integration technique and how can organisations:
- Rapidly unify information from disparate data sources to make accurate decisions and analyse data in real-time
- Build a single engine for security that provides audit and control by geographies
- Accelerate delivery of insights from your advanced analytics project
Improve IT Security and Compliance with Mainframe Data in Splunk
Avoid security blind spots with an enterprise-wide view.
If your organization relies on Splunk as its security nerve center, you can’t afford to leave out your mainframes.
They work with the rest of your IT infrastructure to support critical business applications–and they need to be
viewed in that wider context to address potential security blind spots.
Although the importance of including mainframe data in Splunk is undeniable, many organizations have left it out
because Splunk doesn’t natively support IBM Z® environments. Learn how Precisely Ironstream can help with a
straight-forward, powerful approach for integrating your mainframe security data into Splunk, and making it actionable
once it’s there.
Cybersecurity Legos - We're all part of something bigger
Presentation by Ben Boyd during the 2018 Northwest Arkansas Community College Cyber Security Awareness Symposium.
Building a Cyber Security program is more than just technology or architecture. Managing Cyber Risk is the duty of anyone with a digital asset.
Aligning Application Security to Compliance
Regulatory compliance mandates have historically focused on IT & endpoint security as the primary means to protect data. However, as our digital economy has increasingly become software dependent, standards bodies have dutifully added requirements as they relate to development and deployment practices. Enterprise applications and cloud-based services constantly store and transmit data; yet, they are often difficult to understand and assess for compliance.
This webcast will present a practical approach towards mapping application security practices to common compliance frameworks. It will discuss how to define and enact a secure, repeatable software development lifecycle (SDLC) and highlight activities that can be leveraged across multiple compliance controls. Topics include:
* Consolidating security and compliance controls
* Creating application security standards for development and operations teams
* Identifying and remediating gaps between current practices and industry accepted "best practices”
Understanding Zero Trust Security for IBM i
As security threats continue to evolve and increase, companies need to also adapt their approach to IT security. One important concept that is gaining in popularity and adoption is zero trust security. The main concept behind the zero trust security model is "never trust, always verify,” which means that devices should not be trusted by default, even if they are connected to a permissioned network such as a corporate LAN and even if they were previously verified.
Zero Trust means moving beyond a perimeter security strategy. As companies offer customers and business partners new digital experiences and processes, networks can be local, in the cloud, or a combination or hybrid with resources anywhere as well as workers in any location. This dynamic is impacting IBM i customers and zero trust security is an important element of a modern security strategy.
Join us for this webcast to hear about:
• Understanding zero trust security concepts
• Zero trust security in the real world
• Zero trust security for IBM i environments
CASE STUDY: UK NATIONAL HEALTH SERVICE
The document provides an overview of the Care Identity Services (CIS) project to build an identity management and smartcard provisioning system to replace the expiring NHS Spine Identity Management Services. Key points:
- The Spine system currently supports over 1 million registered users but the contract is expiring with an aging architecture.
- Building the system internally through a "DevOps" model was chosen over outsourcing to a prime contractor or systems integrator due to concerns over cost, control, and ability to change quickly.
- The CIS project scope includes rebuilding core services like user identity management, the user directory, and smartcard management on new open source software like ForgeRock, OpenDJ and integrating with other
Big Data Everywhere Chicago: The Big Data Imperative -- Discovering & Protect...
Today, no industry is immune from a potential data breach and the havoc it can create. According to a 2013 Global Data Breach study by the Ponemon Institute, the average cost of data loss exceeds $5.4 million per breach, and the average per person cost of lost data is approaching $200 per record in the US. Protecting sensitive data in Hadoop is now the imperative for IT and business. With the emergence of Hadoop as a business-critical data platform, Hadoop offers organizations opportunities to improve performance, better understand customers and develop a competitive advantage. But reaching these desirable analytic outcomes depends on the ability to use data without exposing the organization to unnecessary risk. This presentation will cover best practices for a data-centric security, compliance and data governance approach, with a particular focus on two customer use cases within the financial services and insurance industries. You'll learn how these companies are reducing their security exposure through automated data-centric protection of sensitive data in Hadoop.
Securing Microsoft Technologies for HITECH Compliance
SharePoint Saturday Philadelphia presentation 2/4/2012. Special thanks to CipherPoint Software and AvePoint for preparation and sharing panel time.
Best practices in planning and design of enterprise security and master data management strategies for HITECH compliance of both SharePoint 2010 on premise and Office 365 instances for use in covered healthcare entities.
Ciso executive forum 2013
This document discusses the future of information security based on Netflix's experience and perspective. It predicts that social, mobile, and cloud computing will drive new security challenges as traditional controls become lacking. Netflix relies heavily on cloud computing and aims to be fully cloud-based. It uses various "monkey" programs to test systems and identify weaknesses. Looking ahead, the document predicts that security teams will take more of an advisory role using analytics and automation. Device, network, and data security will need new approaches as boundaries shift. Security will rely more on continuous testing, monitoring, and automated protection.
Guardium Data Activiy Monitor For C- Level Executives
IT infrastructure is changing and needs controls for mobile, cloud, and big data
Guardium is the leader in database and big data security
Heterogeneous support is a great asset to leverage across the infrastructure to reduce risk
Supports separation of duties
Integration with other security products
No additional training for multiple products
Security data deluge
A Security Data Warehouse (SDW) is a massive database built using Hadoop and Hive that aggregates security and fraud-related event data from across an entire enterprise for long-term analytics. Zions Bank built an SDW to address limitations of their SIEM in dealing with large, unstructured datasets and to provide a common platform where security and fraud teams could collaborate by analyzing the complete historical data in one system. The SDW utilizes various Hadoop features for scalability, fault tolerance and handling different data types to support petabytes of stored data and thousands of daily analysis jobs.
Protect your Database with Data Masking & Enforced Version Control
Your database holds your company's most sensitive and important assets- your data. All those customers' personal details, credit card numbers, social security numbers- you can't afford leaving them vulnerable to any- outside or inside- breaches.
ICP for Data- Enterprise platform for AI, ML and Data Science
IBM Cloud Private for Data, an ultimate platform for all AI, ML and Data Science workloads. Integrated analytics platform based on Containers and micro services. Works with Kubernetes and dockers, even with Redhat openshift. Delivers the variety of business use cases in all industries- FS, Telco, Retail, Manufacturing etc
Securing Microsoft Technologies for HITECH Compliance
This document provides a summary of a presentation on securing Microsoft technologies for HITECH compliance. It discusses how Microsoft business solutions can help with healthcare information security and compliance with privacy laws like HIPAA and HITECH. It outlines challenges around managing unstructured data, compliance, security, and recruitment in healthcare. The presentation addresses privacy, data security, and how technologies like SharePoint can help with managing electronic protected health information and personally identifiable information according to healthcare regulations.
Securing your esi_piedmont
This document provides an overview and guidance on securing electronically stored information (ESI). It discusses the importance of ESI security due to increasing data breaches and costs. It then covers ESI and security overview topics like the CIA triad and vendor selection. The document concludes with security tips and guidance on topics like encryption, identity and access management, change management, and incident response. The overall document aims to educate readers on securing ESI and provides high-level summaries of best practices.
Beyond the Scan: The Value Proposition of Vulnerability Assessment
Vulnerability Assessment is, by some, regarded as one of the least “sexy” capabilities in information security. However, it is the presenter’s view that it is also a key component of any successful infosec program, and one that is often overlooked. Doing so serves an injustice to the organization and results in many missed opportunities to help ensure success in protecting critical information assets. The presenter will explore how Vulnerability Assessment can be leveraged “Beyond the Scan” and provide tangible value to not only the security team, but the entire business that it supports.
Turn Big Data into Big Value on Informatica and AWS
Speaker: Eddie Hui, Principal Sales Consultant, Informatica
These Informatica Cloud offerings are pre-built packages for quick time-to-value for customers looking to fast-track cloud data management initiatives. For example, customers can quickly kick start a new Amazon Redshift data warehouse project and use Informatica Cloud Connector for Amazon Redshift to load it with meaningful connected data from cloud sources such as Salesforce.com or on-premises sources such as relational databases -- all within hours, not months.
A Study in Borderless Over Perimeter
A Breakout Session by Amit Saha, COO at Saviynt. Presented at the 2014 IRM Summit in Phoenix, Arizona.
Who will guard the guards
This document discusses privileged identity and session management. It begins by outlining some ground-level realities of how system administrators operate and common insider threats. It then discusses compliance regulations related to privileged accounts. The rest of the document defines different types of privileged accounts, outlines the scope of the problem of managing privileged accounts, and provides examples of privileged access needed for application-to-application communication. It concludes by advocating for a comprehensive privileged identity and session management platform and controls framework to help isolate and protect privileged credentials and activities.
Protect Sensitive Data on Your IBM i (Social Distance Your IBM i/AS400)
The document discusses protecting sensitive data on IBM i systems. It provides an agenda for a webcast covering key concepts for protecting IBM i data privacy including encryption, tokenization, and secure file transfer. It will also introduce the Assure Security solution from Precisely for IBM i compliance and security. The webcast includes segments on protecting data privacy, demonstrating Assure Security, and a question and answer period.
Similar to Securing and governing a multi-tenant data lake within the financial industry (20)
A Key to Real-time Insights in a Post-COVID World (ASEAN)
A Key to Real-time Insights in a Post-COVID World (ASEAN)
Improve IT Security and Compliance with Mainframe Data in Splunk
Improve IT Security and Compliance with Mainframe Data in Splunk
Cybersecurity Legos - We're all part of something bigger
Cybersecurity Legos - We're all part of something bigger
Big Data Everywhere Chicago: The Big Data Imperative -- Discovering & Protect...
Big Data Everywhere Chicago: The Big Data Imperative -- Discovering & Protect...
Securing Microsoft Technologies for HITECH Compliance
Securing Microsoft Technologies for HITECH Compliance
Guardium Data Activiy Monitor For C- Level Executives
Guardium Data Activiy Monitor For C- Level Executives
Protect your Database with Data Masking & Enforced Version Control
Protect your Database with Data Masking & Enforced Version Control
ICP for Data- Enterprise platform for AI, ML and Data Science
ICP for Data- Enterprise platform for AI, ML and Data Science
Securing Microsoft Technologies for HITECH Compliance
Securing Microsoft Technologies for HITECH Compliance
Beyond the Scan: The Value Proposition of Vulnerability Assessment
Beyond the Scan: The Value Proposition of Vulnerability Assessment
Turn Big Data into Big Value on Informatica and AWS
Turn Big Data into Big Value on Informatica and AWS
Protect Sensitive Data on Your IBM i (Social Distance Your IBM i/AS400)
Protect Sensitive Data on Your IBM i (Social Distance Your IBM i/AS400)
More from DataWorks Summit
Data Science Crash Course
Introduction: This workshop will provide a hands-on introduction to Machine Learning (ML) with an overview of Deep Learning (DL).
Format: An introductory lecture on several supervised and unsupervised ML techniques followed by light introduction to DL and short discussion what is current state-of-the-art. Several python code samples using the scikit-learn library will be introduced that users will be able to run in the Cloudera Data Science Workbench (CDSW).
Objective: To provide a quick and short hands-on introduction to ML with python’s scikit-learn library. The environment in CDSW is interactive and the step-by-step guide will walk you through setting up your environment, to exploring datasets, training and evaluating models on popular datasets. By the end of the crash course, attendees will have a high-level understanding of popular ML algorithms and the current state of DL, what problems they can solve, and walk away with basic hands-on experience training and evaluating ML models.
Prerequisites: For the hands-on portion, registrants must bring a laptop with a Chrome or Firefox web browser. These labs will be done in the cloud, no installation needed. Everyone will be able to register and start using CDSW after the introductory lecture concludes (about 1hr in). Basic knowledge of python highly recommended.
Floating on a RAFT: HBase Durability with Apache Ratis
In a world with a myriad of distributed storage systems to choose from, the majority of Apache HBase clusters still rely on Apache HDFS. Theoretically, any distributed file system could be used by HBase. One major reason HDFS is predominantly used are the specific durability requirements of HBase's write-ahead log (WAL) and HDFS providing that guarantee correctly. However, HBase's use of HDFS for WALs can be replaced with sufficient effort.
This talk will cover the design of a "Log Service" which can be embedded inside of HBase that provides a sufficient level of durability that HBase requires for WALs. Apache Ratis (incubating) is a library-implementation of the RAFT consensus protocol in Java and is used to build this Log Service. We will cover the design choices of the Ratis Log Service, comparing and contrasting it to other log-based systems that exist today. Next, we'll cover how the Log Service "fits" into HBase and the necessary changes to HBase which enable this. Finally, we'll discuss how the Log Service can simplify the operational burden of HBase.
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Utilizing Apache NiFi we read various open data REST APIs and camera feeds to ingest crime and related data real-time streaming it into HBase and Phoenix tables. HBase makes an excellent storage option for our real-time time series data sources. We can immediately query our data utilizing Apache Zeppelin against Phoenix tables as well as Hive external tables to HBase.
Apache Phoenix tables also make a great option since we can easily put microservices on top of them for application usage. I have an example Spring Boot application that reads from our Philadelphia crime table for front-end web applications as well as RESTful APIs.
Apache NiFi makes it easy to push records with schemas to HBase and insert into Phoenix SQL tables.
Resources:
https://community.hortonworks.com/articles/54947/reading-opendata-json-and-storing-into-phoenix-tab.html
https://community.hortonworks.com/articles/56642/creating-a-spring-boot-java-8-microservice-to-read.html
https://community.hortonworks.com/articles/64122/incrementally-streaming-rdbms-data-to-your-hadoop.html
HBase Tales From the Trenches - Short stories about most common HBase operati...
Whilst HBase is the most logical answer for use cases requiring random, realtime read/write access to Big Data, it may not be so trivial to design applications that make most of its use, neither the most simple to operate. As it depends/integrates with other components from Hadoop ecosystem (Zookeeper, HDFS, Spark, Hive, etc) or external systems ( Kerberos, LDAP), and its distributed nature requires a "Swiss clockwork" infrastructure, many variables are to be considered when observing anomalies or even outages. Adding to the equation there's also the fact that HBase is still an evolving product, with different release versions being used currently, some of those can carry genuine software bugs. On this presentation, we'll go through the most common HBase issues faced by different organisations, describing identified cause and resolution action over my last 5 years supporting HBase to our heterogeneous customer base.
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
LocationTech GeoMesa enables spatial and spatiotemporal indexing and queries for HBase and Accumulo. In this talk, after an overview of GeoMesa’s capabilities in the Cloudera ecosystem, we will dive into how GeoMesa leverages Accumulo’s Iterator interface and HBase’s Filter and Coprocessor interfaces. The goal will be to discuss both what spatial operations can be pushed down into the distributed database and also how the GeoMesa codebase is organized to allow for consistent use across the two database systems.
Managing the Dewey Decimal System
OCLC has been using HBase since 2012 to enable single-search-box access to over a billion items from your library and the world’s library collection. This talk will provide an overview of how HBase is structured to provide this information and some of the challenges they have encountered to scale to support the world catalog and how they have overcome them.
Practical NoSQL: Accumulo's dirlist Example
Many individuals/organizations have a desire to utilize NoSQL technology, but often lack an understanding of how the underlying functional bits can be utilized to enable their use case. This situation can result in drastic increases in the desire to put the SQL back in NoSQL.
Since the initial commit, Apache Accumulo has provided a number of examples to help jumpstart comprehension of how some of these bits function as well as potentially help tease out an understanding of how they might be applied to a NoSQL friendly use case. One very relatable example demonstrates how Accumulo could be used to emulate a filesystem (dirlist).
In this session we will walk through the dirlist implementation. Attendees should come away with an understanding of the supporting table designs, a simple text search supporting a single wildcard (on file/directory names), and how the dirlist elements work together to accomplish its feature set. Attendees should (hopefully) also come away with a justification for sometimes keeping the SQL out of NoSQL.
HBase Global Indexing to support large-scale data ingestion at Uber
Danny Chen presented on Uber's use of HBase for global indexing to support large-scale data ingestion. Uber uses HBase to provide a global view of datasets ingested from Kafka and other data sources. To generate indexes, Spark jobs are used to transform data into HFiles, which are loaded into HBase tables. Given the large volumes of data, techniques like throttling HBase access and explicit serialization are used. The global indexing solution supports requirements for high throughput, strong consistency and horizontal scalability across Uber's data lake.
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Recently, Apache Phoenix has been integrated with Apache (incubator) Omid transaction processing service, to provide ultra-high system throughput with ultra-low latency overhead. Phoenix has been shown to scale beyond 0.5M transactions per second with sub-5ms latency for short transactions on industry-standard hardware. On the other hand, Omid has been extended to support secondary indexes, multi-snapshot SQL queries, and massive-write transactions.
These innovative features make Phoenix an excellent choice for translytics applications, which allow converged transaction processing and analytics. We share the story of building the next-gen data tier for advertising platforms at Verizon Media that exploits Phoenix and Omid to support multi-feed real-time ingestion and AI pipelines in one place, and discuss the lessons learned.
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
This document discusses using Apache NiFi to build a high-speed cyber security data pipeline. It outlines the challenges of ingesting, transforming, and routing large volumes of security data from various sources to stakeholders like security operations centers, data scientists, and executives. It proposes using NiFi as a centralized data gateway to ingest data from multiple sources using a single entry point, transform the data according to destination needs, and reliably deliver the data while avoiding issues like network traffic and data duplication. The document provides an example NiFi flow and discusses metrics from processing over 20 billion events through 100+ production flows and 1000+ transformations.
Supporting Apache HBase : Troubleshooting and Supportability Improvements
This document discusses supporting Apache HBase and improving troubleshooting and supportability. It introduces two Cloudera employees who work on HBase support and provides an overview of typical troubleshooting scenarios for HBase like performance degradation, process crashes, and inconsistencies. The agenda covers using existing tools like logs and metrics to troubleshoot HBase performance issues with a general approach, and introduces htop as a real-time monitoring tool for HBase.
Security Framework for Multitenant Architecture
In the healthcare sector, data security, governance, and quality are crucial for maintaining patient privacy and ensuring the highest standards of care. At Florida Blue, the leading health insurer of Florida serving over five million members, there is a multifaceted network of care providers, business users, sales agents, and other divisions relying on the same datasets to derive critical information for multiple applications across the enterprise. However, maintaining consistent data governance and security for protected health information and other extended data attributes has always been a complex challenge that did not easily accommodate the wide range of needs for Florida Blue’s many business units. Using Apache Ranger, we developed a federated Identity & Access Management (IAM) approach that allows each tenant to have their own IAM mechanism. All user groups and roles are propagated across the federation in order to determine users’ data entitlement and access authorization; this applies to all stages of the system, from the broadest tenant levels down to specific data rows and columns. We also enabled audit attributes to ensure data quality by documenting data sources, reasons for data collection, date and time of data collection, and more. In this discussion, we will outline our implementation approach, review the results, and highlight our “lessons learned.”
Presto: Optimizing Performance of SQL-on-Anything Engine
Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Airbnb, Bloomberg, Comcast, Facebook, FINRA, LinkedIn, Lyft, Netflix, Twitter, and Uber, in the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Stores, HDFS, NoSQL and RDBMS data stores.
With the ever-growing list of connectors to new data sources such as Azure Blob Storage, Elasticsearch, Netflix Iceberg, Apache Kudu, and Apache Pulsar, recently introduced Cost-Based Optimizer in Presto must account for heterogeneous inputs with differing and often incomplete data statistics. This talk will explore this topic in detail as well as discuss best use cases for Presto across several industries. In addition, we will present recent Presto advancements such as Geospatial analytics at scale and the project roadmap going forward.
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Specialized tools for machine learning development and model governance are becoming essential. MlFlow is an open source platform for managing the machine learning lifecycle. Just by adding a few lines of code in the function or script that trains their model, data scientists can log parameters, metrics, artifacts (plots, miscellaneous files, etc.) and a deployable packaging of the ML model. Every time that function or script is run, the results will be logged automatically as a byproduct of those lines of code being added, even if the party doing the training run makes no special effort to record the results. MLflow application programming interfaces (APIs) are available for the Python, R and Java programming languages, and MLflow sports a language-agnostic REST API as well. Over a relatively short time period, MLflow has garnered more than 3,300 stars on GitHub , almost 500,000 monthly downloads and 80 contributors from more than 40 companies. Most significantly, more than 200 companies are now using MLflow. We will demo MlFlow Tracking , Project and Model components with Azure Machine Learning (AML) Services and show you how easy it is to get started with MlFlow on-prem or in the cloud.
Extending Twitter's Data Platform to Google Cloud
Twitter's Data Platform is built using multiple complex open source and in house projects to support Data Analytics on hundreds of petabytes of data. Our platform support storage, compute, data ingestion, discovery and management and various tools and libraries to help users for both batch and realtime analytics. Our DataPlatform operates on multiple clusters across different data centers to help thousands of users discover valuable insights. As we were scaling our Data Platform to multiple clusters, we also evaluated various cloud vendors to support use cases outside of our data centers. In this talk we share our architecture and how we extend our data platform to use cloud as another datacenter. We walk through our evaluation process, challenges we faced supporting data analytics at Twitter scale on cloud and present our current solution. Extending Twitter's Data platform to cloud was complex task which we deep dive in this presentation.
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
At Comcast, our team has been architecting a customer experience platform which is able to react to near-real-time events and interactions and deliver appropriate and timely communications to customers. By combining the low latency capabilities of Apache Flink and the dataflow capabilities of Apache NiFi we are able to process events at high volume to trigger, enrich, filter, and act/communicate to enhance customer experiences. Apache Flink and Apache NiFi complement each other with their strengths in event streaming and correlation, state management, command-and-control, parallelism, development methodology, and interoperability with surrounding technologies. We will trace our journey from starting with Apache NiFi over three years ago and our more recent introduction of Apache Flink into our platform stack to handle more complex scenarios. In this presentation we will compare and contrast which business and technical use cases are best suited to which platform and explore different ways to integrate the two platforms into a single solution.
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Companies are increasingly moving to the cloud to store and process data. One of the challenges companies have is in securing data across hybrid environments with easy way to centrally manage policies. In this session, we will talk through how companies can use Apache Ranger to protect access to data both in on-premise as well as in cloud environments. We will go into details into the challenges of hybrid environment and how Ranger can solve it. We will also talk through how companies can further enhance the security by leveraging Ranger to anonymize or tokenize data while moving into the cloud and de-anonymize dynamically using Apache Hive, Apache Spark or when accessing data from cloud storage systems. We will also deep dive into the Ranger’s integration with AWS S3, AWS Redshift and other cloud native systems. We will wrap it up with an end to end demo showing how policies can be created in Ranger and used to manage access to data in different systems, anonymize or de-anonymize data and track where data is flowing.
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Advanced Big Data Processing frameworks have been proposed to harness the fast data transmission capability of Remote Direct Memory Access (RDMA) over high-speed networks such as InfiniBand, RoCEv1, RoCEv2, iWARP, and OmniPath. However, with the introduction of the Non-Volatile Memory (NVM) and NVM express (NVMe) based SSD, these designs along with the default Big Data processing models need to be re-assessed to discover the possibilities of further enhanced performance. In this talk, we will present, NRCIO, a high-performance communication runtime for non-volatile memory over modern network interconnects that can be leveraged by existing Big Data processing middleware. We will show the performance of non-volatile memory-aware RDMA communication protocols using our proposed runtime and demonstrate its benefits by incorporating it into a high-performance in-memory key-value store, Apache Hadoop, Tez, Spark, and TensorFlow. Evaluation results illustrate that NRCIO can achieve up to 3.65x performance improvement for representative Big Data processing workloads on modern data centers.
Computer Vision: Coming to a Store Near You
Background: Some early applications of Computer Vision in Retail arose from e-commerce use cases - but increasingly, it is being used in physical stores in a variety of new and exciting ways, such as:
● Optimizing merchandising execution, in-stocks and sell-thru
● Enhancing operational efficiencies, enable real-time customer engagement
● Enhancing loss prevention capabilities, response time
● Creating frictionless experiences for shoppers
Abstract: This talk will cover the use of Computer Vision in Retail, the implications to the broader Consumer Goods industry and share business drivers, use cases and benefits that are unfolding as an integral component in the remaking of an age-old industry.
We will also take a ‘peek under the hood’ of Computer Vision and Deep Learning, sharing technology design principles and skill set profiles to consider before starting your CV journey.
Deep learning has matured considerably in the past few years to produce human or superhuman abilities in a variety of computer vision paradigms. We will discuss ways to recognize these paradigms in retail settings, collect and organize data to create actionable outcomes with the new insights and applications that deep learning enables.
We will cover the basics of object detection, then move into the advanced processing of images describing the possible ways that a retail store of the near future could operate. Identifying various storefront situations by having a deep learning system attached to a camera stream. Such things as; identifying item stocks on shelves, a shelf in need of organization, or perhaps a wandering customer in need of assistance.
We will also cover how to use a computer vision system to automatically track customer purchases to enable a streamlined checkout process, and how deep learning can power plausible wardrobe suggestions based on what a customer is currently wearing or purchasing.
Finally, we will cover the various technologies that are powering these applications today. Deep learning tools for research and development. Production tools to distribute that intelligence to an entire inventory of all the cameras situation around a retail location. Tools for exploring and understanding the new data streams produced by the computer vision systems.
By the end of this talk, attendees should understand the impact Computer Vision and Deep Learning are having in the Consumer Goods industry, key use cases, techniques and key considerations leaders are exploring and implementing today.
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Whole genome shotgun based next generation transcriptomics and metagenomics studies often generate 100 to 1000 gigabytes (GB) sequence data derived from tens of thousands of different genes or microbial species. De novo assembling these data requires an ideal solution that both scales with data size and optimizes for individual gene or genomes. Here we developed an Apache Spark-based scalable sequence clustering application, SparkReadClust (SpaRC), that partitions the reads based on their molecule of origin to enable downstream assembly optimization. SpaRC produces high clustering performance on transcriptomics and metagenomics test datasets from both short read and long read sequencing technologies. It achieved a near linear scalability with respect to input data size and number of compute nodes. SpaRC can run on different cloud computing environments without modifications while delivering similar performance. In summary, our results suggest SpaRC provides a scalable solution for clustering billions of reads from the next-generation sequencing experiments, and Apache Spark represents a cost-effective solution with rapid development/deployment cycles for similar big data genomics problems.
More from DataWorks Summit (20)
Floating on a RAFT: HBase Durability with Apache Ratis
Floating on a RAFT: HBase Durability with Apache Ratis
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
HBase Tales From the Trenches - Short stories about most common HBase operati...
HBase Tales From the Trenches - Short stories about most common HBase operati...
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
HBase Global Indexing to support large-scale data ingestion at Uber
HBase Global Indexing to support large-scale data ingestion at Uber
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
Supporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability Improvements
Presto: Optimizing Performance of SQL-on-Anything Engine
Presto: Optimizing Performance of SQL-on-Anything Engine
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Recently uploaded
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Dmitrii Kamaev, PhD
Senior Product Owner - QIAGEN
What is an RPA CoE? Session 2 – CoE Roles
In this session, we will review the players involved in the CoE and how each role impacts opportunities.
Topics covered:
• What roles are essential?
• What place in the automation journey does each role play?
Speaker:
Chris Bolin, Senior Intelligent Automation Architect Anika Systems
Session 1 - Intro to Robotic Process Automation.pdf
👉 Check out our full 'Africa Series - Automation Student Developers (EN)' page to register for the full program:
https://bit.ly/Automation_Student_Kickstart
In this session, we shall introduce you to the world of automation, the UiPath Platform, and guide you on how to install and setup UiPath Studio on your Windows PC.
📕 Detailed agenda:
What is RPA? Benefits of RPA?
RPA Applications
The UiPath End-to-End Automation Platform
UiPath Studio CE Installation and Setup
💻 Extra training through UiPath Academy:
Introduction to Automation
UiPath Business Automation Platform
Explore automation development with UiPath Studio
👉 Register here for our upcoming Session 2 on June 20: Introduction to UiPath Studio Fundamentals: https://community.uipath.com/events/details/uipath-lagos-presents-session-2-introduction-to-uipath-studio-fundamentals/
JavaLand 2024: Application Development Green Masterplan
My presentation slides I used at JavaLand 2024
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
These are the slides for the presentation, "Component Testing: Bridging the gap between frontend applications" that was presented at QA or the Highway 2024 in Columbus, OH by Zachary Hamm.
Leveraging the Graph for Clinical Trials and Standards
Katja Glaß
OpenStudyBuilder Community Manager - Katja Glaß Consulting
Marius Conjeaud
Principal Consultant - Neo4j
Apps Break Data
How information systems are built or acquired puts information, which is what they should be about, in a secondary place. Our language adapted accordingly, and we no longer talk about information systems but applications. Applications evolved in a way to break data into diverse fragments, tightly coupled with applications and expensive to integrate. The result is technical debt, which is re-paid by taking even bigger "loans", resulting in an ever-increasing technical debt. Software engineering and procurement practices work in sync with market forces to maintain this trend. This talk demonstrates how natural this situation is. The question is: can something be done to reverse the trend?
Introduction of Cybersecurity with OSS at Code Europe 2024
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
Astute Business Solutions | Oracle Cloud Partner |
Your goto partner for Oracle Cloud, PeopleSoft, E-Business Suite, and Ellucian Banner. We are a firm specialized in managed services and consulting.
"Choosing proper type of scaling", Olena Syrota
Imagine an IoT processing system that is already quite mature and production-ready and for which client coverage is growing and scaling and performance aspects are life and death questions. The system has Redis, MongoDB, and stream processing based on ksqldb. In this talk, firstly, we will analyze scaling approaches and then select the proper ones for our system.
Mutation Testing for Task-Oriented Chatbots
Conversational agents, or chatbots, are increasingly used to access all sorts of services using natural language. While open-domain chatbots - like ChatGPT - can converse on any topic, task-oriented chatbots - the focus of this paper - are designed for specific tasks, like booking a flight, obtaining customer support, or setting an appointment. Like any other software, task-oriented chatbots need to be properly tested, usually by defining and executing test scenarios (i.e., sequences of user-chatbot interactions). However, there is currently a lack of methods to quantify the completeness and strength of such test scenarios, which can lead to low-quality tests, and hence to buggy chatbots.
To fill this gap, we propose adapting mutation testing (MuT) for task-oriented chatbots. To this end, we introduce a set of mutation operators that emulate faults in chatbot designs, an architecture that enables MuT on chatbots built using heterogeneous technologies, and a practical realisation as an Eclipse plugin. Moreover, we evaluate the applicability, effectiveness and efficiency of our approach on open-source chatbots, with promising results.
Dandelion Hashtable: beyond billion requests per second on a commodity server
This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Join us to learn how UiPath Apps can directly and easily interact with prebuilt connectors via Integration Service--including Salesforce, ServiceNow, Open GenAI, and more.
The best part is you can achieve this without building a custom workflow! Say goodbye to the hassle of using separate automations to call APIs. By seamlessly integrating within App Studio, you can now easily streamline your workflow, while gaining direct access to our Connector Catalog of popular applications.
We’ll discuss and demo the benefits of UiPath Apps and connectors including:
Creating a compelling user experience for any software, without the limitations of APIs.
Accelerating the app creation process, saving time and effort
Enjoying high-performance CRUD (create, read, update, delete) operations, for
seamless data management.
Speakers:
Russell Alfeche, Technology Leader, RPA at qBotic and UiPath MVP
Charlie Greenberg, host
Principle of conventional tomography-Bibash Shahi ppt..pptx
before the computed tomography, it had been widely used.
Main news related to the CCS TSI 2023 (2023/1695)
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
Recently uploaded (20)
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Principle of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptx
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Securing and governing a multi-tenant data lake within the financial industry
- 1. Securing and Governing a Multi-Tenant Data Lake within the Financial Industry Presenters: Bradley Smith Ian Pillay
- 2. Data Lake Security, governance, multi-tenancy
- 3. • Hadoop Administrator – Jack of all trades • Bachelor of Information Science with Honors in Big Data • Interests: track racing Bradley Smith • Hadoop Administrator – Open source connoisseur • Bachelor of Information & Computer Science • Interests: game development (Unity 3D, Blender) Ian Pillay Who are we?
- 4. 155
- 5. “Transparent security focused on user experience.” Security
- 6. Enterprise security model: Proactive Reactive Prevent Predict Detect Respond and contain • POPI – Protection of Personal Information Act • GDPR – General Data Protection Regulation • PCI DSS – Payment Card Industry Data Security Standard Defining requirements - compliance
- 7. 5 pillars of enterprise security: Pillar Intent Tool Administration How do I set policy? Apache Ambari / Apache Ranger Password vault Authentication Who am I? Kerberos / LDAP Authorization What can I do? Apache Ranger / LDAP client Apache Knox Audit What did I do? Apache Ranger / Lucene indexer LDAP Client Data Protection How can I encrypt data? FPE / Tokenization / SSL
- 9. Data lake security model:
- 10. Security challenges: • Kerberos • Siloed security teams • Data encryption & tokenization • SSL • Integration conflicts • Elevated accounts
- 11. “Responsible approach to an autonomous experience.” Governance
- 14. How we govern: • Policies • Data • Resources • Compliance • POPI – Protection of Personal Information Act • GDPR – General Data Protection Regulation • PCI DSS – Payment Card Industry Data Security Standard
- 15. Chinese Wall
- 16. Governance challenges: • Internal enterprise policies • Master Data Management • Homegrown metadata solution (OPMD – Operational Metadata) • Global data ownership • Auditing • Alerting • Reporting
- 17. “All for one, and one for all” Multi-tenancy
- 19. RESOURCES QUEUES
- 20. Business Unit 1 Business Unit 2 Enterprise Lake Proprietary Data Science Workbench KVM (Active/Passive) Load Balanced Virtual Machines Application Development Test Application Development Test Repo (in DMZ) e.g. IDE Managed Queues Managed Storage Common OS Apps Production Workbench Common OS Apps
- 21. Multi-tenancy challenges: • Cloud computing • Queue performance • Transparent real time performance dashboards • Data science workbenches • Distributed application management • Legacy users • API framework • Web services
- 22. “We don’t know what we don’t know.” Disruptive technologies
- 23. Disruptive technologies: • Quantum computing • Adversarial machine learning • Cryptocurrencies • Blockchain • Mixed reality • Cloud computing • Emerging API threats • IOT • Cybersecurity wars • Autonomous cars
- 24. Q & A You now have 100% elasticity on your Q
- 25. Speaker Information • Bradley Smith • Twitter: @Tryxster • LinkedIn: https://www.linkedin.com/in/bradgsmith/ • Email Address: bradley.g.smith@outlook.com • Ian Pillay • Twitter: @IanJamie26 • LinkedIn: https://www.linkedin.com/in/ian-pillay-development/ • Email Address: miraj26@gmail.com
Editor's Notes
- Philosophy: transparent security focused on user experience
- Bank related security requirements Proactive vs reactive to solve for those requirements
- The pillars and how we are achieving them
- Security architecture High-level view of whats happening in our environment with the pillars in place
- Integrated security model Shows how the types of security initiatives is spread throughout the pillars of big data security and more indepth- how we are pulling it off
- Philosophy: responsible approach to an autonomous experience
- What does it mean to govern? What can we govern? – Focus on can, not can’t, we don’t want to limit usability/flexibility/adapatibility Data, can and can’t do MetaData (lucene indexer, Atlas) Access Rest Motion (EL) Users, can and can’t do Access Data Platform Users, data
- Users and data in harmony Governance that doesn’t prohibit data science and other users from doing their jobs. How we are achieving this, and planning to improve Comes down to AD integration throughout the bank too
- Go over what we can and can’t govern, and the types of regulations in place for governance
- The concept of the Chinese wall, No data leaves, and if it does, it is useless, which brings up encryption too for the regulations mentioned in the above slide
- Philosophy: All for one, and one for all
- What is a data lake, What is a data analytics platform What do they mean to each other and what purpose do they serve And lastly what we are doing, and why
- Balancing users with queues and resource management Go indepth on how we are using queue management with YARN QM with elasticity And go indepth in how we are handling resources with Dynamic Resource Allocation with SPARK/ YARN etc.
- Edge Node architecture overview How we split users while maintaining confidentiality and security , whilst also allowing the whole ‘multi-tenancy’ thing to the cluster, sharing responsibly.
- Be open to change
- Prepare, it is inevitable
- Question time! With a nerdy pun to accompany it