Hybrid Data Architecture: Integrating Hadoop with a Data Warehouse

•

12 likes•3,718 views

Mather Economics wanted a data architecture that could integrate Hadoop and a data warehouse to provide a responsive user experience for data slicing, aggregating, and modeling on 100% of data samples. A hybrid approach was implemented that uses Hadoop for ingestion and storage and a data warehouse for transformation, integration, and dimensional modeling to support both internal analysts and external customers. This hybrid approach meets the goals of being data and technology agnostic while providing speed for analytics.

Technology

HYBRID DATA
ARCHITECTUREIntegrating Hadoop with a Data Warehouse
Chuck Currin
Principal Consultant @ Key2 Consulting
Arvid Tchivzhel
Director @ Mather Economics

INTRODUCTIONS
Key2 Consulting – Who are we? Mather Economics – Who are we?

PROJECT BACKGROUND
• Project Team Turnover
• Requirements Refactoring
• Existing Code Review
• Speeding up the SDLC

SO, WHAT DID MATHER WANT?
• Data Agnostic
• Technology Agnostic
• Responsive End User Experience
• On the fly data slicing and aggregating
• 100% sample sizing for Modeling

Customer Experience Needs
Responsive Data Architecture
PROCESS REQUIREMENTS PYRAMID
The Architecture must
Support Ease of Data
Extensibility and Scalability
Statistical Modelling (100% Sample Size)
Capability to capture any data format

TWO SEPARATE APPROACHES
DW Hadoop
Cube
Drill
ETL
Facts
Slice
Star
Schema
Dice
Multi
Dimensional
Aggregate
Dimensions
CDC
Slowly
changing
HUE
Beeline
HBase
Cluster
HDFS
Node
Python
SQOOP
PIG
Hive

HYBRID APPROACH
SQOOP
Python
Hive
PIG
AggregateDimensions
HUE
Beeline
HBase
Cluster
HDFS
Node
Cube
Drill
ETL
Facts
Slice
Star
Schema
Dice
Multi
Dimensional
CDC
Slowly
changing

IMPLEMENTATION PYRAMID
Data Sourcing
Data Lake
Dashboard
Data Warehouse

HDFS
Customer
Interaction
Reducing &
Linking
Mapping & Storage
Segmenting &
Staging
Ingestion
Transformation &
Integration
Dimensional
Data Warehouse
Analyst
Interaction
Relational

DATA LAKE
Mapping & Storage
Segmenting &
Staging
Reducing &
Linking

DATA WAREHOUSE
Transformation &
Integration
Dimensional
Data Warehouse

USER INTERACTION
Analyst
Interaction
Customer
Interaction

DATA IS OIL… BUT ORGANIZATIONS NEED
GASOLINE
“…Data is the new oil — it’s very valuable to the companies that have it, but
only after it has been mined and processed.
The analogy makes some sense, but it ignores the fact that people and
companies don’t have the means to collect the data they need or the ability
to process it once they have it. A lot of us just need gasoline.”
Derrick Harris, Gigaom.com, March 4, 2015
Copyright 2015 Mather Economics LLC. All rights reserved.

THE GASOLINE: ANALYTICS AND
OPTIMIZATION ENABLED BY BIG DATA
• Digital Customer Lifetime Value (CLV)
• e-Commerce flow
• Advertising targeting and pricing
• Editorial programming decisions
• Product offerings, bundles, and pricing
• Acquisition targeting
• Retention strategy
• Content recommendation
Copyright 2015 Mather Economics LLC. All rights reserved.

INTEGRATING FORMERLY DISPARATE SYSTEMS TO
CREATE 360-DEGREE CUSTOMER INTERACTION
VIEW
Copyright 2015 Mather Economics LLC. All rights reserved.
Centralized Big
Data & Analytics:
• Customer- Level
• Online Data
• Offline Data
Data
Analytics
Customer
Transactions
Demographics
Online
Behavior
Marketing
Analytics
Dashboard

ENTITY RESOLUTION AND BIG DATA
• Big Data and traditional warehousing
are critical for successful entity
resolution
• Clients understand number of
customers that interact through
different channels
Copyright 2015 Mather Economics LLC. All rights reserved.
Transactions
Demographics
Registrations
Email
Campaigns
Newsletter

TRADITIONAL OFFLINE METRICS CAN BE
MEASURED BY ONLINE ENGAGEMENT
Copyright 2015 Mather Economics LLC. All rights reserved.

CUSTOMER LIFETIME VALUE
Copyright 2015 Mather Economics LLC. All rights reserved.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%0
60
120
180
240
300
360
420
480
540
600
660
720
Expected Value
(Area Under Curve)
CLV = [(Revenues – Costs)*(Expected Lifetime)]*PV – Acquisition Cost

OPERATIONALIZING CLV
• Find high/low value segments
• Integrate with campaign targeting
• Apply to pricing and product offerings
• Available to customer service reps
• Apply to online audience segmentation
• Overlay against content, psychographics…etc.
Copyright 2015 Mather Economics LLC. All rights reserved.

V
a
l
u
e
Copyright 2015 Mather Economics LLC. All rights reserved.

CUSTOMER LIFETIME VALUE HELPS MANAGE
PROFITABILITY ON A CUSTOMER LEVEL
Copyright 2015 Mather Economics LLC. All rights reserved.

CONCLUSION
• DW and Hadoop not mutually exclusive
• Mather requirements met by a hybrid approach
• Hybrid key to satisfying both internal/external users
• Speed of dimensional data critical for analytics
Copyright 2015 Mather Economics LLC. All rights reserved.

Digital transformations require a new hybrid cloud—one that’s open by design, and frees clients to choose and change environments, data and services as needed. This approach allows cloud apps and services to be rapidly composed using the best relevant data and insights available, while maintaining clear visibility, control and security—everywhere. How do you decide where to put data on a hybrid cloud and how to use it? What’s the best hybrid cloud strategy in terms of data and workload? How should you leverage a 50/50 rule or a 80/20 rule and user interaction to evaluate which data/workload to move to the cloud and which data/workload to keep on-premise? Hybrid cloud provides an open platform for innovation, including cognitive computing. Organizations are looking for taking shadow IT out of the shadows by providing a self-service way to the information and a hybrid cloud strategy is allowing that. Also, how to use hybrid cloud for better manage data sovereignty & compliance?

The Value of the Modern Data Architecture with Apache Hadoop and Teradata

Hortonworks

Extending Data Lake using the Lambda Architecture June 2015

DataWorks Summit

The Modern Data Architecture for Predictive Analytics with Hortonworks and Re...

Revolution Analytics

Hortonworks and Revolution Analytics have teamed up to bring the predictive analytics power of R to Hortonworks Data Platform. Hadoop, being a disruptive data processing framework, has made a large impact in the data ecosystems of today. Enabling business users to translate existing skills to Hadoop is necessary to encourage the adoption and allow businesses to get value out of their Hadoop investment quickly. R, being a prolific and rapidly growing data analysis language, now has a place in the Hadoop ecosystem. This presentation covers: - Trends and business drivers for Hadoop - How Hortonworks and Revolution Analytics play a role in the modern data architecture - How you can run R natively in Hortonworks Data Platform to simply move your R-powered analytics to Hadoop Presentation replay at: http://www.revolutionanalytics.com/news-events/free-webinars/2013/modern-data-architecture-revolution-hortonworks/

Hortonworks Oracle Big Data Integration

Hortonworks

Moving Health Care Analytics to Hadoop to Build a Better Predictive Model

DataWorks Summit

Come to this deep dive on how Pivotal's Data Lake Vision is evolving by embracing next generation in-memory data exchange and compute technologies around Spark and Tachyon. Did we say Hadoop, SQL, and what's the shortest path to get from past to future state? The next generation of data lake technology will leverage the availability of in-memory processing, with an architecture that supports multiple data analytics workloads within a single environment: SQL, R, Spark, batch and transactional.

Hortonworks and Red Hat Webinar_Sept.3rd_Part 1

Hortonworks

As the enterprise's big data program matures and Apache Hadoop becomes more deeply embedded in critical operations, the ability to support and operate it efficiently and reliably becomes increasingly important. To aid enterprise in operating modern data architecture at scale, Red hat and Hortonworks have collaborated to integrate Hortonworks Data Platform with Red Hat's proven platform technologies. Join us in this interactive 3-part webinar series, as we'll demonstrate how Red Hat JBoss Data Virtualization can integrate with Hadoop through Hive and provide users easy access to data.

IlOUG Tech Days 2016 - Unlock the Value in your Data Reservoir using Oracle B...

Mark Rittman

It Takes a Village: Organizational Alignment to Deliver Big Data Value in Hea...

DataWorks Summit

The business and technology teams within a health insurer must align the company’s central data platform with its data strategy. That requires substantial organizational alignment. Hear the firsthand perspective from Health Care Service Corporation (HCSC), the largest customer-owned health insurance company in the United States. The speaker will cover how they integrated membership information, regulatory compliance, and the general ledger, to improve overall healthcare management. At HCSC, the strong alignment between executive leadership, business portfolio direction, architectural strategy, technology delivery, and program management have helped create leading-edge capabilities which help the company respond nimbly to a quickly evolving healthcare industry.

Hortonworks and Clarity Solution Group

Hortonworks

Many organizations are leveraging social media to understand consumer sentiment and opinions about brands and products. Analytics in this area, however, is in its infancy and does not always provide a compelling result for effective business impact. Learn how consumer organizations can benefit by integrating social data with enterprise data to drive more profitable consumer relationships. This webinar is presented by Hortonworks and Clarity Solution Group, and will focus on the evolution of Hadoop, the clear advantage of Hortonworks distribution, and business challenges solved by “Consumer720.”

Microsoft and Hortonworks Delivers the Modern Data Architecture for Big Data

Hortonworks

Solving Big Data Problems using Hortonworks

DataWorks Summit/Hadoop Summit

IDC Retail Insights - What's Possible with a Modern Data Architecture?

Hortonworks

Software engineering practices for the data science and machine learning life...

DataWorks Summit

With the advent of newer frameworks and toolkits, data scientists are now more productive than ever and starting to prove indispensable to enterprises. Typical organizations have large teams of data scientists who build out key analytics assets that are used on a daily basis and an integral part of live transactions. However, there is also quite a lot of chaos and complexities that get introduced because of the state of the industry. Many packages used by data scientists are from open source, and even if they are well curated, there is a growing tendency to pick out the cutting-edge or unstable packages and frameworks to accelerate analytics. Different data scientists may use different versions of runtimes, different Python or R versions, or even different versions of the same packages. Predominantly data scientists work on their laptops and it becomes difficult to reproduce their environments for use by others. Since data science is now a team sport across multiple personas, involving non-practitioners, traditional application developers, execs, and IT operators, how does an enterprise create a platform for productive cross-role collaboration? Enterprises need a very reliable and repeatable process, especially when it results in something that affects their production environments. They also require a well managed approach that enables the graduation of an asset from development through a testing and staging process to production. Given the pace of businesses nowadays, the process needs to be quite agile and flexible too—even enabling an easy path to reversing a change. Compliance and audit processes require clear lineage and history as well as approval chains. In the traditional software engineering world, this lifecycle has been well understood and best practices have been followed for ages. But what does it mean when you have non-programmers or users who are not really trained in software engineering philosophies or who perceive all of this as "big process" roadblocks in their daily work ? How do you we engage them in a productive manner and yet support enterprise requirements for reliability, tracking, and a clear continuous integration and delivery practice? The presenters, in this session, will bring up interesting techniques based on their user research, real life customer interviews, and productized best practices. The presenters also invite the audience to share their stories and best practices to make this a lively conversation. Speaker Sriram Srinivasan, Senior Technical Staff Member, Analytics Platform Architect, IBM

Verizon: Finance Data Lake implementation as a Self Service Discovery Big Dat...

DataWorks Summit

Finance Data Lake objective is to create a centralized enterprise data repository for all Finance and Supply Chain data. It serves as the single source of truth. It enables a self-service discovery Analytics platform for business users to answer adhoc business questions and derive critical insights. The data lake is based on open source Hadoop big data platform and a very cost effective solution in breaking the ERP data silos and simplifying the data architecture in the enterprise. POCs were conducted on in-house Hortonworks Hadoop data platform to validate the cluster performance for Production volumes. Based on business priorities, an initial roadmap was defined using 3 data sources including 2 SAP ERPs and Peoplesoft (OLTP systems). Development environment was established in AWS Cloud for agile delivery. The near real time data ingestion architecture for the data lake was defined using replication tools and custom SQOOP based micro-batching framework and data persisted in Apache Hive DB in ORC format. Data and user security is implemented using Apache Ranger and sensitive data stored at rest in encryption zones. Business data sets were developed in Hive scripts and scheduled using Oozie. Multiple reporting tools connectivity including SQL tools, Excel and Tableau were enabled for Self-service Analytics. Upon successful implementation of the initial phase, a full roadmap is established to extend the Finance data lake to over 25 data sources and enhance data ingestion to scale as well as enable OLAP tools on Hadoop.

Hadoop 2.0: YARN to Further Optimize Data Processing

Hortonworks

Data is exponentially increasing in both types and volumes, creating opportunities for businesses. Watch this video and learn from three Big Data experts: John Kreisa, VP Strategic Marketing at Hortonworks, Imad Birouty, Director of Technical Product Marketing at Teradata and John Haddad, Senior Director of Product Marketing at Informatica. Multiple systems are needed to exploit the variety and volume of data sources, including a flexible data repository. Learn more about: - Apache Hadoop 2 and YARN - Data Lakes - Intelligent data management layers needed to manage metadata and usage patterns as well as track consumption across these data platforms.

Stream based Data Integration

Jeffrey T. Pollock

Ambari Meetup: 2nd April 2013: Teradata Viewpoint Hadoop Integration with AmbariHortonworks

The Next Generation of Big Data Analytics

Hortonworks

Apache Hadoop has evolved rapidly to become a leading platform for managing and processing big data. If your organization is examining how you can use Hadoop to store, transform, and refine large volumes of multi-structured data, please join us for this session where we will discuss, the emergence of "big data" and opportunities for deriving business value, the evolution of Apache Hadoop and future directions, essential components required in a Hadoop-powered platform, and solution architectures that integrate Hadoop with existing data discovery and data warehouse platforms.

Big Data, Hadoop, Hortonworks and Microsoft HDInsight

Hortonworks

Big Data is everywhere. And at the center of the big data discussion is Apache Hadoop, a next-generation enterprise data platform that allows you to capture, process and share the enormous amounts of new, multi-structured data that doesn’t fit into transitional systems. With Microsoft HDInsight, powered by Hortonworks Data Platform, you can bridge this new world of unstructured content with the structured data we manage today. Together, we bring Hadoop to the masses as an addition to your current enterprise data architectures so that you can amass net new insight without net new headache.

Hadoop Powers Modern Enterprise Data ArchitecturesDataWorks Summit

Rob Bearden Keynote Hadoop Summit San Jose

DataWorks Summit/Hadoop Summit

Big Data at Oracle - Strata 2015 San Jose

Jeffrey T. Pollock

Multi-tenant Hadoop - the challenge of maintaining high SLAS

DataWorks Summit

In shared configuration, the same Hadoop environment supports many applications. Each has specific requirements and criticality (SLA). Yet they all rely on an assembly of shared application bricks. At the same time, the life cycle of a cluster is not static in time. It evolves horizontally, with the arrival of new applications, but also vertically, as the applications grow in load or evolve in terms of functionality. With this in mind, a multi-tenant production cluster presents several challenges including and not limited to: - Maintain a high level of SLA for a set of use cases with heterogeneous needs - Plan and implement the architecture evolution of a cluster in production to ensure the maintenance of SLA throughout the integration of new use cases on it EDF will present how it manages this heterogeneity of SLA inherent of any Big Data cluster. EDF is focusing on how it is renovating its cluster, its organization, its processes and its approach in order to deliver a platform with strong SLA throughout its life cycle. Speaker Edouard Rousseaux, Tech Lead, EDF

Fighting Financial Crime with Artificial Intelligence

DataWorks Summit

How can we take the state of the art in deep learning and AI research, and transplant it into a large bank to deliver useful results which impact the general public? To answer this broad-reaching question, we take the viewer through a solution Think Big Analytics recently deployed at a major European bank for fraud detection, using state of the art AI techniques and a near-real time open-source architecture. We show how financial transactions can be transposed into a form where the latest AI techniques in image recognition can be leveraged, in surprisingly novel ways. We have been able to more accurately detect fraud and reduce financial crime, cutting losses and improving customer experience. We describe some architectures which can be used to do this in production, at scale, in global financial institutions. Speaker: Tim Seears, Director of Data Science, Think Big Analytics, a Teradata Company

Big Data

Charter Global

Using ML and Azure to improve Customer Lifetime Value

Navin Albert

What's hot

Alexandre Vasseur - Evolution of Data Architectures: From Hadoop to Data Lake...

NoSQLmatters

Hortonworks and Red Hat Webinar_Sept.3rd_Part 1

Hortonworks

IlOUG Tech Days 2016 - Unlock the Value in your Data Reservoir using Oracle B...

Mark Rittman

It Takes a Village: Organizational Alignment to Deliver Big Data Value in Hea...

DataWorks Summit

Hortonworks and Clarity Solution Group

Hortonworks

Microsoft and Hortonworks Delivers the Modern Data Architecture for Big Data

Hortonworks

Solving Big Data Problems using Hortonworks

DataWorks Summit/Hadoop Summit

IDC Retail Insights - What's Possible with a Modern Data Architecture?

Hortonworks

Software engineering practices for the data science and machine learning life...

DataWorks Summit

Verizon: Finance Data Lake implementation as a Self Service Discovery Big Dat...

DataWorks Summit

Hadoop 2.0: YARN to Further Optimize Data Processing

Hortonworks

Stream based Data Integration

Jeffrey T. Pollock

Ambari Meetup: 2nd April 2013: Teradata Viewpoint Hadoop Integration with AmbariHortonworks

The Next Generation of Big Data Analytics

Hortonworks

Big Data, Hadoop, Hortonworks and Microsoft HDInsight

Hortonworks

Hadoop Powers Modern Enterprise Data ArchitecturesDataWorks Summit

Rob Bearden Keynote Hadoop Summit San Jose

DataWorks Summit/Hadoop Summit

Big Data at Oracle - Strata 2015 San Jose

Jeffrey T. Pollock

Multi-tenant Hadoop - the challenge of maintaining high SLAS

DataWorks Summit

Fighting Financial Crime with Artificial Intelligence

DataWorks Summit

What's hot (20)

Alexandre Vasseur - Evolution of Data Architectures: From Hadoop to Data Lake...

Hortonworks and Red Hat Webinar_Sept.3rd_Part 1

IlOUG Tech Days 2016 - Unlock the Value in your Data Reservoir using Oracle B...

It Takes a Village: Organizational Alignment to Deliver Big Data Value in Hea...

Hortonworks and Clarity Solution Group

Microsoft and Hortonworks Delivers the Modern Data Architecture for Big Data

Solving Big Data Problems using Hortonworks

IDC Retail Insights - What's Possible with a Modern Data Architecture?

Software engineering practices for the data science and machine learning life...

Verizon: Finance Data Lake implementation as a Self Service Discovery Big Dat...

Hadoop 2.0: YARN to Further Optimize Data Processing

Stream based Data Integration

Ambari Meetup: 2nd April 2013: Teradata Viewpoint Hadoop Integration with Ambari

The Next Generation of Big Data Analytics

Big Data, Hadoop, Hortonworks and Microsoft HDInsight

Hadoop Powers Modern Enterprise Data Architectures

Rob Bearden Keynote Hadoop Summit San Jose

Big Data at Oracle - Strata 2015 San Jose

Multi-tenant Hadoop - the challenge of maintaining high SLAS

Fighting Financial Crime with Artificial Intelligence

Similar to Hybrid Data Architecture: Integrating Hadoop with a Data Warehouse

Big Data

Charter Global

Using ML and Azure to improve Customer Lifetime Value

Navin Albert

MongoDB World 2019: re:Innovate from Siloed to Deep Insights on Your Data

MongoDB

Hadoop and Your Enterprise Data Warehouse

Edgar Alejandro Villegas

Big Data BI Simplified

InMobi Technology

Today's world, Big Data seems to be buzz word and enabling BI seems to be the dream come true. In traditional world, all BI systems have run on RDBMS and embraced Star Model to enable DWH queries. Imagine, enabling the same for data lying in Hadoop clusters along with RDBMS and bringing down the barrier for business to be able to play with this data. The slides essentially covers around this theme.

Building a Modern Analytic Database with Cloudera 5.8

Cloudera, Inc.

Analytic workloads and the ability to determine “what happened” are some of the most common use cases across enterprises today - helping you understand and adapt based on changing trends. However, for most businesses today, they are only able to see a piece of the story. Analytics are limited by the amount of data able to be stored and ultimately accessed, it’s time-intensive to bring in new datasets or fit unstructured data into rigid schemas, and user access is constrained to a select few who must already know the questions they’re trying to answer. It’s no surprise that big data is disrupting this modus operandi for analytics. A modern, Hadoop-based platform is designed to help businesses break free of these analytic limitations, providing a new kind of adaptive, high-performance analytic database. The recent release of Cloudera 5.8 continues to advance Cloudera Enterprise as the foundation for these analytic workloads. Join Justin Erickson, Senior Director of Product Management at Cloudera, and Andy Frey, Chief Technology Officer at Marketing Associates, as they discuss: -What technology is needed to build a modern analytic database with Hadoop -What’s new with Cloudera 5.8 -How to align your teams around agile analytics -Real world success from Marketing Associates -What’s next for Cloudera Enterprise’s Analytic Database

Business Analytics & Big Data Trends and Predictions 2014 - 2015

Brad Culbert

Smart Enterprise Big Data Bus for the Modern Responsive Enterprise

DataWorks Summit

PTC Channel Advantage Partner Program Overview

PTC

Cloud Adoption Framework - Walking Deck (L100).pptx

Sherman37

Simplify Your Digital Journey in Progress

JK Tech

Drive Business Outcomes for Big Data Environments

Cisco Services

Business Intelligence Solutions

Charter Global

The 5 Keys to a Killer Data Lake

DataWorks Summit

In the age of IoT, most everyone is talking about data lakes. For the most part, we all agree on the value data lakes deliver, but beyond this conceptual agreement, there are still many practical questions that need answers. The key to success comes down to how data lakes are implemented and managed. Chuck Yarbrough outlines the 5 keys for creating a data lake along with strategies for defining, ingesting, governing, managing, and analyzing the data lake in ways that will enable transformative benefits in Iot and other use cases. This session will show and share how real-world data lake implementations are changing the world. Chuck focuses on automation of the data lake from ingesting data to managing metadata at scale and applying machine learning to drive significant results. Along the way, Chuck explores tools and procedures that help create a well-organized, -governed, and -managed data lake—without the risk of creating a dreaded data swamp. You’ll leave armed with the five keys to successfully creating and managing a killer data lake.

Digital Velocity London 2018 - James Morgan, Sainsbury's

Tealium

Enable Better Decision Making with Power BI Visualizations & Modern Data Estate

CCG

Vadlamudi saketh30 (ml)

Vadlamudi Saketh

Consumption based analytics enabled by Data Virtualization

Denodo

Watch full webinar here: https://buff.ly/2NM5Jtf An eclectic mix of old and new data drives every decision and every interaction, but too many organisations are attempting unsuccessfully to consolidate this data into a single repository which is time-consuming, resource-intensive, expensive, and risky. Join this Denodo and HCL Webinar to discover how data virtualization provides an effective modern day architecture and an alternative to data consolidation and the challenges of fragmented data ecosystems and traditional integration approaches. We will share stories and provide multiple perspectives on best practices and solutions. Content will include: - Business use cases that highlight challenges and solutions that result in faster time-to-market and greater ROI. - Suggested approaches to achieve extreme agility for competitive advantage.

BizTrans SysTech_Analytics_Serv_SAP_v1.0BizTrans SysTech

Connecta Event: Big Query och dataanalys med Google Cloud Platform

ConnectaDigital

Avancerad dataanalys och ”big data” har under de senaste åren klättrat på trendlistorna och är nu ett av de mest prioriterade områdena i utvecklingen av nya tjänster och produkter för ledarföretag i det digitala landskapet. Informationen som byggs upp i systemen när kundmötena digitaliseras har visat sig vara guld värt. Här finns allt vi behöver veta för att göra våra affärer mer effektiva. Sedan sommaren 2013 har Connecta tillsammans med Google ett etablerat samarbete för att hjälpa våra kunder med övergången till moln-tjänster för bland annat avancerad dataanalys. För att göra oss själva redo att hjälpa våra kunder har vi under ett antal år utvecklat såväl kunskaper som skaffat oss erfarenheter kring Googles olika moln-produkter, som exempelvis ”Big Query”. Big Query är ett molnbaserat analysverktyg och en del av Google Cloud Platform. Big Query gör det möjligt att ställa snabba frågor mot enorma dataset på bara någon sekund. Big Query och Google Cloud Platform erbjuder färdiga lösningar för att sätta upp och underhålla en infrastruktur som med enkla medel gör allt detta möjligt. På Connecta Digital Consultings tredje event för våren introducerade vi våra kunder och partners i koncepten dataanalys och Big Query. Under eventet berördes följande punkter: - Big Data och Business Intelligence (BI) - “The Google Big Data tools” – framgångsfaktorer och hur man kommer igång - Google Cloud Platform och hur man genomför en framgångsrik molnsatsning Vi presenterade case och berättade om viktiga lärdomar vi dragit i samarbetet med Google och våra kunder.

Similar to Hybrid Data Architecture: Integrating Hadoop with a Data Warehouse (20)

Big Data

Using ML and Azure to improve Customer Lifetime Value

MongoDB World 2019: re:Innovate from Siloed to Deep Insights on Your Data

Hadoop and Your Enterprise Data Warehouse

Big Data BI Simplified

Building a Modern Analytic Database with Cloudera 5.8

Business Analytics & Big Data Trends and Predictions 2014 - 2015

Smart Enterprise Big Data Bus for the Modern Responsive Enterprise

PTC Channel Advantage Partner Program Overview

Cloud Adoption Framework - Walking Deck (L100).pptx

Simplify Your Digital Journey in Progress

Drive Business Outcomes for Big Data Environments

Business Intelligence Solutions

The 5 Keys to a Killer Data Lake

Digital Velocity London 2018 - James Morgan, Sainsbury's

Enable Better Decision Making with Power BI Visualizations & Modern Data Estate

Vadlamudi saketh30 (ml)

Consumption based analytics enabled by Data Virtualization

BizTrans SysTech_Analytics_Serv_SAP_v1.0

Connecta Event: Big Query och dataanalys med Google Cloud Platform

More from DataWorks Summit

Data Science Crash Course

DataWorks Summit

Introduction: This workshop will provide a hands-on introduction to Machine Learning (ML) with an overview of Deep Learning (DL). Format: An introductory lecture on several supervised and unsupervised ML techniques followed by light introduction to DL and short discussion what is current state-of-the-art. Several python code samples using the scikit-learn library will be introduced that users will be able to run in the Cloudera Data Science Workbench (CDSW). Objective: To provide a quick and short hands-on introduction to ML with python’s scikit-learn library. The environment in CDSW is interactive and the step-by-step guide will walk you through setting up your environment, to exploring datasets, training and evaluating models on popular datasets. By the end of the crash course, attendees will have a high-level understanding of popular ML algorithms and the current state of DL, what problems they can solve, and walk away with basic hands-on experience training and evaluating ML models. Prerequisites: For the hands-on portion, registrants must bring a laptop with a Chrome or Firefox web browser. These labs will be done in the cloud, no installation needed. Everyone will be able to register and start using CDSW after the introductory lecture concludes (about 1hr in). Basic knowledge of python highly recommended.

Floating on a RAFT: HBase Durability with Apache Ratis

DataWorks Summit

In a world with a myriad of distributed storage systems to choose from, the majority of Apache HBase clusters still rely on Apache HDFS. Theoretically, any distributed file system could be used by HBase. One major reason HDFS is predominantly used are the specific durability requirements of HBase's write-ahead log (WAL) and HDFS providing that guarantee correctly. However, HBase's use of HDFS for WALs can be replaced with sufficient effort. This talk will cover the design of a "Log Service" which can be embedded inside of HBase that provides a sufficient level of durability that HBase requires for WALs. Apache Ratis (incubating) is a library-implementation of the RAFT consensus protocol in Java and is used to build this Log Service. We will cover the design choices of the Ratis Log Service, comparing and contrasting it to other log-based systems that exist today. Next, we'll cover how the Log Service "fits" into HBase and the necessary changes to HBase which enable this. Finally, we'll discuss how the Log Service can simplify the operational burden of HBase.

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi

DataWorks Summit

Utilizing Apache NiFi we read various open data REST APIs and camera feeds to ingest crime and related data real-time streaming it into HBase and Phoenix tables. HBase makes an excellent storage option for our real-time time series data sources. We can immediately query our data utilizing Apache Zeppelin against Phoenix tables as well as Hive external tables to HBase. Apache Phoenix tables also make a great option since we can easily put microservices on top of them for application usage. I have an example Spring Boot application that reads from our Philadelphia crime table for front-end web applications as well as RESTful APIs. Apache NiFi makes it easy to push records with schemas to HBase and insert into Phoenix SQL tables. Resources: https://community.hortonworks.com/articles/54947/reading-opendata-json-and-storing-into-phoenix-tab.html https://community.hortonworks.com/articles/56642/creating-a-spring-boot-java-8-microservice-to-read.html https://community.hortonworks.com/articles/64122/incrementally-streaming-rdbms-data-to-your-hadoop.html

HBase Tales From the Trenches - Short stories about most common HBase operati...

DataWorks Summit

Whilst HBase is the most logical answer for use cases requiring random, realtime read/write access to Big Data, it may not be so trivial to design applications that make most of its use, neither the most simple to operate. As it depends/integrates with other components from Hadoop ecosystem (Zookeeper, HDFS, Spark, Hive, etc) or external systems ( Kerberos, LDAP), and its distributed nature requires a "Swiss clockwork" infrastructure, many variables are to be considered when observing anomalies or even outages. Adding to the equation there's also the fact that HBase is still an evolving product, with different release versions being used currently, some of those can carry genuine software bugs. On this presentation, we'll go through the most common HBase issues faced by different organisations, describing identified cause and resolution action over my last 5 years supporting HBase to our heterogeneous customer base.

Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...

DataWorks Summit

LocationTech GeoMesa enables spatial and spatiotemporal indexing and queries for HBase and Accumulo. In this talk, after an overview of GeoMesa’s capabilities in the Cloudera ecosystem, we will dive into how GeoMesa leverages Accumulo’s Iterator interface and HBase’s Filter and Coprocessor interfaces. The goal will be to discuss both what spatial operations can be pushed down into the distributed database and also how the GeoMesa codebase is organized to allow for consistent use across the two database systems.

Managing the Dewey Decimal System

DataWorks Summit

Practical NoSQL: Accumulo's dirlist Example

DataWorks Summit

Many individuals/organizations have a desire to utilize NoSQL technology, but often lack an understanding of how the underlying functional bits can be utilized to enable their use case. This situation can result in drastic increases in the desire to put the SQL back in NoSQL. Since the initial commit, Apache Accumulo has provided a number of examples to help jumpstart comprehension of how some of these bits function as well as potentially help tease out an understanding of how they might be applied to a NoSQL friendly use case. One very relatable example demonstrates how Accumulo could be used to emulate a filesystem (dirlist). In this session we will walk through the dirlist implementation. Attendees should come away with an understanding of the supporting table designs, a simple text search supporting a single wildcard (on file/directory names), and how the dirlist elements work together to accomplish its feature set. Attendees should (hopefully) also come away with a justification for sometimes keeping the SQL out of NoSQL.

HBase Global Indexing to support large-scale data ingestion at Uber

DataWorks Summit

Data serves as the platform for decision-making at Uber. To facilitate data driven decisions, many datasets at Uber are ingested in a Hadoop Data Lake and exposed to querying via Hive. Analytical queries joining various datasets are run to better understand business data at Uber. Data ingestion, at its most basic form, is about organizing data to balance efficient reading and writing of newer data. Data organization for efficient reading involves factoring in query patterns to partition data to ensure read amplification is low. Data organization for efficient writing involves factoring the nature of input data - whether it is append only or updatable. At Uber we ingest terabytes of many critical tables such as trips that are updatable. These tables are fundamental part of Uber's data-driven solutions, and act as the source-of-truth for all the analytical use-cases across the entire company. Datasets such as trips constantly receive updates to the data apart from inserts. To ingest such datasets we need a critical component that is responsible for bookkeeping information of the data layout, and annotates each incoming change with the location in HDFS where this data should be written. This component is called as Global Indexing. Without this component, all records get treated as inserts and get re-written to HDFS instead of being updated. This leads to duplication of data, breaking data correctness and user queries. This component is key to scaling our jobs where we are now handling greater than 500 billion writes a day in our current ingestion systems. This component will need to have strong consistency and provide large throughputs for index writes and reads. At Uber, we have chosen HBase to be the backing store for the Global Indexing component and is a critical component in allowing us to scaling our jobs where we are now handling greater than 500 billion writes a day in our current ingestion systems. In this talk, we will discuss data@Uber and expound more on why we built the global index using Apache Hbase and how this helps to scale out our cluster usage. We’ll give details on why we chose HBase over other storage systems, how and why we came up with a creative solution to automatically load Hfiles directly to the backend circumventing the normal write path when bootstrapping our ingestion tables to avoid QPS constraints, as well as other learnings we had bringing this system up in production at the scale of data that Uber encounters daily.

Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix

DataWorks Summit

Recently, Apache Phoenix has been integrated with Apache (incubator) Omid transaction processing service, to provide ultra-high system throughput with ultra-low latency overhead. Phoenix has been shown to scale beyond 0.5M transactions per second with sub-5ms latency for short transactions on industry-standard hardware. On the other hand, Omid has been extended to support secondary indexes, multi-snapshot SQL queries, and massive-write transactions. These innovative features make Phoenix an excellent choice for translytics applications, which allow converged transaction processing and analytics. We share the story of building the next-gen data tier for advertising platforms at Verizon Media that exploits Phoenix and Omid to support multi-feed real-time ingestion and AI pipelines in one place, and discuss the lessons learned.

Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi

DataWorks Summit

Cybersecurity requires an organization to collect data, analyze it, and alert on cyber anomalies in near real-time. This is a challenging endeavor when considering the variety of data sources which need to be collected and analyzed. Everything from application logs, network events, authentications systems, IOT devices, business events, cloud service logs, and more need to be taken into consideration. In addition, multiple data formats need to be transformed and conformed to be understood by both humans and ML/AI algorithms. To solve this problem, the Aetna Global Security team developed the Unified Data Platform based on Apache NiFi, which allows them to remain agile and adapt to new security threats and the onboarding of new technologies in the Aetna environment. The platform currently has over 60 different data flows with 95% doing real-time ETL and handles over 20 billion events per day. In this session learn from Aetna’s experience building an edge to AI high-speed data pipeline with Apache NiFi.

Supporting Apache HBase : Troubleshooting and Supportability Improvements

DataWorks Summit

Security Framework for Multitenant Architecture

DataWorks Summit

In the healthcare sector, data security, governance, and quality are crucial for maintaining patient privacy and ensuring the highest standards of care. At Florida Blue, the leading health insurer of Florida serving over five million members, there is a multifaceted network of care providers, business users, sales agents, and other divisions relying on the same datasets to derive critical information for multiple applications across the enterprise. However, maintaining consistent data governance and security for protected health information and other extended data attributes has always been a complex challenge that did not easily accommodate the wide range of needs for Florida Blue’s many business units. Using Apache Ranger, we developed a federated Identity & Access Management (IAM) approach that allows each tenant to have their own IAM mechanism. All user groups and roles are propagated across the federation in order to determine users’ data entitlement and access authorization; this applies to all stages of the system, from the broadest tenant levels down to specific data rows and columns. We also enabled audit attributes to ensure data quality by documenting data sources, reasons for data collection, date and time of data collection, and more. In this discussion, we will outline our implementation approach, review the results, and highlight our “lessons learned.”

Presto: Optimizing Performance of SQL-on-Anything Engine

DataWorks Summit

Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Airbnb, Bloomberg, Comcast, Facebook, FINRA, LinkedIn, Lyft, Netflix, Twitter, and Uber, in the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Stores, HDFS, NoSQL and RDBMS data stores. With the ever-growing list of connectors to new data sources such as Azure Blob Storage, Elasticsearch, Netflix Iceberg, Apache Kudu, and Apache Pulsar, recently introduced Cost-Based Optimizer in Presto must account for heterogeneous inputs with differing and often incomplete data statistics. This talk will explore this topic in detail as well as discuss best use cases for Presto across several industries. In addition, we will present recent Presto advancements such as Geospatial analytics at scale and the project roadmap going forward.

Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...

DataWorks Summit

Specialized tools for machine learning development and model governance are becoming essential. MlFlow is an open source platform for managing the machine learning lifecycle. Just by adding a few lines of code in the function or script that trains their model, data scientists can log parameters, metrics, artifacts (plots, miscellaneous files, etc.) and a deployable packaging of the ML model. Every time that function or script is run, the results will be logged automatically as a byproduct of those lines of code being added, even if the party doing the training run makes no special effort to record the results. MLflow application programming interfaces (APIs) are available for the Python, R and Java programming languages, and MLflow sports a language-agnostic REST API as well. Over a relatively short time period, MLflow has garnered more than 3,300 stars on GitHub , almost 500,000 monthly downloads and 80 contributors from more than 40 companies. Most significantly, more than 200 companies are now using MLflow. We will demo MlFlow Tracking , Project and Model components with Azure Machine Learning (AML) Services and show you how easy it is to get started with MlFlow on-prem or in the cloud.

Extending Twitter's Data Platform to Google Cloud

DataWorks Summit

Twitter's Data Platform is built using multiple complex open source and in house projects to support Data Analytics on hundreds of petabytes of data. Our platform support storage, compute, data ingestion, discovery and management and various tools and libraries to help users for both batch and realtime analytics. Our DataPlatform operates on multiple clusters across different data centers to help thousands of users discover valuable insights. As we were scaling our Data Platform to multiple clusters, we also evaluated various cloud vendors to support use cases outside of our data centers. In this talk we share our architecture and how we extend our data platform to use cloud as another datacenter. We walk through our evaluation process, challenges we faced supporting data analytics at Twitter scale on cloud and present our current solution. Extending Twitter's Data platform to cloud was complex task which we deep dive in this presentation.

Event-Driven Messaging and Actions using Apache Flink and Apache NiFi

DataWorks Summit

At Comcast, our team has been architecting a customer experience platform which is able to react to near-real-time events and interactions and deliver appropriate and timely communications to customers. By combining the low latency capabilities of Apache Flink and the dataflow capabilities of Apache NiFi we are able to process events at high volume to trigger, enrich, filter, and act/communicate to enhance customer experiences. Apache Flink and Apache NiFi complement each other with their strengths in event streaming and correlation, state management, command-and-control, parallelism, development methodology, and interoperability with surrounding technologies. We will trace our journey from starting with Apache NiFi over three years ago and our more recent introduction of Apache Flink into our platform stack to handle more complex scenarios. In this presentation we will compare and contrast which business and technical use cases are best suited to which platform and explore different ways to integrate the two platforms into a single solution.

Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger

DataWorks Summit

Companies are increasingly moving to the cloud to store and process data. One of the challenges companies have is in securing data across hybrid environments with easy way to centrally manage policies. In this session, we will talk through how companies can use Apache Ranger to protect access to data both in on-premise as well as in cloud environments. We will go into details into the challenges of hybrid environment and how Ranger can solve it. We will also talk through how companies can further enhance the security by leveraging Ranger to anonymize or tokenize data while moving into the cloud and de-anonymize dynamically using Apache Hive, Apache Spark or when accessing data from cloud storage systems. We will also deep dive into the Ranger’s integration with AWS S3, AWS Redshift and other cloud native systems. We will wrap it up with an end to end demo showing how policies can be created in Ranger and used to manage access to data in different systems, anonymize or de-anonymize data and track where data is flowing.

Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...

DataWorks Summit

Advanced Big Data Processing frameworks have been proposed to harness the fast data transmission capability of Remote Direct Memory Access (RDMA) over high-speed networks such as InfiniBand, RoCEv1, RoCEv2, iWARP, and OmniPath. However, with the introduction of the Non-Volatile Memory (NVM) and NVM express (NVMe) based SSD, these designs along with the default Big Data processing models need to be re-assessed to discover the possibilities of further enhanced performance. In this talk, we will present, NRCIO, a high-performance communication runtime for non-volatile memory over modern network interconnects that can be leveraged by existing Big Data processing middleware. We will show the performance of non-volatile memory-aware RDMA communication protocols using our proposed runtime and demonstrate its benefits by incorporating it into a high-performance in-memory key-value store, Apache Hadoop, Tez, Spark, and TensorFlow. Evaluation results illustrate that NRCIO can achieve up to 3.65x performance improvement for representative Big Data processing workloads on modern data centers.

Computer Vision: Coming to a Store Near You

DataWorks Summit

Background: Some early applications of Computer Vision in Retail arose from e-commerce use cases - but increasingly, it is being used in physical stores in a variety of new and exciting ways, such as: ● Optimizing merchandising execution, in-stocks and sell-thru ● Enhancing operational efficiencies, enable real-time customer engagement ● Enhancing loss prevention capabilities, response time ● Creating frictionless experiences for shoppers Abstract: This talk will cover the use of Computer Vision in Retail, the implications to the broader Consumer Goods industry and share business drivers, use cases and benefits that are unfolding as an integral component in the remaking of an age-old industry. We will also take a ‘peek under the hood’ of Computer Vision and Deep Learning, sharing technology design principles and skill set profiles to consider before starting your CV journey. Deep learning has matured considerably in the past few years to produce human or superhuman abilities in a variety of computer vision paradigms. We will discuss ways to recognize these paradigms in retail settings, collect and organize data to create actionable outcomes with the new insights and applications that deep learning enables. We will cover the basics of object detection, then move into the advanced processing of images describing the possible ways that a retail store of the near future could operate. Identifying various storefront situations by having a deep learning system attached to a camera stream. Such things as; identifying item stocks on shelves, a shelf in need of organization, or perhaps a wandering customer in need of assistance. We will also cover how to use a computer vision system to automatically track customer purchases to enable a streamlined checkout process, and how deep learning can power plausible wardrobe suggestions based on what a customer is currently wearing or purchasing. Finally, we will cover the various technologies that are powering these applications today. Deep learning tools for research and development. Production tools to distribute that intelligence to an entire inventory of all the cameras situation around a retail location. Tools for exploring and understanding the new data streams produced by the computer vision systems. By the end of this talk, attendees should understand the impact Computer Vision and Deep Learning are having in the Consumer Goods industry, key use cases, techniques and key considerations leaders are exploring and implementing today.

Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark

DataWorks Summit

Whole genome shotgun based next generation transcriptomics and metagenomics studies often generate 100 to 1000 gigabytes (GB) sequence data derived from tens of thousands of different genes or microbial species. De novo assembling these data requires an ideal solution that both scales with data size and optimizes for individual gene or genomes. Here we developed an Apache Spark-based scalable sequence clustering application, SparkReadClust (SpaRC), that partitions the reads based on their molecule of origin to enable downstream assembly optimization. SpaRC produces high clustering performance on transcriptomics and metagenomics test datasets from both short read and long read sequencing technologies. It achieved a near linear scalability with respect to input data size and number of compute nodes. SpaRC can run on different cloud computing environments without modifications while delivering similar performance. In summary, our results suggest SpaRC provides a scalable solution for clustering billions of reads from the next-generation sequencing experiments, and Apache Spark represents a cost-effective solution with rapid development/deployment cycles for similar big data genomics problems.

More from DataWorks Summit (20)

Data Science Crash Course

Floating on a RAFT: HBase Durability with Apache Ratis

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi

HBase Tales From the Trenches - Short stories about most common HBase operati...

Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...

Managing the Dewey Decimal System

Practical NoSQL: Accumulo's dirlist Example

HBase Global Indexing to support large-scale data ingestion at Uber

Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix

Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi

Supporting Apache HBase : Troubleshooting and Supportability Improvements

Security Framework for Multitenant Architecture

Presto: Optimizing Performance of SQL-on-Anything Engine

Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...

Extending Twitter's Data Platform to Google Cloud

Event-Driven Messaging and Actions using Apache Flink and Apache NiFi

Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger

Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...

Computer Vision: Coming to a Store Near You

Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark

Recently uploaded

National Security Agency - NSA mobile device best practices

Quotidiano Piemontese

20240605 QFM017 Machine Intelligence Reading List May 2024

Matthew Sinclair

FIDO Alliance Osaka Seminar: Overview.pdf

FIDO Alliance

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

FIDO Alliance

GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...

Neo4j

Leonard Jayamohan, Partner & Generative AI Lead, Deloitte This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.

Epistemic Interaction - tuning interfaces to provide information for AI support

Alan Dix

Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024 https://alandix.com/academic/papers/synergy2024-epistemic/ As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.

GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...

Neo4j

Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.

Generative AI Deep Dive: Advancing from Proof of Concept to Production

Aggregage

GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...

Neo4j

Dr. Sean Tan, Head of Data Science, Changi Airport Group Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.

Monitoring Java Application Security with JDK Tools and JFR Events

Ana-Maria Mihalceanu

Elizabeth Buie - Older adults: Are we really designing for our future selves?

Nexer Digital

Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx

nkrafacyberclub

Essentials of Automations: The Art of Triggers and Actions in FME

Safe Software

In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation. We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios. Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!

Communications Mining Series - Zero to Hero - Session 1

DianaGray10

This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered: • Communication Mining Overview • Why is it important? • How can it help today’s business and the benefits • Phases in Communication Mining • Demo on Platform overview • Q/A

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Prayukth K V

The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development. The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers: State of global ICS asset and network exposure Sectoral targets and attacks as well as the cost of ransom Global APT activity, AI usage, actor and tactic profiles, and implications Rise in volumes of AI-powered cyberattacks Major cyber events in 2024 Malware and malicious payload trends Cyberattack types and targets Vulnerability exploit attempts on CVEs Attacks on counties – USA Expansion of bot farms – how, where, and why In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East Why are attacks on smart factories rising? Cyber risk predictions Axis of attacks – Europe Systemic attacks in the Middle East Download the full report from here: https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/

Securing your Kubernetes cluster_ a step-by-step guide to success !

KatiaHIMEUR1

Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster. However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks. In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

FIDO Alliance

A tale of scale & speed: How the US Navy is enabling software delivery from l...

sonjaschweigert1

Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved: - Reduction in onboarding time from 5 weeks to 1 day - Improved developer experience and productivity through actionable findings and reduction of false positives - Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO) Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production. We will cover: - How to remove silos in DevSecOps - How to build efficient development pipeline roles and component templates - How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence) - How to streamline operations with automated policy checks on container images

Video Streaming: Then, Now, and in the Future

Alpen-Adria-Universität

In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.

20240607 QFM018 Elixir Reading List May 2024

Matthew Sinclair

Recently uploaded (20)