Дмитрий Лавриненко "Blockchain for Identity Management, based on Fast Big Data"

•

0 likes•346 views

Dmitry Lavrinenko is a solutions architect who specializes in blockchain for identity management, big data, and related technologies. He proposes a vendor-agnostic SaaS platform that utilizes fast data technologies like continuous loading, parallel processing, and data consolidation. The solution would include a data warehouse, processing, analytics, visualization, machine learning, and identity management capabilities. Blockchain provides benefits like cryptographic security, privacy, consensus, audibility and smart contracts for managing identities. The proposed architecture features data lakes, batch and speed layers, serving layers, storage, functions, streaming, and machine learning components.

Technology

Dmitry Lavrinenko
SoftServe
Blockchain for Identity Management, based on Fast Big Data

About Me
• Solutions Architect
• DevOps
• IoT
• Big Data
• Blockchain
• AI
• ML
dlavrin@softserveinc.com

Business Goal
Deliver a vendor-agnostic and open-source based SaaS platform
by using most up to date automation technologies and utilizing the
partnership
with major software vendors and independent developers.

What is Fast Data for us
• Continuous data loading
• Massively parallel processing
• Data consolidation
• Dimensional processing
• Data normalization & denormalization (depends on tech stack)
• Structured & dimensional data models
• Hybrid distributed warehouses

What is Fast & Big Data Solution
• Data WH
• Processing
• Analytics
• Visualisation
• Machine Learning
• Data Virtualization
• Data Ingestion
• Identity 
As a Service

Why Blockchain
• Cryptographic environment
• IoT Privacy
• Blockchain Identity
• Consensus
• Integration
• Internal assurance
• Resilience  
• Tracking
• Smart contracts
• Digital fingerprint
• Audibility

Portal
Solution Architecture
Event
Strea
m
data
cach
retention
scenario
s
Data Lake
Batch layer
Speed layer
Serving layer
Consumer
Dashboard
Storage
Stream
Serving
Storage
Function
s
Stream
Processin
Ad-Hoc  
Queries
BI
Analytics
Visualizati
on engine
ML
ML
IoT Edge
Orchestratio
n
P2P
Blockchain
Storage
Consens
us
Data
Storage

Blockchain Layer Architecture
Batch LayerDistributed Data Storage
Smart
IoT Edge
Orchestratio
n
P2
P
Serving
Layer
API layer
Consensus
Identity
Blockchain
framework

Data
Integration &
Processing
Data
Warehouse
BI
Platforms
Big Data &
NoSQL
Data
Science
Using Best-of-Breed Technologies

• ~ 10,000 Identity requests
• ~1,000 per min
• Independent clusters in scale
• Independent Streaming in scale
• Distributed Data Storage
• Custom Analytics platforms, cloud
based.
• Machine learning on flight
Identity Processing: Facts

Thank you
Dmitry Lavrinenko, DevOps Solutions Architect

We will start from understanding how Real-Time Analytics can be implemented on Enterprise Level Infrastructure and will go to details and discover how different cases of business intelligence be used in real-time on streaming data. We will cover different Stream Data Processing Architectures and discus their benefits and disadvantages. I'll show with live demos how to build Fast Data Platform in Azure Cloud using open source projects: Apache Kafka, Apache Cassandra, Mesos. Also I'll show examples and code from real projects.

Azure Big Data Story

Lynn Langit

This document discusses Azure big data capabilities including the 5 V's of big data: volume, velocity, variety, veracity, and value. It notes that 60% of big data projects fail to move beyond pilot according to Gartner. It then provides details on Azure persistence choices for storing big data including storage, Data Lake, HDInsight, DocumentDB, SQL databases, and Hadoop options. It also discusses load and data cleaning choices on Azure like Stream Analytics, SQL Server, and Azure Machine Learning. Finally, it presents 5 architectural patterns for using Azure big data capabilities.

Case studies session 2

HBaseCon

Blackbird is Rocket Fuel's real-time bidding platform that processes billions of rows and bid requests in milliseconds. It uses HBase for its data store and has implemented various optimizations to achieve high performance, such as using protocol buffers, caching, append-only writes, and off-peak compactions. It also focuses on reliability through techniques like aggressive monitoring, dynamic blacklisting, and handling client failures without relying on a proxy server.

Presto: Fast SQL on Everything

David Phillips

Presto is an open source distributed SQL query engine that was originally developed by Facebook. It allows for fast SQL queries on large datasets across multiple data sources. Presto uses various optimizations like code generation, predicate pushdown, and data layout awareness to improve query performance. It is used at Facebook and other companies for interactive analytics, batch ETL, A/B testing, and app analytics where low latency and high concurrency are important.

Move your on prem data to a lake in a Lake in Cloud

CAMMS

With the boom in data; the volume and its complexity, the trend is to move data to the cloud. Where and How do we do this? Azure gives you the answer. In this session, I will give you an introduction to Azure Data Lake and Azure Data Factory, and why they are good for the type of problem we are talking about. You will learn how large datasets can be stored on the cloud, and how you could transport your data to this store. The session will briefly cover Azure Data Lake as the modern warehouse for data on the cloud,

MongoDB on Azure

Norberto Leite

Disney+ Hotstar: Scaling NoSQL for Millions of Video On-Demand Users

ScyllaDB

Disney+ Hotstar is the fastest growing branch of Disney+. Join Disney+ Hotstar Architect Vamsi Subhash and senior data engineer Balakrishnan Kaliyamoorthy to learn… How Disney+ Hotstar architected their systems to handle massive data loads Why they chose to replace both Redis and Elasticsearch Their requirements for massively scalable data infrastructure and evolving data models How they migrated their data to Scylla Cloud, ScyllaDB’s fully managed NoSQL database-as-a-service, without suffering downtime

Vitalii Bondarenko "Machine Learning on Fast Data"

DataConf

This document discusses machine learning on fast data. It presents an agenda covering ML on production systems, TensorFlow, Kafka, Docker and Kubernetes. It then describes the machine learning process and shows how an enterprise analytics platform can integrate data sources, a machine learning cluster using Kafka, and data destinations. Details are provided on using TensorFlow for linear regression and neural networks. Apache Kafka is explained as a distributed streaming platform using topics, brokers, and consumer groups. The Confluent platform, KStream and KTable APIs are also summarized. Docker and Kubernetes are mentioned for containerization.

This document discusses MongoDB sharding as a case study for scaling MongoDB. It provides background on CIGNEX Datamatics and their big data analytics practice. It then describes a use case of 7 million users accessing digital assets across 8 devices each. It recommends MongoDB due to its flexibility and performance. The solution involves sharding across multiple MongoDB nodes to distribute the data and handle the high volume of concurrent requests. Benchmarking shows that sharding significantly improves performance of inserts and updates over non-sharded architecture. The key takeaway is that sharding is very effective but requires careful planning, benchmarking, and choice of shard key.

Bootstrap SaaS startup using Open Source Tools

botsplash.com

An afternoon with mongo db new delhi

Rajnish Verma

The document summarizes MongoDB as a modern database designed to solve problems of volume, velocity, and variety of data that traditional relational databases are not well-suited for. It highlights key MongoDB features like scalability, flexible schemas, and high availability. The document also discusses how MongoDB compares favorably to other databases in security capabilities and is a good fit for applications involving user data management, content delivery, and mobile apps.

How to Build Modern Data Architectures Both On Premises and in the Cloud

VMware Tanzu

Enterprises are beginning to consider the deployment of data science and data warehouse platforms on hybrid (public cloud, private cloud, and on premises) infrastructure. This delivers the flexibility and freedom of choice to deploy your analytics anywhere you need it and to create an adaptable and agile analytics platform. But the market is conspiring against customer desire for innovation... Leading public cloud vendors are interested in pushing their new, but proprietary, analytic stacks, locking customers into subpar Analytics as a Service (AaaS) for years to come. In tandem, Legacy Data Warehouse vendors are trying to extend the lifecycle of their costly and aging appliances with new features of marginal value, simply imitating the same limiting models of public cloud vendors. New vendors are coming up with interesting ideas, but these ideas are often lacking critical features that don’t provide support for hybrid solutions, limiting the immediate value to users. It is 2017—you can, in fact, have your analytics cake and eat it too! Solve your short term costs and capabilities challenges, and establish a long term hybrid data strategy by running the same open source analytics platform on your infrastructure as it exists today. In this webinar you will learn how Pivotal can help you build a modern analytical architecture able to run on your public, private cloud, or on-premises platform of your choice, while fully leveraging proven open source technologies and supporting the needs of diverse analytical users. Let’s have a productive discussion about how to deploy a solid cloud analytics strategy. Presenter : Jacque Istok, Head of Data Technical Field for Pivotal https://content.pivotal.io/webinars/jul-20-how-to-build-modern-data-architectures-both-on-premises-and-in-the-cloud

Shift: Real World Migration from MongoDB to Cassandra

DataStax

DBP-010_Using Azure Data Services for Modern Data Applications

decode2016

This document discusses using Azure data services for modern data applications based on the Lambda architecture. It covers ingestion of streaming and batch data using services like Event Hubs, IoT Hubs, and Kafka. It describes processing streaming data in real-time using Stream Analytics, Storm, and Spark Streaming, and processing batch data using HDInsight, ADLA, and Spark. It also covers staging data in data lakes, SQL databases, NoSQL databases and data warehouses. Finally, it discusses serving and exploring data using Power BI and enriching data using Azure Data Factory and Machine Learning.

Cassandra Community Webinar: MySQL to Cassandra - What I Wish I'd Known

DataStax

A brief intro to how Barracuda Networks uses Cassandra and the ways in which they are replacing their MySQL infrastructure, with Cassandra. This presentation will include the lessons they've learned along the way during this migration. Speaker: Michael Kjellman, Software Engineer at Barracuda Networks Michael Kjellman is a Software Engineer, from San Francisco, working at Barracuda Networks. Michael works across multiple products, technologies, and languages. He primarily works on Barracuda's spam infrastructure and web filter classification data.

MongoDB vs Mysql. A devops point of view

Pierre Baillet

Scylla Summit 2018: Adventures in AdTech: Processing 50 Billion User Profiles...

ScyllaDB

AdTech requires high speed at massive scale. Sizmek serves millions of requests every second. Requests need to be processed in tens of milliseconds, while involving 10 simultaneous lookups into a database that contains tens of billions of profiles. In this presentation, you will discover how Scylla enables Sizmek’s real-time bidders to query a gigantic user profile store quickly and reliably with only a few nodes. We’ll discuss data modeling, server and driver configuration, techniques to minimize disk access, as well as considerations for leveraging Spark while migrating from HBase.

Converging Database Transactions and Analytics

SingleStore

MongoDB San Francisco 2013: Storing eBay's Media Metadata on MongoDB present...

MongoDB

This session will be a case study of eBay’s experience running MongoDB for project Zoom, in which eBay stores all media metadata for the site. This includes references to pictures of every item for sale on eBay. This cluster is eBay's first MongoDB installation on the platform and is a mission critical application. Yuri Finkelstein, an Enterprise Architect on the team, will provide a technical overview of the project and its underlying architecture.

What's new in MongoDB 2.6

Matias Cascallares

The document summarizes the key improvements in MongoDB version 2.6, including improved operations, integrated search capabilities, query system enhancements, improved security features, and better performance and stability. Some of the main updates are bulk write operations, background indexing and replication, storage allocation improvements to reduce fragmentation, full text search integration, index intersection capabilities, aggregation framework enhancements, and auditing functionality. The presentation provides details on each of these areas.

Integration Monday - Analysing StackExchange data with Azure Data Lake

Tom Kerkhove

Big data is the new big thing where storing the data is the easy part. Gaining insights in your pile of data is something different. Based on a data dump of the well-known StackExchange websites, we will store & analyse 150+ GB of data with Azure Data Lake Store & Analytics to gain some insights about their users. After that we will use Power BI to give an at a glance overview of our learnings. If you are a developer that is interested in big data, this is your time to shine! We will use our existing SQL & C# skills to analyse everything without having to worry about running clusters.

The Evolution of the Fashion Retail Industry in the Age of AI with Kshitij Ku...

Databricks

AI is fundamentally transforming how we live and work. Zalando is a data driven company. We deliver an optimal customer experience that drives engagement. We continue to improve this experience by leveraging the latest technologies and machine learning techniques — such as building a cutting edge cloud based infrastructure to support our operations at scale. We provide our data scientists across Zalando with the means to implement artificial intelligence use cases, leveraging data from all parts of our company and the best machine learning techniques from across the industry. Apache Spark delivered through Databricks is at the core of this strategy. In this keynote, I’ll share our AI journey thus far, and share how we are exploring ways to unify data through A.I. with Spark and Databricks.

Not only SQL - Database Choices

Lynn Langit

This document discusses database choices and provides an overview of different database technologies including relational databases, NoSQL databases, and Hadoop. It highlights key-value, columnar, document, and graph NoSQL databases and provides demos of technologies like DynamoDB, MongoDB, Neo4j, and Hadoop. The document also discusses using these database options on premises or in the cloud with providers like AWS, Google, and Microsoft and how to query data from NoSQL databases.

Data Privacy with Apache Spark: Defensive and Offensive Approaches

Databricks

In this talk, we’ll compare different data privacy techniques & protection of personally identifiable information and their effects on statistical usefulness, re-identification risks, data schema, format preservation, read & write performance. We’ll cover different offense and defense techniques. You’ll learn what k-anonymity and quasi-identifier are. Think of discovering the world of suppression, perturbation, obfuscation, encryption, tokenization, watermarking with elementary code examples, in case no third-party products cannot be used. We’ll see what approaches might be adopted to minimize the risks of data exfiltration.

MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

MongoDB

This document discusses MongoDB implementation at Medtronic Energy and Component Center (MECC) to address data management challenges. MECC manufactures components for medical devices and generates a large volume of operational data from various sources. Previously, this data was stored in spreadsheets and relational databases, making it difficult to analyze. MongoDB was implemented to provide a flexible schema for storing component manufacturing and test data as documents. This has allowed for faster querying of complete historical data and improved reporting and analytics. While some gaps remain around enterprise acceptance and tool integration, MongoDB has provided benefits over the previous data management approaches.

Webinar: Faster Big Data Analytics with MongoDB

MongoDB

Learn how to leverage MongoDB and Big Data technologies to derive rich business insight and build high performance business intelligence platforms. This presentation includes: - Uncovering Opportunities with Big Data analytics - Challenges of real-time data processing - Best practices for performance optimization - Real world case study This presentation was given in partnership with CIGNEX Datamatics.

Maximizing MongoDB Performance on AWS

MongoDB

AWS is an incredibly popular environment for running MongoDB deployments. Today you have many choices about instance type, storage, network config, security, how you configure MongoDB processes, and more. In addition, you now have options when it comes to tooling to help you manage and operate your deployment. In this session, we’ll take a look at several recommendations that can help you get the best performance out of AWS.

TechDays NL 2016 - Building your scalable secure IoT Solution on Azure

Tom Kerkhove

The Internet-of-Things was one of the big hypes in 2015 but it’s more than that – Customers want to build out their own infrastructures and act on their data. Today we’ll look at how Microsoft Azure helps us to build scalable solutions to process events from thousands of devices in a secure manner and the challenges it has. Once the data is in the cloud we’ll also take a look at ways we can learn from our measurements.

Дмитрий Лавриненко "Big & Fast Data for Identity & Telemetry services"

Дмитрий Лавриненко "Blockchain for Identity Management, based on Fast Big Data"

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Дмитрий Лавриненко "Blockchain for Identity Management, based on Fast Big Data"

Similar to Дмитрий Лавриненко "Blockchain for Identity Management, based on Fast Big Data" (20)

More from Fwdays

More from Fwdays (20)

Recently uploaded

Recently uploaded (20)

Дмитрий Лавриненко "Blockchain for Identity Management, based on Fast Big Data"