PLNOG 3: Tomasz Mikołajczyk - Data scalability. Why you should care?

•

0 likes•12 views

This document discusses data scalability and introduces GridwiseTech, a vendor-independent scalable technology expert. It explains that IT systems are constantly growing due to increased users, applications, and data which can lead to infrastructure bottlenecks. To improve efficiency, GridwiseTech introduces scalability through distributed processing, load balancing, and scaling out data. It then summarizes a case study where GridwiseTech helped an electronic manufacturer scale its infrastructure to ensure scalability on each functional layer and achieve significant performance improvements like 10x faster data processing.

Technology

Data scalability. Why you
should care?
Tomasz Mikolajczyk
& the GridwiseTech team
www.gridwisetech.com
tomek@gridwisetech.com

GridwiseTech www.gridwisetech.com
Presentation survey
Presentation plan
1) About GridwiseTech
2) Building efficient database oriented, IT systems
3) Momentum + case study
Presentation duration
40 minutes

GridwiseTech www.gridwisetech.com
About GridwiseTech
Leading vendor-
independent
scalable technology
expert.
Solve scalability and
performance problems
with virtualization,
Grid, distributed data
processing, caching.
Implement vision for
distributed
infrastructures.
Mission: turn avant-garde technology into customer’s business benefits
About us

GridwiseTech www.gridwisetech.com
Chosen customers
Chosen partners
Our references
About us

GridwiseTech www.gridwisetech.com
Building efficient IT system
About us
Application
WWW
SOA
Middleware
Processing
Data provisioning
Data persistence

GridwiseTech www.gridwisetech.com
Vendors puzzle
About us
Application
WWW
SOA
Platform Computing, Sun
Microsystems, Altair,
UnivaUD, Cluster Resources
Gigaspaces, Gemstone,
Coherence, eXludus,
Terracotta
Oracle, IBM, Microsoft
PostreSQL, MySQL
Citrix, Vmvare,Novell, Virtual
Iron, RedHat

GridwiseTech www.gridwisetech.com
Improving the system efficiency

GridwiseTech www.gridwisetech.com
Simple data oriented IT system
Internal or
external
data sources
applications
working on data

GridwiseTech www.gridwisetech.com
IT systems are constantly growing
Increased number
of users
Increased number
of applications
Increased amount
of data

GridwiseTech www.gridwisetech.com
IT systems are constantly growing
Infrastructure
bottleneck

GridwiseTech www.gridwisetech.com
Hardware scale up – the first idea
Typical solution:
buy bigger and faster
hardware

GridwiseTech www.gridwisetech.com
Hardware scale up – the first idea
Recurring software and
hardware migrations will
always generate substantial
costs

GridwiseTech www.gridwisetech.com
Introduction of scalability
Simultaneous
calculation
Load balancing

GridwiseTech www.gridwisetech.com
Introduction of scalability
Simultaneous
calculation
Cloud
Load balancing

GridwiseTech www.gridwisetech.com
Introduction of distributed processing
Cloud
I/O bottleneck

GridwiseTech www.gridwisetech.com
Problem with data access?
Database / configuration / storage optimization
In-memory databases
Database clustering
Sharding
Data grids
Distributed caching

GridwiseTech www.gridwisetech.com
Fully scalable system
Scaling out data

GridwiseTech www.gridwisetech.com
Project for electronic manufacturer
Technical scope of the problem:
2,5 million transactions per day
many databases, different vendors and/or
versions
4TB of data to maintain
constant environment modifications
400,000 lines of PL/SQL code to analyze
Improve report generation time by 70% allowing for increase in daily production.
Project main goal
Business problem:
lack of possibility to increase production
limited process manageability due to report
availability delay
Repetitive infrastructure down time
resulting in data inconstancy
Data normalization problem due to many
different data sources

GridwiseTech www.gridwisetech.com
Initial customer architecture
1
2

GridwiseTech www.gridwisetech.com
Designed architecture
Distributed caching

GridwiseTech www.gridwisetech.com
Momentum

GridwiseTech www.gridwisetech.com
Project for electronic manufacturer
Results of scaling manufacturer infrastructure:
Scalability ensured on each functional layer
Real-time report generation
10x times data processing speed-up
100,000 of reads/writes per second (7500 before)
Fully manageable production process
Improve report generation time by 70% allowing for increase in daily production.
Project main goal

GridwiseTech www.gridwisetech.com
GridwiseTech's value add
GridwiseTech's value add:
 Application for automatic mapping database tables into
objects.
 Scalable Feeder application that provide fast data replication
between multiple data sources and data grid.
 Data normalization from multiple data sources.
 Mechanism, that prevent from RAM overloading.
 Mechanism, that provide synchronisation possibility between
services and applications.
 Technique of building thread enabled and fully scalable
external services with batch processing support.

Thank you
www.gridwisetech.com
contact@gridwisetech.com

A world-leading manufacturer was in search of an IoT solution that could ingest, integrate, and manage data being generated from various types of connected machinery located on factory floors around the globe. The company needed to manage the devices generating the data, integrate the flow of data into existing back-end systems, run advanced analytics on that data, and then deliver services to generate real-time decision making at the edge. In this session, learn how Clairvoyant, a leading systems integrator and Red Hat partner, was able to accelerate digital transformation for their customer using Internet of Things (IoT) and machine learning in a hybrid cloud environment. Specifically, Clairvoyant and Eurotech will discuss: • The approach taken to optimize manufacturing processes to cut costs, minimize downtime, and increase efficiency. • How a data processing pipeline for IoT data was built using an open, end-to-end architecture from Cloudera, Eurotech, and Red Hat. • How analytics and machine learning inferencing powered at the IoT edge will allow predictions to be made and decisions to be executed in real time. • The flexible and hybrid cloud environment designed to provide the key foundational elements to quickly and securely roll out IoT use cases.

The Yellowbrick Impact for MicroStrategy

Yellowbrick Data

How Yellowbrick Data Integrates to Existing Environments Webcast

Yellowbrick Data

This document discusses how Yellowbrick can integrate into existing data environments. It describes Yellowbrick's data warehouse capabilities and how it compares to other solutions. The document recommends upgrading from single server databases or traditional MPP systems to Yellowbrick when data outgrows a single server or there are too many disparate systems. It also recommends moving from pre-configured or cloud-only systems to Yellowbrick to significantly reduce costs while improving query performance. The document concludes with a security demonstration using a netflow dataset.

Leveraging a big data model in the IT domain

VSS Monitoring

Case Study: Big Data Analytics

Abhinav Das

The client is a US-based product company that offers an enterprise search and discovery platform based on big data technologies. They added a new SQL interface and wanted to benchmark its performance against other alternatives like open source Hadoop+Hive. TenXLabs researched the TPC-H benchmark, generated test data, built clusters for the client's product and Hadoop+Hive, and developed a test harness and web tool to run queries and collect performance statistics to compare the systems. The client was able to identify areas for improvement and was surprised their product performed well compared to MySQL in some cases.

Agile Data Management with Enterprise Data Fabric (Middle East)

Denodo

Watch full webinar here: https://bit.ly/3td9ICb In a world where machine learning and artificial intelligence are changing our everyday lives, digital transformation tops the strategic agenda in many private and government organizations. Data is becoming the lifeblood of a company, flowing seamlessly through it to enable deep business insights, create new opportunities, and optimize operations. Chief Data Officers and Data Architects are under continuous pressure to find the best ways to manage the overwhelming volumes of the data that tend to become more and more distributed and diverse. Moving data physically to a single location for reporting and analytics is not an option anymore – this is the fact accepted by the majority of the data professionals. Join us for this webinar to know about the modern virtual data landscapes including: - Virtual Data Fabric - Data Mesh - Multi-Cloud Hybrid architecture and to learn how to leverage Denodo Data Virtualization platform to implement these modern data architectures.

VP of WW Partners by Alan Chhabra

Big Data Spain

Bigquery 101

Cesar Orozco Manotas

This document discusses key capabilities needed for real-time analytics. It notes that real-time data, combined with historical data, provides important context for decision making. Building data pipelines with fewer systems and steps leads to greater scalability and reliability. The document outlines needs for real-time analytics like ingesting streaming data, powering analytic applications, delivering massive capacity, and guaranteeing performance. It emphasizes that both real-time and historical data are important for analytics and that the right architecture can incorporate multiple data sources and workloads.

Data Virtualization to Survive a Multi and Hybrid Cloud World

Denodo

Watch full webinar here:https://buff.ly/2Edqlpo Hybrid cloud computing is slowing becoming the standard for businesses. The transition to hybrid can be challenging depending on the environment and the needs of the business. A successful move will involve using the right technology and seeking the right help. At the same time, multi-cloud strategies are on the rise. More enterprise organizations than ever before are analyzing their current technology portfolio and defining a cloud strategy that encompasses multiple cloud platforms to suit specific app workloads, and move those workloads as they see fit. In this session, you will learn: *Key challenges of migration to the cloud in a complex data landscape *How data virtualization can help build a data driven, multi-location cloud architecture for real time integration *How customers are taking advantage of data virtualization to save time and costs with limited resources

Data Virtualization enabled Data Fabric: Operationalize the Data Lake (APAC)

Denodo

Watch full webinar here: https://bit.ly/3aIofv9 The best of breed big data fabrics should deliver actionable insights to the business users with minimal effort, provide end-to-end security to the entire enterprise data platform and provide real-time data integration, while delivering self-service data platform to business users. While big data initiatives have become necessary for any business to generate actionable insights, big data fabric has become a necessity for any successful big data initiative. The best of breed big data fabrics should deliver actionable insights to the business users with minimal effort, provide end-to-end security to the entire enterprise data platform and provide real-time data integration, while delivering self-service data platform to business users. Attend this session to learn how big data fabric enabled by data virtualization: - Provides lightning fast self-service data access to business users - Centralizes data security, governance and data privacy - Fulfills the promise of data lakes to provide actionable insights

End to End Supply Chain Control Tower

Databricks

The Virtualization of Clouds - The New Enterprise Data Architecture Opportunity

Denodo

Watch full webinar here: https://bit.ly/3x7xVuR Organizations worldwide are adopting a variety of the public cloud service providers (i.e. AWS, Google, Microsoft) and each have a portfolio of storage, compute, network, and security options. All of which create significant challenges in managing a hybrid and multi-cloud enterprise architecture. Even worse is the impact to the governance and integration of data from the clouds and physical infrastructure to support the broad array of analytics and operational requirements. Can one public cloud provider meet all your needs today and in the future? How do you manage across multiple public and private clouds you have today and where your data exists? And, how would you manage and operate your multi-cloud and on-premises systems to gain value from your data in any of them? The Chief Research Officer at Ventana Research, Mark Smith, will expound the challenges and path ahead for virtualization and integration of your data and the clouds, setting an architectural path for best success.

Partner Keynote: How Logical Data Fabric Knits Together Data Visualization wi...

Denodo

What's New In Neo4j 3.4 & Bloom Update

Neo4j

Neo4j is a native graph database platform company that provides graph databases, tools, and services. In this document: - Neo4j announced new features for version 3.4 including 70% faster queries, native string indexes for 5x faster writes, and geospatial search. - A new product, Neo4j Bloom, was introduced as a graph visualization and exploration tool targeting business users. - Neo4j also discussed its Graph Query Language initiative to develop a standard query language for graphs across platforms and vendors.

ironSource Atom BigData Berlin

Shimon Tolts

Gianluigi Vigano, Senior Architect and Fouad Teban, Regional Presales Manager...

Dataconomy Media

Scaling Multi-Cloud Deployments with Denodo: Automated Infrastructure Management

Denodo

Quick Intro to Google Cloud Technologies

Chris Schalk

This document provides an introduction to Google's cloud technologies including Google App Engine, Google Storage, the Prediction API, and BigQuery. It describes each technology's capabilities and how developers can use them. Google App Engine is an application development platform, Storage provides cloud data storage, Prediction API enables machine learning predictions, and BigQuery allows fast, SQL-based analysis of large datasets. Examples and demos of each technology are also presented.

Managing R&D Data on Parallel Compute Infrastructure

Databricks

Clinical genomic analytics pipelines using Databricks and the Delta Lake for the benefit of loading individual reads from raw sequencing or base-call files have significant advantages over more traditional methods. Analysis pipelines that perform genomic mapping to purpose-built reference data artifacts persisted to tables allows for enhanced performance that is magnitudes greater than previous mapping methods. These scalable, reproducible, and potentially open sourced methods have the ability to transform bioinformatics and R&D data management / governance.

Democratizing Data Quality Through a Centralized Platform

Databricks

Bad data leads to bad decisions and broken customer experiences. Organizations depend on complete and accurate data to power their business, maintain efficiency, and uphold customer trust. With thousands of datasets and pipelines running, how do we ensure that all data meets quality standards, and that expectations are clear between producers and consumers? Investing in shared, flexible components and practices for monitoring data health is crucial for a complex data organization to rapidly and effectively scale. At Zillow, we built a centralized platform to meet our data quality needs across stakeholders. The platform is accessible to engineers, scientists, and analysts, and seamlessly integrates with existing data pipelines and data discovery tools. In this presentation, we will provide an overview of our platform’s capabilities, including: Giving producers and consumers the ability to define and view data quality expectations using a self-service onboarding portal Performing data quality validations using libraries built to work with spark Dynamically generating pipelines that can be abstracted away from users Flagging data that doesn’t meet quality standards at the earliest stage and giving producers the opportunity to resolve issues before use by downstream consumers Exposing data quality metrics alongside each dataset to provide producers and consumers with a comprehensive picture of health over time

Cloud Modernization with Data Virtualization

Denodo

Watch full webinar here: [https://buff.ly/2sLhFAc] TransAlta is an electric power generator company headquartered in Calgary, Alberta. TransAlta's IT department initiated "Zero Data Center" project to move their entire data layer to the cloud for flexibility, agility and lower TCO. Data virtualization technology played a central role in TransAlta's real-time data integration, while helping them move to the cloud with zero down-time Attend this Denodo DataFest 2018 session to learn: Who is TransAlta and why TransAlta wanted to move their entire enterprise data layer to the cloud Why data virtualization played a critical role in TransAlta's cloud modernization effort How TransAlta uses DV in their energy trading, wind icing forecast and HR fuctions

KEYNOTE: Edge optimized architecture for fabric defect detection in real-time

Shuquan Huang

This document discusses edge computing and an edge optimized architecture for real-time fabric defect detection. It summarizes that traditional centralized cloud architectures have high latency and bandwidth issues for industrial IoT applications. An edge optimized architecture places computing resources such as data processing, analytics and control planes closer to IoT devices and sensors at the edge to enable low latency insights and responsiveness needed for applications like textile defect detection. The document invites the reader to attend a session providing more details on an edge optimized architecture for real-time fabric defect detection.

Virtual BenchLearning - Data Bench Framework

Big Data Value Association

At the heart of this DataBench webinar is the goal to share a benchmarking process helping European organisations developing Big Data Technologies to reach for excellence and constantly improve their performance, by measuring their technology development activity against parameters of high business relevance. The webinar aims to provide the audience with a framework and tools to assess the performance and impact of Big Data and AI technologies, by providing real insights coming from DataBench. In addition, representatives from other projects part of the BDV PPP such as DeepHealth and They-Buy-for-You will participate to share the challenges and opportunities they have identified on the use of Big Data, Analytics, AI. The perspective of other projects that also have looked into benchmarking, such as Track&Now and I-BiDaaS will be introduced.

Evolving From Monolithic to Distributed Architecture Patterns in the Cloud

Denodo

Watch full webinar here: https://goo.gl/rSfYKV Gartner states in its Predicts 2018: Data Management Strategies Continue to Shift Toward Distributed, “As data management activities are becoming more widespread in both distributed processing use cases, like IoT, and demands for new types of data, emerging roles such as data scientists or data engineers are expected to be driving the new data management requirements in the coming two years. These trends indicate that both the collection of data as well as the need to connect to data are rapidly becoming the new normal, and that the days of a single data store with all the data of interest — the enterprise data warehouse — are long gone.” Data management solutions are becoming distributed, heterogeneous and extremely diverse. Attend this session to learn: • How to evolve architecture patterns in the cloud using data virtualization. • How data virtualization accelerates cloud migration and modernization. • Successful cloud implementation case studies.

A Modern Data Architecture in Azure, presented by Armand van Oijen (Kadenza) ...

Patrick Van Renterghem

1. The document discusses the evolution of data integration infrastructures from pre-data warehouse systems using custom reporting and local data collections to traditional data warehouses using on-premise databases and ETL processes to modern cloud-based architectures. 2. It describes an airline company's move from an on-premise data warehouse to a new cloud-based data architecture on Azure using services like Azure Data Factory and data virtualization to provide a more scalable, flexible, and agile solution. 3. The key takeaways are that a new era of data usage has begun, solutions should be proven in parts to show business value, and an agile approach using pay-per-

Perth Microsoft Data & Analytics User Group - Building Solutions with Azure D...

Sergio Zenatti Filho

This document introduces Sergio Zenatti Filho, a Cloud Solution Architect for Data Platforms at Microsoft. He has over 18 years of experience designing and delivering data and analytics solutions internationally. He enjoys learning new technologies and helping customers define the best solutions for their business needs. Sergio has extensive expertise with Microsoft's data and analytics platforms in both the cloud and on-premises.

Learn to Use Databricks for the Full ML Lifecycle

Databricks

Machine learning development brings many new complexities beyond the traditional software development lifecycle. Unlike traditional software development, ML developers want to try multiple algorithms, tools and parameters to get the best results, and they need to track this information to reproduce work. In addition, developers need to use many distinct systems to productionize models. In this talk, learn how to operationalize ML across the full lifecycle with Databricks Machine Learning.

Analyzing petabytes of smartmeter data using Cloud Bigtable, Cloud Dataflow, ...

Edwin Poot

Webinar: SQL for Machine Data?

Crate.io

By 2020, 50% of all new software will process machine-generated data of some sort (Gartner). Historically, machine data use cases have required non-SQL data stores like Splunk, Elasticsearch, or InfluxDB. Today, new SQL DB architectures rival the non-SQL solutions in ease of use, scalability, cost, and performance. Please join this webinar for a detailed comparison of machine data management approaches.

What's hot

Yellowbrick Webcast with DBTA for Real-Time Analytics

Yellowbrick Data

Data Virtualization to Survive a Multi and Hybrid Cloud World

Denodo

Data Virtualization enabled Data Fabric: Operationalize the Data Lake (APAC)

Denodo

End to End Supply Chain Control Tower

Databricks

The Virtualization of Clouds - The New Enterprise Data Architecture Opportunity

Denodo

Partner Keynote: How Logical Data Fabric Knits Together Data Visualization wi...

Denodo

What's New In Neo4j 3.4 & Bloom Update

Neo4j

ironSource Atom BigData Berlin

Shimon Tolts

Gianluigi Vigano, Senior Architect and Fouad Teban, Regional Presales Manager...

Dataconomy Media

Scaling Multi-Cloud Deployments with Denodo: Automated Infrastructure Management

Denodo

Quick Intro to Google Cloud Technologies

Chris Schalk

Managing R&D Data on Parallel Compute Infrastructure

Databricks

Democratizing Data Quality Through a Centralized Platform

Databricks

Cloud Modernization with Data Virtualization

Denodo

KEYNOTE: Edge optimized architecture for fabric defect detection in real-time

Shuquan Huang

Virtual BenchLearning - Data Bench Framework

Big Data Value Association

Evolving From Monolithic to Distributed Architecture Patterns in the Cloud

Denodo

A Modern Data Architecture in Azure, presented by Armand van Oijen (Kadenza) ...

Patrick Van Renterghem

Perth Microsoft Data & Analytics User Group - Building Solutions with Azure D...

Sergio Zenatti Filho

Learn to Use Databricks for the Full ML Lifecycle

Databricks

What's hot (20)

Yellowbrick Webcast with DBTA for Real-Time Analytics

Data Virtualization to Survive a Multi and Hybrid Cloud World

Data Virtualization enabled Data Fabric: Operationalize the Data Lake (APAC)

End to End Supply Chain Control Tower

The Virtualization of Clouds - The New Enterprise Data Architecture Opportunity

Partner Keynote: How Logical Data Fabric Knits Together Data Visualization wi...

What's New In Neo4j 3.4 & Bloom Update

ironSource Atom BigData Berlin

Gianluigi Vigano, Senior Architect and Fouad Teban, Regional Presales Manager...

Scaling Multi-Cloud Deployments with Denodo: Automated Infrastructure Management

Quick Intro to Google Cloud Technologies

Managing R&D Data on Parallel Compute Infrastructure

Democratizing Data Quality Through a Centralized Platform

Cloud Modernization with Data Virtualization

KEYNOTE: Edge optimized architecture for fabric defect detection in real-time

Virtual BenchLearning - Data Bench Framework

Evolving From Monolithic to Distributed Architecture Patterns in the Cloud

A Modern Data Architecture in Azure, presented by Armand van Oijen (Kadenza) ...

Perth Microsoft Data & Analytics User Group - Building Solutions with Azure D...

Learn to Use Databricks for the Full ML Lifecycle

Similar to PLNOG 3: Tomasz Mikołajczyk - Data scalability. Why you should care?

Analyzing petabytes of smartmeter data using Cloud Bigtable, Cloud Dataflow, ...

Edwin Poot

Webinar: SQL for Machine Data?

Crate.io

IoT meets Big Data

ratthaslip ranokphanuwat

ข้อมูล IoT จากอุปกรณ์มีปริมาณมาก และมีความไดนามิกต่อเนื่องตลอดเวลา มีหลายมิติ ทำอย่างไร จึงจะประมวผลได้เร็วและตอบสนองแบบทันทีทันได้ โดยใช้เครื่องมือ Streaming processing and bigdata analytics

Digital transformation and AI @Edge

Institute of Contemporary Sciences

1) The document discusses digital transformation and AI at the edge for industrial applications. It describes challenges like processing vast amounts of distributed data in real-time while ensuring security and reliability with limited resources. 2) Edge computing is important for industrial IoT as it allows data processing and AI inferencing close to where data is generated, improving latency, security, and scalability. The document outlines several open source edge computing projects and technologies being developed. 3) Achieving digital transformation requires bridging gaps between IT and operational systems through approaches like collecting telemetry data, predictive maintenance, and building a data-informed culture across the organization. Standards like OPC UA are also important for interoperability.

Lisa Guess - Embracing the Cloud

centralohioissa

With all the hype around Cloud and SDN, business decision makers are finding themselves trying to navigate through many new concepts and consequently needing to change the way they have traditionally selected their IT infrastructure. Technologies are now becoming more integrated and it is more important than ever to help your business be agile enough to keep up with the demands of your users and your customers. Come hear from Lisa Guess to learn how organizations can embrace Cloud technologies such as automation, SDN and Orchestration platforms to help you build next-generation networks.

The Evolution of Data Architecture

Wei-Chiu Chuang

Digital Business Transformation in the Streaming Era

Attunity

Enterprises are rapidly adopting stream computing backbones, in-memory data stores, change data capture, and other low-latency approaches for end-to-end applications. As businesses modernize their data architectures over the next several years, they will begin to evolve toward all-streaming architectures. In this webcast, Wikibon, Attunity, and MemSQL will discuss how enterprise data professionals should migrate their legacy architectures in this direction. They will provide guidance for migrating data lakes, data warehouses, data governance, and transactional databases to support all-streaming architectures for complex cloud and edge applications. They will discuss how this new architecture will drive enterprise strategies for operationalizing artificial intelligence, mobile computing, the Internet of Things, and cloud-native microservices. Link to the Wikibon report - wikibon.com/wikibons-2018-big-data-analytics-trends-forecast Link to Attunity Streaming CDC Book Download - http://www.bit.ly/cdcbook Link to MemSQL's Free Data Pipeline Book - http://go.memsql.com/oreilly-data-pipelines

abiquo

guestf5c2fa

The document discusses an intelligent network platform that connects heterogeneous devices and shares data and resources in real-time. The platform, called Abiquo, allows users to interconnect different data sources, optimize costs, and facilitate communication between technologies. It provides benefits such as reducing time to market, improving business decisions, and automating processes through distributed and intensive computing capabilities.

Bridging the Last Mile: Getting Data to the People Who Need It (APAC)

Denodo

Watch full webinar here: https://bit.ly/34iCruM Many organizations are embarking on strategically important journeys to embrace data and analytics. The goal can be to improve internal efficiencies, improve the customer experience, drive new business models and revenue streams, or – in the public sector – provide better services. All of these goals require empowering employees to act on data and analytics and to make data-driven decisions. However, getting data – the right data at the right time – to these employees is a huge challenge and traditional technologies and data architectures are simply not up to this task. This webinar will look at how organizations are using Data Virtualization to quickly and efficiently get data to the people that need it. Attend this session to learn: - The challenges organizations face when trying to get data to the business users in a timely manner - How Data Virtualization can accelerate time-to-value for an organization’s data assets - Examples of leading companies that used data virtualization to get the right data to the users at the right time

Choosing the Right Database: Exploring MySQL Alternatives for Modern Applicat...

Mydbops

Choosing the Right Database: Exploring MySQL Alternatives for Modern Applications by Bhanu Jamwal, Head of Solution Engineering, PingCAP at the Mydbops Opensource Database Meetup 14. This presentation discusses the challenges in choosing the right database for modern applications, focusing on MySQL alternatives. It highlights the growth of new applications, the need to improve infrastructure, and the rise of cloud-native architecture. The presentation explores alternatives to MySQL, such as MySQL forks, database clustering, and distributed SQL. It introduces TiDB as a distributed SQL database for modern applications, highlighting its features and top use cases. Case studies of companies benefiting from TiDB are included. The presentation also outlines TiDB's product roadmap, detailing upcoming features and enhancements.

Big Data and Semantic Web in Manufacturing

Nitesh Khilwani

This document discusses big data and semantic web technologies in manufacturing. It outlines how big data is being used in manufacturing for applications like product quality tracking, supply chain management, and forecasting. Semantic web technologies like ontologies and semantic layers are discussed as ways to add meaning to manufacturing data and integrate information from different systems. A case study on using an ontology called ECOS and a text mining tool called TEXT2RDF to extract semantics from manufacturing documents is presented.

Flash session -streaming--ses1243-lon

Jeffrey T. Pollock

Predictive Analytics: Why (I)IoT Is Different

Altoros

This document discusses how predictive analytics can help address challenges with Internet of Things (IoT) data. While existing machine learning frameworks are useful, IoT data often has more "wrinkles" like variability, volume, and veracity that make insights difficult. Edge computing architectures that move some decision making and data processing to the network edge can help address this by filtering data before it reaches core systems. A two-tier machine learning approach combining frameworks with custom models tailored for IoT data wrangling may help bridge the gap between data realities and insight aspirations for predictive analytics with IoT.

AI as a Service, Build Shared AI Service Platforms Based on Deep Learning Tec...

Databricks

I will share the vision and the production journey of how we build enterprise shared AI As A Service platforms with distributed deep learning technologies. Including those topics: 1) The vision of Enterprise Shared AI As A Service and typical AI services use cases at FinTech industry 2) The high level architecture design principles for AI As A Service 3) The technical evaluation journey to choose an enterprise deep learning framework with comparisons, such as why we choose Deep learning framework based on Spark ecosystem 4) Share some production AI use cases, such as how we implemented new Users-Items Propensity Models with deep learning algorithms with Spark,improve the quality , performance and accuracy of offer and campaigns design, targeting offer matching and linking etc. 5) Share some experiences and tips of using deep learning technologies on top of Spark , such as how we conduct Intel BigDL into a real production.

Why Your Data Science Architecture Should Include a Data Virtualization Tool ...

Denodo

Watch full webinar here: https://bit.ly/35FUn32 Presented at CDAO New Zealand Advanced data science techniques, like machine learning, have proven an extremely useful tool to derive valuable insights from existing data. Platforms like Spark, and complex libraries for R, Python, and Scala put advanced techniques at the fingertips of the data scientists. However, most architecture laid out to enable data scientists miss two key challenges: - Data scientists spend most of their time looking for the right data and massaging it into a usable format - Results and algorithms created by data scientists often stay out of the reach of regular data analysts and business users Watch this session on-demand to understand how data virtualization offers an alternative to address these issues and can accelerate data acquisition and massaging. And a customer story on the use of Machine Learning with data virtualization.

Bridging the Last Mile: Getting Data to the People Who Need It

Denodo

Watch full webinar here: https://bit.ly/3cUA0Qi Many organizations are embarking on strategically important journeys to embrace data and analytics. The goal can be to improve internal efficiencies, improve the customer experience, drive new business models and revenue streams, or – in the public sector – provide better services. All of these goals require empowering employees to act on data and analytics and to make data-driven decisions. However, getting data – the right data at the right time – to these employees is a huge challenge and traditional technologies and data architectures are simply not up to this task. This webinar will look at how organizations are using Data Virtualization to quickly and efficiently get data to the people that need it. Attend this session to learn: - The challenges organizations face when trying to get data to the business users in a timely manner - How Data Virtualization can accelerate time-to-value for an organization’s data assets - Examples of leading companies that used data virtualization to get the right data to the users at the right time

Webinar: Faster Big Data Analytics with MongoDB

MongoDB

Learn how to leverage MongoDB and Big Data technologies to derive rich business insight and build high performance business intelligence platforms. This presentation includes: - Uncovering Opportunities with Big Data analytics - Challenges of real-time data processing - Best practices for performance optimization - Real world case study This presentation was given in partnership with CIGNEX Datamatics.

Flexible and Scalable Integration in the Automation Industry/Industrial IoT

confluent

IIoT / Industry 4.0 with Apache Kafka, Connect, KSQL, Apache PLC4X

Kai Wähner

Data integration and processing is a huge challenge in Industrial IoT (IIoT, aka Industry 4.0 or Automation Industry) due to monolithic systems and proprietary protocols. Apache Kafka, its ecosystem (Kafka Connect, KSQL) and Apache PLC4X are a great open source choice to implement this integration end to end in a scalable, reliable and flexible way. This blog post covers a high level overview about the challenges and a good, flexible architecture. At the end, I share a video recording and the corresponding slide deck. These provide many more details and insights. Apache Kafka is the De-facto Standard for Real-Time Event Streaming. It provides Open Source (Apache 2.0 License) Global-scale Real-time Persistent Storage Stream Processing PCL4X allows vertical integration and to write software independent of PLCs using JDBC-like adapters for various protocols like Siemens S7, Modbus, Allen Bradley, Beckhoff ADS, OPC-UA, Emerson, Profinet, BACnet, Ethernet. Github example: https://github.com/kaiwaehner/iiot-integration-apache-plc4x-kafka-connect-ksql-opc-ua-modbus-siemens-s7 More details: http://www.kai-waehner.de/blog/2019/09/02/iiot-data-integr…and-apache-plc4x/ ‎ Video Recording: https://youtu.be/RWKggid25ds

"An introduction to Kx Technology - a Big Data solution", Kyra Coyne, Data Sc...

Maya Lumbroso

Similar to PLNOG 3: Tomasz Mikołajczyk - Data scalability. Why you should care? (20)

Analyzing petabytes of smartmeter data using Cloud Bigtable, Cloud Dataflow, ...

Webinar: SQL for Machine Data?

IoT meets Big Data

Digital transformation and AI @Edge

Lisa Guess - Embracing the Cloud

The Evolution of Data Architecture

Digital Business Transformation in the Streaming Era

abiquo

Bridging the Last Mile: Getting Data to the People Who Need It (APAC)

Choosing the Right Database: Exploring MySQL Alternatives for Modern Applicat...

Big Data and Semantic Web in Manufacturing

Flash session -streaming--ses1243-lon

Predictive Analytics: Why (I)IoT Is Different

AI as a Service, Build Shared AI Service Platforms Based on Deep Learning Tec...

Why Your Data Science Architecture Should Include a Data Virtualization Tool ...

Bridging the Last Mile: Getting Data to the People Who Need It

Webinar: Faster Big Data Analytics with MongoDB

Flexible and Scalable Integration in the Automation Industry/Industrial IoT

IIoT / Industry 4.0 with Apache Kafka, Connect, KSQL, Apache PLC4X

"An introduction to Kx Technology - a Big Data solution", Kyra Coyne, Data Sc...

Recently uploaded

FREE A4 Cyber Security Awareness Posters-Social Engineering part 3

Data Hops

WeTestAthens: Postman's AI & Automation Techniques

Postman

Public CyberSecurity Awareness Presentation 2024.pptx

marufrahmanstratejm

zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...

Alex Pruden

Folding is a recent technique for building efficient recursive SNARKs. Several elegant folding protocols have been proposed, such as Nova, Supernova, Hypernova, Protostar, and others. However, all of them rely on an additively homomorphic commitment scheme based on discrete log, and are therefore not post-quantum secure. In this work we present LatticeFold, the first lattice-based folding protocol based on the Module SIS problem. This folding protocol naturally leads to an efficient recursive lattice-based SNARK and an efficient PCD scheme. LatticeFold supports folding low-degree relations, such as R1CS, as well as high-degree relations, such as CCS. The key challenge is to construct a secure folding protocol that works with the Ajtai commitment scheme. The difficulty, is ensuring that extracted witnesses are low norm through many rounds of folding. We present a novel technique using the sumcheck protocol to ensure that extracted witnesses are always low norm no matter how many rounds of folding are used. Our evaluation of the final proof system suggests that it is as performant as Hypernova, while providing post-quantum security. Paper Link: https://eprint.iacr.org/2024/257

Dandelion Hashtable: beyond billion requests per second on a commodity server

Antonios Katsarakis

This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).

Presentation of the OECD Artificial Intelligence Review of Germany

innovationoecd

A Comprehensive Guide to DeFi Development Services in 2024

Intelisync

DeFi represents a paradigm shift in the financial industry. Instead of relying on traditional, centralized institutions like banks, DeFi leverages blockchain technology to create a decentralized network of financial services. This means that financial transactions can occur directly between parties, without intermediaries, using smart contracts on platforms like Ethereum. In 2024, we are witnessing an explosion of new DeFi projects and protocols, each pushing the boundaries of what’s possible in finance. In summary, DeFi in 2024 is not just a trend; it’s a revolution that democratizes finance, enhances security and transparency, and fosters continuous innovation. As we proceed through this presentation, we'll explore the various components and services of DeFi in detail, shedding light on how they are transforming the financial landscape. At Intelisync, we specialize in providing comprehensive DeFi development services tailored to meet the unique needs of our clients. From smart contract development to dApp creation and security audits, we ensure that your DeFi project is built with innovation, security, and scalability in mind. Trust Intelisync to guide you through the intricate landscape of decentralized finance and unlock the full potential of blockchain technology. Ready to take your DeFi project to the next level? Partner with Intelisync for expert DeFi development services today!

Main news related to the CCS TSI 2023 (2023/1695)

Jakub Marek

An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers. The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 . The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .

System Design Case Study: Building a Scalable E-Commerce Platform - Hiike

Hiike

Columbus Data & Analytics Wednesdays - June 2024

Jason Packer

Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...

saastr

Best 20 SEO Techniques To Improve Website Visibility In SERP

Pixlogix Infotech

How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf

Chart Kalyan

leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...

alexjohnson7307

Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe

Precisely

Inconsistent user experience and siloed data, high costs, and changing customer expectations – Citizens Bank was experiencing these challenges while it was attempting to deliver a superior digital banking experience for its clients. Its core banking applications run on the mainframe and Citizens was using legacy utilities to get the critical mainframe data to feed customer-facing channels, like call centers, web, and mobile. Ultimately, this led to higher operating costs (MIPS), delayed response times, and longer time to market. Ever-changing customer expectations demand more modern digital experiences, and the bank needed to find a solution that could provide real-time data to its customer channels with low latency and operating costs. Join this session to learn how Citizens is leveraging Precisely to replicate mainframe data to its customer channels and deliver on their “modern digital bank” experiences.

Energy Efficient Video Encoding for Cloud and Edge Computing Instances

Alpen-Adria-Universität

Introduction of Cybersecurity with OSS at Code Europe 2024

Hiroshi SHIBATA

I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems. The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS. Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application. I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.

Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...

saastr

Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack

shyamraj55

Your One-Stop Shop for Python Success: Top 10 US Python Development Providers

akankshawande

Recently uploaded (20)

FREE A4 Cyber Security Awareness Posters-Social Engineering part 3

WeTestAthens: Postman's AI & Automation Techniques

Public CyberSecurity Awareness Presentation 2024.pptx

zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...

Dandelion Hashtable: beyond billion requests per second on a commodity server

Presentation of the OECD Artificial Intelligence Review of Germany

A Comprehensive Guide to DeFi Development Services in 2024

Main news related to the CCS TSI 2023 (2023/1695)

System Design Case Study: Building a Scalable E-Commerce Platform - Hiike

Columbus Data & Analytics Wednesdays - June 2024

Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...

Best 20 SEO Techniques To Improve Website Visibility In SERP

How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf

leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...

Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe

Energy Efficient Video Encoding for Cloud and Edge Computing Instances

Introduction of Cybersecurity with OSS at Code Europe 2024

Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...

Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack

Your One-Stop Shop for Python Success: Top 10 US Python Development Providers

PLNOG 3: Tomasz Mikołajczyk - Data scalability. Why you should care?

1. Data scalability. Why you should care? Tomasz Mikolajczyk & the GridwiseTech team www.gridwisetech.com tomek@gridwisetech.com

2. GridwiseTech www.gridwisetech.com Presentation survey Presentation plan 1) About GridwiseTech 2) Building efficient database oriented, IT systems 3) Momentum + case study Presentation duration 40 minutes

3. GridwiseTech www.gridwisetech.com About GridwiseTech Leading vendor- independent scalable technology expert. Solve scalability and performance problems with virtualization, Grid, distributed data processing, caching. Implement vision for distributed infrastructures. Mission: turn avant-garde technology into customer’s business benefits About us

4. GridwiseTech www.gridwisetech.com Chosen customers Chosen partners Our references About us

5. GridwiseTech www.gridwisetech.com Building efficient IT system About us Application WWW SOA Middleware Processing Data provisioning Data persistence

6. GridwiseTech www.gridwisetech.com Vendors puzzle About us Application WWW SOA Platform Computing, Sun Microsystems, Altair, UnivaUD, Cluster Resources Gigaspaces, Gemstone, Coherence, eXludus, Terracotta Oracle, IBM, Microsoft PostreSQL, MySQL Citrix, Vmvare,Novell, Virtual Iron, RedHat

7. GridwiseTech www.gridwisetech.com Improving the system efficiency

8. GridwiseTech www.gridwisetech.com Simple data oriented IT system Internal or external data sources applications working on data

9. GridwiseTech www.gridwisetech.com IT systems are constantly growing Increased number of users Increased number of applications Increased amount of data

10. GridwiseTech www.gridwisetech.com IT systems are constantly growing Infrastructure bottleneck

11. GridwiseTech www.gridwisetech.com Hardware scale up – the first idea Typical solution: buy bigger and faster hardware

12. GridwiseTech www.gridwisetech.com Hardware scale up – the first idea Recurring software and hardware migrations will always generate substantial costs

13. GridwiseTech www.gridwisetech.com Introduction of scalability Simultaneous calculation Load balancing

14. GridwiseTech www.gridwisetech.com Introduction of scalability Simultaneous calculation Cloud Load balancing

15. GridwiseTech www.gridwisetech.com Introduction of distributed processing Cloud I/O bottleneck

16. GridwiseTech www.gridwisetech.com Problem with data access? Database / configuration / storage optimization In-memory databases Database clustering Sharding Data grids Distributed caching

17. GridwiseTech www.gridwisetech.com Fully scalable system Scaling out data

18. GridwiseTech www.gridwisetech.com Project for electronic manufacturer Technical scope of the problem: 2,5 million transactions per day many databases, different vendors and/or versions 4TB of data to maintain constant environment modifications 400,000 lines of PL/SQL code to analyze Improve report generation time by 70% allowing for increase in daily production. Project main goal Business problem: lack of possibility to increase production limited process manageability due to report availability delay Repetitive infrastructure down time resulting in data inconstancy Data normalization problem due to many different data sources

19. GridwiseTech www.gridwisetech.com Initial customer architecture 1 2

20. GridwiseTech www.gridwisetech.com Designed architecture Distributed caching

21. GridwiseTech www.gridwisetech.com Momentum

22. GridwiseTech www.gridwisetech.com Project for electronic manufacturer Results of scaling manufacturer infrastructure: Scalability ensured on each functional layer Real-time report generation 10x times data processing speed-up 100,000 of reads/writes per second (7500 before) Fully manageable production process Improve report generation time by 70% allowing for increase in daily production. Project main goal

23. GridwiseTech www.gridwisetech.com GridwiseTech's value add GridwiseTech's value add:  Application for automatic mapping database tables into objects.  Scalable Feeder application that provide fast data replication between multiple data sources and data grid.  Data normalization from multiple data sources.  Mechanism, that prevent from RAM overloading.  Mechanism, that provide synchronisation possibility between services and applications.  Technique of building thread enabled and fully scalable external services with batch processing support.

24. Thank you www.gridwisetech.com contact@gridwisetech.com

PLNOG 3: Tomasz Mikołajczyk - Data scalability. Why you should care?

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to PLNOG 3: Tomasz Mikołajczyk - Data scalability. Why you should care?

Similar to PLNOG 3: Tomasz Mikołajczyk - Data scalability. Why you should care? (20)

Recently uploaded

Recently uploaded (20)

PLNOG 3: Tomasz Mikołajczyk - Data scalability. Why you should care?