[Paper Reading] QAGen: Generating query-aware test databases
•Download as PPTX, PDF•
0 likes•103 views
Today, a common methodology for testing a database management system (DBMS) is to generate a set of test databases and then execute queries on top of them. However, for DBMS testing, it would be a big advantage if we can control the input and/or the output (e.g., the cardinality) of each individual operator of a test query for a particular test case. Unfortunately, current database generators generate databases independent of queries. As a result, it is hard to guarantee that executing the test query on the generated test databases can obtain the desired (intermediate) query results that match the test case. In this paper, we propose a novel way for DBMS testing. Instead of first generating a test database and then seeing how well it matches a particular test case (or otherwise use a trial-and-error approach to generate another test database), we propose to generate a query-aware database for each test case. To that end, we designed a query-aware test database generator called QAGen. In addition to the database schema and the set of basic constraints defined on the base tables, QAGen takes the query and the set of constraints defined on the query as input, and generates a query-aware test database as output. The generated database guarantees that the test query can get the desired (intermediate) query results as defined in the test case. This approach of testing facilitates a wide range of DBMS testing tasks such as testing of memory managers and testing the cardinality estimation components of query optimizers.
Report
Share
Report
Share
Recommended
Ad Placement Challenge
The document summarizes Alexey Grigorev's approach to the NIPS'17 Ad Placement Challenge. It describes using an FTRL model trained on historical Criteo data to create a policy for scoring and selecting ad placements. Several post-processing techniques were tested, with the best performing being to sigmoid and scale the model outputs, set the predicted click to a higher value, and select hyperparameters through cross-validation. This approach achieved a score of 45-55 on the challenge, showing the model could predict some clicks correctly but was limited and not truly counterfactual.
Outbrain Click Prediction
The document discusses click prediction for ads on Outbrain's platform. It describes using a recommender system to predict which ad a user will click based on their characteristics and past interactions. A variety of machine learning models are evaluated like FTRL, XGBoost, and FFM. Features include user ID, location, time, ad/page details, and tree model leaves. The goal is to reorder ad groups to optimize MAP@12 performance on click-through rates.
Deep learning
Deep learning uses neural networks with multiple layers to simulate the human brain. It consists of an input layer, hidden layers, and an output layer with data passed between each layer. The example task uses a neural network for multi-class classification of handwritten digits, with an output softmax activation and minimization of error through gradient descent backpropagation. After experimentation, the model achieved 98% validation accuracy. Deep learning is widely used for computer vision, natural language processing, and reinforcement learning problems.
Avito Duplicate Ads Detection @ kaggle
This document summarizes the process of building a machine learning model to identify duplicate listings using data from Avito, a Russian online marketplace. It discusses the dataset containing over 3 million training pairs and 1 million test pairs of listings with images and text data. Many features were computed from the text, images, attributes, locations and other fields to quantify similarities and differences between listings. These features were then used to train models like XGBoost and ensembles of models to generate the final predictions. Key lessons learned included the importance of cross validation and managing a large number of computed features.
Functional Approach to Software Engineering
This document outlines functional programming concepts like pure functions, shared mutable state, null, and monads. It discusses how pure functions have no side effects or mutations, the problems with shared mutable state and null values, and how monads support mapping and lifting values to address these issues. Examples of applying these concepts in JavaScript are also mentioned.
Feel++ webinar 9 27 2012
The document summarizes the state of Feel++, an open source C++ finite element library. Recent developments include improvements to the programming environment, numerics, language, and coupling capabilities. Projects using Feel++ were presented at conferences. Upcoming events include a workshop on Feel++ and reduced order modeling. Work continues on certification reduced basis methods and non-conforming discretization techniques for parallel computing.
CxStates
This document presents a dynamically defined state model called CxStates that is not based on the state pattern. CxStates allows for a user-configurable state model that allows easy communication with external systems. It includes basic classes like CxBaseState that defines a state, CxTransitionEntry that defines transitions between states, and CxActionSequence to execute methods in external systems during transitions. Transitions are executed by executing pre-transition and post-transition actions. The document demonstrates CxStates with an example of modeling the states of an automatic coffee machine.
3slides
The document discusses the research focus of the TAO group at Inria Saclay, which includes machine learning and optimization applications for energy management. The group has one permanent member and others part-time. It collaborates closely with partners in Taiwan and the company Artelys. The research aims to address challenges in power grid simulation like variable demand and renewable energy sources using techniques like mathematical programming, reinforcement learning, and direct policy search combined with heuristics and Monte Carlo tree search.
Recommended
Ad Placement Challenge
The document summarizes Alexey Grigorev's approach to the NIPS'17 Ad Placement Challenge. It describes using an FTRL model trained on historical Criteo data to create a policy for scoring and selecting ad placements. Several post-processing techniques were tested, with the best performing being to sigmoid and scale the model outputs, set the predicted click to a higher value, and select hyperparameters through cross-validation. This approach achieved a score of 45-55 on the challenge, showing the model could predict some clicks correctly but was limited and not truly counterfactual.
Outbrain Click Prediction
The document discusses click prediction for ads on Outbrain's platform. It describes using a recommender system to predict which ad a user will click based on their characteristics and past interactions. A variety of machine learning models are evaluated like FTRL, XGBoost, and FFM. Features include user ID, location, time, ad/page details, and tree model leaves. The goal is to reorder ad groups to optimize MAP@12 performance on click-through rates.
Deep learning
Deep learning uses neural networks with multiple layers to simulate the human brain. It consists of an input layer, hidden layers, and an output layer with data passed between each layer. The example task uses a neural network for multi-class classification of handwritten digits, with an output softmax activation and minimization of error through gradient descent backpropagation. After experimentation, the model achieved 98% validation accuracy. Deep learning is widely used for computer vision, natural language processing, and reinforcement learning problems.
Avito Duplicate Ads Detection @ kaggle
This document summarizes the process of building a machine learning model to identify duplicate listings using data from Avito, a Russian online marketplace. It discusses the dataset containing over 3 million training pairs and 1 million test pairs of listings with images and text data. Many features were computed from the text, images, attributes, locations and other fields to quantify similarities and differences between listings. These features were then used to train models like XGBoost and ensembles of models to generate the final predictions. Key lessons learned included the importance of cross validation and managing a large number of computed features.
Functional Approach to Software Engineering
This document outlines functional programming concepts like pure functions, shared mutable state, null, and monads. It discusses how pure functions have no side effects or mutations, the problems with shared mutable state and null values, and how monads support mapping and lifting values to address these issues. Examples of applying these concepts in JavaScript are also mentioned.
Feel++ webinar 9 27 2012
The document summarizes the state of Feel++, an open source C++ finite element library. Recent developments include improvements to the programming environment, numerics, language, and coupling capabilities. Projects using Feel++ were presented at conferences. Upcoming events include a workshop on Feel++ and reduced order modeling. Work continues on certification reduced basis methods and non-conforming discretization techniques for parallel computing.
CxStates
This document presents a dynamically defined state model called CxStates that is not based on the state pattern. CxStates allows for a user-configurable state model that allows easy communication with external systems. It includes basic classes like CxBaseState that defines a state, CxTransitionEntry that defines transitions between states, and CxActionSequence to execute methods in external systems during transitions. Transitions are executed by executing pre-transition and post-transition actions. The document demonstrates CxStates with an example of modeling the states of an automatic coffee machine.
3slides
The document discusses the research focus of the TAO group at Inria Saclay, which includes machine learning and optimization applications for energy management. The group has one permanent member and others part-time. It collaborates closely with partners in Taiwan and the company Artelys. The research aims to address challenges in power grid simulation like variable demand and renewable energy sources using techniques like mathematical programming, reinforcement learning, and direct policy search combined with heuristics and Monte Carlo tree search.
Kubernetes Workload Rebalancing
All the content of this website is informative and non-commercial, does not imply a commitment to develop, launch or schedule delivery of any feature or functionality, should not rely on it in making decisions, incorporate or take it as a reference in a contract or academic matters. Likewise, the use, distribution and reproduction by any means, in whole or in part, without the authorization of the author and / or third-party copyright holders, as applicable, is prohibited.
How to build TiDB
This is the speech Max Liu gave at Percona Live Open Source Database Conference 2016.
Max Liu: Co-founder and CEO, a hacker with a free soul
The slide covered the following topics:
- Why another database?
- What kind of database we want to build?
- How to design such a database, including the principles, the architecture, and design decisions?
- How to develop such a database, including the architecture and the core technologies for TiKV and TiDB?
- How to test the database to ensure the quality and stability?
from Binary to Binary: How Qemu Works
This document discusses how Qemu works to translate guest binaries to run on the host machine. It first generates an intermediate representation called TCG-IR from the guest binary code. It then translates the TCG-IR into native host machine code. To achieve high performance, it chains translated blocks together by patching jump targets. Key techniques include just-in-time compilation, translation block finding, block chaining, and helper functions to emulate unsupported guest instructions.
Etl confessions pg conf us 2017
This document discusses strategies for efficiently loading and transforming large datasets in PostgreSQL for analytics use cases. It presents several case studies:
1) Loading a large CSV file - different methods like pgloader, COPY, and temporary foreign tables are compared. Temporary foreign tables perform best when filtering columns.
2) Pre-aggregating ("rolling up") data into multiple tables at different granularities for optimized querying. Chained INSERTs and CTEs are more efficient than individual inserts.
3) Creating a "dumb rollup table" using GROUPING SETS to pre-aggregate into a single temp table and insert into final tables in one pass. This outperforms multiple round trips or inserts.
Engineering data quality
Garbage in, garbage out - we have all heard about the importance of data quality. Having high quality data is essential for all types of use cases, whether it is reporting, anomaly detection, or for avoiding bias in machine learning applications. But where does high quality data come from? How can one assess data quality, improve quality if necessary, and prevent bad quality from slipping in? Obtaining good data quality involves several engineering challenges. In this presentation, we will go through tools and strategies that help us measure, monitor, and improve data quality. We will enumerate factors that can cause data collection and data processing to cause data quality issues, and we will show how to use engineering to detect and mitigate data quality problems.
Holistic data application quality
The quality of data-powered applications depends not only on code, but also on collected data, as well as models trained on data. This renders traditional quality assurance inadequate. We will take a look in our toolbox for more holistic tactics that bridge the gap between code and data quality assurance.
Centernet
Slide for study session given by Christian Saravia at Arithmer inc.
It is a summary of recent method for object detection, centernet.
Arithmer株式会社は東京大学大学院数理科学研究科発の数学の会社です。私達は現代数学を応用して、様々な分野のソリューションに、新しい高度AIシステムを導入しています。AIをいかに上手に使って仕事を効率化するか、そして人々の役に立つ結果を生み出すのか、それを考えるのが私たちの仕事です。
Arithmer began at the University of Tokyo Graduate School of Mathematical Sciences. Today, our research of modern mathematics and AI systems has the capability of providing solutions when dealing with tough complex issues. At Arithmer we believe it is our job to realize the functions of AI through improving work efficiency and producing more useful results for society.
Data Analysis with TensorFlow in PostgreSQL
In this talk I'll discuss how we can combine the power of PostgreSQL with TensorFlow to perform data analysis. By using the pl/python3 procedural language we can integrate machine learning libraries such as TensorFlow with PostgreSQL, opening the door for powerful data analytics combining SQL with AI. Typical use-cases might involve regression analysis to find relationships in an existing dataset and to predict results based on new inputs, or to analyse time series data and extrapolate future data taking into account general trends and seasonal variability whilst ignoring noise. Python is an ideal language for building custom systems to do this kind of work as it gives us access to a rich ecosystem of libraries such as Pandas and Numpy, in addition to TensorFlow itself.
NoSQL Database Migration Masterclass - Session 2: The Anatomy of a Migration
In this talk, Felipe Mendes, Solutions Architect at ScyllaDB, shares how 4 companies managed their migration. He covers:
Disney+ – No migration needed!
Discord – Shadow cluster
OpenWeb – TTL expiration, cover Load and Stream
MyHeritage – Counters
ShareChat – Bonus: A bit of everything
Validating big data pipelines - FOSDEM 2019
This document discusses validating data pipelines built with Apache Spark and Apache Airflow. It emphasizes that tests are not perfect and failures will occur, so validation is important to minimize impacts. Simple validation rules can check for invalid records, changes in data distributions, and schema mismatches. Validation rules can run as separate Spark jobs and metrics from jobs can be compared against expected values. Airflow can coordinate validation jobs and check for anomalies before publishing results. Overall, the key is to have validation rules that alert infrequently but catch meaningful issues.
Validating big data jobs - Spark AI Summit EU
As big data jobs move from the proof-of-concept phase into powering real production services, we have to start consider what will happen when everything eventually goes wrong (such as recommending inappropriate products or other decisions taken on bad data). This talk will attempt to convince you that we will all eventually get aboard the failboat (especially with ~40% of respondents automatically deploying their Spark jobs results to production), and its important to automatically recognize when things have gone wrong so we can stop deployment before we have to update our resumes.
Figuring out when things have gone terribly wrong is trickier than it first appears, since we want to catch the errors before our users notice them (or failing that before CNN notices them). We will explore general techniques for validation, look at responses from people validating big data jobs in production environments, and libraries that can assist us in writing relative validation rules based on historical data.
For folks working in streaming, we will talk about the unique challenges of attempting to validate in a real-time system, and what we can do besides keeping an up-to-date resume on file for when things go wrong. To keep the talk interesting real-world examples (with company names removed) will be presented, as well as several creative-common licensed cat pictures and an adorable panda GIF.
If you’ve seen Holden’s previous testing Spark talks this can be viewed as a deep dive on the second half focused around what else we need to do besides good testing practices to create production quality pipelines. If you haven’t seen the testing talks watch those on YouTube after you come see this one
Model selection and tuning at scale
This document discusses model selection and tuning at scale using large datasets. It describes using different percentages of a 1TB Criteo click-through dataset to test and tune gradient boosted trees (GBTs) and other models. Testing on small slices found GBT performed best. Tuning GBT on larger slices up to 10% of the data showed tree depth should increase logarithmically with data size. Online learning with VW was also efficient, needing minimal tuning. The document cautions that true model selection and tuning at scale involves starting with larger data samples than GBs to avoid extrapolating from small data.
Data engineering in 10 years.pdf
If we could only predict the future of the software industry, we could make better investments and decisions. We could waste less resources on technology and processes we know will not last, or at least be conscious in our decisions to choose solutions with a limited life time. It turns out that for data engineering, we can predict the future, because it has already happened. Not in our workplace, but at a few leading companies that are blazing ahead. It has also already happened in the neighbouring field of software engineering, which is two decades ahead of data engineering regarding process maturity. In this presentation, we will glimpse into the future of data engineering. Data engineering has gone from legacy data warehouses with stored procedures, to big data with Hadoop and data lakes, on to a new form of modern data warehouses and low code tools aka "the modern data stack". Where does it go from here? We will look at the points where data leaders differ from the crowd and combine with observations on how software engineering has evolved, to see that it points towards a new, more industrialised form of data engineering - "data factory engineering".
From Zero to Streaming Healthcare in Production (Alexander Kouznetsov, Invita...
This document provides an overview of a company's first Kafka Streams project to build a streaming data pipeline. Some key lessons learned include adopting a data-first mindset where the data defines the application behavior and architecture. All business logic is modeled as data transformations. Testing was done using TopologyTestDriver for unit tests and emulators for external systems. Kafka Streams was determined to be a good fit as it provided an ordered, fault-tolerant processing pipeline with exactly-once guarantees. Future work includes open sourcing components and improving the declarative side effect handling in the KStreams DSL.
Willump: Optimizing Feature Computation in ML Inference
Systems for performing ML inference are increasingly important, but are far slower than they could be because they use techniques designed for conventional data serving workloads, neglecting the statistical nature of ML inference. As an alternative, this talk presents Willump, an optimizer for ML inference.
Druid
This document discusses Druid in production at Fyber, a company that indexes 5 terabytes of data daily from various sources into Druid. It describes the hardware used, including 30 historical nodes and 2 broker nodes. Issues addressed include slow query times with many dimensions, some as lists, and data cleanup steps to reduce cardinality like replacing values. Segment sizing and partitioning are also discussed. Hardware, data ingestion, querying, and optimizations used to scale Druid for Fyber's analytics needs are covered in under 3 sentences.
Building real time Data Pipeline using Spark Streaming
This document summarizes the key challenges and solutions in building a real-time data pipeline that ingests data from a database, transforms it using Spark Streaming, and publishes the output to Salesforce. The pipeline aims to have a latency of 1 minute with zero data loss and ordering guarantees. Some challenges discussed include handling out of sequence and late arrival events, schema evolution, bootstrap loading, data loss/corruption, and diagnosing issues. Solutions proposed use Kafka, checkpointing, replay capabilities, and careful broker/connect setups to help meet the reliability requirements for the pipeline.
Mirko Damiani - An Embedded soft real time distributed system in Go
An embedded system usually involves low level languages like C and highly customized hardware. In this talk we will see a use case of a soft real time system which was developed taking a very different approach, written in Go. We will see what are the advantages of this choice, along with its limits.
Beyond unit tests: Deployment and testing for Hadoop/Spark workflows
As a Hadoop developer, do you want to quickly develop your Hadoop workflows? Do you want to test your workflows in a sandboxed environment similar to production? Do you want to write unit tests for your workflows and add assertions on top of it?
In just a few years, the number of users writing Hadoop/Spark jobs at LinkedIn have grown from tens to hundreds and the number of jobs running every day has grown from hundreds to thousands. With the ever increasing number of users and jobs, it becomes crucial to reduce the development time for these jobs. It is also important to test these jobs thoroughly before they go to production.
We’ve tried to address these issues by creating a testing framework for Hadoop/Spark jobs. The testing framework enables the users to run their jobs in an environment similar to the production environment and on the data which is sampled from the original data. The testing framework consists of a test deployment system, a data generation pipeline to generate the sampled data, a data management system to help users manage and search the sampled data and an assertion engine to validate the test output.
In this talk, we will discuss the motivation behind the testing framework before deep diving into its design. We will further discuss how the testing framework is helping the Hadoop users at LinkedIn to be more productive.
[Paper Reading] Efficient Query Processing with Optimistically Compressed Has...
Modern query engines rely heavily on hash tables for query processing. Overall query performance and memory
footprint is often determined by how hash tables and the
tuples within them are represented. In this work, we propose three complementary techniques to improve this representation:
Domain-Guided Prefix Suppression bit-packs keys and values tightly to reduce hash table record width. Optimistic Splitting decomposes values (and operations on them) into (operations on) frequently-accessed and infrequently-accessed value slices.
By removing the infrequently-accessed value slices from the hash table record, it improves cache locality. The Unique Strings Selfaligned Region (USSR) accelerates handling frequently-occurring strings, which are very common in real-world data sets, by creating an on-the-fly dictionary of the most frequent strings. This allows executing many string operations with integer logic and reduces memory pressure.
We integrated these techniques into Vectorwise. On the TPC-H benchmark, our approach reduces peak memory consumption by 2–4× and improves performance by up to 1.5×. On a real-world BI workload, we measured a 2× improvement in performance and in micro-benchmarks we observed speedups of up to 25×.
[Paper Reading]Orca: A Modular Query Optimizer Architecture for Big Data
The performance of analytical query processing in data management systems depends primarily on the capabilities of the system's query optimizer. Increased data volumes and heightened interest in processing complex analytical queries have prompted Pivotal to build a new query optimizer.
In this paper we present the architecture of Orca, the new query optimizer for all Pivotal data management products, including Pivotal Greenplum Database and Pivotal HAWQ. Orca is a comprehensive development uniting state-of-the-art query optimization technology with own original research resulting in a modular and portable optimizer architecture.
In addition to describing the overall architecture, we highlight several unique features and present performance comparisons against other systems.
More Related Content
Similar to [Paper Reading] QAGen: Generating query-aware test databases
Kubernetes Workload Rebalancing
All the content of this website is informative and non-commercial, does not imply a commitment to develop, launch or schedule delivery of any feature or functionality, should not rely on it in making decisions, incorporate or take it as a reference in a contract or academic matters. Likewise, the use, distribution and reproduction by any means, in whole or in part, without the authorization of the author and / or third-party copyright holders, as applicable, is prohibited.
How to build TiDB
This is the speech Max Liu gave at Percona Live Open Source Database Conference 2016.
Max Liu: Co-founder and CEO, a hacker with a free soul
The slide covered the following topics:
- Why another database?
- What kind of database we want to build?
- How to design such a database, including the principles, the architecture, and design decisions?
- How to develop such a database, including the architecture and the core technologies for TiKV and TiDB?
- How to test the database to ensure the quality and stability?
from Binary to Binary: How Qemu Works
This document discusses how Qemu works to translate guest binaries to run on the host machine. It first generates an intermediate representation called TCG-IR from the guest binary code. It then translates the TCG-IR into native host machine code. To achieve high performance, it chains translated blocks together by patching jump targets. Key techniques include just-in-time compilation, translation block finding, block chaining, and helper functions to emulate unsupported guest instructions.
Etl confessions pg conf us 2017
This document discusses strategies for efficiently loading and transforming large datasets in PostgreSQL for analytics use cases. It presents several case studies:
1) Loading a large CSV file - different methods like pgloader, COPY, and temporary foreign tables are compared. Temporary foreign tables perform best when filtering columns.
2) Pre-aggregating ("rolling up") data into multiple tables at different granularities for optimized querying. Chained INSERTs and CTEs are more efficient than individual inserts.
3) Creating a "dumb rollup table" using GROUPING SETS to pre-aggregate into a single temp table and insert into final tables in one pass. This outperforms multiple round trips or inserts.
Engineering data quality
Garbage in, garbage out - we have all heard about the importance of data quality. Having high quality data is essential for all types of use cases, whether it is reporting, anomaly detection, or for avoiding bias in machine learning applications. But where does high quality data come from? How can one assess data quality, improve quality if necessary, and prevent bad quality from slipping in? Obtaining good data quality involves several engineering challenges. In this presentation, we will go through tools and strategies that help us measure, monitor, and improve data quality. We will enumerate factors that can cause data collection and data processing to cause data quality issues, and we will show how to use engineering to detect and mitigate data quality problems.
Holistic data application quality
The quality of data-powered applications depends not only on code, but also on collected data, as well as models trained on data. This renders traditional quality assurance inadequate. We will take a look in our toolbox for more holistic tactics that bridge the gap between code and data quality assurance.
Centernet
Slide for study session given by Christian Saravia at Arithmer inc.
It is a summary of recent method for object detection, centernet.
Arithmer株式会社は東京大学大学院数理科学研究科発の数学の会社です。私達は現代数学を応用して、様々な分野のソリューションに、新しい高度AIシステムを導入しています。AIをいかに上手に使って仕事を効率化するか、そして人々の役に立つ結果を生み出すのか、それを考えるのが私たちの仕事です。
Arithmer began at the University of Tokyo Graduate School of Mathematical Sciences. Today, our research of modern mathematics and AI systems has the capability of providing solutions when dealing with tough complex issues. At Arithmer we believe it is our job to realize the functions of AI through improving work efficiency and producing more useful results for society.
Data Analysis with TensorFlow in PostgreSQL
In this talk I'll discuss how we can combine the power of PostgreSQL with TensorFlow to perform data analysis. By using the pl/python3 procedural language we can integrate machine learning libraries such as TensorFlow with PostgreSQL, opening the door for powerful data analytics combining SQL with AI. Typical use-cases might involve regression analysis to find relationships in an existing dataset and to predict results based on new inputs, or to analyse time series data and extrapolate future data taking into account general trends and seasonal variability whilst ignoring noise. Python is an ideal language for building custom systems to do this kind of work as it gives us access to a rich ecosystem of libraries such as Pandas and Numpy, in addition to TensorFlow itself.
NoSQL Database Migration Masterclass - Session 2: The Anatomy of a Migration
In this talk, Felipe Mendes, Solutions Architect at ScyllaDB, shares how 4 companies managed their migration. He covers:
Disney+ – No migration needed!
Discord – Shadow cluster
OpenWeb – TTL expiration, cover Load and Stream
MyHeritage – Counters
ShareChat – Bonus: A bit of everything
Validating big data pipelines - FOSDEM 2019
This document discusses validating data pipelines built with Apache Spark and Apache Airflow. It emphasizes that tests are not perfect and failures will occur, so validation is important to minimize impacts. Simple validation rules can check for invalid records, changes in data distributions, and schema mismatches. Validation rules can run as separate Spark jobs and metrics from jobs can be compared against expected values. Airflow can coordinate validation jobs and check for anomalies before publishing results. Overall, the key is to have validation rules that alert infrequently but catch meaningful issues.
Validating big data jobs - Spark AI Summit EU
As big data jobs move from the proof-of-concept phase into powering real production services, we have to start consider what will happen when everything eventually goes wrong (such as recommending inappropriate products or other decisions taken on bad data). This talk will attempt to convince you that we will all eventually get aboard the failboat (especially with ~40% of respondents automatically deploying their Spark jobs results to production), and its important to automatically recognize when things have gone wrong so we can stop deployment before we have to update our resumes.
Figuring out when things have gone terribly wrong is trickier than it first appears, since we want to catch the errors before our users notice them (or failing that before CNN notices them). We will explore general techniques for validation, look at responses from people validating big data jobs in production environments, and libraries that can assist us in writing relative validation rules based on historical data.
For folks working in streaming, we will talk about the unique challenges of attempting to validate in a real-time system, and what we can do besides keeping an up-to-date resume on file for when things go wrong. To keep the talk interesting real-world examples (with company names removed) will be presented, as well as several creative-common licensed cat pictures and an adorable panda GIF.
If you’ve seen Holden’s previous testing Spark talks this can be viewed as a deep dive on the second half focused around what else we need to do besides good testing practices to create production quality pipelines. If you haven’t seen the testing talks watch those on YouTube after you come see this one
Model selection and tuning at scale
This document discusses model selection and tuning at scale using large datasets. It describes using different percentages of a 1TB Criteo click-through dataset to test and tune gradient boosted trees (GBTs) and other models. Testing on small slices found GBT performed best. Tuning GBT on larger slices up to 10% of the data showed tree depth should increase logarithmically with data size. Online learning with VW was also efficient, needing minimal tuning. The document cautions that true model selection and tuning at scale involves starting with larger data samples than GBs to avoid extrapolating from small data.
Data engineering in 10 years.pdf
If we could only predict the future of the software industry, we could make better investments and decisions. We could waste less resources on technology and processes we know will not last, or at least be conscious in our decisions to choose solutions with a limited life time. It turns out that for data engineering, we can predict the future, because it has already happened. Not in our workplace, but at a few leading companies that are blazing ahead. It has also already happened in the neighbouring field of software engineering, which is two decades ahead of data engineering regarding process maturity. In this presentation, we will glimpse into the future of data engineering. Data engineering has gone from legacy data warehouses with stored procedures, to big data with Hadoop and data lakes, on to a new form of modern data warehouses and low code tools aka "the modern data stack". Where does it go from here? We will look at the points where data leaders differ from the crowd and combine with observations on how software engineering has evolved, to see that it points towards a new, more industrialised form of data engineering - "data factory engineering".
From Zero to Streaming Healthcare in Production (Alexander Kouznetsov, Invita...
This document provides an overview of a company's first Kafka Streams project to build a streaming data pipeline. Some key lessons learned include adopting a data-first mindset where the data defines the application behavior and architecture. All business logic is modeled as data transformations. Testing was done using TopologyTestDriver for unit tests and emulators for external systems. Kafka Streams was determined to be a good fit as it provided an ordered, fault-tolerant processing pipeline with exactly-once guarantees. Future work includes open sourcing components and improving the declarative side effect handling in the KStreams DSL.
Willump: Optimizing Feature Computation in ML Inference
Systems for performing ML inference are increasingly important, but are far slower than they could be because they use techniques designed for conventional data serving workloads, neglecting the statistical nature of ML inference. As an alternative, this talk presents Willump, an optimizer for ML inference.
Druid
This document discusses Druid in production at Fyber, a company that indexes 5 terabytes of data daily from various sources into Druid. It describes the hardware used, including 30 historical nodes and 2 broker nodes. Issues addressed include slow query times with many dimensions, some as lists, and data cleanup steps to reduce cardinality like replacing values. Segment sizing and partitioning are also discussed. Hardware, data ingestion, querying, and optimizations used to scale Druid for Fyber's analytics needs are covered in under 3 sentences.
Building real time Data Pipeline using Spark Streaming
This document summarizes the key challenges and solutions in building a real-time data pipeline that ingests data from a database, transforms it using Spark Streaming, and publishes the output to Salesforce. The pipeline aims to have a latency of 1 minute with zero data loss and ordering guarantees. Some challenges discussed include handling out of sequence and late arrival events, schema evolution, bootstrap loading, data loss/corruption, and diagnosing issues. Solutions proposed use Kafka, checkpointing, replay capabilities, and careful broker/connect setups to help meet the reliability requirements for the pipeline.
Mirko Damiani - An Embedded soft real time distributed system in Go
An embedded system usually involves low level languages like C and highly customized hardware. In this talk we will see a use case of a soft real time system which was developed taking a very different approach, written in Go. We will see what are the advantages of this choice, along with its limits.
Beyond unit tests: Deployment and testing for Hadoop/Spark workflows
As a Hadoop developer, do you want to quickly develop your Hadoop workflows? Do you want to test your workflows in a sandboxed environment similar to production? Do you want to write unit tests for your workflows and add assertions on top of it?
In just a few years, the number of users writing Hadoop/Spark jobs at LinkedIn have grown from tens to hundreds and the number of jobs running every day has grown from hundreds to thousands. With the ever increasing number of users and jobs, it becomes crucial to reduce the development time for these jobs. It is also important to test these jobs thoroughly before they go to production.
We’ve tried to address these issues by creating a testing framework for Hadoop/Spark jobs. The testing framework enables the users to run their jobs in an environment similar to the production environment and on the data which is sampled from the original data. The testing framework consists of a test deployment system, a data generation pipeline to generate the sampled data, a data management system to help users manage and search the sampled data and an assertion engine to validate the test output.
In this talk, we will discuss the motivation behind the testing framework before deep diving into its design. We will further discuss how the testing framework is helping the Hadoop users at LinkedIn to be more productive.
Similar to [Paper Reading] QAGen: Generating query-aware test databases (20)
DAA Slides for Multiple topics such as different algorithms
DAA Slides for Multiple topics such as different algorithms
NoSQL Database Migration Masterclass - Session 2: The Anatomy of a Migration
NoSQL Database Migration Masterclass - Session 2: The Anatomy of a Migration
From Zero to Streaming Healthcare in Production (Alexander Kouznetsov, Invita...
From Zero to Streaming Healthcare in Production (Alexander Kouznetsov, Invita...
Willump: Optimizing Feature Computation in ML Inference
Willump: Optimizing Feature Computation in ML Inference
Building real time Data Pipeline using Spark Streaming
Building real time Data Pipeline using Spark Streaming
Mirko Damiani - An Embedded soft real time distributed system in Go
Mirko Damiani - An Embedded soft real time distributed system in Go
Beyond unit tests: Deployment and testing for Hadoop/Spark workflows
Beyond unit tests: Deployment and testing for Hadoop/Spark workflows
More from PingCAP
[Paper Reading] Efficient Query Processing with Optimistically Compressed Has...
Modern query engines rely heavily on hash tables for query processing. Overall query performance and memory
footprint is often determined by how hash tables and the
tuples within them are represented. In this work, we propose three complementary techniques to improve this representation:
Domain-Guided Prefix Suppression bit-packs keys and values tightly to reduce hash table record width. Optimistic Splitting decomposes values (and operations on them) into (operations on) frequently-accessed and infrequently-accessed value slices.
By removing the infrequently-accessed value slices from the hash table record, it improves cache locality. The Unique Strings Selfaligned Region (USSR) accelerates handling frequently-occurring strings, which are very common in real-world data sets, by creating an on-the-fly dictionary of the most frequent strings. This allows executing many string operations with integer logic and reduces memory pressure.
We integrated these techniques into Vectorwise. On the TPC-H benchmark, our approach reduces peak memory consumption by 2–4× and improves performance by up to 1.5×. On a real-world BI workload, we measured a 2× improvement in performance and in micro-benchmarks we observed speedups of up to 25×.
[Paper Reading]Orca: A Modular Query Optimizer Architecture for Big Data
The performance of analytical query processing in data management systems depends primarily on the capabilities of the system's query optimizer. Increased data volumes and heightened interest in processing complex analytical queries have prompted Pivotal to build a new query optimizer.
In this paper we present the architecture of Orca, the new query optimizer for all Pivotal data management products, including Pivotal Greenplum Database and Pivotal HAWQ. Orca is a comprehensive development uniting state-of-the-art query optimization technology with own original research resulting in a modular and portable optimizer architecture.
In addition to describing the overall architecture, we highlight several unique features and present performance comparisons against other systems.
[Paper Reading]KVSSD: Close integration of LSM trees and flash translation la...
Log-Structured-Merge (LSM) trees are a write-optimized data structure for lightweight, high-performance Key-Value (KV) store. Solid State Disks (SSDs) provide acceleration of KV operations on LSM trees. However, this hierarchical design involves multiple software layers, including the LSM tree, host file system, and Flash Translation Layer (FTL), causing cascading write amplifications. We propose KVSSD, a close integration of LSM trees and the FTL, to manage write amplifications from different layers. KVSSD exploits the FTL mapping mechanism to implement copy-free compaction of LSM trees, and it enables direct data allocation in flash memory for efficient garbage collection. In our experiments, compared to the hierarchical design, our KVSSD reduced the write amplification by 88% and improved the throughput by 347%.
[Paper Reading]Chucky: A Succinct Cuckoo Filter for LSM-Tree
Modern key-value stores typically rely on an LSM-tree in storage (SSD) to handle writes and Bloom filters in memory (DRAM) to optimize reads. With ongoing advances in SSD technology shrinking the performance gap between storage and memory devices, the Bloom filters are now emerging as a performance bottleneck.
We propose Chucky, a new design that replaces the multiple Bloom filters by a single Cuckoo filter that maps each data entry to an auxiliary address of its location within the LSM-tree. We show that while such a design entails fewer memory accesses than with Bloom filters, its false positive rate off the bat is higher. The reason is that the auxiliary addresses occupy bits that would otherwise be used as parts of the Cuckoo filter's fingerprints. To address this, we harness techniques from information theory to succinctly encode the auxiliary addresses so that the fingerprints can stay large. As a result, Chucky achieves the best of both worlds: a modest access cost and a low false positive rate at the same time.
[Paper Reading]The Bw-Tree: A B-tree for New Hardware Platforms
The emergence of new hardware and platforms has led to reconsideration of how data management systems are designed. However, certain basic functions such as key indexed access to records remain essential. While we exploit the common architectural layering of prior systems, we make radically new design decisions about each layer. Our new form of B-tree, called the Bw-tree achieves its very high performance via a latch-free approach that effectively exploits the processor caches of modern multi-core chips. Our storage manager uses a unique form of log structuring that blurs the distinction between a page and a record store and works well with flash storage. This paper describes the architecture and algorithms for the Bw-tree, focusing on the main memory aspects. The paper includes results of our experiments that demonstrate that this fresh approach produces outstanding performance.
[Paper Reading] Leases: An Efficient Fault-Tolerant Mechanism for Distribute...
In a distributed system, caching must deal ,with the additional complications of communication and host failures. Leases are proposed as a time-based mechanism that provides efficient consistent access to cached data in distributed systems. Non-Byzantine failures affect performance, not correctness, with their effect minimized by short leases. An analytic model and an evaluation for file access in the V system show that leases of short duration provide good performance. The impact of leases on performance grows more significant in systems of larger scale and higher processor performance.
Paper: https://web.stanford.edu/class/cs240/readings/89-leases.pdf
[Paper reading] Interleaving with Coroutines: A Practical Approach for Robust...
This paper proposes interleaving with coroutines for
any type of index join. It showcases the proposal on SAP
HANA by implementing binary search and CSB+-tree traversal for an instance of index join related to dictionary compression. Coroutine implementations not only perform similarly to prior interleaving techniques, but also resemble the original code closely, while supporting both interleaved and non-interleaved execution. Thus, this paper claims that coroutines
make interleaving practical for use in real DBMS codebases.
Paper: http://www.vldb.org/pvldb/vol11/p230-psaropoulos.pdf
Follow PingCAP on Twitter: https://twitter.com/PingCAP
Follow PingCAP on LinkedIn: https://www.linkedin.com/company/13205484/
[Paperreading] Paxos made easy (by sen han)
This is a sub-project of open-source project TuringCell. Turing Cell Model is a computing model running on top of distributed consistency algorithms (such as Paxos/Raft). TuringCell is an open-source implementation of Turing Cell Model. This means that you can add features as high-availability, fault-tolerance and strong-consistency to existing software very easily. At the same time, TuringCell is an industry-friendly project, at its core is the force from an open, tolerant community. Wherever you are from, whichever language you speak, you are all welcome to equally join in, discuss and build TuringCell !
Paper: https://github.com/turingcell/paxos-made-easy/blob/feature/translation/README_en.md
[Paper Reading] Generalized Sub-Query Fusion for Eliminating Redundant I/O fr...
This document describes RESIN, a query optimizer that eliminates redundant I/O for big data queries. RESIN introduces two new operators - ResinMap and ResinReduce - and two optimization rules - sub-query fusion and binary-operator elimination. These optimizations were found to benefit 40% of queries in the TPC-DS benchmark, improving performance by an average of 1.4x. The optimizer works by fusing operators applied to the same table, eliminating redundant joins or unions, and combining grouped aggregations. An evaluation on a 10GB TPC-DS dataset found RESIN's optimizations significantly reduced redundant I/O for many real-world analytical queries.
[Paper Reading] Steering Query Optimizers: A Practical Take on Big Data Workl...
This document discusses methods for optimizing query performance in a query optimizer called Scope by selecting alternative rule configurations. It proposes using rule signatures to group similar queries and generate candidate rule configurations to execute for each group. A learning model is then trained on execution results to select the best configuration for future queries in each group. The goal is to improve upon the default configuration by adapting to workloads and addressing inaccuracies in cardinality estimation that can lead to suboptimal plans.
The Dark Side Of Go -- Go runtime related problems in TiDB in production
Ed Huang, CTO of PingCAP, talked at Go System Conference about dealing with the typical and profound issues related to Go’s runtime as your systems become more complex. Taking TiDB as an example, he demonstrated how these problems can be reproduced, located, and analyzed in production.
TiDB DevCon 2020 Opening Keynote
At TiDB DevCon 2020, Max Liu, CEO at PingCAP, gave a keynote speech. He believes that today’s database should be more real-time, more flexible, and easier to use, and TiDB, an elastic, cloud-native, real-time HTAP database, is exactly that kind of database.
Finding Logic Bugs in Database Management Systems
Dr. Manuel Rigger from ETH talks about the testing framework that helps find 400+ logic bugs in DBMSs like TiDB, MySQL, PostgreSQL, and SQLite.
Chaos Practice in PingCAP
Building a distributed database is very difficult because we have to make sure that the user data is secure. At PingCAP, we did a lot of testing, including Chaos Engineering, to sure the security of TiDB. In this sharing, we talk about PingCAP's testing philosophy, Chaos practices. We'll also introduce Chaos Mesh, a Chaos Engineering platform based on K8s, and how to use Chaos Mesh to test against your application.
TiDB at PayPay
PayPay migrated their payment database from Amazon Aurora to TiDB in 3 months. They chose TiDB for its horizontal scalability, high availability, and ability to remove the need for application-level sharding. They performed an accuracy verification by comparing data between the old and new databases, as well as across microservices. Performance and availability testing was also conducted during the migration to validate the migration was successful. After 3 months of the new TiDB database in production, PayPay saw the expected performance improvements and zero incidents, finding TiDB to be a reliable replacement.
Paper Reading: FPTree
This paper presents a hybrid SCM-DRAM persistent tree, which stores leaf nodes in SCM (Storage Class Memory) and keeps inner nodes in DRAM for better performance. Its design ideas are worth taking further for SCM devices.
Paper: https://wwwdb.inf.tu-dresden.de/misc/papers/2016/Oukid_FPTree.pdf
Paper Reading: Smooth Scan
In seek of a robust, ideal execution plan, this paper proposes an access path operator, Smooth Scan, which can continuously morphs between IndexScan and TableScan as selectivity knowledge evolves at run-time.
Paper link: https://scholar.harvard.edu/files/stratos/files/smooth_vldbj.pdf
Paper Reading: Flexible Paxos
This PaperReading recaps the basics of Paxos first, then introduces Flexible Paxos which no longer requires quorums from the same Paxos phase intersect. It allows a better trade-off between resilience and performance.
Paper reading: Cost-based Query Transformation in Oracle
Query transformation in Oracle can be heuristic or cost-based. This 2006 paper presents a cost-based transformation framework that combines logical transformation and physical optimization for an optimal execution plan, as well as some efficient algorithms for enumerating the search space of it.
Paper reading: HashKV and beyond
HashKV aims for efficient updates and GC atop KV separation. This #PaperReading highlights HashKV's design—hash-based data grouping and hotness-awareness, weighs its pros and cons from 5 metrics, and shares some insights into workflow optimization.
More from PingCAP (20)
[Paper Reading] Efficient Query Processing with Optimistically Compressed Has...
[Paper Reading] Efficient Query Processing with Optimistically Compressed Has...
[Paper Reading]Orca: A Modular Query Optimizer Architecture for Big Data
[Paper Reading]Orca: A Modular Query Optimizer Architecture for Big Data
[Paper Reading]KVSSD: Close integration of LSM trees and flash translation la...
[Paper Reading]KVSSD: Close integration of LSM trees and flash translation la...
[Paper Reading]Chucky: A Succinct Cuckoo Filter for LSM-Tree
[Paper Reading]Chucky: A Succinct Cuckoo Filter for LSM-Tree
[Paper Reading]The Bw-Tree: A B-tree for New Hardware Platforms
[Paper Reading]The Bw-Tree: A B-tree for New Hardware Platforms
[Paper Reading] Leases: An Efficient Fault-Tolerant Mechanism for Distribute...
[Paper Reading] Leases: An Efficient Fault-Tolerant Mechanism for Distribute...
[Paper reading] Interleaving with Coroutines: A Practical Approach for Robust...
[Paper reading] Interleaving with Coroutines: A Practical Approach for Robust...
[Paper Reading] Generalized Sub-Query Fusion for Eliminating Redundant I/O fr...
[Paper Reading] Generalized Sub-Query Fusion for Eliminating Redundant I/O fr...
[Paper Reading] Steering Query Optimizers: A Practical Take on Big Data Workl...
[Paper Reading] Steering Query Optimizers: A Practical Take on Big Data Workl...
The Dark Side Of Go -- Go runtime related problems in TiDB in production
The Dark Side Of Go -- Go runtime related problems in TiDB in production
Paper reading: Cost-based Query Transformation in Oracle
Paper reading: Cost-based Query Transformation in Oracle
Recently uploaded
3D Hybrid PIC simulation of the plasma expansion (ISSS-14)
3D Particle-In-Cell (PIC) algorithm,
Plasma expansion in the dipole magnetic field.
Comparing Evolved Extractive Text Summary Scores of Bidirectional Encoder Rep...
Comparing Evolved Extractive Text Summary Scores of Bidirectional Encoder Rep...University of Maribor
Slides from:
11th International Conference on Electrical, Electronics and Computer Engineering (IcETRAN), Niš, 3-6 June 2024
Track: Artificial Intelligence
https://www.etran.rs/2024/en/home-english/Deep Software Variability and Frictionless Reproducibility
Deep Software Variability and Frictionless ReproducibilityUniversity of Rennes, INSA Rennes, Inria/IRISA, CNRS
The ability to recreate computational results with minimal effort and actionable metrics provides a solid foundation for scientific research and software development. When people can replicate an analysis at the touch of a button using open-source software, open data, and methods to assess and compare proposals, it significantly eases verification of results, engagement with a diverse range of contributors, and progress. However, we have yet to fully achieve this; there are still many sociotechnical frictions.
Inspired by David Donoho's vision, this talk aims to revisit the three crucial pillars of frictionless reproducibility (data sharing, code sharing, and competitive challenges) with the perspective of deep software variability.
Our observation is that multiple layers — hardware, operating systems, third-party libraries, software versions, input data, compile-time options, and parameters — are subject to variability that exacerbates frictions but is also essential for achieving robust, generalizable results and fostering innovation. I will first review the literature, providing evidence of how the complex variability interactions across these layers affect qualitative and quantitative software properties, thereby complicating the reproduction and replication of scientific studies in various fields.
I will then present some software engineering and AI techniques that can support the strategic exploration of variability spaces. These include the use of abstractions and models (e.g., feature models), sampling strategies (e.g., uniform, random), cost-effective measurements (e.g., incremental build of software configurations), and dimensionality reduction methods (e.g., transfer learning, feature selection, software debloating).
I will finally argue that deep variability is both the problem and solution of frictionless reproducibility, calling the software science community to develop new methods and tools to manage variability and foster reproducibility in software systems.
Exposé invité Journées Nationales du GDR GPL 2024
Remote Sensing and Computational, Evolutionary, Supercomputing, and Intellige...
Remote Sensing and Computational, Evolutionary, Supercomputing, and Intellige...University of Maribor
Slides from talk:
Aleš Zamuda: Remote Sensing and Computational, Evolutionary, Supercomputing, and Intelligent Systems.
11th International Conference on Electrical, Electronics and Computer Engineering (IcETRAN), Niš, 3-6 June 2024
Inter-Society Networking Panel GRSS/MTT-S/CIS Panel Session: Promoting Connection and Cooperation
https://www.etran.rs/2024/en/home-english/ANAMOLOUS SECONDARY GROWTH IN DICOT ROOTS.pptx
Abnormal or anomalous secondary growth in plants. It defines secondary growth as an increase in plant girth due to vascular cambium or cork cambium. Anomalous secondary growth does not follow the normal pattern of a single vascular cambium producing xylem internally and phloem externally.
The binding of cosmological structures by massless topological defects
Assuming spherical symmetry and weak field, it is shown that if one solves the Poisson equation or the Einstein field
equations sourced by a topological defect, i.e. a singularity of a very specific form, the result is a localized gravitational
field capable of driving flat rotation (i.e. Keplerian circular orbits at a constant speed for all radii) of test masses on a thin
spherical shell without any underlying mass. Moreover, a large-scale structure which exploits this solution by assembling
concentrically a number of such topological defects can establish a flat stellar or galactic rotation curve, and can also deflect
light in the same manner as an equipotential (isothermal) sphere. Thus, the need for dark matter or modified gravity theory is
mitigated, at least in part.
如何办理(uvic毕业证书)维多利亚大学毕业证本科学位证书原版一模一样
原版纸张【微信：741003700 】【(uvic毕业证书)维多利亚大学毕业证】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
ESR spectroscopy in liquid food and beverages.pptx
With increasing population, people need to rely on packaged food stuffs. Packaging of food materials requires the preservation of food. There are various methods for the treatment of food to preserve them and irradiation treatment of food is one of them. It is the most common and the most harmless method for the food preservation as it does not alter the necessary micronutrients of food materials. Although irradiated food doesn’t cause any harm to the human health but still the quality assessment of food is required to provide consumers with necessary information about the food. ESR spectroscopy is the most sophisticated way to investigate the quality of the food and the free radicals induced during the processing of the food. ESR spin trapping technique is useful for the detection of highly unstable radicals in the food. The antioxidant capability of liquid food and beverages in mainly performed by spin trapping technique.
Nucleic Acid-its structural and functional complexity.
This presentation explores a brief idea about the structural and functional attributes of nucleotides, the structure and function of genetic materials along with the impact of UV rays and pH upon them.
Nucleophilic Addition of carbonyl compounds.pptx
Nucleophilic addition is the most important reaction of carbonyls. Not just aldehydes and ketones, but also carboxylic acid derivatives in general.
Carbonyls undergo addition reactions with a large range of nucleophiles.
Comparing the relative basicity of the nucleophile and the product is extremely helpful in determining how reversible the addition reaction is. Reactions with Grignards and hydrides are irreversible. Reactions with weak bases like halides and carboxylates generally don’t happen.
Electronic effects (inductive effects, electron donation) have a large impact on reactivity.
Large groups adjacent to the carbonyl will slow the rate of reaction.
Neutral nucleophiles can also add to carbonyls, although their additions are generally slower and more reversible. Acid catalysis is sometimes employed to increase the rate of addition.
Shallowest Oil Discovery of Turkiye.pptx
The Petroleum System of the Çukurova Field - the Shallowest Oil Discovery of Türkiye, Adana
What is greenhouse gasses and how many gasses are there to affect the Earth.
What are greenhouse gasses how they affect the earth and its environment what is the future of the environment and earth how the weather and the climate effects.
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...
Functional Magnetic Resonance Imaging (fMRI) provides means to characterize brain activations in response to behavior. However, cognitive neuroscience has been limited to group-level effects referring to the performance of specific tasks. To obtain the functional profile of elementary cognitive mechanisms, the combination of brain responses to many tasks is required. Yet, to date, both structural atlases and parcellation-based activations do not fully account for cognitive function and still present several limitations. Further, they do not adapt overall to individual characteristics. In this talk, I will give an account of deep-behavioral phenotyping strategies, namely data-driven methods in large task-fMRI datasets, to optimize functional brain-data collection and improve inference of effects-of-interest related to mental processes. Key to this approach is the employment of fast multi-functional paradigms rich on features that can be well parametrized and, consequently, facilitate the creation of psycho-physiological constructs to be modelled with imaging data. Particular emphasis will be given to music stimuli when studying high-order cognitive mechanisms, due to their ecological nature and quality to enable complex behavior compounded by discrete entities. I will also discuss how deep-behavioral phenotyping and individualized models applied to neuroimaging data can better account for the subject-specific organization of domain-general cognitive systems in the human brain. Finally, the accumulation of functional brain signatures brings the possibility to clarify relationships among tasks and create a univocal link between brain systems and mental functions through: (1) the development of ontologies proposing an organization of cognitive processes; and (2) brain-network taxonomies describing functional specialization. To this end, tools to improve commensurability in cognitive science are necessary, such as public repositories, ontology-based platforms and automated meta-analysis tools. I will thus discuss some brain-atlasing resources currently under development, and their applicability in cognitive as well as clinical neuroscience.
Unveiling the Energy Potential of Marshmallow Deposits.pdf
Unveiling the Energy Potential of Marshmallow Deposits: A Revolutionary
Breakthrough in Sustainable Energy Science
SAR of Medicinal Chemistry 1st by dk.pdf
In this presentation include the prototype drug SAR on thus or with their examples .
Syllabus of Second Year B. Pharmacy
2019 PATTERN.
THEMATIC APPERCEPTION TEST(TAT) cognitive abilities, creativity, and critic...
THEMATIC APPERCEPTION TEST(TAT) cognitive abilities, creativity, and critic...Abdul Wali Khan University Mardan,kP,Pakistan
hematic appreciation test is a psychological assessment tool used to measure an individual's appreciation and understanding of specific themes or topics. This test helps to evaluate an individual's ability to connect different ideas and concepts within a given theme, as well as their overall comprehension and interpretation skills. The results of the test can provide valuable insights into an individual's cognitive abilities, creativity, and critical thinking skillsRecently uploaded (20)
3D Hybrid PIC simulation of the plasma expansion (ISSS-14)
3D Hybrid PIC simulation of the plasma expansion (ISSS-14)
Comparing Evolved Extractive Text Summary Scores of Bidirectional Encoder Rep...
Comparing Evolved Extractive Text Summary Scores of Bidirectional Encoder Rep...
Deep Software Variability and Frictionless Reproducibility
Deep Software Variability and Frictionless Reproducibility
Remote Sensing and Computational, Evolutionary, Supercomputing, and Intellige...
Remote Sensing and Computational, Evolutionary, Supercomputing, and Intellige...
The binding of cosmological structures by massless topological defects
The binding of cosmological structures by massless topological defects
ESR spectroscopy in liquid food and beverages.pptx
ESR spectroscopy in liquid food and beverages.pptx
Nucleic Acid-its structural and functional complexity.
Nucleic Acid-its structural and functional complexity.
What is greenhouse gasses and how many gasses are there to affect the Earth.
What is greenhouse gasses and how many gasses are there to affect the Earth.
aziz sancar nobel prize winner: from mardin to nobel
aziz sancar nobel prize winner: from mardin to nobel
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...
Unveiling the Energy Potential of Marshmallow Deposits.pdf
Unveiling the Energy Potential of Marshmallow Deposits.pdf
THEMATIC APPERCEPTION TEST(TAT) cognitive abilities, creativity, and critic...
THEMATIC APPERCEPTION TEST(TAT) cognitive abilities, creativity, and critic...
[Paper Reading] QAGen: Generating query-aware test databases
- 3. How to simulate clients’ bussiness? ● Ideal circumstance: Database Schema + Real Data + SQL ○ Sometimes Real Data is not available ● Alternatives: Database Schema + SQL + (Statistics?) ● Mock distribution/ domains for an attribution ○ pk -> uniform distribution ○ pk + fk -> Intergration Constriant ○ Zipfian ○ Enum Type (every low cardinality) ● Some SQL are too complex to understand: ○ TPC… SSB… ○ Heavy Anylitical Tasks ● We need a query-aware generator
- 4. Hassles of Making a Generator ● For simple query, the solution is simple and direct ○ select [cols...] from t1 join t2 on t1.pk = t2.fk ● What if we need to simulate some correlation ○ where a < ‘xx’ and b > ‘yy’ ● Or proccess complex projection ○ select case when cond1 then ***, case when cond2 ● And the combination of above circumstances….
- 5. Two papers with two different methods ● QAGen ● DGL
- 6. QAGen: Generating query- aware test databases
- 7. Symbolic Execution ● Volcano mode iterator ● Data are symbols instead of real data ● Generate constraints during execution
- 8. Symbolic Execution ● Volcano mode iterator ● Data are symbols instead of real data ● Generate constraints during execution
- 9. Agenda of QAGen ● Analyze the Query && Find out the Available “Knob”s ○ A knob of operator is the cardinality of the output of the operator ○ or the Distribution ○ if pre-grouped ● Do the Symbolic Execution (and Get the Constraint of the Data) ● Instantiating the Data ○ Always with expensive cost
- 11. Symbolic Execution - Selection ● GetNext for child, read tuple “t” ● [Positive Tuple Annotation] if output not reach c ○ for each symbol s in t, insert <s, p> ○ return t to parent ● [Negative Tuple Processing] ○ insert <s, !p>
- 12. Symbolic Execution - Join ● Generate Join Distribution ● GetNext for (outer) child, read tuple “t” ● [Positive Tuple Annotation] replace the fk with pk ● [Negative Tuple Processing]
- 13. Instantiazing ● A typical process of model checking. ● Low efficiency
- 14. Do we need a state-of-the-art generator? ● Pros: ○ without meddling, we can definitely get effective data ○ work for complex quries that are not understandable ○ perfectly guarantee constriants ■ Cardinality constriants ■ Integretry constriants ● Cons: ○ low efficiency ○ some cases still can’t be processed ● How about introducing an intermediate DSL
- 16. Another alternative ● What if we have understood the workload pretty well? ● Then use a more flexiable DSL instead of coding for every workload? ● DGL (Data Generate Language): ○ Scalar ○ Rows ○ Iterator ■ Distribution ■ Real SQL (Query / Persist) ○ Tables ○ Expression / Functions
- 17. An example of TPC-H
- 18. Thanks