Strategies for SQL Server Index Analysis
•
1 like•1,358 views
Properly managing indexes for a database is a common use in many environments. When reviewing an index, should it have a single or multiple columns? Should the table be a heap, a clustered index or clustered columnstore index? These considerations are often at the forefront when analyzing your indexes. In this session, we'll look at easy methods for identifying new indexes, we'll review patterns for index consolidation, and discuss how and when to remove indexes. At the end, you'll have a strategy that helps design indexes to improve performance.
Report
Share
Report
Share
Download to read offline
Recommended
Introduction to Columnstore Indexes
Columnstore indexes were first introduced in SQL Server 2012. Columnstore indexes provide the ability to flip-over your indexes from column-based to row-based, leading to dramatic changes in the storage and performance of your data. In this session, we'll review the basic concepts of columnstore indexes and look at the changes coming to them in SQL Server 2014
Introducing Azure Databases
This document discusses various database options available on Azure, including SQL Database, SQL Data Warehouse, DocumentDB, MongoDB, and Redis Cache. It explores how to set up an Azure SQL database and DocumentDB database, including configuring firewall settings. It also covers restoring databases, using elastic pools, querying data, monitoring performance, and maintaining databases using tools like Dynamic Management Views and Powershell. The goal is to understand Azure database options and how to control, monitor, and maintain databases on Azure.
Intoduction to sql 2012 Tabular Modeling
New modeling option in SQL 2012 Analysis Services – Tabular Server
•BISM Vision
•Table-like modeling
•Finding Remote of James Bond Car
•ABCD – Anybody Can Dance
Understanding Tableau's Visual Pipeline
Powerpoint deck of breakout session on the topic from Tableau customer conference in 2015 at MGM Grand Las Vegas.
Tuning and Optimizing U-SQL Queries (SQLPASS 2016)
This document discusses tuning and optimizing U-SQL queries for maximum performance. It provides an overview of U-SQL query execution, performance analysis, and various tuning and optimization techniques such as cost optimizations, data partitioning, predicate pushing, and column pruning. The document also discusses how to write UDOs (user defined operators) and how they can impact performance.
Modernizing ETL with Azure Data Lake: Hyperscale, multi-format, multi-platfor...
More and more customers who are looking to modernize analytics needs are exploring the data lake approach in Azure. Typically, they are most challenged by a bewildering array of poorly integrated technologies and a variety of data formats, data types not all of which are conveniently handled by existing ETL technologies. In this session, we’ll explore the basic shape of a modern ETL pipeline through the lens of Azure Data Lake. We will explore how this pipeline can scale from one to thousands of nodes at a moment’s notice to respond to business needs, how its extensibility model allows pipelines to simultaneously integrate procedural code written in .NET languages or even Python and R, how that same extensibility model allows pipelines to deal with a variety of formats such as CSV, XML, JSON, Images, or any enterprise-specific document format, and finally explore how the next generation of ETL scenarios are enabled though the integration of Intelligence in the data layer in the form of built-in Cognitive capabilities.
U-SQL Federated Distributed Queries (SQLBits 2016)
U-SQL Federated Distributed Queries (SQLBits 2016 ADL/U-SQL Pre-Conference)
Data Sources, External Tables, Credentials, Federated Queries
Mapping Data Flows Training deck Q1 CY22
Mapping data flows allow for code-free data transformation at scale using an Apache Spark engine within Azure Data Factory. Key points:
- Mapping data flows can handle structured and unstructured data using an intuitive visual interface without needing to know Spark, Scala, Python, etc.
- The data flow designer builds a transformation script that is executed on a JIT Spark cluster within ADF. This allows for scaled-out, serverless data transformation.
- Common uses of mapping data flows include ETL scenarios like slowly changing dimensions, analytics tasks like data profiling, cleansing, and aggregations.
Recommended
Introduction to Columnstore Indexes
Columnstore indexes were first introduced in SQL Server 2012. Columnstore indexes provide the ability to flip-over your indexes from column-based to row-based, leading to dramatic changes in the storage and performance of your data. In this session, we'll review the basic concepts of columnstore indexes and look at the changes coming to them in SQL Server 2014
Introducing Azure Databases
This document discusses various database options available on Azure, including SQL Database, SQL Data Warehouse, DocumentDB, MongoDB, and Redis Cache. It explores how to set up an Azure SQL database and DocumentDB database, including configuring firewall settings. It also covers restoring databases, using elastic pools, querying data, monitoring performance, and maintaining databases using tools like Dynamic Management Views and Powershell. The goal is to understand Azure database options and how to control, monitor, and maintain databases on Azure.
Intoduction to sql 2012 Tabular Modeling
New modeling option in SQL 2012 Analysis Services – Tabular Server
•BISM Vision
•Table-like modeling
•Finding Remote of James Bond Car
•ABCD – Anybody Can Dance
Understanding Tableau's Visual Pipeline
Powerpoint deck of breakout session on the topic from Tableau customer conference in 2015 at MGM Grand Las Vegas.
Tuning and Optimizing U-SQL Queries (SQLPASS 2016)
This document discusses tuning and optimizing U-SQL queries for maximum performance. It provides an overview of U-SQL query execution, performance analysis, and various tuning and optimization techniques such as cost optimizations, data partitioning, predicate pushing, and column pruning. The document also discusses how to write UDOs (user defined operators) and how they can impact performance.
Modernizing ETL with Azure Data Lake: Hyperscale, multi-format, multi-platfor...
More and more customers who are looking to modernize analytics needs are exploring the data lake approach in Azure. Typically, they are most challenged by a bewildering array of poorly integrated technologies and a variety of data formats, data types not all of which are conveniently handled by existing ETL technologies. In this session, we’ll explore the basic shape of a modern ETL pipeline through the lens of Azure Data Lake. We will explore how this pipeline can scale from one to thousands of nodes at a moment’s notice to respond to business needs, how its extensibility model allows pipelines to simultaneously integrate procedural code written in .NET languages or even Python and R, how that same extensibility model allows pipelines to deal with a variety of formats such as CSV, XML, JSON, Images, or any enterprise-specific document format, and finally explore how the next generation of ETL scenarios are enabled though the integration of Intelligence in the data layer in the form of built-in Cognitive capabilities.
U-SQL Federated Distributed Queries (SQLBits 2016)
U-SQL Federated Distributed Queries (SQLBits 2016 ADL/U-SQL Pre-Conference)
Data Sources, External Tables, Credentials, Federated Queries
Mapping Data Flows Training deck Q1 CY22
Mapping data flows allow for code-free data transformation at scale using an Apache Spark engine within Azure Data Factory. Key points:
- Mapping data flows can handle structured and unstructured data using an intuitive visual interface without needing to know Spark, Scala, Python, etc.
- The data flow designer builds a transformation script that is executed on a JIT Spark cluster within ADF. This allows for scaled-out, serverless data transformation.
- Common uses of mapping data flows include ETL scenarios like slowly changing dimensions, analytics tasks like data profiling, cleansing, and aggregations.
Azure Data Factory
Azure Data Factory is a data integration service that allows for data movement and transformation between both on-premises and cloud data stores. It uses datasets to represent data structures, activities to define actions on data with pipelines grouping related activities, and linked services to connect to external resources. Key concepts include datasets representing input/output data, activities performing actions like copy, and pipelines logically grouping activities.
Harnessing the power of both worlds
The document discusses harnessing the power of SQL Server columnstore indexes and Analysis Services ROLAP. It finds that combining clustered columnstore indexes with ROLAP in Analysis Services provides very fast performance for aggregates and distinct counts on large datasets of over 1 billion records, returning results within seconds. It recommends settings like enabling ROLAP distinct counts at the data source and maintaining statistics to optimize query plans when using this solution.
Jaws - Data Warehouse with Spark SQL by Ema Orhian
1) Jaws is a highly scalable and resilient data warehouse explorer that allows submitting Spark SQL queries concurrently and asynchronously through a RESTful API.
2) It provides features like persisted query logs, results pagination, and pluggable storage layers. Queries can be run on Spark SQL contexts configured to use data from HDFS, Cassandra, Parquet files on HDFS or Tachyon.
3) The architecture allows Jaws to scale on standalone, Mesos, or YARN clusters by distributing queries across multiple worker nodes, and supports canceling running queries.
U-SQL Partitioned Data and Tables (SQLBits 2016)
This document discusses data partitioning and distribution in U-SQL. It explains how to use partitioned tables to get benefits like partition elimination in queries. Finely partitioning tables on keys like date and hashing on other keys can improve query performance by pruning partitions and distributions. The document also covers data skew that can occur if one partition receives too much data, and provides options to address it like repartitioning the data or using multiple partitioning keys.
Azure data factory
What are the evolving approaches to analytics? What is Azure Data Factory? Capabilities of Azure Data Factory
SQL Server Workshop for Developers - Visual Studio Live! NY 2012
Here are the key enhancements to the OVER clause in SQL Server 2012:
- Ability to specify both PARTITION BY and ORDER BY clauses, allowing window functions to operate over ordered partitions of data. This enables more advanced windowing scenarios.
- New window frame specifications - RANGE and ROWS, which allow window functions to consider rows before and/or after the current row within the partition. This enables window functions like moving averages.
- Ability to use window functions with aggregate functions like SUM(), COUNT(), etc. This allows calculations like cumulative sums within partitions.
- Performance improvements for window functions through optimizations like push-down of predicates and better parallelization.
So in summary, SQL Server 2012 significantly expands the
Nosql data models
The document discusses different NoSQL data models including key-value, document, column family, and graph models. It provides examples of popular NoSQL databases that implement each model such as Redis, MongoDB, Cassandra, and Neo4j. The document argues that these NoSQL databases address limitations of relational databases in supporting modern web applications with requirements for scalability, flexibility, and high performance.
The olap tutorial 2012
This document provides an introduction to creating an OLAP (Online Analytical Processing) project in Microsoft SQL Server Analysis Services (SSAS) 2012. It discusses connecting to data sources, creating dimensions and hierarchies, building cubes, and defining calculations and KPIs. The tutorial uses a sample product inventory dataset to demonstrate how to design and deploy an SSAS project that can then be accessed using Microsoft Excel for analysis and reporting.
Microsoft's Hadoop Story
This document discusses Hadoop and its relationship to Microsoft technologies. It provides an overview of what Big Data is, how Hadoop fits into the Windows and Azure environments, and how to program against Hadoop in Microsoft environments. It describes Hadoop capabilities like Extract-Load-Transform and distributed computing. It also discusses how HDFS works on Azure storage and support for Hadoop in .NET, JavaScript, HiveQL, and Polybase. The document aims to show Microsoft's vision of making Hadoop better on Windows and Azure by integrating with technologies like Active Directory, System Center, and SQL Server. It provides links to get started with Hadoop on-premises and on Windows Azure.
Azure Analysis Services (Azure Bootcamp 2018)
An overview of AAS and the slides presented during Azure Bootcamp Day 2018 in the SafeNet offices, downtown Milwaukee, WI.
Azure Data Factory Data Flow Preview December 2019
Visual Data Flow in Azure Data Factory provides a limited preview of data flows that allow users to visually design transformations on data. It features implicit staging of data in data lakes, explicit selection of data sources and transformations through a toolbox interface, and setting of properties for transformation steps and destination connectors. The preview is intended to get early feedback to help shape the future of Visual Data Flow.
Big Challenges in Data Modeling: NoSQL and Data Modeling
Big Data and NoSQL have led to big changes In the data environment, but are they all in the best interest of data? Are they technologies that "free us from the harsh limitations of relational databases?"
In this month's webinar, we will be answering questions like these, plus:
Have we managed to free organizations from having to do Data Modeling?
Is there a need for a Data Modeler on NoSQL projects?
If we build Data Models, which types will work?
If we build Data Models, how will they be used?
If we build Data Models, when will they be used?
Who will use Data Models?
Where does Data Quality happen?
Finally, we will wrap with 10 tips for data modelers in organizations incorporating NoSQL in their modern Data Architectures.
Data visualization with sql analytics
SQL Analytics provides data visualization and dashboarding capabilities on data lakes to help analysts, sales executives, marketing teams, and finance departments make more data-driven decisions. It allows users to build live visualizations and dashboards from their data, analyze it using SQL queries for reports or alerts, and connect to over 40 different data sources. Key features include a simple query editor, browsing table schemas, autocomplete functions, secure collaboration through shared dashboards and queries, and automation through alerts and parameterized queries and dashboards.
ADF Mapping Data Flows Level 300
Azure Data Factory can now use Mapping Data Flows to orchestrate ETL workloads. Mapping Data Flows allow users to visually design transformations on data from disparate sources and load the results into Azure SQL Data Warehouse for analytics. The key benefits of Mapping Data Flows are that they provide a visual interface for building expressions to cleanse and join data with auto-complete assistance and live previews of expression results.
Business intelligence
Business intelligence (BI) refers to technologies, applications, and practices used to collect, integrate, analyze, and present data to facilitate informed decision-making. BI involves collecting data from various structured and unstructured sources, integrating the data through extraction, transformation, and loading (ETL) processes, and analyzing and presenting the information to users. Common BI tools include Hadoop, Apache Spark, Apache Hive, SAP HANA, Sybase IQ, Informatica, and Tableau.
Sap business objects data services toc
This 40 hour course provides an introduction to SAP Business Objects Data Services (BODS), which is SAP's ETL platform. It allows users to extract, transform and load data from various sources into a single location. The course covers fundamental concepts like the BODS architecture and components. It teaches how to design ETL jobs using the BODS designer tool. Additional topics include defining source and target metadata, creating and troubleshooting batch jobs, using functions and scripts, platform and data quality transforms, addressing cleansing, data cleansing, matching and consolidating data, SAP interfaces, data profiling, and migration. The trainer has over 14 years of experience in data warehousing and business intelligence projects.
Data virtualization using polybase
This document provides an overview of using Polybase for data virtualization in SQL Server. It discusses installing and configuring Polybase, connecting external data sources like Azure Blob Storage and SQL Server, using Polybase DMVs for monitoring and troubleshooting, and techniques for optimizing performance like predicate pushdown and creating statistics on external tables. The presentation aims to explain how Polybase can be leveraged to virtually access and query external data using T-SQL without needing to know the physical data locations or move the data.
Data Modeling for NoSQL
- Data modeling for NoSQL databases is different than relational databases and requires designing the data model around access patterns rather than object structure. Key differences include not having joins so data needs to be duplicated and modeling the data in a way that works for querying, indexing, and retrieval speed.
- The data model should focus on making the most of features like atomic updates, inner indexes, and unique identifiers. It's also important to consider how data will be added, modified, and retrieved factoring in object complexity, marshalling/unmarshalling costs, and index maintenance.
- The _id field can be tailored to the access patterns, such as using dates for time-series data to keep recent
Spark SQL In Depth www.syedacademy.com
Spark SQL allows users to perform relational operations on Spark's RDDs using a DataFrame API. It addresses challenges in existing systems like limited optimization and data sources by providing a DataFrame API that can query both external data and RDDs. Spark SQL leverages a highly extensible optimizer called Catalyst to optimize logical query plans into efficient physical query plans using features of Scala. It has been part of the Spark core distribution since version 1.0 in 2014.
The Critical Missing Component in the Production ML Stack
The day the ML application is deployed to production and begins facing the real world is the best and the worst day in the life of the model builder. The joy of seeing accurate predictions is quickly overshadowed by a myriad of operational challenges. Debugging, troubleshooting & monitoring takes over the majority of their day, leaving little time for model building. In DevOps, software operations are taken to a level of an art. Sophisticated tools enable engineers to quickly identify and resolve issues, continuously improving software stability and robustness. In the ML world, operations are still largely a manual process that involves Jupyter notebooks and shell scripts. One of the cornerstones of the DevOps toolchain is logging. Traces and metrics are built on top of logs enabling monitoring and feedback loops. What does logging look like in an ML system?
In this talk we will demonstrate how to enable data logging for an AI application using MLflow in a matter of minutes. We will discuss how something so simple enables testing, monitoring and debugging in an AI application that handles TBs of data and runs in real-time. Attendees will leave the talk equipped with tools and best practices to supercharge MLOps in their team.
The Plan Cache Whisperer - Performance Tuning SQL Server
Execution plans tell SQL Server how to execute queries. If you listen closely, execution plans can also tell you when performance-tuning opportunities exist in your environment. By listening to your queries, you can understand how SQL Server is operating and gain insight into how your environment is functioning. In this session, learn how to use XQuery to browse and search the plan cache, enabling you to find potential performance issues and opportunities to tune your queries. In addition, learn how a performance issue on a single execution plan can be used to find similar issues on other execution plans, enabling you to scale up your performance tuning effectiveness. You can use this information to help reduce issues related to parallelism, shift queries from using scans to using seek operations, or discover exactly which queries are using which indexes. All this and more is readily available through the plan cache.
Index in sql server
Index is a database object, which can be created on one or more columns (16 Max column combinations). When creating the index will read the column(s) and forms a relevant data structure to minimize the number of data comparisons. The index will improve the performance of data retrieval and adds some overhead on data modification such as create, delete and modify. So it depends on how much data retrieval can be performed on table versus how much of DML (Insert, Delete and Update) operations
More Related Content
What's hot
Azure Data Factory
Azure Data Factory is a data integration service that allows for data movement and transformation between both on-premises and cloud data stores. It uses datasets to represent data structures, activities to define actions on data with pipelines grouping related activities, and linked services to connect to external resources. Key concepts include datasets representing input/output data, activities performing actions like copy, and pipelines logically grouping activities.
Harnessing the power of both worlds
The document discusses harnessing the power of SQL Server columnstore indexes and Analysis Services ROLAP. It finds that combining clustered columnstore indexes with ROLAP in Analysis Services provides very fast performance for aggregates and distinct counts on large datasets of over 1 billion records, returning results within seconds. It recommends settings like enabling ROLAP distinct counts at the data source and maintaining statistics to optimize query plans when using this solution.
Jaws - Data Warehouse with Spark SQL by Ema Orhian
1) Jaws is a highly scalable and resilient data warehouse explorer that allows submitting Spark SQL queries concurrently and asynchronously through a RESTful API.
2) It provides features like persisted query logs, results pagination, and pluggable storage layers. Queries can be run on Spark SQL contexts configured to use data from HDFS, Cassandra, Parquet files on HDFS or Tachyon.
3) The architecture allows Jaws to scale on standalone, Mesos, or YARN clusters by distributing queries across multiple worker nodes, and supports canceling running queries.
U-SQL Partitioned Data and Tables (SQLBits 2016)
This document discusses data partitioning and distribution in U-SQL. It explains how to use partitioned tables to get benefits like partition elimination in queries. Finely partitioning tables on keys like date and hashing on other keys can improve query performance by pruning partitions and distributions. The document also covers data skew that can occur if one partition receives too much data, and provides options to address it like repartitioning the data or using multiple partitioning keys.
Azure data factory
What are the evolving approaches to analytics? What is Azure Data Factory? Capabilities of Azure Data Factory
SQL Server Workshop for Developers - Visual Studio Live! NY 2012
Here are the key enhancements to the OVER clause in SQL Server 2012:
- Ability to specify both PARTITION BY and ORDER BY clauses, allowing window functions to operate over ordered partitions of data. This enables more advanced windowing scenarios.
- New window frame specifications - RANGE and ROWS, which allow window functions to consider rows before and/or after the current row within the partition. This enables window functions like moving averages.
- Ability to use window functions with aggregate functions like SUM(), COUNT(), etc. This allows calculations like cumulative sums within partitions.
- Performance improvements for window functions through optimizations like push-down of predicates and better parallelization.
So in summary, SQL Server 2012 significantly expands the
Nosql data models
The document discusses different NoSQL data models including key-value, document, column family, and graph models. It provides examples of popular NoSQL databases that implement each model such as Redis, MongoDB, Cassandra, and Neo4j. The document argues that these NoSQL databases address limitations of relational databases in supporting modern web applications with requirements for scalability, flexibility, and high performance.
The olap tutorial 2012
This document provides an introduction to creating an OLAP (Online Analytical Processing) project in Microsoft SQL Server Analysis Services (SSAS) 2012. It discusses connecting to data sources, creating dimensions and hierarchies, building cubes, and defining calculations and KPIs. The tutorial uses a sample product inventory dataset to demonstrate how to design and deploy an SSAS project that can then be accessed using Microsoft Excel for analysis and reporting.
Microsoft's Hadoop Story
This document discusses Hadoop and its relationship to Microsoft technologies. It provides an overview of what Big Data is, how Hadoop fits into the Windows and Azure environments, and how to program against Hadoop in Microsoft environments. It describes Hadoop capabilities like Extract-Load-Transform and distributed computing. It also discusses how HDFS works on Azure storage and support for Hadoop in .NET, JavaScript, HiveQL, and Polybase. The document aims to show Microsoft's vision of making Hadoop better on Windows and Azure by integrating with technologies like Active Directory, System Center, and SQL Server. It provides links to get started with Hadoop on-premises and on Windows Azure.
Azure Analysis Services (Azure Bootcamp 2018)
An overview of AAS and the slides presented during Azure Bootcamp Day 2018 in the SafeNet offices, downtown Milwaukee, WI.
Azure Data Factory Data Flow Preview December 2019
Visual Data Flow in Azure Data Factory provides a limited preview of data flows that allow users to visually design transformations on data. It features implicit staging of data in data lakes, explicit selection of data sources and transformations through a toolbox interface, and setting of properties for transformation steps and destination connectors. The preview is intended to get early feedback to help shape the future of Visual Data Flow.
Big Challenges in Data Modeling: NoSQL and Data Modeling
Big Data and NoSQL have led to big changes In the data environment, but are they all in the best interest of data? Are they technologies that "free us from the harsh limitations of relational databases?"
In this month's webinar, we will be answering questions like these, plus:
Have we managed to free organizations from having to do Data Modeling?
Is there a need for a Data Modeler on NoSQL projects?
If we build Data Models, which types will work?
If we build Data Models, how will they be used?
If we build Data Models, when will they be used?
Who will use Data Models?
Where does Data Quality happen?
Finally, we will wrap with 10 tips for data modelers in organizations incorporating NoSQL in their modern Data Architectures.
Data visualization with sql analytics
SQL Analytics provides data visualization and dashboarding capabilities on data lakes to help analysts, sales executives, marketing teams, and finance departments make more data-driven decisions. It allows users to build live visualizations and dashboards from their data, analyze it using SQL queries for reports or alerts, and connect to over 40 different data sources. Key features include a simple query editor, browsing table schemas, autocomplete functions, secure collaboration through shared dashboards and queries, and automation through alerts and parameterized queries and dashboards.
ADF Mapping Data Flows Level 300
Azure Data Factory can now use Mapping Data Flows to orchestrate ETL workloads. Mapping Data Flows allow users to visually design transformations on data from disparate sources and load the results into Azure SQL Data Warehouse for analytics. The key benefits of Mapping Data Flows are that they provide a visual interface for building expressions to cleanse and join data with auto-complete assistance and live previews of expression results.
Business intelligence
Business intelligence (BI) refers to technologies, applications, and practices used to collect, integrate, analyze, and present data to facilitate informed decision-making. BI involves collecting data from various structured and unstructured sources, integrating the data through extraction, transformation, and loading (ETL) processes, and analyzing and presenting the information to users. Common BI tools include Hadoop, Apache Spark, Apache Hive, SAP HANA, Sybase IQ, Informatica, and Tableau.
Sap business objects data services toc
This 40 hour course provides an introduction to SAP Business Objects Data Services (BODS), which is SAP's ETL platform. It allows users to extract, transform and load data from various sources into a single location. The course covers fundamental concepts like the BODS architecture and components. It teaches how to design ETL jobs using the BODS designer tool. Additional topics include defining source and target metadata, creating and troubleshooting batch jobs, using functions and scripts, platform and data quality transforms, addressing cleansing, data cleansing, matching and consolidating data, SAP interfaces, data profiling, and migration. The trainer has over 14 years of experience in data warehousing and business intelligence projects.
Data virtualization using polybase
This document provides an overview of using Polybase for data virtualization in SQL Server. It discusses installing and configuring Polybase, connecting external data sources like Azure Blob Storage and SQL Server, using Polybase DMVs for monitoring and troubleshooting, and techniques for optimizing performance like predicate pushdown and creating statistics on external tables. The presentation aims to explain how Polybase can be leveraged to virtually access and query external data using T-SQL without needing to know the physical data locations or move the data.
Data Modeling for NoSQL
- Data modeling for NoSQL databases is different than relational databases and requires designing the data model around access patterns rather than object structure. Key differences include not having joins so data needs to be duplicated and modeling the data in a way that works for querying, indexing, and retrieval speed.
- The data model should focus on making the most of features like atomic updates, inner indexes, and unique identifiers. It's also important to consider how data will be added, modified, and retrieved factoring in object complexity, marshalling/unmarshalling costs, and index maintenance.
- The _id field can be tailored to the access patterns, such as using dates for time-series data to keep recent
Spark SQL In Depth www.syedacademy.com
Spark SQL allows users to perform relational operations on Spark's RDDs using a DataFrame API. It addresses challenges in existing systems like limited optimization and data sources by providing a DataFrame API that can query both external data and RDDs. Spark SQL leverages a highly extensible optimizer called Catalyst to optimize logical query plans into efficient physical query plans using features of Scala. It has been part of the Spark core distribution since version 1.0 in 2014.
The Critical Missing Component in the Production ML Stack
The day the ML application is deployed to production and begins facing the real world is the best and the worst day in the life of the model builder. The joy of seeing accurate predictions is quickly overshadowed by a myriad of operational challenges. Debugging, troubleshooting & monitoring takes over the majority of their day, leaving little time for model building. In DevOps, software operations are taken to a level of an art. Sophisticated tools enable engineers to quickly identify and resolve issues, continuously improving software stability and robustness. In the ML world, operations are still largely a manual process that involves Jupyter notebooks and shell scripts. One of the cornerstones of the DevOps toolchain is logging. Traces and metrics are built on top of logs enabling monitoring and feedback loops. What does logging look like in an ML system?
In this talk we will demonstrate how to enable data logging for an AI application using MLflow in a matter of minutes. We will discuss how something so simple enables testing, monitoring and debugging in an AI application that handles TBs of data and runs in real-time. Attendees will leave the talk equipped with tools and best practices to supercharge MLOps in their team.
What's hot (20)
Jaws - Data Warehouse with Spark SQL by Ema Orhian
Jaws - Data Warehouse with Spark SQL by Ema Orhian
SQL Server Workshop for Developers - Visual Studio Live! NY 2012
SQL Server Workshop for Developers - Visual Studio Live! NY 2012
Azure Data Factory Data Flow Preview December 2019
Azure Data Factory Data Flow Preview December 2019
Big Challenges in Data Modeling: NoSQL and Data Modeling
Big Challenges in Data Modeling: NoSQL and Data Modeling
The Critical Missing Component in the Production ML Stack
The Critical Missing Component in the Production ML Stack
Viewers also liked
The Plan Cache Whisperer - Performance Tuning SQL Server
Execution plans tell SQL Server how to execute queries. If you listen closely, execution plans can also tell you when performance-tuning opportunities exist in your environment. By listening to your queries, you can understand how SQL Server is operating and gain insight into how your environment is functioning. In this session, learn how to use XQuery to browse and search the plan cache, enabling you to find potential performance issues and opportunities to tune your queries. In addition, learn how a performance issue on a single execution plan can be used to find similar issues on other execution plans, enabling you to scale up your performance tuning effectiveness. You can use this information to help reduce issues related to parallelism, shift queries from using scans to using seek operations, or discover exactly which queries are using which indexes. All this and more is readily available through the plan cache.
Index in sql server
Index is a database object, which can be created on one or more columns (16 Max column combinations). When creating the index will read the column(s) and forms a relevant data structure to minimize the number of data comparisons. The index will improve the performance of data retrieval and adds some overhead on data modification such as create, delete and modify. So it depends on how much data retrieval can be performed on table versus how much of DML (Insert, Delete and Update) operations
Before you optimize: Understanding Execution Plans
You know what your query does, but do you know how it does it? Do you know what type of resources your query uses? This session covered these questions and more as we walked through reading execution plans. We saw how SQL breaks down the execution of your query and what each step tells us about the overall query. These slides provide the additional resources that go into the depth we couldn't get into in the session.
Presentation interpreting execution plans for sql statements
Download & Share Technology
Presentations http://goo.gl/k80oY0
Student Guide & Best http://goo.gl/6OkI77
Overview of query evaluation
The document discusses Oracle system catalogs which contain metadata about database objects like tables and indexes. System catalogs allow accessing information through views with prefixes like USER, ALL, and DBA. Examples show how to query system catalog views to get information on tables, columns, indexes and views. Query optimization and evaluation are also covered, explaining how queries are parsed, an execution plan is generated, and the least cost plan is chosen.
Introduction of sql server indexing
SQL Server supports two main types of indexes - clustered and nonclustered. A clustered index physically orders the data on disk based on the index key. Only one clustered index is allowed per table. A nonclustered index contains key values and row locators but does not determine the physical order of data. SQL Server supports up to 999 nonclustered indexes per table. The choice of index depends on the query patterns against the table and the desired performance characteristics.
"Using Indexes in SQL Server 2008" by Alexander Korotkiy, part 1
Speakers Corner Event at Ciklum Dnepropetrovsk Office. Alexander Korotkiy, Senior .NET Developer talking Indexes in MS SQL Server.Speakers Corner Event at Ciklum Dnepropetrovsk Office. Alexander Korotkiy, Senior .NET Developer talking Indexes in MS SQL Server.
Columnstore indexes in sql server 2014
This document provides information about an upcoming presentation on Columnstore Indexes in SQL Server 2014. It notes that the presentation will be recorded so that those who could not attend live can view it later. It requests that anyone with issues about being recorded should leave immediately, and remaining will be taken as consent to the recording. It also states the presentation will be free and will begin in 1 minute.
Database index(sql server)
This slide is introduced What is index in database system .
It's very simple to let you understand what is index pros and cons.
there has chinese voice version youtube :)
https://www.youtube.com/watch?v=BOBZAMfQfrQ&feature=youtu.be
https://www.youtube.com/watch?v=hkl5CcmZ4OI&feature=youtu.be
MySQL Performance Tuning: Top 10 Tips
This document provides tips for tuning a MySQL database to optimize performance. It discusses why tuning is important for cost effectiveness, performance, and competitive advantage. It outlines who should be involved in tuning including application designers, developers, DBAs and system administrators. The document covers what can be tuned such as applications, databases structures, and hardware. It provides best practices for when and how much to tune a database. Specific tuning techniques are discussed for various areas including application development, database design, server configuration, and storage engine optimizations.
An Introduction to Factor analysis ppt
This document discusses exploratory factor analysis (EFA). EFA is used to identify underlying factors that explain the pattern of correlations within a set of observed variables. The document outlines the steps of EFA, including testing assumptions, constructing a correlation matrix, determining the number of factors, rotating factors, and interpreting the factor loadings. It provides an example of running EFA on a dataset with 11 physical performance and anthropometric variables from 21 participants. The analysis extracts 3 factors that explain over 80% of the total variance.
How to Design Indexes, Really
MySQL users commonly ask: Here's my table, what indexes do I need? Why aren't my indexes helping me? Don't indexes cause overhead? This talk gives you some practical answers, with a step by step method for finding the queries you need to optimize, and choosing the best indexes for them.
Factor analysis
Factor analysis is a statistical technique used to reduce a large set of variables into a smaller set of underlying factors or dimensions. It examines the interrelationships among variables to define common dimensions called factors that can help explain correlations. Factor analysis is used to identify the underlying structure in a data set and reduce many variables into a smaller number of factors for subsequent analysis like regression or discriminant analysis.
Viewers also liked (13)
The Plan Cache Whisperer - Performance Tuning SQL Server
The Plan Cache Whisperer - Performance Tuning SQL Server
Before you optimize: Understanding Execution Plans
Before you optimize: Understanding Execution Plans
Presentation interpreting execution plans for sql statements
Presentation interpreting execution plans for sql statements
"Using Indexes in SQL Server 2008" by Alexander Korotkiy, part 1
"Using Indexes in SQL Server 2008" by Alexander Korotkiy, part 1
Similar to Strategies for SQL Server Index Analysis
AnalytixLabs - Data Science 360 (Nasscom)-1648178720283 (1).pdf
The document provides information about the Data Science 360 certification program offered by AnalytixLabs. It summarizes that the program focuses 80% on practical skills and industry best practices. It is led by experienced analytics professionals from top firms and includes high-quality course material, real-life case studies, and regular sessions with industry experts to help students become job-ready. Upon completion, students will receive a certificate and potential scholarship and job assistance.
DATA SCIENCE AND BIG DATA ANALYTICSCHAPTER 2 DATA ANA.docx
DATA SCIENCE AND BIG DATA
ANALYTICS
CHAPTER 2:
DATA ANALYTICS LIFECYCLE
DATA ANALYTICS LIFECYCLE
• Data science projects differ from BI projects
• More exploratory in nature
• Critical to have a project process
• Participants should be thorough and rigorous
• Break large projects into smaller pieces
• Spend time to plan and scope the work
• Documenting adds rigor and credibility
DATA ANALYTICS LIFECYCLE
• Data Analytics Lifecycle Overview
• Phase 1: Discovery
• Phase 2: Data Preparation
• Phase 3: Model Planning
• Phase 4: Model Building
• Phase 5: Communicate Results
• Phase 6: Operationalize
• Case Study: GINA
2.1 DATA ANALYTICS
LIFECYCLE OVERVIEW
• The data analytic lifecycle is designed for Big Data problems and
data science projects
• With six phases the project work can occur in several phases
simultaneously
• The cycle is iterative to portray a real project
• Work can return to earlier phases as new information is uncovered
2.1.1 KEY ROLES FOR A
SUCCESSFUL ANALYTICS
PROJECT
KEY ROLES FOR A
SUCCESSFUL ANALYTICS
PROJECT
• Business User – understands the domain area
• Project Sponsor – provides requirements
• Project Manager – ensures meeting objectives
• Business Intelligence Analyst – provides business domain
expertise based on deep understanding of the data
• Database Administrator (DBA) – creates DB environment
• Data Engineer – provides technical skills, assists data
management and extraction, supports analytic sandbox
• Data Scientist – provides analytic techniques and modeling
2.1.2 BACKGROUND AND OVERVIEW
OF DATA ANALYTICS LIFECYCLE
• Data Analytics Lifecycle defines the analytics process and
best practices from discovery to project completion
• The Lifecycle employs aspects of
• Scientific method
• Cross Industry Standard Process for Data Mining (CRISP-DM)
• Process model for data mining
• Davenport’s DELTA framework
• Hubbard’s Applied Information Economics (AIE) approach
• MAD Skills: New Analysis Practices for Big Data by Cohen et al.
https://en.wikipedia.org/wiki/Scientific_method
https://en.wikipedia.org/wiki/Cross_Industry_Standard_Process_for_Data_Mining
http://www.informationweek.com/software/information-management/analytics-at-work-qanda-with-tom-davenport/d/d-id/1085869?
https://en.wikipedia.org/wiki/Applied_information_economics
https://pafnuty.wordpress.com/2013/03/15/reading-log-mad-skills-new-analysis-practices-for-big-data-cohen/
OVERVIEW OF
DATA ANALYTICS LIFECYCLE
2.2 PHASE 1: DISCOVERY
2.2 PHASE 1: DISCOVERY
1. Learning the Business Domain
2. Resources
3. Framing the Problem
4. Identifying Key Stakeholders
5. Interviewing the Analytics Sponsor
6. Developing Initial Hypotheses
7. Identifying Potential Data Sources
2.3 PHASE 2: DATA PREPARATION
2.3 PHASE 2: DATA
PREPARATION
• Includes steps to explore, preprocess, and condition
data
• Create robust environment – analytics sandbox
• Data preparation tends to be t.
5 ways to improve performance through indexing
The document discusses 5 ways to improve database performance through indexing: 1) addressing lookup imbalance by moving clustered indexes or adding included columns, 2) reducing scans on large indexes by adding alternative indexes, 3) using filtered indexes to improve performance on queries involving a subset of rows, 4) addressing missing index recommendations to optimize queries, and 5) indexing foreign keys to avoid performance issues during DML operations.
Bdf16 big-data-warehouse-case-study-data kitchen
The list of failed big data projects is long. They leave end-users, data analysts and data scientists frustrated with long lead times for changes. This case study will illustrate how to make changes to big data, models, and visualizations quickly, with high quality, using the tools teams love. We synthesize techniques from devOps, Demming, and direct experience.
Geek Sync | Why Did My Clever Index Change Backfire?
You can watch the replay for this Geek Sync webcast in the IDERA Resource Center: http://ow.ly/LBvZ50A5a8P
You've got a great indexing strategy to speed up your SQL Server! You deploy the change and then... it all goes wrong.
Join IDERA and Kendra Little as she explains why index changes may slow down queries, cause queries to fail, cause unexpected outages, or just not work like they did when you tested them. You’ll leave the session with a set of practical steps to ensure you sidestep common indexing blunders and have confidence that your clever index change will speed up your queries instead of slowing them down. This is a demo-packed session you will not want to miss.
About Kendra: Kendra Little teaches database administrators and developers how to consistently speed up their SQL Servers. She is a Microsoft Certified Master in SQL Server, a Microsoft MVP, and one of the few artists in the world to draw both dinosaurs and databases.
Using Metrics for Fun, Developing with the KV Store + Javascript & News from ...
We explore "Metrics, mstats and Me: Splunking Human Data” and also have some insights into the KV Store and javascript use in dashboards. We’ll also re-cover the conf18 updates for those who couldn’t attend our last session.
Syd601 – assignment 1
This document outlines the key concepts of object-oriented systems development using the Unified Process methodology. It discusses objects, classes, principles like single responsibility and open/closed, the phases of Unified Process including inception, elaboration, construction and transition. It also covers the core disciplines like requirements, design, implementation and testing. Finally, it mentions some UML diagram types and the environments that will be used for the project.
Statistics and Indexes Internals
The document provides biographical information about Antonios Chatzipavlis, a SQL Server expert and evangelist. It then summarizes his presentation on statistics and index internals in SQL Server, which covers topics like cardinality estimation, inspecting and updating statistics, index structure and types, and identifying missing indexes. The presentation includes demonstrations of analyzing cardinality estimation and picking the right index key.
SAG_Indexing and Query Optimization
This document discusses indexing and query optimization in SQL Server. It provides an overview of indexes including clustered and non-clustered indexes. It describes how data is stored at the page and extent level and differences between tables with and without clustered indexes. The document also outlines the query optimization process including parsing, optimization, execution and the cost-based optimizer. Finally, it reviews common execution plan operators like table scans, index scans and seeks and when they would be considered good or bad.
System Analysis Methods
This document discusses system analysis methods. It defines system analysis as a problem solving technique that decomposes a system into component pieces to study how well they work and interact. The document outlines the objectives, stages, and phases of system analysis. It describes two main approaches: model-driven analysis which uses models like data flow diagrams; and accelerated system analysis which uses prototypes. Model-driven analysis can be structured analysis, information engineering, or object-oriented analysis. Accelerated approaches emphasize prototypes to rapidly identify requirements.
Query Optimization in SQL Server
Optimizing Query is very important to improve the performance of the database. Analyse query using query execution plan, create cluster index and non-cluster index and create indexed views
Get Testing with tSQLt - SQL In The City Workshop 2014
Get Testing with tSQLt. Practical Examples and automation.
Steve Jones presentation from SQL In The City.
DTecH IT Education- Best Obiee training institute in bangalore
Best OBIEE training institute & Oracle Partner in Bangalore! DTecH IT Education, No.1, 1st Cross, 1st Main, Ashwini Layout, Near Sony World, Koramangala 6th Blk, Bangalore-47. Contact: 080-4150-1359, 080-3221-0825
Optimising Queries - Series 1 Query Optimiser Architecture
Optimising Queries - Series 1 Query Optimiser Architecture
=> Phases
=> Strategies
=> Data access plans
=> Auto-parameterisation
=> Avoiding recompilation of queries
by: DR. SUBRAMANI PARAMASIVAM
lec1.pdf
1) The document provides an introduction to a course on data analytics with Python. It outlines the objectives of the course, which are to conceptually understand data analytics using practical examples rather than just procedures.
2) The course will cover defining data and its importance, different types of data analytics, why analytics is important today, how statistics, analytics and data science are related, and why Python is used. It will also explain the four levels of data measurement.
3) The levels of data from lowest to highest are nominal, ordinal, interval, and ratio. The type of data affects what operations and statistical methods can be meaningfully applied. Descriptive, diagnostic, predictive, and prescriptive analytics will also
RecSys 2015 Tutorial - Scalable Recommender Systems: Where Machine Learning m...
RecSys 2015 Tutorial - Scalable Recommender Systems: Where Machine Learning m...Joaquin Delgado PhD.
This tutorial gives an overview of how search engines and machine learning techniques can be tightly coupled to address the need for building scalable recommender or other prediction based systems. Typically, most of them architect retrieval and prediction in two phases. In Phase I, a search engine returns the top-k results based on constraints expressed as a query. In Phase II, the top-k results are re-ranked in another system according to an optimization function that uses a supervised trained model. However this approach presents several issues, such as the possibility of returning sub-optimal results due to the top-k limits during query, as well as the prescence of some inefficiencies in the system due to the decoupling of retrieval and ranking.
To address this issue the authors created ML-Scoring, an open source framework that tightly integrates machine learning models into Elasticsearch, a popular search engine. ML-Scoring replaces the default information retrieval ranking function with a custom supervised model that is trained through Spark, Weka, or R that is loaded as a plugin in Elasticsearch. This tutorial will not only review basic methods in information retrieval and machine learning, but it will also walk through practical examples from loading a dataset into Elasticsearch to training a model in Spark, Weka, or R, to creating the ML-Scoring plugin for Elasticsearch. No prior experience is required in any system listed (Elasticsearch, Spark, Weka, R), though some programming experience is recommended. RecSys 2015 Tutorial – Scalable Recommender Systems: Where Machine Learning...
Search engines have focused on solving the document retrieval problem, so their scoring functions do not handle naturally non-traditional IR data types, such as numerical or categorical. Therefore, on domains beyond traditional search, scores representing strengths of associations or matches may vary widely. As such, the original model doesn’t suffice, so relevance ranking is performed as a two-phase approach with 1) regular search 2) external model to re-rank the filtered items. Metrics such as click-through and conversion rates are associated with the users’ response to items served. The predicted selection rates that arise in real-time can be critical for optimal matching. For example, in recommender systems, predicted performance of a recommended item in a given context, also called response prediction, is often used in determining a set of recommendations to serve in relation to a given serving opportunity. Similar techniques are used in the advertising domain. To address this issue the authors have created ML-Scoring, an open source framework that tightly integrates machine learning models into a popular search engine (SOLR/Elasticsearch), replacing the default IR-based ranking function. A custom model is trained through either Weka or Spark and it is loaded as a plugin used at query time to compute custom scores.
Qualitative and quantitative analysis
Nellie Deutsch will be discussing Qualitative and Quantitative Analysis for Action Research in today's webinar July 30, 2015 at 12 PM EST on WizIQ: http://www.wiziq.com/online-class/2866384-ar-qualitative-and-quantitative-data-analysis Recordings will be available to those who join the class.
Hpd 1
The document discusses physical database design and tuning. It covers topics such as understanding the workload, index selection guidelines, clustering and indexing tools. Database tuning involves tuning indexes, tuning the conceptual schema such as normalization, and tuning queries. The goal is to refine the initial design based on the actual database usage to improve performance.
Lecture2 (1).ppt
Data mining is the process of extracting patterns from large data sets to identify useful information. It involves applying machine learning algorithms to detect patterns in sample data and then using the learned patterns to predict future behaviors or outcomes. Data mining utilizes techniques from machine learning, statistics, databases, and visualization to analyze large datasets and discover hidden patterns. The goal of data mining is to extract useful information from large datasets and transform it into an understandable structure for further use.
Similar to Strategies for SQL Server Index Analysis (20)
AnalytixLabs - Data Science 360 (Nasscom)-1648178720283 (1).pdf
AnalytixLabs - Data Science 360 (Nasscom)-1648178720283 (1).pdf
DATA SCIENCE AND BIG DATA ANALYTICSCHAPTER 2 DATA ANA.docx
DATA SCIENCE AND BIG DATA ANALYTICSCHAPTER 2 DATA ANA.docx
Geek Sync | Why Did My Clever Index Change Backfire?
Geek Sync | Why Did My Clever Index Change Backfire?
Using Metrics for Fun, Developing with the KV Store + Javascript & News from ...
Using Metrics for Fun, Developing with the KV Store + Javascript & News from ...
Get Testing with tSQLt - SQL In The City Workshop 2014
Get Testing with tSQLt - SQL In The City Workshop 2014
DTecH IT Education- Best Obiee training institute in bangalore
DTecH IT Education- Best Obiee training institute in bangalore
Optimising Queries - Series 1 Query Optimiser Architecture
Optimising Queries - Series 1 Query Optimiser Architecture
RecSys 2015 Tutorial - Scalable Recommender Systems: Where Machine Learning m...
RecSys 2015 Tutorial - Scalable Recommender Systems: Where Machine Learning m...
RecSys 2015 Tutorial – Scalable Recommender Systems: Where Machine Learning...
RecSys 2015 Tutorial – Scalable Recommender Systems: Where Machine Learning...
More from Jason Strate
Accelerating Business Intelligence Solutions with Microsoft Azure pass
Business Intelligence (BI) solutions need to move at the speed of business. Unfortunately, roadblocks related to availability of resources and deployment often present an issue. What if you could accelerate the deployment of an entire BI infrastructure to just a couple hours and start loading data into it by the end of the day. In this session, we'll demonstrate how to leverage Microsoft tools and the Azure cloud environment to build out a BI solution and begin providing analytics to your team with tools such as Power BI. By end of the session, you'll gain an understanding of the capabilities of Azure and how you can start building an end to end BI proof-of-concept today.
Leveraging Cloud for the Modern SQL Developer
In the past, developing a database was a simple affair where you built and deployed the database for a server down the hall in your datacenter, or in some cases to a server under someone's desk. While those good ole days were a much more simple time, they were quite limiting in how a database could be deployed and there were much fewer options for ensuring stability and performance. Today, SQL Developers have many more options for deployment including the services provided through Windows Azure. In this session, we'll dive into what you need to know to start developing databases with Windows Azure and discuss the value that cloud delivery can provide to your database development.
http://pragmaticworks.com/LearningCenter/FreeTrainingWebinars/PastWebinars.aspx?ResourceId=640
Getting Started with Windows Azure and SQL Databases
The Microsoft Windows Azure platform provides service-based offerings for your data needs. These include services from file storage to SQL databases that are scalable to your specific needs. Regardless of the size of your data platform, Windows Azure has options that fit to your business needs. In this session, we'll discuss the Windows Azure platform and discuss how the SQL Database and other services enhance and extend your data platform. By session end, you'll understand how Windows Azure is able to fit into your environment and be ready to start using Windows Azure.
http://pragmaticworks.com/LearningCenter/FreeTrainingWebinars/PastWebinars.aspx?ResourceId=639
5 SQL Server Indexing Myths
There are many "best practices" around that help people decide how to index databases. Having these practices can help alleviate the time that it takes to design an indexing strategy for a database. These practices can be of great use, except when they are wrong. Join us in this session, as we discuss some common myths associated with indexes and then dive into the myths to demonstrate how they can be debunked. At the end of the session, you'll know a few more things about indexes and leave armed with scripts that can help you debunk these myths on your own.
http://pragmaticworks.com/LearningCenter/FreeTrainingWebinars/PastWebinars.aspx?ResourceId=654
Introduction to Clustered Indexes and Heaps
It's the age old question, do I put a clustered index on the table or leave it as a heap? While a simple question, this can have serious impacts on the performance of the table in queries. In this session, we'll review the basics of both clustered indexes and heaps; identifying key differences and cases where you might choose one over another.
The Flavors of Non-Clustered Indexes
A non-clustered index is just another way to get to data. Add in a couple columns and off you go. Not too much to think about, just fire and forget. But it’s not really that easy. Non-clustered indexes have a number of various ways in which they can be built and designed that can have significant impacts on performance. In this session we’ll look at what can go into a non-clustered index. We will explore what you can put into a non-clustered index and what you can expect back out from performance.
Necessary Evils, Building Optimized CRUD Procedures
Every developer loves them and a lot of DBAs hate them. But there are many and valid reasons for creating generic SELECT, INSERT, UPDATE, and DELETE procedures. In this session, we’ll go through designing CRUD procedures that utilize new and existing SQL features to create CRUD procedures that are optimized for performance.
How Do Non-Clustered Indexes Improve Performance?
We've all heard the mantra about the importance of clustered indexes, but what about non-clustered indexes? Do you really need non-clustered indexes or are they just something that consultants recommend to increase their billable hours? In this session, we'll investigate the various flavors of non-clustered indexes and how they can be used to provide significant performance improvements.
Choosing Your Clustered Index
We often hear that you should always have clustered indexes on your tables. What's left off of this discussion is the how to choose the columns for this index. Should it be the primary key? Or some other column or set of columns? Having the wrong key column(s) in the clustered index can lead to performance problems, possibly worse than having no clustered index on the table at all. In this session, we'll review common patterns for selecting clustered indexes and how to determine which pattern you need. Also, we'll look at how to analyze tables to help select the best clustered index for every table.
http://www.jasonstrate.com/go/20131010-IX/
BuildingSecurity Audits with Extended Events
The document discusses using Extended Events in SQL Server to perform security audits. It describes the components of Extended Events including events, actions, predicates, targets, and packages. It then provides examples of how to build audits to track logins and queries, demonstrating how to capture useful security information without invasive monitoring. Templates are shown for login and query audits that can be directed to file, ring buffer, or event stream targets.
5 Amazing Reasons DBAs Need to Love Extended Events
Extended events provide DBAs with a powerful tool that can be used to troubleshoot and investigate SQL Server. Throughout this session, you’ll walk through five great reasons, with demos. By the end of the webcast, you’ll be itching to grab the scripts from the demos to start building your own extended event sessions today.
The Side Effect of NOLOCK
This document discusses the side effects of using NOLOCK hints in SQL queries. It notes that NOLOCK is not an acceptable way to fix locking issues and risks corrupting data. The document recommends researching SNAPSHOT ISOLATION as an alternative and provides contact information for Jason Strate, a database architect, who discusses locking and isolation levels on his website.
Discovering the plan cache (sql sat175)
This document discusses exploring the SQL Server plan cache. It begins with an overview of the plan cache and how it stores execution plans. It then covers various methods for viewing the plan cache, including using dynamic management objects. The document demonstrates how to interpret plan cache contents, including the showplan XML structure. It shows scenarios and demos related to missing indexes, parameter sniffing, physical operators, and key lookups. The document emphasizes that the plan cache contains valuable performance information and techniques for accessing it.
Introduction to SQL Server Security
This document discusses securing Microsoft SQL Server. It covers securing the SQL Server installation, controlling access to the server and databases, and validating security. Key points include using least privilege for service accounts, controlling access through logins, roles and permissions, auditing with SQL Server Audit and Policy Based Management, and services available from Pragmatic Works related to SQL Server security, training and products.
What are you waiting for? (#SQLSat211)
The document discusses wait statistics and wait types in SQL Server. It provides an overview of defining, viewing, collecting and reviewing wait data using dynamic management views and wait statistics. It also simplifies the SQL Server execution model using a grocery store checkout line analogy. Specific wait types are explained such as CXPACKET, OLEDB, PAGEIOLATCH_X, PAGELATCH_X and LATCH_X. Guidelines and potential resolutions are provided for analyzing and addressing top wait events.
Discovering the plan cache (#SQLSat211)
The document discusses discovering and analyzing the plan cache in SQL Server. It covers viewing the plan cache, the structure of showplan XML, and demonstrates scenarios and techniques for querying the plan cache including identifying missing indexes, parameter sniffing issues, and physical operator details. The goal is to help attendees leverage the plan cache to troubleshoot performance issues and tune queries.
Discovering the Plan Cache (#SQLSat 206)
The document discusses viewing and analyzing the plan cache in SQL Server. It covers viewing the plan cache, exploring the structure of showplan XML, and demonstrating methods to query the plan cache through demos. The goals are to discuss the plan cache, explore it, demonstrate querying methods, and demonstrate performance tuning concepts.
A Function by Any Other Name is a Function
This document discusses user defined functions in SQL Server and their performance impacts. It identifies purposes for creating user defined functions, discusses the different types including scalar, inline, and multi-statement functions. It demonstrates the performance impact of each type of function through examples and benchmarks. Scalar functions have linear performance effects while inline and multi-statement functions allow for more complex logic but can introduce duplication of access. In summary, functions provide benefits like consolidation and reuse but their performance impact depends on the type and how they are used.
What are you waiting for
This document discusses wait statistics and wait analysis in SQL Server. It defines waits as occurring when a process requests a resource that is unavailable. It provides an overview of wait types categorized as background, external, or queue waits. The document demonstrates how to view wait statistics using DMVs and third party tools. It provides translations and guidelines for some common wait types like CXPACKET, PAGEIOLATCH, and LATCH. It concludes with additional resources for learning about waits.
More from Jason Strate (19)
Accelerating Business Intelligence Solutions with Microsoft Azure pass
Accelerating Business Intelligence Solutions with Microsoft Azure pass
Getting Started with Windows Azure and SQL Databases
Getting Started with Windows Azure and SQL Databases
Necessary Evils, Building Optimized CRUD Procedures
Necessary Evils, Building Optimized CRUD Procedures
5 Amazing Reasons DBAs Need to Love Extended Events
5 Amazing Reasons DBAs Need to Love Extended Events
Recently uploaded
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
Full-RAG: A modern architecture for hyper-personalization
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
20 Comprehensive Checklist of Designing and Developing a Website
Dive into the world of Website Designing and Developing with Pixlogix! Looking to create a stunning online presence? Look no further! Our comprehensive checklist covers everything you need to know to craft a website that stands out. From user-friendly design to seamless functionality, we've got you covered. Don't miss out on this invaluable resource! Check out our checklist now at Pixlogix and start your journey towards a captivating online presence today.
A tale of scale & speed: How the US Navy is enabling software delivery from l...
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Essentials of Automations: The Art of Triggers and Actions in FME
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Building RAG with self-deployed Milvus vector database and Snowpark Container...
This talk will give hands-on advice on building RAG applications with an open-source Milvus database deployed as a docker container. We will also introduce the integration of Milvus with Snowpark Container Services.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Securing your Kubernetes cluster_ a step-by-step guide to success !
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Pushing the limits of ePRTC: 100ns holdover for 100 days
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
Recently uploaded (20)
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Strategies for SQL Server Index Analysis
- 2. INTRODUCTION Jason Strate e: jstrate@pragmaticworks.com e: jasonstrate@gmail.com b: www.jasonstrate.com t: StrateSQL Resources jasonstrate.com/go/indexing
- 5. AGENDA Introduction Catalog Views Dynamic Management Objects Index Analysis Analysis Case Studies
- 7. GOALS Provide a basis for tools in index analysis Introduce index analysis stored procedure Demonstrate index analysis case studies
- 8. CATALOG VIEWS Introduction Catalog Views Dynamic Management Objects Index Analysis Analysis Case Studies
- 9. INDEX METADATA • Methodology: • Heaps or Clustered? • Identical Indexes • Overlapping Indexes • Objects • sys.indexes • sys.index_columns • sys.dm_db_partition_stats
- 10. FOREIGN KEYS? Methodology • Index your foreign key columns. Objects: • sys.foreign_keys • sys.foreign_key_columns Primarily an OLTP best practice
- 12. DYNAMIC MANAGEMENT OBJECTS Introduction Catalog Views Dynamic Management Objects Index Analysis Analysis Case Studies
- 13. CAPTAIN OBVIOUS Methodology • Keep the indexes that you use Objects • sys.dm_db_index_usage_stats • sys.dm_db_index_operational_stats
- 14. CAPTAIN OBVIOUS, AGAIN Methodology • Build indexes that your queries will find useful Objects • sys.dm_db_missing_index_details • sys.dm_db_missing_index_groups • sys.dm_db_missing_index_group_stats
- 15. IS IT MONEY? Methodology • Is it worth the maintenance? Objects • sys.dm_db_index_operational_stats
- 16. DID I JUST SAY MAINTENANCE? Methodology • Tuning is better than troubleshooting Object • sys.dm_db_index_physical_stats
- 18. INDEX ANALYSIS Introduction Catalog Views Dynamic Management Objects Index Analysis Analysis Case Studies
- 19. INDEX ANALYSIS • Stored procedure • SQL Server 2005 – 2014 • SQL Database • Compiles Catalog/DMO in single view • Filter by tables • Provides single look at indexes • Missing indexes • Missing foreign key indexes • Multiple outputs • Detailed • Realign • Duplicate • Overlapping
- 21. ANALYSIS CASE STUDIES Introduction Catalog Views Dynamic Management Objects Index Analysis Analysis Case Studies
- 22. ALIGNMENT ISSUES • Excessive key lookups • Exceed number of seeks • Problems • Increased CPU • Increased IO • Increased duration
- 23. ALIGNMENT ISSUES • Ignore the problem • Choose high key lookup rate • Move clustered index • Maybe the key is wrong • Add included columns • Improve NC indexes
- 24. CASE STUDY 1 • What’s wrong? • Do you see the pattern? • What do you do? • Move clustered index
- 25. CASE STUDY 2 • What’s wrong? • Do you see the pattern? • What do you do? • Move clustered index • Add included columns
- 26. CASE STUDY 3 • What’s wrong? • Do you see the pattern? • What do you do? • Add included columns
- 28. SCAN HEAVY INDEXES • High scans on large indexes • Exceed seeks • Problems • Increased IO • Increased locking • Potential deadlocks • Affect page life expectancy • Impact memory distribution
- 29. CASE STUDY 1 • Metric is high, so what? • How do you tie into performance counter? • Sys.dm_db_index_ usage_stats • Plan cache Access Methods:Full Scans/sec
- 30. CASE STUDY 2 • What’s wrong? • Why aren’t the other indexes used? • Are the right indexes in place? • What do you do? • Examine execution plans • Add in correct indexes
- 32. MISSING INDEXES • Optimize through materialization • Index versus statistics • Three options • Dynamic management views • Database Engine Tuning Advisor • Plan Cache
- 35. INDEXING FOREIGN KEYS • Foreign keys enforcement • INSERT • UPDATE • DELETE • Performed in background • Unseen until a problem
- 36. INDEXING FOREIGN KEY ISSUES • Unindexed foreign key • Constraint columns • Performance issues • Scan of column values • Increased locking • Blocking likely • Potential deadlocking
- 37. CASE STUDY Delete from SalesOrderHeader?
- 38. CASE STUDY Delete from SalesOrderDetail?
- 41. THANK YOU Jason Strate e: jstrate@pragmaticworks.com e: jasonstrate@gmail.com b: www.jasonstrate.com t: StrateSQL Resources jasonstrate.com/go/indexing