This document discusses principles for designing systems and databases for high performance. It emphasizes that:
1. Performance requirements should be specified upfront and the architecture should support meeting those requirements.
2. The database design impacts performance, so logical and physical designs like normalization, indexing, partitioning, and caching should be considered.
3. Application architecture also affects performance through techniques like minimizing network traffic and SQL parsing overhead.
Stormwater Management Using State Revolving FundsOHM Advisors
http://www.ohm-advisors.com. A look at stormwater management projects, including stormwater best management practices (BMPs) funded using State Revolving Fund program. Michigan's State Revolving Fund (SRF) is a funding source dedicated to improving water quality. SRF was created under the Federal Clean Water Act. Section 319 authorizes SRF loans to implement non-point source pollution projects.
Regulatory, Technical and Modeling Challenges to Developing a Frequency Based...OHM Advisors
Wayne County’s North Huron Valley / Rouge Valley (NHVRV) interceptor system collects
sewage from 15 communities located in Southeast Michigan and transports flows to the Detroit Water and Sewerage Department (DWSD) for treatment and discharge. The County is evaluating a regional approach to controlling wet weather sanitary sewer overflows (SSOs). A new methodology called the i3D antecedent moisture (AM) model, was used to perform the hydrologic modeling. The i3D model is a continuous model that produces a good match to observed flow data over time. The accuracy of the model resulted in a high level of confidence in the frequency analysis for SSOs and will serve as the basis for recommending improvements to control wet weather SSOs. The use of the AM model combined with a frequency analysis for sizing improvements eliminated the need to select a design storm event based on “average” conditions. This reduced many of the conservatisms that are frequently included in event models such as the capture coefficient and seasonal effects. The use of spatially varied rainfall also improved the accuracy of the analysis over the use of a point rain gauge. This paper presents the modeling and analysis innovations used and the preliminary development of a regional project.
On March 11, 2015, Gary Sebach, OHM Advisors' Director of Architectural Services and Marysville ECHS Principal Kathy McKinnis, presented a landmark STEM education project at the Buckeye Association of School Administrators (BASA) conference: “STEM in Motion” Ohio’s first grant-funded STEM Early College High School.
Internal Search - The Lost Child of Web AnalyticsCharles Meaden
Discover why you should take the time to analyse the words and phrases that people enter into the search engine on your web site. These words show the users intent and allow you to better understand their thinking.
Voted 'Best Presentation' at the 2012 MeasureCamp in London
Stormwater Utilities: A regional and national perspective on planning and imp...OHM Advisors
As part of an umbrella discussion about funding options to improve Michigan’s aging sanitary and stormwater infrastructure, OHM Advisor's Greg Kacvinsky provides an overview of stormwater utility fees. His presentation covers how they initially came into practice and have been used nationally, typical revenues and revenue uses, and planning and implementation tips and strategies.
Building a Community Around your Blog 2 - Let the Comments be your Content!Brendan Sera-Shriar
This is number 2 in a series of presentations that discuss the hows and whys of building a community around your blog.
Let the comments be your content!
Supplemental Info: Say on Pay and Dodd Frank 20100723PERFORMENSATION
This presentation answers questions asked during the live event July 23, 2010. It also provides additional evidence regarding Pay and Performance in the United Kingdom.
Stormwater Management Using State Revolving FundsOHM Advisors
http://www.ohm-advisors.com. A look at stormwater management projects, including stormwater best management practices (BMPs) funded using State Revolving Fund program. Michigan's State Revolving Fund (SRF) is a funding source dedicated to improving water quality. SRF was created under the Federal Clean Water Act. Section 319 authorizes SRF loans to implement non-point source pollution projects.
Regulatory, Technical and Modeling Challenges to Developing a Frequency Based...OHM Advisors
Wayne County’s North Huron Valley / Rouge Valley (NHVRV) interceptor system collects
sewage from 15 communities located in Southeast Michigan and transports flows to the Detroit Water and Sewerage Department (DWSD) for treatment and discharge. The County is evaluating a regional approach to controlling wet weather sanitary sewer overflows (SSOs). A new methodology called the i3D antecedent moisture (AM) model, was used to perform the hydrologic modeling. The i3D model is a continuous model that produces a good match to observed flow data over time. The accuracy of the model resulted in a high level of confidence in the frequency analysis for SSOs and will serve as the basis for recommending improvements to control wet weather SSOs. The use of the AM model combined with a frequency analysis for sizing improvements eliminated the need to select a design storm event based on “average” conditions. This reduced many of the conservatisms that are frequently included in event models such as the capture coefficient and seasonal effects. The use of spatially varied rainfall also improved the accuracy of the analysis over the use of a point rain gauge. This paper presents the modeling and analysis innovations used and the preliminary development of a regional project.
On March 11, 2015, Gary Sebach, OHM Advisors' Director of Architectural Services and Marysville ECHS Principal Kathy McKinnis, presented a landmark STEM education project at the Buckeye Association of School Administrators (BASA) conference: “STEM in Motion” Ohio’s first grant-funded STEM Early College High School.
Internal Search - The Lost Child of Web AnalyticsCharles Meaden
Discover why you should take the time to analyse the words and phrases that people enter into the search engine on your web site. These words show the users intent and allow you to better understand their thinking.
Voted 'Best Presentation' at the 2012 MeasureCamp in London
Stormwater Utilities: A regional and national perspective on planning and imp...OHM Advisors
As part of an umbrella discussion about funding options to improve Michigan’s aging sanitary and stormwater infrastructure, OHM Advisor's Greg Kacvinsky provides an overview of stormwater utility fees. His presentation covers how they initially came into practice and have been used nationally, typical revenues and revenue uses, and planning and implementation tips and strategies.
Building a Community Around your Blog 2 - Let the Comments be your Content!Brendan Sera-Shriar
This is number 2 in a series of presentations that discuss the hows and whys of building a community around your blog.
Let the comments be your content!
Supplemental Info: Say on Pay and Dodd Frank 20100723PERFORMENSATION
This presentation answers questions asked during the live event July 23, 2010. It also provides additional evidence regarding Pay and Performance in the United Kingdom.
No more Three Tier - A path to a better code for Cloud and AzureMarco Parenzan
Three Tier is no more a model for Cloud and in general Scalable Web Applications. Scaffolding from SQL Database is, sure, the worst way to do. A road to do this is changing modeling activity, from scaffolded DB model to Command/DTO model, through ViewModel pattern. Command and DTOs are pillars of CQRS model.
What Your Database Query is Really DoingDave Stokes
Do you ever wonder what your database servers is REALLY doing with that query you just wrote. This is a high level overview of the process of running a query
Machine Learning with ML.NET and Azure - Andy CrossAndrew Flatters
ML.NET is an open source, machine learning framework built in .NET and runs on Windows, Linux and macOS. It allows developers to integrate custom machine learning into their applications without any prior expertise in developing or tuning machine learning models. Enhance your .NET apps with sentiment analysis, price prediction, fraud detection and more using custom models built with ML.NET
About Andy Cross
Andy Cross (@andyelastacloud) is a co-founder of Elastacloud, an Azure Insider, co-founder of the UK London Azure User Group, an Azure MVP and a Microsoft Regional Director. An international speaker, Andy has lead teams building the largest Hadoop and HDInsight specialist deployments on Azure.
His passion for embedded software and high performance compute clusters gives him a unique insight into a sphere of computation from the very small and resource constrained to the massively scalable, limitless potential of the cloud.
Hekaton is the original project name for In-Memory OLTP and just sounds cooler for a title name. Keeping up the tradition of deep technical “Inside” sessions at PASS, this half-day talk will take you behind the scenes and under the covers on how the In-Memory OLTP functionality works with SQL Server.
We will cover “everything Hekaton”, including how it is integrated with the SQL Server Engine Architecture. We will explore how data is stored in memory and on disk, how I/O works, how native complied procedures are built and executed. We will also look at how Hekaton integrates with the rest of the engine, including Backup, Restore, Recovery, High-Availability, Transaction Logging, and Troubleshooting.
Demos are a must for a half-day session like this and what would an inside session be if we didn’t bring out the Windows Debugger. As with previous “Inside…” talks I’ve presented at PASS, this session is level 500 and not for the faint of heart. So read through the docs on In-Memory OLTP and bring some extra pain reliever as we move fast and go deep.
This session will appear as two sessions in the program guide but is not a Part I and II. It is one complete session with a small break so you should plan to attend it all to get the maximum benefit.
Defense Against the Dark Arts: Protecting Your Data from ORMsVanessa Hurst
Object-relational mappers, or ORMs, enable rapid software development by allowing application developers to treat database entities similar to objects within an application. This can increase productivity drastically, but has unfortunate implications for DBAs and anyone who actually looks at data created by the application. This talk will help DBAs, Developers-turned-DBAs, and anyone in between understand how to leverage and limit, as necessary, ORMs. The talk covers what types of data ORMs are really great for, and how to look for and understand the nuances that may impact performance or compromise data integrity in an application.
Hadoop clusters can store nearly everything in a cheap and blazingly fast way to your data lake. Answering questions and gaining insights out of this ever growing stream becomes the decisive part for many businesses. Increasingly data has a natural structure as a graph, with vertices linked by edges, and many questions arising about the data involve graph traversals or other complex queries, for which one does not have an a priori given bound on the length of paths.
Whether you're a MongoDB professional or totally new to document databases, our MongoDB performance success factors & evaluation framework has something for you,
Curious about MongoDB performance?
Mydbops CTO, Manosh Malai illustrates the secret sauce for MongoDB performance best practices & analysis tool.
Cloud Architecture Patterns for Mere Mortals - Bill Wilder - Vermont Code Cam...Bill Wilder
How do you design applications for the cloud so that they will be scalable and reliable? In this talk, we will explain several architectural patterns which are popular for cloud computing: we will look at the need for the patterns generally, then look concretely at how you might realize them using capabilities of the Windows Azure Platform. CQRS, NoSQL, Sharding, and a few smaller patterns will be considered.
Presented by Bill Wilder at Vermont Code Camp III on Saturday September 10, 2011. http://blog.codingoutloud.com/2011/09/12/vermont-code-camp-iii/
Optimizing Oracle databases with SSD - April 2014Guy Harrison
Presentation on using Solid State Disk (SSD) with Oracle databases, including the 11GR2 db flash cache and using flash in Exadata. Last given at Collaborate 2014 #clv14.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
5. Core message Design limits performance Architecture maps requirements to design Make sure performance requirements are specified Make sure architecture allows for performance Make sure performance requirements are realized
15. “Twitter is, fundamentally, a messaging system. Twitter was not architected as a messaging system, however. For expediency's sake, Twitter was built with technologies and practices that are more appropriate to a content management system.”
21. Other logical design thoughts Artificial keys Generally more efficient than long composite keys Null values Not a good idea if you intend to search for “unknown” or “incomplete” values Null should not mean something But beneficial as long as you don’t need to look for them. Data types Constraints on precision can sometimes reduce row lengths Variable length strings usually better Carefully consider CLOBs vs long VARCHARs
23. Indexing, clustering and weird table types Lots’ of options: B*-Tree index Bitmap index Hash cluster Index Cluster Nested table Index Organized Table Most often useful: B*-Tree (concatenated) indexes Bitmap indexes Hash Clusters
24.
25. Concatenated index effectiveness SELECT cust_id FROM sh.customers c WHERE cust_first_name = 'Connor' AND cust_last_name = 'Bishop' AND cust_year_of_birth = 1976;
26. Concatenated indexing guidleines Create a concatenated index for columns from a table that appear together in the WHERE clause. If columns sometimes appear on their own in a WHERE clause, place them at the start of the index. The more selective a column is, the more useful it will be at the leading end of the index (better single key lookups) But indexes compress better when the leading columns are less selective. (better scans) Index skip scans can make use of an index even if the leading columns are not specified, but it’s a poor second choice to a “normal” index range scan.
30. Bitmap join performance SELECT SUM (amount_sold) FROM customers JOIN sales s USING (cust_id) WHERE cust_email='flint.jeffreys@company2.com';
36. Summary tables Aggregate queries on big tables often the most expensive Pre-computing them makes a lot of sense Balance accuracy with overhead Aggregate Query MV on COMMIT Manual Summary Result set cache MV stale tolerated Accuracy Efficiency
41. The best SQL is no SQL Avoid asking for the same data twice.
42. 11g client side cache CLIENT_RESULT_CACHE_SIZE: this is the amount of memory each client program will dedicate to the cache. Use RESULT_CACHE hint or (11GR2) table property Optionally set the CLIENT_RESULT_CACHE_LAG
43. Parse overhead It’s easy enough in most programming languages to create a unique SQL for every query:
57. Geek quiz stuff: High probability answers (keep standing if): Know what Alice and Wally have in common You know the next number in this series 3 . 1 4 Know what “M” is in E=MC2
58. Know (or can work out) your age in hex Have an opinion about of ST vs SW If you know who Leonard McCoy is Think there is an important distinction between Nerd and Geek Can quote Monty Python …. Other than dead parrot? You’ve ever watched Jerry Springer
59. There are more networked devices in your house than people, pets and cars Know the names of two of Thomas the tank engines friends Know the names of any of Angelina and Brad’s babies Low probability answers: (sit down if you): Have a twitter account # Azure is your new favourite color
60. You’ve ever played Zork You have a favourite Dr Who companion Your favourite is Sarah Jane Know your age in binary (or can work it out in your head) You are proficient in some form of assembler
61. # You are proficient in some for or English There is a rubicks cube in your house Have your own domain Have ever been to Azeroth Who is Know who said “Dude I am not your nemesis”
62. Worn a star trek or star wars costume Played a game that uses a non-six sided dice Get email on my phone – before getting out of bed Calculator watch Binary time piece Was on the internet prior to the WWW
63. # Met my current partner on line Know the next thing in this sequence: Hydrogen, Helium, Lithuim, Berilium, …. Know what a Gigaquads in a megaquad is
64. Saw a sci-fi movie more than twice at the movies ========================================================= You cleaned up at home before going to work
Editor's Notes
TelsavsMasseratiLatency vs throughputVolume?Economics?
MethodsRequirementsMeasurementPrototypeBenchmarkData modelOptimize physical model to queryDenormalizeIndexing, clustering, partitionApplicationMinimize database accessOptimize database access
http://www.steveschmidtracing.com/our-team-5.html
Generally not easy to change architectures.....
Normal form is the best starting point for an efficient database design. However, don’t go overboard in eliminating redundancy. A normalized data model is one in which any data redundancy has been eliminatedand in which data and relationships are uniquely identifiable by primaryand foreign keys. Although the normalized data model is rarely the final destinationfrom a performance point of view, the normalized data model is almost alwaysthe best starting point. Indeed, failing to normalize your logical model isfrequently a recipe for a poorly performing physical model.Relational theory provides for various levels of normal form, some of whichare of academic interest only. Third normal form is the most commonly adoptednormalization level, and it has the following characteristics:❏ All data in an entity (table) is dependent on the primary key.❏ There should be no repeating groups of attributes (columns).❏ No data in an entity is dependent on only part of the key.❏ No data in an entity is dependent on any nonkey attribute.These characteristics are often remembered through the adage “the key, thewhole key, and nothing but the key.”
Subtypes categorize or partition a logical entity and help to classify thetypes of information that is within the entity. A subtype usually has a set of attributesthat are held in common with the parent entity (the super-type) andother attributes that are not shared with the super-type or other subtypes. Figure4-1 shows how a PEOPLE entity could be split into subtypes of CUSTOMERSand EMPLOYEES.When translating entity subtypes into tables, we have the following options:❏ Create tables for the super-type and for each subtype. The super-type tablecontains only columns that are common to both subtypes.❏ Create a table for the super-type only. Attributes from all subtypes becomecolumns in this super-table. Typically, columns from subtype attributes willbe nullable, and a category column indicates the subtype in which a row belongs.❏ Create separate tables for each subtype without creating a table for thesuper-type. Attributes from the super-type are duplicated in each table.The three solutions result in very different performance outcomes. In particular,creating tables for the super-type and each subtype is likely to reduce performancein most circumstances, except where only the super-type is subject to afull table scan. Table 4-1 compares the performance of each of the three solutionsfor common database operations.
Remember, you don’t always have control over the network – in particular client side code may sometimes be located anywhere
The further the code is (in network terms) from the database, the more the network effects will magnify. You can’t get any closer to the database than being inde the database as PL/SQL