Become a Big Data Quality Hero
•
1 like•327 views
Many believe that regression testing an application with minimal data is sufficient. With big data applications, the data testing methodology becomes far more complex. Testing can now be done within the data fabrication process as well as in the data delivery process. Today, comprehensive testing is often mandated by regulatory agencies—and more importantly by customers. Finding issues before deployment and saving your company’s reputation—and in some cases preventing litigation—are critical. Jason Rauen presents an overview of the architecture, processes, techniques, and lessons learned by an original big data company. Detecting defects up-front is vital. Learn how to test thousands, millions, and in some cases billions—yes, billions—of records directly, rendering sampling procedures obsolete. Save time and money for your organization with better data test coverage than ever before.
Report
Share
Report
Share
Download to read offline
Recommended
Data Visualization Workflow
This document outlines the workflow for preparing data visualizations, including getting the data, exploring and cleansing it, identifying key variables, testing for usability, building the data, and creating models and visualizations. It discusses downloading data from various sources, examining a sample of rows to understand the data, cleaning invalid values, determining useful variables, ensuring data quality, and developing visualizations like scatter plots, linear regressions, density plots, bar charts, and heat maps to analyze the data.
Making Open the Default
Keynote address at OpenCon 2015 in Brussels, Belgium. Video here:
https://www.youtube.com/watch?v=3rK-tt3UlYs
Making Open the Default - Bjorn Brembs
The document discusses issues with the current scholarly publishing system, including limited access, lack of impact analysis and functional search capabilities, poor peer review, and lack of archiving for publications, software, and data. It notes there are over 575 proposed solutions but the system remains dysfunctional. Issues highlighted include the weakening relationship between journal impact factors and actual citation rates, bias in genetic association studies toward higher impact journals, and that retracted studies are disproportionately published in high impact journals. The document argues for publishing in journals that can be widely read instead of solely high impact journals, and for making science open by default to save time and money while benefiting more people.
The Mindset Change for the Agile Tester
On traditional projects, testers usually join the project after coding has started, or even later when coding is almost finished. Testers have no role in advising the project team early regarding quality issues but focus only on finding defects. They become accustomed to this style of working and adjust their mental processes accordingly. In agile, testers must collaborate closely with customers and programmers throughout the development lifecycle, where their focus changes from finding defects to preventing them. Janet Gregory shares ways to change the tester’s mindset from “How can I break the software?” to “How can I help deliver excellent software?”—a critical mental shift on agile projects. Another facet of the mind-set change is learning how to test early and incrementally. Janet uses interactive exercises and examples to help you understand how effective this mindset change is—and how you can apply it on your agile projects.
Leveraging Core Values for Healthier, More Productive Teams
Although all teams require a healthy level of interaction, high-performing teams' interactions are all based on trust, respect, and shared goals. Such teams find ways to overcome the fear of conflict, and quickly identify and resolve issues that are getting in the way. Scott Ross shares how, when the Omnyx software R&D department determined their culture was hindering performance, they crafted a core values statement that has served them well for the past three years. Scott describes the ways they proactively and intentionally use their value statement to drive the culture they seek and discusses the results they have achieved. Take back the list of resources that Scott uses daily to help himself and others see how their actions add to and take away from their core values. Return to the office prepared to use this same process with your team and start on the road to a high-performing team whose members love to come to work every day.
ADC-BSC EAST 2013 Keynote: Worse Is Better—For Better or for Worse
More than two decades ago, Richard P. Gabriel proposed the idea that “Worse Is Better” to explain why some things that are designed to be pure and perfect are eclipsed by solutions that are compromised and imperfect. This is not simply an observation that things should be better but are not, or that flawed and ill-considered solutions sometimes win out over those created with intention. Rather, many solutions that are narrow and incomplete work out better than those seen as comprehensive and complete. Whether programming languages, operating systems, or development practices, we find many examples in software development, some more provocative and surprising than others—Unix, Linux, C, COM, the web, the Wiki, incremental development, and TDD, to name but a few. The Worse Is Better philosophy pre-dates and anticipates agile development, but some of its finer points are still overlooked by agile and fragile projects everywhere. As Kevlin Henney revisits the original premise and question, he shares examples that can teach us new and surprising things.
Testing Challenges within Agile Teams
In her book Agile Testing: A Practical Guide for Testers and Agile Teams, Janet Gregory recommends using the automation pyramid as a model for test coverage. In the pyramid model, most automated tests are unit tests written and maintained by the programmers,and tests that execute below the user interface—API-level tests that can be developed and maintained collaboratively by programmers and testers. However, as agile becomes mainstream, some circumstances may challenge this model. Many applications are transferring logic back to the client side by using programming languages such as JavaScript. Legacy systems, using languages such as COBOL, don’t have access to unit testing frameworks. Janet shows how to adapt the model to your needs and addresses some of these automation issues. During the session, delegates are encouraged to share their challenges and success stories.
Test Automation on Large Agile Projects: It's Not a Cakewalk
1) The document discusses test automation on large agile projects. It describes how test automation grew and responsibilities were divided across multiple teams over 2.5 years on one project.
2) Key reasons for test automation included keeping development on track, capturing knowledge, and automating repetitive work. Issues that arose included accumulating test debt and tools contributing to segmented responsibilities.
3) The presentation provides lessons learned around defining "Done", managing test debt, and organizing testing responsibilities across teams as projects increase in size and complexity. Setting expectations for test code quality and having integrated discussions on story size were emphasized.
Recommended
Data Visualization Workflow
This document outlines the workflow for preparing data visualizations, including getting the data, exploring and cleansing it, identifying key variables, testing for usability, building the data, and creating models and visualizations. It discusses downloading data from various sources, examining a sample of rows to understand the data, cleaning invalid values, determining useful variables, ensuring data quality, and developing visualizations like scatter plots, linear regressions, density plots, bar charts, and heat maps to analyze the data.
Making Open the Default
Keynote address at OpenCon 2015 in Brussels, Belgium. Video here:
https://www.youtube.com/watch?v=3rK-tt3UlYs
Making Open the Default - Bjorn Brembs
The document discusses issues with the current scholarly publishing system, including limited access, lack of impact analysis and functional search capabilities, poor peer review, and lack of archiving for publications, software, and data. It notes there are over 575 proposed solutions but the system remains dysfunctional. Issues highlighted include the weakening relationship between journal impact factors and actual citation rates, bias in genetic association studies toward higher impact journals, and that retracted studies are disproportionately published in high impact journals. The document argues for publishing in journals that can be widely read instead of solely high impact journals, and for making science open by default to save time and money while benefiting more people.
The Mindset Change for the Agile Tester
On traditional projects, testers usually join the project after coding has started, or even later when coding is almost finished. Testers have no role in advising the project team early regarding quality issues but focus only on finding defects. They become accustomed to this style of working and adjust their mental processes accordingly. In agile, testers must collaborate closely with customers and programmers throughout the development lifecycle, where their focus changes from finding defects to preventing them. Janet Gregory shares ways to change the tester’s mindset from “How can I break the software?” to “How can I help deliver excellent software?”—a critical mental shift on agile projects. Another facet of the mind-set change is learning how to test early and incrementally. Janet uses interactive exercises and examples to help you understand how effective this mindset change is—and how you can apply it on your agile projects.
Leveraging Core Values for Healthier, More Productive Teams
Although all teams require a healthy level of interaction, high-performing teams' interactions are all based on trust, respect, and shared goals. Such teams find ways to overcome the fear of conflict, and quickly identify and resolve issues that are getting in the way. Scott Ross shares how, when the Omnyx software R&D department determined their culture was hindering performance, they crafted a core values statement that has served them well for the past three years. Scott describes the ways they proactively and intentionally use their value statement to drive the culture they seek and discusses the results they have achieved. Take back the list of resources that Scott uses daily to help himself and others see how their actions add to and take away from their core values. Return to the office prepared to use this same process with your team and start on the road to a high-performing team whose members love to come to work every day.
ADC-BSC EAST 2013 Keynote: Worse Is Better—For Better or for Worse
More than two decades ago, Richard P. Gabriel proposed the idea that “Worse Is Better” to explain why some things that are designed to be pure and perfect are eclipsed by solutions that are compromised and imperfect. This is not simply an observation that things should be better but are not, or that flawed and ill-considered solutions sometimes win out over those created with intention. Rather, many solutions that are narrow and incomplete work out better than those seen as comprehensive and complete. Whether programming languages, operating systems, or development practices, we find many examples in software development, some more provocative and surprising than others—Unix, Linux, C, COM, the web, the Wiki, incremental development, and TDD, to name but a few. The Worse Is Better philosophy pre-dates and anticipates agile development, but some of its finer points are still overlooked by agile and fragile projects everywhere. As Kevlin Henney revisits the original premise and question, he shares examples that can teach us new and surprising things.
Testing Challenges within Agile Teams
In her book Agile Testing: A Practical Guide for Testers and Agile Teams, Janet Gregory recommends using the automation pyramid as a model for test coverage. In the pyramid model, most automated tests are unit tests written and maintained by the programmers,and tests that execute below the user interface—API-level tests that can be developed and maintained collaboratively by programmers and testers. However, as agile becomes mainstream, some circumstances may challenge this model. Many applications are transferring logic back to the client side by using programming languages such as JavaScript. Legacy systems, using languages such as COBOL, don’t have access to unit testing frameworks. Janet shows how to adapt the model to your needs and addresses some of these automation issues. During the session, delegates are encouraged to share their challenges and success stories.
Test Automation on Large Agile Projects: It's Not a Cakewalk
1) The document discusses test automation on large agile projects. It describes how test automation grew and responsibilities were divided across multiple teams over 2.5 years on one project.
2) Key reasons for test automation included keeping development on track, capturing knowledge, and automating repetitive work. Issues that arose included accumulating test debt and tools contributing to segmented responsibilities.
3) The presentation provides lessons learned around defining "Done", managing test debt, and organizing testing responsibilities across teams as projects increase in size and complexity. Setting expectations for test code quality and having integrated discussions on story size were emphasized.
Test Estimation for Managers
The document announces a presentation titled "Test Estimation for Managers" to be given by Julie Gardiner of The Test People. Julie Gardiner has over 20 years of experience in IT in roles including analyst programmer, Oracle DBA, project manager, test analyst, test team leader, test consultant, and test manager. She is now head of agile, talent and transition management with The Test People and specializes in risk-based testing, agile testing, test management, and people issues.
Yin and Yang: Metrics within Agile and Traditional Lifecycles
Metrics are powerful tools when used to effect positive change in a project or organization. However, the value and benefits of metrics are often dependent on the context. While certain metrics provide information and insight to drive decision making for a traditional development approach, they may not be useful in an agile landscape—and vice versa. QA and agile experts Shaun Bradshaw and Bob Galen delve into the value, pitfalls, pros, and cons of various metrics in agile and waterfall development environments. Hear their experiences as they discuss and explore a variety of project-level, software development, and software testing metrics through the lens of both traditional and agile development contexts. Although Bob and Shaun respect each other’s knowledge and skill, they don’t often agree on metrics. And in this showdown, you’ll see why! Be prepared to learn, be entertained, and be ready to get in on the action as these two metrics titans go head-to-head.
How to Survive the Coming Test Automation Zombie Apocalypse
Test automation is software development. To automate tests well, you have to have brains. Unfortunately, the very brains that make you good at your job also make you highly attractive to zombies. Like all zombies, test automation zombies are brainless, insatiable, and relentless. Unlike human zombies, test automation zombies can be difficult to recognize. They don’t look like people at all. Some look like org charts. Some look like best practices. Some you can’t see at all—bad habits that “seemed like a good idea at the time.” Dale Emery has survived numerous test automation zombie attacks with almost no permanent damage. Dale knows zombies, and teaches you to recognize them. Automating only at the end of a sprint? Zombie. Sprinkling fixed delays through your tests to accommodate unresponsive web pages? Zombie. Automating only by record and playback? Zombie. Make no mistake—the test automation zombies are coming. Learn how to spot them before they eat your brains.
Lean Management: Lessons from the Field
The document summarizes a presentation by Sanjiv Augustine on timeless lean management solutions. It discusses three key solutions: 1) Organizing around a network of small, self-organizing teams. 2) Driving lean innovation through techniques like lean startup methodology. 3) Practicing wise leadership through empowering teams, flattening hierarchies, and trusting the team. The presentation provides examples of how companies like Ericsson have successfully implemented these lean-agile principles at scale.
Agile Development in a Regulated Environment
There is no doubt that agile is an accepted development methodology. However, if you work in a regulated industry like health care where you have to comply with its standard operating procedures, heaps of paperwork, and frequent audits, don’t these conflict with agile’s core tenets? Chris Ampenberger describes his operating environment and the applicable regulations that define the constraints for the software development process he can use. He shares how they overcame the incongruity between agile and regulatory requirements. With real-world examples, Chris demonstrates how you can produce the required documentation as a byproduct of the scrum team’s everyday work and illustrates how his teams succeeded in an agile way, achieving significant increases in productivity. Chris points out common pitfalls, details the hurdles they had to overcome, and discusses how to obtain buy-in from stakeholders at all levels of the organization. If you are working in a regulated environment, this session is for you.
Keynote: Surviving or Thriving: Top Ten Lessons for the Professional Tester
As testers and test managers we often find ourselves struggling just to survive within our organization—sometimes with the possibility of job loss due to outsourcing looming. Often, we are told to become more “effective,” “efficient,” and do “more with less.” However, most testers and test managers are unsure of what those mandates actually mean. Today, it is not sufficient to just survive; we must take initiatives to thrive. Lloyd Roden shares ten valuable lessons on how you can become better at testing and thrive in your career. Lloyd's lessons include the importance of using modern technology in testing, using test design techniques when reviewing documentation, testing the testers with techniques such as bug seeding, reporting project waste, providing management with feedback on decisions that they made, becoming a pioneer or explorer rather than a settler or outlaw in your organization, and more. Lloyd’s advice is practical—and challenging—for all testers, test leads, and test managers.
An Introduction to SAFe: The Scaled Agile Framework
Many organizations have achieved agility at the team level only to be unable to achieve it across teams. The Scaled Agile Framework (SAFe) provides both a vision and method for how to achieve this. SAFe is the first documented framework that can be used to scale agile throughout an organization. It is a combination of lean, kanban, and Scrum—lean to provide a context for an organization, kanban to manage the flow of projects, and Scrum to provide agile at the team level. Beginning with an introduction to lean and kanban, Ken Pugh explains why they are required for agile at scale. Ken then describes the framework of SAFe—specifically how it creates a structure to manifest the behaviors required for agile at scale. In particular, learn how to coordinate your organization’s portfolio, programs, and projects. Ken concludes by discussing when it is advisable to use the framework and when a more emergent method is preferable.
Measurement and Metrics for Test Managers
To be most effective, test managers must develop and use metrics to help direct the testing effort and make informed recommendations about the software’s release readiness and associated risks. Because one important testing activity is to “measure” the quality of the software, test managers must measure the results of both the development and testing processes. Collecting, analyzing, and using metrics is complicated because many developers and testers are concerned that the metrics will be used against them. Join Rick Craig as he addresses common metrics—measures of product quality, defect removal efficiency, defect density, defect arrival rate, and testing status. Learn the guidelines for developing a test measurement program, rules of thumb for collecting data, and ways to avoid “metrics dysfunction.” Rick identifies several metrics paradigms and discusses the pros and cons of each. Delegates are urged to bring their metrics problems and issues for use as discussion points.
Testing After You’ve Finished Testing
Stakeholders always want to release when they think we’ve “finished testing”. They believe we have revealed “all of the important problems” and “verified all of the fixes,” and now it’s time to reap the rewards. However, as testers we still can assist in improving software by learning about problems after code has rolled “live-to-site”—especially if it’s a website. At eBay we have a post-ship “site quality” mindset in which testers continue to learn from A/B testing, operational issues, customer sentiment analysis, discussion forums, and customer call patterns—just to name a few. Jon Bach explains how and what eBay’s Live Site Quality team learns every day about what they just released to production. Take away some ideas on what you can do to test and improve value—even after you’ve shipped.
Patters for Team Collaboration: Toward Whole Team Quality
The half-day tutorial covered patterns for team collaboration toward whole-team quality, presented by Janet Gregory and Matt Barcomb. The presentation defined patterns as models or reliable traits and discussed various collaboration patterns including generalizing specialists, pairing, shared vision using tools like business model canvases and impact mapping, specification techniques like user story mapping and acceptance test-driven development, and paired development practices like continuous testing, test-driven development, and configuration management. The goal was to provide teams with collaborative techniques to improve quality.
Introducing Keyword-Driven Test Automation
The document describes a presentation on keyword-driven test automation given by Hans Buwalda of LogiGear Corporation. The presentation introduces keyword-driven testing and compares it to other automation techniques. It provides recommendations for successfully applying keyword-driven testing to test design, automation, and organization. The presentation also covers topics like data-driven testing, non-UI testing, and test automation best practices.
Query or Not to Query? Using Apache Spark Metrics to Highlight Potentially Pr...
John submits a query and expects it to run smoothly. Based on his prior experience, he anticipates the query to finish in 20 mins.
Scenario-1: John’s query finishes execution in the expected timeframe and doesn’t impact any other concurrent query in the workload.
Scenario-2: John’s query takes twice the expected time, and also slows down multiple other concurrent queries. John now wonders “should I have submitted this query?”.
Testing the Data Warehouse
The document discusses testing strategies for data warehouses. It recommends testing at the primary, secondary, and tertiary levels, with primary level testing focusing on unit testing individual transformations to verify data is mapped and loaded correctly. Common issues found in data warehouse testing are also outlined such as errors in data mapping specifications and load processes leading to invalid data. Planning tests to address typical problems and ensuring data quality is also emphasized.
Testing the Data Warehouse—Big Data, Big Problems
Data warehouses are critical systems for collecting, organizing, and making information readily available for strategic decision making. The ability to review historical trends and monitor near real-time operational data is a key competitive advantage for many organizations. Yet the methods for assuring the quality of these valuable assets are quite different from those of transactional systems. Ensuring that appropriate testing is performed is a major challenge for many enterprises. Geoff Horne has led numerous data warehouse testing projects in both the telecommunications and ERP sectors. Join Geoff as he shares his approaches and experiences, focusing on the key “uniques” of data warehouse testing: methods for assuring data completeness, monitoring data transformations, measuring quality, and more. Geoff explores the opportunities for test automation as part of the data warehouse process, describing how you can harness automation tools to streamline the work and minimize overhead.
Test Data, Information, Knowledge, Wisdom: past, present & future of standing...
The document discusses test data for software testing from both past and present perspectives. It begins by covering definitions and concepts related to test data from ISO 29119 standards. It then discusses the basics of test data and how test data requirements and information are developed through the test planning, design and implementation processes. The document also discusses different sources of test data and how test data relates to different testing levels and techniques like unit testing, integration testing and system testing.
Data Science Popup Austin: Back to The Future for Data and Analytics
JD Stanley, Global Director of Strategy and Integrated Solutions at Cisco's Data and Analytics Business Unit, gave a keynote on data and analytics going "Back to the Future." He discussed how most data will be processed at the edge by mobile devices and IoT solutions in the near future. Cisco is focusing on network analytics, edge computing, data relationships, and accelerating time to value through tools for data exploration, preparation, and prediction. The talk concluded that initial focus should be on data discovery rather than predictions, improving data quality, and unleashing teams of various roles to iteratively improve outcomes through experience rather than just technology.
How Celtra Optimizes its Advertising Platform with Databricks
Leading brands such as Pepsi and Macy’s use Celtra’s technology platform for brand advertising. To inform better product design and resolve issues faster, Celtra relies on Databricks to gather insights from large-scale, diverse, and complex raw event data. Learn how Celtra uses Databricks to simplify their Spark deployment, achieve faster project turnaround time, and empower people to make data-driven decisions.
In this webinar, you will learn how Databricks helps Celtra to:
- Utilize Apache Spark to power their production analytics pipeline.
- Build a “Just-in-Time” data warehouse to analyze diverse data sources such as Elastic Load Balancer access logs, raw tracking events, operational data, and reportable metrics.
- Go beyond simple counting and group events into sequences (i.e., sessionization) and perform more complex analysis such as funnel analytics.
Testing the Data Warehouse―Big Data, Big Problems
The document is a presentation on testing data warehouses. It discusses why data warehouses need to be tested, what aspects should be tested, and where and in what order testing should be conducted. It also covers typical issues encountered in data warehouse testing and strategies for planning testing of continually changing source systems and common data warehouse problems.
Frank Bien Opening Keynote - Join 2016
Frank Bien, CEO of Looker - along with Amazon, Google and other data disrupters - discuss how innovators are deeply integrating analytics into every aspect of their businesses, from mobile to warehouse to cloud.
Frank shares Looker’s vision for the future of business intelligence and data analytics and reveal pivotal product and partnership updates.
Frank Bien Opening Keynote - Join 2016
Frank Bien, CEO of Looker - along with Amazon, Google and other data disrupters - discuss how innovators are deeply integrating analytics into every aspect of their businesses, from mobile to warehouse to cloud.
Frank shares Looker’s vision for the future of business intelligence and data analytics and reveal pivotal product and partnership updates.
351315535-Module-1-Intro-to-Data-Science-pptx.pptx
This document provides an overview of the data science process, including historical notes on related frameworks like KDD, CRISP-DM, and big data. It discusses the typical stages in the knowledge discovery process, including business understanding, data understanding, data preparation, model building, evaluation, and deployment. It also provides an example walking through these stages to predict power failures in Manhattan, covering opportunities assessment, data acquisition and cleaning, model building, policy construction, and evaluation.
DATA @ NFLX (Tableau Conference 2014 Presentation)
I presented this at a 2014 Tableau Conference session with Albert Wong.
Netflix relies on data to make decisions ranging from buying and recommending content, to improving the streaming experience on devices.
This presentation shares our Big Data analytics architecture and the tools used to make data accessible throughout our business, focusing on how Tableau fits into our organization and why it aligns well with our culture.
More Related Content
Viewers also liked
Test Estimation for Managers
The document announces a presentation titled "Test Estimation for Managers" to be given by Julie Gardiner of The Test People. Julie Gardiner has over 20 years of experience in IT in roles including analyst programmer, Oracle DBA, project manager, test analyst, test team leader, test consultant, and test manager. She is now head of agile, talent and transition management with The Test People and specializes in risk-based testing, agile testing, test management, and people issues.
Yin and Yang: Metrics within Agile and Traditional Lifecycles
Metrics are powerful tools when used to effect positive change in a project or organization. However, the value and benefits of metrics are often dependent on the context. While certain metrics provide information and insight to drive decision making for a traditional development approach, they may not be useful in an agile landscape—and vice versa. QA and agile experts Shaun Bradshaw and Bob Galen delve into the value, pitfalls, pros, and cons of various metrics in agile and waterfall development environments. Hear their experiences as they discuss and explore a variety of project-level, software development, and software testing metrics through the lens of both traditional and agile development contexts. Although Bob and Shaun respect each other’s knowledge and skill, they don’t often agree on metrics. And in this showdown, you’ll see why! Be prepared to learn, be entertained, and be ready to get in on the action as these two metrics titans go head-to-head.
How to Survive the Coming Test Automation Zombie Apocalypse
Test automation is software development. To automate tests well, you have to have brains. Unfortunately, the very brains that make you good at your job also make you highly attractive to zombies. Like all zombies, test automation zombies are brainless, insatiable, and relentless. Unlike human zombies, test automation zombies can be difficult to recognize. They don’t look like people at all. Some look like org charts. Some look like best practices. Some you can’t see at all—bad habits that “seemed like a good idea at the time.” Dale Emery has survived numerous test automation zombie attacks with almost no permanent damage. Dale knows zombies, and teaches you to recognize them. Automating only at the end of a sprint? Zombie. Sprinkling fixed delays through your tests to accommodate unresponsive web pages? Zombie. Automating only by record and playback? Zombie. Make no mistake—the test automation zombies are coming. Learn how to spot them before they eat your brains.
Lean Management: Lessons from the Field
The document summarizes a presentation by Sanjiv Augustine on timeless lean management solutions. It discusses three key solutions: 1) Organizing around a network of small, self-organizing teams. 2) Driving lean innovation through techniques like lean startup methodology. 3) Practicing wise leadership through empowering teams, flattening hierarchies, and trusting the team. The presentation provides examples of how companies like Ericsson have successfully implemented these lean-agile principles at scale.
Agile Development in a Regulated Environment
There is no doubt that agile is an accepted development methodology. However, if you work in a regulated industry like health care where you have to comply with its standard operating procedures, heaps of paperwork, and frequent audits, don’t these conflict with agile’s core tenets? Chris Ampenberger describes his operating environment and the applicable regulations that define the constraints for the software development process he can use. He shares how they overcame the incongruity between agile and regulatory requirements. With real-world examples, Chris demonstrates how you can produce the required documentation as a byproduct of the scrum team’s everyday work and illustrates how his teams succeeded in an agile way, achieving significant increases in productivity. Chris points out common pitfalls, details the hurdles they had to overcome, and discusses how to obtain buy-in from stakeholders at all levels of the organization. If you are working in a regulated environment, this session is for you.
Keynote: Surviving or Thriving: Top Ten Lessons for the Professional Tester
As testers and test managers we often find ourselves struggling just to survive within our organization—sometimes with the possibility of job loss due to outsourcing looming. Often, we are told to become more “effective,” “efficient,” and do “more with less.” However, most testers and test managers are unsure of what those mandates actually mean. Today, it is not sufficient to just survive; we must take initiatives to thrive. Lloyd Roden shares ten valuable lessons on how you can become better at testing and thrive in your career. Lloyd's lessons include the importance of using modern technology in testing, using test design techniques when reviewing documentation, testing the testers with techniques such as bug seeding, reporting project waste, providing management with feedback on decisions that they made, becoming a pioneer or explorer rather than a settler or outlaw in your organization, and more. Lloyd’s advice is practical—and challenging—for all testers, test leads, and test managers.
An Introduction to SAFe: The Scaled Agile Framework
Many organizations have achieved agility at the team level only to be unable to achieve it across teams. The Scaled Agile Framework (SAFe) provides both a vision and method for how to achieve this. SAFe is the first documented framework that can be used to scale agile throughout an organization. It is a combination of lean, kanban, and Scrum—lean to provide a context for an organization, kanban to manage the flow of projects, and Scrum to provide agile at the team level. Beginning with an introduction to lean and kanban, Ken Pugh explains why they are required for agile at scale. Ken then describes the framework of SAFe—specifically how it creates a structure to manifest the behaviors required for agile at scale. In particular, learn how to coordinate your organization’s portfolio, programs, and projects. Ken concludes by discussing when it is advisable to use the framework and when a more emergent method is preferable.
Measurement and Metrics for Test Managers
To be most effective, test managers must develop and use metrics to help direct the testing effort and make informed recommendations about the software’s release readiness and associated risks. Because one important testing activity is to “measure” the quality of the software, test managers must measure the results of both the development and testing processes. Collecting, analyzing, and using metrics is complicated because many developers and testers are concerned that the metrics will be used against them. Join Rick Craig as he addresses common metrics—measures of product quality, defect removal efficiency, defect density, defect arrival rate, and testing status. Learn the guidelines for developing a test measurement program, rules of thumb for collecting data, and ways to avoid “metrics dysfunction.” Rick identifies several metrics paradigms and discusses the pros and cons of each. Delegates are urged to bring their metrics problems and issues for use as discussion points.
Testing After You’ve Finished Testing
Stakeholders always want to release when they think we’ve “finished testing”. They believe we have revealed “all of the important problems” and “verified all of the fixes,” and now it’s time to reap the rewards. However, as testers we still can assist in improving software by learning about problems after code has rolled “live-to-site”—especially if it’s a website. At eBay we have a post-ship “site quality” mindset in which testers continue to learn from A/B testing, operational issues, customer sentiment analysis, discussion forums, and customer call patterns—just to name a few. Jon Bach explains how and what eBay’s Live Site Quality team learns every day about what they just released to production. Take away some ideas on what you can do to test and improve value—even after you’ve shipped.
Patters for Team Collaboration: Toward Whole Team Quality
The half-day tutorial covered patterns for team collaboration toward whole-team quality, presented by Janet Gregory and Matt Barcomb. The presentation defined patterns as models or reliable traits and discussed various collaboration patterns including generalizing specialists, pairing, shared vision using tools like business model canvases and impact mapping, specification techniques like user story mapping and acceptance test-driven development, and paired development practices like continuous testing, test-driven development, and configuration management. The goal was to provide teams with collaborative techniques to improve quality.
Introducing Keyword-Driven Test Automation
The document describes a presentation on keyword-driven test automation given by Hans Buwalda of LogiGear Corporation. The presentation introduces keyword-driven testing and compares it to other automation techniques. It provides recommendations for successfully applying keyword-driven testing to test design, automation, and organization. The presentation also covers topics like data-driven testing, non-UI testing, and test automation best practices.
Viewers also liked (11)
Yin and Yang: Metrics within Agile and Traditional Lifecycles
Yin and Yang: Metrics within Agile and Traditional Lifecycles
How to Survive the Coming Test Automation Zombie Apocalypse
How to Survive the Coming Test Automation Zombie Apocalypse
Keynote: Surviving or Thriving: Top Ten Lessons for the Professional Tester
Keynote: Surviving or Thriving: Top Ten Lessons for the Professional Tester
An Introduction to SAFe: The Scaled Agile Framework
An Introduction to SAFe: The Scaled Agile Framework
Patters for Team Collaboration: Toward Whole Team Quality
Patters for Team Collaboration: Toward Whole Team Quality
Similar to Become a Big Data Quality Hero
Query or Not to Query? Using Apache Spark Metrics to Highlight Potentially Pr...
John submits a query and expects it to run smoothly. Based on his prior experience, he anticipates the query to finish in 20 mins.
Scenario-1: John’s query finishes execution in the expected timeframe and doesn’t impact any other concurrent query in the workload.
Scenario-2: John’s query takes twice the expected time, and also slows down multiple other concurrent queries. John now wonders “should I have submitted this query?”.
Testing the Data Warehouse
The document discusses testing strategies for data warehouses. It recommends testing at the primary, secondary, and tertiary levels, with primary level testing focusing on unit testing individual transformations to verify data is mapped and loaded correctly. Common issues found in data warehouse testing are also outlined such as errors in data mapping specifications and load processes leading to invalid data. Planning tests to address typical problems and ensuring data quality is also emphasized.
Testing the Data Warehouse—Big Data, Big Problems
Data warehouses are critical systems for collecting, organizing, and making information readily available for strategic decision making. The ability to review historical trends and monitor near real-time operational data is a key competitive advantage for many organizations. Yet the methods for assuring the quality of these valuable assets are quite different from those of transactional systems. Ensuring that appropriate testing is performed is a major challenge for many enterprises. Geoff Horne has led numerous data warehouse testing projects in both the telecommunications and ERP sectors. Join Geoff as he shares his approaches and experiences, focusing on the key “uniques” of data warehouse testing: methods for assuring data completeness, monitoring data transformations, measuring quality, and more. Geoff explores the opportunities for test automation as part of the data warehouse process, describing how you can harness automation tools to streamline the work and minimize overhead.
Test Data, Information, Knowledge, Wisdom: past, present & future of standing...
The document discusses test data for software testing from both past and present perspectives. It begins by covering definitions and concepts related to test data from ISO 29119 standards. It then discusses the basics of test data and how test data requirements and information are developed through the test planning, design and implementation processes. The document also discusses different sources of test data and how test data relates to different testing levels and techniques like unit testing, integration testing and system testing.
Data Science Popup Austin: Back to The Future for Data and Analytics
JD Stanley, Global Director of Strategy and Integrated Solutions at Cisco's Data and Analytics Business Unit, gave a keynote on data and analytics going "Back to the Future." He discussed how most data will be processed at the edge by mobile devices and IoT solutions in the near future. Cisco is focusing on network analytics, edge computing, data relationships, and accelerating time to value through tools for data exploration, preparation, and prediction. The talk concluded that initial focus should be on data discovery rather than predictions, improving data quality, and unleashing teams of various roles to iteratively improve outcomes through experience rather than just technology.
How Celtra Optimizes its Advertising Platform with Databricks
Leading brands such as Pepsi and Macy’s use Celtra’s technology platform for brand advertising. To inform better product design and resolve issues faster, Celtra relies on Databricks to gather insights from large-scale, diverse, and complex raw event data. Learn how Celtra uses Databricks to simplify their Spark deployment, achieve faster project turnaround time, and empower people to make data-driven decisions.
In this webinar, you will learn how Databricks helps Celtra to:
- Utilize Apache Spark to power their production analytics pipeline.
- Build a “Just-in-Time” data warehouse to analyze diverse data sources such as Elastic Load Balancer access logs, raw tracking events, operational data, and reportable metrics.
- Go beyond simple counting and group events into sequences (i.e., sessionization) and perform more complex analysis such as funnel analytics.
Testing the Data Warehouse―Big Data, Big Problems
The document is a presentation on testing data warehouses. It discusses why data warehouses need to be tested, what aspects should be tested, and where and in what order testing should be conducted. It also covers typical issues encountered in data warehouse testing and strategies for planning testing of continually changing source systems and common data warehouse problems.
Frank Bien Opening Keynote - Join 2016
Frank Bien, CEO of Looker - along with Amazon, Google and other data disrupters - discuss how innovators are deeply integrating analytics into every aspect of their businesses, from mobile to warehouse to cloud.
Frank shares Looker’s vision for the future of business intelligence and data analytics and reveal pivotal product and partnership updates.
Frank Bien Opening Keynote - Join 2016
Frank Bien, CEO of Looker - along with Amazon, Google and other data disrupters - discuss how innovators are deeply integrating analytics into every aspect of their businesses, from mobile to warehouse to cloud.
Frank shares Looker’s vision for the future of business intelligence and data analytics and reveal pivotal product and partnership updates.
351315535-Module-1-Intro-to-Data-Science-pptx.pptx
This document provides an overview of the data science process, including historical notes on related frameworks like KDD, CRISP-DM, and big data. It discusses the typical stages in the knowledge discovery process, including business understanding, data understanding, data preparation, model building, evaluation, and deployment. It also provides an example walking through these stages to predict power failures in Manhattan, covering opportunities assessment, data acquisition and cleaning, model building, policy construction, and evaluation.
DATA @ NFLX (Tableau Conference 2014 Presentation)
I presented this at a 2014 Tableau Conference session with Albert Wong.
Netflix relies on data to make decisions ranging from buying and recommending content, to improving the streaming experience on devices.
This presentation shares our Big Data analytics architecture and the tools used to make data accessible throughout our business, focusing on how Tableau fits into our organization and why it aligns well with our culture.
SQL Server Managing Test Data & Stress Testing January 2011
A quick look at some of the available functionality for SQL Server developers who have access to Visual Studio 2010 and SQL-Hero.
With Visual Studio 2010 Premium (and Professional to a degree) delivering similar capabilities to what was available in VS 2008 Database Pro Edition, the ability to generate a mass amount of sample data for your database has only gotten more accessible with time.
Realizing that other tools exist in this space and not all SQL developers use Visual Studio, we’ll also take a look at the third party data generation facility available in SQL-Hero, seeing how we can create thousands (or millions!) of records very quickly using a powerful rules engine, plus automate this process to support continuous integration strategies.
Transforming and Scaling Large Scale Data Analytics: Moving to a Cloud-based ...
The Census Bureau is the U.S. government's largest statistical agency with a mission to provide current facts and figures about America's people, places and economy. The Bureau operates a large number of surveys to collect this data, the most well known being the decennial population census. Data is being collected in increasing volumes and the analytics solutions must be able to scale to meet the ever increasing needs while maintaining the confidentiality of the data. Past data analytics have occurred in processing silos inhibiting the sharing of information and common reference data is replicated across multiple system. The use of the Hortonworks Data Platform, Hortonworks Data Flow and other open-source technologies is enabling the creation of a cloud-based enterprise data lake and analytics platform. Cloud object stores are used to provide scalable data storage and cloud compute supports permanent and transient clusters. Data governance tools are used to track the data lineage and to provide access controls to sensitive data.
Data DevOps: An Overview
Data, the way that we process it and store it, is one of many important aspects of IT. Data is the lifeblood of our organizations, supporting real-time business processes and decision-making. For our DevOps strategy to be truly effective we must be able to safely and quickly evolve production databases, just as we safely and quickly evolve production code. Yet for many organizations their data sources prove to be less than trustworthy and their data-oriented development efforts little more than productivity sinkholes. We can, and must, do better.
This presentation begins with a collection of agile principles for data professionals and of data principles for agile developers - the first step in working together is to understand and appreciate the priorities and strengths of the people that we work with. Our focus is on a collection of practices that enable development teams to easily and safely evolve and deploy databases. These techniques include agile data modeling, database refactoring, database regression testing, continuous database integration, and continuous database deployment.
We also work through operational strategies required of production databases to support your DevOps strategy. If data sources aren’t an explicit part of your DevOps strategy then you’re not really doing DevOps, are you?
Big Data Matching - How to Find Two Similar Needles in a Really Big Haystack
When consolidating multiple sources of information from across your organization, how do you find the records that relate to the same customer, the same company or the same product? This is the challenge faced by many businesses today when putting a data lake to work. The problem is made far worse when different systems may not have the same contact entered the same way. Is Bob Smith the same as Robert Smith? How about Dr. Robert L. Smith - is he the same person? What about Syncsort, Inc and Sinksort Corp.? Are those the same company? One must compare each individual record to every other record in the dataset with some very sophisticated matching algorithms to determine who is who, and you may have to compare the data multiple times in multiple ways to resolve each entity.
Just to add to the difficulty, let’s say your organization has very large volumes of records in your data lake - you don’t have to compare a thousand records to a thousand other records multiple times - you must compare a million to a million, or 100 million to 100 million. This kind of compute intensive comparison can bring even a powerful cluster to its knees.
This is a problem Syncsort customers must solve, and we have developed some very powerful and intelligent software to tackle it.
View this presentation as we discuss the challenges of entity resolution at scale, how Syncsort’s Trillium data quality software line has tackled them successfully in production clusters and see a demonstration of this software in action.
Testing the Data Warehouse―Big Data, Big Problems
Data warehouses have become a popular mechanism for collecting, organizing, and making information readily available for strategic decision making. The ability to review historical trends and monitor near real-time operational data has become a key competitive advantage for many organizations. Yet the methods for assuring the quality of these valuable assets are quite different from those of transactional systems. Ensuring that the appropriate testing is performed is a major challenge for many enterprises. Geoff Horne has led a number of data warehouse testing projects in both the telecommunications and ERP sectors. Join Geoff as he shares his approaches and experiences, focusing on the key “uniques” of data warehouse testing including methods for assuring data completeness, monitoring data transformations, and measuring quality. He also explores the opportunities for test automation as part of the data warehouse process, describing how it can be harnessed to streamline and minimize overhead.
Introduction to Business and Data Analysis Undergraduate.pdf
The document provides an introduction to business and data analytics. It discusses how businesses are recognizing the value of data analytics and are hiring and upskilling people to expand their data analytics capabilities. It also notes the significant demand for skilled data analysts. The document outlines the modern data ecosystem, including different data sources, key players in turning data into insights, and emerging technologies shaping the ecosystem. It defines data analysis and provides an overview of the data analyst ecosystem.
Data kitchen 7 agile steps - big data fest 9-18-2015
This document discusses applying agile principles and practices to data and analytics teams to address the complexity they face. It outlines seven steps to doing agile data work: 1) adding tests, 2) modularizing and containerizing work, 3) using branching and merging, 4) employing multiple environments, 5) giving analysts tools to experiment, 6) using simple storage, and 7) supporting small team, feature branch, and data governance workflows. The goal is to enable rapid experimentation and integration of new data sources through these agile practices adapted for analytics teams and their unique needs.
Big Data Analytics - Best of the Worst : Anti-patterns & Antidotes
This document discusses best practices for big data analytics. It emphasizes the importance of data curation to ensure semantic consistency and quality across diverse data sources. It warns against simply accumulating large amounts of ungoverned data ("data swamps") without relevant analytics or business applications. Instead, it advocates taking a full stack approach by building incremental decision models and data products to demonstrate value from the beginning. The document also stresses the need for data management layers, appropriate computing frameworks, and real-time and batch analytics capabilities to enable flexible exploration and insights.
Big Data Day LA 2016/ Use Case Driven track - Reliable Media Reporting in an ...
OnPrem Solution Partners worked with NBCU to profile in-house data to determine data quality, and recommend process and quality improvements. We present our process for data import, improvements we want to make, and lessons learned regarding various tools used, including MariaDB, ElasticSearch, Cassandra, and others.
Similar to Become a Big Data Quality Hero (20)
Query or Not to Query? Using Apache Spark Metrics to Highlight Potentially Pr...
Query or Not to Query? Using Apache Spark Metrics to Highlight Potentially Pr...
Test Data, Information, Knowledge, Wisdom: past, present & future of standing...
Test Data, Information, Knowledge, Wisdom: past, present & future of standing...
Data Science Popup Austin: Back to The Future for Data and Analytics
Data Science Popup Austin: Back to The Future for Data and Analytics
How Celtra Optimizes its Advertising Platform with Databricks
How Celtra Optimizes its Advertising Platform with Databricks
351315535-Module-1-Intro-to-Data-Science-pptx.pptx
351315535-Module-1-Intro-to-Data-Science-pptx.pptx
DATA @ NFLX (Tableau Conference 2014 Presentation)
DATA @ NFLX (Tableau Conference 2014 Presentation)
SQL Server Managing Test Data & Stress Testing January 2011
SQL Server Managing Test Data & Stress Testing January 2011
Transforming and Scaling Large Scale Data Analytics: Moving to a Cloud-based ...
Transforming and Scaling Large Scale Data Analytics: Moving to a Cloud-based ...
Big Data Matching - How to Find Two Similar Needles in a Really Big Haystack
Big Data Matching - How to Find Two Similar Needles in a Really Big Haystack
Introduction to Business and Data Analysis Undergraduate.pdf
Introduction to Business and Data Analysis Undergraduate.pdf
Data kitchen 7 agile steps - big data fest 9-18-2015
Data kitchen 7 agile steps - big data fest 9-18-2015
Big Data Analytics - Best of the Worst : Anti-patterns & Antidotes
Big Data Analytics - Best of the Worst : Anti-patterns & Antidotes
Big Data Day LA 2016/ Use Case Driven track - Reliable Media Reporting in an ...
Big Data Day LA 2016/ Use Case Driven track - Reliable Media Reporting in an ...
More from TechWell
Failing and Recovering
Isabel Evans stopped drawing and painting after being told she was not very good at it, which led to a loss of confidence in her creative and professional abilities. However, she realized that attempting creative activities is important for cognitive and emotional development, and that making mistakes and learning from failures allows for growth. By reengaging with failure through art and with support from others, Isabel was able to regain confidence in her abilities and reboot her career. The document discusses different perspectives on failure and the importance of learning from mistakes.
Instill a DevOps Testing Culture in Your Team and Organization
The DevOps movement is here. Companies across many industries are breaking down siloed IT departments and federating them into product development teams. Testing and its practices are at the heart of these changes. Traditionally, IT organizations have been staffed with mostly manual testers and a limited number of automation and performance engineers. To keep pace with development in the new “you build it, you own it” environment, testing teams and individuals must develop new technical skills and even embrace coding to stay relevant and add greater value to the business. DevOps really starts with testing. Join Adam Auerbach as he explains what DevOps is and how it relates to testing. He describes how testing must change from top to bottom and how to access your own environment to identify improvement opportunities. Adam dives into practices like service virtualization, test data management, and continuous testing so you can understand where you are now and identify steps needed to instill a DevOps testing culture in your team and organization.
Test Design for Fully Automated Build Architecture
This document summarizes a half-day tutorial on test design for fully automated build architectures presented by Melissa Benua of mParticle at STAREAST 2018. The tutorial covered guiding principles for test design including prioritizing important and reliable tests, structuring automated pipelines around components, packages, and releases, and monitoring test results through code coverage, flaky test handling, and logging versus counters. It also included exercises mapping test cases to functional boundaries and categories of tests to pipeline stages.
System-Level Test Automation: Ensuring a Good Start
Many organizations invest a lot of effort in test automation at the system level but then have serious problems later on. As a leader, how can you ensure that your new automation efforts will get off to a good start? What can you do to ensure that your automation work provides continuing value? This tutorial covers both “theory” and “practice”. Dot Graham explains the critical issues for getting a good start, and Chris Loder describes his experiences in getting good automation started at a number of companies. The tutorial covers the most important management issues you must address for test automation success, particularly when you are new to automation, and how to choose the best approaches for your organization—no matter which automation tools you use. Focusing on system level testing, Dot and Chris explain how automation affects staffing, who should be responsible for which automation tasks, how managers can best support automation efforts to promote success, what you can realistically expect in benefits and how to report them. They explain—for non-techies—the key technical issues that can make or break your automation effort. Come away with your own clarified automation objectives, and a draft test automation strategy to use to plan your own system-level test automation.
Build Your Mobile App Quality and Test Strategy
Let’s build a mobile app quality and testing strategy together. Whether you have a web, hybrid, or native app, building a quality and testing strategy means (1) knowing what data and tools you have available to make agile decisions, (2) understanding your customers and your competitors, and (3) testing your app under real-world conditions. Jason Arbon guides you through the latest techniques, data, and tools to ensure the awesomeness of your mobile app quality and testing strategy. Leave this interactive session with a strategy for your very own app—or one you pretend to own. The information Jason shares is based on data from Appdiff’s next-gen mobile app testing platform, lessons from Applause/uTest’s crowd, text mining hundreds of millions of app store reviews, and in-depth discussions with top mobile app development teams.
Testing Transformation: The Art and Science for Success
Technologies, testing processes, and the role of the tester have evolved significantly in the past few years with the advent of agile, DevOps, and other new technologies. It is critical that we testing professionals evaluate ourselves and continue to add tangible value to our organizations. In your work, are you focused on the trivial or on real game changers? Jennifer Bonine describes critical elements that help you artfully blend people, process, and technology to create a synergistic relationship that adds value. Jennifer shares ideas on mastering politics, maneuvering core vs. context, and innovating your technology strategies and processes. She explores how new processes can be introduced in an organization, what the role of organizational culture is in determining the success of a project, and how you can know what tools will add value vs. simply adding overhead and complexity. Jennifer reviews critically needed tester skills and discusses a continual learning model to evolve your skills and stay relevant. This discussion can lead you to technologies, processes, and skills you can stake your career on.
Implement BDD with Cucumber and SpecFlow
We’ve all been there. We work incredibly hard to develop a feature and design tests based on written requirements. We build a detailed test plan that aligns the tests with the software and the documented business needs. And when we put the tests to the software, it all falls apart because the requirements were changed without informing everyone. Mary Thorn says help is at hand. Enter behavior-driven development (BDD), and Cucumber and SpecFlow, tools for running automated acceptance tests and facilitating BDD. Mary explores the nuances of Cucumber and SpecFlow, and shows you how to implement BDD and agile acceptance testing. By fostering collaboration for implementing active requirements via a common language and format, Cucumber and SpecFlow bridge the communication gap between business stakeholders and implementation teams. In this workshop, practice writing feature files with the best practices Mary has discovered over numerous implementations. If you experience developers not coding to requirements, testers not getting requirements updates, or customers who feel out of the loop and don’t get what they ask for, Mary has answers for you.
Develop WebDriver Automated Tests—and Keep Your Sanity
Many teams go crazy because of brittle, high-maintenance automated test suites. Jim Holmes helps you understand how to create a flexible, maintainable, high-value suite of functional tests using Selenium WebDriver. Learn the basics of what to test, what not to test, and how to avoid overlapping with other types of testing. Jim includes both philosophical concepts and hands-on coding. Testers who haven't written code should not be intimidated! We'll pair you up to make sure you're successful. Learn to create practical tests dealing with advanced situations such as input validation, AJAX delays, and working with file downloads. Additionally, discover when you need to work together with developers to create a system that's more easily testable. This tutorial focuses primarily on automating web tests, but many of the same concepts can be applied to other UI environments. Demos and labs will be in C# and Java using WebDriver. Leave this tutorial having learned how to write high-value WebDriver tests—and stay sane while doing so.
Ma 15
DevOps is a cultural shift aimed at streamlining intergroup communication and improving operational efficiency for development and operations groups. Over time, inclusion of other IT groups under the DevOps umbrella has become the norm for many organizations. But even broadening the boundaries of DevOps, the conversation has been largely devoid of the business units’ place at the table. A common mistake organizations make while going through the DevOps transformation is drawing a line at the IT boundary. If that occurs, a larger, more inclusive silo within the organization is created, operating in an informational vacuum and causing operational inefficiency and goal misalignment. Sharing his experiences working on both sides of the fence, Leon Fayer describes the importance of including business units in order to align technology decisions with business goals. Leon discusses inclusion of business units in existing agile processes, benefits of cross-departmental monitoring, and a business-first approach to technology decisions.
Eliminate Cloud Waste with a Holistic DevOps Strategy
Chris Parlette maintains that renting infrastructure on demand is the most disruptive trend in IT in decades. In 2016, enterprises spent $23B on public cloud IaaS services. By 2020, that figure is expected to reach $65B. The public cloud is now used like a utility, and like any utility, there is waste. Who's responsible for optimizing the infrastructure and reducing wasted expenses? It’s DevOps. The excess expense, known as cloud waste, comprises several interrelated problems: services running when they don't need to be, improperly sized infrastructure, orphaned resources, and shadow IT. There are a few core tenets of DevOps—holistic thinking, no silos, rapid useful feedback, and automation—that can be applied to reducing your cloud waste. Join Chris to learn why you should include continuous cost optimization in your DevOps processes. Automate cost control, reduce your cloud expenses, and make your life easier.
Transform Test Organizations for the New World of DevOps
With the recent emergence of DevOps across the industry, testing organizations are being challenged to transform themselves significantly within a short period of time to stay meaningful within their organizations. It’s not easy to plan and approach these changes considering the way testing organizations have remained structured for ages. These challenges start from foundational organizational structures and can cut across leadership influence, competencies, tools strategy, infrastructure, and other dimensions. Sumit Kumar shares his experience assisting various organizations to overcome these challenges using an organized DevOps enablement framework. The framework includes radical restructuring, turning the tools strategy upside down, a multidimensional workforce enablement supported by infrastructure changes, redeveloped collaborations models, and more. From his real world experiences Sumit shares tips for approaching this journey and explains the roadmap for testing organizations to transform themselves to lead the quality in DevOps.
The Fourth Constraint in Project Delivery—Leadership
All too often, the triple constraints—time, cost, and quality—are bandied about as if they are the be-all, end-all. While they are important, leadership—the fourth and larger underpinning constraint—influences the first three. Statistics on project success and failure abound, and these measurements are usually taken against the triple constraints. According to the Project Management Institute, only 53 percent of projects are completed within budget, and only 49 percent are completed on time. If so many projects overrun budget and are late, we can’t really say, “Good, fast, or cheap—pick two.” Rob Burkett talks about leadership at every level of a team. He shares his insights and stories gleaned from his years of IT and project management experience. Rob speaks to some of the glaring difficulties in the workplace in general and some specifically related to IT delivery and project management. Leave with a clearer understanding of how to communicate with teams and team members, and gain a better understanding of how you can be a leader—up and down your organization.
Resolve the Contradiction of Specialists within Agile Teams
As teams grow, organizations often draw a distinction between feature teams, which deliver the visible business value to the user, and component teams, which manage shared work. Steve Berczuk says that this distinction can help organizations be more productive and scale effectively, but he recognizes that not all shared work fits into this model. Some work is best handled by “specialists,” that is people with unique skills. Although teams composed entirely of T-shaped people is ideal, certain skills are hard to come by and are used irregularly across an organization. Since these specialists often need to work closely with teams, rather than working from their own backlog, they don’t fit into the component team model. The use of shared resources presents challenges to the agile planning model. Steve Berczuk shares how teams such as those providing infrastructure services and specialists can fit into a feature+component team model, and how variations such as embedding specialists in a scrum team can both present process challenges and add significant value to both the team and the larger organization.
Pin the Tail on the Metric: A Field-Tested Agile Game
Metrics don’t have to be a necessary evil. If done right, metrics can help guide us to make better forward-looking decisions, rather than being used for simply managing or monitoring. They can help us identify trade-offs between options for what to do next versus punitive or worse, purely managerial measures. Steve Martin won’t be giving the Top Ten List of field-tested metrics you should use. Instead, in this interactive mini-workshop, he leads you through the critical thinking necessary for you to determine what is right for you to measure. First, Steve explores why you want to measure something—whether it’s for a team, a portfolio, or even an agile transformation. Next, he provides multiple real-life metrics examples to help drive home concepts behind characteristics of good and bad metrics. Finally, Steve shows how to run his field-tested agile game—Pin the Tail on the Metric. Take back this activity to help you guide metrics conversations at your organization.
Agile Performance Holarchy (APH)—A Model for Scaling Agile Teams
A hierarchy is an organizational network that has a top and a bottom, and where position is determined by rank, importance, and value. A holarchy is a network that has no top or bottom and where each person’s value derives from his ability, rather than position. As more companies seek the benefits of agile, leaders need to build and sustain delivery capability while scaling agile without introducing unnecessary process and overhead. The Agile Performance Holarchy (APH) is an empirical model for scaling and sustaining agility while continuing to deliver great products. Jeff Dalton designed the APH by drawing from lessons learned observing and assessing hundreds of agile companies and teams. The APH helps implement a holarchy—a system composed of interacting organizational units called holons—centered on a series of performance circles that embody the behaviors of high performing agile organizations. Jeff describes how APH provides guidelines in the areas of leadership, values, teaming, visioning, governing, building, supporting, and engaging within an all-agile organization. Join Jeff to see what the APH is all about and how you can use it in your team and organization.
A Business-First Approach to DevOps Implementation
DevOps is a cultural shift aimed at streamlining intergroup communication and improving operational efficiency for development and operations groups. Over time, inclusion of other IT groups under the DevOps umbrella has become the norm for many organizations. But even broadening the boundaries of DevOps, the conversation has been largely devoid of the business units’ place at the table. A common mistake organizations make while going through the DevOps transformation is drawing a line at the IT boundary. If that occurs, a larger, more inclusive silo within the organization is created, operating in an informational vacuum and causing operational inefficiency and goal misalignment. Sharing his experiences working on both sides of the fence, Leon Fayer describes the importance of including business units in order to align technology decisions with business goals. Leon discusses inclusion of business units in existing agile processes, benefits of cross-departmental monitoring, and a business-first approach to technology decisions.
Databases in a Continuous Integration/Delivery Process
The document summarizes a presentation about including databases in a continuous integration/delivery process. It discusses treating database code like application code by placing it under version control and integrating databases into the DevOps software development pipeline. This allows databases to be built, tested, and released like other software through continuous integration, delivery, and deployment.
Mobile Testing: What—and What Not—to Automate
Organizations are moving rapidly into mobile technology, which has significantly increased the demand for testing of mobile applications. David Dangs says testers naturally are turning to automation to help ease the workload, increase potential test coverage, and improve testing efficiency. But should you try to automate all things mobile? Unfortunately, the answer is not always clear. Mobile has its own set of complications, compounded by a wide variety of devices and OS platforms. Join David to learn what mobile testing activities are ripe for automation—and those items best left to manual efforts. He describes the various considerations for automating each type of mobile application: mobile web, native app, and hybrid applications. David also covers device-level testing, types of testing, available automation tools, and recommendations for automation effectiveness. Finally, based on his years of mobile testing experience, David provides some tips and tricks to approach mobile automation. Leave with a clear plan for automating your mobile applications.
Cultural Intelligence: A Key Skill for Success
Diversity is becoming the norm in everyday life. However, introducing global delivery models without a proper understanding of intercultural differences can lead to difficulty, frustration, and reduced productivity. Priyanka Sharma and Thena Barry say that in our diverse world, we need teams with people who can cross these boundaries, communicate effectively, and build the diverse networks necessary to avoid problems. We need to learn about cultural intelligence (CI) and cultural quotient (CQ). CI is the ability to relate and work effectively across cultures. CQ is the cognitive, motivational, and behavioral capacity to understand and respond to beliefs, values, attitudes, and behaviors of individuals and groups. Together, CI and CQ can help us build behavioral capacities that aid motivation, behavior, and productivity in teams as well as individuals. Priyanka and Thena show how to build a more culturally intelligent place with tools and techniques from Leading with Cultural Intelligence, as well as content from the Hofstede cultural model. In addition, they illustrate the model with real-life experiences and demonstrate how they adapted in similar circumstances.
Turn the Lights On: A Power Utility Company's Agile Transformation
Why would a century-old utility with no direct competitors take on the challenge of transforming its entire IT application organization to an agile methodology? In an increasingly interconnected world, the expectations of customers continue to evolve. From smart meters to smart phones, IoT is creating a crisis point for industries not accustomed to rapid change. Glen Morris explains that pizzas can be tracked by the minute and packages at every stop, and customers now expect this same customer service model should exist for all industries—including power. Glen examines how to create momentum and transform non-IT-focused industries to an agile model. If you are struggling with gaining traction in your pursuit of agile within your business, Glen gives you concrete, practical experiences to leverage in your pursuit. Finally, he communicates how to gain buy-in from business partners who have no idea or concern about agile or its methodologies. If your business partners look at you with amusement when you mention the need for a dedicated Product Owner, join Glen as he walks you through the approaches to overcoming agile skepticism.
More from TechWell (20)
Instill a DevOps Testing Culture in Your Team and Organization
Instill a DevOps Testing Culture in Your Team and Organization
Test Design for Fully Automated Build Architecture
Test Design for Fully Automated Build Architecture
System-Level Test Automation: Ensuring a Good Start
System-Level Test Automation: Ensuring a Good Start
Testing Transformation: The Art and Science for Success
Testing Transformation: The Art and Science for Success
Develop WebDriver Automated Tests—and Keep Your Sanity
Develop WebDriver Automated Tests—and Keep Your Sanity
Eliminate Cloud Waste with a Holistic DevOps Strategy
Eliminate Cloud Waste with a Holistic DevOps Strategy
Transform Test Organizations for the New World of DevOps
Transform Test Organizations for the New World of DevOps
The Fourth Constraint in Project Delivery—Leadership
The Fourth Constraint in Project Delivery—Leadership
Resolve the Contradiction of Specialists within Agile Teams
Resolve the Contradiction of Specialists within Agile Teams
Pin the Tail on the Metric: A Field-Tested Agile Game
Pin the Tail on the Metric: A Field-Tested Agile Game
Agile Performance Holarchy (APH)—A Model for Scaling Agile Teams
Agile Performance Holarchy (APH)—A Model for Scaling Agile Teams
A Business-First Approach to DevOps Implementation
A Business-First Approach to DevOps Implementation
Databases in a Continuous Integration/Delivery Process
Databases in a Continuous Integration/Delivery Process
Turn the Lights On: A Power Utility Company's Agile Transformation
Turn the Lights On: A Power Utility Company's Agile Transformation
Recently uploaded
Epistemic Interaction - tuning interfaces to provide information for AI support
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
UiPath Test Automation using UiPath Test Suite series, part 5
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
Full-RAG: A modern architecture for hyper-personalization
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
Pushing the limits of ePRTC: 100ns holdover for 100 days
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
20240605 QFM017 Machine Intelligence Reading List May 2024
Everything I found interesting about machines behaving intelligently during May 2024
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
A tale of scale & speed: How the US Navy is enabling software delivery from l...
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Microsoft - Power Platform_G.Aspiotis.pdf
Revolutionizing Application Development
with AI-powered low-code, presentation by George Aspiotis, Sr. Partner Development Manager, Microsoft
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
GridMate - End to end testing is a critical piece to ensure quality and avoid...
End to end testing is a critical piece to ensure quality and avoid regressions. In this session, we share our journey building an E2E testing pipeline for GridMate components (LWC and Aura) using Cypress, JSForce, FakerJS…
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
DevOps and Testing slides at DASA Connect
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
20240609 QFM020 Irresponsible AI Reading List May 2024
Everything I found interesting about the irresponsible use of machine intelligence in May 2024
Essentials of Automations: The Art of Triggers and Actions in FME
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Recently uploaded (20)
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
Become a Big Data Quality Hero
- 1. T8 Concurrent Class 10/3/2013 11:15:00 AM "Become a Big Data Quality Hero" Presented by: Jason Rauen LexisNexis Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
- 2. Jason Rauen LexisNexis Jason Rauen is a senior quality test analyst at Georgia-based LexisNexis Risk Solutions. With more than fifteen years of experience, Jason has led the data testing team in big data from its inception. He has presented big data scripting techniques at HPCC Systems national Data Summit. His background includes working at companies including Microsoft, AT&T, and LexisNexis, and instructing at Intel, Boeing, Executrain, and the Department of the Navy.
- 3. 9/19/2013 Interesting Quotes…… “Quality isn’t measured by how many clients you obtain; it’s measured by how many clients you retain.” “QA isn’t the bottom of the totem pole; it’s the dirt holding it up.” Become a Big Data Quality Hero A look inside QA for Big Data Presented by 01001010 01100001 01110011 01101111 01101110 00100000 01010010 01100001 01110101 01100101 01101110 (Jason Rauen) 1
- 4. 9/19/2013 Overview • Why Test and How it’s Different – Issues – Benefits • Architecture and why you need to know – HPCC Systems/Hadoop – Know Your Data/Environment • Strategies and Concepts – What to look for – Sample Gathering (AUB) – Stats – Profiling Why Test and How it’s Different Why Test Data: • Traditional methods not adequate – Traditional sampling needs improvement and is scenario based, not enough samples, human error, etc…. • Tied into current environment • Government regulatory compliances • Auditing requirements • Company wide initiatives 2
- 5. 9/19/2013 Why Test and How it’s Different Want to keep your customers? Why Test and How it’s Different • When? o Testing - SDLC o Routine Testing o Frequency - Yearly/Monthly/Weekly/Daily/Hourly/On Demand • What? Types Testing New Project – Source to Target (Transform) Standard - Production Validation Emergency releases • How? o Using what you have available o Freebies – Profiling tools, etc… 3
- 6. 9/19/2013 Why Test and How it’s Different Issues: • Lack of control Timing of builds Samples and location of samples • 3rd Party Apps Lack of licenses, Costs, Training, and existing knowledge • Extra hardware • Upgrades Why Test and How it’s Different Benefits: • Cost savings • Better Coverage No Samples Increased Sampling Focused Samples • Faster (Time is $) • Quicker to Diagnosing issues • Better Data Integrity • Collaboration with other groups 4
- 7. 9/19/2013 Architecture and why you need to know Typical Generic Architecture input DB Architecture and why you need to know Data Fabrication Engines • HDFS Hadoop and HPCC THOR • Made of several nodes • Where the ETL happens • Where the Keys are made Data Delivery Engines • HPCC ROXIE, HBASE, etc… • Keys moved to and referenced here • Queries reside 5
- 8. 9/19/2013 Architecture and why you need to know Architecture and why you need to know HDFS Hadoop Mapreduce HBASE 6
- 9. 9/19/2013 Architecture and why you need to know Architecture and why you need to know HDFS Map Shuffle Reduce 7
- 10. 9/19/2013 Architecture and why you need to know HPCC Systems DISTRIBUTE/PROJECT/TRANSFORM Rollup Strategies and Concepts • What to look for…… Brand New, Incomplete, or Missing Builds (Data Cops) Data progression Today/Yesterday FatherKey/Grandfatherkey Count of Deltas in release/deploy Keys updated Missing keys/New keys Field Validations Indexed and Non Indexed Key Layout issues Corruption unprintable or invalid characters Duplicate records of new and existing records Data Fabrication Engine to Data delivery Engine deploys/sync Queries with new data 8
- 11. 9/19/2013 Strategies and Concepts JOIN • Sample gathering • New Key for testing • Deployment Validation - Data Fabrication • Deployment Validation - Data Delivery And get a free cookie… Strategies and Concepts AUB for JOIN A = Left key (New) B = Right key (Old) Types of JOINS Inner Join Left Outer Join Full Outer Join Right Outer Join Minus or Left Only 9
- 12. 9/19/2013 Strategies and Concepts AUB for JOIN A = Left key (New) B = Right key (Old) VENN Strategies and Concepts Statistics: What you try to remember with this swimming behind you. 10
- 13. 9/19/2013 Strategies and Concepts Statistics: • On data sets and keys - Gives you a high level look at the release - Ranges - You’ll start to notice a trend line • On Releases - Done over time you’ll see the trend of new data sets and keys - Done over time you’ll see the trend of changed or modified data sets and keys Strategies and Concepts RELEASE NUMBERS 400 350 AVERAG 175.4 300 CEILING 210.6 250 FLOOR 135.1 200 150 100 50 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 11
- 14. 9/19/2013 Strategies and Concepts Data Profiling: • Data Profiling Summary Report • Data Profiling Field Detail Report http://www.hpccsystems.com/demos/dataprofiling-demo • Data Profiling Field Combination Report Strategies and Concepts Data Profiling Summary Report 12
- 15. 9/19/2013 Strategies and Concepts Data Profiling Field Detail Report Strategies and Concepts Data Profiling Field Combination Report 13
- 16. 9/19/2013 Strategies and Concepts SQL Pig ECL SELECT * FROM Products; DUMP Products; Products; SELECT * FROM Products WHERE productcode = ‘R2D2C3PO’; Products= FILTER Products BY productcode = ‘R2D2C3PO’; DUMP Products; Products(productcode = ‘R2D2C3PO’); Products= GROUP Products ALL; Products =FOREACH Products GENERATE COUNT (Products); DUMP Products; COUNT(Products); SELECT COUNT (*) FROM Products; Strategies and Concepts SQL Pig ECL SELECT * FROM Products ORDER BY productcode; Products= ORDER SORT( Products BY productcode; Products,productcode); DUMP Products; SELECT * FROM Products FULL OUTER JOIN OtherProducts ON Products.col1 = OtherProducts.col1; Products= JOIN Products BY col1 FULL OUTER, OtherProducts BY col1; DUMP Products; JOIN(Products,OtherPro ducts, LEFT.col1 = RIGHT.col1,FULL OUTER); 14
- 17. 9/19/2013 Summary Why Test and How it’s Different Architecture and why you need to know Strategies and Concepts Questions? 15
- 18. 9/19/2013 Contact / Useful links www.linkedin/in/jasonrauen • HPCC Systems/ECL Links: http://hpccsystems.com http://hpccsystems.com/demos • Hadoop/Pig Latin Links: http://pig.apache.org http://hadoop.apache.org • SQL Links: http://sql.org/ http://msdn.microsoft.com/en-US/sqlserver/default.aspx 16