This document discusses implementing Agile methodology for business intelligence (BI) projects. It begins by addressing common misconceptions about Agile BI, noting that it does not require specific tools or methodologies and can be applied using existing technologies. The document then examines extract, transform, load (ETL) tools and how some may not be well-suited for Agile due to issues like proprietary coding and lack of integration with version control and continuous integration practices. However, ETL tools can still be used when appropriate. The document provides recommendations for setting up an Agile BI environment, including using ETL tools judiciously and mitigating issues through practices like sandboxed development environments and test data sets to enable test-driven development.
Agile BI Development Through AutomationManta Tools
How can code life cycle automation satisfy the growing demands in modern enterprise business intelligence?
Whilst an agile approach to BI development is useful for delivering value in general, the use of advanced automation techniques can also save significant resources, prevent production errors, and shorten time to market.
Gentlemen from Data To Value, Manta Tools, Volkswagen and M&G investments presented and discussed different approaches to agile BI development. Take a look!
This 1 day, hands-on, workshop will introduce the processes and workflows necessary to manage a Business Intelligence team in a flexible, iterative and agile manner. Through standard agile management methods (Scrum, Kanban and Test-Driven Development), this workshop will provide you with the tools to manage your workflow, BI development, demand management, and customer engagement.
The goal of this workshop is to expose you to different ways of working and give you potential tactics and techniques to improve your BI project delivery.
Agile Data Warehouse Design for Big Data PresentationVishal Kumar
Synopsis:
[Video link: http://www.youtube.com/watch?v=ZNrTxSU5IQ0 ]
Jim Stagnitto and John DiPietro of consulting firm a2c) will discuss Agile Data Warehouse Design - a step-by-step method for data warehousing / business intelligence (DW/BI) professionals to better collect and translate business intelligence requirements into successful dimensional data warehouse designs.
The method utilizes BEAM✲ (Business Event Analysis and Modeling) - an agile approach to dimensional data modeling that can be used throughout analysis and design to improve productivity and communication between DW designers and BI stakeholders. BEAM✲ builds upon the body of mature "best practice" dimensional DW design techniques, and collects "just enough" non-technical business process information from BI stakeholders to allow the modeler to slot their business needs directly and simply into proven DW design patterns.
BEAM✲ encourages DW/BI designers to move away from the keyboard and their entity relationship modeling tools and begin "white board" modeling interactively with BI stakeholders. With the right guidance, BI stakeholders can and should model their own BI data requirements, so that they can fully understand and govern what they will be able to report on and analyze.
The BEAM✲ method is fully described in
Agile Data Warehouse Design - a text co-written by Lawrence Corr and Jim Stagnitto.
About the speaker:
Jim Stagnitto Director of a2c Data Services Practice
Data Warehouse Architect: specializing in powerful designs that extract the maximum business benefit from Intelligence and Insight investments.
Master Data Management (MDM) and Customer Data Integration (CDI) strategist and architect.
Data Warehousing, Data Quality, and Data Integration thought-leader: co-author with Lawrence Corr of "Agile Data Warehouse Design", guest author of Ralph Kimball’s “Data Warehouse Designer” column, and contributing author to Ralph and Joe Caserta's latest book: “The DW ETL Toolkit”.
John DiPietro Chief Technology Officer at A2C IT Consulting
John DiPietro is the Chief Technology Officer for a2c. Mr. DiPietro is responsible
for setting the vision, strategy, delivery, and methodologies for a2c’s Solution
Practice Offerings for all national accounts. The a2c CTO brings with him an
expansive depth and breadth of specialized skills in his field.
Sponsor Note:
Thanks to:
Microsoft NERD for providing awesome venue for the event.
http://A2C.com IT Consulting for providing the food/drinks.
http://Cognizeus.com for providing book to give away as raffle.
Improving the customer experience using big data customer-centric measurement...Business Over Broadway
This presentation provides an overview of some of the content of my new book, TCE: Total Customer Experience. In the presentation, I discuss customer experience management, customer loyalty, the optimal customer survey, the value of analytics and using a Big Data customer-centric approach to improve the value of all your business data
Agile BI Development Through AutomationManta Tools
How can code life cycle automation satisfy the growing demands in modern enterprise business intelligence?
Whilst an agile approach to BI development is useful for delivering value in general, the use of advanced automation techniques can also save significant resources, prevent production errors, and shorten time to market.
Gentlemen from Data To Value, Manta Tools, Volkswagen and M&G investments presented and discussed different approaches to agile BI development. Take a look!
This 1 day, hands-on, workshop will introduce the processes and workflows necessary to manage a Business Intelligence team in a flexible, iterative and agile manner. Through standard agile management methods (Scrum, Kanban and Test-Driven Development), this workshop will provide you with the tools to manage your workflow, BI development, demand management, and customer engagement.
The goal of this workshop is to expose you to different ways of working and give you potential tactics and techniques to improve your BI project delivery.
Agile Data Warehouse Design for Big Data PresentationVishal Kumar
Synopsis:
[Video link: http://www.youtube.com/watch?v=ZNrTxSU5IQ0 ]
Jim Stagnitto and John DiPietro of consulting firm a2c) will discuss Agile Data Warehouse Design - a step-by-step method for data warehousing / business intelligence (DW/BI) professionals to better collect and translate business intelligence requirements into successful dimensional data warehouse designs.
The method utilizes BEAM✲ (Business Event Analysis and Modeling) - an agile approach to dimensional data modeling that can be used throughout analysis and design to improve productivity and communication between DW designers and BI stakeholders. BEAM✲ builds upon the body of mature "best practice" dimensional DW design techniques, and collects "just enough" non-technical business process information from BI stakeholders to allow the modeler to slot their business needs directly and simply into proven DW design patterns.
BEAM✲ encourages DW/BI designers to move away from the keyboard and their entity relationship modeling tools and begin "white board" modeling interactively with BI stakeholders. With the right guidance, BI stakeholders can and should model their own BI data requirements, so that they can fully understand and govern what they will be able to report on and analyze.
The BEAM✲ method is fully described in
Agile Data Warehouse Design - a text co-written by Lawrence Corr and Jim Stagnitto.
About the speaker:
Jim Stagnitto Director of a2c Data Services Practice
Data Warehouse Architect: specializing in powerful designs that extract the maximum business benefit from Intelligence and Insight investments.
Master Data Management (MDM) and Customer Data Integration (CDI) strategist and architect.
Data Warehousing, Data Quality, and Data Integration thought-leader: co-author with Lawrence Corr of "Agile Data Warehouse Design", guest author of Ralph Kimball’s “Data Warehouse Designer” column, and contributing author to Ralph and Joe Caserta's latest book: “The DW ETL Toolkit”.
John DiPietro Chief Technology Officer at A2C IT Consulting
John DiPietro is the Chief Technology Officer for a2c. Mr. DiPietro is responsible
for setting the vision, strategy, delivery, and methodologies for a2c’s Solution
Practice Offerings for all national accounts. The a2c CTO brings with him an
expansive depth and breadth of specialized skills in his field.
Sponsor Note:
Thanks to:
Microsoft NERD for providing awesome venue for the event.
http://A2C.com IT Consulting for providing the food/drinks.
http://Cognizeus.com for providing book to give away as raffle.
Improving the customer experience using big data customer-centric measurement...Business Over Broadway
This presentation provides an overview of some of the content of my new book, TCE: Total Customer Experience. In the presentation, I discuss customer experience management, customer loyalty, the optimal customer survey, the value of analytics and using a Big Data customer-centric approach to improve the value of all your business data
Re-Architect Your Legacy Environment To Enable An Agile, Future-Ready EnterpriseDell World
It’s time to re-architect your legacy environment in order to lay the foundation for an adaptive enterprise. In this session, you'll learn how to increase your business and technical agility using a fit-to-purpose .NET or Java architecture, while deploying your apps intelligently in the cloud and integrating with your complex IT environment, customers and partners.
This exam measures your ability to accomplish the technical tasks listed below. The percentages indicate the relative weight of each major topic area on the exam. https://www.pass4sureexam.com/70-461.html
This presentation will help you understand the basic building blocks of Business Intelligence. Learn how decisions are triggered, the complete decision process and who makes decisions in the corporate world.
More importantly, understand core components of a Business Intelligence architecture such as a data warehouse, data mining, OLAP (Online analytical procession) , OLTP (Online Transaction Processing) and data reporting. Each component plays an integral part which enables today's managers and decision makers collect, analyze and interpret data to make it actionable for decision making.
Business intelligence has become an integral part that needs to be incorporated to ensure business survival. It is a tool that helps analyze historical data and forecast future so that your are always one step ahead in your business.
Please feel free to like, share and comment as you please!
Validation and Business Considerations for Clinical Study MigrationsPerficient, Inc.
There are a variety of essential validation and business considerations that should be evaluated when migrating clinical studies from one database to another (with or without existing data). Having a clear understanding of downstream study processes and receiving input from cross-functional teams are just some of the keys to a successful migration.
In this SlideShare we discuss several case studies that provide insight into:
The steps followed during a study migration process, from validation and business perspectives
Validation considerations for migrating into an empty database, as well as a database that already contains data
Suggested documentation for the migration process
Business continuity considerations to ensure a smooth study migration for all team members
Best practices and lessons learned
As powerful as graph technology is, the potential still exists for every graph - and the ontologies used to create them - to become YAS (Yet Another Silo). This deck, first presented at @Dave McCombs' Semantic Arts Conference in Colorado this February, contains a few tips to avoid that from happening.
What's New with SAP BusinessObjects Business Intelligence 4.1?SAP Analytics
http://spr.ly/sapbusinessobjectsbi - Learn more about SAP's strategy for an enterprise BI and the new features in SAP BusinessObjects Business Intelligence 4.1.
For more information on “SAP BusinessObjects BI 4.1 Upgrade”, register for our upcoming webcast on October 8, 2013, 8 AM PST: http://bit.ly/1bamzuh
Workday Integration Cloud Connect consists of a growing number of pre-built, packaged integrations and connectors to complementary solutions that are 100% built, maintained, and supported by Workday. http://www.workday.com/solutions/technology/integration_cloud/integration_cloud_connect.php
Re-Architect Your Legacy Environment To Enable An Agile, Future-Ready EnterpriseDell World
It’s time to re-architect your legacy environment in order to lay the foundation for an adaptive enterprise. In this session, you'll learn how to increase your business and technical agility using a fit-to-purpose .NET or Java architecture, while deploying your apps intelligently in the cloud and integrating with your complex IT environment, customers and partners.
This exam measures your ability to accomplish the technical tasks listed below. The percentages indicate the relative weight of each major topic area on the exam. https://www.pass4sureexam.com/70-461.html
This presentation will help you understand the basic building blocks of Business Intelligence. Learn how decisions are triggered, the complete decision process and who makes decisions in the corporate world.
More importantly, understand core components of a Business Intelligence architecture such as a data warehouse, data mining, OLAP (Online analytical procession) , OLTP (Online Transaction Processing) and data reporting. Each component plays an integral part which enables today's managers and decision makers collect, analyze and interpret data to make it actionable for decision making.
Business intelligence has become an integral part that needs to be incorporated to ensure business survival. It is a tool that helps analyze historical data and forecast future so that your are always one step ahead in your business.
Please feel free to like, share and comment as you please!
Validation and Business Considerations for Clinical Study MigrationsPerficient, Inc.
There are a variety of essential validation and business considerations that should be evaluated when migrating clinical studies from one database to another (with or without existing data). Having a clear understanding of downstream study processes and receiving input from cross-functional teams are just some of the keys to a successful migration.
In this SlideShare we discuss several case studies that provide insight into:
The steps followed during a study migration process, from validation and business perspectives
Validation considerations for migrating into an empty database, as well as a database that already contains data
Suggested documentation for the migration process
Business continuity considerations to ensure a smooth study migration for all team members
Best practices and lessons learned
As powerful as graph technology is, the potential still exists for every graph - and the ontologies used to create them - to become YAS (Yet Another Silo). This deck, first presented at @Dave McCombs' Semantic Arts Conference in Colorado this February, contains a few tips to avoid that from happening.
What's New with SAP BusinessObjects Business Intelligence 4.1?SAP Analytics
http://spr.ly/sapbusinessobjectsbi - Learn more about SAP's strategy for an enterprise BI and the new features in SAP BusinessObjects Business Intelligence 4.1.
For more information on “SAP BusinessObjects BI 4.1 Upgrade”, register for our upcoming webcast on October 8, 2013, 8 AM PST: http://bit.ly/1bamzuh
Workday Integration Cloud Connect consists of a growing number of pre-built, packaged integrations and connectors to complementary solutions that are 100% built, maintained, and supported by Workday. http://www.workday.com/solutions/technology/integration_cloud/integration_cloud_connect.php
Hybrid Data Warehouse Hadoop ImplementationsDavid Portnoy
Data Warehouse vendors are evolving to incorporate the best Hadoop has to offer. Similarly, the Hadoop ecosystem is growing to include capabilities previously available only to large scale (MPP) DW platforms.
Agile Business Intelligence (or how to give management what they need when th...Evan Leybourn
If you like the ideas raised in this presentation, don't forget to check out my latest book, Directing the Agile Organisation (http://theagiledirector.com/book).
Based on common agile management methods, this presentation will demonstrate the processes and workflows required to manage a Business Intelligence team or project in a flexible, iterative and agile manner. We will also examine the open source technologies that assist in supporting and automating the processes.
These processes draw on the underlying principles of agile and utilises a combination of Scrum, Test Driven Development, Feature Driven Design and XP. These methods can be applied in both a low maturity environment to develop business intelligence capability, or a high maturity environment to encourage greater customer engagement.
Learn how agile methods can dramatically improve the design of BI/DW systems: database schemas. View the webinar video recording and download this deck: http://www.senturus.com/resources/agile-bi-demystified/.
Special guest, Lawrence Corr, author of the book Agile Data Warehouse Design, introduces BEAM (business event analysis and modeling) with a set of agile tools and techniques for dimensional modelstorming (modeling + brainstorming) with BI stakeholders and users.
Senturus, a business analytics consulting firm, has a resource library with hundreds of free recorded webinars, trainings, demos and unbiased product reviews. Take a look and share them with your colleagues and friends: http://www.senturus.com/resources/.
This presentation was presented by Martin Kersten (CWI), well known in the Dutch eScience and scientific computing community, at the Netherlands eScience Center (NLeSC) on November 9, 2011 in Amsterdam, Netherlands.
Abstract of the presentation:
This presentation gives an introduction to NoSQL (Not only SQL) (pdf) databases with examples from MonetDB and discussed, applications and limitations.
Building a heterogeneous Hadoop Olap system with Microsoft BI stack. PABLO DO...Big Data Spain
Session presented at Big Data Spain 2012 Conference
16th Nov 2012
ETSI Telecomunicacion UPM Madrid
www.bigdataspain.org
More info: http://www.bigdataspain.org/es-2012/conference/building-a-heterogeneous-hadoop-olap-system-with-microsoft-bi-stack/pablo-doval-and-ibon-landa
Stockage de données dans les SGBD
Cette présentation traite des diverses manières de stocker der informations dans les bases de donées ainsi que des approches techniques permettant d'optimiser le traitement de ces données tout en consommant le moins de ressources possibles
Shorter time to insight more adaptable less costly bi with end to end modelst...Daniel Upton
For Data Project Leaders: This data warehouse data modeling approach enables shorter time to insights, lower cost and greater adaptability to external changes by combining my End to End Data Modelstorming concept with Data Vault modeling.
An exciting talk on the main difficulties and how to overcome them when building and scaling data teams with Florian Douetteau
- Technological issues: What stack should they choose for the company’s architecture? And what about big data technologies; should they accept being a polyglot or rather assume being a ruthless dictator?
- HR issues: Who should they hire? Should they build their data team as an extension of the BI team? Or should they build it from scratch?
- Data issues: How are they supposed to get data inside his data lake? Which strategy should they adopt: the cicada, the spider or the fox one?
- Product issues: What is big data really about? And eventually, what are they willing to do with this bunch of data?
The talk aims at demonstrating how tough it can be to build and scale a data department, and at giving some insights about the strategy Florian thinks they should adopt.
Canadian Experts Discuss Modern Data Stacks and Cloud Computing for 5 Years o...Daniel Zivkovic
Two #ModernDataStack talks and one DevOps talk: https://youtu.be/4R--iLnjCmU
1. "From Data-driven Business to Business-driven Data: Hands-on #DataModelling exercise" by Jacob Frackson of Montreal Analytics
2. "Trends in the #DataEngineering Consulting Landscape" by Nadji Bessa of Infostrux Solutions
3. "Building Secure #Serverless Delivery Pipelines on #GCP" by Ugo Udokporo of Google Cloud Canada
We ran out of time for the 4th presenter, so the event will CONTINUE in March... stay tuned! Compliments of #ServerlessTO.
Product Analysis Oracle BI Applications IntroductionAcevedoApps
Oracle Business Intelligence (BI) Applications are complete, prebuilt BI solutions that deliver intuitive, role-based intelligence for everyone in an organization—from front line employees to senior management—that enable better decisions, actions, and business processes. Designed for both “single source” and heterogeneous environments, these solutions enable organizations to gain insight from a range of data sources and applications including Siebel, Oracle E-Business Suite, PeopleSoft Enterprise, JD Edwards, and third party systems such as SAP.
Business Process De Pillis Tool ComparisonG.J. dePillis
Comparing the two giants in the industry: ProVision by Metastorm and ARIS by IDS Scheer/AG Software.
Has your company outgrown Visio? Do you need to graduate to a Business Process tool? this presentation will help answer your questions.
Pentaho Data Integration: Extrayendo, integrando, normalizando y preparando m...Alex Rayón Jerez
Sesión de Pentaho Data Integration impartida en Noviembre de 2015 en el marco del Programa de Big Data y Business Intelligence de la Universidad de Deusto (detalle aquí http://bit.ly/1PhIVgJ).
Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)Rittman Analytics
Set of product roadmap + capabilities slides from Oracle Data Integration Product Management, and thoughts on data integration on big data implementations by Mark Rittman (Independent Analyst)
How AI is transforming DevOps | Calidad InfotechCalidad Infotech
DevOps is a remarkable asset to start-ups. The growing technology over the last two decades has made it easier to build & scale all sizes of businesses & organizations. In this fast-paced growing technology world, DevOps has paved its way with its innovative & effective tools & practices that have turned out to be a… Continue reading.. https://calidadinfotech.com/devops-services
Applying the Lean Startup approach to data management
Speaker: David Portnoy, Former Entrepreneur-in-Residence, U.S. Department of Health and Human Services (HHS)
Abstract: Discussed at the White House Open Data Roundtable on data quality (and a subsequent government-wide Data Cabinet meeting), the Demand-Driven Open Data (DDOD) program was demonstrated to be an effective approach to improving data quality. DDOD is best described as “Lean Startup for open data”. It’s a framework for implementing open data initiatives in a more systematic and practical way, delivering improvements in data assets and creation of shared technical capabilities. We’ll also cover how a Lean Startup approach to data management is applicable to industry as well.
This infographic demonstrates the accomplishments of the DDOD program within the pilot for the U.S. Department of Health and Human Services (HHS).
Impact of DDOD on Data Quality - White House 2016David Portnoy
"The Impact of Demand-Driven Open Data (DDOD) on Data Quality" was presented on April 27, 2016 at Open Data Roundtable held at the White House Office of Science and Technology Policy.
It discusses the data quality problems prevalent in open data and their impact, the origins of the DDOD concept, how it works, progress towards its goals, several use case examples, and how to implement it at other organizations.
More information:
* DDOD http://ddod.healthdata.gov
* Open Data Roundtables https://www.data.gov/meta/open-data-roundtables/
* White House Office of Science and Technology Policy: https://www.whitehouse.gov/blog/2016/02/05/open-data-empowering-americans-make-data-driven-decisions
There are a growing number of examples demonstrating compelling and creative uses of data provided by U.S. Department of Health and Human Services (HHS) agencies.
HHS provides a wealth of open data sources and APIs. Industry, researchers and media have been able to put these data assets to good use, creating significant economic value, informing the public and improving public health.
This document explores the concepts behind how DDOD (Demand-Driven Open Data) can be used in conjunction with FOIA (Freedom of Information Act) requests. It describes how DDOD and FOIA can leverage each other's strengths to help overcome their inherent challenges.
DDOD is an initiative by the U.S. Department of Health and Human Services (HHS) started in November 2014 as part of its IDEA Lab program. The goal is to leverage the vast data assets throughout HHS’s agencies (CMS, FDA, NIH, CDC, NCHS, AHRQ and others) to create additional economic and public health value.
DDOD provides a systematic, ongoing and transparent mechanism for anybody to tell HHS and its agencies what data would be valuable to them. It's the Lean Startup approach to open data. With this initiative HHS can move from measuring Open Data in terms of number of datasets released to value in terms of use cases enabled.
DDOD website: http://ddod.us
Intro to Demand-Driven Open Data for Data OwnersDavid Portnoy
This document is intended for use by data owners within government to learn how Demand-Driven Open Data (DDOD) could benefit their organizations.
DDOD is an initiative by the U.S. Department of Health and Human Services (HHS) started in November 2014 as part of its IDEA Lab program. The goal is to leverage the vast data assets throughout HHS’s agencies (CMS, NIH, CDC, FDA, AHRQ and others) to create additional economic and public health value.
DDOD provides a systematic, ongoing and transparent mechanism for anybody to tell HHS and its agencies what data would be valuable to them. With this initiative HHS can move from measuring Open Data in terms of number of datasets released to value in terms of use cases enabled.
DDOD website: http://ddod.us
Intro to Demand Driven Open Data for Data UsersDavid Portnoy
This document is intended for any commercial or academic organization to learn how Demand-Driven Open Data (DDOD) could benefit them.
DDOD is an initiative by the U.S. Department of Health and Human Services (HHS) started in November 2014 as part of its IDEA Lab program. The goal is to leverage the vast data assets throughout HHS’s agencies (CMS, NIH, CDC, FDA, AHRQ and others) to create additional economic and public health value.
DDOD provides a systematic, ongoing and transparent mechanism for anybody to tell HHS and its agencies what data would be valuable to them. With this initiative HHS can move from measuring Open Data in terms of number of datasets released to value in terms of use cases enabled.
DDOD website: http://ddod.us
Case Study in Linked Data and Semantic Web: Human GenomeDavid Portnoy
The National Human Genome Research Institute's "GWAS Catalog" (Genome-Wide Association Studies) project is a successful implementation of Linked Data (http://linkeddata.org/) and Semantic Web (http://www.w3.org/standards/semanticweb/) concepts. This deck discusses how this project has been implemented, challenges faced and possible paths for the future.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
2. This group extends the TDWI community
online and is designed to foster peer
network and discussion of key issues
relevant to business intelligence and data
warehousing managers.
TDWI (The Data Warehousing
Institute™) provides education, training,
certification, news, and research for
executives and information technology
(IT) professionals worldwide. Founded in
1995, TDWI is the premier educational
institute for business intelligence and
data warehousing. Our Web site is
www.tdwi.org.
Why this topic?
There’s a lot of confusion and misconception about the
meaning of Agile, especially as it applies to BI
Many in corporate IT still believe that Agile cannot easily be
applied to BI
Posts on this topic in the TDWI forum in LinkedIn would benefit
from being organized and summarized
3. What we’ll cover
Misconceptions about Agile BI
Core techniques of Agile BI
Review of ETL tool landscape and benefits
Decision factors for choosing the ETL environment
Mitigating aspects of ETL tools that make Agile harder
How to implement an Agile BI development environment
Due to the prevailing confusion
and misconceptions, it’s easier to
start with what Agile BI is not
4. Misconceptions about Agile in the BI community
There’s a common misconception that Agile BI applies to practically any
methodology or tool that helps develop BI projects faster or in a more flexible way.
Some examples of misconceptions:
Agile is primarily adding iterations to
typical projects
Agile implies starting to code without
planning or design
Agile involves particular data models,
such as Data Vault
Agile involves rapid prototyping
techniques, as can be achieved by
certain metadata driven tools
Agile involves self-serve reporting, such
as Tableau
Agile involves moving ETL from a
separate code base into the reporting
layer, as made possible by in-memory
processing, such as with QlickView
Agile involves building real-time or low-
latency DW, rather than traditional batch
Agile operates in a hosted cloud
environment, especially PaaS (Platform
as a Service)
5. The culprits for the myths and misconceptions
#1 Vendors claim that their products are agile.
#2 The BI community as a whole does not have a long history or
substantial practice with agile development. Therefore they are more likely
to be swayed by vendor pitches.
6. The culprits for the myths and misconceptions (cont.)
In the software development world, that’s equivalent to saying
that new frameworks, such as Ruby on Rails, are needed for
Agile development. (Few credible publications or developers
would make such a claim.)
The implication that other BI tools can’t be used to achieve
Agile BI is simply not true. (Even general purpose
development platforms can be applied to BI.)
In reality, team composition, proficiency with existing
technologies and management’s acceptance of agile is a
bigger impact than a specific type of BI tool.
“...Agile BI methodology differs from
[agile software development] in that it
requires new and different technologies
and architectures for support. Metadata-
generated BI applications are one such
example...”
Example source of
misconceptions:
The article goes on to claim that these
particular tools are needed in order to
achieve “development done faster”,
“react[ing] more quickly to ...
requirements“, incremental product
delivery, “rapid prototypes versus
specifications”, “reacting versus
planning”, “personal interactions ...
versus documentation”, etc.
Forrester Research article “Agile Out of the
Box”, 2010
This list is just buzz words associated with agile without
substantial evidence of why other tools are insufficient.
Rapid prototyping is confused with the role of end-to-end
working software.
On the contrary, arguments can be made why the tools
identified could be detrimental to agile teams. (See TDWI
LinkedIn group discussion “The Role of ETL tools in Agile
BI”.)
What’s wrong with itWhat’s being said
7. The reality
Yes, many of the items misclassified as necessary for
Agile still help projects ramp up and complete faster.
Yes, many improve the flexibility of dealing with
changes in source data, business logic and reporting.
Yes, many provide additional visibility into complex
logic and functional changes across team members and
stakeholders.
Data Vault model
Rapid prototyping tools
Metadata driven BI tools
Self-serve reporting
In-memory processing
Hosted cloud (PaaS)
environment
But none of them are required
to have successful Agile BI projects
8. So what are the requirements for implementing Agile BI?
Productive Agile BI teams operate almost identically to Agile methodology used for
software development.
...With just the minimal tweaks to accommodate:
1. Integration of available ETL and reporting tools into the development
environment
2. Changes to regression testing due to the fact that databases have state
3. Challenges of managing large data sets in the deployment process
9. Techniques for implementing Agile in BI
Timebox deliverables – of course
Measure completion with working
software! (Prototypes using non-
production tools are OK. But need to get
end-to-end data flow working ASAP.)
Highly efficient, daily team
synchronization in which entire team
participates.
Monitor completion of features (stories),
not time spent. Calculate team velocity to
improve planning.
Hold sprint retrospectives to learn from
mistakes.
Leverage techniques of Agile app dev:
Manage everything in version control,
including data model and test data sets
Assume refactoring of working code can
occur later to improve performance and
maintainability
Use Test Driven Development (TDD), to
ensure understanding of requirements and
reduce rework
Implement Continuous Integration to
automate build, tests, deployment
Measure project success by delivery of
business value, not delivery of predefined
requirements on time and on budget
Accept that it’s OK to fail, but fail early and
adapt. (Non agile projects don’t recognize
failure until time or budget runs out.)
10. What’s the reason for low adoption of Agile in BI?
Application Development Business Intelligence
Development
Environment
Custom app development using
standard, general purpose
languages well suited for
automation
Proprietary vendor architectures and
DSLs (domain specific languages) not
well suited for automation
Team skills Have skills to write automation for
continuous integration
Rely on vendors to provide these
features
Costs Low up front investment by
leveraging open source platforms
High up front investment in vendor-
specific tools: DW appliance, data
modeling, ETL, OLAP, Reporting, etc.
Releases Software is stateless and therefore
easier to test and deploy with each
build
Databases have state, with each build
needing to start with a certain data set.
High data volumes may take hours to
load a changed data model or roll back
changes.
Agile is widely adopted in application development
...but not in BI
Potential reasons might stem from differences between the two worlds
12. ETL tools have evolved over the years
Graphical development accomplishing ETL through parameterization and
configuration, rather than code generation
Avoids complexities with code management and deployment
Intuitive development UI enabling developers to manipulate ETL metadata
From metadata, generate code in a general purpose (such as C or Java) or
domain specific (such as SQL or MDX) language
Types: One-shot generators (that require switching to a native dev env) vs.
full development environments with managed version deployments
Origin: Reusable code compiled from a few similar projects
Just change parameters to reuse for specific loading, logging, change data
capture, database connections, etc.
One-time solutions
Built with focus on short-term delivery and minimal up front cost
Custom
Code
Frameworks
Code
Generators
Engines
13. We can categorize the major ETL players
The vendors
Traditional vendors: Informatica, SSIS, DataStage
Open source: Talend, Pentaho Kettle
Metadata driven, automated discovery, federated integration:
Kalido, BI Ready, Wherescape, Composite Software
The most common alternative
SQL + shell scripts
Native DB load utilities
14. ETL tools have lots of value
Built-in commonly used features for transformation and job control
Without ETL tools, we’re reinventing the wheel on many BI design patterns that
have been implemented countless times throughout history
Abstracts complex logic into a graphical components or domain specific language
that leverages best practices and is often more maintainable over the potentially
long project life span
Graphical representation of data model, data flow and job flow provide visibility
into business logic, especially useful for less technical team members
Provides a degree of self-documentation without the need to update the graphical
representation of logic separately from source code
Master Data Management (MDM)
Data cleansing
Change Data Capture (CDC)
Data lineage and data dependency
functionality
Processing of SCD (Slowly Changing
Dimensions)
Parallelization of tasks that can be run
concurrently
Advanced merging functionality
15. But many ETL tools are
not well suited to an
Agile BI environment
16. First, these tools may not be ideal for Agile in general...
Some ETL tools are...
Not well suited for code refactoring, branching, and merging because
the code is not in text files that can be used modern version control, such
as Git
Not well suited for use with automation in Continuous Integration,
because they’re often standalone environments with no provisions for
external automation
Not well suited for TDD (Test Driven Development), unless the vendors
explicitly made provisions for unit test automation
Proprietary and have “black box” features that might make testing more
challenging or decrease portability of test cases
Expensive, with high up-front license cost also putting more capital at
risk – unless open source ETL, of course
17. Second, they may negatively impact productivity of Agile teams
ETL tools may...
Require a proprietary, vendor-specific skill set not present in the organization
Cause work priority to be stove-piped and limited to skill set, rather than overall
business value
Prevent the ability to leverage the full dev team, since they fall under a
separate development environment from the rest of apps
Result in a productivity hit, since some professional developers are more
productive writing code in native languages than using GUI tools, even after
training
Not provide compelling enough reasons for developers to learn any one ETL
tool, since the lack of industry standards decreases skill portability
18. Third, there are other challenges and considerations
There are challenges and limitations with ETL tools even outside of Agile
Require allocation of additional resources to manage version upgrades of the
ETL tool, even if the code base hasn’t been changing
When the type of processing needed is outside of core ETL tool features,
complexity can grow quickly
Usefulness of visual representations for data models, data flows and job flows is
reduced as complexity increases
Some find GUI development less efficient than traditional coding, especially for
complex or unique type of processing
Often the sophisticated features are underutilized, resulting in expensive tools
being used just for job scheduling
19. Fourth, BI is increasingly involving Big Data
Big Data implementations often make ETL tools less compelling
Large volumes make it more efficient to
Manipulate data in place using ELT, rather than have multiple staging areas
Use native methods (MapReduce /Java, SQL, Hive, etc.) that allow for more control
and performance optimization
High velocity of data makes it harder to use ETL tools that have traditionally
been designed around batch-oriented processing.
High variability of data makes ETL tools less attractive, since they expect a
fixed schema and don’t gracefully accommodate changes. Common examples
include unstructured web log data in flat files and logical objects from apps
stored in key-value pair format.
MPP vendors, such as Teradata and Netezza make a case for doing ELT (rather
than ETL) processing natively and provide built-in features to do so
Currently ETL tools are rarely used with the Hadoop ecosystem for many of the
reasons stated, as well as licensing cost
20. That said, how do we
implement an-
Agile BI environment?
21. First, use ETL tools when it makes sense
Pick the right ETL tool for the job...
We covered the potential benefits and problems of using such ETL tools
for Agile BI. Look for situations where benefits outweigh the problems.
For example, a good situation to employ ETL tools might be: A use case
requiring sophisticated data cleansing transformations, complex job control
logic, and data volumes easily handled by traditional SMP database
architectures.
Outside of such situations, consider using SQL, DB-specific native code, or
general purpose languages already in use elsewhere in the organization.
Is it OK to start with using an ETL tool as a job scheduler?
Yes, assuming it’s an efficient way to handle much needed job control
logic, including failures, event triggers, and dependencies.
Plus, you get the option to adopt other capabilities of the tool over time
with low project risk.
While traditional ETL tools
can simplify a complex task,
they can also overcomplicate
a simple task.
22. Second, when you do use ETL tools, look
for ways to mitigate these issues identified
So what’s the solution?
L
Issue Approach
High up-front
license cost
Use open source tools or less expensive licenses like with SQL Server.
Aggressive vendor negotiations, in light of lower cost alternatives.
Use with
Continuous
Integration
See following slides. Some vendors, like Microsoft, may make provisions for
automated builds within their environment. Otherwise look for opportunities to
simplify, partially automate, and notify team of build state.
Use with
version control
Where possible, save ETL logic to XML, create dumps of repository, and
generate code from metadata. Then manage in common version control tool.
Decreased
portability
Move code to general purpose development languages, including SQL and
MDX. Consider tools that generate generic code from GUI or metadata.
Vendor-specific
skill set
Build cross-functional team by...
Training existing developers
Hiring well-rounded developers willing to learn ETL tools
Risk of
introducing
another
development
environment
Start using ETL tools now and “grow” into using the functionality
Continue coding in what you know: native RDBMS code or even general
app dev languages
Start using ETL as a glorified job scheduler to wrap native code
When refactoring code, take the opportunity to push more logic into the
ETL tool
Gradually start using other features such as MDM, data quality,
notifications, enterprise service bus, etc.
23. Continuous Integration: Methodology
Each developer should have a sandbox:
1-to-1 app instance to DB instance (CI by Martin Fowler)
Automate: Table deployment, usage stats, schema
verification, data migration verification, DB testing,
migration to prod
Version control all DB assets, ideally using a
distributed tool like Git
Use tool like dbDeploy and link app build, DB version, and forward/reverse DDL & DML
scripts
Generate a test data set with a dimension annotating what each is testing; Becomes a
company asset that enables TDD of BI
For cases where an application consumes data from the data warehouse:
BI developers should learn software coding practices; Application developers should learn
data modeling, SQL, DB tuning
Consuming apps use 2 phased builds:
Build 1, DB is stubbed out and runs within minutes
Build 2, includes real DB for end-to-end testing, but might run for a while
Bugs found in Build 2, trigger additions to the test data set; Next time same bug is caught
in Build 1
Shared
developer
schema
Dev 1
Dev 2
Dev 3
Typical BI dev env
with contention
during development
Sandboxed dev env
appropriate for agile
development
Schema Dev 1Dev 1
Dev 2
Dev 3
Schema Dev 2
Schema Dev 3
24. Continuous Integration: Tools & Configuration
How dbDeploy works
dbDeploy is treated as a custom Ant task:
1. Logs & assigns version #s to changes in
SQL files
2. Save changelog table since prior version
3. Generates DDL & DML scripts to apply to
DB in other envs
Tool Type Purpose
Ant Build tool Automates steps to build & deploy software
Jenkins Continuous
Integration
Monitors source code repository (Git) for checkins,
automatically launching build-test cycles and publishing
results.
Git Source control /
repository
Source code repository optimized for branching and merging,
making it efficient for each developer to have their own
sandbox environment. It triggers CI built-test cycles.
dbDeploy,
dbMaintain,
etc.
Database
refactoring manager
Automates the process of establishing
which database refactorings need to be
run against a specific database in order
to migrate it to a particular build.
DbUnit, DbFit,
SQLUnit
Unit test automation Common tool to aid TDDD (Test-driven DB development).
Manage DB state between test runs, import/export test
datasets, run unit tests and log exceptions. Regression testing
of DDL, DML, stored procedures.
Developer
Env.
Repository
(Git)
CI Environment
Check
out
Build Tool
Deploy
& Test
Test server
Prod server
Project
Code
Check
in
Success /
Fail Tag
25. Continuous Refactoring & Releases of Databases
Dev
Sandbox
Project
Integration
Sandbox
Test / QA
Sandbox
Production
Highly iterative
development
Characteristics
Environment
Deployment
Frequency
Risk / impact
of bug
Project-Level
Testing
System Integration
Testing
Operations &
Support
Frequent Infrequent Controlled
Low impact
Medium impact
High impact
Based on presentation by Pramod Sadalage
Testing Test data set
(Used for TDD)
Test data set
Benchmark data
Production data
26. Continuous Integration:
Possible Configuration for Microsoft BI Stack
PowerDelivery
Addresses TFS’s weakness in coordinating the promotion of builds
through multiple environments of the delivery pipeline: triggering build
on commit, promoting commit build to test, promoting test build to
prod
Windows PowerShell
Task-based command-line shell & scripting language (built on .NET)
for task automation
Team Foundation Server
Microsoft's application lifecycle management (ALM) solution.
Collaboration platform that supports agile delivery practices
Build machine is configured for continuous integration, so latest
working version is refreshed and available to the entire distributed
team
SQL Server Data Tools
Develop, debug, and execute database unit tests interactively
in Visual Studio.
Puts database testing on an equal footing with application testing.
Can then be run from command line or from a build machine
Integrated with testing, bug tracking, and project management using
TFS