Recipes 6 of Data Warehouse and Business Intelligence - Naming convention tec...Massimo Cenci
The naming convention is a key component of any IT project.
The purpose of this article is to suggest a standard for a practical and effective Data Warehouse design
Recipes 6 of Data Warehouse and Business Intelligence - Naming convention tec...Massimo Cenci
The naming convention is a key component of any IT project.
The purpose of this article is to suggest a standard for a practical and effective Data Warehouse design
Recipes 8 of Data Warehouse and Business Intelligence - Naming convention tec...Massimo Cenci
The naming convention is a key component of any IT project.
The purpose of this article is to suggest a standard for a practical and effective Data Warehouse design in Oracle environment
Recipe 16 of Data Warehouse and Business Intelligence - The monitoring of the...Massimo Cenci
For each calendar day, we must have clear what I expect to receive on that day and, for any given data file, what is the reference day that I expect to find inside.
There can be no doubt or ambiguity: is information that we need to know in advance and we have to configure.
This Doc Consist of ER diagram of University and NHL, Introduction to posgres SQL and installation,DML and its various commands,implementation of constraints with examples,DML Implementation with set operations & Functions,Implementation of nested Queries.
Recipes 9 of Data Warehouse and Business Intelligence - Techniques to control...Massimo Cenci
In the loading of a Data Warehouse is important to have full control of the processing units that compose it.
Each processing unit must be carefully monitored both in the detection of errors that may occur,
both in the analysis of the execution times
With the introduction of SQL Server 2012 data developers have new ways to interact with their databases. This session will review the powerful new analytic windows functions, new ways to generate numeric sequences and new ways to page the results of our queries. Other features that will be discussed are improvements in error handling and new parsing and concatenating features.
MySQL Replication Evolution -- Confoo Montreal 2017Dave Stokes
MySQL Replication has evolved since the early days with simple async master/slave replication with better security, high availability, and now InnoDB Cluster
This presentation is an INTRODUCTION to intermediate MySQL query optimization for the Audience of PHP World 2017. It covers some of the more intricate features in a cursory overview.
This presentations shows how to create a time/date dimension for PowerPivot from the date data in your fact table. I also shows the DAX functions that you can use to add columns to the fact table or a separate dimension table.
Recipes 8 of Data Warehouse and Business Intelligence - Naming convention tec...Massimo Cenci
The naming convention is a key component of any IT project.
The purpose of this article is to suggest a standard for a practical and effective Data Warehouse design in Oracle environment
Recipe 16 of Data Warehouse and Business Intelligence - The monitoring of the...Massimo Cenci
For each calendar day, we must have clear what I expect to receive on that day and, for any given data file, what is the reference day that I expect to find inside.
There can be no doubt or ambiguity: is information that we need to know in advance and we have to configure.
This Doc Consist of ER diagram of University and NHL, Introduction to posgres SQL and installation,DML and its various commands,implementation of constraints with examples,DML Implementation with set operations & Functions,Implementation of nested Queries.
Recipes 9 of Data Warehouse and Business Intelligence - Techniques to control...Massimo Cenci
In the loading of a Data Warehouse is important to have full control of the processing units that compose it.
Each processing unit must be carefully monitored both in the detection of errors that may occur,
both in the analysis of the execution times
With the introduction of SQL Server 2012 data developers have new ways to interact with their databases. This session will review the powerful new analytic windows functions, new ways to generate numeric sequences and new ways to page the results of our queries. Other features that will be discussed are improvements in error handling and new parsing and concatenating features.
MySQL Replication Evolution -- Confoo Montreal 2017Dave Stokes
MySQL Replication has evolved since the early days with simple async master/slave replication with better security, high availability, and now InnoDB Cluster
This presentation is an INTRODUCTION to intermediate MySQL query optimization for the Audience of PHP World 2017. It covers some of the more intricate features in a cursory overview.
This presentations shows how to create a time/date dimension for PowerPivot from the date data in your fact table. I also shows the DAX functions that you can use to add columns to the fact table or a separate dimension table.
Presenter: Dean Richards of Confio Software
If you're a developer or DBA, this presentation will outline a method for determining the best execution plan for a query every time by utilizing SQL Diagramming techniques.
Whether you're a beginner or expert, this approach will save you countless hours tuning a query.
You Will Learn:
* SQL Tuning Methodology
* Response Time Tuning Practices
* How to use SQL Diagramming techniques to tune SQL statements
* How to read executions plans
This presentation deals with the fundamentals of SQL, Installation and Database concepts. Presented by our team in Alphalogic Inc: https://www.alphalogicinc.com/
MySQL 8 -- A new beginning : Sunshine PHP/PHP UK (updated)Dave Stokes
MySQL 8 has many new features and this presentation covers the new data dictionary, improved JSON functions, roles, histograms, and much more. Updated after SunshinePHP 2018 after feedback
Open Source 101 2022 - MySQL Indexes and HistogramsFrederic Descamps
Nobody complains that the database is too fast. But when things slow down, the complaints come quickly. The two most popular approaches to speeding up queries are indexes and histograms. But there are so many options and types on indexes that it can get confusing. Histograms are fairly new to MySQL but they do not work for all types of data. This talk covers how indexes and histograms work and show you how to test just how effective they are so you can measure the performance of your queries.
Data Modeling, Normalization, and De-Normalization | PostgresOpen 2019 | Dimi...Citus Data
As a developer using PostgreSQL one of the most important tasks you have to deal with is modeling the database schema for your application. In order to achieve a solid design, it’s important to understand how the schema is then going to be used as well as the trade-offs it involves.
As Fred Brooks said: “Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious.”
In this talk we're going to see practical normalisation examples and their benefits, and also review some anti-patterns and their typical PostgreSQL solutions, including Denormalization techniques thanks to advanced Data Types.
Il controllo temporale dei data file in staging areaMassimo Cenci
Dobbiamo conoscere prima le precise caratteristiche temporali del data file che dobbiamo caricare nel Data Warehouse.
Per ogni giorno solare, dobbiamo avere chiaro quali data file mi aspetto di ricevere in quel giorno e, per ogni data file, quale è il giorno di riferimento dei dati che mi aspetto di trovare al suo interno
Recipe 12 of Data Warehouse and Business Intelligence - How to identify and c...Massimo Cenci
In a presentation of some time ago, I had highlighted the importance of the control
between the reference day and the expected one of a data file received from the host system.
Given the importance of the topic,
I would further deepen both the meaning of the control and the identification of the reference day of the data.
This argument may seem strange, or too technical, mainly for those who are
not very experienced about Data Warehouse but this is a very useful example to understand the
pitfalls inside the ETL process.
Recipes 10 of Data Warehouse and Business Intelligence - The descriptions man...Massimo Cenci
The descriptions management, is a subject little discussed within the Data Warehouse and Business Intelligence community. It speaks little in books and articles, although it is a crucial issue because it affects the way we see the information.
So, we'll talk about descriptions. In particular, the descriptions of the codes that we can found in the dimension tables of a Data Warehouse.
Note di Data Warehouse e Business Intelligence - Tecniche di Naming Conventio...Massimo Cenci
La naming convention è una componente fondamentale di ogni progetto informatico.
L’obiettivo di questo articolo è quello di suggerire uno standard di nomenclatura pratico ed efficace per un progetto di Data Warehouse.
(parte 2)
Note di Data Warehouse e Business Intelligence - Tecniche di Naming Conventio...Massimo Cenci
La naming convention è una componente fondamentale di ogni progetto informatico.
L’obiettivo di questo articolo è quello di suggerire uno standard di nomenclatura
pratico ed efficace per un progetto di Data Warehouse.
(parte 1)
Oracle All-in-One - how to send mail with attach using oracle pl/sqlMassimo Cenci
Oracle All-in-One (All you need in only one slide)
OAiO1 - How to send mail with attach using Oracle plsql
The goal is to have in a single image all the objects involved
in a common need of a Data Warehouse in Oracle environment.
Note di Data Warehouse e Business Intelligence - Le Dimensioni di analisi (pa...Massimo Cenci
Le dimensioni di analisi sono le componenti fondamentali per definire gli spazi analitici all’interno di un Data Warehouse.
Analizziamo in dettaglio il loro design e la loro implementazione (parte 2)
Note di Data Warehouse e Business Intelligence - Le Dimensioni di analisiMassimo Cenci
Le dimensioni di analisi sono le componenti fondamentali per definire gli spazi analitici all’interno di un Data Warehouse. L’obiettivo è quello di analizzare in dettaglio il loro design e la loro implementazione
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Data Warehouse and Business Intelligence - Recipe 4 - Staging area - how to verify the reference day
1. Recipes of Data Warehouse and
Business Intelligence
Staging Area: How to verify the reference day of the
data source
2. The Micro ETL Foundation
•
•
•
•
•
The Micro ETL Foundation is a set of ideas and solutions for Data Warehouse and
Business Intelligence Projects in Oracle environment.
It doesn’t use expensive ETL tools, but only your intelligence and ability to
think, configure, build and load data using the features and the programming
language of your RDBMS.
This recipes is another easy example based on the slides of Recipes 1 and 2 of Data
Warehouse and Business Intelligence.
Copying the content of the following slides with your editor and SQL Interface
utility, you can reproduce this example.
The solution presented here is the check of the reference day of a data source.
This is another little component to permit us to achive the complete control of the
Staging Area loading.
3. The problem
• In the «Recipe 3» we have seen the check about the rows number. We
have ensured this: You have received, for example, 943 rows from the data
source file. You have loaded 943 rows in the Staging Area table.
• This is not enough.
• Now we need to ensure that the reference day (DAY_KEY) of the data is
correct.
• Suppose that you start the loading of account balances file now:
wednesday at 01:00. You expect that the reference day of the balances is
yesterday, tuesday. The loading day is today, the expected day is today-1.
• If now is monday, and there is no process during week-end, the expected
day is today-3, friday
4. What to check
• At the end of the loading, we need to control that it is gone all ok. We
need to ensure that in this loading day, the expected day is correct. To
have this safety, we must:
1. Build a configuration table.
2. Initialize the configuration table with the expected day for every
loading day
3. Create a log table
4. Fill the log table at the end of the load of Staging Area.
• Without this check we risk to reload the same day if the source system
had some problems or it has not been able to overwrite the previous file.
5. The daily configuration table
•
•
•
•
•
•
•
IO_COD is the same of the configuration
table created in «Recipes2». It is unique.
DAY_KEY is the loading day.
DY_TXT is the day of the week
WD_FLG is the flag of working day.
FROM_EDAY_KEY/TO_EDAY_KEY are the
expected range of day. For daily data
source file they will have the same value.
To have a range can be useful for
monthly data files that contains all the
days of the month.
Now we see how to configure 3 years.
DROP TABLE STA_IODAY_CFT;
CREATE TABLE STA_IODAY_CFT
(
IO_COD
VARCHAR2(12),
DAY_KEY
NUMBER,
DY_TXT
VARCHAR2(3),
WD_FLG
NUMBER,
FROM_EDAY_KEY NUMBER,
TO_EDAY_KEY NUMBER
);
6. The load of daily table
•
•
•
The load of this table may be done only one time.
In this example we configure a daily loading (excluding week-end) from a year in
the future (sysdate+365) and 2 years in the past.
Configure holidays according to your country calendar. Here are configured only
Christmas Day, New Year's Day and Independence Day.
INSERT INTO STA_IODAY_CFT (IO_COD,DAY_KEY,DY_TXT,WD_FLG,FROM_EDAY_KEY,TO_EDAY_KEY)
WITH X AS (
SELECT TO_CHAR((SYSDATE+365)-LEVEL,'YYYYMMDD') AS DAY_KEY
,TO_CHAR((SYSDATE+365)-LEVEL,'dy') DY_TXT
FROM DUAL CONNECT BY LEVEL <= (365*3))
,Y AS (
SELECT TO_CHAR((SYSDATE+365)-LEVEL,'YYYYMMDD') AS DAY_KEY
,FIRST_VALUE(TO_CHAR((SYSDATE+365)-LEVEL,'YYYYMMDD'))
OVER (ORDER BY TO_CHAR((SYSDATE+365)-LEVEL,'YYYYMMDD')
ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS EDAY_KEY
FROM DUAL
WHERE TO_CHAR((SYSDATE+365)-LEVEL,'dy') NOT IN ('sun','sat')
AND SUBSTR(TO_CHAR(SYSDATE+365-LEVEL,'YYYYMMDD'),5)
NOT IN ('0101','0704','1225')
CONNECT BY LEVEL <= (365*3) ORDER BY 1 DESC)
SELECT 'employees1'
,X.DAY_KEY,X.DY_TXT
,(CASE WHEN (EDAY_KEY IS NULL) THEN 0 ELSE 1 END)
,Y.EDAY_KEY,Y.EDAY_KEY
FROM X
LEFT OUTER JOIN Y ON (X.DAY_KEY=Y.DAY_KEY)
ORDER BY 1,2 DESC;
7. The daily log table
•
•
•
•
This table must contain a row for
every data source file.
The DAY_KEY is the loading day
The others day columns are the
expected reference day and the
effective reference day.
The RET_COD contains the result
code of the check and will be ‘OK ‘ or
‘NOT OK’.
DROP TABLE STA_IODAY_LOT;
CREATE TABLE STA_IODAY_LOT
(
IO_COD
VARCHAR2(12) NOT NULL,
SOURCE_COD VARCHAR2(80) NOT NULL,
DAY_KEY
NUMBER,
FROM_EDAY_KEY NUMBER,
TO_EDAY_KEY NUMBER,
FROM_DAY_KEY NUMBER,
TO_DAY_KEY NUMBER,
RET_COD
VARCHAR2(30),
STAMP_DTS DATE
);
8. The load of the log table
•
•
•
We use the test case of the «Recipe 2» and the data loaded into
STA_EMPLOYEES1_STT table. (so it has an old day_key)
We calculate the min and max day_key from the Staging Area table.
We compare the received day(s) with the expected day(s)
INSERT INTO STA_IODAY_LOT (
IO_COD, SOURCE_COD, DAY_KEY,
FROM_EDAY_KEY, TO_EDAY_KEY, FROM_DAY_KEY,
TO_DAY_KEY, RET_COD, STAMP_DTS)
WITH X AS (
SELECT SOURCE_COD
,TO_CHAR(SYSDATE,'YYYYMMDD') DAY_KEY
,MIN(DAY_KEY) FROM_DAY_KEY
,MAX(DAY_KEY) TO_DAY_KEY
FROM STA_EMPLOYEES1_STT
GROUP BY SOURCE_COD)
SELECT A.IO_COD
,X.SOURCE_COD
,X.DAY_KEY
,A.FROM_EDAY_KEY,A.TO_EDAY_KEY
,X.FROM_DAY_KEY,X.TO_DAY_KEY
,(CASE WHEN (A.FROM_EDAY_KEY+A.TO_EDAY_KEY=X.FROM_DAY_KEY+X.TO_DAY_KEY) THEN 'OK' ELSE 'NOT OK' END)
,SYSDATE
FROM STA_IODAY_CFT A
LEFT OUTER JOIN X ON (A.DAY_KEY = X.DAY_KEY)
WHERE A.IO_COD = 'employees1'
AND A.DAY_KEY = TO_CHAR(SYSDATE,'YYYYMMDD');
COMMIT;
9. Conclusion
We are at the end of this recipe. The configuration and the log tables are:
With only two tables, we have reached the control of the reference day of the source
data without ETL tools.
This is the philosophy of Micro ETL Foundation.
Email - massimo_cenci@yahoo.it
Blog (italian/english) - http://massimocenci.blogspot.it/