CASE Excel vs. Database Solution

CASE: Excel vs. Database Solution
Work experience showcase
Mika Kulmala
2021-01-31 ver. 1.1

2021-01-31 ver 1.1 Mika Kulmala https://www.linkedin.com/in/mtkulmala/ Page 2 (16)
Purpose
●
I made a case study in 2013 comparing different
work flows with a very large product range, complex
IT system landscape, specific Customer
requirements and subjected to multi-national
regulations.
●
These results are comparable to a certain degree
on the high-level view to other complex situations,
e.g. SME import/export sector in the UK and the
EU27.
●
The purpose is to compare feasibility of different
work flows and to highlight the benefits of a properly
designed, executed and managed IT process.

Case Study 2013
Chemical and Pharmaceutical
eCatalogue creation process
●
I compared how to create
quarterly eCatalogues for
eProcurement (UN/EDIFACT and
NA/ASC X12 standards) using:
1)Existing manual Excel work
flow, in production
2)Mostly automated Access work
flow, demonstrated with a
prototype
3)Mostly automated live database
work flow, demonstrated as a
mature development idea
●
About the information use under NDA: I follow
it, even when it’s long expired. This study is a
generic view, where I have removed all the
company specific information and used numeric
approximates for data. Work flows are generic
in nature. The company situation and practices
have evolved greatly since 2013, so there is no
sensitive data or information to be found.
Master, Source
& Customer Data
Existing: Manual
Excel work flow
Prototyped: Mostly
automated Access
work flow
eCatalogue
(Products, prices,
descriptions, etc.)
Concept: Mostly
automated live DB
work flow

Background Information
eProcurement EDI transaction
and data chain to customer
●
Global company with multiple production
locations and international sales with
established customers.
●
EDI (EDIFACT and X12) requirements and
transfer file definitions from customers and
3rd
Party Service Providers.
●
Internal rules, reviewed by Legal dept.,
how to apply laws and regulations to
eProcurement processes and Customers.
●
Narrowed down to the eCatalogue only:
– 100 maintained eCatalogues for same
number of customers in EMEA region.
– 15 international eProcurement 3rd
Party
Service Providers.
– 2 different global ERPs: Chemicals and
Pharmaceuticals.
– An internal collaboration platform for
Master and Source data, Business
Intelligence reports and DB views.
– An external collaboration platform for
Customer files, prices, discounts and
requirements.
●
Company
EDI Process
Customer
EDI Process
Peer-to-Peer or
3rd
Party Service
Provider
Company
ERP
Customer
ERP
Company:
Offering products,
WebShop, eCatalogue,
PunchOut (peer-to-peer),
Order management, Status
updates, Customer Service,
Placing eInvoice,
Receivables, Customer
product and price sheet
maintenance, eProcurement
maintenance
Customer:
Browsing products,
Placing eOrder, Order
changes, Status
checks, Payments,
Product and price sheet
requirements,
eProcurement
requirements
UN/EDIFACT and NA/ASC X12

Chemicals and Pharmaceuticals
●
Chemicals and
Pharmaceuticals give a
prime example of highly
regulated product series
with a lot of data
dependencies.
●
Hundreds of thousands
products with a plenty of
Master Data and
supporting information,
unique data selection for
each region and country,
with complex legal and
other requirements.
Min. 12 Identifiers
and Databases
(e.g. IUPAC, CAS,
PubChem, etc.)
Drug Class,
Chemical Class,
Legal Classification
etc.
Regulations,
Distribution Laws,
Buyer Restrictions,
Black Lists, etc.
Perishable Cargo
Regulations,
Special Handling,
Dangerous Goods
Global, Regional
and National
Price Lists and
VAT levels
Customer
Requirements and
Restrictions, Prices
and Discounts
Product Data and
Information
(e.g. descriptions,
white papers, etc.)
Technical Data
Requirements
(e.g. EDI, sanity,
normalisation, etc.)

In Production: Excel Process
●
10 top level process steps
described in the process
guide.
●
30-70 sub-level process steps
depending on the individual
eCatalogue complexity.
●
Combined eCatalogues (both
Chemicals and
Pharmaceuticals) double the
number of process steps due
to two different sources.
●
Fastest single eCatalogue
10+30=40 process steps in
60 minutes.
●
Slowest combined eCatalogue
2x(10+70)=160 process steps
in 960 minutes (2 work days).
Start acceptance
process with
Customer
Two Company ERPs
Collaboration
Platforms
(Internal & External)
Master Data, DB
views, BI reports,
Price sheets
Reference files,
Customer prices
and discounts,
VAT and currency
Manual process in Excel
Data gathering ETL (data import)
Data sanitisation
Data queries
(VLOOKUP)
Data content
verification
Data format
validation
EDI file export EDI file validation
Publish
eCatalogue to
eProcurement
Platform

Manual Data Validation in Excel
●
Notes:
- Multiple different unique
identifiers used in different places
(due to Customer requirements
and difference between Chemicals
and Pharmaceuticals).
- Common input and output file
formats: IDoc, CSV, XML, XLS.
- Excel has known, undocumented
ETL import limitations.
●
ETL input validation for all files: file
format, character encoding,
number and position of columns,
decimal separator, thousand
separator, SI/imperial units,
currency, date format, unique
identifier, etc. (depending on file)
●
Customer data input validation:
product ranges, date ranges,
prices, discounts
●
Validations to a completed Excel
file before exporting to EDI file:
data sanity, price change-% from
previous export, product range
changes from previous export +
all ETL input validation checks
●
Validation for final EDI export file:
file format, named elements,
header and footer, character
encoding, EOL and EOF format,
decimal separator, thousand
separator, file naming format
●
Escalations: Master Data and
ERP source data related
(technical) issues raised with Data
Administrators. Customer data
related (business) issues raised
with Sales (who communicate
with the Customer).

Excel Process Takt Times
F
a
s
t
e
s
t
s
i
n
g
l
e
S
l
o
w
e
s
t
s
i
n
g
l
e
A
v
e
r
a
g
e
s
i
n
g
l
e
F
a
s
t
e
s
t
d
o
u
b
l
e
S
l
o
w
e
s
t
d
o
u
b
l
e
A
v
e
r
a
g
e
d
o
u
b
l
e
T
o
t
a
l
a
v
e
r
a
g
e
0
200
400
600
800
1000
1200
0
1
2
3
4
5
6
7
40
80
60
80
160
120
100
60
480
270
120
960
540
510
1.5
6
4.5
1.5
6
4.5
5.1
Process steps Production minutes
Min/step (right axis)
●
Takt Times, here minutes per process step,
are based on the number of process steps
and the estimated production times.
●
Notes:
– Separately timed process steps would
have allowed more precise
estimations, but they were deemed
unnecessary in a high-level case
study.
– Master Data (e.g. monthly price sheets
for each product category) preparation
and input time are not included.
– Customer specific data preparation
and input time are included.
– The complexity of process increases
with the number of steps due to larger
amounts of data to be processed.
– Averages are only based on the
calculation. They are not weighed by
any parameters.
●
Observation: 2x process steps = 8x time in
production, demonstrating the impact of the
increased complexity.

Prototyped: Access Process
●
Automating all ETL import processes:
– ETL rules written and implemented for
each new file only once. Updated when
there are changes to files or metadata.
– On-the-fly the same normalisation and
validation steps as in manual process.
– Summary of new, changed, removed or
potentially corrupted data for each step
(evidence file).
●
Automating eCatalogue creation process:
– All queries, data enrichment and
normalisation automated.
– Shareable report (evidence file) of new,
changed, removed or potentially
corrupted data.
– Special attention to Customer data,
changes and requirements.
●
Process time 15-30 minutes leaving more
time for EDI file validation, Customer
communication and other tasks.
Start acceptance
process with
Customer
Collaboration
Platforms
(Internal & External)
Reference files,
Customer prices
and discounts,
VAT and currency
Manual process steps in Access
Import Customer
Prices & Discounts
Verify changes to
Customer data
Run eCatalogue
creation process
Validated EDI
file export
Confirm EDI
file validation
results & Save
evidence report
Publish
eCatalogue to
eProcurement
Platform
Two Company ERPs
Master Data, DB
views, BI reports,
Price sheets

Automatic Data Validation in Access
●
Notes:
- Multiple different unique
identifiers used in different
places (same as in Excel).
- Common input and output file
formats: IDoc, CSV, XML, XLS.
- Access shares some known,
undocumented ETL import
limitations with Excel.
- ETL and EDI rules need to be
defined for each input and
output file, but only once.
●
ETL input validation for all files:
same as in Excel, but
automated. Only known ETL
import issues need manual file
manipulation before importing.
●
Customer data input validation:
same as in Excel, but
automated.
●
Validations to a completed
Excel file before exporting to
EDI file: same as in Excel, but
automated.
●
Validation for final EDI export
file: same as in Excel, but
automated.
●
All validation steps: Summary
of new, changed, removed or
potentially corrupted data for
each step, which can be saved
as an evidence file.
●
Escalations: Same as in Excel.

Access Process Takt Times
F
a
s
t
e
s
t
s
i
n
g
l
e
S
l
o
w
e
s
t
s
i
n
g
l
e
A
v
e
r
a
g
e
s
i
n
g
l
e
F
a
s
t
e
s
t
d
o
u
b
l
e
S
l
o
w
e
s
t
d
o
u
b
l
e
A
v
e
r
a
g
e
d
o
u
b
l
e
T
o
t
a
l
a
v
e
r
a
g
e
0
5
10
15
20
25
30
35
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
13 13 13 13 13 13 13
15
30
22.5
15
30
22.5 22.5
0.9
0.4
0.6
0.9
0.4
0.6 0.6
Process steps Production minutes
Min/step (right axis)
●
Takt Times, here minutes per process step, are
based on the number of process steps and the
estimated production times.
●
Notes:
– Master Data (e.g. monthly price sheets for
each product category) preparation and
input time are not included.
– Customer specific data preparation and
input time are included.
– The complexity of process stays the same
regardless the size and complexity of the
eCatalogue.
– The time consuming part is in creating
rules and queries for new eCatalogues and
data sources. These tasks need to be done
only once (as opposed to Excel solution,
where all tasks are always repetitive).
– Averages are only based on the
calculation. They are not weighed by any
parameters.
●
Observation: eCatalogue complexity has no
significant change in production time. The
biggest time of production is spent in confirming
the results of automatic EDI file export and
storing the evidence files.

Concept: Live Database Process
●
Scheduling data queries:
– Direct queries from ERP and internal
collaboration platform.
– On-the-fly all the same normalisation
and validation steps as in the Access
process.
– Emailed summary of new, changed,
removed or potentially corrupted data.
●
Automating eCatalogue creation process:
– Emailed work status and reports.
– Rest same as in the Access process.
●
Master and Source Data processes
streamlined. No need for slow change
requests for BI reports and other data.
●
Process time 15-30 minutes leaving more
time for EDI file validation, Customer
communication and other tasks.
●
Takt Times not calculated as they are in
practice the same (or better) as in Access
process.
Start acceptance
process with
Customer
Internal
Collaboration
Platform
Customer prices
and discounts
Manual process steps in live database
Import Customer
Prices & Discounts
Verify changes to
Customer data
Run eCatalogue
creation process
Validated EDI
file export
Confirm EDI
file validation
results & Save
evidence report
Publish
eCatalogue to
eProcurement
Platform
Two Company
ERPs
Scheduled
data queries
External
Collaboration
Platform
Scheduled
data queries

Comparison of Solutions
In Production: Excel Solution Prototyped: Access Solution Concept: Live DB Solution
Steps: 10 top + 30-70 sub 3 top + 10 sub 3 top + 10 sub (estimated)
Time: 60-960 minutes per eCatalogue 15-30 minutes per eCatalogue 15-30 minutes per eCatalogue
Pros:
+
- Tested method in use
- Every step is monitored personally (also a Con:
danger of micro-management)
- Basic method can be copied and adjusted to
new Customer and 3rd
Party Service Provider
requirements with a relative ease
- Advanced user can recycle already validated
price sheets and product groups to save
production time (also a Con: possibility of copying
existing, undetected errors)
- All Master and Source data are imported only
once a month
- All ETL import and data validation steps are
automated (disclaimer: known ETL limitations)
- Automatic report of new, changed, removed and
potentially corrupted data on each import
- One tool for all possible Customer, product
range and EDI file combinations
- Always accurate, sanitised and normalised data
with single point of search
- Human errors minimised and roll-backs enabled
- Possibility to create ad-hoc queries and
reconcile work process with evidence files
- Semi-automatic work and status reporting
- Easy to change and add new Customer and 3rd
Party Service Provider requirements
- Low entry level for new personnel
- Greatly improved SLA compliance
- A proper, defined, managed and
documented IT process
- All Master and Source data imports
are scheduled with direct data
queries, faster than monthly imports in
Access
- Emailed report of new, changed,
removed and potentially corrupted
data on each scheduled import
- No need to produce dedicated BI
reports, DB views and other source
files for eCatalogue, freeing up
resources
- Can utilise existing database tools,
no need to introduce nor maintain low-
tech Access, no ETL limitations
- All remaining Pros of Access
Solution
Cons:
–
- All data from all source files imported every time
- Manual work reporting, lack of evidence files
- Long production time due to repetitive tasks (one
VLOOKUP can take several minutes to complete),
requires a lot of local processing power for Excel
- Undocumented Excel ETL limitations
- Until Excel 2016 limitation of 1 million rows
(workaround split files, increasing mistake risk)
- Quick error cumulation: a mistake at early stage
can lead to a complete reprocess to correct errors
- Prone to human error due to large number of
process steps and different working files
- Even with Excel macros a large number of
validations is still a lot of manual work
- Teaching new personnel is time consuming and
complex process
- Access database needs development,
maintenance and SQL skills (also a Pro: can be
dedicated to an Application Manager as a proper,
defined, managed and documented IT process)
- Requires more precise documentation (also a
Pro: more carefully documented)
- Higher resource costs to start (also a Pro:
comparably low running costs)
- Access shares the undocumented ETL
limitations with Excel (workaround manual
preprocessing and data conversion)
- Requires dedicated ERP and DB
user accounts with sufficient rights for
scheduled queries (also a Pro: better
monitored and separated from normal
user accounts)
- Requires ERP and/or DB user
accounts for end-users (also a Pro:
better monitored)
- Requires established, documented
and maintained ERP and DB
connections (also a Pro: better
monitored)
- All remaining Cons of Access
solution

●
Benefits are even more obvious when aggregated on an annual level:
– With Excel: 100 single eCatalogues x average production time
270 mins x 4 quarterly updates = 108.000 mins = 1.800 hrs = 225
working days (8 hrs) – estimated with the lowest (best) average.
– With Access/DB: 100 eCatalogues x maximum production time 30
mins x 4 quarterly updates = 12.000 mins = 200 hrs = 25 working
days (8 hrs) – estimated with the highest (worst) average. The
complexity or size of eCatalogue has no longer any significant
impact on the duration.
– Result with Access/DB: 200 working days of resources freed.
– Even better: After piloting in the EMEA region the Access/DB
solution could have been introduced to all other regions creating
global savings and introducing more robust, documented IT
processes.
●
These benefits are not directly applicable to other businesses but they
do prove the point: repetitive business processes with Excel should
be subjected to a business analysis or a feasibility study.
Case Study 2013 Outcome

If an organisation is using Excel
to do all this:
●
Import large amounts of data
from different sources on weekly
basis
●
Cross-reference data between
different sources
●
Validate against various
requirements and regulations
●
Implement country and customer
specific prices, VAT and currency
●
Publish hundreds or more files in
strictly regulated manner with a
tight schedule
●
Repeating work with increased
workload spikes around month
and quarter ends
Final Conclusions
That organisation should consider:
●
Retiring Excel from database
duties
●
Developing proper, managed IT
processes built around a database
or incorporated to an ERP
●
Allocating resources to the startup
and change periods
So they could benefit from:
●
Lowered risks
●
Increased and reported risk
mitigation and contingency
●
Decreased human mistakes and
increased job satisfaction
●
Dramatically improved production
times with superior, repeatable
data quality
Disclaimer: Every business and IT landscape is unique. Any substantial IT and
process changes should be first subjected to a business analysis or a feasibility study.

Thank You for Reading!
●
I hope you found this presentation interesting.
You can always contact me for the original
files and/or more information. Comments and
constructive critique are welcome, too.
●
This showcase is the 11th in the series of my
work examples. They are easy to recognise
from the banana for scale.
●
I believe in the power of the education, so I
am happy to share my knowledge and
experience. In addition I am trying to attract
employer interest and get a job :-)
https://www.linkedin.com/in/mtkulmala/
Credits and Licensing: Free to use!
●
Revision 1.1 2021-01-31
●
Banana photo, presentation lay-out, text and
drawings (except for CC buttons): myself
●
Programs: LibreOffice
●
Typefaces: Liberation font family
●
License: Creative Commons Attribution-
ShareAlike 4.0 International (CC BY-SA 4.0),
also source of CC buttons.
●
This means: You can use, edit and share my
presentation or parts of it freely (without a
charge) as long as you give credit for my work
and share your work forward in the same way.
●
LinkedIn trademark and copyrights belong to
LinkedIn Corporation.
https://creativecommons.org/licenses/by-sa/4.0/

CASE Excel vs. Database Solution

Recommended

Recommended

More Related Content

Similar to CASE Excel vs. Database Solution

Similar to CASE Excel vs. Database Solution (20)

Recently uploaded

Recently uploaded (20)

CASE Excel vs. Database Solution