Driving Outcomes (Kirkpatrick Lvl 4) through Pragmatic Learning Governance

WWW.LEARNINGCAFE.COM.AU
@lrncafe
Sydney | Melbourne |Brisbane
Session Tips
• Apologies in advance for
cramming good speakers
and content in a 1 hr.
webinar
• Mute your microphone till
you are ready to speak.
• Use the chat window to
post your questions,
comments and insights.
• This is a free and frank
discussion. Diversity of
views is welcome.
• This session is being
recorded and will be
available at our website.
1
Practice
Leadership In
Organisational
Learning
Knowledge
Performance
Driving Outcomes (Kirkpatrick Lvl 4) through
Pragmatic Learning Governance
Sandhya Krishnan
General Manager,
Organisation Solutions.
RAC WA
Brendan Hill
Learning & Capability
Lead – Digital, nbn
Jeevan Joshi
Community & Research
LearningCafe
Thursday 28, March 2024, Sydney 12 pm | Singapore 10 am | India 7.30 am

Community of forward-looking professionals interested in
frank discussions about current issues and trends in the
science and art of organisational learning, capability
knowledge and collaboration; and its application at work.
◼ Practice leadership in organisational learning systems/ ecosystems.
◼ Community driven. Research based, free and frank discussions.
◼ Integration with Business KM, Productivity, Collaboration etc
Greg Palmer
Director, Leadership
and Learning, Deloitte
and others
https://bit.ly/lcweb0324z

◼ By designing programs and Connexgen
aims to create spaces, where the
community can connect, learn from one
another, and cultivate shared
perspectives.
3
Providing industrial strength
learning infrastructure and
expertise to Community
Learning & Not for Profits to
scale their efforts for greater
impact.
Sat 21 Sept 11 -12 am
Gordon Library Meeting Room 2,Sydney
https://bit.ly/cnxgen1

WHAT IS GOVERNANCE ?
◼ Governance encompasses the system by which an
organisation is controlled and operates, and the
mechanisms by which it, and its people, are held to
account.
◼ Brandon Hall POV
◼ Align Learning with Business
◼ Establish Clear Governance Structures and Processes
◼ Foster a Culture of Continuous Learning
◼ Foster a Culture of Continuous Learning
https://brandonhall.com/importance-of-learning-strategy-and-governance/#
Benefits
• Alignment with Business Goals
• Improved Efficiency and Effectiveness
• Enhanced Employee Engagement and Retention
• Improved Performance and Productivity
• Better Risk Management
• Increased Agility and Innovation
• Better Talent Management
• Enhanced Brand Reputation
◼ Alignment with Business Strategy
◼ Resource Optimisation
◼ Data-Driven Decision-Making

Brendan Hill
Learning & Capability
Lead – Digital, nbn

nbn-COMMERCIAL
KIRKPATRICK EVALUATION- OVERVIEW
1.Aligns Learning with Business Objectives
Ensures training supports organisational goals and strategy,
providing evidence that learning leads to desired business
outcomes (e.g., profitability, customer satisfaction).
2.Drives Accountability and Transparency
Tracks the impact of learning on performance, fostering
accountability by linking training results to tangible outcomes.
3.Informs Decision-Making
Data-driven insights help prioritise resources for high-impact
programs, enabling informed decisions on future L&D
investments.
4.Ensures Continuous Improvement
Evaluating results allows for program adjustments and ongoing
optimisation to ensure learning remains effective and aligned
with governance goals.
How Level 4 Evaluation Enables Governance

nbn-COMMERCIAL
Kirkpatrick Model – Defining roles & responsibilities
Role/Activity Responsible (R) Accountable (A) Consulted (C) Informed (I)
1. Set Business Goals Business Leaders Executive Sponsor L&D Team Employees
2. Agree on Business Changes
(ways of working, IT testing,
new processes, etc.)
Business Leaders Executive Sponsor
L&D Team,
IT Team
SMEs
Business Analysts
Employees
3. Define Learning Objectives L&D Team L&D Manager Business Leaders Employees
4. Design Training Program Instructional Designers L&D Manager Subject Matter Experts (SMEs) Employees
5. Deliver Training Trainers/Facilitators/SMEs L&D Manager Business Leaders Employees
6. Measure Business Impact
(Level 4)
Business Analysts L&D Manager Business Leaders Employees
7. Evaluate Learning
Outcomes (ROE)
L&D Team L&D Manager Business Leaders Employees
8. Continuous Improvement L&D Team L&D Manager Business Leaders Employee

nbn-COMMERCIAL
Level 4 in practice
A national commercial real estate company,
aimed to improve operational efficiency and
customer satisfaction through Generative AI
(Gen AI) tools. The goal: to achieve
Kirkpatrick Level 4 results, demonstrating a
direct impact on business outcomes like
profitability, efficiency, & customer
experience.
Background
Focus on driving measurable business outcomes
by aligning Gen AI training with key organisational
goals:
• Customer satisfaction through personalised
AI recommendations.
• Operational efficiency via automation.
• Profitability increases by streamlining
workflows
Objectives
1. Expectation Setting with Leaders: Clear
goals on profitability, efficiency, and
satisfaction were agreed with leadership
2. Defined KPIs: Measurable outcomes (CSAT,
operational efficiency, profitability) were
established upfront.
3. An Evaluation Plan: What activities need to
change in the workplace.
4. Ongoing Monitoring: Data collection post-
training to track impact and adjust strategies
5. Accountability: Regular check-ins with
leadership ensured alignment with strategic
business goals.
Governance Steps

nbn-COMMERCIAL
Level 4 in practice
Customer Satisfaction:
• AI-powered recommendations increased
CSAT by 15%, showing improved
customer interactions driven by AI usage.
Operational Efficiency:
• Automation reduced manual tasks by
30%, resulting in faster processes and
fewer errors. (More level 3)
Profitability:
• AI optimisation led to an 11% increase in
profitability within six months, due to
improved efficiency and customer
engagement.
Results
• The governance steps ensured that the
training programme was fully aligned with
business objectives, allowing the organisation
to track and measure its impact effectively.
• By focusing on well-defined KPIs up-front and
regularly reviewing progress with leadership,
we were to deliver on the agreed business
expectations, achieving a clear Return on
Expectations (ROE).
• The success of this initiative demonstrates
the power of combining skill development
with strong governance practices to ensure
that learning translates directly into tangible
business results
Conclusion

nbn-COMMERCIAL
Challenges
faced
• Ambitious timeline: Achieving results within six months
was difficult due to the scope of changes.
• Busy leaders: Competing priorities limited leadership’s
ability to fully engage and support the initiative.
• Stakeholder buy-in: Stakeholders needed to agree
upfront to a robust plan and take accountability.
Retroactive adjustments were not feasible.
• Pursuing additional results: Delivering extra outcomes in
a short timeframe required creative, sustained effort.
• Communication challenges: Maintaining effective
communication and engagement was tough with other
business priorities.
• Performance incentives: Incentives were introduced to
keep participants and leaders motivated.
• Gamification: A gamified training approach boosted
engagement through competition.
Lessons learnt from roll-out

nbn-COMMERCIAL
Making it
Happen
Jeevan Joshi
Community & Research
LearningCafe

PYRAMID FRAMEWORK WORKS WELL
13
https://www.processpark.com/index.cgi?sayfa=hizmetdetay&id=6&lang=EN
https://ub-ba.de/governance-key-success-factor-outsourcing-
engagements/

MANY WAYS TO STRUCTURE GOVERNANCE
14
https://elearning.qmul.ac.uk/about-us/groups-and-committees/governance-model/

CHALLENGES FOR KIRKPATRICK LEVEL 4 MEASUREMENT
15
• Attribution Complexity: It's hard to isolate the impact of
training from other factors affecting business outcomes.
• Data Challenges: Accessing relevant, consistent data is often
problematic.
• Defining Success: Agreeing on clear, measurable business
metrics can be difficult.
• Resource Constraints: Measurement requires significant
time, money, and expertise.
• Organizational Barriers: Lack of buy-in and focus on short-
term results hinder efforts.
• Impacted by Power Struggles

LEARNINGCAFE HACK
LEARNING GOVERNANCE CAN BE
SET UP TO INDICATE KIRKPATRICK
LVL 4 MEASUREMENT
MAYBE THE ONLY PRACTICAL
SUSTAINABLE AND ROBUST APPROACH
AVAILABLE TO L&D

ROUGH GUIDE FOR LEARNING GOVERNANCE
19
-Establish need
- Define the roadmap
- Set up a Governance
Committee/ Working Group
- Keep scope and process
light weight and tight
- Use 80/20 rule
- Run a tight ship.
- Keep discussion focussed
on big question
- Do not let the discussion
become tactical
- Admit lack of data
/resources
- Build consensus on forward
direction
- Introduce Data based
discussion
- Introduce Risk
Management
- Ask resources to be
prioritised and what should
be axed.
- Keep impacted stakeholder
about decisions made.
First Instance
Next Iteration
Setup

ALWAYS ASK THE RIGHT & BIG QUESTIONS IF DON’T HAVE THE ANSWERS
◼ IS L&D IS DELIVERING WHAT IT IS SUPPOSED TO DELIVER ? RIGHT CAPABILITY, RIGHT QUANTITY,
RIGHT TIME. IS L&D ADRESSING ORGANISATIONAL PRIORITIES ?
20

80/20 RULE
21
High
Business
Priority
Permanence of Capability
Low
High
High Priority
Low Permanence
High Priority
High Permanence
Low Priority
High Permanence
Low Priority
Low Permanence
Tactical/MVP Measures
Embed in Systems/Processes
Invest Strategically
Top 20%
Manage Tactically
Top 60%
Ignore
Bottom 20%

22
GET THE RIGHT
PEOPLE INVOLVED
FOR THE RIGHT
REASONS
• Seniority/Leaders
• Decision makers/ budget holders
• Passion for Employee Development/Learning
• Minimise gatekeepers
START SIMPLE BUT
RAISE THE BAR VERY
SIMPLE
• Scope sharp and small
• Recognise immaturity/ norming forming
FOCUSSED AND
RIGOUROUS
FOLLOW UP – NO
TIME WASTING
• Start with follow up and operational updates
• Plan to improve every time.
LEAVE OUT THE
LEARNING
LANGUAGE – TREAT
IT LIKE A BUSINESS
PROBLEM
• Focus on what are delivering and not how
• Use business language only. Try not to use learning buzz words –
“learning in the flow of work, ADDIE” etc.
• Use data where you have and can.

Case
Study
Sandhya Krishnan
General Manager, Organisation Solutions. RAC WA

Data Literacy @ RAC
A summary
Sandhya Krishnan

RAC Data Capability Strategy
Technology - a
single, rich and
personalised view
of the member
Deep, data
enabled member
knowledge
Analytics capability
that enables data
driven decision
modelling
Build and leverage data, digital and technology capabilities
In support of our Social Impact; Member Value and Experience; People and Commercial goals
Build and leverage our data, digital and technology capabilities
By end of FY25, RAC will have increased its organisational data capability
through commitment to four key areas –
People; Organisation; Systems & Processes and Technology & assets - to
ensure that data is used to achieve strategic outcomes in an appropriate,
ethical and sustainable way that aligns with our business values and
member needs

Purpose What will the Leaders/ Employees See Key Benefits
Build and secure the critical capabilities
required now and in the future with a
capability index of base +10% by 2025
• Data literacy embedded in employee lifecycle through recruitment, development, mobility and rewards
• Skills-based learning, development and internal opportunity suggestions for better career growth
• People programs drive development of our capability focus areas
• Focus on skills for growth and retention
• Greater engagement in career development for
prioritized segments
• Learning and development aligned to skills gaps in
capability focus areas
26
What is in Scope for FY24 Cost Breakdown Incl. Salary or BAU expenses Metrics to Measure Success
Data Literacy
Implement Data Literacy Program by:
• addressing data skills gaps in agreed priority business areas
• identifying opportunities to embed data literacy in the employee lifecycle
• Agree approach to skills assessment
• establishing baseline measure through RBL capability audit
Skills Framework
Agree on our approach to developing skills and leverage system capability to manage skills and
career pathways in line with People Tech Roadmap
• Support agreed pilots to encourage maximum consistency
Strategic Capabilities
• Align existing programs and processes to support building our capability focus areas
• Identify skill gaps from the roll out of XXX, in addition to leveraging engagement survey and
XXX data
<Redacted> • Business area Data Literacy Measures (TBD, dependent
on pilot group)
• RBL Capability Audit
• Consistency and usage of skills framework and
associated technology
Data Capability- Idea on a Page
Key Resources
<Redacted>

What is the Data Literacy program?
The program provides learning opportunities for all roles
across the business, aligned to the way we work with
data now and how we may work with data in the future.
The goal is to support all our people in using data to
make better decisions faster.

Data Literacy – Sample plan details
Education (Feb - April)
Curated learning paths for each persona with
minimum standard capabilities
Workshop events for learning of key concepts
Online learning through Workday for skill build
on supporting software tools
Learning integrated into and tracked through
development plans
Application (Feb - April)
Mapping priority data literacy skills to business
needs
» Topic - Data Analysis– Increased capability
in generating and sharing insights
» Events – ‘All In’ - high level launch of
program to all staff (@ end Feb)
» Embedding - leader led team activities
aligned to the priority topic.
Communication (Oct-Dec)
Build awareness and engagement through
events and activities
» Senior Team – Responsible for messaging
about the value of the upcoming program.
» Champions – Design events and activities
relevant to the business. Embed data
literacy messages into existing business
initiatives
» Leadership Group – Attend leadership
meetings to build common understanding
of data literacy in the business context.
Responsible for cascading messaging about
the value of the program.
» Change/Champions – Design and deliver All
staff communications and activities (Blogs /
Team Meetings etc)

Business area
FY24 Q1 FY24 Q2 FY24 Q3 FY24 Q4
Jul Aug Sept Oct Nov Dec Jan Feb Mar April May June
Area 1
Area 2
Area 3
Area 4
Area 5
Sample Rollout Timeline and Reporting
Phase 2 Data Quality
Phase 1 Discovery Phase 2 – Value of Data
Phase 1 Discovery and Design Phase 2 – Analysing Data
Phase 3 Analytics Solutions
Phase 3 Quality of Data
Phase 1 Discovery Phase 2 – Topic TBC
Phase 1 Discovery and Design Phase 2 – Skills for the Modern Finance Function
Sample monthly reporting
on progress in a business
area

DISCUSSION
What is the Kirkpatrick Level 4 measurement approach ?
How can Governance be set up ?

Driving Outcomes (Kirkpatrick Lvl 4) through Pragmatic Learning Governance

More Related Content

Similar to Driving Outcomes (Kirkpatrick Lvl 4) through Pragmatic Learning Governance

More from LearningCafe

Recently uploaded

Driving Outcomes (Kirkpatrick Lvl 4) through Pragmatic Learning Governance