1. The Chronicles of APEX:
Experiences from the PM Development Frontier
Presented by
Ron Johnson, Chief of Ames’ Systems Management Office
Claire Smith, APEX Deputy Program Manager
February 2007
1

2. Topics
• APEX Program Overview
• Top Ten Lessons-Being-Learned So Far
• Where We Go From Here
2

3. Ames Project Excellence (APEX) Development Program
One year development program designed to demonstrate a measurable
increase in the participants’ ability to execute project management and
systems engineering
Two integrated pathways
APEX PM (10 project manager participants) and APEX SE (10 systems engineering participants)
Participants selected by Center Management because of their leadership positions in critical ARC
projects, examples: CEV, LCROSS, SOFIA, Kepler, Air Traffic, NGATS, Columbia (Super Computing)
Participants spend approximately 25% of their time on APEX activities (paid by assigned project/home
organization)
Development areas for both pathways
Technical Development: Agency-unique APEX Online tool created for PM and SE core competency
technical assessment and automatic creation of Individual Development Plan (IDP)
Leadership Development: APPEL funded 4D Leadership model (workshop, assessments, and monthly
coaching)
3

4. Profiles
Of the
Participants
Focus of
APEX Pilot
Year
4

5. PM and SE Competencies used as foundation for the Program
- Consistent with APPEL model
Project Managers Systems Engineers
1.1 New Business Development 1.1 Trade Studies
1.2 Bid and Proposal 1.2 Proposal Development
1.3 Requirements Development and Management 2.1 Baseline System Architecture
1.4 Project Planning and Scheduling 2.2 Alternative Design Concept Development
1.5 Acquisition Strategies 3.1 Requirements Development
1.6 Risk Management 3.2 Requirements Management
2.1 Budget and Full Cost Management 4.1 Margin Determination
3.1 Systems Engineering 4.2 Technical Resource Allocation
3.2 Design and Development 5.1 System Design
3.3 Technology Assessment, Development, and Transfer 5.2 System Analysis
3.4 Contract Management 5.3 Uncertainty Analysis
3.5 Stakeholder Management 6.1 System Verification
4.1 Project Control 6.2 System Validation
4.2 Project Review and Evaluation 7.1 Risk Identification and Assessment
5.1 System Safety 7.2 Risk Mitigation
5.2 Mission Assurance 7.3 Risk Tracking
6.1 Agency Structure and Mission 8.1 Review Planning
6.2 NASA and Ames PM Procedures 8.2 Review Implementation and Support
6.3 International Agreements 9.1 Baselining
7.1 Team Dynamics and Management 9.2 Change Control
7.2 Communication 10.1 Systems Engineering Planning
7.3 Problem Solving 10.2 SEMP Implementation
8.1 Knowledge Management 11.1 Team Dynamics and Management
8.2 Knowledge Sharing 11.2 Communication
11.3 Problem Solving
5

6. Critical Tasks identified for each Competency
and Profile
Sample
6

7. Key Program Components
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Team Learning
One-On-One
Mentoring APEX Development
Council
IDP
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Classroom On-The-Job
Development APEX Management
Training
Assignments Staff
7

8. Process for Demonstrating a Measurable Increase in
Capability
Establish
Establish Individual
Individual Achieving Next
Achieving Next
Beginning
Beginning
Assessment Development
Development Higher Capability
Higher Capability
Assessment Capability Level
Capability Level
4 Month Rev.
(APEX Online) (for each Critical
Plan w/ Milestones
Plan w/ Milestones Level - -Graduation
Level Graduation
(APEX Online) (for each Critical
Task) 8 Month Rev.
Task)
- Participant - “Developing” - One on One Mentoring - Evaluated by Development
- Supervisor - “Functional” - Team Mentoring Council based on APEX
- Mentor - “Proficient” - On the Job Training achievements
- Training Classes
- Knowledge Sharing
All focused on improving capability
in a minimum of four critical tasks
8

9. Process for Demonstrating a Measurable Increase in
Capability
Establish
Establish Individual
Individual Achieving Next
Achieving Next
Beginning
Beginning
Assessment Development
Development Higher Capability
Higher Capability
Assessment Capability Level
Capability Level
4 Month Rev.
(APEX Online) (for each Critical
Plan w/ Milestones
Plan w/ Milestones Level - -Graduation
Level Graduation
(APEX Online) (for each Critical
Task) 8 Month Rev.
Task)
- Participant - “Developing” - One on One Mentoring - Evaluated by Development
- Supervisor - “Functional” - Team Mentoring Council based on APEX
- Mentor - “Proficient” - On the Job Training achievements
- Training Classes
- Knowledge Sharing Proficient
Functional
APEX Capability Levels Functional
APEX Capability Levels Developing
• Developing: Limited/No knowledge. Limited/No experience in executing task.
• Developing: Limited/No knowledge. Limited/No experience in executing task.
• Functional: Can define core concepts and practices, and has executed task in routine
situations underCan defineof expert practitioner.
• Functional: guidance core concepts and practices, and has executed task in routine
situations under guidance of expert practitioner.
• Proficient: Breadth of knowledge, skills and experience applied to executing task in
complex environments.of knowledge, skills and experience applied to executing task in
• Proficient: Breadth
complex environments.
9

10. Example IDP from a Systems Engineer
10

11. APEX Graduation Documentation
Participant’s Name:
APEX Pathway (PM or SE):
APEX Profile (Selected before beginning of APEX Online Assessment):
Immediate Supervisor:
One-On-One Mentor:
1. How have your APEX program activities (Mentoring, OJT, Courses, etc) improved your technical and leadership capabilities?
2. Please complete the following chart. Format allows you to be as detailed as necessary:
Please provide a concise and detailed (as necessary)
IDP Objective* description of how you successfully satisfied or
APEX Focused *Enter Proficient exceed ed both the knowledge and performance criteria
Tasks Ready if Applicable associated with this objective
1
2
3
4
Participant Signature__________________________ Date:__/__/__
One-On-One Mentor:
Do you confirm that the participant has satisfied the APEX graduation criteria, I.e. accomplished the IDP Objectives listed above?
__ Yes __No. Explain:
What are your recommendations for the participant's future learning and development?
Mentor Signature: __________________________ Date: __/__/__ 11

12. APEX 12 Month Timeline
Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb
Assessment
Individual
Living document
Development Plan
Team Mentoring
Monthly
One-on-One Mentoring
OJT and Mentor-Assigned
Tasks
Leadership Development
and Courses/Workshops
Milestone Reviews 4-month 8-month Final
12

13. Top Ten Lessons-Being-Learned
so far
13

14. 1. Program Leadership Needs To Combine Technical and
Learning Expertise
Pilot Year : :APEX co-managed/led by Systems Management Office
Pilot Year APEX co-managed/led by Systems Management Office
and Workforce Development Branch
and Workforce Development Branch
Key Strength: Combined best practices of both disciplines.
Key Strength: Combined best practices of both disciplines.
Key Challenges: Differences in language, perspectives and approaches
Key Challenges: Differences in language, perspectives and approaches
to technical development, and complications from splitting budgets (vs.
to technical development, and complications from splitting budgets (vs.
embedded in one organization).
embedded in one organization).
Recommendation: Anticipate difficulties. Focus on collaboration vs.
Recommendation: Anticipate difficulties. Focus on collaboration vs.
and commitment to shared vision.
and commitment to shared vision.
14

15. 2. Learning and Development is a Voluntary
Activity, or
You Can Lead Them To Water But You Can’t
Make Them Drink
Pilot Year : :Candidates Nominated & Participants Selected by Center
Pilot Year Candidates Nominated & Participants Selected by Center
Senior Leadership
Senior Leadership
Key Strength: Participants were project practitioners working on
Key Strength: Participants were project practitioners working on
mission critical programs and projects. Senior leadership nomination
mission critical programs and projects. Senior leadership nomination
provided status and supported importance of program.
provided status and supported importance of program.
Key Challenge: Participants felt “conscripted” into program. .Impacted
Key Challenge: Participants felt “conscripted” into program Impacted
attachment and commitment.
attachment and commitment.
Recommendation: Nomination Process should include Candidate
Recommendation: Nomination Process should include Candidate
Application indicating interest, commitment and self-defined
Application indicating interest, commitment and self-defined
development needs and objectives.
development needs and objectives.
15

16. 3. Integrate Technical and Leadership Development In One
Program
Pilot Year : :Participants required to demonstrate increase in capability
Pilot Year Participants required to demonstrate increase in capability
in both technical and leadership* skills.
in both technical and leadership* skills.
**Utilizing 4D Leadership Assessments, Workshop and Monthly Coaching.
Utilizing 4D Leadership Assessments, Workshop and Monthly Coaching.
Key Strength: Leadership component gave participants opportunity to
Key Strength: Leadership component gave participants opportunity to
address all aspects of aasuccessful project, e.g. people skills.
address all aspects of successful project, e.g. people skills.
Key Challenge: leadership development activities were “one more
Key Challenge: leadership development activities were “one more
thing” added to participant’s plate. Note: Resistance decreased as value of coaching
thing” added to participant’s plate. Note: Resistance decreased as value of coaching
was proven.
was proven.
Recommendation: Integrate technical and leadership components into
Recommendation: Integrate technical and leadership components into
any PM and/or SE development program.
any PM and/or SE development program.
16

17. 4. Focus Technical Development On Increasing
Participants’ Ability to Execute PM and/or SE.
Pilot Year : :APEX Competency Matrices (PM and SE) Defined Specific
Pilot Year APEX Competency Matrices (PM and SE) Defined Specific
“Critical Tasks”* Associated with Each Competency.
“Critical Tasks”* Associated with Each Competency.
Key Strength: Defined critical tasks offered specific guidance on what aa
Key Strength: Defined critical tasks offered specific guidance on what
PM or SE is supposed to do.
PM or SE is supposed to do.
Key Challenges: Can only focus on 1-2 tasks* vs. all tasks, i.e. aafew trees,
Key Challenges: Can only focus on 1-2 tasks* vs. all tasks, i.e. few trees,
not the forest.
not the forest. **Necessary But Not Sufficient
Necessary But Not Sufficient
Recommendation: Each center should prioritize competencies based on
Recommendation: Each center should prioritize competencies based on
mission, and define critical tasks for each, i.e. actionable items.
mission, and define critical tasks for each, i.e. actionable items.
17

18. 5. Pave A Clear Development Pathway
A Map From Here To There
Pilot Year : :APEX Assessment Process Defined Beginning Capability
Pilot Year APEX Assessment Process Defined Beginning Capability
Levels for each Critical Task. APEX Individual Development Plan
Levels for each Critical Task. APEX Individual Development Plan
(IDP) Defined Development Objective. .
(IDP) Defined Development Objective
Key Strength: Provided aamodel for observing and/or measuring actual
Key Strength: Provided model for observing and/or measuring actual
learning product,i.e. increased capability.
learning product,i.e. increased capability.
Key Challenges: Pilot year definitions of capability levels too vague,
Key Challenges: Pilot year definitions of capability levels too vague,
and, in some cases, unrealistic.
and, in some cases, unrealistic.
Recommendation: Be willing to experiment with metric to measure
Recommendation: Be willing to experiment with metric to measure
development. Define stepping stones (improvements) that are
development. Define stepping stones (improvements) that are
attainable and traced to Center and Agency priorities.
attainable and traced to Center and Agency priorities.
18

19. 6. Participants’ Program/Project Assignments The Most
Critical Factor Promoting Actual Learning and
Development
Pilot Year : :Participants’ IDPs focused on “OJT” activities tied to
Pilot Year Participants’ IDPs focused on “OJT” activities tied to
existing project work. Where necessary, OJT assignment on external
existing project work. Where necessary, OJT assignment on external
project.
project.
Key Strength: APEX activities (IDP, mentoring, etc) tailored and
Key Strength: APEX activities (IDP, mentoring, etc) tailored and
applied to participants’ current project assignments and responsibilities
applied to participants’ current project assignments and responsibilities
increasing chances of project success.
increasing chances of project success.
Key Challenge: Difficult (almost impossible) to take participants outside
Key Challenge: Difficult (almost impossible) to take participants outside
of project to develop an expertise.
of project to develop an expertise.
Recommendation: Tailor program, as much as possible, to participants
Recommendation: Tailor program, as much as possible, to participants
current project assignments.
current project assignments.
19

20. 7. Mentoring is Second Most Critical Factor
Influencing Actual Learning and Development.
Pilot Year : :Participants matched with one-On-One Mentor and Team-
Pilot Year Participants matched with one-On-One Mentor and Team-
based mentoring (PMs and SEs). Mentors and participants trained in
based mentoring (PMs and SEs). Mentors and participants trained in
mentoring partnerships. Mentoring Agreements initiated relationship,
mentoring partnerships. Mentoring Agreements initiated relationship,
set direction.
set direction.
Key Strength: Individual mentoring focused on participant’s immediate
Key Strength: Individual mentoring focused on participant’s immediate
issues and executing IDP. Team mentoring helped build knowledge
issues and executing IDP. Team mentoring helped build knowledge
sharing network i.e. “community of practice”
sharing network i.e. “community of practice”
Key Challenges: Making appropriate matches (changed two mentors).
Key Challenges: Making appropriate matches (changed two mentors).
Mentor and protégé’s work commitments interfered with partnership.
Mentor and protégé’s work commitments interfered with partnership.
Recommendation: Define criteria for mentors. Provide mentor training.
Recommendation: Define criteria for mentors. Provide mentor training.
Involve mentors in reviews.
Involve mentors in reviews.
20

21. 8. A Healthy Program Has Regular Check-Ups
Pilot Year: Program activities included IDP reviews at 44and 88months,
Pilot Year: Program activities included IDP reviews at and months,
regular “House Calls” on participants and mentors and periodic brown-
regular “House Calls” on participants and mentors and periodic brown-
bags to discuss program issues.
bags to discuss program issues.
Key Strength: Activities provided opportunity to any required course
Key Strength: Activities provided opportunity to any required course
corrections/improvements to program and/or participants’ IDPs.
corrections/improvements to program and/or participants’ IDPs.
Activities increased sense of ownership of and commitment to program
Activities increased sense of ownership of and commitment to program
among participants and mentors.
among participants and mentors.
Key Challenge: Multiple, and sometimes conflicting needs and
Key Challenge: Multiple, and sometimes conflicting needs and
approaches to defining success, how to review progress, etc.
approaches to defining success, how to review progress, etc.
Recommendation: Design and schedule reviews throughout program.
Recommendation: Design and schedule reviews throughout program.
Clarify expectations regarding focus and scope of reviews. Involve
Clarify expectations regarding focus and scope of reviews. Involve
participants and mentors in program decision-making, including review
participants and mentors in program decision-making, including review
design.
design.
21

22. 9. Developing A Community Of Project Practitioners* is
Easier Said Than Done *
Vehicle for generating and sharing practical knowledge, lessons-learned and for fostering innovation.
Pilot Year : :Activities designed to develop Community Of Practice, e.g.
Pilot Year Activities designed to develop Community Of Practice, e.g.
brown bags, 8th. Month review and team-based mentoring.
brown bags, 8th. Month review and team-based mentoring.
Key Strength: Participants and mentors have voice in program design
Key Strength: Participants and mentors have voice in program design
and implementation = increased sense of ownership and commitment.
and implementation = increased sense of ownership and commitment.
Spontaneous eruptions of informal get-togethers.
Spontaneous eruptions of informal get-togethers.
Key Challenge: Developing consensus around definition of COP, how
Key Challenge: Developing consensus around definition of COP, how
to develop it and apply it to Agency and Center mission. Value of COP
to develop it and apply it to Agency and Center mission. Value of COP
cannot be demonstrated, can only be discovered.
cannot be demonstrated, can only be discovered.
Recommendation: Define clear objectives for COP and trace objectives
Recommendation: Define clear objectives for COP and trace objectives
to mission. Try aalot of different things!
to mission. Try lot of different things!
22

23. 10. First Build A Prototype, Then A Program
Pilot Year : :APEX Pilot year defined as “experiment”, aaprototype to
Pilot Year APEX Pilot year defined as “experiment”, prototype to
test and discover best methods for implementing learning and
test and discover best methods for implementing learning and
development activities resulting in measurable improvements.
development activities resulting in measurable improvements.
Key Strength: Actual experience and practice leads to improved,
Key Strength: Actual experience and practice leads to improved,
relevant and effective design and execution. Everyone is involved in the
relevant and effective design and execution. Everyone is involved in the
process of improvement, especially users.
process of improvement, especially users.
Key Challenges: Last 99things mentioned --plus 100 others!
Key Challenges: Last things mentioned plus 100 others!
Recommendation: Define Pilot year as prototype, test and improve
Recommendation: Define Pilot year as prototype, test and improve
design in real-time as you proceed (don’t wait until pilot year is over).
design in real-time as you proceed (don’t wait until pilot year is over).
Use participants and mentors experience, not theories, as guide for
Use participants and mentors experience, not theories, as guide for
ongoing improvements.
ongoing improvements.
23

24. Where We Go From Here
Graduate our first class
Continue to support APEXers as the core of our growing
community of practice
– Continued access to assessment tool and mentors, assist in selection of
new APEXers and become their mentors, host knowledge sharing sessions
Capture final lessons from participants, mentors, Development
Council, management staff, other stakeholders
Prioritize and make improvements to selected program elements
– Include participants and mentors in design improvements
Plan for the next APEX class
Continue to share our experiences and make available our
products and tools to APPEL and other Centers
24