Neal Fultz, Principal Consultant, njnm Consulting The California Cloud Workforce is an initiative in LA-area community colleges to develop the skills for future employment in Cloud and DevOps roles, spearheaded by Santa Monica College. Because there are more than 20 colleges participating and because the technology and required skills evolves rapidly, we have developed an NLP ensemble using federal data to identify occupational specializations in Cloud Computing, and the relevant coursework across many different institutions. Training and using the model consisted of several phases: * Extracting occupational data from the O*NET system and curricula data from Course Outlines of Record * Creating component models using DistilBERT, traditional NLP topic models, and the Bloom taxonomy of educational objectives * Ensembling the component models using PaCMAP * Deployment, and aggregating and visualizing results Using PaCMAP and DistilBERT produced a more parsimonious model that can leverage both transformers architecture and domain specific knowledge, and can be calibrated for Cloud Computing or other programs, and is so easy to manage that students deploy it themselves as part of their coursework.

1. PaCMAP Ensembles
Neal Fultz
neal@njnm.co
bit.ly/3vwx0W3

2. Executive Summary
● LACC system needed a way to identify
curricula gaps in Cloud Computing program
● By applying BERT, PaCMAP and elbow
grease, can find the gaps.

3. Background: The Client
● CA Cloud Workforce Consortia (LACC)
● > 2,000 annual openings in LA County
● “industry standard skills to understand and
develop applications for the cloud”

4. Background: The Problem
● “Cloud classes” lag behind “Cloud jobs”
● More generally: how do we figure out if /
where there is a mismatch between what
industry needs and what is taught in
classrooms?

5. Data: Curriculum
● Course Outline of Record
○ Tentative schedule,
assignments, textbook
○ Course Objectives
○ Student Learning
Outcomes
● Programs are
sets of courses

6. Data: Jobs
● O*Net
○ by Dept of Labor
○ CC 4.0
○ Tasks
○ Work Activities
○ Wages & Growth

7. Idea: “Skill Space”
Soft Technical

8. “Interpretation”
● “Closeness”
● “Matches”
● “Misses”
Soft Technical

9. Implementation
1. Stock NN + fine tune
2. Dimensionality Reduction
3. Hyperparameter tuning
4. Score & Aggregate

10. Hard Mode
Because the client has free devops in the form of
students, they wanted to make downstream
applications a class project.
=> Therefore, can’t use anything students can’t
access or figure out in final deliverable.

11. Implementation pt 1: NN
● DistilBERT, a distilled version of BERT:
smaller, faster, cheaper and lighter. Sanh,
Debut, Julien, Wolf. Neurips 2019.
● “40% smaller, 60% faster, that retains 97% of
the language understanding capabilities.”
● Runs comfortably on typically student
workstation or in Collab. GPU optional.
“40% smaller” is still
768-dimensional

12. Implementation pt 2: DR
● Uniform Pairwise Controlled
Manifold Approximation Projection
○ Multi stage optimization with “Far neighbors”
○ Review paper is extremely good

14. Implementation Pt 2
● “Bleeding edge” - multiple breaking changes
during engagement, non-standard interfaces,
“interesting” defaults, etc
○ But devs @ Duke very responsive
● Based on spot checking, /very/ good at
consolidating redundant information in this
type of data set

15. Implementation Pt2
● Issue: Only “as good” as inputs.
● Solution: Leverage domain knowledge
○ Bloom’s Taxonomy
● Solution: go even wider and let PaCMAP strip
out redundancies

16. Implementation Pt 2

17. Implementation Pt 2
Level % Job Tasks
creating 16.3
evaluating 12.8
analyzing 7.9
applying 10.3
understanding 2.8
remembering 4.7

18. PaCMAP ensemble
DistilBERT
Bag of
Words
Bloom’s
Taxonomy
PaCMAP
Ensemble
Embedding

19. Implementation pt 2
● PaCMAP ensemble provides reasonable and
structured way to blend together three
different NLP models
● Have to deal with extra complexity (as with all
ensembles)

20. Implementation Pt 2

21. Implementation pt 3: Tune
● Need to tune all component models +
PaCMAP
● Choose a good loss / metric:
○ “Stress”
○ “K-fold Stress”
○ “K-fold Spearman Stress”

22. ● Choosing # of dimensions
○ In past, would use scree plots / intuition
○ Use Gavish & Donaho instead (270)
○ NB: that’s under linearity, PaCMAP can do as
good with fewer, treat as hyperparameter
Implementation pt 3: Tune

23. Implementation pt 4: Scoring
Now have this thing:
Note mismatch between what that is and what
the actual problem is. Need to distill to a metric
of “closeness”
Soft Technical

24. Implementation pt 4: Scoring
Soft Technical
a c
b
d
e
x
y
z x y z
a
b
c
d
e

25. Implementation pt 4: Scoring
● What to fill in cells?
○ cos() - problems
○ Distances
probably ok
○ CDF
x y z
a
b
c
d
e

26. Implementation pt 4: Scoring
● How to aggregate?
○ mean: attenuation
○ mean(max())
○ Note: 2 scores
○ Forcibly
Symmetrize
x y z
a
b
c
d
e

27. “Design and conduct
hardware or software tests”

29. PaCMAP ensemble (scoring)
DistilBERT
Bag of
Words
Bloom’s
Taxonomy
PaCMAP
Ensemble
Embedding
Fine Tuning

30. Executive Summary
● LACC system needed a way to identify
curricula gaps in Cloud Computing program
● By applying BERT, PaCMAP and elbow
grease, can find the gaps.

31. Q&A
bit.ly/3vwx0W3

32. Shoutouts?
Special thanks to:
● Salomon / ScopeWave
● Nancy / Santa Monica College
● Ankush, Jeremy, Rebecca / Handshake
● PaCMAP Team / Duke

33. Who Are You?
Neal Fultz, neal@njnm.co - data science and
machine learning consultant and recovering
software engineer. Primarily AdTech and
FinTech, but I do other things as well.

34. How did you find this Project?
After presenting at IDEAS 2017 (DTLA) on a
project I did with DataKind for University of
Wisconsin Parkside, another attendee
remembered me 4 years later and reached out.

35. What about The Program Level?
● BEWARE between-course sparsity.
● Concatenate sets (Course Outcome x Job
Task) similarity matrices and reaggregate.
● This allows different programs to tune to
specific niches and specialties.

36. Future Work?
“Skill space” is generic, NNs are very flexible:
● Determine if courses can transfer or substitute
● Resume generator, job recommender from
student transcripts
● Join to wages data, estimate ROI per course
● Identify “missing Bloom level”