Recommended
More Related Content
Similar to How good of developer are you v1 .1 ASML.pptx
Similar to How good of developer are you v1 .1 ASML.pptx (20)
More from Roy Wasse
Recently uploaded
Recently uploaded (20)
How good of developer are you v1 .1 ASML.pptx
- 1. How good of a developer are you? And can we objectively measure this?
- 2. Bio • Roy Wasse • Bsc. Technical Computer Science • Msc. Philosophy of Technology • Co Founder OpenValue • Dev & Management roles • 15+ years of hiring experience
- 3. Is someone is a ‘good’ dev! • Do you know YAGNI or HATEOS, SOLID? • Explain balanced binary tree search algo? • Use 20 best interview questions from internet • Apply Open Hiring? • Proxy variables e.g., CV, experience, education, references, questionnaire • Whiteboard coding • Make (lengthy) assignment
- 4. Google Hiring process • Assessments: You may be asked to do a brief online assessment, like a coding quiz, after you’ve submitted your resume. • Short virtual chats: Before diving into more in-depth interviews, you’ll typically have one or two shorter conversations over phone or video. These will usually be with a recruiter and then with either the hiring manager or peer on the team, and are designed to assess key skills you’ll need for the role. • Project work: We sometimes ask candidates to complete a small project prior to their in- depth interviews. This could range from prepping a case study to providing writing or code samples (don’t stress, they’re not that scary and we won’t spring this on you without warning), and helps us understand how you think and approach problems. We’ll let you know about any additional materials we’ll need early on. • Interviews: We get excited about interviewing and take it seriously because, at the risk of sounding cliché, Google is what Googlers make it. Our process can be rigorous (typically 3-4 interviews in one day, either over video or in person).
- 5. • Can science help us and how? • Let’s dive into literature • And see how we can predict of someone will be a good dev! Who knows what works?!!
- 6. Use of Aptitude tests • Measure if someone is dispositioned well to excel • Popular in the 1970s (e.g., Alspaugh, 1972; Mazlack, 1980) • due to lack of good results from these tests not used much anymore (Bornat, Dehnadi, & Simon, 2008; Curtis, 1991; Mayer & Stalnaker, 1968) • Also: in a sample of over 3,500 students, no incremental validity was found for specialized aptitude tests for programmers over the more general aptitude tests used by the United States Air Force (Besetsny, Ree, & Earles, 1993)
- 7. Question • Who thinks a computer science study is an important predictor for the expert level of a developer?
- 8. Role of education • Large meta study (Roth, BeVier, Switzer, & Schippmann, 1996) reported a correlation of r = 0.16 • Shortly after graduation, correlation between grades and job performance much higher • 1 year, r = 0.23, n = 1,288 • 6 years, r = 0.05, n = 866.
- 9. Question • Who thinks a senior developer should have at least 5 years of experience?
- 10. Role of experience • Studies show positive effect of having programming experience (Agarwal, Sinha & Tanniru, 1996; Arisholm & Sjøberg, 2004) • Other studies have reported no effect of experience beyond the first few years (Jeffery & Lawrence, 1979) or no effect of experience (Jørgensen, 1995; Wohlin, 2002, 2004)
- 11. Question • Who thinks LLMs will replace us within the next few years for the better part of our job?
- 12. Role of LLMs • Refactoring vs. Refuctering (Tornhill, Borg, Mones, 2024) • Ideal for who? • Generation works pretty well, but what about code deletion?
- 13. Question • Who thinks Java/.Net/... certification programs are important for a developer?
- 14. Role of Programming knowledge • Central predictor of skill is knowledge (Chi et al., 1988) • Used in Java certification • But…software development is a skill Acquired by practice
- 15. What is skill? • Ability (Ferguson, 1956): “An ability is a trait defined by what an individual can do” • “Skill is specialized type of ability, one that improves with practice and is well adjusted, well organized and goal oriented” (Fitts & Posner, 1967)
- 16. Model of skill acquisition • “A Five-Stage Model of the Mental Activities Involved in Directed Skill Acquisition”, (Dreyfus & Dreyfus, 1980) • Flow is at level 5 • Also think of driving / playing chess
- 17. It’s all about skill? • Skill cannot by equated with performance • “the individual performed well because of his high level of skill” is a generalization (Messick, 1994) • 8 major components of job performance research show 3 direct determinants of performance on the job: knowledge, skill and motivation (Campbell, McCloy, Oppler & Sager, 1993)
- 18. Skill is just one..? Campbell’s work performance framework proposed eight work performance dimensions: 1. job-specific task proficiency <= skill 2. non–job-specific task proficiency 3. written and oral communications 4. demonstrating effort 5. maintaining personal discipline 6. facilitating peer and team performance 7. supervision 8. management and administration. According to Campbell, these eight dimensions are sufficient to describe the latent structure of performance at a general level. However, he also noted that the eight factors can have different patterns of subdimensions, and their content and salience can vary across jobs.
- 19. Technical skill is the most important one (McGill, 2008)
- 20. Skill & project success emphasizing low price when selecting software providers in bidding rounds increases the likelihood project will fail.” (Jørgensen 2011) Tested by analyzing a dataset of 4,791,067 bids for 785,326 small- scale projects Clients emphasizing a low price selected providers with lower skill, resulted in 9% increase in failure rate
- 21. What about teamwork • Team performance is more complex to understand than individual performance (see, e.g., Baker & Salas, 1992; Volmer, 2006), individual skill or expertise is nevertheless a central component in team performance (Land, Wong, & Jeffery, 2003).
- 22. Importance of developers • The claim of substantial programming performance differences on success is confirmed by additional data, reviews & experiments (Curtis, 1981; DeMarco & Lister, 1985, 1999; Trendowicz & Munch, 2009; Prechelt, 1999a) • Variability in individual performance more important than technology or methods used to increase productivity • In the book “Software Economics”, Boehm (1981) reported that the “people factor” was the largest of all investigated factors in the success of software projects and therefore concluded that developer attributes are by far the best opportunity for improving software productivity.
- 23. Pair experts for extra speed • The benefits of pair programming has been studied from the perspective of forming the pair based on different levels of expertise (Lui & Chan, 2006) • …or seniority (Arisholm, Gallis, Dybå, & Sjøberg, 2007). • A systematic literature review on pair programming found that, among students, this practice was most beneficial when the pair was comprised of individuals with a similar level of programming skill (Salleh, Mendes, & Grundy, 2011).
- 24. So common sense wins? • A software developer is someone who usually holds “a bachelor’s degree in computer science and strong computer programming skills” (US Bureau of Labor Statistics) • So let’s hire based on skill level!!
- 25. Deep dive into skills • And…how to objectively measure coding skills • Soft skills
- 27. Research overview • Relations between effort estimates, skill indicators, and measured programming skill. Magne Jørgensen, Gunnar R. Bergersen and Knut Liestøl (2021) • Measuring Programming Skill: Construction and Validation of an Instrument for Evaluating Java Developers. Gunnar R. Bergersen, PhD thesis (2015) • Construction and validation of an instrument for measuring programming skill. Gunnar R. Bergersen, Dag I. K. Sjøberg, and Tore Dybå (2014) • Evaluating methods and technologies in software engineering with respect to developers’ skill level. Gunnar R. Bergersen and Dag I. K. Sjøberg (2012) • Inferring skill from tests of programming performance: combining time and quality. Gunnar R. Bergersen, Jo E. Hannay, Dag I. K. Sjøberg, Tore Dybå, and Amela Karahasanovic (2011) • Programming skill, knowledge, and working memory among professional developers from an investment theory perspective. Gunnar R. Bergersen and Jan-Eric Gustafsson (2011)
- 28. How to measure skill? • Home work assignment (better rotate those) • Whiteboard coding (under pressure) • Online questionaires (assuming correlation between real world coding) • All are subjective • And probably you want to get it right!
- 29. Measure via self assessment • Ratings of self may work better when people compare themselves relative to each other (people they know) rather than when providing absolute ratings (Kruger and Dunning (1999) Source: wikipedia
- 30. Measuring programming skill • Research started in 50’s with punch card operators traits (McNamara & Hughes, 1955) • 60’s use of test in selection researched (Perry & Cantley, 1965) • Behavorial study of devs (Shneiderman, 1976) • Theories concerning the comprehension of computer programs (Brooks, 1983)
- 31. Task performance is key • >20 studies about job performance assessed • Almost all frameworks mentioned task performance as an important dimension of individual work performance.
- 32. Predict skill • Study of novice LISP programmers (Anderson & Jeffries, 1985) found that “the best predictor of individual subject differences in errors on problems that involved one LISP concept was the number of errors on other problems that involved different concepts” (Anderson, 1987, p. 203).
- 33. How to put this into a system • Time (Bergersen, 2011) • Quality (Ebert et. al. 2005) • Process improvement (Dybå, 2000) Create a measurement model & test the model on new data (Bergersen et al., 2014)
- 34. Rasch model • More than 60 years ago, Rasch (1960) used tests to study the reading performance of students. • Rasch models have become a practical way to measure psychological abilities
- 35. Task construction 1.Picked existing tasks from literature 2.Reanalyzed data sets to combine time & quality 3.Large control group with 255 developers from 9 countries used to score tasks
- 36. • Collection of tasks • VM in the cloud (Linux + IntelliJ / Eclipse • Interchangeable Building a new standard to measure & analyze software engineering skills_
- 37. Detailed analytics of skills and meaningful comparisons with empirically validated norm groups_
- 38. DEMO
- 41. OCA/OCP • Makes you think like a compiler • Check for API knowledge • Does not tell anything about your actual skill level • Doesn’t measure skills like GrepS does • So, what if we would consult the community for a GrepS test variant that uses both
- 42. Foojay community • Over 1000 members • 2 news articles every week day • Podcast • Conference information • Tech questions • And the certification initiative!
- 44. Foojay certified developer • Multiple sessions with champions and 80+ community members • Defined what should be tested and how • Universal test • Never do a test again! • Or actually do test again to prove your progress! • Current status = beta! • Want a free test, let me know
- 45. Recap • Coding is mostly about skills • Research shows how to measure skills • Let’s base Java certification this! • Questions?
Editor's Notes
- GB: ..are you?
- GB: drop the second «is» and add a «?» Q & A with Bert Jan about how to test if someone is a good dev, ask audience Hypermedia as the Engine of Application State Single responsibility principle - Class has one job to do. Each change in requirements can be done by changing just one class. Open/closed principle - Class is happy (open) to be used by others. Class is not happy (closed) to be changed by others. Liskov substitution principle - Class can be replaced by any of its children. Children classes inherit parent's behaviours. Interface segregation principle - When classes promise each other something, they should separate these promises (interfaces) into many small promises, so it's easier to understand. Dependency inversion principle - When classes talk to each other in a very specific way, they both depend on each other to never change. Instead classes should use promises (interfaces, parents), so classes can change as long as they keep the promise. Open Hiring: prevent ruling out certain people because of bias. Just see how they do FAANG companies
- Programming aptitude is a term that is often used to identify the reason that some developers are dispositionally better at becoming good programmers. Thus, when new tests of programming aptitude are proposed (e.g., Dehnadi, 2006; Harris, 2014; Tukiainen & M¨onkk¨onen, 2002), the main challenge is not whether such tests predict the success in learning to program, but rather that such tests should be better than other tests that are already available for use and, furthermore, may be used in many more situations besides programming. Would this mean that asking to explain a binary tree is not a good predictor of programming performance? Research also shows that brain training doesn’t really work. Practising your brain by doing a lot of chess excersices doesn’t mean you become a good programmer fast. So again measure the actuall skill and not a proxy skill. (from refactoring vs refuctoering)
- I used to say Bsc Informatics at a minimum. But seen too many counterexamples. It sure has a role 0,3 is very weak Research about job performace and grades So especially in the beginning academic background is important, later not so much. That would coincide with intuition no? Important moderator was found to be the time between graduation and performance measurement.
- Best prediction of someones future performance is past performance Studies/papers say that in the end experience is only a limited predictor of developer performace
- Key difference between senior and junior is that they’ve seen it before. This allows them to focus more on semantics rather than syntax. If you already understand an annotation in Java, you can focus on getting the job done. Before we had internet this mattered more, since it wasn’t so easy to look up syntax, but now with rich IDE’s that help us autocomplete and tools like CoPilot and ChatGPT the advantage of having it seen before becomes smaller one could argue, also according to a recent sudy from Tornhill, Borg and Moes). But learning details matter
- Skilled behavior can be traced back to Aristotle (1999), who di↵erentiated between epistˆemˆe (i.e., knowledge) and technˆe (i.e., crafts, art). The theory of skill describes three overlapping phases in the acquisition of psychomotoric skill, that is, bodily movement in relation to mental activities. In the first (cognitive) phase, facts about the domain where the developed skill will later be applied are initially acquired. Another central predictor of skill is knowledge (Chi et al., 1988). That knowledge is central to skill follows directly from phase one of skill acquisition: Kyllonen and Woltz (1989) name this phase “knowledge acquisition”.6 Studies also show that the best way to learn something is by simulating the to be learned task as closely as possible
- https://xray-delta.com/2011/10/12/dreyfus-model-a-richer-understanding-of-competency-building/ Skill acuisition via three phases (learn how to drive a car) 1. learn the facts 2. active thinking 3. embody skill Become a car driver, learn how to play the piano Become a chess player Become a programmer
- E’ve already seen some predictors of skill, like education and experiemce. Level 5 skill is not enough High performance is, in many occasions, most likely due to high skill. However, high performance may also be due to luck. Or motivation can cause bad performace, work extra hard
- Context: Bidding rounds are frequently used to select competent and cost-efficient providers for software projects. Objective: We hypothesize that emphasizing low price when selecting software providers in such bidding rounds substantially increases the likelihood the project will fail. Method: The hypothesis is tested by analyzing a dataset of 4,791,067 bids for 785,326 small-scale projects registered at a web-based marketplace connecting software clients and providers. Results: We find evidence supporting our hypothesis. For example, selecting providers with bids 25% lower than the average bid is connected to a 9% increase in the frequency of project failures for the same level of provider skill. In addition, we found that clients emphasizing a low price, on average, selected providers with lower skill levels. This decrease in provider skill level further strengthened the negative effect of a strong focus on low price on project failures. For example, selecting a provider with a 15% failure rate for previous projects instead of 5% increased the failure rate by 33%. Conclusion: We interpret the findings to suggest that a client may substantially reduce the likelihood of project failure by reducing the emphasis on low price when selecting a provider. (https://www.researchgate.net/publication/262345882_A_strong_focus_on_low_price_when_selecting_software_providers_increases_the_likelihood_of_failure_in_software_outsourcing_projects) Selecting a provider with a 15% failure rate for previous projects instead of 5% increased the failure rate by 33%
- Work focused on issues concerning the measurement of productivity (e.g., Jones, 1997; Walston & Felix, 1977) was often based on counting lines of code that are produced within a given time unit. Nevertheless, it appears that a valid measure of programming productivity has been elusive as there is no readily available way to compare the performance of individuals across di↵erent systems.
- Land of the blind one eye is king
- https://www.wilmarschaufeli.nl/publications/Schaufeli/358.pdf Task performance can be defined as the proficiency (ie, competency) with which one performs central job task
- Individual’s current level of programming performance on one set of tasks predicts an individual’s future level of performance on another set of tasks. Generally, one can say that performance increases with higher expertise. But there are situations where novice programmers have been found to outperform experts (see, e.g., Adelson, 1984; Haerem & Rau, 2007). In those situations, factors other than programming expertise may dominate an individual’s performance on a task. For example, implementing a calculation may pose a complex mathematical problem rather than a programming challenge Master of Java Similar to the “conventional wisdom [that]‘The best indicator of future performance is past performance”’ (Wernimont & Campbell, 1968, p. 372).
- Whereas performance may use units such as time, degree of correctness, efficacy, reliability, etc., none of these units are meaningful to characterize skill. For example, for an airplane passenger, the di↵erence between landing safely (good flying performance) and dying in a crash (the worst possible flying performance) is probably infinitely large (what does this tell about the skills of a pilot)
- the Rasch model conceptualizes abilities using an interval scale where the unit of measurement is the logarithm of the odds The relation between ability and difficulty is expressed stochastically in the Rasch model, so that when the level of an individual’s ability equals the difficulty of an item, the probability of a correct answer is 50%
- Time & Quality First, The construction and sampling of tasks in the Pre Study was based on previously published tasks. Second, to establish principles to combine time and quality as performance, two data sets were reanalyzed. Data Set 1 was reported in (Arisholm & Sjøberg, 2004) and used only the data from the professionals in the study Third, concerning the subject sampling and data collection, Table 4 shows all the involved subjects and data sets. In addition to Data Sets 1 and 2, another data set (Data Set 3) was used in the Main Study and is discussed in detail in Papers II, III, and IV. As shown in the table, 255 developers from 9 unique countries participated in total, representing 320 person days of programming performance available for analysis.
- Start at 50 sec
- Donning – kruger at work with GrepS data