2. 200 3D E V E L O P E R S T E S T E R S
O V E R O N L Y
3. Don’t hire too many testers…
Everyone who writes code at Google is a tester.“
4. Software Engineer
Software Engineer in Test
Test Engineer
Test Engineering Manager
테스트 관련 엔지니어들
Key Testing-Related Roles
Why so many?
많이 고용할 필요가 없다더니…
왜 이리 다양할까요?
5. Google SW Engineers are feature developers.
일반 소프트웨어 개발자는 기능을 개발
Google SW Engineers in Test are test developers.
테스트 담당 소프트웨어 엔지니어는 테스트 효율/품질을 높일 수 있는 부분 담당
Google Test Engineers are user developers.
테스트 엔지니어는 유저의 입장에서 테스트 담당
+ Google Test Engineering Manager
테스트 관련 엔지니어들
Key Testing-Related Roles
6. Software Engineer in Test Test Engineering ManagerTest Engineer
SW Engineer in Test는
SW Engineer가 테스트를
더 효율적으로 수행할 수
있도록 하는 것이 목표
Sync 기능 개발
Software Engineer
(SWE)
Test Code
Test-Driven Design
Unit Tests
Software Engineer in Test
(SET)
Sync 기능을 효율적으로 테스트
할 수 있는 코드 개발
Refactoring
Automation
Unit Testing Frameworks
7. Ted Mao
Interview with Software Engineer in Test
이전의 BugsDB (구글이 사용하던 이전 BugsDB) 는 우리의 개발 과정을
도와주긴 커녕 방해하는 일이 대부분이었습니다. 느린 UI 속도, 직관적이
지 않은 Workflow 등 문제가 많았습니다. 그래서 Buganizer를 개발할 때
에는 개발자들의 실제 Process를 도울 수 있도록 설계했습니다.
“
8. Software Engineer in Test Test Engineer Test Engineering Manager
Test Engineer는 유저 관점에서 테스트
• SW Engineer in Test 보다 더 넓은 관점으로 프로젝트를 바라보게 됨
• Technical Skills + User Focus
• User Scenario가 정상적으로 작동하는지 확인
Security
Privacy
Performance
Reliability
Usability
Compatibility
Globalization
Google Chrome
Test Engineer
9. Sync 기능 개발
Software Engineer
(SWE)
크롬 개발 과정의 테스터들
Testers Involved in Chrome Development
Test Code
Test-Driven Design
Unit Tests
Software Engineer in Test
(SET)
Sync 기능을 효율적으로 테스트
할 수 있는 코드 개발
Refactoring
Automation
Unit Testing Frameworks
Test Engineer
(TE)
사용자 관점에서 테스트
사용성, 보안 등 사용자 관점에서
의 비기능 테스트 수행
10. Software Engineer in Test Test Engineer Test Engineering Manager
Test Engineering Manager는 SW Engineer in Test들과 Test Engineer들
사이의 협업이 잘 될 수 있도록 하고 Leadership 역할을 담당
• 절반 이상의 Test Engineering Manager들은 Test Engineer 출신
• “Know your product” 정신
크롬 익스텐션 설치 방법
브라우저 스킨 변경하기
Sync 설정
Proxy 설정 변경
DOM 보는 방법
Cookie 저장 위치
새로운 버전 릴리즈 정책 및 방법
+ Backend & Other Technical Details
Google Chrome
Test Engineering Manager
11. Sync 기능 개발
Software Engineer
(SWE)
크롬 개발 과정의 테스터들
Testers Involved in Chrome Development
Software Engineer in Test
(SET)
Sync 기능을 효율적으로 테스트
할 수 있는 코드 개발
Test Engineer
(TE)
사용자 관점에서 테스트
Test Engineering Manager (TEM) 프로젝트 전반에 걸쳐 테스팅 관리
12. Joel Hynoski
Interview with YouTube Test Engineering Manager
제가 저희 팀의 Tester들에게 가장 먼저 시키는 것은 제품을 완전히 이
해시키는 것입니다. 팀원 모두가 제품을 완벽히 이해해야 합니다. 완벽
히 이해한 후에는 테스팅 문제를 이해하게 되고, 그것을 위한 해결책을
제시할 수 있습니다.
“
14. 테스트의 종류
Types of Tests
테스트의 형태(form)보다는 범위(scope)를 강조하기 위해 code, integration,
and system testing 대신 small, medium, and large tests 사용합니다.
Small Medium Large
15. Small Test Large TestMedium Test
Automated
Single Function or Module
Typical function issues
Data corruption
Faked, Mocks
No external dependencies
Single Unit of Code (Unit Test)
16. Small Test Large TestMedium Test
One or more application modules
Usually automated
Nearest neighbor functions
SW Engineers in Test are heavily involved
“가까운 모듈들이 서로 의도하던 대로 동작하는가?”
Two or More Units of Code (Integration Test)
17. Small Test Large TestMedium Test
Covers three or more modules
Real user scenarios
Real user data sources
“사용자가 의도하는 대로 제품이 동작하는가?”
Three or More Units of Code (System Test)
19. What events are we concerned about?
How likely are these events?
How bad would they be to the enterprise?
How bad would they be for customers?
리스크 측정
Estimating Risks
Frequency
Rarely (Download Page for Chrome)
Seldom (Forward button in Chrome)
Occasionally (Chrome Sync)
Often (Rendering Web Pages)
Impact
Minimal (Chrome Labs)
Some (Refresh button)
Considerable (Chrome Extensions)
Maximal (Autoupdate)
20. 리스크 관리
Risk Management
Eliminate the riskiest components
Write user stories around risky capabilities
Recovery and fallback features
Watchdog code on risks with large impact
Test Engineers들에게 risk에 대응할 책임이 있음
21. Apple Chow
Interview with YouTube Test Engineer
구글에서의 Test Engineer들과 Software Engineer in Test들은 대부분의 다른
회사 엔지니어들보다 기술적인 경험이 많습니다. 코드를 작성할 줄 모르는
Test Engineer를 이곳에서 찾아보기 힘듭니다. 이러한 코딩 능력을 바탕으로
테스팅의 임팩트를 높일 수 있습니다.
“
23. 배워야 할 점
Thins to Take Away
1. 유저 시나리오에는 기술적인 부분이 아예 없어야 한다.
2. 자동화시킬 수 있는 것들은 모두 자동화하고 manual 테스팅이 필요한 곳에
집중하라!
3. 테스팅과 품질 관리는 개발 프로세스에 관련된 모든 사람들이 함께 만들어
가고, Productivity Engineering의 임무는 효율 향상이다.
4. 테스팅의 중요성을 모두가 인지하고 있는 test-centric 문화는 필수이다.
5. 기능보다 품질을 중요시해야 한다.
6. 테스팅이 품질을 보장하진 못한다!
2001년 구글이 AdWords, Search 등을 통해 수익을 내기 시작할 때 200명정도의 개발자들이 있었지만 테스터들은 단 3명
테스터들을 많이 고용할 필요는 없다. 구글에서는 코드를 작성하는 모든이가 테스터다.
SETs are partners in the SWE codebase, but are more concerned with increasing quality and test coverage than adding new features or increasing performance. SETs write code that allows SWEs to test their features.
BugsDB was impeding our development process rather than supporting it. To be honest, it was wasting a lot of valuable engineering time and this was a tax paid by every team that used it.
The TE role puts testing on behalf of the user first. TEs organize the overall quality practices, interpret test results, drive test execution, and build end-to-end test automation.
Google TEs are user developers, responsible for taking the users’ perspectives in all things that have to do with quality. From a development perspective, they create automation for user scenarios and from a product perspective, they assess the overall coverage and effectiveness of the ensemble of testing activity performed by the other engineering roles. It is not utopia, but it is our best attempt at achieving it in a practical way where real-world concerns have a way of disrupting best intentions in the most unforeseen and unforgiving way
The TEM is an engineering colleague taking on important work as an individual contributor and a single point of contact through which all the support teams (development, product management, release engineers, document writers, and so on).
Big challenges in those days. The first was really to get our head around the product. One thing I need all my testers to be is product experts. Everyone on my team must know every aspect of the product stack. Period. Once you get that level of knowledge, you understand what the hard testing problems are and you can start building your team around those needs.
“Productivity is our job; testing and quality are the job of everyone involved in the development”
Engineering Productivity 팀은 continuous integration을 전파하고 tool을 계속 만들어내는 팀
Testers are essentially on loan to the product teams and are free to raise quality concerns and ask questions about functional areas that are missing tests or that exhibit unacceptable bug rates.
The actual language of small, medium, and large isn’t important. Call them whatever you want as long as the terms have meaning that everyone agrees with. The important thing is that Google testers share a common language to talk about what is getting tested and how those tests are scoped. When some enterprising testers begin talking about a fourth class they dubbed enormous tests, every other tester in the company could imagine a systemwide test covering every feature and that runs for a very long time. No additional explanation is necessary.
Small tests verify the behavior of a single unit of code generally isolated from its environment. Examples include a single class or a small group of related functions. Small tests should have no external dependencies. Outside of Google, small tests are commonly known as “unit tests.”
Medium tests validate the interaction of one or more application modules, as depicted in Figure 2.3. Larger scopes and longer running times differentiate medium tests from small tests. Whereas small tests might attempt to exercise all of a single function’s code, medium tests are aimed at testing interaction across a limited subset of modules. Outside of Google, medium tests are often called “integration tests.”
Large and enormous tests are commonly referred to as “system tests” or “end-to-end tests” outside of Google. Large tests operate on a high level and verify that the application as a whole works. These tests are expected to exercise any or all application subsystems from the UI down to backend data storage, as shown in Figure 2.4, and might make use of external resources such as databases, file systems, and network services.
Sheer volume of automation + tooling (no commercial tools in general) + developer-owns-quality and test-centric SWE culture
“Productivity is our job; testing and quality are the job of everyone involved in the development”
