This document summarizes some assumptions and challenges around testing the security of open source versus closed source software. It discusses how automated testing through unit tests and fuzzing works in theory but can be limited in practice. Manual testing or "bug hunting" is also explored for both commercial and free open source software, noting it may be easier to find targets in closed source code. The document aims to start a discussion on how security is evaluated differently for open versus closed source programs.