Web applications pose significant risks for organizations. The selection of an appropriate scanning product or service can be challenging because every organization develops their web applications differently and decisions made by developers can cause wide swings in the value of different scanning technologies. To make a solid, informed decision, organizations need to create development team- and organization-specific benchmarks for the effectiveness of potential scanning technologies. This involves creating a comprehensive model of false positives, false negatives and other factors prior to mandating analysis technologies and making decisions about application risk management. This presentation provides a model for evaluating application analysis technologies, introduces an open source tool for benchmarking and comparing tool effectiveness, and outlines a process for making organization-specific decisions about analysis technology selection.