Network OS Code Coverage demo using Bullseye tool.

1. Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 1
Network OS Bullseye code
coverage demo
Vikram Hosakote
vhosakot@cisco.com
June 21, 2012
Software Engineer – NOSTG

2. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
• What is Bullseye ?
• Building a Bullseye NXOS image
• Generating Bullseye code-coverage numbers in NXOS
• Post-processing on NXOS Linux build server
• Merging coverage reports
• Viewing coverage-numbers in the Bullseye GUI
• Demo
• Automation ideas, eARMS integration ideas, HA testing like SSO and ISSU
• Bullseye help
• Q&A

3. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3
• A tool to capture coverage of C and C++ code.
• Used to generate code-coverage numbers in NXOS at Cisco.
• What is code coverage ?
The ratio of number of lines of code tested to number of lines of code developed. A number less
than 100.
Example –
If LDP is developed with 100 lines of code, and just 50 lines tested, the code coverage is
50 / 100 = 50%

4. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4
• Talk to component DEs and find out
• The component name that needs Bullseye instrumentation (e.g. BGP, MPLS, LDP)
• The directory structure of code and binaries (e.g. build/x86s/gdb/ldp/sup, routing-
sw/mpls/ldp/nxos/cradle/)
• New filenames, function names, and new lines of code developed (e.g. ldp_feature.c,
ldp_cmi.h, ldp_cmi_show_neighbor_handler()). The new lines of code depend on the function
name, component, and the feature.
• If code-coverage is needed on supervisor, or line card, or both ?
• Access to NXOS Linux build server
• The exact RELEASE TAG in the build tree (e.g. REL_5_2_5_S30)
• Use this link to build the Bullseye NXOS image using ACME, gmake, and isanadd CLIs
http://zed.cisco.com/confluence/display/DC3SW/NX-OS+Quality+Initiative

5. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
• Load the Bullseye NXOS images – both kickstart and system
• Run the tests
• Code coverage file bootflash:code_coverage/<component>.cov is created both on active and
standby
• Copy this coverage file from both active and standby to the NXOS Linux build server via FTP /
TFTP
• Do post-processing (next slide) and generate coverage-numbers
• In eARMS, submit Bullseye images like regular NXOS images

6. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
• Install Bullseye on the NXOS Linux build server by doing
source /auto/andatc/independent/shellrc-files/current/rc/.cshrc.build
setenv PATH "${PATH}:/auto/andatc/linux/bullseye/7.5.41/bin/"
setenv BULLSEYE_VERSION "7.5.41"
module add bullseye ; which covbr
• Use these Bullseye CLIs with the coverage file <component>.cov for post-processing
covsrc (coverage by source filename)
covdir (coverage by source file directory)
covfn (function coverage, whether a function was invoked or not)
• Bullseye CLIs to generate coverage Reports –
covhtml or –html option (generate HTML report)
-c option to generate output as comma separated values (CSV)

7. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
NXOS-Linux-Build-Server# covsrc -f ldp.cov
Source Function Coverage C/D Coverage
--------------------------------- --------------------- ---------------------
...p/nxos/cradle/ldp_cltmgr_pss.c 21 / 29 = 72% 35 / 94 = 37%
...pls/ldp/nxos/cradle/ldp_main.c 13 / 17 = 76% 71 / 111 = 63%
...s/ldp/nxos/cradle/ldp_cltmgr.c 45 / 57 = 78% 123 / 268 = 45%
...mpls/ldp/nxos/cradle/ldp_mts.c 47 / 57 = 82% 173 / 312 = 55%
...mpls/ldp/nxos/cradle/ldp_cdb.c 21 / 25 = 84% 61 / 129 = 47%
...s/ldp/nxos/cradle/ldp_sysmgr.c 34 / 40 = 85% 84 / 148 = 56%
...mpls/ldp/nxos/cradle/ldp_pss.c 22 / 25 = 88% 61 / 106 = 57%
...p/nxos/cradle/ldp_utils_nxos.c 25 / 26 = 96% 52 / 87 = 59%
...os/cradle/ldp_cltmgr_pss_tlv.c 3 / 3 = 100% 14 / 27 = 51%
.../nxos/cradle/ldp_cdb_pss_tlv.c 3 / 3 = 100% 22 / 36 = 61%
.../ldp/nxos/cradle/ldp_feature.c 11 / 11 = 100% 43 / 68 = 63%
...cmn/include_lcs/ldpx_mem_api.h 1 / 1 = 100% 0 / 0
...nxos/libldpclt/ldpc_cmn_mmsg.h 1 / 1 = 100% 0 / 0

8. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
NXOS-Linux-Build-Server# covdir -f ldp.cov
Directory Function Coverage C/D Coverage
--------------------------------- --------------------- ---------------------
...-bxb/Bullseye/include/routing/ 3 / 141 = 2% 14 / 366 = 3%
...g-sw/mpls/ldp/cmn/include_lcs/ 2 / 27 = 7% 0 / 10 = 0%
...ng-sw/mpls/ldp/nxos/libldpclt/ 7 / 12 = 58% 14 / 47 = 29%
...uting-sw/mpls/ldp/nxos/cradle/ 388 / 617 = 62% 1197 / 3349 = 35%

9. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
NXOS-Linux-Build-Server# covfn -f ldp.cov
Function Source Line FnCov C/D Coverage
-------- --------------------------------------- ----- ---------------------
ldp...me ...mpls/ldp/nxos/cradle/ldp_cmi.c 3699 1 / 1 8 / 10 = 80%
ldp...it ...mpls/ldp/nxos/cradle/ldp_pss.c 472 1 / 1 8 / 10 = 80%
ldp...rs ...s/ldp/nxos/cradle/ldp_sysmgr.c 465 1 / 1 8 / 10 = 80%
ldp...ng ...mpls/ldp/nxos/cradle/ldp_cdb.c 55 1 / 1 4 / 5 = 80%
ldp...it ...pls/ldp/nxos/cradle/ldp_main.c 201 1 / 1 10 / 12 = 83%
ldp...sz .../nxos/cradle/ldp_cdb_pss_tlv.c 490 1 / 1 5 / 6 = 83%
ldp...ey ...s/ldp/nxos/cradle/ldp_cltmgr.c 295 1 / 1 5 / 6 = 83%
ldp...it ...p/nxos/cradle/ldp_utils_nxos.c 259 1 / 1 5 / 6 = 83%
ldp...et ...mpls/ldp/nxos/cradle/ldp_mts.c 1637 1 / 1 12 / 14 = 85%
ldp...ad ...mpls/ldp/nxos/cradle/ldp_mts.c 1850 1 / 1 14 / 16 = 87%
ldp...ad ...mpls/ldp/nxos/cradle/ldp_mts.c 1992 1 / 1 23 / 26 = 88%
ldp...ch ...mpls/ldp/nxos/cradle/ldp_mts.c 1351 1 / 1 9 / 10 = 90%
ldp...en ...mpls/ldp/nxos/cradle/ldp_pss.c 32 1 / 1 9 / 10 = 90%

10. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
NXOS-Linux-Build-Server# covfn -f ldp.cov --html > ldp_covfn.html

11. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
NXOS-Linux-Build-Server# covfn -f ldp.cov -c > ldp_covfn_csv
"ldp_base_init","/ws/vhosakot-bxb/Bullseye/routing-
sw/mpls/ldp/nxos/cradle/ldp_main.c",201,1,10,12,83%
"ldp_cdb_pss_union_key_get_tlv_sz","/ws/vhosakot-bxb/Bullseye/routing-
sw/mpls/ldp/nxos/cradle/ldp_cdb_pss_tlv.c",490,1,5,6,83%
"ldp_cm_fsb_lookup_by_clt_key","/ws/vhosakot-bxb/Bullseye/routing-
sw/mpls/ldp/nxos/cradle/ldp_cltmgr.c",295,1,5,6,83%
"ldp_thread_info_init","/ws/vhosakot-bxb/Bullseye/routing-
sw/mpls/ldp/nxos/cradle/ldp_utils_nxos.c",259,1,5,6,83%
"ldp_mts_msg_options_set","/ws/vhosakot-bxb/Bullseye/routing-
sw/mpls/ldp/nxos/cradle/ldp_mts.c",1637,1,12,14,85%
"ldp_mts_sysmgr_thread","/ws/vhosakot-bxb/Bullseye/routing-
sw/mpls/ldp/nxos/cradle/ldp_mts.c",1850,1,14,16,87%
"ldp_mts_ctrl_thread","/ws/vhosakot-bxb/Bullseye/routing-
sw/mpls/ldp/nxos/cradle/ldp_mts.c",1992,1,23,26,88%
"ldp_dispatch","/ws/vhosakot-bxb/Bullseye/routing-sw/mpls/ldp/nxos/cradle/ldp_mts.c",1351,1,9,10,90%
"ldp_pss_uri_open","/ws/vhosakot-bxb/Bullseye/routing-
sw/mpls/ldp/nxos/cradle/ldp_pss.c",32,1,9,10,90%

12. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
• Bullseye CLI covmerge is used for merging multiple coverage files (*.cov) from multiple N7K
routers.
• Example to merge coverage files from the active and standby of two N7Ks
covmerge -c –f merged_ldp_coverage.cov N7K1_active.cov N7K1_standby.cov N7K2_active.cov N7K2_standby.cov
• If
• A function has 4 decision paths (say 1,2,3 and 4), one N7K runs tests to cover paths 1 and 4,
and another N7K runs tests to cover paths 1 and 2,
• then
• The merged coverage file from both the N7Ks will show 75 % coverage (1,2,4). Path 3 will be
shown "uncovered"

13. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
/auto/andatc/linux/bullseye/7.5.41/bin/CoverageBrowser ldp.cov &

14. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14

15. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15
# Inside package CodeCoverage
proc get_bullseye_coverage_data {
{Bullseye_Build_Server_IP “101.0.0.102”}
{FTP_username “rcpuser”}
{FTP_Password “123rcp!”}
Component_Name} {
# Refer to flowchart on right 
USAGE
package require CodeCoverage
get_bullseye_coverage_data “ldp”

16. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
• Read variable in job file –
set BULLSEYE_COVERAGE_NEEDED “yes”
OR
set BULLSEYE_COVERAGE_NEEDED $env(EARMS_CODE_COVERAGE)
• eARMS enhancement similar to CFLOW (CSCtw78100) -
• In eARMS, submit Bullseye images like regular NXOS images

17. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
• Unlike CFLOW, Bullseye coverage data will NOT get erased and will remain intact through
• Router reload
• Stateful Switchover (SSO)
• In Service Software Upgrade (ISSU)
• In Service Software Downgrade (ISSD)

18. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
• http://www.bullseye.com/help/index.html
• http://www.bullseye.com/
• http://zed.cisco.com/confluence/display/DC3SW/NX-OS+Quality+Initiative
• https://and-sw-web.cisco.com/twiki/bin/view/Main/WebHome
• ACME commands - https://and-sw-web.cisco.com/twiki/bin/view/ACME/ACMECommands
• http://www.bullseye.com/contact.html
• and-sw-build@cisco.com
• Vikram Hosakote (vhosakot@cisco.com)

19. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19
• Webex recording of Bullseye demo to group.bstoner -
https://cisco.webex.com/ciscosales/lsr.php?AT=pb&SP=MC&rID=61685767&rKey=0ec1c04f55cf4a23
• Webex recording of Bullseye demo to nostg-nxos-pi-test and nxos-pi-ft-core-team –
https://cisco.webex.com/ciscosales/lsr.php?AT=pb&SP=MC&rID=61860282&rKey=8c23197cd63a9a18
• Webex recording of Bullseye demo to ASG-core and STEP teams –
https://cisco.webex.com/ciscosales/lsr.php?AT=pb&SP=MC&rID=61903277&rKey=0a391701dee0c555

20. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20

21. Thank you.