Your SlideShare is downloading. ×
0
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Juniper Trouble Shooting
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Juniper Trouble Shooting

97,075

Published on

Juniper M, T and MX trouble shooting tips and real world cases

Juniper M, T and MX trouble shooting tips and real world cases

Published in: Technology
1 Comment
14 Likes
Statistics
Notes
  • I requested the file more than one hour ago still did not receive it . Is there something wrong or joining was just a way to expand your mailing list ??
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
97,075
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
872
Comments
1
Likes
14
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Control and Forwarding plane <ul><li>Synchronization </li></ul><ul><li>100-Mbps fxp1 Ethernet link is used between RE and PFE </li></ul><ul><li>For M320 case, 100-Mbps Ethernet switch is being used to provide a dedicated link to each FPC. For RE, these links are presented at bcm0 </li></ul><ul><li>Fxp0: management interface </li></ul><ul><li>Fxp2: communication between Primary RE and backup RE </li></ul><ul><li>3) Forwarding table (FT) can hold over 800,000 routes. </li></ul>
  • 2. Difference between M7i and M10i <ul><li>Redundant RE: M10i support, not M7i </li></ul><ul><li>Built-in Adaptive Service: M7i. M10i needs an external AS PIC. </li></ul><ul><li>RE: the same </li></ul>
  • 3. System storage <ul><li>3 types of storages: </li></ul><ul><li>Compact Flash(ad0) : built-in at the board. </li></ul><ul><li>Hard Drive(ad1) </li></ul><ul><li>External storage </li></ul><ul><li>-PCMCIA card(da0??) </li></ul><ul><li>-USB(da1??) </li></ul>
  • 4. JUNOS CLI basics <ul><li>Space bar to complete a command </li></ul><ul><li>Command :Help topic <command> for general concepts </li></ul><ul><li>Command : help reference <command to look> for configuration syntax </li></ul><ul><li>Rebooting system: request system reboot </li></ul><ul><li>Shut down system: request system halt </li></ul><ul><li>Log and Trace files are located at /var/log </li></ul><ul><li>Command: Show log | messages | file-name </li></ul><ul><li>At more prompt, use forward slash(/) to search or use “h” to get a context help screen </li></ul><ul><li>Log commands examples: </li></ul><ul><li>- show log messages | match so-0/3/1 | match TRAP --- AND -- </li></ul><ul><li>- show log messages | mach “fpc | sfm | kernel” --- OR --- </li></ul><ul><li>Monitor log/trace in real time: monitor start file-name | match fail </li></ul><ul><li>Stop monitoring in real time: monitor stop </li></ul><ul><li>Enable/disable real-time output to screen: Esc-Q </li></ul><ul><li>Stop traceing operation: delete flag open </li></ul><ul><li>Truncate(clear) log/trace files: clear log file-name </li></ul><ul><li>Delete log/trace files: file delete file-name </li></ul>
  • 5. JUNOS CLI basics <ul><li>Entering configuration: Type configure or edit </li></ul><ul><li>Exclusive configuration (configure exclusive) and Private configuration (configure private??) </li></ul><ul><li>Moving within the configuration hierarchy: edit (equivalent to cd), up, top, exit (to previous location in the hierarchy) </li></ul><ul><li>Show command at configuration mode vs. show command at operational-mode </li></ul><ul><li>Relative configuration commands Starting with JUNOS5.3 : top </li></ul><ul><li>- top show system login (show system login no matter where you are. Examples: </li></ul><ul><li>- top edit protocols ospf ( to enter protocols ospf no matter where you are) </li></ul><ul><li>Viewing configuration in operational mode: show configuration < configuration path> </li></ul><ul><li>View configuration with set: show xxx | display set </li></ul><ul><li>Viewing candidate configuration: show chassis alarm, show (at the current sub-hierarchy) </li></ul>
  • 6. JUNOS CLI basics <ul><li>Change the candidate configuration. Examples: </li></ul><ul><li>- set alarm sonet lol red </li></ul><ul><li>- delete alarm sonet pll </li></ul><ul><li>Display difference between the candidate and active configurations: </li></ul><ul><li>At the current statement-path, show | compare </li></ul><ul><li>Viewing difference in files. Example: </li></ul><ul><li>- file show filename1 | compare file filename2 </li></ul><ul><li>- show configuarion | compare rollback number </li></ul><ul><li>Removing statements: delete </li></ul><ul><li>Delete the statements and all its subordinate statements and identifieres. </li></ul><ul><li>Wildcard delete. Example: wildcard delte interfaces fe-* </li></ul><ul><li>Ignore portion of the configuration hierarchy: deactivate / activate </li></ul><ul><li>Disable an interface: set disable interface </li></ul><ul><li>Delete and disabled interface: delete interface <interface-name> disable </li></ul>
  • 7. JUNOS CLI basics <ul><li>Activate a configuration </li></ul><ul><li>commit ----- candidate file is checked, actived and marked as the current </li></ul><ul><li>operational sofware configuration file. </li></ul><ul><li>commit check ----- only validate a candidate configuration without </li></ul><ul><li>placing it into effect. </li></ul><ul><li>rollback n -------- recover the previous configuration. And then commit </li></ul><ul><li>rollback 0 is current configuration </li></ul><ul><li>First 3 roll back (1-3) are stored in solid-state flash disk </li></ul><ul><li>/config/juniper.conf.n (n=1-3) </li></ul><ul><li>rest roll back (4-49) are stored in hard disk /var/db/config </li></ul><ul><li>commit confirmed time-out ---- temporarily activate a configuration (default is 10 minutes). If the final commit is not executed, the system will performs a “rollback 1, commit” commands. </li></ul><ul><li>commit synchronize ---- after committed on the master RE internally </li></ul><ul><li>copied and committed on the backup RE automatically. </li></ul><ul><li>commit at time ----- commit at some time </li></ul><ul><li>clear system commit ---- cancel a pending commit </li></ul>
  • 8. JUNOS CLI basics <ul><li>Save a configuration </li></ul><ul><li>save filename </li></ul><ul><li>save terminal -- for copy and paste into other others </li></ul><ul><li>show | display set – create configuration for simplifying configuration editing. </li></ul><ul><li>Loading configuration files ( load and then commit) </li></ul><ul><li>load override filename – override the current config with the loaded one. Do it at the root of the configuration hierarchy. </li></ul><ul><li>load merge filename - combine the new and old </li></ul><ul><li>load merge terminal (then copy/paste hierarchical configuration) </li></ul><ul><li>load replace filename – statements with replace tag will replace the statements with the same name </li></ul><ul><li>load relative – load at where it is current at the configuration hierarchy. </li></ul>
  • 9. Junos CLI Basics <ul><li>Only save the configuration under certain hierarchy. To save the whole configure, issue this command at the top of the hierarchy. </li></ul><ul><li># Save <filenam> </li></ul><ul><li>Display the contents of the file you saved </li></ul><ul><li># Run file show <filename> </li></ul><ul><li>To load a configuration after clear the current configuration </li></ul><ul><li># delete </li></ul><ul><li>#show </li></ul><ul><li>#load override <filename> </li></ul><ul><li>To recover a mistake made previously after committing. </li></ul><ul><li>#rollback 1 </li></ul>
  • 10. Junos CLI Basics <ul><li>show log messages | last </li></ul><ul><li>Show log interactive-commands | match restart </li></ul><ul><li>Use sysctl –a to display kernel parameters. </li></ul><ul><li>sysctl –a | grep icmp (under shell prompt) </li></ul><ul><li>show chassis 0 pic slot 1 information. </li></ul><ul><li>Show chassis pic fpc-slot 0 pic-slot 1 </li></ul><ul><li>Master switchover </li></ul><ul><li>Request chassis cfeb master switch </li></ul><ul><li>Request chassis routing-engine master switch </li></ul>
  • 11. Junos CLI Basics <ul><li>Find out who logins the system and kick out some particular users. </li></ul><ul><li>show system users </li></ul><ul><li>reequest system logout </li></ul><ul><li>help syslog <log strings> </li></ul><ul><li>Example: </li></ul><ul><li>lab@santro-re0> help syslog ACCT_ACCOUNTING_FERROR </li></ul><ul><li>Name: ACCT_ACCOUNTING_FERROR </li></ul><ul><li>Message: Unexpected error <error-code> from file <filename> </li></ul><ul><li>Help: Error occurred during file processing </li></ul><ul><li>Description: An error prevented the accounting statistics process from processing the indicated file. </li></ul><ul><li>Type: Error: An error occurred </li></ul><ul><li>Severity: warning </li></ul>
  • 12. Junos CLI Basic <ul><li>show configuration with inheritance </li></ul><ul><li>show configuration interfaces ge-4/3/3 | display inheritance </li></ul>
  • 13. Syslog <ul><li>set system syslog file messages any notice </li></ul>
  • 14. Hardware troubleshooting process <ul><li>Show chassis alarms </li></ul><ul><li>Show chassis craft-interface </li></ul><ul><li>Show log messages </li></ul><ul><li>Show log chassid </li></ul><ul><li>Monitor start [message | chassid] </li></ul><ul><li>Show chassis hardware </li></ul><ul><li>Show chassis fpc </li></ul><ul><li>Show pfe stat error </li></ul><ul><li>Show interface terse </li></ul><ul><li>Show interface detail </li></ul><ul><li>Show log <log-file-name> </li></ul>
  • 15. Display PIC status <ul><li>Show chassis pic fpc-slot 0 pic-slot 1 </li></ul><ul><li>Example: </li></ul><ul><li>lab@santro-re0> show chassis pic fpc-slot 0 pic-slot 1 </li></ul><ul><li>FPC slot 0, PIC slot 1 information: </li></ul><ul><li>Type 10x 1GE(LAN), 1000 BASE </li></ul><ul><li>ASIC type H chip </li></ul><ul><li>State Online </li></ul><ul><li>PIC version 1.13 </li></ul><ul><li>Uptime 1 day, 22 hours, 25 minutes, 17 seconds </li></ul><ul><li>PIC port information: </li></ul><ul><li>Fiber Xcvr vendor </li></ul><ul><li>Port Cable type type Xcvr vendor part number Wavelength </li></ul><ul><li>0 GIGE 1000SX SM FINISAR CORP. FTRJ8519P1BNL-J2 850 nm </li></ul>
  • 16. Boot image <ul><li>If you need to reboot from PCMCIA card, you need to copy a special image called jinstall-mediaxxxx . </li></ul><ul><li>Interrupt normal boot </li></ul><ul><li>Hit space when the system is rebooting until it goes to either boot: or OK prompt. </li></ul><ul><li>If you get boot: prompt, the loader is not run yet. You need to do this: </li></ul><ul><li>Boot: /boot/loader </li></ul><ul><li>Change a boot device at OK prompt </li></ul><ul><li>Ok nextboot compact-flash </li></ul><ul><li>Ok reboot </li></ul>
  • 17. Interfaces <ul><li>Disable(admin down) an interface </li></ul><ul><li>Admin Link </li></ul><ul><li>So-0/1/1 down up </li></ul><ul><li>So-0/1/1.0 up down </li></ul><ul><li>Deactivate an interface </li></ul><ul><li>Admin Link </li></ul><ul><li>So-0/1/1 up up </li></ul>
  • 18. RE overview (Q: how to find out RE <-> Platform compatibility list?) <ul><li>Primary coopy of JUNOS resides on the flash memory. Use this command to create a backup copy: </li></ul><ul><li>request system snapshot </li></ul><ul><li>Mgd manages CLI </li></ul><ul><li>RE has different versions: RE-333, RE-400, RE-600, RE-1600. Each RE is supported by certain platforms. </li></ul><ul><li>RE uses Intel processor from P III to P IV. </li></ul><ul><li>Use this command to find out what RE is being used: show chassis hardware. </li></ul><ul><li>Hard disk monitoring: Self-Monitoring Analysis and Reporting Technology System(SMART). From 5.5, SMART is enabled by default. To disable: </li></ul><ul><li>set system processes disk-monitoring disable </li></ul><ul><li>Configuration file compression: default starting Release 7.0 (maybe). To enable: </li></ul><ul><li>set system compress-configuration-file </li></ul><ul><li>RE versions </li></ul><ul><li>RE5(RE-400): only supported in M7i and M10i </li></ul><ul><li>RE4(RE-600): All M and T series. Except M7i/M10i/M320. The only RE to have flash memory upgrade </li></ul><ul><li>RE3 (RE-333): M5/10/20/40/40e, and M160 </li></ul><ul><li>RE-1600: M320 and T320/T640. Using Broadcom chipset for Ethernet connectivity to PFE. </li></ul><ul><li> While used on M320, the GE link is supported as bcm0. While on T-series, 100- Mbps is supported(???) </li></ul>
  • 19. PFE overview on M-series <ul><li>Different names but referring to the route lookup module: </li></ul><ul><li>M40 – System Control Board (SCB) </li></ul><ul><li>M20 – System Switch Board (SSB) </li></ul><ul><li>M5/10 – FPC and SCB are combined into a single board called the Forwarding Engine Board (FEB) </li></ul><ul><li>M7i/10i – Compact FEB (CFEB) </li></ul><ul><li>M40e and M160 – Switching and Forwarding Module (SFM). 4 SFM on M160, each one provides 25% of lookup capability. 2 SFM on M40e, only one can be active. </li></ul><ul><li>Special stuff on M40e and M160 platform: </li></ul><ul><li>MCS card (Miscellaneous Control Subsystem): provide control and monitoring functions for the various components in the chassis </li></ul><ul><li>PCG (PFE clock generation): 125-MHZ signal. Redundant PCGS </li></ul>
  • 20. PFE on T-series and M320 <ul><li>M320 is different than T and M-series. It is a combio of two using I and J chips. </li></ul><ul><li>T640 PFC2 has single PFE, PFC3 has two PFE </li></ul><ul><li>T-Series nonblocking cross-bar switch fabric – Switch Interface Boards(SIBs). </li></ul><ul><li>T320: 3 SIBs with 2 are active. SIB 1 and 2 are active, SIB0 is standby. SIB0 has only one high-speed line (HSL) connected to FPC. SIB1 and SIB2 has 2 HSL. So when SIB0 becomes active, system performance is degraded. </li></ul><ul><li>T640: 5 switch fabric cards or SIBs, 4 are active, 1 standby. Something like Cisco’s GSR. </li></ul><ul><li>M320: 4 SIBs. </li></ul><ul><li>M320 FPC1: use single I chip </li></ul><ul><li>M320 FPC2: dual I chip, thus two PFE </li></ul><ul><li>M320 FPC3: dual J chip, thus two PFE </li></ul>
  • 21. Physical Interface Cards (PIC) <ul><li>IP service PIC is to hardware assist complex packet processing and has no physical ports. </li></ul><ul><li>IP service PIC include: </li></ul><ul><li>1)Tunnel service PIC for IP-IP, GRE tunnel and PIM-SM tunnel. </li></ul><ul><li>2)Multlink PIC: Multilink Point-to-Point (MLPPP) and Multilink Frame Relay (MLFR, FRF 1.5) </li></ul><ul><li>Hot-Pluggable except M20 and M40 which need to remove FPC. </li></ul><ul><li>Take PIC offline before physically removing it. Otherwise would cause system damage or PFE reset. </li></ul><ul><li>Packet loss is expected on M-serials except M320 because of FPC reset. </li></ul>
  • 22. Flexible PIC Concentrator (FPC) <ul><li>Support 1 to 4 PICs. M160 OC-192 has an FPC support only one PIC. </li></ul><ul><li>Each FPC on M-serial pooled to create shared memory switch fabric. So hot-swap FPC cause system to repartition the shared memory pool; 200 ms packet loss. </li></ul><ul><li>FPC is hot-swappable in all platforms except M5 and M10 which is using FEB. However M7i and M10i are OK even using CFEB. </li></ul><ul><li>Build-in FPC at some high-speed quad-wide PICs such as OC-48c/STM-16 for M20/40. OC-192c/STM-64 SONET/SDH on M160. </li></ul><ul><li>New FPC to support reuse of old PICs: </li></ul><ul><li>M160 FPC1: intend to reuse M20/40 PIC </li></ul><ul><li>M160 FPC2: design to support M160 only PIC, such as OC-48c </li></ul><ul><li>FPC3: support native T-series PICs. </li></ul><ul><li>T640 only support FPC2 and FPC3. </li></ul><ul><li>How to power off FPC? </li></ul><ul><li>set chassis fpc power off </li></ul>
  • 23. M-series System Board <ul><li>General functions </li></ul><ul><li>Names very by platforms </li></ul><ul><li>M40 – System Control Board (SCB) </li></ul><ul><li>M20 – System Switch Board (SSB) </li></ul><ul><li>M5/10 – FPC and SCB are combined into a single board called the Forwarding Engine Board (FEB) </li></ul><ul><li>M7i/10i – Compact FEB (CFEB) </li></ul><ul><li>M40e and M160 – Switching and Forwarding Module (SFM). 4 SFM on M160, each one provides 25% of lookup capability. 2 SFM on M40e, only one can be active. </li></ul><ul><li>Enhanced System Boards: </li></ul><ul><li>- 2 nd generation Internet Processor II ASIC (not on M5/10 and M7i/10i) </li></ul><ul><li>- support 840K routing entries, double from old board 420K. </li></ul><ul><li>- Double on-chip memory to 16MB on IP II </li></ul><ul><li>- CPU memory 128 M for M40, 256M for M20, M40e and M160. </li></ul><ul><li>- Increased CPU speed to 256 MHZ. </li></ul><ul><li>- First shipped with JUNOS 5.5 Sep 2002. </li></ul>
  • 24. IP II ASIC <ul><li>Performance: 40 Mpps, 40 byte with 80K prefixes at routing table. </li></ul><ul><li>Packet processing features: </li></ul><ul><li>Filtering, sampling, logging, counting, load balancing </li></ul><ul><li>All M-series have enhanced S-board which as IP II ASIC. M5/10 doesn’t have enhanced S-board. </li></ul><ul><li>T-series might contain as many as 16 IP II ASIC. Each FPC has one or two PFE which contains its own IP II ASIC. </li></ul>
  • 25. Craft Interface <ul><li>What is it? </li></ul><ul><li>Collection of mechanisms on M-series and T-series </li></ul><ul><li>View System status messages </li></ul><ul><li>Trouble shooting </li></ul><ul><li>Where is it? </li></ul><ul><li>On the front of the chassis </li></ul><ul><li>What does it have? </li></ul><ul><li>System status LEDs </li></ul><ul><li>FPC/PIC online/offline buttons. </li></ul><ul><li>LCD screen provide status reporting for the entire system. </li></ul><ul><li>What alternatives on other platforms? </li></ul><ul><li>M7i: FIC (Fixed Interface Card)provide PIC offline/online buttons </li></ul><ul><li>M10i: HCM (High-Availability Chassis Manager) Card provide PIC offline/online bottons. </li></ul>
  • 26. Password recovery <ul><li>Connect to console </li></ul><ul><li>Power cycle the RE and watch it booting up </li></ul><ul><li>Enter a space character at the boot loader quick help manue to get a command prompt (don’t enter space too quickly) </li></ul><ul><li>Enter “boot –s” </li></ul><ul><li>When system boots up, answer “ recovery” to recover password </li></ul><ul><li>Follow the on-screen steps to change password </li></ul><ul><li>Commit the change </li></ul><ul><li>Reboot the system again. </li></ul><ul><li> </li></ul>
  • 27. Coredump analysis – using syslog message <ul><li>Step 1: Get the stack trace from syslog messages </li></ul><ul><li>lab@hissy> show log messages | find &quot;machine check&quot; </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 machine check caused by error on the PC </li></ul><ul><li>I Bus </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 error detect register 1: 0x08, 2: 0x00 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 error ack count = 0 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 error address: 0x08004014 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 PCI bus error status register: 0x02 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 was the PCI master </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 C/BE bits: I/O read [0b0010] </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 error detection reg1: PCI cycle </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 PCI status reg: parity error </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 ^B </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 last message repeated 7 times </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Registers: </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 R00: 0x000e8c4c R01: 0x0775dad4 R02: 0x0000334 </li></ul><ul><li>4 R03: 0x00000000 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 R04: 0x0775dae0 R05: 0x00142e34 R06: 0x06006b3 </li></ul><ul><li>6 R07: 0x00006b36 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 R08: 0x00142e4c R09: 0x88000000 R10: 0x0000000 </li></ul><ul><li>0 R11: 0x00000000 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 R12: 0x00100004 R13: 0x000cc411 R14: 0x0000c43 </li></ul><ul><li>0 R15: 0x00040000 </li></ul>
  • 28. Coredump analysis – using syslog message <ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 R16: 0x00000000 R17: 0x00041410 R18: 0x0004c42 </li></ul><ul><li>0 R19: 0x8004c618 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 R20: 0x0002c490 R21: 0x00110000 R22: 0x0000000 </li></ul><ul><li>Juniper Confidential. For Internal use only. </li></ul><ul><li>0 R23: 0x001151cc </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 R24: 0x00000001 R25: 0x00000000 R26: 0x0775db1 </li></ul><ul><li>4 R27: 0x06006b36 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Stack Traceback: </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 01: sp = 0x0775dad4, pc = 0x000e8c4c </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 02: sp = 0x0775db0c, pc = 0x0005cd9c </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 03: sp = 0x0775db34, pc = 0x00108914 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 04: sp = 0x0775db4c, pc = 0x00108888 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 05: sp = 0x0775db54, pc = 0x000eec84 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 06: sp = 0x0775db5c, pc = 0x00037e78 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 07: sp = 0x0775dc1c, pc = 0x000380f8 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 08: sp = 0x0775dcfc, pc = 0x000eeadc </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 09: sp = 0x0775dd2c, pc = 0x000eefd0 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 10: sp = 0x0775dd3c, pc = 0x000f0184 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 11: sp = 0x0775dd74, pc = 0x000b28cc </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 12: sp = 0x0775dd84, pc = 0x000b29f4 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 13: sp = 0x0775ddac, pc = 0x000b2a8c </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 14: sp = 0x0775ddcc, pc = 0x000b2c80 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 15: sp = 0x0775ddec, pc = 0x000b2d5c </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 16: sp = 0x0775de04, pc = 0x0002665c </li></ul>
  • 29. Coredump analysis – using syslog message <ul><li>What do I want? I will copy the following into a file called “stack” </li></ul><ul><li>single% cat stack </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Stack Traceback: </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 01: sp = 0x0775dad4, pc = 0x000e8c4c </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 02: sp = 0x0775db0c, pc = 0x0005cd9c </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 03: sp = 0x0775db34, pc = 0x00108914 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 04: sp = 0x0775db4c, pc = 0x00108888 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 05: sp = 0x0775db54, pc = 0x000eec84 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 06: sp = 0x0775db5c, pc = 0x00037e78 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 07: sp = 0x0775dc1c, pc = 0x000380f8 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 08: sp = 0x0775dcfc, pc = 0x000eeadc </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 09: sp = 0x0775dd2c, pc = 0x000eefd0 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 10: sp = 0x0775dd3c, pc = 0x000f0184 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 11: sp = 0x0775dd74, pc = 0x000b28cc </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 12: sp = 0x0775dd84, pc = 0x000b29f4 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 13: sp = 0x0775ddac, pc = 0x000b2a8c </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 14: sp = 0x0775ddcc, pc = 0x000b2c80 </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 15: sp = 0x0775ddec, pc = 0x000b2d5c </li></ul><ul><li>Dec 5 01:51:17 hissy tnp_sfm_3 Frame 16: sp = 0x0775de04, pc = 0x0002665c </li></ul>
  • 30. Coredump analysis – using syslog message <ul><li>Step2: Find out which version and build of the image. </li></ul><ul><li>So it is on M160, 4.4B3.2 and build 4.4-20010408-b20191 </li></ul><ul><li>lab@hissy> show version brief </li></ul><ul><li>Hostname: hissy </li></ul><ul><li>Model: m160 </li></ul><ul><li>JUNOS base [4.4B3.2] (Export restricted edition) </li></ul><ul><li>JUNOS Kernel Software Suite [4.4-20010408-b20191] </li></ul><ul><li>JUNOS Routing Software Suite [4.4-20010408-b20191] </li></ul><ul><li>JUNOS Packet Forwarding Engine Support [4.4-20010408-b20191] </li></ul><ul><li>JUNOS Online Documentation Files [4.4-20010408-b20191] </li></ul>
  • 31. Coredump analysis – using syslog message <ul><li>Step 3: Find out which symbol file to use. </li></ul><ul><li>‘ debug’ package for the crashing code if the crash is in the kernel or routing, or the normal </li></ul><ul><li>package for the PFE. The perl script ‘jemsym’ can be used to decode </li></ul><ul><li>the stack. </li></ul><ul><li>Recent dailies; </li></ul><ul><li>single% cd /volume/build </li></ul><ul><li>single% ls </li></ul><ul><li>20010201-0805@ 20010217-0805@ 20010305-0805@ 20010320-0910@ 20010405-0810@ </li></ul><ul><li>20010202-0805@ 20010218-0805@ 20010306-0805@ 20010321-0910@ 20010406-0810@ </li></ul><ul><li>older dailies for released versions; </li></ul><ul><li>single% cd /volume/ftp/private/unregressed/ </li></ul><ul><li>single% ls </li></ul><ul><li>3.4/ 4.0/ 4.1/ 4.2/ 4.3/ 4.4/ 5.0/ </li></ul><ul><li>released code; </li></ul><ul><li>single% cd /volume/ftp/private/junos/ </li></ul><ul><li>single% ls </li></ul><ul><li>4.0B1/ 4.0R5/ 4.1R4/ 4.3B1.2/ 4.4B2.1/ </li></ul><ul><li>4.0B2/ 4.1B1.1/ 4.2B1.1/ 4.3B2.1/ 4.4B3.2/ </li></ul>
  • 32. Coredump analysis – using syslog message <ul><li>single% cp /volume/build/20010408-0810/jpfe-4.4-20010408-b20191-debug.tgz . </li></ul><ul><li>single% tar zxfv jpfe-4.4-20010408-b20191-debug.tgz </li></ul><ul><li>+CONTENTS </li></ul><ul><li>+COMMENT </li></ul><ul><li>+DESC </li></ul><ul><li>+INSTALL </li></ul><ul><li>+REQUIRE </li></ul><ul><li>usr/share/pfe/scb.jbf </li></ul><ul><li>usr/share/pfe/scb.sym </li></ul><ul><li>usr/share/pfe/scb.elf </li></ul><ul><li>usr/share/pfe/fpc.jbf </li></ul><ul><li>usr/share/pfe/fpc.sym </li></ul><ul><li>usr/share/pfe/fpc.elf </li></ul><ul><li>usr/share/pfe/sfm.jbf </li></ul><ul><li>usr/share/pfe/sfm.sym </li></ul><ul><li>usr/share/pfe/sfm.elf </li></ul><ul><li>usr/share/pfe/fpc160.jbf </li></ul><ul><li>usr/share/pfe/fpc160.sym </li></ul><ul><li>usr/share/pfe/fpc160.elf </li></ul><ul><li>usr/share/pfe/sbr.jbf </li></ul><ul><li>usr/share/pfe/sbr.sym </li></ul><ul><li>usr/share/pfe/sbr.elf </li></ul>fpc.sym - M20/M40 fpc stack traces fpc160.sym -- M160 fpc stack traces sbr.sym -- M5/M10 stack traces scb.sym -- M40/M20 S-Board traces sfm.sym --M160 SFM traces.
  • 33. Coredump analysis – using syslog message <ul><li>What is Jemsym file? </li></ul><ul><li>#!/usr/local/bin/perl </li></ul><ul><li>## </li></ul><ul><li>$Id: jemsym,v 1.7 1998/04/21 01:15:33 jim Exp $ </li></ul><ul><li>## </li></ul><ul><li>This file takes a Juniper panic stack trace and turns it </li></ul><ul><li># into a user-readable output from the symbol table file </li></ul><ul><li># for the running micro-kernel. </li></ul><ul><li>Juniper Confidential. For Internal use only. </li></ul><ul><li>## </li></ul><ul><li>By default, gmake produces a symbol table file for each </li></ul><ul><li># target, and then you run the text of the panic stack trace, </li></ul><ul><li># perhaps saved to a temporary file, as follows: </li></ul><ul><li>## </li></ul><ul><li>cat temp-backtrace_file | jemsym target.sym </li></ul>
  • 34. Coredump analysis – using syslog message <ul><li>Step 4: Do the stack trace </li></ul><ul><li>single% cat stack | ~dbovis/bin/jemsym usr/share/pfe/sfm.sym </li></ul><ul><li>0x000e8c4c cchip_ab_pio (0x000e8b2c) +0x120 </li></ul><ul><li>0x0005cd9c pfe_bmemchip_pio_write (0x0005cd44) +0x58 </li></ul><ul><li>0x00108914 bchip_write_sram_opaque (0x00108898) +0x7c </li></ul><ul><li>0x00108888 bchip_write_sram_hton (0x00108878) +0x10 </li></ul><ul><li>0x000eec84 bchip_write_sram_mem_val (0x000eec64) +0x20 </li></ul><ul><li>0x00037e78 diags_pfe_mem_address_test (0x00037dfc) +0x7c </li></ul><ul><li>0x000380f8 diags_pfe_mem_test (0x0003802c) +0xcc </li></ul><ul><li>0x000eeadc bchip_mem_test (0x000eea08) +0xd4 </li></ul><ul><li>0x000eefd0 bchip_diags_sram_test (0x000eef30) +0xa0 </li></ul><ul><li>0x000f0184 bchip_probe_diag (0x000f00fc) +0x88 </li></ul><ul><li>0x000b28cc cm_probe_slot (0x000b284c) +0x80 </li></ul><ul><li>0x000b29f4 cm_probe_slots (0x000b297c) +0x78 </li></ul><ul><li>0x000b2a8c cm_probe_chassis (0x000b2a64) +0x28 </li></ul><ul><li>0x000b2c80 cm_probe_event_loop (0x000b2b98) +0xe8 </li></ul><ul><li>0x000b2d5c cm_probe_thread_init (0x000b2ca8) +0xb4 </li></ul><ul><li>0x0002665c thread_suicide (0x0002665c) +0x0 </li></ul>
  • 35. Coredump analysis – using syslog message <ul><li>Step 4: Do the stack trace </li></ul><ul><li>single% cat stack | ~dbovis/bin/jemsym usr/share/pfe/sfm.sym </li></ul><ul><li>0x000e8c4c cchip_ab_pio (0x000e8b2c) +0x120 </li></ul><ul><li>0x0005cd9c pfe_bmemchip_pio_write (0x0005cd44) +0x58 </li></ul><ul><li>0x00108914 bchip_write_sram_opaque (0x00108898) +0x7c </li></ul><ul><li>0x00108888 bchip_write_sram_hton (0x00108878) +0x10 </li></ul><ul><li>0x000eec84 bchip_write_sram_mem_val (0x000eec64) +0x20 </li></ul><ul><li>0x00037e78 diags_pfe_mem_address_test (0x00037dfc) +0x7c </li></ul><ul><li>0x000380f8 diags_pfe_mem_test (0x0003802c) +0xcc </li></ul><ul><li>0x000eeadc bchip_mem_test (0x000eea08) +0xd4 </li></ul><ul><li>0x000eefd0 bchip_diags_sram_test (0x000eef30) +0xa0 </li></ul><ul><li>0x000f0184 bchip_probe_diag (0x000f00fc) +0x88 </li></ul><ul><li>0x000b28cc cm_probe_slot (0x000b284c) +0x80 </li></ul><ul><li>0x000b29f4 cm_probe_slots (0x000b297c) +0x78 </li></ul><ul><li>0x000b2a8c cm_probe_chassis (0x000b2a64) +0x28 </li></ul><ul><li>0x000b2c80 cm_probe_event_loop (0x000b2b98) +0xe8 </li></ul><ul><li>0x000b2d5c cm_probe_thread_init (0x000b2ca8) +0xb4 </li></ul><ul><li>0x0002665c thread_suicide (0x0002665c) +0x0 </li></ul>
  • 36. Coredump analysis – using core files <ul><li>Where to get coredump files? </li></ul><ul><li>1) Coredump files are stored at: /volume/ftp/pub/incomfing/<case_number>/<core_filenma> </li></ul><ul><li>For Example: </li></ul><ul><li>/volume/ftp/pub/incoming/2008-0104-0511 </li></ul><ul><li>2) For some freaking .tgz file, you need to do this </li></ul><ul><li>gunzip < cosd.core-tarball.0.tgz.2 | tar -xvf - </li></ul><ul><li>Using GUI </li></ul><ul><li>http://jtac-tools.juniper.net/crashdecode/coredump.html </li></ul><ul><li>Using Manual methods: </li></ul><ul><li>Step 1 : Using Jdebug to find out the stack traces. </li></ul><ul><li>jdebug='/volume/buildtools/bin/jdebug‘ </li></ul><ul><li>/volume/buildtools/bin/jdebug <core_file name> </li></ul><ul><li>Examples: The core file is saved at /volume/ftp/pub/incoming/2008-0104-0511/ core-SSB0.core.0 </li></ul><ul><li>Step 2 : Use query-pr to find out the possible PRs based on the stack trace. </li></ul><ul><li>query-pr -m &quot;thread_debug&quot; -m &quot;sched_suspend_thread&quot; –summary </li></ul>
  • 37. Coredump analysis – using core (continued) <ul><li>-bash-2.05b$ /volume/buildtools/bin/jdebug core-SSB0.core.0 </li></ul><ul><li>GNU gdb 6.5 juniper_2006a_411 </li></ul><ul><li>Copyright (C) 2006 Free Software Foundation, Inc. </li></ul><ul><li>GDB is free software, covered by the GNU General Public License, and you are </li></ul><ul><li>welcome to change it and/or distribute copies of it under certain conditions. </li></ul><ul><li>Type &quot;show copying&quot; to see the conditions. </li></ul><ul><li>There is absolutely no warranty for GDB. Type &quot;show warranty&quot; for details. </li></ul><ul><li>This GDB was configured as &quot;--host=i386-unknown-freebsd4.11 --target=powerpc-juniper-eabi&quot;. </li></ul><ul><li>#0 0x000330a0 in panic ( </li></ul><ul><li>format_string=0x25f204 &quot;CCHIP: Too many SRAM parity errors; restart required &quot;) </li></ul><ul><li>at ../ukern/cpu-ppc/ppc603e_panic.c:63 </li></ul><ul><li>63 asm volatile (&quot;sc&quot;); </li></ul><ul><li>(gdb) bt </li></ul><ul><li>#0 0x000330a0 in panic ( </li></ul><ul><li>format_string=0x25f204 &quot;CCHIP: Too many SRAM parity errors; restart required &quot;) </li></ul><ul><li>at ../ukern/cpu-ppc/ppc603e_panic.c:63 </li></ul><ul><li>#1 0x0018bf7c in cchip_error_hardware (C=0x35, hwerror=402653184) </li></ul><ul><li>at ../common/drivers/cchip/cchip_int.c:238 </li></ul><ul><li>#2 0x0018c158 in cchip_error_scan () at ../common/drivers/cchip/cchip_int.c:352 </li></ul><ul><li>#3 0x0006baec in pfe_error_scan (info=0x0) at ../common/toolkits/pfe/pfe_scb.c:172 </li></ul><ul><li>#4 0x000da8c8 in cm_handle_pfe_error (rate_limit=FALSE) </li></ul><ul><li>at ../common/applications/cm/cm_pfe_restart.c:1463 </li></ul><ul><li>#5 0x000dabc0 in cm_restart_handle_timer_event (timer=0x35) </li></ul><ul><li>at ../common/applications/cm/cm_pfe_restart.c:1652 </li></ul><ul><li>#6 0x000daff0 in cm_restart_event_loop () at ../common/applications/cm/cm_pfe_restart.c:1898 </li></ul><ul><li>#7 0x00026fa0 in thread_wake (thread=0x210000) at ../ukern/common/thread.c:572 </li></ul><ul><li>(gdb) </li></ul>
  • 38. Coredump analysis – core file from special image <ul><li>Step 1: to find out the image path using “what” on image or core file. </li></ul><ul><li>-bash-2.05b$ what core-SSB0[1].core.3 core-SSB0[1].core.3: </li></ul><ul><li>scb release 8.2I20071212_2313_pgoyette built by pgoyette on 2007-12-12 23:14:53 UTC </li></ul><ul><li>jtac-bbuild01.juniper.net:/b/pgoyette/VZ-8.2-20071012/src/juniper/pfe/obj-scb </li></ul><ul><li>-bash-2.05b$ cd /volume/nfsbuild40 </li></ul><ul><li>-bash-2.05b$ ls </li></ul><ul><li>jcano pgoyette ramanathan sdoshi yuris </li></ul><ul><li>So the whole path is: </li></ul><ul><li>/volume/nfsbuild40/pgoyette/VZ-8.2-20071012/src/juniper/pfe/obj-scb </li></ul><ul><li>Step 2: Find out the *.elf file. In the above case, it is scb.elf under the above path. </li></ul>
  • 39. Coredump analysis – core file from special image <ul><li>Soemtimes it take more trouble to untar the compressed jpfe file to get the elf file. </li></ul><ul><li>lab@iggy> show version brief | grep packet </li></ul><ul><li>JUNOS Packet Forwarding Engine Support [4.0-20000608-s22432] </li></ul><ul><li>( From above number I don’t know where to get the jpfe file ) </li></ul><ul><li>single% tar zxfv jpfe-4.0-20000608-regressed-debug.tgz </li></ul><ul><li>+CONTENTS </li></ul><ul><li>+COMMENT </li></ul><ul><li>+DESC </li></ul><ul><li>+INSTALL </li></ul><ul><li>+REQUIRE </li></ul><ul><li>usr/share/pfe/scb.jbf </li></ul><ul><li>usr/share/pfe/scb.sym </li></ul><ul><li>usr/share/pfe/scb.elf </li></ul><ul><li>usr/share/pfe/fpc.jbf </li></ul><ul><li>usr/share/pfe/fpc.sym </li></ul><ul><li>usr/share/pfe/fpc.elf </li></ul><ul><li>usr/share/pfe/sfm.jbf </li></ul><ul><li>usr/share/pfe/sfm.sym </li></ul><ul><li>usr/share/pfe/sfm.elf </li></ul><ul><li>usr/share/pfe/fpc160.jbf </li></ul><ul><li>usr/share/pfe/fpc160.sym </li></ul><ul><li>usr/share/pfe/fpc160.elf </li></ul>fpc.sym M20/M40 fpc stack traces fpc160.sym M160 fpc stack traces sbr.sym M5/M10 stack traces scb.sym M40/M20 S-Board traces sfm.sym M160 SFM traces.
  • 40. Coredump analysis – core file from special image <ul><li>-bash-2.05b$ /volume/cross/cygnus-i386-ppc/bin/gdb-core.ppc -nw /volume/nfsbuild40/pgoyette/VZ-8.2-20071012/src/juniper/pfe/obj-scb/scb.elf core-SSB0[1].core.3 </li></ul><ul><li>GNU gdb 4.16-97r2a </li></ul><ul><li>Copyright 1997 Free Software Foundation, Inc. </li></ul><ul><li>GDB is free software, covered by the GNU General Public License, and you are </li></ul><ul><li>This GDB was configured as &quot;--host=i386-unknown-freebsd2.2.5 --target=powerpc-eabi&quot;... </li></ul><ul><li>#0 topo_connect (topo=0xd5af08, next=0x28, reconnect=FALSE) </li></ul><ul><li>at ../common/toolkits/topo/topo.c:428 </li></ul><ul><li>../common/toolkits/topo/topo.c:428: No such file or directory. </li></ul><ul><li>(gdb) bt  ----------------------------------------------------------- </li></ul><ul><li>#0 topo_connect (topo=0xd5af08, next=0x28, reconnect=FALSE) </li></ul><ul><li>at ../common/toolkits/topo/topo.c:428 </li></ul><ul><li>#1 0x155a84 in nh_indirect_add_sub (nh=0x2163a3c, unilist=0x0, </li></ul><ul><li>indirect_elementpp=0x2163a98) </li></ul><ul><li>at ../common/applications/nh/nh_indirect.c:193 </li></ul><ul><li>#2 0x155a84 in nh_indirect_add_sub (nh=0x2163a3c, unilist=0x0, </li></ul><ul><li>indirect_elementpp=0x2163a98) </li></ul><ul><li>at ../common/applications/nh/nh_indirect.c:193 </li></ul><ul><li># </li></ul><ul><li>at ../common/applications/pfeman/pfeman_rt.c:413 </li></ul><ul><li>#11 0x276cc in thread_suicide () at ../ukern/common/thread.c:951 </li></ul>
  • 41. Coredump analysis – Kernel core of special image <ul><li>Find out where is the symbol file by using what. </li></ul><ul><li>Ex: /volume/nfsbuild40/pgoyette/VZ-8.2I20071212_2313/ ship / jkernel-8.2I20080311_1541_jtac-builder-debug.tgz </li></ul><ul><li>copy the jkernel file to your home directory and unzip it. </li></ul><ul><li>Ex: gunzip < jkernel-8.2I20080311_1541_jtac-builder-debug.tgz | tar -xvf- </li></ul><ul><li>Debug the vmcore.0 file </li></ul><ul><li>Ex: gdb -k kernel.debug vmcore.0 </li></ul>
  • 42. Coredump analysis – daemon crash <ul><li>1) uncompress the freaking core *.tgz file </li></ul><ul><li>gunzip < cosd.core-tarball.2.tgz | tar -xvf - cosd.core.0 juniper.conf messages cosd.info.0 juniper.conf.1.gz </li></ul><ul><li>2) Where is the symbol file by doing “what” </li></ul><ul><li>bash-2.05b$ what cosd.core.0 </li></ul><ul><li>cosd.core.0: </li></ul><ul><li>COSD release 7.3R3.6 built by builder on 2006-02-01 08:03:43 UTC </li></ul><ul><li>xathanon.juniper.net :/build/xathanon-c/7.3R3.6/obj-i386/juniper/usr.sbin/cosd </li></ul><ul><li>getsubopt.c 8.1 (Berkeley) 6/4/93 </li></ul><ul><li>Copyright (c) 1994 Powerdog Industries. All rights reserved. </li></ul>
  • 43. Coredump analysis – daemon crash <ul><li>3) Decode the core file </li></ul><ul><li>-bash-2.05b$ gdb /build/xathanon-c/7.3R3.6/obj-i386/juniper/usr.sbin/cosd/cosd cosd.core.0 </li></ul><ul><li>GNU gdb 4.18 (FreeBSD) </li></ul><ul><li>Copyright 1998 Free Software Foundation, Inc. </li></ul><ul><li>-bash-2.05b$ gdb /build/xathanon-c/7.3R3.6/obj-i386/juniper/usr.sbin/cosd/cosd cosd.core.0 </li></ul><ul><li>GNU gdb 4.18 (FreeBSD) </li></ul><ul><li>Copyright 1998 Free Software Foundation, Inc. </li></ul><ul><li>Core was generated by `cosd'. </li></ul><ul><li>Program terminated with signal 11, Segmentation fault. </li></ul><ul><li>/usr/lib/libisc.so.2: No such file or directory. </li></ul><ul><li>#0 0x806d6f2 in cos_ifd_configure (dop=0x81e4300, conf=0x81ba000, </li></ul><ul><li>name=0xbfbff850 &quot;ge-0/3/0&quot;, match_len=10, wc_match=0 '00', </li></ul><ul><li>ifd_has_ieee_classifier=1 '01', errmsg=0xbfbffc70 &quot;&quot;, errmsglen=256) </li></ul><ul><li>at ../../../../src/juniper/usr.sbin/cosd/cosd_parser.c:2705 </li></ul><ul><li>2705 cos_ifd->if_flags |= COS_IFD_CONF_F_IEEE_CLASSIFIER; </li></ul><ul><li>(gdb) bt </li></ul><ul><li>#0 0x806d6f2 in cos_ifd_configure (dop=0x81e4300, conf=0x81ba000, </li></ul><ul><li>name=0xbfbff850 &quot;ge-0/3/0&quot;, match_len=10, wc_match=0 '00', </li></ul><ul><li>ifd_has_ieee_classifier=1 '01', errmsg=0xbfbffc70 &quot;&quot;, errmsglen=256) </li></ul><ul><li>at ../../../../src/juniper/usr.sbin/cosd/cosd_parser.c:2705 </li></ul><ul><li>#1 0x806f851 in cos_config_interfaces (dop=0x81e4280, conf=0x81ba000, </li></ul><ul><li>errmsg=0xbfbffc70 &quot;&quot;, errmsglen=256) </li></ul><ul><li>at ../../../../src/juniper/usr.sbin/cosd/cosd_parser.c:3944 </li></ul>#2 0x807bb53 in cos_config (conf=0x81ba000, errmsg=0xbfbffc70 &quot;&quot;, errmsglen=256) at ../../../../src/juniper/usr.sbin/cosd/cosd_parser.c:10816 #3 0x807be0e in cosd_parse_config (cos_conf=0x81ba000, check_only=0 '00') at ../../../../src/juniper/usr.sbin/cosd/cosd_parser.c:10924 #4 0x8069ac4 in main (argc=1, argv=0xbfbffe0c) at ../../../../src/juniper/usr.sbin/cosd/cosd_main.c:330 (gdb) l 2700 } else { 2701 cos_ifd = cos_pat_to_ifd(pnode); 2702 } 2703 2704 if (ifd_has_ieee_classifier) { 2705 cos_ifd->if_flags |= COS_IFD_CONF_F_IEEE_CLASSIFIER; 2706 } 2707 2708 /* 2709 * in commit check, cosd hasn't built its interface data
  • 44. Coredump analysis – Software or Hardware issues? <ul><li>Case #1 </li></ul><ul><li>Panic, TLB Data miss, Data access etc type of system exceptions:most probably software related. What you should do is to enable the coredump on the chassisd and gather all the base information mentioned above. </li></ul><ul><li>Case #2: </li></ul><ul><li>pci parity error being reported on the CPU DRAM address space, this means that this is </li></ul><ul><li>a bogus pci error. The reason is, there is no pci bus connected to the CPU DRAM. </li></ul><ul><li>Action: In this case, we have to enable the coredump on chassisd and get the coredump of the PFE component along with the base information. No RMA should be issued. </li></ul><ul><li>Example: </li></ul><ul><li>mpc106 machine check caused by error on the PCI Bus </li></ul><ul><li>mpc106 error detect register 1: 0x08, 2: 0x00 </li></ul><ul><li>mpc106 error ack count = 2 </li></ul><ul><li>mpc106 error address: 0x001d0048        < belongs to CPU DRAM </li></ul><ul><li>mpc106 PCI bus error status register: 0x02 </li></ul><ul><li>mpc106 was the PCI master </li></ul><ul><li>    C/BE bits: I/O read [0b0010] </li></ul><ul><li>mpc106 error detection reg1: PCI cycle </li></ul><ul><li>mpc106 PCI status reg: parity error     < parity error. </li></ul>
  • 45. Coredump analysis – Software or Hardware issues? <ul><li>Case #3: </li></ul><ul><li>There is parity protection enabled (ECC is disabled) on the CPU </li></ul><ul><li>DRAM, if a hw failure occurs here, the message that you should </li></ul><ul><li>see is: &quot;memory parity/ECC error&quot;. </li></ul><ul><li>Action: Run the memory diagnostics tests and RMA. </li></ul><ul><li>Example: </li></ul><ul><li>mpc106 machine check caused by error on the Processor Bus       < reported by Processor Bus </li></ul><ul><li>mpc106 error detect register 1: 0x04, 2: 0x00 </li></ul><ul><li>mpc106 error ack count = 0 </li></ul><ul><li>mpc106 error address: 0x02f39e18 </li></ul><ul><li>mpc106 Processor bus error status register: 0x72 </li></ul><ul><li>  transfer type 0b01110, transfer size 2 </li></ul><ul><li>mpc106 error detection reg1: memory parity/ECC error    < parity error. </li></ul><ul><li>mpc106 PCI status reg: parity error </li></ul>
  • 46. Coredump analysis – Software or Hardware issues? <ul><li>Case #3: </li></ul><ul><li>There is parity protection enabled (ECC is disabled) on the CPU </li></ul><ul><li>DRAM, if a hw failure occurs here, the message that you should </li></ul><ul><li>see is: &quot;memory parity/ECC error&quot;. </li></ul><ul><li>Action: Run the memory diagnostics tests and RMA. </li></ul><ul><li>Example: </li></ul><ul><li>mpc106 machine check caused by error on the Processor Bus       < reported by Processor Bus </li></ul><ul><li>mpc106 error detect register 1: 0x04, 2: 0x00 </li></ul><ul><li>mpc106 error ack count = 0 </li></ul><ul><li>mpc106 error address: 0x02f39e18 </li></ul><ul><li>mpc106 Processor bus error status register: 0x72 </li></ul><ul><li>  transfer type 0b01110, transfer size 2 </li></ul><ul><li>mpc106 error detection reg1: memory parity/ECC error    < parity error. </li></ul><ul><li>mpc106 PCI status reg: parity error </li></ul>
  • 47. Monitoring - logs <ul><li>Step 1: configure logging file </li></ul><ul><li>Example: </li></ul><ul><li>isis { </li></ul><ul><li>traceoptions { </li></ul><ul><li>file mike-isis; </li></ul><ul><li>flag state; </li></ul><ul><li>flag error; </li></ul><ul><li>flag spf; </li></ul><ul><li>flag lsp receive detail; </li></ul><ul><li>} </li></ul><ul><li>Step 2: monitor start <log-file-name> </li></ul><ul><li>Step 3: monitor start message </li></ul><ul><li>Example: </li></ul><ul><li>lab@falcons> monitor start mike-isis </li></ul><ul><li>lab@falcons> monitor start messages </li></ul><ul><li>lab@falcons> </li></ul><ul><li>*** mike-isis *** </li></ul><ul><li>Feb 5 20:05:53.517506 Updating LSP falcons.00-00 in database </li></ul><ul><li>Feb 5 20:05:53.517654 Updating L2 LSP falcons.00-00 in TED </li></ul>
  • 48. Booting up system <ul><li>request system snapshot partition as-primary </li></ul><ul><li>request system media usb </li></ul><ul><li>request system reboot media usb - when reboot from another media, all the file systems will be under this media. </li></ul><ul><li>request system snapshot part as-primary media compact-flash </li></ul><ul><li>request system reboot media compact </li></ul><ul><li>request system software add /var/tmp/junojseries-8.4R2.4-domestic.tgz no-validate </li></ul><ul><li>Request system snapshot -- make a image at another storage(if you are using disk, this will mirror the image to CF. If you are using CF, this will makes an image at disk. </li></ul><ul><li>request system software delete backup </li></ul><ul><li>request system storage cleanup </li></ul><ul><li>To remove swap space at the compact-flash: </li></ul><ul><li>http://www.juniper.net/techpubs/software/junos/junos85/rn-sw-85 </li></ul>
  • 49. Tools and quick reference <ul><li>http://clie.juniper.net </li></ul><ul><li>/volume/build - junos releases and source code. After 8.4, go to extra hierarchy /volume/build/junos. For example: /volume/build/junos/8.4/release/8.4R2.4/ship </li></ul><ul><li>http:// jam.jnpr.net </li></ul><ul><li>http://www- in.juniper.net/eng/cvs_pdf / </li></ul><ul><li>https:// deepthought.juniper.net /app/ </li></ul><ul><li>http://cvs/cgi-bin/viewcvs.cgi / </li></ul><ul><li>http:// confluence.jnpr.net / </li></ul><ul><li>/volume/current - cvs functional specs </li></ul><ul><li>/volume/labcores </li></ul><ul><li>http://rogers.jtac-emea.jnpr.net/wiki/index.php?title = Enginee </li></ul>
  • 50. How to find out what syslog means? <ul><li>[email_address] > </li></ul><ul><li>help syslog SNMPD_SUBAGENT_NO_RESOURCES Name:          SNMPD_SUBAGENT_NO_RESOURCES Message:       No resources available for subagent (<subagent-name>): <error-message> Help:          Subagent resources were temporarily exhausted Description:   The SNMP agent process (snmpd) uses certain resources for communication with subagents. Resources were not available                for communication with the indicated subagent. Type:          Error: An error occurred Severity:      notice Cause:         An internal software failure occurred. Action:        Contact your technical support representative. </li></ul>
  • 51. How to find out the data between 2 proc sockets? <ul><li>Find out the processes ID (use snmpd and mib2d as example) </li></ul><ul><li>root@Kelly_RE0% ps -aux | egrep -i &quot;snmpd|mib2d&quot; </li></ul><ul><li>root 8322 0.0 0.2 5036 3932 ?? S 4Feb08 0:12.24 /usr/sbin/snmpd -N </li></ul><ul><li>root 8302 0.0 0.2 4464 3892 ?? I 4Feb08 0:10.35 /usr/sbin/mib2d –N </li></ul><ul><li>Find out socket stream. </li></ul><ul><li>root@Kelly_RE0% fstat -p 8302 </li></ul><ul><li>USER CMD PID FD MOUNT INUM MODE SZ|DV R/W </li></ul><ul><li>..... </li></ul><ul><li>root mib2d 8302 17* local stream faab6c80 <-> fab03e60 </li></ul><ul><li>root@Kelly_RE0% fstat -p 8322 </li></ul><ul><li>USER CMD PID FD MOUNT INUM MODE SZ|DV R/W </li></ul><ul><li>..... </li></ul><ul><li>root snmpd 8322 15* local stream fab03e60 <-> faab6c80 </li></ul><ul><li>3. Then, check the socket data. </li></ul><ul><li>root@Kelly_RE0% netstat -Aan | egrep -i &quot;mib2d|snmpd|Send&quot; </li></ul><ul><li>PCB Proto Recv-Q Send-Q Local Address Foreign Address (state) </li></ul><ul><li>PCB Proto Recv-Q Send-Q Local Address Foreign Address (state) </li></ul><ul><li>Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr </li></ul><ul><li>f5f4e6c0 stream 0 0 0 faad35a0 0 0 /var/run/snmpd_stream </li></ul><ul><li>f5f4b300 stream 0 0 0 faa47aa0 0 0 /var/run/snmpd_stream </li></ul><ul><li>f5f4fc20 stream 0 0 0 fab67dc0 0 0 </li></ul>
  • 52. How to do RMA? <ul><li>Logistics </li></ul><ul><li>csr-apac(emea, usa) </li></ul>
  • 53. Trouble shoot T-series <ul><li>show chassis hardware </li></ul><ul><li>show pfe statistics traffic </li></ul><ul><li>show interface [int] extensive </li></ul><ul><li>start shell </li></ul><ul><li>su </li></ul><ul><li>vty fpc[x] </li></ul><ul><li>show sys mess </li></ul><ul><li>show nvram </li></ul><ul><li>show lchip ifd </li></ul><ul><li>show ifl brief </li></ul><ul><li>show lchip [x] error </li></ul><ul><li>show lchip [x] lout stat </li></ul><ul><li>show lchip [x] lout sw lsif </li></ul><ul><li>show lchip [x] lout sw desrd </li></ul><ul><li>show lchip [x] lout sw hdrf </li></ul><ul><li>show lchip [x] lout sw nlif </li></ul><ul><li>show lchip [x] lout hw lsif </li></ul><ul><li>show lchip [x] lout hw nlif </li></ul><ul><li>show lchip [x] lout hw hdrf </li></ul>show lchip [x] lout hw nlif show lchip [x] stream [stream_#] show lchip [x] lout registers lsif lsif [stream_#] ( where [stream_#] is the stream you found which corresponds to the interface that has the problem using the show lchip ifd output above ) show lchip [x] lout registers nlif nlif
  • 54. Trouble shoot T-series <ul><li>start shell </li></ul><ul><li>su </li></ul><ul><li>vty fpc[x] </li></ul><ul><li>show sys mess </li></ul><ul><li>show nvram </li></ul><ul><li>show lchip ifd </li></ul><ul><li>show ifl brief </li></ul><ul><li>show lchip [x] error </li></ul><ul><li>show lchip [x] lout stat </li></ul><ul><li>show lchip [x] lout sw lsif </li></ul><ul><li>show lchip [x] lout sw desrd </li></ul><ul><li>show lchip [x] lout sw hdrf </li></ul><ul><li>show lchip [x] lout sw nlif </li></ul><ul><li>show lchip [x] lout hw lsif </li></ul><ul><li>show lchip [x] lout hw nlif </li></ul><ul><li>show lchip [x] lout hw hdrf </li></ul><ul><li>show lchip [x] lout hw nlif </li></ul><ul><li>show lchip [x] stream [stream_#] </li></ul><ul><li>show lchip [x] lout registers lsif lsif [stream_#] </li></ul><ul><li>(where [stream_#] is the stream you have seen on the &quot;show lchip ifd&quot; </li></ul><ul><li>output under the lchip [x]) </li></ul><ul><li>show lchip [x] lout registers nlif nlif </li></ul><ul><li>show lchip [x] lout reg nlif dbufpart </li></ul><ul><li>show lchip [x] lout reg nlif bdispmon </li></ul><ul><li>Wait a little, hopefully after a few more errors go by. </li></ul><ul><li>show nchip [x] all </li></ul><ul><li>show mq [x] wan stat </li></ul><ul><li>show mq [x] wan stream active stat </li></ul><ul><li>Show chassis fabric topology Show chassis fabric sibs Show chassis fabric fpcs </li></ul>
  • 55. How to trouble shoot SNMP and MIB2d <ul><li>rtsockmon -c mib2d </li></ul><ul><li>rtsockmon -ge mib2d </li></ul><ul><li>show snmp statistics extensive </li></ul><ul><li>netstat –an </li></ul><ul><li>show system virtual-memory </li></ul><ul><li>[edit snmp] </li></ul><ul><li>lab@Johnny-re1# show </li></ul><ul><li>community public; </li></ul><ul><li>traceoptions { </li></ul><ul><li>file test size 10m; </li></ul><ul><li>flag all; </li></ul><ul><li>} </li></ul>
  • 56. How to trouble shoot routing and forwarding issues? <ul><li>FPC7(FED1DSRJ01-LAB-re0 vty)# show route ip prefix 192.12.1.2 </li></ul><ul><li>IPv4 Route Table 0, default.0, 0x0: </li></ul><ul><li>Destination NH IP Addr Type NH ID Interface </li></ul><ul><li>--------------------------------- --------------- -------- ----- --------- </li></ul><ul><li>192.12.1.2 Hold 716 ge-7/0/4.0 </li></ul>
  • 57. How to trouble shoot routing and forwarding issues? <ul><li>install@FED1DSRJ01-LAB-re0> show route forwarding-table destination 192.12.1.2 </li></ul><ul><li>Routing table: inet </li></ul><ul><li>Internet: </li></ul><ul><li>Destination Type RtRef Next hop Type Index NhRef Netif </li></ul><ul><li>192.12.1.2/32 dest 1 192.12.1.2 hold 716 2 ge-7/0/4.0 </li></ul><ul><li>Routing table: __juniper_private1__.inet </li></ul><ul><li>Internet: </li></ul><ul><li>Destination Type RtRef Next hop Type Index NhRef Netif </li></ul><ul><li>default perm 0 rjct 116 1 </li></ul><ul><li>Routing table: __juniper_private2__.inet </li></ul><ul><li>Internet: </li></ul><ul><li>Destination Type RtRef Next hop Type Index NhRef Netif </li></ul><ul><li>default perm 0 rjct 196 1 </li></ul><ul><li>Routing table: FED1J1MIS.inet </li></ul><ul><li>Internet: </li></ul><ul><li>Destination Type RtRef Next hop Type Index NhRef Netif </li></ul><ul><li>default perm 0 rjct 521 1 </li></ul><ul><li>Routing table: TEST-L3VPN.inet </li></ul><ul><li>Internet: </li></ul><ul><li>Destination Type RtRef Next hop Type Index NhRef Netif </li></ul><ul><li>default perm 0 rjct 530 1 </li></ul>
  • 58. How to trouble shoot routing and forwarding issues? <ul><li>install@FED1DSRJ01-LAB-re0> show arp </li></ul><ul><li>MAC Address Address Name Interface Flags </li></ul><ul><li>02:01:00:00:00:05 10.0.0.5 10.0.0.5 em0.0 none </li></ul><ul><li>00:04:80:9d:b5:00 10.1.1.1 10.1.1.1 fxp0.0 none </li></ul><ul><li>00:0c:29:9a:e5:38 10.1.1.115 10.1.1.115 fxp0.0 none </li></ul><ul><li>00:05:85:9b:5d:f5 31.1.1.2 31.1.1.2 ge-7/0/3.493 none </li></ul><ul><li>00:14:f6:56:b8:7e 68.1.0.204 68.1.0.204 ge-7/1/0.0 none </li></ul><ul><li>02:01:00:00:00:05 128.0.0.5 128.0.0.5 em0.0 none </li></ul><ul><li>00:00:c0:10:01:02 192.16.1.2 192.16.1.2 ge-7/0/5.0 none </li></ul><ul><li>Total entries: 7 </li></ul>
  • 59. How to trouble shoot routing and forwarding issues? <ul><li>install@FED1DSRJ01-LAB-re0> show arp </li></ul><ul><li>MAC Address Address Name Interface Flags </li></ul><ul><li>02:01:00:00:00:05 10.0.0.5 10.0.0.5 em0.0 none </li></ul><ul><li>00:04:80:9d:b5:00 10.1.1.1 10.1.1.1 fxp0.0 none </li></ul><ul><li>00:0c:29:9a:e5:38 10.1.1.115 10.1.1.115 fxp0.0 none </li></ul><ul><li>00:05:85:9b:5d:f5 31.1.1.2 31.1.1.2 ge-7/0/3.493 none </li></ul><ul><li>00:14:f6:56:b8:7e 68.1.0.204 68.1.0.204 ge-7/1/0.0 none </li></ul><ul><li>02:01:00:00:00:05 128.0.0.5 128.0.0.5 em0.0 none </li></ul><ul><li>00:00:c0:10:01:02 192.16.1.2 192.16.1.2 ge-7/0/5.0 none </li></ul><ul><li>Total entries: 7 </li></ul>
  • 60. How to trouble shoot routing and forwarding issues? <ul><li>install@FED1DSRJ01-LAB-re0> show route protocol ospf </li></ul><ul><li>inet.0: 260 destinations, 387 routes (186 active, 0 holddown, 77 hidden) </li></ul><ul><li>@ = Routing Use Only, # = Forwarding Use Only </li></ul><ul><li>+ = Active Route, - = Last Active, * = Both </li></ul><ul><li>0.0.0.0/0 *[OSPF/10] 09:25:03, metric 16777215 </li></ul><ul><li>Discard </li></ul><ul><li>3.1.1.0/24 *[OSPF/150] 09:23:28, metric 0, tag 0 </li></ul><ul><li>> via so-0/1/0.108 </li></ul><ul><li>10.1.0.0/16 [OSPF/150] 09:23:28, metric 0, tag 0 </li></ul><ul><li>> via so-0/1/0.108 </li></ul><ul><li>10.1.1.0/24 [OSPF/150] 09:23:28, metric 0, tag 0 </li></ul><ul><li>> via so-0/1/0.108 </li></ul><ul><li>10.1.200.0/28 [OSPF/150] 09:23:28, metric 0, tag 0 </li></ul><ul><li>> via so-0/1/0.108 </li></ul><ul><li>10.99.0.0/16 [OSPF/150] 09:23:28, metric 0, tag 0 </li></ul><ul><li>> via so-0/1/0.108 </li></ul><ul><li>10.99.99.0/24 [OSPF/150] 09:23:28, metric 0, tag 0 </li></ul><ul><li>> via so-0/1/0.108 </li></ul><ul><li>24.234.6.0/24 *[OSPF/10] 00:54:30, metric 182 </li></ul><ul><li>> to 68.1.0.204 via ge-7/1/0.0 </li></ul><ul><li>24.234.6.0/27 *[OSPF/10] 00:54:30, metric 166 </li></ul><ul><li>> to 68.1.0.204 via ge-7/1/0.0 </li></ul><ul><li>24.248.129.0/27 [OSPF/150] 09:23:28, metric 0, tag 0 </li></ul><ul><li>> via so-0/1/0.108 </li></ul>
  • 61. How to trouble shoot routing and forwarding issues? <ul><li>FFPC7(FED1DSRJ01-LAB-re0 vty)# show route ip prefix 192.12.1.2 </li></ul><ul><li>IPv4 Route Table 0, default.0, 0x0: </li></ul><ul><li>Destination NH IP Addr Type NH ID Interface </li></ul><ul><li>--------------------------------- --------------- -------- ----- --------- </li></ul><ul><li>192.12.1.2 192.12.1.2 Unicast 716 ge-7/0/4.0 </li></ul><ul><li>FFPC7(FED1DSRJ01-LAB-re0 vty)# show route ip lookup 192.12.1.2 </li></ul><ul><li>Route Information (192.12.1.2): </li></ul><ul><li>interface : ge-7/0/4.0 (87) </li></ul><ul><li>Nexthop prefix : 192.12.1.2 </li></ul><ul><li>Nexthop ID : 716 </li></ul><ul><li>MTU : 1514 </li></ul><ul><li>Class ID : 0 </li></ul><ul><li>FFPC7(FED1DSRJ01-LAB-re0 vty)# </li></ul>
  • 62. How to trouble shoot routing and forwarding issues? <ul><li>install@FED1DSRJ01-LAB-re0> show interfaces filters ge-7/0/4 </li></ul><ul><li>Interface Admin Link Proto Input Filter Output Filter </li></ul><ul><li>ge-7/0/4 up up </li></ul><ul><li>ge-7/0/4.0 up up inet </li></ul><ul><li>multiservice </li></ul><ul><li>FFPC7(FED1DSRJ01-LAB-re0 vty)# show nhdb interface ge-7/0/4 </li></ul><ul><li>ID Type Interface Next Hop Addr Protocol Encap MTU </li></ul><ul><li>----- -------- ------------- --------------- ---------- ------------ ---- </li></ul><ul><li>625 Bcast ge-7/0/4.0 - IPv4 Ethernet 0 </li></ul><ul><li>626 Receive ge-7/0/4.0 192.12.1.0 IPv4 Ethernet 0 </li></ul><ul><li>628 Resolve ge-7/0/4.0 - IPv4 Ethernet 0 </li></ul><ul><li>716 Unicast ge-7/0/4.0 192.12.1.2 IPv4 Ethernet 1514 </li></ul>
  • 63. Lab stuff <ul><li>Agilent Router Tester. Remote access: </li></ul><ul><li>Top 3 chassis: 172.19.59.28 </li></ul><ul><li>Bottom 3 chassis: 172.19.58.12 </li></ul><ul><li>User name: Administrator </li></ul><ul><li>Password: n2x </li></ul><ul><li>Launch pad </li></ul><ul><li>Create new session </li></ul><ul><li>For FE, need to config SFP </li></ul><ul><li>IXIA: VNC 172.19.58.2 (SV) 172.25.84.219(HD) </li></ul><ul><li>ixia-2.jtac-west </li></ul><ul><li>IXIA application server: 172.19.58.17 </li></ul>
  • 64. How to trouble shoot EOAM? <ul><li>http://www.juniper.net/techpubs/software/junos/junos82/swconfig82-network-interfaces/html/interfaces-ethernet-config50.html#1272612 </li></ul><ul><li>http://www.juniper.net/techpubs/software/junos/junos82/swconfig82-network-interfaces/html/interfaces-summary298.html#11618684 </li></ul><ul><li>Known PRs: </li></ul><ul><li>-PR81057 </li></ul>
  • 65. How to trouble shoot EOAM? <ul><li>protocols { </li></ul><ul><li>oam { </li></ul><ul><li>ethernet { </li></ul><ul><li>link-fault-management { </li></ul><ul><li>interfaces { </li></ul><ul><li>[xge/ge/fe]-<fpc>/<pic>/<port> { </li></ul><ul><li>pdu-interval <value>; </li></ul><ul><li>link-discovery <active|passive>; </li></ul><ul><li>pdu-threshold <count>; </li></ul><ul><li>remote-loopback; </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>
  • 66. How to trouble shoot EOAM? <ul><li>protocols { </li></ul><ul><li>oam { </li></ul><ul><li>ethernet { </li></ul><ul><li>link-fault-management { </li></ul><ul><li>interfaces { </li></ul><ul><li>[xge/ge/fe]-<fpc>/<pic>/<port> { </li></ul><ul><li>pdu-interval <value>; </li></ul><ul><li>link-discovery <active|passive>; </li></ul><ul><li>pdu-threshold <count>; </li></ul><ul><li>remote-loopback; </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>
  • 67. How to Manually mount a USB/CF storage? <ul><li>http://kb.juniper.net/KB8017 </li></ul><ul><li>First upload the desired JUNOS image to the router via ftp to /var/tmp. </li></ul><ul><li>Connect the USB mass storage device. </li></ul><ul><li>Format the USB device by dropping to shell (start shell) then enter &quot;dd if=/dev/zero of=/dev/da0 bs=128k&quot; (root access required). Note this step can take several minutes to complete with no output to the CLI window. </li></ul><ul><li>Label the device by entering &quot;disklabel -r -w da0 auto&quot;. (!! if you move the USB/CF around, you need to execut this command before mounting) </li></ul><ul><li>Create the file system with &quot;newfs -U /dev/da0c&quot;. </li></ul><ul><li>Create a dir to be used as a mount point with &quot;mkdir /var/tmp/usb&quot;. </li></ul><ul><li>Mount the USB device using &quot;mount /dev/da0c /var/tmp/usb&quot;. </li></ul><ul><li>df -h can be used to verify the mount. </li></ul><ul><li>Copy the JUNOS install image to the USB device. </li></ul><ul><li>cp /var/tmp/junos-jseries-8.0R2.8-domestic.tgz  /var/tmp/usb </li></ul><ul><li>Delete the original image to free up space on the CF. </li></ul><ul><li>rm /var/tmp/junos-jseries-8.0R2.8-domestic.tgz </li></ul><ul><li>Use the &quot;request system software add /var/tmp/usb/junos-jseries-8.0R2.8-domestic.tgz&quot; command to install the new JUNOS version. </li></ul>
  • 68. How to do tcpdump at Junos? <ul><li>You have to login as root </li></ul><ul><li>You have to know which incoming interface? </li></ul><ul><li>Command: </li></ul><ul><li>root@bananas-re0% tcpdump -xvf -i so-1/1/0 </li></ul>
  • 69. Ethernet OAM <ul><li>Ethernet OAM types      In short, there are two types of Ethernet OAM:      1.  Ethernet OAM as defined by 802.3ah   This is referred as LFM (Link Fault Management) and are identified   by the ether-type 0x8809 (slow protocol type packets), sub-type 3.      2.  Ethernet OAM as defined by IEEE 802.1ag   This is referred as CFM (Connectivity Fault Management) and can be   by the ether-type 0x8902. </li></ul><ul><li>Ethernet OAM implementation in JunOS      Ethernet OAM is implemented using the RE user space daemons &quot;lfmd&quot;      and &quot;cfmd&quot;.  Also, both &quot;lfmd&quot; and &quot;cfmd&quot; use the &quot;ppmd&quot; daemon on      the PFE for some periodic packet processing.      There is a packet processing path in the RE kernel as well in addition      to the daemons mentioned above. </li></ul>
  • 70. Ethernet OAM <ul><li>Ethernet OAM for regular Ethernet interfaces      Both 802.3ah (LFM) and 802.1ag (CFM) type Ethernet OAMs are supported in JunOS for the regular Ethernet interfaces with the following      restrictions.      802.3ah (LFM) type OAM can be configured only on the Ethernet IFDs      and NOT on the Ethernet IFLs.  Also, these packets are always VLAN      untagged.      However, 802.1ag (CFM) type OAM can be configured either on an Ethernet      IFD or IFL.  If this is configured on an IFD, the packets will be      always VLAN untagged.  If this is configured on an IFL, it will be      either VLAN tagged or untagged based on the &quot;vlan-tagging&quot; keyword      configuration on an Ethernet IFD. </li></ul>
  • 71. Ethernet OAM <ul><li>Link Monitoring </li></ul><ul><li>Link monitoring in Ethernet OAM detects and indicates link faults under a variety of conditions. Link monitoring uses the event notification OAMPDU and sends events to the remote OAMentity when there are problems detected on the link. The error events include the following: </li></ul><ul><li>• Error Symbol Period (error symbols per second)—The number of symbol errors that occurred during a specified period exceeded a threshold. These errors are coding symbol errors. </li></ul><ul><li>• Error Frame (error frames per second)—The number of frame errors detected during a specified period exceeded a threshold. </li></ul><ul><li>• Error Frame Period (error frames per n frames)—The number of frame errors within the last n frames has exceeded a threshold. </li></ul><ul><li>• Error Frame Seconds Summary (error seconds per m seconds)—The number of error seconds (1-second intervals with at least one frame error) within the last m seconds has exceeded a threshold. Since IEEE 802.3ah OAM does not provide a guaranteed delivery of any OAM PDU, the event </li></ul><ul><li>notification OAM PDU may be sent multiple times to reduce the probability of a lost notification. A sequence number is used to recognize duplicate events </li></ul>
  • 72. Ethernet OAM <ul><li>Ethernet OAM for regular Ethernet interfaces      Both 802.3ah (LFM) and 802.1ag (CFM) type Ethernet OAMs are supported in JunOS for the regular Ethernet interfaces with the following      restrictions.      802.3ah (LFM) type OAM can be configured only on the Ethernet IFDs      and NOT on the Ethernet IFLs.  Also, these packets are always VLAN      untagged.      However, 802.1ag (CFM) type OAM can be configured either on an Ethernet      IFD or IFL.  If this is configured on an IFD, the packets will be      always VLAN untagged.  If this is configured on an IFL, it will be      either VLAN tagged or untagged based on the &quot;vlan-tagging&quot; keyword      configuration on an Ethernet IFD. </li></ul>
  • 73. Ethernet OAM one scenario ( 2008-0401-0623) <ul><li>Scenario: Two T640s with JUNOS 8.2SR are connected together through an optical transport network (e.g., Fujitsu 7500/7600), using LAN-PHY on 10GE IQ2 PICs. </li></ul><ul><li>Question: If there is a link failure in the transport network and the 10GE links between the Fujitsu switches and the T640s stay up, will the Local T640 send out Ethernet 802.3ah OAMPDUs with the Flags for Critical Link Events(1) and the Link Event TLVs(2) to the Remote T640? </li></ul><ul><li>Answer: No. None of that will happen. What will happen is, the OAM Discovery INFO PDUs will timeout and both sides will detect that and mark a failure on their respective links. If only one direction of the link is down, one side will be in &quot;Active Send Local&quot; state and the other side will be in &quot;Send Local Remote&quot; state. There is no reason to send Link Event TLVs in the above situation as it's a link fault, not a framing error. </li></ul><ul><li>The reason we do not send Link-Fault or Dying Gasp is, by the time we detect a Rx fault, the ifd is marked down and the Tx is also brought down. The Critical Event is not defined in the 802.3ah for any specific purposes,and is implementation dependant. In Juniper implementation, we use Critical event to simulate RDI functionality. We only send Critical event in case we have a CCC-DOWN on the ifls on the interface marked by RPD and an action profile to send a critical event is defined. </li></ul>
  • 74. Ethernet OAM one scenario ( 2008-0401-0623) <ul><li>syslog {     </li></ul><ul><li>archive {        </li></ul><ul><li>files number ;        </li></ul><ul><li>  size size ;        </li></ul><ul><li>  ( world-readable | no- world-readable );    }     console {         facility severity ;    }     file filename {         facility severity ;         explicit-priority ;         match &quot; regular-expression &quot;;         archive {             files number ;             size size ;            ( world-readable | no- world-readable );        }    }     host ( hostname | other-routing-engine | scc-master) {         facility severity ;         explicit-priority ;         facility-override facility ;         log-prefix string ;         match &quot; regular-expression &quot;;    }     source-address source-address ; time -format (year | millisecond | year millisecond);     user ( username | *) {         facility severity ;         match &quot; regular-expression &quot;;    }} </li></ul>
  • 75. CoS configuration ( 2008-0523-0448) <ul><li>http://www.juniper.net/techpubs/software/junos/junos90/swconfig-cos/frameset.html </li></ul><ul><li>In the following classifier example, packets with EXP bits 000 are assigned to the data-queue forwarding class with a low loss priority, and packets with EXP bits 001 are assigned to the data-queue forwarding class with a high loss priority. </li></ul><ul><li>[edit class-of-service] </li></ul><ul><li>classifiers { </li></ul><ul><ul><li>exp exp_classifier { </li></ul></ul><ul><ul><ul><li>forwarding-class data-queue { </li></ul></ul></ul><ul><ul><ul><ul><li>loss-priority low code-points 000; </li></ul></ul></ul></ul><ul><ul><ul><ul><li>loss-priority high code-points 001; </li></ul></ul></ul></ul><ul><ul><ul><ul><li>} </li></ul></ul></ul></ul><ul><ul><li>} </li></ul></ul><ul><li>} </li></ul><ul><li>In the following drop-profile map example, the scheduler includes two drop-profile maps, which specify that packets are evaluated by the low-drop drop profile if they have a low loss priority and are from any protocol. Packets are evaluated by the high-drop drop profile if they have a high loss priority and are from any protocol. </li></ul><ul><li>[edit class-of-service] </li></ul><ul><li>schedulers { </li></ul><ul><ul><li>best-effort { </li></ul></ul><ul><ul><ul><li>drop-profile-map loss-priority low protocol any drop-profile low-drop; </li></ul></ul></ul><ul><ul><ul><li>drop-profile-map loss-priority high protocol any drop-profile high-drop; </li></ul></ul></ul><ul><ul><li>} </li></ul></ul><ul><li>} </li></ul><ul><li>In the following rewrite rule example, packets in the be forwarding class with low loss priority are assigned the EXP bits 000, and packets in the be forwarding class with high loss priority are assigned the EXP bits 001. </li></ul><ul><li>[edit class-of-service] </li></ul><ul><li>rewrite-rules { </li></ul><ul><ul><li>exp exp-rw { </li></ul></ul><ul><ul><ul><li>forwarding-class be { </li></ul></ul></ul><ul><ul><ul><ul><li>loss-priority low code-point 000; </li></ul></ul></ul></ul><ul><ul><ul><ul><li>loss-priority high code-point 001; </li></ul></ul></ul></ul>
  • 76. How to verify packages are corrupted? <ul><li>root@% mount /altroot </li></ul><ul><li>root@% mount /altconfig </li></ul><ul><li>root@% cd /altroot/packages/ </li></ul><ul><li>root@% sha1 j*8.5R3.4 </li></ul><ul><li>SHA1 (jbase-8.5R3.4) = 51a9f2cfe95a53d1dbda2daedd6b5dd6dd66213c </li></ul><ul><li>SHA1 (jdocs-8.5R3.4) = c56296f2016d5ddbf8b22c00cb8c06dc5c664271 </li></ul><ul><li>SHA1 (jkernel-8.5R3.4) = fedc82d6e8edb6b5ff972ac4c0f22885841ee48e </li></ul><ul><li>SHA1 (jpfe-T-8.5R3.4) = f8ea2b28cf27a168a1023b0e544cdfb047ac2f0e ---> corrupted </li></ul><ul><li>SHA1 (jpfe-common-8.5R3.4) = 0034ccbd5bd1b2bbd9b9ee41d3b42c50443e5562 ---> corrupted </li></ul><ul><li>SHA1 (jroute-8.5R3.4) = 5c22ca387a78d4a3cb47af79ef6bdcfa0e0bc26f </li></ul><ul><li>root@% sha1 /packages/j*8.5R3.4 </li></ul><ul><li>SHA1 (/packages/jbase-8.5R3.4) = 51a9f2cfe95a53d1dbda2daedd6b5dd6dd66213c </li></ul><ul><li>SHA1 (/packages/jdocs-8.5R3.4) = c56296f2016d5ddbf8b22c00cb8c06dc5c664271 </li></ul><ul><li>SHA1 (/packages/jkernel-8.5R3.4) = fedc82d6e8edb6b5ff972ac4c0f22885841ee48e </li></ul><ul><li>SHA1 (/packages/jpfe-T-8.5R3.4) = f14de1eb8e537a35088864192d6838bb24804492 </li></ul><ul><li>SHA1 (/packages/jpfe-common-8.5R3.4) = 270c4f4cc9c0afb6ba52c6916c2213eeba851ddc </li></ul><ul><li>SHA1 (/packages/jroute-8.5R3.4) = 5c22ca387a78d4a3cb47af79ef6bdcfa0e0bc26f </li></ul>
  • 77. Class-of-Service trouble shooting <ul><li>There is bug in Gimlet FPC where the PLP high defined at classifier will *NOT* be copied to notification. Thus if egress FPC might have rewrite rule messed up. </li></ul><ul><li>Gimlet FPC to Gimlet FPC has no problem. </li></ul><ul><li>Gimble FPC to Stoli FPC has problem </li></ul><ul><li>Gimlet FPC to Gimlet FPC with drop-profile has problem. </li></ul><ul><li>To work around this problem for scenario 2 & 3: </li></ul><ul><li>lab@slayer-re1# set class-of-service copy-plp </li></ul><ul><li>Default forwarding class: </li></ul><ul><li>Queue Forwarding-class </li></ul><ul><li>0 best-effort </li></ul><ul><li>1 Assured-forwarding </li></ul><ul><li>2 expedited-forwarding </li></ul><ul><li>3 network-control </li></ul>
  • 78. Class-of-Service trouble shooting <ul><li>http://www.juniper.net/techpubs/software/junos/junos90/swconfig-cos/swconfig-cos.pdf </li></ul><ul><li>Table 43: Default MPLS EXP Rewrite Table(P230) </li></ul><ul><li>------------------------------------------------ </li></ul><ul><li>Forwarding Class Loss Priority CoS Value </li></ul><ul><li>best-effort(0) low 000 </li></ul><ul><li>best-effort high 001 </li></ul><ul><li>expedited-forwarding(1) low 010 </li></ul><ul><li>expedited-forwarding high 011 </li></ul><ul><li>assured-forwarding(2) low 100 </li></ul><ul><li>assured-forwarding high 101 </li></ul><ul><li>network-control(3) low 110 </li></ul><ul><li>network-control high 111 </li></ul>
  • 79. Class-of-Service trouble shooting <ul><li>http://www.juniper.net/techpubs/software/junos/junos90/swconfig-cos/swconfig-cos.pdf </li></ul><ul><li>Table 42: Default Packet Header Rewrite Mappings (p225) </li></ul><ul><li>Map from Forwarding Class PLP Value Map to DSCP/DSCP IPv6/ EXP/IEEE/IP </li></ul><ul><li>expedited-forwarding low ef </li></ul><ul><li>expedited-forwarding high ef </li></ul><ul><li>assured-forwarding low af11 </li></ul><ul><li>assured-forwarding high af12 (DSCP/DSCP IPv6/EXP) </li></ul><ul><li>best-effort low be </li></ul><ul><li>best-effort high be </li></ul><ul><li>network-control low nc1/cs6 </li></ul><ul><li>network-control high nc2/cs7 </li></ul><ul><li>The mapping of alias to EXP code point is at next slide. Same thing to look up alias to DSCP code point. </li></ul>
  • 80. Class-of-Service trouble shooting <ul><li>lab@slayer-re1> show class-of-service code-point-aliases exp </li></ul><ul><li>Code point type: exp </li></ul><ul><li>Alias Bit pattern </li></ul><ul><li>af11 100 </li></ul><ul><li>af12 101 </li></ul><ul><li>be 000 </li></ul><ul><li>be1 001 </li></ul><ul><li>cs6 110 </li></ul><ul><li>cs7 111 </li></ul><ul><li>ef 010 </li></ul><ul><li>ef1 011 </li></ul><ul><li>nc1 110 </li></ul><ul><li>nc2 111 </li></ul>
  • 81. PLP Treatment on LMNR Platforms Overview
  • 82. Problem <ul><li>Customer Cox was seeing an increase of Non-Real-Time class traffic in the network when replacing IQ2 10GE PICs by 10GE XENPAK (non-IQ2) PICs. </li></ul><ul><li>Hard to isolate as there was a mix of traffic from different sources. </li></ul><ul><li>Initially though the problem was due to missclasification. </li></ul>
  • 83. Topology IP unlabeled Traffic IP unlabeled Traffic LSP xe-0/1/0
  • 84. Configuration: Forwarding Classes <ul><li>> ...service forwarding-classes </li></ul><ul><li>queue 0 BEST-EFFORT; </li></ul><ul><li>queue 1 NON-REAL-TIME; </li></ul><ul><li>queue 2 INTERACTIVE; </li></ul><ul><li>queue 3 REAL-TIME; </li></ul><ul><li>queue 4 VIDEO; </li></ul><ul><li>queue 5 VOICE; </li></ul><ul><li>queue 6 NETWORK-CONTROL; </li></ul>
  • 85. Configuration: IP-Prec. Classifier <ul><li>forwarding-class BEST-EFFORT { </li></ul><ul><li>loss-priority high code-points BEST-EFFORT-be; </li></ul><ul><li>} </li></ul><ul><li>forwarding-class NON-REAL-TIME { </li></ul><ul><li>loss-priority high code-points NON-REAL-TIME-af11; </li></ul><ul><li>} </li></ul><ul><li>forwarding-class INTERACTIVE { </li></ul><ul><li>loss-priority low code-points INTERACTIVE-af21; </li></ul><ul><li>} </li></ul><ul><li>forwarding-class REAL-TIME { </li></ul><ul><li>loss-priority low code-points REAL-TIME-af31; </li></ul><ul><li>} </li></ul><ul><li>forwarding-class VIDEO { </li></ul><ul><li>loss-priority low code-points VIDEO-af41; </li></ul><ul><li>} </li></ul><ul><li>forwarding-class VOICE { </li></ul><ul><li>loss-priority low code-points VOICE-ef; </li></ul><ul><li>} </li></ul><ul><li>forwarding-class NETWORK-CONTROL { </li></ul><ul><li>loss-priority low code-points NETWORK-CONTROL-nc1; </li></ul><ul><li>} </li></ul>inet-precedence { BEST-EFFORT-be 000; NON-REAL-TIME-af11 001; INTERACTIVE-af21 010; REAL-TIME-af31 011; VIDEO-af41 100; VOICE-ef 101; NETWORK-CONTROL-nc1 110; }
  • 86. Configuration: EXP Classifier <ul><li>forwarding-class BEST-EFFORT { </li></ul><ul><li>loss-priority high code-points BEST-EFFORT-be; </li></ul><ul><li>} </li></ul><ul><li>forwarding-class NON-REAL-TIME { </li></ul><ul><li>loss-priority high code-points NON-REAL-TIME-af11; </li></ul><ul><li>} </li></ul><ul><li>forwarding-class INTERACTIVE { </li></ul><ul><li>loss-priority low code-points INTERACTIVE-af21; </li></ul><ul><li>} </li></ul><ul><li>forwarding-class REAL-TIME { </li></ul><ul><li>loss-priority low code-points REAL-TIME-af31; </li></ul><ul><li>} </li></ul><ul><li>forwarding-class VIDEO { </li></ul><ul><li>loss-priority low code-points VIDEO-af41; </li></ul><ul><li>} </li></ul><ul><li>forwarding-class VOICE { </li></ul><ul><li>loss-priority low code-points VOICE-ef; </li></ul><ul><li>} </li></ul><ul><li>forwarding-class NETWORK-CONTROL { </li></ul><ul><li>loss-priority low code-points NETWORK-CONTROL-nc1; </li></ul><ul><li>} </li></ul>BEST-EFFORT-be 000; NON-REAL-TIME-af11 001; INTERACTIVE-af21 010; REAL-TIME-af31 011; VIDEO-af41 100; VOICE-ef 101; NETWORK-CONTROL-nc1 110;
  • 87. Configuration: Rewrite Rules, EXP <ul><li>exp WRITE-EXP { </li></ul><ul><li>forwarding-class BEST-EFFORT { </li></ul><ul><li>loss-priority low code-point BEST-EFFORT-be; </li></ul><ul><li>loss-priority high code-point BEST-EFFORT-be; </li></ul><ul><li>} </li></ul><ul><li>forwarding-class NON-REAL-TIME { </li></ul><ul><li>loss-priority low code-point NON-REAL-TIME-af11; </li></ul><ul><li>loss-priority high code-point NON-REAL-TIME-af11; </li></ul><ul><li>} </li></ul><ul><li>forwarding-class INTERACTIVE { </li></ul><ul><li>loss-priority low code-point INTERACTIVE-af21; </li></ul><ul><li>loss-priority high code-point INTERACTIVE-af21; </li></ul><ul><li>} </li></ul><ul><li>forwarding-class REAL-TIME { </li></ul><ul><li>loss-priority low code-point REAL-TIME-af31; </li></ul><ul><li>loss-priority high code-point REAL-TIME-af31; </li></ul><ul><li>} </li></ul><ul><li>forwarding-class VIDEO { </li></ul><ul><li>loss-priority low code-point VIDEO-af41; </li></ul><ul><li>loss-priority high code-point VIDEO-af41; </li></ul><ul><li>} </li></ul><ul><li>forwarding-class VOICE { </li></ul><ul><li>loss-priority low code-point VOICE-ef; </li></ul><ul><li>loss-priority high code-point VOICE-ef; </li></ul><ul><li>} </li></ul><ul><li>forwarding-class NETWORK-CONTROL { </li></ul><ul><li>loss-priority low code-point NETWORK-CONTROL-nc1; </li></ul><ul><li>loss-priority high code-point NETWORK-CONTROL-nc1; </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>
  • 88. PLP handling
  • 89. Which PLP ? <ul><li>The L to N notification cell contains two bits (three with tri-color marking) of interest: </li></ul><ul><li>The pseudo-plp bit: This is bit 2 of the QoS field (6-bits), and it’s used by the Lin BA Classifier and Rewrite rules </li></ul><ul><li>The real plp bit: this is a separate bit, see the Lin functional description for location. </li></ul>
  • 90. PLP On LMNR
  • 91. Example: IP packet, precedence 000, non-IQ2 PIC <ul><li>Let’s say we receive a packet with IP-Prec bits 000. Let’s say we have a BA Classifier that classifies IP-Prec: 000 as Best-Effort (queue 0) and plp=high: </li></ul><ul><li># show class-of-service code-point-aliases inet-precedence </li></ul><ul><li>BEST-EFFORT-be 000 ; </li></ul><ul><li>NON-REAL-TIME-af11 001; </li></ul><ul><li>INTERACTIVE-af21 010; </li></ul><ul><li>REAL-TIME-af31 011; </li></ul><ul><li>VIDEO-af41 100; </li></ul><ul><li>VOICE-ef 101; </li></ul><ul><li>NETWORK-CONTROL-nc1 110; </li></ul>
  • 92. Contd… <ul><li># show class-of-service classifiers inet-precedence CLASSIFY-IPP </li></ul><ul><li>forwarding-class BEST-EFFORT { </li></ul><ul><li>loss-priority high code-points 000; </li></ul><ul><li>} </li></ul><ul><li># show class-of-service forwarding-classes </li></ul><ul><li>queue 0 BEST-EFFORT; </li></ul><ul><li>queue 1 NON-REAL-TIME; </li></ul><ul><li>queue 2 INTERACTIVE; </li></ul><ul><li>queue 3 REAL-TIME; </li></ul><ul><li>queue 4 VIDEO; </li></ul><ul><li>queue 5 VOICE; </li></ul><ul><li>queue 6 NETWORK-CONTROL; </li></ul>
  • 93. Ctd… <ul><li>Because this packet’s real-plp bit will remain 0, RED will treat it as such. If we have the following rewrite rule: </li></ul><ul><li>apena@austinp-re0# show class-of-service rewrite-rules </li></ul><ul><li>exp WRITE-EXP { </li></ul><ul><li>forwarding-class BEST-EFFORT { </li></ul><ul><li>loss-priority low code-point 000; </li></ul><ul><li>loss-priority high code-point 000; <<<< </li></ul><ul><li>} </li></ul>
  • 94. Will this work ? <ul><li>The answer is: </li></ul><ul><ul><li>It depends on the incoming PIC. </li></ul></ul><ul><ul><li>By default we OR the LSB of EXP and DSCP with the real PLP (see flow chart): </li></ul></ul><ul><ul><ul><li>EXP 000 ORed with plp=1 gives EXP=001 </li></ul></ul></ul><ul><ul><ul><li>This produces incorrect classification at next hop router </li></ul></ul></ul><ul><ul><ul><li>With IQ2 PIC, Lin can write proper real PLP thanks to cookie. </li></ul></ul></ul><ul><ul><ul><li>Without IQ2, Lin can’t write real plp, just pseudo plp. </li></ul></ul></ul>
  • 95. Workaround: <ul><li>Use compatible markings </li></ul><ul><li>Enable copy-plp hidden knob. </li></ul><ul><li>Enable tri-color marking </li></ul>
  • 96. Multicast trouble shooting <ul><li>lab@ 320_1> show pim rps extensive </li></ul><ul><li>Instance: PIM.master </li></ul><ul><li>Address family INET </li></ul><ul><li>RP: 198.140.33.2 Learned from 198.140.33.7 via: auto-rp </li></ul><ul><li>Time Active: 17w5d 05:03:53 </li></ul><ul><li>Holdtime: 150 with 139 remaining </li></ul><ul><li>Device Index: 134 </li></ul><ul><li>Subunit: 32780 </li></ul><ul><li>Interface: pe-2/0/0.32780 </li></ul><ul><li>Group Ranges: </li></ul><ul><li>224.0.2.64/32, 139s remaining </li></ul><ul><li>224.0.2.65/32, 139s remaining </li></ul><ul><li>224.0.2.66/32, 139s remaining </li></ul><ul><li>224.0.2.67/32, 139s remaining </li></ul><ul><li>Active groups using RP: </li></ul><ul><li>233.43.202.9 </li></ul><ul><li>233.43.202.8 </li></ul>
  • 97. IPSec configuration and troubleshooting <ul><li>This is a wiki for a very bad Google IPSeT defrag case. </li></ul><ul><li>http://confluence.jnpr.net/confluence/display/IPGE/Google+2009-0106-+IPSec+Fragmentation+Issue+-+PR+414885 </li></ul>
  • 98. IPSec configuration and troubleshooting <ul><li>lab@kings-re0# show services </li></ul><ul><li>service-set ny2ny02jt-payload { </li></ul><ul><li>max-flows 2m; </li></ul><ul><li>next-hop-service { </li></ul><ul><li>inside-service-interface sp-0/0/0.1; </li></ul><ul><li>outside-service-interface sp-0/0/0.2; </li></ul><ul><li>} </li></ul><ul><li>ipsec-vpn-options { </li></ul><ul><li>local-gateway 200.1.1.2; </li></ul><ul><li>} </li></ul><ul><li>ipsec-vpn-rules ny2ny02jt-payload; </li></ul><ul><li>} </li></ul><ul><li>ipsec-vpn { </li></ul><ul><li>rule ny2ny02jt-payload { </li></ul><ul><li>term 1 { </li></ul><ul><li>then { </li></ul><ul><li>remote-gateway 200.1.1.1; </li></ul><ul><li>dynamic { </li></ul><ul><li>ike-policy ny2ny02jt-payload; </li></ul><ul><li>ipsec-policy stream; </li></ul><ul><li>} </li></ul><ul><li>tunnel-mtu 9188; </li></ul><ul><li>anti-replay-window-size 1024; </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>match-direction input; </li></ul><ul><li>} </li></ul>ipsec { proposal brook { protocol esp; authentication-algorithm hmac-md5-96; encryption-algorithm 3des-cbc; } policy stream { proposals brook; } } ike { proposal rivlet { authentication-method pre-shared-keys; dh-group group1; authentication-algorithm md5; encryption-algorithm 3des-cbc; } policy ny2ny02jt-payload { mode main; proposals rivlet; pre-shared-key ascii-text &quot;$9$O4v9BEyleWXxd&quot;; ## SECRET-DATA } } establish-tunnels immediately; }
  • 99. IPSec configuration and troubleshooting <ul><li>On T640 or other platforms where you have service PIC, you need to configure </li></ul><ul><li>the SP interfaces. </li></ul><ul><li>lab@kings-re0# show interfaces sp-0/0/0 </li></ul><ul><li>description ipsec-vpn; </li></ul><ul><li>mtu 9192; </li></ul><ul><li>unit 1 { </li></ul><ul><li>description ipsec-vpn-inside; </li></ul><ul><li>family inet; </li></ul><ul><li>service-domain inside; </li></ul><ul><li>} </li></ul><ul><li>unit 2 { </li></ul><ul><li>description ipsec-vpn-outside; </li></ul><ul><li>family inet; </li></ul><ul><li>service-domain outside; </li></ul><ul><li>} </li></ul>Direct traffic to the IPSec tunnel. 1) Static route lab@kings-re0# show routing-options static { route 172.0.0.0/8 { next-hop 172.25.44.1; retain; no-readvertise; } route 0.0.0.0/0 { next-hop sp-0/0/0.1; retain; } } 2) IGP 3) BGP
  • 100. IPSec configuration and troubleshooting <ul><li>lab@kings-re0# run ping 111.0.0.1 </li></ul><ul><li>PING 111.0.0.1 (111.0.0.1): 56 data bytes </li></ul><ul><li>64 bytes from 111.0.0.1: icmp_seq=0 ttl=64 time=1.335 ms </li></ul><ul><li>64 bytes from 111.0.0.1: icmp_seq=1 ttl=64 time=1.026 ms </li></ul><ul><li>64 bytes from 111.0.0.1: icmp_seq=2 ttl=64 time=1.050 ms </li></ul><ul><li>64 bytes from 111.0.0.1: icmp_seq=3 ttl=64 time=1.065 ms </li></ul><ul><li>64 bytes from 111.0.0.1: icmp_seq=4 ttl=64 time=1.032 ms </li></ul><ul><li>64 bytes from 111.0.0.1: icmp_seq=5 ttl=64 time=0.869 ms </li></ul><ul><li>64 bytes from 111.0.0.1: icmp_seq=6 ttl=64 time=1.078 ms </li></ul><ul><li>64 bytes from 111.0.0.1: icmp_seq=7 ttl=64 time=0.905 ms </li></ul><ul><li>64 bytes from 111.0.0.1: icmp_seq=8 ttl=64 time=1.073 ms </li></ul><ul><li>64 bytes from 111.0.0.1: icmp_seq=9 ttl=64 time=1.084 ms </li></ul><ul><li>64 bytes from 111.0.0.1: icmp_seq=10 ttl=64 time=0.885 ms </li></ul><ul><li>64 bytes from 111.0.0.1: icmp_seq=11 ttl=64 time=1.095 ms </li></ul><ul><li>64 bytes from 111.0.0.1: icmp_seq=12 ttl=64 time=0.948 ms </li></ul><ul><li>64 bytes from 111.0.0.1: icmp_seq=13 ttl=64 time=0.912 ms </li></ul>lab@jazz-re0> monitor traffic interface sp-0/0/0.1 verbose output suppressed, use <detail> or <extensive> for full protocol decode Address resolution is ON. Use <no-resolve> to avoid any reverse lookup delay. Address resolution timeout is 4s. Listening on sp-0/0/0.1, capture size 96 bytes Reverse lookup for 111.0.0.1 failed (check DNS reachability). Other reverse lookup failures will not be reported. Use <no-resolve> to avoid reverse lookups on IP addresses. 19:03:10.506267 In IP 101.1.1.1 > 111.0.0.1: ICMP echo request, id 51991, seq 6, length 64 19:03:10.506285 Out SERVICES service id 64 flags 0x82 service set id 1 iif 78 IP 111.0.0.1 > 101.1.1.1: ICMP echo reply, id 51991, seq 6, length 64 19:03:11.507050 In IP 101.1.1.1 > 111.0.0.1: ICMP echo request, id 51991, seq 7, length 64 19:03:11.507061 Out SERVICES service id 64 flags 0x82 service set id 1 iif 78 IP 111.0.0.1 > 101.1.1.1: ICMP echo reply, id 51991, seq 7, length 64 19:03:12.507977 In IP 101.1.1.1 > 111.0.0.1: ICMP echo request, id 51991, seq 8, length 64 19:03:12.507988 Out SERVICES service id 64 flags 0x82 service set id 1 iif 78 IP 111.0.0.1 > 101.1.1.1: ICMP echo reply, id 51991, seq 8, length 64 19:03:13.508794 In IP 101.1.1.1 > 111.0.0.1: ICMP echo request, id 51991, seq 9, length 64 19:03:13.508802 Out SERVICES service id 64 flags 0x82 service set id 1 iif 78 IP 111.0.0.1 > 101.1.1.1: ICMP echo reply, id 51991, seq 9, length 64 19:03:14.509561 In IP 101.1.1.1 > 111.0.0.1: ICMP echo request, id 51991, seq 10, length 64
  • 101. IPSec configuration and troubleshooting <ul><li>lab@jazz-re0# run show log kmd </li></ul><ul><li>Jul 17 18:32:20 jazz-re0 clear-log[8331]: logfile cleared </li></ul><ul><li>Jul 17 18:33:26 Initialising the KMD ipsec-interface-id pool </li></ul><ul><li>Jul 17 18:33:26 Deleted SA pair with index=0 tunnel index=1 to kernel </li></ul><ul><li>Jul 17 18:33:26 Initializing certificate manager </li></ul><ul><li>Jul 17 18:33:26 Added SA pair with index=0 tunnel index=1 PIC index=0 Interface name: sp-0/0/0 Length:1392 to kernel </li></ul><ul><li>Jul 17 18:34:06 Added SA pair with index=1 tunnel index=1 PIC index=0 Interface name: sp-0/0/0 Length:1392 to kernel </li></ul><ul><li>Jul 17 18:34:11 Added SA pair with index=2 tunnel index=1 PIC index=0 Interface name: sp-0/0/0 Length:1392 to kernel </li></ul><ul><li>Jul 17 18:57:25 Initialising the KMD ipsec-interface-id pool </li></ul><ul><li>Jul 17 18:57:38 Initialising the KMD ipsec-interface-id pool </li></ul><ul><li>Jul 17 18:58:53 Initialising the KMD ipsec-interface-id pool </li></ul><ul><li>Jul 17 19:31:56 Deleted SA pair with index=1 tunnel index=1 to kernel </li></ul><ul><li>Jul 17 19:31:56 Added SA pair with index=3 tunnel index=1 PIC index=0 Interface name: sp-0/0/0 Length:1392 to kernel </li></ul><ul><li>Jul 17 19:34:11 Deleted SA pair with index=2 tunnel index=1 to kernel </li></ul>
  • 102. IPSec configuration and troubleshooting <ul><li>lab@jazz-re0# run show log kmd </li></ul><ul><li>Jul 17 18:32:20 jazz-re0 clear-log[8331]: logfile cleared </li></ul><ul><li>Jul 17 18:33:26 Initialising the KMD ipsec-interface-id pool </li></ul><ul><li>Jul 17 18:33:26 Deleted SA pair with index=0 tunnel index=1 to kernel </li></ul><ul><li>Jul 17 18:33:26 Initializing certificate manager </li></ul><ul><li>Jul 17 18:33:26 Added SA pair with index=0 tunnel index=1 PIC index=0 Interface name: sp-0/0/0 Length:1392 to kernel </li></ul><ul><li>Jul 17 18:34:06 Added SA pair with index=1 tunnel index=1 PIC index=0 Interface name: sp-0/0/0 Length:1392 to kernel </li></ul><ul><li>Jul 17 18:34:11 Added SA pair with index=2 tunnel index=1 PIC index=0 Interface name: sp-0/0/0 Length:1392 to kernel </li></ul><ul><li>Jul 17 18:57:25 Initialising the KMD ipsec-interface-id pool </li></ul><ul><li>Jul 17 18:57:38 Initialising the KMD ipsec-interface-id pool </li></ul><ul><li>Jul 17 18:58:53 Initialising the KMD ipsec-interface-id pool </li></ul><ul><li>Jul 17 19:31:56 Deleted SA pair with index=1 tunnel index=1 to kernel </li></ul><ul><li>Jul 17 19:31:56 Added SA pair with index=3 tunnel index=1 PIC index=0 Interface name: sp-0/0/0 Length:1392 to kernel </li></ul><ul><li>Jul 17 19:34:11 Deleted SA pair with index=2 tunnel index=1 to kernel </li></ul>
  • 103. How to compare rollback? <ul><li>rprivette@CHRL-HAGG-03> show system rollback compare 0 2 </li></ul><ul><li>[edit interfaces ge-3/3/1 unit 3478] </li></ul><ul><li>- description &quot;16/VLXX/010009/TWCS - FREEMAN WHITE # 255277 [ENLAN]&quot;; </li></ul><ul><li>+ description &quot;16/KDFN/010010/TWCS - Freeman White # FW115671&quot;; </li></ul><ul><li>- encapsulation vlan-vpls; </li></ul><ul><li>+ encapsulation vlan-ccc; </li></ul><ul><li>+ family ccc { </li></ul><ul><li>+ policer { </li></ul><ul><li>+ input LIMIT_10M; </li></ul><ul><li>+ output LIMIT_10M; </li></ul><ul><li>+ } </li></ul><ul><li>+ } </li></ul><ul><li>- family vpls { </li></ul><ul><li>- policer { </li></ul><ul><li>- input LIMIT_10M; </li></ul><ul><li>- output LIMIT_10M; </li></ul><ul><li>- } </li></ul><ul><li>- } </li></ul>
  • 104. MX VLAN configuration – what are the new stuff? <ul><li>STP’s: original 802.1D </li></ul><ul><li>MSTP: based on 802.1s </li></ul><ul><li>RSTP: based on 802.1w </li></ul><ul><li>MISTP: Cisco Proprietary Multiple Instance STP </li></ul><ul><li>PVST+: Per-VLAN spanning-tree plus </li></ul><ul><li>Rapid PVST+ </li></ul>
  • 105. MX VLAN Trunking configuration – General guideline <ul><li>Generally, there are four things that you must configure in an L2 environment: </li></ul><ul><li>Interfaces and virtual LAN (VLAN) tags—L2 interfaces are usually various type of Ethernet links with VLAN tags used to connect to customer devices or other bridges or routers. </li></ul><ul><li>Bridge domains and virtual switches—Bridge domains limit the scope of media access control (MAC) learning (and thereby the size of the MAC table) and also determine where the device should propagate frames sent to broadcast, unknown unicast, and multicast (BUM) MAC addresses. Virtual switches allow for the configuration of multiple, independent bridge domains. </li></ul><ul><li>Spanning Tree Protocols (xSTP, where the “x” represents the STP type)—Bridges function by associating a MAC address with an interface, similar to the way a router associates an IP network address with a next-hop interface. Just as routing protocols use packets to detect and prevent routing loops, bridges use xSTP frames to detect and prevent bridging loops. (L2 loops are more devastating to a network because of the broadcast nature of Ethernet LANs.) </li></ul><ul><li>Integrated bridging and routing (IRB)—Support for both Layer 2 bridging and Layer 3 routing on the same interface. Frames are bridged if they are not sent to the router's MAC address. Frames sent to the router's MAC address are routed to other interfaces configured for Layer 3 routing. </li></ul>
  • 106. MX VLAN Trunking configuration – vlan tagging <ul><li>interfaces ge-2/2/6 { </li></ul><ul><ul><li>encapsulation flexible-ethernet-services; </li></ul></ul><ul><ul><li>vlan-tagging; # Customer interface uses singly-tagged frames </li></ul></ul><ul><ul><li>unit 200 { </li></ul></ul><ul><ul><ul><li>encapsulation vlan-bridge; </li></ul></ul></ul><ul><ul><ul><li>vlan-id 200; </li></ul></ul></ul><ul><ul><li>} </li></ul></ul><ul><li>} </li></ul><ul><li>interfaces ae1 { </li></ul><ul><ul><li>encapsulation extended-vlan-bridge; </li></ul></ul><ul><ul><li>vlan-tagging; </li></ul></ul><ul><ul><li>unit 100 { </li></ul></ul><ul><ul><ul><li>vlan-id 100; </li></ul></ul></ul><ul><ul><li>} </li></ul></ul><ul><ul><li>unit 200 { </li></ul></ul><ul><ul><ul><li>vlan-id 200; </li></ul></ul></ul><ul><ul><li>} </li></ul></ul><ul><li>} </li></ul>
  • 107. MX VLAN Trunking configuration – bridge domain <ul><li>Configure the virtual switches and bridge domains on all three routers. There is always a default virtual switch in the router for L2 functions; however, if there is only one L2 network, then the virtual switch instance type is not needed. </li></ul><ul><li>Configure a bridge domain on Router 1: </li></ul><ul><li>[edit] </li></ul><ul><li>bridge-domains { </li></ul><ul><ul><li>vlan100 { </li></ul></ul><ul><ul><ul><li>domain-type bridge; </li></ul></ul></ul><ul><ul><ul><li>vlan-id 100; </li></ul></ul></ul><ul><ul><ul><li>interface ge-2/2/1.100; </li></ul></ul></ul><ul><ul><ul><li>interface ae1.100; </li></ul></ul></ul><ul><ul><ul><li>interface ae2.100; </li></ul></ul></ul><ul><ul><li>} </li></ul></ul><ul><ul><li>vlan200 { </li></ul></ul><ul><ul><ul><li>domain-type bridge; </li></ul></ul></ul><ul><ul><ul><li>vlan-id 200; </li></ul></ul></ul><ul><ul><ul><li>interface ge-2/2/1.200; </li></ul></ul></ul><ul><ul><ul><li>interface ge-2/2/6.200; </li></ul></ul></ul><ul><ul><ul><li>interface ae1.200; </li></ul></ul></ul><ul><ul><ul><li>interface ae2.200; </li></ul></ul></ul><ul><ul><li>} </li></ul></ul><ul><li>} </li></ul>
  • 108. MX VLAN Trunking configuration – MSTP-1 <ul><li>Key words: </li></ul><ul><li>MSTI: Multiple Spanning Tree Instances </li></ul><ul><li>CIST: Common and Internal Spanning Tree </li></ul><ul><li>MSTP: Multiple Spanning Tree Protocol </li></ul><ul><li>Configuration name: The names must match to be in the same region </li></ul><ul><li>Revision Level: must be the same across the same region. </li></ul><ul><li>VLAN-to-MSTI mapping: vlans mapped to this MSTP instance. </li></ul>
  • 109. MX VLAN Truncking configuration – MSTP-2 <ul><li>protocols { </li></ul><ul><ul><li>mstp { </li></ul></ul><ul><ul><ul><li>configuration-name mstp-for-R1-2-3; # The names must match to be in the same region </li></ul></ul></ul><ul><ul><ul><li>revision-level 3; # The revision levels must match </li></ul></ul></ul><ul><ul><ul><li>bridge-priority 0; # This bridge acts as root bridge for VLAN 100 and 200 </li></ul></ul></ul><ul><ul><ul><li>interface ae1; </li></ul></ul></ul><ul><ul><ul><li>interface ae2; </li></ul></ul></ul><ul><ul><ul><li>msti 1 { </li></ul></ul></ul><ul><ul><ul><ul><li>vlan100; # This VLAN corresponds to MSTP instance 1 </li></ul></ul></ul></ul><ul><ul><ul><li>} </li></ul></ul></ul><ul><ul><ul><li>msti 2 { </li></ul></ul></ul><ul><ul><ul><ul><li>vlan200; # This VLAN corresponds to MSTP instance 2 </li></ul></ul></ul></ul><ul><ul><ul><li>} </li></ul></ul></ul><ul><ul><li>} </li></ul></ul><ul><li>} </li></ul>
  • 110. MX VLAN Truncking configuration – IRB-1 <ul><li>You configure IRB in two steps: </li></ul><ul><li>(1) Configure the IRB interface using the irb statement. </li></ul><ul><li>(2) Reference the IRB interface at the bridge domain level of the configuration. </li></ul><ul><li>IRB supports Layer 2 bridging and Layer 3 routing on the same interface. If the MAC address on the arriving frame is the same as that of the IRB interface, then the packet inside the frame is routed. Otherwise, the MAC address is learned or looked up in the MAC address database. </li></ul>
  • 111. MX VLAN configuration – IRB-2 <ul><li>edit interfaces] </li></ul><ul><li>xe-2/1/0 { </li></ul><ul><ul><li>unit 0 { </li></ul></ul><ul><ul><ul><li>family inet { </li></ul></ul></ul><ul><ul><ul><ul><li>address 10.0.10.2/24; # Routing interface </li></ul></ul></ul></ul><ul><ul><ul><li>} </li></ul></ul></ul><ul><ul><li>} </li></ul></ul><ul><li>} </li></ul><ul><li>irb { </li></ul><ul><ul><li>unit 0 { </li></ul></ul><ul><ul><ul><li>family inet { </li></ul></ul></ul><ul><ul><ul><ul><li>address 10.0.1.2/24 { </li></ul></ul></ul></ul><ul><ul><ul><ul><li>vrrp-group 1 { </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>virtual-address 10.0.1.51; </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>priority 254; </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><li>} </li></ul></ul></ul></ul><ul><ul><ul><li>} </li></ul></ul></ul><ul><ul><li>} </li></ul></ul><ul><li>} </li></ul><ul><ul><li>unit 1 { </li></ul></ul><ul><ul><ul><li>family inet { </li></ul></ul></ul><ul><ul><ul><ul><li>address 10.0.2.2/24 { </li></ul></ul></ul></ul><ul><ul><ul><ul><li>vrrp-group 2 { </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>virtual-address 10.0.2.51; </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>priority 100; </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><li>} </li></ul></ul></ul></ul><ul><ul><ul><li>} </li></ul></ul></ul><ul><ul><li>} </li></ul></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>bridge-domains { </li></ul><ul><ul><li>vlan-100 { </li></ul></ul><ul><ul><ul><li>domain-type bridge; </li></ul></ul></ul><ul><ul><ul><li>vlan-id 100; </li></ul></ul></ul><ul><ul><ul><li>interface ge-2/2/2.100; </li></ul></ul></ul><ul><ul><ul><li>interface ae1.100; </li></ul></ul></ul><ul><ul><ul><li>interface ae3.100 </li></ul></ul></ul><ul><ul><ul><li>routing-interface irb.0; </li></ul></ul></ul><ul><ul><li>} </li></ul></ul><ul><ul><li>vlan-200 { </li></ul></ul><ul><ul><ul><li>domain-type bridge; </li></ul></ul></ul><ul><ul><ul><li>vlan-id 200; </li></ul></ul></ul><ul><ul><ul><li>interface ge-3/3/3.200; </li></ul></ul></ul><ul><ul><ul><li>interface ae1.200; </li></ul></ul></ul><ul><ul><ul><li>interface ae3.200 </li></ul></ul></ul><ul><ul><ul><li>routing-interface irb.1; </li></ul></ul></ul><ul><ul><li>} </li></ul></ul><ul><li>} </li></ul>
  • 112. MX VLAN configuration- host interface <ul><li>New CLI introduced at in the fix of PR 299511 </li></ul><ul><li>lab@Atlas_re0# show interfaces ge-5/0/4 </li></ul><ul><li>encapsulation ethernet-bridge; </li></ul><ul><li>unit 0 { </li></ul><ul><li>    family bridge; </li></ul><ul><li>} </li></ul><ul><li>[edit] </li></ul><ul><li>lab@Atlas_re0# show interfaces ge-0/0/4    </li></ul><ul><li>encapsulation ethernet-bridge; </li></ul><ul><li>unit 0 { </li></ul><ul><li>    family bridge; </li></ul><ul><li>} </li></ul><ul><li>Bridge-domain{ </li></ul><ul><li>vlan333 { </li></ul><ul><li>    domain-type bridge; </li></ul><ul><li>    vlan-id 333; </li></ul><ul><li>    interface ge-5/0/4.0; </li></ul><ul><li>    interface ge-0/0/4.0; </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>
  • 113. Firewall Troubleshooting <ul><li>lab@slayer-re1> show firewall filter log-as0.0-i </li></ul><ul><li>Filter: log-as0.0-i </li></ul><ul><li>Counters: </li></ul><ul><li>Name Bytes Packets </li></ul><ul><li>rsvp-as0.0-i 0 0 </li></ul><ul><li>ospf-as0.0-i 0 0 </li></ul><ul><li>bgp-as0.0-i 0 0 </li></ul><ul><li>all-as0.0-i 149963421000 99975614 </li></ul>
  • 114. Firewall Troubleshooting -temp <ul><li>lab@slayer-re1> show firewall filter log-as0.0-i </li></ul><ul><li>Filter: log-as0.0-i </li></ul><ul><li>Counters: </li></ul><ul><li>Name Bytes Packets </li></ul><ul><li>rsvp-as0.0-i 0 0 </li></ul><ul><li>ospf-as0.0-i 0 0 </li></ul><ul><li>bgp-as0.0-i 0 0 </li></ul><ul><li>all-as0.0-i 149963421000 99975614 </li></ul>
  • 115. Firewall Troubleshooting -temp <ul><li>lab@slayer-re1> show firewall filter log-as0.0-i </li></ul><ul><li>Filter: log-as0.0-i </li></ul><ul><li>Counters: </li></ul><ul><li>Name Bytes Packets </li></ul><ul><li>rsvp-as0.0-i 0 0 </li></ul><ul><li>ospf-as0.0-i 0 0 </li></ul><ul><li>bgp-as0.0-i 0 0 </li></ul><ul><li>all-as0.0-i 149963421000 99975614 </li></ul>
  • 116. Firewall Troubleshooting -temp <ul><li>lab@slayer-re1> show firewall filter log-as0.0-i </li></ul><ul><li>Filter: log-as0.0-i </li></ul><ul><li>Counters: </li></ul><ul><li>Name Bytes Packets </li></ul><ul><li>rsvp-as0.0-i 0 0 </li></ul><ul><li>ospf-as0.0-i 0 0 </li></ul><ul><li>bgp-as0.0-i 0 0 </li></ul><ul><li>all-as0.0-i 149963421000 99975614 </li></ul>
  • 117. Firewall Troubleshooting -temp <ul><li>lab@slayer-re1> show firewall filter log-as0.0-i </li></ul><ul><li>Filter: log-as0.0-i </li></ul><ul><li>Counters: </li></ul><ul><li>Name Bytes Packets </li></ul><ul><li>rsvp-as0.0-i 0 0 </li></ul><ul><li>ospf-as0.0-i 0 0 </li></ul><ul><li>bgp-as0.0-i 0 0 </li></ul><ul><li>all-as0.0-i 149963421000 99975614 </li></ul>
  • 118. MX-960 pegasus DPC auto-nego <ul><li>https://tools.online.juniper.net/cm/case_note_detail.jsp?cid=Up9%2FoWPEU57FR9OFIsO0vQ%3D%3D&type=WQDDoTj%2Bp28%3D&num=fF6aYIYjhYCr4QBubu3%2BXg%3D%3D&isInternal=false </li></ul><ul><li>http://cvs.juniper.net/cgi-bin/viewcvs.cgi/sw-projects/platform/atlas/pegasus/pegasus_unit_test_plan.txt?rev=1.3&view=markup </li></ul><ul><li>7. Speed/Duplex selection from RE CLI - 100m/full-duplex </li></ul><ul><li>Goal: Test configuration of speed, link-mode from RE CLI </li></ul><ul><li>Test Steps: 1. Issue the below command on RE CLI -> set interfaces ge-x/y/z speed 100m link-mode full-duplex -> commit </li></ul><ul><li>2. Issue the below command on DPC console -> &quot;show bcm5466 registers y z&quot; </li></ul><ul><li>3. Compare the values from &quot;MII Control Register&quot; with Broadcom 5466 data sheet. </li></ul><ul><li>4. Issue the below command on DPC console -> &quot;show npez y rgmii z&quot; </li></ul><ul><li>Success Criteria: Description in the Data sheet should match with the values read. From output of step 4 verify rgmii rate </li></ul><ul><li>Result: PASS Output: Step 2: MII Control Register (0x00) : 0x3100 </li></ul><ul><li>Step 4: The rate of the RGMII port is 100Mb </li></ul>
  • 119. How to trouble shoot RSVP/LSP issues? <ul><li>RSVP related operational mode commands: </li></ul><ul><li>- clear rsvp session </li></ul><ul><li>- show rsvp session </li></ul><ul><li>- clear mpls lsp </li></ul><ul><li>- show mpls lsp </li></ul><ul><li>- show rsvp interface </li></ul><ul><li>- show ted database extensive </li></ul><ul><li>- </li></ul>
  • 120. How to trouble shoot RSVP/LSP issues? jtac@ar51.den1> show ted database 168.215.52.177 extensive TED database: 0 ISIS nodes 671 INET nodes NodeID: 168.215.52.177 Type: Rtr, Age: 271072 secs, LinkIn: 2, LinkOut: 2 Protocol: OSPF(0.0.0.0) To: 66.192.245.68-1 , Local: 66.192.245.78 , Remote: 0.0.0.0 Local interface index: 0, Remote interface index: 0 Color: 0 <none> Metric: 100 Static BW: 1000Mbps Reservable BW: 700Mbps Available BW [priority] bps: [0] 699.21Mbps [1] 699.21Mbps [2] 699.21Mbps [3] 699.21Mbps [4] 699.21Mbps [5] 699.21Mbps [6] 699.21Mbps [7] 699.21Mbps Interface Switching Capability Descriptor(1): Switching type: Packet Encoding type: Packet Maximum LSP BW [priority] bps: [0] 699.21Mbps [1] 699.21Mbps [2] 699.21Mbps [3] 699.21Mbps [4] 699.21Mbps [5] 699.21Mbps [6] 699.21Mbps [7] 699.21Mbps To: 66.192.245.116-1, Local: 66.192.245.126, Remote: 0.0.0.0 Local interface index: 0, Remote interface index: 0 Color: 0 <none> Metric: 100 Static BW: 1000Mbps Reservable BW: 700Mbps Available BW [priority] bps: [0] 699.07Mbps [1] 699.07Mbps [2] 699.07Mbps [3] 699.07Mbps [4] 699.07Mbps [5] 699.07Mbps [6] 699.07Mbps [7] 699.07Mbps Interface Switching Capability Descriptor(1): Switching type: Packet Encoding type: Packet Maximum LSP BW [priority] bps: [0] 699.07Mbps [1] 699.07Mbps [2] 699.07Mbps [3] 699.07Mbps [4] 699.07Mbps [5] 699.07Mbps [6] 699.07Mbps [7] 699.07Mbps
  • 121. How to trouble shoot commit problem? <ul><li>Commit synch | display details </li></ul><ul><li>Show log ksyncd, same as the /var/log/ksyncd </li></ul><ul><li>Roll back configuration of backup RE and sych up from RE0 </li></ul><ul><li>Copy configuration from master RE to backup RE: </li></ul><ul><li>Configure files are saved under /config. The running config is juniper.conf.gz.(execute this command from master RE, be careful of the permission on backup RE’s directory) </li></ul><ul><li>            rcp –T juniper.config.gz re1:/var/tmp </li></ul><ul><li>will copy the file to backup RE1’s /var/tmp directory </li></ul><ul><li># commit check </li></ul><ul><li>ushesh@re0-bb02.pdx02> show system commit </li></ul>
  • 122. Trouble shoot PFE CPU high <ul><li>start shell </li></ul><ul><li>vty fpc6 </li></ul><ul><li>sh nvram </li></ul><ul><li>sh syslog messages </li></ul><ul><li>FFPC4(cer-core-01 vty)# show pfe statistics traffic </li></ul><ul><li>FFPC4(cer-core-01 vty)# show pfe statistics notification </li></ul><ul><li>FFPC4(cer-core-01 vty)# show icmp statistics </li></ul><ul><li>Show chassis fpc (to find out fpc cpu utilization) </li></ul>
  • 123. Trouble shoot PFE CPU high <ul><li>start shell </li></ul><ul><li>vty fpc6 </li></ul><ul><li>sh nvram </li></ul><ul><li>sh syslog messages </li></ul><ul><li>FFPC4(cer-core-01 vty)# show pfe statistics traffic </li></ul><ul><li>FFPC4(cer-core-01 vty)# show pfe statistics notification </li></ul><ul><li>FFPC4(cer-core-01 vty)# show icmp statistics </li></ul><ul><li>Show chassis fpc (to find out fpc cpu utilization) </li></ul>
  • 124. 6PE trouble shooting <ul><li>PE configuration </li></ul><ul><li>lab@Magenta# show protocols </li></ul><ul><li>rsvp { </li></ul><ul><li>interface as0.0; </li></ul><ul><li>} </li></ul><ul><li>mpls { </li></ul><ul><li>ipv6-tunneling; </li></ul><ul><li>label-switched-path to_PE2 { </li></ul><ul><li>to 4.4.4.4; </li></ul><ul><li>} </li></ul><ul><li>interface as0.0; </li></ul><ul><li>} </li></ul><ul><li>bgp { </li></ul><ul><li>group purple { </li></ul><ul><li>type internal; </li></ul><ul><li>local-address 2.2.2.2; </li></ul><ul><li>family inet6 { </li></ul><ul><li>labeled-unicast { </li></ul><ul><li>explicit-null; </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>peer-as 100; </li></ul><ul><li>neighbor 4.4.4.4; </li></ul><ul><li>} </li></ul>group to_CE2 { type external; local-address 8002::1; family inet6 { unicast; } peer-as 300; neighbor 8002::2; } } isis { interface as0.0 { level 2 metric 10; } interface lo0.0; } fe-0/1/0 { unit 0 { family inet { address 99.1.1.1/24; } } } gr-1/2/0 { // GSR tunnel unit 100 { tunnel { source 99.1.1.1; destination 99.1.1.2; } family inet6 { address 8002::1/126; } } } lo0 { unit 0 { family inet { address 2.2.2.2/32; } family iso { address 49.0001.0005.0005.0005.00; } } }
  • 125. 6PE trouble shooting <ul><li>CE configuration </li></ul><ul><li>interfaces { </li></ul><ul><li>fe-0/1/0 { </li></ul><ul><li>unit 0 { </li></ul><ul><li>family inet { </li></ul><ul><li>address 99.1.1.2/24; </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>gr-1/2/0 { </li></ul><ul><li>unit 100 { </li></ul><ul><li>tunnel { </li></ul><ul><li>source 99.1.1.2; </li></ul><ul><li>destination 99.1.1.1; </li></ul><ul><li>} </li></ul><ul><li>family inet6 { </li></ul><ul><li>address 8002::2/126; </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>lo0 { </li></ul><ul><li>unit 0 { </li></ul><ul><li>family inet { </li></ul><ul><li>address 127.0.0.1/32; </li></ul><ul><li>} </li></ul><ul><li>family inet6 { </li></ul><ul><li>address 9001::5/128; </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>routing-options { static { route 172.0.0.0/8 { next-hop 172.19.58.1; no-readvertise; } } autonomous-system 300; } protocols { bgp { group to_PE2 { type external; local-address 8002::2; family inet6 { unicast; } export policy1; peer-as 100; neighbor 8002::1; } } }
  • 126. MPLS Auto-bandwidth <ul><li>Auto-bandwidth configuration </li></ul><ul><li>mpls { </li></ul><ul><li>apply-groups [ lspHigh-common lspStnd-common lsp-optimize-timer ]; </li></ul><ul><li>path-mtu { </li></ul><ul><li>rsvp mtu-signaling; </li></ul><ul><li>} </li></ul><ul><li>statistics { </li></ul><ul><li>file mpls.stat size 300k files 20 world-readable; </li></ul><ul><li>interval 300; </li></ul><ul><li>auto-bandwidth; </li></ul><ul><li>display-id; </li></ul><ul><li>} </li></ul><ul><li>traceoptions { </li></ul><ul><li>file mpls.log size 10m files 21 world-readable; </li></ul><ul><li>flag error; </li></ul><ul><li>flag state; </li></ul><ul><li>flag cspf; </li></ul><ul><li>flag connection; </li></ul><ul><li>flag graceful-restart; </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>label-switched-path lspStndT6toT1 { to 166.34.95.71; optimize-timer 60; node-link-protection; adaptive; auto-bandwidth { adjust-interval 300; adjust-threshold 10; minimum-bandwidth 100k; maximum-bandwidth 10g; adjust-threshold-overflow-limit 5; } primary use-ge-620; } path use-ge-620 { 192.100.36.37; }
  • 127. MPLS Auto-bandwith trouble shooting lab@Magenta> file show /var/log/mpls.stat Oct 30 15:41:21 trace_on: Tracing to &quot;/var/log/mpls.stat&quot; started to_PE2 132491 pkt 139233752 Byte Oct 30 15:41:21 2008 UTC Total 2 sessions: 1 success, 0 fail, 1 ignored Oct 30 15:43:09 trace_on: Tracing to &quot;/var/log/mpls.stat&quot; started to_PE2 132491 pkt 139233752 Byte 0 pps 0 Bps auto-bw 0 pkt 0 Byte Oct 30 15:43:09 2008 UTC Total 3 sessions: 2 success, 0 fail, 1 ignored Oct 30 15:44:19 trace_on: Tracing to &quot;/var/log/mpls.stat&quot; started auto-bw 0 pkt 0 Byte 0 pps 0 Bps Util 0.00% lab@Magenta> file show /var/log/mpls.log Oct 30 15:48:20 trace_on: Tracing to &quot;/var/log/mpls.log&quot; started Oct 30 16:03:09.172425 RPD_MPLS_PATH_BANDWIDTH_CHANGE: MPLS path (lsp auto-bw) bandwidth changed, path bandwidth 4140760 bps Oct 30 16:03:10.173337 RPD_MPLS_LSP_BANDWIDTH_CHANGE: MPLS LSP auto-bw bandwidth changed, lsp bandwidth 4140760 bps Oct 30 16:08:09.173234 RPD_MPLS_PATH_BANDWIDTH_CHANGE: MPLS path (lsp auto-bw) bandwidth changed, path bandwidth 1000 bps Oct 30 16:08:10.174771 RPD_MPLS_LSP_BANDWIDTH_CHANGE: MPLS LSP auto-bw bandwidth changed, lsp bandw
  • 128. MPLS Auto-bandwith trouble shooting edit protocols mpls statistics] lab@Magenta# run show mpls lsp extensive Ingress LSP: 1 sessions 4.4.4.4 From: 2.2.2.2, State: Up, ActiveRoute: 0, LSPname: auto-bw Description: test2 ActivePath: (primary) Node/Link protection desired LoadBalance: Random Autobandwidth MinBW: 1000bps MaxBW: 10Gbps AdjustTimer: 300 secs AdjustThreshold: 10% Max AvgBW util: 0bps, Bandwidth Adjustment in 5 second(s). Overflow limit: 5, Overflow sample count: 0 Encoding type: Packet, Switching type: Packet, GPID: IPv4 *Primary State: Up Priorities: 7 0 Bandwidth: 1.824kbps OptimizeTimer: 60 SmartOptimizeTimer: 180 Reoptimization in 18 second(s). Computed ERO (S [L] denotes strict [loose] hops): (CSPF metric: 10) 5.5.5.1 S Received RRO (ProtectionFlag 1=Available 2=InUse 4=B/W 8=Node 10=SoftPreempt): 5.5.5.1(Label=3) 90 Oct 30 17:27:24.553 CSPF: computation result ignored[5 times] 89 Oct 30 17:23:09.175 Record Route: 5.5.5.1(Label=3) 88 Oct 30 17:23:09.175 Up 87 Oct 30 17:23:09.175 Automatic Autobw adjustment succeeded
  • 129. NAT stuff To enable random port allocation, user has to configure &quot;set services nat pool <pool-name> port automatic random-allocation&quot; or &quot;set services nat pool <pool-name> port range low <low-port-num> high <high-port-num> random-allocation&quot;.
  • 130. How to look up RE CPU and Memory? lab@jazz-re0> show chassis routing-engine Routing Engine status: Slot 0: Current state Master Election priority Master (default) Temperature 41 degrees C / 105 degrees F CPU temperature 43 degrees C / 109 degrees F DRAM 3584 MB Memory utilization 13 percent CPU utilization: User 0 percent Background 0 percent Kernel 2 percent Interrupt 0 percent Idle 97 percent Model RE-A-2000 Serial ID 9009002764 Start time 2008-11-18 08:15:10 PST Uptime 8 hours, 54 minutes, 29 seconds Load averages: 1 minute 5 minute 15 minute 0.06 0.10 0.05
  • 131. Translate Cisco ATM to Juniper ATM interface ATM1/0/0 description ### Google DEDICADA### bandwidth 155000 no ip address no ip directed-broadcast no ip proxy-arp no ip mroute-cache load-interval 30 atm sonet stm-1 atm uni-version 3.1 no atm ilmi-keepalive no atm enable-ilmi-trap no snmp trap link-status ! interface ATM1/0/0.1 point-to-point description Link Google_Akwan (50Mbps)*5531004003 bandwidth 50000 ip address 200.162.89.161 255.255.255.252 no ip redirects no ip unreachables no ip directed-broadcast no ip proxy-arp no atm enable-ilmi-trap snmp trap link-status pvc 5531004003 2/901 vbr-nrt 55209 55209 1 no ilmi manage oam-pvc manage oam retry 10 5 1 encapsulation aal5snap ! !----------------------------
  • 132. Translate Cisco ATM to Juniper ATM chassis { fpc 0 { pic 3 { framing sdh; } } } interfaces { at-0/3/0 { atm-options { pic-type atm2; vpi 2; } unit 1 { encapsulation atm-snap; point-to-point; no-traps; vci 2.901; shaping { vbr peak 55209000 sustained 55209000 burst 1; } oam-period 10; oam-liveness { up-count 10; down-count 5; } family inet { address 200.162.89.162/30; } } }
  • 133. Translate Cisco ATM to Juniper ATM http://www.juniper.net/techpubs/software/junos/junos90/swconfig-network-interfaces/frameset.html
  • 134. T1 / T3 trouble shooting <ul><li>Loopback testing </li></ul><ul><li>http://www.juniper.net/techpubs/software/erx/erx41x/swconfig-physical-link/html/t1-e1-ji-config8.html </li></ul><ul><li>Either Local loopback or remote loopback can be configured at any given time. </li></ul><ul><li>For local loopback, best use an external loopback plug because it can also tests the PICs transmit and receive circuitry. </li></ul><ul><li>SONET, T1/DS1 type P-T-P interfaces support remote loopback </li></ul><ul><li>Configuring remote loopback only results in a line loop on local router. </li></ul><ul><li>Configuration: </li></ul><ul><li>sonet-options { </li></ul><ul><li>loopback local/remote; </li></ul><ul><li>} </li></ul>
  • 135. A good status write up [Action] Spoke with Bob Walsh and Mark Rippe. [Issue summarized] The issue was they were seeing physical layer T1 issues as well as intermittent ping loss. [Issue details] For T1 errors they were seeing BEE and LOF errors. When looking at the ping loss issue, [Start of cause analysis – top layer of root cause] I determined that the reason for network outage was due to PPP going down and renegotiating over and over again. [ real root cause] This was due to the T1 error condition. [ here is why the real root cause is] Setting t1-0/0/3 hold-time up 0 down 100 stabilized the PPP connection. But that does not resolve the underlying issue with the T1 errors. BEE and LOF indicates a problem with upstream provider equipment. BEE is typically triggered when upstream switch has a problem in TX side and then notifies the upstream equipment of the problem. LOF implies that we are not seeing frames on the link for a period of time. Bob had also tested same J2300 router and cable on Verizon T1 circuit and observed no errors. So not likely a J2300 hardware issue. [address possible doubt to prove the root cause] Cox testing with end-to-end loopback and all zeroes testing indicated no errors. However, it is possible that the testing equipment sensitivity may not be great enough detect the failure compared to Juniper router T1 interfaces which tend to be very sensitive to any errors on the line. Going forward we [workaround recommendation] recommend keeping hold-time configured on the T1 interface for this very reason. But ultimately it would be up to provider to correct any line defects. [game plan] Current action plan is to wait for new ATM circuit to be installed to bypass the Amica equipment that this J2300 connects to. That will likely occur within the next several days. Will keep case open in the interim.
  • 136. Juniper Smartd Issues PSN-2008-10-046 apparently covers multiple hdd related PRs. I looked at these PRs. If smartd is off, it may help PR/288011. However, I don't see how it would help PR/278580, PR/389540 and PR/390306.
  • 137. VPLS tagging configuration ****Old Way to config**** unit 25 { description &quot;DSH - ubr02 : 28/GCXG/061828//COXC&quot;; encapsulation vlan-ccc; vlan-id 25; input-vlan-map { swap; vlan-id 1212; } output-vlan-map swap; } *****New Way to config***** unit 4000 { description &quot;Lab - Todd SPN Test 1&quot;; encapsulation vlan-ccc; vlan-tags outer 4000 inner-range 1-4094; input-vlan-map { swap; vlan-id 1101; } output-vlan-map swap; } Got a case with vpls tagging. Customer closed this case immediately for the reason of mis-configuration. Might worth for reference in the future
  • 138. Juniper interface trouble shooting To disable keepalive on a point-to-point interface. This is a tricky one as I have kept forgot it. set no-keepalive
  • 139. Platform code name Atlas - The MX960, 14-slot carrier-class Ethernet platform, part of Harry. ATLAS Alexander - M40e ALEXANDER Autobahn - JUNOS upgrade to FreeBSD 6.1 Bellini - Bellini - Fine-grained (per VLAN) queuing for DPC (Dense Port Cards) on ATLAS Bombay - T320 BOMBAY Callypso - 7-slot chassis Ethernet switch MX480 Matrix takes Atlas cards, part of Harry.(IPG) Calvin - M7i CALVIN Chaser - M5 / M10 CHASER Cosmo - M 20 COSMO Dr Pepper - JUNOS on Saipan Flamingo - M320 FPCs Gibson - T640 GIBSON-LLC GIBSON-SHMC Gimlet - LMNR chipset GIMLET Greyhound - SONET OC768 PIC Haddock - HGE-PIC qpp HADDOCK Harry - Ethernet switch/router platforms HARRY Havana HAVANA
  • 140. Platform code name Heavy Metal - T640 based platform (IPG) Hobbes - M10i Hobson - TX platform HOBSON Hurricane - Hardware Stackable switch - Java Fixed configuration switches: - Espresso (Fixed configuration switch) - Latte (Virtual chassis Switch) - Caffeine : - Biscotti (Software) - Grande (8 slot 1.6Tbps chassis Switch) - Venti (16 slot 3.2Tbps chassis switch)
  • 141. Jsim Procedure (M120) lab@blackjack-re0> show chassis fpc-feb-connectivity lab@blackjack-re0> start shell pfe network feb0 RFEB0(blackjack-re0 vty)# show ichip ifd RFEB0(blackjack-re0 vty)# show ichip 0 r counters RFEB0(blackjack-re0 vty)# show ichip 0 iif statistics RFEB0(blackjack-re0 vty)# jsim reset full 0 (must reset) RFEB0(blackjack-re0 vty)# show ifl brief RFEB0(blackjack-re0 vty)# set jsim iif 73 (must bind intf) RFEB0(blackjack-re0 vty)# set jsim ipsrc 201.1.1.2 RFEB0(blackjack-re0 vty)# set jsim ipdst 200.1.1.2 RFEB0(blackjack-re0 vty)# jsim lookup verbose
  • 142. Jsim Procedure (M120) <ul><li>1) Find out which FPC (cFPC) is connected to which FEB </li></ul><ul><li>lab@blackjack-re0> show chassis fpc-feb-connectivity </li></ul><ul><li>FPC FPC type FPC state Connected FEB FEB state Link status </li></ul><ul><li>0 cFPC Online None </li></ul><ul><li>1 cFPC Online 1 Online OK </li></ul><ul><li>2 Type 3 Online 0 Online OK </li></ul><ul><li>3 Type 2 Online 3 Online OK </li></ul><ul><li>4 Type 2 Online 4 Online OK </li></ul><ul><li>Empty 5 Online </li></ul><ul><li>2) Console to the corresponding FEB (FEB 0 is connected to FPC3 @ slot 2) </li></ul><ul><li>lab@blackjack-re0> start shell pfe network feb0 </li></ul><ul><li>RFEB platform (666Mhz MPC 8541 processor, 512MB memory, 512KB flash) </li></ul><ul><li>RFEB0(blackjack-re0 vty)# exit </li></ul>
  • 143. Jsim Procedure (M120) 3) Find out which iCHIP is being used (from here, we know ICHIP 0 is being used) RFEB0(blackjack-re0 vty)# show ichip ifd I-chip global information: ICHIP 0: Initialized, Version 2, STREAM 32 (wan stream 0) has 1 IFDs. IFD 191: so-2/0/0 ICHIP 1: Not Initialized, ICHIP 2: Not Initialized, ICHIP 3: Not Initialized,
  • 144. Jsim Procedure (M120) 4) Collect some statistics of iCHIP 0 RFEB0(blackjack-re0 vty)# show ichip 0 r counters Traffic stats: Counter Name Total Rate Peak Rate ---------------------- ---------------- -------------- -------------- rcp_input_ucast 167035601285 31638906 39270060 (BYTE) 6868449722474 1265556255 1832927823 rcp_output_ucast 164600940855 31638902 39270077 (BYTE) 6771063304262 1265556088 1832926045 RFEB0(blackjack-re0 vty)# show ichip 0 iif statistics Traffic stats: Counter Name Total Rate Peak Rate ---------------------- ---------------- -------------- -------------- GFAB_BCNTR 91405146968728 592351311 784316693 KA_PCNTR 0 0 0 KA_BCNTR 0 0 0 Discard counters: Counter Name Total Rate Peak Rate ---------------------- ---------------- -------------- -------------- WAN_DROP_CNTR 2194246089959 7582075 11888478 FAB_DROP_CNTR 15144376205 0 2380431 KA_DROP_CNTR 0 0 0 HOST_DROP_CNTR 194 0 0
  • 145. Jsim Procedure (M120) 5) Reset JSIM ( everytime you change something, you need to reset JSIM) RFEB0(blackjack-re0 vty)# jsim reset full 0 6) Find out the interface ifl ( here it is 73) we will bind to JSIM lookup RFEB0(blackjack-re0 vty)# show ifl brief Index Name Type Encapsulation Flags ----- -------------------- ------------- -------------- ------ 71 ge-4/2/0.0 VLAN Tagged Ethernet 0x000000000000c000 73 so-2/0/0.0 Cisco HDLC Cisco HDLC 0x0000000000008010 72 ge-4/2/0.32767 VLAN Tagged Ethernet 0x000000000000c000 64 lo0.0 Unspecified Unspecified 0x0000000000000052
  • 146. Jsim Procedure (M120) 7) Bind iif to jsim and setup stream lookup key RFEB0(blackjack-re0 vty)# set jsim iif 73 RFEB0(blackjack-re0 vty)# set jsim ipsrc 201.1.1.2 RFEB0(blackjack-re0 vty)# set jsim ipdst 200.1.1.2 8) Finally, do the lookup (this is the data we are looking for) RFEB0(blackjack-re0 vty)# jsim lookup verbose Step Kp Address Data Description ---- -- ----------- -------- ----------- [ 1] 16 reg 000000 0000a679 nh: TID itable tid=10 offset=-7 itid 00000a 00040000 itable address (seg 0) 04000010 itable descriptor addr=0x000100 size=65536 idx_bits=16 bit_offset=0 lookup index=73 [ 2] 9 sram 00014b 10292f28 nh: extended buff-modify intermediate-nh addr=0x040a4a sram 040a4a 7840b2ab Buffer Translate: write kb(8), off 42, bits 12, data 0xffffc40 [ 3] 9 sram 040a4b 44060b61 nh: multiple SER(no SE) hops=1 addr=0x110182
  • 147. Tethereal to decode ixia packets. -bash-2.05b$ tethereal -r cap.enc -V Frame 1 (70 bytes on wire, 70 bytes captured) Arrival Time: Feb 4, 2017 16:03:16.453824000 Time delta from previous packet: 0.000000000 seconds Time relative to first packet: 0.000000000 seconds Frame Number: 1 Packet Length: 70 bytes Capture Length: 70 bytes Ethernet II, Src: 00:1f:12:23:e6:02, Dst: 00:00:c8:01:01:64 Destination: 00:00:c8:01:01:64 (AltosCom_01:01:64) Source: 00:1f:12:23:e6:02 (00:1f:12:23:e6:02) Type: IP (0x0800) Internet Protocol, Src Addr: 100.4.4.3 (100.4.4.3), Dst Addr: 200.1.1.100 (200.1.1.100) Version: 4 Header length: 20 bytes Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00) 0000 00.. = Differentiated Services Codepoint: Default (0x00) .... ..0. = ECN-Capable Transport (ECT): 0 .... ...0 = ECN-CE: 0
  • 148. IPSec SP-MTU and Tunnel-MTU(M/J series) On m-series: with sp-mtu of 1440, the max IP payload size that is 8 byte aligned is 1416, adding 20 bytes of IP header len results in 1436. On j-series: with mtu of 1446 (tunnel-mtu-ipsec overheads), the max IP payload size that is 8 byte aligned is 1424, adding 20 bytes of IP header len becomes 1444.
  • 149. Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net

×