• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Ann over

Ann over






Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Ann over Ann over Presentation Transcript

    • Flight Schedule “Killers”:Problems/Solutions in Phase C/D PM Challenge February 22-23, 2012 Presented by: Ann Over/GRC Co-Author: Diane Malarik/GRC
    • IntroductionMost every NASA flight project has technical challenges leadingto schedule and budget issues Our project had more than typical, especially in Phase C/DThis briefing covers these real world “schedule killers” and howwe overcame them to meet our requirementsThe name of the project is CoNNeCT: Communications, Navigation, and Networking reConfigurable Testbed 2
    • CoNNeCT / SCAN Testbed OverviewThe CoNNeCT Project will advance the Technology Readiness Level (TRL) of SpaceCommunications for future NASA missions. Launch to the International Space Station (ISS) on JAXA H-II Transfer Vehicle (HTV-3) NET May 2012 Utilizes a Flight Releasable Attachment Mechanism (FRAM)-based payload interface and is installed on the Expedite the Processing of Experiments to Space Station (ExPRESS) Logistics Carrier (ELC) at the ISS P3 location.CoNNeCT is a $100M+ payload funded by the NASASpace Communications and Navigation (SCaN) Program. Partners include GSFC, JPL, Harris and GD; other NASA Centers provide key support: KSC, JSC, MSFC. Flight System is the “SCAN Testbed”CoNNeCT is Class D and plans to operate for five yearson ISS (minimum design life is two years). Includes Ground System with a flight-like Ground Integration Unit (GIU) certified for S/W V&V 3
    • SCAN Testbed - Flight System Overview S-Band MGA JPL SDR Ka-Band HGA GD SDR S-Band LGA RF Subsystem Plate Assembly Thermostat Control L-Band LGA Assembly (GPS) Avionics Harris SDR ExPA (FRAM) Antenna Pointing S-Band LGASystem Integrated Gimbal Assembly (APS IGA) TWTA Power Supply Unit Harris Up/Dn Converter Gimbal Control Electronics (GCE) 4
    • Software Defined Radios (SDRs) Are the “Instruments” JPL/L-3 CE S-band SDR 6 MHz wide channel L-band receive (GPS) Virtex II, Sparc Processor , RTEMs OS 10 Mbps Class STRS Compliant Harris Up/Down ConvertorHarris General Dynamics Ka-band SDR S-band SDR 6 MHz wide channel 225 MHz wide channel Virtex IV, PowerPC Virtex II, ColdFire Processor (60 MIPS), Proc, DSP (1 VxWorks OS, CRAM (Chalcogenide GFLOP), VxWorks OS RAM) Memory >100 Mbps Class 10 Mbps Class STRS Compliant STRS Compliant
    • CoNNeCT Schedule – How it came out Preliminary Design Review to ship in two years PDR – September 2009 CDR – April 2010 Flight System in EMI Completed all major Environmental and Testing Vibe – Jan 2011 TVAC – Mar 2011 EMI – June 2011 Completed all major Interface Testing Closed Loop with ELC Power and C&DH TDRSS Emulator TDRSS Compatibility (Oct 2010 and Apr 2011) Completed Closed Loop Antenna Pointing System (APS) Pointing and Tracking & Comm Performance Testing July-Oct 2011Flight System in Vibe Flight System in TVAC 6
    • Top “Schedule Killers” for CoNNeCTPost PDR, there were literally too many issues to countPresented here are the most significant issues, someunique, many notIt is our hope that others can learn from our real worldproblems, mistakes, and ultimate solutions 7
    • Killer #1: RequirementsContext: Requirements were originally developed via bottoms-upapproach (instruments before bus); at PDR initial baselineincluded new top-down decomposition, but still were greatlyflawed – science and carrier interfacesChallenge: Major requirements work needed in parallel with finaldesign; critical path impact for schedule to CDRAction: Held a series of workshops to finalize the Flight SystemRequirements document, including real time edits, led byexperienced system engineer, with participation by keystakeholders, in priority order; by CDR completed gap analysis oflevel 4 and below and issued changes as neededResults: Requirements were acceptable for flight development(but still not perfect for all verifications, especially science)Take-away: Complete full decomposition of requirements at SRR,and assign verification owners by PDR; find system engineer(s)who can bridge science and engineering teams 8
    • Killer #2: Structural LoadsContext: Carrier was in co-development with payload andinterfaces were not well defined; radio instrument loadrequirements set too lowChallenge: Post-PDR, first coupled loads analysis showed thepayload had numerous negative margins and the radio loads werehigher than design; a non-linearity was discovered between thepassive and active mounting mechanisms and a new factor of 1.6was added to the requirements to accommodateAction: Conducted structural test article sine sweep test tovalidate the dynamic model; redesigned the thickness of radiatorplates and added significant number of fasteners; worked withradio vendors to increase the loads tolerance via analysis andtest; updated the model using better carrier interface model;implemented force-limiting for testing and analysis verifications 9
    • Killer #2: Structural Loads (cont) System Vibration Test • Model Statistics – 1,756,498 nodes – 938,874 elements– 10,538,988 degrees of freedom – With 1% of actual weight – First payload mode 64 Hz 10
    • Killer #2: Structural Loads (cont)Results: Made it through system vibration testing; final systemanalysis (close to ship ready) showed positive margins with someresidual risk for components not tested to full protoflight levels;final design changes very late to address negative margins(mounting pads).Take-away: It was very painful to have little to no structuralmargins in the beginning. Project had to accept more risk toproceed with manufacturing and system testing before loadsclosed. Build in ample structural margins from the beginning. 11
    • Killer #3: Heritage GimbalContext: For cost savings, CoNNeCT used heritage “flightqualified” designs from Lunar Reconnaissance Orbiter (LRO)including the Traveling Wave Tube Amplifier (TWTA) and theAntenna Pointing System (APS) Gimbal; the APS was a fixed pricepurchase drove the project critical path from day 1; TWTA was notan issueChallenge: The APS is safety critical, but was not human-ratedand was not designed for the structural or thermal environmentsAction: To meet ISS safety and environment requirements thegimbal was significantly redesigned. No RF radiation was allowedon ISS structure using geometrical analysis and hard stops; theAPS spec was rewritten three times based on a NASA-contractorco-design including 13 mods and cost growth from $3.4M to$6.6M; the entire system schedule was adjusted to accommodatea 12-month delivery slip including production of a high fidelityvibration simulator for system vibe and other simulators forsystem performance 12
    • Killer #3: Heritage Gimbal (cont) Baseline Design for CDR “Table Top”PDR design System CDR revised designRange of motion acceptable by ISS Flight Antenna Pointing System Safety Gimbal and Antennas 13
    • Killer #3: Heritage Gimbal (cont)Results: The critical path schedule never had down time waitingfor the gimbal; the vibration simulator was significant cost, butbought seven months of schedule to allow system vibrationtesting without the APS; system thermal vac testing proceededwith a flight-like simulator; significant development with closed-loop pointing algorithm completed with software simulator, thenbrassboard gimbal system; flight gimbal installation was nominaland EMI and System performance testing passed without issue(“miracle”).Take-away: Beware of heritage “flight qualified”hardware, especially for human rated systems that generally havestricter requirements. Do not use a purchase order if the designis not final. 14
    • Killer #4a & 4b: Spacewire Cables & Firmware/SoftwareContext: Given data rate requirements of >100 Mbps and otherNASA success with Spacewire (SpW), it was chosen as thebaseline. Spacewire FPGAs developed by two vendors.Challenge: SpW hardware is not robust and interface standards are not mature, leaving room for interpretation for software and firmware implementation. One SpW cable failed after moving payload to the Structural Dynamic Lab and data rates were not obtained. Others failed later in the development. Performance issues - high data rates worked but low rates didn‟t. Software drivers were only tool since FPGAs could not readily be changed. 15
    • Killer #4a/b: Spacewire Cables & Firmware/SoftwareAction: Hardware: rebuilt cable and replaced while on vibe table to proceed with testing; investigation showed root cause inside connector; further drop outs found and changed as technician moved cables; root cause determined to be combination of bend radii and tie downs; cable rebuilt and others re- routed (see next slide) Firmware/Software: created Tiger Team to investigate performance issues via analysis and test; eventually found major cause - interface incompatibility between firmware (see follow-on slide) and other performance problems in the firmware and software 16
    • Killer #4a: Spacewire Cables (cont) Puncturing caused failure.Wires are insulated/protected from rubbing with the SpaceWire Cable Routing/Mounting – addition of kynar sleeving. minimize bend radius and tie down with padding 17
    • Killer #4b: Spacewire Cables (cont) Problem Area JPL Aitech S950 SBC D.E. SpW PMC Data SpW Phy Phy FIFO SpW (2) Core Core PCI DMA FIFO Phy SpW PPC Local (2) & PCI (8) & Core Data GD 750 PCI Bus 60X Core DMA FIFO Phy SpW SpW Phy Arbiter (2) Core Core 60x Bus FIFO Data Phy SpW (2) Core Memory HarrisSolution Controller Cmd/Tlm SpW Phy Core Driver PLL Code SDRAM Problem was data interface (DMA) between the single board computer and the SpaceWire interface board. This is technically not a “SpaceWire” issue. 18
    • Killer #4: Spacewire (cont)Results: Hardware: Repairs done in parallel with other testing that didn‟t require SpW to minimize loss of schedule (radios included self-test functionality used instead of flowing through avionics); pulled in experts from GSFC for second hardware issue and followed their best practices that worked for other spacecraft Firmware/Software: Tiger Team (GRC/other NASA, vendor consultants) first looked at software-only solutions; after further investigation and determination of critical root cause(s), worked with vendor to change firmware and software team rewrote the drivers post-EMI testing; entire avionics unit was pulled to access firmware and new environmental testing completed; analysis showed FPGA change did not impact System EMI results; throughout year of SpW issues functional & environmental testing was conducted with incremental SpW performance capabilitiesTake-away: Spacewire performance is great, but buyer beware.Spacewire hardware is not robust and firmware/software interfacestandards/algorithms are not mature. Utilize FPGAs that arereliably reprogrammable without removal from system. 19
    • Killer #5: CM951 FPGA (Digital I/O)Context: The main processor utilized a CM951 board for digitalinput/output operations. During payload integratedtesting, vendor design flaw was uncovered: Uncommandedpower-on/power-off of key subsystems was observed. Behaviortraced to erroneous design in CM951 DIO card FPGA. Vendorconfirmed existence of design error (other customers alsoreported it)Challenge: This problem was uncovered very late and the onlyway to fix it was to remove the entire avionics unit, pull theboard, and replace the FPGA; DIO functions are also safetycritical.Action: Detailed problem investigation was initiated on theCoNNeCT Ground Integration Unit (GIU). The GIU is a high-fidelity, flight-like test unit that allowed thorough investigativemethods. Determined during high traffic FTP operations, correctDIO state values were being commanded and set for a fewmicroseconds, but DIO card would self-corrupt output uponsubsequent DIO activity. Vendor provided firmware fix at no cost; 20
    • Killer #5: CM951 FPGA (Digital I/O) DIO Board showing location of FPGADIO Boards inAvionics unit Photo showing DIO cards in Flight Avionics unit Flight-spare Avionics Unit Flight Avionics Unit (s/n prior to final close-out (s/n 002) 001) installed 21
    • Killer #5: CM951 FPGA (Digital I/O)Results: CM951 board returned to full functionality with minimalimpact on schedule (back-up used while prime avionicsrecertified), but significant project costs were incurred for extrawork. FPGA change-out done for SpaceWire board at same time.Take-away: Provide software team flight-like hardware to developand stress test core software drivers as early as possible; selectFPGA’s that can be reprogrammed reliably without systemdisassembly. 22
    • Killer #6: JPL Radio (“Gravity”)Context: The JPL radio includes S-band and GPS “slices” forcomm and navigation. The GPS slice can only fully be tested witha live GPS signal, so only limited signal generator testing wasdone at the system level.Challenge: The GPS risk was considered low, so signal generatortesting was not done at the system level until post-vibe. At thattime, significant performance degradation was observed for theGPS signals, especially L5 (see next slide).Action: JPL action team formed to develop root cause andconduct trouble-shooting. Measured accurate quantitativeperformance with live GPS test. Performance marginal to meetminimum requirements and decided to pull flight radio for vendorrepair.Results: Radio passed visual inspection and performance testingon bench with no issues. After much deliberation, finallydetermined it was a gravity issue with an RF seal and largergasket installed. Unit was requalified and reinstalled in system.Take-away: System installation can change test results. Don’t skiptesting even if perceived to be low risk. 23
    • Killer #6: JPL Radio (“Gravity”) Lid (warped) 0.044” max gap Gasket grove .029” deep 0.051” wideGasket 0.04” diameter Lid (warped) Correlation plots 24
    • Killer #7: Safety Critical SoftwareContext: The Phase 0/1 Flight Safety Review was conducted afterthe Avionics subsystem design was firm, limiting the designoptions to adequately address additional safety hazards identifiedduring the review process.Challenge: During EVA and EVRs, when crew and equipment couldcome into the direct line of sight of the Ka-band antenna, thepayload must provide two fault tolerance against inadvertentactivation of RF. The ELC carrier could provide only one powerinhibit; payload had to provide two more. Due to maturity of theavionics subsystem, HW solutions were ruled out.Action: Significant project resources were expended to modify theSW architecture to implement two verifiable and independentinhibits within a single CPU and to develop Safety Critical SW tocontrol the power to the subsystems that produce RF. 25
    • Killer #7: Safety Critical SoftwareResults: CoNNeCT is the first ISS Payload to demonstrateadequate Control Path Separation to allow two independent inhibitsto be controlled by a single CPU. The commands to power on thesubsystems that produce RF are identified as “Critical” and canonly be sent by POIC, yet are part of every day nominal operations.Take-away: Complete Phase 0/1 Flight Safety before thesubsystem designs are solidified. Safety Critical SW in human-rated space environment is expensive and should be last resort. 26
    • The “Perfect” Plan (Post-CDR) Top Level View of Baseline 17.6 days 27
    • “Reality” – Issues, But Minimized Schedule Impacts 17.6 days 28
    • ConclusionsSCAN Testbed is a technical and programmatic advancement ofcommunications technology that will pave the way for futurespace communication architectures.The NASA and Industry partnership provided significantadvancements on an extremely tight schedule Overcame many issues to meet the HTV-3 schedule Key to meeting schedule was to make progress with available functionalityLook ahead: Launch June 26, 2012 Experiments Program planned after commissioning in late 2012 29
    • AcknowledgementsThe authors wish to acknowledge the extraordinary team ofindividuals across the Glenn Research Center, the Jet PropulsionLab, the Goddard Space Flight Center, the Johnson SpaceCenter, the Kennedy Space Center, the Marshall Space FlightCenter, and our SDR partners, General Dynamic and HarrisCorporation.We extend our sincere appreciation to each team member for themany individual sacrifices. 30
    • 31
    • Back-up 32
    • CoNNeCT Operations & Objectives On-Orbit Networking TechnologiesSpace Network Communications S-Band and Ka-Band Next Generation Navigation Techniques Advance SDR/STRS Communications Technology to TRL-7, Compliant to STRS Common Architecture (Commercial/International) Near Earth Network Communications
    • Payload Delivered to ISS by JAXA HTV berths to ISS, EP MP is Payload will be removed from removed and installed to JAXA Multi-Purpose Exposed Pallet(EPExperiment Module-Exposed Pallet MP) and installed on ELC3, via EVR, with EVA back up. GRC TNSC, Japan Tanegashima Space Center (TNSC): NASA GRC: Payload processed for Payload installed onto EP MP, EP MP international shipping and customs installed into HTV Unpressurized Logistics Carrier and launched to ISS. 34
    • CoNNeCT Design & Testing ChallengesRequirementsAntenna Pointing System “heritage” hardwareStructural Redesign, Margin, and Flight Releasable AttachmentMechanism Non-linearityISS ELC Co-development (& JAXA EPMP)SpaceWire hardware, firmware, and softwareDigital Input/Output FirmwareSingle Board Computer Antenna Pointing System SpaceWire Cable Routing/Mounting & SpaceWire Board Gimbal and Antennas 35
    • Executive SummaryAn incompatibility in the PCI interface between the AiTechprocessor and Dynamic Engineering Spacewire Card wasdiscovered in March 2011.Initial attempts at fixing the DE firmware in March and April provedunsuccessful.Software workarounds were developed but initially limited high rateperformance.The DIO problem discovered in June 2011 provided anotheropportunity to fix the firmware.Dynamic Engineering delivered Rev_AA firmware on June 23rd thatsuccessfully eliminated the PCI bus lock-ups.Testing showed much better multi-SDR performance whenworkaround was disabled.Firmware updated on Avionics #2 and Avionics #1 in late July In parallel with the CM951 DIO fix 36
    • SCAN Testbed UseThe Dynamic Engineering Spacewire cardis a PCI Mezzanine Card (PMC) thatattaches directly to the AiTech SingleBoard Computer (SBC)It provides the Spacewire interfacebetween Avionics and SDRs AiTech Single Board Computer Harris - Data + Command/Telemetry JPL - Data GD – DataAn FPGA with Rev F firmware provides thePCI interface functionalityUsing the original firmware + softwareworkaround would have threatened: Dynamic Engineering High rate communications from Avionics 4-channel Spacewire PMC card to/from Harris SDR Processor availability for experimenters 37
    • Problem DefinitionThis was never a “Spacewire” problem. It was a PCI bus interface incompatibility between the processor and SpW board.Under certain conditions the local PMC bus would becomelocked-up, halting all DMA Spacewire communications. Most all duplex operations Simplex comm with Harris command and telemetry activeIt took 6 weeks of intensive testing, outside expert help, andspecialized test equipment to discover the root cause. 38
    • Root CauseThe DE Spacewire board uses eight (8) Direct Memory Access(DMA) “engines” to move data to and from the processor Tx and Rx DMA for each of 4 channelsThe eight DMAs are managed by arbitration logic that determineswhich DMA gets to use the single PMC bus at any given timeThe DMA arbiter had a fault where it would switch channels in themiddle of a “RETRY” while the AiTech processor fetched data.This resulted in an endless contention, or lock-up, as the DE boardasked for new data and the processor kept waiting for the originaldata to be read. Neither board would time-out. 39
    • PCI Lockup DetailsD.E. SpaceWire board is acting as the PCI bus MasterDMA #1 requests data from Address ASBC doesn’t have the data, issues a RETRY, and goes to fetch the dataDMA Arbiter then moves to the next DMA pending.DMA #2 requests data from Address BSBC returns with Address A data in a buffer – issues retryDMA #2 endlessly asks for Address B dataSBC does not time-out and issues endless RETRYs As Master the SpW board is required to re-issue the pending request for Address A data.
    • Root Cause ValidationPCI traces clearly show the root cause problemNew firmware (Rev_AA) proved effective at preventing the PCI buslock problems As tested on SDS2 with PMC analyzer traces confirming no lock-upsThe new firmware was installed on the GIU on 7/7/11 Process witnessed and approved by QA (M. Luboski)Multi-SDR testing of the new firmware (Rev AA) with new SpWdrivers and SpW manager code on the GIU With semaphore protections in place, Harris forward link data was corrupted at higher rates when performing multi-SDR tests. With semaphores removed, data was transferred perfectly at composite rates around 65-70Mbps 41
    • Corrective ActionSpacewire Card Firmware Upgrade Requires de-integration of the SpW card from the SBC Card is connected to a PMC programmer interface installed in a PC New firmware is “flashed” onto the board – no hardware mods requiredFirmware initially installed onto SDS and GIU for testing Done here at GRC using hardware/software approved by the vendor GIU upgrade was witnessed by QA and the process documentedFlight Upgrades Performed in parallel with the CM951 upgrades on Avionics #1 and #2 SBC removed from chassis and SpW board de-integrated SpW board flashed with QA witness using approved procedures and GSE SpW board re-integrated to SBC (required new staking) SBC installed back into chassis Functional and environmental testing documented in CM951 presentation 42
    • Reacceptance and Regression TestingDoesn’t apply to Spacewire as there has never been a fullyfunctional baseline test performed. Best validation of the “repaired” hardware is to demonstrate performance improvements above initial capabilities.Functional Spacewire Testing Extensive SpW testing on the Flight System with Avionics #2 was performed 7/27 and 8/2. Results proved new firmware was functional and DMA was arbitrating correctly Uncovered more software bugs Further testing on Flight System performed on 8/16 Harris and JPL – single duplex and multi-SDR duplex testing successful at high rates. Performance matches testing on GIU Uncovered additional software bugs Additional SpW functionality was added to T-Vac test code to exercise multiple channels and DMA arbitration. 43
    • Summary of Recent ResultsSoftware build on Sept 2nd (4707) fixed last major SpW bugs Multi-SDR low-rate problem due to unknown DMA arbitration issue Slow rate testing caused unexpected interruptions in Harris data flow Fix was simple - Changed how new DMAs are started on JPL and GD Stale buffer problem Internal avionics buffers not being reset upon test start caused test failuresExtensive GIU Testing 9/6 and 9/7 Successful testing of all SDRs at all rates Harris - 3M/12.5M, 12.5M/50M, 12.5M/100M JPL – 18k/24k, 155k/192k, 769k/769k GD – 18k/24k, 72k/192k, 72k/1M Successful multi-SDR tests with all combinations of above Success = 0 errors, 0 idle, and 0 null frames on forward or return link Except for Harris return link which has null frames at very high ratesFinal Validation on F.S during imminent Comm Performance Tests 44
    • ConclusionsThe root cause of the PCI incompatibility that caused poorSpacewire performance has been fixed on the flight unit and GIUThe re-integrated flight hardware successfully passed all box-levelenvironmental tests as documented in the CM951 presentationSpacewire performance has never been better All required rates have been tested successfully All single SDR and two-SDR combinations Limited three-SDR tests have been successful as well 45
    • DIO Corruption Root CauseTelecon was held with vendor (AITech) on 5/16/2011Vendor confirmed the problem was a fundamental design error inone of the FPGAs used on their CM951 DIO card Other customers had previously reported problems with CM951 digital output erroneous state changesCM951 card has a PCI bridge and an FPGA that contains a PCIcore, 1553 logic, and the digital I/O After previous customer reports, vendor investigation replicated problem and led to suspect subvendor PCI core as source of problemVendor acquired source code for PCI core and established thatthere were issues Address lines for DIO transfers were taken from within PCI core (poor design practice); this allowed DIO to be improperly clockedVendor upgraded the core with a redesigned versionSubsequent vendor testing did not reproduce the problemVendor product line modified to include redesigned FPGA firmware 46
    • Root Cause ValidationAn existing CM951 card was tested in a CoNNeCT SoftwareDevelopment System (SDS) and problem was again replicated CoNNeCT SDSs are lower-fidelity flight system emulators that do not replicate all flight avionics functions Problem consistently replicated; near-immediate corruption at FTP initiationA redesigned CM951 card was then tested in a CoNNeCT SDS inits basic configuration (no daughter board, i.e., PMC locationunpopulated) New card requires new modified driver due to memory addressing on FPGA Corruption did not occur even with hours of high-traffic FTP calls Note: Phenomenon of concern is on a microsecond timescale; hours of testing represents statistically very large data set of error-free operation. 47
    • Additional Vendor Design Flaw UncoveredWhen GRC tested the redesigned CM951 with a daughter boardinstalled in the PMC location (our flight configuration), corruptiondid not occur. However, the 1553 function did not work properly. Vendor did not test redesigned CM951 in this configuration (even though it is a „catalog‟ configuration offered by the vendor) Vendor sent representative to GRC to review our testing and our software interface code Vendor rep concluded GRC interface code and driver was correct Rep discovered additional problem with vendor firmware Test configuration replicated at vendor facility; subsequent testing reproduced 1553 problem Vendor RE-redesign and validation test of FPGA completed on 7/5/2011 GRC validation testing of re-redesigned CM951 showed proper 1553 function and no corruption of DIO states Extensive testing (80+ hours as of 7/17/2011) completed on SDS and GIU using high-traffic FTP transfers and other flight-like test cases (FS committal data) Many more hours since accumulated on GIU and Flight System No corruption encountered; 1553 functions properly 48
    • Examples of GIU Validation TestingSoftware/firmware Test Evaluation• The following tests were performed on the GIU with two CM951 boards containing the upgraded FPGAs: – Payload MDM transfers to and from Avionics – Nominal ELC 1553 commanding and telemetry – JPL SDR Uploads – Nominal JPL commanding and telemetry – FTP transfer to Avionics – DIO commanding to APS, JPL, heater, and RF switches – APS movement, calibration and track file runs –JPL SDR SpaceWire data between Avionics and SDR• Logic analyzer was connected to redundant I/O lines to detect any inadvertent state changes. No inadvertent state changes were observed.• No MIL-STD-1553 anomalies were encountered.Faulty FPGA firmware validated as root cause 49
    • Corrective ActionSCAN Testbed flight CM951 cards modified to include upgraded FPGA• CM951 cards removed from flight-spare Avionics unit (s/n 002)• Cards shipped to AITech for rework• FPGA replaced is a 208 pin Quad Flat Pack.• Used same manufacturer part number but has new firmware.• Heat sink and thermal interface material removed and replaced.• Vendor touched up conformal coating and performed functional test.• Boards thermal cycled for 8 cycles over a range of –40C to +71C.• Vendor produced a PRC (Production Route Card) that details the steps performed and provided sign-off records for QA and technicians. (Similar to original build process.)• GRC Quality Assurance witnessed/inspected FPGA replacement on the first commercial board.• Cards reinstalled in flight-spare Avionics unit (s/n 002)• Temporary R&R of flight Avionics unit (s/n 001) with flight spare (s/n 002) • Allowed for early verification of new CM951 performance in flight system• CM951 cards in flight Avionics unit (s/n 001) reworked and reinstalled• Flight Avionics unit (s/n 001) acceptance tested including environmentals• Flight-spare Avionics unit (s/n 002) replaced with recertified flight Avionics unit (s/n 001) 50