Uploaded on


  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Remote Operations and Ground Control Centers May 17-21, 2004 Kim Lankford [email_address] 256-544-3519 Marshall Space Flight Center Flight Projects Directorate Colsa Corporation Barry S. Bryant [email_address] 256-544-7404 Marshall Space Flight Center Flight Projects Directorate Computer Sciences Corporation R. Lee Pitts [email_address] 256-544-0666 Marshall Space Flight Center Flight Projects Directorate Computer Sciences Corporation
  • 2. Agenda
    • Introduction to Remote Operations introduces aspects of remote operations that will be discussed throughout the presentation
    • Introduction to the POIC introduces the POIC remote operations model
    • Remote Operations Options and Services briefly describes the remote operations options/services provided to our customers
    • POIC Architecture defines the architectural groundwork for successful remote operations
    • Remote Operations Objectives identifies the key drivers to consider when developing a system to support remote users
    • Lessons Learned discusses some of the key items to consider prior to taking on remote operations
    • Potential Benefits for Future Programs discusses how these successes might be applied to future programs to same money and provide wider access to space
  • 3. Introduction to POIC
    • Remote Operations is the ability for complete operations of a payload experiment from a remote site with all the capabilities typically provided to control center operations personnel
    • Marshall has supported multi-mission, remote operations since inception
      • MSFC installed a facsimile machine in 1961 to exchange data and documents with KSC and JSC to expedite Saturn development
      • Von Braun established the Mission Operations Office (MOO) in 1965
        • MOO created the Launch Information Exchange Facility (LIEF)
        • The Huntsville Operations Support Center (HOSC) was created as part of the LIEF
        • Launch data from KSC and flight data from JSC evaluated remotely from the HOSC
        • LIEF was the “all hearing center during a Saturn launch”
      • In 1973, the HOSC was online and supporting Skylab
        • Supported from problem detection to resolution
        • Created a special troubleshooting team
      • In 1975, the HOSC was online and supporting the Apollo-Soyuz mission
      • Throughout the 80s and 90s, the HOSC provided continuous support to STS launches and Spacelab.
        • Users from around the world involved
        • Most support started local but evolved to a tele-presence for many Payload Investigators
  • 4. Introduction to POIC
    • Today’s missions include ISS payload operations, STS support, MCC-H backup, and Chandra support
    • The HOSC is the dissemination point for all ISS science data
      • ISS Operations supports a diverse user base of payload investigators
      • Facility class payloads and individual experiments supported simultaneously
      • HOSC ground systems are accessed by users from around the world
        • All payload users utilize web services in support of operational activities
        • All science data are distributed from the HOSC
        • Realtime components allow users to command, manage payloads, and process health and status telemetry
      • Payload systems managers (Cadre) are located at the HOSC
      • Payload users may be located locally or remotely with all services available
    Today the HOSC is a multi-mission facility
  • 5. Introduction to POIC MCC-H HOSC Station-Wide and U.S. User Functions RSA Russian Assets P/L Integration and Operations TDRSS WSC Command/Telemetry/Video/Voice Command Video Voice P/L Planning Ku-Band S-Band P/L Planning, Voice MRPO, SPD, EXPR Int, EXPR Eng, WORF Eng Remote User Community NASA Telescience Support Centers (TSCs)
    • JSC (HRF, Earthkam, MELFI, CBOSS)
    U.S. Commercial Space Centers (CSCs)
    • UC San Diego (Earthkam)
    • Univ. of Colorado (CGBA, Yeast Gap)
    • Cal Tech (FOAM)
    • Univ. of Wisconsin (PASC)
    • Wyle Labs, Houston (CBOSS)
    • MIT, Cambridge (AMS)
    Other NASA Centers
    • KSC (Space Life Sciences)
    • JPL (Earthkam)
    • GSFC (SEM)
    U.S. Commercial Facilities
    • Boeing VSR, Houston (ARIS, PEI)
    International Partner (IP) Facilities
    • ESA Columbus Control Center
    • ESA Belgium USOC (Promiss)
    • ESA MOST, Moscow (ARGES, HEAT)
    • ESA PMSR , Netherlands (ARGES, HEAT)
    • ESA European Astronaut Center (Promiss)
    • ESA Norwegian USOC (EMCS)
    • CERN, Switzerland (AMS)
    • Thomas Electronics, Canada (HRF)
    • St. Johns, Newfoundland (SPACEDRUMS)
    Other non U.S. Locations
  • 6. Introduction to Remote Operations
    • Other considerations make remote operations a critical part of the overall operations concept
      • Reduced budgets forced a paradigm shift and initiated a search for cheaper ways to do the same jobs
      • The number of orbiting experiments translates into more people than the control center can support locally
      • The ability to share information immediately in a large number of locations such as schools and universities
      • More users want to do their work at home in a familiar environment close to their colleagues
  • 7. Introduction to Remote Operations
    • Key technology advances make remote operations possible
      • The performance and cost of Intel-based platforms provides an easily managed, cost effective platform for remote users
      • The bandwidth and ease of access to the web/internet provides remote users with an inexpensive avenue for access to many control center services
      • The introduction of Virtual Private Network (VPN) technology provides remote users with the ability to have a secure communications mechanism with the control center
      • The development of high available (HA) services provides users with continuous access to their services even when remote
  • 8. Introduction to Remote Operations
    • Key technology advances make remote operations possible
      • The development of Open Source operating systems such as Linux support the model of cost effective servers to support remote operations
      • The stability of the Intel-based platforms, running both Linux and Windows 2000, provide users with stability comparable to their predecessor UNIX, VMS and MVS systems
      • The acceptance and use of IPSEC compliance in multiple products like routers and firewalls provides interoperability
      • Ubiquitous use of standards like POSIX and CORBA allow for migration of our products to inexpensive platforms
  • 9. Introduction to Remote Operations Payload Operations Integration Center (POIC) Payload Planning System Payload Data Services System Enhanced HOSC System (EHS) TReK GSE EPC
    • Any Remote Payload Option can utilize the EHS Web interfaces
    EHS Copy *not baselined IVoDS
  • 10. Introduction to Remote Operations
    • Several Remote Payload Options are provided depending on the needs of the individual user
      • TReK - Remote users can obtain a copy of the Telescience Resource Kit (TreK) software and associated COTS to do their telemetry and commanding functions
        • TReK is a suite of Windows 2000 software applications that provide:
          • Local ground support system functions for Telemetry and Commanding
          • An interface with the POIC to utilize POIC remote ground support system services
        • TReK is suitable for individuals or small payload teams that need to monitor and control low/medium data rate payloads
        • TReK is primarily for the Science users who want full control, with local autonomy
      • GSE - Remote users can develop or procure their own software to utilize both POIC telemetry and command programmatic interfaces
  • 11. Introduction to Remote Operations
    • Several Remote Payload Options are provided depending on the needs of the individual user (Cont.)
      • EPC - Remote users that desire to user the same capabilities provided in the POIC can obtain a copy of the Enhanced HOSC System PC (EPC) Application and associated COTS
        • EPC provides remote users with integrated access to POIC developed telemetry processing/display and command system services on a Windows 2000 PC platform
        • A remote EPC is a copy of the same EPC that local users utilize at the POIC
        • EPC provides some local applications with a built in login capability to the POIC
      • IVoDS - Remote users can obtain a copy of the Internet Voice Distribution System (IVoDS) to utilize the ISS mission voice services from the POIC
        • IVoDS is MSFC-developed software provided to remote payload users enabling Windows 2000 PC-based multiple voice loop talk/monitor capability
        • IVoDS can monitor up to 8 loops/conferences simultaneously and talk on one
  • 12. Introduction to Remote Operations
    • Several Remote Payload Options are provided depending on the needs of the individual user (Cont.)
      • Web - POIC Web interface provides remote users with web-based access to several POIC capabilities from a JAVA compliant Windows 2000 platform
        • POIC Web is accessible via Browser and Java Plugin
        • POIC Web is highly reliable, available, and secure (within the EHS Web infrastructure)
        • Regardless of the Remote Payload Option the payload user selects, all payload users will use the POIC web interface to access certain POIC capabilities
      • EHS Copy - If EHS copy is baselined, remote facilities can obtain a copy of EHS to support a remote control center with its own set of local and remote users
  • 13. POIC Architecture
    • Focus for the HOSC will be on ISS support
    • The HOSC offers services through 3 primary systems
      • Major systems in the HOSC architecture are
        • E nhanced H OSC S ystem (EHS)
        • P ayload D ata S ervices S ystem (PDSS)
        • P ayload P lanning S ystem (PPS)
      • A number of auxiliary services are available as well
        • Traditional Voice Distribution System
        • Internet Voice Distribution System (IVoDS)
        • Video Distribution System
        • T elescience R esource K it (TReK)
      • The HOSC systems are designed to provide support from single users up to large facilities
  • 14. POIC Architecture
    • The HOSC provides an extensive suite of Ground System capabilities to support all users, local and remote
      • Data Acquisition & Distribution Services
        • Acquires telemetry data from multiple sources
        • Error checks and generates data quality information
        • Provides highly available storage and playback
        • Provides a standard delivery method to all local and remote users
      • Database Services
        • Converts Payload Data Library (PDL) inputs into ISS Command and Telemetry Databases
        • Users have local database control to change parameter characteristics
        • Provides DBMS monitoring for status and performance
        • Provides a centralized Database Change Request (DCR) system
  • 15. POIC Architecture
    • The HOSC provides an extensive suite of Ground System capabilities to support all users, local and remote
      • Telemetry Services
        • Traditional telemetry and data routing services for packet data
        • Logging, management, retrieval, and playback of processed data
        • Ability for a user to build and use graphical, textual, and tabular displays in realtime using a simple point and click display generation tool
        • Provides the capability to request a selection of parameters over a specified time period
        • Capability to create Custom Data Packets which contain only the parameters required by an individual user
      • Payload Information Management System Services
        • Provides a workflow engine with user notifications and status updates
        • Contains a data vault for the storage and management of operational procedures, documents, and change requests
        • Acts as the operational hub for payload users to interchange flight items with the cadre
  • 16. POIC Architecture
    • The HOSC provides an extensive suite of Ground System capabilities to support all users, local and remote
      • Command Services
        • Provides users the capability to view, modify, initiate, and status commands
        • Provides unique controls and security functions necessary to safely execute hazardous and critical command functions for a variety of missions
        • Ability for a user to build and use scripts in realtime for the command, control, and analysis of all monitored components
          • Scripts range from a single function to very complex and nested scripts to accomplish many functions
          • Scripts may be used to start displays, send commands or execute other scripts based on telemetry values or user input
        • Ability for a user to build and uplink automated procedures and files
        • Provides realtime command tracking and long term command delogs
  • 17. POIC Architecture
    • Systems are partitioned into Private and Public Domains
      • Private Domain is secured by an advanced firewall
        • Stateful inspection of protocols
        • Support ISS required protocols and is IPSec 1.0b compliant
        • Vendor is knowledgeable, responsive, and constantly improves the product
      • Public Domain is secured by router access lists and firewalls
      • Network traffic between the Private Domain and the Public Domain is secured through protocol controls
      • HOSC Mission LANs reside in the Private Domain under the most strict access requirements
  • 18. POIC Architecture Remote clients
  • 19. POIC Architecture
    • Tier 1 represents the client level
      • Remote user access is supported on PCs
        • Connections are protected by secure network interfaces
          • IPSec compliant Virtual Private Networks (VPN) for remote users
          • Local users via switched VLAN
        • Accessible remotely or locally
        • Local HOSC management of the clients is by Microsoft Advanced Server
        • HOSC PCs are fully protected behind the HOSC firewall on dedicated VLANs
      • Local PCs are generic devices with Windows O/S
        • Current version is Windows 2000
        • XP will be supported soon through the normal change process
        • Written in Visual Basic, JAVA, C, and C++
        • EHS provides the product for access to the Interface level
      • Remote users can use any compliant device for their APPs
        • HOSC provides EHS applications for Windows platforms
        • Remote users can write their own applications
  • 20. POIC Architecture
    • Tier 2 represents the Interface level
      • Interface servers represent 2 logical server types; Web and RIS/Login
        • Interface servers decouple users from the application servers
        • Accessible remotely or locally
        • Web servers are protected in the HOSC firewall DMZ on a dedicated VLAN
          • Shared by local and remote users
          • Not utilized for realtime functions
          • Proxies for web based requests to internal servers
        • RIS/Login servers are fully protected behind the HOSC firewall on a dedicated switched VLAN
          • Proxies for all non-web based requests
          • RIS is the source of GSE Web initiated UDP packets, CDP data, and remote commanding
          • Login servers handle all EPC user server processing
  • 21. POIC Architecture
    • Tier 2 represents the Interface level
      • Interface servers are bankable compute resources which can be sacrificed
        • Non-persistent data resident
        • Each server is functionally similar to others
        • Web servers will be clustered to provide a Fault Tolerant (FT) group in OPS
        • FT clustering is loosely coupled
        • Commodity Intel ™ platforms
  • 22. POIC Architecture
    • Tier 3 represents the Application level
      • Application servers represent logical server types; Database (DB), NRT, PIMS, Project, FEP, SMAC, PPS, and PDSS servers
        • DB houses the project databases for telemetry and commanding
        • NRT is a shareable compute intensive resource for Near-RealTime (NRT) Data retrieval
        • PIMS is the ISS Payload Information Management System
        • Project servers house various services to include commanding, mission Exception Monitoring (EM), etc
        • Front End Processors (FEP) provide the ingest point and initial processing for spacecraft data
        • SMAC servers are utilized to configure mission systems
        • PPS servers are utilized by the CADRE to plan payload operation
        • PDSS provides EHS data to HOSC internal and external users while logging data to NRT data stores
    • Tier 4 represents the Infrastructure support level
  • 23. Remote Operations Objectives
    • Two sets of objectives exist; the user’s and the system’s
      • A typical user has basic objectives that allow for easy operations concept integration:
        • Simple, tailored, convenient, and stable interfaces for voice, video, data, and command services
        • Support through all mission phases
          • pre-mission planning
          • mission preparation testing
          • real-time support
          • post-mission analysis
        • Getting the right information (data) to the right people at the right time
        • Being provided a “just right” solution
          • “ Don’t provide a luxury car when all I need is a compact”
        • All data only when needed
  • 24. Remote Operations Objectives
    • Two sets of objectives exist; the user’s and the system’s
      • The system has another set of objectives which are supportive of users but sometimes appear to be in conflict due to constraints imposed on the users
        • Provide responsive support services
          • Mission integration
          • Customer support
          • Training
          • Realtime support
        • Insure the integrity and availability of data even under near catastrophic events
          • RAID on critical systems
          • Backups
          • Downtime for maintenance activities
        • Provide a secure environment and communications channels
        • Provide a cost effective solution for ongoing activities
        • Provide facilitization of users and equipment
  • 25. Remote Operations Options and Services
    • A cost benefit analysis should be conducted that weighs the operational needs against the cost to determine the optimal solution
      • User can develop his tools inhouse
        • Consider development, test, and maintenance cost
        • Consider user needs for local autonomy versus extra cost
      • User can use the capabilities of a current control center
        • Capabilities shared across projects
        • Shared infrastructure items
        • Rich suite of tools to choose from
      • Combination of custom tools with baselined control center capabilities
        • Take advantage of generic solutions where applicable
        • Expend effort only for custom tools in unique and critical areas
  • 26. Remote Operations Options and Services
  • 27.
    • Four general categories of lessons learned have been identified through the experiences of the HOSC and our remote operations history:
      • Understand onboard system requirements and limitations
      • Understand the ground system complexities and how architecture can provide easy integration of changing baselines
        • Mandated security requirements
        • COTS product integration into the operations environment
        • Fiscal drivers requiring an evaluation and implementation of different systems
      • Understand the total end-to-end user needs
        • Adequate emphasis should be placed on all phases of the mission
        • Often a disproportionate effort is placed on the “real-time” portion of the mission
        • Adequate pre-mission preparation might well avoid “real-time” difficulties
        • Adequate post mission close-out and analysis phase may ensure a successful mission
    Lessons Learned
  • 28.
    • Four general categories of lessons learned have been identified through the experiences of the HOSC and our remote operations history:
      • Ground systems can never provide a 100% solution to meet all customers’ needs
        • Provide standard interfaces and user tools that are easy to use and extensible
        • Allow customers to develop custom solutions for those “unmet” requirements
        • Each user must understand limitations and take advantage of the provided tools
    Lessons Learned
  • 29. Potential Benefits for Future Programs
    • Future programs can design, implement, and maintain a cost effective and efficient ground infrastructure in support of all mission phases
    • Remote operations can be an important part of any solution
    • The items that should be taken into consideration when implementing an operations facility include:
      • Understanding the existing capabilities and limitations of spacecraft systems and interface facilities prior to building your control facility
      • Evaluate existing tools that may be used to satisfy requirements
      • Maximize the use of commodity-based systems and open source products
      • Realize that a user does not have to be local to a control center to conduct realtime operations
      • A tiered approach to security minimizes access to crucial systems
      • Conduct a cost analysis to help derive a “just right” cost effective solution