Sigtran Workshop


Published on

Useful documentation for better undersstand the signalization protocol on IP

Published in: Technology, Business

Sigtran Workshop

  1. 1. Usage of SIGTRAN in the Vodafone Italy NP Project Version 1.0 Ivrea, January 26th - 27th , 2009
  2. 2. Agenda » Ulticom’s Signalware SIGTRAN Overview » SCTP Overview » M3UA Overview » Signalware M3UA: Installation, Configuration & Monitoring » References End
  3. 3. Agenda » Ulticom’s Signalware SIGTRAN Overview » SCTP Overview » M3UA Overview » Signalware M3UA: Installation, Configuration & Monitoring » References Back
  4. 4. Ulticom’s Signalware SIGTRAN Overview » SIGTRAN (SIGnalling TRANsport) is a Framework Architecture for Signaling Transport defined by IETF within the SIGTRAN working group (RFC2719 – Oct 1999) » The framework defines the architecture and the functional requirements for transport of signalling information over IP. » A major application of SIGTRAN framework concerns SS7 signalling; in this case, a set of protocols (Adaptation Layers) has been defined by SIGTRAN WG (xxUA, xxPA protocols). » Besides, the specific needs of signalling transport over a connection-less, unreliable packet network, have led to the introduction of a new transport protocol (SCTP)
  5. 5. Signalware SIGTRAN Architecture » Signalware SIGTRAN is an integrated platform that provides an environment for the development and deployment of SS7 applications and services over IP » Signalware is a middleware between the platform (HW platform/OS) and the applications. » In order to grant high availability and scalability, HW platform is a cluster made up by 1 to 8 CEs (Computer Equipments), which loadshare traffic and minimize service disruption. » Applications may be configured to run on 1 to 8 LNs (Logical Nodes) - LN:an instance of a protocol stack and a unique Point Code (PC). » Applications exploit an API to access Signalware transport service (legacy SS7/MTP or IP based transport) HW platform/Operating System Signalware LN SS7 Application LN SS7 Application CE1 CEn… …
  6. 6. Signalware architecture in NP application » The NP application is built on top of Signalware: » The stack is hosted on a single IBM x3755 server with Solaris 10 Operating System (FMP node) » The whole NP solution consists of several applications running over Signalware platform » Signalware provides M3UA/SCCP layers, while above layers are part of the NP application (SCTP is embedded in Solaris 10) IBM x3755 / Solaris 10 Signalware LN SS7 Application (Atos Platform, FMRDB, NP, SSA) CE xxFMP0x
  7. 7. Agenda » Ulticom’s Signalware SIGTRAN Overview » SCTP Overview » M3UA Overview » Signalware M3UA: Installation, Configuration & Monitoring » References Back
  8. 8. SCTP Overview » Stream Control Transmission Protocol » IETF standard - RFC 2960 Stream Control Transmission Protocol (October 2000) - RFC 3309 Stream Control Transmission Protocol (SCTP) Checksum Change (September 2002) - RFC 4960 Stream Control Transmission Protocol (September 2007) - RFC 4460 (*) Stream Control Transmission Protocol (SCTP) Specification, Errata and Issues (September 2007) (*) Signalware is compliant with RFC2960 and RFC3309, compliance with RFC4960 is still not implemented (no compatibility issues)
  9. 9. SCTP Overview » SCTP is a reliable transport protocol operating specifically designed for signalling transport on top of a connectionless packet network such as IP » TCP is not suited for signalling transport (why?) » SCTP features » Connection oriented (associations) - Heartbeats - Flow Control » Message based » Orderered and out of order message sending (within streams) - No head-of-line blocking » Multihoming - Increased robustness against TCP » DoS Protection - Cookies - 4-way handshake
  10. 10. Functional view of SCTP Association Startup and Takedown Sequenced delivery within streams User Data Fragmentation Acknowledgement and Congestion Avoidance ChunkBundling PacketValidation PathManagement The SCTP transport service can be decomposed into a number of functions
  11. 11. SCTP Packets » An SCTP packet is composed of a common header and one or more chunks. » Valid chunk types: » 0 Payload Data (DATA) » 1 Initiation (INIT) » 2 Initiation Acknowledgement (INIT ACK) » 3 Selective Acknowledgement (SACK) » 4 Heartbeat Request (HEARTBEAT) » 5 Heartbeat Acknowledgement (HEARTBEAT ACK) » 6 Abort (ABORT) » 7 Shutdown (SHUTDOWN) » 8 Shutdown Acknowledgement (SHUTDOWN ACK) » 9 Operation Error (ERROR) » 10 State Cookie (COOKIE ECHO) » 11 Cookie Acknowledgement (COOKIE ACK) » 12 Reserved for Explicit Congestion Notification Echo (ECNE) » 13 Reserved for Congestion Window Reduced (CWR) » 14 Shutdown Complete (SHUTDOWN COMPLETE) » …
  12. 12. SCTP Association State Machine » During its lifetime, an association can have different states
  13. 13. Agenda » Ulticom’s Signalware SIGTRAN Overview » SCTP Overview » M3UA Overview » Signalware M3UA: Installation, Configuration & Monitoring » References Back
  14. 14. M3UA Overview » Message Transfer Part 3 (MTP3) - User Adaptation Layer » IETF standard - RFC 3332Signaling System 7 (SS7) Message Transfer Part 3 (MTP3) – User Adaptation Layer (M3UA) (September 2002) - RFC 4666Signaling System 7 (SS7) Message Transfer Part 3 (MTP3) – User Adaptation Layer (M3UA) (September 2006) (*) Minor changes between RFC3332 and RFC4666 (same protocol version, i.e. 1.0) (**)Signalware is compliant with RFC3332, compliance with RFC4666 is still not implemented (no compatibility issues)
  15. 15. M3UA Overview » M3UA supports the transport of SS7 MTP3 user signaling over IP using the services of the SCTP: » The list of these protocol layers includes, but is not limited to, ISUP, SCCP, and telephone user part (TUP) » The M3UA layer provides the same interface to its upper layer as MTP3 does: - In this way, MTP3 user is not aware that MTP3 services are offered remotely from an MTP3 layer at an SG, and not by a local MTP3 layer. - Similarly, MTP3 layer at the SG may also be unaware that its local users are actually remote user parts over M3UA. - In practice, the M3UA extends access to the MTP3 layer services to a remote IP–based application.
  16. 16. M3UA Network Scenarios » Two distinct scenarios: » AS-SG interconnection - In the AS, M3UA replaces MTP3, allowing ISUP, SCCP and TCAP based application to operate without an underlying SS7 network(*) - In the SG, M3UA gives SS7 network access to applications located on IP networks (i.e. it enables signaling transport between SS7 network and new-generation IP networks) » IPSP-IPSP interconnection -M3UA enables SS7 applications in an all-IP environment (same as an AS without an SS7 network) (*) The AS may or may not have an own Signalling Point Code (e.g the PC may not be needed if there is one-to-one correspondence between AS and SG) Legacy SS7 node (e.g. MSC) xxAP NIF MTP3 MTP2 MTP1 M3UA SCTP IP M3UA SCTP IP xxAP TCAP SCCP Application Server (AS)Signalling Gateway (SG) SS 7 SS 7 IPIP TCAP SCCP MTP3 MTP2 MTP1 M3UA SCTP IP xxAP TCAP SCCP IP Signalling Point (IPSP) IPIP M3UA SCTP IP xxAP TCAP SCCP IP Signalling Point (IPSP)
  17. 17. M3UA Key Concepts » Application Server (AS) A logical entity handling a specific type of traffic (e.g. a virtual data base element, or the FMP node in the NP Network Scenario). The AS contains a set of one or more unique Application Server Processes, of which one or more is normally actively processing traffic. » Application Server Process (ASP) A process instance handling traffic for an AS » Signaling Gateway (SG) Virtual entity that acts as a bridge between IP and SS7 networks. It is perceived as an SS7 Point Code within the SS7 network. » Signaling Gateway Process (SGP) Process instance handling traffic within the SG » IP Server Process (IPSP) An ASP that interacts with other ASPs (instead of SGPs). » Routing Key (RK) Set of SS7 parameters that identify the type of traffic handled by an AS (DPC, SI,…). An AS registers a RK with a SG or with another AS (there is one-to-one correspondence between an AS and a RK) » Routing Context (RC) 32 bits RK identifier
  18. 18. M3UA Messages » Management messages (ERR, NOTIFY) » Transfer Messages (DATA) » SS7 Signalling Network Management (SSNM) messages (DUNA, DAVA, DAUD, SCON, DUPU, DRST) » ASP State Maintenance (ASPSM) messages (ASPUP, ASPDN, BEAT, ASPUP ACK, ASPDN ACK, BEAT ACK) » ASP Traffic Maintenance (ASPTM) messages (ASPAC, ASPIA, ASPAC ACK, ASPIA ACK) » Routing Key Management (RKM) messages (optional) (REG REQ, REG RSP, DEREG REQ, DEREG RSP)
  19. 19. M3UA Activation (single ASP in single AS) » The assumption is that the SCTP association has been already established prior to M3UA message exchange » Routing Key Registration is optional (e.g. Eagle5 does not support it, static provisioning is needed) » The ASP Up message is used to indicate to the remote peer that M3UA is ready to receive any ASPSM/ASPTM messages for all Routing Keys that the ASP is configured to serve. » The ASP Active message is sent by an ASP to indicate to a remote M3UA peer that it is ready to process signalling traffic for a particular Application Server » As an example grey text reports Signalware MML commands that trigger M3UA ASPSM/ASPTM messages. RegReq (LRK=x,RK=y) RegRsp (LRK=x,RC=z) ASPUP ASPUP ACK ASP SGP ASPAC (RC=z) ASPAC ACK (RC=z) Notify (AS-ACTIVE) CREATE-M3UA-RKEY ACTIVATE-M3UA-RKEY ACTIVATE-M3UA-LSET Notify (AS-INACTIVE)
  20. 20. M3UA Activation (2 ASP in single AS, Loadsharing) » The graph on the left depicts a scenario with 2 ASPs in a single Application Server, which: » handle traffic in loadsharing mode (ASPAC message): - this means that the SGP can direct traffic to all ASPs active at a certain moment in time, choosing them according to an implementation dependent algorithm. » are configured in “1+1” sparing: - More generally, failover model supports an "n+k" redundancy model, where "n" ASPs is the minimum number of redundant ASPs required to handle traffic and "k" ASPs are available to take over for a failed or unavailable ASP. ASPUP ASPUP ACK ASP1 SGP ASPAC (LoadSHR) ASPAC ACK Notify (AS-ACTIVE) ASP2 ASPUP ASPUP ACK ASPAC (LoadSHR) ASPAC ACK Notify (AS-ACTIVE)
  21. 21. M3UA Signalling » The graph on the left depicts a scenario where an SS7 Legacy Node (e.g. MSC) exchange SS7 signalling with an Application Server (e.g. FMP node) through a Signalling Gateway TFP SS7 legacy node ASSG DUNA DAUD DUNA TFA DAVA MSU DATA MSU DATA
  22. 22. M3UA Signalling » The graph on the left depicts a scenario where a local SSN on the AS becomes prohibited SSP AS SG SSA MSU SST SST MSU SST
  23. 23. Agenda » Ulticom’s Signalware SIGTRAN Overview » SCTP Overview » M3UA Overview » Signalware M3UA: Installation, Configuration, Operation & Monitoring » References Back
  24. 24. Signalware M3UA: Installation, Configuration, Operation & Monitoring » Signalware product is composed of a set of packages that are automatically installed, removed, checked and repaired through automated scripts (shipped as part of the whole software package) » This is the set of packages installed on Vodafone xxFMP0y nodes: - OMNI (base system) - OMNI-X (support for application development) - OMNI-GUI (GUI support) - OMNI-MAN (man pages) - OMNI-OLU (support for on line upgrade) - OMNI-C3 (ITU MTP and drivers) - OMNI-C7X (ITU MTP development) - OMNI-C4 (ITU SCCP) - OMNI-AC7 (ANSI TCAP over ITU SCCP/MTP drivers) - OMNI-C3M (ITU M3UA) - OMNI-SCTP (Stream Control Transmission Protocol)
  25. 25. Signalware M3UA: Installation » Signalware installation requires the creation of a special user account for installation, operation and provisioning (omni on Vodafone xxFMP0y nodes) » A valid license file is also needed: » this file (.lic extension) is provided by Ulticom through Ulticom portal » it must be copied into /etc directory before installations » it is an XML file customized for the target machine on which Signalware will be installed (host name and host id) <?xml version="1.0" encoding="US-ASCII" standalone="yes"?> <LICENSE> <CUSTID> LSN: P507000S090028 … <CE> <NAME>t1fmp01</NAME> <HOSTID>3856b52e-SunOS-t1fmp01-7D2AF259E968E1BAF756BFDBD0</HOSTID> … <ITEM TYPE="FEATURE"> <NAME>M3UA</NAME> <VALUE>ENABLED</VALUE> </ITEM> … -----BEGIN SIGNATURE----- iOKdfu2H0mm1nHOIWtVlrFgmAtTwsOF/SGnxGKyFLhPXJKhVT5974wC7z+oThEhb3J9YC9EXW9/h … -----END SIGNATURE-----
  26. 26. Signalware M3UA: Installation » Installation can be done in 3 ways: » Ulticom Menu Interface (script swsetup provided in the software bundle) - This is strongly recommended - Details on installation and configuration can be found in the following document: » Ulticom command line interface (swxxxx scripts) » Native Solaris installation tools (pkgadd, pkgrm commands), taking dependencies into account SW Installation Configuration
  27. 27. Signalware characteristics & capabilities » Features enabled based on the purchased license: » In Vodafone, the following protocols/variants are available according to the shipped license: - M3UA - SCCP - TCAP - ATM available in C7, A7, CH7 and J7 variants (even if not all has been installed) » The following protocols/variants are not available with Vodafone license (even if they are part of the Ulticom Signalware suite) - M2PA - M2UA - SUA - INAP - TUP, ISUP - CAP - SIP - Diameter and many others…
  28. 28. Signalware characteristics & capabilities » The main provisioning limits affecting Signalware M3UA configuration are reported below: - Up to 8 Logical Nodes (LN) - Up to 256 M3UA links per CE for all nodes - Up to 32 SCTP streams per association - Up to 10,000 Route SETs (RSETs) per platform node - Up to 255 Link SETs (LSETs) per platform node - Up to 16 LSETs per RSET - Up to 4,095 Routing Keys per platform node (default is 512) - Outbound GTT entry number is in range [1000 - 20000] (default is 1000) - Inbound GTT entry number defaults to 200
  29. 29. Signalware Configuration » There are several configuration files, located in $OMNI_HOME/conf (i.e. /users/omni/conf on xxFMP0y nodes). » Values specified here overwrite default configuration parameters: - omni_conf_info Definition of general Signalware parameters, such as: MEAS_30MIN_TIMER=5 (interval interval by which 30-minute protocol measurements are collected) - usam_conf_info Definition of USAM specific parameters, which is the Signalware SCTP association manager for M3UA and SUA., such as: MAX_NB_THREAD=127 (Maximum number of threads in charge of establishing outbound SCTP associations) - m3ua_conf_info.<SHM> Definition of M3UA specific parameters, such as: M3UA_SLK_ADDR_SORTED=FALSE (If FALSE, SCTP addresses are kept as provisioned by CREATE-M3UA-SLK MML command → SCTP primary path ) SCTP_PATH_REXMIT=2 (maximun path retransmission) SCTP_HB_INTERVAL=500000 (Heartbeat interval set to 500 ms) » For each file there is a detailed man page which provides an explanation for all configuration parameters » When changes are made to those files, Signalware must be terminated and restarted
  30. 30. Signalware Configuration » Signalware over Solaris 10 exploits the SCTP protocol stack embedded in the Operating System: » Some configuration parameters must be configured directly in the Kernel (Solaris SCTP tunable parameters): - /etc/init.d/ndd-set (or similar startup script) ndd –set /dev/sctp sctp_pa_max_retr 4 Controls the maximum number of retransmissions (over all paths) for an SCTP association. The SCTP association is closed when this number is exceeded. » Priority order when considering SCTP Configuration Parameters - SCTP tunable parameters - m3ua_conf_info settings overwrite previous ones - MML commands specific settings overwrite all the previous ones (association specific parameters in CREATE-M3UA-SLK command)
  31. 31. Signalware Operation » NP application requires that Signalware is controlled through the Solaris Service Management facility (not started manually): » Services: svc:/application/Ulticom/DFdaemon:default svc:/application/Ulticom/Signalware:default controls respectively the Ulticom Distributed File System and the core Signalware processes » Manifest files are stored in /var/svc/manifest/application/Ulticom/ » Signalware is started through: /usr/sbin/svcadm enable DFdaemon /usr/sbin/svcadm enable Signalware and stopped through: /usr/sbin/svcadm disable Signalware /usr/sbin/svcadm disable DFdaemon » Status can be checked through: svcs DFdaemon svcs Signalware
  32. 32. Signalware Processes omni@t1fmp01> ps -ef | grep omni root 22854 22844 0 16:15:43 ? 0:06 /users/omni/bin/pup -start omni 22866 22844 0 16:15:44 ? 0:01 /users/omni/bin/meas -x7m3ua -start omni 22856 22844 0 16:15:43 ? 0:01 /users/omni/bin/guiserver -start root 22847 22844 0 16:15:43 ? 0:01 /users/omni/bin/log -start root 22844 22137 0 16:15:43 ? 0:17 /users/omni/bin/pop /tmp/start.100.22137 omni 22849 22844 0 16:15:43 ? 0:19 /users/omni/bin/dly -start omni 22137 1 0 16:15:40 ? 0:00 /bin/bash -p /users/omni/bin/go.omni root 22864 22844 0 16:15:44 ? 0:00 /users/omni/bin/resmon -start omni 22855 22844 0 16:15:43 ? 0:00 /users/omni/bin/oos -start omni 22851 22844 0 16:15:43 ? 0:00 /users/omni/bin/pevt -start omni 11154 27225 0 12:05:21 pts/17 0:00 grep omni omni 22828 22137 0 16:15:42 ? 0:00 /users/omni/bin/nrplinker root 22858 22844 0 16:15:43 ? 0:01 /users/omni/bin/watch -f /users/omni/conf/omni_watch.100 -h /users/omni/conf/om root 22852 22844 0 16:15:43 ? 0:19 /users/omni/bin/pm -nocheck -start root 22848 22844 0 16:15:43 ? 0:01 /users/omni/bin/sig -start omni 22868 22844 0 16:15:44 ? 0:05 /users/omni/bin/c7scmg -start omni 22870 22844 0 16:15:44 ? 15:47 /users/omni/bin/ln_daemon -f /users/omni/conf/nodeConf.N100.100 -start root 22853 22844 0 16:15:43 ? 0:23 /users/omni/bin/tap -start omni 10406 1 13 Oct 22 ? 8540:41 /usr/bin/perl -w /users/omni/bin/swmml omni 26759 26601 0 11:04:00 pts/13 0:00 -pfksh omni 26966 26759 0 11:04:14 pts/13 0:00 pevt omni 22850 22844 0 16:15:43 ? 0:01 /users/omni/bin/mon -start omni 22867 22844 0 16:15:44 ? 0:03 /users/omni/bin/sw_nm -protocol C7 -nokill -start omni 22869 22844 0 16:15:44 ? 0:02 /users/omni/bin/c7tcmg -start root 21863 1 0 16:14:32 ? 0:25 /users/omni/bin/DFdaemon -D root 22865 22844 0 16:15:44 ? 0:24 /users/omni/bin/c7m3ua AUTO_RST -start omni 22857 22844 0 16:15:43 ? 0:02 /users/omni/bin/scosprocess -start root 22832 22137 0 16:15:42 ? 0:02 /users/omni/bin/usam_daemon TRACE_FILE usam_trace
  33. 33. Signalware Processes » Solaris Service Management facility controls both DFdaemon and Signalware execution, therefore they are started by root process: omni 22137 1 0 16:15:40 ? 0:00 /bin/bash -p /users/omni/bin/go.omni root 21863 1 0 16:14:32 ? 0:25 /users/omni/bin/DFdaemon -D » The first process to be started by go.omni is the Parent Operation Process (POP), which in turn starts all the others: root 22844 22137 0 16:15:43 ? 0:17 /users/omni/bin/pop tmp/start.100.22137 ... omni 22850 22844 0 16:15:43 ? 0:01 /users/omni/bin/mon –start omni 22867 22844 0 16:15:44 ? 0:03 /users/omni/bin/sw_nm -protocol C7 -nokill –start omni 22869 22844 0 16:15:44 ? 0:02 /users/omni/bin/c7tcmg -start ... » A detailed explanation for the main Signalware processes can be found here. » A man page is available for each of the processes listed in the previous slide.
  34. 34. Signalware Provisioning » Provisioning is done through MML commands/files » Signalware provides an enhanced terminal handler, called swmml » swmml can be used in two different ways: - Interactive mode: - it provides an interactive environment where commands can be entered manually - Available features include auto-completion (both for commands and arguments), command history, on-line help (man pages) +---------+ Signalware Enhanced Terminal Handler v9.41 | swmml | Copyright 1993-2008 Ulticom, Inc. +---------+ All Rights Reserved Welcome to swmml. To get some help, type 'man'. Current editing mode set to: emacs mode. N100:t1fmp01:100 > - Batch mode: - swmml –e ‘<MML command>’ allows the execution of a single MML command - swmml –f <MML script> allows the execution of an MML script - swmml –x <ext. MML script> allows the execution of an extended MML script (i.e. a script which combines MML commands with PERL statements) The same script can be entered in interactive mode through RUN-MML-SCRIPT - In case of multiple logical nodes, swmml –n <node_name> is used
  35. 35. Signalware Provisioning » Provisioning order for a Logical Node configured as an M3UA Application Server » Create Own Point Code ⇒ CREATE-OSPC » Create IP Link SETs ⇒ CREATE-M3UA-LSET » Create IP links (i.e. associations) ⇒ CREATE-M3UA-SLK » Create Route SETs ⇒ CREATE-RSET » Create M3UA Routing Keys ⇒ CREATE-M3UA-RKEY » Create remote SSNs (if needed) ⇒ CREATE-REMSSN » Create Concerned Point Codes ⇒ CREATE-OSPC » Allow Route SETs ⇒ ALLOW-RSET » Activate Link SETs (or directly IP links) ⇒ ACTIVATE-M3UA-LSET or ACTIVATE-M3UA-SLK » Activate routing keys ⇒ ACTIVATE-M3UA-RKEY » Define rules for GTT (if needed) ⇒ CREATE-GT » For each command there is a detailed man page that explain how to use it » Click here to see the configuration of MIFMP01 node
  36. 36. Signalware Provisioning » Similarly, there is an unprovisioning order for the LN configured as M3UA AS » Deactivate routing keys ⇒ DEACTIVATE-M3UA-RKEY » Deactivate Link SETs ⇒ DEACTIVATE-M3UA-LSET » Inhibit Route SETs ⇒ INHIBIT-RSET » Delete Concerned Point Codes ⇒ DELETE-CPC » Delete remote SSNs ⇒ DELETE-REMSSN » Delete Routing Keys ⇒ DELETE-M3UA-RKEY » Delete Route SETs ⇒ DELETE-RSET » Delete IP links ⇒ DELETE-M3UA-LSET » Delete IP Link SETs ⇒ DELETE-M3UA-SLK » Delete Own Point Code ⇒ DELETE-OSPC
  37. 37. Monitoring and Troubleshooting » There is a set of logs in $OMNI_HOME/Logs directory (i.e. /users/omni/Logs) » Monitor.log.<SHM>.<file_number> - Log collected by the mon process - Log file in displayable HEX/ASCII format. <SHM> is the shared memory value and <file_number> is a five digit progressive number (numbering begins from a file number, when the file reaches 2 Mbytes it is closed, the file number is increased and a new log file is created). - Files are purged after a given time period (see DISPLAY-PURGE and SCHEDULE-PURGE commands) - Those files logs messages exchanged between Signalware processes » Monitor.pcap.<SHM>.<file_number> - Log collected by the mon process - Log file that contains monitored SS7 Message Signaling Units (MSUs) in PCAP format (to be read through Ethereal/Wireshark) - The .pcap file is closed along with the corresponding .log (files are always synchronized) - NEVER FORGET: This is a facility that logs SS7 messages as seen by the application, but troubleshooters are encouraged to capture “real” SCTP packets directly from the network » The MonControl and the monitor utilities can be used to handle and to configure mon process logging
  38. 38. Monitoring and Troubleshooting » A set of logs in $OMNI_HOME/Logs directory (cont.): » Logging into the previous files van be controlled through the MonControl and the monitor utilities (see man pages for details): - The MonControl utility controls the monitoring activity of mon process. It allows: - Enabling/disabling monitoring of specific user space processes (or all) - Logging in both ASCII and pcap files, or only one of them - Force flushing or purge&open - The monitor standalone process enables/disables monitoring in the UNIX Streams wrapper environment. - Monitor commands are limited to L3RT, SCCP, ISUP, and TCAP stream routers - monitoring can be enabled at four points: message entering the module on the write side, message exiting the module on the write side, message entering the module on the read side, and message exiting the module on the read side.
  39. 39. Monitoring and Troubleshooting » A set of logs in $OMNI_HOME/Logs directory (cont.) » usam_trace.<mmdd>.<ext> - Log collected by the usam_daemon process (SCTP protocol log) - The samdiag utility allows to change the trace level and also to see contents of the usam shared memory (internal structures. etc.) - When an error (errno) is displayed, refer to /usr/include/sys/errno.h to check the meaning For example, if usam_trace.1023.1 contains: 2008/10/23 23:31:50 | ERROR | sam_create_connect | sctp_connect error, errno 145 then cat /usr/include/sys/errno.h | grep 145 shows #define ETIMEDOUT 145 /* Connection Timed Out */ » Event.<SHM>.diag.<mmdd>.<ext> - Collected daily (from 00:00 to 24:00) from log process - if size increases over a given limit, by default set to 10 Mbytes, the files are rotated - The trace level can be set through MML commands (SET-M3UA-TRACE, SET-SUA-TRACE) » Other log files: » /data/omni/omnilog this is the file to which Signalware output is redirected. » $OMNI_HOME/<ce name>/tmp/Event.<SHM>.txt.<mmdd>.<ext> (e.g. /users/omni/t1fmp01/tmp/Event.100.txt.1105.1) this file contains an ASCII string for each logged event
  40. 40. Monitoring and Troubleshooting » Diagnostic tools: » c7m3uadiag - The c7m3uadiag tool allows the user to display the data structures and other relevant information of all M3UA modules. It performs the following: - Displays M3UA shared memory table - Displays traces - Displays M3UA kernel virtual memory - Displays relevant information (e.g., global variables, size of data structures) - It is a menu driven utility - Navigating through sub-menus it allows a lot of operations (e.g. it shows last messages, measurements, RK filters, etc.) - See man page for details » c7sccp_dump - A menu driven tool similar to the previous one, it is used to examine internal SCCP runtime information - See man page for details » samdiag - The samdiag tool is a menu driven utility that performs the following - Displays USAM shared memory (i.e. SCTP data structures, users, remote peers, configuration data, etc.); - Change of USAM trace level (up to single function calls) - See man page for details
  41. 41. Monitoring and Troubleshooting » Diagnostic tools (cont.) » c7MonDecode - The c7MonDecode decodes SCCP/TCAP and ISUP messages stored in a monitor log file (i.e. $OMNI_HOME/Logs/Monitor.log.<SHM>.<file_number>) - It can also be used to convert those files in .pcap format if not available » dmp - This is the Shared Memory Dump Analysis Utility - Shared memory contains a number of measurements and diagnostics that are useful if problems occur. DMP also dumps/formats these variables. - See man page for details » omnimon - Deprecated dump utility, replaced by the newest dumpce » dumpce - dumpce utility collects as much debugging information as possible about the user's system and Signalware configurations and stores them into a .tar file for easy delivery to technical support
  42. 42. Monitoring and Troubleshooting » Other tools: » snoop + Wireshark/Tshark - snoop captures IP packets from the network and displays their contents (or alternatively stores them into a capture file to be viewed through Wireshark) snoop –r –d <netIf> sctp snoop –r –d <netIf> -o <outfile.cap> sctp - Example: $ snoop –r –d e1000g3 sctp -> SCTP D=2905 S=2905 Heartbeat ACK -> SCTP D=2905 S=2905 SACK tsn 6072e817 win 16384 gaps/dups 0/0 -> SCTP D=2905 S=2905 Heartbeat » swmml commands - Basically all the DISPLAY-* commands are useful for debugging purpose
  43. 43. Monitoring and Troubleshooting » An example: » In order to see and log all messages exchanged between M3UA, SCCP and TCAP stream routers, issue the following commands: monitor –ln N100 –m 15 N100.L3RT monitor –ln N100 –m 15 N100.SCCP monitor –ln N100 –m 15 N100.TCAP » The information will be logged into the /users/omni/Logs/Monitor.log.100.xxxxx - To check what happens in real time, use: tail –f /users/omni/Logs/Monitor.log.100.xxxxx - To decode the contents of the file, use: c7MonDecode Monitor.log.100.xxxxx » Do not forget to disable logging when not needed: monitor –ln N100 –m 0 N100.L3RT monitor –ln N100 –m 0 N100.SCCP monitor –ln N100 –m 0 N100.TCAP
  44. 44. Events, Statistics and Measurements » Signalware detects and reports events related to communications and platform management. » Event reporting is done by the log process » Event are stored into daily log files and are also printed to stdout through the pevt tool (see man page for details) » An event reported on the system has the following components: - Event code: 6 digits, the left 3 digits identify the event type, the right 3 digits the event identification (see the manual page of omni_events for a detailed description of available events) - Date and time - System name Name of the CE that reported the event - Severity One of the following: CRITICAL, ILLOGICAL, ERROR, INFO (The severity can be changed through MML command CHANGE-SEVERITY) - Logical name of the process reporting the event - Text Textual description of the events. They can be changed through the NLS feature (man Event_Customization for details).
  45. 45. Events, Statistics and Measurements » Some examples: … 003010 27-Oct-2008 09:04:49 t1fmp01.LOG Info Log Files Closed/Shut Down … … 018280 07-Oct-2008 13:22:12 t1fmp01.L3RT Error MSU discarded due to NI error … … 069893 12-Oct-2008 11:57:23 t1fmp01.M3UA Error SCTP <association> net status change state NOT ACTIVE ipaddr <ipaddr> … Event code (type + identification) Date and time System name Logical process name Severity Textual Description
  46. 46. Events, Statistics and Measurements » The Node Measurements (MEAS) process collects and records measurement data produced by the protocol components of the logical node: » Collected measurements are stored into the following file: $OMNI_HOME/<CE_name>/dffile/Meas.<node>.<SHM>.<mmdd> (e.g. /users/omni/t1fmp01/dffile/Meas.N100.100.1105) » A single file is opened at 00:00 and closed at 24:00 and contains both 5-minutes and 30- minutes measurements for the whole day (for a single logical node). » Measurement data are supplied by MTP, M3UA, SCCP, TCAP and MAP. » MTP/M3UA measurements are available at 5- and 30-minute intervals, while others are available only at 30-minute intervals. » Measurement data can also be retrieved through the MML command RETRIEVE-MEAS » Measurement files are kept for a given period, and are automatically purged after expiration. » The purge period is the same applied to logs and events and is configurable through SCHEDULE-PURGE MML command.
  47. 47. Events, Statistics and Measurements » SCTP measurements are handled differently: » In the Solaris 10 implementation, SCTP is embedded in the Operating System - All SCTP stuff, including statistics, is directly handled by the OS itself - The RETRIEVE-MEAS MML command does not provide SCTP statistics - Similarly, the following MML commands: DISPLAY-SCTP-VARIABLES DISPLAY-SCTP-STATISTICS DISPLAY-SCTP-ASSOCIATIONS are not available on Solaris 10 platform. - SCTP statistics are available through the sctp_stat shell command
  48. 48. Events, Statistics and Measurements » Ulticom events, measurements and alarms are integrated into the np application: » This allows a unique access point to statistics and alarms for the whole NP application » Ulticom measurements are collected by the NP application and stored together with all other statistics into the following directory: $SDP_HOME/stat/ » Alarm management is carried out by SSA application and stored into log files located into: $PLAT_HOME/log » The alarms raised are forwarded to the Vodafone Network Management System using the SNMP agent available in the SSA application.
  49. 49. Signalware Backup » The backup of the logical node’s configuration is performed by Signalware automatically at periodically: » The backup period is set by default to 5 days: - Backup occurs at midnight - Backup period can be changed through the MML command SCHEDULE-BKUP (between 1 and 365 days) - Configuration changes between two backups are stored in a recent change files » The configuration is saved in the form of MML commands into the following files: - is a single configuration file that is saved for the logical node as a whole - files are used for automatic recovery of the configuration when the logical node is restarted. There is a separate file for each protocol manager - are the recent change files, which store changes applied to the configuration since the last backup - Files are located both in $OMNI_HOME/conf and into the DF directory $OMNI_HOME/<CE_name>/dffile - Old backups are purged after the purge period defined by SCHEDULE-PURGE
  50. 50. Signalware Backup » Backup can be forced at any moment by the BACKUP-NODE MML command, which: » stores an updated configuration both into and into files » deletes all recent change files » A given configuration stored in an archive file can be restored by the RESTORE-NODE MML command: » The archive file overwrites the recent change files and files are deleted, so that configuration is reloaded as soon as Signalware is restarted » The command is not allowed if the recent change files are not empty
  51. 51. Agenda » Ulticom’s Signalware SIGTRAN Overview » SCTP Overview » M3UA Overview » Signalware M3UA: Installation, Configuration, Operation & Monitoring » References Back
  52. 52. References » IETF Web Site: and specifically IETF SIGTRAN Working Group: » Some interesting tutorials from Internet Engineering Consortium (IEC): » Ulticom Web site:
  53. 53. Thank you
  54. 54. Back Up Slides
  55. 55. Why TCP is not suited for signalling transport? » TCP provides both reliable data transfer and strictly ordered delivery of data: » Some applications need reliable transfer without sequence maintenance, while others would be satisfied with partial ordering of the data. » In both of these cases, the head-of-line blocking offered by TCP causes unnecessary delay. » TCP is stream-oriented, not message-oriented. Applications must: » add their own record marking to delineate their messages, and » make explicit use of the PSH facility to ensure that a complete message is transferred in a reasonable time » Poor robustness against failures: » No support of multi-homed hosts » Vulnerability against denial-of-service attacks: » e.g. SYN attacks Back
  56. 56. Association Startup and Takedown » An association may be inititiated either locally by the SCTP user or by the remote peer » 4-way handshake with Cookie mechanism » Protection against SYN attacks » Graceful (SHUTDOWN) and ungraceful (ABORT) association termination Back INIT INIT_ACK (cookie) COOKIE (cookie) COOKIE_ACK SCTP (ASP) SCTP (SGP)
  57. 57. Sequenced delivery within streams » SCTP STREAM = Sequence of user messages that are to be delivered in order to the upper-layer protocol » The SCTP user specifies the number of streams to be supported at association startup time. This number is negotiated with the remote end. » No Head-Of_Line blocking → while one stream may be blocked waiting for the next in-sequence user message, delivery from other streams may proceed. » SCTP provides also a mechanism for unordered delivery. Back DATA (Stream ID=1, SSN=2, TSN=12) SACK (CTSN=13) SCTP SCTP DATA (Stream ID=2, SSN=345, TSN=13) DATA (Stream ID=1, SSN=3, TSN=14) DATA (Stream ID=2, SSN=346, TSN=15) DATA (Stream ID=2, SSN=347, TSN=16) SACK (CTSN=16, GAP blocks=1, GAP block start=15, GAP block end = 15)
  58. 58. Acknowledgement and Congestion Avoidance » SCTP assigns a Transmission Sequence Number (TSN) to each user data fragment or unfragmented message. » The TSN is independent of any Stream Sequence Number (SSN) assigned at the stream level. » The receiving end acknowledges all TSNs received, even if there are gaps in the sequence. » In this way, reliable delivery is kept functionally separate from sequenced stream delivery. Back
  59. 59. User Data Fragmentation » For transmission efficiency, SCTP defines mechanisms for bundling of small user messages and fragmentation of large user messages: » When converting user messages into DATA chunks, an endpoint will fragment user messages larger than the current association Path MTU into multiple DATA chunks. The data receiver will normally reassemble the fragmented message from DATA chunks before delivery to the user. » Fragmentation is performed end-to-end (IPv4 supports only hop-by-hop fragmentation, which is less efficient) » Path MTU can be statically provisioned - Some implementations supports also a mechanism for Path MTU discovery. Back
  60. 60. Chunk Bundling » For transmission efficiency, SCTP defines mechanisms for bundling of small user messages and fragmentation of large user messages: » Multiple DATA and control chunks may be bundled by the sender into a single SCTP packet for transmission, as long as the final size of the packet does not exceed the current path MTU. » On the receiving side, the receiver will unbundle the packet back into the original chunks. Back
  61. 61. Packet Validation » Each SCTP Packet contains a mandatory Verification Tag field and a 32-bit checksum field: » The Verification Tag value is chosen by each end of the association during association startup. - Packets received without the expected Verification Tag value are discarded, as a protection against blind masquerade attacks and against stale SCTP packets from a previous association. » The checksum is set by the sender of each SCTP packet to provide additional protection against data corruption in the network. - The receiver of an SCTP packet with an invalid checksum silently discards the packet. - RFC 2960 chose the Adler32 algorithm for packet validation, but this algorithm suffers of poor performance with small packets - RFC 3309 introduced the new CRC32 algorithm Back
  62. 62. Path Management » The SCTP path management function is responsible for choosing the destination transport address for outgoing SCTP packets, based on: » the current reachability status of the remote IP addresses » the SCTP user's instructions » Remote IP addresses’ reachability is continuosuly monitored through SCTP heartbeats when other packet traffic is inadequate to provide this information » SCTP user is notified upon remote IP address status change » At association startup, a primary path is defined for each SCTP endpoint, and is used for normal sending of SCTP packets. Back Multi-Homed SCTP End Point Multi-Homed SCTP End Point Primary Path Alternate Paths IP1 IP2 IP3 IP4
  63. 63. Main Signalware Processes » Parent Operation Process (POP) » First process started when Signalware is launched. » POP provides local process management functions (within a CE). It is responsible of: - Creating the process set’s memory mapped file - Starting other processes - Detecting processes’ failure or termination - Offering services such as automatic process recovery in the event of a failure - Handling shutdown of local process sets and managing the deallocation of resources Back
  64. 64. Main Signalware Processes » Platform Manager (PM) » The Platform Management (PM) process is responsible for the following: - Configuring, unconfiguring, starting, and stopping all processes described in the cestart.<SHM> file (created automatically during SW installation; see example below) $ DFcat cestart.100 .CE t1fmp01 @N100 C7M3UA - Startup and shutdown of all logical nodes - Routing MML commands to logical nodes and other processes - Maintenance of a per-operator command history file of MML commands - Direct handling of Platform Management MML commands, such as: - DISPL-PURGE - SCH-PURGE - DISPLAY-CE* commands - DISPLAY-PLATFORM-STATUS - Automatic purge of measurement and history files - Centralizing the results of test conditions sent by various processes, computing the state of each CE based on these conditions, and logging events. Back
  65. 65. Main Signalware Processes » Signalware Node Manager (SW_NM) » The Node Management (SW_NM) process is responsible for: - startup and shutdown of the Signalware logical node including configuration recovery - routing of MML commands to the concerned protocol managers - maintenance of a Recent Change Log of MML commands - direct handling of node management MML commands, such as: - BACKUP-NODE - DISPLAY-BKUP - RETRIEVE-MEAS - SCHEDULE-BKUP - STOP-NODE - automatic backup of configurations. Back
  66. 66. Main Signalware Processes » Watch Process (WATCH) » Signalware automatically starts the watch process to monitor specified Signalware processes. - Watch performs two distinct types of process monitoring: - Critical Process Watching - List of processes to be monitored contained in a file specified with –f option - When the death of any critical-watch process is detected, watch will allow the process 30 seconds to restart. If the critical process does not restart within this time, watch will shut down Signalware. - Process Health Check Monitoring - List of processes to be monitored contained in a file specified with –h option - Each process in the list will be sent a priority message (API_WATCH_CHECK) to which it must respond (API_WATCH_RESPONSE) - If a process fails to respond to a health check, watch will allow the process up to three such health-cycle failures. If the process does not respond before the allowance expires, watch will kill the process. If this fails, watch will shut down Signalware. Back
  67. 67. Main Signalware Processes » USAM Daemon (USAM_DAEMON) » The usam_daemon process runs on top of the SCTP protocol state machine implemented in the Solaris 10 Operating System. » It is started together with POP when Signalware is launched » It performs the following tasks: - waits for incoming commands from M3UA or SUA - manages SCTP associations (i.e. opens and closes SCTP associations) - waits for incoming calls on an opened SCTP listening port » The TRACE_FILE argument sets the tracing filename. - The usam_daemon does not use the same trace file as other Signalware Logical Node processes - Trace level can be modified by means of the samdiag utility Back
  68. 68. Main Signalware Processes » C7 Protocol Managers (C7xxx processes) » This is a set of processes that handles specific protocols of the C7 protocol stack - The c7m3ua process manages the M3UA protocol activities for a logical node instance that is running in the Signalware environment. - For each logical node, a copy of the c7m3ua process is executed in each CE - c7m3ua controls both network management requests and M3UA related MML operator commands. - The c7scmg process manages the SCCP protocol activities for a logical node instance that is running in the Signalware environment. - For each logical node, a copy of the c7scmg process is executed in each CE - c7scmg controls both network management requests and SCCP related MML operator commands. - The c7tcmg process manages TCAP application-related capabilities for an instance of a logical node running in the Signalware environment. - For each logical node, a copy of the c7tcmg process is executed in each CE - c7tcmg controls both network management requests and TCAP related MML operator commands. Back
  69. 69. Main Signalware Processes » Message Monitor (MON) » MON collects information generated by other Signalware processes and stores them in log files within the $OMNI_HOME/Logs directory (i.e. /users/omni/Logs) » Two separate log files are created by default: - one in displayable HEX/ASCII - Log file name is Monitor.log.<SHM>.<file_number> where <SHM> is the shared memory value and <file_number> is a five digit progressive number (numbering begins from a file number, when the file reaches 2 Mbytes it is closed, the file number is increased and a new log file is created) - Files are purged after a given time period (see DISPLAY-PURGE and SCHEDULE-PURGE commands) - Those files logs messages exchanged between Signalware processes - the other is in binary form (PCAP) - It contains only monitored SS7 Message Signaling Units (MSUs) and can be read through Ethereal/Wireshark - Log file name is Monitor.pcap.<SHM>.<file_number> - The .pcap file is closed along with the corresponding .log (files are always synchronized) » Use MonControl utility to select specific file-type logging and monitor to define monitoring points Back
  70. 70. Main Signalware Processes » Event Logging and Distribution Process (LOG) » The log process records and distributes process-set-generated Events, Error Reports, and Information Notices. » The log process writes data to the following files: - $OMNI_HOME/<ce name>/tmp/Event.<SHM>.txt.<mmdd>.<ext> (e.g. /users/omni/t1fmp01/tmp/Event.100.txt.1105.1 this file contains an ASCII string of each logged event) - $OMNI_HOME/<ce name>/tmp/Event.<SHM>.nls.<mmdd>.<ext> (same as above, but events are logged according to the Native Language Support feature) - $OMNI_HOME/Logs/Event.<SHM>.diag.<mmdd>.<ext> (e.g. /users/omni/Logs/Event.100.diag.1105.1 This file contains diagnostic information provided by the local log process to log critical errors) » Files are opened at 00:00 and closed at 24:00 (if size increases over a given limit, by default set to 10 Mbytes, the files are rotated) » The trace level can be set through MML commands (SET-M3UA-TRACE, SET-SUA- TRACE) Back
  71. 71. Other Signalware Processes » Parent Update Process (PUP) » Used by POP in a multiple-CE environment to distribute information on table and state change to other CEs. » Signalware Resource Usage Monitor (RESMON) » RESMON samples the system resources of the local CE at regular intervals to determine whether CPU, memory, and disk utilization do not exceed given thresholds. » If such threshold are exceeded, RESMON notifies PM, which marks the corresponding CE as not fully operational » Thresholds and sampling rate are configurable in omni_conf_info file » The operator can monitor platform status through the DISPLAY-PLATFORM-STATUS MML command » Node Measurements Process (MEAS) » The Node Measurements (MEAS) process collects and records measurement data produced by the protocol components of the logical node. » Delayed Message Delivery Process (DLY) » The Delayed Message Delivery (dly) process collects and forwards IPC messages when a specified day/time is reached. » Signal Distribution Process (SIG) » The Signal Distribution process (sig) allows a process to send a signal to another process set member that does not operate using the same user ID or group ID. Back
  72. 72. An example of node provisioning (MIFMP01) » Node MIFMP01 [PC=H’0918] has 4 Ethernet Interfaces on 2 distinct Control LANs: » 2 ASPs, each with 2 IP addresses laying on different Control LANs and hosted on different PCI cards for maximum redundancy » ASPs load share traffic » MIFMP01 is connected to 4 Signalling Gateways (Eagle 5): » BOSGW03 [PC=H’100C] » MISGW03 [PC=H’080F] » NASGW03 [PC=H’200C] » RMSGW03 [PC=H’180C] » An M3UA Link Set is defined towards each SGW: » Link set name is MI01YYSGW03, where YY may be MI, NA, RM, BO » Each Link Set contains 8 SCTP associations, 4 from ASP 1 (A) and 4 from ASP 2 (B) Back BOSGW03 MISGW03 NASGW03 RMSGW03 MIFMP01 MI01BOSGW03 MI01MISGW03 MI01NASGW03 MI01RMSGW03 Primary Route Alternate Route
  73. 73. An example of node provisioning (MIFMP01) » As an example, here are the 8 SCTP associations (M3UA SLK according to Signalware terminology) included in M3UA LSET MI01BOSGW03 (to BOSGW03 SGW) » 4 Associations from ASP A (local IP addresses & » 4 Associations from ASP B (local IP addresses & » Association name is MI01xxxxzw, where xxxx is the SGW, z is A or B according to local ASP, w is an incremental number Back Local ASP Remote ASP M3UA SLK Primary LOCAL IP Secondary LOCAL IP Primary REMOTE IP Secondary REMOTE IP MIFMP01_A BOSGW03_W MI01BO03A1 MIFMP01_A BOSGW03_X MI01BO03A2 MIFMP01_A BOSGW03_Y MI01BO03A3 MIFMP01_A BOSGW03_Z MI01BO03A4 MIFMP01_B BOSGW03_A MI01BO03B1 MIFMP01_B BOSGW03_AA MI01BO03B2 MIFMP01_B BOSGW03_AB MI01BO03B3 MIFMP01_B BOSGW03_V MI01BO03B4
  74. 74. An example of node provisioning (MIFMP01) » MML commands used for MIFMP01 provisioning: » Create Own Point Code: CREATE-OSPC:PC=2328,NI=NAT1; # PC=H’0918=2328 » Create IP Link SETs CREATE-M3UA-LSET:LSET=MI01BOSGW03,TYPE=ASP-SGP, RADDR=; CREATE-M3UA-LSET:LSET=... » Create IP links CREATE-M3UA-SLK:SLK=MI01BO03A1,LSET=MI01BOSGW03, LADDR=, RADDR=,ASPID=1,MODE=CONNECT, RPORT=2905,LPORT=2905; # This establishes SCTP association CREATE-M3UA-SLK:SLK=... » Create Route SETs CREATE-RSET:RSET=MI01BO, PC=4108, RTES=MI01BOSGW03&MI01MISGW03, LOADSHR=NO; # LOADSHR=NO implies first LSET is primary, second one is alternate CREATE-RSET:RSET=... Back
  75. 75. An example of node provisioning (MIFMP01) » MML commands used for MIFMP01 provisioning (cont.): » Create M3UA Routing Key CREATE-M3UA-RKEY:RKEY=RKMI01,TYPE=STATIC-AS,LSET=MI01BOSGW03& MI01MISGW03&MI01NASGW03&MI01RMSGW03,DPC=2328,CONTEXT=14,SI=SCCP; » Create remote SSNs (not needed) # CREATE-REMSSN not used, to avoid undesired SST messages to SGWs » Create Concerned Point Codes # Each SGW is also CPC # 6 is MAP, 12 INAP, 11 and 146 provisioned for future use (SGW ITZ) CREATE-CPC:PC=4108&2063&8204&6156,SSN=6; CREATE-CPC:PC=4108&2063&8204&6156,SSN=11; CREATE-CPC:PC=4108&2063&8204&6156,SSN=12; CREATE-CPC:PC=4108&2063&8204&6156,SSN=146; » Allow Route SETs ALLOW-RSET:RSET=MI01BO; ALLOW-RSET:RSET=... Back
  76. 76. An example of node provisioning (MIFMP01) » MML commands used for MIFMP01 provisioning (cont.): » Activate Link SETs/IP Links ACTIVATE-M3UA-SLK:SLK=MI01ABOW; ACTIVATE-M3UA-SLK:SLK=... » Activate routing keys ACTIVATE-M3UA-RKEY:RKEY=RKMI01; » Define rules for inbound GTT CREATE-GT:TT=0,NP=ISDN-TEL,NA=INT,DIG="0",PC=2328,SSN=0,RI=GT; CREATE-GT:TT=0,NP=ISDN-TEL,NA=INT,DIG="1",PC=2328,SSN=0,RI=GT; CREATE-GT:TT=0,NP=ISDN-TEL,NA=INT,DIG="2",PC=2328,SSN=0,RI=GT; CREATE-GT:TT=0,NP=ISDN-TEL,NA=INT,DIG="3",PC=2328,SSN=0,RI=GT; CREATE-GT:TT=0,NP=ISDN-TEL,NA=INT,DIG="4",PC=2328,SSN=0,RI=GT; CREATE-GT:TT=0,NP=ISDN-TEL,NA=INT,DIG="5",PC=2328,SSN=0,RI=GT; CREATE-GT:TT=0,NP=ISDN-TEL,NA=INT,DIG="6",PC=2328,SSN=0,RI=GT; CREATE-GT:TT=0,NP=ISDN-TEL,NA=INT,DIG="7",PC=2328,SSN=0,RI=GT; CREATE-GT:TT=0,NP=ISDN-TEL,NA=INT,DIG="8",PC=2328,SSN=0,RI=GT; CREATE-GT:TT=0,NP=ISDN-TEL,NA=INT,DIG="9",PC=2328,SSN=0,RI=GT; Back