Smf deepdive-tran

3,988 views

Published on

SMF Deep Dive

Published in: Education, Technology
  • Be the first to comment

Smf deepdive-tran

  1. 1. SOLARIS 10 DEEP DIVE: SMFSOLARIS 10DEEP DIVE: SMFChristine TranUS Solaris Adoption PracticeSun Microsystems Sun Proprietary/Use by Permission Only
  2. 2. SOLARIS 10 DEEP DIVE: SMFTopics• Part I: SMF Fundamentals > Motivation for SMF > Core concepts and terminology > Command overview > New boot process• Part II: SMF in Action > Quick tour of service administration > Managing inetd services > Tips for troubleshooting > Recovering a repository Sun Proprietary/Use by Permission Only
  3. 3. SOLARIS 10 DEEP DIVE: SMFTopics• Part III: Advanced SMF > Inside the manifest > A little about XML > Writing a manifest from scratch > Exploring the repository with svccfg and svcprop• Part IV: More Advanced SMF > Security aspects > Snaphots > Handling signals > Portable services Sun Proprietary/Use by Permission Only
  4. 4. SOLARIS 10 DEEP DIVE: SMFTopics• Part V: SMF Tools and SMF Community > Tools: Webmin > Tools: xmllint, jedit, Morphon > SMF Community on OpenSolaris > Resources > Review > Q&A > Closing Sun Proprietary/Use by Permission Only
  5. 5. SOLARIS 10 DEEP DIVE: SMFPart I:SMF Fundamentals Sun Proprietary/Use by Permission Only
  6. 6. SOLARIS 10 DEEP DIVE: SMFMotivation for SMF• What is a service?• Is running a software program the same as delivering a service? What exactly is a service? > Answer > Answer > Answer• Does there currently exist an OS framework for service support and management? > Answer > Answer > Answer Sun Proprietary/Use by Permission Only
  7. 7. SOLARIS 10 DEEP DIVE: SMFHow are services started today?• init(1M) vi rc scripts in /etc/rc?.d/* > Long-time running or one-time initializations• inetd(1M) as defined by inetd.conf > Short-lived to provide network functions• init(1M) as defined by /etc/initab > Restartable or one-time functions• How are these services grouped? Could they be related? How are the relationships described? Sun Proprietary/Use by Permission Only
  8. 8. SOLARIS 10 DEEP DIVE: SMFDiagnostic ability• An application fails to start > Missing configuration file? > Failed to mount a filesystem? > Database is late in starting up? > Missing another component?• Lack of knowledge of service boundary and service interdependencies limits error handling ability of system.• Lacks common framework of error-handling• Lacks restart capability Sun Proprietary/Use by Permission Only
  9. 9. SOLARIS 10 DEEP DIVE: SMFService Management Today• Thousands of different text files, arbitrarily grouped and managed with multiple administration techniques.• Undeclared, often unknown dependencies; linear startup is a by-product.• Lacks common interface, each service is started a different way.• Does not address multiple instances of a service, for example, web servers.• Does not address services that may span multiple hosts, for example, a grid. Sun Proprietary/Use by Permission Only
  10. 10. SOLARIS 10 DEEP DIVE: SMFMotivation for SMFA Service Oriented Architecture requires a definition of serviceand a more robust framework to deliver and manage it. Sun Proprietary/Use by Permission Only
  11. 11. SOLARIS 10 DEEP DIVE: SMFSMF gives you• A consistent service model in a common framework: command interface, service description, property specification, status view, etc.• A meaningful system view.• Ability to state dependencies (by-product is parallel boot).• Restart capabilities• All data stored in persistent, transaction-based repository.• Snapshots allow undo and rollback to a working configuration. Sun Proprietary/Use by Permission Only
  12. 12. SOLARIS 10 DEEP DIVE: SMFCore concepts and terminology• A service is: > An abstract description of a long-lived software object. > May describe object on a collection of machines (grid). > May have multiple instances (httpd). > Each instance of a service has a well-defined state and a well- defined error boundary [process contract]. > Each service defines methods: start, stop, refresh, etc. > Each service defines dependencies: what it depends on, and may define dependents: what depends on it. Sun Proprietary/Use by Permission Only
  13. 13. SOLARIS 10 DEEP DIVE: SMFCore concepts and terminology• A service has a unique identifier called FMRI – Fault Management Resource ID svc://localhost/network/login:rlogin Scheme svc – SMF managed service lrc – legacy RC script Sun Proprietary/Use by Permission Only
  14. 14. SOLARIS 10 DEEP DIVE: SMFCore concepts and terminology• A service has a unique identifier called FMRI – Fault Management Resource ID svc://localhost/network/login:rlogin Location localhost – current hostname – future release Sun Proprietary/Use by Permission Only
  15. 15. SOLARIS 10 DEEP DIVE: SMFCore concepts and terminology• A service has a unique identifier called FMRI – Fault Management Resource ID svc://localhost/network/login:rlogin Functional category application system device network milestone platform site Sun Proprietary/Use by Permission Only
  16. 16. SOLARIS 10 DEEP DIVE: SMFFunctional Categories• Application – traditional daemons• Device – useful for dependencies• Milestone – similar to run levels• Network – converted services from inetd.conf• Platform – platform-specific services• System – platform-independent system services• Site – reserved for local site use Sun Proprietary/Use by Permission Only
  17. 17. SOLARIS 10 DEEP DIVE: SMFCore concepts and terminology• A service has a unique identifier called FMRI – Fault Management Resource ID svc://localhost/network/login:rlogin Service Description – related to method or RC script Sun Proprietary/Use by Permission Only
  18. 18. SOLARIS 10 DEEP DIVE: SMFCore concepts and terminology• A service has a unique identifier called FMRI – Fault Management Resource ID svc://localhost/network/login:rlogin Instance – “default” is the default instance Sun Proprietary/Use by Permission Only
  19. 19. SOLARIS 10 DEEP DIVE: SMFCore concepts and terminology• FMRI can be addressed by the shortest unique match, for example: > svc://localhost/network/login:rlogin > svc:/network/login:rlogin > network/login:rlogin > rlogin• Some common names have changed to a different FMRI, for example: syslog is now system-log Sun Proprietary/Use by Permission Only
  20. 20. SOLARIS 10 DEEP DIVE: SMFService states• online – the service instance is enabled and has successfully started.• offline – the service instance is enabled, but the service is not yet running or available to run, usually due to a dependency that has not been satisfied, or an error in the start method.• disabled – the service instance is not enabled and is not running.• maintenance – the service instance has encountered an error that must be resolved before it could be started again. There are many reasons why a service could be in this state. Sun Proprietary/Use by Permission Only
  21. 21. SOLARIS 10 DEEP DIVE: SMFService states• legacy_run – the legacy service is not managed by SMF, but the service can be observed. > Faults not handled by SMF, no automated restart > Administrative error undetected > Software or hardware error results in process death• degraded – the service instance is enabled, but is running at a limited capacity.• uninitialized – this state is the initial state for all services before their configuration has been read. Sun Proprietary/Use by Permission Only
  22. 22. SOLARIS 10 DEEP DIVE: SMFService dependency• A dependency is a formal description of other services that are required to start a service.• A service can be dependent on another service or files. When a dependency is not met, the service stays offline. Whether a dependency is satisfied depends on its type: > require_all – all services are running or all files are present. > require_any – at least one is running or at least one file is present. > optional_all – all are running, disabled, in maintenance, or not present. For files, this type is the same as require_all. > exclude_all – all are disabled, in maintenance, or when files are not present. Sun Proprietary/Use by Permission Only
  23. 23. SOLARIS 10 DEEP DIVE: SMFDependency actions• Once running (online or degraded states are considered running) if a service dependency is stopped or refreshed, SMF determines whether to restart the service depending on the restart_on attribute specified. restart_on attribute Reason for dependency stop None Error Restart Refresh Error no yes yes yes Non-error stop no no yes yes Refresh no no no yes Sun Proprietary/Use by Permission Only
  24. 24. SOLARIS 10 DEEP DIVE: SMFService restarter• /lib/svc/bin/svc.startd is the master process starter and restarter. It > reads the Service Configuration Repository and start services on boot; > restart services that have failed; > shutdown services whose dependencies are no longer satisfied; > run legacy RC scripts at run-level transitions; > provides system view of service status.• There is a delegated restarter, inetd. Sun Proprietary/Use by Permission Only
  25. 25. SOLARIS 10 DEEP DIVE: SMFDelegated restarter• Assumes responsibility for executing a set of services.• Manages service faults; may gracefully handle conditions such as signals or core dumps.• May define/require additional service configuration.• Often provides functionality common to a specific set of services, for example: inetd Sun Proprietary/Use by Permission Only
  26. 26. SOLARIS 10 DEEP DIVE: SMFSMF Manifest• Description of the initial configuration of a service in XML.• Loaded into the SMF repository at boot time.• Sun-delivered services live in /var/svc/manifest.• For ISVs, manifests should be placed in the appropriate subdirectory /var/svc/manifest.• Manifests for anything that is specific to the customers site only can go in /var/svc/manifest/site. This is reserved for local use. Sun Proprietary/Use by Permission Only
  27. 27. SOLARIS 10 DEEP DIVE: SMFSMF Manifest• A manifest identifies, at the very least: > the service name > what the service is dependent on > ways to start and stop the service• Can contain other attributes, such as the delegated restarter, config files, log files and others. Sun Proprietary/Use by Permission Only
  28. 28. SOLARIS 10 DEEP DIVE: SMFService Configuration Repository• Located in /etc/svc.• Distributed between local memory (volatile) and local file (repository.db).• repository.db holds the persistent service description read from the manifests.• volatile is a tmpfs allocated from swap containing transient data (lock files, init state, log files).• The underlying DB engine is sqlite 2.8• Zones have their own repository.• Changes to services should be made against repository. Sun Proprietary/Use by Permission Only
  29. 29. SOLARIS 10 DEEP DIVE: SMFProfile and Archive• Profile – a set of service instances and their enabled or disabled state. Useful for copying service states between systems.• Generated by svccfg extract• Archive – a complete set of persistent data for all service instances. Useful for copying service definitions between systems.• Generated by svccfg archive• Output is an XML file similar to manifests. Sun Proprietary/Use by Permission Only
  30. 30. SOLARIS 10 DEEP DIVE: SMFSMF Snapshots• snapshot – a complete collection of properties for a service instance.• It provides a historical view of a service and simplifies rollback of service configuration changes.• No snapshot-at-will capability yet.• Snapshot types: > initial – taken on the first import of the manifest > last_import – taken during last import > running – properties of the currently running service instance > previous – properties of the service instance just before running > start – taken at the last successful service start Sun Proprietary/Use by Permission Only
  31. 31. SOLARIS 10 DEEP DIVE: SMFKey files and directories• /var/svc – manifests, logs and profiles• /lib/svc – binaries, methods, seed, support files• /etc/svc/repository.db – global configuration database• /lib/svc/seed/global.db – minimal configuration database• /lib/libscf.so.1 – repository APIs Sun Proprietary/Use by Permission Only
  32. 32. SOLARIS 10 DEEP DIVE: SMFNew filesystems• /etc/svc/volatile > Directory where svc.startd stores log files in early stages of boot, before /var is mounted read-write. Contains transient data.• /system/contracts > Primary interface into contract subsystem, to allow service instances to be restarted. Indirectly related to SMF. Sun Proprietary/Use by Permission Only
  33. 33. SOLARIS 10 DEEP DIVE: SMFCommand overview• General commands > svcs(1) service status listings > svcadm(1M) administrative actions > svccfg(1M) general property manipulation > svcprop(1) property reporting• inetd management commands > inetadm(1M) administrative actions/property modification > inetconv(1M) conversion of legacy inetd.conf entries Sun Proprietary/Use by Permission Only
  34. 34. SOLARIS 10 DEEP DIVE: SMFCommand overview• Daemons > svc.startd(1M) dependency engine, master restarter > svc.configd(1M) repository, authorization > inetd(1M) delegated restarter• Contracts subsystem > ctrun(1M) execute with process contract > ctstat(1M) display active contracts > ctwatch(1M) monitor contract events > libcontract(3LIB) contract APIs Sun Proprietary/Use by Permission Only
  35. 35. SOLARIS 10 DEEP DIVE: SMFNew boot process• Instead of booting to run level, SMF introduces the concept of milestone.• milestone – a service which specifies a collection of services which declare a specific state of system- readiness.• You can boot to specific milestone, services not part of that milestone are temporarily disabled.• The default milestone is the milestone you will transition to at every boot. It is usually milestone “all”.• init S, boot -s and friends still work. Sun Proprietary/Use by Permission Only
  36. 36. SOLARIS 10 DEEP DIVE: SMFMilestones and run levels SVR4 Run Level SMF Milestone - none s, S single-user 2 multi-user 3 multi-user-server - all Sun Proprietary/Use by Permission Only
  37. 37. SOLARIS 10 DEEP DIVE: SMFLegacy services• Milestones are services, as such, they have manifests which have defined dependencies.• After all its dependencies have been satisfied, the start method of the SMF major milestone executes each S script within /etc/rc?.d/ with the argument start.• /var/svc/manifest/milestone/single-user.xml specifies a series of dependencies, and the start method is /sbin/rcS start.• SMF services first, legacy services last, run once only. Sun Proprietary/Use by Permission Only
  38. 38. SOLARIS 10 DEEP DIVE: SMFPart II:SMF in Action Sun Proprietary/Use by Permission Only
  39. 39. SOLARIS 10 DEEP DIVE: SMFService administrationsvcs – reports service status• svcs -a show all services• svcs -x show services not running, impact• svcs -d show dependencies (what this depends on)• svcs -D show dependents (what depends on this)• svcs -p show member processes• svcs -l/-v show additional details Sun Proprietary/Use by Permission Only
  40. 40. SOLARIS 10 DEEP DIVE: SMFsvcs# svcs -aSTATE STIME FMRIlegacy_run Oct_12 lrc:/etc/rc2_d/S20sysetuponline Oct_12 svc:/network/ssh:defaultdisabled Oct_12 svc:/application/print/server:defaultoffline Oct_12svc:/application/print/rfc1179:default... Sun Proprietary/Use by Permission Only
  41. 41. SOLARIS 10 DEEP DIVE: SMFsvcs# svcs -xsvc:/application/print/server:default (LP print server) State: disabled since Wed Oct 12 16:18:38 2005Reason: Disabled by an administrator. See: http://sun.com/msg/SMF-8000-05 See: lpsched(1M)Impact: 2 dependent services are not running. (Use -v for list.) Sun Proprietary/Use by Permission Only
  42. 42. SOLARIS 10 DEEP DIVE: SMFsvcs# svcs -d httpSTATE STIME FMRIonline Oct_12 svc:/network/loopback:defaultonline Oct_12 svc:/network/physical:default# svcs -D loopbackSTATE STIME FMRIdisabled Oct_12 svc:/network/dns/client:defaultdisabled Oct_12 svc:/network/dns/server:defaultdisabled Oct_12 svc:/network/rarp:default Sun Proprietary/Use by Permission Only
  43. 43. SOLARIS 10 DEEP DIVE: SMFsvcs# svcs -p sendmailSTATE STIME FMRIonline 16:09:32 svc:/network/smtp:sendmail 16:10:32 23724 sendmail 16:10:32 23725 sendmail Sun Proprietary/Use by Permission Only
  44. 44. SOLARIS 10 DEEP DIVE: SMFsvcs# svcs -l rarpfmri svc:/network/rarp:defaultname Reverse Address Resolution Protocol (RARP) serverenabled falsestate disablednext_state nonestate_time Wed Oct 12 16:18:41 2005restarter svc:/system/svc/restarter:defaultdependency require_all/error file://localhost/etc/nsswitch.conf (online)dependency require_any/error Sun Proprietary/Use by Permission Only
  45. 45. SOLARIS 10 DEEP DIVE: SMFService Administrationsvcadm – administer services -t temporary -r recursive -s synchronously waits for completion• svcadm enable/disable enable/disable services• svcadm enable/disable -t temporarily, until rebooted• svcadm restart restart specified service• svcadm refresh re-read service configuration• svcadm clear/mark clear/mark maintenance state• svcadm milestone transition to a milestone• svcadm -d milestone transition and set as default Sun Proprietary/Use by Permission Only
  46. 46. SOLARIS 10 DEEP DIVE: SMFsvcadm# svcs sendmailSTATE STIME FMRIonline 16:49:59 svc:/network/smtp:sendmail# svcadm disable -t sendmail# svcs sendmailSTATE STIME FMRIdisabled 16:51:27 svc:/network/smtp:sendmail• Does not show service in temporary state. The presence of this property tells us that this setting is temporary.# svcprop sendmail | grep general_ovr/enabledgeneral_ovr/enabled boolean false Sun Proprietary/Use by Permission Only
  47. 47. SOLARIS 10 DEEP DIVE: SMFsvcadm keyserv# svcs keyservSTATE STIME FMRIdisabled Oct_12 svc:/network/rpc/keyserv:default# svcadm enable keyserv# svcs keyservSTATE STIME FMRImaintenance 16:58:57 svc:/network/rpc/keyserv:default• Keyserv is in maintenance mode, why? Sun Proprietary/Use by Permission Only
  48. 48. SOLARIS 10 DEEP DIVE: SMFsvcadm keyserv# svcs -xsvc:/network/rpc/keyserv:default (RPC encryption key storage) State: maintenance since Wed Oct 19 16:58:57 2005Reason: Start method exited with $SMF_EXIT_ERR_CONFIG. See: http://sun.com/msg/SMF-8000-KS See: keyserv(1M) See: /var/svc/log/network-rpc-keyserv:default.logImpact: This service is not running.• Actually, it is because the domain name is not set on this host. Sun Proprietary/Use by Permission Only
  49. 49. SOLARIS 10 DEEP DIVE: SMFsvcadm keyserv# domainname test.com# domainnametest.com• There is a service called domain, which takes care of system identifying info; it has to be restarted.# svcs -o fmri,desc domainFMRI DESCsvc:/system/identity:domain system identity(domainname)# svcs domainSTATE STIME FMRIonline Oct_12 svc:/system/identity:domain# svcadm restart domain Sun Proprietary/Use by Permission Only
  50. 50. SOLARIS 10 DEEP DIVE: SMFsvcadm keyserv# svcs keyservSTATE STIME FMRImaintenance 16:58:57 svc:/network/rpc/keyserv:default• Keyserv is still in maintenance, why?# svcadm clear keyserv# svcs keyservSTATE STIME FMRIonline 17:02:48 svc:/network/rpc/keyserv:default• Will this survive a reboot? Sun Proprietary/Use by Permission Only
  51. 51. SOLARIS 10 DEEP DIVE: SMFsvcadm milestone• What milestone am I in?# svcprop -p options_ovr/milestone system/svc/restarter:defaultsvcprop: Couldnt find property `options_ovr/milestone for instance `svc:/system/svc/restarter:default.• If you see this, you are in milestone “all”.# svcadm milestone single-user# Requesting System Maintenance ModeConsole login service(s) cannot runOct 19 17:19:00 zone1 syslogd:going down on signal 15svc.startd: The system is coming down for administration. Please wait.svc.startd: Killing user processes: done. Sun Proprietary/Use by Permission Only
  52. 52. SOLARIS 10 DEEP DIVE: SMFsvcadm milestonesvc.startd: Killing user processes: done.Root password for system maintenance (control-d to bypass):single-user privilege assigned to /dev/console.Entering System Maintenance Mode# svcprop -p options_ovr/milestone system/svc/restarter:defaultsvc:/milestone/single-user:default# reboot# svcprop -p options_ovr/milestone system/svc/restarter:defaultsvcprop: Couldnt find property `options_ovr/milestone for instance `svc:/system/svc/restarter:default.• Im back in milestone “all”. Sun Proprietary/Use by Permission Only
  53. 53. SOLARIS 10 DEEP DIVE: SMFsvcadm milestone# svcadm milestone -d single-user# Requesting System Maintenance Mode# reboot# svcprop -p options_ovr/milestone system/svc/restarter:defaultsvc:/milestone/single-user:default• Every subsequent reboot will get me back here. To get back to full service, set default milestone to be “all”.# svcadm milestone -d all Sun Proprietary/Use by Permission Only
  54. 54. SOLARIS 10 DEEP DIVE: SMFManaging inetd services• Definition for inetd-started services were in inetd.conf but inetd.conf is now deprecated. Look how short it is!• inetd.conf entries are converted to SMF manifests and imported into the repository on initial boot.• The old way of adding a service by modifying inetd.conf and HUPing inetd no longer works.• If you have an application that depends on whats in inetd.conf, there is a tool called inetconv that will convert the inetd.conf entry to an SMF manifest. Sun Proprietary/Use by Permission Only
  55. 55. SOLARIS 10 DEEP DIVE: SMFinetd• inetd is now a delegated restarter for inet services.• It manages inet-specific properties such as > tcp_wrappers > max connection rate• It manages state for inet services.• It listens for connections, etc. Sun Proprietary/Use by Permission Only
  56. 56. SOLARIS 10 DEEP DIVE: SMFinetconv# grep tftp inetd.conf# TFTPD - tftp server (primarily used for booting)# tftp dgram udp6 wait root /usr/sbin/in.tftpd in.tftpd -s /tftpboot• Its commented out. Uncomment the entry. You can HUP inetd but take my word that that wont do anything. Sun Proprietary/Use by Permission Only
  57. 57. SOLARIS 10 DEEP DIVE: SMFinetconv# inetconvinetconv: Notice: Service manifest for 100235/1 already generated as / var/svc/manifest/network/rpc/100235_1- rpc_ticotsord.xml, skippedtftp -> /var/svc/manifest/network/tftp-udp6.xmlImporting tftp-udp6.xml ...Doneinetconv: Notice: Service manifest for 100083/1 already generated as / var/svc/manifest/network/rpc/100083_1-rpc_tcp.xml, skippedinetconv: Notice: Service manifest for 100068/2-5 already generated as / var/svc/manifest/network/rpc/100068_2-5-rpc_udp.xml, skipped Sun Proprietary/Use by Permission Only
  58. 58. SOLARIS 10 DEEP DIVE: SMFinetconv# svcs tftpsvcs: Pattern tftp doesnt match any instances# svcs -l udp6fmri svc:/network/tftp/udp6:defaultname tftpenabled truestate onlinenext_state nonestate_time Wed Oct 19 17:40:16 2005restarter svc:/network/inetd:default Sun Proprietary/Use by Permission Only
  59. 59. SOLARIS 10 DEEP DIVE: SMFService Administrationinetadm – administer inetd-started services• inetadm list all services managed by inetd• inetadm -p show default inetd service property values• inetadm -l show all properties for a service• inetadm -e/-d enable/disable a service• inetadm -m modify property values• inetadm -M modify inetd default property values Sun Proprietary/Use by Permission Only
  60. 60. SOLARIS 10 DEEP DIVE: SMFinetadm# inetadm -l udp6SCOPE NAME=VALUE name="tftp" endpoint_type="dgram" proto="udp6" isrpc=FALSE wait=TRUE exec="/usr/sbin/in.tftpd -s /tftpboot" user="root" ...• Lets say you want to run with some other option, -X instead of -s. Sun Proprietary/Use by Permission Only
  61. 61. SOLARIS 10 DEEP DIVE: SMFinetadm# inetadm -m udp6 exec="/usr/sbin/in.tftpd -X / tftpboot"# inetadm -l udp6SCOPE NAME=VALUE name="tftp" endpoint_type="dgram" proto="udp6" isrpc=FALSE wait=TRUE exec="/usr/sbin/in.tftpd -X /tftpboot" user="root" Sun Proprietary/Use by Permission Only
  62. 62. SOLARIS 10 DEEP DIVE: SMFinetadm• Can also use inetadm to manage inetd-services as well as svcadm.# svcs udp6STATE STIME FMRIonline 17:49:44 svc:/network/tftp/udp6:default# inetadm -d udp6# svcs udp6STATE STIME FMRIdisabled 18:01:53 svc:/network/tftp/udp6:default Sun Proprietary/Use by Permission Only
  63. 63. SOLARIS 10 DEEP DIVE: SMFinetadm• Usually if you dont use a service, its enough to disable it using svcadm. But you want it really gone then delete it from your repository using svccfg delete FMRI. Sun Proprietary/Use by Permission Only
  64. 64. SOLARIS 10 DEEP DIVE: SMFService Administrationsvcprop – get service configuration properties• -p property FMRI fetch a property• -s snapshot FMRI fetch property from the snapshot# svcprop -p inetd_start/exec tftp/udp6/usr/sbin/in.tftpd -s /tftpboot# svcprop -p restarter/state tftp/udp6online Sun Proprietary/Use by Permission Only
  65. 65. SOLARIS 10 DEEP DIVE: SMFService Administrationsvccfg – modify service configurations• select FMRI select a service• listprop show properties for a service• setprop change a property value for a service• delprop delete a property value for a service• delete delete a service• export|import export|import a service description in XML• extract|apply extract|apply a profile• archive dump a full XML service description for all services Sun Proprietary/Use by Permission Only
  66. 66. SOLARIS 10 DEEP DIVE: SMFsvccfg# svccfg -s print/server listprop lpsched/fd_limitlpsched/fd_limit count 0# svccfgsvc:> select print/serversvc:/application/print/server> listprop lpsched/fd_limitlpsched/fd_limit count 0svc:/application/print/server> setprop lpsched/fd_limit = 8192svc:/application/print/server> exit# svcadm refresh print/server# svccfg -s print/server listprop lpsched/fd_limitlpsched/fd_limit count 8192 Sun Proprietary/Use by Permission Only
  67. 67. SOLARIS 10 DEEP DIVE: SMFsvccfg• Can be used in batch or interactive mode.• Understands globbing to form regex• Well work through a more pertinent example later. Sun Proprietary/Use by Permission Only
  68. 68. SOLARIS 10 DEEP DIVE: SMFTips for troubleshooting• If a single service is broken, -x is your friend. svcs -x will display the service logfile, if it exists.• See service start message: > boot -m verbose.• For a system that hangs during boot: > boot -m verbose,milestone=none > log in at prompt > svcadm milestone all• Watch system progress with svcs. Sun Proprietary/Use by Permission Only
  69. 69. SOLARIS 10 DEEP DIVE: SMFTips for troubleshooting• truss the service starting up by trussing the restarter• Or modify the start method, wrapping it in truss# svcprop -p start/exec apache2/lib/svc/method/http-apache2 start# svccfg -s apache2svc:/network/http:apache2> setprop start/exec = “/usr/bin/truss -ealfo /tmp/apache2.truss /lib/svc/method/http-apache2 start”• Dont forget svcadm clear once fault is rectified. Sun Proprietary/Use by Permission Only
  70. 70. SOLARIS 10 DEEP DIVE: SMFLast resorts• Have changes been made to repository? > How many services are affected? > What snapshots exist? > Can you revert back to a previous snapshot?• Read /lib/svc/share/README.• Restore repository from backup. Sun Proprietary/Use by Permission Only
  71. 71. SOLARIS 10 DEEP DIVE: SMFRevert a snapshot# svccfg -s svc:/network/dns/server:defaultsvc:/network/dns/server:default> listsnapinitiallast-importpreviousrunningstartsvc:/network/dns/server:default> revert startsvc:/network/dns/server:default> exit# svcadm refresh dns/server# svcadm restart dns/server Sun Proprietary/Use by Permission Only
  72. 72. SOLARIS 10 DEEP DIVE: SMFRecovering a repository• The SMF repository can be recovered in case of corruption.• You may want to save the current repository in /etc/svc/repository.db for post mortem.• Restore script is /lib/svc/bin/restore_repository.• If you restore from seed, next boot would be the initial boot, you will see SMF reading and importing manifests.# zlogin -C zone1[Connected to zone zone1 console] 40/104 Sun Proprietary/Use by Permission Only
  73. 73. SOLARIS 10 DEEP DIVE: SMFrestore_repository# /lib/svc/bin/restore_repositoryThe following backups of /etc/svc/repository.db exist, from oldest to newest:boot-20051012_125317manifest_import-20051012_130044boot-20051012_161835boot-20051019_172301boot-20051019_180815# Please enter one of:1) boot, for the most recent post-boot backup2) manifest_import, for the most recent manifest_import backup.3) a specific backup repository from the above list Sun Proprietary/Use by Permission Only
  74. 74. SOLARIS 10 DEEP DIVE: SMFrestore_repository4) -seed-, the initial starting repository. (All customizations will be lost.)5) -quit-, to cancel.Enter response [boot]: -seed-After confirmation, the following steps will be taken:svc.startd(1M) and svc.configd(1M) will be quiesced, if running./etc/svc/repository.db -- renamed --> /etc/svc/repository.db_old_20051019_182048/lib/svc/seed/nonglobal.db -- copied --> /etc/svc/repository.dband the system will be rebooted with reboot(1M).Proceed [yes/no]? yes Sun Proprietary/Use by Permission Only
  75. 75. SOLARIS 10 DEEP DIVE: SMFPart III:Advanced SMF Sun Proprietary/Use by Permission Only
  76. 76. SOLARIS 10 DEEP DIVE: SMFManifest revisited• Manifest is the description of the initial configuration of a service, delivered in XML.• Automatically imported into repository on boot, install, upgrade, and pkgadd by svc:/system/manifest-import.• Ignored once imported unless modified.• Do not modify Sun-delivered manifests! If you need to change service properties, do it in the repository. Well talk more about this. Sun Proprietary/Use by Permission Only
  77. 77. SOLARIS 10 DEEP DIVE: SMFOrder is important• Service bundle• Service name• Create default instance• Single instance• Dependencies• Exec methods• Property groups• Stability level• Template Sun Proprietary/Use by Permission Only
  78. 78. SOLARIS 10 DEEP DIVE: SMFA little about XML• XML markups contain data about data, metadata, on the information they bracket. > <time>1200</time>• Tells you at 1200 is a time unit, instead of weight or distance. Whatever processes XML data can recognize that data for what it is.• Can contain any kind of data. How do we differentiate? An XML file of train schedule will have different formatting than an XML file of TV schedule, not just formatting, but different processing and different usage. Yet, <time>1200</time> can appear in both XML files. Sun Proprietary/Use by Permission Only
  79. 79. SOLARIS 10 DEEP DIVE: SMFIntroducing DOCTYPE• Look at the top of each manifest, they all have this line: > <!DOCTYPE service_bundle SYSTEM /usr/share/lib/xml/dtd/service_bundle.dtd.1>• DTD – Document Type Definition, contrast with DOCTYPE for HTML.• Tells you whether you (especially if you are an XML- processing program) are reading a train schedule or a TV Guide listing.• DTD is one example of a type of markup called declaration. Specifically, the DTD is used to constrain the grammar of an XML file. Sun Proprietary/Use by Permission Only
  80. 80. SOLARIS 10 DEEP DIVE: SMFGrammar?• Grammar governs the “correctness” of an XML document. A train schedule must have: source, terminus, track number, departure and arrival time. It wont have channel, show name, show time, duration, starring, rating.• Looking at just data, they look very similar. The DTD defines and enforces how they must be marked up in XML, one way and one way only.• DTD is a collection of rules, or declarations, describing markup objects.• An element declaration defines what it can contain and in which order, also known as a content model. Sun Proprietary/Use by Permission Only
  81. 81. SOLARIS 10 DEEP DIVE: SMFDefine your own language• Strictly speaking, XML is not a markup language. It has no tag with absolute meaning. <time> can mean absolute time or elapsed time. Contrast this with <b></b>, this always means bold.• XML lays down framework for you to construct your own markup language.• Adding elements in the DTD adds objects to your language. An element not declared in the DTD is illegal. An element containing anything not declared in the DTD is illegal.• Well see why this is important later on. Sun Proprietary/Use by Permission Only
  82. 82. SOLARIS 10 DEEP DIVE: SMFWhats in the DTD?• A bunch of things! None resembling what you might find in a manifest. > <!-- comments --> are comments > <!ELEMENT> or <!ATTLIST> are declarations > <blah /> is same as <blah></blah>• Ordering in DTD is not same as the required ordering in manifest. For example, declaration for property group is before declaration of service FMRI, and service bundle is declared last. Sun Proprietary/Use by Permission Only
  83. 83. SOLARIS 10 DEEP DIVE: SMFHeres an snippet<!ELEMENT dependency ( service_fmri*, stability?, ( propval | property )* ) ><!ATTLIST dependency name CDATA #REQUIRED grouping ( require_all | require_any | exclude_all | optional_all ) #REQUIRED restart_on ( error | restart | refresh | none ) #REQUIRED type CDATA #REQUIRED delete ( true | false ) "false" > Sun Proprietary/Use by Permission Only
  84. 84. SOLARIS 10 DEEP DIVE: SMFWhy XML?• Think about whats in /etc/rc?.d today. Anyone can write anything in any format and drop it in /etc/rc?.d. In fact, whats in /etc/rc?.d is closer to methods than manifests.• A unified service management tool depends on having uniform data to process. The repository is a database which relies on having correct data in the correct “cell”.• With XML you get a parser and a validator for free. Anything that XML cant validate wont get imported.• The parser is in a library called libxml2. It comes with xmllint, which is a generic XML validator. Sun Proprietary/Use by Permission Only
  85. 85. SOLARIS 10 DEEP DIVE: SMFA quick tour inside the manifest• We said a manifest must have at least > the service name > what the service is dependent on > ways to start and stop the service > in theory you need a restarter too but there is the default svc.startd.• House-keeping robot analogy > Lights > Depends on Power > On/Off switch on the wall, flip up or down > Robot is the default restarter Sun Proprietary/Use by Permission Only
  86. 86. SOLARIS 10 DEEP DIVE: SMFtftp-udp6.xmlservice name=network/tftp/udp6 type=service <exec_method type=method name=inetd_start exec=/usr/sbin/in.tftpd -s /tftpboot<exec_method type=method name=inetd_offline exec=:kill_process Sun Proprietary/Use by Permission Only
  87. 87. SOLARIS 10 DEEP DIVE: SMFtftp-udp6.xml<restarter> <service_fmri value=svc:/network/inetd:default /> </restarter>• Note that the method isnt “start”, but “inetd_start”.• Wait! No dependencies? Well, having inetd as your delegated restarter implicitly marks you as dependent on inetd. Who starts inetd, then? Its started in /var/svc/profile/generic_open.xml• If you look at /var/svc/manifest/network/nfs/server.xml, you will see that there is no explicitly defined restarter. Sun Proprietary/Use by Permission Only
  88. 88. SOLARIS 10 DEEP DIVE: SMFWriting a manifest from scratch• Not all Sun products are delivered in a manifest, the reason seems to be time and effort. The README file for MySQL still says to link a startup script in /etc/rc3.d to the mysql.server script.• If no manifest is delivered, the service can still run as a legacy service.• Were going to convert MySQL into an SMF service.• You may want to do this in a zone. Remember that each zone has their own manifests and SMF repository. Sun Proprietary/Use by Permission Only
  89. 89. SOLARIS 10 DEEP DIVE: SMFSetting up MySQL database• First, we need to set up the database. Instructions are in /etc/sfw/mysql/README.solaris.mysql. The startup script is /etc/sfw/mysql/mysql.server.root@mysql~> /usr/sfw/bin/mysql_install_dbPreparing db tablePreparing host tablePreparing user tablePreparing func tablePreparing tables_priv tablePreparing columns_priv tableInstalling all prepared tables13:38:46 /usr/sfw/sbin/mysqld: Shutdown Complete Sun Proprietary/Use by Permission Only
  90. 90. SOLARIS 10 DEEP DIVE: SMFMySQL• Add users and groupsroot@mysql~> groupadd mysqlroot@mysql~> useradd -g mysql mysqlroot@mysql~> chgrp -R mysql /var/mysqlroot@mysql~> chmod -R 770 /var/mysql• Add configuration filesroot@mysql~> installf SUNWmysqlr /var/mysql d 770 root mysqlroot@mysql~> cp /usr/sfw/share/mysql/my-medium.cnf/var/mysql/my.cnf Sun Proprietary/Use by Permission Only
  91. 91. SOLARIS 10 DEEP DIVE: SMFMySQL• Start MySQL manually to see that everything worksroot@mysql~> /usr/sfw/sbin/mysqld_safe --user=mysql &Starting mysqld daemon with databases from /var/mysql• Set the passwordsroot@mysql~> /usr/sfw/bin/mysqladmin -u root password mysqlroot@mysql~> /usr/sfw/bin/mysqladmin -u root -h mysql password mysql• Check the status of MySQLroot@mysql~> /usr/sfw/bin/mysqladmin -u root -pmysql statusUptime: 37 Threads: 1 Questions: 1 Slow queries: 0 Opens: 6 Flush tables: 1 Open tables: 0 Queries per second avg: 0.02 Sun Proprietary/Use by Permission Only
  92. 92. SOLARIS 10 DEEP DIVE: SMFCopy and modify• Great! Everything works. Now you are ready to write your manifest!• Find a manifest similar to your application and tweak it. Since MySQL is an application, Ill look in /var/svc/manifest/application. The print/server.xml seems closest to what I want, so Ill start from that. Remember that site-specific manifests go into “site”.• Most everything is already filled out, I just change a few relevant properties.root@mysql~> cp /var/svc/manifest/application/print/server.xml /var/svc/manifest/site/mysql.xml Sun Proprietary/Use by Permission Only
  93. 93. SOLARIS 10 DEEP DIVE: SMFName• Change the name<service_bundle type=manifest name=SUNWsqlr:mysql><service name=application/mysql type=service version=1> Sun Proprietary/Use by Permission Only
  94. 94. SOLARIS 10 DEEP DIVE: SMFDependency• I leave the existing dependencies alone. I may want to add a dependency on the config file, so MySQL doesnt start if its missing.<dependency name=database grouping=require_all restart_on=none type=path> <service_fmri value=file://localhost/var/mysql/my.cnf/> </dependency> Sun Proprietary/Use by Permission Only
  95. 95. SOLARIS 10 DEEP DIVE: SMFThis just in• File dependencies going may be going away.• CR 6240573 > Synopsis: File dependencies are currently overused by SMF consumers. The problem is that a file dependency means "wait for this file to become available before starting". Thats not the usual case. Most users of file dependencies are trying to express the concept that "if this file is missing, there is a problem."• No file event to signal SMF if a file has changed.• Actions in a file dependency doesnt do anything, yet. Sun Proprietary/Use by Permission Only
  96. 96. SOLARIS 10 DEEP DIVE: SMFMethods• Change the start/stop methods<exec_method type=method name=start exec=/etc/sfw/mysql/mysql.server start timeout_seconds=60 /><exec_method type=method name=stop exec=/etc/sfw/mysql/mysql.server stop timeout_seconds=60 /> Sun Proprietary/Use by Permission Only
  97. 97. SOLARIS 10 DEEP DIVE: SMFDescription and documentation• For now, delete the property groups as they dont apply to mysql• Change the description and documentation location<loctext xml:lang=C> MySQL server</loctext><documentation> <manpage title=mysql section=1M manpath=/usr/sfw/man /></documentation> Sun Proprietary/Use by Permission Only
  98. 98. SOLARIS 10 DEEP DIVE: SMFImporting your manifest• Youre done! Now import it.root@mysql~> svccfg import mysql.xmlroot@mysql~> svcs -l mysqlfmri svc:/application/mysql:defaultname MySQL serverenabled falsestate disablednext_state nonestate_time Tue Jun 07 19:20:51 2005restarter svc:/system/svc/restarter:default Sun Proprietary/Use by Permission Only
  99. 99. SOLARIS 10 DEEP DIVE: SMFImporting your manifestdependency require_all/none svc:/system/filesystem/local (online)dependency require_all/none svc:/system/filesystem/usr (online)dependency require_all/refresh svc:/system/identity:domain (online)dependency optional_all/none svc:/system/system-log (online)dependency require_all/none file://localhost/var/mysql/my.cnf (onlineroot@mysql~> svcadm enable mysqlroot@mysql~> svcs mysqlSTATE STIME FMRIonline 19:21:19 svc:/application/mysql:default Sun Proprietary/Use by Permission Only
  100. 100. SOLARIS 10 DEEP DIVE: SMFDone!• Congratulations! You have successfully written a service manifest. Now MySQL is an SMF-managed service. Sun Proprietary/Use by Permission Only
  101. 101. SOLARIS 10 DEEP DIVE: SMFSMF on OpenSolaris• Collection of manifests and methods are at http://www.opensolaris.org/os/community/smf/manifests/• There is a different manifest for MySQL submitted there. Sun Proprietary/Use by Permission Only
  102. 102. SOLARIS 10 DEEP DIVE: SMFExploring the repository• We said the repository is a database containing all descriptions of all services SMF knows about, and their states.• Make changes to service against its content in the repository, not in the manifest. The days of looking in a text file to find whats running is over. In fact, whats in a manifest can differ vastly with the running configuration.• Well fix Bug ID 6270245 as an example.• Bug ID: 6270245 > Synopsis: apache2 needs svc:/system/filesystem/local dependency when Sun Volume Manager metadevices are in use Sun Proprietary/Use by Permission Only
  103. 103. SOLARIS 10 DEEP DIVE: SMFTwo ways to fix• Two ways to fix:• Delete the service from the repository, fix the manifest and re-import. This is not the recommended way.• Fix the currently running service configuration. This is the recommended way, and well talk about why after weve finished the exercise.• Were going to do this both ways, in order to better understand the repository and the manifest.• In this case the service can be addressed as http or apache2, http is the service, apache2 is the instance.• Disable the service, then delete it from the repository. Sun Proprietary/Use by Permission Only
  104. 104. SOLARIS 10 DEEP DIVE: SMFFix 1: deleting http from repository# svccfgsvc:> list network/httpnetwork/httpsvc:> delete network/httpsvc:> exit• Or# svccfg delete apache2# svcs apache2svcs: Pattern apache2 doesnt match any instancesSTATE STIME FMRI Sun Proprietary/Use by Permission Only
  105. 105. SOLARIS 10 DEEP DIVE: SMFModifying manifest• Once again, look around for something to copy from, I find my filesystem/local dependency in inetd-upgrade.xml and ssh.xml, respectively: inetd-upgrade.xml: <dependency name=filesystem ssh.xml: <dependency name=fs-local• Note the name, filesystem and fs-local. These are just common names, they are not FMRI. In fact, these can be anything descriptive.• Add this to my /var/svc/manifest/network/http-apache2.xml• Import your fixed manifest and check. Sun Proprietary/Use by Permission Only
  106. 106. SOLARIS 10 DEEP DIVE: SMFfilesystem/local dependencydependency name=myfs-local grouping=require_all restart_on=none type=service> <service_fmri value=svc:/system/filesystem/local/></dependency> Sun Proprietary/Use by Permission Only
  107. 107. SOLARIS 10 DEEP DIVE: SMFCompare with DTD<!ELEMENT dependency ( service_fmri*, stability?, ( propval | property )* ) ><!ATTLIST dependency name CDATA #REQUIRED grouping ( require_all | require_any | exclude_all | optional_all ) #REQUIRED restart_on ( error | restart | refresh | none ) #REQUIRED type CDATA #REQUIRED delete ( true | false ) "false" > Sun Proprietary/Use by Permission Only
  108. 108. SOLARIS 10 DEEP DIVE: SMFIn depth• name is myfs-local, a descriptive name.• grouping is require_all, self-explanatory, but when is grouping not require_all? When the dependency is a milestone, for example.• restart_on is none. If you depend on a service, and its state changes, you may want to re-examine your own service.• Refer back to slide “reason for dependency stop”• type is service; can be service or file. Sun Proprietary/Use by Permission Only
  109. 109. SOLARIS 10 DEEP DIVE: SMFrestart_on• restart_on none: do not restart, you require this dependency to be satisfied once, on startup, thats all.• restart_on error: restart when this service faults. Why? Opposite of previous case, you depend on this service all the time. If it faults (restart is implicit) and does not come back up, you need to restart and check your dependency. What state would you be in?• restart_on restart: restart when this service has an error stop, or a non-error stop.• restart_on refresh: restart on all the above conditions, and when this service re-read its configuration file. Sun Proprietary/Use by Permission Only
  110. 110. SOLARIS 10 DEEP DIVE: SMFsvccfg import# svccfg import http-apache2.xml# svcs -l apache2fmri svc:/network/http:apache2name Apache 2 HTTP serverenabled falsestate disablednext_state nonestate_time Mon Jun 06 18:26:41 2005restarter svc:/system/svc/restarter:defaultdependency require_all/error svc:/network/loopback:default (online)dependency optional_all/error svc:/network/physical:default (online)dependency require_all/none svc:/system/filesystem/local (online) Sun Proprietary/Use by Permission Only
  111. 111. SOLARIS 10 DEEP DIVE: SMFFix 2: modify properties in repository• Fix by modifying current service properties in the repository.• Now that Ive written the manifest, I know better what to do. Basically, take that block of dependency and some how get it into the running service configuration of apache2.• Disable and delete apache2 from the repository.• Re-import the original, unrepaired manifest; it should only have two dependencies. Sun Proprietary/Use by Permission Only
  112. 112. SOLARIS 10 DEEP DIVE: SMFPut back original manifest# svccfg -s apache2 delete# svccfg import http-apache2.xml# svccfg -s apache2 listpgloopback dependencyphysical dependencystart methodstop methodrefresh methodhttpd applicationstartd frameworkgeneral frameworkrestarter framework NONPERSISTENTrestarter_actions framework NONPERSISTENT Sun Proprietary/Use by Permission Only
  113. 113. SOLARIS 10 DEEP DIVE: SMFWhat Im interested in# svccfgsvc:> helpGeneral commands: help set repository endManifest commands: inventory validate import export archiveProfile commands: apply extractEntity commands: list select unselect add deleteSnapshot commands: listsnap selectsnap revertProperty group commands: listpg addpg delpgProperty commands: listprop setprop delprop editpropProperty value commands: addpropvalue delpropvalue setenv unsetenvsvc:> Sun Proprietary/Use by Permission Only
  114. 114. SOLARIS 10 DEEP DIVE: SMFAdding properties• Add a new dependency to my property groupsvc:/network/http:apache2> addpg myfs-local dependency• In your manifest, this conforms to<dependency name=myfs-local• Add other propertiessvc:/network/http:apache2> setprop myfs- local/grouping = astring: ”require_all”svc:/network/http:apache2> setprop myfs- local/restart_on = astring: ”none”svc:/network/http:apache2> setprop myfs-local/type = astring: ”servicr” Sun Proprietary/Use by Permission Only
  115. 115. SOLARIS 10 DEEP DIVE: SMFOops! A typo.svc:/network/http:apache2> listpropmyfs-local dependencymyfs-local/group astring require_allmyfs-local/restart_on astring nonemyfs-local/type astring servicrsvc:/network/http:apache2> editprop (invokes vi)# Property group "myfs-local"# delprop myfs-local# addpg myfs-local dependency# setprop myfs-local/group = astring: (require_all)# setprop myfs-local/restart_on = astring: (none)# setprop myfs-local/type = astring: (servicr) Sun Proprietary/Use by Permission Only
  116. 116. SOLARIS 10 DEEP DIVE: SMFAdd the FMRI• Change this line tosetprop myfs-local/type = astring: “service”• Upon quitting vi, the temp file is executed. This corresponds togrouping=require_allrestart_on=nonetype=service>• Now add the FMRI of the dependencysvc:/network/http:apache2> setprop myfs-local/ entities = fmri: “svc:/system/filesystem/local” Sun Proprietary/Use by Permission Only
  117. 117. SOLARIS 10 DEEP DIVE: SMFCheck your configurationsvc:/network/http:apache2> listprop...filesystem-local dependencyfilesystem-local/group astring require_allfilesystem-local/restart_on astring nonefilesystem-local/type astring servicefilesystem-local/entities fmri svc:/system/filesystem/local...svc:/network/http:apache2> exit Sun Proprietary/Use by Permission Only
  118. 118. SOLARIS 10 DEEP DIVE: SMFRefresh the service# svcadm refresh apache2# svcs -l apache2fmri svc:/network/http:apache2name Apache 2 HTTP server...restarter svc:/system/svc/restarter:defaultdependency require_all/error svc:/network/loopback:default (online)dependency optional_all/error svc:/network/physical:default (online)dependency require_all/none svc:/system/filesystem/local (online) Sun Proprietary/Use by Permission Only
  119. 119. SOLARIS 10 DEEP DIVE: SMFNew property added# svccfg -s apache2 listpgloopback dependencyphysical dependencystart methodstop methodrefresh methodhttpd applicationstartd frameworkgeneral frameworkrestarter framework NONPERSISTENTrestarter_actions framework NONPERSISTENTmyfs-local dependency Sun Proprietary/Use by Permission Only
  120. 120. SOLARIS 10 DEEP DIVE: SMFThe reason why ...• Youve modified the service properties for apache2 in the repository. Dont be confused the next time you look at http-apache2.xml and wonder what happened. Manifests are read and imported only once (unless they are modified.)• Dont modify manifests in place. Upon boot svc:/system/manifest-import searches /var/svc/manifest for new XML files and ones which have changed and import them.• This allows us to upgrade service properties (via a patch, for example) by dropping new manifest in, instead of manipulating the repository using svccfg. Sun Proprietary/Use by Permission Only
  121. 121. SOLARIS 10 DEEP DIVE: SMFThe reason why ...• When svccfg import is run on a new manifest for services which already exist, it first compares the current values to those in the last-import snapshot.• If values dont differ, it assumes that theres been no customization, and upgrades the properties to whats in the manifest.• Otherwise it issues warnings to the user, but leaves the customizations in place.• So any patch which alters a manifest will cause manifest- import to try to upgrade the corresponding service, no post-install action necessary. Sun Proprietary/Use by Permission Only
  122. 122. SOLARIS 10 DEEP DIVE: SMFThe reason why ...• If you had modified that manifest in place and re-imported it, or even if you copied it somewhere else, modified it, and re-imported it, svccfg will think that the modified version is the original, official version (via the last-import snapshot comparison), will will override all your customization after the patch.• The moral of the story: leave those manifests alone. Make changes against the repository with svccfg.• man smf_bootstrap(5) for the gory details. Sun Proprietary/Use by Permission Only
  123. 123. SOLARIS 10 DEEP DIVE: SMFPart IV:More Advanced SMF Sun Proprietary/Use by Permission Only
  124. 124. SOLARIS 10 DEEP DIVE: SMFSMF security• SMF uses RBAC to delegate service management responsibility. Well talk about managing specific services first.• SMF provides these propval to be put into individual manifest to authorize specific services: > action_authorization – can execute method > value_authorization – can change value of existing properties > modify_authorization – change, add, delete properties• These are fixed keywords.• Lets take a look smtp-sendmail.xml and ssh.xml as examples. Sun Proprietary/Use by Permission Only
  125. 125. SOLARIS 10 DEEP DIVE: SMFaction_authorizationsmtp-sendmail.xml:<property_group name=general type=framework> <propval name=action_authorization type=astring value=solaris.smf.manage.sendmail /></property_group>ssh.xml:<property_group name=general type=framework><!-- to start stop sshd --> <propval name=action_authorization type=astring value=solaris.smf.manage.ssh /></property_group> Sun Proprietary/Use by Permission Only
  126. 126. SOLARIS 10 DEEP DIVE: SMFprof_auth• Where will you find this? /etc/security/prof_authMail Management:::Manage sendmail & queues:auths=solaris.smf.manage.sendmail;help=RtMail Mngmnt.htmlNetwork Security:::Manage network and host security:auths=solaris.smf.manage.ssh,solaris.network.*; help=RtNetSecure.html• These RBAC roles have this privilege, which privilege is this? The one defined by the keyword action_authorization, which allows you to execute methods. Sun Proprietary/Use by Permission Only
  127. 127. SOLARIS 10 DEEP DIVE: SMFDelegating to a user• But you havent set up roles properly, you want to delegate service administration to a user.# echo christine::::auths=solaris.smf.manage.ssh >> /etc/user_attr• You have to re-login for authorization to take effect.# su – christine$ iduid=85228(christine) gid=10(staff) Sun Proprietary/Use by Permission Only
  128. 128. SOLARIS 10 DEEP DIVE: SMFDelegating to a user$ svcs sshSTATE STIME FMRIonline Oct_19 svc:/network/ssh:default$ dateFri Oct 21 15:47:40 EDT 2005$ svcadm restart ssh$ tail /var/svc/log/network-ssh:default.log[ Oct 21 15:47:42 Stopping because service restarting. ][ Oct 21 15:47:42 Executing stop method (:kill) ][ Oct 21 15:47:42 Executing start method ("/lib/svc/method/sshd start") ][ Oct 21 15:47:42 Method "start" exited with status 0 ] Sun Proprietary/Use by Permission Only
  129. 129. SOLARIS 10 DEEP DIVE: SMFA little problem$ svcadm disable sshsvcadm: svc:/network/ssh:default: Permission denied.$ svcadm disable -t ssh$ svcs sshSTATE STIME FMRIdisabled 15:55:43 svc:/network/ssh:default$ svcprop ssh |grep general_ovrgeneral_ovr/enabled boolean false• Service can only be temporarily disabled, why? See BugID 6303496. Sun Proprietary/Use by Permission Only
  130. 130. SOLARIS 10 DEEP DIVE: SMFNeeds value_authorization• Bug ID 6303496 > Synopsis cron cannot be smf managed with solaris.smf.manage.cron• It turns out that you also need value_authorization, the keyword to change existing property values.• Why? Because youre actually changing the value of some property. Which property?• And exactly where is that general_ovr property listed?• general and general_ovr appear to be property groups of type framework. Check the DTD.• Well talk more about instances later. Sun Proprietary/Use by Permission Only
  131. 131. SOLARIS 10 DEEP DIVE: SMFLooking for general_ovr# svccfg -s sshsvc:/network/ssh> listprop general*general frameworkgeneral/action_authorization astring solaris.smf.manage.sshgeneral/entity_stability astring Unstablegeneral/single_instance boolean true# svccfg -s ssh:defaultsvc:/network/ssh:default> listprop general*general frameworkgeneral/enabled boolean truegeneral_ovr framework NONPERSISTENTgeneral_ovr/enabled boolean false Sun Proprietary/Use by Permission Only
  132. 132. SOLARIS 10 DEEP DIVE: SMFvalue_authorization# svccfg -s sshsvc:/network/ssh> listpropgeneral/action_authorization astring solaris.smf.manage.ssh• Do you need to addpg?svc:/network/ssh> setprop general/value_authorization = astring: "fooblat"svc:/network/ssh> exit• You still need to make one change for this to work. Sun Proprietary/Use by Permission Only
  133. 133. SOLARIS 10 DEEP DIVE: SMFvalue_authorization# tail /etc/user_attrchristine::::auths=solaris.smf.manage.ssh,fooblat# svcadm refresh ssh# su - christine$ svcadm disable ssh$ svcs sshSTATE STIME FMRIdisabled 16:42:47 svc:/network/ssh:default$ svcprop ssh|grep general_ovr$• Yes, its a permanent change. Sun Proprietary/Use by Permission Only
  134. 134. SOLARIS 10 DEEP DIVE: SMFValue can be anything• The value of the auth used has no specific meaning except that it must exist in /etc/user_attr.• In the manifest<propval name=value_authorization type=astring value=fooblat />• ... is equal to<propval name=value_authorization type=astringvalue=solaris.smf.manage.ssh />• ... but the later is more descriptive. In fact:# auths christinesolaris.smf.manage.ssh,fooblat,solaris.device.cdrw,s Sun Proprietary/Use by Permission Only
  135. 135. SOLARIS 10 DEEP DIVE: SMFSMF Security• If you look at /etc/security/prof_attr, you will seeService Management:::Manage services:auths=solaris.smf.manage, solaris.smf.modify• These are two main authorizations in SMF: > solaris.smf.modify – add, delete, modify services > solaris.smf.manage – restart, refresh, or other state modification• They dont appear in any manifest because they apply all services.• There are others: method, dependency, application, and framework. man smf_security(5) for the gory details. Sun Proprietary/Use by Permission Only
  136. 136. SOLARIS 10 DEEP DIVE: SMFBlueprint Restricting Service Administration in the Solaris 10 Operating System by Glenn Brunette (June 2005) This Sun BluePrints Cookbook describes how to use the Solaris 10 Service Management Facility (SMF) to require specific authorizations for certain types of operations. Using this capability, it is possible to delegate access to core service management functions based on the concept of least privilege--if a user or service does not strictly need to have some degree of privilege, then that privilege should not be granted. SMF allows organizations to have much finer grained access control policies than was possible before the Solaris 10 Operating System. http://www.sun.com/blueprints/0605819-2887.pdf Sun Proprietary/Use by Permission Only
  137. 137. SOLARIS 10 DEEP DIVE: SMFSnapshots# svccfg -s sshsvc:/network/ssh> listsnapInstance not selected.svc:/network/ssh> select defaultsvc:/network/ssh:default> listsnaplast-importinitialrunning• Where are snapshot start and previous?# svcs sshSTATE STIME FMRIdisabled Oct_21 svc:/network/ssh:default Sun Proprietary/Use by Permission Only
  138. 138. SOLARIS 10 DEEP DIVE: SMFWhats the difference?# svccfg -s ssh listprop *general*general frameworkgeneral/entity_stability astring Unstablegeneral/single_instance boolean truegeneral/action_authorization astring solaris.smf.manage.sshgeneral/value_authorization astring yadayada# svccfg -s ssh:default listprop *general*general frameworkgeneral/enabled boolean falsegeneral_ovr framework NONPERSISTENT Sun Proprietary/Use by Permission Only
  139. 139. SOLARIS 10 DEEP DIVE: SMFinstance• Instances are units of administrations.• Services are mechanism for sharing properties.• Instance can have properties, and its parent service can have properties.• Instances have snapshots, services do not.• If unnamed, that instance is called “default”.• Snapshots are composite view of the instance and service properties at one point in time. Sun Proprietary/Use by Permission Only
  140. 140. SOLARIS 10 DEEP DIVE: SMFrevertsvc:/network/ssh:default> listsnaplast-import initialrunning startsvc:/network/ssh:default> selectsnap initial[initial]svc:/network/ssh:default> revertsvc:/network/ssh:default> unselectsvc:/network/ssh> listprop general*general frameworkgeneral/action_authorization astring solaris.smf.manage.sshgeneral/entity_stability astring Unstablegeneral/single_instance boolean true• value_authorization has disappeared. Sun Proprietary/Use by Permission Only
  141. 141. SOLARIS 10 DEEP DIVE: SMFSnapshots• Snapshots are not good for rolling back a single change because > No control over when snapshots are taken > Alters the service properties as well• To undo a property change, its simpler to use svccfg setprop commands• Can use svcprop or svccfg export to take textual snapshots.• However, when you use svccfg export, you use the service name, even though the output file contains the description of the instance(s). Sun Proprietary/Use by Permission Only
  142. 142. SOLARIS 10 DEEP DIVE: SMFignore_error property• From http-apache2.xml:<property_group name=startd type=framework> <!-- sub-process core dumps shouldnt restart session --> <propval name=ignore_error type=astring value=core,signal /></property_group>• SMF is instructed to ignore signals sent to a service or core dumps by a service; otherwise SMF will try to restart the service.• man svc.startd(1M) for the gory details; description at the startd/ignore_error paragraph. Sun Proprietary/Use by Permission Only
  143. 143. SOLARIS 10 DEEP DIVE: SMFSignaling httpd# ps -ef|grep httpwebservd 29855 29853 0 11:46:46 ? 0:00 /usr/apache2/bin/httpd -k startroot 29853 27175 0 11:46:45 ? 0:00 /usr/apache2/bin/httpd -k startwebservd 29857 29853 0 11:46:46 ? 0:00 /usr/apache2/bin/httpd -k startwebservd 29856 29853 0 11:46:46 ? 0:00 /usr/apache2/bin/httpd -k startwebservd 29858 29853 0 11:46:46 ? 0:00 /usr/apache2/bin/httpd -k startwebservd 29854 29853 0 11:46:46 ? 0:00 /usr/apache2/bin/httpd -k start# kill -HUP 29853 <-- started by root Sun Proprietary/Use by Permission Only
  144. 144. SOLARIS 10 DEEP DIVE: SMFLogs# tail -f /var/apache2/logs/error_log(nothing ...)# tail -f /var/svc/log/network-http:apache2.log[ Oct 26 11:46:44 Executing start method ("/lib/svc/method/http-apache2 start") ][ Oct 26 11:46:45 Method "start" exited with status 0 ](nothing ... )• What would happen, by the way, if you kill just httpd processes started by webservd? Sun Proprietary/Use by Permission Only
  145. 145. SOLARIS 10 DEEP DIVE: SMFKeep on killing httpd# ps -ef|grep httpwebservd 29854 27175 0 11:46:46 ? 0:00 /usr/apache2/bin/httpd -k start# kill -9 29854 <-- last one# tail -f /var/apache2/logs/error_log[Wed Oct 26 12:46:53 2005] [notice] Digest: generating secret for digest authentication ...[Wed Oct 26 12:46:53 2005] [notice] Digest: done[Wed Oct 26 12:46:54 2005] [notice] Apache/2.0.52 (Unix) DAV/2 configured -- resuming normal operations Sun Proprietary/Use by Permission Only
  146. 146. SOLARIS 10 DEEP DIVE: SMFhttpd restarted by SMF# tail -f /var/svc/log/network-http:apache2.logStopping because all processes in service exited. ][ Oct 26 11:55:05 Executing stop method ("/lib/svc/method/http-apache2 stop") ]httpd (pid 29853?) not running[ Oct 26 11:55:05 Method "stop" exited with status 0 ][ Oct 26 11:55:05 Executing start method ("/lib/svc/method/http-apache2 start") ][ Oct 26 11:55:06 Method "start" exited with status 0 ]# svcs apache2STATE STIME FMRIonline 12:46:53 svc:/network/http:apache2 Sun Proprietary/Use by Permission Only
  147. 147. SOLARIS 10 DEEP DIVE: SMFRemoving ignore_error property# svccfg -s apache2svc:/network/http:apache2> listprop startd*startd frameworkstartd/ignore_error astring core,signalsvc:/network/http:apache2> delprop startd/ignore_errorsvc:/network/http:apache2> listprop startd*startd frameworksvc:/network/http:apache2> exit# svcadm refresh apache2 Sun Proprietary/Use by Permission Only
  148. 148. SOLARIS 10 DEEP DIVE: SMFLog of httpd refreshing# tail -f /var/apache2/logs/error_log[Wed Oct 26 12:55:30 2005] [notice] Graceful restart requested, doing restart[Wed Oct 26 12:55:30 2005] [warn] (128)Network is unreachable: connect to listener[Wed Oct 26 12:55:30 2005] [notice] Digest: generating secret for digest authentication ...[Wed Oct 26 12:55:30 2005] [notice] Digest: done[Wed Oct 26 12:55:31 2005] [notice] Apache/2.0.52 (Unix) DAV/2 configured -- resuming normal operations# tail -f /var/svc/log/network-http:apache2.log[ Oct 26 12:55:30 Rereading configuration. ][ Oct 26 12:55:30 Executing refresh method ("/lib/svc/method/http-apache2 refresh") ][ Oct 26 12:55:30 Method "refresh" exited with status 0 ] Sun Proprietary/Use by Permission Only
  149. 149. SOLARIS 10 DEEP DIVE: SMFhttpd without ignore_error# kill -9 29999 <-- started by root# tail -f /var/svc/log/network-http:apache2.log[ Oct 26 13:13:02 Stopping because process received fatal signal from outside the service. ][ Oct 26 13:13:02 Executing stop method ("/lib/svc/method/http-apache2 stop") ]httpd (pid 29999?) not running[ Oct 26 13:13:02 Method "stop" exited with status 0 ][ Oct 26 13:14:02 Method or service exit timed out. Killing contract 720 ]# svcs apache2STATE STIME FMRImaintenance 13:14:02 svc:/network/http:apache2 Sun Proprietary/Use by Permission Only
  150. 150. SOLARIS 10 DEEP DIVE: SMFClearing and restarting# svcadm clear apache2# tail -f /var/svc/log/network-http:apache2.log[ Oct 26 13:16:04 Leaving maintenance because clear requested. ][ Oct 26 13:16:04 Enabled. ][ Oct 26 13:16:04 Executing start method ("/lib/svc/method/http-apache2 start") ][ Oct 26 13:16:05 Method "start" exited with status 0 ]• What would happen if you kill any httpd processes started by webservd, now that you no longer have the ignore_error property? Sun Proprietary/Use by Permission Only
  151. 151. SOLARIS 10 DEEP DIVE: SMFLogs# tail -f /var/apache2/logs/error_log[Wed Oct 26 13:24:23 2005] [notice] caught SIGTERM, shutting down[Wed Oct 26 13:24:25 2005] [notice] Digest: generating secret for digest authentication ...[Wed Oct 26 13:24:25 2005] [notice] Digest: done[Wed Oct 26 13:24:26 2005] [notice] Apache/2.0.52 (Unix) DAV/2 configured -- resuming normal operations• SMF stops and restarts httpd:apache because svc.startd will interpret any externally-generated signal as a failure of the service, and will execute the stop and start method to try and recover. Sun Proprietary/Use by Permission Only
  152. 152. SOLARIS 10 DEEP DIVE: SMFLogs# tail -f /var/svc/log/network-http:apache2.log[ Oct 26 13:24:23 Stopping because process received fatal signal from outside the service. ][ Oct 26 13:24:23 Executing stop method ("/lib/svc/method/http-apache2 stop") ][ Oct 26 13:24:23 Method "stop" exited with status 0 ][ Oct 26 13:24:24 Executing start method ("/lib/svc/method/http-apache2 start") ][ Oct 26 13:24:25 Method "start" exited with status 0 ]• Whats different though? Why isnt it in maintenance mode? Sun Proprietary/Use by Permission Only
  153. 153. SOLARIS 10 DEEP DIVE: SMFPortable SMF• Perhaps you have a large number of servers, or even zones, on which youd like to imprint an identical set of services.• Well, this is not easy. This is “provisioning”, and SMF is a service management tool.• You cant just “pack up” the database, or the XML dump generated by svccfg archive, or all the manifests ... Why not?• What about the methods, the config files, the binaries, everything else that makes up the apps? Sun Proprietary/Use by Permission Only
  154. 154. SOLARIS 10 DEEP DIVE: SMFDuplicating service properties• If, however, you have an application already installed, and you would like to transfer service properties (such as dependencies, the exec method, anything that might be defined by the manifest), you can do that by exporting and then importing a manifest.# svccfg export svc:/network/http > apache2.export• Move apache2.export to another system.# svccfg import apache2.export• Note: the argument to svccfg export is a service, not an instance, although the XML file will contain definition of the instance(s). Sun Proprietary/Use by Permission Only
  155. 155. SOLARIS 10 DEEP DIVE: SMFDuplicating service states• You can get a profile, or an XML file that describes the state of all services by using svccfg extract.• This XML file can be used to duplicate the on|off state of services from system to system. Use svccfg apply. Sun Proprietary/Use by Permission Only
  156. 156. SOLARIS 10 DEEP DIVE: SMFProfiles• A profile is an XML description of all service instance states on a system.• Profiles are in /var/svc/profile.• Profiles are applied in order: generic, platform, site• Never modify existing profiles, use site.xml for local customization.• An example of something that may create a use a profile is the Solaris Security Toolkit 4.2 (aka JASS). Sun Proprietary/Use by Permission Only
  157. 157. SOLARIS 10 DEEP DIVE: SMFPart V:SMF Tools and Community Sun Proprietary/Use by Permission Only
  158. 158. SOLARIS 10 DEEP DIVE: SMFWebmin Sun Proprietary/Use by Permission Only
  159. 159. SOLARIS 10 DEEP DIVE: SMFAbout Webmin• A web-based interface for UNIX system administration• It comes with Solaris 10, or get it at http://www.webmin.com• Version 1.190 an later supports SMF, even has a feature to create a new manifest. Sun Proprietary/Use by Permission Only
  160. 160. SOLARIS 10 DEEP DIVE: SMFSMF in Webmin Sun Proprietary/Use by Permission Only
  161. 161. SOLARIS 10 DEEP DIVE: SMFCreate new service• This button will walk you through creating a new service manifest.• You dont need to know XML, but you do need to understand what goes into a manifest.• Will ask you a series of questions such as: > Name of service > What are the dependencies, > Which is the delegated restarter > Service properties > Man pages Sun Proprietary/Use by Permission Only
  162. 162. SOLARIS 10 DEEP DIVE: SMFGUI Tools• With Webmin you can monitor and manage SMF services through a browser.• A GUI interface to SMF is being developed.• Watch for a SunMC SMF module due near the end of this year. Sun Proprietary/Use by Permission Only
  163. 163. SOLARIS 10 DEEP DIVE: SMFMore tools• Use xmllint to check for errors in the manifest file without importing the manifest.• svccfg validate does something similar, without changing the repository.• Jedit is a free text editor with XML with syntax checker and validator. Comes with Solaris 10 (/usr/demo/jds/bin/jedit) or get it from http://www.jedit.org.• Morphon Technologies has a good and free XML editor, but sadly no longer in development, get it from http://www.morphon.com. Sun Proprietary/Use by Permission Only
  164. 164. SOLARIS 10 DEEP DIVE: SMFSFM community on OpenSolaris• http://www.opensolaris.org/os/community/smf/• Submit manifests or methods youve written here: http://www.opensolaris.org/os/community/smf/manifests/• greenline-interest@sun.com migrating to smf-discuss@opensolaris.org• Subscribe here: http://www.opensolaris.org/os/community/smf/discussions• Read it here: http://www.opensolaris.org/jive/forum.jspa?forumID=24• File bugs here:• http://bugs.opensolaris.org/ Sun Proprietary/Use by Permission Only
  165. 165. SOLARIS 10 DEEP DIVE: SMFResources• Internal > Greenline project page: http://greenline.eng > greenline-interest alias.• External > BigAdmin: http://www.sun.com/bigadmin > Especially the Service Developer Intro at http://www.sun.com/bigadmin/content/selfheal/sdev_intro.html > SMF community: http://opensolaris.org/os/community/smf/ > SMF mailing list: smf-discuss@opensolaris.org• Blogs > Liane Prazas blog: http://blogs.sun.com/roller/page/lianep > Stephen Hahns blog: http://blogs.sun.com/sch Sun Proprietary/Use by Permission Only
  166. 166. SOLARIS 10 DEEP DIVE: SMFReview••• Sun Proprietary/Use by Permission Only
  167. 167. SOLARIS 10 DEEP DIVE: SMFQuestions?••• Sun Proprietary/Use by Permission Only
  168. 168. SOLARIS 10 DEEP DIVE: SMFAnd lastly ... Thank you! Sun Proprietary/Use by Permission Only
  169. 169. SOLARIS 10 DEEP DIVE: SMFReferences• Presentations L. Praza. smf(5): Service Management Facility. R. Manus, G. Hiregoudar. Solaris Management Framework – Workshop. Sun Proprietary/Use by Permission Only
  170. 170. SOLARIS 10 DEEP DIVE: SMFChanges• Last modified: 03/10/2005 Sun Proprietary/Use by Permission Only
  171. 171. SOLARIS 10 DEEP DIVE: SMFSOLARIS 10DEEP DIVE: SMFchristine.tran@sun.comUS Solaris Adoption PracticeSun Microsystems Sun Proprietary/Use by Permission Only

×