The init Phase
The init phase has undergone major changes in Solaris 10. Even if you are experienced on previous versions of Solaris OE, this section introduces the svc.startd daemon and the Service Management Facility (SMF), which are new in Solaris 10 and will be tested heavily on the exam.
After control of the system is passed to the kernel, the system begins the last stage of the boot processthe init stage. In this phase of the boot process, the init daemon (/sbin/init) reads the /etc/default/init file to set any environment variables for the shell that init invokes. By default, the CMASK and variables are set. These values get passed to any processes that init starts. Then, init reads the /etc/inittab file and executes any process entries that have sysinit in the action field so that any special initializations can take place before users log in.
After reading the /etc/inittab file, init starts the svc.startd daemon, which is responsible for starting and stopping other system services such as mounting file systems and configuring network devices. In addition, svc.startd will execute legacy run control (rc) scripts, which are described later in this section.
The /sbin/init command sets up the system based on the directions in /etc/inittab. Each entry in the /etc/inittab file has the following fields:
Table 3.23 provides a description of each field.
Valid action keywords are listed in Table 3.24:
The following example shows a default /etc/inittab file:
ap::sysinit:/sbin/autopush -f /etc/iu.ap sp::sysinit:/sbin/soconfig -f /etc/sock2path smf::sysinit:/lib/svc/bin/svc.startd >/dev/msglog 2<>/dev/msglog </dev/console p3:s1234:powerfail:/usr/sbin/shutdown -y -i5 -g0 >/dev/msglog 2<>/dev/msglog
The init process performs the following tasks based on the entries found in the default /etc/inittab file:
The Solaris Management Facility (SMF) Service
Explain the Service Management Facility and the phases of the boot process.
In Solaris 10, the svc.startd daemon replaces the init process as the master process starter and restarter. Where in previous versions of Solaris, init would start all processes and bring the system to the appropriate "run level" or "init state." Now SMF, or more specifically, the svc.startd daemon, assumes the role of starting system services.
SMF Services A service can be described as an entity that provides a resource or list of capabilities to applications and other services. This entity can be running locally or remote, but at this phase of the boot process, the service is running locally. A service does not have to be a process; it can be the software state of a device or a mounted file system. Also, a system can have more than one instance of a service, such as with multiple network interfaces, multiple mounted file systems, or a set of other services.
The advantages of using SMF to manage system services over the traditional Unix startup scripts that, in the past, were run by the init process are
Those of you who have experience on previous versions of Solaris will notice a few differences immediately:
The service instance is the fundamental unit of administration in the SMF framework, and each SMF service has the potential to have multiple versions of it configured. A service instance is either enabled or disabled with the svcadm command described later in this chapter. An instance is a specific configuration of a service, and multiple instances of the same service can run in the same Solaris instance. For example, a web server is a service. A specific web server daemon that is configured to listen on port 80 is an instance. Another instance of the web server service could have different configuration requirements listening on port 8080. The service has system-wide configuration requirements, but each instance can override specific requirements, as needed.
Services are represented in the SMF framework as service instance objects, which are children of service objects. These instance objects can inherit or override the configuration settings of the parent service object. Multiple instances of a single service are managed as child objects of the service object.
Services are not just the representation for standard long-running system services such as httpd or nfsd. Services also represent varied system entities that include third-party applications such as Oracle software. In addition, a service can include less traditional entities such as the following:
The services started by svc.startd are referred to as milestones. The milestone concept replaces the traditional run levels that were used in previous versions of Solaris. A milestone is a special type of service that represents a group of services. A milestone is made up of several SMF services. For example, the services that instituted run levels S, 2, and 3 in previous version of Solaris are now represented by milestone services named:
milestone/single-user (equivalent to run level S)
milestone/multi-user (equivalent to run level 2)
milestone/multi-user-server (equivalent to run level 3)
Other milestones that are available in the Solaris 10 OE are
milestone/name-services milestone/devices milestone/network milestone/sysconfig
An SMF manifest is an XML (Extensible Markup Language) file that contains a complete set of properties that are associated with a service or a service instance. The properties are stored in files and subdirectories located in /var/svc/manifest. Manifests should not be edited directly to modify the properties of a service. The service configuration repository is the authoritative source of the service configuration information, and the service configuration repository can only be manipulated or queried using SMF interfaces, which are command-line utilities described later in this section.
Each service instance is named with a Fault Management Resource Identifier or FMRI. The FMRI includes the service name and the instance name. For example, the FMRI for the ftp service is svc:/network/ftp:default, where network/ftp identifies the service and default identifies the service instance.
You may see various forms of the FMRI that all refer to the same service instance, as follows:
svc://localhost/network/inetd:default svc:/network/inetd:default network/inetd:default
An FMRI for a legacy service will have the following format:
where the lrc (legacy run control) prefix indicates that the service is not managed by SMF. The pathname /etc/rc3_d refers to the directory where the legacy script is located, and S90samba is the name of the run control script. See the section titled "Using the Run Control Scripts to Stop or Start Services" later in this chapter for information on run control scripts.
In earlier versions of Solaris, processes were started at bootup by their respective shell scripts, which ran in a pre-determined sequence. Sometimes, one of these shell scripts failed for various reasons. Perhaps it was an error in the script or one of the daemons did not start for various reasons. When a script failed, the other scripts were started regardless, and sometimes these scripts failed because a previous process failed to start. Tracking the problem down was difficult for the system administrator.
To remedy the problem with sequencing scripts, Sun uses the SMF to manage the starting and stopping of services. The SMF understands the dependencies that some services have on other services. With SMF, if a service managed by the SMF fails or is terminated, all dependent processes will be taken offline until the required process is restarted. The interdependency is started by means of a service contract, which is maintained by the kernel and is where the process interdependency, the restarter process, and the startup methods are all described.
Most service instances have dependencies on other services or files. Those dependencies control when the service is started and automatically stopped. When the dependencies of an enabled service are not satisfied, the service is kept in the offline state. When the service instance dependencies are satisfied, the service is started or restarted by the svc.startd daemon. If the start is successful, the service is transitioned to the online state. There are four types of service instance dependencies listed below.
Each service or service instance must define a set of methods that start, stop, and optionally refresh the service. These methods can be listed and modified for each service using the svccfg command described later in this chapter.
A service instance is satisfied and started when its criteria, for the type of dependency, are met. Dependencies are satisfied when cited services move to the online state. Once running (online or degraded), if a service instance with a require_all, require_any, or optional_all dependency is stopped or refreshed, the SMF considers why the service was stopped and uses the restart_on attribute of the dependency to decide whether to stop the service. restart_on attributes are defined in Table 3.25
A service is considered to have stopped due to an error if the service has encountered a hardware error or a software error such as a core dump. For exclude_all dependencies, the service is stopped if the cited service is started and the restart_on attribute is not none.
You can use the svcs command, described later in this chapter, to view service instance dependencies and to troubleshoot failures. You'll also see how to use the svccfg command to modify service dependencies.
SMF Command-line Administration Utilities
The SMF provides a set of command-line utilities used to administer and configure the SMF. Table 3.26 describes these utilities.
To report the status of all enabled service instances and get a list of the various services that are running, use the svcs command with no options as follows:
svcs | more
The svcs command obtains information about all service instances from the service configuration repository and displays the state, start time, and FMRI of each service instance as follows:
STATE STIME FMRI legacy_run 14:10:49 lrc:/etc/rc2_d/S10lu legacy_run 14:10:49 lrc:/etc/rc2_d/S20sysetup legacy_run 14:10:50 lrc:/etc/rc2_d/S40llc2 legacy_run 14:10:50 lrc:/etc/rc2_d/S42ncakmod legacy_run 14:10:50 lrc:/etc/rc2_d/S47pppd legacy_run 14:10:50 lrc:/etc/rc2_d/S70uucp Output has been truncated . . . . online 14:09:37 svc:/system/svc/restarter:default online 14:09:48 svc:/network/pfil:default online 14:09:48 svc:/network/loopback:default online 14:09:48 svc:/milestone/name-services:default online 14:09:50 svc:/system/filesystem/root:default online 14:09:54 svc:/system/filesystem/usr:default online 14:09:56 svc:/system/device/local:default online 14:09:57 svc:/milestone/devices:default online 14:09:57 svc:/network/physical:default online 14:09:58 svc:/milestone/network:default
Listing Legacy Services You'll notice that the list includes legacy scripts that were used to start up processes. Legacy services can be viewed, but cannot be administered with SMF.
Running the svcs command without options will display the status of all enabled services. Use the -a option to list all services, including disabled services as follows:
The result is a listing of all services as follows:
. .. . <output has been truncated> disabled 15:48:41 svc:/network/shell:kshell disabled 15:48:41 svc:/network/talk:default disabled 15:48:42 svc:/network/rpc/ocfserv:default disabled 15:48:42 svc:/network/uucp:default disabled 15:48:42 svc:/network/security/krb5_prop:default disabled 15:48:42 svc:/network/apocd/udp:default online 15:47:44 svc:/system/svc/restarter:default online 15:47:47 svc:/network/pfil:default online 15:47:48 svc:/network/loopback:default online 15:47:50 svc:/system/filesystem/root:default . .. . <output has been truncated>
To display information on selected services, you can supply the FMRI as an argument to the svcs command as follows:
svcs -l network
With the -l option, the system displays detailed information about the network service instance. The network FMRI specified in the previous example is a general functional category and is also called the network milestone. The information displayed by the previous command is as follows:
fmri svc:/milestone/network:default name Network milestone enabled true state online next_state none state_time Wed Jul 27 14:09:58 2005 alt_logfile /etc/svc/volatile/milestone-network:default.log restarter svc:/system/svc/restarter:default dependency require_all/none svc:/network/loopback (online) dependency require_all/none svc:/network/physical (online)
Use the -d option to view which services are started at the network:default milestone, as follows:
svcs -d milestone/network:default
The system displays
STATE STIME FMRI online Jul_27 svc:/network/loopback:default online Jul_27 svc:/network/physical:default
Another milestone is the multi-user milestone, which is displayed as follows:
svcs -d milestone/multi-user
The system displays all of the services started at the multi-user milestone:
STATE STIME FMRI online Jul_27 svc:/milestone/name-services:default online Jul_27 svc:/milestone/single-user:default online Jul_27 svc:/system/filesystem/local:default online Jul_27 svc:/network/rpc/bind:default online Jul_27 svc:/milestone/sysconfig:default online Jul_27 svc:/network/inetd:default online Jul_27 svc:/system/utmp:default online Jul_27 svc:/network/nfs/client:default online Jul_27 svc:/system/system-log:default online Jul_27 svc:/network/smtp:sendmail
Many of these services have their own dependencies, services that must be started before they get started. We refer to these as sub-dependencies. For example, one of the services listed is the svc:/network/inetd:default service. A listing of the sub-dependencies for this service can be obtained by typing
svcs -d network/inetd
STATE STIME FMRI disabled 15:47:57 svc:/network/inetd-upgrade:default online 15:47:48 svc:/network/loopback:default online 15:48:01 svc:/milestone/network:default online 15:48:30 svc:/milestone/name-services:default online 15:48:33 svc:/system/filesystem/local:default online 15:48:34 svc:/network/rpc/bind:default online 15:48:36 svc:/milestone/sysconfig:default
The -d option, in the previous example, lists the services or service instances upon which the multi-user service instance is dependent. These are the services that must be running before the multi-user milestone is reached. The -D option shows which other services depend on the milestone/multi-user service as follows:
svcs -D milestone/multi-user
The system displays the following output indicating that the dhcp-server and multi-user-server services are dependent on the multi-user service:
STATE STIME FMRI online Jul_27 svc:/network/dhcp-server:default online Jul_27 svc:/milestone/multi-user-server:default
To view processes associated with a service instance, use the -p option as follows:
svcs -p svc:/network/inetd:default
The system displays processes associated with the svc:/network/inetd:default service. In this case, information about the inetd process is shown as follows:
STATE STIME FMRI online Jul_27 svc:/network/inetd:default Jul_27 231 inetd
Viewing processes using svcs -p instead of the traditional ps command makes it easier to track all of the processes associated with a particular service.
If a service fails for some reason and cannot be restarted, you can list the service using the -x option as follows:
The system will display:
svc:/application/print/server:default (LP print server) State: disabled since Thu Sep 22 18:55:14 2005 Reason: Disabled by an administrator. See: http://sun.com/msg/SMF-8000-05 See: lpsched(1M) Impact: 2 dependent services are not running. (Use -v for list.)
Starting and Stopping Services Using SMF
To disable services in previous versions of Solaris, the system administrator had to search out and rename the relevant RC script(s) or comment out statements in a configuration file such as modifying the inetd.conf file when disabling ftp.
SMF makes it much easier to locate services and their dependencies. To start a particular service using SMF, the service instance must be enabled using the svcadm enable command. By enabling a service, the status change is recorded in the service configuration repository. The enabled state will persist across reboots as long as the service dependencies are met. The following example demonstrates how to use the svcadm command to enable the ftp server:
svcadm enable network/ftp:default
To disable the ftp service, use the disable option as follows:
svcadm disable network/ftp:default
To verify the status of the service, type
The system displays the following:
STATE STIME FMRI online 16:07:08 svc:/network/ftp:default
The svcadm command allows the following subcommands:
The svcadm command can also be used to change milestones. In the following step by step, I'll use the svcadm command to determine my current system state (milestone) and then change the system default milestone to single-user.
At bootup, svc.startd retrieves the information in the service configuration repository and starts services when their dependencies are met. The daemon is also responsible for restarting services that have failed and for shutting down services whose dependencies are no longer satisfied.
svcs -x telnet
The results show that the service is not running:
svc:/network/telnet:default (Telnet server) State: disabled since Fri Sep 23 10:06:46 2005 Reason: Temporarily disabled by an administrator. See: http://sun.com/msg/SMF-8000-1S See: in.telnetd(1M) See: telnetd(1M) Impact: This service is not running.
I'll enable the service using the svcadm command as follows:
svcadm enable svc:/network/telnet:default
After enabling the service, check the status using the svcs command as follows:
# svcs -x telnet
The system responds with:
svc:/network/telnet:default (Telnet server) State: online since Fri Sep 23 10:20:47 2005 See: in.telnetd(1M) See: telnetd(1M) Impact: None.
Also, if a service that has been running but stops, try restarting the service using the svcadm restart command as follows:
svcadm restart svc:/network/telnet:default
Starting Services During Boot
Under SMF, the boot process is much quieter than previous versions of Solaris. This was done to reduce the amount of uninformative "chatter" that might obscure any real problems that might occur during boot.
Some new boot options have been added to control the verbosity of boot. One that you may find particularly useful is -m verbose, which prints a line of information when each service attempts to start up. This is similar to previous versions of Solaris where the boot messages were more verbose.
You can also boot the system using one of the milestones as follows:
boot -m milestone=single-user
The system will boot into single-user mode where only the basic services are started as shown when the svcs command is used to display services.
STATE STIME FMRI disabled 17:10:27 svc:/system/filesystem/local:default disabled 17:10:27 svc:/system/identity:domain disabled 17:10:27 svc:/system/sysidtool:net disabled 17:10:28 svc:/system/cryptosvc:default disabled 17:10:28 svc:/network/initial:default disabled 17:10:28 svc:/network/rpc/bind:default disabled 17:10:28 svc:/system/sysidtool:system disabled 17:10:28 svc:/milestone/sysconfig:default Output has been truncated . . . . .
This method of booting is slightly different than using the boot -s command. When the system is explicitly booted to a milestone, exiting the console administrative shell will not transition the system to multi-user mode, as boot -s does. To move to multi-user mode after boot -m milestone=single-user, use the following command:
svcadm milestone milestone/multi-user-server:default
The milestones that can be specified at boot time are
If you boot a system using one of the milestones and you do not include the -s option, the system will stay in the milestone state that you booted the system in. The system will not go into multi-user state automatically when you press Ctrl+D. You can however, get into the multi-user state by using the following command and all services will be restored:
svcadm milestone all
To boot the system without any milestones, type
boot -m milestone=none
The boot command instructs the svc.startd daemon to temporarily disable all services except for the master restarter named svc:/system/svc/restarter:default and start sulogin on the console. The "none" milestone can be very useful in troubleshooting systems that have failures early in the boot process.
To bring the system back down to single-user mode from multi-user mode, type
svcadm milestone milestone/single-user
The -d option can be used with the previous example to cause svcadm to make the given milestone the default boot milestone, which persists across reboots. This would be the equivalent of setting the default run level in the /etc/inittab file on previous versions of Solaris.
Other options that can be used with svcadm include
SMF Message Logging
In addition to the system logging methods described earlier in this chapter, each service has a log file in the /var/svc/log directory (or the /etc/svc/volatile directory, for services started before the single-user milestone) indicating when and how the system was started, whether it started successfully, and any messages it may have printed during its initialization. If a severe problem occurs during boot, you will be able to log in on the console in maintenance mode, and you can use the svcs command to help diagnose the problem, even on problems which would have caused boot to hang. Finally, the new boot -m boot option allows the system administrator to configure the boot process to be more verbose, printing a simple message when each service starts.
Creating New Service Scripts
As you customize your system, you'll create custom scripts to start and stop processes or services on your system. The correct procedure for incorporating these scripts into the SMF is as follows:
This next step by step describes how to create a new service and incorporate it into the SMF. Taking the time to convert your existing RC scripts to SMF allows them to take advantage of automated restart capabilities that could be caused by hardware failure, unexpected service failure, or administrative error. Participation in the service management facility also brings enhanced visibility with svcs (as well as future-planned GUI tools) and ease of management with svcadm and other Solaris management tools. The task requires the creation of a short XML file and making a few simple modifications to the service RC script. The following step by step will take you through the process.
Use Service Management Facility or legacy commands and scripts to control both the boot and shutdown procedures.
Solaris 10 still supports legacy RC scripts referred to as legacy services, but you will notice that the /etc/inittab file used by the init daemon has been significantly reduced. In addition, RC scripts that were located in the /etc/init.d directory and linked to the /etc/rc#.d directory have also been reduced substantially. For many of the scripts that remain, simply run the svcadm command to start the appropriate service.
SMF-managed services no longer use RC scripts or /etc/inittab enTRies for startup and shutdown, so the scripts corresponding to those services have been removed. In future releases of Solaris, more services will be managed by SMF, and these directories will become less and less populated. RC scripts and /etc/inittab enTRies that manage third-partyprovided or locally developed services will continue to be run at boot. These services may not run at exactly the same point in boot as they had before the advent of SMF, but they are guaranteed to not run any earlierany services which they had implicitly depended on will still be available.
For those readers who are experienced on Solaris versions prior to Solaris 10, you are accustomed to starting and stopping services via rc scripts. For instance, to stop and start the sshd daemon, you would type:
/etc/init.d/sshd stop /etc/init.d/sshd start
In SMF, the correct procedure to start sshd is to type
svcadm enable -t network/ssh:default
To temporarily stop sshd, you would type
svcadm disable -t network/ssh:default
Or simply type
svcadm restart network/ssh:default
to stop and restart the sshd daemon.
Prior to Solaris 10, to send a HUP signal to the ssh daemon, we would have typed
kill -HUP 'cat /var/run/sshd.pid'
In Solaris 10, the correct procedure is to type
svcadm refresh network/ssh:default
Using the Run Control Scripts to Stop or Start Services
Although it is recommended that you use SMF to start and stop services as described in the previous section, "Creating New Service Scripts," functionality still exists to allow the use of run control scripts to start and stop system services at various run levels. Run control scripts were used in previous versions of Solaris to start and stop system services before SMF was introduced.
A run level is a system state (run state), represented by a number or letter that identifies the services and resources that are currently available to users. The who -r command can still be used to identify a systems run state as follows:
The system responds with the following indicating that run-level 3 is the current run state:
. run-level 3 Aug 4 09:38 3 1 1
Since the introduction of SMF in Solaris 10, we now refer to these run states as milestones, and Table 3.26 describes how the legacy run states coincide with the Solaris 10 milestones.
To support legacy applications that still use them, run control scripts have been carried over from Solaris 9. With run control scripts, each init state (milestone) has a corresponding series of run control scriptswhich are referred to as rc scripts and are located in the /sbin directoryto control each run state. These rc scripts are as follows:
Run Control Scripts Solaris startup scripts can be identified by their rc prefix or suffix, which means run control.
You can still use the init command to transition to between the various run states. The init daemon will simply pass the required run state to the svc.startd daemon for execution.
The SMF will execute the /sbin/rc<n> scripts, which in turn execute a series of other scripts that are located in the /etc directory. For each rc script in the /sbin directory, a corresponding directory named /etc/rc<n>.d contains scripts to perform various actions for that run state. For example, /etc/rc3.d contains files that are used to start and stop processes for run state 3.
The /etc/rc<n>.d scripts are always run in ASCII sort order shown by the ls command and have names of this form:
A file that begins with K is referred to as a stop script and is run to terminate (kill) a system process. A file that begins with S is referred to as a start script and is run to start a system process. Each of these start and stop scripts is called by the appropriate /sbin/rc# script. For example, the /sbin/rc0 script runs the scripts located in the /etc/rc0.d directory. The /sbin/rc# script will pass the argument start or stop to each script, based on their prefix and whether the name ends in .sh. There are no arguments passed to scripts that end in .sh.
All run control scripts are also located in the /etc/init.d directory, and all scripts must be /sbin/sh scripts. These files are hard linked to corresponding run control scripts in the /etc/rc<n>.d directories.
These run control scripts can also be run individually to start and stop services. For example, you can turn off NFS server functionality by typing /etc/init.d/nfs.server stop and pressing Enter. After you have changed the system configuration, you can restart the NFS services by typing /etc/init.d/nfs.server start and pressing Enter. If you notice, however, many of these RC scripts simply have svcadm commands embedded in them to perform the task of stopping and starting the service.
In addition to the svcs -p command, you can still use the pgrep command to verify whether a service has been stopped or started:
pgrep -f <service>
The pgrep utility examines the active processes on the system and reports the process IDs of the processes. See Chapter 5, "Managing System Processes," for details on this command.
Adding Scripts to the Run Control Directories
If you add a script to the run control directories, you put the script in the /etc/init.d directory and create a hard link to the appropriate rc<n>.d directory. You need to assign appropriate numbers and names to the new scripts so that they will be run in the proper ASCII sequence, as described in the previous section.
To add a new run control script to a system, follow the process in Step by Step 3.5.
The following example creates an rc script named program that starts up at run level 2 and stops at run level 0. Note the use of hard links versus soft links:
# cp program /etc/init.d # cd /etc/init.d # chmod 744 program # chown root:sys program # ln /etc/init.d/program /etc/rc2.d/S99program # ln /etc/init.d/program /etc/rc0.d/K01program
# ls -li /etc/init.d/program /etc/rc?.d/[SK]*program
The system displays the following:
389928 -rwxr--r-- 3 root sys 69 Oct 26 23:31 /etc/init.d/program 389928 -rwxr--r-- 3 root sys 69 Oct 26 23:31 /etc/rc0.d/K01program 389928 -rwxr--r-- 3 root sys 69 Oct 26 23:31 /etc/rc2.d/S99program
Disabling a Run Control Script If you do not want a particular script to run when the system is entering a corresponding init state, you can change the uppercase prefix (S or K) to some other character; I prefer to prefix the filename with an underscore. Only files beginning with uppercase prefixes of S or K are run. For example, you could change S99mount to _S99mount to disable the script.