Previous Page
Next Page

Certification Objective 8.01–Identify, Detect, and Protect Against Trojan Horse Programs and Backdoors

Before we delve into specifics about remote access attacks, let's define some of the terms this chapter exemplifies. A Trojan horse program (Trojan) is basically a malicious program that is disguised as some useful software. Upon execution, Trojans typically install some form of backdoor program on the system that will allow a remote attacker unfettered access at a later time. However, in some cases, successfully implemented Trojans that are disguised as legitimate programs will actually perform the anticipated functions while also secretly executing the intended malicious actions. The most popular examples provided by Sun Microsystems are a malicious substitute switch user (su) program placed in a public directory that is waiting to be accidentally executed and a shell script that spoofs the login program. On the flipside, a popular form of permissible backdoor that can potentially be exploitable is a program set up by a programmer to provide remote access to the system to perform debugging and troubleshooting tasks.

A worm is not the same thing as a Trojan; a worm is a self-replicating program that copies itself from system-to-system, sometimes using up all available resources on infected systems. Of late, worms have been receiving a lot of press, given their quick replication by rapidly spreading to thousands of computers all over the Internet and affecting their availability. In addition, some worms include payloads that open up backdoors on systems similar to that of Trojans. For example, upon execution, the latest Bagel.U variant has a backdoor that listens at TCP port 4751. At that point, the worm will attempt to connect to a web page hosting a PHP script, which is how the worm notifies the attacker that the system on which it currently resides can be remotely accessed through TCP port 4751.

Exam Watch 

Make sure you know the differences between, and similarities among, the types of malicious attacks discussed in this chapter. Know that a worm is not the same as a Trojan horse, and know the difference between the two. Also, you should know enough about each type of attack to understand how each affects the system. A Trojan horse program is a malicious program that is disguised as some useful software, a worm is a self- replicating program that will copy itself from system-to-system, a logic bomb is code that is inserted into programming code, and a fork bomb is a system process that replicates itself.

A logic bomb is malicious source code that is inserted into legitimate programming code and is designed to execute under specific circumstances, such as upon someone opening a particular file, when the system is rebooted, or on a specific date. Be sure not to confuse a logic bomb with a fork bomb. The latter is a system process that replicates itself until it exceeds the maximum number of allowable processes; therefore, no new processes will be able to initiate. In essence, a fork bomb is a form of denial of service (DoS) attack. Refer back to Chapter 7 for information regarding DoS attacks and remediation.

One final bad boy that should be mentioned is a virus. A virus can be defined as a program or code that can self-replicate and that is typically loaded onto your computer without you being aware of its presence. Viruses have various payloads, such as using up available memory or halting your system. Refer to Figure 8-1 for an excellent virus-worm-Trojan comparison, courtesy of Brian and Mark at ThinkQuest.

Image from book
Figure 8-1: Virus, worm, and Trojan comparison

Securing the System from Trojans and Backdoors

Several techniques can be exercised to help detect and protect against Trojans and backdoors on a system running Solaris. In this section, we'll focus on methods that Sun recommends and that are also covered by the exam. These include tasks that can be performed to harden your system, such as educating users, installing anti-virus software, removing unnecessary compilers, securing file and directory permissions, monitoring path variables, and using file digests, checksums, and the Solaris Fingerprint Database. Let's start by examining ways to harden the Solaris operating system with regard to Trojans and backdoors.

Taking Protective Measures

Probably the most important key to protecting against Trojans and backdoors is user awareness. Executing unknown programs from unknown sources is a risk that users should understand completely. User awareness training and frequent security policy memorandums are critical to help mitigate risks associated with Trojans and backdoors. Anti-virus software should be installed and kept up-to-date on every desktop and server in the infrastructure. In addition, e-mail filters should be deployed and configured to block dangerous attachments.

Some of the most popular anti-virus solutions for Solaris are the following:

In addition to user awareness and maintaining anti-virus definition updates, Sun recommends a few Solaris-specific tasks to harden operating systems and protect against Trojans:

  • Removing unnecessary compilers

  • Securing file and directory permissions

  • Monitoring path variables

We'll cover in detail user, file, and directory permissions in Chapters 9 and 10; in this section, we'll talk about removing compilers and monitoring path variables.

Removing Unnecessary Compilers  A compiler is a program that processes code written in a programming language and builds the machine code (also known as compiling) that the system can understand. One method used to harden Solaris systems so that a user or attacker cannot compile malicious code on it is to remove compilers that are not needed for normal operation. Some administrators go so far as to remove all compilers, and then reinstall them only as needed. Among the most popular development suites for Solaris that contain compilers is the Sun ONE Studio. As an example, we'll remove the Sun ONE Studio 8 compilers with the command-line uninstaller that was built during the standard installation. The uninstaller for the Sun ONE Studio 8 compilers is uninstall_Compilers.class.

To remove the Sun ONE Studio 8 compilers with the command-line uninstaller, follow these steps:

  1. Log in with an account that has root privileges, or use the su command to become superuser. As superuser, you'll have full privileges; for example, to switch to root, simply open a terminal session and type su root, press ENTER, and then enter root's password at the prompt.

  2. In a terminal session, go to the product directory using the following command:

    cd /var/sadm/prod/com.sun.s1s8_compiler_collection_fcs
  3. To determine the uninstaller .class filename for the software you want to uninstall, type ls –l to see the list of uninstaller .class filenames or use the standard uninstaller filename for the Sun ONE Studio 8 compilers, which is uninstall_Compilers.class.

  4. Type the following command to run the command-line uninstaller:

    /usr/bin/java uninstall_Compilers -nodisplay

    Or to uninstall the entire product without any additional prompts (in silent mode), use this:

    /usr/bin/java uninstall_uninstaller-class-file-name -nodisplay –noconsole

    where uninstaller-class-file-name is the name of the class file.

  5. Press ENTER to continue.

  6. If you want to uninstall all components of the product, type 1 and skip to step 9.

  7. If you want to uninstall only certain components of the product, type 2. The uninstaller displays a list of the components of the product with checked boxes indicating which components will be uninstalled.

  8. If you do not want a component uninstalled, type its number to deselect it. Otherwise, type 0. The uninstaller displays an updated list showing which components of the product will be uninstalled. Select another component that you want uninstalled and type its number, or type 0 to proceed with uninstallation.

  9. The uninstaller lists the components that will be uninstalled and asks you what you would like to do. Type 1 if you want to uninstall the product components now, and proceed to step 10. Type 2 if you want to start over with the uninstallation process, and go back to step 3. Type 3 if you want to exit the uninstaller, and proceed to step 11.

  10. When uninstallation is complete, you have the option of viewing the product's log file by typing its corresponding number. When finished, type the number that corresponds to Done. The installer exits.

  11. (Optional) If you performed the uninstallation using a remote display, on the display computer, disable client access by typing % xhost - source-computer- name.

  12. Exit from superuser privileges on the source computer by typing exit.

Monitoring Paths  The next step with regard to hardening your Solaris operating system, specifically to help protect against Trojans, is more of an ongoing task. Monitoring paths, in particular the PATH and LD_LIBRARY_PATH environment variables, is an important preventative measure, especially for that of privileged users (that is, root). Directories that are referenced should have the appropriate permissions as well as the files that reside in those directories, which we'll talk more about in Chapters 9 and 10.

As an example of this predicament, take a look at root's path. This path—or any path, for that matter—is set at login (through the startup files .login, .profile, or .cshrc, depending on users' default shell) and should never contain a parameter indicated with a dot (.). This parameter would cause the system to search for executables or libraries within that path. If a user placed a Trojan unknowingly or an attacker maliciously planted one in a directory in that path with which privileges were set, at some point root could come by and execute the program by accident.

In addition, you should not allow a search path for root to contain the current directory such as in the following path statement: PATH=./:/usr/bin. In other words, the previously mentioned PATH variables that include the period would permit a condition where root could potentially run the wrong program and execute a Trojan. Other good examples of this concern include these PATH variables: PATH=/usr/bin:/usr/sbin:.:/usr/tools/bin and PATH=/usr/bin:/usr/sbin:. If you find a dot parameter in your PATH variables, you should consider removing it at once, unless otherwise temporarily implemented by administration. You should set up the user search path so that the current directory (.) comes last. The path variable for superuser should never include the current directory.

Exam Watch 

Almost every version of Sun's exam asks a question regarding protection against Trojan programs. You should know that path variables that do not contain a parameter indicated with a dot (.) are recommended to avoid having the system searching for executables or libraries within that path.

Using the Automated Security Enhancement Tool

To help ensure that path variables are correct, among other things, Sun recommends using the Automated Security Enhancement Tool (ASET). ASET enables you to monitor and restrict access to system files and directories with automated administration governed by a preset security level—low, medium, or high. At the low level, ASET does not take action or affect system services, it ensures that attributes of system files are set to standard release values, and it performs several checks while reporting any potential security weaknesses. At the medium level, ASET does not affect system services; it modifies settings of system files and parameters, restricts system access, and reports security weaknesses and modifications that were previously made to restrict access. Finally, at the high level setting, ASET adjusts system files and parameter settings to minimum access permissions. At this level, most system applications and commands continue to function normally; however, security considerations take precedence over other system behaviors.

Sun recommends running regularly scheduled ASET tasks during off-peak hours or when system activities as well as the user load are low. These tasks include system files permissions tuning, system files checks, user and group checks, system configuration files check, environment variables check, EEPROM (electrically erasable programmable read-only memory) check, and firewall setup. In the remaining sections we'll take a closer look at each of these tasks.

Table 8-1 contains a list of the ASET report filenames for each of the tasks that ASET performs. You should note that messages in each report are enclosed with a beginning and an ending banner. Be sure to examine these files closely every time ASET is run to monitor and solve security problems. The reports are stored in the /usr/aset/reports directory, within subdirectories that are named in accordance with the time and date the reports are created.

Table 8-1: ASET Tasks and Associated Report Files


Report File

System files permissions tuning


System files checks


User and group checks


System configuration files check


Environment variables check


EEPROM check


Firewall setup


System Files Permissions Tuning  Tuning system files permissions automatically sets system file permissions according to the security level you choose. At the high level, setting permissions are assigned to restrict access; at the medium level, permissions are tightened just enough for most normal operating environments; and at the low level setting, permissions are set for open sharing. For example, if a system is placed in a demilitarized zone (DMZ) or configured as a perimeter firewall, many administrators tend to set a high level to secure system file permissions further. On the other hand, if a system is placed in a local area network (LAN) that is behind a perimeter, security administrators tend to use the medium level setting. Finally, when a system is placed in an offline network or for a system that is unconditionally trusted—maybe even a development subnet or for testing—administrators can use the low setting for an open information-sharing environment.

When the system is installed, the system files permissions tuning task will run by default; however, after making security-level adjustments, it should be run again to modify system file permissions accordingly. Any changes made using this task will be logged in the tune.rpt file.

System Files Checks  Fundamentally, system files checks is a file comparison check from a master file that is created when the task is first executed. For each security level, a list of directories that contains files to check is automatically defined; however, this list can be modified. Upon running this task, the following criteria are checked for each file in each directory in the list:

  • Owner and group

  • Permission bits

  • Size and checksum

  • Number of links

  • Last modification time

Probably one of the most useful criteria checks in system files checks is the checksum. Traditionally, a checksum is a special count of bits in a particular transmission, which is included with the transmission, so that the receiver can verify that the same bits arrived during the transmission. If the counts match, it's assumed that the transmission was successfully received intact. Similarly, the system files checks checksum uses a sum command to produce a Cyclic Redundancy Check (CRC) and block count for files that exist in specified directories. Upon running the task, the checksum component can help prevent backdoor attacks by verifying the integrity of files. When the system files checks task finds inconsistencies between the criteria in the master file and the file being checked, they are reported in the cklist.rpt file.

Exam Watch 

It's common to come across questions about preventing backdoor attacks on the exam. Be sure to know what checksum is and how it can help prevent backdoor attacks.

User and Group Checks  The user and group checks task is used to verify the integrity of user accounts, their passwords, and their groups. The primary check is made from the passwd and group files, and the passwords in local, NIS, and NIS+ files. Upon running this task, the following criteria are checked for violations and are reported in the usrgrp.rpt file:

  • Duplicate names or IDs

  • Entries in incorrect format

  • Accounts without a password

  • Invalid login directories

  • The nobody account

  • Null group password

  • A plus sign (+) in the /etc/passwd file on an NIS server or an NIS+ server

System Configuration Files Check  During the system configuration files check, ASET checks the integrity of, inspects, and makes modifications to system configuration files mostly found in the /etc directory, and reports problems in the sysconf.rpt file. The files that this task checks are of great importance:

  • /etc/default/login

  • /etc/hosts.equiv

  • /etc/inetd.conf

  • /etc/aliases

  • /var/adm/utmpx

  • /.rhosts

  • /etc/vfstab

  • /etc/dfs/dfstab

  • /etc/ftpd/ftpusers

Environment Variables Check  The environment variables check task inspects the PATH and UMASK environment variables. These are found in the /.profile, /.login, and /.cshrc files. As you may recall from earlier in this chapter, the path should never contain a parameter indicated with a dot (.). This parameter would cause the system to search for executables or libraries within that path (the current directory). If a user placed a Trojan unknowingly or an attacker maliciously planted one in a directory in that path with which privileges were set, at some point root could execute the program by accident. This potential problem is one of the primary concerns of the environment variables check. Environment security results are reported in the env.rpt file.

EEPROM Check  In short, the EEPROM check inspects the EEPROM security parameter to ensure that it is set to the appropriate security level and has not been tampered with. The EEPROM security level options can be set either to none, command, or full. If set to command or full, the system will prompt for the PROM security password; however, the default is none. During the EEPROM check, ASET will not modify this setting but will report recommendations in the eeprom.rpt report file.

Firewall Setup  The firewall setup task simply ensures that the system can be safely used as a perimeter gateway—or as Sun calls it, a secure "network relay." This task can help secure a system by setting up a firewall daemon that can be used not only to separate a trusted network from an untrusted network, but to disable Internet Protocol (IP) packet forwarding and conceal routing information from the untrusted or external network, among other things. Of course, you do have the option of eliminating the firewall setup task by editing the asetenv file if you choose not to use the system as a network relay. This task reports findings in the firewall.rpt report file.

Exam Watch 

ASET runs several tasks to help secure the Solaris operating system, and each task performs specific checks and adjustments. For the exam, you should know the seven ASET tasks, how to configure them, and what files contain individual task reports.

Configuring and Running ASET  To configure ASET, and although you can simply run the suite using its predefined default parameters, you can adjust settings in the configuration files that can be found in /usr/aset/asetenv, /usr/aset/masters/tune.low, /usr/aset/masters/, and /usr/aset/masters/tune.high. To begin, let's take a look at the /usr/aset/asetenv file:

#                                         #
#      User Configurable Parameters       #
#                                         #
TASKS="firewall env sysconf usrgrp tune cklist eeprom"

The asetenv file has two sections—the user configurable parameters section and the internal environment variables section—but only the user configurable parameters part is shown in the extract, and for good reason. To avoid corrupting the program, you should not make any changes to the internal environment variables section. However, user configurable parameters can be modified by specifying the directories for the system files checks task, extending checks to NIS+ tables, specifying a UID aliases file, scheduling ASET execution, and choosing which tasks to run:

  • Specifying the directories for the system files checks task is accomplished by defining which directories to check for each security level; the CKLISTPATH_LOW variable defines the directories to be checked at the low security level, the CKLISTPATH_MED for the medium security level, and CKLISTPATH_HIGH for the high level. As shown in the extract, the lower level directories need to be specified in the higher level path; for example, in CKLISTPATH_MED=${CKLISTPATH_LOW}:/usr/bin:/usr/ucb, the CKLISTPATH_LOW directories are automatically included.

  • To extend checks to NIS_ tables, the YPCHECK environment variable would have to be enabled with a true parameter. This is particularly useful if it is required to check the NIS+ passwd table; otherwise only the local passwd file is checked. We'll talk more about that in the sections that follow.

  • The UID_ALIASES variable specifies an aliases file that lists shared UIDs.

  • The PERIODIC_SCHEDULE parameter is used by ASET to determine when to execute tasks. The format of PERIODIC_SCHEDULE follows that of crontab entries.

  • In the remaining parts of this section, we'll take a closer look at each individual task that ASET runs. Choosing which tasks to run in the user configurable parameters section of the /usr/aset/asetenv file is relatively easy. By default, it contains all seven ASET tasks shown in this extract:

    TASKS="firewall env sysconf usrgrp tune cklist eeprom"

    To exclude tasks, simply remove them from the list.

By editing the /usr/aset/masters/tune.low, /usr/aset/masters/, and /usr/aset/ masters/tune.high files, you can restrict permissions to values that are appropriate for default system settings. The higher the security level tune file (that is, tune.high), the more restrictive permissions can be. According to Sun, each entry in a tune file occupies one line. The fields in an entry appear in the following order:

  1. pathname  The full path name of the file

  2. mode  A five-digit number that represents the permission setting

  3. owner  The owner of the file

  4. group  The group owner of the file

  5. type  The type of file

In addition, you must follow these nine rules when making modifications to the tune files:

  1. You can use regular shell wildcard characters (that is, asterisks and question marks) in the path name.

  2. mode represents the least restrictive value. If the current setting is already more restrictive than the specified value, ASET does not loosen the permission settings.

  3. You must use names for owners and groups instead of numeric IDs.

  4. You can use a question mark (?) in place of owner, group, and type.

  5. type can be a symlink, directory, or file.

  6. Higher security level tune files set file permissions to be at least as restrictive as file permissions at lower levels.

  7. A file can match more than one tune file entry.

  8. File permission is set to the most restrictive value despite entries with different permissions.

  9. The last entry has priority when two entries have different owner or group designations.

When you're ready to run ASET, you can manually execute ASET tasks or simply schedule ASET to run periodically. To run ASET manually at any given time, simply log in as root or become superuser and then issue this command:

/usr/aset/aset -l level -d pathname

where level is security level value (low, medium, or high), and pathname is the working directory for ASET (the default is /usr/aset).

On the other hand, to run ASET automatically at a predefined time, simply edit the PERIODIC_SCHEDULE variable in the /usr/aset/asetenv file to the desired start time and then add ASET to the crontab file (which is a list of tasks that are run on regular schedules). For more information on the crontab utility, visit on the Web or use the command man crontab.

File/Message Digests

One popular method that Sun supports for verifying whether files were maliciously altered is to use message digest algorithms. A message digest is a one-way function for a stream of binary data as verification that the data was not altered since the message digest was first generated. For example, many times when downloading a patch, update, or program, you'll see output such as the following along with your download link:

 Creation Date: April 12, 2004
 Release Date: April 12, 2004
 File Size: 4.63 MB
MD5: B12A24F23E37B0EFC4B9C42B1247B8B9

This information was published with a message digest signature (that is, an MD5 hash) when the file was created. In this example, after downloading the file, you would run the MD5 utility on it to verify that you get the same MD5 hash output that was posted with the download link. If the signature or hash is different from that posted, you can assume that the file was either corrupted during transfer or possibly maliciously altered since the hash was first published.

The MD5 and the Secure Hashing Algorithm (SHA1) are among the most popular message digest algorithms. MD5 was developed by Ron Rivest in 1991 at the MIT Laboratory for Computer Science and RSA Data Security, Inc. The MD5 algorithm takes a message of arbitrary length and produces a 128-bit message digest. SHA1—a revision to the original SHA that was published in 1994—is similar to the MD4 family of hash functions developed by Rivest. The algorithm takes a message and produces a 160-bit message digest. Although MD5 and SHA1 were developed to help detect corrupt or maliciously altered files, Sun recommends using a more comprehensive package called Tripwire. In addition to Tripwire, to help prevent unauthorized changes from being made to system files, Sun also recommends using ASET (discussed in the previous section) and the Basic Security Module (BSM), which is discussed in Chapter 5. Finally, the Solaris cryptographic framework supports a command that can also be used to check the integrity of files. For example, using the digest command, you can compute a message digest for one or more files.

We'll talk more about the cryptographic framework, including creating symmetric keys and encrypting files, in Chapter 11; here, we'll take a succinct look at how to create digests of files using the MD5 and SHA1 algorithms.

Computing a Digest of a File  By comparing digests of a file, you are checking the file's integrity to ensure that it has not been corrupted or altered. In the Solaris cryptographic framework environment, we can perform digest computations using the following syntax:

digest -v -a algorithm input-file > digest-listing

Where -v displays the output with file information, -a algorithm is the algorithm used to compute a digest (that is, MD5 or SHA1), input-file is the input file for the digest to be computed, and digest-listing is the output file for the digest command.

The Solaris Fingerprint Database

The Solaris Fingerprint Database (sfpDB) is a free tool from Sun that allows you to check the integrity of system files through cryptographic checksums. By doing this, you can determine whether system binaries and patches are safe in accordance with their original checksums stored at Sun. This includes files distributed with Solaris OE media kits, unbundled software, and patches.

Exam Watch 

Preventing unauthorized changes from being made to system files is a primary security concern and Sun requires that every certification candidate understands how. Be sure to study the Automated Security Enhancement Tool (ASET), the Basic Security Module (BSM), Tripwire, and the Solaris cryptographic framework for the exam.

MD5 software binaries that can be used with sfpDB for Intel and SPARC architectures can be freely downloaded from Sun at Currently, foreign language versions are not supported, but suggestions and requests can be sent to

The MD5 software is compressed in the tar file format; therefore, after downloading to a directory (such as /usr/local), unpack the archive with the following command:

zcat md5.tar.Z | tar xvf -

The software will be unpacked into a MD5 subdirectory.

Installing MD5 Software  After downloading and unpacking the MD5 archive, the file permissions must be modified before they can be executed. To permit only root, for example, to read, write, and execute the MD5 programs, issue the chmod 700 /usr/local/md5/* command (chmod is a command used to change file access permissions). Additionally, the owner and group of the MD5 files must be modified to belong to a system user and associated group. Given the particular functionality, traditionally they should be owned by root; therefore, you should also issue the command chown root:root /usr/local/md5/* (chown is a command used to change file owners; see Chapter 10 for more on this command).

Using MD5 Software  Creating hexadecimal MD5 digital fingerprints is simple. For example, based on the installation mentioned previously, to create an MD5 fingerprint for the su program on a SPARC system, you would enter this command:

/usr/local/md5/md5-sparc /usr/bin/su

The output should look something like this:

MD5 (/usr/bin/su) = cb2b71c32f4eb00469cbe4fd529e690c

Furthermore, by placing a space between target files, you can create MD5 fingerprints for more than one file at a time, such as for the su and ls programs, by issuing this command:

/usr/local/md5/md5-sparc /usr/bin/su /usr/bin/ls

The output should look something like this:

MD5 (/usr/bin/su) = cb2b71c32f4eb00469cbe4fd529e690c
MD5 (/usr/bin/ls) = 351f5eab0baa6eddae391f84d0a6c192

Finally, to create MD5 fingerprints for all files in the /usr/bin directory, you could issue the MD5 command along with the find command, such as in the following:

find /usr/bin -type f -print \
| xargs -n100 /usr/local/md5/md5-sparc > /tmp/md5output.txt

As indicated, the output will be printed to the /tmp directory in file md5output.txt. The contents of the file can easily be copied into the online fingerprint database for integrity checks.

You can check the integrity of your files against the sfpDB portal database at (see Figure 8-2).

Image from book
Figure 8-2: Solaris Fingerprint Database portal

The online front-end interface to the database is easy to use—no assembly required. Simply scroll down to the query form (shown in Figure 8-3) and copy and paste one or more hexadecimal MD5 file signatures (one per line for a maximum of 256 possible entries) in signature format, followed by an optional comment string or in canonical MD5 output format. Currently, the database includes 2,717,303 fingerprints, 22,702 package names, and 23,658 patches. In other words, if your Sun binary, patch, or file is legitimate, it's likely listed in the database.

Image from book
Figure 8-3: Solaris Fingerprint Database query form

Upon submitting your query, the database search results will contain information regarding a signature match or mismatch, as depicted in Figure 8-4. File-specific information with regard to signature matches includes the following:

Image from book
Figure 8-4: Solaris Fingerprint Database results
  • Canonical-path

  • Package

  • Version

  • Architecture

  • Source

Previous Page
Next Page