Proctool User's Guide



Walter Nielsen
Morgan Herrington







Table Of Contents


Introduction to Proctool

Proctool is a tool for process monitoring and management. It is targeted for system administration, performance analysis, as well as general observations by end users. Proctool encompasses much of the the functionality provided by ps(1), renice(1), and priocntl(1). It also provides some of what dispadmin(1M) and sar(1M) do.

The key difference between Proctool and the above mentioned tools is that Proctool provides a Motif graphical user interface. Also unlike ps, Proctool will periodically sample the processes and the kernel at user specified intervals. It also provides a way of graphing the data and in general, much more functionality.

The basic idea of Proctool was inspired by top, a freeware Unix tool available through most ftp archives.

Support, Distribution Rights, etc.

Proctool is available in binary form for free. It is not an official Sun Microsystems Inc product and there is no implied support or endorsement from Sun Microsystems. However, the authors plan to support the product on their own time. For bug reports, request for enhancements, questions or comments, send email to:

System Requirements

Version 2.x of Proctool requires Solaris 2.x FCS on SPARC systems only. As a rule, Proctool is compatible only with a particular major version of Solaris. This is because it uses unadvertised interfaces in the kernel which may change between Solaris releases. To make this relationship more explicit, the Proctool version numbering scheme matches Solaris's. i.e. Proctool v2.4.5 is for Solaris 2.4.


Feature Highlights

Some of the features of Proctool:


Basic Features

Proctool Main Window

Proctool's main window is split into three main panes:

The Control Area on the top contains control gadgets for Proctool. The Process Area is a list of processes similar to a ps(1) output. Processes in this window are selectable with the mouse button. The bottom pane is the System Area. This contains the system-wide information along with a per-CPU information line. The footer area of the main window is used as the message area. Most error messages are printed there.

There are a number of popup command and property windows that are also useful and interesting. The graph window is the most visually entertaining. There is a process property window that lets you examine a process in detail and a system property window that lets you delve more deeply into the system as a whole.

Monitors

A powerful mechanism provided by Proctool is the concept of "monitors". A monitor consists of a logical expression and a set of actions. The logical expression is composed of relational expressions where the operands are various per process parameters such as UID, Resident Set size, etc. and values. Monitors are evaluated every time an update occurs at the end of a sampling period or when a new monitor is inserted or redefined.

A typical monitor expression is as follows:

which evaluates to all C-shell processes owned by the user walter.

Actions are invoked when a monitor's logical expression evaluates to true. Actions may consist of reminders such as beeps, mail and popups. Actions may also change the visual representation of the process in proctool, for instance displaying a process in another color such as red or yellow.

Some typical use of a monitor by a system's administrator:

An extremely territorial and paranoid person can set up a monitor so that if (UID!= <their UID> AND UID!= 0), then execute a command: kill -9 $PID. This will make sure no one does anything on their system (and probably win a couple of popularity contests as well).

Monitors are defined through the COMMANDS => MONITOR menu. All monitors must have a name associated with them and must be entered into the monitor list. Monitors may be active or inactive. An inactive monitor is ignored until activated. Normally activated monitors remain active until user intervention. Monitors may also be defined so that their actions are executed exactly once. These monitors deactivate themselves once their actions have been triggered.

Monitors are explained in greater detail in the section titled Monitor Window. The syntax of a monitor expression is shown in Appendix A.

Privileges

Proctool can be used in both privileged or unprivileged mode. Privileged mode is granted to users who are root, i.e. UID == 0. A user can also enable privileged mode by entering the root password through the COMMANDS => SET UID menu.

Without root privileges, the user is restricted so they are unable to kill, renice, change the priority, or bring up a process property popup window on a process not owned by them.


Control Area Menus

Control Area Menus

The following is a description of the menus and menu items at the top of the main Proctool window.

File Menu

View Menu

Commands Menu

Graph Menu

Properties Menu


Control Area

Control Area

The following is description of the gadgets available in Proctool control area, the portion between the menu area and the process pane.


Process Area

Process Area

The Process Area show the list of current processes running on the system. The fields displayed for each process depends upon the viewpoint selected. The order of processes displayed in this window depends on the sort criteria selected. One may also use monitors to selectively hide or color processes displayed within the Process Area.

Processes may be selected from the Process Area. Many of the property and command windows requires the user to select one or more processes first. To select a process, point to it with the mouse and clicking it with either the left or middle button. Multiple processes may be selected by sweeping with the middle button down. (The selection behavior is based roughly on OpenWindows mechanism).

Double clicking on a process will bring up the Process Property Window. This window allows the user to examine in detail various aspects of the process.


System Pane

System Area

The system pane shows system wide and per CPU information.

The system-wide statistics show the total number of processes and the number categorized by whether they are sleeping, running, idle, etc. On a single CPU system, there will be exactly one running process, namely Proctool itself.

Proctool will automatically detect the number of CPUs in a system and show the utilization times for each. One a single CPU system, the last PID field is not too interesting as it will always be Proctool. The time utilization field may be interesting on a multiple CPU system since it may show the load balance on the system.


Define Viewpoint Window

Define Viewpoint Window

The viewpoint command window allows the user to select which process parameters (resource variables) they want to see in the main process window pane. To change the current viewpoint, just enter a list of variables in the lower text field and hit the Apply button.

Viewpoints can be named and saved. This is done by entering the list of resource variables, then entering a viewpoint name in the top half of the command window, then hitting the Insert button. To replace an existing viewpoint definition, first select a viewpoint, edit the resource variable list on the bottom then hit replace. Hitting Reset resets the viewpoint to the previously set viewpoint.


Sort Window

Define Custom Sort Window

The sort command window allows one to define the sort keys used to sort the processes visible in the Process Area. The default sort key is by PID.

Multiple sort keys can be specified, up to 20. By default, Proctool sorts in ascending order. One can choose to sort on a key in descending order by selecting the Sort Descending toggle switch before selecting a resource variable in the upper scrolling list.

Sort keys that are used in descending order are preceded by a tilde '~' in the textual representation.

For example, the sort window shows two keys, ~CPU% and RSS. This means that processes will be sorted so that the process consuming the most CPU resources (highest CPU% value) will be show first. In the event that two or more processes have the same value of CPU%, they will be sorted in ascending RSS usage.

A new set of keys specified in this window will take affect once the Apply button is selected.


Monitor Window

Define Monitor Window

The monitor window consists of the following components:

Usage

Typically one enters a new monitor by first entering a name, then an expression, and selecting a set of actions. Once this is done, the user can insert the monitor by hitting the Insert button. The user has the option of inserting a new monitor at the top or bottom of the list, or before and after another selected monitor. The order is important due to the order monitors are evaluated (see section on Monitor Evaluation below).

To edit the definition of an existing monitor, first select it. The name, expression, and action fields will show its current definitions. The user may then change any of these fields and then replace it by hitting the Replace button.

Monitor Evaluation

Monitors can be either be active or inactive. Deactivated monitors are not triggered. At every sampling update or at after each command that affects the state of a process (nice, kill, signal, priocntl), the list of active monitors is evaluated. Monitors are evaluated from top to bottom in the order they are found in the monitor list. For each monitor, the monitor expression is applied against all the processes, not just the ones that are visible. Processes are evaluated in ascending PID number for each monitor.

Conflicts may occur during monitor evaluation. For example, one monitor may want to change a process to be shown in red, another to be shown in orange. In these cases, the action of the last monitor takes precedence, using the rules described previously.

Monitor Actions

Visual

This monitor action is used to change the color of a process in the main process window pane. Monitors triggered with this action are not transcripted.

Beep

Monitors that are triggered will generate a bell (^G) on the terminal. In almost all usage, this would not be the only action selected.

Mail

There is a default message that is sent when a monitor mail action is triggered. The default message is shown in the figure below.

Monitor Mail Window.

The user may choose to customize the standard template. To do so, just edit this popup window. The edited template now becomes associated with the particular action. All other unedited monitor mail templates are not affected.

Note that the $USER, $CMD, $PID, and $EXPR are macros that are expanded at the time the action is invoked. They are replaced with the actual values of the process that triggered the monitor. In addition, the macro $EXPR expands to the actual monitor expression that was triggered. Note that all process resource variables may be used in the template (e.g. $RSS, $CPU, $NICE, etc.).

Command

Select this button to define a Monitor Action which can be any Unix command. That command will be invoked when the monitor is invoked. The command string is evaluated in the Bourne shell (/bin/sh).

The command string does macro expansion just as the Monitor Mail text does (see above).

NOTE. Discretion should be observed on what kind of command is used here. In this version of Proctool, Proctool will hang until the command is completed.

Hide

Use this action to hide a process in Proctool's main process window pane. A typical example of this usage is to hide all processes owned by root using this expression:

Popup

Use this action to generate a popup when the monitor is triggered. Only one popup is triggered per monitor. The popup will show the process ID's of up to the first ten processes that triggered the monitor. The others can be seen in the transcript.

Repetition Count

For each monitor, you can specify how many times it should trigger. The options are:

ONCE Trigger once and deactivate itself.
ONCE PER PID Trigger once per unique PID. The monitor keeps trackof each PID it triggered on.
INDEFINITELY Keep on triggering.

Activation

The default init file has a bunch of monitors defined. Usability hint: the easiest way to selectively activate/deactivate is to use the popup menu on the monitor scrolling list at the top of the Monitor Window. Do this by pressing the right (menu) mouse button over it.


Signal Window

Signal Command Window

Allows one to send any signal to the list of processes selected in Proctool main process window area. If more than one process is selected, the same signal is sent to each of the selected processes but the order is undefined. The user must have root privileges to send a signal to a process which they do not own. The results of sending the signal is shown in the transcript window. Hint - not a good idea to try this on Proctool itself!


Process Control Window

Process Control Window

The Process Control Window provides two set of functions.

For either one of these options, one can apply the changes to a list of processes selected in the main process area, to a list of processes with the same User ID, Parent Process ID, Group ID, etc.


Set UID Window

Set UID Window

Use this command window to obtain root privileges. Proctool starts up in a non-privileged mode unless invoked as root. By entering the root password, the user is then granted super-user privileges. Having this permission allows users to nice, kill, send signals, set run-time priorities and examine processes that they do not own.


Processor Control Window

Processor Control Window

This option is available only on multi-processor systems. It allows one to selectively turn off and on processors. When using this option, Proctool will check the status of the processors when this window is first popped up. After the user selects a change in status for one or more of the processors and hits the APPLY button, it will once again check the current status of the processors before applying the change. This is to avoid getting into situations where ALL of the processors are turned off !!

Processor ids are not necessarily numbered sequentially starting from 0. For example, on an SS10, a 2 processor system has CPUs number 0 and 2.


Graphs

Proctool provides three graph windows. Depending on the type of graph window, there can be up to 4 graph areas within each window. Graph windows can be used to visually display various system and process parameters. The graphs are updated at every sampling period, which is in sync with the main process window. Proctool maintains a history of up to 60 samples per graph.

All graphs can be shown in either line mode or bar mode. The line mode charts the history, up to 60 samples. In bar mode, Proctool will always show at least two bar graphs, one representing the most current values and the other one showing the average values for all samples in its history buffer. Currently there are no facilities for history greater than 60 samples or graph replay capabilities.

Once a graph window is opened and it starts graphing, the updates occur in sync every time the main process window is updated. If a graph window is iconized, it's data is still updated though not rendered. If an user QUITs a graph window, then updating terminates.

System Graph Window.

There can be up to three graph areas in this window.

CPU Utilization Graph

System Paging Graph

Process Graph Window

Process Statistics Graph

This window allows the user to graph a set of variables for a set of processes. There can be up to 4 graph areas in this window and you can choose up to 8 processes for each graph area.

Here are the steps to graph a set of processes (see Processor Control Window):

(1) Bring up the Process Graph Window.
(2) Select up to 8 processes in the main window
(single click plus middle button select).
(3) Bring up the popup-menu in the graph and pick "SELECT PIDS".
(4) Bring up the same popup-menu and pick "SELECT VARIABLE".

By selecting DUPLICATE GRAPH, one can bring up another graph area. You can graph a different set of processes on this graph and/or another variable.

Process Page Usage Graph Window

Page Usage Graph

This is a specialized graph for showing detail virtual memory page access patterns for one process. On Solaris 2.x, one can instruct the kernel to keep detailed accounting of which virtual memory pages a process accesses. Proctool breaks down the selected processes page usage with some detail, showing the of new text vs. data and read vs. write accesses on a per page granularity.

Only one process can be displayed in this graph. The selection mechanism is exactly the same as in the process graph.

NOTE: Using this option on a process will put a slight load on a system, it should not be used while doing a benchmark.

Also note that if you use "kill -9" to terminate Proctool, then it will not have any opportunity to disable the accounting which is being done by the kernel. If proctool is allowed to exit normally (or via any catchable signal), then it will reset this state gracefully.

Changing Graph Colors

The default colors for the variables in a graph are GREEN, BLUE, YELLOW, RED, PURPLE, ORANGE, AQUAMARINE, and MAGENTA. You can change the colors by adding the following lines to your X resources file:

proctool.graph_color_0: pink
proctool.graph_color_1: yellow
...

In this example, the first variable for all the graphs will be PINK instead of GREEN. And the second would be YELLOW instead of BLUE.


Tool Property Window

Tool Property Window

The tool property window allows the user to set various parameters which apply to all of Proctool. The control gadgets in the window are described below:


Process Property Window

Process Property Window

The process property window presents information for a particular process. The kind of information presented here is generally too expensive (CPU-wise) to show for all processes in the main process window.

To invoke this window, double click on a process in the main window's process area. The contents of the Process Property Window can be updated by hitting the Update button. One can also turn on the auto-update mode. When this is done, the contents are updated after every sampling time interval.


System Property Window

System Property Window

The system property window presents information that apply to the system as a whole. There are three available views. Selecting the Update button updates the data should it be changed from within or without Proctool. This window may be selected through the PROPERTIES menu -> SYSTEM PROPERTIES menu item or by double clicking on the system pane.


Configuration

Configuration is done in the following steps:


Implementation

Proctool is actually two processes, proctool and pmon. proctool is the GUI front-end. pmon is a setuid (to root) program that reads the /proc and kernel data structures to get the necessary information and passes it to proctool. There are a number of reasons for having two processes:

1) To preserve security of the system. Many of the operations done by pmon require root privileges hence pmon is setuid to root; proctool is not.

2) Future distributed systems. A future enhancement may involve having multiple pmons on various systems and a centralized proctool. This would allow monitoring multiple systems.

Proctool's GUI was implemented using Motif on Solaris 2.x using ANSI C.


Known bugs and Deficiencies

Graphs - In a line graph, if two or more variables have the same exact value for a more than one sampling period, they overwrite each other and only the last one is seen.

Traceback - Occasionally is unable to find an entry point, and prints it out as "???????". Does not work on a.out files running under Binary compatibility mode. Tracebacks are extremely slow and could use optimization work.

Memory map- There is some guesswork in the description of some memory map segments. For example, if a segment maps to a shared library, then if it has execute but not write privileges, it is assumed to be the text. Doesn't work on a.out files running under Binary compatibility mode.

Error messages- When displayed in the footer area of the main window or a popup, they sometimes get quickly overwritten by something else. Most errors are also written into the transcript but not all.

Customizing fonts, etc. Most fonts, colors, etc. can be customized using your standard Xt resources. In some cases, Proctool has some fixed non-customizable resources. Examples are the font type in the process and system window. This may change in the future.


Trouble Shooting

In case you have trouble running proctool, here is a short checklist of potential problems:


Bug Reporting

If you happen to find a bug in proctool, please report it directly to the authors (and not through Sun's bug reporting system). In all cases, please report:

If the problem isn't easy to reproduce, and if either of the programs dump core, then please include the core file in the bug report (or at least include a traceback from dbx or adb).

If the display program (proctool) is dying without leaving a core file, you can invoke it under a debugger and ask for a traceback when it quits. This could also be done with the monitor program (pmon), but is more difficult since you would have to attach to the running program.

For pmon problems, try setting the environment variable PROCTOOL_DEBUG and re-running. This will generally cause the program to dump information about what went wrong and what it was doing.

One other useful debugging technique is to run Proctool under truss; and include the last few hundred lines of output in the bug report, for example:


Monitor Expression Syntax


Process Area Variables