prex [-o trace_file_name] [-l libraries] [-s kbytes_size] cmd 
 [cmd-args]...

prex [-o trace_file_name] [-l libraries] [-s kbytes_size] -p pid

prex -k [-s kbytes_size]

The prex command is the part of the Solaris tracing architecture that controls probes in a process or the kernel. See tracing(3TNF) for an overview of this tracing architecture, including example source code using it.

prex is the application used for external control of probes. It automatically preloads the libtnfprobe library. prex locates all the probes in a target executable or the kernel and provides an interface for the user to manipulate them. It allows a probe to be turned on for tracing, debugging, or both. Tracing generates a TNF (Trace Normal Form) trace file that can be converted to ASCII by tnfdump(1) and used for performance analysis. Debugging generates a line to standard error whenever the probe is hit at run time.

prex does not work on static executables. It only works on dynamic executables.

There are three ways to invoke prex:

In a future release of prex, the command language may be moved to a syntax that is supported by an existing scripting language like ksh(1). In the meantime, the interface to prex is uncommitted.

There are two different methods of communicating with prex:

$HOME/
./

The command language for both methods is the same and is specified in USAGE. The commands that return output will not make sense in a control file. The output will go to standard output.

When using prex on a target process, the target will be in one of two states, running or stopped. This can be detected by the presence or absence of the prex> prompt. If the prompt is absent, it means that the target process is running. Typing Control-C will stop the target pr ocess and return the user to the prompt. There is no guarantee that Control-C will return to a prex prompt immediately. For example, if the target process is stopped on a job control stop (SIGSTOP), then Control-C in prex will wait until the target has been continued (SIGCONT). See Signals to Target Program below for more information on signals and the target process.

The following options are supported:

-k

-l libraries

-o trace_file_name

-s kbytes_size

This section describes the usage of the prex utility.

Probes are specified by a list of space-separated selectors. Selectors are of the form:

attribute=value

(See TNF_PROBE(3TNF)). The "attribute=" is optional. If it is not specified, it defaults to "keys=".

The attribute or value (generically called "spec") can be any of the following:

IDENT

QUOTED_STR

REGEXP

The following grammar explains the syntax.

selector_list ::= | /* empty */
 selector_list selector
selector ::= spec=spec | /* whitespace around `=' opt */
 spec
spec ::= IDENT |
 QUOTED_STR |
 REGEXP

The terminals in the above grammar are:

IDENT = [a-zA-Z_\e.%]{[a-zA-Z0-9_\e.%]}+
QUOTED_STR = '[^\en']*' /* any string in single quotes */
REGEXP = /[^\en/]*/ /* regexp's have to be in / / */

This is a list of the remaining grammar that is needed to understand the syntax of the command language (defined in next subsection):

filename ::= QUOTED_STR /* QUOTED_STR defined above */
spec_list ::= /* empty */ |
 spec_list spec /* spec defined above */
fcn_handle ::= &IDENT /* IDENT defined above */
set_name ::= $IDENT /* IDENT defined above */

create $set_name selector_list
list sets # list the defined sets

list fcns # list the available fcn_handle

connect &fcn_handle $set_name
connect &fcn_handle selector_list
clear $set_name
clear selector_list

trace $set_name
trace selector_list
untrace $set_name
untrace selector_list

enable $set_name
enable selector_list
disable $set_name
disable selector_list

Enabled or Tracing State Debug State Results
Disabled (On/Off) (Connected/Cleared) In
------------------------------------------------------------
Enabled On Connected Tracing and
 Debugging

Enabled On Cleared Tracing only

Enabled Off Connected Debugging only

Enabled Off Cleared Nothing

Disabled On Connected Nothing

Disabled On Cleared Nothing

Disabled Off Connected Nothing

Disabled Off Cleared Nothing

list history # lists probe control command history

list spec_list probes $set_name # list probes $all
list spec_list probes selector_list # list name probes file=test.c
list values spec_list # list values keys given in spec_list
list tracefile # list tracefile

help topic

source filename

continue # resumes the target process
quit kill # quit prex, kill target
quit resume # quit prex, continue target
quit suspend # quit prex, leave target suspended
quit # quit prex (continue or kill target)

Probes in shared objects that are brought in by dlopen(3C) are automatically set up according to the command history of prex. When a shared object is removed by a dlclose(3C), prex again needs to refresh its understanding of the probes in the target program. This implies that there is more work to do for dlopen(3C) and dlclose(3C) \(emso they will take slightly longer. If a user is not interested in this feature and doesn't want to interfere with dlopen(3C) and dlclose(3C), detach prex from the target to inhibit this feature.

prex does not interfere with signals that are delivered directly to the target program. However, prex receives all signals normally generated from the terminal, for example, Control-C (SIGINT), and Control-Z (SIGSTOP), and does not forward them to the target program. To signal the target program, use the kill(1) command from a shell.

Process managing applications like dbx, truss(1), and prex cannot operate on the same target program simultaneously. prex will not be able to attach to a target which is being controlled by another application. A user can trace and debug a program serially by the following method: first attach prex to target (or start target through prex), set up the probes using the command language, and then type quit suspend. The user can then attach dbx to the suspended process and debug it. A user can also suspend the target by sending it a SIGSTOP signal, and then by typing quit resume to prex. In this case, the user should also send a SIGCONT signal after invoking dbx on the stopped process (else dbx will be hung).

There are a few failure points that are possible when writing out events to a trace file, for example, system call failures. These failures result in a failure code being set in the target process. The target process continues normally, but no trace records are written. Whenever a user enters Control-C to prex to get to a prex prompt, prex will check the failure code in the target and inform the user if there was a tracing failure.

If the target program does a fork(2), any probes that the child encounters will cause events to be logged to the same trace file. Events are annotated with a process id, so it will be possible to determine which process a particular event came from. In multi-threaded programs, there is a race condition with a thread doing a fork while the other threads are still running. For the trace file not to get corrupted, the user should either use fork1(2), or make sure that all other threads are quiescent when doing a fork(2),

If the target program itself (not any children it may fork(2)) does an exec(2), prex detaches from the target and exits. The user can reconnect prex with prex -p pid.

A vfork(2) is generally followed quickly by an exec(2) in the child, and in the interim, the child borrows the parent's process while the parent waits for the exec(2). Any events logged by the child from the parent process will appear to have been logged by the parent.

Invoking prex with the -k flag causes prex to run in kernel mode. In kernel mode, prex controls probes in the Solaris kernel. See tnf_kernel_probes(4) for a list of available probes in the Solaris kernel. A few prex commands are unavailable in kernel mode; many other commands are valid in kernel mode only.

The -l, -o, and -p command-line options are not valid in kernel mode (that is, they may not be combined with the -k flag).

The rest of this section describes the differences in the prex command language when running prex in kernel mode.

ktrace [ on | off ]

quit resume

quit kill

buffer [ alloc [ size ] | dealloc ]

pfilter [ on | off | add pidlist | delete pidlist ]

See tracing(3TNF) for complete examples showing, among other things, the use of prex to do simple probe control.

When either the process or kernel is started, all probes are disabled.

Example 1 Set creation and set listing

create $out name=/out/ # $out = probes with "out" in
 # value of "name" attribute
create $foo /page/ name=biodone # $foo = union of
 # probes with "page" in value of keys attribute
 # probes with "biodone" as value of "name" attribute
list sets # list the defined sets
list fcns # list the defined probe fcns

Example 2 Commands to trace and connect probe functions

trace foobar='on' # exact match on foobar attribute
trace $all # trace all probes (predefined set $all)
connect &debug $foo # connect debug func to probes in $foo

Example 3 Commands to enable and disable probes

enable $all # enable all probes
enable /vm/ name=alloc # enable the specified probes
disable $foo # disable probes in set $foo
list history # list probe control commands issued

Example 4 Process control

continue # resumes the target process
^C # stop target; give control to prex
quit resume # exit prex, leave process running
 # and resume execution of program

Example 5 Kernel mode

buffer alloc 2m # allocate a 2 Megabyte buffer
enable $all # enable all probes
trace $all # trace all probes
ktrace on # turn tracing on
ktrace off # turn tracing back off
pfilter on # turn process filter mode on
pfilter add 1379 # add pid 1379 to process filter
ktrace on # turn tracing on
 # (only pid 1379 will be traced)

.prexrc

~/.prexrc

/proc/nnnnn

ed(1), kill(1), ksh(1), ld(1), tnfdump(1), tnfxtract(1), truss(1), exec(2), fork(2), fork1(2), vfork(2), TNF_DECLARE_RECORD(3TNF), TNF_PROBE(3TNF), dlclose(3C), dlopen(3C), gethrtime(3C), libtnfctl(3TNF), tnf_process_disable(3TNF), tracing(3TNF), tnf_kernel_probes(4), attributes(5)

Currently, the only probe function that is available is the &debug function. When this function is executed, it prints out the arguments sent in to the probe as well as the value associated with the sunw%debug attribute in the detail field (if any) to stderr.

For example, for the following probe point:

TNF_PROBE_2(input_values, "testapp main",
 "sunw%debug 'have read input values successfully'",
 tnf_long, int_input, x,
 tnf_string, string_input, input);

If x was 100 and input was the string "success", then the output of the debug probe function would be:

probe input_values; sunw%debug "have read input values successfully";
int_input=100; string_input="success";

Some non-SPARC hardware lacks a true high-resolution timer, causing gethrtime() to return the same value multiple times in succession. This can lead to problems in how some tools interpret the trace file. This situation can be improved by interposing a version of gethrtime(), which causes these successive values to be artificially incremented by one nanosecond:

hrtime_t
gethrtime()
{
 static mutex_t lock;
 static hrtime_t (*real_gethrtime)(void) = NULL;
 static hrtime_t last_time = 0;

 hrtime_t this_time;

 if (real_gethrtime == NULL) {
 real_gethrtime =
 (hrtime_t (*)(void)) dlsym(RTLD_NEXT, "gethrtime");
 }
 this_time = real_gethrtime();

 mutex_lock(&lock);
 if (this_time <= last_time)
 this_time = ++last_time;
 else
 last_time = this_time;
 mutex_unlock(&lock);

 return (this_time);
}

Of course, this does not increase the resolution of the timer, so timestamps for individual events are still relatively inaccurate. But this technique maintains ordering, so that if event A causes event B, B never appears to happen before or at the same time as A.

dbx is available with the Sun Workshop Products.

prex should issue a notification when a process id has been automatically deleted from the filter set.

There is a known bug in prex which can result in this message:

Tracing shut down in target program due to an internal
error - Please restart prex and target

When prex runs as root, and the target process is not root, and the tracefile is placed in a directory where it cannot be removed and re-created (a directory with the sticky bit on, like /tmp),mm then the target process will not be able to open the tracefile when it needs to. This results in tracing being disabled.

Changing any of the circumstances listed above should fix the problem. Either don't run prex as root, or run the target process as root, or specify the tracefile in a directory other than /tmp.