Scripting: Alfred Language Summary


Alfred input scripts (a.k.a. worklists) are composed of expressions using the following operators:

##AlfredToDo 3.0

Job [options]

Task {title} [options]

Cmd {launch_expr} [options]

RemoteCmd {launch_expr} [options]

Instance {task_title_to_reference}

Iterate varname -from n -to m -by i -template {script} [options]

Assign  varname {value_string}

See the Operator Details document for detailed descriptions of options and operator syntax.  



A worklist is a structured script which describes the work to be dispatched and monitored by alfred. Worklists are typically created by application programs as they spool new rendering jobs to alfred. That is: the following syntax discussion will probably be useful only to people developing alfred-compliant applications.

Worklists support a simple hierarchical structuring scheme which can encode dependencies that in turn determine the order of task execution; worklists are somewhat similar to makefiles in this regard. The predefined worklist operators are: Task, Cmd, and Instance. There are also some special conventions for embedding certain run-time values into command expressions. As the worklist script is parsed during job initialization, alfred constructs both the internal execution hierarchy for the dispatcher as well as the widget hierarchy for the monitor. The dispatcher then continuously traverses this internal representation looking for opportunities to launch new work.

The alfred expression syntax consists of some simple list-formation rules. Specifically, each operator accepts a list of argument strings, some of which may be optional. Strings are delimited by either double-quotes or curly-braces; they may contain newlines, they are also allowed to be empty. Case is usually important, while whitespace is generally not (see the description of Instances for an exception). Newlines and semicolons, when not in strings, are end-of-expression markers. Backslash is used (as in C) to produce literal versions of syntactically significant characters and other special character codes (and to disable newline processing). Leading sharp-signs (#) introduce one-line comments. See the special characters discussion for more information.

Hierarchical nesting is accomplished by embedding child Task descriptions within the subtask field of the parent Task operator. It is only possible to create nested strings using the curly-brace delimiters.


See the dispatching overview for a general discussion of job concepts such as task hierarchies, launching sequences, and basic Alfred terminology.

A simple example

Here is a simple script which defines an example task hierarchy. Assume we have two shadow maps to compute, and a final frame which references them; in this case the shadow maps must be complete before rendering can begin on the final frame. The alfred "reserved words" are Job, Task, and Cmd, and these operators can take options, such as "-subtasks", which begin with a leading hyphen:

	##AlfredToDo 3.0

	Job -title {A Simple Job} -subtasks {

		Task {Scene One} -subtasks {

			Task {Shadow A} -cmds {
				Cmd {render light.1.rib}

			Task {Shadow B} -cmds {
				Cmd {render light.2.rib}

		} -cmds {
			Cmd {render scene01.rib}

This worklist describes a two-level execution tree in which the task named Scene One will launch its command (render scene01.rib) after the two tasks upon which it depends have completed successfully (the Shadows). The nested task descriptions define the (depth-first) order of execution; hence, the render command for light.1.rib is launched first, then the one for light.2.rib follows. Then, when they are both complete, scene01.rib is rendered. Commands are launched as separate sub-processes, in the order that they appear in the script.

Note that the two shadow commands are not dependent on each other, i.e. they aren't nested with respect to each other; hence, they represent a parallel processing opportunity. Given sufficient resources, alfred will typically launch both commands and allow them to execute concurrently. Hence, the script defines a launching order, some commands may still be running when another is launched, and these concurrent commands may complete in any order. In no case, however, will a Task launch its own commands until all of its sub-tasks have completed their commands successfully.

Note also that the first line of worklists start with the identifying comment
##AlfredToDo 3.0
by which alfred distinguishes scripts from other input (i.e. RIB files).
The "3.0" indicates the worklist language-version in use.

The assumptions made in this example are:

Alfred will not attempt to confirm these sorts of assumptions, that's the domain of the script creator. If an error does occur, the task becomes blocked, the corresponding UI widget converts to the error state, and all tasks which depend on the broken one will remain unexecuted (others may proceed however).

Things to note about Tasks:

Process Launch and Tracking

The Job traversal and command launching are described in detail in the dispatching discussion.  

Remote Servers, Launch Expressions, and Runtime Substitution

The launch expression argument to the Cmd operator specifies the actual programs that alfred should launch. As in the simple example above, these can be straightforward command strings which simply specify an executable and its arguments. Many times the named executable is just a locally written shell script.

However, alfred also provides some powerful features beyond hierarchical ordering of launches, most notably the binding of remote servers. Some commands, such as netrender, are executed on the local processor but communicate with remote servers. For example, consider a typical (non-alfred) invocation:

	netrender -h antigone -h percival  a.rib
The single RIB file is parceled out to two rendering servers which have been previously configured to wait for rendering requests (i.e. each has a running alfserver). An equivalent alfred invocation would be: 
	Cmd {netrender %H a.rib} -service {pixarNRM} -atleast 2
Given a command and its server requirements, alfred takes care of locating servers of the requested type and launching the command with the names of the hosts it found. In this case, the simple service key expression {pixarNRM} assumes that the master schedule file has been configured to contain a list of server slots on remote hosts, and that the arbitrary keyword "pixarNRM" has been attached to some of these slots.

Note: the schedule file also allows the site administrator or project coordinator to restrict server access to certain users at certain times of day, and it defines the relative priority of groups of users on subsets of the servers. The Alfred user-interface provides schedule editors which modify these settings.

When the dispatcher is processing a job and it encounters a Cmd like the one above which specifies a service key, it requests an available slot from the maitre-d, which in turn searches for slots that match the service keys. So, these service key expressions are used to filter through the list of remote server slots which are available to the requesting user, and find one which matches the Cmd requirements.

The schedule file entry for each remote server contains a field called Selection Keys, which is just a blank-delimited list of words. The keywords chosen are arbitrary, but obviously they must be coordinated between the script writers and the schedule administrators. A typical use for service keys is to indicate the type of software (server) running on the associated remote host. Pixar applications which create Alfred scripts, such as MTOR, assume the use of a few descriptive service key conventions.

Sometimes it is also useful to include keys which describe the hardware or operating system of the remote system as well, since some applications only run on certain types of machines. Keys are also a handy way to indicate project "ownership" of particular servers. System attributes which vary over time, such as resource utilization, are typically treated using dynamic server metrics rather than static keys.

Service keywords can also be combined using a simple logical-expression syntax. For example:

	Cmd {ralf %h render a.rib} -service {pixarNRM|pixarRender}
The expression "pixarNRM|pixarRender" indicates that this command requires a remote server whose description contains either the keyword "pixarNRM OR pixarRender". Use comma "," for logical AND; use exclamation-point "!" for logical NOT; and parenthesis "()" for grouping. String matches are case insensitive.

For example, consider a schedule file which has the following Services defined (among others):

    Name    host    UDI  Selection Keys
   [Gumby] [iris42] [5] [pixarRAT pixarRender SGI big fast FieldTrip]
   [Pokey] [sun007] [1] [pixarRender pixarNRM Sun fast NightWalrus]
and let's say you have an alfred script with the following command in it:
   Cmd {someApp %h args} -service {NightWalrus,(big|fast)}
the maitre_d will first look at Gumby (since it has a higher Universal Desirability Index, 5 vs. 1) and check if it has the "NightWalrus" service key and either the "fast" or "big" keys. Gumby doesn't, but Pokey does, so it gets bound.

Note: in addition to the keywords specified in the schedule, each server slot always has two additional implicit keywords: the name of the slot, and the name of the host. This feature is seldom used, but by specifying a slot or host name as the the service key, it allows a script to restrict execution of a particular command to a predefined host (or slot on a host), rather than allowing alfred to pick the best slot from all available servers.

A special scoping directive can be added to the -service specification to limit the selections to Services defined on the dispatcher's host (i.e. the local host, from the dispatcher's point of view).

   Cmd {netrender -f -Progress some.rib} -service {local:pixarNRM}
The local: prefix requests the local host. Note that this command will check out the matching Service (on the local host), and so other concurrent commands that need the same Service will block waiting for it. This is different from a Cmd with no service specification, which just executes locally as soon as it is encountered during job traversal. The -tags option can be used (with the alfLimitLocal configuration) to limit the number of concurrent executions of commands with the same tag; for example ator uses -tags intensive on some of its CPU-intensive operations to limit how many are running simultaneously (the default alfred.ini file sets alfLimitLocal(intensive) to 1).

Note: It is important to remember that the Cmd directive does not actually send work to remote hosts. It launches local applications, such as netrender or rsh, which then manage interactions with a remote server themselves. If there is a copy of Pixar's alfserver running on the remote server machines, then RemoteCmd can be used to cause direct remote execution to occur with no local client.

The following symbols are expanded at launch time when they occur within launch or message expressions:

~ home directory expansion, as in csh(1)
%h substitute the hostnames bound to the current Cmd via the dynamic -service mechanism. This is a simple blank-delimited list of hostnames (useful for rsh, etc).
%H like %h but formatted as -h hostname pairs (as required by netrender).
%n converted to the count of bound slots; an integer indicating how many slots were bound to this command.
%s converted to the currently bound service names, this is the name of the service (slot) from the schedule which matched the current Cmd's -service request (as distinct from the name of the host on which the service resides). The schedule allows multiple service slots to be defined on each real host, typically to allow each CPU on a multiprocessor system to be scheduled separately.
%r converted to the "retry state" of the current command.   Alfred will substitute a zero or a one in place of the %r depending on whether the launch is the first attempt (0), or a retry (1) of a previously failed or ejected command.
%x execution state, intended for use in Cmds in the -cleanup section of a task. It is sometimes useful for a clean-up Cmd to know the context in which it is being executed, the %x symbol will be converted to one of the following strings: Active, Done, Delete or Restart (there may be additional states added in the future). Commands in the regular Task -cmds section will only execute when the Task is Active; clean-up commands are typically only invoked either when all the regular commands are finished (the Done state). Clean-ups of in active Task trees will also be run when a job is being deleted while still running (the Delete state). The Restart state arises when a shared-server sub-DAG (Task -service) encounters an error and must be reset. The Done and Delete states also apply to commands in the Job -cleanup section.
%b host binding history, intended for use in Job -cleanup commands, this symbol is replaced with a blank-delimited list of all the hosts ever bound to the current job; each hostname will appear only once.
%B server binding history, intended for use in Job -cleanup commands, this symbol is replaced with a blank-delimited list of all the server slot names ever bound to the current job; each name will appear only once. See the notes on %s, above.
%J expands to the batcave SQL Job "jid" for the current job.
%j expands to the internal dispatcher job-id for the current job. Note that these are not globally unique.
%t expands to the Task "tid" for the current task. While not globally unique, it is unique within the job, and is used both internally by alfred and in the batcave SQL tables (as "tid").
%c expands to the batcave SQL Cmd "cmdid" for the current Cmd/RemoteCmd.
%idref(host) like %h but using the hostnames from the command whose -id value is idref
%idref(-host) as above, formatted as -h hostname pairs
%idref(result) substitute the result string from the command whose -id value is idref
%D(path) DirMaps, apply per-architecture remapping of paths using a site-defined mapping table.
%% a single percent-sign is substituted


Sharing One Server Check-Out Among Several Commands
Sometimes it can be useful to acquire a remote server and run several commands in sequence on that server. This is called a shared server scenario. The mechanism for accomplishing this goal is to add a server check-out specification to the enclosing Task and then reference the task-id on each Cmd or RemoteCmd that needs the server slot. For example:

  Task {SharedServer example} -id {bob} -service {pixarRender} -cmds {
	Cmd {rsh %h mkdir /tmp/somedir} -refersto {bob}
	Cmd {rsh %h render -Progress some.rib} -refersto {bob}
  } -cleanup {
	# Clean-up commands go here.
	# This example uses RemoteCmd just to illustrate its use as an
	# alternative to rsh above.  Note that there is an implicit '%h'
	# used to determine where the command should be run.
	RemoteCmd {/bin/rm -rf /tmp/somedir} -refersto bob
  } -subtasks {
	# the description of any nested dependent tasks go here
The lifetime of the check-out is governed by two factors. The initial task-level check-out is done lazily in the sense that it only occurs when one of the commands that references it actually becomes the next command to be executed. A reference count is used to determine when it is safe to check the slot back in, it is freed when the last command to reference it has completed or errored out.

Syntax note: the characters allowed in the idref names are letters, numbers, period, and underbar. For all of the % substitutions above, braces may be used delimit names, as in csh or Tcl; for example, if the current schedule has a service of type "renderserver" defined for the host named "darwin", then a Task containing this command:

Cmd {renderer -use %{h}.profile} -service {renderserver}
would cause the following command to be launched:
renderer -use darwin.profile
The braces are required in this case because the simpler string "%h.profile" would cause the substitution mechanism to look for a Task or Cmd with "-id h.profile" and use it's current values.  


Special Characters and Escapes in Alfred Scripts
It is important to remember that alfred makes two, very different, passes through the scripts submitted to it. The first spool-time parsing pass is used to construct the "shape" of the job; it sets up the hierarchy of tasks. The second pass (and subsequent identical passes) is the dispatching, run-time, pass during which commands are launched and run-time substitutions are made, etc.

The initial parsing of the alfred script is done using a system built around John Ousterhaut's TCL interpreter. As a result, the TCL syntax rules apply to command arguments and string formation. Alfred scripts consist of calls to the Job, Task, Cmd, and Instance commands (operators), which in turn take strings as arguments. In the case of Task, the -subtasks option can contain additional script which is parsed recursively. In general the TCL rules, as they apply to alfred operator arguments, are much simpler than those for the Bourne shell (sh) or csh(1). Shell programmers should be aware of several things:

For example, consider using find(1) to remove files which start with "preview" from a directory tree. In a csh(1) script the command might look this way:
   find ~bob -type f -name 'preview.*' -exec /bin/rm {} \;
that is, the asterisk must be escaped from the shell filename expansion because it is used directly by the find command; similarly with the semicolon after the exec expression. The curly braces are untouched by the shell (since they're not part of a variable expression). If a similar command was part of an alfred script (as a Task's clean-up command for example), it would look like this:
   Cmd {find ~bob -type f -name preview.* -exec /bin/rm \{\} ;}

Shell Pipelines and other Expressions
Given the restrictions just described above on Cmd launch expressions there are nonetheless occasions when shell constructs, such as command pipelines or run-time filename expansion, can be very useful. In these situations, a simple solution is to launch an appropriate shell as the Cmd, passing the pipeline expression to the shell via command-line arguments

   Cmd {/bin/sh -e "cd /tmp/frames; ls -1 | xargs -I+ cp + /DDR"}
Script authors should read the documentation for their shell of choice to understand the implications of various invocation options. For example, you must decide whether the user's .cshrc or .profile should be executed when the dispatcher launches a command like the above.

Another approach is to use the Cmd -msg option to pass arbitrary expressions to a launched shell. This is essentially equivalent to the above approach, but not as compact, however it does allow for persistent reuse of the shell, if that's desired. Consider the following examples which send mail, which can sometimes be handy at the end of job. Recall that the -s option to Mail looks for the next "word" as the subject, so spaces need to be escaped (using csh(1) syntax this time):

   Cmd {/bin/csh -fc "/usr/sbin/Mail -s 'job done' jean < ~/errlog"}
   Cmd {/bin/csh -fet} -msg {/usr/sbin/Mail -s 'job done' jean < ~/errlog}
Often the SIMPLEST SOLUTION for complex expressions is to write a short shell script in your favorite language and launch the script from alfred:
   Cmd {myscript}
If a remote server is required, the run-time host selection can be passed to the script as an argument:
   Cmd {myscript %h} -service {someServerType}

See the notes on writing alfred-compliant application programs for details on the behavior requirements for apps launched by alfred.  



Assign: Initializing Global Job Variables

The Job -init {initializations} block can be used to define variables which have global scope with respect to the Tasks of the job. Typically these variables are just used to make the job script more compact by using them instead of frequently occurring (long) filenames. The init-block can contain multiple Assign statements, for example:

   ##AlfredToDo 3.0

   Job -title {A Job} -init {

      Assign frmdir {/usr/people/buzz/projects/toys/ribfiles}

   } -subtasks {

      Task {Frame 0} -cmds {
         Cmd {render $frmdir/frame.0.rib}

      Task {Frame 1} -cmds {
         Cmd {render $frmdir/frame.1.rib}

      [... etc ...]
the Assign statement takes two arguments: the name of the variable and its string value.




Using Iterate to Simplify Repetitive Scripts

Many scripts have a repetitive structure, especially those used to render a sequence of frames from the same shot. The Iterate operator can often be used to simplify the construction of these scripts: basically a template of the repetitive structure is described just once, and it is replicated by Alfred while the job is executing. Here's a simple example:

   ##AlfredToDo 3.0

   Job -title {A Repetitive Job} -subtasks {

      Task {Environment.0} -cmds {
         Cmd {render environment.0.rib}

      Iterate frame -from 1 -to 100 -by 1 -subtasks {
         Instance {Environment.0}
      } -template {
         Task {Frame $frame} -cmds {
            Cmd {render Frm.$frame.rib}

This job will render 100 RIB files after rendering their shared environment map. This particular example assumes that all the RIB files already exist, and that they are named Frm.1.rib, Frm.2.rib, etc. The Iterate operator defines an arbitrary variable name, in this case "frame", which is incremented from 1 through 100 in steps of 1. During each iteration cycle the script in the template block is copied into the job execution tree, and each reference to "$frame" is replaced with the variable's current value.



On-Demand Processing, Walk-Ahead Limits, and Thwarting

It is important to note that this copy procedure occurs incrementally while the job is executing, not all at once when the job script is initially submitted. The template code is dynamically inserted into the the job's execution tree when the dispatcher determines that it is appropriate to generate more work for itself.

This determination is based on the walk-ahead metric which is a measure, at a particular task node, of all the ready-to-execute leaf nodes which precede it. Recall that alfred traverses a job tree in depth-first order looking for tasks which have no outstanding subtasks (dependencies). At any given moment, all such leaf-nodes can potentially be launched in parallel, although typically there are scarce resources such as remote servers or tag limits which restrict the number of active tasks.

An Iterate node will perform one substitute-and-copy cycle when the number of unlaunched leaf nodes preceding it, in the top-to-bottom, depth-first sense, is less than the user-defined walk-ahead limit, which is set in the preferences dialog. Since an Iterate cycle usually results in additional unlaunched nodes, the next cycle is delayed by the walk-ahead test until the job can catch up. When a node is delayed in this way it is said to have been thwarted.

This gating behavior is particularly useful when the template script contains tasks which generate, render, and then clean up RIB input files on a per-frame basis. The disk space footprint will be approximately (RIB_file_size) x (walk_ahead_limit).

Formatting During Substition and Leading Zero Padding

Sometimes it's useful to apply special formatting to the iteration variable's value during the template substitution. This is especially true of frame numbers which often want to be padded with leading zeros so that the resulting file names will sort properly. Of course, in the case of frames rendered from RIB files the output file name is embedded in the RIB stream and therefore isn't Alfred's "problem". For other cases in which formatting is important (for example if the input file name is zero-padded), Alfred does not provide a direct formatting mechanism but instead relies on the built-in Tcl interpreter to format values as the substitution is taking place. In the particular case of padding with leading zeros, use a construction such as the following:
    Iterate fn -from 9 -to 15 -by 1 -template {
        Task {Frame $fn} -cmds {
            Cmd {render Frm.[format "%0.5d" $fn].rib}
Which will render the RIB files named:   Frm.0009.rib, Frm.0010.rib,  etc.
See the Tcl format command documentation for more details.

Iterate Nodes and Job Percent-Done

When an Iterate node cycles it adds new tasks to the job tree. In some circumstances this can cause the job progress estimates to fluctuate dramatically. The best way to achieve smooth estimates is to specify an appropriate value for the "Job -etalevel" parameter. This causes time and percent-done estimates to track a particular level of the task hierarchy. The topmost tasks are at level zero, deeper nodes have numerically higher values. Pick the level at or hierarchically above your iterate nodes rather than a deeper level.

In the short job example above the default level of zero will track the iterated nodes properly.

Note that when a job has only one level-zero node   - a sort of master task with frame tasks below it -   then you should track the subtasks, e.g. level 1, since progress estimates based on the single level-zero node are unlikely to be accurate.  



Conditional Cmd Launches using if Expressions

The Cmd  -if {expression} conditional test is useful in situations where certain individual commands may not need to be executed. For example:

    Cmd {render backdrop.rib} -if {[file exists backdrop.rib]}
The expression string is evaluated using the Tcl expr command. If the expression produces any non-zero value then the conditional test passes (it is "true") and the Cmd will launch as usual. An expression which evaluates to zero causes the Cmd to be skipped, it is not launched and is marked "done" as if it had completed successfully.

The Tcl file command, used in the example above, can also be used to construct other kinds of tests related to files.

Alfred adds a utility command "fnewer" which can be used in if-expressions to compare file modification times. For example, if an artist is running many test renders of foreground elements over a fixed background, it might be useful to only render the background geometry once, during the first job of the day, or if the RIB-file is updated:

    Cmd {render backdrop.rib} -if {[fnewer backdrop.rib backdrop.tif]}
the rendering will proceed if the RIB file is newer than the TIFF file, otherwise it is skipped. The fnewer command returns "0" (false) if the first file is older than the second. If the second file can't be found it returns "1" (true), so that in cases like this the file will be created; if the first file doesn't exist, fnewer returns "-1", and since this is also non-zero the launch is attempted anyway (which might result in a error).

Note that both the curly and square brackets are typically required in these expressions, as described in expr(n) and more generally in Tcl(n).  



Referencing results from previous commands

Alfred provides a limited mechanism by which commands can pass information to each other. For example, one command may generate RIB and choose a temporary filename for its results, and another command needs to know the filename to render it. This scheme has two components; first, a launched app writes a single line to its standard-out of the form:

	ALF_RET text text text
Alfred scans the output of commands under its control looking for lines beginning with 'ALF_RET', if it finds one it stores the rest of the line in a local buffer associated with the command. The second step is that subsequent commands may refer to these result strings in their launch and message expressions using the %idref(result) substitution syntax. The first Cmd must be named with the -id option, and the second must declare its intent to use the value with the -refersto option. For example:
  Task {command result example} -subtasks {
	# the description of any nested dependent tasks go here
  } -cmds {
	# these are the commands for this task, executed in sequence
	Cmd {someApp -genrib 2} -id Frm2
	Cmd {netrender -f -Progress %H %Frm2(result)} \
		-service {nrmserver} -refersto Frm2
  } -cleanup {
	# clean-up commands go here
	Cmd {/bin/rm -f %Frm2(result)} -refersto Frm2
This assumes of course that 'someApp' actually writes an ALF_RET line to stdout which names the RIB file! As an aside, these three commands are executed serially, each one is launched only when the previous one has completed successfully. In this sense the worklist is locally a simple script. Parallel execution happens only among commands from different, non-dependent, Tasks. So, commands specify work to be done, Tasks are containers which specify command ordering; also, Tasks are the objects which are represented in the user-interface diagram of the running job. As a different aside, RIB generation can often be a lengthy process, and a Task structured as in the example above will appear 'Active' (in the monitor) during the RIB generation as well as during the rendering.

Progress or percent-done indication

Alfred also scans the stdout of its launched apps for strings of this form:
The integer nnn, in the range 0-100, is sent to the UI and used to control the percent-done bars drawn on active task nodes.  

Exit status

Alfred also scans for this directive:
The integer nnn is used to determine success or failure instead of the actual program exit status (0 indicates successful completion, any non-zero value indicates failure and results in a blocking error).


The program called ralf ships with alfred; it functions like the remote shell rsh(1), except that it also provides a means of retrieving the exit status of the remotely launched app, and it will transmit SIGTERM signals to the remote side.


Pixar Animation Studios
(510) 752-3000 (voice)   (510) 752-3151 (fax)
Copyright © 1996- Pixar. All rights reserved.
RenderMan® is a registered trademark of Pixar.