SPECaccel® 2023 Config Files

$Id$ Latest: www.spec.org/accel2023/Docs/

I. Config file basics

A. What is a config file?

B. Benchmark selection

C. Five consumers

D. Syntax

1. Sections and scope

2. Comments

3. Whitespace

4. Quoting

5. Line continuation

6. Included files

7. Section markers

E. Section merging

Precedence Examples
1. Benchmark specifier
2. Tuning specifier
3. Label specifier
4. Combining sections
5. Conflicting sections
6. Section order
7. Repeated Sections

F. Variables

1. Defining variables

2. $[square] substitution

3. Useful $[variables]

4. ${curly} interpolation

5. Useful ${variables}

6. Unsetting "%undef%"

7. Debug tips

All variable types

$(MAKEVAR)
\$SHELLVAR
$[startup]
${during_run}/$during_run
%{macro}
%{ENV_var} New with SPECaccel 2023

II. Config file options for runaccel

A. Precedence: config file vs. runaccel command line

B. Options

action allow_label_override backup_config basepeak bind check_version command_add_redirect delay deletework difflines enable_monitor env_vars expand_notes expid fail fail_build fail_run flagsurl force_monitor http_proxy http_timeout ignore_errors ignore_sigint info_wrap_columns iterations keeptmp line_width label locking log_line_width log_timestamp mail_reports mailcompress mailmethod mailport mailserver mailto make make_no_clobber makeflags mean_anyway minimize_builddirs minimize_rundirs no_input_handler no_monitor nobuild notes_wrap_columns notes_wrap_indent output_format output_root pmodel New preenv ranks rebuild reportable runlist save_build_files section_specifier_fatal setprocgroup size src.alt strict_rundir_verify table teeout threads tune use_submit_for_speed verbose verify_binaries version_url

III. Config file options for specmake

A. Commonly used Make variables

CC, CXX, FC
PORTABILITY (+ several similar options)
OPTIMIZE (+etc.)
EXTRA_CFLAGS (+etc.)
LIBS
PASS1_OPTIMIZE (+etc.)

B. Parallel Model Selection

C. Creating your own Make variables

D. += is available but use with caution

E. Using buildsetup to create a sandbox

F. About Automatic Rebuilds

{C|CXX|F}C_VERSION_OPTION New and required with SPECaccel 2023

IV. Config file options for the shell

A. \$SHELLVAR variable substitution

B. Options

bench_post_setup build_pre_bench build_post_bench mointor_X post_setup submit

C. Using submit

1. Basic usage

2. Reporting

Submit Examples:

1. mpirun

2. bind list

3. Generating scripts and command_add_redirect

4. Generating a script for numactl

5. Line continuation

6. Magical notes_submit

V. Config file options for human readers

A. Descriptive fields

hw_accel_connect hw_accel_desc hw_accel_ecc hw_accel_model hw_accel_name hw_accel_type hw_accel_vendor hw_avail hw_cpu_name hw_cpu_char hw_cpu_mhz hw_cpu_max_mhz hw_disk hw_fpu hw_memory hw_model hw_nchips hw_ncores hw_ncoresperchip hw_ncpuorder hw_nthreadspercore hw_ocache hw_other hw_pcache hw_power_{id}_cal_date hw_power_{id}_cal_label hw_power_{id}_cal_org hw_power_{id}_met_inst hw_power_{id}_connection hw_power_{id}_label hw_power_{id}_model hw_power_{id}_serial hw_power_{id}_setup hw_power_{id}_vendor hw_psu hw_psu_info hw_scache hw_tcache hw_temperature_{id}_connection hw_temperature_{id}_label hw_temperature_{id}_model hw_temperature_{id}_serial hw_temperature_{id}_setup hw_temperature_{id}_vendor hw_vendor license_num prepared_by sw_accel_driver sw_avail sw_base_ptrsize sw_compiler sw_file sw_os sw_state sw_other sw_peak_ptrsize tester test_sponsor

notes_comp_NNN notes_port_NNN notes_base_NNN notes_peak_NNN notes_submit_NNN notes_os_NNN notes_plat_NNN notes_part_NNN notes_NNN

2 Note numbering

3 Additional tags

4 Links in notes sections

5 Attachments

VII. The config file preprocessor

A. Introduction

1. Example: Picking Model

2. Syntax Basics

B. Defining macros

1. In a config file

2. On the command line

3. Predefined macros: endian, hostname...
%{ENV_variable_name} New with SPECaccel 2023

4. Example: Adjusting the Environment

5. Redefinition

C. Un-doing macro definition

D. Using macros

E. Conditionals

1. %ifdef .. %endif

2. %ifndef .. %endif

3. %if .. %endif

4. %else

5. %elif

F. Informational directives

1. %warning

2. %error

3. %info New with SPECaccel 2023

4. %dumpmacros New with SPECaccel 2023

VII. Output files - and how they relate to your config file

A. Help, I've got too many config files!

B. The log file

1. Useful Search Strings

2. About Temporary Debug Logs

3. Verbosity levels

D. Help, I've got too many log files!

E. Finding the build directory

F. Files in the build directory

G. For more information

Appendices

A. Other benchmark sets

B. Obsolete features

 

I. Config file basics

I.A. What is a config file?

A SPECaccel config file is a file that defines how to build, run, and report on the SPECaccel benchmarks in a particular environment. It defines how SPECaccel 2023 interacts with your test environment, and lets you describe that environment in human-readable form.

A config file provides detailed control of testing, including:

  • Tuning and portability, which you may customize for your compilers.
  • The runtime environment, which you may customize for your operating system.
  • A description of the SUT (System Under Test), which you should customize to create an accurate report.

Using your customized options, the SPECaccel tools automatically create Makefiles, build the benchmarks, run them, generate reports, and write log files.

Because they collect your options into one place, config files are key to result reproducibility.

For example, if a vendor publishes SPECaccel 2023 results for the SuperHero Model 42 at the SPEC web site www.spec.org/accel2023, it is expected that a customer can demonstrate similar performance using only 3 ingredients:

  1. The SPECaccel 2023 media, ordered from SPEC.
  2. The software and hardware as specified in the full disclosure from the published result.
  3. The config file, which is posted along with the results.

q. This document looks big and intimidating. Where do I start?

a. Don't start here. Start with the Overview and Using SPEC SPECaccel 2023 - the 'runaccel' Command.

After that, please read section I.C carefully, which explains that config files contain options for five (5) different consumers. You need to recognize which options are for which consumers.

Please notice that config files have 3 kinds of sections. You need to know how named sections work.

From that point on, you can probably skip around among topics as they may interest you.

Tip: Most topics can be found by adding '#topic' to the URL for this document. Examples: #consumers  #shell  #readers  #sw_c_compiler 

Still lost? The config file directory contains several example config files (Prefixed as "Example_"). Copy the example that best matches the compiler version you're using to a new config file, and edit from there. The "Example_SUT.inc" file includes an example for the system description fields which you can either copy into your config file, or copy it to a new include file for your particular system.

I.B. Benchmark selection

In a config file, you can reference: One or more individual benchmarks, such as 404.lbm, or entire suites, using the Short Tags in the table below.

Short
Tag
Contents Metrics What do Higher Scores Mean?
accel 12 benchmarks SPECaccel 2023_base
SPECaccel 2023_peak
Higher scores indicate that less time is needed.

I.C. Five consumers

A config file has content for five (5) distinct consumers, as shown in the table.
To understand a config file, you must understand which consumer is addressed at any given point.
Column 3 below provides a few examples for each; click the roman numerals in column 2 for many more.

Consumer List of
options
Examples Brief description
runaccel II ranks
output_format
reportable
threads
pmodel

Options that change how runaccel itself works. Many should be familiar from Using SPECaccel 2023.
Putting your desired options in the config file can simplify the command line. For example, if jimmy.cfg includes the lines:

output_format = text,csv
tune          = base
reportable    = yes
pmodel        = TGT

then the defaults for the runaccel command would change as shown. Both of these would do the same thing:

runaccel --config=jimmy --output=text,csv --tune=base --reportable --pmodel=TGT accel 
specmake III OPTIMIZE
PORTABILITY

Make variables, to control benchmark builds via specmake.

Commonly used specmake variables: section III
Many more: Make Variables
Using specmake on the command line: Utilities

shell IV
post_setup
submit

Commands to be executed by the Unix shell (/bin/sh)
Example:

submit = $[top]/bind.sh $command  

Above, a user supplied "bind.sh" shell script to perform CPU core binding.

Warning: SPECaccel config files can execute arbitrary shell commands.
Read a config file before using it.
Don't be root. Don't run as Administrator. Turn privileges off.

readers V hw_model
notes100
sw_compiler

System Under Test (SUT) description, with enough detail so that the reader can understand what was tested and can reproduce your results. If a SPECaccel 2023 result is published (whether at SPEC or independently) it must be fully described.

preprocessor VII %define
%ifdef
%if
%elif
%error

Preprocessing directives and definitions to adjust your config file prior to use. All preprocessor directives begin in column 1. Example:

%if %{model} eq 'omp_target'
   pmodel = TGT
%elif %{model} eq 'omp_inner_simd'
   pmodel = SMD
%else
# Model not defined for OpenMP, use default 'loop'  
   pmodel = LOP
%endif
      

I.D. Config file syntax

I.D.1. Sections and scope

A config file contains: a header section, named sections, and a HASH section.

Scope: Every line is considered to be within the scope of one of these three. Lines prior to the first section marker are in the scope of the header section. All other lines are either in the scope of the most recently preceding section marker or in the HASH section.

Sections Description Example
header section

The header section is the first section, prior to the any named section. Most attempts to address runaccel itself are done in the header section. In the example, lines 1 through 6 are in the header section.

$ cat -n hiroshi.cfg
 1  flagsurl             = $[top]/config/flags/gcc.xml
 2  iterations           = 1
 3  output_format        = text
 4  runlist              = 404.lbm
 5  size                 = ref
 6  tune                 = peak
 7  openacc=peak:
 8     CC_VERSION_OPTION = -v
 9     CC                = gcc
10     OPTIMIZE          = -O2
11  404.lbm=peak:
12     OPTIMIZE          = -O3
13  
14  __HASH__
15  404.lbm=peak=none:
16  # Last updated 2021-02-06 14:29:40
17  opthash=ff6059d6d9ec9577f7f49d05178c58688f31004089
18  baggage=
19  compiler_version=\
20  @eNo1jbEOgjAYhPc+RUcdaFEJIWwGHEhQjBLjRn7LLzapLWkL8
21  kpP9OCqkdX070hmtk0bTjMUspiuhQA9RFgfDki3brEkLdkCf0y
22  0NO36VHldDROLqTSzoNS2JfS5pT/DqUAH54cv4tAsjDEC6M9au
23  FuH/CZ+c5Q+pyRd+tUlX
24  compile_options=\
25  @eNp1T11PgzAUfe+vaPrOMjNfJGMJlLqhQBtLX3xptHYGBWoKM
26  vNp93Vjs3obadX2I+sHXZtD+0D3VXr9bX+8/InJBEMAeKBFZLq
27  KgWj06epUADLlCVqe+rFquJaKiHumJSaC1YWAuOAr/DPWvfu4I
28  mALF6zzeSpj96+RITFhJd3rG/dMaQTzEIJYV2T0DJl8RlGfl7Z
29  hjyd8vw+D9MirnY6z5LRW9NOC1yNkc/OfAJtxnuD
30  exehash=5290fe504238c6de1a13e275ab8ca11e035fbb4e7e
31  
$ 
         
  or
default:

Options for the header section may also be entered in section

default=default=default:

or a shorter spelling thereof, such as

default:

This can be helpful if you need to effectively return to the header section, perhaps when using include.

Tip: Nevertheless, it is usually easier to maintain a config files that keeps all runaccel options near the top, preferably in alphabetical order.

Named sections

A named section is a portion of the config file that follows a line that contains a section marker.

Briefly, a section marker is a 1-to-3 part string with equals signs in the middle and a colon at the end; see detail below. The example has 2 named sections, delimited by 2 section markers on lines 7 and 11.

Notice that the example sets OPTIMIZE in both of the named sections. To understand which one gets used, see the precedence rules, which describe how sections interact.

HASH section

The HASH Section is the final, automatically-generated section. It starts with __HASH__ and contains checksums. The example starts the HASH section at line 14. (For readability, the lines are truncated on the right.)

The automatically-updated checksums ensure that binaries match their config file. You can optionally disable checking, but doing so is strongly discouraged. See verify_binaries.

Config files printed by --output_format=config do not include the HASH section.

I.D.2. Comments

Comments begin with '#'. There are two types:

Syntax Type Saved? Detail
# Regular Yes Regular comments can be full-line or trailing. A copy of your config file is saved with the test results. If you submit your results to SPEC for publication, the regular comments can be read by anyone.
#> Protected No

A full-line comment that begins with #> is a protected comment and will not be saved with your results. You can use protected comments for proprietary information, for example:

#> I didn't use the C++ beta because of Bob's big back-end bug.  

Both types of comments are ignored when processing the config file.

Full-line comments: If # is the first non-blank item on a line, then the whole line is a comment. Comment lines can be placed anywhere in a config file.

Trailing comments: If a line has non-blank items, you can (usually) add regular comments. You cannot write a protected trailing comment. If you try to use a protected comment after some other element, it is treated as a regular comment.

All comments below will be saved except the one that says 'NOT saved'.

                       # New optimizers.
default=base:          # Most optimizers go up to ten.
   OPTIMIZE   = -O11   #    These go to eleven.
                       #> This comment is NOT saved
   COPTIMIZE  = -std   #> This comment is saved

Not a comment: These instances of # do not start a comment:

I.D.3. Whitespace

Blank lines can be placed anywhere in a config file. They are ignored.

Spaces at the beginning of lines are ignored, with the exception that preprocessor directives always begin with a percent sign (%) in the first column. You can put spaces after the percent sign, if you wish (example).

Spaces within a line are usually ignored. Don't try to break up a section marker, and you can't break up a single word (say 'OPTIMIZE' not 'OPT I MIZE'). If multiple spaces separate line elements, it is as if there were only one space. Each of these have the same meaning:

   OPTIMIZE=-O2 -noalias
   OPTIMIZE   =  -O2      -noalias

One place where spaces are considered significant is in notes, where you can use spaces to line up your comments. Notes are printed in a fixed-width font.

Trailing spaces and tabs are ignored, unless they are preceded by a backslash.

I.D.4. Quoting

If you use double (") or single (') quotes within a config file, runaccel leaves them alone. The assumption is that you put them there because one of the consumers (such as a shell) needs them. The quotes are not significant to runaccel but may be highly significant to the consumer. See the section on quote traps.

If you use a backslash (\) it is usually not significant. The exceptions are:

I.D.5. Line continuation

Many fields, including most reader fields, can be continued by adding a number:

node_compute_hw_tcache000= 256 MB I+D on chip per chip
node_compute_hw_tcache001 = 16 MB shared / 4 cores
node_compute_sw_os1 = SUSE Linux Enterprise Linux Server 12
node_compute_sw_os2 = 4.12.14-94.41-default

The fields which cannot be continued are the ones that are expecting a simple integer, such as node_hw_nchips and license_num; and the ones that expect a date, such as hw_avail. You can pick your own style of numbering, as in the examples above. (Note: the stored results from your test always use three-digit numbers, and have slightly different syntax, as discussed in utility.html.)

Shell-style "here documents" with double angle brackets and a delimiter word (e.g. <<EOT) can be used to set multi-line values. Backslash-continued lines are also supported. For example:

$ cat continued_lines.cfg
expand_notes   = 1
output_format  = text
output_root    = /tmp/fake_lines
runlist        = 404.lbm

here_continued = <<EOT
      + This is +
       + a test  +
EOT

backslash_continued = + So is +\
 + this  +

notes1 = ${here_continued}
notes2 = ${backslash_continued}
$ cat continued_lines.sh 
runaccel --config=continued_lines --fakereport | grep txt
grep '+' /tmp/fake_lines/result/accel2023*txt
$ ./continued_lines.sh 
    format: Text -> /tmp/fake_lines/result/accel2023.001.ref.txt
           + This is +
           + a test  +
     + So is +
     + this  +
$   

I.D.6. Included files

You can include other files in your config file using include:
Multiple files may be included.
Included files may use macros (and you can use configpp to check the effect).
Included files may write to arbitrary sections, including (effectively) the header section.

Example: a config file is developed on one system, and applied on a different System Under Test (SUT). The compilers are, of course, installed on the development system. They might not be present on the SUT.

The compiler is described in the main config file on lines 10-12.
The hardware is described in SUT.inc which is included via line 6.

$ cat -n include.cfg
 1  iterations    = 1
 2  output_format = text
 3  output_root   = /tmp/example
 4  runlist       = 404.lbm
 5  size          = test
 6  test_sponsor    = Turboblaster
 7  tester          = Turboblaster
 8  include: SUT.inc
 9  default:
10    CC                = gcc
11    CC_VERSION_OPTION = -V
12    sw_compiler000   = C/C++/Fortran: Version 12.2 of
13    sw_compiler001   = GNU Compiler Collection
14    sw_avail         = Aug-2022
$

Note the software date (sw_avail) on line 14 above.
Today's SUT is the Turboblaster SuperHero IV.
Copy its information to the expected place.

$ cp Turboblaster.inc SUT.inc
$ cat Turboblaster.inc 
default:
node_compute_label = TurboBlaster 5000
node_compute_order = 1
node_compute_count = 2
node_compute_purpose = compute
node_compute_hw_vendor = Mega Technology
node_compute_hw_model = Turblaster 5000
node_compute_hw_cpu_name = Turbo CPU
node_compute_hw_ncpuorder = 1 chips
hw_avail  = Feb-2023
$

Note the hardware date (hw_avail) in the include file.
Both dates are referenced in reports:

$ cat include.sh 
runaccel --config=include | grep txt
grep avail /tmp/example/result/*txt
$ ./include.sh 
format: Text -> /tmp/example/result/accel2023.001.test.txt
Test sponsor: Turboblaster      Hardware availability: Feb-2023
Tested by:    Turboblaster      Software availability: Aug-2022
$ 

I.D.7. Section markers and Named sections

A named section begins with a section marker and continues until the next section marker or the HASH section is reached.

Named sections can be entered in any order.
Section markers can be repeated. Material from repeated sections is automatically consolidated.

A section marker is a one- to three-part string of the form:

   benchmark[,...]=tuning[,...]=label[,...]:

The three parts of a section marker are called the section specifiers, with allowed values:

Section
specifier
Allowed values
benchmark default
A metric: accel
Any individual benchmark, such as 404.lbm
A list, such as: 450.md,455.seismic,463.swim
tuning default
base
peak
A list of tuning levels: base,peak
label default
Any specific label: an arbitrary tag to identify binaries and directories
A list of labels, separated by commas

Trailing default section specifiers may be omitted from a section marker. In the pairs below, in each case, the second line is equivalent to the first:

accel=default:            404.lbm=base=default:      default=default:
accel:                    404.lbm=base:              default:

I.E. Section merging and precedence rules

By constructing section markers, you determine how you would like your options applied. Benchmarks are built according to instructions in the sections that they match, subject to rules for combining sections and resolving conflicts among them. Sections are combined using these rules.
Click down arrow to go to an example.

  • down arrow For the benchmark specifier, the precedence is:

    highest   named benchmark(s)
              suite name 
    lowest    default 

    (See also Appendix A)

  • down arrow For the tuning specifier, base or peak has higher precedence than default.
  • down arrow For the label specifier, any named label is at a higher precedence level than default.
  • Combining sections:
    1. down arrow Combine sections that apply to a benchmark, if there is no conflict among them.
    2. down arrow If sections conflict with each other, the order of precedence is:

      highest     benchmark
                  tuning
      lowest      label 
  • Section order:
    • down arrow For sections at differing precedence levels, order does not matter.

    • down arrow If a section occurs more than once, the settings are combined. If there are conflicts, the last instance wins.

Precedence Example 1: benchmark specifiers

For the benchmark specifier, the precedence is:

  highest   named benchmark(s)
            suite name
  lowest    default

The flagsurl line picks up definitions that are provided with SPECaccel 2023.

404.lbm is an accel benchmark, but it does not use the setting on lines 12-14. Instead, it uses the higher precedence lines 10-11 (named benchmark).

456.spF is an accel benchmark. The highest precedence section for it is on lines 15-17.

450.md is an accel benchmark. It gets the OPTIMIZE setting from lines 12-14.

$ cat -n precedence_example1.cfg
     1  flagsurl             = $[top]/config/flags/gcc.xml
     2  iterations           = 1
     3  output_format        = text
     4  output_root          = /tmp/ptest
     5  runlist              = 404.lbm,450.md,456.spF
     6  size                 = test
     7  default:
     8     CC_VERSION_OPTION = --version
     9     CC                = gcc
    10  404.lbm:
    11     OPTIMIZE          = -O3
    12  450.md:
    13     pmodel            = ACC
    14     OPTIMIZE          = -O2 -fopenacc
    15  456.spF:
    16     pmodel            = TGT
    17     OPTIMIZE          = -O1 -fopenmp
Precedence Example 2: tuning specifiers

For the tuning specifier, base or peak has higher precedence than default.

The first few lines of the config file use similar features as the previous example.

The tuning from line 12 is used for base, and line 14 for peak.

$ cat -n precedence_example2.cfg 
     1  flagsurl             = $[top]/config/flags/gcc.xml
     2  iterations           = 1
     3  output_format        = text
     4  output_root          = /tmp/ptest2
     5  runlist              = 404.lbm
     6  size                 = test
     7  default=default:
     8     CC_VERSION_OPTION = --version
     9     CC                = gcc
    10     OPTIMIZE          = -fopenacc
    10  default=base:
    11     OPTIMIZE          += -O1
    12  default=peak:
    13     OPTIMIZE          += -O3
$ cat precedence_example2.sh 
runaccel --config=precedence_example2 --tune=base | grep txt
runaccel --config=precedence_example2 --tune=peak | grep txt
cd /tmp/ptest2/result
grep 'O[0-9]' *txt
$ ./precedence_example2.sh 
    format: Text -> /tmp/ptest2/result/accel2023.001.acc.test.txt
    format: Text -> /tmp/ptest2/result/accel2023.002.acc.test.txt
accel2023.001.acc.test.txt: 404.lbm: -fopenacc -O1
accel2023.002.acc.test.txt: 404.lbm: -fopenacc -O3
$
Precedence Example 3: label specifiers

For the label specifier, any named label has higher precedence than the default.

This config file is simpler than the previous examples, because we don't even bother to run it; instead, --fake is used.

The runaccel command uses --label=OhTwo. Therefore, the default setting for OPTIMIZE on line 6 is over-ridden by the settings on lines 7-8.

$ cat -n precedence_example3.cfg 
 1  runlist              = 404.lbm
 2  size                 = test
 3  accel=base=default:
 4     CC_VERSION_OPTION = --version
 5     CC                = gcc
 6     OPTIMIZE          = -O0 -fopenacc
 7     pmodel            = ACC
 8  accel=base=OhTwo:
 9     OPTIMIZE          = -O2 -fopenacc
$ cat precedence_example3.sh 
runaccel --config=precedence_example3 --fake --label=OhTwo | grep lbm.c
$ ./precedence_example3.sh 
gcc -c -o lbm.o -DSPEC  -DSPEC_OPENACC -DNDEBUG -O2 -fopenacc lbm.c
$  
Precedence Example 4: Combining sections

Combine sections that apply to a benchmark, if there is no conflict among them.

Note that line 1 sets the label, and line 3 sets the tuning.

All sections -- including lines 6, 8, 10, and 12 -- contribute to the compile command, which has been wrapped for readability.

$ cat -n precedence_example4.cfg 
  1  label                = wall
  2  runlist              = 404.lbm
  3  tune                 = peak
  4  default:
  5     CC_VERSION_OPTION = --version
  6     CC                = gcc
  7  accel:
  8     OPTIMIZE          = -O1 -fopenmp
  9  default=peak:
 10     COPTIMIZE         = -ftree-vectorize
 11  default=default=wall:
 12     EXTRA_COPTIMIZE   = -Wall
$ cat precedence_example4.sh  
runaccel --config=precedence_example4 --fake | grep lbm.c
$ ./precedence_example4.sh 
mpicc -c -o lbm.o -DSPEC -DSPEC_OPENMP -DNDEBUG -O1 -fopenmp -ftree-vectorize -Wall  lbm.c
$  
Precedence 5: Conflicting sections

If sections conflict with each other, the order of precedence is:

highest     benchmark
            tuning
lowest      label 
Precedence Example 6: Section order

For sections at differing precedence levels, order does not matter.

These two config files use a different order but perform the same functions when runaccel applies them..

$ diff --side-by-side precedence_example6a.cfg precedence_example6b.cfg
iterations           = 1                            iterations           = 1
output_format        = text                         output_format        = text
output_root          = /tmp/ptest                   output_root          = /tmp/ptest
runlist              = 404.lbm                      runlist              = 404.lbm
size                 = test                         size                 = test
default:                                            default:
   CC_VERSION_OPTION = --version                       CC_VERSION_OPTION = --version
   CC                = gcc                             CC                = gcc
404.lbm:                                          <
   OPTIMIZE          = -O3                        <
accel:                                            <
   OPTIMIZE          = -O1                        <
default:                                            default:
   OPTIMIZE          = -O0                             OPTIMIZE          = -O0
                                                  > accel:
                                                  >    OPTIMIZE          = -O1
                                                  > 404.lbm:
                                                  >    OPTIMIZE          = -O3
$ cat precedence_example6.sh
runaccel --fake --config=precedence_example6a | grep lbm.c
runaccel --fake --config=precedence_example6b | grep lbm.c
$ chmod +x precedence_example6.sh
$ ./precedence_example6.sh 
mpicc -c -o lbm.o -DSPEC -DNDEBUG -O3  lbm.c
mpicc -c -o lbm.o -DSPEC -DNDEBUG -O3  lbm.c
$  
Precedence Example 7: Repeated Sections

If a section occurs more than once, the settings are combined. If there are conflicts, the last instance wins.

Recall that trailing default specifiers can be dropped. Therefore, these three section markers:

 7  accel:
12  accel=default:
17  accel=default=default:

actually name the same section, which is entered three times.

The EXTRA_CFLAGS, COPTIMIZE, and EXTRA_COPTIMIZE settings (lines 9, 14, 19) are combined.

The OPTIMIZE settings on lines 8, 12 and 18 conflict with each other. Only the last one is retained.

$ cat -n precedence_example7.cfg
     1  label                = wall
     2  runlist              = 404.lbm
     3  tune                 = base
     4  default:
     5       CC_VERSION_OPTION = --version
     6       CC                = gcc
     7  accel:
     8       OPTIMIZE          = -O1 -fopenmp
     9       EXTRA_CFLAGS      = -finline-functions
    10  accel=default:
    11       pmodel            = ACC
    12       OPTIMIZE          = -O2 -fopenacc
    13       COPTIMIZE         = -ftree-vectorize
    14  accel=peak:
    15       OPTIMIZE          = -O4 -fopenacc
    16  accel=default=default:
    17       pmodel            = LOP
    18       OPTIMIZE          = -O3 -fopenmp
    19       EXTRA_COPTIMIZE   = -Wall
$ cat precedence_example7.sh 
runaccel --config=precedence_example7 --fake | grep lbm.c 
$ ./precedence_example7.sh 
gcc -c -o lbm.o -DSPEC -DNDEBUG -DSPEC  -O3 -fopenmp -ftree-vectorize -finline-functions -DSPEC_OPENMP -DSPEC_OPENMP_LOOP   -Wall     lbm.c
$  

I.F. Variables and Variable Substitution

You can do variable substitution within a config file.
As described above, the contents of a config file are directed to various consumers.
Therefore, to use variable substitution, you need to know which consumer does the substitution.
Differing syntax is used for each.

q. Wait a minute... all these choices for substitution? Which do I want?

a. Probably either the first in the list: specmake; or the last: the preprocessor.

Format Example Description + consumer Traps for the unwary
$(round) $(COMPILER_DIR) Make variables, interpreted by specmake  
\$SHELLVAR \$SPEC Shell or command interpreter variables. Quoting
$[square] $[top]/config Unchanging items, substituted by runaccel at startup. Timing
${curly}
$unbracketed
${hw_avail}ish
$hw_avail
Changeable items subject to perl interpolation.
$unbracketed is allowed if it is not ambiguous.
Try not to confuse with %{curly}
%{ENV_var} %{ENV_LIBRARY_PATH} Predefined macros for environment variables, handled by the preprocessor.  
%{curly} %{bits} macros, handled by the preprocessor Try not to confuse with ${curly}

I.F.1. Defining variables

You can create your own runaccel variables using a line of the form

name = value

The name may contain only letters, digits, and underscores (a hyphen is NOT allowed).
Start with a letter.
You may indent your definitions if you wish (see: whitespace)

Exception: preprocessor macros are different on all of the above. Hyphens are allowed, use

%define name value

and the % must appear in column 1. You can indent after the % if you wish.

Conventions: Although not required, certain conventions are usually followed:

  1. lower case is typically used for macros. When combining words, CamelCase may be useful.
  2. lower case is also typically used for runaccel variables, combining words with underscores
  3. CAPITALS are typically used when creating make variables
  4. CAPITALS are also typically used for shell or environment variables.

Examples of the above (in the same order)

%ifndef processorNumaControl
%   define processorNumaControl firstTouch # macro
%endif
default:
    my_submit_cmd       = numactl -C $BIND # runaccel variable
    MYTUNE              = -O2 --math=SIMD  # make variable
    ENV_LD_LIBRARY_PATH = /opt/lib         # environment variable

The remainder of this section I.F is about runaccel variables -- the $[square] and ${curly} rows from the table at the top.

I.F.2. $[square] substitution at startup

Immediately after preprocessing, variables that are delimited by $[square brackets] are substituted.
Any value set in the config file can be substituted, provided that is visible in the current scope.
You can access the value of additional variables that you may have created.
See the list of useful variables below.
Perhaps the most useful is $[top] for the top of the SPECaccel 2023 tree, often found in contexts such as:

    flagsurl01          = $[top]/config/compiler.xml
    flagsurl02          = $[top]/config/platform.xml
    EXTRA_LIBS          = $[top]/mylibs
    preENV_LIBRARY_PATH = $[top]/lib64:$[top]/lib

Traps for the unwary: In some cases it may be obvious to the human which value to use, but the tools aren't as smart as you.

  1. Timing: You cannot substitute variables that do not exist, or whose values aren't known when the config file is first read. For example, the label is not sorted out until later. Values that pertain to individual benchmarks (such as the current iteration number or the name of the benchmark being run) are not set until much later, when benchmarks are actually being run.
  2. Command line: If a value is mentioned on both the runaccel command line and in the config file, the substitution might not do what you expect.
  3. output_root:
    You cannot set an output_root that depends on a runaccel variable.
    You can set one that uses a macro:

    output_root=$[top]/my/path      # wrong
    output_root=${top}/my/path      # wrong
    output_root=%{ENV_SPEC}/my/path # right

Square substitution is done early. That comes in handy if you need a variable right away, for example, in order to use it with preENV.

$ cat EarlySub.cfg 
SW_DIR                 = /opt/path/to/compilers
preENV                 = 1
preENV_LD_LIBRARY_PATH = $[SW_DIR]/lib

$ cat EarlySub.sh
runaccel --config=EarlySub --fake 404.lbm | grep '^LD' | uniq
$ ./EarlySub.sh 
LD_LIBRARY_PATH=/opt/path/to/compilers/lib
$  

The example below uses variables defined in several named sections. The sections are delimited by section markers default: (line 8), default=base: (line 11), and default=peak: (line 15).

$ cat -n square.cfg
     1  expand_notes         = 1
     2  iterations           = 1
     3  output_format        = text
     4  output_root          = /tmp/square
     5  runlist              = 404.lbm
     6  size                 = test
     7  tune                 = base,peak
     8  default:
     9     CC                = gcc
    10     CC_VERSION_OPTION = --version
    11  default=base:
    12     the_system        = STAR
    13     OPTIMIZE          = -O1 -fopenacc
    14     notes_base_100    = base tuning uses '$[CC]' '$[OPTIMIZE]' on system '$[the_system]'
    15  default=peak:       
    16     OPTIMIZE          = -O2 -fopenacc
    17     notes_peak_100    = peak tuning uses '$[CC]' '$[OPTIMIZE]' on system '$[the_system]'
    18  
$ cat square.sh 
runaccel --config=square | grep txt
grep tuning /tmp/square/result/accel2023.001.omp.test.txt
$ ./square.sh 
    format: Text -> /tmp/square/result/accel2023.001.small.test.txt
     base tuning uses 'gcc' '-O1 -fopenacc' on system 'STAR'
     peak tuning uses 'gcc' '-O2 -fopenacc' on system ''
$  

Note that line 14 finds all three variables that it is looking for, but line 17 does not. If it is not clear why this happens, please see the descriptions of named sections and precedence above.

I.F.3. Useful $[square] variables

Useful $[variables] include:

$[configfile] Your config file name
$[configpath] The full path to your config file
$[dirprot] protection that is applied to directories created by runaccel
$[endian] 4321 or 87654321 for big endian; 1234 or 12345678 for little
$[flag_url_base] directory where flags files are looked up
$[OS] unix or windows
$[os_exe_ext] exe for windows, nil elsewhere
$[realuser] the user name according to the OS
$[top] the top directory of your installed SPECaccel 2023 tree
$[username] the username for purposes of tagging run directories
$[uid] the numeric user id

You can access the initial value of most options that you can enter into a config file, including:

action, allow_label_override, backup_config, basepeak, check_version, command_add_redirect, copies, delay, deletework, difflines, env_vars, expand_notes, expid, fake, flagsurl, http_proxy, http_timeout, ignore_errors, ignore_sigint, info_wrap_columns, iterations, label, line_width, locking, log_line_width, mail_reports, mailcompress, mailmethod, mailport, mailserver, mailto, make, make_no_clobber, makeflags, mean_anyway, minimize_builddirs, minimize_rundirs, no_input_handler, no_monitor, notes_wrap_columns, notes_wrap_indent, output_format, output_root, plain_train, rawformat, rebuild, reportable, runlist, section_specifier_fatal, setprocgroup, size, strict_rundir_verify, sysinfo_program, table, teeout, train_single_thread, tune, use_submit_for_speed, username, verbose, verify_binaries, version_url.

I.F.4. ${curly} substitution (interpolation) during a run
+ $unbracketed substitution

During a run, variables that are delimited by ${curly brackets} are substituted: ${command}
Usually, variables can be spelled with or without the curlies: $command or ${command}
Exception 1: curlies are required if the variable is adjacent to other text ${command}s

runaccel uses perl interpolation.
Only scalars (not: perl arrays and hashes) can be interpolated.

Example: on the notes100 line, you could optionally add say either ${lognum} or $lognum, but don't try taking the curly brackets away from ${size}.

$ cat just.cfg 
expand_notes      = 1
notes100          = Just ${size}ing, in run $lognum 
output_root       = /tmp/just
runlist           = 404.lbm
size              = test
CC                = gcc
CC_VERSION_OPTION = --version
$ 
$ cat just.sh
runaccel -c just | grep txt
grep Just /tmp/just/result/accel2023.001.acc.test.txt
$   
$ ./just.sh 
    format: Text -> /tmp/just/result/accel2023.001.acc.test.txt
     Just testing, in run 001
$ 

Traps for the unwary

  1. Try not to confuse ${curly} interpolation vs. %{curly} (macros)
  2. Timing: Some variables are only defined at certain times, and a line that uses it might be interpolated at a different time. Therefore interpolation won't always do what you might wish. In particular, notes are not expanded in the context of a particular benchmark run, and therefore variables such as ${tune} are not useful within them.

  3. output_root:
    You cannot set an output_root that depends on a runaccel variable.
    You can set one that uses a macro:

    output_root=$[top]/my/path      # wrong
    output_root=${top}/my/path      # wrong
    output_root=%{ENV_SPEC}/my/path # right

I.F.5. Useful ${curly} variables

These variables may be of interest:

${baseexe} The first part of the executable name, which is <baseexe>_<tune>.<label>. For example, in "lbm_base.foo", baseexe is "lbm".
${benchmark} The number and name of the benchmark currently being run, for example 404.lbm
${benchname} The name of the benchmark currently being run, for example lbm
${benchnum} The number of the benchmark currently being run, for example 605
${benchtop} The top directory for the benchmark currently being run, for example /spec/accel2023/benchspec/ACCEL/404.lbm
${command} The shell command line to run the current benchmark, for example ../run_base_test_sticky.0000/lbm_base.sticky output6.ref.txt 24000 10000 3000 1250 600 100 6
${commandexe} The executable for the current command, for example ../run_base_test_none.0000/lbm_base.sticky
${label} The label for the benchmark being run
${iter} The current iteration number
${logname} The complete log file name, for example /spec/accel2023/result/accel2023.168.log
${lognum} The log file number, for example 168
${tune} The tuning for the benchmark being run (base or peak)
${workload} The current workload number (within the iteration)

For a complete list of the available variables relative to the current config file, set

expand_notes = 1
verbose      = 35 # or higher

Then, do a run that causes a command substitution to happen.
In the log, you will find many lines of the form:

 - Variables available for interpolation that have changed since the last list:
    (From config) $runmode = "speed"
    (From config) $size = "test"
 - Variables available for interpolation that have changed since the last list:
    (From config) $size = "ref"

I.F.6. Unsetting a variable with "%undef%"

It is sometimes useful to undo the setting of a variable that would otherwise be included from another section. This can be accomplished using the special value %undef%. In the example, line 14 undefines COPTIMIZE when compiling peak:

 $ cat -n dave.cfg 
 1  flagsurl             = $[top]/config/flags/gcc.xml
 2  iterations           = 1
 3  output_format        = text
 4  output_root          = /tmp/undef
 5  runlist              = 404.lbm
 6  size                 = test
 7  tune                 = base,peak
 8  default:
 9     CC_VERSION_OPTION = --version
10     CC                = gcc
11     OPTIMIZE          = -O2 -fopenacc
12     COPTIMIZE         = -fno-tree-pre
13  404.lbm=peak:
14     COPTIMIZE         = %undef%
15  
$ runaccel --config=dave | grep txt
    format: Text -> /tmp/undef/result/accel2023.001.small.test.txt
$ cd /tmp/undef/benchspec/ACCEL/404.lbm/build
$ grep OPTIMIZE build_base_none.0000/Makefile.spec 
COPTIMIZE        = -fno-tree-pre
OPTIMIZE         = -O2 -fopenacc
$ grep OPTIMIZE build_peak_none.0000/Makefile.spec 
COPTIMIZE        = 
OPTIMIZE         = -O2 -fopenacc
$ 

Notice that in the build directory, COPTIMIZE is present for base and absent for peak.

I.F.7. Debug tips for runaccel variables

When debugging a config file that uses runaccel variables, try:

iterations       = 1
minimize_rundirs = 0
reportable       = 0
runlist          = (one or two benchmarks)
size             = test 
verbose          = 40

Using --fake will probably be informative. Look inside the log for the (case-sensitive) word 'From'.

II. Config file options for runaccel

This section documents options that control the operation of runaccel itself.

II.A. Precedence: config file vs. runaccel command line

In the list that follows, some items are linked to the document Using SPECaccel 2023 - the 'runaccel' Command because they can be specified either in a config file, or on the runaccel command line.

New with SPECaccel 2023, If an option is specified in both places, the command line wins.

II.B. Options

In the table that follows, if an option is documented as accepting the values "no" and "yes", these may also be specified as "false" and "true", or as "0" and "1".

The "Use In" column indicates where the option can be used:

H use only in header section
N use in a named section.
H,N can be used in both the header section and in named sections. The item can therefore be applied on a global basis, and/or can be applied to individual benchmarks.
Option Use In Default Meaning
action H validate What to do. The available actions are defined in the runaccel document. See also the buildsetup example in the section on specmake.
allow_label_override H no

The runaccel command can use --label to select sections that apply to a specific label. If the label mentioned on the runaccel command does not occur in any section marker:

  • If allow_label_override is no (or 0 or false), the tools will print an error message and exit.
  • If allow_label_override is yes (or 1 or true), then the tools will attempt to build (or run) the same binary they would have built by default, but will name the binary with specified label.
This option was known as allow_extension_override prior to SPECaccel 2023.
backup_config H yes When updating the hashes in the config file, make a backup copy first. Highly recommended to defend against full-file-system errors, system crashes, or other unfortunate events.
basepeak H,N no Use base binary and/or base result for peak. If applied to the whole suite then only base is run, and its results are reported for both the base and peak metrics. If applied to a single benchmark, the same binary will be used for both base and peak runs, and the base results will be reported for both. (The reason for running the binary during both base and peak is to remove the possibility that skipping a benchmark altogether might somehow change the performance of some other benchmark.)
check_version H no (yes for
reportable
runs)

When set, before doing a reportable run, runaccel will download a small file (~15 bytes) from www.spec.org containing the current version of the suite and the date it was released, and check your copy vs. that file. In this way, you can be notified if the version of the suite that you're using is out-of-date.

Setting this variable to no will disable this check.

If you'd like to check a local file instead, you can modify version_url to point to your internal copy.

If you would like to check your version for a NON-reportable run, you will need to add --check_version to your command line. Setting check_version=yes in the config file only causes the check for reportable runs.

command_add_redirect H no

If set, the generated ${command} will include redirection operators (stdout, stderr), which are passed along to the shell that executes the command. If this variable is not set, specinvoke does the redirection. This option is commonly used with submit.

When used with submit, the command_add_redirect feature lets you choose whether redirection operators (such as <this_benchmark.in or >that_benchmark.out) are applied to your entire modified submit command (the default) or just to the portion that has ${command}.
In pseudo-code:

A benchmark runs exe; it sends output to out.
If you use:
submit = send_job job="${command}" rank_id=$rank
   Do you prefer this?
         send_job job="exe >out" rank_id=n
Or this?
         send_job job="exe" rank_id=n >out

The above is only pseudo-code; see the section on Using Submit for real examples.

Option Use In Default Meaning
delay H,N 0 Insert a delay of the specified number of seconds before and after benchmark execution. This delay does not contribute to the measured runtime of the benchmark. This delay is also not available in a reportable run.
deletework H,N no If set to yes, always delete existing benchmark working directories. An extra-careful person might want to set this to ensure no unwanted leftovers from previous benchmark runs, but the tools are already trying to enforce that property.
difflines H,N 10 Number of lines of differences to print when comparing results.
enable_monitor H,N yes

If this variable is set to no, then all of the monitoring hooks are disabled. This can be overridden by setting force_monitor. force_monitor is new with SPECaccel 2023

env_vars
or
envvars
H,N no

If set to yes, environment settings can be changed using ENV_* options in the config file. Note that you cannot change OMP_NUM_THREADS or ACC_NUM_CORES using this feature. Use threads.

Example: Consider the config file below, which sets the LD_LIBRARY_PATH to the location of the compiler runtime libraries for an alternate gcc thats not used for the base.

 size        = test
 tune        = peak
 label       = srini
+env_vars    = 1
 #
 404.lbm=peak:
+   ENV_LD_LIBRARY_PATH = /tmp/compiler/lib64:%{ENV_LD_LIBRARY_PATH}
    CC                  = alt_gcc
    CC_VERSION_OPTION   = -v
    OPTIMIZE            = -O1 -fopenacc
 
  1. The syntax %{ENV_x} uses the preprocessor to import both the definition for SPEC and the definition for LD_LIBRARY_PATH as they were at runaccel startup. If the definitions for SPEC and LD_LIBRARY_PATH at the start of the run are, respectively, /A and /B, then runaccel will add 'LD_LIBRARY_PATH=/A/libraries:/B'  to the environment when it invokes the executable lbm_s_peak.srini.
  2. For SPECaccel 2023, the environment must be consistent during base (rule 3.4). You may vary it for peak.
  3. When developing a config file that uses env_vars, you may find it useful to set verbose to 35 (or higher), which will cause the tools to log environment settings.
  4. Which environment? If you are attempting to set the environment for your entire run, this is not the feature that you are looking for. Try preenv instead. The env_vars option and ENV* are about communication from the config file to the environment of the invoked benchmark.
expand_notes H no If set, will expand variables in notes. This capability is limited because notes are NOT processed by specmake, so you cannot do repeated substitutions.
expid H If set to a non-empty value, will cause executables, run directories, results, and log files to be put in a subdirectory (with the same name as the value set) in their normal directories. For example, setting expid = CDS will cause benchmark binaries to end up in exe/CDS, run directories to end up in run/CDS, and results and logs in $SPEC/result/CDS.
fail H,N no If set, will cause a build or run to fail.
fail_build H,N 0

If set, will cause a build to fail. For example, you could say something like this:

404.lbm=default:
#> I am posting this config file for use by others in the
#> company, but am forcing it to fail here because
#> I want to force users to review this section.
#>
#> Once you find your way here, you should test whether
#> bug report 234567 has been fixed, by using the first
#> line below.  If it has not been fixed, then use the
#> second.  In either case, you'll need to remove the
#> fail_build.
#>
#>   - Pney Guvaxre
#>     Boomtime, the 66th day of Confusion in the YOLD 3172

OPTIMIZE = -Osuperduper
# OPTIMIZE = -Omiddling
fail_build = yes

In the example above, the build is forced to fail until the user examines and modifies that section of the config file. Notice that Pney has used protected comments to cause the comments about the internal bug report to disappear if the config file were to be published as part of a reportable run.

fail_run H,N no If set, will cause a run to fail.
Option Use In Default Meaning
flagsurl H none

If set, retrieve the named URL or filename and use that as the "user" flags file. If the special value "noflags" is used, runaccel will not use any file and (if formatting previously run results) will remove any stored file. Automated processing of flags is explained in flag-description.html.

If you want to list more than one flagsfile, the recommended method is by using numbered continuation lines, for example:

     flagsurl1 = mycompiler.xml
     flagsurl2 = myplatform.xml 

Using other methods (such as backslash continuation) to specify multiple flags files may appear to work, but may result in unexpected differences between the original config file and the config file as written by output format config.

Multiple flags files will typically be needed, because flags files are separated into two types, "compiler", and "platform".

force_monitor H,N no

If this variable is set to yes, then all of the monitoring hooks are enabled, regardless of settings that would otherwise turn them off. This means that every invocation of specinvoke will be subject to monitor_specrun_wrapper, and all command invocations will be subject to monitor_wrapper. This includes things that would normally not be subject to monitoring, such as FDO training runs, input generation commands, and commands used for validating benchmark output such as specdiff. force_monitor is new with SPECaccel 2023.

http_proxy H

In some cases, such as when doing version checks and loading flag description files, runaccel will use HTTP to fetch a file. If you need to specify the URL of a proxy server, this is the variable to use. By default, no proxy is used. Note that this setting will override the value of the http_proxy environment variable. For example, one might set:

http_proxy = http://webcache.tom.spokewrenchdad.com:8080 
http_timeout H 30 This is the amount of time (in seconds) to wait while attempting to fetch a file via HTTP. If the connection cannot be established in this amount of time, the attempt will be aborted.
ignore_errors H no Ignore certain errors which would otherwise cause the run to stop. Very useful when debugging a new compiler and new set of options: with this option set, you'll find out about all the benchmarks that have problems, instead of only finding out about the first one.
ignore_sigint H no Ignore SIGINT. If this is set, runaccel will attempt to continue running when you interrupt one of its child processes by pressing ^C (assuming that you have ^C mapped in the common way). Note that this does NOT cause runaccel itself to ignore SIGINT.
info_wrap_columns H 50 When set to a value greater than 0, attempts to split non-notes informational lines such that they are no longer than info_wrap_columns columns wide. Lines are split on whitespace, and newly created lines are guaranteed to have at least the same indentation as the original line. If a line contains an item that is longer than info_wrap_columns, a warning is logged and the original line is left unchanged.
iterations H 3 Number of iterations to run.
keeptmp H no

Whether or not to keep various temporary files. If you leave keeptmp at its default setting, temporary files will be automatically deleted after a successful run. If not, temporary files may accumulate at a prodigious rate, and you should be prepared to clean them by hand. Temporary files include:

  • accel2023.nnn.log.debug in your result directory ($SPEC/result, %SPEC%\result, or the corresponding location under your output_root).
  • A directory named tmp/ under one of these places:

    $SPEC/               or        %SPEC%\
    Whatever you set for an output_root directory
  • If tmp/ is not usable, you might find temporaries under tmp.accel2023 or, possibly, tmp.accel2023.n, where n may range from 1 to 15.
label H none

An arbitrary tag for executables, build directories, and run directories.
Legal characters are: alphanumerics, underscore, hyphen, and period. For SPECaccel 2023, this field replaces the field ext
Examples:

label=jun12.old.CC 
label=jun12.new.CC 
label=jun14-flagday 
label=jun15-robert.wants.yet.another.test

If a label is used as a section specifier, it can be referenced from the runaccel command line.
If a runaccel command uses a label that is not defined in a config file, by default the run will stop with an error:

runaccel --label=sandra [...]
ERROR: The label 'sandra' defines no settings in the config file!

The error can be disabled if you set allow_label_override=yes.

Tip: Make a habit of setting the label to anything that is meaningful to you. Doing so makes it easier to compare binaries, easier to track history, and easier to clean up by removing directories that match the label. Example:

This config file runs 404.lbm using the (very short) test workload. It uses differing tuning depending on the label:

output_format        = text
runlist              = 404.lbm
default:
   CC_VERSION_OPTION = -v
   CC                = gcc
default=base=OhZero:
   OPTIMIZE          = -O0
default=base=OhThree:
   OPTIMIZE          = -O3 

The label can be used when hunting down directories - either to analyze them or to delete them, as in the Unix find command. On other systems, try searching in a GUI file manager.

$ find . -type d -name '*OhZ*'
./benchspec/ACCEL/404.lbm/build/build_base_OhZero.0000
./benchspec/ACCEL/404.lbm/run/run_base_test_OhZero.0000
./benchspec/ACCEL/404.lbm/run/run_base_test_OhZero.0001
./benchspec/ACCEL/404.lbm/run/run_base_test_OhZero.0002
$ cd benchspec/ACCEL/404.lbm/build
$ grep OPTIMIZE */Makefile.spec
build_base_OhThree.0000/Makefile.spec:OPTIMIZE   = -O3
build_base_OhZero.0000/Makefile.spec:OPTIMIZE   = -O0
$ 
Option Use In Default Meaning
line_width H 0 Line wrap width for screen. If left at the default, 0, then lines will not be wrapped and may be arbitrarily long.
locking H yes Try to use file locking to avoid race conditions, e.g. if more than one copy of runaccel is in use. Although performance tests are typically done with only one copy of runaccel active, it can be handy to run multiple copies if you are just testing for correctness, or if you are compiling the benchmarks.
log_line_width H 0 Line wrap width for logfiles. If your editor complains about lines being too long when you look at logfiles, try setting this to some reasonable value, such as 80 or yes. If left at the default, yes, then lines will not be wrapped and may be arbitrarily long.
log_timestamp H no Whether or not to prepend time stamps to log file lines.
mailcompress H no When using the 'mail' output format, turning this on will cause the various report attachments to be compressed with gzip.
mailmethod H smtp When using the 'mail' output format, this specifies the method that should be used to send the mail. On UNIX and UNIX-like systems, there are three choices: 'smtp' (communicate directly with an SMTP server over the network), 'sendmail' (try invoking sendmail directly from locations where it is commonly installed), and 'qmail' (try invoking qmail-inject from locations where it is commonly installed).
Using a sendmail or qmail-inject program from a non-standard location is possible only by setting the PERL_MAILER environment variable. See the Mail::Mailer documentation for details.
mailport H 25 When using the 'mail' output format, and when the mailmethod is 'smtp', this specifies the port to use on the mail server. The default is the standard SMTP port and should not be changed.
mailserver H 127.0.0.1 When using the 'mail' output format, and when the mailmethod is 'smtp', this specifies the IP address or hostname of the mailserver through which to send the results.
mailto H ''

The address or addresses to which results should be sent when using the 'mail' output format. If multiple addresses are specified, they should be separated by commas or whitespace. Each address should consist only of the name@domain part (i.e. no "full name" type info). The addresses are not checked for correct formatting; if a mistake is made, the results may be sent to an unknown location. Think: comp.arch. OK, probably not there, but seriously be careful about security on this one.

Config files as posted at www.spec.org/accel2023 will not include whatever you put on this line (thus, spambots will not see the contents of this field).

Note that to get your reports mailed to you, you need to specify both mail as an output_format and an address to which they should be mailed. For example:

   mailto=fast.guy@welovebenchmarks.org
   output_format=text,mail

If no addresses are specified, no mail will be sent.

mail_reports H all The list of report types to mail. The format and possible values are the same as for output_format, with the addition of log, which will cause the current log file to be sent. The default is for all files associated with the run to be mailed (so, this will include what you listed as your desired output_format plus log (the log file) and rsf (the rawfile). You can cut your email down to the bare essentials with something like this:
   mailto=fast.guy@welovebenchmarks.org
   output_format=text,mail
   mail_reports=text
If none of the requested report types were generated, no mail will be sent.
make H,N specmake Name of make executable. Note that the tools will enforce use of specmake for reportable results.
make_no_clobber H,N no Don't delete directories when building executables. The default is no, meaning "clobber". The "yes" option, meaning "avoid clobbering", should only be used for troubleshooting a problematic compile. The tools will not allow you to use this option when building binaries for a reportable result. Note that you could issue multiple successive runaccel commands with this option set (either in the config file, or with the --make_no_clobber switch), and the build directories will be preserved. But once you remove make_no_clobber (allowing it to default back to no), then the tools will attempt a normal build with a fresh build directory.
makeflags H,N ''

Extra flags for make (such as -j). Set this to -j n where n is the number of concurrent processes to run during a build. Omitting n or setting it to zero unlimits the number of jobs that will be run in parallel. Use with care! Make flags should be used here only if you are familiar with GNU make. (The program specmake is GNU Make under another name to ensure no accidental conflicts with other Make utilities you might have. The GNU Make Manual can be consulted, and you can also say specmake --help.)

Note that requesting a parallel build with makeflags = -j N causes multiple processors to be used at build time. It has no effect on whether multiple processors are used at run time, and so does not affect how you report on parallelism.

Option Use In Default Meaning
mean_anyway H no Calculate mean even if invalid. DANGER: this will write a mean to all reports even if no valid mean can be computed (e.g. half the benchmarks failed). A mean from an invalid run is not "reportable" (that is, it cannot be represented in public as the SPEC metric).
minimize_rundirs H no During a run, try to keep working disk size down. Cannot be used in a reportable run.
minimize_builddirs H no Try to keep working disk size down during builds.
nobuild H no Do not attempt to build benchmarks. Useful to prevent attempts to rebuild benchmarks that cannot be built. Also comes in handy when testing whether proposed config file options would potentially force an automatic rebuild.
no_monitor H,N '' Exclude the listed workloads from monitoring via the various monitor_* hooks.
no_input_handler H,N close

Method to use to simulate an empty input. Choices are:

  • close - Just close stdin before executing the benchmark (this is the default).
  • zero - Create a zero-length file and attach it to stdin.
  • file - Synonym for zero.
  • null - Open the null device (/dev/null or NUL:) and attach it to stdin.

Normally, this option should be left at the default; it was actually added to the tools for the benefit of a different SPEC suite that needed the feature. If a reportable run for SPECaccel 2023 uses this feature, an explanation should be provided as to why it was used.

notes_wrap_columns H 0 When set to a value greater than 0, attempts to split notes lines such that they are no longer than notes_wrap_columns columns wide. Lines are split on whitespace, and newly created lines are guaranteed to have at least the same indentation as the original line. If a line contains an item that is longer than notes_wrap_columns, a warning is logged and the original line is left unchanged.
notes_wrap_indent H ' ' When line wrapping is enabled (see notes_wrap_columns), this is the string that will be prepended to newly created lines after the indentation from the original line is applied. The default is four spaces, but it can be set to any arbitrary string.
output_format H all Format for reports. Valid options are listed at runaccel.html under --output_format; major options include txt (ASCII text), html, pdf, and ps. You might prefer to set this to txt if you're going to be doing lots of runs, and only create the pretty reports at the end of the series. See also the information in runaccel.html about --rawformat.
output_root H

If set to a non-empty value, all output files will be rooted under the named directory, instead of under $SPEC (or %SPEC%).
If the output_root is not an absolute path (one that begins with "/" on Unix, or a device name on Windows), the path will be created under $SPEC or %SPEC%. You probably will find it more useful if you pick some place other than the default, for convenience when cleaning directories, allocating quotas, or optimizing I/O.

You can navigate a rooted directory with ogo.
This option can be useful for sharing an installation.
It can also can be useful to optimize your I/O for a big SPECrate run: you can put the output_root on your faster device.

Example: Junjie has a 500 TB optimized file system under /local/home. If he types

runaccel --output_root=/local/home/junjie --config=mine 404.lbm 
These directories will be created For these purposes
log files and reports /local/home/junjie/result/
404.lbm build directories /local/home/junjie/benchspec/ACCEL/404.lbm/build/
404.lbm benchmark binaries (executables) /local/home/junjie/benchspec/ACCEL/404.lbm/exe/
404.lbm run directories /local/home/junjie/benchspec/ACCEL/404.lbm/run/
Option Use In Default Meaning
pmodel N LOP Select OpenACC or OpenMP parallel model. With OpenMP also select the methodology. If not set, the default is to build with LOP.

Important! This option only sets the appropriate macro define flag for the model's directives to be visible in the post-processed source. You must also set the appropriate compiler option to enable the use of the model (ex. "-fopenmp") in OPTIMIZE. Also, not all models are supported by all compilers. Please consult your compiler's documentation for the appropriate options and which models they support.

pmodel setting
ACCUse OpenACC. Only available with the 'openacc' suite.
LOPOpenMP using the "loop" construct.
TGTOpenMP using the "distribute/parallel" constructs collapsing all nested loop levels.
SMDOpenMP using the "distribute/parallel" constructs with "simd" applied to inner loops.

preenv
or
pre_env
H yes

Use preENV_ lines in the config file. When this option is set (the default), lines of the form preENV_<variable> = <value> will cause runaccel to set the specified environment variable to value and re-exec runaccel to perform the run. The restart is done in order to ensure that the entire run takes place with the new settings. You can set preENV_SOME_VARIABLE = value only in the header section or a section using one these benchmark specifiers:

default:
accel:

Any attempt to use preENV_ in other sections is silently ignored.
You cannot change OMP_NUM_THREADS or ACC_NUM_CORES using this feature. Use threads.
Multiple preENV_ settings may appear in the config file.
If you are looking for a way to affect the environment of an individual benchmark, try env_vars, not preenv.

Example: Above, the section on env_vars showed how a library path can be adjusted for a single benchmark.
Instead of adjusting one benchmark, the environment could be set for all benchmarks by putting a line for preENV_LD_LIBRARY_PATH at the top of the config file:

preENV_LD_LIBRARY_PATH = %{ENV_SPEC}/libraries:%{ENV_LD_LIBRARY_PATH}
ranksNot used in SPECaccel H 1 The number of ranks to be used to run each benchmark. The value is required to be set, either in the config file or on the runaccel command line, so the reports will be formatted correctly. Use $ranks to pass the count to the MPI driver, as shown in the submit section.
reportable H yes Strictly follow reporting rules, to the extent that it is practical to enforce them by automated means. The tester remains responsible for ensuring that the runs are rule compliant. You must set reportable to generate a valid run suitable for publication and/or submission to SPEC.
rebuild H no Rebuild binaries even if they exist.
runlist H none Benchmarks or sets to run. Names can be abbreviated, just as on the command line. See the long discussion of run order in runaccel.html.
save_build_files H,N none

After a build is finished, files matching any of the space-delimited wildcard patterns in this variable will be gathered up and saved. When a non-reportable run is being set up, those files will be unpacked into the run directory when the executable is copied in. The auxiliary file package is ignored when setting up reportable runs. The save_build_files feature is new with SPECaccel 2023.

section_specifier_fatal H yes

While parsing the config file, if a section specifier is found that refers to an unknown benchmark or benchset, an error is output and the run stops. Set section_specifier_fatal=no in the header section of your config file to convert this error into a warning and allow the run to continue. The ability to convert section specifier errors into warnings is probably of use only for benchmark developers.

setprocgroup H yes Set the process group. On Unix-like systems, improves the chances that ^C gets the whole run, not just one of the children.
size H ref

Size of input set: test or ref
If you are in the early stages of testing a new compiler or new set of options, you might set this to test.
The reference workload (ref) is the only size whose time appears in reports.

You might choose to use runaccel --size=test while debugging a new set of compilation options.

Reportable runs only invoke the ref workload

Option Use In Default Meaning
src.alt N none

Name of a SPEC-approved alternate source.
Set the option in the named section for the benchmark(s) where you wish to use it.
Multiple src.alts may be specified a single benchmark. Separate them wth commas.
The effect of using a src.alt is to modify the sources in the build directory.
You may also spell this option as srcalt.

 

About alternate sources: Sometimes a portability issue may require use of different source code for a benchmark, and SPEC may issue a src.alt, which is a compressed tar file containing modifications, created by makesrcalt.

To use a src.alt, see the instructions posted with it at www.spec.org/accel2023/src.alt.
Typically, you begin by downloading it to your $SPEC or %SPEC% directory and unpacking it:

$ cd $SPEC   
$ specxz -dc nnn.benchmark.FixMumble.tar.xz | spectar -xvf -
$ cat README.nnn.benchmark.src.alt.FixMumble.txt

The README will explain what to add to your config file and any other instructions that are needed.

After you unpack it, a directory is created under under <benchmark>/src/src.alt/ and a set of patches are stored there. You can look at the patches using dumpsrcalt, but it may be easier to just apply the src.alt and look at a build directory.

Example: This config file builds with or without a src.alt, depending on the runaccel setting for the --label.

$ cat testme.cfg
action               = buildsetup
runlist              = nnn.benchmark
tune                 = base
default:
   CC                = gcc
   CC_VERSION_OPTION = --version
nnn.benchmark=base=without:
   OPTIMIZE          = -O2
nnn.benchmark=base=with:
   OPTIMIZE          = -O2
   srcalt            = FixMumble

To populate build directories:

runaccel --label=without --config=testme
runaccel --label=with    --config=testme

Then, if you visit benchspec/ACCEL/nnn.benchmark/build you can compare the directories.

Option Use In Default Meaning
strict_rundir_verify H yes

When set, the tools will verify that the file contents in existing run directories match the expected checksums. Normally, this should always be on, and reportable runs will force it to be on. Turning it off might make the setup phase go a little faster while you are tuning the benchmarks.

Developer notes: setting strict_rundir_verify=no might be useful when prototyping a change to a workload or testing the effect of differing workloads. Note, though, that once you start changing your installed tree for such purposes it is easy to get lost; you might as well keep a pristine tree without modifications, and use a second tree that you convert_to_development.

table H yes In ASCII reports, include information about each execution of the benchmark.
teeout H no Run output through tee so you can see it on the screen. Primarily affects builds, but also provides some information about progress of runtime, by showing you the specinvoke commands.
threads H,N 1 Value to be set for OMP_NUM_THREADS or ACC_NUM_CORES when benchmarks are run.
If you have set these in your environment prior to using runaccel, that setting will be ignored. To use multiple benchmark threads, you must use either this config file field or the corresponding option runaccel --threads=N.
tune H base default tuning level. In a reportable run, must be either all or base.
verbose H 5 Verbosity level. Select level 1 through 99 to control how much debugging info runaccel prints out. For more information, see the section on log files, below.
verify_binaries H yes

runaccel uses checksums to verify that executables match the config file that invokes them, and if they do not, runaccel forces a recompile. You can turn that feature off by setting verify_binaries=no.

Warning: It is strongly recommended that you keep this option at its default, yes (that is, enabled). If you disable this feature, you effectively say that you are willing to run a benchmark even if you don't know what you did or how you did it -- that is, you lack information as to how it was built!

The feature can be turned off because it may be useful to do so sometimes when debugging (for an example, see env_vars), but it should not be routinely disabled.

Since SPEC requires that you disclose how you build benchmarks, reportable runs (using the command-line switch --reportable or config file setting reportable=yes) will cause verify_binaries to be automatically enabled. For SPECaccel 2023, this field replaces the field check_md5

version_url H http://
www.
spec.org/
accel2023/
current_
version
If version checking is enabled, this specifies the location from which the version information should be fetched.

III. Config file options for specmake

For SPECaccel you do not write Makefiles. Instead, you set Make variables in the config file, which are sent to a SPEC-supplied copy of GNU Make, known as specmake. Variables with a dollar sign and parentheses, aka "round brackets", are substituted by specmake. For example:

COMPILER_DIR = /usr/local/bin/
CC           = $(COMPILER_DIR)cc
CXX          = $(COMPILER_DIR)c++
FC           = $(COMPILER_DIR)f90

See below for more information on syntax of variables that you create and reference.

III.A. Commonly used Make variables

  • Basic specmake variables are described in this section.
  • Many more are described in the Make Variables document.
  • The specmake utility can be run directly from the command line, as described in Utilities.

The following Make variables are frequently useful. When selecting where to put a flag, please bear in mind that the run rules require that portability flags must use PORTABILITY variables.

CC How to invoke your C compiler.
CXX How to invoke your C++ compiler.
FC How to invoke your Fortran compiler.
CC_VERSION_OPTION
CXX_VERSION_OPTION
FC_VERSION_OPTION

New with SPECaccel 2023: You must specify how to ask each compiler "Please tell me your version" because the method varies from compiler to compiler. The version information is recorded contemporaneously with the build.

You must specify the option for all compilers that you use.
If you change compilers for some benchmarks, be sure to also change the corresponding version option.

PORTABILITY
EXTRA_PORTABILITY
Portability flags to be applied no matter what the compiler.
{C|CXX|F}PORTABILITY
EXTRA_{C|CXX|F}PORTABILITY
Portability flags to be applied to modules of the designated language
(For example, CXXPORTABILITY is for the C++ modules).
OPTIMIZE EXTRA_OPTIMIZE Optimization flags to be applied for all compilers.
{C|CXX|F}OPTIMIZE
EXTRA_{C|CXX|F}OPTIMIZE
Optimization flags to be applied to modules of the designated language.
EXTRA_{C|CXX|F}FLAGS Flags that are neither optimization nor portability
LIBS Libraries to add to the link line
Many more See chart in document Make Variables

III.B. Parallel Model Selection

SPECaccel contains one suite for OpenACC and OpenMP. Previously, accel 1.4 contained three separate suites, one for each model. To select OpenACC choose "ACC". For OpenMP, you may select between three different methods of the directives, "loop", "distribute/parallel", or "distribute/parallel" with "simd" applied to inner loops.

The choice of directives will depend on the target device and compiler support. Not all compiler support all forms, and the applied parallelism may differ depending on the target. For example, using "loop" may be better when targeting offload devices such as GPUs, but "simd" better when targeting multi-core CPUs. The different forms are provide to allow users the best choice for their target and mitigate bias when favoring one form over another.

Note the same model must be used for all benchmarks in base. For peak, you may select the model on a per benchmark basis.

To enable, use the appropriate 'pmodel' setting. The tools then set the appropriate preprocessor define flag which makes the directives visible in the source. Note that not all compilers support all models and the compiler flag that enables the model must be set in your OPTIMIZE variable in the config file. Please consult your compiler's documentation for the supported models and options.

pmodel settingModelDefine flag implicitly added
ACCOpenACC-DSPEC_OPENACC
LOPOpenMP using the "loop" directive-DSPEC_OPENMP -DSPEC_OPENMP_LOOP
TGTOpenMP using the "distribute" and "parallel" directives collapsing all nested loop levels.-DSPEC_OPENMP -DSPEC_OPENMP_TARGET
SMDOpenMP using the "distribute" and "parallel" directives with "simd" applied to inner loops.-DSPEC_OPENMP -DSPEC_OPENMP_INNER_SIMD

Note that using the define flags directly in you OPTIMIZE or PORTABILITY flags will cause the tools to error. This is to help reduce the possibiltiy of setting the pmodel to one model but then accidently enabling a different model.

The config file fragment below demonstrates how to conditionally include a parallel model selected via runaccel --define option:

% cat -n config/pmodel.cfg
     1  # If model is not set use OpenACC
     2  %ifndef %{model}
     3  %   define model acc
     4  %endif
     5
     6  # Compiler flags applied to all models
     7  default:
     8    CC             = nvc
     9    OPTIMIZE       = -O3
    10    CXXPORTABILITY = --c++14
    11    submit = $command
    12
    13  # OpenACC flags
    14  %if %{model} eq 'acc'
    15      pmodel = ACC
    16      OPTIMIZE += -acc
    17  %endif
    18
    19  # OpenMP Loop flags
    20  %if %{model} eq 'loop'
    21      pmodel = LOP
    22      OPTIMIZE += -mp=gpu
    23  %endif
    24
    25  # OpenMP Distribute flags
    26  %if %{model} eq 'dist'
    27      pmodel=TGT
    28      OPTIMIZE += -mp=gpu
    29  %endif
    30
    31  # OpenMP Distribute w/ Inner SIMD flags
    32  %if %{model} eq 'simd'
    33      pmodel=SMD
    34      OPTIMIZE += -mp=gpu
    35  %endif
    36
    37  # This will error since the model define flags
    38  # should only be set by the tools
    39  404.lbm=peak:
    40      OPTIMIZE = -mp -fast -DSPEC_OPENMP_TARGET
    41
% runaccel -c pmodel --fake 404.lbm | grep lbm.c
nvc -c -o lbm.o -DSPEC -DNDEBUG -DSPEC  -O3 -acc           -DSPEC_OPENACC       lbm.c
% runaccel -c pmodel --fake --define model=acc 404.lbm | grep lbm.c
nvc -c -o lbm.o -DSPEC -DNDEBUG -DSPEC  -O3 -acc           -DSPEC_OPENACC       lbm.c
% runaccel -c pmodel --fake --define model=loop 404.lbm | grep lbm.c
nvc -c -o lbm.o -DSPEC -DNDEBUG -DSPEC  -O3 -mp=gpu           -DSPEC_OPENMP -DSPEC_OPENMP_LOOP        lbm.c
% runaccel -c pmodel --fake --define model=dist 404.lbm | grep lbm.c
nvc -c -o lbm.o -DSPEC -DNDEBUG -DSPEC  -O3 -mp=gpu           -DSPEC_OPENMP -DSPEC_OPENMP_TARGET        lbm.c
% runaccel -c pmodel --fake --define model=simd 404.lbm | grep lbm.c
nvc -c -o lbm.o -DSPEC -DNDEBUG -DSPEC  -O3 -mp=gpu           -DSPEC_OPENMP -DSPEC_OPENMP_INNER_SIMD        lbm.c
% runaccel -c pmodel --fake --define model=simd -T peak 404.lbm | grep ERROR
ERROR: pmodel define flag "-DSPEC_OPENMP_TARGET" may only be set by the tools.

III.C. Creating your own Make variables

Variables with a dollar sign and parentheses, aka "round brackets", are substituted by specmake.

Deprecated feature alert: Although it is also possible to pass information to specmake using curly brackets: ${COMPILER_DIR}, this is not recommended. Instead, you should consistently use curly brackets to address runaccel and round brackets to address specmake. It is possible that a future version of runaccel may insist on interpolating curly brackets itself, rather than allowing specmake to do so.

Example:

$ cat makevar.cfg 
action            = build
runlist           = 404.lbm
default:
   DEBUG_SYMBOLS  = --debug:symbols=expanded_info_level:42
   EXTRA_FFLAGS   = $(GEE)
small=base:
   GEE            = -g $(DEBUG_SYMBOLS)
small=peak:
   GEE            = -g 

$ cat makevar.sh
runaccel --config=makevar --fake --tune=base | grep COMP:
runaccel --config=makevar --fake --tune=peak | grep COMP:
$ ./makevar.sh 
COMP: "cc -c -o lbm.o -g --debug:symbols=expanded_info_level:42 <source>"
COMP: "cc -c -o lbm.o -g <source>"
$  

The config file above creates two variables options (DEBUG_SYMBOLS and GEE). Both are passed to specmake, which interprets them. The results are shown above using the runaccel --fake option.

For an extensive example of variable substitution handled by specmake, see the SPEC CPU®2000 example at www.spec.org/cpu2000/docs/example-advanced.cfg. Search that file for LIBS, and note the long comment which provides a walk-through of a complex substitution handled by specmake.

III.D. The operator "+=" is available (but should be used with caution)

The operator "+=" adds to specmake variables. It may be convenient; it also may cause hard-to-diagnose bugs. Example:

$ cat tmp.cfg  
action  = build
runlist = 404.lbm
tune    = peak
default:
   OPTIMIZE   = -O1
fprate=default:
   OPTIMIZE  += --unroll
default=peak:
   OPTIMIZE  += --inner_unroll
404.lbm=peak:
   OPTIMIZE  += --outer_unroll
default=default=breakfast:
   OPTIMIZE   = --jelly_roll

$ runaccel --fake --config=tmp | grep lbm.c
cc -c -o lbm.o -DSPEC -DSPEC_CPU -DNDEBUG -O1 --inner_unroll --unroll --outer_unroll lbm.c
$   

Note that the options accumulate.

Caution: although the += operator adds flexibility, it may introduce hard-to-predict behavior, depending on precedence of section specifiers, the order of your config file, and other features, such as include files. Instead of using '+=' try picking different make variables for different purposes. For an example of hard-to-predict behavior, what will happen if you add --label=breakfast to the above runaccel command? (Try it.)

Recommendations:
Avoid += to prevent surprises.
Keep it simple.
Pick different makevars for different purposes.
Create conventions for your config files and write them down, in config file comments.
If you must use += review its effects carefully (--fake is your friend).

III.E Using action=buildsetup to create a sandbox

When debugging a set of build options, it is often useful to create a "sandbox" - that is, a directory where you can play with the benchmark and its options. This example creates a build sandbox with action buildsetup.

$ cat sandbox.cfg 
action        = buildsetup
label         = fast
output_root   = /tmp/demo_buildsetup
runlist       = 404.lbm
tune          = peak
default=peak:
   OPTIMIZE   = -fast
$ cat sandbox.sh 
runaccel --config=sandbox | grep log
grep Makefile.spec /tmp/demo_buildsetup/result/accel2023.001.log
$ ./sandbox.sh 
The log for this run is in /tmp/demo_buildsetup/result/accel2023.001.log
Wrote to makefile '/tmp/demo_buildsetup/benchspec/ACCEL/404.lbm/build/build_peak_fast.0000/Makefile.spec':

The action causes a directory to be created
There, the Makefile can be examined, used in a dry run, or modified as part of a testing effort.

$ cd /tmp/demo_buildsetup/benchspec/ACCEL/404.lbm/build/build_peak_fast.0000/
$ grep OPTIMIZE Makefile.spec 
OPTIMIZE         = -fast
$ specmake --dry-run
cc -c -o lbm.o -DSPEC -DNDEBUG   -fast                  lbm.c
cc -c -o main.o -DSPEC -DNDEBUG   -fast                  main.c
cc -c -o specrand/specrand.o -DSPEC -DNDEBUG   -fast                  specrand/specrand.c
cc      -fast          lbm.o main.o specrand/specrand.o             -lm         -o lbm

See also the chapter on specmake in SPECaccel 2023 Utilities and the sandbox examples in Avoiding runaccel.

III.F. About Automatic Rebuilds

The SPECaccel tools try to keep config files and binaries synchronized with each other.(*)
Edits to a config file may cause binaries to be rebuilt, sometimes to the surprise(**) of testers.

Testing option sensitivity: The first thing that happens in a rebuild is to delete the old binary.
If that is a potential problem (perhaps it takes a long time to build), you can test whether a config file change will cause a rebuild:

  1. Copy the config file to a test version (say, mycopy.cfg)
  2. Edit the copy.
  3. Remove any lines that change defaults for ignore_errors or verify_binaries
  4. Use: runaccel --config=mycopy --nobuild --size=test --iterations=1 ...

The command:

No binaries are harmed.

Notes:

(*) Unless you change verify_binaries. Recommendation: Don't change it.

(**) Recent implementations surprise less often. FIXME FIXME FIXME For detail, see the CPU2006 version of this section.

IV. Config file options for the shell

Some options in your config file cause commands to be executed by your shell (/bin/sh).

IV.A. \$SHELLVAR variable substitution

Substitution by the shell uses backslash dollar sign.
The backslash protects the variables from interpretation by runaccel.

Example: This config file runs 404.lbm twice, with base and peak options. Only peak uses backslashes:

$ cat tmp.cfg
expand_notes         = 1
iterations           = 1
runlist              = 404.lbm
size                 = test
tune                 = base,peak
default:
   CC                = gcc
   CC_VERSION_OPTION = --version
default=base:
   submit    = echo home=$HOME, spec=$SPEC   > /tmp/chan; ${command}
default=peak:
   submit    = echo home=\$HOME, spec=\$SPEC > /tmp/nui;  ${command}

In base, $HOME and $SPEC are gobbled up by runaccel, which obediently retuns their values: nothing at all.
In peak, backslashes prevent runaccel from interpreting the variables, and the shell provides their values.

IV.B. Shell Options

Warning: SPECaccel config files can execute arbitrary shell commands.
Read a config file before using it.
Don't be root. Don't run as Administrator. Turn privileges off.

These options cause commands to be executed:

bench_post_setup

Command to be executed after each benchmark's run directory setup phase. The rules say that this feature may be used to cause data to be written to stable storage (e.g. sync). The command must be the same for all benchmarks. It will be run after each benchmark is setup, and for all workloads (test/train/ref). It is not affected by the setting of parallel_test.

Use the header section for bench_post_setup.

build_pre_bench
build_post_bench
Commands for benchmark monitoring, described in the document on the Monitoring Facility.
monitor_X Commands that allow benchmark monitoring, described in the document on the Monitoring Facility.
post_setup

Command to be executed after all benchmark run directories have been set up. The rules say that this feature may be used to cause data to be written to stable storage (e.g. sync).

Notes:

  1. This option may only be specified in the header section.
  2. The post_setup option is not executed if minimize_rundirs is set.
  3. If parallel_test is greater than 1, post_setup is not executed for test and train.
submit

Modified command to actually run a benchmark. The default is in ${command}, which the rules allow you to supplement, for example by including the mpiexec command or batch scheduler command.

IV.C. Using submit

The config file feature submit allows you specify the command needed to launch your command as well as include and process to core bindings. This section provides examples to demonstrate how submit works with several other config file features. You might also want to search published results at www.spec.org/accel2023 for systems that are similar to your system.

IV.C.1. Basic usage of submit

You can use submit to specify scripts to perform binding before the benchmark command.

Submit Example 1: ${command} and mpirun

The example below runs user supplied "bind.sh" which binds CPU threads to cores, or which GPU to use.

$ cat submit.cfg 
runlist  = 404.lbm
submit   = sh $[top]/bind.sh ${command}

IV.C.2. Reporting of submit usage

If you use the submit feature, a notes section will automatically be created to indicate that you have done so.

                                 Submit Notes
                                 ------------
    The config file option 'submit' was used.

You can add notes to that section, or customize it as you wish, by creating lines with notes_submit_NNN. The phrase The config file option 'submit' was used must appear somewhere in your customized notes. You can vary the capitalization of the phrase, you can even break it across multiple lines; it just needs to be present. If it is not, it will automatically be added. If you used scripts in your submit command, the full text of the script should be listed in the submit notes.

V. Config file options for readers

Whether or not you send your result to SPEC, you should fully disclose how you achieved the result. If it requires the installation of the GoFastLinker, you should say so. By setting the appropriate fields in the config file, you can cause information about the GoFastLinker to appear in the reports that are intended for humans.

V.A. Descriptive fields

Here are the fields that you can set to describe your testbed to readers:

fw_bios The CPU Firware and BIOS version.
hw_accel_connect Describes how the accelerator is connected to the system under test. Possible values include but not limited to integrated, PCIe, none.
hw_accel_desc Provide further details about the accelerator that are important to performance. Things that might be of interest include memory sizes, number of cores, number of vectors, etc.
hw_accel_ecc This is a Yes or No field stating if the accelerator uses ECC for its memory.
hw_accel_model The model name of the accelerator.
hw_accel_name The name of the accelerator.
hw_accel_type The type of accelerator. Possible values include, but not limited to: GPU, APU, CPU, etc.
hw_accel_vendor The company/vendor of the accelerator.
hw_avail Date hardware first shipped. If more than one date applies, use the LATEST one.
hw_cpu_max_mhz Maximum speed of the CPUs, in MHz. This may be what is referred to as the maximum turbo speed of the CPUs. If turbo is disabled, or there is no other speed of the CPUs, please just list the normal operating speed (same as hw_cpu_mhz).
hw_cpu_name Manufacturer-determined formal processor name.
hw_cpu_nominal_mhz Nominal speed of the CPUs, in MHz. This may be what is referred to as the normal operating speed of the CPUs.
hw_disk

Disk subsystem for the SPEC run directories. Three important Notes:

  1. Only a single file system may be used for the run directories, as described in the run rules at Rule 3.1.2.
  2. If the output_root feature is used, its disk subsystem should be the one described here.
  3. Disks used for other purposes - such as the system disk - are usually not a performance-relevant variable, so need not be described. Note that if you wish to do so, you can extend this field using the same continuation rules as other fields (so, you could use hw_disk01, hw_disk02, etc).
hw_memory Size of main memory (and other performance-relevant information about memory, as discussed in the run rules.)
hw_model Model name.
hw_nchips Number of CPU chips configured. See the discussion of CPU counting in the run rules.
hw_ncores Number of CPU cores configured. See the discussion of CPU counting in the run rules.
hw_ncpuorder Valid number of processors orderable for this model, including a unit. For example, "2, 4, 6, or 8 chips".
hw_nthreadspercore Number of hardware threads per core. See the discussion of CPU counting in the run rules.
hw_other Any other performance-relevant hardware.
hw_pcache 1st level (primary) cache.
hw_scache 2nd level cache.
hw_tcache 3rd level cache.
hw_ocache 4th level or other form of cache.
hw_vendor The hardware vendor. An example of usage of this and related fields is given in the test_sponsor section.
license_num The SPEC license number for either the tester or the test_sponsor.
prepared_by Is never outputed. If you wish, you could set this to your own name, so that the rawfile will be tagged with your name but not the formal reports.
sw_accel_driver The name and version of the software driver used to control the accelerator.
sw_avail Availability date for the software used. If more than one date, use the LATEST one.
sw_compiler Name and version of compiler. Note that if more than one compiler is used, you can employ continuation lines. (This applies to most of the fields discussed here, but is emphasized for sw_compiler because it is common that testers first find themselves wanting to use continuation lines when documenting their compiler set.)
sw_file

File system (ntfs, ufs, nfs, etc) for the SPEC run directories. Three important Notes:

  1. Only a single file system may be used for the run directories, as described in the run rules at Rule 3.1.2.
  2. If the output_root feature is used, its file system should be the one described here.
  3. File systems used for other purposes - such as for the operating system - are usually not a performance-relevant variable, so need not be described. But if you wish to do so, you can extend this field using the same continuation rules as other fields (so, you could use sw_file01, sw_file02, etc).
sw_os Operating system name and version.
sw_other Any other performance-relevant non-compiler software used, including third-party libraries, accelerators, etc.
sw_state Multi-user, single-user, default, etc. The rules have been clarified regarding documentation of system state and tuning, in rules 3.1.1, 4.2.4 (paragraphs c and d), and 4.2.5 (paragraph g).
tester The entity actually carrying out the tests. An optional field; if not specified, defaults to test_sponsor. An example is given in the test_sponsor section.
test_date When the tests were run. This field is populated automatically based on the clock in the system under test. Setting this in the config file will generate a warning and the setting will be ignored. If your system clock is incorrect, then the value may be edited in the raw file (see utility.html). It's better to avoid the necessity to edit, by setting your system clock properly.
test_sponsor

The entity sponsoring this test. An optional field; if not specified, defaults to hw_vendor.

For example, suppose that the Genius Compiler Company wants to show off their new compiler on the TurboBlaster 9000 computer, but does not happen to own a maxed-out system with eight thousand processors. Meanwhile, the Pawtuckaway State College Engineering department has just taken delivery of such a system. In this case, the compiler company could contract with the college to test their compiler on the big machine. The fields could be set as:

      test_sponsor = Genius Compilers
      tester       = Pawtuckaway State College
      hw_vendor    = TurboBlaster

V.B. Fields can vary by scope.

Fields can appear and disappear based upon scope. For example, you can define the compiler optimizations you use for all benchmarks while adding those for peak only in the peak section. You can also have notes which apply to all runs and add additional notes for another section.

Most (*) fields can be continued to another line by appending a numeral, for example sw_os1, sw_os2, sw_os3 if you need 3 lines to fully identify your operating system.

Here is an example that uses both of these features:

   default:
   OPTIMIZE    = -fast -xipo=2
   notes_000   =  System used mega-threading scheduler.

   default=peak:
   COPTIMIZE    = -xhyper -lsmash
   FOPTIMIZE    = -xblaster -lpile
   notes_001    = Smash library was used to allow potater tool optimization.
   notes_002    = Pile library allowed dynamic adjustment of spin delay, set with BLASTER=17.

In the above example, the information about the mega-threading scheduler will be printed for both base and peak runs. The information about the smash and pile libraries will only be printed for peak runs.

Note that the numerals above will be converted to a more formal style of three-digit numbers with leading zeros in the raw file; as discussed in utility.html, if you edit a rawfile you must use the more formal style of 000, 001, etc.

(*) The fields which cannot be continued are the ones that are expecting a simple integer, such as: hw_nchip, hw_ncores, hw_ncoresperchip, hw_nthreadspercore, license_num; and the ones that are expecting a date: hw_avail, and sw_avail.

V.C. Additional notes for the reader

In addition to the pre-defined fields, you can write as many notes as you wish. These notes are printed in the report, using a fixed-width font. For example, you can use notes to describe software or hardware information with more detail beyond the predefined fields:

   notes_os_001  = The operating system used service pack 2 plus patches 
   notes_os_002  = 31415, 92653, and 58979.  At installation time, the 
   notes_os_003  = optional "Numa Performance Package" was selected.

V.C.1. Notes sections

There are various notes sections. If there are no notes in a particular section, it is not output, so you don't need to worry about making sure you have something in each section.

The sections, in order of appearance, are as follows:

  1. notes_comp_NNN -- Notes about compiler invocation.
  2. notes_port_NNN -- Notes about portability options.
  3. notes_base_NNN -- Notes about base optimization options.
  4. notes_peak_NNN -- Notes about peak optimization options.
  5. notes_submit_NNN -- Notes about use of the submit option.
  6. notes_os_NNN -- Notes about operating system tuning and changes.
  7. notes_plat_NNN -- Notes about platform tuning and changes.
  8. notes_part_NNN -- Notes about component parts (for kit-built systems).
  9. notes_NNN -- General notes.

Notes about the submit command are described above, with the description of the submit option.

V.C.2. Note numbering

Start your notes with the name of the notes section where you want the note to appear, and then add numbers to define the order of the lines. Within a section, notes are sorted by line number. The NNN above is not intended to indicate that you are restricted to 3 digits; you can use a smaller or larger number of digits as you wish, and you can skip around as you like: for example, ex-BASIC programmers might naturally use line numbers 100, 110, 120... But note that if you say notes_plat782348320742972403 you just might encounter the dreaded (and highly unusual) "out of memory" error, so don't do that.

You can optionally include an underscore just before the number, but beware: if you say both notes_plat_105 and notes_plat105, both are considered to be the same line. The last one mentioned will replace the first, and it will be the only one output.

V.C.3. Additional tags

For all sections you can add an optional additional tag of your choosing before the numbers. Notes will be organized within the tags.

The intent of the feature is that it may allow you to organize your system information in a manner that better suits your own categories for describing it.

For example:

$ cat notes_tags.cfg 
iterations      = 1
output_format   = text
output_root     = /tmp/notes_tags
runlist         = 404.lbm
size            = test
small=base:
   CC                       = mpicc
   CC_VERSION_OPTION        = -V
   notes_part_greeting_011  = ++ how
   notes_part_greeting_20   = ++ you?
   notes_part_greeting_012  = ++ are
   notes_part_aname_1       = ++ Veronica,
   notes_part_080           = ++ hi

$ cat notes_tags.sh  
runaccel --config=notes_tags --fakereport 404.lbm | grep txt
grep '++' /tmp/notes_tags/result/accel2023.001.test.txt
$ ./notes_tags.sh
    format: Text -> /tmp/notes_tags/result/accel2023.001.test.txt
     ++ hi
     ++ Veronica,
     ++ how
     ++ are
     ++ you?
$ 

V.C.4. Links in notes sections

You can mention URLs in your notes section, and html reports will correctly render them as hyperlinks. For example:

notes_plat_001 = Additional detail may be found at
notes_plat_002 = http://www.turboblaster.com/servers/big/green/

If you like, you can use descriptive text for the link by preceding it by the word LINK and adding the descriptive text in square brackets:

LINK url AS [descriptive text]

The brackets may be omitted if your descriptive text is a single word, without blanks.

For example:

notes_plat_001 = Additional detail may be found at 
notes_plat_002 = LINK http://www.turboblaster.com/servers/big/green/ AS [TurboBlaster Servers]

When the above appears in an html report, it is rendered as:

 Additional detail may be found at 
 TurboBlaster Servers
      

And in a text report, it appears as:

                                Platform Notes
                                --------------
     Additional detail may found at 
     TurboBlaster Servers (http://www.turboblaster.com/servers/big/green/)

Since the text report is not a context in which the reader can click on a link, it is spelled out instead. Note that because the text report spells the link out, the text line is wider than in HTML, PS, and PDF reports. When deciding where to break your notes lines, you'll have to pick whether to plan line widths for text (which may result in thin-looking lines elsewhere) or plan your line widths for HTML/PS/PDF (which may result in lines that fall of the right edge with text). The feature notes_wrap_columns won't help you here, since it is applied before the link is spelled out.

VI. The config file preprocessor

VI.A. Introduction

The SPECaccel tools include a configuration file macro preprocessor. This section introduces preprocessor examples, and then discusses a few basics of syntax.

VI.A.1. Preprocessor Example: Picking a Model

In this example, the user is testing several configurations, and wants to pick an appropriate pmodel and set the appropriate optimization and portability flags..

$ cat -n preprocessor_example1.cfg
     1  %if %{model} eq 'acc'
     2  pmodel=ACC
     3  OPTIMIZE     = -fast -acc
     4  %endif
     5
     6  %if %{model} eq 'lop'
     7  pmodel=LOP
     8  OPTIMIZE     = -fast -mp=gpu
     9
    10  403.stencil:
    11  PORTABILITY += -DSPEC_NO_NOTHING
    12
    13  %endif

Lines that start with percent (%) are preprocessor directives. The percent (%) character must be in column 1.

The config file preprocessor is called a macro processor because it allows you to define macros, which are brief abbreviations for longer constructs. In the example above, two macros are used: (%{model}, and %{usegpu}). Macro values are tested, and hunks of config file are included or excluded, using the conditionals %if (lines 2 and 5), and %elif (line 4). One of the conditionals includes an %else clause (line 7).

The preprocessor is automatically run whenever you use runaccel, and macros may be set on the runaccel command line. Or, you can run it separately, as configpp:

% specperl bin/harness/configpp -c preprocessor_example1.cfg --define model=acc | grep -e pmodel
pmodel=ACC
% specperl bin/harness/configpp -c preprocessor_example1.cfg --define model=lop | grep -e pmodel -e stencil -e PORT
pmodel=LOP
403.stencil:
PORTABILITY += -DSPEC_NO_NOTHING

Notice above that preprocessor variables can be set on the command line, using either --define or '-S'.

VI.A.2. Preprocessor Syntax Basics

Emphasis: Column 1. Always punch column 1.

All preprocessor directives begin with the percent sign (%) in column 1. You can -- and probably should -- use indenting, but keep the percent sign on the far left. Any amount of spaces or tabs may separate the percent from the directive. The following are okay:

%define foo
%    define bar
%		undef hello!

The following are not okay:

 %define foo           Space in the first column
	%define foo    Tab in the first column
#define foo            <--wrong!

Did you catch why the last one is wrong? This isn't CPP - use percent, not hash!

One line per directive

The macro preprocessor does NOT follow the same quoting rules described elsewhere for config files. In particular, you may not use line continuation, line appending, or block quotes. You may have a value of arbitrary length, but in the interests of config file readability and maintainability, please keep them relatively short.

VI.B. Defining macros

Macro names may only be composed of alphanumeric characters, underscores, and hyphens, and they ARE case-sensitive.

Macros can be defined in a config file or on the command line. Both ways are equivalent.

Macros set on the command-line are defined first; therefore, it is not possible to use the command line to override a macro definition that occurs in the config file itself. If you want the command line to "win", see the notes below about redefinition.

A macro that has not been defined will not be substituted.

A macro that has been defined, but not given a value,

  • behaves as if it were 1 in a numeric context
  • behaves as if it were true in a logical context (because for Perl, 1 is true)
  • behaves as an empty string in a string context

See the example in the next section.

Having no value is not the same as having an empty string as the value. The following are NOT equivalent:

--define FOO="" FOO is defined, is false, and behaves as an empty string in a string context.
--define FOO FOO is defined, is true,  and behaves as an empty string in a string context.

See the table about Truth and definition, below.

VI.B.1. In a config file

To define a macro in a config file, use the %define preprocessor directive. Note that no quoting is necessary when specifying the names of macros or their values.

# Define a simple macro
%define foo bar
# Now the macro called 'foo' has the value 'bar'

# It's not necessary for a macro to have a value to be useful
%define baz
# Now the macro called 'baz' is defined, but it has no value.

Above, the macro baz has been defined, but has not been assigned a value.
Nevertheless, if you use it in an expression, it will behave as if it has the value 1.

$ cat macroDefinedButNoValue.cfg 
%define baz
%ifdef %{baz}
%   info baz is defined. In text contexts, it looks like this: "%{baz}".
%endif
%if %{baz} 
%   info In a logical context, it behaves as if it were 1 (i.e. true)
%endif
%if %{baz} + 3  == 4
%   info In a numeric context, it behaves as if it were 1.
%endif
$ cat macroDefinedButNoValue.sh  
specperl bin/harness/configpp --config=macroDefinedButNoValue | grep INFO
$ ./macroDefinedButNoValue.sh 
  INFO: baz is defined. In text contexts, it looks like this: "".
  INFO: In a logical context, it behaves as if it were 1 (i.e. true)
  INFO: In a numeric context, it behaves as if it were 1.
$ 

VI.B.2. On the command line

Macros can also be defined on the command line, using the --define switch. You may find it convenient to abbreviate that switch, if you prefer. The following are entirely equivalent:

$ runaccel --define mymacro=something
$ runaccel -S mymacro=something

VI.B.3. Predefined macros endian, hostname... %{ENV_variable_name}

Some useful macros are predefined, as shown in the table below. New with SPECaccel 2023: The config file preprocessor can do environment variable substitution. Environment variables to be substituted use a percent sign and curly brackets, and the name is prefixed with the string "ENV_", as %{ENV_variable_name}.

%{configfile} The name of the config file specified on the command line (possibly not including directory path)
%{endian} Indicator of endian characteristics, for example '12345678' indicating a little-endian system
%{hostname} System Under Test name
%{runaccel} Your original runaccel command
%{top_config_file} The name of the top-level config file (likely the same as %{configfile}, one may have the full path and the other not)
%{current_config_file} The name of the config file currently being read (same as %{configfile} unless reading an included config file)
%{parent_config_file} The name of the config file that included the config file currently being read (empty for the top-level config file)
%{config_nesting_level} The number of nested includes in effect (0 for the top-level config file)
%{ENV_variable}

Any of your environment variables.

For a complete list, see dumpmacros.

Automating output_root: You can automatically bring an output_root into your config file using %{ENV_GO} and navigate around it, as shown in the example for the ogo utility.

VI.B.4. Example: Adjusting the Environment

Notice that %{ENV_variable_name} brings an environment variable into your config file; what if you want to push it back out to affect other things? In that case, you will also need the feature preenv, which causes runaccel to re-invoke itself with your requested variable in the environment.

For example, you could set up paths for your compiler using something like this:

$ cat -n test.cfg
     1  action       = build
     2  output_root  = /tmp/me
     3  rebuild      = 1
     4  runlist      = 404.lbm
     5  verbose      = 99
     6
     7  %define tentop /SW/compilers/GCC/Linux/x86_64/gcc-10.1.0
     8  %define ninetop  /SW/compilers/GCC/Linux/x86_64/gcc-9.2.0
     9
    10  %ifdef %{wantGcc9}
    11     preENV_PATH            = %{ninetop}/bin:%{ENV_PATH}
    12     preENV_LD_LIBRARY_PATH = %{ninetop}/lib64:%{ENV_LD_LIBRARY_PATH}
    13  % else
    14     preENV_PATH            = %{tentop}/bin:%{ENV_PATH}
    15     preENV_LD_LIBRARY_PATH = %{tentop}/lib64:%{ENV_LD_LIBRARY_PATH}
    16  %endif
    17
    18  default:
    19     CC                = gcc
    20     CC_VERSION_OPTION = --version
    21
$ 

The config file above builds a single benchmark, 404.lbm, using the local paths for GCC 9.2 or 10.1. Notice that in both cases, the desired path is added to the existing path: for example, on line 12, the right side accesses the existing setting via %{ENV_LD_LIBRARY_PATH} and the left side applies it to the build, via preENV_LD_LIBRARY_PATH. On line 5, the config file sets verbose=99, so that runaccel will print all possible detail and we will be able to confirm the adjusted variables.

VI.B.5. Redefinition

You will receive a warning if a previously defined macro is re-defined. In the example below, the first config file just sets a macro; the second one tests before acting.

$ cat preproc.symbol.just_define.cfg 
%define build_ncpus 20
makeflags   = -j%{build_ncpus}
$ 
$ cat preproc.symbol.test_first.cfg 
%ifndef %{build_ncpus}
%   define build_ncpus 20
%endif
makeflags   = -j%{build_ncpus}
$ 

A run script uses both config files with and without a definition on the runaccel command line:

$ cat preproc.symbol.sh 
JUST_DEFINE="configpp --config=preproc.symbol.just_define.cfg"
 TEST_FIRST="configpp --config=preproc.symbol.test_first.cfg"
    GREP_IT='grep -e WARNING -e makeflags'
echo original:
$JUST_DEFINE --define build_ncpus=8 | $GREP_IT
$JUST_DEFINE                        | $GREP_IT
echo
echo test1st: 
$TEST_FIRST  --define build_ncpus=8 | $GREP_IT
$TEST_FIRST                         | $GREP_IT
$

Result: testing first with ifndef avoids a warning and does a better job of respecting the user's wishes.

$ ./preproc.symbol.sh 
original:
WARNING: Redefinition of preprocessor macro 'build_ncpus' on line 1
makeflags   = -j20
makeflags   = -j20

test1st:
makeflags   = -j8
makeflags   = -j20
$  

VI.C. Un-doing macro definition

Sometimes you want to make the preprocessor forget about a macro that you taught it. This is easily accomplished.

Macros can be undefined in two ways:

  1. On the command line, using the --undef switch, or
  2. In the configuration file, using the '%undef' preprocessor directive.
    %define foo bar
    # Now the macro called 'foo' has the value 'bar'
    
    %undef foo
    # Now it doesn't
    

    Note that no quoting is necessary when specifying the names of macros.

Like macro definition, the undefinition requests can't affect macros set in the config file because those definitions effectively happen after the un-definition. For this reason, command-line undefinition is basically useless; it can only undo macros also set on the command line.
So why was such a useless ability added to the tools? The writer likes orthogonality.

VI.D. Using macros

By now you're probably over that initial euphoric rush that comes from wantonly defining and undefining macros, and you're looking for something more. This is it!

When you want to use the value of a macro, you refer to it by name. Unfortunately, the syntax for this is not as simple as you might hope. It's not too complicated, though; to have the preprocessor expand the macro 'foo', just write

  %{foo}

in the place where you'd like it to appear. Given the following config file snippet:

%define foo Hello_
%define bar baz
%define foobar Huh?
%define foobaz What?
%define Hello_baz Please don't do this

Here's a handy table to see the various ways you can reference these values:

Macro reference Value
%{foo} Hello_
%{bar} baz
%{foobar} Huh?
%{foobaz} What?
%{Hello_baz} Please don't do this

Easy, right? The following is also possible:

%{foo%{bar}} What?
%{%{foo}%{bar}} Please don't do this

Because macro values can only be one line long, it's not possible to use the preprocessor to macro-ize large chunks of your config file at once, as may be common practice for advanced users of CPP.

A macro that has not been defined will not be substituted. Note the mention of %{FOO} on the last line of this config file.

$ cat macroDef.cfg 
action          = build
output_format   = text
output_root     = /tmp/mDef
rebuild         = 1
runlist         = 404.lbm
size            = test
teeout          = 1
default:
   CC                 = gcc
   C_VERSION_OPTION   = --version 
   OPTIMIZE           = %{FOO}
$

The failure for the first test below is expected: the GNU C compiler does not know what to do with ${FOO}.

$ cat macroDef1.sh 
runaccel --config=macroDef | grep -e lbm.o -e ^Error -e uccess
$ ./macroDef1.sh 
rm -rf *.o  lbm.out
gcc -c -o lbm.o -DSPEC -DNDEBUG -DSPEC  %{FOO}           -DSPEC_OPENMP -DSPEC_OPENMP_LOOP        lbm.c
specmake: *** [/tmp/SPECACCEL/benchspec/Makefile.defaults:365: lbm.o] Error 1
Error with make 'specmake --output-sync build':
$ cat /tmp/mDef/benchspec/ACCEL/404.lbm/build/build*/make.out
gcc -c -o lbm.o -DSPEC -DNDEBUG -DSPEC  %{FOO}           -DSPEC_OPENMP -DSPEC_OPENMP_LOOP        lbm.c
gcc: error: %{FOO}: No such file or directory
specmake: *** [/tmp/SPECACCEL/benchspec/Makefile.defaults:365: lbm.o] Error 1
$

If you want to substitute an empty string, then assign it one.

$ cat macroDef2.sh 
runaccel --config=macroDef --define FOO="" | grep -e lbm.o -e ^Error -e uccess
$ ./macroDef2.sh 
rm -rf *.o  lbm.out
gcc -c -o lbm.o -DSPEC -DNDEBUG -DSPEC             -DSPEC_OPENMP -DSPEC_OPENMP_LOOP        lbm.c
gcc                lbm.o main.o             -lm         -o lbm
$ 

VI.E. Conditionals

Defining, undefining, and expanding macros is quite an enjoyable activity in and of itself, and can even be useful on occasion. However, conditionals add an entirely new dimension to config file processing: the ability to include and exclude entire sections of text based on macros and their values.

VI.E.1. %ifdef .. %endif

The %ifdef conditional provides a way to determine whether or not a particular macro has been defined. If the named macro has been defined, the conditional is true, and the text to the matching %endif is included in the text of the config file as evaluated by runaccel. Note that the matching %endif may not necessarily be the next %endif; conditionals may be nested.

For example, given the following section of a config file:

%define foo
%ifdef %{foo}
This text will be included
%endif

%ifdef %{bar}
This text will not be included
%endif

The preprocessor would produce the following output:

This text will be included

Note especially the quoting used for the macro names in the conditional; the only time macro name quoting may be omitted is when defining or undefining it.

VI.E.2. %ifndef .. %endif

The %ifndef conditional is the converse of %ifdef; If the named macro has not been defined, the conditional is true, and the text to the matching %endif is included in the text of the config file as evaluated by runaccel. Note that the matching %endif may not necessarily be the next %endif; conditionals may be nested.

Given a slightly modified version of the example from earlier:

%define foo
%ifndef %{foo}
Now THIS text will not be included
%endif

%ifndef %{bar}
This text WILL be included
%endif

The preprocessor would produce the following output:

This text WILL be included

VI.E.3. %if .. %endif

Checking whether or not a macro is defined is quite useful, but it's just a subset of the more general conditional facility available. This general form is

%if expression
...
%endif

The expression is evaluated using a subset of the Perl interpreter, so the possibilities for testing values are fairly broad. For example,

%ifdef %{foo}
...
%endif

is exactly equivalent to

%if defined(%{foo})
...
%endif

Likewise,

%ifndef %{foo}
...
%endif

is exactly equivalent to

%if !defined(%{foo})
...
%endif

Using the general form, it's possible to string conditionals together:

%if defined(%{foo}) && !defined(%{bar}) || %{baz} == 0
...
%endif

To compare versus a string value, you must supply quotes:

%if %{foo} eq 'Hello, Dave.'
...
%endif

You may perform basic math on macro values:

%if %{foo} * 2004 > 3737
...
%endif

You can do basic regular expression matching, as shown in the example for %info.

More precisely, the Perl operations allowed are the :base_core and :base_math bundles, with the ability to dereference and modify variables disallowed. For more details, see the source code for config.pl (the eval_pp_conditional subroutine) and Perl's own Opcode documentation.

Truth and definition
%ifdef %{v} and
%if defined(%{v})
  vs.
%if %{v}

Conditional expressions are evaluated using perl's rules for Truth and Falsehood.
Therefore 0 (zero) is defined, and is false:

$ cat test_define.cfg
%ifdef %{a}
   OPTIMIZE = -a
%endif
%if defined(%{b})   
   LDOPTIMIZE = -b
%endif
%if %{c}
   COPTIMIZE = -c
%endif
$ specperl bin/harness/configpp --config=test_define -S a=1 -S b=1 -S c=1 | grep OPT  
   OPTIMIZE = -a
   LDOPTIMIZE = -b
   COPTIMIZE = -c
$ specperl bin/harness/configpp --config=test_define -S a=0 -S b=0 -S c=0 | grep OPT
   OPTIMIZE = -a
   LDOPTIMIZE = -b

VI.E.4. %else

It's possible to get by without the "else" part of the classic "if .. then .. else" trio, but it's not any fun. It works as you'd expect:

%define foo
%ifndef %{foo}
This text will not be included
%else
This text WILL be included (from the else clause)
%endif

The preprocessor would produce the following output:

This text WILL be included (from the else clause)

Only one %else per conditional is allowed.

VI.E.5. %elif

%elif is another convenience that's been added. For those not familiar with CPP, it's an "else if" construct. You may have as many of these as you'd like. Given:

%define foo Hello!

%if !defined(%{foo})
This text will not be included
%elif defined(%{bar})
This text won't be included either
%elif '%{foo}' eq 'Hello!'
This text WILL be included (from the second elif clause)
%else
Alas, the else here is left out as well.
%endif

The preprocessor would produce the following output:

This text WILL be included (from the second elif clause)

VI.F. Informational Directives

It's often helpful to be able to warn or exit on certain conditions. Perhaps there's a macro that must be set to a particular value, or maybe it's just very highly recommended.

VI.F.1. %warning

%warning does just what you'd expect; when the preprocessor encounters this directive, it prints the text following to stdout and the current log file, along with its location within the file being read, and continues on.

$ cat warning.cfg 
%if !defined(%{somewhat_important_macro})
%   warning You have not defined somewhat_important_macro!
%endif
$ 
$ cat warning.sh 
specperl bin/harness/configpp --config=warning | grep -C2 WARNING
$ 
$ ./warning.sh 

***
  WARNING: You have not defined somewhat_important_macro!
           (From line 2 of
           /accel2023/config/warning.cfg.)
$  

VI.F.2. %error

Like %warning, %error logs an error to stderr and the log file. Unlike %warning, though, it then stops the run.

Consider a slightly modified version of the previous example:

$ cat error.cfg 
%if !defined(%{REALLY_important_macro})
%   error You have not defined REALLY_important_macro!
%endif
$ 
$ cat error.sh 
specperl bin/harness/configpp --config=error > /tmp/out
echo runaccel exit code: $?
grep -C2 ERROR /tmp/out
$ ./error.sh 
runaccel exit code: 1

************************************************************************
  ERROR: You have not defined REALLY_important_macro!
         (From line 2 of
         /accel2023/config/error.cfg.)
$ 

Unlike a warning, the error will be close to the last thing output. As you can see from the output of echo $? above, runaccel exited with an error code 1.

VI.F.3. %info

The %info directive prints a message preceded by the word INFO:. For example,


$ cat -n info.cfg 
   1  %if !defined(%{chip}) || %{chip} !~ m/(sparc|x86)/
   2  %   error Please use --define chip=sparc or --define chip=x86
   3  %endif
   4  
   5  %if  %{chip} eq "sparc"
   6  %   define default_build_ncpus 64
   7  %elif %{chip} eq "x86"
   8  %   define default_build_ncpus 20
   9  %endif
  10  %ifndef   %{build_ncpus}
  11  %   define  build_ncpus   %{default_build_ncpus}
  12  %endif
  13  
  14  %info Preprocessor selections: 
  15  %info    .    build_ncpus      %{build_ncpus}
  16  %info    .    chip             %{chip}
  17  
  18  make = specmake --jobs=%{build_ncpus} --load-average=%{build_ncpus} 
$
$ specperl bin/harness/configpp -c info --define chip=sparc | grep -e make -e INFO
INFO: Preprocessor selections:
INFO: .    build_ncpus      64
INFO: .    chip             sparc
make = specmake --jobs=64 --load-average=64
$ 
 

Note in the example above:

  • Line 1 demonstrates basic perl regular expression matching
  • Lines 14-16 use %info
  • After all preprocessing has been completed, line 18 tells the specmake utility that it can start up to 64 compile jobs, provided that the system load average is no higher than 64.

VI.F.4. %dumpmacros

The %dumpmacros prints the values of all macros currently defined at the point where it appears. Each macro value is preceded by the word 'DBG:'. For example,


$ cat -n dumpmacros.cfg 
     1	%if !defined(%{chip}) || %{chip} !~ m/(sparc|x86)/
     2	%   error Please use --define chip=sparc or --define chip=x86
     3	%endif
     4	
     5	%if  %{chip} eq "sparc"
     6	%   define default_build_ncpus 64
     7	%elif %{chip} eq "x86"
     8	%   define default_build_ncpus 20
     9	%endif
    10	%ifndef   %{build_ncpus}
    11	%   define  build_ncpus   %{default_build_ncpus}
    12	%endif
    13	
    14	%dumpmacros
$
$ specperl bin/harness/configpp -c dumpmacros --define chip=sparc | grep -e DBG
  DBG: build_ncpus: '%{default_build_ncpus}'
  DBG: chip: 'sparc'
  DBG: default_build_ncpus: '64'
  DBG: runaccel: 'configpp -c dumpmacros --define chip=sparc'
$
 

The %dumpmacros directive is new with SPECaccel 2023.

VII. Output files - and how they relate to your config file

This section describes how the location and contents of several kinds of output files are influenced by your config file.

VII.A. Help, I've got too many config files!

It was mentioned above that the HASH section of the config file is written automatically by the tools. Each time your config file is updated, a backup copy is made. Thus your config directory may soon come to look like this:

$ cd $SPEC/config
$ ls tune.cfg*
tune.cfg                        tune.cfg.2021-02-05T124831      tune.cfg.2021-02-05T125733
tune.cfg.2021-02-05T120242      tune.cfg.2021-02-05T125557      tune.cfg.2021-02-05T125738
tune.cfg.2021-02-05T122021      tune.cfg.2021-02-05T125603      tune.cfg.2021-02-05T125744
tune.cfg.2021-02-05T122026      tune.cfg.2021-02-05T125608      tune.cfg.2021-02-05T125749
tune.cfg.2021-02-05T124215      tune.cfg.2021-02-05T125614      tune.cfg.2021-02-05T125756
tune.cfg.2021-02-05T124222      tune.cfg.2021-02-05T125620      tune.cfg.2021-02-05T125802
tune.cfg.2021-02-05T124728      tune.cfg.2021-02-05T125626      tune.cfg.2021-02-05T125807
tune.cfg.2021-02-05T124739      tune.cfg.2021-02-05T125632      tune.cfg.2021-02-05T125812
tune.cfg.2021-02-05T124821      tune.cfg.2021-02-05T125727
$  

If this feels like too much clutter, you can disable the backup mechanism, as described under backup_config. Note that doing so may leave you with a risk of losing the config file in case of a filesystem overflow or system crash. A better idea may be to periodically remove just portions of the clutter, with selective removal of older version; or sweep them to a wastebasket every now and then:

$ cd $SPEC/config
$ mkdir wastebasket
$ mv *cfg.2021* wastebasket/
$

VII.B. The log file and verbosity levels

$SPEC/result (Unix) contains reports and log files. When you are doing a build, you will probably find that you want to pay close attention to the log files such as accel2023.001.log. Depending on the verbosity level that you have selected, it will contain detailed information about how your build went.

The SPECaccel 2023 tool suite provides for varying amounts of output about its actions during a run. These levels range from the bare minimum of output (level 0) to copious streams of information that are probably useful only to tools developers (level 99). Selecting one output level gives you the output from all lower levels, which may cause you to wade through more output than you might like.

VII.B.1. Useful Search Strings

When you are trying to find your way through a log file, you will probably find these (case-sensitive) search strings useful:

runaccel:The runaccel command for this log.
RunningPrinted at the top of a run of a benchmark.
#Printed at the top of a run of a benchmark for runs with multiple iterations. Useful for finding the ref workloads in reportable runs.
runtimePrinted at the end of a benchmark run.
BuildingPrinted at the beginning of a benchmark compile.
Elapsed compilePrinted at the end of a benchmark compile.

VII.B.2. About Temporary Debug Logs

There are also temporary debug logs, such as accel2023.001.log.debug. A debug log contains very detailed debugging output from the SPEC tools, as if --verbose 99 had been specified.

For a successful run, the debug log will be removed automatically, unless you specify "--keeptmp" on the command line, or "keeptmp=yes" in your config file.

For a failed run, the debug log is kept. The debug log may seem overwhelmingly wordy, repetitive, detailed, redundant, repetitive, and long-winded, and therefore useless. Suggestion: after a failure, try looking in the regular log first, which has a default verbosity level of 5. If your regular log doesn't have as much detail as you wish, then you can examine the additional detail in the debug log.

If you file a support request, you may be asked to send in the debug log.

VII.B.3. Verbosity levels

The 'level' referred to in the table below is selected either in the config file verbose option or in the runaccel command as in 'runaccel --verbose n'.

Levels higher than 99 are special; they are always output to your log file. You can also see them on the screen if you set verbosity to the specified level minus 100. For example, the default log level is 5. This means that on your screen you will get messages at levels 0 through 5, and 100 through 105. In your log file, you'll find the same messages, plus the messages at levels 106 through 199.

Level What you get
0 Basic status information, and most errors. These messages can not be turned off.
1 List of the benchmarks which will be acted upon.
2 A list of possible output formats, as well as notification when beginning and ending each phase of operation (build, setup, run, reporting).
3 A list of each action performed during each phase of operation (e.g. "Building 404.lbm", "Setting up 635.weather_s")
4 Notification of benchmarks excluded
5 (default) Notification if a benchmark somehow was built but nevertheless is not executable.
6 Time spent doing automatic flag reporting.
7 Actions to update SPEC-supplied flags files.
10 Information on basepeak operation.
12 Errors during discovery of benchmarks and output formats.
15 Information about certain updates to stored config files
24 Notification of additions to and replacements in the list of benchmarks.
30 A list of options which are included in the hash of options used to determine whether or not a given binary needs to be recompiled.
35 A list of key=value pairs that can be used in command and notes substitutions, and results of env_var settings.
40 A list of 'submit' commands for each benchmark.
Note: If you would like to see all of the submit commands for every copy and every benchmark invocation, with all your variables (such as $BIND) resolved, try runaccel --verbose=40 --fake, or go to the run directory and use specinvoke -n (dry run).
70 Information on selection of median results.
89 Progress comparing run directory checksum for executables.
90 Time required for various internal functions in the tools.
95, 96, 97, 98 Flag parsing progress during flag reporting (progressively more detail)
99 Gruesome detail of comparing hashes of files being copied during run directory setup.
Messages at the following levels will always appear in your log files
100 Various config file errors, such as bad preprocessor directives, bad placement of certain options, illegal characters...
102 Information about output formats that could not be loaded.
103 A tally of successes and failures during the run broken down by benchmark.
106 A list of runtime and calculated ratio for each benchmark run.
107 Dividers to visually block each phase of the run.
110 Elapsed time for each portion of a workload (if an executable is invoked more than once).
120 Messages about which commands are being issued for which benchmarks.
125 A listing of each individual child processes' start, end, and elapsed times.
130 A nice header with the time of the runaccel invocation and the command line used. Information about what happened with your sysinfo program
140 General information about the settings for the current run.
145 Messages about file comparisons.
150 List of commands that will be run, and details about the settings used for comparing output files. Also the contents of the makefile written.
155 Start, end, and elapsed times for benchmark run.
160 Start, end, and elapsed times for benchmark compilation.
180 stdout and stderr from commands run
190 Start and stop of delays
191 Notification of command line used to run specinvoke.

VII.D. Help, I've got too many log files!

If you do many builds and runs, you may find that your result directory gets too cluttered. Within a result directory, all output formats other than .rsf can be regenerated from your .rsf files. Therefore, you could reduce clutter by deleting HTML, PDF, and other reports. You can delete old .debug logs unless you plan to submit a support request.

Still feel cluttered? A simple solution is to move your result directory aside, giving it a new name. Don't worry about creating a new directory; runaccel will do so automatically. You should be careful to ensure no surprises for any currently-running users. If you move result directories, it is a good idea to also clean temporary directories at the same time.
Example:

cd $SPEC
mv result old-result
rm -Rf tmp/
cd output_root     # (If you use an output_root)
rm -Rf tmp/

VII.E. Finding the build directory

As described under "About Disk Usage" in runaccel.html, the SPECaccel 2023 tools do the actual builds and runs in newly created directories. The benchmark sources are never modified in the src directory.

The build directories for a benchmark are located underneath that benchmarks' top-level directory, typically $SPEC/benchspec/ACCEL/nnn.benchmark/build (Unix)

(If you are using the output_root feature, then the first part of that path will change to be your requested root instead of SPEC.)

The build directories have logical names, typically of the form build_<tune>_<label>.0000.

If the directory build_<tune>_<label>.0000 already exists when a new build is attempted for the same tuning and label, the directory will be re-used, unless:

  • You have set the environment variable SPEC_NO_RUNDIR_DEL, or
  • The directory has been left in a "locked" state by an aborted previous build.

In such cases, the 0000 will be incremented until a name is generated that is available. You can find locked directories by searching for lock=yes in the file $SPEC/benchspec/ACCEL/<nnn.benchmark>/run/list (Unix)

When more than one build directory has been created for a given tuning and label, you may need to trace the directory back to the specific build attempt that created it. You can do so by searching for the directory name in the log files:

$ grep build_peak_blue271.0000 *log | grep Building
accel2023.838.log:  Building 404.lbm peak blue271: (build_peak_blue271.0000) [2021-02-14 05:26:09]
$  

In the above example, the grep command locates log #838 as the log that corresponds to this run directory.

VII.F. Files in the build directory

A variety of files are output to the build directory. Here are some of the key files which can usefully be examined:

Makefile.spec The components for make that were generated for the current config file with the current set of runaccel options.
make.out For 1 pass compile: detailed commands generated.

VII.G. For more information

For more information about how the run directories work, see the descriptions of specinvoke, specmake, and specdiff in utility.html.

Appendix A. Troubleshooting

When something goes wrong, here are some things to check:

  1. Are there any obvious clues in the log file? Search for the word "Building". Keep searching until you hit the next benchmark AFTER the one that you are interested in. Now scroll backward one screen's worth of text.

  2. Did your desired switches get applied? Go to the build directory, and look at options*out.

  3. Did the tools or your compilers report any errors? Look in the build directory at *err.

  4. What happens if you try the build by hand? See the section on specmake in utility.html.

  5. If an actual run fails, what happens if you invoke the run by hand? See the information about "specinvoke -n" in utility.html

  6. Do you understand what is in your path, and why? Sometimes confusion can be greatly reduced by ensuring that you have only what is needed, avoiding, in particular, experimental and non-standard versions of standard utilities.

    Note: on Windows systems, SPEC recommends that Windows/Unix compatibility products should be removed from the %PATH% prior to invoking runaccel, in order to reduce the probability of certain difficult-to-diagnose error messages.

  7. Try asking the tools to leave more clues behind, with keeptmp.

Appendix B. Obsolete/removed items