Supporting tools
To enable the usage of the SCL and AMML, a minimum set of supporting tools is necessary. These tools include a parser, a correctness checker, an interpreter, and a syntax highlighter. All these tools (or subsets of them) can be used via the command-line interface or the Jupyter/iPython kernel.
Using the command-line interface
The command-line interface is invoked with the command texts.
Script mode
The command texts script runs a DSL program (also called a model) in one of three different evaluation modes. The basics on how to write a program in the Scientific Computing Language can be found in this section and Atomic and Molecular Modeling Language in this section.
In all evaluation modes, the model is first parsed using a grammar. The grammar can be optionally specified by the flag -g GRAMMAR_PATH, --grammar-path. The default grammar used for parsing is in the file src/virtmat/language/grammar/virtmat.tx. Syntax errors are issued in this phase. The interpreter then starts applying static constraints such as type checks. Static errors (such as type errors) are issued at this stage. Then the evaluation of the model is performed. The evaluation (or execution) mode can be specified using the -m | --mode option. These modes are instant, deferred, and workflow. The instant evaluation mode is the default. More details are available in the following subsections. The model source file (with extension .vm ) must always be specified using the flag -f MODEL_FILE, --model-file MODEL_FILE even if it is empty.
Instant and deferred modes
In the instant mode the evaluation is performed immediately and locally. In deferred mode the evaluation is deferred but still performed locally. In instant and deferred modes the model does not persist after the interpreter has finished, i.e. the texts script has returned. Only parameters of print statements in the program will be evaluated.
Workflow mode
In the workflow mode the intepreter creates a workflow from the model and stores the workflow in a database. Therefore, the model is persistent after the interpreter returns, i.e. the model instance continues existing on the database. A model instance can be retrieved via a universally unique identifier (UUID) that is generated at the time when the model instance is created. A model instance can only be evaluated and extended but not otherwise modified.
If no additional flags are specified the model is not evaluated after interpretation except for parameters in print statements that contain no references to variables. Otherwise n.c. will be printed instead of the not yet computed values. Optionally, a model can be evaluated locally after the intepreter finishes by specifying the -r | --autorun flag. In this case all variables will be evaluated but only those in print statements will be printed. Additionally, the -d, --on-demand flag restricts the evaluation to only those variables that are referenced in print statements.
In order to work in workflow mode, a launchpad describing a connection to a MongoDB database must be configured. Using the flag -l LAUNCHPAD_FILE, --launchpad-file LAUNCHPAD_FILE the launchpad data (such as hostname, database name and credentials) can be optionally specified. Otherwise the script looks up the current working directory and then the folder $HOME/.fireworks for a launchpad configuration. More information about configuring a launchpad can be found here.
By using the -h, --help flag a summarized help can be obtained:
texts script --help
usage: texts script [-h] -f MODEL_FILE [-g GRAMMAR_PATH] [-m {instant,deferred,workflow}] [-t] [-l LAUNCHPAD_FILE] [-u UUID] [-r] [-d] [-q QADAPTER_FILE] [-w] [--no-unique-launchdir] [--enable-logging]
[--logging-level {NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}] [--no-duplicate-detection]
Run a script
options:
-h, --help show this help message and exit
-f MODEL_FILE, --model-file MODEL_FILE
path to model file
-g GRAMMAR_PATH, --grammar-path GRAMMAR_PATH
path to grammar root file
-m {instant,deferred,workflow}, --mode {instant,deferred,workflow}
execution mode
-t, --textx-debug enable textx debug mode
-l LAUNCHPAD_FILE, --launchpad-file LAUNCHPAD_FILE
path to launchpad file (workflow mode only)
-u UUID, --uuid UUID UUID of a model to extend/run (workflow mode only)
-r, --autorun run the model (workflow mode only)
-d, --on-demand run on demand
-q QADAPTER_FILE, --qadapter-file QADAPTER_FILE
path to default qadapter file (workflow mode only)
-w, --ignore-warnings
do not display warnings
--no-unique-launchdir
disable unique launchdir
--enable-logging enable logging messages
--logging-level {NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}
logging level
--no-duplicate-detection
disable duplicate detection
A persistent model can be retrieved in workflow mode by using the flag -u UUID, --uuid UUID. A model can be optionally evaluated in workflow mode by adding the -r, --autorun and -d, --on-demand flags or/and extended by specifying a model extension with the flag -f MODEL_FILE, --model-file MODEL_FILE. The evaluation with the -r, --autorun flag is effective only for workflow nodes of interactive category.
By default, in workflow mode every statement will be evaluated in a unique launch directory. Because nodes, that are evaluated in parallel and that share a launch directory, may interfere due to possible local file I/O operations. The default use of unique launch directories can be deactivated for interactive workflow nodes by the flag --no-unique-launchdir. In this case, one has to make sure that statements giving rise to interactive workflow nodes do not cause conflicting file I/O operations. Note that batch workflow nodes are always evaluated in unique launch directories.
Interactive session
The interactive session, started with texts session, is a tool managing a session for processing a model in workflow mode. Some of the startup flags are the same as for texts script.
By default no evaluation is performed. The evaluation with the -r, --autorun without the -a, --async-run flag is effective only for workflow nodes of interactive category. The flag -a, --async-run activates background evaluation using a workflow engine with a launcher thread. The evaluation includes workflow nodes of both interactive and batch categories. For this feature the vre-middleware package must be installed. The WFEngine object used in the run is dumped into a file in the same folder as the launchpad file. This object can be used independently with the Python API or the GUI of the vre-middleware package. The flag -q, --qadapter-file is explained in detail [in this section](#evaluation-of-nodes-of-batch-category the-qadapter-object).
By using the -h, --help flag a summarized help can be obtained:
texts session --help
usage: texts session [-h] [-g GRAMMAR_PATH] [-l LAUNCHPAD_FILE] [-u UUID] [-r] [-a] [-d] [-s SLEEP_TIME] [-q QADAPTER_FILE] [-f MODEL_FILE] [-w] [--no-unique-launchdir]
[--enable-logging] [--logging-level {NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}] [--no-duplicate-detection]
Start an interactive session
options:
-h, --help show this help message and exit
-g GRAMMAR_PATH, --grammar-path GRAMMAR_PATH
path to grammar root file
-l LAUNCHPAD_FILE, --launchpad-file LAUNCHPAD_FILE
path to launchpad file
-u UUID, --uuid UUID UUID of a model
-r, --autorun run the model
-a, --async-run run in background
-d, --on-demand run on demand
-s SLEEP_TIME, --sleep-time SLEEP_TIME
sleep time for background evaluation in seconds
-q QADAPTER_FILE, --qadapter-file QADAPTER_FILE
path to default qadapter file
-f MODEL_FILE, --model-file MODEL_FILE
path to model file
-w, --ignore-warnings
do not display warnings
--no-unique-launchdir
disable unique launchdir
--enable-logging enable logging messages
--logging-level {NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}
logging level
--no-duplicate-detection
disable duplicate detection
Session behavior
In the session, inputs are interactively typed and the outputs are shown immediately. An empty model instance is created at startup and with every input the model instance is extended with the model source extension typed as input. The editing capabilities of the input fields are very limited.
Specific features
In the interactive session, if a valid parameter is typed it is interpreted as a print statement. For example, a typed input a is changed to print(a) before parsing. Multiline comments are not supported in the interactive session.
A successfully parsed and interpreted input cannot be modified. It becomes permanently part of the model source.
The session manager accepts additional inputs that are no model sources in SCL or AMML. These are sometimes called magic commands (like in iPython and Jupyter) and begin with the symbol %.
magic commands |
action |
|---|---|
|
close (quit) the session |
|
activate evaluation |
|
deactivate evaluation |
|
sleep time for background evaluation, in seconds |
|
prints the UUID of the active model |
|
switch to model with UUID |
|
create a new session and a new model |
|
prints the current model source |
|
re-evaluate var1, var2, etc. |
|
print the tag section of the current model |
|
perform a global search |
The %history command prints all statements (without print statements that are not persisted), one per line, with timestamp of most recent evaluation update and current evaluation state. The evaluation state is currently passed through from the FireWorks workflow management system and is described in detail here.
The %rerun var command can be used to re-evaluate the parameter of a variable var and its descendants, i.e. all parameters containing references to var. This is useful when there was some error during evaluation of var that was not due to the provided input to var but due to side effects, for example there was a computing node or file system failure, network or power outage etc.
Duplicate detection
Both script and session modes have a duplicate detection feature to identify variables in other persistent models that have identical names, parameters and values. This feature ensures safe reuse of such variables and helps to avoid unnecessary repeated evaluations. The duplicate detection can be used in any model but particularly useful in the cases of sub-model reuse and of bulk processing.
Duplicate detection is activated in workflow evaluation mode by default but it can be deactivated with the --no-duplicate-detection flag.
Using the Jupyter kernel
The Jupyter kernel is still work in progress. Currently, some help can be obtained in this documentation.
Interpreting the error messages
Either the parser or the interpreter may detect static errors in the program. Then they issue specific error messages in the following format:
Error type: /absolute/path/to/model_source_file.vm:line:column --> context <--
Detailed error description
The --> context <-- displays the section of the model source where the error has occurred. If the Jupyter kernel or texts session is used then None is printed instead of an absolute path to file with the model source.
NOTE: If an error message is not in this particular format, especially if it is a Python exception with a stack trace beginning with the word “Traceback”, then it is most likely due to a bug in the interpreter and not due to an error in the model source. In such cases please submit an issue with the tag Bug and provide a minimal input to reproduce the error.
Simple examples of errors
Input:
b = 1 a = 2
Error:
Syntax error: None:1:7 --> b = 1 *a = 2 <--
Expected '[' or '^|;|\,|\)|$' or '**' or '*' or '/' or '+' or '-' or '<=' or '>=' or '>' or '<' or '!=' or '==' or 'if' or 'for' or 'on' or '^|;' or EOF
All syntax errors are issued by the parser.
Input:
b = a
Error:
Unknown object: None:1:5
Unknown object "a" of class "GeneralVariable"
Unknown object error is a semantic error issued by the parser when it cannot resolve a reference.
Input:
s = 1
s = 'hello'
Error:
Initialization error: None:2:1 --> s = 'hello' <--
Repeated initialization of "s"
Repeated initialization error is issued by the interpreter when it applies semantic constraints before the beginning of evaluation.
Runtime errors
Some errors occur only during the evaluation. These are called runtime errors. Runtime errors have the same format as the static errors. Runtime errors only occur if the evaluation is requested, e.g. by print statements. For example, this input:
length = 1 [m]
b = length + 1
will produce no error message. Adding print(b) to the model and running the model in instant or deferred evaluation mode will yield this error:
Dimensionality error: None:2:5 --> length + 1 <--
Cannot convert from 'meter' ([length]) to 'dimensionless' (dimensionless)
In workflow evaluation mode this error occurs only if the evaluation is enabled and will be issued only if there is print(b) statement.
Evaluation of nodes of batch category: the qadapter object
Once a model has been created in workflow mode, thus added to the database, it can be evaluated using either texts script or texts session with the --autorun, -r flag. In this mode, the workflow nodes of interactive category are evaluated synchronously while the nodes of batch category are not. More information about the workflow node categories can be found here. To enable asynchronous background evaluation of nodes of batch category one has to add the --async-run, -a flag.
Default qadapter
The evaluation of nodes of batch category happens in an HPC environment governed by a batch system such as SLURM and requires an object called qadapter. A default qadapter will be automatically created by the workflow engine and used by the launcher thread. To this end, the default resources in the resource configuration (resconfig) object are used. These resources are: default worker, default queue, default launch directory, default group used for accounting, default queue with default computing resources (such as walltime, number of CPUs, amount of memory etc.), default loaded environment modules, default virtual environment. The resconfig object is stored in a resconfig file with default location $HOME/.fireworks/res_config.yaml that can be overridden by the environment variable RESCONFIG_LOC. If the resconfig file does not exist at the beginning of texts session, then it is generated automatically (a prompt will be displayed to accept this step). Further more technical information about res_config can be found elsewhere.
If necessary, the default qadapter can be overridden by using the optional --qadapter-file, -q flag, which takes the name of a qadapter file (e.g. “qadapter.yaml”), containing a qadapter object, as argument. This can be necessary if the default qadapter does not specify some resource needed by the batch job to run. However, the recommended way to fix this is to modify the res_config object instead of providing a default qadapter. If the user has to write a qadapter file then this tutorial can be used.
Custom qadapter
Different batch nodes may have different resource requirements and the (either auto-generated or manually provided) default qadapter may not fit to all of them. For this reason, a custom qadapter is created automatically for nodes of batch category. The resources specified in the custom qadapter override these in the default qadapter during node launch.
The custom qadapter is created automatically for models with resource annotations and added to the node pertinent to the statement with the resource annotation in the model. All resources specified in the resource annotation are included in the custom qadapter.
In addition, the custom qadapter includes the default virtual environment and all default modules so that these will be activated and loaded in the batch job before the evaluation step starts. These default settings are sourced from the res_config.
In addition, the application-specific environment modules and variables will be set. These are not sourced from the res_config but from the calculator or the algorithm that requires these resources.