# Supporting tools To enable the usage of the SCL and AMML, a minimum set of supporting tools is necessary. These tools include a parser, a correctness checker, an interpreter, and a syntax highlighter. All these tools (or subsets of them) can be used via the command-line interface or the Jupyter/iPython kernel. ## Using the command-line interface The command-line interface is invoked with the command `texts`. ### Script mode The command `texts script` runs a DSL *program* (also called a *model*) in one of three different evaluation modes. The basics on how to write a program in the Scientific Computing Language can be found in [this section](scl.md) and Atomic and Molecular Modeling Language in [this section](amml.md). In all evaluation modes, the model is first parsed using a grammar. The grammar can be optionally specified by the flag `-g GRAMMAR_PATH, --grammar-path`. The default grammar used for parsing is in the file `src/virtmat/language/grammar/virtmat.tx`. Syntax errors are issued in this phase. The interpreter then starts applying static constraints such as type checks. Static errors (such as type errors) are issued at this stage. Then the evaluation of the model is performed. The evaluation (or execution) mode can be specified using the `-m | --mode` option. These modes are *instant*, *deferred*, and *workflow*. The instant evaluation mode is the default. More details are available in the following subsections. The *model source file* (with extension `.vm` ) must always be specified using the flag `-f MODEL_FILE, --model-file MODEL_FILE` even if it is empty. #### Instant and deferred modes In the *instant* mode the evaluation is performed *immediately* and *locally*. In *deferred* mode the evaluation is deferred but still performed *locally*. In *instant* and *deferred* modes the model does not persist after the interpreter has finished, i.e. the `texts script` has returned. Only parameters of `print` statements in the program will be evaluated. #### Workflow mode In the *workflow* mode the intepreter creates a workflow from the model and stores the workflow in a database. Therefore, the model is *persistent* after the interpreter returns, i.e. the *model instance* continues existing on the database. A model instance can be retrieved via a *universally unique identifier* (UUID) that is generated at the time when the model instance is created. A model instance can only be evaluated and extended but not otherwise modified. If no additional flags are specified the model is not evaluated after interpretation except for parameters in `print` statements that contain no references to variables. Otherwise `n.c.` will be printed instead of the not yet computed values. Optionally, a model can be evaluated locally after the intepreter finishes by specifying the `-r | --autorun` flag. In this case all variables will be evaluated but only those in print statements will be printed. Additionally, the `-d | --on-demand` flag restricts the evaluation to only those variables that are referenced in `print` and `view` statements. In order to work in workflow mode, a launchpad describing a connection to a MongoDB database must be configured. Using the flag `-l LAUNCHPAD_FILE, --launchpad-file LAUNCHPAD_FILE` the launchpad data (such as hostname, database name and credentials) can be optionally specified. Otherwise the script looks up the current working directory and then the folder `$HOME/.fireworks` for a launchpad configuration. More information about configuring a launchpad can be found [here](https://vre-middleware.readthedocs.io/en/latest/launchpad.html). By using the `-h, --help` flag a summarized help can be obtained: ``` texts script --help usage: texts script [-h] -f MODEL_FILE [-g GRAMMAR_PATH] [-m {instant,deferred,workflow}] [-l LAUNCHPAD_FILE] [-u UUID] [-r] [-d] [-q QADAPTER_FILE] [-w] [--no-unique-launchdir] [--enable-logging] [--logging-level {NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}] [--no-duplicate-detection] Run a script options: -h, --help show this help message and exit -f MODEL_FILE, --model-file MODEL_FILE path to model file -g GRAMMAR_PATH, --grammar-path GRAMMAR_PATH path to grammar root file -m {instant,deferred,workflow}, --mode {instant,deferred,workflow} execution mode -l LAUNCHPAD_FILE, --launchpad-file LAUNCHPAD_FILE path to launchpad file (workflow mode only) -u UUID, --uuid UUID UUID of a model to extend/run (workflow mode only) -r, --autorun run the model (workflow mode only) -d, --on-demand run on demand -q QADAPTER_FILE, --qadapter-file QADAPTER_FILE path to default qadapter file (workflow mode only) -w, --ignore-warnings do not display warnings --no-unique-launchdir disable unique launchdir --enable-logging enable logging messages --logging-level {NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL} logging level --no-duplicate-detection disable duplicate detection ``` A persistent model can be retrieved in workflow mode by using the flag `-u UUID, --uuid UUID`. A model can be optionally evaluated in workflow mode by adding the `-r, --autorun` and `-d, --on-demand` flags or/and extended by specifying a model extension with the flag `-f MODEL_FILE, --model-file MODEL_FILE`. The evaluation with the `-r, --autorun` flag is effective only for workflow nodes of [_interactive_ category](resources.md#policies-for-interactive-and-batch-execution). By default, in workflow mode every statement will be evaluated in a unique launch directory. Because nodes, that are evaluated in parallel and that share a launch directory, may interfere due to possible local file I/O operations. The default use of unique launch directories can be deactivated for [_interactive_](resources.md#policies-for-interactive-and-batch-execution) workflow nodes by the flag `--no-unique-launchdir`. In this case, one has to make sure that statements giving rise to _interactive_ workflow nodes do not cause conflicting file I/O operations. Note that [_batch_](resources.md#policies-for-interactive-and-batch-execution) workflow nodes are always evaluated in unique launch directories. ### Interactive session The interactive session, started with `texts session`, is a tool managing a session for processing a model in *workflow mode*. Some of the startup flags are the same as for `texts script`. By default no evaluation is performed. The evaluation with the `-r, --autorun` without the `-a, --async-run` flag is effective only for workflow nodes of [_interactive_ category](resources.md#policies-for-interactive-and-batch-execution). The flag `-a, --async-run` activates background evaluation using a workflow engine with a launcher thread. The evaluation includes workflow nodes of both [_interactive_ and _batch_ categories](resources.md#policies-for-interactive-and-batch-execution). For this feature the [vre-middleware package](https://vre-middleware.readthedocs.io) must be installed. The `WFEngine` object used in the run is dumped into a file in the same folder as the launchpad file. This object can be used independently with the Python API or the GUI of the [vre-middleware package](https://vre-middleware.readthedocs.io) as described in more detail [in this section](#evaluation-of-nodes-of-batch-category). The flag `-q, --qadapter-file` is explained in detail [in this section](#evaluation-of-nodes-of-batch-category). By using the `-h, --help` flag a summarized help can be obtained: ``` texts session --help usage: texts session [-h] [-g GRAMMAR_PATH] [-l LAUNCHPAD_FILE] [-u UUID] [-r] [-a] [-d] [-s SLEEP_TIME] [-q QADAPTER_FILE] [-f MODEL_FILE] [-w] [--no-unique-launchdir] [--enable-logging] [--logging-level {NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}] [--no-duplicate-detection] Start an interactive session options: -h, --help show this help message and exit -g GRAMMAR_PATH, --grammar-path GRAMMAR_PATH path to grammar root file -l LAUNCHPAD_FILE, --launchpad-file LAUNCHPAD_FILE path to launchpad file -u UUID, --uuid UUID UUID of a model -r, --autorun run the model -a, --async-run run in background -d, --on-demand run on demand -s SLEEP_TIME, --sleep-time SLEEP_TIME sleep time for background evaluation in seconds -q QADAPTER_FILE, --qadapter-file QADAPTER_FILE path to default qadapter file -f MODEL_FILE, --model-file MODEL_FILE path to model file -w, --ignore-warnings do not display warnings --no-unique-launchdir disable unique launchdir --enable-logging enable logging messages --logging-level {NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL} logging level --no-duplicate-detection disable duplicate detection ``` #### Session behavior In the session, inputs are interactively typed and the outputs are shown immediately. An empty model instance is created at startup and with every input the model instance is extended with the model source extension typed as input. The editing capabilities of the input fields are very limited. #### Specific features In the interactive session, if a valid parameter is typed it is interpreted as a `print` statement. For example, a typed input `a` is changed to `print(a)` before parsing. Multiline comments are not supported in the interactive session. A successfully parsed and interpreted input cannot be modified. It becomes permanently part of the model source. The session manager accepts additional inputs that are no model sources in SCL or AMML. These are sometimes called *magic commands* (like in iPython and Jupyter) and begin with the symbol `%`. magic commands | action -----------------------------------|-------- `%exit`, `%bye`, `%close`, `%quit` | close (quit) the session `%start` | activate evaluation `%stop` | deactivate evaluation `%sleep ` | set sleep time (sec) for background evaluation `%uuid` | print the UUID of the active model `%uuid ` | switch to model with UUID `%new` | create a new session and a new model `%hist`, `%history` | print the current model source `%rerun var1[, var2][, ...]` | re-evaluate var1, var2, etc. `%vary` | print the [varied parameters](bulk.md#bulk-processing) `%tag` | print the [tag section](query.md#the-tag-statement) `%find [action]` | perform a global [search](query.md#the-find-command) `%help` | show this help The `%history` command prints all statements (without `print` statements that are not persisted), one per line, with timestamp of most recent evaluation update and current evaluation state. The evaluation state is currently passed through from the FireWorks workflow management system and is described in detail [here](https://materialsproject.github.io/fireworks/reference.html#fireworks-states). The `%rerun var` command can be used to re-evaluate the parameter of a variable `var` and its descendants, i.e. all parameters containing references to `var`. This is useful when there was some error during evaluation of `var` that was not due to the provided input to `var` but due to side effects, for example there was a computing node or file system failure, network or power outage etc. ### Duplicate detection Both *script* and *session* modes have a duplicate detection feature to identify variables in other persistent models that have identical names, parameters and values. This feature ensures safe reuse of such variables and helps to avoid unnecessary repeated evaluations. The duplicate detection can be used in any model but particularly useful in the cases of [sub-model reuse](submodel.md) and of [bulk processing](bulk.md). Duplicate detection is activated in *workflow* evaluation mode by default but it can be deactivated with the `--no-duplicate-detection` flag. ## Using the Jupyter kernel The Jupyter kernel is still work in progress. Currently, some help can be obtained in [this documentation](jupyter.md). ## Interpreting the error messages Either the parser or the interpreter may detect *static errors* in the program. Then they issue specific error messages in the following format: ``` Error type: /absolute/path/to/model_source_file.vm:line:column --> context <-- Detailed error description ``` The `--> context <--` displays the section of the model source where the error has occurred. If the Jupyter kernel or `texts session` is used then `None` is printed instead of an absolute path to file with the model source. **NOTE:** If an error message is not in this particular format, especially if it is a Python exception with a stack trace beginning with the word "Traceback", then it is most likely due to a bug in the interpreter and not due to an error in the model source. In such cases please [submit an issue](https://gitlab.kit.edu/kit/virtmat-tools/vre-language/-/issues) with the tag *Bug* and provide a minimal input to reproduce the error. ### Simple examples of errors Input: ``` b = 1 a = 2 ``` Error: ``` Syntax error: None:1:7 --> b = 1 *a = 2 <-- Expected '[' or '^|;|\,|\)|$' or '**' or '*' or '/' or '+' or '-' or '<=' or '>=' or '>' or '<' or '!=' or '==' or 'if' or 'for' or 'on' or '^|;' or EOF ``` All syntax errors are issued by the parser. Input: ``` b = a ``` Error: ``` Unknown object: None:1:5 Unknown object "a" of class "GeneralVariable" ``` Unknown object error is a semantic error issued by the parser when it cannot resolve a reference. Input: ``` s = 1 s = 'hello' ``` Error: ``` Initialization error: None:2:1 --> s = 'hello' <-- Repeated initialization of "s" ``` Repeated initialization error is issued by the interpreter when it applies semantic constraints before the beginning of evaluation. ### Runtime errors Some errors occur only during the evaluation. These are called *runtime errors*. Runtime errors have the same format as the static errors. Runtime errors only occur if the evaluation is requested, e.g. by `print` statements. For example, this input: ``` length = 1 [m] b = length + 1 ``` will produce no error message. Adding `print(b)` to the model and running the model in instant or deferred evaluation mode will yield this error: ``` Dimensionality error: None:2:5 --> length + 1 <-- Cannot convert from 'meter' ([length]) to 'dimensionless' (dimensionless) ``` In workflow evaluation mode this error occurs only if the evaluation is enabled and will be issued only if there is `print(b)` statement. ## Evaluation of nodes of batch category Once a model has been created in workflow mode, thus added to the database, it can be evaluated using either `texts script` or `texts session` with the `--autorun, -r` flag. In this mode, the workflow nodes of _interactive_ category are evaluated synchronously while the nodes of _batch_ category are not. More information about the workflow node categories can be found [here](resources.md#policies-for-interactive-and-batch-execution). To enable asynchronous background evaluation of nodes of batch category one has to add the `--async-run, -a` flag. As soon as the `texts session` is exited the background evaluation is terminated. This means that no further _batch_ nodes will be scheduled for execution. The already scheduled or running _batch_ nodes will be further processed without interruption or any user actions. The evaluation of a model can be continued either by loading the model in a new session with `texts session -u UUID -r -a` or by running the `wfengine` CLI tool from the [VRE Middware package](https://vre-middleware.readthedocs.io/en/latest/resconfig.html) independently. ```bash wfengine --load-from-file wfengine-.yaml -r -c all ``` The `wfengine-.yaml` file is always created by `texts session` in the same folder as the launchpad file used in the session when the `--async-run, -a` flag is used. The `wfengine-.yaml` file contains a dumped `WEngine` object with a complete configuration, including launchpad, model UUID, default qadapter (see below), default worker, etc. Further options for running the `wfengine` tool can be shown with the `--help` flag. The evaluation of nodes of batch category happens in an HPC environment governed by a batch system such as SLURM and requires an object called *qadapter*. ### Default qadapter A default qadapter will be automatically created by the workflow engine and used by the launcher thread. To this end, the default resources in the resource configuration (*resconfig*) object are used. These resources are: default worker, default queue, default launch directory, default group used for accounting, default queue with default computing resources (such as walltime, number of CPUs, amount of memory etc.), default loaded environment modules, default virtual environment. The resconfig object is stored in a resconfig file with default location `$HOME/.fireworks/res_config.yaml` that can be overridden by the environment variable `RESCONFIG_LOC`. If the resconfig file does not exist at the beginning of `texts session`, then it is generated automatically (a prompt will be displayed to accept this step). Further more technical information about res_config can be found [elsewhere](https://vre-middleware.readthedocs.io/en/latest/resconfig.html). If necessary, the default qadapter can be overridden by using the optional `--qadapter-file, -q` flag, which takes the name of a qadapter file (e.g. "qadapter.yaml"), containing a qadapter object, as argument. This can be necessary if the default qadapter does not specify some resource needed by the batch job to run. However, the recommended way to fix this is to modify the res_config object instead of providing a default qadapter. If the user has to write a qadapter file then [this tutorial](https://materialsproject.github.io/fireworks/qadapter_programming.html) can be used. ### Custom qadapter Different batch nodes may have different resource requirements and the (either auto-generated or manually provided) default qadapter may not fit to all of them. For this reason, a *custom qadapter* is created automatically for nodes of batch category. The resources specified in the custom qadapter override these in the default qadapter during node launch. The custom qadapter is created automatically for models with [resource annotations](resources.md#resource-annotation) and added to the node pertinent to the statement with the resource annotation in the model. All resources specified in the resource annotation are included in the custom qadapter. In addition, the custom qadapter includes the default virtual environment and all default modules so that these will be activated and loaded in the batch job before the evaluation step starts. These default settings are sourced from the res_config. In addition, the application-specific environment modules and variables will be set. These are not sourced from the res_config but from the calculator or the algorithm that requires these resources.