ecFlow -- Installation & Usage
By micromamba (Recommend)
1 | |
and it installs,
1 | |
where boost is 1.84.0
ecFlow Quickstart
1 | |
Inspect and update the suite definition
The file test.def defines a suite called "test". Open this file in a text editor and note the structure.
1 | |
We define a suite called "test"; it defines a variable called ECF_HOME, then it defines two tasks ("tasks" are things that actually run) and specifies that task t2 will be run as soon as task t1 has completed. The scripts corresponding to the tasks are specified in ".ecf" files in the test directory. Have a look - they simply print some information about themselves, then sleep for 2 seconds.
The first thing you must do is to complete the path to your working directory in the ECF_HOME definition in test.def and save the file.
Start an ecFlow server
We will start a new running instance of an ecFlow server using the default port (# start ecFlow with default port 3141). It is possible to use a different port by adding --port=3500 (for example) to every ecFlow command-line action. Note also that we start it as a background task - it will run until the server is stopped. It can be run in the foreground, but in that case you will need to use a new terminal for any subsequent commands!
To start the ecflow server:
1 | |
make sure this port can be used {:.info}
check what port numbers are being used
netstat -lnptu
You can check what port numbers are being used, with netstat: To list all open network ports on your machine, run netstat -lnptu. Here is a breakdown of the parameters:
l - List all listening ports n - Display the numeric IP addresses (i.e., don't do reverse DNS lookups p - List the process name that is attached to that port t - List all TCP connections u - List all UDP connections
1 | |
Check that it is running:
1 | |
or check
1 | |
Please note “Host” and “Port Number” here. Also note that each user must use a unique port number (we recommend using a random number between 2500 and 9999)
To stop the ecflow server:
1 | |
e.g.
1 | |
and then,
1 | |
has shown nothing now.
Load your suite definition into the server **
1 | |
More,
1 | |
Check that it is loaded by asking the server to give you back the suite definition:
1 | |
Monitor and interact via the GUI
Start ecFlowUI: (Local PC)
When opening the ecflow GUI flow for the first time you will need to add your server to the GUI. In the GUI click on “Servers” and then “Manage servers”. A new window will appear. Click on “Add server”. Here you need to add the Name, Host, and Port of your server. For “Host” and “Port” please refer to the last section of output from the previous step.
1 | |
ssh tunnel
SSH tunneling, or SSH port forwarding, is a method of transporting arbitrary data over an encrypted SSH connection. SSH tunnels allow connections made to a local port (that is, to a port on your own desktop) to be forwarded to a remote machine via a secure channel.
on local PC terminal,
1 | |
and you can set ecflow_ui config, - host=localhost - port=local PC's port 9090
ecFlowUI has started
Once ecFlowUI has started, you must tell it how to reach your server. Go to the
Servers → Manage Servers menu, click "Add server", then enter the details of your server. Name can be anything you want - it's for you to identify the server to your self; something like "localtest" would be fine here. Host should in this case be "localhost", and Port should be XXX. The other fields can be left blank, but keep the "Add server to current view" box ticked.
You should now see your suite loaded into the GUI! To make the server active, right-click on the top-level node representing the server ("localtest in our case) and choose "Restart". Now right-click on the "test" node and choose "Begin" to make the suite active. The default behaviour is to only refresh its view of the suite every 60 seconds, so you will need to click the green refresh button at the top every so often to see the progress of the tasks.
The diagram shows the typical status changes for a Task.
Start
At this time, the service is stopped and needs to be started manually.
1 | |
Since no triggering mechanism is added, t1 will be executed immediately after the service starts.
Replace suite
Replace the entire suite,
1 | |
check the status of the server
To check the status of the server, enter the following unix command
1 | |
ecFlow variables
ECF_HOME ECF_INCLUDE ECF_FILES
Execute, rerun, requeue
In ecflow_ui, it is important to understand the distinction between execute, rerun and re-queue.
These options are available with the right mouse button over a task.
Execute: This means run the task immediately. (Hence ignores any dependency that holds the task).
This option preserves the previous output, from the job. You should see output files t1.1, t1.2, t1.3, each time the task is run.
Rerun: This places the task, back into the queued state.
The task will now honour any dependencies that would hold the job. i.e. time dependencies, limits, triggers, suspend, etc. (You will be introduced to these terms later on in the tutorial) If the task does run, the previous output is preserved.
Re-queue: This resets the task back to the queued state.
If the task has a default status, this is applied. The task output number is reset, such that the next output will be written to t1.1. This will overwrite any existing output with that extension when the task runs. Any subsequent calls to execute or rerun will now overwrite the output files, t1.2,t1.3 etc.
Add description
The manual page makes the documentation in the ecf script visible in ecflow_ui.
The description page is the combination of all the text before the directives %manual and %end. (or create f1.man)
e.g. edit t2.ecf
1 | |
familys
Tasks can be organized into a tree structure similar to a Unix file system, where task is like a file and family is like a folder.
A suite is a family with additional properties (see Dates). family can contain other families. Similar to directories, tasks in different families can have the same name.
Unless a file location is specified, ecFlow assumes that the structure of the suite corresponds to the location of the task file in the file system.
1 | |
Time Dependencies
Time Triggers
Add a Cron
Repeat *** Repeat
It is sometimes useful to repeat the same task or family several times, looping on a specific value. You can do that by defining a repeat attribute. You can iterate over sequences of:
- strings
- integers
- dates
A sequence of integers or dates is created by specifying the first and the last element, with an optional increment (the default is one).
From ecflow5 we can also loop over an arbitrary list of dates.
1 | |
The following generated variables, are accessible for trigger expressions YMD, YMD_YYYY, YMD_MM, YMD_DD, YMD_DOW,YMD_JULIAN
e.g.
1 | |
When we use relative time attributes under a Repeat. They are automatically reset when the repeat loops. Take for example:
1 | |
Repeat increment
A repeat under a family node will only increment when all the child nodes are complete.
In the example above, once task t1, and t2 are complete, the repeat integer rep will increment to the next value.
with SBATCH
sbatch.h
1 | |