easyvvuq.actions package¶
Submodules¶
easyvvuq.actions.action_statuses module¶
Implements ActionPool - a thin wrapper around the Python Executor interface that is meant to simplify the execution of actions and retrieval of results. This object is instantiated by the Campaign. The user would never instantiate it manually. The user does interact with it to track the progress of execution.
- class easyvvuq.actions.action_statuses.ActionPool(campaign, actions, inits, sequential=False)[source]¶
Bases:
object
A class that handles the execution of Actions.
- Parameters:
campaign (Campaign) – An instance of an EasyVVUQ campaign.
actions (Actions) – An instance of Actions containing things to be done as part of the simulation.
inits (iterable) – Initial inputs to be passed to each Actions representing a sample. Will usually contain dictionaries with the following information: {‘run_id’: …, ‘campaign_dir’: …, ‘run_info’: …}.
sequential (bool) – Will run the actions sequentially.
- add_collate_callback(fn)[source]¶
Adds a callback to be called after collation is done.
- Parameters:
fn - A callable that takes previous as it’s only input.
- collate(progress_bar=False)[source]¶
A command that will block until all Futures in the pool have finished. It will also store the results gather from Actions in the database.
- Parameters:
progress_bar (bool) – Whether to show progress bar
- progress()[source]¶
Some basic stats about the action statuses status.
- Returns:
A dictionary with four keys - ‘ready’, ‘active’ and ‘finished’, ‘failed’. Values under “ready” correspond to Actions waiting for execution, “active” corresponds to the number of currently running tasks.
- Return type:
dict
easyvvuq.actions.execute_kubernetes module¶
Provides an action element to execute a simulation on a Kubernetes cluster and retrieve the output. The successful use of this actions requires that the Kubernetes cluster is properly set-up on the users system. Namely the ~/.kube/config file should contain valid information. Exact details will depend on the cloud service provider. Otherwise this action works similarly to how ExecuteLocal works. The difference is that the simulations are executed on a Kubernetes cluster. The input files are passed to the Pods via the ConfigMap mechanism. This probably limits the size of the configuration files but this can be alleviated with some kind of a pre-processing script on the Pod side. Likewise, output from the simulation is retrieved using the Kubernetes log mechanism. Therefore the simulation output needs to be printed to stdout on the Pod side. Again, if the simulation produces complicated or large output you should extract the quantitities of interest on the Pod using some kind of script and print them to stdout.
Examples
- class easyvvuq.actions.execute_kubernetes.ExecuteKubernetes(image, command, input_file_names=None, output_file_name=None)[source]¶
Bases:
object
- Parameters:
image (str) – Name of the repository e.g. orbitfold/easyvvuq:tagname.
command (str) – A command to run the simulation from within the container.
input_file_names (list) – A list of input files.
output_file_names (list) – A list of output files.
- create_config_maps(file_names)[source]¶
Create Kubernetes ConfigMaps for the input files to the simulation.
- Parameters:
file_names (list) – Will go through every filename in this list and create a Kubernetes ConfigMap with it’s contents.
- create_volumes(file_names, dep)[source]¶
Create descriptions of Volumes that will hold the input files.
- Parameters:
filenames (list) – A list of file names to be mounted under /config/ in the running image.
- finalise()[source]¶
Will read the logs from the Kubernetes pod, output them to a file and delete the Kubernetes resources we have allocated.
easyvvuq.actions.execute_local module¶
This module provides an assortment of actions generally concerned with executing simulations locally. Some Actions will also be useful when using Dask.
- class easyvvuq.actions.execute_local.Actions(*args)[source]¶
Bases:
object
- class easyvvuq.actions.execute_local.CreateRunDirectory(root, flatten=False)[source]¶
Bases:
object
Creates a directory structure for storing simulation input and output files.
- Parameters:
root (str) – Root directory to create a directory structure in.
flatten (bool) – If set to True will result in a flat directory structure (each run gets a directory under root). If left as False will create a hierarchical structure. This is useful so as not to overload the filesystem.
- class easyvvuq.actions.execute_local.ExecuteLocal(full_cmd, stdout=None, stderr=None)[source]¶
Bases:
object
- easyvvuq.actions.execute_local.local_execute(encoder, command, decoder, root='/tmp')[source]¶
A helper function for a simple local execution. It will create a directory under your specified root folder, encode the sampler output, execute a command and decode the results of the simulation.
- Parameters:
encoder (Encoder) – an encoder to use
command (list of str) – a command to run your simulation (same as argument to popen, e.g. [‘ls’, ‘-al’])
decoder (Decoder) – a decoder to use
root (str) – root folder, for example ‘/tmp’ or if you want to use ram based filesystem it could be ‘/dev/shm’
- Return type:
easyvvuq.actions.execute_qcgpj module¶
- class easyvvuq.actions.execute_qcgpj.EasyVVUQBasicTemplate(*args: Any, **kwargs: Any)[source]¶
Bases:
QCGPJTemplate
A basic template class for submission of QCG-PilotJob tasks that run on a single core
The class can be used only for the most simple use-cases. For example it doesn’t allow to specify resource requirements. Thus, for more advanced use-cases, it is recommended to provide custom implementation of QCGPJTemplate. For complete reference of QCG-PilotJob task’s description parameters please look at https://qcg-pilotjob.readthedocs.io/en/latest/fileinterface.html#submit
- class easyvvuq.actions.execute_qcgpj.EasyVVUQParallelTemplate(*args: Any, **kwargs: Any)[source]¶
Bases:
QCGPJTemplate
A template class for submission of QCG-PilotJob tasks that run on exact number cores / nodes
With this class it is possible to define basic resource requirements for tasks. For advanced use-cases, it is recommended to provide custom implementation of QCGPJTemplate. For complete reference of QCG-PilotJob task’s description parameters please look at https://qcg-pilotjob.readthedocs.io/en/latest/fileinterface.html#submit
- class easyvvuq.actions.execute_qcgpj.ExecuteQCGPJ(action)[source]¶
Bases:
object
A utility decorator over action that marks the action as configured for parallel execution by QCG-PilotJob Currently it has no influence on the processing.
- Parameters:
action (Action) – an action that will be decorated in order to enable parallel execution inside a QCG-PilotJob task.
- class easyvvuq.actions.execute_qcgpj.QCGPJPool(qcgpj_executor=None, template=None, template_params=None, polling_interval=1)[source]¶
Bases:
Executor
A Pool that manages execution of actions with QCG-PilotJob.
- Parameters:
qcgpj_executor (str) – An instance of QCGPJExecutor. If not provided, an instance of QCGPJExecutor with default settings will be created
template (QCGPJTemplate) – An object which contains only a single method template that returns a tuple. The first element of a tuple should be a string representing a QCG-PilotJob task’s description with placeholders (identifiers preceded by $ symbol) and the second a dictionary that assigns default values for selected placeholders. If not provided, a default EasyVVUQBasicTemplate will be used
template_params (dict) – A dictionary that contains parameters that will be used to substitute placeholders defined in the template
polling_interval (int) – An interval between queries to the QCG-PilotJob Manager service about state of the tasks, in seconds.
- convert_results(result_qcgpj)[source]¶
Converts results generated by QCG-PilotJob task to EasyVVUQ-compatible form
The method loads results data from a file where it was stored by QCG-PilotJob’s task and then converts it to a dictionary which can be further processed by EasyVVUQ.
- Parameters:
result_qcgpj (list or None) – A list of results returned by a QCG-PilotJob task (only the first element will be used), or None if the task hasn’t finished with the status SUCCEED
- Returns:
A dictionary containing results
- Return type:
dict
- property executor¶
Returns current QCGPJExecutor instance.
It gives you an access to QCG-PilotJob Manager instance, which in turn can be used to get information about the QCG-PilotJob execution environment.
easyvvuq.actions.execute_qcgpj_task module¶
easyvvuq.actions.execute_slurm module¶
Provides a simple action element for interacting with a SLURM job. This lets you execute your simulation on a SLURM cluster.
- class easyvvuq.actions.execute_slurm.ExecuteSLURM(template_script, variable)[source]¶
Bases:
object
An Action to launch and track the execution of a SLURM job.
- Parameters:
template_script (str) – Filename of a file containing the script template.
variable (str) – A string to be replaced with the directory in which the job is meant to be executed. This is to be used to make sure that the simulation can find the correct input files and knows where to put output files.
Module contents¶
This module contains implementations of various Actions. Actions in EasyVVUQ are responsible for anything that is related to the execution of the simulation. That includes: actually executing the simulation, preparing the input files, parsing the output files, creating directory structures necessary to execute the simulation, cleaning up after, delegating work to external execution back-ends such as Dask, etc.