easyvvuq.actions package

Submodules

easyvvuq.actions.action_statuses module

Implements ActionPool - a thin wrapper around the Python Executor interface that is meant to simplify the execution of actions and retrieval of results. This object is instantiated by the Campaign. The user would never instantiate it manually. The user does interact with it to track the progress of execution.

class easyvvuq.actions.action_statuses.ActionPool(campaign, actions, inits, sequential=False)

Bases: object

A class that handles the execution of Actions.

Parameters:

• campaign (Campaign) – An instance of an EasyVVUQ campaign.

• actions (Actions) – An instance of Actions containing things to be done as part of the simulation.

• inits (iterable) – Initial inputs to be passed to each Actions representing a sample. Will usually contain dictionaries with the following information: {‘run_id’: …, ‘campaign_dir’: …, ‘run_info’: …}.

• sequential (bool) – Will run the actions sequentially.

add_collate_callback(fn)

Adds a callback to be called after collation is done.

Parameters:

fn - A callable that takes previous as it’s only input.

collate(progress_bar=False)

A command that will block until all Futures in the pool have finished. It will also store the results gather from Actions in the database.

Parameters:

progress_bar (bool) – Whether to show progress bar

progress()

Some basic stats about the action statuses status.

Returns:

A dictionary with four keys - ‘ready’, ‘active’ and ‘finished’, ‘failed’. Values under “ready” correspond to Actions waiting for execution, “active” corresponds to the number of currently running tasks.

Return type:

dict

start(pool=None)

Start the actions.

Parameters:

pool (An Executor instance (e.g. ThreadPoolExecutor))

Returns:

Starts execution and returns a reference to itself for tracking progress and for collation.

Return type:

ActionPool

easyvvuq.actions.execute_kubernetes module

Provides an action element to execute a simulation on a Kubernetes cluster and retrieve the output. The successful use of this actions requires that the Kubernetes cluster is properly set-up on the users system. Namely the ~/.kube/config file should contain valid information. Exact details will depend on the cloud service provider. Otherwise this action works similarly to how ExecuteLocal works. The difference is that the simulations are executed on a Kubernetes cluster. The input files are passed to the Pods via the ConfigMap mechanism. This probably limits the size of the configuration files but this can be alleviated with some kind of a pre-processing script on the Pod side. Likewise, output from the simulation is retrieved using the Kubernetes log mechanism. Therefore the simulation output needs to be printed to stdout on the Pod side. Again, if the simulation produces complicated or large output you should extract the quantitities of interest on the Pod using some kind of script and print them to stdout.

Examples

class easyvvuq.actions.execute_kubernetes.ExecuteKubernetes(image, command, input_file_names=None, output_file_name=None)

Bases: object

Parameters:

• image (str) – Name of the repository e.g. orbitfold/easyvvuq:tagname.

• command (str) – A command to run the simulation from within the container.

• input_file_names (list) – A list of input files.

• output_file_names (list) – A list of output files.

create_config_maps(file_names)

Create Kubernetes ConfigMaps for the input files to the simulation.

Parameters:

file_names (list) – Will go through every filename in this list and create a Kubernetes ConfigMap with it’s contents.

create_volumes(file_names, dep)

Create descriptions of Volumes that will hold the input files.

Parameters:

filenames (list) – A list of file names to be mounted under /config/ in the running image.

finalise()

Will read the logs from the Kubernetes pod, output them to a file and delete the Kubernetes resources we have allocated.

finished()

Will return True if the pod has finished, otherwise will return False.

start(previous=None)

Will create the Kubernetes pod and hence start the action.

Parameters:

previous (dict) – Data from previous Action.

Returns:

Data from previous Action appended with data from this Action.

Return type:

dict

succeeded()

Will return True if the pod has finished successfully, otherwise will return False. If the job hasn’t finished yet will return False.

easyvvuq.actions.execute_local module

This module provides an assortment of actions generally concerned with executing simulations locally. Some Actions will also be useful when using Dask.

class easyvvuq.actions.execute_local.Actions(*args)

Bases: object

finalise()

finished()

set_wrapper(wrapper)

Adds a wrapper to be called on each Action.

Parameters:

wrapper (callable) – A function to call on each Action. Should pass through the return of the start method.

start(previous=None)

succeeded()

class easyvvuq.actions.execute_local.CleanUp

Bases: object

finalise()

finished()

start(previous=None)

succeeded()

class easyvvuq.actions.execute_local.CreateRunDirectory(root, flatten=False)

Bases: object

Creates a directory structure for storing simulation input and output files.

Parameters:

• root (str) – Root directory to create a directory structure in.

• flatten (bool) – If set to True will result in a flat directory structure (each run gets a directory under root). If left as False will create a hierarchical structure. This is useful so as not to overload the filesystem.

start(previous=None)

Starts the action.

Will read a run_id from a dictionary supplied by the previous Action. Will then create a directory structure based on the numerical value of the run_id.

Returns:

A dictionary to be passed to the following Action.

Return type:

dict

succeeded()

Has the Action finished successfully.

Returns:

True if Action completed successfully. False otherwise.

Return type:

bool

class easyvvuq.actions.execute_local.Decode(decoder)

Bases: object

finalise()

finished()

start(previous=None)

succeeded()

class easyvvuq.actions.execute_local.Encode(encoder)

Bases: object

finalise()

finished()

start(previous=None)

succeeeded()

class easyvvuq.actions.execute_local.ExecuteLocal(full_cmd, stdout=None, stderr=None)

Bases: object

finalise()

Performs clean-up if necessary. In this case it isn’t. I think.

finished()

start(previous=None)

succeeded()

Will return True if the process finished successfully. It judges based on the return code and will return False if that code is not zero.

class easyvvuq.actions.execute_local.ExecutePython(function)

Bases: object

finalise()

finished()

start(previous=None)

succeeded()

easyvvuq.actions.execute_local.local_execute(encoder, command, decoder, root='/tmp')

A helper function for a simple local execution. It will create a directory under your specified root folder, encode the sampler output, execute a command and decode the results of the simulation.

Parameters:

• encoder (Encoder) – an encoder to use

• command (list of str) – a command to run your simulation (same as argument to popen, e.g. [‘ls’, ‘-al’])

• decoder (Decoder) – a decoder to use

• root (str) – root folder, for example ‘/tmp’ or if you want to use ram based filesystem it could be ‘/dev/shm’

Return type:

Actions

easyvvuq.actions.execute_qcgpj module

class easyvvuq.actions.execute_qcgpj.EasyVVUQBasicTemplate(*args: Any, **kwargs: Any)

Bases: QCGPJTemplate

A basic template class for submission of QCG-PilotJob tasks that run on a single core

The class can be used only for the most simple use-cases. For example it doesn’t allow to specify resource requirements. Thus, for more advanced use-cases, it is recommended to provide custom implementation of QCGPJTemplate. For complete reference of QCG-PilotJob task’s description parameters please look at https://qcg-pilotjob.readthedocs.io/en/latest/fileinterface.html#submit

static template() → Tuple[str, Dict[str, Any]]

class easyvvuq.actions.execute_qcgpj.EasyVVUQParallelTemplate(*args: Any, **kwargs: Any)

Bases: QCGPJTemplate

A template class for submission of QCG-PilotJob tasks that run on exact number cores / nodes

With this class it is possible to define basic resource requirements for tasks. For advanced use-cases, it is recommended to provide custom implementation of QCGPJTemplate. For complete reference of QCG-PilotJob task’s description parameters please look at https://qcg-pilotjob.readthedocs.io/en/latest/fileinterface.html#submit

static template() → Tuple[str, Dict[str, Any]]

class easyvvuq.actions.execute_qcgpj.ExecuteQCGPJ(action)

Bases: object

A utility decorator over action that marks the action as configured for parallel execution by QCG-PilotJob Currently it has no influence on the processing.

Parameters:

action (Action) – an action that will be decorated in order to enable parallel execution inside a QCG-PilotJob task.

finalise()

finished()

start(previous=None)

succeeded()

class easyvvuq.actions.execute_qcgpj.QCGPJPool(qcgpj_executor=None, template=None, template_params=None, polling_interval=1)

Bases: Executor

A Pool that manages execution of actions with QCG-PilotJob.

Parameters:

• qcgpj_executor (str) – An instance of QCGPJExecutor. If not provided, an instance of QCGPJExecutor with default settings will be created

• template (QCGPJTemplate) – An object which contains only a single method template that returns a tuple. The first element of a tuple should be a string representing a QCG-PilotJob task’s description with placeholders (identifiers preceded by $ symbol) and the second a dictionary that assigns default values for selected placeholders. If not provided, a default EasyVVUQBasicTemplate will be used

• template_params (dict) – A dictionary that contains parameters that will be used to substitute placeholders defined in the template

• polling_interval (int) – An interval between queries to the QCG-PilotJob Manager service about state of the tasks, in seconds.

as_completed(futures)

Checks for the status of features and yields those that are finished

convert_results(result_qcgpj)

Converts results generated by QCG-PilotJob task to EasyVVUQ-compatible form

The method loads results data from a file where it was stored by QCG-PilotJob’s task and then converts it to a dictionary which can be further processed by EasyVVUQ.

Parameters:

result_qcgpj (list or None) – A list of results returned by a QCG-PilotJob task (only the first element will be used), or None if the task hasn’t finished with the status SUCCEED

Returns:

A dictionary containing results

Return type:

dict

property executor

Returns current QCGPJExecutor instance.

It gives you an access to QCG-PilotJob Manager instance, which in turn can be used to get information about the QCG-PilotJob execution environment.

shutdown(**kwargs)

Clean-up the resources associated with the QCGPJPool.

submit(fn, *args, **kwargs)

Submits a callable to be executed by QCG-PilotJob.

Schedules the callable to be executed inside a QCG-PilotJob’s task and returns a Future representing the execution of the callable.

Returns:

QCGPJFuture representing the given call.

Return type:

QCGPJFuture

easyvvuq.actions.execute_qcgpj_task module

easyvvuq.actions.execute_slurm module

Provides a simple action element for interacting with a SLURM job. This lets you execute your simulation on a SLURM cluster.

class easyvvuq.actions.execute_slurm.ExecuteSLURM(template_script, variable)

Bases: object

An Action to launch and track the execution of a SLURM job.

Parameters:

• template_script (str) – Filename of a file containing the script template.

• variable (str) – A string to be replaced with the directory in which the job is meant to be executed. This is to be used to make sure that the simulation can find the correct input files and knows where to put output files.

finalise()

Performs clean-up if necessary. In this case it isn’t. I think.

start(previous=None)

Start the SLURM job.

Parameters:

previous (dict) – A dictionary containing information provided by previously executed actions.

Module contents

This module contains implementations of various Actions. Actions in EasyVVUQ are responsible for anything that is related to the execution of the simulation. That includes: actually executing the simulation, preparing the input files, parsing the output files, creating directory structures necessary to execute the simulation, cleaning up after, delegating work to external execution back-ends such as Dask, etc.