easyvvuq package

Subpackages

Submodules

easyvvuq.base_element module

class easyvvuq.base_element.BaseElement[source]

Bases: object

Baseclass for all EasyVVUQ elements.

element_category()[source]
element_name()[source]
element_version()[source]

easyvvuq.campaign module

EasyVVUQ Campaign

This module contains the Campaign class that is used to coordinate all EasyVVUQ workflows.

class easyvvuq.campaign.Campaign(name, params=None, actions=None, db_location=None, work_dir='./', change_to_state=False, verify_all_runs=True)[source]

Bases: object

Campaigns organise the dataflow in EasyVVUQ workflows.

The Campaign functions as as state machine for the VVUQ workflows. It uses a database (CampaignDB) to store information on both the target application and the VVUQ algorithms being employed. It also collects data from the simulations and can be used to store and resume your state.

Notes

Multiple campaigns can be combined in a CampaignDB. Hence the particular campaign we are currently working on will be specified using campaign_id.

Parameters:
  • name (str) – Name of the Campaign. Freely chosen, serves as a human-readable way of distinguishing between several campaigns in the same database.
  • params (dict, optional) – Description of the parameters to associated with the application. Will be used to create an app when creating the campaign. It is also possible to add apps manually using add_app method of the Campaign class. But this can be a useful shorthand when working with single app campaigns. To use this functionality both params and actions has to be specified. The name of this app will be the same as the name of the Campaign.
  • actions (Actions, optional) – Actions object associated with an application. See description of the params parameter for more details.
  • db_location (str, optional) – Location of the underlying campaign database - either a path or acceptable URI for SQLAlchemy.
  • work_dir (str, optional, default=’./’) – Path to working directory - used to store campaign directory.
  • change_to_state (bool, optional, default=False) – Should we change to the directory containing any specified state_file in order to make relative paths work.
  • verify_all_runs (bool, optional, default=True) – Check all new runs being added for unrecognised params (not defined for the currently set app), values lying within defined physical range, type checking etc. This should normally always be set to True, but in cases where the performance is too degraded, the checks can be disabled by setting to False.
Variables:
  • campaign_name (str or None) – Name for the campaign/workflow.
  • _campaign_dir (str or None) – Path to the directory campaign uses for local storage (runs inputs etc)
  • db_location (str or None) – Location of the underlying campaign database - either a path or acceptable URI for SQLAlchemy.
  • _log (list) – The log of all elements that have been applied, with information about their application
  • campaign_id (int) – ID number for the current campaign in the db.CampaignDB.
  • campaign_db (easyvvuq.db.Basedb.CampaignDB) – A campaign database object
  • last_analysis – The result of the most recent analysis carried out on this campaign
  • _active_app (dict) – Info about currently set app
  • _active_app_name (str) – Name of currently set app
  • _active_sampler_id (int) – The database id of the currently set Sampler object

Examples

A typical instantiation might look like this.

>>> params = {
        "S0": {"type": "float", "default": 997},
        "I0": {"type": "float", "default": 3},
        "beta": {"type": "float", "default": 0.2},
        "gamma": {"type": "float", "default": 0.04, "min": 0.0, "max": 1.0},
        "iterations": {"type": "integer", "default": 100},
        "outfile": {"type": "string", "default": "output.csv"}
    }
>>> encoder = uq.encoders.GenericEncoder(template_fname='sir.template', delimiter='$', target_filename='input.json')
>>> decoder = uq.decoders.SimpleCSV(target_filename='output.csv', output_columns=['I'])
>>> actions = uq.actions.local_execute(encoder, os.path.abspath('sir') + ' input.json', decoder)
>>> campaign = uq.Campaign(name='sir', params=params, actions=actions)

A simplified one (without an app) might look simply like this.

>>> campaign = Campaign('simple')

An app then can be added.

>>> campaign.add_app('simple_app', params=params, actions=actions)
add_app(name=None, params=None, actions=None, set_active=True)[source]

Add an application to the CampaignDB.

Parameters:
  • name (str) – Name of the application.
  • params (dict) – Description of the parameters to associate with the application.
  • actions (Actions) – An instance of Actions containing actions to be executed
  • set_active (bool) – Should the added app be set to be the currently active app?
add_external_runs(input_files, output_files, input_decoder, output_decoder)[source]

Takes a list of files and adds them to the database. This method is to be used when adding runs to the EasyVVUQ database that were not executed using EasyVVUQ.

Parameters:
  • output_files (list of str) – A list of output file paths to be loaded to the database.
  • decoder (Decoder) – A decoder that will be used to parse these files.
add_runs(runs, mark_invalid=False)[source]

Add runs to the database.

Parameters:
  • runs (list of dicts) – Each dict defines the value of each model parameter listed in self.params_info for a run to be added to self.runs
  • mark_invalid (bool) – Will mark runs that fail verification as invalid (but will not raise an exception)
analyse(**kwargs)[source]

If available will call an appropriate analysis class on the collation result.

Parameters:**kwargs (dict) – Argument to the analysis class constructor (after sampler).
Returns:An object representing analysis results. Can be used to interact with those results in some way. Plot, retrieve surrogate models and so on. See easyvvuq.analysis.AnalysisResults for further information.
Return type:AnalysisResults
apply_analysis(analysis)[source]

Run the analysis element on the output of the last run collation.

Parameters:analysis (Analysis) – Element that performs a VVUQ analysis on a dataframe summary of run outputs.
apply_for_each_sample(actions, status=<Status.NEW: 1>, sequential=False)[source]

For each run in this Campaign’s run list, apply the specified action (an object of type Action).

Parameters:
  • actions (Actions) – Actions to be applied to each relevant run in the database.
  • status (Status) – Will apply the Actions only to those runs whose status is as specified.
  • sequential (bool) – Whether to process samples sequentially (sometimes more efficient or you might want to avoid the concurrent module for some reason).
Returns:

An object containing ActionStatus instances to track action execution.
Return type:

ActionPool
campaign_dir

Get the path in which to load/save files related to the campaign.

Returns:Path to the campaign directory - given as a subdirectory of the working directory.
Return type:str
draw_samples(num_samples=0, mark_invalid=False)[source]

Draws num_samples sets of parameters from the currently set sampler, resulting in num_samples new runs added to the runs list. If num_samples is 0 (its default value) then this method draws ALL samples from the sampler, until exhaustion (this will fail if the sampler is not finite).

Parameters:
  • num_samples (int) – Number of samples to draw from the active sampling element. By default is 0 (draw ALL samples)
  • mark_invalid (bool) – If True will mark runs that go outside valid parameter range as INVALID. This is useful for MCMC style methods where you want those runs to evaluate to low probabilities.
execute(nsamples=0, pool=None, mark_invalid=False, sequential=False)[source]

This will draw samples and execute the Actions on those samples.

Parameters:
  • nsamples (int) – Number of samples to draw. For infinite samplers or when you want to process samples in batches.
  • pool (Executor) – A pool object to be used when processing runs (e.g. instance of ThreadPoolExecutor or ProcessPoolExecutor).
  • mark_invalid (bool) – Mark runs that go outside the specified input parameter range as INVALID.
  • sequential (bool) – Whether to process samples sequentially (sometimes more efficient or you might want to avoid the concurrent module for some reason).
get_active_app()[source]

Returns a dict of information regarding the application that is currently set for this campaign.

get_active_sampler()[source]

Return the active sampler element in use by this campaign.

Returns:
Return type:The sampler currently in use
get_campaign_runs_dir()[source]

Get the runs directory from the CampaignDB.

Returns:Path in which the runs information will be written.
Return type:str
get_collation_result(last_iteration=False)[source]

Return dataframe containing all collated results

Parameters:last_iteration (bool) – Will only return the result of the last iteration.
Returns:A DataFrame with the simulation results along with the inputs used to produce them.
Return type:DataFrame
get_invalid_runs(last_iteration=False)[source]

Return dataframe containing all results marked as INVALID.

Parameters:last_iteration (bool) – Will only return the result of the last iteration.
Returns:A DataFrame with the results form simulations that were marked as INVALID. These will usually be the ones that went outside the specified parameter ranges. These still have to be accounted for in some way by some methods (e.g. MCMC).
Return type:DataFrame
get_last_analysis()[source]

Return the output of the most recently run analysis element.

ignore_runs(list_of_run_IDs)[source]

Flags the specified runs to be IGNORED in future collation. Note that this does NOT remove previously collated results from the collation table. For that you must refresh the collation by running recollate().

Parameters:list – The list of run IDs for the runs that should be set to status IGNORED
init_db(name, work_dir='.')[source]

Initialize the connection with the database and either resume or create the campaign.

Parameters:
  • name (str) – Name of the campaign.
  • work_dir (str) – Work directory, defaults to cwd.
iterate(nsamples=0, pool=None, mark_invalid=False, sequential=False)[source]

This is the equivalent of execute for methods that rely on the output of the previous sampling stage (designed for MCMC, should work for others).

Parameters:
  • nsamples (int) – Number of samples to draw (during a single iteration).
  • pool (Executor) – An Executor instance. For example ThreadPoolExecutor or a Dask Client. Defaults to the ThreadPoolExecutor.
  • mark_invalid (bool) – Mark runs that go outside the specified input parameter range as INVALID.
  • sequential (bool) – Will execute the Actions associated with runs sequentially. Might be more efficient in some situations.
Yields:

ActionPool – An object containing Futures instances to track action execution.
list_runs(sampler=None, campaign=None, app_id=None, status=None)[source]

Get list of runs in the CampaignDB.

Parameters:
  • sampler (int) – Sampler id to filter for.
  • campaign (int) – Campaign id to filter for.
  • app_id (int) – App id to filter for.
  • status (Status) – Status to filter for.
Returns:
Return type:

list of runs
recollate()[source]

Clears the current collation table, changes all COLLATED status runs back to ENCODED, then runs collate() again

relocate(campaign_dir)[source]

Relocate the campaign by specifying a new path where campaign is located.

Parameters:new_path (str) – new runs directory
replace_actions(app_name, actions)[source]

Replace actions for an app with a given name.

Parameters:
  • app_name (str) – Name of the app.
  • actions (Actions) – Actions instance, will replace the current Actions of an app.
rerun(list_of_run_IDs)[source]

Sets the status of the specified runs to ENCODED, so that their results may be recollated later (presumably after extending, rerunning or otherwise modifying the data in the relevant run folder). Note that this method will NOT perform any execution - it simply flags the run in EasyVVUQ as being uncollated. Actual execution is (as usual) the job of the user or middleware.

Parameters:list – The list of run IDs for the runs that should be set to status ENCODED
set_app(app_name)[source]

Set active app for the campaign.

Application information is retrieved from self.campaign_db.

Parameters:app_name (str) – Name of selected app, if None given then first app will be selected.
set_sampler(sampler, update=False)[source]

Set active sampler.

Parameters:
  • sampler (Sampler) – Sampler that will be used to create runs for the current campaign.
  • update (bool) – If set to True it will not add the sampler to the database, just change it as the active sampler.

easyvvuq.constants module

Constants and Enums to set defaults and constrain selections

ivar default_campaign_prefix:
 Text used to ensure campaign names are identifiable and somewhat human readable.
vartype default_campaign_prefix:
 str
class easyvvuq.constants.OutputType[source]

Bases: enum.Enum

Types of data output by UQPs/VVPs

ARRAY = 'array'
SAMPLE = 'sample'
SAMPLE_ARRAY = 'sample_array'
SUMMARY = 'summary'
TRACK = 'track'
class easyvvuq.constants.Status[source]

Bases: enum.IntEnum

Status of runs in the Run Table

COLLATED = 3
ENCODED = 2
IGNORED = 4
INVALID = 5
NEW = 1

easyvvuq.data_structs module

Data structures to ensure consistency during serialization for databases.

class easyvvuq.data_structs.AppInfo(name=None, paramsspec=None, actions=None)[source]

Bases: object

Handles information for particular application.

Variables:
  • name (str or None) – Human readable application name.
  • paramsspec (ParamsSpecification or None) – Description of possible parameter values.
to_dict(flatten=False)[source]

Convert to a dictionary (optionally flatten to single level)

Parameters:flatten (bool) – Should the return dictionary be single level (i.e. should paramsspec be serialized).
Returns:Dictionary representing the application- if flattened then paramsspec is returned as a JSON format sting.
Return type:dict
class easyvvuq.data_structs.CampaignInfo(name=None, easyvvuq_version=None, campaign_dir_prefix=None, campaign_dir=None, runs_dir=None, local=False)[source]

Bases: object

Handles information on Campaign.

Parameters:
  • name (str or None) – Human readable campaign name.
  • easyvvuq_version (str or None) – Version of EasyVVUQ used to create the campaign.
  • campaign_dir_prefix (str or None) – Prefix test for campaign directory.
  • campaign_dir (str or None,) – Path to the campaign directory.
  • runs_dir (str or None) – path to run directory (within the campaign directory)
  • local (bool, default=False) – Is this campaign designed to be created and executed on the same machine?
Variables:
  • name (str or None) – Human readable campaign name.
  • easyvvuq_version (str or None) – Version of EasyVVUQ used to create the campaign.
  • campaign_dir_prefix (str or None) – Prefix test for campaign directory.
  • campaign_dir (str or None,) – Path to the campaign directory.
  • runs_dir (str or None) – path to run directory (within the campaign directory)
easyvvuq_version
to_dict(flatten=False)[source]

Convert this to a dictionary

Parameters:flatten (bool) – Should the return dictionary be single level (always true here).
Returns:Dictionary representing the campaign.
Return type:dict
class easyvvuq.data_structs.RunInfo(run_name=None, run_dir=None, app=None, params=None, sample=None, campaign=None, status=<Status.NEW: 1>)[source]

Bases: object

Handles information for individual application runs.

Parameters:
  • run_name (str) – Human readable name of the run.
  • app (None or int) – ID of the associated application.
  • params (None or dict) – Dictionary of parameter values for this run.
  • sample (None or int) – ID of the sampler that created the run.
  • campaign (None or int) – ID of the associated campaign.
Variables:
  • campaign (int) – ID of the associated campaign.
  • sample (int) – ID of the sampler that created the run.
  • app (int) – ID of the associated application.
  • run_name (str) – Human readable name of the run.
  • status (enum(Status)) –
to_dict(flatten=False)[source]

Convert to a dictionary (optionally flatten to single level)

Parameters:flatten (bool) – Should the return dictionary be single level (i.e. should params or other dictionary variables be serialized).
Returns:Dictionary representing the run - if flattened then params are returned as a JSON format sting.
Return type:dict
easyvvuq.data_structs.check_local_dir(path, dir_type='campaign')[source]

Check that local path exists and if not create it.

Parameters:
  • path (str) – Directory location to check.
  • dir_type (str, default=’campaign’) – Type of directory we are checking (used for user and debugging information.)
easyvvuq.data_structs.check_reference(ref, run_name, ref_type='campaign')[source]

Validation check for a RunInfo reference. Checks that an integer value has been passed to use as a reference to another ‘table’ - i.e. to a specific campaign, app or sampler.

Parameters:
  • ref (int) – Reference to be checked.
  • run_name (str) – Name of run for which the check is being performed (user info/ debugging).
  • ref_type (str, default=’campaign’) – Are we checking for a campaign, sampler or app (user info/ debugging).

easyvvuq.params_specification module

Data structures to ensure consistency during serialization for databases.

class easyvvuq.params_specification.EasyVVUQValidator(*args, **kwargs)[source]

Bases: cerberus.validator.Validator

Validator class. Normalizes and/or validates any mapping against a validation-schema which is provided as an argument at class instantiation or upon calling the validate(), validated() or normalized() method. An instance itself is callable and executes a validation.

All instantiation parameters are optional.

There are the introspective properties types, validators, coercers, default_setters, rules, normalization_rules and validation_rules.

The attributes reflecting the available rules are assembled considering constraints that are defined in the docstrings of rules’ methods and is effectively used as validation schema for schema.

Parameters:
  • schema (any mapping) – See schema. Defaults to None.
  • ignore_none_values (bool) – See ignore_none_values. Defaults to False.
  • allow_unknown (bool or any mapping) – See allow_unknown. Defaults to False.
  • require_all (bool) – See require_all. Defaults to False.
  • purge_unknown (bool) – See purge_unknown. Defaults to to False.
  • purge_readonly (bool) – Removes all fields that are defined as readonly in the normalization phase.
  • error_handler (class or instance based on BaseErrorHandler or tuple) – The error handler that formats the result of errors. When given as two-value tuple with an error-handler class and a dictionary, the latter is passed to the initialization of the error handler. Default: BasicErrorHandler.
checkers = ()
coercers = ()
default_setters = ()
normalization_rules = {'coerce': {'oneof': [{'type': 'callable'}, {'type': 'list', 'schema': {'oneof': [{'type': 'callable'}, {'type': 'string', 'allowed': ()}]}}, {'type': 'string', 'allowed': ()}]}, 'default': {'nullable': True}, 'default_setter': {'oneof': [{'type': 'callable'}, {'type': 'string', 'allowed': ()}]}, 'purge_unknown': {'type': 'boolean'}, 'rename': {'type': 'hashable'}, 'rename_handler': {'oneof': [{'type': 'callable'}, {'type': 'list', 'schema': {'oneof': [{'type': 'callable'}, {'type': 'string', 'allowed': ()}]}}, {'type': 'string', 'allowed': ()}]}}
rules = {'allof': {'logical': 'allof', 'type': 'list'}, 'allow_unknown': {'oneof': [{'type': 'boolean'}, {'type': ['dict', 'string'], 'check_with': 'bulk_schema'}]}, 'allowed': {'type': 'container'}, 'anyof': {'logical': 'anyof', 'type': 'list'}, 'check_with': {'oneof': [{'type': 'callable'}, {'type': 'list', 'schema': {'oneof': [{'type': 'callable'}, {'type': 'string', 'allowed': ()}]}}, {'type': 'string', 'allowed': ()}]}, 'coerce': {'oneof': [{'type': 'callable'}, {'type': 'list', 'schema': {'oneof': [{'type': 'callable'}, {'type': 'string', 'allowed': ()}]}}, {'type': 'string', 'allowed': ()}]}, 'contains': {'empty': False}, 'default': {'nullable': True}, 'default_setter': {'oneof': [{'type': 'callable'}, {'type': 'string', 'allowed': ()}]}, 'dependencies': {'check_with': 'dependencies', 'type': ('dict', 'hashable', 'list')}, 'empty': {'type': 'boolean'}, 'excludes': {'schema': {'type': 'hashable'}, 'type': ('hashable', 'list')}, 'forbidden': {'type': 'list'}, 'items': {'check_with': 'items', 'type': 'list'}, 'keysrules': {'check_with': 'bulk_schema', 'forbidden': ['rename', 'rename_handler'], 'type': ['dict', 'string']}, 'max': {'nullable': False}, 'maxlength': {'type': 'integer'}, 'meta': {}, 'min': {'nullable': False}, 'minlength': {'type': 'integer'}, 'noneof': {'logical': 'noneof', 'type': 'list'}, 'nullable': {'type': 'boolean'}, 'oneof': {'logical': 'oneof', 'type': 'list'}, 'purge_unknown': {'type': 'boolean'}, 'readonly': {'type': 'boolean'}, 'regex': {'type': 'string'}, 'rename': {'type': 'hashable'}, 'rename_handler': {'oneof': [{'type': 'callable'}, {'type': 'list', 'schema': {'oneof': [{'type': 'callable'}, {'type': 'string', 'allowed': ()}]}}, {'type': 'string', 'allowed': ()}]}, 'require_all': {'type': 'boolean'}, 'required': {'type': 'boolean'}, 'schema': {'anyof': [{'check_with': 'schema'}, {'check_with': 'bulk_schema'}], 'type': ['dict', 'string']}, 'type': {'check_with': 'type', 'type': ['string', 'list']}, 'valuesrules': {'check_with': 'bulk_schema', 'forbidden': ['rename', 'rename_handler'], 'type': ['dict', 'string']}}
validation_rules = {'allof': {'logical': 'allof', 'type': 'list'}, 'allow_unknown': {'oneof': [{'type': 'boolean'}, {'type': ['dict', 'string'], 'check_with': 'bulk_schema'}]}, 'allowed': {'type': 'container'}, 'anyof': {'logical': 'anyof', 'type': 'list'}, 'check_with': {'oneof': [{'type': 'callable'}, {'type': 'list', 'schema': {'oneof': [{'type': 'callable'}, {'type': 'string', 'allowed': ()}]}}, {'type': 'string', 'allowed': ()}]}, 'contains': {'empty': False}, 'dependencies': {'check_with': 'dependencies', 'type': ('dict', 'hashable', 'list')}, 'empty': {'type': 'boolean'}, 'excludes': {'schema': {'type': 'hashable'}, 'type': ('hashable', 'list')}, 'forbidden': {'type': 'list'}, 'items': {'check_with': 'items', 'type': 'list'}, 'keysrules': {'check_with': 'bulk_schema', 'forbidden': ['rename', 'rename_handler'], 'type': ['dict', 'string']}, 'max': {'nullable': False}, 'maxlength': {'type': 'integer'}, 'meta': {}, 'min': {'nullable': False}, 'minlength': {'type': 'integer'}, 'noneof': {'logical': 'noneof', 'type': 'list'}, 'nullable': {'type': 'boolean'}, 'oneof': {'logical': 'oneof', 'type': 'list'}, 'readonly': {'type': 'boolean'}, 'regex': {'type': 'string'}, 'require_all': {'type': 'boolean'}, 'required': {'type': 'boolean'}, 'schema': {'anyof': [{'check_with': 'schema'}, {'check_with': 'bulk_schema'}], 'type': ['dict', 'string']}, 'type': {'check_with': 'type', 'type': ['string', 'list']}, 'valuesrules': {'check_with': 'bulk_schema', 'forbidden': ['rename', 'rename_handler'], 'type': ['dict', 'string']}}
class easyvvuq.params_specification.ParamsSpecification(params, appname=None)[source]

Bases: object

static deserialize(serialized_params)[source]
process_run(new_run, verify=True)[source]
serialize()[source]

Module contents