eqi package

class eqi.Executor(campaign, config_file=None, resume=True, log_level='info')

Bases: object

Integrates EasyVVUQ and QCG-PilotJob Manager

Executor allows to process the most demanding operations of EasyVVUQ in parallel using QCG-PilotJob.

create_manager(resources=None, reserve_core=False, log_level='info')

Creates new QCG-PilotJob Manager and sets is as the Executor’s engine.

Parameters:
  • resources (str, optional) – The resources to use. If specified forces usage of Local mode of QCG-PilotJob Manager. The format is compliant with the NODES format of QCG-PilotJob, i.e.: [node_name:]cores_on_node[,node_name2:cores_on_node][,…]. Eg. to run on 4 cores regardless the node use resources=”4” to run on 2 cores of node_1 and on 3 cores of node_2 use resources=”node_1:2,node_2:3”
  • reserve_core (bool, optional) – If True reserves a core for QCG-PilotJob Manager instance, by default QCG-PilotJob Manager shares a core with computing tasks Parameters.
  • log_level (str, optional) – Logging level for QCG-PilotJob Manager (for both service and client part).
Returns:
Return type:

None
set_manager(qcgpjm)

Sets existing QCG-PilotJob Manager as the Executor’s engine

Parameters:qcgpjm (qcg.pilotjob.api.manager.Manager) – Existing instance of a QCG-PilotJob Manager
Returns:
Return type:None
add_task(task)

Add a task to execute with QCG PJ

Parameters:task (Task) – The task that will be added to the execution workflow
Returns:
Return type:None
run(processing_scheme=<ProcessingScheme.SAMPLE_ORIENTED: 'Submits a workflow of EasyVVUQ operations as separate QCG PJ tasks for a sample (e.g. encoding -> execution) and then goes to the next sample'>)

Executes demanding parts of EasyVVUQ campaign with QCG-PilotJob

A user may choose the preferred execution scheme for the given scenario.

Parameters:processing_scheme (ProcessingScheme) – Tasks processing scheme
Returns:
Return type:None
print_resources_info()

Displays resources assigned to QCG-PilotJob Manager

terminate_manager()

Terminates QCG-PilotJob Manager

class eqi.Task(type, requirements=None, name=None, model='default', resume_level=<ResumeLevel.BASIC: 'For the tasks creating run directories, the resume checks if an unfinished task created such directory. If such directory is available, this is automatically removed before the start of the resumed task'>, **params)

Bases: object

Represents a piece of work to be executed by QCG-PilotJob Manager

Parameters:
  • type (TaskType) –
    The type of the task. Allowed tasks are: ENCODING, EXECUTION, ENCODING_AND_EXECUTION,
    and OTHER (currently not supported)
  • requirements (TaskRequirements, optional) – The requirements for the Task
  • name (str, optional) – name of the Task, if not provided the name will take a value of type
  • model (str, optional) – Allows to set the flavour of execution of task adjusted to a given resource. At the moment of writing a user can select from the following models: threads, intelmpi, openmpi, srunmpi, default
  • resume_level (ResumeLevel, optional) – The resume level applied for a task.
  • params (kwargs) – additional parameters that may be used by specific Task types
get_type()
get_requirements()
get_model()
get_resume_level()
get_params()
get_name()
class eqi.TaskType

Bases: enum.Enum

An enumeration.

ENCODING = 'ENCODING'
EXECUTION = 'EXECUTION'
ENCODING_AND_EXECUTION = 'ENCODING_AND_EXECUTION'
OTHER = 'OTHER'
class eqi.ProcessingScheme(description, iterative=False)

Bases: enum.Enum

Specifies scheme of processing of tasks with QCG-PJ

Parameters:
  • description (str) – Description of the ProcessingScheme
  • iterative (bool) – Defines if the ProcessingScheme uses iterative tasks of QCG-PJ
STEP_ORIENTED = 'Submits specific EasyVVUQ operation (e.g. encoding) for all samples as a separate QCG PJ tasksand then goes to the next EasyVVUQ operation (e.g. execution)'
STEP_ORIENTED_ITERATIVE = ('Submits an iterative task for execution of specific EasyVVUQ operation (e.g. encoding) for all samples (a single iteration is here an execution of the encoding operation for a single sample)and then do the same for the next EasyVVUQ operation (e.g. for execution)', True)
SAMPLE_ORIENTED = 'Submits a workflow of EasyVVUQ operations as separate QCG PJ tasks for a sample (e.g. encoding -> execution) and then goes to the next sample'
SAMPLE_ORIENTED_CONDENSED = 'Submits all EasyVVUQ operations for a sample as a single QCG PJ task (e.g. encoding -> execution) and then goes to the next sample'
SAMPLE_ORIENTED_CONDENSED_ITERATIVE = ('Submits an iterative QCG PJ task for all samples, where a single iteration is composed ofall EasyVVUQ operations for a sample (e.g. encoding -> execution)', True)
EXEC_ONLY = 'Submits a workflow of EasyVVUQ operations as separate QCG PJ tasks for execution only'
EXEC_ONLY_ITERATIVE = ('Submits an iterative QCG PJ task for all samples, where a single iteration is an execution of sample ', True)
is_iterative()

Checks if ProcessingScheme makes use of iterative QCG-PJ tasks

Returns:bool
Return type:True if ProcessingScheme uses iterative QCG-PJ tasks
class eqi.TaskRequirements(cores: Union[int, eqi.core.task_requirements.Resources, None] = None, nodes: Union[int, eqi.core.task_requirements.Resources, None] = None)

Bases: object

Requirements for a task executed within QCG Pilot Job

Parameters:
  • cores (int or eqi.Resources) – the resource requirements for cores
  • nodes (int or eqi.Resources) – the resource requirements for nodes
get_resources()

Allows to get resource requirements in a form of dictionary understandable by QCG Pilot Job Manager

Returns:dict
Return type:dictionary with the resource requirements specification
class eqi.Resources(exact=None, min=None, max=None, split_into=None)

Bases: object

Stores typical for QCG Pilot Job resource requirements

Parameters:
  • exact (Number) – The exact number of resources
  • min (Number) – The minimal acceptable number of resources
  • max (Number) – The maximal acceptable number of resources
  • split_into (Number) – The anticipated number of chunks to which the total resources should be split. The minimal number of resources in a chunk will be restricted by the value of ‘min’.
get_dict()
Returns:Dictionary of resource requirements.
Return type:Dict
class eqi.ResumeLevel

Bases: enum.Enum

Typically the resumed task will start in a working directory of the previous, not-completed run. Sometimes the partially generated output or intermediate files could be a problem. Therefore EQI tries to help in this matter by providing dedicated mechanisms for automatic recovery. However, this depends on the use case, how much the automatism can interfere with the resume logic. Therefore there are a few levels of automatic recovery available.

DISABLED = 'Automatic resume is fully disabled'
BASIC = 'For the tasks creating run directories, the resume checks if an unfinished task created such directory. If such directory is available, this is automatically removed before the start of the resumed task'
MODERATE = "This level processes all operations offered by BASIC level, and adds the following.At the beginning of a task's execution, the list of directories and files in a run directory is generated and stored. The resumed task checks for the differences and remove new files and directoriesin order to resurrect the initial state."
class eqi.StateKeeper(directory)

Bases: object

EQI_CAMPAIGN_STATE_FILE_NAME = '.eqi_campaign_state.json'

Stores information about EQI execution

Parameters:
  • directory (string) – the root directory where the information will be stored
  • campaign (Campaign) – the EasyVVUQ Campaign object from which the StateKeeper will be inited
setup(campaign)

Initialises StateKeeper with campaign

Parameters:campaign (Campaign) – the EasyVVUQ Campaign object from which the StateKeeper will be inited
write_to_state_file(data)
get_from_state_file()