The imsi API

This section describes the imsi Application Programming Interface (API).

Command line interface (cli.py) and link to backend (ui_manager.py)

class imsi.user_interface.ui_manager.Override(*, options: str)

Bases: BaseModel

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

options: str

classmethod validate_options(v)

imsi.user_interface.ui_manager.assert_imsi_version_match(version)

Check and return the version requirements contained in the version controlled config files. Raises if versions do not match.

The imsi minor version is toggled when there are config breaking changes, as such we only require the major and minor version match.

Inputs:

version : version to check, usually from a requirements file

Returns:

Raises IMSIVersionMismatchError if versions are mismatched. Otherwise returns True.

imsi.user_interface.ui_manager.build_run_config_on_disk(configuration: Configuration, db: ConfigDatabase, track=True, force=False)

This actually creates the physical config directory on disk, and extracts/modifies various relevant files

imsi.user_interface.ui_manager.check_if_imsi_compliant_version(path: Path, print_message: bool = False) → bool

imsi.user_interface.ui_manager.compile_model_execs(args, force=False)

Builds all component executables by calling an upstream script from the repository. Should be under /bin but not enforceable.

imsi.user_interface.ui_manager.create_imsi_configuration(imsi_config_path: str, setup_params: ~typing.Dict) -> (<class 'imsi.config_manager.config_manager.Configuration'>, <class 'imsi.config_manager.databases.ConfigDatabase'>)

Build and return configuration instance and config db given imsi_config_path

imsi.user_interface.ui_manager.get_all_state_files(state_path)

Return all the files in the state folder

imsi.user_interface.ui_manager.get_current_imsi_version(no_patch=True)

imsi.user_interface.ui_manager.get_init_src_descr(work_dir: str | Path)

Returns the contents of the initial src repo state from setup time, contained in {work_dir}/.imsi/init/state_descriptor_src_setup.txt.

imsi.user_interface.ui_manager.get_init_src_state(work_dir, verbose=False)

Return the initial state description for the src folder, from setup time.

If verbose, returns the contents of the state artefact files for rev-parse (*rev.txt) and status (*status.txt) (separated by newline). Otherwise, returns the single-line git repo status query (hash and clean/dirty flags).

imsi.user_interface.ui_manager.get_init_state_files(work_dir, stateful_folder: str = 'src')

Return list of files for the stateful_folder from init state

imsi.user_interface.ui_manager.get_init_state_folder(work_dir: str | Path, realpath: bool = True) → str

Return the path to the initial imsi state hash folder. Default is the true path (realpath=True, ie readlink). Otherwise the return is {work_dir}/.imsi/init/state.

imsi.user_interface.ui_manager.get_init_state_hash(work_dir: str | Path)

Return the initial imsi state hash, from .imsi/init/state

imsi.user_interface.ui_manager.get_required_imsi_version(source_config_path: Path = PosixPath('src/imsi-config'), version_req_file: str = 'version_requirements.yaml')

Get the imsi version of the source repo.

Raises IMSIVersionRequirementNotFoundError if the version_requirement_file is missing.

Returns a tuple of the version requirement file path and the version

imsi.user_interface.ui_manager.get_sequencer_status()

imsi.user_interface.ui_manager.get_setup_param(setting: str, work_dir: str | Path = None)

Get settings from setup_params in Configuration

imsi.user_interface.ui_manager.get_stateful_folder_files(state_path, stateful_folder)

Return a mapping of files for the stateful_folder from the state folder, keys correspond to type of artefact (rev, diff, status)

imsi.user_interface.ui_manager.parse_override(options: Iterable[str], force=False)

Parse and apply command-line option overrides to the current IMSI run configuration.

Parameters: - options: An iterable of option identifier strings. Each string must contain

a ‘/’ separating the group and option name (e.g. “group/option” or “group/option=value”).

• force: If True, forces rebuilding of on-disk run artifacts

  and other actions performed by build_run_config_on_disk even if not strictly necessary. Defaults to False.

imsi.user_interface.ui_manager.parse_override_options(options: Iterable[str]) → list[dict]

imsi.user_interface.ui_manager.parse_state_components(work_dir, state, stateful_folder: str = 'src', exclude: str | list[str] = 'diff', header=True)

Parse the contents of the stateful_folder contents from the state folder, returned as a single string. exclude will exclude the type of state artefact (rev, status, diff)

imsi.user_interface.ui_manager.query_time()

Instantiate the configuration / SimulationTime instances and enable querying timers

imsi.user_interface.ui_manager.reload_config_from_source(force=False)

Build a new config directory from upstream imsi source

This will re-extract everthing out of the cloned repository to re-create the config directory. I.e. if one made changes in the repo after setup, and wanted to apply them, they would call this update function.

imsi.user_interface.ui_manager.save_restarts(args)

Execute the save restarts script

imsi.user_interface.ui_manager.set_selections(parm_file=None, selections=None, force=False)

Parse key=value pairs of selection given on the command line Try to apply these to the imsi selections for the sim.

imsi.user_interface.ui_manager.submit_run()

Instantiate the configuration object and submit job to queue

imsi.user_interface.ui_manager.tapeload_rs(args)

Execute the tapeload rs script

imsi.user_interface.ui_manager.update_config_from_state(force=False)

Apply changes made in the “imsi_configuration_${runid}” state file to the configuration and update the config directory as appropriate.

imsi.user_interface.ui_manager.validate_version_reqs(source_config_path: Path = PosixPath('src/imsi-config'))

Validates if the imsi-config source repo’s version works with the imsi version being used.

Configuration manager (config_manager.py)

This module provides classes and utilities for managing configurations, including models, experiments, machines, compilers, and their interactions.

The module includes classes for defining configuration elements such as Model, Experiment, Machine, and Compiler. Additionally, it provides a Configuration class for composing these elements into a complete configuration.

ConfigManager is a class used to establish configuration objects and facilitate saving and loading configurations for later use. The module also includes abstract classes such as ConfigDatabase for defining an abstract interface for a configuration database and its concrete implementation JsonConfigDatabase.

Lastly, the module contains Factory classes (eg. CompilerFactory), which provide methods for creating instances of analogous classes (eg. Compiler).

The module is designed to support flexibility in configuration management, allowing configurations to be composed, serialized, and deserialized.

This module is a WIP. It is next required to create infrastructure modules that use the configurations created here.

Neil Swart, April 2024

class imsi.config_manager.config_manager.CompilerFactory

Bases: object

Class containing methods to instantiate an instance of Compiler. Like for machines, several parsing functions are required and encapsulated here.

static create_from_database(db: ConfigDatabase, machine: Machine, compiler_name: str = None)

class imsi.config_manager.config_manager.ConfigManager(db: ConfigDatabase = None)

Bases: object

This class is used to establish configuration objects, as well as save/load them for later use

create_SetupParams(setup_params: Dict, machine: Machine) → SetupParams

create_compiler(machine: Machine, compiler_name: str = None) → Compiler

create_components(model: Model, experiment: Experiment) → Components

create_configuration(model_name: str, experiment_name: str, machine_name: str = '', compiler_name: str = '', sequencer_name: str = '', flow_name: str = '', postproc_profile: str = '', **kwargs)

Create the individual instances of config elements and return a configuration composed of these

create_experiment(experiment_name: str, model_name: str) → Experiment

create_machine(machine_name: str = None) → Machine

create_model(model_name: str) → Model

create_postproc(model: Model, experiment: Experiment, machine: Machine, postproc_profile: str) → PostProcessing

create_sequencing(machine: Machine, experiment: Experiment, model: Model, sequencer_name: str, flow_name: str) → Sequencing

create_utilities() → Utilities

load_configuration(filepath) → Configuration

Load the configuration from a file

load_state(filepath) → Configuration

Load the configuration state from a file

classmethod save_configuration(configuration: Configuration, filepath: str)

Save the configuration to a file

classmethod save_state(configuration: Configuration, filepath: str)

Pickle the configuration object

class imsi.config_manager.config_manager.Configuration(*, model: Model, experiment: Experiment, components: Components, machine: Machine, compiler: Compiler, postproc: PostProcessing, setup_params: SetupParams, utilities: Utilities, sequencing: Sequencing)

Bases: BaseModel

Container class that combines sub-configurations and serves as the goto reference defining the configuration of a simulation.

compiler: Compiler

components: Components

experiment: Experiment

get_unique_key_value(key: str)

Search recursively through the nested dicts of the configuration to try and find a specified key and return its value if the key is unique. If mulitple instances of they key exist, return an error.

machine: Machine

model: Model

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(_Configuration__context)

Used to specifically update defaults only for specific configs

postproc: PostProcessing

sequencing: Sequencing

setup_params: SetupParams

utilities: Utilities

class imsi.config_manager.config_manager.ExperimentFactory

Bases: object

Class containing methods to instantiate an instance of Experiment. On particular check that the model and experiment are consistent

static create_from_database(db: ConfigDatabase, experiment_name: str, model_name: str)

Create a sequencing instance given the config db, the machine, sequencer and flow names

class imsi.config_manager.config_manager.PostprocFactory

Bases: object

Class containing methods to instantiate an instance of Postprocessing. Fetch default from Experiment object (preferred) or Model object (backup) if postproc is not set by the user.

static create_from_database(db: ConfigDatabase, model: Model, experiment: Experiment, machine: Machine, postproc_name: str = None)

static get_default_postproc(model, experiment) → str

Retrieve the default postproc_profile

class imsi.config_manager.config_manager.SequencingFactory

Bases: object

Class containing methods to instantiate an instance of Sequencing. Like for machines, several parsing functions are required and encapsulated here.

static create_from_database(db: ConfigDatabase, machine: Machine, experiment: Experiment, model: Model, sequencer_name: str, flow_name: str = None, model_type: str = None)

Create a sequencing instance given the config db, the machine, sequencer and flow names

static determine_default_flow(model_type: str, sequencer_name: str, sequencer_baseflows: dict, machine_specific_flows, machine: Machine, experiment: Experiment, model: Model)

Get the default sequencing flow, which handles the selected model_type, and also has configuration support for the selected sequencer and machine.

Parameters:

model_type (str): The model configuration, e.g. ESM, AMIP or OMIP

static set_flow_config(flow_name: str, flows: dict, sequencer_name: str, sequencer_config: dict, machine: Machine, model_type: str)

static set_sequencer_config(sequencer_name: str, sequencers: dict, machine_name: str)

static verify_sequencing_structure(sequencing_config: dict)

Checks what we got from the db includes mandatory sections (replaceable by schema validation?)

imsi.config_manager.config_manager.database_factory(imsi_config_path: str, config_type: str = 'yaml') → ConfigDatabase

Factory function to create a ConfigDatabase instance based on the config_type

Shell interface

Generates the shell interface to the modelling system.

In general this means creating a “config” directory with various files that the model and downstream utilities ingest, such as namelists, and shell_parameters files.

imsi.shell_interface.shell_interface_manager.build_config_dir(db: ConfigDatabase, configuration: Configuration, track=True, force=False)

Main composition function to manange the build the config directory with appropriately parsed content

imsi.shell_interface.shell_comp_environment.generate_compilation_template(configuration: Configuration)

Generate the compilation_template file contents based on a Configuraiton instance

Here we just generate and return a list of strings, that will be written to file elsewhere using a common tool.

imsi.shell_interface.shell_comp_environment.generate_computational_environment(comp_env: dict, machine_name, compiler_name: str = None) → List[str]

Generate content of a file that can be sourced in a shell to set the computational environment for compiling and running the model.

Here we just generate and return a list of strings, that will be written to file elsewhere using a common tool.

imsi.shell_interface.shell_comp_environment.generate_computational_environment_controller(machine_config: Machine, run_config_path: str, basename_prefix: str = None) → List[str]

Generate the contents of the computational environment ‘controller’ file.

The controller is a passthrough shell file that simply sources the correct computational_environment file for the machine at runtime. It does so by matching the hostname of the current machine (via regex) to the machine name (used in imsi config). Both the regex and machine name are known from upstream config and supplied via the machine_config (instance of Machine). The file that is sourced is then: {run_config_path}/{basename_prefix}_{machine name}.

Parameters:

machine_config: Machine

machine configuration object

run_config_path: str

path to run config folder (usually {runid}/config)

basename_prefix: str

file basename prefix for file names to source.

Returns:

contents of the controller file, a list of strings (lines)

imsi.shell_interface.shell_config_parameters.generate_flattened_config(full_config_dict: dict)

This creates a far more extenives shell parameters file content for the simulation, that contains every variable defined in the imsi configuration.

The idea is not to practically use this, but to demonstrate it could be possible to make a clean break and do anything in shell downstream of this.

Here we just generate and return a list of strings, that will be written to file elsewhere using a common tool.

imsi.shell_interface.shell_config_parameters.generate_shell_parameters(shell_config: dict)

This creates a shell_parameters file contents for the simulation, that contains variable definitions required by downstream shell scripting. What appears in this file is defined by the “shell_config” template in the imsi json.

Here we just generate and return a list of strings, that will be written to file elsewhere using a common tool.

imsi.shell_interface.shell_config_parameters.set_shell_config(shell_config_template: dict, full_config_dict: dict) → dict

Set shell parameters based on shell_config.yaml and internal imsi variables

Parameters:

shell_config_templatedict

A template of the shell config, from the database (contains {{}} enclosed strings to be replaced)

full_config_dictdict

The full configuration as a single dictionary, which will be searched for key=value in order to do the replacements in the template above

imsi.shell_interface.shell_diag_parameters.generate_diag_parameters_content(diag_config)

This creates content for the diag_parameters file for the simulation, that contains variable definitions required by downstream shell scripting. Currently this is just a pure propagation of variables from the diagnostic config. Ultimately, the interaction of imsi with diagnostics needs to be refined.

imsi.shell_interface.shell_inputs_outputs.add_atmos_forcing_preamble(component_config, script_content)

imsi.shell_interface.shell_inputs_outputs.extract_utility_files(files_to_extract: Dict, imsi_config_path: str, work_dir: str)

Extract utility files from the source repository into workdir for use. The file source path is defined relative to imsi_config_path The destination path is defined relative to the run work_dir The destination path cannot be in: src, config, sequencer

imsi.shell_interface.shell_inputs_outputs.generate_directory_packing_content(component_dict: dict) → List[str]

Generate directory packing commands based on content in an imsi “components” config, to be called post model run

imsi.shell_interface.shell_inputs_outputs.generate_final_file_saving_content(component_dict: dict) → List[str]

Generate all required saved commands for final outputs based on an imsi “components” config (dict) to be called post model run and post directory packing

imsi.shell_interface.shell_inputs_outputs.generate_input_script_content(component_dict: dict, shell_config: dict) → List[str]

Generate content for the input files script (to be called at runtime with access’ etc)

imsi.shell_interface.shell_inputs_outputs.process_cpp_file(ref_file_path, target_file_path, file_content)

Process cpp files Updates default cpp files from ref_file_path with “file_content”

imsi.shell_interface.shell_inputs_outputs.process_model_config_files(component: str, component_config: dict, imsi_config_path: str, run_config_path: str, file_type: str, shell_config: dict)

Process namelist and compilation files for all component defined in component_config

Does replacements of {{}} variables using shell_config inputs Updates reference files from “ref_file_path” with “file_content”

imsi.shell_interface.shell_inputs_outputs.process_namelist_file(ref_file_path, target_file_path, file_content, shell_config)

Process namelist files Does replacements of {{}} variables using shell_config inputs Updates default namelists from ref_file_path with “file_content”

General utility functions for creating the “shell interface” (config directory) and associated files.

imsi.shell_interface.shell_interface_utilities.create_config_dirs(run_config_path: str, components: List[str], track=True, force=False)

Create the run config dir and subdirectories for each component.

Scheduler interface

class imsi.scheduler_interface.schedulers.BatchJob(user_script: ~typing.List[str], job_directives: ~typing.Dict[str, str] = <factory>)

Bases: object

Defines a batch job with specific directives

construct_job_script(scheduler: Scheduler) → List[str]

Constructs the entire job script from the header and user script using the scheduler’s directives

job_directives: Dict[str, str]

user_script: List[str]

class imsi.scheduler_interface.schedulers.PBSScheduler(name: str = 'PBS', directive_prefix: str = '#PBS', submission_command: str = 'qsub', queue_info_command: str = 'qstat', cancel_command: str = 'qdel', output_redirect: str = '-o {PATH}.o', default_directives: list = <factory>)

Bases: Scheduler

cancel_command: str = 'qdel'

cancel_job(job_id: str) → str

Cancels a job given its job ID

construct_job_header(job_directives: List) → str

Constructs the job header based on directives

default_directives: list

directive_prefix: str = '#PBS'

name: str = 'PBS'

output_redirect: str = '-o {PATH}.o'

query_queue() → str

Queries the job queue and returns the queue status

queue_info_command: str = 'qstat'

submission_command: str = 'qsub'

submit(job_script_filename: str) → str

Submits a job and returns the job ID

class imsi.scheduler_interface.schedulers.SLURMScheduler(name: str = 'SLURM', directive_prefix: str = '#SBATCH', submission_command: str = 'sbatch', queue_info_command: str = 'squeue', cancel_command: str = 'scancel', output_redirect: str = '-o {PATH}_%j.out', default_directives: list = <factory>)

Bases: Scheduler

cancel_command: str = 'scancel'

cancel_job(job_id: str) → str

Cancels a job given its job ID

construct_job_header(job_directives: List) → str

Constructs the job header based on directives

default_directives: list

directive_prefix: str = '#SBATCH'

name: str = 'SLURM'

output_redirect: str = '-o {PATH}_%j.out'

query_queue() → str

Queries the job queue and returns the queue status

queue_info_command: str = 'squeue'

submission_command: str = 'sbatch'

submit(job_script_filename: str) → str

Submits a job and returns the job ID

class imsi.scheduler_interface.schedulers.Scheduler(name: str, directive_prefix: str, submission_command: str, queue_info_command: str, cancel_command: str, output_redirect: str)

Bases: ABC

Defines an abstract interface for scheduler classes

cancel_command: str

abstract cancel_job(job_id: str) → str

Cancels a job given its job ID

abstract construct_job_header(job_directives: List) → List

Constructs the job header based on directives

directive_prefix: str

name: str

output_redirect: str

abstract query_queue() → str

Queries the job queue and returns the queue status

queue_info_command: str

submission_command: str

abstract submit(job: BatchJob) → str

Submits a job and returns the job ID

imsi.scheduler_interface.schedulers.create_scheduler(scheduler_name: str) → Scheduler

Factory method to create a scheduler instance based on the scheduler name

imsi.scheduler_interface.schedulers.main()

Sequencer interface

sequencers

Provide a generic sequencer class, as an interface for dedicated sequencer sub-classes. This allows imsi to interface with multiple underlying sequencers.

class imsi.sequencer_interface.sequencers.Sequencer

Bases: ABC

A class that absracts the definition of a sequencer cap for imsi.

Sub-classes for specific sequencers will provide the concerete implementations of the methods below, using the imsi Configuration information, and using their own specifically required methods.

The sequencer specific implementation must expose back, through this interface, all the methods below.

abstract config(configuration: Configuration)

Steps needed to configure sequencer files based on updated imsi input. Examples might include editing resource files.

abstract setup(configuration: Configuration)

Any steps needed to setup the sequencer, including cloning source and creating any directories

abstract status(configuration: Configuration)

Steps needed to configure sequencer files based on updated imsi input. Examples might include editing resource files.

abstract submit(configuration: Configuration)

Steps needed to configure sequencer files based on updated imsi input. Examples might include editing resource files.

imsi.sequencer_interface.sequencers.create_sequencer(seq_name)

Create and return a sequencer object

An interface between IMSI upstream configuration and tooling and the iss.

imsi configuration is taken and processed into the configuration required for iss. Specifically, the IMSISimpleSequencerInterface exposes methods that correspond to imsi setup, config, and submit.

class imsi.sequencer_interface.iss_cap.IMSISimpleSequencerInterface

Bases: Sequencer

An interface between upstream imsi tooling and the IMSISimpleSequencer.

config(configuration: Configuration, force=True)

Write files needed for the imsi shell sequencer, including the .simulation.time.state file, and the submission files for the model and diagnostics.

setup(configuration: Configuration, force=True)

Any steps needed to setup the sequencer, including cloning source and creating any directories

status(configuration: Configuration, setup_params: dict)

Steps needed to configure sequencer files based on updated imsi input. Examples might include editing resource files.

submit(configuration: Configuration)

Submits the job to the queue. Also checks that this command is being run from the location (machine) that was used for configuration.

imsi.sequencer_interface.iss_cap.create_job(run_id, job_name, script_dir, dates_file, user_script, directives=None, clean_scratch=True, depends_on=None, submit_next=True, submit_dependents=True, listings_basename=None)

Creates a SimpleJob object for a given configuration.

script_dir: str, Path

where the generated script file will go and be sourced from (make sure this is accesible by your system/scheduling system)

imsi.sequencer_interface.iss_cap.create_shell_sequencer(configuration: Configuration, scheduler: Scheduler = None)

Create a IMSISimpleSequencer instance from a Configuration and Scheduler

Utility functions

imsi.utils.general.change_dir(path: Path) → Generator

Temporarily changes the working context to the given path.

imsi.utils.general.delete_or_abort(path)

Asks a user for input to abort or delete and replace an existing directory.

imsi.utils.general.get_active_venv()

imsi.utils.general.get_date_string()

Return a datestring of now time

imsi.utils.general.get_pip_freeze(exe=None)

Return a list of currently installed packages and their exact versions

imsi.utils.general.is_broken_symlink(path)

Return True if path provided is a broken symlink

imsi.utils.general.is_path(var: str) → bool

Determines if the given string represents a path or a base filename.

Args:

var (str): The string to check, which may be a base filename or a path.

Returns:

bool: True if var includes a directory component, indicating it’s a path;

False if it’s only a base filename.

Example:

is_path(“woo”) => False is_path(“boo/woo”) => True

imsi.utils.general.is_root_of(base_path: Path, target_path: Path, greedy: bool = False) → bool

Return True if base_path is a root of target_path. If the paths are the same and greedy=True, return True. Paths do not need to exist.

imsi.utils.general.list_dir_contents(directory: Path, extensions: list = None, keep_folders: bool = True) → list

Returns a sorted list of the directory contents.

extensions:

Include only files with the extensions.

keep_folders:

Include folders in addition to the files from extensions.

imsi.utils.general.parse_memory_string_to_bytes(memory: str, base=2)

Parses a string that specifies memory value and unit and returns the value in bytes.

Parameters

memorya string composed of the value and units, where units

are denoted as KB, MB, or GB (case insensitive, with or without trailing ‘b’). Space is permitted between the value and units.

baseeither 2 or 10, denoting the conversion from the input

units to bytes. Use with caution. Default 2.

Returns

value of the information in bytes

Example >>> parse_memory_string_to_bytes(“1 kb”) 1024 >>> parse_memory_string_to_bytes(“10GB”) 10737418240

imsi.utils.general.remove_folder(path, force=False)

Removes a folder. Does not remove files.

Raises OSError if the folder is not empty, unless force=True (force has no effect if the folder is empty).

For folders, this mimics the difference between OS-level “rm <dir>” (force=False) vs “rm -rf <dir>” (force=True).

imsi.utils.general.write_shell_script(file_path: str, script_content: List[str], mode='w', make_executable=False)

Write shell script content to a file.

imsi.utils.general.write_shell_string(file_path: str, script_content: str, mode='w', make_executable=False)

Write string to shell script preserving newlines

imsi.utils.general.yes_or_no(question, compact=True)

Prompts a user if they want to proceed (y) or not (n) given the prompt (the question).

utils

Utility functions used in imsi, largely for parsing json, and updating, searching and modifying nested python dicts.

imsi.utils.dict_tools.combine_json_configs(rootpath)

combine all json files found recursively under rootpath into one dictionary. This effectively provides a dictionary-database to use.

Parameters:

rootpathstr

The path to recursively search for input files ending in .json/.jsonc. Normally this is the path to the imsi-config directory.

Outputs:

config: dict

A dictionary of the contents of the json or combined json files found under rootpath.

imsi.utils.dict_tools.combine_yaml_configs(rootpath, exclude_directories: list = ['options', 'templates'])

Combine all YAML files found recursively under rootpath into one dictionary, excluding anything under an options/ subdirectory.

imsi.utils.dict_tools.dump_config_to_stream(data: dict, file_type: str, stream: str | PathLike | TextIO) → None

Write the config file contents to the stream specified

imsi.utils.dict_tools.dump_config_to_string(data: dict, file_type: str) → None

Dump the configuration data to the desired stream (default stdout)

imsi.utils.dict_tools.flatten(d, parent_key='', sep='_')

Flatten a nested dict, using sep to join keys from levels in order

imsi.utils.dict_tools.get_config_extension(filename: str | Path) → str

Return the filename extension of the configuration file, one of .yaml, .json, or .jsonc. Raises ValueError if .yml or other extension.

imsi.utils.dict_tools.load_config_file(config_file: str | Path) → None

Reads a config file using the appropriate function for the file type (supports yaml or json)

imsi.utils.dict_tools.load_json(config_file)

Reads a json config file

imsi.utils.dict_tools.load_yaml(config_file)

Reads a yaml config file

imsi.utils.dict_tools.nested_get(d: dict, keys: list, strict=True, default=None)

Get the values inside a nested dict given a list of keys

imsi.utils.dict_tools.parse_config_inheritance(configs, selected_config)

Parse a configuration, inheriting attributes from all “parents”.

Parameters:

configs: dict

dictionary of all possible configurations, typically from an imsi_database.

selected_config: str

The configuration to choose

config_hierarchy: list

Typically not user specified, but used in the recursive function call to layer configurations in the correct order of inheritance.

imsi.utils.dict_tools.parse_config_inheritance_org(configs, selected_config, config_hierachy=None)

Parse a configuration recursively inheriting attributes from all “parents”.

Parameters:

configs: dict

dictionary of all possible configurations, typically from an imsi_database.

selected_config: str

The configuration to choose

config_hierachy: dict

Typically not user specified, but used in the recursive function call to layer configurations in the correct order of inheritance.

imsi.utils.dict_tools.parse_var(s)

Parse a key, value pair, separated by ‘=’ That’s the reverse of ShellArgs. On the command line (argparse) a declaration will typically look like:

foo=hello

or:

foo="hello world"

Courtesy https://stackoverflow.com/questions/27146262/create-variable-key-value-pairs-with-argparse-python

imsi.utils.dict_tools.parse_vars(items, none_as_str=True)

Parse a series of key-value pairs and return a dictionary Courtesy https://stackoverflow.com/questions/27146262/create-variable-key-value-pairs-with-argparse-python

imsi.utils.dict_tools.print_dict_as_config(data, format: str) → None

Print the config to stdout

imsi.utils.dict_tools.recursive_lookup(key, d)

Find a unique key in a nested dict. Will not be graceful if there are duplicates!

imsi.utils.dict_tools.replace_curlies_in_dict(input_dict, replacement_dict, show_warnings=True, fill=None)

imsi.utils.dict_tools.replace_variables(obj, inputs)

imsi.utils.dict_tools.replace_variables_in_string(s, inputs)

imsi.utils.dict_tools.resolve_inheritance(configs, selected_config, config_hierarchy)

Resolve the inheritance tree through recursion and exclusion of duplicates. Ensure ancestors precede descendents.

Parameters:

configs: dict

dictionary of all possible configurations, typically from an imsi_database.

selected_config: str

The configuration to choose

config_hierarchy: list

Typically not user specified, but used in the recursive function call to layer configurations in the correct order of inheritance.

imsi.utils.dict_tools.save_config_data(data: dict, filename: str | Path) → None

Saves data to JSON or YAML, determined by extension of filename.

imsi.utils.dict_tools.update(d, u, verbose=False)

Recursively update a dictionary, d, with an update, u. If an item appears in only one dictionary, it is included in the result.

Parameters:

d : dict to update u : dict of updates

Returns:

d : updated dict

nml_tools

This is a module to collect common namelist functions.

This currently exists because neither f90nml nor the rpn nml tools correctly parse the nemo namelists. Ideally all these functions would be replaced with more robust, community packages.

NCS, 10/2021

imsi.utils.nml_tools.cpp_update(cpp_input_file, cpp_output_file, cpp_changes, verbose=False)

An interface function to update cpp keys.

Inputs:

cpp_input_filestr

path to the cpp file to load in

cpp_output_filestr

path to the updated cpp file to write

cpp_changesdict

key = value pairs to change in the file, where key appears in the default cpp file, and value is the value to replace it with

This is a basic python implementation of the mod_nl routine. Ideally the work would be done by a standard fortran namelist parser, such as f90nml. However, no existing parsers work correctly on NEMO namelists.

imsi.utils.nml_tools.nml_read(nml_input_file)

An interface function to read namelists into a dict.

Inputs:

nml_input_filestr

path to the namelist file to read

Ideally the work would be done by a standard fortran namelist parser, such as f90nml. However, no existing parsers work correctly on NEMO namelists.

imsi.utils.nml_tools.nml_update(nml_input_file, nml_output_file, nml_changes)

An interface function to update namelist parameters.

Inputs:

nml_input_filestr

path to the namelist file to load in

nml_output_filestr

path to the updated namelist file to write

nml_changesdict

A nested dict containing. At the top level the keys are the names of the namelists in nml_input_file, the values are dicts containing key = value pairs to change in the namelist.

This is a basic python implementation of the mod_nl routine. Ideally the work would be done by a standard fortran namelist parser, such as f90nml. However, no existing parsers work correctly on NEMO namelists.

imsi.utils.nml_tools.nml_write(nml_output_file, nml)

An interface function to write namelists from a dict.

Inputs:

nml_output_filestr

path to the namelist file to write

nmldict

A nested dict containing. At the top level the keys are the names of the namelists, the values are dicts containing key = value pairs of the namelist.

Ideally the work would be done by a standard fortran namelist parser, such as f90nml. However, no existing parsers work correctly on NEMO namelists.

imsi.utils.nml_tools.update_env_file(infile: str, outfile: str = None, updates: dict = None, commment_char: str = '#', key_value_only: bool = True)

Update simple shell/environment files.

Only replaces values of existing keys, i.e. does not add key-value pairs that are not already in the file.

If key_value_only is True, then only lines that are in the key=value pattern (and commented lines) will be written to the output file. To keep all lines, set key_value_only=False.

git_tools

This is a module to collect common version control operations.

It might be worth checking out e.g. https://gitpython.readthedocs.io/en/stable/

However, for now to reduce dependencies, and given the basic operations needed, I’m just implementing a few functions here.

NCS, 10/2021

exception imsi.utils.git_tools.GitException(message, *args)

Bases: Exception

Exceptions from running git commands

imsi.utils.git_tools.clear_repo(path)

imsi.utils.git_tools.clone(repo_url, ver, local_name=None, path=None, depth: int | None = None, quiet=True) → str

Clone a git repository, for standard repositories or containing submodules. If there are submodules, clone the repository recursively, and checkout a specified version across all submodules.

Inputs:

repo_urlstr

The address to clone from, typically a url.

verstr

The reference to checkout. Can be a branch name or tag (used with the –branch arg in git clone). When a SHA-1 is used, a full clone of the repo is done and then checked out to the SHA-1 (depth is ignored).

local_namestr, None

The name to use for the on disk code. If None, uses the name of the repository.

pathstr, None

Path of where to clone the repo, path/local_name. If None, cwd used.

depthint, None

Depth to which the repository is cloned, including submodules. If None, the full git history is cloned. Can’t be used when ver is a SHA1

Returns:

path: str

Full path to where repo is cloned (includes repo name, equal to local_name if provided)

imsi.utils.git_tools.ensure_git_config(use_fallbacks=True, path=None)

ensures that git config for local repo is set if there isn’t a global one. (necessary for a commit).

If use_fallbacks is True, fallbacks are (order of application if exists):

local git config global git config username, no email

imsi.utils.git_tools.get_head_hash(path=None)

imsi.utils.git_tools.get_repo_name(repo_url: str) → str

imsi.utils.git_tools.git_add_all(path=None)

imsi.utils.git_tools.git_add_commit(path=None, msg='commit')

imsi.utils.git_tools.init_repo(path=None, branch=None)

imsi.utils.git_tools.is_git_repo(path)

imsi.utils.git_tools.is_repo_clean(path=None)

imsi.utils.git_tools.is_sha1(s)

Return True if string is like SHA-1 (7 to 40 hex characters)

imsi.utils.git_tools.repo_has_commits(path=None)