owmeta_core.bundle package

class owmeta_core.bundle.AccessorConfig

Bases: object

Configuration for accessing a Remote. Loaders are added to a remote according to which accessors are available

class owmeta_core.bundle.Bundle(ident, bundles_directory='~/.owmeta/bundles', version=None, conf=None, remotes=None, remotes_directory='~/.owmeta/remotes', transaction_manager=None)

Bases: object

Main entry point for using bundles

Typical usage is something like this:

>>> with Bundle('example/bundleId', version=42) as bnd:
...     for aDataObject in bnd(DataObject)().load():
...         # Do something with `aDataObject`
...         print(aDataObject)
DataObject(<http://example.org/entities#aDataObject>)

Note

Paths, bundles_directory and remotes_directory, will have symbolic links, environment variables, and “~” (for the current user’s home directory) expanded when the Bundle is initialized. To reflect changes to symbolic links or home directories, the bundles_directory or remotes_directory attributes must be updated directly or a new instance must be created.

Parameters

identstr

Bundle ID

bundles_directorystr, optional

Path to the bundles directory. Defaults to DEFAULT_BUNDLES_DIRECTORY

versionint, optional

Bundle version to access. By default, the latest version will be used.

confConfiguration or dict, optional

Configuration to add to the one created for the bundle automatically. Values for the default imports context (IMPORTS_CONTEXT_KEY), the default context (DEFAULT_CONTEXT_KEY) and store ('rdf.store', 'rdf.source', and, 'rdf.store_conf') will be ignored and overwritten.

remotesiterable of Remote or str, optional

A subset of remotes and additional remotes to fetch from. See Fetcher.fetch

remotes_directorystr, optional

The directory to load Remotes from in case a bundle is not in the bundle cache. Defaults to DEFAULT_REMOTES_DIRECTORY

transaction_managertransaction.TransactionManager, optional

Transaction manager

initdb()

Initialize the bundle’s conf Data instance

load_dependencies()

Load direct dependencies of this bundle

Yields

Bundle

A direct dependency of this bundle

load_dependencies_transitive()

Load dependencies from this bundle transitively

Yields

Bundle

A direct or indirect dependency of this bundle

connection

The owmeta_core connection to the bundle’s indexed database

property contexts

List of str. Context IDs in this bundle

class owmeta_core.bundle.BundleDependencyManager(dependencies, **common_bundle_arguments)

Bases: object

Finds the bundle in which a context is defined.

For a given bundle graph, that there is one Bundle that “owns” a given context. Although multiple bundles may provide that context, the one closest to the root of the graph which provides some statements in that context is called the owner. Note that this does not mean that bundles on which the owner depends do not also be queried; however, the exact behavior is up to the component that uses this component.

Parameters

dependenciesfunction

Function that returns a sequence of dependency descriptors

load_dependencies_transitive()

Load dependencies from this bundle transitively.

Any given version of a bundle will be yielded at most once regardless of how many times that version of the bundle appears in the dependency graph. Dependencies will yielded in topological sort order, so every dependency a Bundle declares will be yielded before any of its transitive dependencies.

Yields

Bundle

A direct or indirect dependency of this bundle

class owmeta_core.bundle.BundleDependentStoreConfigBuilder(bundles_directory=None, remotes_directory=None, remotes=None, read_only=True, transaction_manager=None)

Bases: object

Builds an RDFLib store configuration that depends on bundles.

The process of building the store configuration requires traversing the graph of dependencies so that duplicate dependencies in the graph can be omitted. To support this process, this builder will fetch bundles as needed to resolve transitive dependencies

build(indexed_db_path, dependencies, bundle_directory=None)

Builds the store configuration

Parameters

indexed_db_pathstr

Path to the indexed database of the store that depends on the listed dependenices

dependencieslist of dict

List of dependencies’ info, each entry including at least keys for ‘id’ and ‘version’

bundle_directorystr, optional

Path to the bundle directory for the dependent store, if the dependent store is a bundle. Used for information in an exceptional path, but not otherwise used

Returns

str

The type of the store. This is the name used to look up the RDFLib store plugin

object

The configuration for the store. This is the object that will be passed to rdflib.store.Store.open to configure the store.

class owmeta_core.bundle.BundleTransactionManager(explicit=False)

Bases: TransactionManager

Marker class useful in debugging to identify which txn manager we’re using

class owmeta_core.bundle.Cache(bundles_directory)

Bases: object

Cache of bundles

Parameters

bundles_directorystr

The where bundles are stored

list()

Returns a generator of summary bundle info

class owmeta_core.bundle.Deployer(remotes=(), **kwargs)

Bases: _RemoteHandlerMixin

Deploys bundles to Remotes.

A deployer takes a bundle directory tree or bundle archive and uploads it to a remote. Fetcher is, functionally, the dual of this class.

Deployer is responsible for selecting remotes and corresponding uploaders among a set of options. Uploaders are responsible for actually doing the upload.

deploy(bundle_path, remotes=None)

Deploy a bundle to all remotes that are configured to accept uploads

Parameters

bundle_pathstr

Path to a bundle directory tree or archive

remotesiterable of Remote or str

A subset of remotes to deploy to and additional remotes to deploy to

Raises

NoAcceptableUploaders

Thrown when none of the selected uploaders could upload the bundle

class owmeta_core.bundle.Descriptor(ident, **kwargs)

Bases: object

Descriptor for a bundle.

The descriptor is sufficient to build a distributable bundle directory tree from a ConjunctiveGraph and a set of files (see Installer).

dump(output)

Save a descriptor to a file as a YAML record

Parameters

outputfile object

The file to save the descriptor to

classmethod load(descriptor_source)

Load a descriptor from a YAML record

Parameters

descriptor_sourcestr or file object

The descriptor source. Handled by yaml.safe_load

Raises

NotADescriptor

Thrown when the object loaded from descriptor_source isn’t a dict

classmethod make(obj)

Makes a descriptor from the given object.

Parameters

obja dict-like object

An object with parameters for the Descriptor. Typically a dict

Returns

Descriptor

The created descriptor

class owmeta_core.bundle.Fetcher(bundles_root, remotes, transaction_manager=None, **kwargs)

Bases: _RemoteHandlerMixin

Fetches bundles from Remotes

A fetcher takes a list of remotes, a bundle ID, and, optionally, a version number and downloads the bundle to a local directory. Deployer is, functionally, the dual of this class.

Parameters

bundles_rootstr

The root directory of the bundle cache

remoteslist of Remote or str

List of pre-configured remotes used in calls to fetch

transaction_managertransaction.TransactionManager

Transaction manager to use when populating the indexed database after fetching

fetch(bundle_id, bundle_version=None, remotes=None, progress_reporter=None, triples_progress_reporter=None)

Retrieve a bundle by name from a remote and put it in the local bundle cache.

The first remote that can retrieve the bundle will be tried. Each remote will be tried in succession until one downloads the bundle.

Parameters

bundle_idstr

The id of the bundle to retrieve

bundle_versionint, optional

The version of the bundle to retrieve. If not provided, attempt to fetch the latest version available

remotesiterable of Remote or str

A subset of remotes and additional remotes to fetch from. If an entry in the iterable is a string, then it will be looked for amongst the remotes passed in initially.

progress_reportertqdm.tqdm-like object, optional

Receives updates of progress in fetching and installing locally

triples_progress_reportertqdm.tqdm-like object, optional

Receives updates of progress for adding triples for an individual graph

Returns

str

returns the directory where the bundle has been placed

Raises

exceptions.NoBundleLoader

Thrown when none of the loaders are able to download the bundle

FetchTargetIsNotEmpty

Thrown when the requested bundle is already in the cache

class owmeta_core.bundle.FilesDescriptor

Bases: object

Descriptor for files

class owmeta_core.bundle.Installer(source_directory, bundles_directory, graph, imports_ctx=None, default_ctx=None, class_registry_ctx=None, installer_id=None, remotes=(), remotes_directory=None)

Bases: object

Installs a bundle locally

Parameters

source_directorystr

Directory where files come from. All files for a bundle must be below this directory

bundles_directorystr

Directory where the bundles files go. Usually this is the bundle cache directory

graphrdflib.graph.ConjunctiveGraph

The graph from which we source contexts for this bundle

default_ctxstr, optional

The ID of the default context – the target of a query when not otherwise specified.

imports_ctxstr, optional

The ID of the imports context this installer should use. Imports relationships are selected from this graph according to the included contexts.

class_registry_ctxstr, optional

The ID of the class registry context this installer should use. Class registry entries are retrieved from this graph.

installer_iditerable of Remote or str, optional

Name of this installer for purposes of mutual exclusion

remotesiterable of Remote, optional

Remotes to be used for retrieving dependencies when needed during installation. If not provided, the remotes will be collected from remotes_directory

remotes_directorystr, optional

The directory to load Remotes from in case a bundle is not in the bundle cache. Defaults to DEFAULT_REMOTES_DIRECTORY

install(descriptor, progress_reporter=None)

Given a descriptor, install a bundle

Parameters

descriptorDescriptor

The descriptor for the bundle

progress_reportertqdm.tqdm-like object

Used for reporting progress during installation. optional

Returns

str

The directory where the bundle is installed

Raises

TargetIsNotEmpty

Thrown when the target directory for installation is not empty.

class owmeta_core.bundle.Remote(name, accessor_configs=())

Bases: object

A place where bundles come from and go to

Parameters

namestr

The name of the remote

accessor_configsiterable of AccessorConfig

Configs for how you access the remote

add_config(accessor_config)

Add the given accessor config to this remote

Parameters

accessor_configAccessorConfig

The config to add

Returns

bool

True if the accessor config was added (meaning there’s no equivalent one already set for this remote). Otherwise, False.

generate_loaders()

Generate the bundle loaders for this remote.

Loaders are generated from accessor_configs and LOADER_CLASSES according with which type of Loader can load a type of accessor

generate_uploaders()

Generate the bundle uploaders for this remote

classmethod read(inp)

Read a serialized Remote

Parameters

inpfile object

File-like object containing the serialized Remote

write(out)

Serialize the Remote and write to out

Parameters

outfile object

Target for writing the remote

accessor_configs

Configs for how you access the remote.

One might configure mirrors or replicas for a given bundle repository as multiple accessor configs

file_name

If read from a file, the remote should have this attribute set to its source file’s path

name

Name of the remote

class owmeta_core.bundle.URLConfig(url)

Bases: AccessorConfig

Configuration for accessing a remote with just a URL.

Note that URLConfigs should be pickle-able since they are written to a YAML file as part of the Remote they’re apart of.

owmeta_core.bundle.build_indexed_database(dest, bundle_directory, transaction_manager, progress=None, trip_prog=None)

Build the indexed database from a bundle directory

owmeta_core.bundle.retrieve_remotes(remotes_dir, load_entry_points=True)

Retrieve remotes from a project directory or user remotes directory

Parameters

owmdirstr

path to the project directory

load_entry_pointsbool, optional

if True, then the entry points for Loader and Uploader implementations that have been added as entry points

owmeta_core.bundle.DEFAULT_BUNDLES_DIRECTORY = '~/.owmeta/bundles'

Default directory for the bundle cache

owmeta_core.bundle.DEFAULT_REMOTES_DIRECTORY = '~/.owmeta/remotes'

Default directory for descriptors of user-level remotes as opposed to project-specific remotes

owmeta_core.bundle.URL_CONFIG_MAP = {'file': <class 'owmeta_core.bundle.loaders.local.FileURLConfig'>, 'http': <class 'owmeta_core.bundle.loaders.http.HTTPURLConfig'>, 'https': <class 'owmeta_core.bundle.loaders.http.HTTPSURLConfig'>}

URLConfigs by scheme. Can be populated by pkg_resources entry points

Subpackages

• owmeta_core.bundle.loaders package
  • LoadFailed
  • Loader
  • Uploader
  • Submodules

Submodules

• owmeta_core.bundle.archive module
  • ArchiveTargetPathDoesNotExist
  • TargetDirectoryMismatch
  • UnarchiveFailed
  • ArchiveExtractor
  • Archiver
  • Unarchiver
  • ensure_archive()
• owmeta_core.bundle.common module
  • bundle_tree_filter()
  • fmt_bundle_directory()
  • validate_manifest()
  • BUNDLE_ARCHIVE_MIME_TYPE
  • BUNDLE_INDEXED_DB_NAME
  • BUNDLE_MANIFEST_FILE_NAME
  • BUNDLE_MANIFEST_VERSION
• owmeta_core.bundle.exceptions module
  • BundleNotFound
  • CircularDependencyDetected
  • DeployFailed
  • FetchFailed
  • FetchTargetIsNotEmpty
  • InstallFailed
  • MalformedBundle
  • NoAcceptableUploaders
  • NoBundleLoader
  • NoRemoteAvailable
  • NotABundlePath
  • NotADescriptor
  • TargetIsNotEmpty
  • UncoveredImports