haul package

Subpackages

• haul.containers package
  • Submodules
  • haul.containers.base module
  • Module contents

Submodules

haul.cli module

A nifty CLI tool to dump an exported file’s contents for debugging.

Call it like this:

python -m haul.cli /path/to/file.haul

haul.cli.main()

haul.errors module

exception haul.errors.ModelClassNotRegistered

Bases: Exception

haul.policy module

class haul.policy.BaseRelinkAction(fallback: Optional[haul.policy.BaseRelinkAction] = None)

Bases: object

Base class for new relink actions.

Override execute() to return a list of relinked model instances, None or a literal False to discard the object.

execute(model_cls: Type[django.db.models.base.Model], objects: List[haul.types.ObjectData], policy: haul.policy.ImportPolicy) → Sequence[Optional[Union[django.db.models.base.Model, typing_extensions.Literal[False]]]]

fallback: Optional[haul.policy.BaseRelinkAction] = None

class haul.policy.BaseRelinkActionWithFieldsOverwrite(fallback: Optional[haul.policy.BaseRelinkAction] = None, overwrite_fields: Tuple[str, ...] = ())

Bases: haul.policy.BaseRelinkAction

A base relink action class that automatically overwrites model fields set via overwrite_fields.

overwrite_fields: Tuple[str, ...] = ()

class haul.policy.ExportPolicy

Bases: object

A behaviour class that defines how to handle objects during export.

get_attachments(instance: django.db.models.base.Model) → List[haul.types.Attachment]

Override to add file or stream based attachments to an object. See Attachment.

should_export_object(instance: django.db.models.base.Model)

Override to additionally filter objects during export.

should_follow_reference(instance: django.db.models.base.Model, target: haul.types.ID, field: str)

Override to additionally filter FK references during export.

class haul.policy.ImportPolicy

Bases: object

A behaviour class that defines how to handle objects during import. Override its methods and pass it to ImportContainer to customize the behaviour.

post_object_import(instance: django.db.models.base.Model)

postprocess_object_fields(model_cls: Type[django.db.models.base.Model], fields: Dict[str, Any])

Called after relink_object() and before the relink actions have been applied. You can modify the fields dict in place here.

preprocess_object_fields(model_cls: Type[django.db.models.base.Model], fields: Dict[str, Any])

Called after objects have been deserialized. You can modify the fields dict in place here.

process_attachment(instance: django.db.models.base.Model, key: Any, stream: IO[bytes])

Called for each attachment on a model. This is where you read and process the attachment contents. Note that the stream is not guaranteed to be open after this method exits and is guaranteed to be closed once the ImportContainer.import() context exits.

process_many_to_many(instance: django.db.models.base.Model, field: str) → haul.policy.ManyToManyImportAction

Called for each M2M relation on a model. You can either return a ManyToManyImportAction.REPLACE or ManyToManyImportAction.APPEND here. The default is REPLACE.

relink_object(model_cls: Type[django.db.models.base.Model], obj: haul.types.ObjectData) → haul.policy.BaseRelinkAction

Called to decide how the imported objects should be mapped onto existing objects. This method should return an instance of a BaseRelinkAction subclass.

The default implementation just returns RelinkAction.Create()

class haul.policy.ManyToManyImportAction(value)

Bases: enum.Enum

An enumeration.

APPEND = 2

REPLACE = 1

class haul.policy.RelinkAction

Bases: object

class Create(fallback: Optional[haul.policy.BaseRelinkAction] = None, ignore_fields: Tuple[str, ...] = <factory>)

Bases: haul.policy.BaseRelinkAction

A relink action that unconditionally creates new objects.

execute(model_cls: Type[django.db.models.base.Model], objects: List[haul.types.ObjectData], policy: haul.policy.ImportPolicy) → List[Union[django.db.models.base.Model, typing_extensions.Literal[False]]]

ignore_fields: Tuple[str, ...]

A list of fields to ignore during import

class Discard(fallback: Optional[haul.policy.BaseRelinkAction] = None)

Bases: haul.policy.BaseRelinkAction

A relink action that unconditionally discards all objects.

execute(_model_cls, objects: List[haul.types.ObjectData], policy: haul.policy.ImportPolicy) → Sequence[Optional[Union[django.db.models.base.Model, typing_extensions.Literal[False]]]]

class LinkByFields(fallback: Optional[haul.policy.BaseRelinkAction] = None, overwrite_fields: Tuple[str, ...] = (), lookup_fields: Tuple[str, ...] = <factory>)

Bases: haul.policy.BaseRelinkActionWithFieldsOverwrite

A relink action that tries to look up existing objects by a set of fields.

execute(model_cls: Type[django.db.models.base.Model], objects: List[haul.types.ObjectData], policy: haul.policy.ImportPolicy) → Sequence[Optional[Union[django.db.models.base.Model, typing_extensions.Literal[False]]]]

lookup_fields: Tuple[str, ...]

A list of fields to use to look up existing objects

class LinkByPK(fallback: Optional[haul.policy.BaseRelinkAction] = None, overwrite_fields: Tuple[str, ...] = ())

Bases: haul.policy.BaseRelinkActionWithFieldsOverwrite

A relink action that tries to map objects by their literal PK value.

This only works if the PKs are same in the source and destination databases.

execute(model_cls: Type[django.db.models.base.Model], objects: List[haul.types.ObjectData], policy: haul.policy.ImportPolicy) → List[Optional[Union[django.db.models.base.Model, typing_extensions.Literal[False]]]]

class LinkToInstance(fallback: Optional[haul.policy.BaseRelinkAction] = None, overwrite_fields: Tuple[str, ...] = (), pk: Optional[Hashable] = None)

Bases: haul.policy.BaseRelinkActionWithFieldsOverwrite

A relink action that links the object to a specific existing object.

execute(model_cls: Type[django.db.models.base.Model], objects: List[haul.types.ObjectData], policy: haul.policy.ImportPolicy) → Sequence[Union[django.db.models.base.Model, typing_extensions.Literal[False]]]

pk: Hashable = None

PK of the existing object to link to

class LogToConsole(fallback: Optional[haul.policy.BaseRelinkAction] = None)

Bases: haul.policy.BaseRelinkAction

A relink action that prints the objects and links them to None.

execute(model_cls: Type[django.db.models.base.Model], objects: List[haul.types.ObjectData], policy: haul.policy.ImportPolicy) → Sequence[Optional[Union[django.db.models.base.Model, typing_extensions.Literal[False]]]]

haul.serializers module

Haul uses DRF ModelSerializer as a base for its serialization.

This means you can use custom Field and Serializer objects to serialize and deserialize fields.

class haul.serializers.DummyField(*args, **kwargs)

Bases: rest_framework.fields.Field

get_attribute(instance)

Given the outgoing object instance, return the primitive value that should be used for this field.

run_validation(data=None)

Validate a simple representation and return the internal value.

The provided data may be empty if no representation was included in the input.

May raise SkipField if the field should not be included in the validated data.

to_internal_value(data)

Transform the incoming primitive data into a native value.

to_representation(value)

Transform the outgoing native value into primitive data.

class haul.serializers.Exporter(*args, **kwargs)

Bases: rest_framework.serializers.ModelSerializer

build_standard_field(field_name, model_field)

Create regular model fields.

serializer_related_field

alias of haul.serializers.DummyField

class haul.serializers.ForeignKey(*args, **kwargs)

Bases: rest_framework.fields.Field

bind(field_name, parent)

Initializes the field name and parent for the field instance. Called when a field is added to the parent serializer instance.

export_policy: haul.policy.ExportPolicy

get_attribute(instance)

Given the outgoing object instance, return the primitive value that should be used for this field.

to_internal_value(data)

Transform the incoming primitive data into a native value.

to_representation(value)

Transform the outgoing native value into primitive data.

class haul.serializers.ManyToMany(*args, **kwargs)

Bases: haul.serializers._BaseM2X

export_policy: haul.policy.ExportPolicy

weak = False

class haul.serializers.ReverseForeignKey(*args, **kwargs)

Bases: haul.serializers._BaseM2X

export_policy: haul.policy.ExportPolicy

weak = True

haul.types module

class haul.types.Attachment(id: str, key: Any, content_provider: Union[Callable[[], BinaryIO], NoneType] = None, _container_stream: Union[BinaryIO, NoneType] = None)

Bases: object

content_provider: Optional[Callable[[], BinaryIO]] = None

classmethod from_data(key: Any, data: bytes)

classmethod from_path(key: Any, path: str)

id: str

key: Any

classmethod to_yaml(representer, node)

class haul.types.ContainerFormat(value)

Bases: enum.Enum

An enumeration.

COMPRESSED_ZIP = 2

NON_COMPRESSED_ZIP = 3

YAML = 1

class haul.types.ID(kind: str, pk: Any)

Bases: object

classmethod deserialize(s: Any) → haul.types.ID

classmethod from_object(obj: django.db.models.base.Model) → haul.types.ID

classmethod from_yaml(loader, node)

kind: str

static kind_for_model(obj: Union[django.db.models.base.Model, Type[django.db.models.base.Model]])

static model_for_kind(kind: str) → Type[django.db.models.base.Model]

pk: Any

serialize()

classmethod to_yaml(representer, node)

yaml_tag = 'ID'

class haul.types.ObjectData(id: haul.types.ID, serialized_data: Dict[str, Any], fields: Union[Dict[str, Any], NoneType] = None, refs: List[haul.types.Ref] = <factory>, refers_to: Set[haul.types.ID] = <factory>, attachments: List[haul.types.Attachment] = <factory>)

Bases: object

add_reference(ref: haul.types.Ref)

attachments: List[haul.types.Attachment]

fields: Optional[Dict[str, Any]] = None

id: haul.types.ID

refers_to: Set[haul.types.ID]

refs: List[haul.types.Ref]

serialized_data: Dict[str, Any]

class haul.types.Ref(ids: List[haul.types.ID], field: str, nullable: bool = False, weak: bool = False)

Bases: object

field: str

ids: List[haul.types.ID]

nullable: bool = False

weak: bool = False

Module contents

class haul.Attachment(id: str, key: Any, content_provider: Union[Callable[[], BinaryIO], NoneType] = None, _container_stream: Union[BinaryIO, NoneType] = None)

Bases: object

content_provider: Optional[Callable[[], BinaryIO]] = None

classmethod from_data(key: Any, data: bytes)

classmethod from_path(key: Any, path: str)

id: str

key: Any

classmethod to_yaml(representer, node)

class haul.BaseRelinkAction(fallback: Optional[haul.policy.BaseRelinkAction] = None)

Bases: object

Base class for new relink actions.

Override execute() to return a list of relinked model instances, None or a literal False to discard the object.

execute(model_cls: Type[django.db.models.base.Model], objects: List[haul.types.ObjectData], policy: haul.policy.ImportPolicy) → Sequence[Optional[Union[django.db.models.base.Model, typing_extensions.Literal[False]]]]

fallback: Optional[haul.policy.BaseRelinkAction] = None

class haul.ContainerFormat(value)

Bases: enum.Enum

An enumeration.

COMPRESSED_ZIP = 2

NON_COMPRESSED_ZIP = 3

YAML = 1

class haul.ExportContainer(exporters: List[Type[haul.serializers.Exporter]] = [], policy: Optional[haul.policy.ExportPolicy] = None, ignore_unknown=False)

Bases: haul.containers.base.BaseContainer

Your starting point for object export.

export_objects(objects: Iterable[django.db.models.base.Model])

Serializes objects and adds them to the container.

iter_objects() → Iterable[haul.types.ObjectData]

write(stream: IO[bytes], format: haul.types.ContainerFormat = ContainerFormat.YAML, metadata: Optional[Any] = None)

Writes the serialized objects from the container into a data stream.

Parameters

• stream – The stream to write into. For ContainerFormat.ZIP_*, has to be seekable.

• metadata – a free-form metadata object that will be stored in the stream. It’s available later through ImportContainer.metadata.

class haul.ExportPolicy

Bases: object

A behaviour class that defines how to handle objects during export.

get_attachments(instance: django.db.models.base.Model) → List[haul.types.Attachment]

Override to add file or stream based attachments to an object. See Attachment.

should_export_object(instance: django.db.models.base.Model)

Override to additionally filter objects during export.

should_follow_reference(instance: django.db.models.base.Model, target: haul.types.ID, field: str)

Override to additionally filter FK references during export.

class haul.Exporter(*args, **kwargs)

Bases: rest_framework.serializers.ModelSerializer

build_standard_field(field_name, model_field)

Create regular model fields.

serializer_related_field

alias of haul.serializers.DummyField

class haul.ForeignKey(*args, **kwargs)

Bases: rest_framework.fields.Field

bind(field_name, parent)

Initializes the field name and parent for the field instance. Called when a field is added to the parent serializer instance.

export_policy: haul.policy.ExportPolicy

get_attribute(instance)

Given the outgoing object instance, return the primitive value that should be used for this field.

to_internal_value(data)

Transform the incoming primitive data into a native value.

to_representation(value)

Transform the outgoing native value into primitive data.

class haul.ID(kind: str, pk: Any)

Bases: object

classmethod deserialize(s: Any) → haul.types.ID

classmethod from_object(obj: django.db.models.base.Model) → haul.types.ID

classmethod from_yaml(loader, node)

kind: str

static kind_for_model(obj: Union[django.db.models.base.Model, Type[django.db.models.base.Model]])

static model_for_kind(kind: str) → Type[django.db.models.base.Model]

pk: Any

serialize()

classmethod to_yaml(representer, node)

yaml_tag = 'ID'

class haul.ImportContainer(exporters: List[Type[haul.serializers.Exporter]] = [], policy: Optional[haul.policy.ImportPolicy] = None, ignore_unknown=False)

Bases: haul.containers.base.BaseContainer

Your starting point for object import.

import_objects() → haul.containers._import.ImportReport

Untangles the object graph, relinks objects and imports them into the database.

metadata: Any = None

free-form metadata as stored by ExportContainer.write()

read(stream: BinaryIO)

Reads a data stream, deserializes objects in it and stores them inside the container.

This is a context manager which has to be kept open when import_objects() is called:

c = ImportContainer(exporters=...)
with open(...) as f:
    with c.read(f):
        c.import_objects()

class haul.ImportPolicy

Bases: object

A behaviour class that defines how to handle objects during import. Override its methods and pass it to ImportContainer to customize the behaviour.

post_object_import(instance: django.db.models.base.Model)

postprocess_object_fields(model_cls: Type[django.db.models.base.Model], fields: Dict[str, Any])

Called after relink_object() and before the relink actions have been applied. You can modify the fields dict in place here.

preprocess_object_fields(model_cls: Type[django.db.models.base.Model], fields: Dict[str, Any])

Called after objects have been deserialized. You can modify the fields dict in place here.

process_attachment(instance: django.db.models.base.Model, key: Any, stream: IO[bytes])

Called for each attachment on a model. This is where you read and process the attachment contents. Note that the stream is not guaranteed to be open after this method exits and is guaranteed to be closed once the ImportContainer.import() context exits.

process_many_to_many(instance: django.db.models.base.Model, field: str) → haul.policy.ManyToManyImportAction

Called for each M2M relation on a model. You can either return a ManyToManyImportAction.REPLACE or ManyToManyImportAction.APPEND here. The default is REPLACE.

relink_object(model_cls: Type[django.db.models.base.Model], obj: haul.types.ObjectData) → haul.policy.BaseRelinkAction

Called to decide how the imported objects should be mapped onto existing objects. This method should return an instance of a BaseRelinkAction subclass.

The default implementation just returns RelinkAction.Create()

class haul.ManyToMany(*args, **kwargs)

Bases: haul.serializers._BaseM2X

export_policy: haul.policy.ExportPolicy

weak = False

class haul.ManyToManyImportAction(value)

Bases: enum.Enum

An enumeration.

APPEND = 2

REPLACE = 1

exception haul.ModelClassNotRegistered

Bases: Exception

class haul.ObjectData(id: haul.types.ID, serialized_data: Dict[str, Any], fields: Union[Dict[str, Any], NoneType] = None, refs: List[haul.types.Ref] = <factory>, refers_to: Set[haul.types.ID] = <factory>, attachments: List[haul.types.Attachment] = <factory>)

Bases: object

add_reference(ref: haul.types.Ref)

attachments: List[haul.types.Attachment]

fields: Optional[Dict[str, Any]] = None

id: haul.types.ID

refers_to: Set[haul.types.ID]

refs: List[haul.types.Ref]

serialized_data: Dict[str, Any]

class haul.Ref(ids: List[haul.types.ID], field: str, nullable: bool = False, weak: bool = False)

Bases: object

field: str

ids: List[haul.types.ID]

nullable: bool = False

weak: bool = False

class haul.RelinkAction

Bases: object

class Create(fallback: Optional[haul.policy.BaseRelinkAction] = None, ignore_fields: Tuple[str, ...] = <factory>)

Bases: haul.policy.BaseRelinkAction

A relink action that unconditionally creates new objects.

execute(model_cls: Type[django.db.models.base.Model], objects: List[haul.types.ObjectData], policy: haul.policy.ImportPolicy) → List[Union[django.db.models.base.Model, typing_extensions.Literal[False]]]

ignore_fields: Tuple[str, ...]

A list of fields to ignore during import

class Discard(fallback: Optional[haul.policy.BaseRelinkAction] = None)

Bases: haul.policy.BaseRelinkAction

A relink action that unconditionally discards all objects.

execute(_model_cls, objects: List[haul.types.ObjectData], policy: haul.policy.ImportPolicy) → Sequence[Optional[Union[django.db.models.base.Model, typing_extensions.Literal[False]]]]

class LinkByFields(fallback: Optional[haul.policy.BaseRelinkAction] = None, overwrite_fields: Tuple[str, ...] = (), lookup_fields: Tuple[str, ...] = <factory>)

Bases: haul.policy.BaseRelinkActionWithFieldsOverwrite

A relink action that tries to look up existing objects by a set of fields.

execute(model_cls: Type[django.db.models.base.Model], objects: List[haul.types.ObjectData], policy: haul.policy.ImportPolicy) → Sequence[Optional[Union[django.db.models.base.Model, typing_extensions.Literal[False]]]]

lookup_fields: Tuple[str, ...]

A list of fields to use to look up existing objects

class LinkByPK(fallback: Optional[haul.policy.BaseRelinkAction] = None, overwrite_fields: Tuple[str, ...] = ())

Bases: haul.policy.BaseRelinkActionWithFieldsOverwrite

A relink action that tries to map objects by their literal PK value.

This only works if the PKs are same in the source and destination databases.

execute(model_cls: Type[django.db.models.base.Model], objects: List[haul.types.ObjectData], policy: haul.policy.ImportPolicy) → List[Optional[Union[django.db.models.base.Model, typing_extensions.Literal[False]]]]

class LinkToInstance(fallback: Optional[haul.policy.BaseRelinkAction] = None, overwrite_fields: Tuple[str, ...] = (), pk: Optional[Hashable] = None)

Bases: haul.policy.BaseRelinkActionWithFieldsOverwrite

A relink action that links the object to a specific existing object.

execute(model_cls: Type[django.db.models.base.Model], objects: List[haul.types.ObjectData], policy: haul.policy.ImportPolicy) → Sequence[Union[django.db.models.base.Model, typing_extensions.Literal[False]]]

pk: Hashable = None

PK of the existing object to link to

class LogToConsole(fallback: Optional[haul.policy.BaseRelinkAction] = None)

Bases: haul.policy.BaseRelinkAction

A relink action that prints the objects and links them to None.

execute(model_cls: Type[django.db.models.base.Model], objects: List[haul.types.ObjectData], policy: haul.policy.ImportPolicy) → Sequence[Optional[Union[django.db.models.base.Model, typing_extensions.Literal[False]]]]

class haul.ReverseForeignKey(*args, **kwargs)

Bases: haul.serializers._BaseM2X

export_policy: haul.policy.ExportPolicy

weak = True