Object graph import/export framework for Django
Free software: MIT license
Experimental: active in production, but the API is subject to change.
Haul allows you to add model export/import functionality to your Django app. It can export some or all objects out of the Django ORM, store them in a file or a stream, and later import them back into the same or a different database / application instance.
When importing into a different database, you can customize how the imported objects are mapped against existing objects in the DB, and define what gets overwritten and what gets created anew.
Automatically follows FK and M2M references
Flexible serialization behaviours
Flexible object relinking on import
File attachments support
Compressed and plaintext formats
pip install django-haul
Consider following models:
from django.db import models class Tag(models.Model): name = models.CharField(max_length=32) class Author(models.Model): name = models.CharField(max_length=100) tags = models.ManyToManyField(Tag) class Book(models.Model): author = models.ForeignKey(Author, on_delete=models.CASCADE, related_name='books') coauthor = models.ForeignKey(Author, on_delete=models.SET_NULL, null=True) name = models.CharField(max_length=100) isbn = models.CharField(max_length=100, null=True) tags = models.ManyToManyField(Tag)
To expose models to Haul, you need to define
from haul import Exporter, ForeignKey, ReverseForeignKey, ManyToMany class TagExporter(Exporter): class Meta: fields = '__all__' model = Tag class BookExporter(Exporter): author = ForeignKey() coauthor = ForeignKey(allow_null=True) tags = ManyToMany() class Meta: fields = '__all__' model = Book class AuthorExporter(Exporter): books = ReverseForeignKey() tags = ManyToMany() class Meta: fields = '__all__' model = Author
Exporter base class is based on Django REST Framework’s own
ModelSerializer and will auto-discover non-relational fields.
Now, to export all books into a file, you can use:
EXPORTERS = [BookExporter, AuthorExporter, TagExporter] with open('export.haul', 'wb') as f: c = ExportContainer(exporters=EXPORTERS) c.export_objects(Book.objects.all()) c.write(f)
The output file will contain an object graph dump:
--- _: header metadata: null object_kinds: - test_app:book - test_app:author version: 1 --- _: object attachments:  data: !!omap - books: - !<ID> kind: test_app:book pk: 1 - tags:  - name: '1' id: !<ID> kind: test_app:author pk: 1 --- _: object attachments:  data: !!omap - books:  - tags:  - name: '2' id: !<ID> kind: test_app:author pk: 2 --- _: object attachments:  data: !!omap - author: !<ID> kind: test_app:author pk: 1 - coauthor: null - tags:  - name: b1 - isbn: null id: !<ID> kind: test_app:book pk: 1
You can also inspect the objects within the files with the CLI dump tool:
python -m haul.cli export.haul
Note how the
Author objects related to the
Book instances got picked up and exported automatically.
To import this data back into the database, you can simply feed it to an
from haul import ImportContainer c = ImportContainer(exporters=EXPORTERS) with open('export.haul', 'rb') as f: with c.read(f): c.import_objects()
This, however, will unconditionally create new objects, even if books and authors with the same names already exist.
You can flexibly define how Haul should treat existing and duplicate objects. For example, let’s prevent duplicate authors from being imported, but keep duplicate books and link them to the already existing authors:
from haul import ImportPolicy, RelinkAction class BookImportPolicy(ImportPolicy): def relink_object(self, model_cls, obj): if model_cls is Book: # Unconditionally import as a new object return RelinkAction.Create() if model_cls is Author: return RelinkAction.LinkByFields( # Look up authors by their names lookup_fields=('name',), # Fall back to creation if not found fallback=RelinkAction.Create(), ) # Do not import other object types return RelinkAction.Discard() c = ImportContainer(exporters=EXPORTERS, policy=BookImportPolicy()) with open('export.haul', 'rb') as f: with c.read(d): c.import_objects()
haul.policy for other relink actions.