EVOLUTION-MANAGER

Edit File: evolution.txt

================
Django Evolution
================

When you run ``./manage.py syncdb``, Django will look for any new models that
have been defined, and add a database table to represent those new models.
However, if you make a change to an existing model, ``./manage.py syncdb`` will
not make any changes to the database.

This is where **Django Evolution** fits in. Django Evolution is an extension
to Django that allows you to track changes in your models over time, and to
update the database to reflect those changes.

.. admonition:: Work in Progress

Django Evolution is a work in progress. The interface and usage described
    in this document is subject to change as we finesse the details. The SQL
    that is automatically generated may occasionally be wrong. There are some
    significant limitations to the capabilities of the system. In short,
    Django Evolution is not yet ready for a production rollout.

An example
==========

To demonstrate how Django Evolution works, let's work through the lifecycle of
a Django application - a new super-cool blogging application called
``Blogette``. After some initial design work, we come up with the following
models::

class Author(models.Model):
        name = models.CharField(max_length=50)
        email = models.EmailField()
        date_of_birth = models.DateField()

def __unicode__(self):
            return self.name

class Entry(models.Model):
        headline = models.CharField(max_length=255)
        body_text = models.TextField()
        pub_date = models.DateTimeField()
        author = models.ForeignKey(Author)

def __unicode__(self):
            return self.headline

We create a test project, create our ``blogette`` application directory, put
the model definitions in the models.py file, add ``blogette`` to
``settings.INSTALLED_APPS``, and run ``./manage.py syncdb``. This installs
the application, creating the Entry and Author tables.

The first changes
=================

But why do we want to track the date of birth of the Author? In retrospect,
this field was a mistake, so let's delete that field from the model.
However, if we just delete the field and run ``syncdb`` again, Django
won't do anything to respond to the change. We need to use a slightly
different process.

Before we start making changes to ``models.py``, we need to set up the
project to allow evolutions. To do this, we add ``django_evolution`` to the
``settings.INSTALLED_APPS`` and run ``./manage.py syncdb``. This sets up
the tables for tracking model changes::

$ ./manage.py syncdb
    Creating table django_evolution
    Installing baseline version for testproject.blogette
    Loading 'initial_data' fixtures...
    No fixtures found.

Now we can go into ``models.py`` remove the ``date_of_birth`` field.
After removing the field, ``./manage.py syncdb`` will provide a warning
that changes have been detected::

$ ./manage.py syncdb
    Project signature has changed - an evolution is required
    Loading 'initial_data' fixtures...
    No fixtures found.

The evolution process itself is controlled using a new management command --
**evolve**. This command is made available when you install the
``django_evolution`` application in your project.

If you want to know what has changed, you can use ask Django Evolution
to provide a hint::

$ ./manage.py evolve --hint
    #----- Evolution for blogette
    from django_evolution.mutations import *
    from django.db import models

MUTATIONS = [
         DeleteField('Author', 'date_of_birth')
    ]
    #----------------------
    Trial evolution successful.
    Run './manage.py evolve --execute' to apply evolution.

The output of the hint is a Django Evolution. An Evolution is python code
that defines a list of mutations that need to be made to update a model -
in this case, the fact that we need to delete the ``date_of_birth`` field
from the model ``Author``.

If you want to see what the SQL would look like for this evolution, you can
use the ``--sql`` option::

$ ./manage.py evolve --hint --sql
    ;; Compiled evolution SQL for blogette
    ALTER TABLE "blogette_person" DROP COLUMN "date_of_birth" CASCADE;

.. note::

The SQL output shown here is what would be produced by the Postgres 
        backend; if you are using a different backend, the SQL produced by 
        this step will be slightly different.
        
If we wanted to, we could pipe this SQL output directly into our database.
however, we don't need to - we can get Django Evolution to do this for us,
by passing the execute option to ``evolve``. When you do this, you will
be given a stern warning that you may destroy data::

$ ./manage.py evolve --hint --execute

You have requested a database evolution. This will alter tables
    and data currently in the 'blogette' database, and may result in
    IRREVERSABLE DATA LOSS. Evolutions should be *thoroughly* reviewed
    prior to execution.

Are you sure you want to execute the evolutions?

Type 'yes' to continue, or 'no' to cancel:

Assuming you answer ``yes``, Django Evolution should respond::

Evolution successful.

Now if we re-run syncdb, we find that there are no evolution warnings::

$ ./manage.py syncdb
    Loading 'initial_data' fixtures...
    No fixtures found.

If we were to inspect the database itself, we will find that the
``date_of_birth`` field has been deleted.

Stored evolutions
=================

At this point, we are happy with the model definitions. Once we have
developed the views and templates for Blogette, we can deploy Version 1
of Blogette on our production server. We copy our source code to the
production server, run syncdb, and the production server will have tables
that represent the latest version of our models.

Now we can start work on Blogette Version 2. For this release, we decide
to add a 'location' field to Authors, and a 'summary' field to entries.
This means we now will need to make changes to models.py. However, we
now have a production server with the Version 1 models deployed. This
means that we need to keep track of any changes we make during the
development process so that we can duplicate those changes on our
production server.

To do this, we make use of stored evolutions - a mechanism for defining
and reusing collections of evolutions for later use.

Mutation definitions
--------------------

Let's start with adding the ``location`` field to ``Authors``. First, we edit
``models.py`` to add a new CharField definition, with a length of 100.
We will also allow null values, since we have existing data that won't have
an author location. Add the following field definition to authors::

location = models.CharField(max_length=100, null=True)

After making this change to the Author model, we know an evolution will be
required; if we run ``evolve --hint`` we will get the exact change required::

$ ./manage.py evolve --hint
    #----- Evolution for blogette
    from django_evolution.mutations import *
    from django.db import models

MUTATIONS = [
         AddField('Author', 'location', models.CharField, max_length=100, null=True)
    ]
    #----------------------
    Trial evolution successful.
    Run './manage.py evolve --execute' to apply evolution.

At this point, we *could* just run ``evolve --hint --execute`` to update the
development server. However, we want to remember this change, so we need to store
it in a way that Django Evolution can use it.

To do this, we need to create an evolution store. In the blogette directory,
we create an ``evolutions`` module. In this module, we create two files -
``__init__.py``, and ``add_location.py``. The ``blogette`` application
directory should now look something like this::

/blogette
        /evolutions
            __init__.py
            add_location.py
        models.py
        views.py

``add_location.py`` is a file used to describe the evolution we want to
perform. The contents of this file is exactly the same as the
``evolve --hint`` command produced -- just copy the content between the
lines marked ``----`` into ``add_location.py``.

We then need to define an evolution sequence. This sequence defines the order
in which evolutions need to be applied. Since we only have one evolution,
the definition looks like this::

SEQUENCE = ['add_location']

Put this statement in ``__init__.py``, and you've have defined your first
stored evolution.

Now we can apply the evolution. We don't need a hint this time - we have
a stored evolution, so we can just run ``evolve``::

$ ./manage.py evolve
    #----- Evolution for blogette
    from django_evolution.mutations import *
    from django.db import models

This shows us the sequence of mutations described by our stored evolution
sequence. It also tells us that the trial evolution was successful - every
time you run ``evolve``, Django Evolution will simulate the changes to make
sure that the mutations that are defined will reconcile the differences
between the models.py file and the state of the database.

Since the simulation was successful, we can apply the evolution using the
``--execute`` flag::

$ ./manage.py evolve --execute
    ...
    Evolution successful.

.. note::

The warning and prompt for confirmation will be displayed every time
    you evolve your database. It is omitted for clarity in this and later
    examples. If you don't want to be prompted every time, use the
    ``--noinput`` option.

SQL mutations
-------------

Now we need to add the ``summary`` field to ``Entry``. We could follow the
same procedure - however, we going to do something a little different.
Rather than define a Python Evolution file, we're going to define our
mutation in raw SQL.

The process of adding a stored SQL evolution is very similar to adding a
stored Python evolution. In this case, we're going to call our mutation
``add_summary``, so we create a file called ``add_summary.sql`` to the
evolutions directory. Into this file, we put the SQL that will make the
change we require. In our case, this means::

ALTER TABLE blogette_entry ADD COLUMN summary varchar(100) NULL;

Then, we add the new evolution to the evolution sequence in
``evolutions/__init__.py``. The sequence should now look like this::

SEQUENCE = ['add_location', 'add_summary']

We have now defined our SQL mutation, and how it should be executed, so we
can trial the evolution::

$ ./manage.py evolve
    #----- Evolution for blogette
    from django_evolution.mutations import *
    from django.db import models

MUTATIONS = [
        SQLMutation('add_summary')
    ]
    #----------------------
    Evolution could not be simulated, possibly due to raw SQL mutations.

Unfortunately, Django Evolution can't simulate SQL mutations, so we can't be
sure that the mutation is correct. However, we can inspect the SQL that will
be used using the --sql option::

$ ./manage.py evolve --sql
    ;; Compiled evolution SQL for blogette
    ALTER TABLE blogette_entry ADD COLUMN summary varchar(100) NULL;

If we are satisfied that this SQL is correct, we can execute it::

$ ./manage.py evolve --execute
    ...
    Evolution successful.

Meanwhile, on the production site...
------------------------------------

Now that we have finished Blogette Version 2, we can update our existing
production server. We copy the Version 2 source code, including the evolutions,
to the production server. We then run ./manage.py syncdb, which reports that
evolutions are required::

$ ./manage.py syncdb
    Project signature has changed - an evolution is required
    There are unapplied evolutions for blogette.
    Loading 'initial_data' fixtures...
    No fixtures found.

If we run evolve, we can see the full sequence of mutations that will be
applied::

$ ./manage.py evolve
    #----- Evolution for blogette
    from django_evolution.mutations import *
    from django.db import models

MUTATIONS = [
        AddField('Author', 'location', models.CharField, max_length=100, null=True),
        SQLMutation('add_summary')
    ]
    #----------------------
    Evolution could not be simulated, possibly due to raw SQL mutations.

Again, since there is a raw SQL migration involved, we will need to validate
the migration ourselves using the ``--sql`` option::

$ ./manage.py evolve --sql
    ;; Compiled evolution SQL for testproject.blogette
    ALTER TABLE blogette_author ADD COLUMN location varchar(100) NULL;
    ALTER TABLE blogette_entry ADD COLUMN summary varchar(100) NULL;

If we are happy with this sequence, we can apply it::

$ ./manage.py evolve --execute
    ...
    Evolution successful.

Our production site now has a database that mirrors the changes made on
the development server.

Reference
=========

The Contract
------------

Django Evolution imposes one important restriction on your development process:
If you intend to make a change to your database, you **must** do it through
Django Evolution. If you modify the database outside of Django and the Django
Evolution framework, you're on your own.

The operation of Django Evolution is entirely based upon the observing and
storing changes to the models.py file. As a result, any changes to the database
will not be observed by Django Evolution, and won't be used in evaluating hints
or establishing if evolution simulations have been successful.

.. admonition:: Room for improvement

This is one area of Django Evolution that could be significantly improved.
    Adding database introspection to allow for external database modifications
    is on the plan for the future, but for the moment, stick to the contract.

Usage of the ``evolve`` command
-------------------------------

``./manage.py evolve`` does not require any arguments or options. When run 
without any options, ``./manage.py evolve`` will list the evolutions that
have been stored and are available, but have not yet been applied to the
database.

You may optionally provide the name of one or more applications as an argument
to evolve. This will restrict the output of the command to those applications
that are named.

You cannot specify application names if you use the ``--execute``
option. Evolutions cannot be executed on a per-application basis. They must
be applied across the entire project, or not at all.

The following options may also be used to alter the behavior of the ``evolve``
command.

--hint
~~~~~~

Provide a hinted list of mutations for migrating. If no application labels are
provided, hints for the entire project will be generated. If one or more
application names are provided, the hints provided will be restricted to those
applications.

May be combined with ``--sql`` to generate hinted mutations in SQL format.

May be combined with ``--execute`` to apply the changes to the database.

--sql
~~~~~

Convert an evolution from Python syntax to SQL syntax.

If ``--hint`` is specified, the hinted list of mutations will be converted. If
``--hint`` is not specified, the generated SQL will be for any stored
evolutions that have not been applied.

--execute (-x)
~~~~~~~~~~~~~~

Apply evolutions to the database.

If ``--hint`` is specified, the hinted list of mutations will be applied. If
``--hint`` is not specified, this command will apply any unapplied stored
evolutions.

.. note::

You cannot specify an application name if you are trying to execute an
    evolution. Evolutions must be applied across the entire database.

--purge
~~~~~~~

Remove any stale applications from the database.

If you remove an application from the ``INSTALLED_APPS`` list, the database
tables for that application also need to be removed. Django Evolution will 
only remove these tables if you specify ``--purge`` as a command line 
argument.

--noinput
~~~~~~~~~

Use the ``--noinput`` option to suppress all user prompting, such as
"Are you sure?" confirmation messages. This is useful if ``django-admin.py``
is being executed as an unattended, automated script.

--verbosity
~~~~~~~~~~~

Use ``--verbosity`` to specify the amount of notification and debug 
information that the evolve command should print to the console.

* ``0`` means no input.
    * ``1`` means normal input (default).
    * ``2`` means verbose input.

Built-in Mutations
------------------

Django Evolution comes with a number of mutations built-in. These mutations
cover the simple cases for model evolutions.

AddField(model_name, field_name, initial=None, \*\*kwargs)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Add the field 'field_name' to the model, using the provided initial value to
populate the new column. The extra keyword arguments are the
arguments used to construct the new field.

If you are adding a column to a existing table, you will need to specify
either ``null=True`` on the field definition, or provide an initial value for
the new field. This is because after adding the field, every item in the 
database must be able to provide a value for the new field. Since existing 
rows will not have a value for the new field, you must either explicitly allow 
a value of NULL in the model definition, or you must provide an initial value.

The initial value can be a literal (e.g., ``initial=3`` for an `IntegerField`), 
or a callable that takes no arguments and returns a literal. If you use a 
literal, Django Evolution will apply any quoting required to protect the 
literal value. If you use a callable, the value returned by the callable will 
be inserted directly into the SQL ``INSERT`` statement that populates the new 
column, so it should be in a format appropriate for database insertion.

If your newly added field definition provides a default, the default will be
used as the initial value.

Example::

# Add a nickname field, allowing NULL values.
    AddField('Author', 'nickname', models.CharField, max_length=100, null=True)

# Add a nickname field, specifying the nickname for all existing Authors
    # to be 'That Guy'
    AddField('Author', 'nickname', models.CharField, max_length=100, initial='That Guy')

# Add a note field, using a callable for the initial value.
    # Note that the value returned by the callable has been wrapped in
    # quotes, so it is ready for database insertion.
    def initial_note_value():
        return "'Registered %s'" % datetime.now()
    AddField('Author', 'note', models.CharField, max_length=100, initial=initial_note_value)

ChangeField(model_name, field_name, initial=None, \*\*kwargs)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Change database affecting attributes of the field 'field_name' to the model. Changes to the field
are specified using keyword arguments such as ''null=True'' and ''max_length=20''.

Like the AddField, if the change to the field is to set ''null=False'', an initial value must be provided
as either a literal or a callable.

The ChangeField will accept both changed attributes and unchanged attributes of a field. This means
that if a CharField has a modified ''max_length'', its ''null'' attribute may be re-specified in the ChangeField.
The ChangeField will detect that the ''null'' attribute was specified but not changed and will silently ignore
the re-specification.

Example::
    # Adding a null constraint to the birthday field.
    ChangeField('Author', 'date_of_birth', null=True)
    
    # Increasing the size of the author's name.
    ChangeField('Author', 'name', max_length=100)
    
.. note::
    The ChangeField only supports the null, max_length, unique, db_column, db_index and db_table (for many to many fields) attribute at this time.

DeleteField(model_name, field_name)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Remove the field 'field_name' from the model.

Example::

# Remove the author's name from the Author model
    DeleteField('Author', 'name')

RenameField(model_name, old_field_name, new_field_name, db_column=None, db_table=None)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Rename the field 'old_name' to 'new_name'.

``db_column`` and ``db_table`` are optional arguments; if provided, they
specify the new column name (in the case of a value or ForeignKey field) or
the new table name (in the case of a ManyToManyField). If you attempt to 
specify a table name for a normal or ForeignKey rename, an Exception will 
be raised; similarly, and exception will be raised if you attempt to specify 
a column name for a ManyToManyField rename.

Example::

# Rename the field 'name' as 'full_name'
    RenameField('Author', 'name', 'full_name')

DeleteModel(model_name)
~~~~~~~~~~~~~~~~~~~~~~~

Remove the model 'model_name' from the application.

Example::

# Remove the Author model
    DeleteModel('Author')

SQLMutation(tag, sql, update_func=None)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Run a list of SQL statements as a mutation. This can be used as an alternative
to defining an mutation as an SQL file. By default, SQLMutations cannot be
simulated - therefore, you should carefully audit the SQL for any evolution
sequence including an SQLMutation before you execute that SQL on a database.

``tag`` is a text description of the changes. It will usually be the
evolution name.

``update_func`` is a optional function that will be invoked to update the
project signature when the mutation is simulated. This allows SQLMutations
to be included as part of a simulated evolution sequence.

An update function takes the application label and current project signature,
and mutates that signature until it reflects the changes implemented by the
SQL. For example, a simple mutation function to remove an 'Author' table
from an application might look something like::

def my_update_func(app_label, project_signature):
        del project_signature[app_label]['Author']

If you provide an update function, it will be invoked as part of the
simulation sequence. Any discrepancies between the simulated and expected
project signatures will be reported as an error during the evolution process.

Example::

# Add a new column to the Author table
    SQLMutation('add_location', ['ALTER TABLE blogette_author ADD COLUMN location varchar(100);'])

Defining your own mutations
---------------------------

If you have a special set of mutation requirements, you can define your own
mutation. All you need to do is subclass
``django_evolution.mutations.BaseMutation``, and include the mutation in a
stored evolution.

More to come
============

Django Evolution isn't complete - yet. There is still lots to do, and we'd
appreciate any help or suggestions you may have. The following are some examples
of features that we would like to add:

1. Support for more mutation types, including:
       a. ChangeModel - to handle model constraints like ``unique_together``,
       b. ChangeData - to handle data changes without a corresponding field
          definition change.

2. Improving support for the MySQL database backend

3. Support for other database backends (Oracle...)

4. Support for database introspection - the ability to look at the actual
      database structure to determine if evolutions are required.