Category Archives: programming

Migrate Django 1.4 to 1.11

Migrate Django 1.4 to 1.11

I thought these rough notes might be useful to anyone migrating from Django 1.4 (and possibly 1.3) to Django 1.11.

Moving from South to Django Migrations

1. Update your settings files:

Remove ‘south’ from your INSTALLED_APPS

2. Remove migrations.

Remove all app/migrations files except for __initi__.py . Remove any *.py[c|o] files.

Make sure there’s no migrations specific code in your app/migrations/__init__.py files

https://docs.djangoproject.com/en/1.7/topics/migrations/

3. Run new Migrations

python manage.py makemigrations;

python manage.py migrate;

Moving from Django 1.3/1.4 -> 1.11

This is a 1.3 app migrated to 1.4 and now is being migrated to 1.11.

Will probably need a new manage.py file. Create a dummy project and copy manage.py over your existing manage.py file or diff the two). Rename your settings directory to match the name of your project. In my case the project is portal, settings renamed to portal.

URLS files:

django.conf.urls.defaults is replaced by django.conf.urls

patterns() is gone, see:

https://docs.djangoproject.com/en/1.11/releases/1.8/#django-conf-urls-patterns

Views

django.views.generic.simple

django.views.generic.simple is gone, removed in 1.5.

Replace django.views.generic.simple.direct_to_template with TemplateView:

from django.views.generic import TemplateView

Shortcuts

from django.shortcuts import RequestContext

Replaced with:

from django.template import RequestContext

Contrib

django.contrib.databrowse is gone. See:

Forms

This error:

django.core.exceptions.ImproperlyConfigured: Creating a ModelForm without either the 'fields' attribute or the 'exclude' attribute is prohibited; form BlahForm needs updating.

Change in Django 1.8. See:

Quick fix is:

fields = '__all__'

But you will need to confirm that you want the form to expose all fields.

Formtools

Removal of django.contrib.formtools:

Is now separate package, install with pip:

pip install django-formtools

Change imports from django.contrib.formtools to formtools. For example:

from django.contrib.formtools.preview import FormPreview

Becomes:

from formtools.preview import FormPreview

Models

model._meta.get_all_field_names() is gone.

See:

https://docs.djangoproject.com/en/1.9/ref/models/meta/#migrating-from-the-old-api

This is verbatim from the docs:

MyModel._meta.get_all_field_names() becomes:

from itertools import chain
list(set(chain.from_iterable(
    (field.name, field.attname) if hasattr(field, 'attname') else (field.name,)
    for field in MyModel._meta.get_fields()
    # For complete backwards compatibility, you may want to exclude
    # GenericForeignKey from the results.
    if not (field.many_to_one and field.related_model is None)
)))

This provides a 100% backwards compatible replacement, ensuring that both field names and attribute names ForeignKeys are included, but fields associated with GenericForeignKeys are not. A simpler version would be:

[f.name for f in MyModel._meta.get_fields()]

While this isn’t 100% backwards compatible, it may be sufficient in many situations.

Commands

BaseCommand

BaseCommand.option_list is gone.

See https://docs.djangoproject.com/en/1.11/releases/1.8/#extending-management-command-arguments-through-command-option-list .

Arguments

See here for how to add arguments:

https://docs.djangoproject.com/en/1.11/howto/custom-management-commands/#accepting-optional-arguments

These custom options can be added in the add_arguments() method like this:

class Command(BaseCommand):

    def add_arguments(self, parser):

        # Positional arguments

        parser.add_argument(‘poll_id’, nargs=‘+’, type=int)

        # Named (optional) arguments

        parser.add_argument(

            ‘–delete’,

            action=‘store_true’,

            dest=‘delete’,

            default=False,

            help=‘Delete poll instead of closing it’,

        )

Move from MySQL to Postgres

Bigger or more complex databases may want to do a manual migration where the database is migration using django models.

https://www.calazan.com/migrating-django-app-from-mysql-to-postgresql/

I’ll update this as I have time.

Migrate Section of a Git Repo to a new Git Repo

I inherited a git repo that has multiple projects in it. It’s easier to manage the code and dependencies if each of these projects are in their own git repo.

The process is pretty simple:

  1. Export the directory you plan to use as the starting point of the new repo
  2. Create the new repo
  3. Import the old repo

  1. Export the directory
git log --pretty=email --patch-with-stat --reverse --full-index --binary DIRECTORY > /tmp/DIRECTORY_patch;

2. Create the new repo

git init REPONAME;

3. Import the old

cd REPONAME;

git am < /tmp/DIRECTORY_patch;

Run ‘git log’ to verify that you’ve successfully import the repo.

django.db.utils.DataError: invalid value for parameter “TimeZone”

If your Django Apps throws this error:

    cursor.execute(self.ops.set_time_zone_sql(), [tz])

django.db.utils.DataError: invalid value for parameter "TimeZone": "America/New_York"

The problem is that your Database and Django app are set to use different timezones.

Search your Django settings files for this:

TIME_ZONE =

And then connect to your database and query for the timezone. In Postgres run this query:

show timezone;

The fix is to have your Database and Django app use the same time zone. The quickest fix is to set TIME_ZONE in your Django app to be the same as your Database.

Python-LDAP: List the Group Membership of a User

Python-LDAP: List the Group Membership of a  User

Unless your an LDAP expert it’s not obvious how to find the groups a user is member of, and that also describes how you need to tackle this problem.

This code makes a lot of assumptions and also assumes that a user’s UID is named ‘uid’.

If there’s a better way to do this, let me know.

References:
This thread on the Python-LDAP mailing list was quite useful:

Build & Install Osra 1.3.8 on Ubuntu 11.10

Summary

Osra is a a utility designed to convert graphical representations of chemical structures create by Igor Filippov at the National Cancer Institute. This page documents how to compile and install Osra on Ubuntu Linux 11.10. These instructions may work on other versions of Ubuntu Linux and on Debian Linux.  Please leave a comment if you have compiled Osra using these instructions on a different version of Ubuntu, or on other Linux distributions.

Overview

At the time of writing this doc, March 2012, the Osra version is 1.3.8 and is available at:

I copied all the source to a directory in /tmp. If you need the source code at some later point in time, don’t use /tmp as files in /tmp are deleted upon reboot (if they’re older than 14 days). Also, at the time of writing Osra requires a patched version of Gocr for Osra to work. You need to install Gocr before you try to compile and install Osra, you also need to install other packages required for Osra to compile. Most are listed below, but see the Osra Homepage for more details.

An overview of the steps are:

  1. Install required Ubuntu packages
  2. Compile and Install Gocr
  3. Compile and Install Osra

I’ve also written instructions on how to install Osra 1.2.1 on Ubuntu 9.04, however that was written in 2009. Continue reading Build & Install Osra 1.3.8 on Ubuntu 11.10

Install Oracle SQL Developer on Ubuntu Karmic & Lucid

If you want to install Oracle’s SQL Developer on Ubuntu (or another DEB based system such as Debian) you can do one of the following:

  1. Download the RPM package and install using rpm (not advisable).
  2. Download the RPM package and convert to a DEB package using alien
  3. Download the ZIP file titled “Oracle SQL Developer for other platforms” and manually install
  4. Use the make-sqldeveloper-package to convert the ZIP file into a DEB package

I used the make-sqldeveloper-package, which is available for Debian and Ubuntu and it’s derivatives. However, the man page and the instructions are little unclear on how to use it. You need to download the zip file available at Oracle’s (and not the RPM file nor any of the other packages) and then use the make-sqldeveloper-package to convert it to a DEB which you can then install using the dpkg command.

This is preferable to using rpm or alien as you can more easily manage the package using Debian’s and Ubuntu package management tools, plus it will integrate SQL Developer into Gnome’s Menu System . Plus, when Oracle updates their version you can use make-sqldeveloper-package to create an updated DEB package and easily update the version you have installed. The procedure outlined below works on Ubuntu Karmic and should also work on any Debian version that has the make-sqldeveloper-package.

As the man page states:

This utility will require you to download the “Oracle  SQL  Developer  for  other  platforms”  archive  from  <http://otn.oracle.com/software/prod‐ucts/sql/> to create the Debian package from.

1. Download the ZIP file from Oracle’s site

You will need to accept Oracle’s license agreement and download the ZIP file, titled “Oracle SQL Developer for other platforms”, from:

http://www.oracle.com/technology/software/products/sql/index.html

2. Install the make-sqldeveloper-package package

sudo apt-get install sqldeveloper-package

You will need Ubuntu’s multiverse software repository. See the instructions on how to do so.

2.a. Install the tofrodos package

sudo apt-get install tofrodos

Create symbolic links to the tofrodos commands to allow the sqldeveloper-package to work:

sudo ln -s /usr/bin/fromdos /usr/bin/dos2unix
sudo ln -s /usr/bin/todos /usr/bin/unix2dos

Thanks to Miles for the tofrodos tip.

3. Convert the ZIP file to .DEB package

The format of the command is:

make-sqldeveloper-package -b BUILD_LOCATION LOCATION_OF_ZIP_FILE

In my example, I’m going to use /tmp/ORA as the build location and the ZIP file has been download to my Desktop (~Desktop).

make-sqldeveloper-package -b /tmp/ORA/ ~/Desktop/sqldeveloper-2.1.0.63.73-no-jre.zip

The command will create the build directory and remove it when the command finishes. If the command completes successfully you’ll see the following output after a minute or two:

make-sqldeveloper-package: Building sqldeveloper package in "/tmp".

The DEB file will be created in the current directory. In my example I had changed directories to /tmp and that’s where the DEB file will be:

/tmp/sqldeveloper_2.1.0.63.73+0.2.3-1_all.deb

See the man page for more information about the tool:

man make-sqldeveloper-package

4. Install the package

sudo dpkg -i sqldeveloper_2.1.0.63.73+0.2.3-1_all.deb

5. Run SQL Developer

After installing there should he an entry in Gnome’s menu to start the SQL Developer. It’s located under:

Applications->Programming->SQL Developer

Note: You will need a JVM installed for the tool to run.

Obligatory Screen Shot:

Oracle SQL Developer

Updated September 27th 2010. Added instructions from Miles on installing dos2unix alternative for Ubuntu Lucid. The dos2unix command is not available in Ubuntu Lucid.

Compiling EIGENSOFT on Ubuntu Januty

These are rough notes on how to compile Eigensoft 3 on Ubuntu Jaunty.

Update, Eigensoft 4 is available and should compile on more recent versions of Ubuntu:
http://www.hsph.harvard.edu/faculty/alkes-price/software/

Create a location to build the code:

Warning: /tmp is for temporary files, and any files & directories you create there will be remove upon some subsequent reboot.

cd /tmp;
mkdir EIGEN;
cd  EIGEN;

Get the Eigensoft source code:

wget  http://www.hsph.harvard.edu/faculty/alkes-price/files/EIG3.0.tar.gz;
tar xzvf EIG3.0.tar.gz;

Install required dependences: Continue reading Compiling EIGENSOFT on Ubuntu Januty

Compiling Osra on Ubuntu Jaunty

This is a brief HOWTO on compiling OSRA, (Optical Structure Recognition) on Ubuntu Jaunty. To quote the OSRA home page, OSRA is

… is a utility designed to convert graphical representations of chemical structures, as they appear in journal articles, patent documents, textbooks, trade magazines etc., into SMILES (Simplified Molecular Input Line Entry Specification – see http://en.wikipedia.org/wiki/SMILES) or SD file – a computer recognizable molecular structure format. OSRA can read a document in any of the over 90 graphical formats parseable by ImageMagick – including GIF, JPEG, PNG, TIFF, PDF, PS etc., and generate the SMILES or SDF representation of the molecular structure images encountered within that document …

Update: I’ve a newer document that shows how to install Osra on Ubuntu 11.10 (Oneiric):

Make a directory to compile the source:

mkdir /tmp/OSRA; cd /tmp/OSRA;

Be careful doing this in /tmp is cleaned upon reboot the directory may be removed.

Install dependencies needed by the OS:

sudo apt-get install libgraphicsmagick1-dev libmagick++-dev libgraphicsmagick++1-dev potrace gocr  libtclap-dev libopenbabel-dev libopenbabel3 openbabel libnetpbm10 libnetpbm10-dev

Don’t install ocrad and remove it if it’s on your system (you can probably reinstall if you need to after you get Osra to compile):
sudo apt-get remove –purge ocrad;

Source Code:

Instead of manually getting the source packages download the sources used to build the packages for Ubuntu if available.  Make sure the src lines are commented in, in your /etc/apt/sources.list . This will automatically download and extract the code into the current directory:

cd /tmp/OSRA; apt-get source gocr ocrad potrace;

This downloads Gocr 0.46 which the OSRA docs say may not work:

– GOCR/JOCR, optical character recognition library, version 0.43 or later (version 0.45 recommended, do not use 0.46! See special instructions for 0.47 compilation below) Continue reading Compiling Osra on Ubuntu Jaunty

Stuff I’ve read or reading …

Some links to stuff I’ve either read or plan to read which I think is interesting:

Python is not Java, tips for Java programmers starting w/ Python:
http://dirtsimple.org/2004/12/python-is-not-java.html
Python for Bioinformatics:
http://www.pasteur.fr/recherche/unites/sis/formation/python/index.html
BioPython:
http://biopython.org/DIST/docs/tutorial/Tutorial.html

I need to know Selenium better, it’s possible that I might consider using it for a project I might be working on (that sounds like it might have some UI bugs/issues/problems):
http://seleniumhq.org/

10 things every programmer should read (this will take a while to wade through):
http://blog.objectmentor.com/articles/2009/02/26/10-papers-every-programmer-should-read-at-least-twice

Xen Cluster Mgt using Ganeti for Debian Lenny:
http://www.howtoforge.com/xen-cluster-management-with-ganeti-on-debian-lenny
Ganeti admin guide and install guide:
http://ganeti-doc.googlecode.com/svn/ganeti-1.2/admin.html
http://ganeti-doc.googlecode.com/svn/ganeti-1.2/install.html

David Byrne (of Talking Heads) interviews Thom Yorke (of Radiohead):
http://www.wired.com/entertainment/music/magazine/16-01/ff_yorke?currentPage=all

With a great quote, that shows how well the record companies treat their performing artists (that’s sarcasm in case you don’t have a sense of humour). First Radiohead made zip/zilch/nada from EMI sales of digitized versions of their music:

Yorke: In terms of digital income, we’ve made more money out of this record than out of all the other Radiohead albums put together, forever — in terms of anything on the Net. And that’s nuts. It’s partly due to the fact that EMI wasn’t giving us any money for digital sales. All the contracts signed in a certain era have none of that stuff …

Yorke: … It’s about whether the music affects you or not. And why would you worry about an artist or a company going after people copying their music if the music itself is not valued?

Then he talks about how the music iteself isn’t valued, but the business processes surrounding the marketing and selling of music:

Byrne: You’re valuing the delivery system as opposed to the relationship and the emotional thing…

Yorke: You’re valuing the company or the interest of the artists rather than the music itself. I don’t know. We’ve always been quite naive. We don’t have any alternative to doing this. It’s the only obvious thing to do.