Category Archives: programming

Install portal_client on Linux

A user asked for help installing Portal Client. Which is a:

Python-based client for downloading data files hosted by the an instance of the portal software developed by the GDC and further modified by the Institute for Genome Sciences (IGS)

You can install this in any directory that you have write permissions to, and you will need python3 installed.

Here are the quickie install instructions:

git clone https://github.com/IGS/portal_client;
python3 -m venv --copies VENV_portal_client;
source VENV_portal_client/bin/activate;
pip install -r portal_client/requirements.txt;
cd portal_client;
easy_install . ;

Any time you need to run the tool, source the Python Virtual Environment (virtualenv) and run the command. For example:

source VENV_portal_client/bin/activate;
portal_client --help;

If you use a lot of Python virtualenv’s, I suggest using the virtualenvwrapper tool, which makes it easier to manage and run many virtualenv’s:

https://virtualenvwrapper.readthedocs.io/en/latest/

Edx on Debian Stretch

After you clone the Edx repo there you will find a (undocumented) list of packages required to install on Ubuntu:

requirements/system/ubuntu/apt-packages.txt

These packages and others are required for Debian Stretch.  First, install these packages:

sudo apt-get install $(grep -vE "^\s*#" requirements/system/ubuntu/apt-packages.txt   | tr "\n" " ")

And then install these packages:

sudo apt-get install python-scipy python-numpy build-essential python-dev gfortran libfreetype6-dev libjpeg-dev libtiff5-dev zlib1g-dev libpng-dev  libxml2-dev libxslt-dev yui-compressor graphviz libgraphviz-dev graphviz-dev mysql-server libmysqlclient-dev libgeos-dev libreadline7 libreadline7-dev mongodb nodejs mysql-client virtualenvwrapper libgeos-ruby1.8 lynx-cur libxmlsec1-dev swig  libsqlclient-dev libxmlsec1

Like the photo? See a bigger version on Flickr.

Migrate Django 1.4 to 1.11

I thought these rough notes might be useful to anyone migrating from Django 1.4 (and possibly 1.3) to Django 1.11.

Moving from South to Django Migrations

1. Update your settings files:

Remove ‘south’ from your INSTALLED_APPS

2. Remove migrations.

Remove all app/migrations files except for __initi__.py . Remove any *.py[c|o] files.

Make sure there’s no migrations specific code in your app/migrations/__init__.py files

https://docs.djangoproject.com/en/1.7/topics/migrations/

3. Run new Migrations

python manage.py makemigrations;

python manage.py migrate;

Moving from Django 1.3/1.4 -> 1.11

This is a 1.3 app migrated to 1.4 and now is being migrated to 1.11.

Will probably need a new manage.py file. Create a dummy project and copy manage.py over your existing manage.py file or diff the two). Rename your settings directory to match the name of your project. In my case the project is portal, settings renamed to portal.

URLS files:

django.conf.urls.defaults is replaced by django.conf.urls

patterns() is gone, see:

https://docs.djangoproject.com/en/1.11/releases/1.8/#django-conf-urls-patterns

Views

django.views.generic.simple

django.views.generic.simple is gone, removed in 1.5.

Replace django.views.generic.simple.direct_to_template with TemplateView:

from django.views.generic import TemplateView

Shortcuts

from django.shortcuts import RequestContext

Replaced with:

from django.template import RequestContext

Contrib

django.contrib.databrowse is gone. See:

Forms

This error:

django.core.exceptions.ImproperlyConfigured: Creating a ModelForm without either the 'fields' attribute or the 'exclude' attribute is prohibited; form BlahForm needs updating.

Change in Django 1.8. See:

Quick fix is:

fields = '__all__'

But you will need to confirm that you want the form to expose all fields.

Formtools

Removal of django.contrib.formtools:

Is now separate package, install with pip:

pip install django-formtools

Change imports from django.contrib.formtools to formtools. For example:

from django.contrib.formtools.preview import FormPreview

Becomes:

from formtools.preview import FormPreview

Models

model._meta.get_all_field_names() is gone.

See:

https://docs.djangoproject.com/en/1.9/ref/models/meta/#migrating-from-the-old-api

This is verbatim from the docs:

MyModel._meta.get_all_field_names() becomes:

from itertools import chain
list(set(chain.from_iterable(
    (field.name, field.attname) if hasattr(field, 'attname') else (field.name,)
    for field in MyModel._meta.get_fields()
    # For complete backwards compatibility, you may want to exclude
    # GenericForeignKey from the results.
    if not (field.many_to_one and field.related_model is None)
)))

This provides a 100% backwards compatible replacement, ensuring that both field names and attribute names ForeignKeys are included, but fields associated with GenericForeignKeys are not. A simpler version would be:

[f.name for f in MyModel._meta.get_fields()]

While this isn’t 100% backwards compatible, it may be sufficient in many situations.

Commands

BaseCommand

BaseCommand.option_list is gone.

See https://docs.djangoproject.com/en/1.11/releases/1.8/#extending-management-command-arguments-through-command-option-list .

Arguments

See here for how to add arguments:

https://docs.djangoproject.com/en/1.11/howto/custom-management-commands/#accepting-optional-arguments

These custom options can be added in the add_arguments() method like this:

class Command(BaseCommand):

    def add_arguments(self, parser):

        # Positional arguments

        parser.add_argument(‘poll_id’, nargs=‘+’, type=int)

        # Named (optional) arguments

        parser.add_argument(

            ‘–delete’,

            action=‘store_true’,

            dest=‘delete’,

            default=False,

            help=‘Delete poll instead of closing it’,

        )

Move from MySQL to Postgres

Bigger or more complex databases may want to do a manual migration where the database is migration using django models.

https://www.calazan.com/migrating-django-app-from-mysql-to-postgresql/

I’ll update this as I have time.

Migrate Section of a Git Repo to a new Git Repo

I inherited a git repo that has multiple projects in it. It’s easier to manage the code and dependencies if each of these projects are in their own git repo.

The process is pretty simple:

  1. Export the directory you plan to use as the starting point of the new repo
  2. Create the new repo
  3. Import the old repo

  1. Export the directory
git log --pretty=email --patch-with-stat --reverse --full-index --binary DIRECTORY > /tmp/DIRECTORY_patch;

2. Create the new repo

git init REPONAME;

3. Import the old

cd REPONAME;

git am < /tmp/DIRECTORY_patch;

Run ‘git log’ to verify that you’ve successfully import the repo.

django.db.utils.DataError: invalid value for parameter “TimeZone”

If your Django Apps throws this error:

    cursor.execute(self.ops.set_time_zone_sql(), [tz])

django.db.utils.DataError: invalid value for parameter "TimeZone": "America/New_York"

The problem is that your Database and Django app are set to use different timezones.

Search your Django settings files for this:

TIME_ZONE =

And then connect to your database and query for the timezone. In Postgres run this query:

show timezone;

The fix is to have your Database and Django app use the same time zone. The quickest fix is to set TIME_ZONE in your Django app to be the same as your Database.

Python-LDAP: List the Group Membership of a User

Unless your an LDAP expert it’s not obvious how to find the groups a user is a member of, and that also describes how you need to tackle this problem.

This code makes a lot of assumptions and also assumes that a user’s UID is named ‘uid’.

If there’s a better way to do this, let me know.

References:
This thread on the Python-LDAP mailing list was quite useful:

Build & Install Osra 1.3.8 on Ubuntu 11.10

Summary

Osra is a a utility designed to convert graphical representations of chemical structures create by Igor Filippov at the National Cancer Institute. This page documents how to compile and install Osra on Ubuntu Linux 11.10. These instructions may work on other versions of Ubuntu Linux and on Debian Linux.  Please leave a comment if you have compiled Osra using these instructions on a different version of Ubuntu, or on other Linux distributions.

Overview

At the time of writing this doc, March 2012, the Osra version is 1.3.8 and is available at:

I copied all the source to a directory in /tmp. If you need the source code at some later point in time, don’t use /tmp as files in /tmp are deleted upon reboot (if they’re older than 14 days). Also, at the time of writing Osra requires a patched version of Gocr for Osra to work. You need to install Gocr before you try to compile and install Osra, you also need to install other packages required for Osra to compile. Most are listed below, but see the Osra Homepage for more details.

An overview of the steps are:

  1. Install required Ubuntu packages
  2. Compile and Install Gocr
  3. Compile and Install Osra

I’ve also written instructions on how to install Osra 1.2.1 on Ubuntu 9.04, however that was written in 2009. Continue reading Build & Install Osra 1.3.8 on Ubuntu 11.10

Install Oracle SQL Developer on Ubuntu Karmic & Lucid

If you want to install Oracle’s SQL Developer on Ubuntu (or another DEB based system such as Debian) you can do one of the following:

  1. Download the RPM package and install using rpm (not advisable).
  2. Download the RPM package and convert to a DEB package using alien
  3. Download the ZIP file titled “Oracle SQL Developer for other platforms” and manually install
  4. Use the make-sqldeveloper-package to convert the ZIP file into a DEB package

I used the make-sqldeveloper-package, which is available for Debian and Ubuntu and it’s derivatives. However, the man page and the instructions are little unclear on how to use it. You need to download the zip file available at Oracle’s (and not the RPM file nor any of the other packages) and then use the make-sqldeveloper-package to convert it to a DEB which you can then install using the dpkg command.

This is preferable to using rpm or alien as you can more easily manage the package using Debian’s and Ubuntu package management tools, plus it will integrate SQL Developer into Gnome’s Menu System . Plus, when Oracle updates their version you can use make-sqldeveloper-package to create an updated DEB package and easily update the version you have installed. The procedure outlined below works on Ubuntu Karmic and should also work on any Debian version that has the make-sqldeveloper-package.

As the man page states:

This utility will require you to download the “Oracle  SQL  Developer  for  other  platforms”  archive  from  <http://otn.oracle.com/software/prod‐ucts/sql/> to create the Debian package from.

1. Download the ZIP file from Oracle’s site

You will need to accept Oracle’s license agreement and download the ZIP file, titled “Oracle SQL Developer for other platforms”, from:

http://www.oracle.com/technology/software/products/sql/index.html

2. Install the make-sqldeveloper-package package

sudo apt-get install sqldeveloper-package

You will need Ubuntu’s multiverse software repository. See the instructions on how to do so.

2.a. Install the tofrodos package

sudo apt-get install tofrodos

Create symbolic links to the tofrodos commands to allow the sqldeveloper-package to work:

sudo ln -s /usr/bin/fromdos /usr/bin/dos2unix
sudo ln -s /usr/bin/todos /usr/bin/unix2dos

Thanks to Miles for the tofrodos tip.

3. Convert the ZIP file to .DEB package

The format of the command is:

make-sqldeveloper-package -b BUILD_LOCATION LOCATION_OF_ZIP_FILE

In my example, I’m going to use /tmp/ORA as the build location and the ZIP file has been download to my Desktop (~Desktop).

make-sqldeveloper-package -b /tmp/ORA/ ~/Desktop/sqldeveloper-2.1.0.63.73-no-jre.zip

The command will create the build directory and remove it when the command finishes. If the command completes successfully you’ll see the following output after a minute or two:

make-sqldeveloper-package: Building sqldeveloper package in "/tmp".

The DEB file will be created in the current directory. In my example I had changed directories to /tmp and that’s where the DEB file will be:

/tmp/sqldeveloper_2.1.0.63.73+0.2.3-1_all.deb

See the man page for more information about the tool:

man make-sqldeveloper-package

4. Install the package

sudo dpkg -i sqldeveloper_2.1.0.63.73+0.2.3-1_all.deb

5. Run SQL Developer

After installing there should he an entry in Gnome’s menu to start the SQL Developer. It’s located under:

Applications->Programming->SQL Developer

Note: You will need a JVM installed for the tool to run.

Obligatory Screen Shot:

Oracle SQL Developer

Updated September 27th 2010. Added instructions from Miles on installing dos2unix alternative for Ubuntu Lucid. The dos2unix command is not available in Ubuntu Lucid.

Compiling EIGENSOFT on Ubuntu Januty

These are rough notes on how to compile Eigensoft 3 on Ubuntu Jaunty.

Update, Eigensoft 4 is available and should compile on more recent versions of Ubuntu:
http://www.hsph.harvard.edu/faculty/alkes-price/software/

Create a location to build the code:

Warning: /tmp is for temporary files, and any files & directories you create there will be remove upon some subsequent reboot.

cd /tmp;
mkdir EIGEN;
cd  EIGEN;

Get the Eigensoft source code:

wget  http://www.hsph.harvard.edu/faculty/alkes-price/files/EIG3.0.tar.gz;
tar xzvf EIG3.0.tar.gz;

Install required dependences: Continue reading Compiling EIGENSOFT on Ubuntu Januty