All posts by Mick

Debian: Yarn fails to run.

Statue of Samuel Eliot Morison on The Mall on Commonwealth Ave in Boston.

Try setting the following environmental variable:

export NODE_PATH=/usr/lib/nodejs:/usr/share/nodejs

Yarn fails to run and shows a stacktrace like the following? Set the NODE_PATH.


Error: Cannot find module '@babel/runtime/helpers/interopRequireWildcard'
Require stack:
- /usr/share/nodejs/yarn/lib/cli/index.js
- /usr/share/nodejs/yarn/bin/yarn.js
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
    at Function.Module._load (node:internal/modules/cjs/loader:778:27)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/usr/share/nodejs/yarn/lib/cli/index.js:3:31)
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/usr/share/nodejs/yarn/lib/cli/index.js',
    '/usr/share/nodejs/yarn/bin/yarn.js'
  ]
}

The following posting on the Debian bug tracker will help:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=933229

Install portal_client on Linux

A user asked for help installing Portal Client. Which is a:

Python-based client for downloading data files hosted by the an instance of the portal software developed by the GDC and further modified by the Institute for Genome Sciences (IGS)

You can install this in any directory that you have write permissions to, and you will need python3 installed.

Here are the quickie install instructions:

git clone https://github.com/IGS/portal_client;
python3 -m venv --copies VENV_portal_client;
source VENV_portal_client/bin/activate;
pip install -r portal_client/requirements.txt;
cd portal_client;
easy_install . ;

Any time you need to run the tool, source the Python Virtual Environment (virtualenv) and run the command. For example:

source VENV_portal_client/bin/activate;
portal_client --help;

If you use a lot of Python virtualenv’s, I suggest using the virtualenvwrapper tool, which makes it easier to manage and run many virtualenv’s:

https://virtualenvwrapper.readthedocs.io/en/latest/

VAST Storage API & JQ

Glendalough Hiking Trail
Hiking Glendalough Park in Ireland

Vast builds some neat storage systems that provide management functionality via a REST API.

jq is a command-line JSON processor, or as the author describes it “sed for JSON”.

This is how I use both tools to get a quota’s used capacity:

curl -u $USERNAME:$PASSWORD --insecure -X GET "https://your-vast-dashboard/api/quotas/" -H "accept: application/json" | jq  ".[]|select(.used_capacity_tb | length > 0) | {name: .name, used: .used_capacity_tb}" 

USERNAME and PASSWORD are environmental variables set to the username and password you use to access your VAST API. Change to URI to point to your VAST appliance.

Edx on Debian Stretch

After you clone the Edx repo there you will find a (undocumented) list of packages required to install on Ubuntu:

requirements/system/ubuntu/apt-packages.txt

These packages and others are required for Debian Stretch.  First, install these packages:

sudo apt-get install $(grep -vE "^\s*#" requirements/system/ubuntu/apt-packages.txt   | tr "\n" " ")

And then install these packages:

sudo apt-get install python-scipy python-numpy build-essential python-dev gfortran libfreetype6-dev libjpeg-dev libtiff5-dev zlib1g-dev libpng-dev  libxml2-dev libxslt-dev yui-compressor graphviz libgraphviz-dev graphviz-dev mysql-server libmysqlclient-dev libgeos-dev libreadline7 libreadline7-dev mongodb nodejs mysql-client virtualenvwrapper libgeos-ruby1.8 lynx-cur libxmlsec1-dev swig  libsqlclient-dev libxmlsec1

Like the photo? See a bigger version on Flickr.

Migrate Django 1.4 to 1.11

I thought these rough notes might be useful to anyone migrating from Django 1.4 (and possibly 1.3) to Django 1.11.

Moving from South to Django Migrations

1. Update your settings files:

Remove ‘south’ from your INSTALLED_APPS

2. Remove migrations.

Remove all app/migrations files except for __initi__.py . Remove any *.py[c|o] files.

Make sure there’s no migrations specific code in your app/migrations/__init__.py files

https://docs.djangoproject.com/en/1.7/topics/migrations/

3. Run new Migrations

python manage.py makemigrations;

python manage.py migrate;

Moving from Django 1.3/1.4 -> 1.11

This is a 1.3 app migrated to 1.4 and now is being migrated to 1.11.

Will probably need a new manage.py file. Create a dummy project and copy manage.py over your existing manage.py file or diff the two). Rename your settings directory to match the name of your project. In my case the project is portal, settings renamed to portal.

URLS files:

django.conf.urls.defaults is replaced by django.conf.urls

patterns() is gone, see:

https://docs.djangoproject.com/en/1.11/releases/1.8/#django-conf-urls-patterns

Views

django.views.generic.simple

django.views.generic.simple is gone, removed in 1.5.

Replace django.views.generic.simple.direct_to_template with TemplateView:

from django.views.generic import TemplateView

Shortcuts

from django.shortcuts import RequestContext

Replaced with:

from django.template import RequestContext

Contrib

django.contrib.databrowse is gone. See:

Forms

This error:

django.core.exceptions.ImproperlyConfigured: Creating a ModelForm without either the 'fields' attribute or the 'exclude' attribute is prohibited; form BlahForm needs updating.

Change in Django 1.8. See:

Quick fix is:

fields = '__all__'

But you will need to confirm that you want the form to expose all fields.

Formtools

Removal of django.contrib.formtools:

Is now separate package, install with pip:

pip install django-formtools

Change imports from django.contrib.formtools to formtools. For example:

from django.contrib.formtools.preview import FormPreview

Becomes:

from formtools.preview import FormPreview

Models

model._meta.get_all_field_names() is gone.

See:

https://docs.djangoproject.com/en/1.9/ref/models/meta/#migrating-from-the-old-api

This is verbatim from the docs:

MyModel._meta.get_all_field_names() becomes:

from itertools import chain
list(set(chain.from_iterable(
    (field.name, field.attname) if hasattr(field, 'attname') else (field.name,)
    for field in MyModel._meta.get_fields()
    # For complete backwards compatibility, you may want to exclude
    # GenericForeignKey from the results.
    if not (field.many_to_one and field.related_model is None)
)))

This provides a 100% backwards compatible replacement, ensuring that both field names and attribute names ForeignKeys are included, but fields associated with GenericForeignKeys are not. A simpler version would be:

[f.name for f in MyModel._meta.get_fields()]

While this isn’t 100% backwards compatible, it may be sufficient in many situations.

Commands

BaseCommand

BaseCommand.option_list is gone.

See https://docs.djangoproject.com/en/1.11/releases/1.8/#extending-management-command-arguments-through-command-option-list .

Arguments

See here for how to add arguments:

https://docs.djangoproject.com/en/1.11/howto/custom-management-commands/#accepting-optional-arguments

These custom options can be added in the add_arguments() method like this:

class Command(BaseCommand):

    def add_arguments(self, parser):

        # Positional arguments

        parser.add_argument(‘poll_id’, nargs=‘+’, type=int)

        # Named (optional) arguments

        parser.add_argument(

            ‘–delete’,

            action=‘store_true’,

            dest=‘delete’,

            default=False,

            help=‘Delete poll instead of closing it’,

        )

Move from MySQL to Postgres

Bigger or more complex databases may want to do a manual migration where the database is migration using django models.

https://www.calazan.com/migrating-django-app-from-mysql-to-postgresql/

I’ll update this as I have time.

Migrate Section of a Git Repo to a new Git Repo

I inherited a git repo that has multiple projects in it. It’s easier to manage the code and dependencies if each of these projects are in their own git repo.

The process is pretty simple:

  1. Export the directory you plan to use as the starting point of the new repo
  2. Create the new repo
  3. Import the old repo

  1. Export the directory
git log --pretty=email --patch-with-stat --reverse --full-index --binary DIRECTORY > /tmp/DIRECTORY_patch;

2. Create the new repo

git init REPONAME;

3. Import the old

cd REPONAME;

git am < /tmp/DIRECTORY_patch;

Run ‘git log’ to verify that you’ve successfully import the repo.

Let’s Encrypt Cert & Nginx (Using a Web Proxy) on Centos 6.x

This was done on Centos 6 where prebuilt ‘certbot’ packages are not available. On Centos 7, install ‘certbot’ from the EPEL repo.

cd ~root;
mkdir certbot;
wget https://dl.eff.org/certbot-auto;
chmod 755 certbot-auto;
./certbot-auto;

This installs packages, including gcc, which you may want to uninstall as it’s bad practice to have compilers on a external facing (public) Web server.

On my server I have a different configuration file for each server we’re proxying for. These are in /etc/nginx/conf.d/ and using a naming convention of:

domainname.domain.conf

For example, example.com would be:

example.com.conf

Edit your config file and add this after location / { … } :

location /.well-known {
alias /tmp/static/.well-known;
}

Then:

mkdir /tmp/static/;

Run certbot:

./certbot-auto certonly

Continue reading Let’s Encrypt Cert & Nginx (Using a Web Proxy) on Centos 6.x

django.db.utils.DataError: invalid value for parameter “TimeZone”

If your Django Apps throws this error:

    cursor.execute(self.ops.set_time_zone_sql(), [tz])

django.db.utils.DataError: invalid value for parameter "TimeZone": "America/New_York"

The problem is that your Database and Django app are set to use different timezones.

Search your Django settings files for this:

TIME_ZONE =

And then connect to your database and query for the timezone. In Postgres run this query:

show timezone;

The fix is to have your Database and Django app use the same time zone. The quickest fix is to set TIME_ZONE in your Django app to be the same as your Database.