Recently in Linux Category

Getting Spotify to work nicely on Linux

Note: the Linux Spotify client will only work with a premium Spotify account.

I spoke at the NYC PostgreSQL Users' Group meeting in December, and while there someone mentioned that Spotify is a great music service (and that they are using PostgreSQL!). So I decided to give it a try. The issue was that, while it can be made to work on Linux, the process of making it work well on Linux is less than simple. I decided to document what I did (and my sources) as I had to pull information from several sources and added a few modifications of my own.

There are two main problems to deal with:

1. Getting the program itself installed and running


2. Getting Linux and your browser to handle the spotify protocol so that, for example, clicking on playlist URLs will work correctly

The answer to problem number one depends in part on your Linux distribution. I am only going to cover Ubuntu and Fedora here -- extrapolation is left as an exercise for the reader.

On Ubuntu (I'm using 11.10), the directions from Spotify seems to work fine. I'll paste them here for the sake of completeness:

# On Ubuntu
# This gets you the older released client
# From http://www.spotify.com/us/download/previews/
# -----------
# 1. Add this line to your list of repositories by
#    editing your /etc/apt/sources.list
deb http://repository.spotify.com stable non-free

# 2. If you want to verify the downloaded packages,
#    you will need to add our public key
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4E9CFF4E

# 3. Run apt-get update
sudo apt-get update

# 4. Install spotify!
sudo apt-get install spotify-client-qt

I just noticed that the Ubuntu directions result in the older client working, not the shiny new preview version. See below for instructions to get the preview client working

# On Ubuntu, new preview client
# From 
# http://getsatisfaction.com/spotify/topics/try_out_the_linux_apps_client_beta_preview
# -----------
wget \
http://download.spotify.com/preview/spotify-client_0.8.0.1031.ga1569aa.552-1_amd64.deb
ar vx spotify-client_0.8.0.1031.ga1569aa.552-1_amd64.deb
tar -xzvf data.tar.gz
cp -rf ./usr /

# From 
# http://meltingrobot.wordpress.com/2011/11/08/spotify-installation-on-fedora-16/
# modified to handle passing of arguments

vi /usr/local/bin/spotify

# add the following lines to /usr/local/bin/spotify
8<--------------------------
#!/bin/bash

/bin/rm -rf ~/.cache/spotify
/usr/share/spotify/spotify $*
8<--------------------------

# make the script executable
chmod +x /usr/local/bin/spotify

# arrange to use the script in place of the binary to work
# around a known issue causing segfaults
rm /usr/bin/spotify
ln -s /usr/local/bin/spotify /usr/bin/spotify

# create symlinks to work around library mismatches
ln -s /usr/lib/x86_64-linux-gnu/libplc4.so /usr/lib/x86_64-linux-gnu/libplc4.so.0d
ln -s /usr/lib/x86_64-linux-gnu/libnspr4.so /usr/lib/x86_64-linux-gnu/libnspr4.so.0d

On Fedora things are complicated by the fact that Spotify no longer distributes an RPM - at least not that I could find. There are several recipes for solving this dilemma that can be found scattered around the Internet. Here is what I used:

# On Fedora (I am on Fedora 15)
# From http://www.passwdshadow.com/
yum -y install perl-ExtUtils-MakeMaker gcc qt-webkit rpm-build git
cd /tmp
git clone git://git.kitenet.net/alien
cd alien
perl Makefile.PL; make; make install
wget \
http://download.spotify.com/preview/spotify-client_0.8.0.1031.ga1569aa.552-1_amd64.deb
/usr/local/bin/alien --to-rpm \
spotify-client_0.8.0.1031.ga1569aa.552-1_amd64.deb
rpm -Uvh --nodeps spotify-client-0.8.0.1031.ga1569aa.552-2.x86_64.rpm
ln -s /usr/lib64/libssl.so.1.0.0e /usr/lib64/libssl.so.0.9.8
ln -s /lib64/libcrypto.so.1.0.0e /lib64/libcrypto.so.0.9.8
ln -s /usr/lib64/libnss3.so /usr/lib64/libnss3.so.1d
ln -s /usr/lib64/libnssutil3.so /usr/lib64/libnssutil3.so.1d
ln -s /usr/lib64/libsmime3.so /usr/lib64/libsmime3.so.1d
ln -s /lib64/libplc4.so /lib64/libplc4.so.0d
ln -s /lib64/libnspr4.so /lib64/libnspr4.so.0d

# From 
# http://meltingrobot.wordpress.com/2011/11/08/spotify-installation-on-fedora-16/
# modified to handle passing of arguments

vi /usr/local/bin/spotify

# add the following lines to /usr/local/bin/spotify
8<--------------------------
#!/bin/bash

/bin/rm -rf ~/.cache/spotify
/usr/bin/spotify.bin $*
8<--------------------------

# make the script executable
chmod +x /usr/local/bin/spotify

# arrange to use the script in place of the binary to work
# around a known issue causing segfaults
mv /usr/bin/spotify /usr/bin/spotify.bin
ln -s /usr/local/bin/spotify /usr/bin/spotify

At this point you should be able to click on the Spotify desktop shortcut and the program will launch.

So on to problem number two. One of the key features of Spotify is the ability to share playlists. This is done via a "spotify" protocol URL. Unfortunately at this point neither Linux nor your browser know how to handle this protocol. I have only worked out the specifics for gnome and Firefox, but here they are below:

# Handling the spotify protocol -- e.g. to allow use of http://sharemyplaylists.com
# From http://kb.mozillazine.org/Register_protocol
# -------------------------------------------------
# At shell command prompt:
gconftool-2 -s \
/desktop/gnome/url-handlers/spotify/command '/usr/bin/spotify %s' --type String
gconftool-2 -s \
/desktop/gnome/url-handlers/spotify/enabled --type Boolean true

# In Firefox:
#    Type about:config into the Location Bar (address bar) and press Enter.
#    Right-click -> New -> Boolean
#          -> Name: network.protocol-handler.expose.spotify
#          -> Value -> false
# Next time you click a link of protocol-type spotify you will be asked
# which application to open it with. Select /usr/bin/spotify

I think that's everything. I used the preceding successfully on my Fedora 15 desktop and my Ubuntu 11:10 laptop. But use at your own risk -- no guarantees that the foregoing will work or will not eat your data ;-)

Hope this helps someone else!

This month, credativ is pleased to welcome two new members of staff to the team in Rugby; as a leading specialist in Linux and Free Software, we are expanding in order to accommodate the growing demand for our services.

At credativ we invest in our employees - their growth and development is important to us and, by working for a dynamic company which is constantly evolving, our employees gain exposure to a diverse range of opportunities which may not be available so readily in larger, more traditional organizations.

Current Opportunities
credativ UK is looking for competent open source developers and assistant free software engineers to join our development team permanently. We have a small development team in the UK, so you need to learn quickly and be the kind of person who gets things done and cares about your coding and craftmanship. Current customer projects using Python and Ruby include:

back and front end web development for customers to use for training and recording of information security processes

creating and advancing modules for business enterprise systems

Work will involve supporting existing systems, improving free software packages and deploying new technologies for customers.

What do we do?
We develop and support business software solutions using free software; our key business areas are consulting, development, support and training. credativ supports a diverse range of clients and has a long history of contributing to free software projects.

Our technical team is actively involved with software projects such as Debian, PostgreSQL and OpenERP, among others. Over the last decade, credativ has expanded from Germany to the UK, US, Canada and India, and worked to maintain excellent relationships with other free software organisations, companies and upstream projects. This means we have extensive links with the wider free software community and a vast pool of resources we can tap in to for the benefit of our customers.

What do we use?
Our platforms run on Linux and are all built using open source technologies. We use Python, Ruby, Rails, PostgreSQL, Django, Apache, C++, Git, and whatever is the best tool for the job. We use lightweight agile development processes, with a strong emphasis on test driven development; we like to get involved in user groups and open source community initiatives.

Skills and Requirements
Solid development skills, a hunger for learning new things and enthusiasm are the most important things. We are using some interesting technologies to solve some interesting problems, so a good approach to problem-solving is a must.
Our platforms use a few core technologies, the more you are familiar with the better. Here is a sublist; for detailed job descriptions please see the careers pages on our website.

Linux

PostgreSQL, MySQL

Python, other object oriented programming languages

Ruby

Ruby on Rails

Open Object, OpenERP

GNU/Linux system administration

How to get in touch
Please send your CV, a covering letter, and links to your blog, github or any open source project contributions to careers@credativ.co.uk

The System Administrator will often come across a situation where an SSH connection to Host B is only possible by making a detour via SHH to Host A:
client -> ssh A -> ssh B

To shorten this two-step process, an entry can be made in the ~/.ssh/config of Host A as "Jumphost", to ensure that this step is always followed in future.

Host Bdirekt
Hostname $IP_von_B
User rwo 
ProxyCommand ssh root@A.intern.lan nc %h %p

In the first row an alias is defined - this can be arbitrary, but some relation to B would make sense. The second row defines the host name of B - for permissions in every network thereafter, an IP is a good idea as a hostname! The option ProxyCommand defines the underlying Jump function - where access via SSH to A and the pipe of data occurs by means of numerical control.

Where SSH keys are properly allocated, there are no more queries. A simple ssh Bdirect leads directly to host B.

All tips in this blog can be found in the Tip Category. Should you need further Support for Linux, you've come to the right place at credativ.

Recursively finding Windows Internet Shortcut (*.url) files and changing them into GNOME desktop files

Over the past few days I have finally converted my wife's computer from WinXP to Linux (Ubuntu 10.10). One of the many fine points of the negotiation leading to this was that I needed to preserve her many Internet Shortcut files. Since she has literally thousands of them, and they are sprinkled about in many a nested folder, I needed a script that could find them, and create the equivalent GNOME desktop files. The following is my solution. Perhaps not the most elegant way to achieve these ends, but it worked great for me. However I cannot promise this script will not eat your files, so please test and use at your own risk ;-)

Create the following script (e.g. using vi)

vi /usr/local/bin/fix_url.sh

Put the following in fix_url.sh (press "i", and then type or paste):

#!/bin/bash

(
    IFS=$'\n'
    files=$(find . -name *.url)
    for fl in $files; do
        NEWFILE=${fl}.desktop
        cp "${fl}" "${NEWFILE}"
        sed -i 's/InternetShortcut/Desktop\ Entry/g' "${NEWFILE}"
        sed -i '/^\[DEFAULT\]/d' "${NEWFILE}"
        sed -i '/^BASEURL/d' "${NEWFILE}"
        sed -i '/^IconFile/d' "${NEWFILE}"
        sed -i '/^IconIndex/d' "${NEWFILE}"
        sed -i 's/\r$//' "${NEWFILE}"
        echo "Type=Link" >> "${NEWFILE}"
    done
)

Save the file by typing ":x" if you used vi.

Make it executable:

chmod +x /usr/local/bin/fix_url.sh

Test/run the new script. Do this first on an isolated test location, e.g. copy some Windows Internet Shortcut files to /tmp/windows_urls:

cd /tmp
/usr/local/bin/fix_url.sh

Check out the resulting *.desktop files. Verify they look correct, and that they actually work when clicked from Nautilus, etc.

If completely satisfied, change to the root of the real directory tree and re-run the script.

When you are all finished, the original *.url files are still hanging around. If you want to get rid of them (again test first):

cd /tmp
find . -name *.url -delete

Hope this helps someone else!

The vast majority of Debian installations are simplified with the use of Preseeding and Netboot. Friedrich Weber, a school student on a work experience placement with us at our German office has observed the process and captured it in a Howto here:

Imagine the following situation: you find yourself with ten to twenty brand new Notebooks and the opportunity to install them with Debian and customise to your own taste. In any case it would be great fun to manually perform the Debian installation and configuration on each Notebook. This is where Debian Preseed comes into play.

The concept is simple and self-explanatory; usually, whoever is doing the installation will be faced with a number of issues during the process (e.g. language, partitioning, packages, Bootloader, etc.) In terms of Preseed, all of these issues can now be resolved. Only those which are not already accounted for in Preseed remain for the Debian installer. In the ideal situation these would become apparent at the outset of the installation, where the solution would differ depending on the target system and which the administrator must deal with manually - only when these have been dealt with can the installation be left to run unattended.

Preseed functions on some simple inbuilt configuration data: preseed.cfg. It includes, as detailed above, the questions which must be answered during installation, and in debconf-format. Data such as this consists of several rows, each row of which defines a debconf configuration option - a response to a question - for example:

    d-i debian-installer/local	string de_DE.UTF-8

The first element of these lines is the name of the package, which is configured (d-i is here an abbreviation of debian installer), the second element is the name of the option, which is set, as the third element of the type of option (a string) and the rest is the value of the option. In this example, we set the language to German using UTF-8-coding.

You can put lines like this together yourself, even simpler with the tool debconf-get-selections: these commands provide straight forward and simple options, which can be set locally. From the selection you can choose your desired settings, adjusted if necessary and copied into preseed.cfg.

Here is an example of preseed.cfg:

    d-i debian-installer/locale string de_DE.UTF-8
    d-i debian-installer/keymap select de-latin1
    d-i console-keymaps-at/keymap select de
    d-i languagechooser/language-name-fb select German
    d-i countrychooser/country-name select Germany
    d-i console-setup/layoutcode string de_DE

    d-i clock-setup/utc boolean true
    d-i time/zone string Europe/Berlin
    d-i clock-setup/ntp boolean true
    d-i clock-setup/ntp-server string ntp1

    tasksel tasksel/first multiselect standard, desktop, gnome-desktop, laptop
    d-i pkgsel/include string openssh-client vim less rsync

In addition to language and timezone settings, selected tasks and packages are also set with these options. If left competely unattended, the installation will not complete, but will make a good start.

Now onto the question of where Preseed pulls its data from. It is in fact possible to use Preseed with CD and DVD images or USB sticks, but generally more comfortable to use a Debian Netboot Image, essentially an installer, which is started across the network and which can cover its Preseed configuration. This boot across the network is implemented with PXE and requires a system that can boot from a network card.

Next, the system depends on booting from the network card. It travels from a DHCO server to an IP address per broadcast. This DHCP server transmits not only a suitable IP, but also to the IP of a so-called Bootserver. A Bootserver is a TFTP-Server, which provides a Bootloader to assist the Administrator with the desired Debian Installer. At the same time the Debian Installer can be shared with the Boot options that Preseed should use and where he can find the Preseed configuration. Here is a snippet of the PXELINUX configuration data pxelinux.cfg/default:

    label i386
        kernel debian-installer/i386/linux
        append vga=normal initrd=debian-installer/i386/initrd.gz netcfg/choose_interface=eth0 domain=example.com locale=de_DE debian-installer/country=DE debian-installer/language=de debian-installer/keymap=de-latin1-nodeadkeys console-keymaps-at/keymap=de-latin1-nodeadkeys auto-install/enable=false preseed/url=http://$server/preseed.cfg DEBCONF_DEBUG=5 -- quiet 

When the user types i386, the tt>debian-installer/i386/linux kernel (found on the TFTP server) is downloaded and run. This is in addition to a whole load of bootoptions given along the way. The debian installer allows the provision of debconf options as boot parameters. It is good practice for the installer to somehow communicate where to find the Preseed communication on the network (preseed/url). In order to download this Preseed configuration, it must also be somehow built into the network.

The options for that will be handed over (the options for the hostnames would be deliberately omitted here, as every target system has its own Hostname). auto-install/enable would delay the language set up so that it is only enabled after the network configuration, in order that these installations are read through preseed.cfg. It is not necessary as the language set up will also be handed over to the kernel options to ensure that the network configuration is German.

The examples and configuration excerpts mentioned here are obviously summarised and shortened. Even so, this blog post should have given you a glimpse into the concept of Preseed in connection with netboot. Finally, here is a complete version of preseed.cfg:

    d-i debian-installer/locale string de_DE.UTF-8
    d-i debian-installer/keymap select de-latin1
    d-i console-keymaps-at/keymap select de
    d-i languagechooser/language-name-fb select German
    d-i countrychooser/country-name select Germany
    d-i console-setup/layoutcode string de_DE

    # Network
    d-i netcfg/choose_interface select auto
    d-i netcfg/get_hostname string debian
    d-i netcfg/get_domain string example.com

    # Package mirror
    d-i mirror/protocol string http
    d-i mirror/country string manual
    d-i mirror/http/hostname string debian.example.com
    d-i mirror/http/directory string /debian
    d-i mirror/http/proxy string
    d-i mirror/suite string lenny

    # Timezone
    d-i clock-setup/utc boolean true
    d-i time/zone string Europe/Berlin
    d-i clock-setup/ntp boolean true
    d-i clock-setup/ntp-server string ntp.example.com

    # Root-Account
    d-i passwd/make-user boolean false
    d-i passwd/root-password password secretpassword
    d-i passwd/root-password-again password secretpassword

    # Further APT-Options
    d-i apt-setup/non-free boolean false
    d-i apt-setup/contrib boolean false
    d-i apt-setup/security-updates boolean true

    d-i apt-setup/local0/source boolean false
    d-i apt-setup/local1/source boolean false
    d-i apt-setup/local2/source boolean false

    # Tasks
    tasksel tasksel/first multiselect standard, desktop
    d-i pkgsel/include string openssh-client vim less rsync
    d-i pkgsel/upgrade select safe-upgrade

    # Popularity-Contest
    popularity-contest popularity-contest/participate boolean true

    # Command to be followed after the installation. `in-target` means that 
         the following
    # Command is followed in the installed environment, rather than in 
        the installation environment.
    # Here http://$server/skript.sh nach /tmp is downloaded, enabled and 
        implemented.
    d-i preseed/late_command string in-target wget -P /tmp/ http://$server/skript.sh; 
  in-target chmod +x /tmp/skript.sh; in-target /tmp/skript.sh

All Howtos of this blog are grouped together in the Howto category - and if you happen to be looking for Support and Services for Debian you've come to the right place at credativ.

The text editor vim offers several tools for automation. This howto describes a way to auto-include text modules when creating new files.

Often during programming or administration you need the same text modules again and again. The editor vim is very helpful here, as it can detect a file type while it is being created and insert pre-defined text modules accordingly. This behaviour can be configured in the file .vim/plugin/autoinsert.vim, for example with:

if has("autocmd")
augroup autoinsert
  au!
  autocmd BufNewFile *.c call s:Template("c")
  autocmd BufNewFile Makefile call s:Template("make")
  autocmd BufNewFile makefile call s:Template("make-simple")
augroup END
endif

function s:Template(argument)
        if (a:argument == "help")
                echo "Currently available templates:"
                echo " c                - Plain C Template"
                echo " make             - Makefile Template"
                echo " make-simple      - Simple Variant of the Makefile Template"
        else
                " First delete all in the current buffer
                %d

                " The Makefile variants
                if (a:argument == "make")
                        0r ~/.vim/skeletons/template.make
                        set ft=make
                elseif (a:argument == "make-simple")
                        0r ~/.vim/skeletons/template.make_simple
                        set ft=make
                elseif (a:argument == "make-simple-cpp")
                        0r ~/.vim/skeletons/template.make_simple_cpp
                        set ft=make

                " Stuff for plain C
                elseif (a:argument == "c")
                        0r ~/.vim/skeletons/template.c
                        set ft=c
                endif

                silent %!~/.vim/do_header %
        endif
endfunction

command! -nargs=1 Template call s:Template(<f-args>)

The lines 21-35 clearly show the template names and include the text modules. The template for make_simple, ~/.vim/skeletons/template.make_simple, for example includes the compiler flags for building C/C++ programs with gcc:

CC := gcc
CFLAGS := -Wall -pedantic -O3
LDFLAGS :=

PROG := main
OBJS := main.o

all: $(PROG)

$(PROG): $(OBJS)
        $(CC) $(LDFLAGS) -o $@ $^

clean:
        rm -rf $(PROG) $(OBJS)

.PHONY: all clean

Another template is shown below: ~/.vim/skeletons/template.c includes text modules for not only the obligatory GPL header but also includes a basic code structure:

/*
 * %%FILENAME%% - description
 *
 * Copyright (C) %%YEAR%% %%AUTHOR%%
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2, or (at your option)
 * any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with this program; if not, write to the Free Software Foundation,
 * Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA.
 */

#include <stdio.h>

int
main (int argc, char **argv)
{
  return 0;
}

/**This must remain at the end of the file.**********
 * vim600:set sw=2 ts=8 fdm=marker fmr=«««,»»»:     *
 * vim600:set cindent cinoptions={1s,>2s,^-1s,n-1s: *
 ****************************************************/

Variables like %%FILENAME%% or %%AUTHOR%% can also automatically be replaced by a small shell script running during file creation: the script ~/.vim/do_header labelled with the file name as argument detects the full name with getent or /etc/passwd, respectively. Other variables are gathered using the default GNU tools as the listing shows:

#!/usr/bin/env zsh

if which getent > /dev/null; then
        REALNAME=$(getent passwd $USER|awk -F : '{print $5}' | awk -F , '{print $1}')
else
        REALNAME=$(grep $USER /etc/passwd|awk -F : '{print $5}' | awk -F , '{print $1}')
        if which nidump > /dev/null && [ -z "$REALNAME" ]; then
                REALNAME=$(nidump passwd / | grep $USER|awk -F : '{print $8}')
        fi
fi
DATE=$(date)
YEAR=$(date +%Y)
FILENAME=$(echo $1 | sed 's/[^/]*\///')
FILE=$(echo $FILENAME | sed 's/\..*//')
FILEBIG=$(echo $FILE | tr '[:lower:]' '[:upper:]')
sed     "s/%%AUTHOR%%/$REALNAME/g;
        s/%%DATE%%/$DATE/g;
        s/%%YEAR%%/$YEAR/g;
        s/%%FILENAME%%/$FILENAME/g;
        s/%%FILE%%/$FILE/g;
        s/%%FILEBIG%%/$FILEBIG/g;"

Besides the examples shown here other templates can be generated to deal with HTML files, python or whatnot, among others - the possibilities are endless.

This howto has just touched on one small aspect of the many automations possible with vim. You can find other howtos in this blog in the howtos category, which has its own feed - if you need more in-depth support or services for GNU tools or Linux, you've come to the right place at credativ.

Following our earlier introduction to RHCS we now present a real world example: the installation of RHCS with Debian to provide certain virtual machines as services.

Our RHCS overview already explained the basics of RHCS. This time we will take two hosts with shared storage and provide KVM guests as services.

Installation of the nodes

In this setup the nodes are the machines which are running KVM. Each running KVM guest is a service managed by RHCS. While installing the KVM hosts you should make sure you comply with the following suggestions:

• /tmp/ and /var/ should be running on different partitions, this improves performance.
• Activate Debian backports, especially for the Kernel.
• Make sure all IP addresses can be resolved in both directions - /etc/hosts helps here in worst case.
• The host name must not resolve to 127.0.0.1! You would only get problems with the Cluster Management System CMAN.
• /etc/hosts/ and /etc/resolv.conf should be the same on all nodes.
• Create password free ssh keys for all nodes and distribute them.
• For ultimate performance it is best to install the latest Debian Linux kernel. In our example we used linux-image-2.6.32-bpo.2-amd64, which crashes the guest kernels >= 2.6.30. However, a patch is available, see bug #573071.
• The network devices should be named in a way that makes sense, for example: rhcs-backbone and external instead of eth0 and eth1.

Configuring the shared storage

As with almost any HA solution, a key element of RHCS is the shared storage which is accessed by all the nodes. In this example we take a "private" machine and install an iSCSI target on it:

apt-get install iscsitarget iscsitarget-source 
echo 'ISCSITARGET_ENABLE=true' > /etc/default/iscsitarget
m-a a-i iscsitarget

Keep in mind that the iSCSI target must build properly, see bug #566740. The configuration of the shared storage is done via /etc/ietd.conf:

IncomingUser discovery_in YourSecurePwd1
OutgoingUser discovery_out YourSecurePwd2
Target YOURMACHINE:clvm1
       IncomingUser node_in YourSecurePwd1
       OutgoingUser node_out YourSecurePwd2
       Lun 0 Path=/dev/sdx1,Type=blockio

On the nodes the same target must be accessed, so make sure /etc/iscsi/iscsid.conf is correct:

discovery.sendtargets.auth.authmethod = CHAP
discovery.sendtargets.auth.username = discovery_in
discovery.sendtargets.auth.password = YourSecurePwd1
discovery.sendtargets.auth.username_in = discovery_out
discovery.sendtargets.auth.password_in = YourSecurePwd2
node.startup = automatic
node.session.auth.authmethod = CHAP
node.session.auth.username = node_in
node.session.auth.password = YourSecurePwd1
node.session.auth.username_in = node_out
node.session.auth.password_in = YourSecurePwd2

The service is started with /etc/init.d/open-iscsi start. Existing targets can be searched, deleted or added by the following commands:

# discovering the targets
iscsiadm -m discovery -t st -p YOURMACHINE -P 1
# deleting target on wrong interface
iscsiadm -m node -p 192.168.0.100:3260,1 -o delete
# opening the portal
iscsiadm -m node --targetname "iqn.2010-03.YOURMACHINE:clvm1" --portal "YOURMACHINE:3260" --

VM setup

The virtual machines are provided by KVM. Thus the apropriate KVM software must be installed first:

apt-get install linux-image-2.6.32-bpo.2-amd64 kvm libvirt-bin virtinst -t lenny-backports

When configuring the bridge, make sure that the bridge name is the same on all nodes. Also the libvirt configuration must be the same on all hosts, so it makes sense to use puppet or similar techniques.
Afterwards, bring up the guests with:

virt-install -n <NAME> -r 256 --vcpus=1 --disk path=/dev/vg_cluster#/<LV> \
  -c /root/debian-<VERSION>-amd64-netinst.iso --vnc --noautoconsole --os-type linux \
  --os-variant debianLenny --accelerate --network=bridge:bridge0 --hvm -k de

To monitor the process use virt-viewer -c qemu+ssh://:/system .

RHCS setup

The next step is the setup of RHCS itself. Again, first things first, the software: apt-get install redhat-cluster-suite. This pulls quite a number of services which are not needed in our example:

invoke-rc.d nfs-kernel-server stop
invoke-rc.d nfs-common stop
invoke-rc.d portmap stop
update-rc.d -f nfs-kernel-server remove
update-rc.d -f nfs-common remove
update-rc.d -f portmap remove

Btw., system-config-cluster is not available for Lenny, but our Philipp Hübner has created a backport:

wget --no-check-certificate https://www.credativ.com/~phu/lenny-backports/system-config-cluster/system-config-cluster_1.0.53-1_all.deb
dpkg -i system-config-cluster_1.0.53-1_all.deb
apt-get -f install
apt-get install xauth

In order to have locking on the LVM cluster, you now need to modify /etc/lvm/lvm.conf: check for the global part.

 locking_type = 3

With the newer kernels the module lock_dlm also vanished, so CMAN init script must be modified: comment out the line modprobe lock_dlm 2>&1 || return 1. Additionally, RHCS 2 only supports XEN, so for libvirt you need to load the resource handler vm.sh.

wget --no-check-certificate https:///www.credativ.com/~phu/vm.sh -O /usr/share/cluster/vm.sh
chmod +x /usr/share/cluster/vm.sh

RHCS itself is called via

/etc/init.d/cman start
/etc/init.d/clvm start
/etc/init.d/rgmanager start

Fencing

Fencing describes the automagical neutralization of nodes which cease to function properly. In our example we use a power plug which can be controlled via network, NETIO-230A. Currently there is no real fence agent available for the device, but the python library Python-Bibliothek offers the necessary background to quickly write one.

Closing words

This howto has shown the setup of RHCS on Debian in easy steps - but of course, the correct steps depend very much on the targeted services, so this is just an example. If you need help just ask - Open Source HA solutions are our speciality, and we offer services and support for KVM virtualization as part of our day to day business.

The OOM-Killer can cause nasty surprises on machines with a heavy memory load; processes are cancelled or terminated without warning. Fortunately, this behaviour can be adjusted with some clever kernel tweaks.

Administrators of Linux machines with a very high RAM-Usage are sometimes faced with a terrifying scenario: the Linux OOM-Killer (OOM = Out Of Memory). In situations such as a crashed PostgreSQL instance, the following entry can typically be found in the server log:

Out of Memory: Killed process PID (Prozessname)

Why is this?

Virtual Memory and Overcommit

Virtual Memory used by Linux can be allocated in a number of ways: malloc(), mmap(), Swap, Shared Memory, to mention some examples. It is possible to overcommit virtual memory by allocating more than is actually available in the system. If this happens, a so-called "OOM-Condition" occurs; that is, your system no longer has any available space in the virtual memory area and cannot allocate any more. This is when the OOM-Killer is activated - and does what its name suggests: kills any processes which meet certain conditions in order to free memory.

If you have an environment where servers are running PostgreSQL in parallel with other memory-intensive processes on the same machine, it's likely that the OOM-Killer will kill certain PostgreSQL processes. Due to the amount of allocated shared memory and the memory usage of each backend, the OOM-Killer will target PostgreSQL by preference since it counts the complete addressed shared memory area of all backends into summary.

The amount of committed memory of your system at a given time can be examined with the /proc-Filesystem:

$ grep Commit /proc/meminfo 
CommitLimit:    376176 kB
Committed_AS:   265476 kB

This example shows the current amount of committed memory at 265476 kB (Committed_AS). Is this equal or even larger than the amount of Committed_AS the OOM-Killer is likely to be woken up.

However, the kernel provides some interfaces to adjust the behaviour of the OOM-Killer and Overcommit with regard to PostgreSQL installations.

Turn off Overcommit

A radical method is to turn overcommit off entirely, although this is only recommended on systems dedicated to PostgreSQL. The overcommit feature can be configured within three categories with the following kernel parameter:

vm.overcommit_memory = 0

This can hold three different kinds of categories:

• 0: Allow a careful strategy of overcommitting memory: small and reasonable amounts of overcommitting allocations are allowed, but heavy and wild allocations will be denied. In this mode, root can allocate more space than unprivileged users. This is also the kernel default setting.
• 1: Allow overcommit without any constraints
• 2: Turn off overcommit. The effective allocatable memory space cannot be larger than swap + a configurable percentage of physical RAM.

The fraction of physical RAM used by category 2 is defined by the parameter:

vm.overcommit_ratio = 50

While vm.overcommit_memory=1 is useful when tuning certain applications, the categories 0 or 2 are the best ones to use most of the time. If you turn off overcommit with vm.overcommit_memory=2, a process will get an "out of memory"-Exception (depending of vm_overcommit_ratio) if allocating memory when no more free space is available. Depending on the distribution you are using, we recommend that you save those settings in the configuration file /etc/sysctl.conf to ensure that they are activated on server reboot.

$ echo "vm.overcommit_memory=2 >> /etc/sysctl.conf
$ echo "vm.overcommit_ratio=60 >> /etc/sysctl.conf
$ sysctl -p /etc/sysctl.conf

Changes to those parameters are activated immediately. You can recheck this by consulting /proc/meminfo:

$ grep Commit /proc/meminfo 
CommitLimit:    401440 kB
Committed_AS:   266456 kB

The machine has 249848 kB of swap and 252656 kB physical RAM.
According to the formula swap + vm.overcommit_ratio * RAM this results in a CommitLimit of 401440 kB

Configure OOM-Killer per process

Where PostgreSQL is running without dedicated server hardware and in parallel with memory-intensive middleware (e.g. JBoss- or Tomcat-Installations), most admins would prefer to be able to control the OOM-Killer on a per-process basis and allow overcommitting of memory allocations. Since kernel 2.6.1, Linux has been providing an interface for tuning the OOM-Score of a process, which will in turn increase or decrease the affinity of the process to be killed when running in an OOM-Situation. This interface allows a very flexible configuration of processes in such environments regarding their memory requirements. The interface is exposed by the /proc-Filesystem, for example here on a PostgreSQL-Installation on Debian:

$ cat /proc/$(cat /var/run/postgresql/8.4-main.pid)/oom_adj
0

Values allowed range from -17 to +15, a negative value decreases, while a positive value increases the likelihood of being killed by the OOM-Killer. -17 is a special value and turns killing the process in an OOM-Situation off.
The settings are inherited from parent to child processes; in PostgreSQL you'll have to set this one to the PostgreSQL master process:

$ echo -17 >> /proc/$(cat /var/run/postgresql/8.4-main.pid)/oom_adj
$ psql -q postgres
test=# SELECT pg_backend_pid();
 pg_backend_pid 
----------------
           3429
(1 line)

test=# 
[1]+  Stopped                 psql -q test
$ cat /proc/3429/oom_adj
-17

The disadvantage of this method is that all child processes will now be excluded from the OOM-Killer, which is not generally what DBAs prefer. For example, where you want to protect the PostgreSQL system processes (like background writer oder autovacuum) from being killed by the OOM-Killer, but still kill ordinary database connections when running out of memory.

To set the OOM-Score you need to have a privileged user, so the best way to implement this setting is to put it into your PostgreSQL start script.

Enhancements in PostgreSQL 9.0

PostgreSQL 9.0 will have additional support for the pictured /proc-Interface. On one hand PostgreSQL 9.0 will come with a new Linux start script, which supports setting the oom_adj value before starting up PostgreSQL; on the other hand it is possible to build PostgreSQL with the special C-Macro LINUX_OOM_ADJ defined, which will allow DBAs to limit the inheritance of the OOM-Score to backend childs as shown in this example:

$ ./configure CC="ccache gcc" CFLAGS="-DLINUX_OOM_ADJ=0"

This method will save the PostgreSQL system process but will allow the OOM-Killer to kill database backend processes running amok.

Alternatives

An alternative solution is available by an additional kernel patch. This extends the existing /proc-Filesystem with a list of process names which should be excluded from the OOM-Killer. However, this patch is an unoffical extension to the Linux kernel and you may have to maintain your own builds of Linux kernels. In addition, it is not nearly as flexible as adjusting the OOM-Score and process names are not useful for uniquely identifying processes (e.g. Java- or Perlbased processes).

Summary

The Linuxkernel provides a comprehensive interface to adjust processes regarding their memory usage and the OOM-Killer. The most flexible method is the introduced /proc-Filesystem with the oom_adj-Interface. PostgreSQL 9.0 will have additional support for this interface. Dedicated PostgreSQL-Systems can be configured to avoid overcommit at all, but will need a deeper understanding of the number of memory resources the database system demands and the requirements of the VM of the kernel.

This week, credativ launches its Open Source Support Card. With this card Open Source Support can be bought at a fixed price - without a binding contract.

After a long preparation phase we are now offering our trusted services in a new, simple format; with the Open Source Support Card you get a fixed contingent of project-specific, pre-paid services.

Customers using the Open Source Support Card have the unique advantage of full cost control; the card can be purchased as a product, without any obligation to sign an agreement for a specific length of time. This may be of particular benefit to larger companies, where new contracts have to be reviewed and cross-reviewed before they can be authorised. A summary of the advantages of the new pre-paid support format include:

• Open Source Support for a specific project
• Support not restricted to a specific number of desktops and servers within a company
• A tempting price model, starting at just £480
• Full control of costs
• Support available via telephone, e-mail and remote access
• Bilingual support - help given in English or even German, if required! ;-)
• Cost of support NOT determined by the number of CPUs or users
• NO binding contract - easy way to purchase
• NO call centre - direct access to the experts
• Support units can be used for the following services:
• administration
• installation (remote)
• consultancy

All support is provided to the usual credativ standard. Just as you would expect from our usual contracts, the cost of the service is not determined by the number of CPUs, users, or DB entries. Support units purchased through the Support Card can be used for all related problems within a company - no matter which workstation or server they come up on. The support itself is provided by our Open Source Support Centre: you won't have to deal with non-technical staff or battle through FAQ scripts - our Linux experts and Open Source specialists are on hand to take calls directly. Many of us are actively involved in contributing to a number of Open Source projects - as regular readers will already be aware. ;-)

The new Open Source Support Card is also an exciting development for the wider Open Source community. By offering yet another attractive support option for free distributions, we hope to prove that there is now no reason not to consider Debian and CentOS as viable alternatives to commercial distributions.

The Open Source Support Card is designed and marketed in such a way that resellers can also get on board, making access to support that bit easier for consumers: imagine purchasing your server online and while you're at it being able to drop a Support Card into the shopping basket as well - Open Source Support with just one click!

Currently the Support Card is just available for Debian and CentOS in the UK and in Germany, although we will soon be offering it in the US and Canada too. If you have any questions or comments we'd be pleased to hear from you - we've put a lot of effort into this new product, and are looking forward to the response from our customers and the wider community.

Many digital cameras today do not just save an image, but also save various meta data in the Exif standard. This data includes information about the position of the camera when the image was taken (such as vertical or horizontal). However, some image programs use this data to rotate the image when displaying it while others don't, leaving the user to face inconsistent behaviour.

This can be fixed with the tool exiftran; it automatically rotates all images according to the Exif data, which it discards afterwards. It is also very easy to use for mass conversion:

# apt-get install exiftran
$ find -print0 | xargs -0 exifautotran

This tool might be shipped by other distributions under a different name. Fedora, for example, calls it fbida.