How to create a DokuWiki farm

Let’s see how to install DokuWiki, create a farm of wikis and run it privately i.e. with our wikis not being reachable from the Internet.

I assume you have an Apache web server and PHP-7 running on your box. I’ve done my installation on a Ubuntu box.

Install the wiki

First we download the tarball from here and uncompress it. The result is a directory called dokuwiki. Then follow the steps:

$ sudo mv /home/vicent/Downloads/dokuwiki /var/www/dokuwiki
$ cd /var/www
$ sudo chown -R www-data:www-data dokuwiki
$ sudo a2enmod rewrite

Then, using /etc/apache2/sites-available/000-default.conf as a template, we create and enable the following virtual host (/etc/apache2/sites-available/dokuwiki.conf)


<VirtualHost *:80>
ServerName dokuwiki
ServerAdmin webmaster@localhost
DocumentRoot /var/www/dokuwiki
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

Then edit /etc/hosts and add the line:

127.0.0.1 dokuwiki

and restart the Apache service:

$ sudo systemctl restart apache2

Finally run the installation script with the web browser:

http://dokuwiki/install.php

Just fill the fields (it is pretty easy) and you are done 🙂 Now you can run your wiki at http://dokuwiki

About the farm

Suppose that you want to have a wiki for every member of your family (for instance you, your wife and your daughter). Of course you can do it simply creating and organizing properly pages on your wiki instance. But just for fun you can also create a farm of wikis with three independent wikis: one for you, one for your wife and one more for your daughter.

Let’s see how to manually set up a farm of wikis. Usually there is only one farm per server. The farm is made of a farmer (the actual DokuWiki installation, described in the previous section) and one or more animals (i.e. individual wiki instances). The setup of a farm directory is what is needed to start farming. Our setup will be:

/var/www/dokuwiki -> the dokuwiki engine
/var/www/farm -> the dokuwiki farm directory which contains the animals

In the farm directory we can have as many animals as we like:

/var/www/farm/wiki_1 -> one wiki
/var/www/farm/wiki_2 -> another wiki

There are two different setups: virtual host based and .htaccess based. We will use the first one. Its main advantage is that it can create much more flexible URLs which are independent of the underlying file structure. The disadvantage is that this method requires access to the the Apache configuration files.

Beware that in our case (a wiki running locally, not visible from the Internet) the mentioned advantage is not really important and the disadvantage simply doesn’t apply.

To access the farm from your local machine you have to edit the /etc/hosts file as described above

Create the farm directory

Create an empty directory named /var/www/farm. That will be the farm directory and needs to be writeable by the web server.

$ sudo mkdir /var/www/farm
$ sudo chown -R www-data:www-data /var/www/farm
$ sudo chmod -R ug+rwx /var/www/farm
$ sudo chmod -R o-rwx /var/www/farm

Activate the farm

This is easy too.

$ sudo cp /var/www/dokuwiki/inc/preload.php.dist /var/www/dokuwiki/inc/preload.php

Open that file, uncomment the two relevant lines and set the farm directory:

if(!defined('DOKU_FARMDIR')) define('DOKU_FARMDIR', '/var/www/farm');
include(fullpath(dirname(__FILE__)).'/farm.php');

Add an animal

Download the animal template and extract it in the farm directory. The archive contains a directory called _animal which includes an empty data and a pre-filled conf directory. Rename the directory. Beware that virtual host setup needs animal directory names that reflect their URL e.g. the URL vicent.uvemas.org works with a directory named vicent.uvemas.org.

In your /etc/hosts file you should add the line

127.0.0.1 vicent.uvemas.org

Virtual Host Based Setup

For this setup we create and enable a new site in /etc/apache2/sites-available for each new animal. For example for vicent.uvemas.org we will create the file vicent.uvemas.org.conf with the following content:

<VirtualHost *:80>
ServerName vicent.uvemas.org # this is the URL of the wiki animal
ServerAdmin webmaster@localhost
DocumentRoot /var/www/dokuwiki # the document root always needs to be the DokuWiki *farmer* directory
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

Now you should be able to visit the animal’s URL e.g. http://vicent.uvemas.org and enjoy your wiki.

Change the admin password

As the animal template includes a default admin user account with the password admin we should change that password as soon as possible and change the admin’s email address too.

I will stop here but there are a lot of things that you can do now: setup ACLs, install plugins, create namespaces… Please visit the DokuWiki website. There you will find tons of info about everything.

Advertisement

How to install Arch Linux on a VM VirtualBox (II)

In the last entry we began to see howto install Arch Linux on a VM VirtualBox. We created and setup the virtual machine and installed the Arch Linux. Now we are going to see how to setup Arch Linux for running smoothly on the virtual machine.

So we logon as root and continue our work. The first thing to do is to be sure that you can use the keyboard. If your keyboard layout is English you have to do nothing but if it is not then you need to setup the proper keyboard layout. In my case (Spanish layout) I have to run the command:

# localectl set-keymap --no-convert es

which set the value of the KEYMAP variable in the /etc/vconsole.conf file:

KEYMAP=es

This configuration is persistent and also applies to the current session. You can find more information about how to configurate the keyboard in console here.

Next thing is to automatically connect to the Internet when the system boots. We can achieve this goal if we start the DHCP as a service:

# systemctl enable dhcpcd.service
Created symlink from /etc/systemd/system/multi-user.target.wants/dhcpcd.service to /usr/lib/systemd/system/dhcpcd.service
# reboot

We logon again and check the network connection using the ping command:

# ping -c 3 www..google.com
PING www.l.google.com (74.125.224.146) 56(84) bytes of data.
64 bytes from 74.125.224.146: icmp_req=1 ttl=50 time=437 ms
64 bytes from 74.125.224.146: icmp_req=2 ttl=50 time=385 ms
64 bytes from 74.125.224.146: icmp_req=3 ttl=50 time=298 ms

--- www.l.google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 298.107/373.642/437.202/57.415 ms

Now let’s pay attention to the time synchronization. It is an important topic on a virtual machine because the CPU is shared among several systems. For instance you can see time delays on your virtual machine if the host system goes to sleep. There are several options for getting the time synchronized. The following works fine for me:

# pacman -S ntp
# systemctl enable ntpd.service
Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service

i.e. I install the ntp package (which contains an NTP server and client) and start it as a service every time the system boots but I don’t setup my system as an NTP server. This setup causes the hardware clock to be re-synchronised every 11 minutes. In theory there are simpler ways to achieve the synchronization goal (like using SNTP) but I’ve not been able to do they work properly. You can get more information about this topic here and here.

After checking that time synchronization works fine we can go to the next task: adding a new user. It is a typical task when administering a Lynux system and can be done easily:

# useradd -m -s /bin/bash vicent
# passwd vicent
Introduzca la nueva contraseña de Unix:
Vuelva a escribir la nueva contraseña de Unix:
passwd: contraseña actualizada correctamente

The above commands create a user called vicent, create its home directory in /home/vicent, give it a bash shell and set his password.

Next we’ll add the new user to the sudoers file. This way vicent will be able to execute a command with root privileges temporarily granted to that single command. How privileges are scaled depends on how the sudoers file is changed. In order to get both the sudo command and the sudoers file we install the sudo package:

# pacman -S sudo

Instead of editing the sudoers file directly we create files under the /etc/sudoers.d directory. These files will be automatically included in the sudoers file every time the sudo command is issued. This way we keep the sudoers file clean and easy to read. The sudoers file and the files under /etc/sudoers.d are edited with the visudo command which edit the files in a safe fashion (see the man page of the visudo command for details):

# visudo -f /etc/sudoers.d/90-vicent

We add the following line to the file:

vicent ALL=(ALL) ALL

It means that, on all hosts where this sudoers file has been distributed, vicent can execute any command with root privileges (after being prompted with vicent‘s password).

Now it’s time to install the graphical components. We begin installin the X Window System as follows:

# pacman -S xorg-server xorg-server-utils xorg-apps xorg-twm xorg-xinit xterm xorg-xclock ttf-dejavu --noconfirm

The above command will install the main components of the X, including the twm window manager. The X configuration files are:

• /usr/share/X11/xorg.conf.d
• /etc/X11/xorg.conf.d

None of those files contains the keyboard configuration so in order to keep my non-English layout when the X is running I execute the command:

# localectl --no-convert set-x11-keymap es

which creates the file /etc/X11/xorg.conf.d/00-keyboard.conf.

Now, before starting the X, we install the VirtualBox Guest Additions package and configure it:

# pacman -S virtualbox-guest-utils --noconfirm

we load the following modules:

# modprobe -a vboxguest vboxsf vboxvideo

and create the virtualbox.conf configuration file with the following content:

# echo vboxguest >> /etc/modules-load.d/virtualbox.conf
# echo vboxsf >> /etc/modules-load.d/virtualbox.conf
# echo vboxvideo >> /etc/modules-load.d/virtualbox.conf


Now we ensure that the user created before will be able to access with read-write permissions to the shared folder (we created the shared folder in the first part of this tutorial):

# usermod -a -G vboxsf vicent
# chown root.vboxsf /media
# chmod 770 /media

Finally we enable the guest additions service so they will be started et every system boot:

# systemctl enable vboxservice.service
Created symlink from /etc/systemd/system/multi-user.target.wants/vboxservice.service to /usr/lib/systemd/system/vboxservice.service

Now we’are ready to come back to the X Window System. Before to start it we have to create the /root/.xinitrc file (indeed we need that file in the $HOME of every user starting the X) with the following contents:

# Make sure the root user uses the right keyboard map
setxkbmap -model pc104 -layout es
# Start the VirtualBox Guest Additions
/usr/bin/VBoxClient-all
# Start the window manager
exec twm

Then we issue the startx command wich in turns sources the .xinitrc file so the result is a screen like this:

Reboot the system and logon again in a virtual console. We have reached the last step of the process i.e. the installation of a desktop environment. As I adhere to the ‘keep it simple’ philosophy of Arch Linux my choice was the LXDE. In order to install it we have to issue the following commands:

# pacman -S lxde
# systemctl enable lxdm.service
Created symlink from /etc/systemd/system/display-manager.service to /usr/lib/systemd/system/lxdm.service
# vi /etc/lxdm/lxdm.conf
uncomment the line
session=/usr/bin/startlxde

It is important to note that the startx command is not called and so the .xinitrc file is not sourced: the display manager (which is started as a service every time the system boots) calls directly to the startlxde command which is in charge of starting the LXDE desktop environment.

To make sure that the non-English keyboard map will persist between LXDE sessions we edit the file /etc/xdg/lxsession/LXDE/autostart and append the line:

setxkbmap -model pc104 -layout es

Now reboot, logon and you will get a nice LXDE screen. In my case, after some tweaking it looks like this:

How to install Arch Linux on a VM VirtualBox (I)

On this entry I’ll describe the steps I followed to successfully install Arch Linux on a VirtualBox virtual machine. The host system is a Windows 8.1.

My main source of documentation has been the excellent Arch Linux Installation Guide wiki. As the rest of the wiki it has a very high quality.

The first thing to do is to download the latest Arch ISO. While the ISO is downloading you can create the virtual machine. The properties of the VM I created are:

• name: ArchLinuxVM
• type of operating system: ArchLinux (64bit)
• RAM memory size: 2GB
• hard drive type: VDI, dynamically allocated
• hard drive size: 20GB
• bidirectional clipboard: yes
• shared folder: yes (remember that the shared folder must exist on the host system before setting this property)
• shared folder auto-mount: yes

If you aren’t new to VirtualBox and know how to setup the machine described above you can skip the next section.

Creating and Configuring the Virtual Machine

Open the VirtualBox program, click the New button of the toolbar, write down the machine name, and choose the OS type and version.

Choose the RAM size. In my case the host system has 8GB so 2GB of RAM was a sensible choice for my VM.

The next step is to create a virtual hard drive.

In the next screens we choose the disk type to be VDI and to allocate the space dynamically. Then we choose the hard disk size. The default size is 8 GB which is probably too small so we increase the size until 20GB.

We click the Create button and then we start the setup of the VM by clicking the Settings button of the toolbar.

Now we go to the General -> Advanced tab and setup the bidirectional clipboard.

Afterward we setup a shared folder. It will be useful to share data between the host and guest systems. In the host system it is seen as a regular folder. In the guest system it is a folder with the same name but living in the /media directory. Before setting up the shared folder it must be created on the host system.
We go to the Shared Folders tab, enter the path of the shared folder and tip the auto-mount check box.

If everything went O.K. it should look like this:

Eventually we select the Storage tab. Click the Add CD button (the small CD with a plus sign picture) and virtually insert the previously downloaded ISO in the CD drive of the VM.

The VM is now created and configured so we can proceed with the Arch Linux installation.

Installing Arch Linux

Now we are ready, on the VirtualBox program click Start on the toolbar and a boot screen will appear, showing you several boot options. Press Enter (i.e. choose the Boot Arch Linux x86_64). After a few seconds you will get terminal with the root user automatically logged on.

The first thing to do if you’re not using an English keyboard is to set the keyboard layout. I’m living in Spain and using a keyboard with Spanish layout so I have to run the command:

# loadkeys /usr/share/kbd/keymaps/i386/qwerty/es

Next you have to partition the virtual hard disk. But first you need to know how your disk is named, so you issue the command lsblk.

In my case the name is sda (I know it because its type is disk and it is 20 GB big). The last thing to do before partitioning is to choose the format of the partition table. You have two options: the classic MBR format and the modern GPT format. In general, if your boot loader is not GRUB legacy and you are not running a multi-boot system with Windows using BIOS, then it is recommended to use the GPT format so we will use it (you can read more about both formats here).
Now that we know the disk name and the partition table format we can issue the proper command to partition the disk, in our case:

# gdisk /dev/sda

gdisk is the GPT version of fdisk. A prompt asks we what we want to do (create new partitions, set partition start and end sectors, etc.). The following screenshot shows an example:

After partitioning the disk the partitions table looks like:

The partition 1 is for installing the GRUB bootloader, the partition 2 is for the / filesystem, the partition 3 is for /boot filesystem and the partition 4 is for the /home filesystem. As you can see we aren’t creating a swap partition. This is because we have a large amount of RAM ans a swap partition will probably not be necessary.

Next we format our partitions with the proper flavors of the mkfs command.

Now we have to create the mount points for the partitions and mount them (beware that we don’t mount the boot partition):

# mkdir /mnt/boot
# mkdir /mnt/home
# mount /dev/sda2 /mnt
# mount /dev/sda3 /mnt/boot
# mount /dev/sda4 /mnt/home

Next step is to test our connection to the Internet using the ping command:

# ping -c 3 www..google.com
PING www.l.google.com (74.125.224.146) 56(84) bytes of data.
64 bytes from 74.125.224.146: icmp_req=1 ttl=50 time=437 ms
64 bytes from 74.125.224.146: icmp_req=2 ttl=50 time=385 ms
64 bytes from 74.125.224.146: icmp_req=3 ttl=50 time=298 ms

--- www.l.google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 298.107/373.642/437.202/57.415 ms


Everything seems O.K. (no packet loss) so we go to the next step, the selection of download mirrors. We edit the /etc/pacman.d/mirrorlist file and select the desired mirrors. Regional mirrors usually work best, but it may be necessary to consider other concerns. In my case I simply selected the first five mirrors in the list (for selecting just uncomment the line containing the server). The less the score is the better the server works.

Now we download from the Internet the base system and install it:

# pacstrap /mnt base

This is the base system so don’t expect a graphical web browser to be installed 🙂

Now we generate the fstab file:

# genfstab -pU /mnt >> /mnt/etc/fstab

At this point we are ready to change root into the system:

# arch-chroot /mnt

The next steps are pretty easy. First we set the hostname and the time zone (in my case Europe, Madrid):

# echo ArchLinuxVM > /etc/hostname
# ln -sf /usr/share/zoneinfo/Europe/Madrid /etc/localtime

Then we have to generate and setup the wanted locales. It is a three steps process:

First we edit the /etc/locale.gen file and uncomment the needed locales (es_ES.UTF-8 in my case)

Second, we generate the required locales:

# locale-gen

And third, we set the locale preferences in the /etc/locale.conf file:

# echo LANG=es_ES.UTF-8 > /etc/locale.conf

Now we set the password for the root user:

# passwd
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully

Now it’s time to install the GRUB bootlader in the boot partition (/dev/sda1 in our case):

# pacman -S grub
# grub-install --target=i386-pc --recheck --debug /dev/sda
# grub-mkconfig -o /boot/grub/grub.cfg

Note that grub-install installs the bootloader to the desired device and copy GRUB images and modules in /boot/grub/i386-pc. In addition grub-mkconfig generate the GRUB configuration file grub.cfg and saves it under /boot/grub.

Once the bootloader has been installed we can reboot the virtual machine:

• – leave the change root environment

  # exit

• – optionally unmount all the partitions

  # umount -R /mnt

• – remove the installation media (i.e. go to the VM VirtualBox top menu and, in the Devices menu, choose CD/DVD devices and remove disk from the virtual drive)
• – issue the reboot command

  # reboot

And that’s enough for today. In the next blog entry we’ll complete the virtual machine configuration (with a permanent setup of the Internet connection, user’s creation, installation of X Window System, etc.).

DNIe no funciona con Windows 8.1

Esta es la primera entrada al blog que escribo en castellano. La razón es que el tema tratado incumbe principalmente a los usuarios del DNIe español (por tanto ciudadanos del Estado español) que quieran usarlo con Windows 8.1.

Se supone que usar el DNIe con Windows 8 es muy fácil porqué este sistema operativo incluye soporte de fábrica del DNIe, así que no hace falta descargar/instalar software, ni módulos criptográficos ni nada de nada. Lo único que tenemos que hacer para usarlo es insertar un lector de tarjetas en el ordenador, insertar el DNIe en el lector y abrir el navegador web, que automáticamente cargará los certificados del DNIe con lo cual cuando accedamos a una página que requiere autenticación con DNIe lo único que tenemos que hacer es introducir el PIN y ya está, acceso conseguido.

Esa es la teoría. En la práctica hay muchos usuarios que tienen problemas para usar el DNIe con Windows 8 (solo hay que navegar un poco para comprobarlo). Por lo que he podido ver los problemas pueden ser de diferentes tipos así que esta entrada solo sera útil para una parte de esos usuarios porqué trata un problema concreto. En mi caso, cada vez que intentaba acceder a alguna página que requiriera autenticación con DNIe, el navegador (que aparentemente tenía los certificados de autenticación y firma bien cargados) no me pedía ni siquiera el PIN. Directamente me presentaba el siguiente diálogo de error:

El mensaje es absurdo , viene a decir que no puedo autenticarme con el DNIe, una tarjeta inteligente uno de cuyos propósitos es precisamente la autenticación. Podemos solucionar este problema si echamos un vistazo a la información sobre el DNIe que nos da el navegador. La siguiente descripción usa el Chrome, pero podemos hacer algo análogo con el IExplorer 10. Vamos al panel de control (o sea pinchamos el botoncito con 3 líneas horizonales de la parte superior derecha) y luego a Settings -> Show advanced settings -> Manage certificates… Al hacerlo se abre el diálogo de Certificados. En la pestaña de certificados personales aparecerán los dos certificados del DNIe. Pinchamos sobre el botón Opciones avanzadas. Al hacerlo se abre otro diálogo que muestra el propósito de los certificados:

Vemos que la opción Autenticación del cliente no esta activada. La activamos, cerramos el navegador, extreamos el DNIe y el lector de tarjetas y reiniciamos el ordenador. Volvemos a insertar el DNIe, abrimos el navegador y tratamos de acceder a una página que requiera autenticación usando el DNIe. Esta vez nos pide el PIN del DNIe y en unos segundos el navegador pasa a cargar la página. Y ya está.

Si usamos el IExplorer 10, con el DNIe insertado en su lector vamos a Settings -> Opciones de Internet -> Contenidos -> Certificados. Con esto abrimos el diálogo de Certificados. Pinchamos el botón Opciones avanzadas y se muestra otra diálogo igual que el de arriba. Nos aseguramos de que la opción Autenticación del cliente esté activada y ya está. Reiniciamos y ya podemos autenticarnos con el DNIe.

Some whitespace pitfalls in Bash programming

Bash is a Unix shell, a command interpreter. It is a layer between the system function calls and the user that provides its own syntax for executing commands. It can work in interactive mode or in batch mode (i.e., via scripts). In interactive mode the Bash prompt, that usually ends with the $ symbol, tells you that Bash is ready and waiting for your commands.

Because the Bash doesn’t have as many features as other programming languages like Java or Python many people suffer from the misconception that it is a sort of simple tool and that takes only a few hours to get proficient with it. Wrong approach. As it happens with a `decent` programming language, if one doesn’t pay attention and learns it properly she will see lots of unexpected errors when running her scripts.

An endless source of errors in Bash programming is the management of whitespaces. In this blog entry we will collect some whitespace pitfalls that usually hit Bash beginners and also developers that only write Bash scripts from time to time (like me).

Commands and arguments

Bash reads commands from its input (a file, a terminal emulator…). In general, each read line is treated as a command, i.e., a instruction to be carried out. Bash divides each line into words at each whitespace (spaces or tabs). The firs word it finds is the name of the command to be executed and the remaining words become arguments to that command.

The above paragraph is IMHO the golden rule that one should never forget if she wants to use Bash in a painless way. It seems trivial (and in fact, it is) but most programming languages are not so strict when using whitespaces so, if someone regularly develops in Perl, Java, C…, it is likely that she tends to use the whitespace in a wrong way when writing Bash scripts. For instance, in Python one can setup a variable as follows:

++++my_var = 4

In fact, it is recommended to delimit the operator with whitespaces for improving readability. However, doing the same thing in Bash will raise an error:

++++$ my_var = 4
++++my_var: command not found

because Bash interprets the above line as: execute the my_var command with arguments = and 4. But the my_var command doesn’t exist so we get an error. The proper way to do the assignment is:

++++$ my_var=4


Parameter expansion

In order to access the data stored in a variable, Bash uses parameter expansion i.e., the replacement of a variable by its value (on expansion time the parameter or its value can be modified in a number of ways but we will only use simple replacements here). The syntax for telling Bash that we want to do a parameter expansion is to prepend the variable name with a $ symbol. Although not always necessary, it is recommended to put curly braces around the variable name:

++++$ echo "my_var = ${my_var}"
++++my_var = 4

Non double-quoted results of expansions are subject to word splitting. Double-quoted results of expansions are not. An because after the replacement Bash can still execute actions on the result (due to the golden rule mentioned above) we have just met a pervasive mistake involving whitespaces: the wrong quotation of results of parameter expansions. For instance, if we want to create an empty file named My favourite quotations.txt then we shouldn’t issue the following command:

++++$ filename="My favourite quotations.txt"
++++$ touch ${filename}

the result of the parameter expansion is not double-quoted so word splitting happens and we execute the touch command with three arguments, My, favourite and quotation.txt. Instead we should double-quote the result of the expansion for avoiding the word splitting after the parameter expansion:

++++$ touch "${filename}"

The above example is trivial and harmless but the same considerations apply to potentially dangerous commands like rm. If we forget the double quotes we may end up removing the wrong files (which can be annoying if we don’t have a backup of those files). When a variable contains whitespaces we have to decide if its expansion needs double quotation or not. The general advice is to put double quotes around every parameter expansion; bugs caused by unquoted parameter expansions are harder to debug than those due to quoted expansions.

Another source of problems related with parameter expansions and whitespaces is the use of metacharacters because unquoted expansions are also subject to globbing. Globbing happens after word splitting. For instance, if in our current directory we have a TODO.txt and a README.txt files, issuing the following similar commands will produce very different results:

++++$ msg="There are *.txt files in this directory"
++++$ echo $msg
++++There are README.txt TODO.txt files in this directory
++++$echo "$msg"
++++There are *.txt files in this directory


Testing conditions

Another common misuse of whitespaces hits many people when they call the [ builtin or the [[ keyword for testing some expression (you can found a nice explanation about the types of Bash commands here). It seems to be very easy to forget that both of them are commands whose arguments are an expression and a closing ] (or ]]). So one can write the following for checking if the file TODO.txt in current directory is indeed a file:

++++$ [-f TODO.txt]
++++[-f: command not found
++++$ [ -f TODO.txt]
++++-bash: [: missing `]'
++++$ [ -f TODO.txt ] && echo OK
++++OK

In the first case we are trying to execute a non-existing command because the [ is not followed with a whitespace. In the second case the last argument passed is not a ] because the ] is not preceded with a whitespace. The third case is fine.

In addition, [ and [[ manage whitespaces in a different way. Parameter expansions inside a [ are subject to word splitting and globbing so quoting those expansions can be critical. However [[ is a keyword and it parses its arguments and does the expansion itself, taking the result as a single argument even if the result contains whitespaces. So if we want to check the content of:

++++$ my_var="I like quotes"

we can do:

++++$ [ "$my_var" = "I like quotes" ] && echo "Me too!"
++++Me too!
++++$ [[ $my_var = "I like quotes" ]] && echo "Me too!"
++++Me too!

[[ has more nice features (pattern matching, support of control operators for combining expressions) that make it more powerful than [ but those features are not directly related with whitespaces so they will not be considered here.

Also notice that many beginners tend to think that the if keyword must be followed by a [ or a [[ just as if must be followed by ( in C and other programming languages. It is not the case in Bash. if must be followed by one or more commands. Those commands can be [ or [[ but they don’t have to.

Grouping commands

The last example of a common mistake related to whitespaces is the use of the one line version of grouping commands. The proper syntax is::

++++{ <LIST>; }

So we have to pay attention to the whitespaces between the curly braces and the list of commands and also to the semicolon after the list of commands or we can get errors like these:

++++$ {rm TODO.txt || echo "Unable to delete file" >&2; }
++++-bash: syntax error near unexpected token `}'
++++$ { rm TODO.txt || echo "Unable to delete file" >&2 }
++++> ^C
++++$

In the first case the closing brace is unexpected because there is no opening brace (instead we have issued a non existing {rm command). In the second case there is no semicolon after the last command. As a consequence Bash thinks that the closing } is an argument of the last command and waits for the user to close the group of commands. We have aborted the unfinished command and come back to the default interactive prompt by pressing Ctrl-C.

Further reading

A couple of great sources of information about Bash are:

• tutorial for newbies: the Bash guide in the Greg’s wiki
• not for newbies: the Bash hackers wiki

Passing arguments to a function: by-value, by-reference, by-object…

When calling a function, the exact mechanism for passing arguments (assigning arguments to parameters) depends on the evaluation strategy and how it is implemented. Some common ways of passing arguments to a function are:

• by value
• by reference
• by object

This is not a theoretical article about passing arguments and doesn’t contain rigorous definitions. I just will try to show clearly how the above mechanisms work on some specific languages widely used: C, C++, Python and Perl.

C

In C, a variable declaration doesn’t create anything, it just ensures that, at runtime, there will be a chunk of memory in the stack big enough to hold the variable value. Keeping that in mind let’s look at the following simple code:

int a = 3;
void update_the_copy(int b) {b = 5;}
update_the_copy(a);
a == 5; // Evaluates to false


In the above example we have two variable declarations, one for a and one more for b. They point to different chunks of memory. When the function is called its arguments are evaluated and a copy of their value is assigned to the function parameters (so b contains a byte-to-byte copy of a). This is called to pass by value. When doing it there is no way to change the arguments from inside the function simply because one has no access to them. The only way to return a value to the caller is via the function’s return value.

Obviously that is memory inefficient when working with large values or nontrivial types. And it doesn’t allow to change in-place variables defined in the caller. C solves the problem by passing (by value) a pointer. In this case the passed value is a memory address i.e.; just a few bytes, so passing the argument is cheap in terms of memory. Although the original pointer and the passed pointer are different entities, they still point to the same memory address so the function can change the pointed data in-place:

int a = 3;
int *pa = &a;
void update_inplace(int *b) {*b = 5;}
update_inplace(pa);
a == 5; // Evaluates to true


C++

C++ supports passing-by-value. However it doesn’t use pointers to deal with the problems of passing a large amount of data or changing values in-place. Instead it uses references. A C++ reference allows you to create a new name for a variable. The reference doesn’t contain a copy of the variable but behind the scenes it keeps around its address. So you can use the reference to read or change the original value stored in that variable:

void swap_values(int& a, int& b) {
++++temp = a;
++++a = b;
++++b = temp;
}

int first = 1;
int second = 2;
swap_values(first, second);


After the swap first will be 2 and second will be 1. Notice that when we call the function we use regular int variables, not references. But, in the function definition, the parameters are references. As a result when the function is called it receives an implicit reference to the variables used as arguments instead of a copy of them. This is called to pass by reference. It allows the function to access the original arguments, no a copy of them. And changes to those values inside the function will be seen outside the function.

Perl

Perl always passes by reference, but not in the same form that C++. The Perl syntax for defining functions doesn’t include a explicit signature so the programmer cannot declare a given parameter as being a reference as she does in C++. Perl passes by reference using an aliasing mechanism: when a function is called, all incoming arguments are flattened into a single list which is then aliased to the implicit variable @_. In this way the function gets implicit references to the arguments:

sub swap_values
{
++++my $temp = $_[0]; # Make a local copy of the first received argument
++++$_[0] = $_[1];
++++$_[1] = $temp;
}

my $first = 1;
my $second = 2;
swap_values($first, $second);

After the swap first will be 2 and second will be 1. Once again, notice that passing by reference doesn’t mean that we use Perl references as arguments, Perl references are another mechanism provided by Perl for modifying values in-place. The important thing is that when the function is called it receives implicit references to the arguments. Therefore we are passing by reference. Also notice that the flattening of arguments can be tricky (although this quirk has nothing to do with the fact that Perl passes by reference. If you are interested, you can find nice details about flattening here ).

To the end to simulate pass-by-value one needs to do explicit copies of the arguments inside the function:

my $name = "Chuck";
sub reverse_name
{
my $name = shift; # a local copy of the argument
$name = reverse $name; # change the local copy
}

reverse_name($name);
print $name; # output "Chuck"


Python

Things are different in Python. All Python values are objects (so they are first-class entities) that can be tagged with one or more names or can have no name at all. Variable names are just names (you can see it graphically here). As a result talking about pass-by-value or pass-by-reference makes non sense in Python (unless one consider that CPython is implemented in C. But I will stay at Python level in this section).

So how does Python pass arguments to a function? It passes by object. Let’s see what does it means with a couple of examples::

obj1 = 3
def simulate_pass_by_value(obj2):
++++obj2 = 5

simulate_pass_by_value(obj1)
obj1 == 5 # Evaluates to false

We get the same result as passing by value. It is not an unexpected result if one remember what we have said about object names at the beginning of this section. The object named obj1 is passed itself to the function. When doing so a new tag, obj2, is added to it. Now the object has two names, obj1 and obj2. The effect of the assignment inside the function is to move the obj2 name to a new object whose value is 5. When the function ends the object named obj2 goes away and the passed object remains untouched and can still be accessed with the obj1 name.

In the following example we pass a mutable object to a function in order to simulate a pass by reference:

def simulate_pass_by_reference(list2):
++++list2.append(2)

list1 = [1]
simulate_pass_by_reference(list1)
list1 # Evaluates to [1, 2]

Now we get the same result as passing by reference i.e; the argument is changed in-place. The object named list1 is passed to the function and a new tag, list2, is attached to it. In this case the new name is never reassigned to a new value. Instead, a method call modifies the object in-place. When the function ends the list2 tag goes away but the modified object is still accessible via its list1 tag.

In summary

• C always passes by-value. Functions can use pointer arguments to change caller variables in-place.
• C++ supports both, pass-by-value and pass-by-reference.
• Perl always passes by-reference. Functions can do local copies of its arguments in order to simulate passing by-value.
• Python passes by-object. Functions can simulate passing by-value and passing by-reference.

QHeaderView sections: visualIndex vs logicalIndex

Headers provided by QHeaderView are made of sections which can be referred to by using indexes. There are two kind of indexes, visual and logical. I think the reference documentation is somewhat confusing when explaining what those indexes are and how they differ from each other. This blog entry is the result of trying to clarify those issues to myself.

Visual indexes tell us the current position that a given section occupies in the header. So let’s suposse we create a five columns table and that we label the sections of its horizontal header as C0, C1, C2, C3 and C4 (the C being a shorthand for column). The visual indexes of these sections are showed in the next figure:

	C0	C1	C2	C3	C4
Visual Index	0	1	2	3	4

By definition, the visual index of a given section will change if the section is moved to a different position. For example, if we move the C3 section to the second position of the header (via QHeaderView.moveSection(3, 1)) then the position of sections C1, C2 and C3 will change. As a result also their visual indexes will change:

	C0	C3	C1	C2	C4
Visual Index	0	1	2	3	4

As we can see even after moving around some sections the set of visual indexes remains ordered as 0, 1, 2… N. So far so good.

However there is a detail one has to be aware of: hiding/showing a given section (calling QHeaderView.hideSection method) does NOT change its visual index as one could expect. So if in our current header we hide the section C0 no visual index will change:

	C3	C1	C2	C4
Visual Index	1	2	3	4

Time to talk about logical indexes. For a given section the value of its logical index depends on the position at which it was added to the header. So when our sample table was just created we had:

	C0	C1	C2	C3	C4
Logical Index	0	1	2	3	4

because C0 was added at position 0, C1 was added at position 1 and so on. Moving sections around doesn’t change any logical index so if we move the C3 section to the second position we will have:

	C0	C3	C1	C2	C4
Logical Index	0	3	1	2	4

But inserting/removing sections (which can be done inserting/removing columns in our table model) will change the logical indexes of the implied sections. For instance, if now we insert a new column, labeled as NC for the sake of clarity, at position 2 we will have:

	C0	C3	NC	C1	C2	C4
Logical Index	0	4	2	1	3	5

The sections C0 and C1 had a logical index lower than 2 so they will keep their logical index unchanged but sections C2, C3 and C4 had a logical index greater than or equal to the logical index of the new section. As a result their logical index will be increased by one.

Hopefully you have now a better understanding of what visual and logical indexes are and will be able to use them without being puzzled by methods like logicalIndex, logicalIndexAt, visualIndex and visualIndexAt provided by QHeaderView.

About the .htaccess file

Many of you have a fair idea of what an .htaccess file is and what can be done with it. In my case that knowledge was not very accurate so I decided to learn more and to write this brief introduction to the subject.

.htaccess files are hidden text files that contain Apache directives. They are hidden for most operating systems because of the dot at the start of its filename. And they are also hidden on the web because most Apache web servers are configured to ignore them.

.htaccess files provide a way to make configuration changes on a per-directory basis as opposite to a per-server basis. It means that, when an .htaccess file is placed in a particular document directory, the configuration directives contained in the file apply to that directory and all its sub-directories, overriding the setup of those directives in the main configuration files. We have to pay attention to this recursive mechanism because a given .htaccess can override some behavior defined in a different .htaccess placed higher in the directory tree hierarchy.

We said that normally .htaccess files are ignored by the web servers. Why does it happen? Well, .htaccess files are not recommended for several reasons, mainly performance and security.

First, everything that can be done with an .htaccess file can also be done with the main configuration files of the server so, in principle, is not a good idea to put .htaccess files here and there because it makes more difficult to know the real configuration of the server.

Second, if they are enabled, every time the server receives a request, it will look for .htaccess files in every requested directory and its ancestors, and will process the found files in order to know which directives it must override. No caching mechanism here, it happens every time a request is received. In contrast, the main configuration has to be loaded just once. Even worse, if you are using RewriteRule directives in your .htaccess file, then the regex are re-compiled with every request to the directory (the main server is much more efficient as it compiles the regex just once and caches them).

So in general .htaccess files should be avoided: anything that goes into an .htaccess file can go into the main configuration files (using the same syntax in most cases) and performs worse… but they exist for a reason.

A typical case in which .htaccess files are used is that of an ISP hosting multiple sites on a single machine. If the server administrator wants to give users (i.e., content providers) the possibility of changing the configuration of their site without having access to the main configuration files, then .htaccess files are the way to go (of course, it implies security risks because people other than the service administrator will be able to change part of the server configuration). Also many popular CMSs like WordPress, Joomla or Drupal use .htaccess files.

Just one more thing. In order to use .htaccess files, the AllowOverride directive must be set to something different than None. This directive determines which directives are allowed in the file so we have to setup it accordingly to our needs. If AllowOverride is set to None then the .htaccess files are not even read by the web server.

That’s all about it. I’m not going to talk here about all the sorcery and tricky things that you can do using .htaccess files (you can do the same than with the main configuration files and using the very same syntax in most cases). In future posts I’ll talk about using the mod_rewrite module in .htaccess files and how it differs from using that module in the main configuration files.

C loops: for() and the comma operator

Everybody knows that in C a for loop has the syntax:

for (variable initialization; exit condition; variable update) {
++++// the body comes here
}


a typical example being:

int i;
for (i = 0; i < 5; i++) {
++++printf("%i ", i);
}


However, all three expressions in the for statement are optional. Even the body is optional. Due to this flexibility when one starts to read C code written by other people it is easy to find for loops with very different looks. In general, those variations are easy to understand. For instance, the above loop can also be written as follows:

int i = 0;
for(; i < 5; i++) {
++++printf("%i ", i);
}


so the variable initialization has been done before the loop is called. If the exit condition is absent then it always evaluates to True. And if the update variable expression is omitted then it is done in the loop body. If all three expressions have been removed from the for statement then we have a potentially infinite loop and we should insert a break statement somewhere in the body.

Things get interesting when the for statement is combined with the infrequently used comma operator.

The comma operator is a sequence point (as they are && and ||) so the order of evaluation of the operands is fixed. It is a binary operator that evaluates its first operand, performs all side effects and discards the result, and then evaluates the second operand and returns its value and type so:

x = (y, z);


will do y and, after performing all side effects, will discard it, then do z and finally will set x to z.

Because the comma operator discards its first operand, it is useful where the first operand has desirable side effects, such as in the initializer or the counting expression of a for loop.

We can use that operator to combine the variable update and the exit condition expressions of the for statement in a single expression:

int i=-1;
for (;i=i+1,i<5;) {
++++printf("%i ", i);
}


As a result, at each iteration the counter is incremented and discarded, then exit condition is evaluated and its result returned so the value of the comma expression is the value of the exit condition.

The equivalent while loop is:

int i=-1;
i = i + 1;
while(i < 5) {
++++printf("%i ", i);
++++i = i + 1;
}


In the following example the exit condition works as in the previous one. In addition, two variables are set in the variable initialization expression. And the update variable expression is used to append statements to the body of the loop:

int i, j;
for (i=-1,j=0;i=i+1,i<5;j=i+1,printf("%i\n", j)){
++++printf("%i\t", i);
}


It is equivalent to the following while loop:

int i = -1;
int j = 0;
i = i + 1;
while(i<5) {
++++printf("%i\t", i);
++++j = i + 1;
++++printf("%i\n", j);
++++i = i + 1;
}


Notice that the examples above are intentionally simple. They just pretend to show you how the combination for statement + comma operator works. In general, comma operator is used to produce side effects in the following situations:

• calling a function
• entering or repeating an iteration loop
• testing a condition

You can see also:

• this SO thread about the use of comma operator with for loops
• a brief explanation of comma expressions with a nice summary

Setup an Android development environment on Ubuntu

Recently I’ve been involved in a project for developing an Android smartphone application. This is a new working field for me as I had never developed applications for mobile devices before. So it requires an important extra effort on my side (tons of things to learn). As I always do when I find myself turned into a newbie I’ve started to read documentation… and I’ve setup my development environment. For doing the setup I’ve followed the instructions found here. In theory it is an easy process. In practice it can be a little bit complicated so I decided to write this post. The setup described in this post has been tested on my Kubuntu Oneiric laptop.

First of all I’ve installed the openJDK implementation of the Java6 SDK:

$ sudo apt-get install openjdk

Other Java implementations have been discarded due to different reasons:

• Java7 is not supported by Android
• Java Oracle packages are not available on Ubuntu Oneiric official/partner repositories
• the GNU Java compiler is not compatible with the Eclipse IDE so it is not an option if you plan to develop with Eclipse

The recommended IDE for developing Android applications is Eclipse because there is a plugin for integrating the Android SDK with it. The Ubuntu Eclipse package uses openJDK by default but it depends on the GNU Java compiler which, as I said, is not compatible with the Android SDK so I don’t now if it is a good idea to install Eclipse from the Ubuntu repos. Just in case I’ve downloaded Eclipse Classic (the version recommended by Android) from its website and installed it on the /opt/ folder. Installing Eclipse is trivial, just untar it and add the eclipse folder to your PATH.

Next I’ve installed the Android SDK Starter Package under /opt/. Again, the installation is trivial, just untar the package and add the android-sdk-yourplatform/tools and android-sdk-yourplatform/platform-tools folders to your PATH.

Once the Starter Package is installed one should execute the command

$ android &

which launches the Android SDK Manager, a tool included in the Starter Package. It is a graphical program, with a simple UI, that allows you to setup your SDK by downloading the essential packages for your development environment. In my case I’ve installed the following packages:

• SDK Tools (latest version is required)
• SDK Platform-tools (latest version is required)
• SDK Platform (latest one is recommended)
• Documentation for Android SDK

Additional packages that I’ve installed include the Google API, SDK Samples and the sources for Android SDK.

If you plan to publish your application, you will want to download additional SDK platforms corresponding to the Android platform versions on which you want the application to run.

Downloading those packages is sometimes a very slow process. If this problem hits you just cancel the installation and try again later (it is a simple workaround but it worked for me).

The last step is to install the Android Development Tool plugin for Eclipse. It must be done using the Update Manager feature of Eclipse as described here. The plugin configuration is very easy, just follow the wizard steps. At the end you will have an Android toolbar on your Eclipse main window. This new toolbar will contain buttons for launching the Android SDK Manager, managing Android Virtual Devices, etc.

Using this plugin is not mandatory but it seems to be highly recommended. If you don’t want to use it then you aren’t forced to use the Eclipse IDE.

That’s all. Now I’ve to see how it works and decide if I like it or I prefer to look for alternative environments. If you’re writing Android apps with a different development environment (for instance, not using Eclipse or not using an IDE at all) please, leave a comment.