Blog of spblinux

Steps to get  dovecot 2.1.17 on qnap ts 119p II NAS (firmware 3.8.1 or 4.x) up and running:

dovecot has to be compiled from source:

• qnap web admin: qpkg center: available / more / Optware IPKG: install and activate it. Or with newer firmware, 4.x, follow these instructions (using HDA_DATA instead of MD0_DATA and replacing cs05q3armel = gcc-3.4.4 by cs08q1armel = gcc-4.2.3).
• enable ssh on qnap and connect as user admin
• cd /opt
• ./Optware.sh start. Or with newer firmware:
  PATH=/opt/bin:/opt/sbin:$PATH
• ipkg update
• ipkg install mc
  (my preferred editor and file manager)
• ipkg install libidn
  ipkg install optware-devel
  (pulls in many tools used for compilation of packages)
• mkdir src
  cd src
  wget http://www.dovecot.org/releases/2.1/dovecot-2.1.17.tar.gz
  tar -xzf dovecot-2.1.17.tar.gz
  cd dovecot-2.1.17
  (download dovecot source)
• export PATH=/opt/bin:/opt/sbin:$PATH
  (compilation might fail if tools in /bin or /usr/bin are used instead of the ones in /opt/bin)
  ./configure –prefix=/opt –with-storages=maildir
  make
  make install
  (if configure does not find a required library foo the run ipkg list |grep foo and install libfoo-dev with ipkg install)
• child processes of dovecot use /lib and ignore /opt/lib
  (maybe this can be avoided with CPPFLAGS=”-I/opt/include” LDFLAGS=”-L/opt/lib -ldl” ./configure …)
  qnap firmware 3.8.1 installs libssl 0.97 in /lib but ipkg installs libssl 0.98 in /opt/lib; workaround:
  ln -s /opt/lib/libssl.so.0.9.8 /lib/
  ln -s /opt/libcrypto.so.0.9.8 /lib/
  (to be moved into dovecot startup script)
• mkdir /opt/var/log
  addgroup dovenull
  addgroup dovecot
  adduser -D -H -G dovenull dovenull
  adduser -D -H _G dovecot dovecot
• cd /opt/etc
  mkdir dovecot
  mkdir dovecot/conf.d
  (now copy config files from /opt/share/doc/dovecot/example-config to /opt/etc/dovecot, e.g. using mc)
  dovecot.conf: protocols = imap and listen = *
  10-auth.conf: disable_plaintext_auth = no
  10-logging.conf: log_path = /opt/var/log/dovecot.log
  10-mail.conf: mail_location = maildir:/share/dovecot/%n
  10-master.conf: unchanged
  10-ssl.conf: ssl = no and #ssl_cert… and #ssl_key… (disable ssl; might be enabled in a later step)
  20-imap.conf: unchanged
  auth-system.conf.ext: comment out the section driver = pam and uncomment the section driver = shadow
  and add args = blocking=no
  newer firmware 4.x:
  do not use blocking=no because /etc/shadow needs a worker thread which is allowed to access /etc/shadow; so add to auth-system.conf.ext:
  service auth-worker {
  group = administrators
  }
  (mail users will be created with qnap web interface)
• from qnap web interface: access control
  user-groups: create group mail
  shared folders: add folder dovecot (use automatic path and do not share it)
  shared folders: permissions for folder dovecot:
  user: create the mail user yourname with password  and add it to group mail
• start dovecot manually
  dovecot -c /opt/etc/dovecot/dovecot.conf
  and test it with telnet as described in the dovecot wiki (with mail user yourname)
  finally try to connect with a mail client like thunderbird
• start and stop dovecot automatically: there are optware start/stop scripts in /opt/etc/init.d
  but the qnap firmware (currently 3.8.1) does not execute them.
  – mkdir /tmp/config
  – mount /dev/mtd5 /tmp/config
  firmware 4.x: use /dev/mtdblock5
  – add executable file autorun.sh to /tmp/config:
  #!/bin/sh
  #
  for f in /tmp/config/K*.sh; do
  [ -x $f ] && $f start &
  [ -x $f ] && cp $f /etc/rcK_init.d/
  done
  – enable autorun.sh with
  /sbin/setcfg Misc Autorun TRUE
  /sbin/setcfg Misc Autorun TRUE -f /tmp/config/uLinux.conf
  (reference /etc/rcS.d/S98init_nas)
  – copy K010optstartstop.sh to /tmp/config (with extension ‘.sh’, not with extension.txt; partly found here and here)
  – chmod +x /tmp/config/K010optstartstop.sh
  – umount /tmp/config
  this will execute   start/stop scripts in /opt/etc/init.d.
• /opt/etc/init.d/S59dovecot may contain
  #!/bin/sh
  if [ -f /opt/var/run/dovecot/master.pid ] ; then
  kill `cat /opt/var/run/dovecot/master.pid`
  else
  killall dovecot 2>/dev/null
  fi
  rm -f /opt/var/run/dovecot/master.pid
  ln -s /opt/lib/libssl.so.0.9.8 /lib/
  ln -s /opt/libcrypto.so.0.9.8 /lib/
  /opt/sbin/dovecot -c /opt/etc/dovecot/dovecot.conf
• /opt/etc/init.d/K41dovecot may contain
  #!/bin/sh
  if [ -f /opt/var/run/dovecot/master.pid ] ; then
  kill `cat /opt/var/run/dovecot/master.pid`
  else
  killall dovecot 2>/dev/null
  fi
  rm -f /opt/var/run/dovecot/master.pid
  or simply
  #!/bin/sh
  /opt/bin/doveadm stop
• both scripts have to be executable
  chmod +x /opt/etc/init.d/S59dovecot
  chmod +x /opt/etc/init.d/K41dovecot

To be continued with py25-getmail to collect mails from pop3 servers
(getmail delivers mails directly into the maildir folders used by dovecot)

• ipkg install py25-getmail
• instructions for getmail configuration (not written for qnap)
• …

Preparing a NAS for backup with rsync I wanted to copy initial data with nfs because of higher transfer speed.

The NAS nfs export mounts fine on a linux box but not from osx 10.5.8. Some search revealed that

• /etc/hosts on the NAS should have an entry for the osx machine [1]
  (e.g. 1.2.3.9 my_osx_box)
• instead of mount 1.2.3.4:/share /mnt use mount_nfs -P 1.2.3.4:/share /mnt [2]
  (or add insecure to /etc/exports on the NAS)

If you have a triple boot mac mini with OS X, Windows and Linux it might be useful to run OS X inside VirtualBox when Linux or Windows is running.

VirtualBox officially supports OS X Server as guest vm and runs as well other OS X versions as guest. – If Linux runs on a second (external) hard disk the first harddisk can be assigned to a virtualbox vm as raw device. And OS X should boot as guest (of type OS X server). However booting a vm with EFI instead of BIOS is still work in progress. With virtualbox 3.2.12:

• limited screen resolutions
  (VBoxManage setextradata "VM name" VBoxInternal2/EfiGopMode N, N=0,1,2...4)

• only first ide hard disk is bootable
• overrides installed boot manager, e.g. rEFIt; and settings done by OS X start volume utility are completely ignored (missing virtualized hardware)
• sometimes crashes in libata (ide disk access); maybe not related to efi boot

EFI boot can be disabled in system settings of the guest vm. But then you need a BIOS bootloader for the OS X kernel like Chameleon which seems to be a very active project.

• chameleon could be installed to the disk containing OS X – but I prefer an unmodified disk
• so chameleon should boot from cdrom

There are many cdrom iso images with chameleon online, but none met my needs:

• boot leopard and snow leopard
• boot from apple hardware but inside virtualbox
• do not use hackintosh kext patches, except disabling native apple power management with Disabler.kext (else the guest uses 100% cpu)
• screen resolution of guest vm be be selected in boot menu

The chameleon boot loader (2.0 RC4 r684) has lots of usage setups. But many of them are only documented by their source code.

• the stage 2 loader of chameleon, boot, is in multiboot format
• it can be chain loaded by syslinux mboot.c32
• mboot.c in chameleon source reveals 3 command line arguments:
  biosdev=80 (or 81 for second hard disk, …)
  partno=2 (for second partition)
  timeout=5 (autostart after 5 seconds)
• multi boot “kernels” may have “modules” (=initial ramdisk in case of linux kernel); chameleon uses a dmg image whith boot parameters and kext kernel modules
  – dmg format: compressed, with mbr partition, hfs+ filesystem in partition 1
  – /Extra/com.apple.Boot.plist defines boot parameters
  – /Extra/Extensions/ contains kext modules
• isolinux.cfg example:
  kernel /mboot.c32
append biosdev=80 timeout=1 partno=2 /boot --- /p800x600.dmg
• com.apple.Boot.plist entry for Display resolution
  <key>Graphics Mode</key>
<string>1280x1024x32</string>
• com.apple.Boot.plist entry for verbose mode
  <key>Kernel Flags</key>
<string>-v</string>
• the dmg image can be opened inside OS X

Based on this information osxboot.iso is a bootable cdrom image with

• boot menu (syslinux based) to choose between
  800×600, 1024×768, 1280×1024
  and, if set as custom resoltions in virtualbox guest vm,
  1400×1050, 1440×900, 1920×1050, 1920×1200
  (VBoxManage setextradata "vmname" CustomVideoMode1 1920x1200x32)
• boot menu to select silent or verbose boot
  (chameleon allows to add other flags)
• default boot from second partition of first hard disk
  (which can be edited during boot)
• OS X 10.5, leopard, or OS X 10.6, snow leopard, on apple hardware, running as virtualbox guest:
  – only kext module Disabler.kext (part of chameleon distribution) is loaded (allows cpu sleep mode for vm guest)
  – leopard runs only inside a 64 bit version of virtualbox (tested with linux)
• EFI has to be disabled in the system properties of the virtualbox guest
• Seems to run stable if the hard disk inside the vm is attached to the virtual SATA hard disk controller
  (to access raw devices Use Host I/O Cache must be enabled for the SATA controller!)

The chameleon boot loader has been patched to set display resolution and verbosity as multiboot arguments:

• resx=800 (defines horizontal resolution)
• resy=600 (defines vertical resolution)
• verbosity=0 / verbosity=1 (sets verbosity: -v flag)

Google, howto isolinux, should give answers how a bootable isolinux cdrom image can be created.

Known bug: “About this Mac” system information crashes.

Software used for this article: SPBLinux 2.4 live cd

In case you want to access usb attached storage media from a virtual machine (vm) running inside virtualbox there are 3 choices:

1. Make the usb connection available to the vm.
2. Make a shared folder to access data on the usb storage device.
3. Give the vm direct access to the usb disk.

Using a shared folder has the advantage that configuration is straight forward (menu Device/Shared folder …). But

• the host must handle the filesystem used on the usb storage device
• the guest must have guest additions installed
• performance is a bottleneck

Usb inside guest also has performance drawbacks.

So it is worth to fiddle with the command line to get direct access to disk partitions.

• ls /dev/disk*
• this displays all disk partions seen by OSX
• in case finder displays the partition to be used by the vm (that is, OSX has automatically mounted this partition) you have to deactivate this partition with the OSX disk utility program
• VBoxManage internalcommands createrawvmdk -filename yourpath/newdisk1.vmdk -rawdisk /dev/disk1
• this creates a vmdk file which can be opened by virtualbox disk manager; this file gives access to disk1
  Use host I/o-Cache has to be enabled for the controller of this disk!

If the vm cannot be started because of disk access error the partitions of disk1 have to be deactivated again with the hard disk utility program.

The system disk /dev/disk0 would need root access rights; other disks can be accessed as user.

Time to put things together. Xen orchestra has now got a ‘vnc‘ link (like http://localhost/xo/htdocs/vnc.php?url=127.0.0.1:5901&name=w7b) which opens gtk vnc plugin (amd64 binary here). – This link is placed in section Misc of xen orchestra’s virtual machine detail window.

The configuration file of xen orchestra, xen-orchestra.conf, has got a section to control gtk vnc plugin. If a windows vm needs mouse grabbing, it can be set in this file. The handling of these settings is done in htdocs/vnc.php. If virtual machine w7b needs even more specific settings an additional file vnc_w7b.php can be used.

Compared to the description in this post vnc gtk plugin does now automatically grab the focus (keyboard immediately active after loading of html page; no need to click into vnc viewer window first) and a send keys menu has been added (0103-add_autofocus_and_sendkeys.diff).

(Build instructions remain the same except that after (updated) colortoolbar.patch.gz a second patch colortoolbar2.patch.gz has to be applied; updated amd64 debian archive: mozilla-gtk-vnc_0.3.10-6_amd64.deb).

There is one little additional patch to xen orchestra (03-add_vm_name_to_vnc_link.diff). The main changes are in vnc.php (compare previous post about xen orchestra). An updated zip archive with patched xen orchestra is here.

Xen virtual machines transmit their screen contents by vnc. A browser based viewer is the java vnc viewer plugin (see previous post). Advantage of java is that it runs on many platforms. However java tends to use a lot of RAM. On ubuntu 10.04 there is an alternative browser plugin which is gtk based: gtk viewer plugin.

The gtk vnc plugin offers only very basic functions: just an authentication dialog and the vnc viewer window. – This post adds a minimalistic toolbar to the plugin (test.html, gtk-vnc-plugin.so, libgtk-vnc-1.0.so.0.0.1).

Steps to build the plugin described above:

• Start with debian sid source package of gtk-vnc (local copies: gtk-vnc_0.3.10.orig.tar.gz, gtk-vnc_0.3.10-6.diff.gz, gtk-vnc_0.3.10-6.dsc)
• Download color toolbar patches:colortoolbar.patch.gz, colortoolbar2.patch.gz
• Inside download directory run
  tar -xzf gtk-vnc_0.3.10.orig.tar.gz
  zcat gtk-vnc_0.3.10-6.diff.gz |patch -p0
  zcat colortoolbar.patch.gz |patch -p0
  zcat colortoolbar2.patch.gz |patch -p0     (updated: more)
  cd gtk-vnc-0.3.10
  chmod +x debian/rules
  ./debian/rules binary
• Result: gtk-vnc-plugin.so and libgtk-vnc-1.0.so.0.0.1
• but it runs only with libgtk-vnc from debian lenny gtk-vnc library (using gtk-vnc-3.8: libgtk-vnc-1.0.so.0.0.1) – gtk-vnc-3.10 has introduced some incompatibility with gtk vnc plugin (does not display hvm virtual machines after second connection); this is solved by patch 0102-libgtk-vnc_revert_to_v3.8_format_selection.diff (which is included in colortoolbar.patch.gz)
• the deb archives created by debian/rules binary have to be installed with
  dpkg -i libgtk-vnc-1.0-0_0.3.10-6_amd64.deb mozilla-gtk-vnc_0.3.10-6_amd64.deb
• to protect these manually installed packages they should be protected in synaptic with menu Package/Force version
• amd64 versions of the created deb archives can be found here:
  libgtk-vnc-1.0-0_0.3.10-6_amd64.deb mozilla-gtk-vnc_0.3.10-6_amd64.deb python-gtk-vnc_0.3.10-6_amd64.deb libgtk-vnc-1.0-dev_0.3.10-6_amd64.deb gvncviewer_0.3.10-6_amd64.deb libgtk-vnc-1.0-0-dbg_0.3.10-6_amd64.deb

Looking for a graphical, browser based tool to manage xen 4.0 virtual machines I tried xen orchestra):

• pro: needs only a web server with php (mysql optional); user interface not overloaded;  open source, well structured code.
• con: only one detail window can be opened at the same time – needs patch; needs on ubuntu 10.04 a workaround for command xm new (see previous post).

The patch for multiple detail windows just had to replace some html tag id fields in htdocs/scripts/xo.js (01-xo.js.wId.diff).

Because I found it convenient to inspect my virtual machines with a browser based vnc viewer, I added a line to the detail window which displays the id and a link to vnc:
Domain id: 1 (vnc)
This patch uses the xen api to query the consoles of the virtual machine and looks up the ip address and the port of the vnc server connected to the console (02-add_vnc_link.diff, to be applied after 01-xo.js.wId.diff).

Patched version of xen orchestra: xen-orchestra-2010-03-22-patched.zip (local version of original file: xen-orchestra-2010-03-22.zip; local version of tightvnc java viewer: VncViewer.jar)

Minor, negligible bugs:

• Network and Storage section of detail windows are empty.
• If the machine running xen orchestra is too busy, xen orchestra displays an empty window; just press the refresh button of the browser to get the main window back.
• ‘Login again’ button of java vnc viewer window does not reconnect (but refresh button of browser does).
• Error messages are not displayed if a virtual machine fails to start (open a terminal and use the command xm start yourvm).
• the vnc viewer window has a fixed hardcoded size (800×600 in htdocs/vnv.php); as workaround add &w=1024&h=798 to the browser address line to display a 1024×768 virtual machine.
• non captured mouse does not work well in windows, unless graphics tablet mouse driver is installed (see next post about gtk-viewer plugin on linux).

Conclusion: I really appreciate the work of Olivier Lambert, the author of xen orchestra!

Virtual machines of xen can be created with xen-builtin command xm create which uses a config file in /etc/xen. There is a second command to create xen virtual machines: xm new

Virtual machines created with xm new are permanently added to xen (virtual machines created with xm create get removed from xen when the virtual machine is shut down). – Some xen monitoring tools, like xen orchestra, rely on permanently added virtual machines.

But xm new is broken in debian (and ubuntu) because xm uses an outdated xml python library. And the corresponding debian  bug report is closed with status ‘won’t fix’. Debian maintainers support  only xen create.

Using virsh as workaround for xen new:

• xm create myvm (from file /etc/xen/myvm)
• virsh dumpxml myvm > /etc/xen/myvm.xml
• xm shutdown myvm
• virsh define /etc/xen/myvm.xml

Debian sid has the source package of libvirt (currently 0.8.1-1), but package 0.8.1-1 does not work with xen 4.0.  However the build service of opensuse factory includes as well libvirt-0.8.1 and they have xen 4.0 patches included.

To get a libvirt package for ubuntu 10.04 which is compatible with xen 4.0 the patches of opensuse had to be combined with the debian sid package (patch: debian.diff; local copy of debian sources: libvirt_0.8.1.orig.tar.gz, libvirt_0.8.1-1.debian.tar.gz, libvirt_0.8.1-1.dsc).

Build process:

• download sources and patch into new directory and open terminal in this directory
• tar -xzf libvirt_0.8.1.orig.tar.gz
• cd libvirt-0.8.1
• tar -xzf ../libvirt_0.8.1-1.debian.tar.gz
• patch -p0 < ../debian.diff
• ./debian/rules binary

And to install the new packages (ubuntu amd64 packages: libvirt-bin_0.8.1-1_amd64.deb, libvirt0_0.8.1-1_amd64.deb, libvirt-dev_0.8.1-1_amd64.deb, libvirt-doc_0.8.1-1_amd64.deb, python-libvirt_0.8.1-1_amd64.deb, libvirt0-dbg_0.8.1-1_amd64.deb):

• cd ..
• dpkg -i libvirt0_0.8.1-1_amd64.deb libvirt-bin_0.8.1-1_amd64.deb python-libvirt_0.8.1-1_amd64.deb

The resulting virsh command seems to work with xen 4.0 (tested with subcommands dumpxml and define). – You should use Package/Force version in synaptic package manager to protect these manually installed packages.

• Adding and removing of hard disks  and cdrom devices works fine
  (guest not running)
• Acronis Trueimage works well inside hvm virtual machines
  (if empty block devices are not recognized as hard disks, e.g. /dev/sdb7:
  dd if=/dev/zero of=/dev/sdb7 bs=1M count=100
  echo ‘w’ |fdisk /dev/sdb7
  This zeros out the beginning of block device /dev/sdb7 and writes a new partition table into /dev/sdb7; all existing data in /dev/sdb7 is overwritten – you have been warned.)
• Sometimes the window opened by virt-manager fails to display a running virtual machine (does not connect by vnc to the running vm).
  Closing both windows (that of the vm and virt-manager itself) and reopening them usually helps (the vm continues to run).

Found in xen wiki a remark of Jeremy Fitzhardinge about the performance of Xen 4.0 (Xen Summit at Intel Shanghai Nov 19-20, 2009, Slide 7):

Performance still needs attention

• Seems to be ~10-20% hit vs 2.6.18-xen

… but you get a current 2.6.31/32/… Linux kernel.

Preparations to boot a Xen 4.0 dom0  kernel: 2.6.18-to-2.6.31-and-higher

Status of Xen features in Linux kernel: XenParavirtOps