View source

by [mailto:whitpa@users.sourceforge.net Paul Whittaker]

== Introduction ==


I recently acquired a Thecus N4100+ 4-bay NAS appliance for approx US$600 (which considering its poor performance, was probably a mistake - anyone considering buying a Thecus NAS should one of the x86-based models nstead). I bought it both for practical purposes (I needed a RAID-5 mass storage system) and because I was keen to hack the O/S and have a look at a real commercial embedded Linux O/S in action.

The contents of the Thecus firmware can be inspected either by installing an SSH module (not possible on the original N4100) and then logging in as root and investigating the live system, or (if you don't own any Thecus hardware, have an N4100, or intend to make minor changes to the official firmware) by using [http://svn.debian.org/wsvn/d-i/trunk/installer/doc/devel/hardware/arm/n2100/firmware?op=file&rev=0&sc=0 this information] to decrypt and disassemble a downloaded firmware blob. Curiously, the i4500R firmware is just an ordinary tarball and is not encrypted in this way. The firmware tarball contains an installation script that shows how the different components of the bundle (kernel, initrd etc) are written to the MTD - it's pretty simple, basically just "fcp -v filename /dev/mtdN". Presumably fcp is essentially the same thing as flashcp (mtd-utils).

The O/S on the appliance, which is an ARM v5 (XScale) based system, is not dissimilar to DIET-PC in terms of size and operational paradigm. Like DIET-PC, it uses a non-terminating initrd, and is busybox- and glibc-based. The 2.6.20 kernel zImage (~1.5 MB compressed) and gzipped ext2 initrd (~13 MB compressed) are stored in separate partitions on a 16 MB MTD device. Additionally, three read-only filesystems (/opt, /usr/lib and /usr/share/zoneinfo) are mounted from files in the initrd using a compressed loopback (cloop) driver from the Knoppix project, and /app (a misnomer, because most of the apps live in /opt) is mounted as a persistent read-write JFFS2 filesystem. All persistent O/S state information is in /app; /etc is a symlink to /app/etc. The boot loader is RedBoot, which is operationally similar to (although generally less functional than) Micromonitor, which I am already familiar with from my work on ThinLinx Hot-e appliances.

The O/S is similar to Netstation/Thinstation/PXES in that it is variable-driven (although it stores variables in a SQLlite database instead of shell script flat files) and has non-standard directory layout with heavy use of symlinks. Not much effort has been made to miniaturise anything (it uses some fat software, such as Samba 3.0.20b, Apache 2.0.55, Netatalk 2.1dev). The shell scripts and overall design suggest Linux competence rather than Linux expertise; there is quite a lot of "dead code" (code that is only pertinent to other Thecus models and serves no purpose on the N4100+), and a fair few unnecessary inclusions where it seems that the developer had a less-than-perfect understanding of what was required and was erring on the side of caution. The Thecus developers have thought a bit about expandability and upgradeability, though, and have developed APIs (albeit rather convoluted ones) for "third party plugin modules" and firmware upgrades.

The Thecus firmware is okay, but not great. It's pretty good functionally (aside from kernel performance problems that Thecus can do little about), but the underlying O/S is a bit primitive. It looks more like a Linux 2.4 userspace than a 2.6 one. There's no reason for the old glibc, the layout is chaotic, it desperately needs udev, and it could benefit from newer techniques like SquashFS, UnionFS, ARM EABI, and iPKG. As DIET-PC 3 is being designed with all of these things in mind, I am keen to see whether I can devise a drop-in replacement for the official firmware. It would provide a good test case for DIET-PC 3, provide more hardware flexibility for Linux hackers, and maybe squeeze a bit more performance out of the hardware.


== N4100+ performance, or lack thereof ==


For those who'd like some real-world figures, the best FTP throughput I was able to get out of my 4x500 GB RAID5 using 1024k stripes (which took more than 20 hours to initialise!) was about 4.8 Mbyte/sec for writes and 11.5 Mbyte/sec for reads. This is really, really bad. Anecdotally, the N5200 is capable of throughput nearly ten times this.

I had read that the original N4100 (which is based on almost exactly the same hardware, but using a more extensively modified 2.6.9 kernel) supposedly gets better write speeds. If true, this suggests that software issues are partly responsible for the appalling performance of the N4100+.  To gauge the impact of the kernel version on performance, I compiled a 2.6.9 kernel using Thecus' N2100 source code and booted it on my N4100+. This works, except that USB is broken and the RTC doesn't work as expected. I then reran speed tests on the same RAID 5 device. The result? Around 8.3 Mbyte/sec write and 15.5 Mbyte/sec read. So the 2.6.20 kernel has about 60% of the write speed and 75% of the read speed of the 2.6.9 kernel, but in absolute terms, they both perform pretty poorly.

I also tried fine tuning the build of Thecus' 2.6.20 kernel, compiling it using the EABI and selecting options to use the DMA engine for purposes other than just RAID 5. This kernel performed marginally better than the official one, with RAID 5 writes around 5.5 Mbyte/sec. I tested this kernel with both EABI and standard ABI userspace. As you might expect, the speed difference for all the important applications (FTP, CIFS, NFS) was negligible. RAID 5 uses integer operations (XOR), and EABI only accelerates floating point operations.

If XOR checksum speeds are anything to go by, it appears that my experimental 2.6.29.1 kernel performs at the same speed as the 2.6.9 Thecus kernel, so it appears that kernel.org developers have worked some of the kinks out of the DMA engine code since 2.6.20.

So it seems that even if the software can get the best possible performance out of the hardware, the N4100+ and its ARM-based relatives - with or without DMA offload engines - will never come close to the performance of x86-based appliances (such as the N5200) that can make highly effective use of the plethora of multimedia CPU extensions available for this architecture (MMXEXT, SSE, 3dNow!, etc).

== How to hack a Thecus 4100+ ... safely ==


Debian installation HowTos available on the web give the impression that the N4100+ is like the N4100 is like the N2100, such that you can just plug a straight-through 9-pin serial port into the mainboard header pins and get access to RedBoot. Not so. The N4100, the N4100+, and apparently also later versions of the N2100, have no pin block that you can plug a serial cable into. That means that you can't access the serial console unless you're prepared to attack your NAS with a soldering iron and educated guesswork. If you you're just trying to do a one-off recovery , you might be able to avoid the soldering iron and some of the guesswork by following [http://onbeat.dk/thecus/index.php/N2100_Recovering_from_a_bad_config_change these instructions from the Thecus Wiki]. I haven't yet summoned up sufficient courage to attempt soldering a pin block onto my own unit.

Fortunately all is not lost, because RedBoot has a network access feature. When the device is powered up, you have (by default) a one second opportunity to access RedBoot via telnet to (by default) 192.168.1.100 on TCP port 9000, and interrupt the boot sequence before the auto-boot script kicks in. See [http://www.cyrius.com/debian/iop/n2100/telnet.html this document] for details. The command sequence I use to do this is:

 while :; do ping -c1 -W1 192.168.1.100 && telnet 192.168.1.100 9000 && break; done

It took me many attempts to interrupt the boot (by the time the telnet connects, you usually have somewhere around 0.3 seconds left to press <Ctrl>c). A good trick is to hold Ctrl with your other finger just above the c key ready to push it when you see this: == Executing boot script in 0.470 seconds - enter ^C to abort

The first thing I did when I got in was run fconfig to reconfigure RedBoot to use DHCP to determine its IP address (since 192.168.1.100 happens to be the fixed DHCP reservation for my main computer!) and to increase the delay before the boot script kicks in from 1 second to 3 seconds. If you want to do anything similar, be very very careful, because incorrect use of fconfig can turn your NAS into a very expensive brick. Note that even though the telnet session disconcertingly echoes backspace characters literally as ^H while doing this, backspace does still work!

RedBoot's default boot script consists of only the following three lines:

 fis load ramdisk
 fis load kernel
 exec

Ensure that the script remains exactly this if you do choose to run fconfig.

Having interrupted the boot process, you can load a DIET-PC initrd.img and zImage into RAM via TFTP as follows:

 RedBoot>load -r -b 0x00800000 -h 192.168.1.101 initrd.img
 Using default protocl (TFTP)
 Raw file loaded 0x00800000-0x013b6fff, assumed entry at 0x00800000
 RedBoot>load -r -b 0x00200000 -h 192.168.1.101 zImage
 Using default protocol (TFTP)
 Raw file loaded 0x00800000-0x013b6fff, assumed entry at 0x00800000
 RedBoot>exec

192.168.1.101 is the IP address of my TFTP server in this example. Loading files via TFTP will not affect the flash memory of the appliance at all, so it will leave the official Thecus firmware completely untouched. For now, this is what we want.

You should have an TFTP server up and running before trying to interrupt the boot sequence. For ubuntu you can use "atftpd". Howto here: [http://www.ubuntugeek.com/howto-setup-advanced-tftp-server-in-ubuntu.html]

Unfortunately you won't be able to see any kernel messages unless you can redirect them to a working device. To get around this, I bought myself a USB serial adaptor (a no-name single-port adaptor cable using the Prolific PL2303 chipset), built a kernel with compiled-in support for uhci-hcd, ehci-hcd, usb-serial (with USB serial console option) and pl2303, and used a "console=ttyUSB0" kernel parameter. This allows me to see kernel messages, although it only runs at 9600bps and the Linux console doesn't initialise properly (apparently USB serial console support, which used to work fine in Linux 2.4, was partly broken by 2.6 console API changes, and no-one cares enough to fix the drivers). The netconsole driver is another possibility, although you don't get to see any kernel messages prior to the NIC initialisation when using this.

You can see the kernel messages without a USB adapter by having another terminal window open and running
$nc -l -p 6666 -u

At the "exec" stage of the boot interrupt the process the following will need to be appended, with the end command looking like this:

RedBoot>exec -c "root=/dev/ram0 initrd=0x00800000,40M netconsole=6665@192.168.1.100/eth0,6666@192.168.1.101/00:02:a5:bf:70:12

6665@192.1688.1.100/eth0 is the Thecus box, 6666@192.168.2.101 is the computer running the command "nc -l -p 6666 -u". Note that once this command is run nothing will happen, it is awaiting input from the Thecus device.


== Building a Kernel ==


I'm attempting to construct a single kernel image that will support all three models. By borrowing bits of Thecus GPL code and extrapolating, I've succeeded in creating a 2.6.29.1 kernel that supports N4100+ and (presumably) N2100 flawlessly. I've taken an educated guess at platform code for the original N4100, but no guarantees. I'll construct a patch against kernel.org 2.6.29.1 when I get a chance.

As far as source code is concerned, the differences between the N2100 and the N4100+ are trivial; the original N4100 is the odd one out. Apart from the obvious difference in the number of disk slots in the SATA backplane, the N2100 and the N4100+ are very similar - the N2100 has an additional USB port and a mini-PCI slot, and an extra Fintek F75373 fan/temperature sensor chip.  The N4100 also has a mini-PCI slot, but differs from both in that it has no USB, different NICs (e1000 instead of r8169) and a different SATA chipset (Intel GD31244, supported by the sata_vsc driver in recent kernels only). All three models have a RS5C327A real time clock chip and a PCA9532 chip for LED and buzzer control.

It looks to me as if the original N4100 was a prototype that Thecus are keen to wash their hands of. Another gotcha that N4100 owners should be aware of is that the e1000 NICs have no ID chips, and MAC addresses have to be assigned via software. Thecus firmware uses the redboot_config binary (for which they have released no source code) to extract the MAC addresses from Redboot configuration data in flash.

The main complication in constructing a common kernel to support all three is the thecus_rtc/thecus_io "i2c" driver.  In Thecus' GPL source for the N2100, thecus_rtc supports RS(5C)372 + F73373 + PCA9532 chips; in the N4100 source, thecus_rtc supports RS(5C)372 + PCA9532; in the N4100+ source, thecus_io supports the PCA9532 only (because in 2.6.20 the RS5C372 driver is separate).  In all cases, the Thecus-provided driver also provides /proc interfaces for monitoring and controlling button, buzzer and LED events.  In recent kernels, rtc, led and hardware sensor (hwmon) drivers are all completely separate from i2c drivers. The most kernel.org-correct way of supporting all the hardware is to abandon the Thecus i2c glueware altogether and just use the standard kernel.org drivers.  The kernel.org LED support works just fine, but the buzzer is detected as an input device(!), and I haven't yet figured out how to use it.

The Thecus-modified 2.6.20 kernel is relatively modern and fairly close to the mainline kernel, so it's quite functional and easy to deal with. Of Thecus' ARM-based NAS products, only the N4100+ is using this version of the kernel - the others (N2100, N4100) are using a more extensively modified 2.6.9 kernel. Recent mainline (kernel.org) 2.6 kernels include N2100 support (minus Thecus' thecus_io driver, which really doesn't add much). Although the 2.6.9 kernel (which can be compiled for, and will "mostly" work on, the N4100+) has DMA engine code that works more efficiently than the 2.6.20 kernel, the latter has other benefits, such as EABI, inotify, and improved hotplug/udev interaction (non-ancient versions of udev will refuse to work with kernels <2.6.15, and some current Linux distros (e.g. Debian Lenny) have compiled glibc with similar assumptions).

FYI, to avoid problems you should use gcc 3.x to compile the 2.6.9 sources, and gcc 4.x to compile 2.6.20 and 2.6.29.1. I was able to build both using Debian Lenny ARM running under QEMU on Windows (I prefer this to cross-compiling, since I mostly do userspace work). You can download the VMs that I use from the [http://dietpc.org/downloads/virtual_machines/ DIET-PC downloads area].

The 2.6.9 and 2.6.20 kernels that I built use just the Thecus-provided sources ([http://www.thecus.com/Downloads/GPL/n2100_2.01.09_GPL.tar.bz2 n2100 2.01.09 GPL] for 2.6.9, [http://www.thecus.com/Downloads/GPL/n4100plus_2.00.05_GPL.tar.bz2 n4100plus 2.00.05 GPL] for 2.6.20) plus [http://squashfs.sourceforge.net/ Squashfs 3.3] and [http://www.fsl.cs.sunysb.edu/project-unionfs.html UnionFS 2.3.3]. I had to tweak things here and there to address compiler-version-specific complaints, and a couple of minor repairs were needed for the 2.6.9 source (it won't compile as provided).  The 2.6.29.1 kernel is straight kernel.org (which now includes squashfs 4.0), plus a platform patch, a UnionFS 2.5.1 patch and an out-of-tree iscsi_trgt driver from the iscsitarget (IETD) distribution.

Here's the hardware support matrix for my kernel builds:

{| border="1"
!Kernel!!N2100!!N4100!!N4100+
|-
|2.6.9||Untested - expected to work||Untested - expected to mostly work||Mostly works (USB is broken, RTC epoch is 1970 instead of 2000)
|-
|2.6.20||Untested - expected to work||Untested - expected not to work||Works
|-
|2.6.29.1||Untested - expected to work||Untested - might work||Works
|}

== Userspace ==


Completely standard DIET-PC 2 already delivers a lot of the essential functionality provided by the official Thecus firmware, but it will need a lot of tweaking before it is functionally on par with the official firmware with regard to configurability and optional extras.

My prototype DIET-PC firmware has no means of remote configuration, so if you want to do anything with it (including shut it down!), you'll have to SSH into it and tweak the config files by hand. My firmware does not share any config files with the Thecus firmware. I am not using the JFFS2 on /dev/mtdblock3, so all state information is volatile and changes will be lost upon reboot, and my firmware will ignore any hostname or fixed IP address settings stored in flash (so it's a good idea to configure your DHCP server with a static assignment for your NAS, or you may have a hard time figuring out what IP address it ended up using).

The root password in my firmware images is "foobar", as is the VNC access password (if any).

I'm confident that my EXPERIMENTAL firmware will not destroy any of your NAS data, but there's no way I'm going to guarantee that! It will activate swap on /dev/md1, but only if it already contains a valid swap signature. It will mount /dev/md0 read-write, and for safety will attempt to unmount /dev/md0 after one second of inactivity (which may or may not work, depending on whether any running software has open files or directories on that volume at the time).  Like the Thecus firmware, it doesn't ever fsck /dev/md0 (this just isn't practical with terabyte-size filesystems, fscking can take hours). And in case you were wondering, all 2.6 kernels have the same on-disk RAID 5 format, so downgrading from 2.6.20 or upgrading from 2.6.9 won't fatally mangle your RAID 5.

{| border="1"
!Feature!!Thecus Firmware!!DIET-PC Firmware (2.6.29.1 version)
|-
|Size of userspace||~13MB (modules not included)||~12MB
|-
|Userspace libc||glibc 2.2.5||glibc 2.7 (2.3.6 for 2.6.9 kernel)
|-
|Userspace core||busybox 1.1.3, traditional /dev||busybox 1.11.2 with udev 136 (or 104 for 2.6.9 kernel)
|-
|Anonymous FTP||Pure FTPd 1.0.20||WU-FTPD CVS
|-
|SMB/CIFS server||Samba 3.0.20 (domain-joinable)||Samba 2.2.12 subset (not domain-joinable)
|-
|iTunes Server||mt-daapd 0.2.3||mt-daapd 0.2.4.2 (admin web server on port 3689, user=admin pass=mt-daapd)
|-
|Web server||Apache 2.0.55||Hiawatha 6.11
|-
|Web scripting||PHP 5.0.5 (mod_php)||PHP 5.2.8 (FastCGI)
|-
|X11 GUI||(none)||Xvnc (xf4vnc 4.3.99) with QVWM 1.1.12-9-IKu desktop featuring rxvt 2.7.10, accessible via VNC to port 5900
|-
|P2P Direct Connect Hub||(none)||opendchub 0.7.15
|-
|P2P Direct Connect Client||(none)||microdc 0.11.0
|-
|SSH daemon||OpenSSH 4.6.1 (3rd party module)||dropbear 0.51
|-
|NFS Server||nfs-utils 1.1.0 (3rd party module)||nfs-utils 1.1.4
|-
|BitTorrent Client||MLDonkey 2.9.2 (3rd party module) or rTorrent 0.7.9 (3rd party module)||Transmission daemon 1.50b2 with web UI and CLI (web UI on port 9091)
|-
|RAID infrastructure||mdadm 1.12, active RAID monitoring and recovery via polling shell scripts||mdadm 2.6.7 with RAID-5 reshaping kernel support, no RAID monitoring or automatic recovery
|-
|Button and LED support||Yes (polling shell scripts)||Not yet
|-
|Backup from NAS||nsync||rsync 3.0.5
|-
|Backup to NAS||Proprietary software for Windows and Mac OS X||Recommend using Windows rsync port (e.g. DeltaCopy) to do scheduled rsync backups
|-
|iSCSI target||(none - except on i4500R)||iSCSI Enterprise Target 0.4.17
|}

== Downloads ==


{| border="1"
!Kernel!!Kernel!!Config||System.map!!SquashFS Initrd
|-
|[http://diet-pc.sourceforge.net/arm/nX100/2.6.9/ 2.6.9 DIET-PC 2 ARM]||[http://diet-pc.sourceforge.net/arm/nX100/2.6.9/zImage zImage]||[http://diet-pc.sourceforge.net/arm/nX100/2.6.9/config config]||[http://diet-pc.sourceforge.net/arm/nX100/2.6.9/System.map System.map]||[http://diet-pc.sourceforge.net/arm/nX100/2.6.9/initrd.img initrd.img]
|-
|[http://diet-pc.sourceforge.net/arm/nX100/2.6.20/ 2.6.20 DIET-PC 3pre ARM EABI]||[http://diet-pc.sourceforge.net/arm/nX100/2.6.20/zImage zImage]||[http://diet-pc.sourceforge.net/arm/nX100/2.6.20/config config]||[http://diet-pc.sourceforge.net/arm/nX100/2.6.20/System.map System.map]||[http://diet-pc.sourceforge.net/arm/nX100/2.6.20/initrd.img initrd.img]
|-
|[http://diet-pc.sourceforge.net/arm/nX100/2.6.29.1/ 2.6.29.1 DIET-PC 3 ARM EABI]||[http://diet-pc.sourceforge.net/arm/nX100/2.6.29.1/zImage zImage]||[http://diet-pc.sourceforge.net/arm/nX100/2.6.29.1/config config]||[http://diet-pc.sourceforge.net/arm/nX100/2.6.29.1/System.map System.map]||[http://diet-pc.sourceforge.net/arm/nX100/2.6.29.1/initrd.img initrd.img]
|}

== Future Ideas ==


* Various people seem to be interested in using the Thecus NAS as a streaming media server, and a commercial product called TwonkyMedia seems to get mentioned a lot. I've acquired it, and it looks as though integrating this into DIET-PC would be pretty simple. The binaries are standard ABI, which would make integrating it into EABI firmware a bit impractical.
* I'm not sure if there is much interest in doing this, but with the addition of a external USB sound card, a Thecus NAS could also be used as a web-based jukebox. I bought a no-name USB audio dongle and did some playback tests using madplay, and this much works fine. I know of a PHP-based server-side-playback web jukebox called Glirnath[.sourceforge.net] which could be modified to run under DIET-PC with only a little Perl-to-PHP conversion.