ZFS and Ubuntu Home Server howto

A while ago I bought myself a HP Microserver – a cheap, low power box which has four 3.5″ drive bays and an optical drive bay. I bought it to run as a home server which would back up all my data as well as serve up video, music and photos around the house. I had decided before buying that I wanted to store my data using the ZFS filesystem since ZFS was the only filesystem at the time which offered guaranteed data integrity. (It still is the only filesystem of release quality which offers this, although BTRFS is catching up.) I have become almost obsessed with ZFS because of the overwhelming benefits it offers but I won’t go into them here. Instead I recommend watching this talk by the creators of ZFS (Part 1, part 2, part 3) or read through the accompanying slides. [PDF]

HP Microserver - openI meant at the time to write about how I set up my system but never did get around to it, so here is what I did in the end. The server arrived with 2GB of ECC RAM and a 250GB hard disk. I eventually upgraded this to 8GB RAM and added two 2TB hard disks, although I started with one 2TB disk and added the second as a mirror when finances allowed. ZFS checks the integrity of the stored data through checksums and so it can always tell you when there is data corruption but it can only silently heal the problem if it has either a mirror or a RAID-Z/Z2 (Equivalent to RAID 5 or 6.)

ZFS is available as part of FreeNAS, FreeBSD, Solaris, and a number of Solaris derivatives. I initially installed FreeNAS 8. FreeNAS runs from a USB stick which I put in the handy internal USB socket, but while that was great for storing and sharing files it was not so good for running bittorrent on or using SSH to connect from out of the house. I also tried Solaris but I ended up going back to what I know and using Ubuntu Linux 12.04 LTS. Although licensing prevents ZFS from being included with Linux it is trivial to add it yourself.

I have assumed a certain level of knowledge on the reader’s part. If it doesn’t make much sense to you then you might be better off with FreeNAS or an off-the-shelf NAS box.

After installing Ubuntu and fully updating it I did the following:

sudo add-apt-repository ppa:zfs-native/stable

sudo apt-get update

sudo apt-get install ubuntu-zfs

…and that was it. It is a lot more complicated to use ZFS as your root filesystem on Linux, so I don’t.

Update: as of Ubuntu 16.04 ZFS will be supported directly. You will be able to install ZFS with the following rather than adding a third-party repository:

sudo apt-get install zfsutils-linux

Next, I had to set up the ZFS storage pool. The creators of ZFS on Linux recommend that you use disk names starting with /dev/disk/by-id/ rather than /dev/sda, /dev/sdb etc as they are more consistent (particularly the wwn identifier) so look in that folder to see what disk names you have.

ls -l /dev/disk/by-id/

The example pool name given is tank but I strongly recommend that you use something else. To create a single disk storage pool with no mirror:

sudo zpool create tank /dev/disk/by-id/wwn-0x5000c5004f14aa06

To add a mirror to that later you would type:

sudo zpool attach tank /dev/disk/by-id/wwn-0x5000c5004f14aa06 /dev/disk/by-id/wwn-0x5000c500400303dd

Or if starting with two disks to put in a mirror, your initial command would be:

sudo zpool create tank mirror /dev/disk/by-id/wwn-0x5000c5004f14aa06 /dev/disk/by-id/wwn-0x5000c500400303dd

I prefer to use mirrors as they are generally faster, however if you want a RAID5-type setup use:

sudo zpool create tank raidz1 … … … (3 or more disk identifiers)

The system will create your storage pool, create a filesystem of the same name and automatically mount it, in this case under /tank.

“sudo zpool list” will show you that a pool has been created as well as the raw space in the pool and the space available.

“sudo zpool status” will show you the disks that make up the pool.

Screenshot showing output of zpool list and zpool status commandsWhile you can just start storing data in your newly-created filesystem (in /tank in our example) that isn’t the best way to use ZFS. Instead you should create additional filesystems within your storage pool to hold different types of data. This will allow you to do things like set compression, deduplication, quotas and snapshots differently for each set of data or backup an individual filesystem with zfs send. You use the zfs command to create your filesystems. Some examples:

sudo zfs create tank/music

sudo zfs create tank/videos

sudo zfs create tank/backups

The above examples will create filesystems in the pool and will automatically mount them as subfolders of the main filesystem. Note that the name is in the format pool / filesystem name and there is no leading slash on the pool name.

Check that your filesystems have been created:

sudo zfs list

Screenshot showing output of zfs list commandNow we need to share the data, otherwise it’s not much of a server. ZFS will automatically manage sharing through NFS (Unix/Linux) or SMB (Windows) but you must first install the server software. For sharing to Windows clients use:

sudo apt-get install samba

To add NFS use:

sudo apt-get install nfs-kernel-server

You don’t need to configure much because ZFS handles most settings for you, but you might wish to change the workgroup name for Samba in /etc/samba/smb.conf.

To share a ZFS filesystem you change a property using the zfs command. For Windows clients:

sudo zfs set sharesmb=on tank/music

sudo zfs set sharesmb=on tank/videos

For Unix / Linux clients:

sudo zfs set sharenfs=on tank/backups

Or you can share the whole lot at once by sharing the main pool. The sub-filesystems will inherit the sharing property unless you turn them off:

sudo zfs set sharesmb=on tank

sudo zfs set sharesmb=off tank/music

You can check whether your filesystems are shared or not:

sudo zfs get sharesmb,sharenfs

At this point you should be able to see your shares from other computers on the network but you probably won’t have permission to access them. You will need to ensure that the file permissions and owners are set correctly, and you will also have to add an account and set a password for  use when connecting through Samba. If your username is ella then use:

sudo smbpasswd -a ella

to set your Samba password, and make sure that ella has permission to access all the files in your shared folders:

sudo chown -R ella:ella /tank/videos

Other useful features of ZFS that you should look up include snapshots and zfs send/recieve. I hope this short guide has been helpful if you are trying to set up a ZFS server. Let me know in the comments.

Updated 29/02/2016 to remove some personal details, add information about ZFS support in Ubuntu and add some explanations noted in the comments.

What is a Raspberry Pi?

I have in my hands, at last, a Raspberry Pi. No, not a delicious item of food, but a tiny, cheap computer designed to encourage people to learn how it works and make it do something new.

Handheld Raspberry Pi

The Raspberry Pi has been brought into being by the Raspberry Pi Foundation, a charity who put their intentions like this:

We don’t claim to have all the answers. We don’t think that the Raspberry Pi is a fix to all of the world’s computing issues; we do believe that we can be a catalyst. We want to see cheap, accessible, programmable computers everywhere; we actively encourage other companies to clone what we’re doing. We want to break the paradigm where without spending hundreds of pounds on a PC, families can’t use the internet. We want owning a truly personal computer to be normal for children. We think that 2012 is going to be a very exciting year.

The people behind the idea want to address the lack of people studying for computer science degrees and they identified two problems which may cause this. One is that the way the computers (ICT) are taught in school is boring and business focused, teaching office skills and not computer science, and the other is that computers now are far more expensive and complex than those in the 80s, such as the BBC micro or the ZX Spectrum. Those computers, unlike the modern PC or Mac, encouraged tinkering, learning and extending.

I recommend that you watch this report from BBC Click which looks at the problem: Can a £15 computer solve the programming gap? [BBC Click]

To keep costs down, the Raspberry Pi uses an Arm processor – the same as in most mobile phones – and does not have very much memory. It is designed to connect to a TV so that an expensive monitor is not necessary. It uses the free Linux operating system which is also open to being modified by the end user. The intention is that simple programming software will be included with the Pi to enable someone to simply plug it in and start learning to write code. The copy of Linux which is recommended for the Raspberry Pi currently includes Scratch, which allows kids to create and animate “interactive stories, animations, games, music, and art” by dragging and dropping blocks of computer code.

The Raspberry Pi model B sells for approximately £30 once you add VAT and delivery and will be available from RS and from Element 14 / Farnell once they have the stock available. There will eventually be a Model A which will be a few pounds cheaper but will lack a network socket and will have only one USB socket. In addition to the Raspberry Pi itself you will need a memory card (SD card) to hold Linux, a keyboard and mouse with USB connectors, a power supply of the Micro USB sort used by many modern mobile phones, and access to a TV or monitor with a HDMI or a composite video connector. These may already be available but if not will add a few pounds to the cost.

At the moment the Raspberry Pi is mostly selling to hobbyists and enthusiasts rather than into education. That’s OK though; at the moment the software for the Pi requires quite a lot of experimentation and difficult-to-follow steps to make some things work and it isn’t really ready for prime time. Many of those people fortunate enough to get their hands on an early Raspberry Pi are working out the problems and feeding information back to everyone that will use one in the best tradition of open source software. By the time kids start to see these in schools in a few months there will hopefully be a lot more software ready to use without too much knowledge necessary.

Unless you already knew about the Raspberry Pi, you probably can’t have one at the moment. Interest in the Pi has been astonishingly high and I think that several hundred thousand people have placed pre-orders or signed up to reserve one. The initial batch was only ten thousand! I signed up for mine at 1 minute past 6 (am!) on the launch day three months ago and I have only just got one so new orders will take a few months yet.

Coming soon: I plan to write about getting started with the Raspberry Pi, what you need, and how to set it up and do something useful with it.

More Information

Can a £15 computer solve the programming gap? [BBC Click]

Computer programming at school [A Latent Existence]

Raspberry Pi: About Us [Raspberry Pi]


Home server project – HP Microserver

HP MicroserverThere are a few computers in our house. I have a PC on my desk. I have another PC connected to the TV which we use for watching and recording TV, films, music, and playing games. I have a netbook in the bedroom for those days when I can’t get out of bed, and an Android phone which I use for music and video too.  My wife has a laptop and an Android tablet computer.Keeping track of our files on all of that can be a bit chaotic and so I have an old PC which works as a file server – a central repository for all of our music and video files which all of the other devices can fetch files from.

That file server consists of an old under-powered PC which runs FreeNAS and has a couple of not-quite-worn-out laptop disks in it because those were all that I had. Since this server isn’t very fast or reliable, Last week I bought a 2TB hard disk to replace the dying disks. I also had an email from ebuyer (my normal source of computer parts) which featured a HP Microserver for £229 with an offer of £100 cashback from HP. A new server capable of running four disks with low power consumption and a warranty all for £129 seemed like a fantastic deal to me, so yesterday I bought one. (It had gone up by £10 though.) The server came with a 250GB hard disk which is a bonus since hard disks are still very expensive after flooding in Thailand destroyed several factories recently.

HP Microserver - open

I intend to use the new server to hold all our music and video as before, plus our photographs. I will store all my files and documents on it, and I will backup the other computers to it. Now that I have my new server I have to work out what operating system I am going to run on it. My old server has FreeNAS installed onto a 2GB USB memory stick, and the hard disks are formatted with ZFS for high file integrity. I could do the same with my new server but since the new server is a bit more powerful (Dual-core CPU, 6GB RAM) I want to also run a couple of Linux virtual machines on it and I can’t easily run virtual machines with FreeNAS as the host operating system. I don’t want to run a file server from a virtual disk either, to avoid any data loss. Oracle Solaris has ZFS so I could run that on the new server and then run VirtualBox on that to host my Linux VMs. Another option is to run a bare metal hypervisor like VMWare ESXi or Citrix Xenserver, and then run Linux and FreeNAS as clients under that. I still wouldn’t want to serve my files from a virtual disk but I could give the FreeNAS virtual machine direct access to the 2TB hard disk. The only question then is where the virtual machines reside. I would prefer not to use the 250GB disk as that would be a welcome upgrade in my Media Centre PC which currently has a horrifically slow IDE disk which struggles to record two programmes at once. I could possibly host the virtual machine for FreeNAS on a USB memory stick too, maybe even the same one as ESXi is installed on although I don’t know if that can be done. I am curious as to whether ESXi could load a second virtual machine through an iSCSI drive served up by another one of it’s clients but that does seem like a recipe for slow operation and data loss.

Comments on which way to go are welcome.

How to set up a free Linux server on Amazon EC2

I bet you didn’t know that Amazon sell computing time. Well they do. After having to build massive data centres full of web servers to handle the largest book store website in the world, they discovered that they had to have lots of extra computers standing by just to handle Christmas and the new year shopping rush. The logical conclusion? Rent out space on those servers to other people when Amazon don’t need them. That has now expanded massively and Amazon are one of the biggest resellers of virtual private servers. (VPS) A VPS is basically a complete computer contained within another computer. The physical computers can pretend to be many smaller virtual computers, which the customer can have complete control of without affecting anyone else.

The most intriguing part is that Amazon bill for these services in tiny increments, so that if you like, you could rent a massive server with 13 processors and 23GB of memory, but just for ten minutes. Indeed, you can rent an entire supercomputer for a day. The service is called the Elastic Compute Cloud, or EC2 for short. You can also rent a tiny little server with 1 processor and 613MB of RAM. In fact, at the moment, you can have that server completely free for a year! You will only pay if you use extra disk space or more than 15GB of data transfer. The service is really aimed at web servers, but it can also be used to run a game server, and I will show you how. Using Amazon EC2 is quite complex and not for the non technical, but hopefully this guide will get you running.

I will write another article later and tell you how to set up a Source (SRCDS) server for games from Valve such as Team Fortress 2 (TF2) and Half Life 2:Deathmatch. (HL2:DM, HL2MP) For now, here is how to get started with a free Linux server on Amazon EC2.  Continue reading “How to set up a free Linux server on Amazon EC2”

Guest post: Where is my paranoid computer? – Well it’s here

This is a guest post by Frank Roper, on twitter as Walthamfrank

This piece was inspired by latentexistence’s blog article Privacy online: Where is my paranoid computer? http://www.latentexistence.me.uk/privacy-online-where-is-my-paranoid-computer/ which references something that Cory Doctorow in the book Little Brother calls Paranoid Linux. If you haven’t already read this piece then I suggest you do before reading further.

It would perhaps be better if I called this piece Where is my paranoid computer? – Well it’s here (well sort of and only under certain conditions) as we don’t yet have something that meets all of the requirements.

Before we start let’s remind ourselves of the requirements for a Paranoid Linux. These are:

  • Mesh networking
  • Random anonymous routing
  • End-to-end encryption of all traffic such as web pages, email, chat and voice
  • Digital signatures and proof of identity
  • Fake internet traffic (Chaff) to provide a decoy and hide real information

latentexistance then suggests some technologies that could meet these requirements.

After some research and searching I have managed to identify a Linux live system – this is a system which can run off either a CD or a USB memory stick – that has most of these technologies built into it. This is The (Amnesic) Incognito Live System [T(A)ILS] https://amnesia.boum.org/ T(A)ILS is designed to preserve your privacy and anonymity – all outgoing connections to the Internet are forced to go through the Tor network and no trace is left on local storage devices (the computer it’s plugged into) unless explicitly asked.

It includes the following software and services:

  • Tor and the Vidalia graphical frontend
  • Firefox preconfigured with Torbutton for anonymity and protection against evil JavaScript, FireGPG for e-mail encryption and all cookies are treated as session cookies by default; the CS Lite extension provides more fine-grained cookie control for those who need it
  • Claws Mail e-mail client, with user-friendly GnuPG support
  • Pidgin preconfigured with OTR for Off-the-Record Messaging
  • Aircrack-ng for wireless networks auditing
  • Gobby for collaborative text writing
  • onBoard virtual keyboard as a countermeasure against hardware keyloggers
  • Shamir’s Secret Sharing using gfshare and ssss

CD and USB images for T(A)ILS can be downloaded from their website (note that you may receive a security certificate error when accessing the site but it’s fine to accept an exception in this case). I’d recommend the USB distribution as you can save files to the stick.

What it does not include is mesh networking. However some work is being undertaken at present to look into providing mesh networking. The issue is that mesh is not cheap – think of it as being a mobile phone network – if you want to provide it across any significant area. For those who wish to explore providing small scale mesh networking open source systems are available that can be ‘plugged’ into existing hardware. A detailed look at this is beyond the scope of this piece but I could have my arm twisted if enough people ask!

Hey, I’m not a Linux geek I want to run stuff on My Windows PC or Mac (or even my mobile or mobile devise)

This is less easy (for this read very hard) if you don’t want to leave traces of your activities all over your computer or mobile.

As a basic you could install Tor http://www.torproject.org/index.html.en on Windows, Mac or mobile device (Tor supports Android-based phones, tablets, computers; test packages are vaialble for iPhone, iPod Touch, iPad and experimental instructions are available for Nokia Maemo/N900).

An additional basic step is to install email encryption. Many email clients – although it’s easier to install in some of the free /open source packages like Thunderbird – support encryption.

The next possible step is to run your applications from a USB stick, although this will still leave traces of activity on the host computers hard disk. There are a variety of pendrive applications available:

An interesting alternative is MojoPac http://www.mojopac.com which can be installed on any USB mass storage device, even an iPod. Depending on who you believe practically everything, or everything, personal (bookmarks, documents, etc) is kept within MojoPac, always with you and never touching the host computer. Currently this only works with Windows XP systems as the host.