GlusterFS 3.2 on Gentoo

I’ve been looking at cloning ( or somehow maintaining synced ) file-systems between multiple servers, and after looking at other solutions ( regular rsyncs, inotify etc ), a clustered file-system seems to be the best solution. GlusterFS looks to be a popular Open Source based solution, however there doesn’t seem to be a complete walk-through for setting it up on Gentoo ( there is a guide for Fedora 13 which I have based some of this guide from, however it seems to be based on an older version of GlusterFS ), so I’ve decided to write one specifically for Gentoo and GlusterFS 3.2.

For this tutorial, I am only going to document setting up 2 servers, but adding more servers during the initial setup should just require adding the additional servers in the same way as the first two are described ( an extra hosts entry, extra ‘disks’ listed on the volume creation commands ).

In this example, we have two servers, server1 and server2. If there is no DNS setup to do this already, the first step is to add these entries on each machine, replacing the server IPs with the correct ones:

192.168.1.100 server1
192.168.1.101 server2

Next, simply install and start sys-cluster/glusterfs on each machine:

emerge sys-cluster/glusterfs
/etc/init.d/glusterd start

A Glusterfs daemon should now be listening on both servers. Next we need to probe the second server from the first ( which fixes the ‘Host server2 not a friend’ error message which I had before I ran this ), simply run this on server1:

gluster peer probe server2

You can now confirm that the machines can see each other using the gluster peer status command ( you can do this on both machines and they should report each other ):

# gluster peer status
Number of Peers: 1

Hostname: server2
Uuid: 963aee89-fca3-4a5a-90ae-ada89f8a277d
State: Peer in Cluster (Connected)

Now we have GlusterFS up and running, we can create a directory to store the data on each machine:

mkdir -p /export/store1

Now we can actually create the volume for the data to sit on. For this example I have set replica to be 2, so the data will exist on all machines ( this command only needs to be ran on server1 ):

gluster volume create test-volume replica 2 transport tcp server1:/export/store1 server2:/export/store1

This should return successfully, now we can start the volume ( again, only run this on server1 ):

gluster volume start test-volume

We now have a GlusterFS volume up and running, but before we can access the file-system, we need to mount it. First create a mount-point on both servers:

mkdir /mnt/glusterfs

Then ensure GlusterFS runs on boot ( on both servers ):

rc-update add glusterd default

And add a row to fstab on server1 ( so the file-system is mounted on boot ):

server1:/test-volume /mnt/glusterfs glusterfs auto,rw,allow_other,default_permissions,max_read=131072 0 0

Then server2:

server2:
server2:/test-volume /mnt/glusterfs glusterfs auto,rw,allow_other,default_permissions,max_read=131072 0 0

On both servers we can now mount the file-system manually, and should be able to add/remove/copy etc files on one machine and have it appear on the other:

mount /mnt/glusterfs

Assuming there were no errors reporting, the disks should now stay in sync ( it seems that even when one machine is disabled, and files are added to the disk , the files are copied to the other machine when it returns. ), and you have a solution for serving or sharing multiple files across multiple servers.