ZFS, your thoughts?

DrJohnZoidberg

DrJohnZoidberg

Honorary Master
Joined
Jul 24, 2006
Messages
27,997
Reaction score
7,457
Location
Table View
I recently decided its time to start creating a more redundant storage solution. A friend and IT guy recommended I use ZFS for my new mirrored array. Just finished setting it up, but write performance is pretty swak. Do you really think its worth it or should I just return to a normal RAID array formatted with ext4 or something?

Anybody using ZFS in a mirror configuration want to comment on their speeds? Currently only getting 18MB/s write and I have 1.8TB to copy over :cry:
 
Does ZFS not run in userspace via FUSE?

Maybe wait for brtfs to mature.
 
ZFS is a brilliant concept, especially it's data integrity features, which are quite unique.
ponder said:
Does ZFS not run in userspace via FUSE?
Click to expand...
Yeah, I checked, it does. To run ZFS natively you should use FreeNAS or some free Solaris, as they have native implementations. The way ZFS works, you should actually get increased speed with mirroring!
 
Yes, running it with FUSE in Fedora. I was copying stuff over now, seems much lighter on the CPU than regular software RAID in linux. It is copying over slower, but once the copy is done, its done and I don't have to wait for any syncing between drives - so maybe it's about the same speed if you factor that in. Still all new to me, so just getting in to it still, however it is quite cool and very easy to setup.
 
the ZFS implementations in fuse and in FreeBSD are incomplete. Don't use them. Get OpenSolaris ISOs somewhere.
 
Okay, small update from my side.

I really like ZFS and the entire concept, but it was just way too slow - I am sure its fine when run natively, but in Fedora both read and write speeds were considerably slower than a single disk configuration.

I then looked in to btrfs. which works on the same concept as ZFS, but it is also still very young and still in experimental phase. I however did just give it a go to see what to expect. All I can say is wow! Looks really promising, can't wait for it to mature and gain more integration into distributions. Read/write speeds 7x faster than my ZFS array and about 20% faster than standard software RAID formatted with ext4. Unfortunately for now its still a bit experimental, but with Fedora backing it in their next release I expect development will have to get a move on.

For now I have reverted back to a raid1 config with ext4, write speeds sitting at about 60MB/s instead of 18MB/s, so I'm happy. Now just to get myself a small UPS :)
 
I'm running ZFS via fuse on a Debian box. 4x 2TB drives and it seems to fly. Possibly a Fedora issue?

Also if you use the Debian freebsd version then ZFS is native btw. What command are you running to check speeds? I can ssh in to my box and run a test for you. Running Debian Squeeze.

J

timvdwest said:
Okay, small update from my side.

I really like ZFS and the entire concept, but it was just way too slow - I am sure its fine when run natively, but in Fedora both read and write speeds were considerably slower than a single disk configuration.

I then looked in to btrfs. which works on the same concept as ZFS, but it is also still very young and still in experimental phase. I however did just give it a go to see what to expect. All I can say is wow! Looks really promising, can't wait for it to mature and gain more integration into distributions. Read/write speeds 7x faster than my ZFS array and about 20% faster than standard software RAID formatted with ext4. Unfortunately for now its still a bit experimental, but with Fedora backing it in their next release I expect development will have to get a move on.

For now I have reverted back to a raid1 config with ext4, write speeds sitting at about 60MB/s instead of 18MB/s, so I'm happy. Now just to get myself a small UPS :)
Click to expand...
 
My ZFS flies - 4x 2TB Samsung's running in Raidz1.

See below. I can run further tests / comment on config if you like? OS: Debian Squeeze , ZFS install = zfs-fuse , apt-get install zfs-fuse (as easy as that :D)

##WRITE SPEED TEST
root@nasty:/nas# time dd if=/dev/zero of=/nas/test.dbf bs=8k count=1048576
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 100.646 s, 85.3 MB/s

real 1m40.660s
user 0m0.268s
sys 0m29.346s

##READ SPEED TEST
oot@nasty:/nas# time dd if=/nas/test.dbf of=/dev/null bs=8k
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 35.8288 s, 240 MB/s

real 0m35.847s
user 0m0.220s
sys 0m13.709s


##READ AND WRITE SPEED TEST
root@nasty:/nas# time dd if=/nas/test.dbf of=/dev/null bs=8k
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 35.8288 s, 240 MB/s

real 0m35.847s
user 0m0.220s
sys 0m13.709s
root@nasty:/nas# time dd if=/dev/zero of=/nas/test.dbf bs=8k count=1048576
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 87.2957 s, 98.4 MB/s

real 1m27.350s
user 0m0.248s
sys 0m29.086s
 
Last edited:
jsheed_sa said:
I'm running ZFS via fuse on a Debian box. 4x 2TB drives and it seems to fly. Possibly a Fedora issue?

Also if you use the Debian freebsd version then ZFS is native btw. What command are you running to check speeds? I can ssh in to my box and run a test for you. Running Debian Squeeze.

J
Click to expand...

I don't know what causes it, maybe it doesn't like a 2 drive configuration or something. I've been really struggling to make a decision on how I should set my disks up so that I have room for expansion. Did some research and found out I can actually create a RAID 5 array with 2 disks and then add the "missing" disk later. Obviously this means no redundancy, but I'm going to go pick up another drive today and pop it in. This seems like the best solution for me at the moment.
 
Tinuva said:
If you want to make use of ZFS, give Nexenta a try. It runs on a opensolaris kernel with a GNU userland (Linux like utilities).
Click to expand...

Is it just a NAS distribution like FREENAS etc? I prefer not to use those as then the device is limited to being a NAS. I like having the option of using it as a torrent server etc. With Debian I can add roles as I see fit. Right now my HP Proliant Microserver runs my ZFS, my XBMC and soon my torrents / sickbeard / couch potato stuff.

I also tried setting up on OpenIndiana but had problems so I moved to Debian.

J
 
It is indeed a Distro of its own, however its not meant for Desktop stuff.

So perhaps you might also want to have a look at StormOS. I have no experience with it, just found it yesterday actually. It is a Desktop Distro based on Nexenta using Debian package management system.

I have a feeling Xorg doesn't work on Nexenta Core 4 yet, but it seems the StormOS guy is working on it mostly for the Nexenta Project.
 
I am now wishing I had stuck with ZFS or btrfs. It seems very likely that I may lose about 1.7TB of data due to what looks like a faulty SATA controller on my motherboard. I had already set up a clean 3 disk RAID5 array and had copied my data from the remaining 2tb drive that I wanted to now add to the array. I copied over the data to the existing array and made sure everything was intact, did multiple reboots to check if everything was working fine and it was.

Then I partitioned the remaining 2tb drive so I could add it to the array, added it and everything was going fine - it was busy rebuilding and growing the array. A while later went to check its progress and to my horror saw that it was showing that 2 disks has failed, I knew however that it was very unlikely that it was a fault on the disks. I tried everything to get the disks back without having to restart the machine, but I just wasn't able to read from them (couldn't even see SMART data). Inevitably I had to shut down the machine, check the physical connections and so forth.

When I restarted I was hoping that the disks would just fall back in to the array, but no such luck, it was showing that the two disks were now removed from the array and obviously now the array wouldn't start up. Tried stopping the array and then tried to assemble the array again using "mdadm --assemble etc etc" but couldn't get that to work and was getting bad superblock errors.

After a marathon google search I found the only way that I might get it back was to try and re-create the array. This actually ended up working and the array is back online, but now it wound't mount and was giving me filesystem errors. Now I am in the process of running a fsck, which isn't exactly a speedy process and there are a plethora of errors. I've seen in other peoples posts that this process may take days or weeks to complete (depending on the amount of data). I am just hoping that I will be able to recover at least some of the data - only time will tell I guess.

The irony of this whole story is that previously I had no redundancy in place and was trying to improve on this only to lose my data once the redundancy was put in place :confused:

What do you think the chances are that I will get my data back?

EDIT

This is taking a serious amount of time and resources.

fsck.ext4 output:

Code: 
...
...
File /myfolder/mysubfolder/myfile (inode #58196012, mod time Mon Jul 18 02:31:22 2011) 
  has 89239 multiply-claimed block(s), shared with 1 file(s):
	... (inode #87293996, mod time Mon Jul 18 02:31:22 2011)
Clone multiply-claimed blocks? yes
...
...

top output:

Code: 
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4752 root      20   0 3295m 2.0g  684 R 99.7 53.1 984:26.68 fsck.ext4
...
...
...
 
Last edited:
Just hit 4 days of fsck.ext4 continuously running flat out on one of my CPU's cores. Wonder how long this is going to take :confused:
 
koffiejunkie said:
Please elaborate?
Click to expand...

It has something called delayed allocation, so what happens is that your files seem saved but are not yet, then a power failure and boom your data is lost

Ext4 uses a filesystem performance technique called allocate-on-flush, also known as delayed allocation. It consists of delaying block allocation until the data is going to be written to the disk, unlike some other file systems, which may allocate the necessary blocks before that step. This improves performance and reduces fragmentation by improving block allocation decisions based on the actual file size.
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X