ZFS - unable to destroy a snapshot

S

shadow_man

Executive Member
Joined
May 27, 2005
Messages
7,028
Reaction score
1,752
Location
Cape Town
EDIT: Damn you keyboard for making me look dyslexic.

Hi,

I'm unable to destroy a ZFS snapshot/s for a certain ZFS dataset in my ZoL (ZFS on Linux) implementation.

Whether I try to kill ALL snapshot for that dataset or just one, it hangs either via:

root@RSNAS001:~# for s in $(zfs list -H -o name -t snapshot | grep file_storage); do zfs destroy $s; done

OR

zfs destroy nas/file_storage@zuper_2013-11-04T18:22:22

Then nothing further.

Any ideas on how to remove this stubborn ox?

Thanks,
 
Seems its got a lot of items running simultaneously - this may be the issue...

Can't seem to force kill them though...

root@RSNAS001:~# ps aux | grep zfs
root 566 0.0 0.0 0 0 ? S< Sep07 0:00 [zfs_iput_taskq/]
root 1383 3.5 0.0 27696 2132 ? D Nov04 87:55 /sbin/zfs recv -Fdv nas
root 7989 0.0 0.0 27292 1552 ? D 10:16 0:00 zfs destroy nas/file_storage zuper_2013-11-04T18:22:22
root 9518 0.0 0.0 27292 1552 ? D 11:39 0:00 zfs destroy nas/file_storage zuper_2013-10-14T17:00:01
root 9921 0.0 0.0 27292 1552 ? D 11:52 0:00 zfs destroy nas/file_storage zuper_2013-10-10T07:30:01
root 10219 0.0 0.0 8112 880 pts/6 S+ 11:56 0:00 grep --color=auto zfs
root 24134 0.0 0.0 106748 3520 ? Ss Nov05 0:00 sudo /sbin/zfs recv -Fdv nas
root 24135 0.0 0.0 27696 2128 ? D Nov05 0:02 /sbin/zfs recv -Fdv nas
root 25209 0.0 0.0 27292 1548 ? D Nov05 0:00 zfs destroy nas/file_storage zuper_2013-08-22T10:03:32
root 25570 0.0 0.0 27292 1552 ? D Nov05 0:00 zfs destroy nas/file_storage zuper_2013-08-22T10:03:32
root 25867 0.0 0.0 27292 1552 ? D Nov05 0:00 zfs destroy nas/file_storage zuper_2013-08-22T10:03:32
 
First question, are you creating incremental snapshots? if so are you deleting snapshots in a given chain? what is the log output when you try to delete them?

Second question, are you creating snapshots to the same datastore? Are you exporting these with zfs_send?

Third question... What are you making snapshots for and how long is your chain?
 
FFF4Skin said:
First question, are you creating incremental snapshots? if so are you deleting snapshots in a given chain? what is the log output when you try to delete them?

Second question, are you creating snapshots to the same datastore? Are you exporting these with zfs_send?

Third question... What are you making snapshots for and how long is your chain?
Click to expand...

Incremental - yes. I'm trying to clear all the snapshots for the dataset nas/file_storage as they were created during testing and now that I'm going live I don't need them.

Snapshots are created on rsnas002 and replicated to rsnas001 with zfs send / recv - i've cleared the snapshots on rsnas002 with no issues, its just the ones on rsnas001 that won't clear.

Which specific log would you like me to check?

Snapshots are of an SMB share for a file server. Over 1000 created during testing.

EDIT: I see this in dmesg and i noticed tasks being blocked.

[5081041.516276] INFO: task collectd:2180 blocked for more than 120 seconds.
[5081041.520206] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[5081041.524163] collectd D ffffffff81806200 0 2180 1 0x00000000
[5081041.524176] ffff8803f138fd28 0000000000000082 ffff8803f138fcd8 ffffffff81182652
[5081041.524188] ffff8803f138ffd8 ffff8803f138ffd8 ffff8803f138ffd8 0000000000013780
[5081041.524209] ffff880405841700 ffff8803ed039700 ffff8803f138fd38 ffff8803f22343a8
[5081041.524237] Call Trace:
[5081041.524258] [<ffffffff81182652>] ? path_put+0x22/0x30
[5081041.524277] [<ffffffff816579cf>] schedule+0x3f/0x60
[5081041.524319] [<ffffffffa00d6a1d>] cv_wait_common+0xfd/0x1b0 [spl]
[5081041.524335] [<ffffffff8108aa50>] ? add_wait_queue+0x60/0x60
[5081041.524363] [<ffffffffa00d6b25>] __cv_wait+0x15/0x20 [spl]
[5081041.524455] [<ffffffffa01c42fc>] rrw_enter+0x8c/0x190 [zfs]
[5081041.524538] [<ffffffffa020a7c3>] zfs_statvfs+0x43/0x170 [zfs]
[5081041.524617] [<ffffffffa020fd09>] ? zfs_getattr_fast+0xd9/0x160 [zfs]
[5081041.524693] [<ffffffffa022810e>] zpl_statfs+0xe/0x20 [zfs]
[5081041.524711] [<ffffffff811a76a9>] statfs_by_dentry+0x99/0x140
[5081041.524726] [<ffffffff811a7778>] vfs_statfs+0x28/0x60
[5081041.524741] [<ffffffff811a77e7>] user_statfs+0x37/0x50
[5081041.524755] [<ffffffff811a7880>] sys_statfs+0x20/0x40
[5081041.524770] [<ffffffff81661ec2>] system_call_fastpath+0x16/0x1b
[5081041.524787] INFO: task zfs:1383 blocked for more than 120 seconds.
[5081041.528771] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[5081041.532879] zfs D ffffffff81806200 0 1383 1382 0x00000000
[5081041.532890] ffff88027e4b7840 0000000000000086 ffff8801433fd680 0000000000000000
[5081041.532901] ffff88027e4b7fd8 ffff88027e4b7fd8 ffff88027e4b7fd8 0000000000013780
[5081041.532918] ffff880405841700 ffff880401750000 ffff8801433fd988 ffff880401750000
[5081041.532947] Call Trace:
[5081041.532963] [<ffffffff816579cf>] schedule+0x3f/0x60
[5081041.532978] [<ffffffff816597cd>] rwsem_down_failed_common+0xcd/0x170
[5081041.533057] [<ffffffffa01c4be6>] ? sa_find_idx_tab+0x56/0x1f0 [zfs]
[5081041.533072] [<ffffffff816598a5>] rwsem_down_read_failed+0x15/0x17
[5081041.533088] [<ffffffff81318a14>] call_rwsem_down_read_failed+0x14/0x30
[5081041.533104] [<ffffffff81658bc4>] ? down_read+0x24/0x2b
[5081041.533182] [<ffffffffa021402c>] zfs_inactive+0x4c/0x230 [zfs]
[5081041.533198] [<ffffffff811250de>] ? truncate_pagecache+0x5e/0x70
[5081041.533273] [<ffffffffa02281c8>] zpl_evict_inode+0x28/0x30 [zfs]
[5081041.533289] [<ffffffff81192388>] evict+0xb8/0x1c0
[5081041.533302] [<ffffffff81192578>] iput_final+0xe8/0x210
[5081041.533315] [<ffffffff811926de>] iput+0x3e/0x50
[5081041.533390] [<ffffffffa0219019>] zfs_rezget+0xf9/0x4a0 [zfs]
[5081041.533470] [<ffffffffa020c5cd>] zfs_resume_fs+0x1ed/0x290 [zfs]
[5081041.533547] [<ffffffffa0203630>] zfs_ioc_recv+0x860/0xef0 [zfs]
[5081041.533563] [<ffffffff81161ebb>] ? kfree+0x3b/0x140
[5081041.533644] [<ffffffffa01d83b2>] ? spa_lookup+0x62/0xc0 [zfs]
[5081041.533700] [<ffffffffa0188459>] ? dbuf_rele_and_unlock+0x169/0x210 [zfs]
[5081041.533777] [<ffffffffa01d1454>] ? spa_open_common+0x214/0x340 [zfs]
[5081041.533857] [<ffffffffa020450c>] zfsdev_ioctl+0xdc/0x1b0 [zfs]
[5081041.533872] [<ffffffff81189c5a>] do_vfs_ioctl+0x8a/0x340
[5081041.533887] [<ffffffff81177ba0>] ? vfs_write+0x110/0x180
[5081041.533901] [<ffffffff81189fa1>] sys_ioctl+0x91/0xa0
[5081041.533914] [<ffffffff81661ec2>] system_call_fastpath+0x16/0x1b
[5081161.532220] INFO: task collectd:2180 blocked for more than 120 seconds.
[5081161.536354] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.


Thanks,
 
Last edited:
Depending on your enabled settings and how many things zfs need to double check, the zfs destroy can run for a long while, it is not instant. On top of that, that can kill disk i/o if its a long large job.
 
Tinuva said:
Depending on your enabled settings and how many things zfs need to double check, the zfs destroy can run for a long while, it is not instant. On top of that, that can kill disk i/o if its a long large job.
Click to expand...

I let it run overnight. Didn't do anything, the other nas with the same snapshots cleared in all of 20 mins...

I think i'm going to try and reboot it later and see if I can delete after, just can't do that during the middle of the day as its in use...
 
Ok first things first. have you tried to do a recursive destroy instead of using your script that list them by whatever naming convention?

Does zfs list show anything mounted for this dataset?

can you issue this command using strace, always good if you need to report a bug.

Attempt the same delete with -r flag on the snapshot and see what happens.

The logs will generally be messages or syslog.

I have my own setup running on several supermicros but I do not have the need for snapshots, either way this should prove interesting.

Also how about a strace on one of those destroy PID's?

PS, unless you can calculate your uptime to the dmesg time output since last boot those messages in dmesg could be decades old(you get the idea). just to clarify the time context in dmesg is the number of seconds since last boot.
 
Last edited:
Thanks all for the help and suggestions.

A reboot fixed it and the erroneous processes disappeared.

All seems to be fine, clearing snapshots as I type this.
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X