« Setting up a Ubuntu NAS to backup my FreeNAS serverWindows Live Mail Log File »

FreeNAS Continuously Rebooting Problem

05/21/09

Permalink 11:11:15 pm by guy, Categories: Linux, Random Stuff

The other I shutdown (properly) my FreeNAS box to replace a drive. The next day I noticed that it was rebooting itself frequently. Once in a while is a bad frequency, every 15 minutes is unusable! I removed all my external drives in an attempt to narrow down the problem. Nothing changed. I finally plugged in a monitor and watched. As I expected, it was rebooting as a result of a kernel panic. Unfortunately I?ve had lots of problems like this with FreeNAS (I still want to love it!) and I?ve passed them all off as hardware incompatibilities with BSD. This time, it wasn?t. What I found was that the panic was for ?ffs_blkfree?. I?d love to copy the who panic message, but it only displays for about 15 seconds so I just look for key phrases I can google. After rebooting several more times I noticed that the drive was thrashing quite a bit and that the fsck process was running. Once I saw that I disabled an option that I had remembered setting previously which was to perform a background fsck on every boot.

After disabling the background fsck the system stabilized, but I was sure I had some kind of data problem so I dropped to a command prompt through an SSH session and manually executed the following command:

freenas:~# fsck -t ufs /dev/ad0s2
** /dev/ad0s2
** Last Mounted on /mnt/Data1
** Phase 1 - Check Blocks and Sizes
1 DUP I=4
UNEXPECTED SOFT UPDATE INCONSISTENCY

INTERNAL ERROR: dups with -p
UNEXPECTED SOFT UPDATE INCONSISTENCY
** Phase 1b - Rescan For More DUPS
1 DUP I=4
UNEXPECTED SOFT UPDATE INCONSISTENCY

** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
BAD/DUP DIR I=4 OWNER=root MODE=40700
SIZE=2048 MTIME=May 21 20:57 2009
CLEAR? [yn] y

** Phase 5 - Check Cyl groups
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? [yn] y

SUMMARY INFORMATION BAD
SALVAGE? [yn] y

ALLOCATED FRAGS 1-8 MARKED FREE
BLK(S) MISSING IN BIT MAPS
SALVAGE? [yn] y

87964 files, 23572718 used, 212474857 free (6921 frags, 26558492 blocks, 0.0% fragmentation)

Better, but let’s try again and make sure

freenas:~# fsck -t ufs /dev/ad0s2
** /dev/ad0s2
** Last Mounted on /mnt/Data1
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
LINK COUNT DIR I=2 OWNER=jackmang MODE=40777
SIZE=512 MTIME=Apr 4 17:02 2009 COUNT 9 SHOULD BE 8
ADJUST? [yn] y

UNREF FILE I=5 OWNER=root MODE=100400
SIZE=499132436784 MTIME=May 20 19:27 2009
RECONNECT? [yn] y

NO lost+found DIRECTORY
CREATE? [yn] y

UNREF FILE I=8 OWNER=root MODE=100400
SIZE=499132416000 MTIME=May 20 19:55 2009
RECONNECT? [yn] y

UNREF FILE I=11 OWNER=root MODE=100400
SIZE=0 MTIME=May 20 23:19 2009
RECONNECT? [yn] y

** Phase 5 - Check Cyl groups
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? [yn] y

SUMMARY INFORMATION BAD
SALVAGE? [yn] y

87965 files, 23572719 used, 212474856 free (6920 frags, 26558492 blocks, 0.0% fragmentation)

***** FILE SYSTEM WAS MODIFIED *****

and one more time to be sure

freenas:~# fsck -t ufs /dev/ad0s2
** /dev/ad0s2
** Last Mounted on /mnt/Data1
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
87965 files, 23572719 used, 212474856 free (6920 frags, 26558492 blocks, 0.0% fragmentation)

All fixed! I don’t know why the background fsck was causing a panic, but this worked. It may be possible that after stopping the background fsck command and rebooting that I could have used the GUI fsck command successfully, but hopefully I’ll never know. That was not fun.

October 2017
Sun Mon Tue Wed Thu Fri Sat
 << <   > >>
1 2 3 4 5 6 7
8 9 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24 25 26 27 28
29 30 31        
I'm a generalist, at least if I'm honest. In my job I am primarily a developer, but also a sysadmin, and (as little as possible) technical support. I know a little about a lot of things, a lot about some things, and everything about nothing. Here I will post random learnings...

Search

XML Feeds

User tools

blogging soft