Loading...

xsan-users@lists.apple.com

[Prev] Thread [Next]  |  [Prev] Date [Next]

metadata journal corrupt, cant mount, cant fix ? please help - Xsan 1.4.2 Svavar Örn Eysteinsson Thu Sep 15 19:01:05 2011

Just today my XSAN volume, or metadata/client server machine just restarted out of the blue.

I have a XSAN 1.4.2 made of 2 PPC XServe G5's running on Leopard, and Tiger.
Connected to 2x XRAIDS that are made of :

XRaid2-Ctrl2-LUN0 on device: /dev/rdisk2     -> 2.73TB
XRaid-Ctrl1-LUN0 on device: /dev/rdisk3         -> 2.73TB
XRaid-Ctrl2-LUN0 on device: /dev/rdisk4         -> 2.73TB
Raid2-Ctrl1-MD-Mirror on device: /dev/rdisk5 -> METADATA LUN
Xraid2-Ctrl1-LUN1 on device: /dev/rdisk6     -> 2.73TB


The RAID system dosn't say anything, everything seems to be OKEY, except that one of the Controllers where out, so I took it out and inserted it again into the system
and got the green light lid.


This is what my system.log shows just after sucessful reboot :

[code]
PortMapper: FSS 'FitonVOL'[0] (pid 120) at 10.100.0.16:49155 is registered.
Sep 15 15:15:31 elvis fsm[120]: Xsan FSS 'FitonVOL[0]': PANIC: /Library/Filesystems/Xsan/bin/fsm ASSERT failed "htons(di->di_marker_st) == DINODEMARKER_ST" file fsm.c, line 880 Sep 15 15:15:31 elvis fsm[120]: PANIC: /Library/Filesystems/Xsan/bin/fsm ASSERT failed "htons(di->di_marker_st) == DINODEMARKER_ST" file fsm.c, line 880 Sep 15 15:15:31 elvis fsm[120]: Xsan FSS 'FitonVOL[0]': PANIC: aborting threads now.
Sep 15 15:15:31 elvis fsmpm[82]: PortMapper: Local FSD client is registered.
Sep 15 15:15:35 elvis servermgrd[47]: servermgr_calendar: created default calendar virtual host Sep 15 15:15:35 elvis servermgrd[47]: xsan: [47] Done waiting for fsmpm to start Sep 15 15:15:35 elvis servermgrd[47]: *** -[NSLock unlock]: lock (<NSLock: 0x1a9ad0> '(null)') unlocked when not locked
Sep 15 15:15:35 elvis servermgrd[47]: *** Break on _NSLockError() to debug.
Sep 15 15:15:35 elvis servermgrd[47]: xsan: [47/103B80] ERROR: get_fsmvol_at_index: Could not connect to FSM because File System Manager "FitonVOL" on 10.100.0.16 is on standby. Sep 15 15:15:35 elvis servermgrd[47]: xsan: [47/103B80] ERROR: get_quotas_for_fsmvol_named(FitonVOL): Could not connect to FSM because File System Manager "FitonVOL" on 10.100.0.16 is on standby.
PortMapper: FSS 'FitonVOL' disconnected.
Sep 15 15:15:49 elvis fsmpm[82]: PortMapper: Rescanning for available CVFS volumes. Sep 15 15:15:51 elvis fsmpm[82]: Portmapper: FSS 'FitonVOL' (pid 120) exited on signal 4 PortMapper: RESTART FSS service 'FitonVOL[0]' on host elvis.internal.domain.com. Sep 15 15:16:00 elvis fsmpm[82]: PortMapper: Starting FSS service 'FitonVOL[0]' on host elvis.internal.domain.com. Sep 15 15:16:00 elvis fsmpm[82]: PortMapper: FSS 'FitonVOL'[0] (pid 185) at elvis.internal.domain.com:49166 is registered. Sep 15 15:16:00 elvis fsm[185]: Xsan FSS 'FitonVOL[0]': PANIC: /Library/Filesystems/Xsan/bin/fsm ASSERT failed "htons(di->di_marker_st) == DINODEMARKER_ST" file fsm.c, line 880 Sep 15 15:16:00 elvis fsm[185]: PANIC: /Library/Filesystems/Xsan/bin/fsm ASSERT failed "htons(di->di_marker_st) == DINODEMARKER_ST" file fsm.c, line 880 Sep 15 15:16:01 elvis fsm[185]: Xsan FSS 'FitonVOL[0]': PANIC: aborting threads now.
Sep 15 15:16:04 elvis fsmpm[82]: NSS: Cannot acquire port for 10.100.0.20
Sep 15 15:16:07 elvis ReportCrash[188]: Formulating crash report for process fsm[185] Sep 15 15:16:37 elvis servermgrd[47]: xsan: [47/103B80] ERROR: get_fsm_process_stats(FitonVOL): Unable to find pid of fsm
Sep 15 15:16:49 elvis fsmpm[82]: NSS: Cannot acquire port for 10.100.0.20
Sep 15 15:17:37 elvis servermgrd[47]: xsan: [47/103B80] ERROR: get_fsm_process_stats(FitonVOL): Unable to find pid of fsm
Sep 15 15:17:40 elvis fsmpm[82]: NSS: Cannot acquire port for 10.100.0.20
Sep 15 15:18:10: --- last message repeated 1 time ---
Sep 15 15:18:34 elvis fsmpm[82]: NSS: Cannot acquire port for 10.100.0.20
Sep 15 15:18:37 elvis servermgrd[47]: xsan: [47/103B80] ERROR: get_fsm_process_stats(FitonVOL): Unable to find pid of fsm
Sep 15 15:18:48 elvis fsmpm[82]: NSS: Cannot acquire port for 10.100.0.20
Sep 15 15:19:18 elvis fsmpm[82]: NSS: Cannot acquire port for 10.100.0.20
Sep 15 15:19:37 elvis servermgrd[47]: xsan: [47/103B80] ERROR: get_fsm_process_stats(FitonVOL): Unable to find pid of fsm
Sep 15 15:19:55 elvis fsmpm[82]: NSS: Cannot acquire port for 10.100.0.20
---- || ----
---- || ----
[/code]

I issued a sudo cvfsck -vn FitonVOL and it reported many dead ionodes and some stuff.
After that, I was going to run cvfsck -wv FitonVOL but it wouldn't allow me.

[code]
elvis:log root# sudo cvfsck -wv

BUILD INFO:
[EMAIL PROTECTED] Server Revision 2.7.201 Build 7.40
[EMAIL PROTECTED] Built for Darwin 8.0
[EMAIL PROTECTED] Created on Thu Oct 11 19:05:39 PDT 2007

Xsan File Systems on host elvis.internal.domain.com:

1)  FitonVOL

Choose a file system by number (1-1) or zero (0) to exit -> 1


** WARNING ** This file system check may modify the meta-data of the

              --- [FitonVOL] ---

              file system. This procedure cannot be un-done!

Do you want to proceed? (Y/N) -> y



Created directory /tmp/cvfsck500a for temporary files.
journal_check error: Journal Descriptor marker corrupt


*Warning*: The file system journal is corrupt. A new journal must be built
 to recover from this error. If you choose to continue cvfsck
 will verify the file system. After cvfsck verifies or repairs
 the file system, cvupdatefs must be run to rebuild the journal.
Continue? [(y)es/(N)o] y

Super Block information.
  FS Created On               : Thu Jan  1 00:00:00 1970
  Inode Version               : '0.0'
  File System Status          : *Dirty*
  Allocated Inodes            : 0
  Free Inodes                 : 0
  FL Blocks                   : 0
  Next Inode Chunk            : 0x0

Creating MetaDataSP allocation check file.
Creating FitonGogn allocation check file.
Creating FitonGogn2 allocation check file.
Creating FitonGogn3 allocation check file.
Creating FitonGogn4 allocation check file.
Stripe Group MetaDataSP                     (  0) 0x3a6f700 blocks.
Stripe Group FitonGogn                      (  1) 0x2baa1e00 blocks.
Stripe Group FitonGogn2                     (  2) 0x2baa1e00 blocks.
Stripe Group FitonGogn3                     (  3) 0x2baa1e00 blocks.
Stripe Group FitonGogn4                     (  4) 0x2baa0600 blocks.

Building Inode Index Database 0 (100%).
Journal extent (base-0x0 end-0x0) block 0x0 doubly allocated!
[/code]

I have no idea what I can do right now !! Can't find any information
about recreating metadata from a corruption..

Please, some advice and help.

p.s. the FS creation date stamp is totally wrong !!




_______________________________________________
Do not post admin requests to the list. They will be ignored.
Xsan-Users mailing list      ([EMAIL PROTECTED])
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/xsan-users/alexiscircle%40gmail.com

This email sent to [EMAIL PROTECTED]