Recovering from led 552, 554 or 556

 A LED code of 552, 554, or 556 during a standard disk based boot indicates a failure occurred during the varyon of the rootvg volume group.

   Some known causes of an LED 552, 554, or 556 are:

   a corrupted file system
   a corrupted Journaled File System (JFS) log device
   a bad IPL-device record or bad IPL-device magic number; the magic number indicates the device type
   a corrupted copy of the Object Data Manager (ODM) database on the boot logical volume
   a fixed disk (hard disk) in the inactive state in the root volume group

   Recovery procedure

   To diagnose and fix the problem, boot to a Service mode shell and run the fsck command on each file system.
   If the file system check fails, you may need to perform other steps.

   WARNING: Do not use this document if the system is a /usr client, diskless client, or dataless client.

   Boot your system into a limited function maintenance shell from bootable AIX media to use this recovery procedure.
  
   Follow the screen prompts to the Welcome to Base OS menu.
   Choose Start Maintenance Mode for System Recovery (Option 3).
   The next screen displays prompts for the Maintenance menu.
   Choose Access a Root Volume Group (Option 1).
   The next screen displays a warning that indicates you will not be able to return to the Base
   OS menu without rebooting. 
   Choose 0 continue.
   The next screen displays information about all volume groups on the system. 
   Select the root volume group by number.
   The logical volumes in rootvg will be displayed with two options below.
   Choose Access this volume group and start a shell before mounting the file systems (Option 2).

   If you receive errors from the preceding option, do not continue with the rest of this procedure.

   Run the following commands to check and repair file systems.  

   fsck -p /dev/hd4
   fsck -p /dev/hd2
   fsck -p /dev/hd9var
   fsck -p /dev/hd3
   fsck -p /dev/hd1

   NOTE: The -y option gives the fsck command permission to repair file system corruption when necessary.
   This flag can be used to avoid having to manually answer multiple confirmation prompts, however, use of this flag can cause permanent data loss in some situations.

   If fsck indicates that block 8 could not be read, the file system is probably unrecoverable. 
   The easiest way to fix an unrecoverable file system is to recreate it.

   This involves deleting it from the system and restoring it from a very current system backup.
   Note that hd9var and hd3 can be recreated, but hd4 and hd2 cannot be recreated.

   If hd4 and/or hd2 is unrecoverable, AIX must be reinstalled or restored from system backup.

   If fsck indicates that a file system has an unknown log record type, or if fsck fails in the logredo process
   A corruption of the JFS log logical volume has been detected.  Use the logform command to reformat it.

             /usr/sbin/logform /dev/hd8

   Answer yes when asked if you want to destroy the log.

   If the file system checks were successful continue

   The majority of instances of LED 552, 554, and 556 will be resolved at this point.

   If you still have an LED 552, 554, or 556, you may try the following steps.

ATTENTION: The following steps will overwrite your Object Data Manager (ODM) database files with a very primitive, minimal ODM database. Due to the potential loss of user configuration data caused by this procedure, it should only be used as a last resort effort to gain access to your system to attempt to back up any data that you can. It is NOT recommended to use the following procedure in lieu of restoring from a system backup.

   Run the following commands, which remove much of the system's configuration and save it to a backup directory.

   mount /dev/hd4 /mnt
   mount /dev/hd2 /mnt/usr
   mkdir /mnt/etc/objrepos/bak
   cp /mnt/etc/objrepos/Cu* /mnt/etc/objrepos/bak
   cp /etc/objrepos/Cu* /mnt/etc/objrepos
   umount /dev/hd2
   umount /dev/hd4

   Determine which disk is the boot disk with the lslv command.

   lslv -m hd5

   Save the clean ODM database to the boot logical volume. 
   (# is the number of the fixed disk, determined with the previous command.)

   savebase -d /dev/hdisk0

   If you are running the Andrew File System (AFS), use the following commands to find out whether you have    more than one version of the v3fshelper file.

   cd /sbin/helpers
   ls -l v3fshelper*

   If there is a version of v3fshelper marked as original (for example,v3fshelper.orig), run the following commands:

   cp v3fshelper v3fshelper.afs
   cp v3fshelper.orig v3fshelper

   WARNING: Do not proceed further if the system is a /usr client, diskless client, or dataless client.

   Make sure that hd5 is on the edge of the drive and if it is more than 1 partition that the partitions
   are contiguous. For systems of 5.1 and above, make sure that hd5 is greater than 12 MB:

   lslv hd5

   Should be like ...

   lslv -m hd5 LP PP1 PV1 PP2 PV2 PP3 PV3 0001 0001 hdisk2 0002 0002 hdisk2

   Recreate the boot image (hdisk# is the fixed disk determined in step 11):

   bosboot -a -d /dev/hdisk0

   Make sure the bootlist is set correctly:

   bootlist -o normal -m

   Make changes, if necessary:

   bootlist -m normal hdiskX cdX

   Make sure that the disk drive that you have chosen as your bootable device has a yes next to it:

   ipl_varyon -i

   Example:

   PVNAME BOOT DEVICE PVID VOLUME GROUP ID
   hdisk1 NO 0007b53cbfd04a9000000000000000000007b53c00004c00
   hdisk4 NO 0007b53c1244625d00000000000000000007b53c00004c00
   hdisk2 YES 0007b53c8ffd631200000000000000000007b53c00004c00

   From the above example, hdisk2 would be a bootable disk drive while hdisk1 and hdisk4 would not be.

   copy the AFS file system helper back to v3fshelper:

   cp v3fshelper.afs v3fshelper
   sync;sync;sync;reboot