Friday, December 7. 2007
When performing system recovery, it is sometimes useful to capture the data from a disk drive into an image file which can then be manipulated on another system -- this is useful when hardware failure is imminent. You can copy a disk's contents over the network using a command such as:
The -C option specifies compression, which may or may not be beneficial, depending on the speed of the network connection and the speed of the local and remote CPUs.
This resulting file (disk1.img in this example) is an exact copy of the data on the original disk. The same type of disk image is used for Xen and KVM virtualization.
The file command will report this type of file as an x86 boot sector:
You can view the partition table of the disk image using fdisk -l. However, to access the individual partitions within the image, you must first use losetup to make the image accessible as a block device:
This contents of the disk1.img file are now available through the block device /dev/loop0. (The -f option specifies that the first available /dev/loopN device should be used, and the -v enables verbose mode so that you will be told the name of the block device). You can now use kpartx to create a block device for each partition within the image:
The partition block devices are found in /dev/mapper and are named with the original block device name (loop0) with the letter "p" and the partition number appended. The file -s command will analyze the contents of each partition:
In this example, the first partition contains a filesystem which may be directly mounted (e.g., with mkdir /mnt/x1 ; mount /dev/mapper/loop0p1 /mnt/x1).
The second partition contains swap space, which is probably not of any interest. The third partition contains an LVM physical volume (PV), so you will need to use vgscan and vgchange to gain access to logical volumes contained therein:
Note that in this case the discovered VG name is zephyr. Once these steps have been performed, the zephyr LVs can then be mounted in the usual way to provide access to their contents.
When you are done using the disk image, reverse the procedure to take the disk image out of use:
Thursday, December 6. 2007
When you boot from a Fedora installation disc and enter rescue mode, you have the option of having the filesystems from your Fedora installation mounted at /mnt/sysimage. If you do not select this option, or if it fails, RAID devices will not be configured for use (as is also the case with LVM, as discussed yesterday).
On a Fedora system, RAID arrays are managed by the mdadm utility. This program expects the RAID configuration to be available at /etc/mdadm.conf. In order to use mdadm without this configuration file, it is necessary to create a dummy configuration file. This is a multi-step process:
Wednesday, December 5. 2007
Logical Volume Management (LVM) is a powerful storage system layer which abstracts the logical view of storage from the actual physical layout. It is automatically configured in the default Fedora storage configuration. LVM enables you to grow and shrink volumes, add storage from new devices to existing volumes, and migrate volumes between storage devices, all without taking the system offline. In fact, the system-config-lvm package provides a convenient way to perform all of these operations using a graphical interface.
However, you can't (yet) shrink a filesystem while it is mounted. This isn't a limitation of LVM itself, but of most filesystems, including ext3. To shrink a partition that is always mounted when the system is running -- such as the root filesystem -- it's necessary to boot from another medium, which is where rescue mode comes in (see yesterday's article). It may also be necessary to use rescue mode to recover from some particularly nasty corruption or misconfiguration issues.
To do LVM and filesystem-resizing work within the Fedora rescue mode, boot the rescue mode from disc as usual, but select Skip when the system offers to mount your hard-disk partitions. Unfortunately, this will mean that your volume groups will not be detected or activated, so you will need to do that manually.
The commands normally used for logical volume management such as vgdisplay, pvcreate, and lvreduce are actually symbolic links to a single executable named lvm. These symbolic links are not available in rescue mode, so you must explicitly use the lvm command followed by the operation you wish to perform: if you wish to do a vgscan, for example, enter the command lvm vgscan.
Therefore, to discover and activate all volume groups, you must execute these commands:
You can display information about the logical volumes using lvm lvs (or lvm lvdisplay for a more verbose display):
Note that the LV and VG names are not very descriptive -- which is why it's a good idea to override the default names during system installation. In this case, we know the approximate size of the LV containing the root filesystem, so we can determine that it is LogVol00 in VolGroup00.
Armed with this information, you can now check (fsck) the filesystem, shrink the filesystem within that logical volume, and then shrink the LV. Because of the potential for rounding errors, it's best to shrink the filesystem to a size slightly smaller than the new LV size, resize the LV, and then grow the filesystem to fully fill the LV:
Note that the operations following the first resize2fs do not need to be performed in rescue mode -- you can reboot the system at that point and continue after the system has booted normally. Earlier versions of the Fedora rescue mode may not set up the symbolic link /dev/VolumeGroup/LogicalVolume, so you may need to refer to /dev/mapper/VolumeGroup-LogicalVolume (e.g., use /dev/mapper/VolGroup00-LogVol00 in place of /dev/VolGroup00/LogVol00).
Tuesday, December 4. 2007
When a system is too damaged to permit booting from the hard disk drive, it's necessary to boot from another medium. The Fedora installation discs support a "Rescue mode" in which the system is booted from the CD and the hard disk partitions are optionally mounted for access.
To access this mode, boot from your Fedora install media and select "Rescue installed system" from the boot menu using the arrow keys and Enter or by pressing the R key (if you need to edit the boot options first -- to disable ACPI, for example -- navigate to the Rescue option with the arrow keys and press Tab).
The kerenel will boot from CD and the system will prompt you to select a keyboard style and language from scrollable lists of options. You will then be given the opportunity to enable the network interfaces on the system, either by entering the IP information or by using DHCP.
The system will then present a dialog stating that the rescue environment is about to find and mount the filesystems from your hard disk Fedora installation, and asks if you wish to continue. This is a critical question: if your filesystems are intact and you wish to access the data that is in them, you can select Continue, the default option. If you are concerned about the state of your filesystems and want to ensure that they will not be altered, but still want to access them, select Read-Only. If your filesystems are damaged, you have multiple Fedora installations, or you wish to perform an operation such as reducing the size of the root filesystem, choose Skip. After some additional messages, you will be presented with a root shell prompt.
If you have elected to continue with read/write mounting of your filesystems, all of the files from your Fedora installation should be available under /mnt/sysimage -- so the normal /etc/passwd file will be available at /mnt/sysimage/etc/passwd.
Although regular Fedora commands and utilities are available in rescue mode, most of them will not work because of the altered paths. You can work around this issue by temporarily changing the root directory using the chroot command:
However, you need to be aware that files within the mounted Fedora filesystems will not have been updated during the rescue mode boot process, including /etc/mtab and /var/log/messages. You can compensate for this by some degree by getting the information from other places (such as dmesg for kernel messages and /proc/mounts for mount information).
If you have been forced to use rescue mode because your system's Grub bootloader code has become damaged or has been overwritten by another bootloader, you can reinstall the Grub bootloader in rescue mode:
You can also use rescue mode to set the root password, create alternate superuser accounts, or change or remove a boot password. Whether these are important recovery operations or a type of attack depends only on the context in which they are performed. You can slow down such an attack by configuring the system BIOS to boot only from the hard disk and installing a BIOS password, but that can be reset using a motherboard jumper in most cases. The moral of the story: if you don't have physical security, you don't have system security.
When you are finished using rescue mode, type exit or press Ctrl-D twice. The system will then reboot.
Monday, December 3. 2007
The simplest recovery mode available is "single user" or "maintainence mode". This is a special runlevel which will take you directly into a superuser (root) shell prompt without executing most of the normal system startup scripts. Because this mode boots from the normal filesystems, it will only work when the kernel, initrd (initial ramdisk), and basic filesystem are intact; however, it does not require a valid /etc/inittab, /etc/passwd, or /etc/shadow file or a working set of boot scripts, so it can be used in many situations where a normal boot would fail.
To enter single user mode, interrupt the normal grub boot display (which usually shows a message counting down the seconds until Fedora is booted, or which may show a menu of available operating systems if you have altered the default grub boot configuration) by pressing the spacebar. If you have a boot password, press P and enter the password now.
Press the A key to append boot options to the default kernel, and type a space and the letter s to indicate that you want to enter single-user mode, as shown in Figure 1. Press Enter to continue booting.
The system will boot and then go directly to a root shell prompt (Figure 2). You can perform any normal administrative functions at this prompt - but since the normal system startup has not taken place, you will not be able to use networking, printing, or other services. If your init scripts are intact, you can start specific services, such as network or cups, using the service command: service nameOfService start
Operations commonly performed in single user mode include:
Note that single user mode presents an extreme security risk: any person who has physical access to your system can use single user mode to gain root (unrestricted) access to your system. A boot password will make it slightly more difficult to execute this type of attack. If you did not create a boot password at installation, you can add one at any time:
Monday, December 3. 2007
Posted by Chris Tyler in Administrivia, System Recovery Week at 00:01 | Comments (0) | Trackbacks (0)
This week is a special Focus Week where we'll be examining the techniques used by experienced system administrators to recover a system or perform special maintainence in unusual circumstances. This will include the use of single-user mode, rescue mode (booted from optical disc), LVM and RAID recovery, and dealing with disk images from other computers.
(Page 1 of 1, totaling 6 entries)
The Fedora Daily Package exists to highlight lesser-known Fedora Linux packages each weekday-- with a special article each Wednesday taking a behind-the-scenes look at some of the configuration options and packaging details that make Fedora tick. For more information, please see the Fedora Daily Package Welcome posting.
To suggest a future Fedora Daily Package, use the Suggest a Package box below.
Suggest a Package
Chris Tyler's Blog
Error on line 137 of /var/www/fedora-dailypackage/bundled-libs/Onyx/RSS.php: The specified file could not be opened. (#404)
Syndicate This Blog
Original material in the Fedora Daily Package is Copyright ©2006-2008 by Chris Tyler. This material may be distributed under the Creative Commons Attribution-Share Alike 2.5 Canada License or the Open Publication License.