Using Kexec and Kdump For System Troubleshooting

Virtuozzo Containers 4.0 comes with the support of a new crash dumping mechanism based on the following kernel components:

• Kexec: this component, if installed on the Hardware Node, allows you to directly reboot to a new kernel (also known as capture kernel) from the context of an already running one without going through the bootloader stage of the system boot process, which drastically reduces reboot-related system downtime. When used for troubleshooting purposes, Kexec can be configured to boot into a new kernel on the system crash while preserving the crashed kernel memory contents and passing the control over this contents to the capture kernel.
• Kdump: this component provides a highly reliable dump generation and capturing mechanism. It uses Kexec to fast boot into the capture kernel in a system crash event and, after the kernel is loaded, captures the kernel crash dump.

Currently, you can use Kexec and Kdump to troubleshoot Hardware Nodes running Linux distributions with 'RHEL 5'-based kernels installed (e.g. RHEL 5). To start using these components on your Hardware Node, you should perform the following operations:

1. Install the kexec-tools package on the Node. For example:

   # rpm -ihv kexec-tools-1.8-1.i386.rpm

2. Open the bootloader configuration file for editing (/etc/grub.conf) and append the following string to the end of the kernel line:
   • If you are running the 32-bit or x86-64-bit version of Virtuozzo Containers:

     "crashkernel=128M@16M"

   • If you are running the IA64-bit version of Virtuozzo Containers:

     "crashkernel=256M@256M"

   128M and 256M in the examples above denote the amount of memory to be reserved for the capture kernel (please keep in mind that this memory is taken from RAM and cannot be used by the system) and 16M and 256M indicate at what physical address the reserved memory section is to be started.

3. Reboot the Hardware Node:

   # shutdown -r now

4. Enable the Kdump service on the Node:

   # chkconfig kdump on

5. Start the Kdump service:

   # service kdump start

Now in the case of a system crash, Kexec will boot to the capture kernel without clearing the crashed kernel memory and then pass the control to this kernel. Kdump, in its turn, will capture the dump and put it to the /var/crash directory on the Hardware Node. This directory is used by Kdump by default for storing system crash dumps. You can leave the default location or redefine it by editing the Kdump configuration file (/etc/kdump.conf). Anyway, you should make sure that the corresponding directory has enough free space to store system dumps.

In addition to storing kernel dumps on the local filesystem, Kdump can be configured to place crash dumps to the following locations:

• RAW disk partitions;
• dedicated filesystems (i.e. formatted partitions that are not used by the system);
• NFS mounted filesystems;
• remote systems (using ssh and scp).

For example, the following session demonstrates what tasks should be completed to start using a remote server for storing crash dumps. Setting a remote location for keeping dumps may prove useful when the local filesystem is corrupted and saving a crash dump to your local system may only worsen the situation:

1. Create a user on the remote server. This user should have the following rights and permissions in respect of the server:
   • S/he should be able to log in to the remote server from the Hardware Node via passwordless SSH keys.
   • As Kdump will move the created crash dumps to the default /var/crash directory on the remote server using the scp utility, the user should have the permissions to wright to this directory.
2. Add the following string to the Kdump configuration file (/etc/kdump.conf) on the Hardware Node:

   "net <username>@<server>"

   where <username> is the name of the user created on the remote server on Step 1 and <server> denotes the IP address or hostname of the remote server.

3. You can also change the default path on the remote server (/var/crash) where all kernel crash dumps will be collected by adding the following string to the Kdump configuration file:

   "path <dump_path>"

   where <dump_path> is the path to be used for storing crash dumps. In this case you should make sure that the user has the permissions to wright to the specified directory.

4. Set up the passwordless SSH host keys and distribute them to the remote server:

   # service kdump propagate

5. Save the changes made to Kdump:

   # service kdump restart

Please send us your feedback on this help page