一次linux救援模式修复

环境描述：
系统版本：CentOS 6.5 64

故障描述：
/bin/sh目录文件缺失，导致CentOS启动卡条；

排查过程：
昨天晚上接到小伙伴反馈他们的yum源服务器重启之后无响应，遂登录Vcenter检查发现系统卡开机进度条了，检查发现有文件缺失的报错；

init: Failed to spawn readahead-collector main process: unable to execute: No such file or directory
init: Failed to spawn rcS pre-start process: unable to execute: No such file or directory
init: Failed to spawn readahead main process: unable to execute: No such file or directory
init: Failed to spawn readahead-collector post-stop process: unable to execute: No such file or directory
init: Failed to spawn rcS post-stop process: unable to execute: No such file or directory
init: Failed to spawn readahead-disable-services main process: unable to execute: No such file or directory

进入Vcenter挂载Centos6.5 ISO进入救援模式，进行如下操作：
通过tcsh切换根文件系统后用rpm命令安装替换原来的bash

bash-4.1# chroot /mnt/sysimage/
chroot: failed to run command  `/bin/sh `: No such file or directory

提示没有找到/bin/sh,怀疑是Bash文件损坏；进行如下操作:

bash-4.1# chroot /mnt/sysimage/ /bin/tcsh

指定切换后的shell为/bin/tcsh后成功；挂载安装盘重新安装bash

[root@localhost ~]# mkdir /media
[root@localhost ~]# mount /dev/cdrom /media
mount: block device /dev/sr0 is write-protected, mounting read-only
[root@localhost ~]# cd /media/Packages/
[root@localhost Packages]# rpm -ivh bash-4.1.2-15.el6.x86_64.rpm --replacepkgs
Preparing...                ########################################### [100%]
   1:bash                   ########################################### [100%]
[root@localhost Packages]# exit
exit
bash-4.1# reboot

满心欢喜的等待重启，结果还是失败，同样的错误； 然后呢，进入带网络的救援模式，从另外的机器上CP了一个/bin/bash文件，然后ln /bin/sh后机器重启恢复；

事件总结：
留意过程中的每一个报错信息，第一次chroot报错的时候就应该发现应该是/bin/sh出了问题；