Rebooting the Magic Way
August 21st, 2008 by Cory Wright in
If you have ever had a hard drive fail on a remote server you may remember the feeling you had after trying to issue the following commands:
# reboot
bash: /sbin/reboot: Input/output error
# shutdown -r now
bash: /sbin/shutdown: Input/output error
Obviously, there is a problem with your drive. These commands are failing because the kernel is unable to load the /sbin/reboot and /sbin/shutdown binaries from the disk so that it can execute them.
A fsck on the next boot might be able to correct whatever is wrong with the disk, but first you need to get the system to reboot. If your machine is located at a managed hosting provider then you could submit a reboot ticket, but you'll have to wait for someone to take responsibility.
Wouldn't it be nice if there was a way to ask the kernel to reboot without needing to access the failing drive? Well, there is a way, and it is remarkably simple.
The "magic SysRq key" provides a way to send commands directly to the kernel through the /proc filesystem. It is enabled via a kernel compile time option, CONFIG_MAGIC_SYSRQ, which seems to be standard on most distributions. First you must activate the magic SysRq option:
echo 1 > /proc/sys/kernel/sysrq
When you are ready to reboot the machine simply run the following:
echo b > /proc/sysrq-trigger
This does not attempt to unmount or sync filesystems, so it should only be used when absolutely necessary, but if your drive is already failing then that may not be a concern.
In addition to rebooting the system the sysrq trick can be used to dump memory information to the console, sync all filesystems, remount all filesystems in read-only mode, send SIGTERM or SIGKILL to all processes except init, or power off the machine entirely, among other things.
Also, instead of echoing into /proc/sys/kernel/sysrq each time you can activate the magic SysRq key at system boot time using sysctl, where supported:
echo "kernel.sysrq = 1" >> /etc/sysctl.conf
If you would like to learn more about magic SysRq you can read the sysrq.txt file in the kernel documentation.
__________________________
Cory Wright is a blogger for LinuxJournal.com
Special Magazine Offer -- 2 Free Trial Issues!
Receive 2 free trial issues of Linux Journal as well as instant online access to current and past issues. There's NO RISK and NO OBLIGATION to buy. CLICK HERE for offer
Linux Journal: delivering readers the advice and inspiration they need to get the most out of their Linux systems since 1994.
Sorry, offer available in the US only. International orders, click here.
Delicious
Digg
Reddit
Newsvine
TechnoratiSubscribe now!
The Latest
Featured Videos
Linux Journal Live - eBook Readers and DRM
November 14th, 2008 by Shawn Powers in
The November 13, 2008 edition of Linux Journal Live! Shawn Powers and special guest, Linux Journal Author Daniel Bartholomew, talk e-book readers and Daniel's Kindle, DRM, and other goodness.
Run Your Windows Partition Without Rebooting
November 13th, 2008 by Elliot Isaacson in
Dual booting is a necessary evil and very inconvenient. What if you could run your windows partition in a virtual machine, so you wouldn't have to worry about rebooting anymore? With VMWare Workstation, you can.
Recently Popular
From the Magazine
December 2008, #176
The Oxford English Dictionary says the word "gadget" is a placeholder name for a technical item whose precise name one can't remember. Like that book-reader thingy from Amazon...what's it called? Spindle, Gindle...Kindle, that's it. Check it out in this month's gadget issue.
Other gadgets covered include the Nokia tablets, the BlackBerry, the Neo FreeRunner, the Dash Express, the Roku Netflix Player, the Kangaroo TV, The TomTom GO 930 and the MooBella Ice Cream System. On the larger hardware front, read the reviews of the Acer Aspire One and the YDL PowerStation. On the software front, check out the articles and columns on memcached, Samba security, Mutt, desktop gadgets, bash and Puppet. To wrap it all up, read Doc's thoughts on Google and the browser platform.
Than you so much for this
On November 16th, 2008 morgellons (not verified) says:
Than you so much for this trick. It should come in handy when rebooting my vps! for me site
morgellons
printer problems
On November 3rd, 2008 Cristian (not verified) says:
Do you happen to know a magic way to reboot my printers? That would be magic trick which I would love to learn... I just hate when printers start behaving very weird and ruining my whole day.
I like this blog.Nice work.
On September 5th, 2008 bilety lotnicze (not verified) says:
I like this blog.Nice work.
Less typing
On August 25th, 2008 Craig MIller (not verified) says:
If one is truely looking to do less typing and reboot, I suggest:
"init 6"
This however, does require the file system to be functioning (since it will walk thru the /etc/rc?.d/ shutdown scripts).
But really, how often does one need to shut down a system (or reboot) with no filesystem?
# init 6 bash:
On August 26th, 2008 Cory Wright says:
# init 6 bash: /sbin/init: Input/output error:)
Also, as several people have already commented, there are lot's of cases where you may need to reboot without accessing the filesystem. I've had to deal with the Dell RAID issue myself.
__________________________Cory Wright is a blogger for LinuxJournal.com
Excellent information!
On August 26th, 2008 Anonymous (not verified) says:
To Mr. Miller's question: "But really, how often does one need to shut down a system (or reboot) with no filesystem?", I would have to reply more often then we wished. It may just be this particular application, but we have 1500+ instances of a system running Linux offering file shares via nfs with which we encounter this very problem. Being able to remotely reboot such a system so that it can run fsck upon reboot is potentially a great time saver. Without this functionality, the time needed to coordinate with remote sites to physically reboot the boxes may lead to days when this service is unavailable. This also allows us to make a quicker determination of whether we merely have file system corruption, or hardware failure, and dispatch maintenance when required.
Not that I'm a Linux guru or
On August 25th, 2008 TsueDesu (not verified) says:
Not that I'm a Linux guru or anything...but if the system drive is failing, Why would you want to reboot the server/system? Wouldn't that render it totally unusable? After all, you're still linked in remotely, so you could rebuild it onto a working drive or at least move the system to a new drive. Or was this implied?
*scratching head in uncertainty*
I'll say its an interesting article and I'm sure there's a use for this for this.
TsueDesu a.k.a This is stick!
Hmm. I was stupid enough to
On October 25th, 2008 Mad Cow (not verified) says:
Hmm. I was stupid enough to make my a and b nameservers the same host. Now the disk is failing but I can't get to the host until monday at the earliest. This is great. It might just work. ;-)
It didn't work BTW. I have to visit the server on monday anyhow. Ho hum.
Dell RAID Problems
On August 25th, 2008 Will Reese says:
I've had Dell servers with PERC RAID cards crap out under heavy IO and cause the filesystem to become unusable until a reboot. The cards and the disks were fine, but the driver wasn't. Rebooting resolved the issue until we were able to upgrade to better hardware and drivers.
-- Will Reese
sohpet
On August 23rd, 2008 Anonymous (not verified) says:
thanks for sites
Sync and umount with magic sysrq keys
On August 22nd, 2008 messageid (not verified) says:
With the Magic SysRQ keys is also possibile to umount and sync all the mounted disk.
If you issue the following key:
R S E I U B
You will: Sync the disk, Term and the Kill all the process, Umount all the mounted disk and the Reboot the system; so you can "emlate" a correct shoutdown process.
I use
On September 6th, 2008 cuervo (not verified) says:
alt-sysrq-S, U, S, B. Sync, remount all filesystems R/O, sync again, hard reboot. I typically let around half a second between those. There's some standard sequence involving elephants, but I figured mine out when magic syswreck was first introduced, and stuck with it. (Hint: That means parent's sequence is probably better.)
FYI, from /usr/src/linux*/Documentation/sysrq.txt:
'b' - Will immediately reboot the system without syncing or unmounting
your disks.
'c' - Will perform a kexec reboot in order to take a crashdump.
'd' - Shows all locks that are held.
'e' - Send a SIGTERM to all processes, except for init.
'f' - Will call oom_kill to kill a memory hog process.
'g' - Used by kgdb on ppc and sh platforms.
'h' - Will display help (actually any other key than those listed
above will display help. but 'h' is easy to remember :-)
'i' - Send a SIGKILL to all processes, except for init.
'k' - Secure Access Key (SAK) Kills all programs on the current virtual
console. NOTE: See important comments below in SAK section.
'm' - Will dump current memory info to your console.
'n' - Used to make RT tasks nice-able
'o' - Will shut your system off (if configured and supported).
'p' - Will dump the current registers and flags to your console.
'q' - Will dump a list of all running timers.
'r' - Turns off keyboard raw mode and sets it to XLATE.
's' - Will attempt to sync all mounted filesystems.
't' - Will dump a list of current tasks and their information to your
console.
'u' - Will attempt to remount all mounted filesystems read-only.
'v' - Dumps Voyager SMP processor info to your console.
'w' - Dumps tasks that are in uninterruptable (blocked) state.
'x' - Used by xmon interface on ppc/powerpc platforms.
Not that easy
On August 22nd, 2008 Anonymous (not verified) says:
# echo 1 > /proc/sys/kernel/sysrq
bash: /bin/echo: Input/output error
Shell built-in
On August 22nd, 2008 Anonymous (not verified) says:
'echo' is also a shell built-in for any decent shell.
Shell built-in
On August 22nd, 2008 Anonymous (not verified) says:
'echo' is also a shell built-in for any decent shell.
echo IS built into bash, but
On August 22nd, 2008 Anonymous (not verified) says:
echo IS built into bash, but in some distros there's also a /bin/echo which is in the path. Try PATH="" echo > /proc/yourmom
In Bash, the precedence
On August 22nd, 2008 Anonymous (not verified) says:
In Bash, the precedence order is:
1. Alias
2. Function (not sure of the order of the first two)
3. Built-in
4. Program found via a $PATH search
The built-in will get used first. Most Linux distros run with Bash as the primary shell now.
Post new comment