wget giving Segmentation fault – Managing your servers can streamline the performance of your team by allowing them to complete complex tasks faster. Plus, it can enable them to detect problems early on before they get out of hand and compromise your business. As a result, the risk of experiencing operational setbacks is drastically lower.
But the only way to make the most of your server management is to perform it correctly. And to help you do so, this article will share nine tips on improving your server management and fix some problem about linux, debian, vps, wget, segmentation-fault.
I recently did
apt-get update followed by
apt-get upgrade on my Debian VPS server. Suddenly (at least)
wget stopped working with error
Segmentation fault. It’s not even trying to do something. So after some googling I’ve installed
gdb and tried to debug wget:
~# gdb wget (gdb) run Starting program: /usr/bin/wget Error while mapping shared library sections: '/lib/libc.so.5': not in executable format: File truncated [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Program received signal SIGSEGV, Segmentation fault. 0x0000000000000000 in ?? ()
/lib/ gives me this:
~# ls -l /lib/ total 288 lrwxrwxrwx 1 root root 21 Oct 14 08:52 cpp -> /etc/alternatives/cpp drwxr-xr-x 2 root root 4096 May 7 2015 discover drwxr-xr-x 2 root root 4096 May 7 2015 ifupdown drwxr-xr-x 2 root root 4096 May 7 2015 init -rwxr-xr-x 1 root root 71416 Oct 5 2014 klibc-IpHGKKbZiB_yZ7GPagmQz2GwVAQ.so -rwxr-xr-x 1 root root 23904 Jan 26 22:08 libc.so.5 lrwxrwxrwx 1 root root 17 Nov 8 2014 libip4tc.so.0 -> libip4tc.so.0.1.0 -rw-r--r-- 1 root root 31416 Nov 8 2014 libip4tc.so.0.1.0 lrwxrwxrwx 1 root root 17 Nov 8 2014 libip6tc.so.0 -> libip6tc.so.0.1.0 -rw-r--r-- 1 root root 31416 Nov 8 2014 libip6tc.so.0.1.0 lrwxrwxrwx 1 root root 15 Nov 8 2014 libipq.so.0 -> libipq.so.0.0.0 -rw-r--r-- 1 root root 10544 Nov 8 2014 libipq.so.0.0.0 lrwxrwxrwx 1 root root 16 Nov 8 2014 libiptc.so.0 -> libiptc.so.0.0.0 -rw-r--r-- 1 root root 5816 Nov 8 2014 libiptc.so.0.0.0 lrwxrwxrwx 1 root root 20 Nov 8 2014 libxtables.so.10 -> libxtables.so.10.0.0 -rw-r--r-- 1 root root 51896 Nov 8 2014 libxtables.so.10.0.0 drwxr-xr-x 3 root root 4096 May 7 2015 lsb drwxr-xr-x 2 root root 4096 May 7 2015 modprobe.d drwxr-xr-x 3 root root 4096 May 7 2015 modules drwxr-xr-x 2 root root 4096 May 7 2015 startpar drwxr-xr-x 8 root root 4096 Jan 27 15:19 systemd drwxr-xr-x 15 root root 4096 May 7 2015 terminfo drwxr-xr-x 4 root root 4096 Jan 27 15:13 udev drwxr-xr-x 4 root root 12288 Jan 27 15:20 x86_64-linux-gnu drwxr-xr-x 2 root root 4096 May 7 2015 xtables
As we can see that “something happened” to libc.so.5 on Jan-26 but I don’t really know what.
dmesg | grep "wget":
(truncated) [247973.065698] wget: segfault at 0 ip (null) sp 00007ffc31e37558 error 14 in wget[400000+63000] [248272.905735] wget: segfault at 0 ip (null) sp 00007ffcf3e14038 error 14 in wget[400000+63000] [248572.716856] wget: segfault at 0 ip (null) sp 00007fffdcc9e2d8 error 14 in wget[400000+63000] [248873.509595] wget: segfault at 0 ip (null) sp 00007ffd84ad95d8 error 14 in wget[400000+63000] [249173.463737] wget: segfault at 0 ip (null) sp 00007ffca87ba518 error 14 in wget[400000+63000] [249472.948078] wget: segfault at 0 ip (null) sp 00007ffe54fd32a8 error 14 in wget[400000+63000] [249776.082815] wget: segfault at 0 ip (null) sp 00007ffecc36c758 error 14 in wget[400000+63000] [250072.819988] wget: segfault at 0 ip (null) sp 00007fff07f4f9d8 error 14 in wget[400000+63000] [250373.130105] wget: segfault at 0 ip (null) sp 00007ffcec9c01d8 error 14 in wget[400000+63000] [250672.640891] wget: segfault at 0 ip (null) sp 00007ffe9f78cc08 error 14 in wget[400000+63000] [250975.950585] wget: segfault at 0 ip (null) sp 00007fff46300128 error 14 in wget[400000+63000]
Server is a VPS running on Debian at Wedos.com:
~# uname -a Linux server1 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 GNU/Linux
Could anybody help me what’s going on and how to fix segfault? It’s production server with live traffic.
The key here is the line
'/lib/libc.so.5': not in executable format: File truncated
The C library is one of the core components of a Unix-like system, including Linux. That it is corrupted means that virtually nothing will work. You are lucky that your system is limping along, but it’s quite possible that starting or restarting anything will fail.
DO NOT REBOOT THE SYSTEM. In its current state, it almost certainly won’t come back up.
Especially because this is a VPS, and because this happened together with an upgrade, I’m wondering if maybe you ran out of resources like disk space or inodes. I wouldn’t put it beyond something like this to have broken more things on your system, so you may need to implement some form of disaster recovery. The easiest option might very well be to restore your most recent backup.
At the very least, you will have to reinstall the C library. On my Debian system, this is provided by the
libc6 family of packages, primarily
libc6. You can run
dpkg -S libc.so to list all packages that provide files with names containing the substring
libc.so. If you can’t run this on the VPS in question, set up a separate system (could be a low-provisioned VM somewhere; that doesn’t really matter much) with the same release of Debian, update the package lists, and run the command there; it will tell you which packages are candidates for reinstallation.
Once you have the very basics back to working, you may want to check out How to verify that package-installed files match originals? for how to verify that (at least mostly) nothing else has been corrupted as well.
Personally, especially since presumably this is an important system (since you state it is a “production server with live traffic”), I’d probably prefer just initiating whatever disaster recovery plans you have in place. Chances are that it will be easier and quicker than trying to repair things in-place and never quite knowing what errors you might possibly have missed. This is really just a variation of needing to nuke from orbit to be sure.