PROBLEM: Can ping address, but traceroute gets ENETDOWN

From: Terry Phelps
Date: Tue Jul 17 2012 - 09:04:19 EST

I'm seeing, to me, totally illogical behavior with my IPv4 networking.
Can someone please help me isolate the problem better?

I have at least EIGHT servers with the same symptom. All are running
Oracle "Unbreakable Enterprise Kernel 2". Oracle numbers this kernel
2.6.39.*, but it is "based on the 3.0.16 kernel". I don't know exactly
what patches might have been applied. The symptom I see is:

I'm SSH'ed into the server from my desk another network. All is well.
Then either (1) SSH freezes, or (2) I exit SSH, and can't SHH to it
Then I ping the server from my desk. It FAILS.
I ping the server from a second machine on my desk (same network). It works.
If I keep pinging from my desktop, where the SSH just failed, it will
NEVER get a response. I've let it ping for DAYS.
But if I stop pinging for 5 minutes or so, it'll work just fine again.
While things are "hosed", I am able to ping and ssh from my second
desktop to the server just fine.
If I SSH to the server, it CAN ping my desktop, but it CANNOT traceroute to it.
If I leave the ping going (and failing), and go to the server and "ip
route flush cache", the pings start working immediately.
I can get the problem from other desktops on other networks, but I
have never seen it from another server on the same network.

It gets stranger. Here are some commands run on the server, while the
pings from my desktop are failing. The failing pings are coming from The machine right next that one is .23, and it works

I have ONE NIC in the box, and I have no reason to think it isn't
configured properly.

# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:50:56:9A:00:17
inet addr: Bcast: Mask:
RX packets:246266059 errors:0 dropped:85001 overruns:0 frame:0
TX packets:290982046 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:70745127855 (65.8 GiB) TX bytes:27490797799 (25.6 GiB)

lo Link encap:Local Loopback
inet addr: Mask:
RX packets:258548668 errors:0 dropped:0 overruns:0 frame:0
TX packets:258548668 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:226377171068 (210.8 GiB) TX bytes:226377171068 (210.8 GiB)

The server can ping my desktop just fine:

# ping
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=127 time=0.827 ms
64 bytes from icmp_seq=2 ttl=127 time=0.739 ms
64 bytes from icmp_seq=3 ttl=127 time=0.725 ms

But a traceroute to the same destination says "network is down":

# traceroute
traceroute to (, 30 hops max, 40 byte packets
send: Network is down

A syscall trace of traceroute shows the sendto() call getting a
ENETDOWN response:

setsockopt(3, SOL_IP, IP_MTU_DISCOVER, [0], 4) = 0
setsockopt(3, SOL_SOCKET, SO_TIMESTAMP, [1], 4) = 0
setsockopt(3, SOL_IP, IP_TTL, [1], 4) = 0
setsockopt(3, SOL_IP, IP_RECVERR, [1], 4) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(33434),
sin_addr=inet_addr("")}, 28) = 0
sendto(3, "@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_"..., 40, 0, NULL, 0) = -1
ENETDOWN (Network is down)

Yet traceroute (and ping) to a machine on the same network is fine:

# traceroute
traceroute to (, 30 hops max, 40 byte packets
1 ( 1.304 ms 1.614 ms 1.886 ms
2 ( 0.521 ms 0.566 ms 0.562 ms

I have a default route, and no other routes defined:

# netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface UG 0 0 0 eth0 U 0 0 0 eth0 U 0 0 0 eth0

Here are my route cache entries for the network I'm trying to talk to:

# netstat -nrC|grep 192.168.118 1500 0 239 eth0 l 16436 0 0 lo 1500 0 0 eth0 l 16436 0 0 lo 1500 0 239 eth0 1500 0 0 eth0 1500 0 239 eth0 1500 0 0 eth0 1500 0 0 eth0

And finally, tcpdump shows that the pings from my desktop ARE
arriving. They are simply
not being replied to:

# tcpdump -np host
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
10:20:48.950240 IP > ICMP echo request, id
2, seq 35155, length 40
10:20:54.956584 IP > ICMP echo request, id
2, seq 35158, length 40
10:21:00.959048 IP > ICMP echo request, id
2, seq 35161, length 40
10:21:06.964326 IP > ICMP echo request, id
2, seq 35164, length 40

If you could PLEASE advise me on where to go from here, I would
greatly appreciate it. I can't imagine what would cause these

Here is the ver_linux output:

Linux 2.6.39-200.29.1.el5uek #1 SMP Fri Jul 6
08:01:33 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

Gnu C 4.1.2
Gnu make 3.81
util-linux 2.13-pre7
mount 2.13-pre7
module-init-tools 3.3-pre2
e2fsprogs 1.39
pcmciautils 014
quota-tools 3.13.
PPP 2.4.4
Linux C Library 2.5
Dynamic linker (ldd) 2.5
Procps 3.2.7
Net-tools 1.60
Kbd 1.12
Sh-utils 5.97
udev 095
wireless-tools 28
Modules Loaded autofs4 hidp rfcomm bluetooth rfkill lockd
sunrpc be2iscsi iscsi_boot_sysfs ib_iser rdma_cm ib_cm iw_cm ib_sa
ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic uio ipv6 cxgb3i libcxgbi
cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi video sbs sbshc
hed acpi_memhotplug acpi_ipmi ipmi_msghandler lp sg sr_mod cdrom
snd_seq_dummy serio_raw e1000 vmw_balloon snd_seq_oss
snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss
snd_pcm snd_timer snd soundcore snd_page_alloc pcspkr parport_pc
i2c_piix4 i2c_core parport floppy pata_acpi ata_generic dm_snapshot
dm_zero dm_mirror dm_region_hash dm_log dm_mod ata_piix shpchp mptspi
mptscsih mptbase scsi_transport_spi sd_mod crc_t10dif ext3 jbd mbcache

Terry Phelps
American Commercial Lines
Jeffersonville, IN
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at