On Tue, May 16, 2000 at 02:29:45PM +0200, Sasi Peter wrote:
> Hi!
> The following has just happened to me, and caused to lose the trust I had
> for linux's reliability.
> Here in the dormitory of Electrical Engineering of the Technical
> University we have a LAN of ~500 PC-s, and in my box I have a 58GB
> partition on which we share MPEG movies and mp3s. I keep deleting the old
> files, but somehow fee space just did not come back, and now finally I did
> a df and a du -s and recognized, that there was a 15GB difference! Then I
> have shut down all the services, actually all the processes not part of
> the kernel + init + sh, and tied to umount it. No success, it said it was
> busy. fuser -vm showed no users. Restart in single mode, e2fsck: clean.
> e2fsck -f, and here it came:
> - lots of deleted inodes wrong
> - at bitmap differences it listed every single block
> - lots of wrong free count for groups (it was zeroed out)
> - lots of wrong directory couts for groups (random but wrong numbers)
> And this came after 15 days of uptime. It just did not free up space, that
> is what I happened to notice ant then this.
Sounds to me like you have an application which is opening files
and holding them open even after they're deleted. The free space is not
released until the file is closed. The fact that you found the file
system "busy" when you went to unmount it, tends to confirm that. I'm
not totally sure if fuser is going to pick up on just the files that
it finds in the directory structure (missing the open but deleted inodes)
or if it looks at the inodes that are being held open, the man page is
not clear about that. Guess I should find that out, but I use lsof more
than fuser. At least lsof distinctly mentions the appearance of such
orphaned files, so I know it shows them up.
The fsck results are also consistant with what I would expect
if you had an application opening files and not closing them.
> How can this happen???
Several ways and almost all application...
You say you are "sharing" files. How? Is this nfs, or smb
(Samba)? You say these are mp3's and mpegs. Are you using something
like gnapster? My first suspects would be associated with your sharing
methods and servers.
Considering your environment (network of 500+ machines in a
University dorm) and what you are using this system for (file sharing
of mp3's and mpegs) I would also consider it to be a distinct possibility
that your system may have been compromised and is being used by other people
for other purposes. The problem may not be system or application but
may be "liveware" related. Intruders are known to store "warez" files
(illegal software) on systems and hide the presence of those files through
various means. Often, they will also install a "root kit" which hides the
presence of an intrusion by providing fake copies of netstat, ps, du, df,
ls, sum, md5sum, and other utilities so that when you check the system
things "should" seem normal, when they are not. When a root kit is
in place, you won't see the backdoor processes because "ps" will filter
them out and doing an md5sum on ls will return the correct result because
the trojan'ed md5sum will report back the reference copy, not the trojaned
file. You won't notice backdoor ports opened and listening, because netstat
is not reporting them. It's HIGHLY UNLIKELY that you would notice that
a connection FROM port 5000 to your ftp server gives you an immediate root
shell, while connections from any other port gives you a normal ftp prompt.
In other words, when they get in, they hide REAL GOOD, but not perfect.
The reason I raise this possibility is that the first indication
of trouble like this is inconsistancies in the system behavior, often
unexplained loss of diskspace and inconsistancies between utitilities
like df and du. Both the errant application and malicious intruder
can both explain all of the symptoms as you've reported them. Which
is more likely? Normally the former. In your case and your situation,
I certainly wouldn't discount the later.
> I have 2.2.14pre14 with usb raid and ide patches, the partition was on a
> raid0 over 4 disks two occupying the two PIIX4 channes on the
> motherboard (UDMA2), two on the CMD648 (UDMA4).
> Would please somebody explain, how this could have happened?
> PS: after e2fsck now I have the 15GB back...
This makes me inclined to believe that it's more likely an
ill-behaved application rather than an intrusion, but I would still
sweep your system CAREFULLY from known good sources of software.
You don't want to find yourself being an unwitting participant in
the next DDoS attack by providing a home for a zombie...
> -- SaPE
> Peter, Sasi <sape@sch.hu>
Mike
-- Michael H. Warfield | (770) 985-6132 | mhw@WittsEnd.com (The Mad Wizard) | (770) 331-2437 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0xDF1DD471 | possible worlds. A pessimist is sure of it!- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Tue May 23 2000 - 21:00:10 EST