Re: fsync on large files

G.W. Wettstein (greg@wind.enjellic.com)
Mon, 15 Feb 1999 03:18:08 -0600


On Feb 13, 11:29pm, "Theodore Y. Ts'o" wrote:
} Subject: Re: fsync on large files

Good evening/morning to everyone.

> From: "Alan Curry" <pacman-kernel@cqc.com>
> Date: Fri, 12 Feb 1999 22:36:51 -0500 (EST)
>
> I'm coming late to this topic, but with a good reason: I just figured
> out today that syslogd's fsyncs have been the cause of some major
> performance problems on an ISP's central server. Load averages have
> been unreasonably high and gradually getting worse. Today we realized
> that there is a weekly cycle to it, and it matches the cycle of the
> log rotation of /var/log/messages. As this log file grows (currently
> 36 megs with 2 days left before rotation) and beyond, with syslogd
> fsync'ing every line it writes, syslogd hangs for long periods of
> time, and when syslogd is hung, lots of other stuff hangs.

> Hmm. That's an interesting problem. This is the first time someone's
> called it to my attention that syslog was calling fsync() all the time,
> and obviously this causes a problem if there's a lot of syslogging going
> on and the log files are large.

This is fixable from syslogd's perspective as well. One of the
extensions that I incorporated into sysklogd was the ability to
selectively disable fsync'ing of logfiles on a per logfile basis.
This dramatically reduces load on busy servers with the nominal risk
of missing a line or two of log output in the event of system
problems.

The following exerpt is from one of the big IMAP servers at our local
university:

*.=info;*.=notice;cron.none;mail.none /var/log/messages
*.=debug /var/log/debug
*.warn /var/log/syslog
mail.* -/var/log/mail

With this configuration syslogd will fsync each line of output on all
the logfiles except for /var/log/mail. This seems to dramatically
reduce system load and CPU consumption by syslogd in all cases where I
have used this directive.

Greg

}-- End of excerpt from "Theodore Y. Ts'o"

As always,
Dr. G.W. Wettstein Enjellic Systems Development - Specialists in
4206 N. 19th Ave. intranet based enterprise information solutions.
Fargo, ND 58102 WWW: http://www.enjellic.com
Phone: 701-281-1686 EMAIL: greg@wind.enjellic.com
------------------------------------------------------------------------------
"A raccoon tangled with a 23,000 volt line today. The results blacked
out 1400 homes and, of course, one raccoon."
-- Steel City News

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/