Re: jbd2: don't wake kjournald unnecessarily

[Sedat Dilek]

On Mon, Jan 21, 2013 at 3:07 PM, Jan Kara <jack@xxxxxxx> wrote:
> On Mon 21-01-13 13:30:26, Sedat Dilek wrote:
>> On Mon, Jan 21, 2013 at 12:40 PM, Sedat Dilek <sedat.dilek@xxxxxxxxx> wrote:
>> > On Mon, Jan 21, 2013 at 11:47 AM, Jan Kara <jack@xxxxxxx> wrote:
>> >>   The traces don't suggest an ext4/jbd2 problem. What is happening is that
>> >> jbd2 is waiting for IO to finish and that never happens. Seeing that you
>> >> loop device I'd think it's some interaction of the loop device and
>> >> freezing. Can you reproduce the issue without the loop device (i.e. with
>> >> the filesystem directly on e.g. scsi disk)? I suspect the reason is
>> >> something like that the backing filesystem is already frozen so filesystem
>> >> on top of it cannot write all the data and hangs waiting for IO -> suspend
>> >> doesn't happen. Contents of /proc/mounts and losetup -l would help us
>> >> understand what's going on.
>> >>
>> >
>> > As said I am here in a very uncommon WUBI environment means my
>> > Ubuntu/precise rootfs-image lays on the Win7-partition (NTFS).
>> > Your explanation sounds reasonable to me as this line from my attached
>> > testcase causes the troubles.
>> >
>> >      echo mem > /sys/power/state && sleep 1
>> >
>> > So, /sys/ is not writable immediately after freezer ends
>> >
>> > I checked again and again my logs and have seen "starving" lines
>> > reported by rtkit-daemon, but did not really get wiser what they want
>> > to tell me. Stopping rtkit-daemon or resetting all or all-known
>> > threads before running my pm_test/freezer did not help, too.
>> >
>> > /usr/sbin/rtkitctl --help
>> > rtkitctl [options]
>> >
>> >   -h, --help         Show this help
>> >       --version      Show version
>> >
>> >       --reset-known  Reset real-time status of known threads
>> >       --reset-all    Reset real-time status of all threads
>> >       --start        Start RealtimeKit if it is not running already
>> >   -k, --exit         Terminate running RealtimeKit daemon
>> >
>> > Here are the outputs you wanted with some more (fstab, grub-config) etc.
>> > I have here no -l option for losetup command.
>> >
>> >
>> > - Sedat -
>> >
>> > P.S.: Outputs for Honza...
>> >
>> > $ sudo cat /proc/mounts
>> > rootfs / rootfs rw 0 0
>> > sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
>> > proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
>> > udev /dev devtmpfs rw,relatime,size=1966948k,nr_inodes=491737,mode=755 0 0
>> > devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
>> > tmpfs /run tmpfs rw,nosuid,relatime,size=788076k,mode=755 0 0
>> > /dev/sda2 /host fuseblk
>> > rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,blksize=4096
>> > 0 0
>> > /dev/loop0 / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
>> > none /sys/fs/fuse/connections fusectl rw,relatime 0 0
>> > none /sys/kernel/debug debugfs rw,relatime 0 0
>> > none /sys/kernel/security securityfs rw,relatime 0 0
>> > none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
>> > none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
>> > gvfs-fuse-daemon /home/wearefam/.gvfs fuse.gvfs-fuse-daemon
>> > rw,nosuid,nodev,relatime,user_id=1000,group_id=1000 0 0
>> >
>> > $ sudo losetup --all --verbose
>> > /dev/loop0: [0802]:17982 (/host/ubuntu/disks/root.disk)
>> >
>> > $ sudo losetup --find --verbose
>> > Loop device is /dev/loop1
>> > /dev/loop1
>> >
>> >  [ /etc/fstab ]
>> > # /etc/fstab: static file system information.
>> >
>> > # Use 'blkid' to print the universally unique identifier for a
>> > # device; this may be used with UUID= as a more robust way to name devices
>> > # that works even if disks are added and removed. See fstab(5).
>> >
>> > # <file system>                 <mount point>   <type>  <options>
>> >          <dump>  <pass>
>> > proc                            /proc           proc
>> > nodev,noexec,nosuid     0       0
>> > /host/ubuntu/disks/root.disk    /               ext4
>> > loop,errors=remount-ro  0       1
>> > /host/ubuntu/disks/swap.disk    none            swap    loop,sw
>> >          0       0
>> > - EOF -
>> >
>> > [ /boot/grub/grub.cfg ]
>> > ...
>> > menuentry 'Ubuntu, mit Linux 3.8.0-rc4-next20130121-1-iniza-generic'
>> > --class ubuntu --class gnu-linux --class gnu --class os {
>> >         set gfxpayload=$linux_gfx_mode
>> >         insmod part_msdos
>> >         insmod ntfs
>> >         set root='(hd0,msdos2)'
>> >         search --no-floppy --fs-uuid --set=root 001AADA61AAD9964
>> >         loopback loop0 /ubuntu/disks/root.disk
>> >         set root=(loop0)
>> >         linux   /boot/vmlinuz-3.8.0-rc4-next20130121-1-iniza-generic
>> > root=UUID=001AADA61AAD9964 loop=/ubuntu/disks/root.disk ro
>> >         initrd  /boot/initrd.img-3.8.0-rc4-next20130121-1-iniza-generic
>> > }
>> > ...
>> >
>>
>> Here some more useful outputs:
>>
>> $ LC_ALL=C df -h -T
>> Filesystem     Type      Size  Used Avail Use% Mounted on
>> rootfs         rootfs     17G   15G  1.5G  92% /
>> udev           devtmpfs  1.9G   12K  1.9G   1% /dev
>> tmpfs          tmpfs     770M  892K  769M   1% /run
>> /dev/sda2      fuseblk   444G   81G  364G  19% /host
>> /dev/loop0     ext4       17G   15G  1.5G  92% /
>> none           tmpfs     5.0M     0  5.0M   0% /run/lock
>> none           tmpfs     1.9G  260K  1.9G   1% /run/shm
>>
>> $ sudo LC_ALL=C fdisk -l /dev/sda
>>
>> Disk /dev/sda: 500.1 GB, 500107862016 bytes
>> 255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
>> Units = sectors of 1 * 512 = 512 bytes
>> Sector size (logical/physical): 512 bytes / 4096 bytes
>> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
>> Disk identifier: 0xcb9885ab
>>
>>    Device Boot      Start         End      Blocks   Id  System
>> /dev/sda1   *        2048      206847      102400    7  HPFS/NTFS/exFAT
>> /dev/sda2          206848   931299327   465546240    7  HPFS/NTFS/exFAT
>> /dev/sda3       931299328   976773119    22736896   27  Hidden NTFS WinRE
>>
>> I am still reflecting on any shitty userspace app is causing all the
>> trouble, but I have zero clue how to dig that...
>   Well, I think the outputs make it pretty clear. /dev/loop0 is a mounted
> image from fuse filesystem. Fuse daemon making filesystem accessible gets
> frozen before /dev/loop0 gets fully written out and so jbd2 journal thread
> hangs. Maybe Miklos (added to CC) could fill in some details / ideas but I
> think the setup like you have never really worked...
>

Thank you for the explanations.
Before doing some patches for Linux Test Project and replying to
LLVMLinux I was reflecting about FUSE when saw the 'df -h -T' line.
You might be right, that pm_test/freezer could never be OK here.
I wantend to test against Linux v3.8-rc4 to be sure.
Anyway, thanks for sharing your thoughts with me!

Let's see what Miklos says...

Regards,
- Sedat -

>                                                                 Honza
> --
> Jan Kara <jack@xxxxxxx>
> SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/