Re: [Bug #11516] severe performance degradation on x86_64 going from 2.6.26-rc9 -> 2.6.27-rc5

[Jason Vas Dias]

Hi - 
Yes, this bug is still a problem with both the latest 2.6.27-rc6 kernel (from Linus' tree 2008-09-21)
and with the latest fedora 10 kernel .

CPU Frequency switching is completely disabled both when powernow-k8 (the correct cpufreq module for my
x86_64 AMD TL-64x2 2.2GHz CPU) is installed as a module or is built-in , and the CPU frequency remains
at its lowest setting; attempts to modify /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq
and /sys/devices/system/cpu/cpu0/cpufreq/scaling_setspeed are not honored, even though 
/sys/devices/system/cpu/cpu0/cpufreq/governor is "userspace" 
and scaling_min_freq < scaling_setspeed > scaling_max_freq .

I see no messages from powernow-k8 indicating that it is aware it was unable to set the speed, though
I do see a message if I attempt to set an invalid speed (eg 600000) .

With 2.6.26-rc9, I get a default CPU clock frequency of 2200000 ; with 2.6.27-rc6, it becomes 800000 and
is not switchable. For some reason, powernow-k8 does not autoload with UDEV; but I don't really need it if
the speed is already set to its highest level.

On 2.6.27-rc6. after it manages to boot, any low-latency drivers time out (eg. USB, Terminal, Keyboard, Network) 
and the machine does not get through the boot-up sequence without becoming overloaded by the kernel's debugging log messages -
neither the network , the terminal or the keyboard work usably. 

Building a kernel with USB completely disabled and turning off debug log messages allows the machine to boot 
(after @ 15 minutes) but the speed is still at its lowest setting and cannot be changed.

Also, 2.6.27-rc6 is unable to reboot the machine: it can put the machine into the "HALT" state, with nothing displayed
on the screen, but the machine does not power-off until manual reset with the power-button. Then, after the machine
has powered-down, it cannot be powered up until the power-on button is depressed for at least two sections an released TWICE
in a row.

Neither booting with 'acpi=0' or compilation without USB support has any effect; it happens the same with a totally
standard Fedora 10 install with the latest 2.6.27 kernel as with my own heavily customized distro , and occurs when either
udev-{124,126,128} are installed, and regardless of compiling the kernel with gcc-4.2.4 , gcc-3.4.6, or gcc-4.3.2 .

Any help / suggestions would be greatly appreciated ! I've attached the Config, System.map and boot.log (compressed) .

Regards,
Jason 


On Sunday 21 September 2008 14:54:18 Rafael J. Wysocki wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.26.  Please verify if it still should be listed and let me know
> (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11516
> Subject		: severe performance degradation on x86_64 going from 2.6.26-rc9 -&gt; 2.6.27= -rc5
> Submitter	: Jason Vas Dias <jason.vas.dias@xxxxxxxxx>
> Date		: 2008-09-07 13:59 (15 days old)
> 
> 
> 
On Sunday 07 September 2008 18:03:06 Andrew Morton wrote:
> Re: [Bugme-new] [Bug 11516] New: severe performance degradation on x86_64
> going from 2.6.26-rc9 -> 2.6.27-rc5 From:
> Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
>   To:
> linux-usb@xxxxxxxxxxxxxxx
>   CC:
> bugme-daemon@xxxxxxxxxxxxxxxxxxx, jason.vas.dias@xxxxxxxxx
>   Date:
> 2008-09-07 18:03
>   Â
>   Spam Status:ÂSpamassassin 0% probability of being spam.
>
> Full report:
> No, hits=-3.349 required=5 tests=AWL,BAYES_00,OSDL_HEADER_SUBJECT_BRACKETED
> Â (switched to email. ÂPlease respond via emailed reply-to-all, not via the
> bugzilla web interface).
>
> On Sun, Â7 Sep 2008 13:59:04 -0700 (PDT) bugme-daemon@xxxxxxxxxxxxxxxxxxx wrote:
> > http://bugzilla.kernel.org/show_bug.cgi?id=11516
> >
> > Â Â Â Â Â ÂSummary: severe performance degradation on x86_64 going from
> > Â Â Â Â Â Â Â Â Â Â 2.6.26-rc9 -> 2.6.27-rc5
> > Â Â Â Â Â ÂProduct: Other
> > Â Â Â Â Â ÂVersion: 2.5
> > Â Â ÂKernelVersion: 2.6.27-rc9
> > Â Â Â Â Â Platform: All
> > Â Â Â Â OS/Version: Linux
> > Â Â Â Â Â Â Â Tree: Mainline
> > Â Â Â Â Â Â Status: NEW
> > Â Â Â Â Â Severity: normal
> > Â Â Â Â Â Priority: P1
> > Â Â Â Â ÂComponent: Other
> > Â Â Â Â AssignedTo: other_other@xxxxxxxxxxxxxxxxxxxx
> > Â Â Â Â ReportedBy: jason.vas.dias@xxxxxxxxx
> >
> >
> > Latest working kernel version:
> > 2.6.26-rc9
> > (git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
> > @2008-07-10)
> >
> > Earliest failing kernel version:
> > 2.6.27-rc9
> > (git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
> > @2008-09-07)
> >
> > Distribution:
> > Gentoo / my Own + Fedora 10
> >
> > Hardware Environment:
> > x86_64 AMD Turion TL-64 64-bit dual core
> >
> > Software Environment:
> > gcc-4.2.4 , glibc-2.8.90, both latest from savannah CVS @ 2008-07-10
> >
> > Problem Description:
> >
> > After my monthly kernel update ( git-pull from Linus' tree + build )
> > the whole machine started crawling instead of sprinting - this was
> > on my "own distro" (originally Gentoo based).
> >
> > Updating Fedora 10 from 2.6.26-0.115.rc9.git2.fc10 to
> > 2.6.27-0.305.rc5.git6.fc10 had the same effect
> > on a bog-standard Fedora 10 installation.
> >
> > Before I get out the toolbox and start investigating in-depth
> > (which I don't really have time to do at the moment), I thought
> > I'd check here first if anyone has seen this issue and if there
> > is any quick configuration / boot arg fix ?
> >
> > To quantify, normally boot-up takes less than 1 minute with the 2.6.26
> > kernel; with the 2.6.27 kernel, it takes over 10 minutes, and the screen
> > is flooded with these "hub event change" messages - and the terminal
> > scrolls visibly slowly, and seems to be unable to keep up with even
> > my extremely slow typing rate.
> >
> > I'm attaching the config-2.6.26-rc9.jvd and config-2.6.27-rc5.jvd files,
> > and klogd logs from a bootup of each kernel.
> >
> > Any advice suggestions as to how to fix would be gratefully received.
> >
> >
> > Steps to reproduce:
> >
> > build and install latest kernel from Linus' tree.
>
> I've tentatively categorised this post-2.6.26 regression under USB.
>
> Sep Â7 15:56:33 localhost kernel: [ Â 21.190720] device: 'vcsa11':
> device_unregister Sep Â7 15:56:33 localhost kernel: [ Â 21.190721] PM:
> Removing info for No Bus:vcsa11 Sep Â7 15:56:33 localhost kernel: [ Â
> 21.190788] device: 'vcsa11': device_create_release Sep Â7 15:56:33
> localhost kernel: [ Â 21.205252] hub 1-0:1.0: state 7 ports 2 chg 0000 evt
> 0000 Sep Â7 15:56:33 localhost kernel: [ Â 21.205305] hub 3-0:1.0: state 7
> ports 2 chg 0000 evt 0000 Sep Â7 15:56:33 localhost kernel: [ Â 21.460430]
> hub 1-0:1.0: state 7 ports 2 chg 0000 evt 0000 Sep Â7 15:56:33 localhost
> kernel: [ Â 21.460454] hub 3-0:1.0: state 7 ports 2 chg 0000 evt 0000 Sep
> Â7 15:56:33 localhost kernel: [ Â 21.704357] hub 1-0:1.0: state 7 ports 2
> chg 0000 evt 0000 Sep Â7 15:56:33 localhost kernel: [ Â 21.704366] hub
> 3-0:1.0: state 7 ports 2 chg 0000 evt 0000 Sep Â7 15:56:33 localhost
> kernel: [ Â 21.954352] hub 1-0:1.0: state 7 ports 2 chg 0000 evt 0000 Sep
> Â7 15:56:33 localhost kernel: [ Â 21.954374] hub 3-0:1.0: state 7 ports 2
> chg 0000 evt 0000 Sep Â7 15:56:33 localhost kernel: [ Â 22.204433] hub
> 1-0:1.0: state 7 ports 2 chg 0000 evt 0000 Sep Â7 15:56:33 localhost
> kernel: [ Â 22.204441] hub 3-0:1.0: state 7 ports 2 chg 0000 evt 0000 Sep
> Â7 15:56:33 localhost kernel: [ Â 22.454376] hub 1-0:1.0: state 7 ports 2
> chg 0000 evt 0000 Sep Â7 15:56:33 localhost kernel: [ Â 22.454384] hub
> 3-0:1.0: state 7 ports 2 chg 0000 evt 0000 Sep Â7 15:56:33 localhost
> kernel: [ Â 22.704406] hub 1-0:1.0: state 7 ports 2 chg 0000 evt 0000 Sep
> Â7 15:56:33 localhost kernel: [ Â 22.704414] hub 3-0:1.0: state 7 ports 2
> chg 0000 evt 0000 Sep Â7 15:56:33 localhost kernel: [ Â 22.960311] hub
> 1-0:1.0: state 7 ports 2 chg 0000 evt 0000 Sep Â7 15:56:33 localhost
> kernel: [ Â 22.960335] hub 3-0:1.0: state 7 ports 2 chg 0000 evt 0000 Sep
> Â7 15:56:34 localhost kernel: [ Â 23.204362] hub 1-0:1.0: state 7 ports 2
> chg 0000 evt 0000 Sep Â7 15:56:34 localhost kernel: [ Â 23.204384] hub
> 3-0:1.0: state 7 ports 2 chg 0000 evt 0000 Sep Â7 15:56:34 localhost
> kernel: [ Â 23.454386] hub 1-0:1.0: state 7 ports 2 chg 0000 evt 0000 Sep
> Â7 15:56:34 localhost kernel: [ Â 23.454394] hub 3-0:1.0: state 7 ports 2
> chg 0000 evt 0000 Sep Â7 15:56:34 localhost kernel: [ Â 23.704357] hub
> 1-0:1.0: state 7 ports 2 chg 0000 evt 0000 Sep Â7 15:56:34 localhost
> kernel: [ Â 23.704366] hub 3-0:1.0: state 7 ports 2 chg 0000 evt 0000 Sep
> Â7 15:56:34 localhost kernel: [ Â 23.954377] hub 1-0:1.0: state 7 ports 2
> chg 0000 evt 0000 Sep Â7 15:56:34 localhost kernel: [ Â 23.954696] hub
> 3-0:1.0: state 7 ports 2 chg 0000 evt 0000 Sep Â7 15:56:35 localhost
> kernel: [ Â 24.204380] hub 1-0:1.0: state 7 ports 2 chg 0000 evt 0000 Sep
> Â7 15:56:35 localhost kernel: [ Â 24.204389] hub 3-0:1.0: state 7 ports 2
> chg 0000 evt 0000 Sep Â7 15:56:35 localhost kernel: [ Â 24.454398] hub
> 1-0:1.0: state 7 ports 2 chg 0000 evt 0000 Sep Â7 15:56:35 localhost
> kernel: [ Â 24.454406] hub 3-0:1.0: state 7 ports 2 chg 0000 evt 0000 Sep
> Â7 15:56:35 localhost kernel: [ Â 24.704379] hub 1-0:1.0: state 7 ports 2
> chg 0000 evt 0000 Sep Â7 15:56:35 localhost kernel: [ Â 24.704402] hub
> 3-0:1.0: state 7 ports 2 chg 0000 evt 0000 Sep Â7 15:56:35 localhost
> kernel: [ Â 24.954381] hub 1-0:1.0: state 7 ports 2 chg 0000 evt 0000
>
> but it might be an x86 problem or anything, it's unclear.
>
> Jason, perhaps you could test a CONFIG_USB=n kernel?

Attachment: kernel-2.6.27-rc6.jvd.logs.bz2
Description: BZip2 compressed data