[DEBUG PATCH] demonstration of hang in sched domain partitioning code
From: Paul Jackson
Date: Fri Oct 05 2007 - 05:24:45 EST
Nick,
I'm running into a problem trying to use your suggested
partition_sched_domains(cpumask_t *partition) routine the way
I thought I was supposed to be able to use it.
If I ask to set up the same partition twice in a row having
just one CPU in it (a singleton) the kernel hangs.
I don't know enough about kernel/sched.c to be able to debug
this very efficiently. Any chance you could take a look?
The following patch can be used to reproduce the problem
I'm seeing. I'm on an 8 CPU Altix using sn2_defconfig.
This patch applies to 2.6.23-rc8-mm2 immediately -after- the
existing patch:
cpuset-remove-sched-domain-hooks-from-cpusets.patch
To see the problem, do:
1) apply the patch, build and boot it
2) mount -t cpuset cpuset /dev/cpuset
3) cd /dev/cpuset
4) echo 1 > memory_pressure_enabled
5) echo 1 > memory_pressure_enabled
6) system hung ... big red switch time ...
Well, not entirely hung - but that task, and sooner
or later other tasks, hang enough so that you can't
login anymore or complete a natural reboot.
P.S. - the second writing, step (6) above, apparently
is not needed. I just had another boot freeze a couple
seconds after the first write.
The patch 'borrows' the infrequently used top cpuset file
file memory_pressure_enabled to let you directly invoke
partition_sched_domains() with a singleton containing
just whatever CPU number you write to it.
Here's the syslog output generated from the above
(I wrote "2", not "1" as above, when I generated the
following):
================================================================
Oct 5 01:38:27 margin kernel: Invoking partition_sched_domains for singleton on CPU 2
Oct 5 01:38:27 margin kernel: CPU2 attaching NULL sched-domain.
Oct 5 01:38:27 margin kernel: CPU2 attaching sched-domain:
Oct 5 01:38:27 margin kernel: domain 0: span 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000004
Oct 5 01:38:27 margin kernel: groups: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000004
Oct 5 01:38:27 margin kernel: Returned from partition_sched_domains.
Oct 5 01:38:30 margin kernel: Invoking partition_sched_domains for singleton on CPU 2
Oct 5 01:38:30 margin kernel: CPU2 attaching NULL sched-domain.
================================================================
---
include/linux/sched.h | 2 ++
kernel/cpuset.c | 12 ++++++++++++
kernel/sched.c | 14 +++++++++++++-
3 files changed, 27 insertions(+), 1 deletion(-)
--- 2.6.23-rc8-mm2.orig/include/linux/sched.h 2007-10-05 00:39:27.256416155 -0700
+++ 2.6.23-rc8-mm2/include/linux/sched.h 2007-10-05 00:41:21.918141772 -0700
@@ -718,6 +718,8 @@ struct sched_domain {
#endif
};
+extern int partition_sched_domains(cpumask_t *partition);
+
#endif /* CONFIG_SMP */
/*
--- 2.6.23-rc8-mm2.orig/kernel/cpuset.c 2007-10-05 00:38:13.195313659 -0700
+++ 2.6.23-rc8-mm2/kernel/cpuset.c 2007-10-05 00:50:28.258426364 -0700
@@ -1005,10 +1005,22 @@ done:
static int update_memory_pressure_enabled(struct cpuset *cs, char *buf)
{
+#ifdef THE_USUAL_CODE
if (simple_strtoul(buf, NULL, 10) != 0)
cpuset_memory_pressure_enabled = 1;
else
cpuset_memory_pressure_enabled = 0;
+#else /* partition_sched_domains debug hook */
+ int i;
+ cpumask_t singleton;
+
+ i = simple_strtoul(buf, NULL, 10);
+ cpus_clear(singleton);
+ cpu_set(i, singleton);
+ printk("Invoking partition_sched_domains for singleton on CPU %d\n", i);
+ partition_sched_domains(&singleton);
+ printk("Returned from partition_sched_domains.\n");
+#endif
return 0;
}
--- 2.6.23-rc8-mm2.orig/kernel/sched.c 2007-10-05 00:39:11.884187317 -0700
+++ 2.6.23-rc8-mm2/kernel/sched.c 2007-10-05 01:21:49.359083211 -0700
@@ -5493,7 +5493,7 @@ int __init migration_init(void)
int nr_cpu_ids __read_mostly = NR_CPUS;
EXPORT_SYMBOL(nr_cpu_ids);
-#undef SCHED_DOMAIN_DEBUG
+#define SCHED_DOMAIN_DEBUG
#ifdef SCHED_DOMAIN_DEBUG
static void sched_domain_debug(struct sched_domain *sd, int cpu)
{
@@ -6327,6 +6327,18 @@ static void detach_destroy_domains(const
arch_destroy_sched_domains(cpu_map);
}
+int partition_sched_domains(cpumask_t *partition)
+{
+ cpumask_t m;
+
+ cpus_and(m, *partition, cpu_online_map);
+ cpus_andnot(m, m, cpu_isolated_map);
+
+ /* Detach sched domains from all of the affected cpus */
+ detach_destroy_domains(&m);
+ return build_sched_domains(&m);
+}
+
#if defined(CONFIG_SCHED_MC) || defined(CONFIG_SCHED_SMT)
static int arch_reinit_sched_domains(void)
{
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <pj@xxxxxxx> 1.650.933.1373
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/