Re: [Lxc-users] Containers slow to start after 1600

From: Eric W. Biederman
Date: Fri Mar 22 2013 - 17:32:14 EST

Benoit Lourdelet <blourdel@xxxxxxxxxxx> writes:

> Hello,
> I tried multiple kernels with similar results.
> I ran the following on 3.8.2
> $ cat > << 'EOF'
> #!/bin/bash
> for i in $(seq 1 2000) ; do
> ip link add a$i type veth peer name b$i
> done
> $ perf record -a
> $ perf report

Interesting. And the snmp_fold_field is an interesting hint.

It looks like something is listing all of your interfaces probably
between interface creation. And that will definitely take this
process from O(NlogN) to O(N^2).

Although I find the find_next_bit call also concerning.

snmp_fold_field is used when reading /proc/net/snmp /proc/net/netstat
and when running ip link to show all of the interfaces.

I suspect find_next_bit is coming out of the per cpu allocator, and it
might be partially responsible. But that doesn't feel right.

Regardless from this trace it looks like the performance problem on
creation is something in userspace is calling /sbin/ip to list all of
the interfaces and that is causing the interface creation to slow down.

Oh I see what is happening. In iproute.

So the limitation at this point is in iproute, and not in the kernel.

And with the following patch the time to create 5000 devices drops from
120 to 11 seconds for me. So it looks like the kernel is fine it is
whatever userspace tools that are of interest that are the bottleneck.

Although a quicker way to do a name to ifindex mapping might be


diff --git a/ip/iplink.c b/ip/iplink.c
index ad33611..58af369 100644
--- a/ip/iplink.c
+++ b/ip/iplink.c
@@ -533,8 +533,6 @@ static int iplink_modify(int cmd, unsigned int flags, int argc, char **argv)

- ll_init_map(&rth);
if (!(flags & NLM_F_CREATE)) {
if (!dev) {
fprintf(stderr, "Not enough information: \"dev\" "
@@ -542,6 +540,7 @@ static int iplink_modify(int cmd, unsigned int flags, int argc, char **argv)

+ ll_init_map(&rth);
req.i.ifi_index = ll_name_to_index(dev);
if (req.i.ifi_index == 0) {
fprintf(stderr, "Cannot find device \"%s\"\n", dev);
@@ -555,6 +554,7 @@ static int iplink_modify(int cmd, unsigned int flags, int argc, char **argv)
if (link) {
int ifindex;

+ ll_init_map(&rth);
ifindex = ll_name_to_index(link);
if (ifindex == 0) {
fprintf(stderr, "Cannot find device \"%s\"\n",

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at