Re: [PATCH] memcg: fix NULL pointer dereference in __mem_cgroup_usage_unregister_event

[brookxu]

Sorry, the calltrace in the email is from the old system, i will fix it later. this problem also exists in the latest master branch. The
following is the calltrace corresponding to the latest kernel:

[Â 135.675108] BUG: kernel NULL pointer dereference, address: 0000000000000004
[Â 135.675350] #PF: supervisor write access in kernel mode
[Â 135.675579] #PF: error_code(0x0002) - not-present page
[Â 135.675816] PGD 800000033058e067 P4D 800000033058e067 PUD 3355ce067 PMD 0
[Â 135.676080] Oops: 0002 [#1] SMP PTI
[Â 135.676332] CPU: 2 PID: 14012 Comm: kworker/2:6 Kdump: loaded Not tainted 5.6.0-rc4 #3
[Â 135.676610] Hardware name: LENOVO 20AWS01K00/20AWS01K00, BIOS GLET70WW (2.24 ) 05/21/2014
[Â 135.676909] Workqueue: events memcg_event_remove
[Â 135.677192] RIP: 0010:__mem_cgroup_usage_unregister_event+0xb3/0x190
[Â 135.677473] Code: c0 31 d2 48 63 ca 48 c1 e1 04 4c 39 64 0e 08 0f 95 c1 83 c2 01 0f b6 c9 01 c8 39 fa 75 e5 85 c0 4d 8b 7d 08 0f 84 ad 00 00 00 <41> 89 47 04 41 c7 07 ff ff ff ff 31 c0 49 8b 4d 00 31 d2 8b 71 04
[Â 135.677825] RSP: 0018:ffffb47e01c4fe18 EFLAGS: 00010202
[Â 135.678186] RAX: 0000000000000001 RBX: ffff8bb223a8a000 RCX: 0000000000000001
[Â 135.678548] RDX: 0000000000000001 RSI: ffff8bb22fb83540 RDI: 0000000000000001
[Â 135.678912] RBP: ffffb47e01c4fe48 R08: 0000000000000000 R09: 0000000000000010
[Â 135.679287] R10: 000000000000000c R11: 071c71c71c71c71c R12: ffff8bb226aba880
[Â 135.679670] R13: ffff8bb223a8a480 R14: 0000000000000000 R15: 0000000000000000
[Â 135.680066] FS:Â 0000000000000000(0000) GS:ffff8bb242680000(0000) knlGS:0000000000000000
[Â 135.680475] CS:Â 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Â 135.680894] CR2: 0000000000000004 CR3: 000000032c29c003 CR4: 00000000001606e0
[Â 135.681325] Call Trace:
[Â 135.681763]Â memcg_event_remove+0x32/0x90
[Â 135.682209]Â process_one_work+0x172/0x380
[Â 135.682657]Â worker_thread+0x49/0x3f0
[Â 135.683111]Â kthread+0xf8/0x130
[Â 135.683570]Â ? max_active_store+0x80/0x80
[Â 135.684034]Â ? kthread_bind+0x10/0x10
[Â 135.684506]Â ret_from_fork+0x35/0x40
[Â 135.684981] Modules linked in: netconsole cmac xt_CHECKSUM xt_MASQUERADE tun bridge stp llc ccm ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep sunrpc intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm rtl8192cu rtl_usb rtl8192c_common irqbypass rtlwifi mac80211 uvcvideo snd_hda_codec_realtek crct10dif_pclmul snd_hda_codec_hdmi snd_hda_codec_generic iTCO_wdt crc32_pclmul wmi_bmof mei_wdt iTCO_vendor_support snd_hda_intel btusb videobuf2_vmalloc ghash_clmulni_intel snd_intel_dspcfg btrtl videobuf2_memops snd_hda_codec btbcm videobuf2_v4l2 btintel snd_hda_core videobuf2_common bluetooth aesni_intel crypto_simd cryptd cfg80211
videodev
[Â 135.685003]Â snd_hwdep glue_helper snd_seq mc ecdh_generic libarc4 snd_seq_device ecc pcspkr joydev snd_pcm thinkpad_acpi snd_timer ledtrig_audio sg i2c_i801 lpc_ich wmi snd rfkill mei_me soundcore mei ie31200_edac sch_fq_codel ip_tables xfs libcrc32c sr_mod sd_mod cdrom t10_pi i915 i2c_algo_bit drm_kms_helper crc32c_intel syscopyarea sysfillrect sysimgblt serio_raw fb_sys_fops drm e1000e ahci libahci video libata ptp pps_core dm_mirror dm_region_hash dm_log dm_mod
[Â 135.689733] CR2: 0000000000000004

We can reproduce this problem in the following waysï

1. We create a new cgroup subdirectory and a new eventfd, and then we monitor multiple memory thresholds of the cgroup through this eventfd
2. closing this eventfd, the system will call __mem_cgroup_usage_unregister_event () multiple times to delete all events related to this eventfd.

The first time __mem_cgroup_usage_unregister_event () is called, the system will clear all items related to this eventfd in thresholds-> primary.
Since there is currently only one eventfd, thresholds-> primary becomes empty, so the system will set thresholds-> primary and hresholds-> spare
to NULL. If at this time, the user creates a new eventfd and monitor the memory threshold of this cgroup, then the system will re-initialize
thresholds-> primary. Then when the system call __mem_cgroup_usage_unregister_event () is called for the second time, because thresholds-> primary
is not empty, the system will access thresholds-> spare, but thresholds-> spare is NULL, which will trigger a crash.

In general, the longer it takes to delete all events related to this eventfd, the easier it is to trigger this problem.

thanks

Michal Hocko wrote on 2020/3/5 17:24:
> Thank you for the report!
>
> On Thu 05-03-20 13:52:03, brookxu wrote:
>> One eventfd monitors multiple memory thresholds of cgroup, closing it, the
>> system will delete related events. Before all events are deleted, another
>> eventfd monitors the cgroup's memory threshold.
> Could you describe the race scenario please? Ideally 
>> As a result, thresholds->primary[] is not empty, but thresholds->sparse[]
>> is NULL, __mem_cgroup_usage_unregister_event() leading to a crash:
>>
>> [Â 138.925809] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
>> [Â 138.926817] IP: [<ffffffff8116c9b7>] mem_cgroup_usage_unregister_event+0xd7/0x1f0
>> [Â 138.927701] PGD 73bce067 PUD 76ff3067 PMD 0
>> [Â 138.928384] Oops: 0002 [#1] SMP
>> [Â 138.935218] CPU: 1 PID: 14 Comm: kworker/1:0 Not tainted 3.10.107-1-tlinux2-0047 #1
> Also you seem to be running a very old kernel. Does the problem exist in
> the current Vanilla kernel?