Re: [RESEND PATCH v7 1/8] kernfs: Introduce interface to access global kernfs_open_file_mutex.

[Al Viro]

On Tue, Apr 05, 2022 at 03:36:03PM +1000, Imran Khan wrote:
> Hello Al,
> 
> On 18/3/22 8:34 am, Al Viro wrote:
> > On Thu, Mar 17, 2022 at 06:26:05PM +1100, Imran Khan wrote:
> > 
> >> @@ -570,9 +571,10 @@ static void kernfs_put_open_node(struct kernfs_node *kn,
> >>  				 struct kernfs_open_file *of)
> [...]
> 
> > As the matter of fact, we can do even better - make freeing
> > that thing rcu-delayed, use rcu_assign_pointer() for stores,
> > rcu_dereference() for loads and have kernfs_notify() do
> > 	rcu_read_lock();
> > 	on = rcu_dereference(kn->attr.open);
> > 	if (on) {
> > 		atomic_inc(&on->event);
> > 		wake_up_interruptible(&on->poll);
> > 	}
> > 	rcu_read_unlock();
> > and kernfs_open_node_lock becomes useless - all places that
> > grab it are under kernfs_open_file_mutex.
> 
> There are some issues in freeing ->attr.open under RCU callback.

Such as?

> There
> are some users of ->attr.open that can block and hence can't operate
> under rcu_read_lock. For example in kernfs_drain_open_files we are
> traversing list of open files corresponding to ->attr.open and unmapping
> those as well. The unmapping operation can block in i_mmap_lock_write.

Yes.

> So even after removing refcnt we will still need kernfs_open_node_lock.

What for?  Again, have kernfs_drain_open_files() do this:
{
        struct kernfs_open_node *on;
	struct kernfs_open_file *of;

	if (!(kn->flags & (KERNFS_HAS_MMAP | KERNFS_HAS_RELEASE)))
		return;
	if (rcu_dereference(kn->attr.open) == NULL)
		return;
	mutex_lock(&kernfs_open_file_mutex);
	// now ->attr.open is stable (all stores are under kernfs_open_file_mutex)
	on = rcu_dereference(kn->attr.open);
	if (!on) {
		mutex_unlock(&kernfs_open_file_mutex);
		return;
	}
	// on->files contents is stable
	list_for_each_entry(of, &on->files, list) {
		struct inode *inode = file_inode(of->file);

		if (kn->flags & KERNFS_HAS_MMAP)
			unmap_mapping_range(inode->i_mapping, 0, 0, 1);

		if (kn->flags & KERNFS_HAS_RELEASE)
			kernfs_release_file(kn, of);
	}
	mutex_unlock(&kernfs_open_file_mutex);
}

What's the problem?  The caller has already guaranteed that no additions will
happen.  Once we'd grabbed kernfs_open_file_mutex, we know that
	* kn->attr.open value won't change until we drop the mutex
	* nothing gets removed from kn->attr.open->files until we drop the mutex
so we can bloody well walk that list, blocking as much as we want.

We don't need rcu_read_lock() there - we are already holding the mutex used
by writers for exclusion among themselves.  RCU *allows* lockless readers,
it doesn't require all readers to be such.  kernfs_notify() can be made
lockless, this one can't and that's fine.

BTW, speaking of kernfs_notify() - can calls of that come from NMI handlers?
If not, I'd consider using llist for kernfs_notify_list...