Re: [RFC PATCH v3 13/16] cxl/mem: Create concept of enabled commands

[Ben Widawsky]

On 21-01-14 17:25:31, Jonathan Cameron wrote:
> On Mon, 11 Jan 2021 14:51:18 -0800
> Ben Widawsky <ben.widawsky@xxxxxxxxx> wrote:
> 
> > CXL devices must implement the Device Command Interface (described in
> > 8.2.9 of the CXL 2.0 spec). While the driver already maintains a list of
> > commands it supports, there is still a need to be able to distinguish
> > between commands that the driver knows about from commands that may not
> > be supported by the hardware. No such commands currently are defined in
> > the driver.
> > 
> > The implementation leaves the statically defined table of commands and
> > supplements it with a bitmap to determine commands that are enabled.
> > 
> > ---
> > 
> > There are multiple approaches that can be taken, but this is nice for a
> > few reasons.
> > 
> > Here are some of the other solutions:
> > 
> > Create a per instance table with only the supported commands.
> > 1. Having a fixed command id -> command mapping is much easier to manage
> >    for development and debugging.
> > 2. Dealing with dynamic memory allocation for the table adds unnecessary
> >    complexity.
> > 3. Most tables for device types are likely to be quite similar.
> > 4. Makes it difficult to implement helper macros like cxl_for_each_cmd()
> > 
> > If the per instance table did preserve ids, #1 above can be addressed.
> > However, as "enable" is currently the only mutable state for the
> > commands, it would yield a lot of overhead for not much gain.
> > Additionally, the other issues remain.
> > 
> > If "enable" remains the only mutable state, I believe this to be the
> > best solution. Once the number of mutable elements in a command grows,
> > it probably makes sense to move to per device instance state with a
> > fixed command ID mapping.
> Agreed with the logic.   
> 
> However, patch wise, should either drop the --- above or move this below the
> --- after your sign off.  Otherwise you run the risk of git dropping your
> sign off and resulting complaints from anyone run validation scripts
> of the kernel tree that check for that.
> 
> > 
> > Signed-off-by: Ben Widawsky <ben.widawsky@xxxxxxxxx>
> > ---
> >  drivers/cxl/cxl.h |  4 ++++
> >  drivers/cxl/mem.c | 38 +++++++++++++++++++++++++++++++++++++-
> >  2 files changed, 41 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> > index 537ac4d8e6bd..963ba30cb200 100644
> > --- a/drivers/cxl/cxl.h
> > +++ b/drivers/cxl/cxl.h
> > @@ -17,6 +17,9 @@
> >  
> >  #define CXL_GET_FIELD(word, field) FIELD_GET(field##_MASK, word)
> >  
> > +/* XXX: Arbitrary max */
> > +#define CXL_MAX_COMMANDS 32
> 
> If going this way, probably want a build time check that you don't
> go beyond this value for a given command set.  I haven't actually
> thought about how to construct that but should be easy enough.
> 

I think BUILD_BUG_ON of ARRAY_SIZE mem_commands is sufficient. Let me know if
you want anything more.

> > +
> >  /* Device  (CXL 2.0 - 8.2.8.3) */
> >  #define CXLDEV_CAP_ARRAY_REG 0x0
> >  #define CXLDEV_CAP_ARRAY_CAP_ID 0
> > @@ -88,6 +91,7 @@ struct cxl_mem {
> >  	} ram;
> >  
> >  	char firmware_version[0x10];
> > +	DECLARE_BITMAP(enabled_cmds, CXL_MAX_COMMANDS);
> >  
> >  	/* Cap 0001h - CXL_CAP_CAP_ID_DEVICE_STATUS */
> >  	struct {
> > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> > index a824cfd4342a..20b26fa2c466 100644
> > --- a/drivers/cxl/mem.c
> > +++ b/drivers/cxl/mem.c
> > @@ -114,6 +114,8 @@ static struct {
> >   *    would typically be used for deprecated commands.
> >   *  * %CXL_CMD_FLAG_MANDATORY: Hardware must support this command. This flag is
> >   *    only used internally by the driver for sanity checking.
> > + *  * %CXL_CMD_INTERNAL_FLAG_PSEUDO: This is a pseudo command which doesn't have
> > + *    a direct mapping to hardware. They are implicitly always enabled.
> >   *
> >   * The cxl_mem_command is the driver's internal representation of commands that
> >   * are supported by the driver. Some of these commands may not be supported by
> > @@ -129,6 +131,7 @@ struct cxl_mem_command {
> >  #define CXL_CMD_INTERNAL_FLAG_NONE 0
> >  #define CXL_CMD_INTERNAL_FLAG_HIDDEN BIT(0)
> >  #define CXL_CMD_INTERNAL_FLAG_MANDATORY BIT(1)
> > +#define CXL_CMD_INTERNAL_FLAG_PSEUDO BIT(2)
> >  };
> >  
> >  /*
> > @@ -140,7 +143,7 @@ struct cxl_mem_command {
> >  static struct cxl_mem_command mem_commands[] = {
> >  	CXL_CMD(INVALID, KERNEL, 0, 0, HIDDEN),
> >  	CXL_CMD(IDENTIFY, NONE, 0, 0x43, MANDATORY),
> > -	CXL_CMD(RAW, NONE, ~0, ~0, MANDATORY),
> > +	CXL_CMD(RAW, NONE, ~0, ~0, PSEUDO),
> >  };
> >  
> >  #define cxl_for_each_cmd(cmd)                                                  \
> > @@ -618,6 +621,10 @@ static int cxl_validate_cmd_from_user(struct cxl_mem *cxlm,
> >  	c = &mem_commands[send_cmd->id];
> >  	info = &c->info;
> >  
> > +	/* Check that the command is enabled for hardware */
> > +	if (!test_bit(cxl_cmd_index(c), cxlm->enabled_cmds))
> > +		return -ENOTTY;
> > +
> >  	if (info->flags & CXL_MEM_COMMAND_FLAG_KERNEL)
> >  		return -EPERM;
> >  
> > @@ -1029,6 +1036,31 @@ static int cxl_mem_add_memdev(struct cxl_mem *cxlm)
> >  	return rc;
> >  }
> >  
> > +/**
> > + * cxl_mem_enumerate_cmds() - Enumerate commands for a device.
> > + * @cxlm: The device.
> > + *
> > + * Returns 0 if enumerate completed successfully.
> > + *
> > + * CXL devices have optional support for certain commands. This function will
> > + * determine the set of supported commands for the hardware and update the
> > + * enabled_cmds bitmap in the @cxlm.
> > + */
> > +static int cxl_mem_enumerate_cmds(struct cxl_mem *cxlm)
> > +{
> > +	struct cxl_mem_command *c;
> > +
> > +	/* All commands are considered enabled for now (except INVALID). */
> > +	cxl_for_each_cmd(c) {
> > +		if (c->flags & CXL_CMD_INTERNAL_FLAG_HIDDEN)
> > +			continue;
> > +
> > +		set_bit(cxl_cmd_index(c), cxlm->enabled_cmds);
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> >  /**
> >   * cxl_mem_identify() - Send the IDENTIFY command to the device.
> >   * @cxlm: The device to identify.
> > @@ -1147,6 +1179,10 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
> >  	if (rc)
> >  		goto err;
> >  
> > +	rc = cxl_mem_enumerate_cmds(cxlm);
> > +	if (rc)
> > +		return rc;
> > +
> >  	rc = cxl_mem_identify(cxlm);
> >  	if (rc)
> >  		goto err;
>