RE: [RFC PATCH net-next v4 2/2] macb: Enable 1588 support in SAMA5Dx platforms.

From: Rafal Ozieblo
Date: Tue Dec 20 2016 - 11:39:02 EST

Next message: Roman Kagan: "[PATCH 12/15] hyperv: move VMBus connection ids to uapi"
Previous message: Eric Dumazet: "Re: [PATCH] net_sched: sch_fq: use rb_entry()"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

From: Andrei Pistirica [mailto:andrei.pistirica@xxxxxxxxxxxxx]
Sent: 14 grudnia 2016 13:56

> This patch does the following:
> - Enable HW time stamp for the following platforms: SAMA5D2, SAMA5D3 and
> SAMA5D4.
> - HW time stamp capabilities are advertised via ethtool and macb ioctl is
> updated accordingly.
> - HW time stamp on the PTP Ethernet packets are received using the
> SO_TIMESTAMPING API. Where timers are obtained from the PTP event/peer
> registers.
>
> Note: Patch on net-next, on December 7th.
>
> Signed-off-by: Andrei Pistirica <andrei.pistirica@xxxxxxxxxxxxx>
> ---
> Patch history:
>
> Version 1:
> Integration with SAMA5D2 only. This feature wasn't tested on any other platform that might use cadence/gem.
>
> Patch is not completely ported to the very latest version of net-next, and it will be after review.
>
> Version 2 modifications:
> - add PTP caps for SAMA5D2/3/4 platforms
> - and cosmetic changes
>
> Version 3 modifications:
> - add support for sama5D2/3/4 platforms using GEM-PTP interface.
>
> Version 4 modifications:
> - time stamp only PTP_V2 events
> - maximum adjustment value is set based on Richard's input
>
> Note: Patch on net-next, on December 14th.
>
> drivers/net/ethernet/cadence/macb.c | 168 ++++++++++++++++++++++++++++++++++--
> 1 file changed, 163 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/net/ethernet/cadence/macb.c b/drivers/net/ethernet/cadence/macb.c
> index 538544a..8d5c976 100644
> --- a/drivers/net/ethernet/cadence/macb.c
> +++ b/drivers/net/ethernet/cadence/macb.c
> @@ -714,6 +714,8 @@ static void macb_tx_interrupt(struct macb_queue *queue)
>
> /* First, update TX stats if needed */
> if (skb) {
> + gem_ptp_do_txstamp(bp, skb);
> +
I think, you can not do it in that way.
It will hold two locks. If you enable appropriate option in kernel (as far as I
remember CONFIG_DEBUG_SPINLOCK) you will get a warning here.

Please look at following call-stack:

1. macb_interrupt() // spin_lock(&bp->lock) is taken
2. macb_tx_interrupt()
3. macb_handle_txtstamp()
4. skb_tstamp_tx()
5. __skb_tstamp_tx()
6. skb_may_tx_timestamp()
7. read_lock_bh() // second lock is taken

I know that those are different locks and different types. But this could lead
to deadlocks. This is the reason of warning I could see.
And this is the reason why I get timestamp in interrupt routine but pass it to
skb outside interrupt (using circular buffer).

Please, refer to this:
https://lkml.org/lkml/2016/11/18/168

1. macb_tx_interrupt()
2. macb_tx_timestamp_add() and schedule_work(&queue->tx_timestamp_task)

Then, outside interrupt (without holding a lock) :
1. macb_tx_timestamp_flush()
2. macb_tstamp_tx()
3. skb_tstamp_tx()

> netdev_vdbg(bp->dev, "skb %u (data %p) TX complete\n",
> macb_tx_ring_wrap(bp, tail),
> skb->data);
> @@ -878,6 +880,8 @@ static int gem_rx(struct macb *bp, int budget)
> GEM_BFEXT(RX_CSUM, ctrl) & GEM_RX_CSUM_CHECKED_MASK)
> skb->ip_summed = CHECKSUM_UNNECESSARY;
>
> + gem_ptp_do_rxstamp(bp, skb);
> +
> bp->stats.rx_packets++;
> bp->stats.rx_bytes += skb->len;
>
> @@ -2080,6 +2084,9 @@ static int macb_open(struct net_device *dev)
>
> netif_tx_start_all_queues(dev);
>
> + if (bp->ptp_info)
> + bp->ptp_info->ptp_init(dev);
> +
> return 0;
> }
>
> @@ -2101,6 +2108,9 @@ static int macb_close(struct net_device *dev)
>
> macb_free_consistent(bp);
>
> + if (bp->ptp_info)
> + bp->ptp_info->ptp_remove(dev);
> +
> return 0;
> }
>
> @@ -2374,6 +2384,133 @@ static int macb_set_ringparam(struct net_device *netdev,
> return 0;
> }
>
> +#ifdef CONFIG_MACB_USE_HWSTAMP
> +static unsigned int gem_get_tsu_rate(struct macb *bp) {
> + /* Note: TSU rate is hardwired to PCLK. */
> + return clk_get_rate(bp->pclk);
> +}
Not exactly. There could be separate TSU clock.
In my solution I check tsu_clk in DT before I decide to take pclk.
But it could be change in macb_ptp_info.

> +
> +static s32 gem_get_ptp_max_adj(void)
> +{
> + return 3921508;
> +}
> +
> +static int gem_get_ts_info(struct net_device *dev,
> + struct ethtool_ts_info *info)
> +{
> + struct macb *bp = netdev_priv(dev);
> +
> + ethtool_op_get_ts_info(dev, info);
> + info->so_timestamping =
> + SOF_TIMESTAMPING_TX_SOFTWARE |
> + SOF_TIMESTAMPING_RX_SOFTWARE |
> + SOF_TIMESTAMPING_SOFTWARE |
> + SOF_TIMESTAMPING_TX_HARDWARE |
> + SOF_TIMESTAMPING_RX_HARDWARE |
> + SOF_TIMESTAMPING_RAW_HARDWARE;
> + info->phc_index = -1;
> +
> + if (bp->ptp_clock)
> + info->phc_index = ptp_clock_index(bp->ptp_clock);
> +
> + return 0;
> +}
> +
> +static int gem_set_hwtst(struct net_device *netdev,
> + struct ifreq *ifr, int cmd)
> +{
> + struct hwtstamp_config config;
> + struct macb *priv = netdev_priv(netdev);
> + u32 regval;
> +
> + netdev_vdbg(netdev, "macb_hwtstamp_ioctl\n");
> +
> + if (copy_from_user(&config, ifr->ifr_data, sizeof(config)))
> + return -EFAULT;
> +
> + /* reserved for future extensions */
> + if (config.flags)
> + return -EINVAL;
> +
> + switch (config.tx_type) {
> + case HWTSTAMP_TX_OFF:
> + priv->hwts_tx_en = false;
> + break;
> + case HWTSTAMP_TX_ON:
> + priv->hwts_tx_en = true;
> + break;
> + default:
> + return -ERANGE;
> + }
> +
> + switch (config.rx_filter) {
> + case HWTSTAMP_FILTER_NONE:
> + if (priv->hwts_rx_en)
> + priv->hwts_rx_en = false;
> + break;
> + case HWTSTAMP_FILTER_PTP_V2_L4_EVENT:
> + case HWTSTAMP_FILTER_PTP_V2_L2_EVENT:
> + case HWTSTAMP_FILTER_PTP_V2_L2_SYNC:
> + case HWTSTAMP_FILTER_PTP_V2_L4_SYNC:
> + case HWTSTAMP_FILTER_PTP_V2_L2_DELAY_REQ:
> + case HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ:
> + case HWTSTAMP_FILTER_PTP_V2_EVENT:
> + case HWTSTAMP_FILTER_PTP_V2_SYNC:
> + case HWTSTAMP_FILTER_PTP_V2_DELAY_REQ:
> + config.rx_filter = HWTSTAMP_FILTER_PTP_V2_EVENT;
> + regval = macb_readl(priv, NCR);
> + macb_writel(priv, NCR, (regval | MACB_BIT(SRTSM)));
> +
> + if (!priv->hwts_rx_en)
> + priv->hwts_rx_en = true;
> + break;
> + default:
> + config.rx_filter = HWTSTAMP_FILTER_NONE;
> + return -ERANGE;
> + }
> +
> + return copy_to_user(ifr->ifr_data, &config, sizeof(config)) ?
> + -EFAULT : 0;
> +}
> +
> +static int gem_get_hwtst(struct net_device *netdev,
> + struct ifreq *ifr)
> +{
> + struct hwtstamp_config config;
> + struct macb *priv = netdev_priv(netdev);
> +
> + config.flags = 0;
> + config.tx_type = priv->hwts_tx_en ? HWTSTAMP_TX_ON : HWTSTAMP_TX_OFF;
> + config.rx_filter = (priv->hwts_rx_en ?
> + HWTSTAMP_FILTER_ALL : HWTSTAMP_FILTER_NONE);
> +
> + return copy_to_user(ifr->ifr_data, &config, sizeof(config)) ?
> + -EFAULT : 0;
> +}
> +
> +static struct macb_ptp_info gem_ptp_info = {
> + .ptp_init = gem_ptp_init,
> + .ptp_remove = gem_ptp_remove,
> + .get_ptp_max_adj = gem_get_ptp_max_adj,
> + .get_tsu_rate = gem_get_tsu_rate,
> + .get_ts_info = gem_get_ts_info,
> + .get_hwtst = gem_get_hwtst,
> + .set_hwtst = gem_set_hwtst,
> +};
> +#endif
> +
> +static int macb_get_ts_info(struct net_device *netdev,
> + struct ethtool_ts_info *info)
> +{
> + struct macb *bp = netdev_priv(netdev);
> +
> + if (bp->ptp_info)
> + return bp->ptp_info->get_ts_info(netdev, info);
> +
> + return ethtool_op_get_ts_info(netdev, info); }
> +
> static const struct ethtool_ops macb_ethtool_ops = {
> .get_regs_len = macb_get_regs_len,
> .get_regs = macb_get_regs,
> @@ -2391,7 +2528,7 @@ static const struct ethtool_ops gem_ethtool_ops = {
> .get_regs_len = macb_get_regs_len,
> .get_regs = macb_get_regs,
> .get_link = ethtool_op_get_link,
> - .get_ts_info = ethtool_op_get_ts_info,
> + .get_ts_info = macb_get_ts_info,
> .get_ethtool_stats = gem_get_ethtool_stats,
> .get_strings = gem_get_ethtool_strings,
> .get_sset_count = gem_get_sset_count,
> @@ -2404,6 +2541,7 @@ static const struct ethtool_ops gem_ethtool_ops = { static int macb_ioctl(struct net_device *dev, struct ifreq *rq, int cmd) {
> struct phy_device *phydev = dev->phydev;
> + struct macb *bp = netdev_priv(dev);
>
> if (!netif_running(dev))
> return -EINVAL;
> @@ -2411,7 +2549,20 @@ static int macb_ioctl(struct net_device *dev, struct ifreq *rq, int cmd)
> if (!phydev)
> return -ENODEV;
>
> - return phy_mii_ioctl(phydev, rq, cmd);
> + switch (cmd) {
> + case SIOCSHWTSTAMP:
> + if (bp->ptp_info)
> + return bp->ptp_info->set_hwtst(dev, rq, cmd);
> +
> + return -EOPNOTSUPP;
> + case SIOCGHWTSTAMP:
> + if (bp->ptp_info)
> + return bp->ptp_info->get_hwtst(dev, rq);
> +
> + return -EOPNOTSUPP;
> + default:
> + return phy_mii_ioctl(phydev, rq, cmd);
> + }
> }
>
> static int macb_set_features(struct net_device *netdev, @@ -2485,6 +2636,12 @@ static void macb_configure_caps(struct macb *bp,
> dcfg = gem_readl(bp, DCFG2);
> if ((dcfg & (GEM_BIT(RX_PKT_BUFF) | GEM_BIT(TX_PKT_BUFF))) == 0)
> bp->caps |= MACB_CAPS_FIFO_MODE;
> +
> + /* iff HWSTAMP is configure and gem has the capability */ #ifdef
> +CONFIG_MACB_USE_HWSTAMP
> + if (gem_has_ptp(bp))
> + bp->ptp_info = &gem_ptp_info;
> +#endif
> }
>
> dev_dbg(&bp->pdev->dev, "Cadence caps 0x%08x\n", bp->caps); @@ -3041,7 +3198,7 @@ static const struct macb_config pc302gem_config = { };
>
> static const struct macb_config sama5d2_config = {
> - .caps = MACB_CAPS_USRIO_DEFAULT_IS_MII_GMII,
> + .caps = MACB_CAPS_USRIO_DEFAULT_IS_MII_GMII | MACB_CAPS_GEM_HAS_PTP,
There are many IP cores with many configuration. If it is possible, capabilities should be read from IP directly.
And it is possible in that case:
Design Configuration Register 5 (0x290)
bit 8: tsu
There is now PTP hardware support without that bit.

> .dma_burst_length = 16,
> .clk_init = macb_clk_init,
> .init = macb_init,
> @@ -3049,14 +3206,15 @@ static const struct macb_config sama5d2_config = {
>
> static const struct macb_config sama5d3_config = {
> .caps = MACB_CAPS_SG_DISABLED | MACB_CAPS_GIGABIT_MODE_AVAILABLE
> - | MACB_CAPS_USRIO_DEFAULT_IS_MII_GMII,
> + | MACB_CAPS_USRIO_DEFAULT_IS_MII_GMII
> + | MACB_CAPS_GEM_HAS_PTP,
> .dma_burst_length = 16,
> .clk_init = macb_clk_init,
> .init = macb_init,
> };
>
> static const struct macb_config sama5d4_config = {
> - .caps = MACB_CAPS_USRIO_DEFAULT_IS_MII_GMII,
> + .caps = MACB_CAPS_USRIO_DEFAULT_IS_MII_GMII | MACB_CAPS_GEM_HAS_PTP,
> .dma_burst_length = 4,
> .clk_init = macb_clk_init,
> .init = macb_init,
> --
> 2.7.4

In macb_start_xmit() there is also invoked skb_tx_timestamp() for software timestamping.
I think, it should be disabled if you do hardware timestamping.

Best regards,
Rafal Ozieblo | Firmware System Engineer,
www.cadence.com

Next message: Roman Kagan: "[PATCH 12/15] hyperv: move VMBus connection ids to uapi"
Previous message: Eric Dumazet: "Re: [PATCH] net_sched: sch_fq: use rb_entry()"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]