• Paul E. McKenney's avatar
    rcu: Make expedited GPs correctly handle hardware CPU insertion · 313517fc
    Paul E. McKenney authored
    The update of the ->expmaskinitnext and of ->ncpus are unsynchronized,
    with the value of ->ncpus being incremented long before the corresponding
    ->expmaskinitnext mask is updated.  If an RCU expedited grace period
    sees ->ncpus change, it will update the ->expmaskinit masks from the new
    ->expmaskinitnext masks.  But it is possible that ->ncpus has already
    been updated, but the ->expmaskinitnext masks still have their old values.
    For the current expedited grace period, no harm done.  The CPU could not
    have been online before the grace period started, so there is no need to
    wait for its non-existent pre-existing readers.
    But the next RCU expedited grace period is in a world of hurt.  The value
    of ->ncpus has already been updated, so this grace period will assume
    that the ->expmaskinitnext masks have not changed.  But they have, and
    they won't be taken into account until the next never-been-online CPU
    comes online.  This means that RCU will be ignoring some CPUs that it
    should be paying attention to.
    The solution is to update ->ncpus and ->expmaskinitnext while holding
    the ->lock for the rcu_node structure containing the ->expmaskinitnext
    mask.  Because smp_store_release() is now used to update ->ncpus and
    smp_load_acquire() is now used to locklessly read it, if the expedited
    grace period sees ->ncpus change, then the updating CPU has to
    already be holding the corresponding ->lock.  Therefore, when the
    expedited grace period later acquires that ->lock, it is guaranteed
    to see the new value of ->expmaskinitnext.
    On the other hand, if the expedited grace period loads ->ncpus just
    before an update, earlier full memory barriers guarantee that
    the incoming CPU isn't far enough along to be running any RCU readers.
    This commit therefore makes the required change.
    Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
tree_exp.h 22.4 KB