• Alex Shi's avatar
    PM / CPU: replace raw_notifier with atomic_notifier · 313c8c16
    Alex Shi authored
    This patch replaces an rwlock and raw notifier by an atomic notifier
    protected by a spin_lock and RCU.
    
    The main reason for this change is due to a 'scheduling while atomic'
    bug with RT kernels on ARM/ARM64. On ARM/ARM64, the rwlock
    cpu_pm_notifier_lock in cpu_pm_enter/exit() causes a potential
    schedule after IRQ disable in the idle call chain:
    
    cpu_startup_entry
      cpu_idle_loop
        local_irq_disable()
        cpuidle_idle_call
          call_cpuidle
            cpuidle_enter
              cpuidle_enter_state
                ->enter :arm_enter_idle_state
                  cpu_pm_enter/exit
                    CPU_PM_CPU_IDLE_ENTER
                      read_lock(&cpu_pm_notifier_lock); <-- sleep in idle
                         __rt_spin_lock();
                            schedule();
    
    The kernel panic is here:
    [    4.609601] BUG: scheduling while atomic: swapper/1/0/0x00000002
    [    4.609608] [<ffff0000086fae70>] arm_enter_idle_state+0x18/0x70
    [    4.609614] Modules linked in:
    [    4.609615] [<ffff0000086f9298>] cpuidle_enter_state+0xf0/0x218
    [    4.609620] [<ffff0000086f93f8>] cpuidle_enter+0x18/0x20
    [    4.609626] Preemption disabled at:
    [    4.609627] [<ffff0000080fa234>] call_cpuidle+0x24/0x40
    [    4.609635] [<ffff000008882fa4>] schedule_preempt_disabled+0x1c/0x28
    [    4.609639] [<ffff0000080fa49c>] cpu_startup_entry+0x154/0x1f8
    [    4.609645] [<ffff00000808e004>] secondary_start_kernel+0x15c/0x1a0
    
    Daniel Lezcano said this notification is needed on arm/arm64 platforms.
    Sebastian suggested using atomic_notifier instead of rwlock, which is not
    only removing the sleeping in idle, but also improving latency.
    
    Tony Lindgren found a miss use that rcu_read_lock used after rcu_idle_enter
    Paul McKenney suggested trying RCU_NONIDLE.
    Signed-off-by: default avatarAlex Shi <alex.shi@linaro.org>
    Tested-by: default avatarTony Lindgren <tony@atomide.com>
    Acked-by: default avatarSebastian Andrzej Siewior <bigeasy@linutronix.de>
    [ rjw: Subject & changelog ]
    Signed-off-by: default avatarRafael J. Wysocki <rafael.j.wysocki@intel.com>
    313c8c16