1. 03 Apr, 2019 40 commits
    • Gilles Chanteperdrix's avatar
      gpio: mxc: ipipe: enable interrupt pipelining · 20914e3c
      Gilles Chanteperdrix authored
      Fix up the MXC GPIO driver (i.MX series) in order to channel
      interrupts through the interrupt pipeline.
    • Gilles Chanteperdrix's avatar
      gpio: mvebu: ipipe: enable interrupt pipelining · 623ab377
      Gilles Chanteperdrix authored
      Fix up the Marvell EBU GPIO controller driver in order to channel
      interrupts through the interrupt pipeline.
    • Gilles Chanteperdrix's avatar
      gpio: davinci: ipipe: enable interrupt pipelining · ee0c4603
      Gilles Chanteperdrix authored
      Fix up the Davinci GPIO driver in order to channel interrupts through
      the interrupt pipeline.
    • Gilles Chanteperdrix's avatar
      gpio: omap: ipipe: enable interrupt pipelining · 835c065f
      Gilles Chanteperdrix authored
      Fix up the omap GPIO driver in order to channel interrupts through the
      interrupt pipeline.
    • Dongjiu Geng's avatar
      clocksource: arm_arch_timer: ipipe: correct the wrong ipipe_timer setup · 420b167b
      Dongjiu Geng authored
      The arch_timer_mem_use_virtual variable is used to judge the type
      of memory-mapped timer, it can not be used to judge the type of CP15
      timer, so fix this issue, otherwise it will lead to kernel panic when
      running as a VM.
      Signed-off-by: default avatarDongjiu Geng <gengdongjiu@huawei.com>
    • Gilles Chanteperdrix's avatar
      clocksource: arch_timer: ipipe: enable interrupt pipelining · 715399c3
      Gilles Chanteperdrix authored
      Fix up the ARM architected timer driver in order to channel interrupts
      through the interrupt pipeline.
      FIXME: split.
    • Greg Gallagher's avatar
      clocksource: ipipe: add support for BCM2835 chips · 1a962c9f
      Greg Gallagher authored
      This change will add support for bcm2835 chips.  This includes the raspberry pi
      0 and 1 family of boards.
      FIXME: split.
    • Gilles Chanteperdrix's avatar
      clocksource: dw_apb: ipipe: add support for user-visible TSC · b8172a6d
      Gilles Chanteperdrix authored
      Expose the clocksource to userland processes via the ARM-specific
      user-TSC interface if enabled.
    • Gilles Chanteperdrix's avatar
      clocksource: arm/gt: ipipe: add support for user-visible TSC · 3004a824
      Gilles Chanteperdrix authored
      Expose the clocksource to userland processes via the ARM-specific
      user-TSC interface.
    • Gilles Chanteperdrix's avatar
      clocksource: imx/gpt: ipipe: add support for user-visible TSC · 251efb26
      Gilles Chanteperdrix authored
      Expose the clocksource to userland processes via the ARM-specific
      user-TSC interface.
    • Gilles Chanteperdrix's avatar
      clocksource: sp804: ipipe: add support for user-visible TSC · ad9366e0
      Gilles Chanteperdrix authored
      Expose the clocksource to userland processes via the ARM-specific
      user-TSC interface.
    • Philippe Gerum's avatar
      ipipe: add cpuidle control interface · b9f77b00
      Philippe Gerum authored
      Add a kernel interface for sharing CPU idling control between the host
      kernel and a co-kernel. The former invokes ipipe_cpuidle_control()
      which the latter should implement, for determining whether entering a
      sleep state is ok. This hook should return boolean true if so.
      The co-kernel may veto such entry if need be, in order to prevent
      latency spikes, as exiting sleep states might be costly depending on
      the CPU idling operation being used.
    • Gilles Chanteperdrix's avatar
      pinctrl: ipipe: enable interrupt pipelining · 54bedcd0
      Gilles Chanteperdrix authored
      Do the fixups in the generic pin controller driver required to channel
      the incoming interrupts through the interrupt pipeline.
    • Philippe Gerum's avatar
      gpio: ipipe: convert to hard locking · e14adb41
      Philippe Gerum authored
      The generic GPIO chip needs to be hard locked as its handlers may be
      called from out-of-band code running in the head domain. To this end,
      convert the regular spinlock protecting the gpio_chip descriptor to a
      hard lock.
    • Philippe Gerum's avatar
      ftrace: ipipe: enable tracing from the head domain · 0e05bbe2
      Philippe Gerum authored
      Enabling ftrace for a co-kernel running in the head domain of a
      pipelined interrupt context means to:
      - make sure that ftrace's live kernel code patching still runs
        unpreempted by any head domain activity (so that the latter can't
        tread on invalid or half-baked changes in the .text section).
      - allow the co-kernel code running in the head domain to traverse
        ftrace's tracepoints safely.
      The changes introduced by this commit ensure this by fixing up some
      key critical sections so that interrupts are still disabled in the
      CPU, undoing the interrupt flag virtualization in those particular
    • Philippe Gerum's avatar
      mm: ipipe: disable ondemand memory · 466697cd
      Philippe Gerum authored
      Co-kernels cannot bear with the extra latency caused by memory access
      faults involved in COW or
      overcommit. __ipipe_disable_ondemand_mappings() force commits all
      common memory mappings with physical RAM.
      In addition, the architecture code is given a chance to pre-load page
      table entries for ioremap and vmalloc memory, for preventing further
      minor faults accessing such memory due to PTE misses (if that ever
      makes sense for them).
      Revisit: Further COW breaking in copy_user_page() and copy_pte_range()
      may be useless once __ipipe_disable_ondemand_mappings() has run for a
      co-kernel task, since all of its mappings have been populated, and
      unCOWed if applicable.
    • Philippe Gerum's avatar
      fork: ipipe: announce mm dismantling · 0e1b063d
      Philippe Gerum authored
      IPIPE_KEVT_CLEANUP is emitted before a process memory context is
      entirely dropped, after all the mappings have been exited. Per-process
      resources which might be maintained by the co-kernel could be released
      there, as all tasks have exited.
    • Philippe Gerum's avatar
      sched: ipipe: announce CPU affinity change · bbba26eb
      Philippe Gerum authored
      Emit IPIPE_KEVT_SETAFFINITY to the co-kernel when the target task is
      about to move to another CPU.
      CPU migration can only take place from the root domain, the pipeline
      does not provide any support for migrating tasks from the head domain,
      and derives several key assumptions based on this invariant.
    • Philippe Gerum's avatar
      sched: ipipe: announce signal receipt · cfe28fb1
      Philippe Gerum authored
      Emit IPIPE_KEVT_SIGWAKE when the target task is about to receive a
      (regular) signal. The co-kernel may decide to schedule a transition of
      the recipient to the root domain in order to have it handle that
      signal asap, which is commonly required for keeping the kernel sane.
      This notification is always sent from the context of the issuer.
    • Philippe Gerum's avatar
      sched: ipipe: announce task exit · d94ddc97
      Philippe Gerum authored
      Emit IPIPE_KEVT_EXIT from do_exit() to the co-kernel before the
      current task has dropped the files and mappings it owns.
    • Philippe Gerum's avatar
      KVM: ipipe: keep hypervisor state consistent across domain preemption · 88e379f8
      Philippe Gerum authored
      In order for the hypervisor to operate properly in presence of a
      co-kernel, we need:
      - the virtualization core to know when the hypervisor stalls due
        to a preemption by the co-kernel.
      - to know when the VM enters and leaves guest mode.
    • Philippe Gerum's avatar
      sched: ipipe: add domain debug checks to common scheduling paths · 0088b024
      Philippe Gerum authored
      Catch invalid calls of root-only code from the head domain from common
      paths which may lead to blocking the current task linux-wise. Checks
      are enabled by CONFIG_IPIPE_DEBUG_CONTEXT.
    • Philippe Gerum's avatar
      sched: ipipe: enable task migration between domains · bf559171
      Philippe Gerum authored
      This is the basic code enabling alternate control of tasks between the
      regular kernel and an embedded co-kernel. The changes cover the
      following aspects:
      - extend the per-thread information block with a private area usable
        by the co-kernel for storing additional state information
      - provide the API enabling a scheduler exchange mechanism, so that
        tasks can run under the control of either kernel alternatively. This
        includes a service to move the current task to the head domain under
        the control of the co-kernel, and the converse service to re-enter
        the root domain once the co-kernel has released such task.
      - ensure the generic context switching code can be used from any
        domain, serializing execution as required.
      These changes have to be paired with arch-specific code further
      enabling context switching from the head domain.
    • Philippe Gerum's avatar
      clockevents: ipipe: connect clock chips to abstract tick device · f8f47d4c
      Philippe Gerum authored
      Announce all clock event chips as they are registered to the
      out-of-band tick device infrastructure, so that we can interpose on
      key handlers in their descriptors.
    • Philippe Gerum's avatar
      ipipe: add kernel event notifiers · 79db6170
      Philippe Gerum authored
      Add the core API for enabling (regular) kernel event notifications to
      a co-kernel running over the head domain. For instance, such a
      co-kernel may need to know when a task is about to be resumed upon
      signal receipt, or when it gets an access fault trap.
      This commit adds the client-side API for enabling such notification
      for class of events, but does not provide the notification points per
      se, which comes later.
    • Philippe Gerum's avatar
      printk: ipipe: add raw console channel · 56ed1979
      Philippe Gerum authored
      A raw output handler (.write_raw) is added to the console descriptor
      for writing (short) text output unmodified, without any logging,
      header or preparation whatsoever, usable from any pipeline domain.
      The dedicated raw_printk() variant formats the output message then
      passes it on to the handler holding a hard spinlock, irqs off.
      This is a very basic debug channel for situations when resorting to
      the fairly complex printk() handling is not an option. Unlike early
      consoles, regular consoles can provide a raw output service past the
      boot sequence. Raw output handlers are typically provided by serial
      console devices.
    • Philippe Gerum's avatar
      dump_stack: ipipe: make dump_stack() domain-aware · 6f013d57
      Philippe Gerum authored
      When dumping a stack backtrace, we neither need nor want to disable
      root stage IRQs over the head stage, where CPU migration can't
      Conversely, we neither need nor want to disable hard IRQs from the
      head stage, so that latency won't skyrocket either.
    • Philippe Gerum's avatar
      driver core: ipipe: defer dev_printk() from head domain · 5dbea799
      Philippe Gerum authored
      Just like printk(), dev_printk() cannot run from the head domain
      and/or with hard IRQs disabled. In such a case, log the output
      directly into the staging buffer we use for printk().
      NOTE: when redirected to the buffer, the output does not include the
      dev_printk() header text but is merely sent as-is to the log.
    • Philippe Gerum's avatar
      printk: ipipe: defer printk() from head domain · c469670d
      Philippe Gerum authored
      The printk() machinery cannot immediately invoke the console driver(s)
      when called from the head domain, since such driver code belongs to
      the root domain and cannot be shared between domains.
      Output issued from the head domain is formatted then logged into a
      staging buffer, and a dedicated virtual IRQ is posted to the root
      domain for notification. When the virtual IRQ handler runs, the
      contents of the staging buffer is flushed to the printk() interface
      anew, which may eventually pass the output on to the console drivers
      from such a context.
    • Philippe Gerum's avatar
      PM / hibernate: ipipe: protect against out-of-band interrupts · 5fb7e241
      Philippe Gerum authored
      We must not allow out-of-band activity to resume while we are busy
      suspending the devices in the system, until the PM sleep state has
      been fully entered.
      Pair existing virtual IRQ disabling calls which only apply to the root
      domain with hard ones.
    • Philippe Gerum's avatar
      module: ipipe: enable try_module_get() from hard atomic context · b7c424c4
      Philippe Gerum authored
      We might have out-of-band code calling try_module_get() from the head
      domain, or from a section covered by a hard spinlock where the root
      domain must not reschedule. This requires the preemption management
      calls in try_module_get() (and the converse module_put()) to be
      converted to their hard variant.
      REVISIT: using try_module_get() from such contexts is questionable,
      client domains should be fixed.
    • Philippe Gerum's avatar
      KGDB: ipipe: enable debugging over the head domain · 5f37bade
      Philippe Gerum authored
      Make the KGDB stub runnable over the head domain since we may take
      traps and interrupts from that context too, by converting the locks to
      hard spinlocks.
    • Philippe Gerum's avatar
      context_tracking: ipipe: do not track over the head domain · 5ac01b95
      Philippe Gerum authored
      Context tracking is a prerequisite for FULL_NOHZ, so that the RCU
      subsystem can detect CPU idleness without relying on the (regular)
      timer tick.
      Out-of-band activity running over the head domain should by definition
      not be involved in such detection logic, as the root domain has no
      knowledge of what happens - and when - on the head domain whatsoever.
    • Philippe Gerum's avatar
      lib/smp_processor_id: ipipe: exclude head domain from preemption check · bc085b2b
      Philippe Gerum authored
      There can be no CPU migration from the head stage, however the
      out-of-band code currently running smp_processor_id() might have
      preempted the regular kernel code from within a preemptible section,
      which might cause false positive in the end.
      These are the two reasons why we certainly neither need nor want to do
      the preemption check in that case.
    • Philippe Gerum's avatar
      atomic: ipipe: keep atomic when pipelining IRQs · 43381d19
      Philippe Gerum authored
      Because of the virtualization of interrupt masking for the regular
      kernel code when the pipeline is enabled, atomic helpers relying on
      common interrupt disabling helpers such as local_irq_save/restore
      pairs would not be atomic anymore, leading to data corruption.
      This commit restores true atomicity for the atomic helpers that would
      be otherwise affected by interrupt virtualization.
    • Philippe Gerum's avatar
      preempt: ipipe: : add preemption-safe hard_preempt_{enable, disable}() ops · e68fafcc
      Philippe Gerum authored
      Some inner code of the interrupt pipeline may have to traverse regular
      kernel code which manipulates the preemption count, expecting full
      serialization including with out-of-band contexts.
      The hard_preempt_*() variants are substituted to the original
      preempt_{enable, disable}() calls in these cases.
    • Philippe Gerum's avatar
      genirq: ipipe: protect generic chip against domain preemption · 4d7b948f
      Philippe Gerum authored
      As described in Documentation/ipipe.rst, irq_chip drivers need to be
      specifically adapted for dealing with interrupt pipelining safely.
      The basic issue to address is proper serialization between some
      irq_chip handlers which may be called from out-of-band context
      immediately upon receipt of an IRQ, and the rest of the driver which
      may access the same data / IO registers from the regular - in-band -
      context on the same CPU.
      This commit converts the generic irq_chip lock to a hard spinlock,
      which ensures such serialization.
    • Philippe Gerum's avatar
      ipipe: add latency tracer · a8de4bad
      Philippe Gerum authored
      The latency tracer is a variant of ftrace's 'function' tracer
      providing detailed information about the current interrupt state at
      each function entry (i.e. virtual interrupt flag and CPU interrupt
      disable bit). This commit introduces the generic tracer code, which
      builds upon the regular ftrace API.
      The arch-specific code should provide for ipipe_read_tsc(), a helper
      routine returning a 64bit monotonic time value for timestamping
      purpose. HAVE_IPIPE_TRACER_SUPPORT should be selected by the
      arch-specific code for enabling the tracer, which in turn makes
      CONFIG_IPIPE_TRACE available from the Kconfig interface.
    • Philippe Gerum's avatar
      ipipe: add out-of-band tick device · 216ce73c
      Philippe Gerum authored
      The out-of-band tick device manages the timer hardware by interposing
      on selected clockevent handlers transparently, so that a client domain
      (e.g. a co-kernel) eventually controls such hardware for scheduling
      the high-precision timer events it needs to. Those events are
      delivered to out-of-hand activities running on the head stage,
      unimpeded by (only virtually) interrupt-free sections of the regular
      kernel code.
      This commit introduces the generic API for controlling the out-of-band
      tick device from a co-kernel. It also provides for the internal API
      clock event chip drivers should use for enabling high-precision
      timing for their hardware.
    • Philippe Gerum's avatar
      locking: ipipe: add hard lock alternative to regular spinlocks · d46d1207
      Philippe Gerum authored
      Hard spinlocks manipulate the CPU interrupt mask, without affecting
      the kernel preemption state in locking/unlocking operations.
      This type of spinlock is useful for implementing a critical section to
      serialize concurrent accesses from both in-band and out-of-band
      contexts, i.e. from root and head stages.
      Hard spinlocks exclusively depend on the pre-existing arch-specific
      bits which implement regular spinlocks. They can be seen as basic
      spinlocks still affecting the CPU's interrupt state when all other
      spinlock types only deal with the virtual interrupt flag managed by
      the pipeline core - i.e. only disable interrupts for the regular
      in-band kernel activity.