1. 01 Nov, 2018 20 commits
    • Philippe Gerum's avatar
      sched: ipipe: enable task migration between domains · 957ac4c9
      Philippe Gerum authored
      This is the basic code enabling alternate control of tasks between the
      regular kernel and an embedded co-kernel. The changes cover the
      following aspects:
      
      - extend the per-thread information block with a private area usable
        by the co-kernel for storing additional state information
      
      - provide the API enabling a scheduler exchange mechanism, so that
        tasks can run under the control of either kernel alternatively. This
        includes a service to move the current task to the head domain under
        the control of the co-kernel, and the converse service to re-enter
        the root domain once the co-kernel has released such task.
      
      - ensure the generic context switching code can be used from any
        domain, serializing execution as required.
      
      These changes have to be paired with arch-specific code further
      enabling context switching from the head domain.
      957ac4c9
    • Philippe Gerum's avatar
      clockevents: ipipe: connect clock chips to abstract tick device · 1befe95a
      Philippe Gerum authored
      Announce all clock event chips as they are registered to the
      out-of-band tick device infrastructure, so that we can interpose on
      key handlers in their descriptors.
      1befe95a
    • Philippe Gerum's avatar
      timekeeping: ipipe: forward clock shift value to DSO helpers · 9e2ee43a
      Philippe Gerum authored
      In order to propagate the "host real-time update" event to a co-kernek
      (IPIPE_KEVT_HOSTRT), we need the clock shift value of the monotonic
      clock to be passed to the legacy vDSO handler, for (re)calculating the
      new wall clock time which is eventually announced to the co-kernel.
      
      Only architectures which still implement the legacy
      update_vsyscall_old() interface need this change.
      9e2ee43a
    • Philippe Gerum's avatar
      ipipe: add kernel event notifiers · 96df83cb
      Philippe Gerum authored
      Add the core API for enabling (regular) kernel event notifications to
      a co-kernel running over the head domain. For instance, such a
      co-kernel may need to know when a task is about to be resumed upon
      signal receipt, or when it gets an access fault trap.
      
      This commit adds the client-side API for enabling such notification
      for class of events, but does not provide the notification points per
      se, which comes later.
      96df83cb
    • Philippe Gerum's avatar
      printk: ipipe: add raw console channel · 4cd03bbb
      Philippe Gerum authored
      A raw output handler (.write_raw) is added to the console descriptor
      for writing (short) text output unmodified, without any logging,
      header or preparation whatsoever, usable from any pipeline domain.
      
      The dedicated raw_printk() variant formats the output message then
      passes it on to the handler holding a hard spinlock, irqs off.
      
      This is a very basic debug channel for situations when resorting to
      the fairly complex printk() handling is not an option. Unlike early
      consoles, regular consoles can provide a raw output service past the
      boot sequence. Raw output handlers are typically provided by serial
      console devices.
      4cd03bbb
    • Philippe Gerum's avatar
      dump_stack: ipipe: make dump_stack() domain-aware · 9ac7f531
      Philippe Gerum authored
      When dumping a stack backtrace, we neither need nor want to disable
      root stage IRQs over the head stage, where CPU migration can't
      happen.
      
      Conversely, we neither need nor want to disable hard IRQs from the
      head stage, so that latency won't skyrocket either.
      9ac7f531
    • Philippe Gerum's avatar
      driver core: ipipe: defer dev_printk() from head domain · ce97be14
      Philippe Gerum authored
      Just like printk(), dev_printk() cannot run from the head domain
      and/or with hard IRQs disabled. In such a case, log the output
      directly into the staging buffer we use for printk().
      
      NOTE: when redirected to the buffer, the output does not include the
      dev_printk() header text but is merely sent as-is to the log.
      ce97be14
    • Philippe Gerum's avatar
      printk: ipipe: defer printk() from head domain · 5f8436dc
      Philippe Gerum authored
      The printk() machinery cannot immediately invoke the console driver(s)
      when called from the head domain, since such driver code belongs to
      the root domain and cannot be shared between domains.
      
      Output issued from the head domain is formatted then logged into a
      staging buffer, and a dedicated virtual IRQ is posted to the root
      domain for notification. When the virtual IRQ handler runs, the
      contents of the staging buffer is flushed to the printk() interface
      anew, which may eventually pass the output on to the console drivers
      from such a context.
      5f8436dc
    • Philippe Gerum's avatar
      PM / hibernate: ipipe: protect against out-of-band interrupts · c9a32947
      Philippe Gerum authored
      We must not allow out-of-band activity to resume while we are busy
      suspending the devices in the system, until the PM sleep state has
      been fully entered.
      
      Pair existing virtual IRQ disabling calls which only apply to the root
      domain with hard ones.
      c9a32947
    • Philippe Gerum's avatar
      module: ipipe: enable try_module_get() from hard atomic context · 0c527f67
      Philippe Gerum authored
      We might have out-of-band code calling try_module_get() from the head
      domain, or from a section covered by a hard spinlock where the root
      domain must not reschedule. This requires the preemption management
      calls in try_module_get() (and the converse module_put()) to be
      converted to their hard variant.
      
      REVISIT: using try_module_get() from such contexts is questionable,
      client domains should be fixed.
      0c527f67
    • Philippe Gerum's avatar
      KGDB: ipipe: enable debugging over the head domain · 4e5d5bc7
      Philippe Gerum authored
      Make the KGDB stub runnable over the head domain since we may take
      traps and interrupts from that context too, by converting the locks to
      hard spinlocks.
      4e5d5bc7
    • Philippe Gerum's avatar
      context_tracking: ipipe: do not track over the head domain · 58992a82
      Philippe Gerum authored
      Context tracking is a prerequisite for FULL_NOHZ, so that the RCU
      subsystem can detect CPU idleness without relying on the (regular)
      timer tick.
      
      Out-of-band activity running over the head domain should by definition
      not be involved in such detection logic, as the root domain has no
      knowledge of what happens - and when - on the head domain whatsoever.
      58992a82
    • Philippe Gerum's avatar
      lib/smp_processor_id: ipipe: exclude head domain from preemption check · e8a89820
      Philippe Gerum authored
      There can be no CPU migration from the head stage, however the
      out-of-band code currently running smp_processor_id() might have
      preempted the regular kernel code from within a preemptible section,
      which might cause false positive in the end.
      
      These are the two reasons why we certainly neither need nor want to do
      the preemption check in that case.
      e8a89820
    • Philippe Gerum's avatar
      atomic: ipipe: keep atomic when pipelining IRQs · 76498343
      Philippe Gerum authored
      Because of the virtualization of interrupt masking for the regular
      kernel code when the pipeline is enabled, atomic helpers relying on
      common interrupt disabling helpers such as local_irq_save/restore
      pairs would not be atomic anymore, leading to data corruption.
      
      This commit restores true atomicity for the atomic helpers that would
      be otherwise affected by interrupt virtualization.
      76498343
    • Philippe Gerum's avatar
      preempt: ipipe: : add preemption-safe hard_preempt_{enable, disable}() ops · 0c4d0889
      Philippe Gerum authored
      Some inner code of the interrupt pipeline may have to traverse regular
      kernel code which manipulates the preemption count, expecting full
      serialization including with out-of-band contexts.
      
      The hard_preempt_*() variants are substituted to the original
      preempt_{enable, disable}() calls in these cases.
      0c4d0889
    • Philippe Gerum's avatar
      genirq: ipipe: protect generic chip against domain preemption · 0a80a322
      Philippe Gerum authored
      As described in Documentation/ipipe.rst, irq_chip drivers need to be
      specifically adapted for dealing with interrupt pipelining safely.
      
      The basic issue to address is proper serialization between some
      irq_chip handlers which may be called from out-of-band context
      immediately upon receipt of an IRQ, and the rest of the driver which
      may access the same data / IO registers from the regular - in-band -
      context on the same CPU.
      
      This commit converts the generic irq_chip lock to a hard spinlock,
      which ensures such serialization.
      0a80a322
    • Philippe Gerum's avatar
      ipipe: add latency tracer · 20700661
      Philippe Gerum authored
      The latency tracer is a variant of ftrace's 'function' tracer
      providing detailed information about the current interrupt state at
      each function entry (i.e. virtual interrupt flag and CPU interrupt
      disable bit). This commit introduces the generic tracer code, which
      builds upon the regular ftrace API.
      
      The arch-specific code should provide for ipipe_read_tsc(), a helper
      routine returning a 64bit monotonic time value for timestamping
      purpose. HAVE_IPIPE_TRACER_SUPPORT should be selected by the
      arch-specific code for enabling the tracer, which in turn makes
      CONFIG_IPIPE_TRACE available from the Kconfig interface.
      20700661
    • Philippe Gerum's avatar
      ipipe: add out-of-band tick device · 88f3dede
      Philippe Gerum authored
      The out-of-band tick device manages the timer hardware by interposing
      on selected clockevent handlers transparently, so that a client domain
      (e.g. a co-kernel) eventually controls such hardware for scheduling
      the high-precision timer events it needs to. Those events are
      delivered to out-of-hand activities running on the head stage,
      unimpeded by (only virtually) interrupt-free sections of the regular
      kernel code.
      
      This commit introduces the generic API for controlling the out-of-band
      tick device from a co-kernel. It also provides for the internal API
      clock event chip drivers should use for enabling high-precision
      timing for their hardware.
      88f3dede
    • Philippe Gerum's avatar
      locking: ipipe: add hard lock alternative to regular spinlocks · 7c28f350
      Philippe Gerum authored
      Hard spinlocks manipulate the CPU interrupt mask, without affecting
      the kernel preemption state in locking/unlocking operations.
      
      This type of spinlock is useful for implementing a critical section to
      serialize concurrent accesses from both in-band and out-of-band
      contexts, i.e. from root and head stages.
      
      Hard spinlocks exclusively depend on the pre-existing arch-specific
      bits which implement regular spinlocks. They can be seen as basic
      spinlocks still affecting the CPU's interrupt state when all other
      spinlock types only deal with the virtual interrupt flag managed by
      the pipeline core - i.e. only disable interrupts for the regular
      in-band kernel activity.
      7c28f350
    • Philippe Gerum's avatar
      genirq: add generic I-pipe core · d9f057db
      Philippe Gerum authored
      This commit provides the arch-independent bits for implementing the
      interrupt pipeline core, a lightweight layer introducing a separate,
      high-priority execution stage for handling all IRQs in pseudo-NMI
      mode, which cannot be delayed by the regular kernel code. See
      Documentation/ipipe.rst for details about interrupt pipelining.
      
      Architectures which support interrupt pipelining should select
      HAVE_IPIPE_SUPPORT, along with implementing the required arch-specific
      code. In such a case, CONFIG_IPIPE becomes available to the user via
      the Kconfig interface for enabling the feature.
      d9f057db
  2. 20 Oct, 2018 20 commits