[WIP] Salt Resources

.gitignore

@@ -26,6 +26,10 @@ Pipfile.lock
 /.?env/
 /bin/
 /etc/
+# Allow repo-local Salt dev configs (parent must be un-ignored first).
+!/etc/
+!/etc/salt/
+!/etc/salt/**
 /include/
 /lib/
 /lib64/

doc/ref/grains/all/index.rst

@@ -19,4 +19,5 @@ grains modules
     opts
     package
     pending_reboot
+    resources
     rest_sample

doc/ref/grains/all/salt.grains.resources.rst

@@ -0,0 +1,5 @@
+salt.grains.resources
+=====================
+
+.. automodule:: salt.grains.resources
+    :members:

doc/ref/modules/all/index.rst

@@ -67,6 +67,7 @@ execution modules
     dpkg_lowpkg
     dummyproxy_pkg
     dummyproxy_service
+    dummyresource_test
     environ
     etcd_mod
     ethtool
@@ -215,6 +216,10 @@ execution modules
     ssh_pkg
     ssh_pki
     ssh_service
+    sshresource_cmd
+    sshresource_pkg
+    sshresource_state
+    sshresource_test
     state
     status
     supervisord

doc/ref/modules/all/salt.modules.dummyresource_test.rst

@@ -0,0 +1,6 @@
+salt.modules.dummyresource_test
+===============================
+
+.. automodule:: salt.modules.dummyresource_test
+    :members:
+    :undoc-members:

doc/ref/modules/all/salt.modules.sshresource_cmd.rst

@@ -0,0 +1,6 @@
+salt.modules.sshresource_cmd
+============================
+
+.. automodule:: salt.modules.sshresource_cmd
+    :members:
+    :undoc-members:

doc/ref/modules/all/salt.modules.sshresource_pkg.rst

@@ -0,0 +1,6 @@
+salt.modules.sshresource_pkg
+============================
+
+.. automodule:: salt.modules.sshresource_pkg
+    :members:
+    :undoc-members:

doc/ref/modules/all/salt.modules.sshresource_state.rst

@@ -0,0 +1,6 @@
+salt.modules.sshresource_state
+==============================
+
+.. automodule:: salt.modules.sshresource_state
+    :members:
+    :undoc-members:

doc/ref/modules/all/salt.modules.sshresource_test.rst

@@ -0,0 +1,6 @@
+salt.modules.sshresource_test
+=============================
+
+.. automodule:: salt.modules.sshresource_test
+    :members:
+    :undoc-members:

doc/ref/runners/all/index.rst

@@ -20,6 +20,7 @@ runner modules
     fileserver
     git_pillar
     http
+    index
     jobs
     manage
     match
@@ -30,6 +31,7 @@ runner modules
     pki
     queue
     reactor
+    resource
     salt
     saltutil
     sdb

doc/ref/runners/all/salt.runners.index.rst

@@ -0,0 +1,9 @@
+.. _all-salt.runners.index:
+
+==================
+salt.runners.index
+==================
+
+.. automodule:: salt.runners.index
+    :members:
+    :undoc-members:

doc/ref/runners/all/salt.runners.resource.rst

@@ -0,0 +1,9 @@
+.. _all-salt.runners.resource:
+
+=====================
+salt.runners.resource
+=====================
+
+.. automodule:: salt.runners.resource
+    :members:
+    :undoc-members:

doc/topics/performance/pki_index.rst

@@ -40,13 +40,17 @@ using these options:
 Monitoring and Management
 =========================
 
-You can check the status of your PKI index or force a manual rebuild using the
-:ref:`PKI runner <all-salt.runners.pki>`:
+Check status or rebuild the minion-key mmap index with the
+:ref:`index runner <all-salt.runners.index>` (name ``pki``):
 
 .. code-block:: bash
 
     # Check index status and load factor
-    salt-run pki.status
+    salt-run index.status name=pki
 
     # Manually rebuild the index from the filesystem
-    salt-run pki.rebuild_index
+    salt-run index.compact name=pki
+
+The legacy :ref:`pki runner <all-salt.runners.pki>` (``salt-run pki.status`` /
+``salt-run pki.rebuild_index``) still works but is deprecated and forwards to
+the same implementation.

doc/topics/releases/3009.0.md

@@ -0,0 +1,19 @@
+(release-3009.0)=
+# Salt 3009.0 release notes
+
+<!---
+Do not edit this file. This is auto generated.
+Edit the templates in doc/topics/releases/templates/
+for a given release.
+-->
+
+
+<!--
+Add release specific details below
+-->
+
+<!--
+Do not edit the changelog below.
+This is auto generated.
+-->
+## Changelog

doc/topics/releases/index.rst

@@ -20,6 +20,7 @@ Upcoming release
     :glob:
 
     3008.*
+    3009.*
 
 See `Install a release candidate <https://docs.saltproject.io/salt/install-guide/en/latest/topics/release-candidate.html>`_
 for more information about installing an RC when one is available.

doc/topics/releases/templates/3009.0.md.template

@@ -0,0 +1,14 @@
+(release-3009.0)=
+# Salt 3009.0 release notes{{ unreleased }}
+{{ warning }}
+
+<!--
+Add release specific details below
+-->
+
+<!--
+Do not edit the changelog below.
+This is auto generated.
+-->
+## Changelog
+{{ changelog }}

doc/topics/targeting/index.rst

@@ -111,6 +111,7 @@ There are many ways to target individual minions or groups of minions in Salt:
     nodegroups
     batch
     range
+    resources
 
 
 Loadable Matchers

doc/topics/targeting/resources.rst

@@ -0,0 +1,177 @@
+.. _targeting-resources:
+
+================================
+Targeting Salt Resources
+================================
+
+.. versionadded:: 3009.0
+
+A *Salt resource* is something a minion manages on behalf of the master —
+an SSH host, a virtual appliance, an external API endpoint — addressed
+by an id of the operator's choosing. Resources extend Salt's targeting
+system: every targeting expression that selects minions can also select
+resources.
+
+This page is the targeting reference. For the data model and registration
+plumbing see :py:mod:`salt.utils.resource_registry`.
+
+
+Targeting forms
+===============
+
+Every form below treats resources alongside minions: a single command
+returns one entry per matched id, whether that id belongs to a minion
+or a resource.
+
+Glob and exact-id
+-----------------
+
+A wildcard glob automatically expands to include every resource managed
+by every matched minion::
+
+    salt '*' test.ping
+
+A specific bare id matches a resource directly::
+
+    salt 'web-01' test.ping
+
+A specific minion id targets only the minion (not its resources)::
+
+    salt 'minion-1' test.ping
+
+
+Compound ``T@`` (resource type)
+-------------------------------
+
+``T@<type>`` matches every resource of the given type::
+
+    salt -C 'T@ssh' state.apply
+
+``T@<type>:<id>`` targets exactly one resource::
+
+    salt -C 'T@ssh:web-01' test.ping
+
+
+Grain-based ``-G`` / ``G@``
+---------------------------
+
+A resource carries its own grains, produced by the ``grains`` function
+in the resource's connection module (e.g.
+:func:`salt.resource.dummy.grains`). The master records each minion's
+per-resource grain dicts in the ``resource_grains`` cache bank when the
+minion registers, and ``salt -G`` matches against that bank in addition
+to the per-minion grain bank::
+
+    salt -G 'environment:prod' test.ping
+
+Compound ``G@`` works the same way and supports the full boolean
+algebra (``and``, ``or``, ``not``, parens)::
+
+    salt -C 'G@environment:prod and G@role:web' state.apply
+    salt -C 'T@ssh and not G@environment:staging' test.ping
+
+The boolean form is evaluated **per resource**, so a compound matches a
+resource iff that resource's identity and grains satisfy the entire
+expression.
+
+
+PCRE grain ``-P`` / ``P@``
+--------------------------
+
+Identical semantics to ``-G`` / ``G@`` but values are regex patterns::
+
+    salt -P 'environment:^production-.*' test.ping
+    salt -C 'P@environment:^production-.*' state.apply
+
+
+List ``-L``
+-----------
+
+A bare resource id appearing in a list expression matches::
+
+    salt -L 'web-01,web-02,db-01' test.ping
+
+
+Pillar ``-I`` / ``I@``
+----------------------
+
+.. note::
+
+    Pillar-based targeting of resources is **not** wired up. Resources
+    do not carry per-resource pillar data today. ``-I`` and ``I@`` only
+    match minions; resources are skipped silently. This is tracked as
+    future work — see the gap notes in
+    :py:mod:`salt.utils.resource_registry`.
+
+
+How master and minion split the work
+====================================
+
+Master side
+-----------
+
+The master's ``CkMinions`` augments grain matches with resource ids
+read from the ``resource_grains`` cache bank. The augment runs for
+``-G``, ``-P``, and any ``G@`` / ``P@`` term inside a compound. The
+matched bare resource ids are added to the response wait set so the
+master accepts the corresponding returns.
+
+Minion side
+-----------
+
+When a publish arrives, the minion's ``_resolve_resource_targets``
+walks every locally managed resource and decides, **per resource**,
+whether the targeting expression matches. For glob / list / ``T@``
+this is a string match; for ``G@`` / ``P@`` the minion uses the
+grains it cached during its last registration; for compound, the
+minion evaluates the full boolean expression against each resource's
+identity and grains.
+
+Each matched resource gets its own job dispatch with ``__grains__``
+swapped to the resource's grain dict (so ``salt 'web-01' grains.items``
+returns ``web-01``'s grains, not the managing minion's).
+
+
+Freshness and refresh
+=====================
+
+The master's ``resource_grains`` bank is updated only when a minion
+re-registers via ``_register_resources_with_master``. Triggers that
+re-register are:
+
+* Minion start / reconnect (``tune_in``);
+* A ``saltutil.refresh_pillar`` (the minion's pillar refresh handler
+  re-discovers resources before re-registering); and
+* The ``resource_refresh`` event on the minion event bus.
+
+A per-resource ``<type>.grains_refresh()`` invocation does **not**
+auto-propagate to the master. To force the master's view to refresh
+without waiting for a pillar refresh, fire the ``resource_refresh``
+event for the relevant minion::
+
+    salt-run resource.refresh minion=resources-minion
+
+That runner publishes ``minion/<id>/resource_refresh`` on the master
+event bus; the minion's handler re-runs resource discovery and
+re-publishes its full grain set.
+
+
+Operator inspection
+===================
+
+Two read-only runners expose what the master sees:
+
+.. code-block:: bash
+
+    # Show every SRN currently in the resource_grains bank with a
+    # one-line summary (top-level grain keys + count).
+    salt-run resource.list_grains
+
+    # Show the full grain dict for one resource.
+    salt-run resource.show_grains type=ssh id=web-01
+
+When ``salt -G '<key>:<value>' test.ping`` returns less than expected,
+``resource.list_grains`` is the first place to check: if a resource
+isn't in the bank, the master will not match it, and the resource needs
+a ``saltutil.refresh_pillar`` (or a ``resource.refresh``) on its
+managing minion.