Workflow concepts

The command reference documents each command’s own arguments. This page covers the cross-cutting behaviour that sits behind those commands — how fan-out works, how reference hosts are drawn from the pool, and how locking keeps concurrent testers from colliding. It applies equally to the mtui REPL and the mtui-mcp tools, since both dispatch through the same engine.

Templates and the active template

Loading a testreport with load_template adds it to the session as a template; previously loaded templates stay loaded and the new one becomes active. Each template owns its own reference hosts and its own report. list_templates shows the loaded set (the active one marked *), switch changes the active one, and unload drops one (closing only its host connections).

Session-level commands (load_template, unload, switch, list_templates, list_refhosts, updates, config, whoami, set_log_level, help, quit) take no template flag: each runs exactly once, never fans out, and accepts neither -T nor --all-templates.

Fan-out across templates

When more than one template is loaded, a command falls into one of two groups:

• Read/annotate commands (list_*, show_*, openqa_*, checkers, checkout, export, set_workflow, comment, …) fan out across every loaded template by default — none of them mutates shared remote state, and re-running one is always safe. Each template’s output is prefixed with an === <RRID> === banner so results stay attributable.
• Host-mutating and remote-write commands (update, prepare, downgrade, install, uninstall, set_repo, run, reboot, lock, unlock, add_host, remove_host, get, put, approve, reject, assign, unassign, request_review, commit) never implicitly fan out: bare, they act on exactly one template — the active one in the REPL, or the sole loaded one — and refuse rather than guess when several are loaded headlessly and none is named.

Scope a single call three ways:

• -T <RRID> / --template <RRID> — act on one loaded template only.
• --all-templates — force fan-out explicitly, on either group. Mutually exclusive with -T.
• --all-templates=false — suppress fan-out on a read/annotate command, narrowing it to the same single-template resolution the other group uses by default.

A headless caller (MCP) with several templates loaded and neither flag named gets AmbiguousTemplate, not a silent guess — the error names the loaded RRIDs and how to scope the call.

Failure handling is per-template: if a fanned-out command fails on one template it keeps running on the rest and reports an aggregate failure at the end. When a host-phase command (one that accepts -t) fans out without explicit -t hosts, a loaded template with no connected host is skipped with a warning rather than counted as a failure — so an unscoped lock/run/… still succeeds on the templates that do have hosts. If every template is host-less the command ran nowhere and fails with “No refhosts defined”. Naming a disconnected host with -t <host> is still a per-template failure, and a -T-scoped call keeps single-template error behaviour.

regenerate also runs exactly once, on the active template by default; -T names another loaded one.

Over MCP the same model applies through per-call template= / all_templates= parameters — see MCP server.

Reference-host pool selection

When add_host connects hosts from a testplatform (and during the autoconnect that runs when a template loads), mtui connects one reference host per test-target slot — a slot being a unique (product, version, arch, addons) combination — rather than every matching candidate. So a testplatform that resolves to several interchangeable hosts on the same architecture does not fan a command out across all of them.

• The host drawn for each slot is chosen at random among the free candidates, spreading load across interchangeable refhosts.
• If the chosen host fails to connect, mtui automatically tries another candidate from the same slot; only when every candidate for a slot is unreachable or claimed does add_host name the slot and why in its result and leave that slot unconnected.
• Under fan-out, each test-target slot draws a distinct free host from the shared pool, arbitrated in-process so two templates never collide on the same host. When every candidate for a slot is busy, the claim queues on the exhausted pool according to [lock] wait / [lock] wait_poll (see Configuration).

To search the inventory without connecting, use list_refhosts — it reads the same source offline (no SSH, no lock, no loaded report).

Product-drift check on connect

When a host connects, mtui checks that the products actually installed on it (from /etc/products.d) match what refhosts.yml records for that host. Any drift — a wrong or wrong-version base product, a wrong architecture, missing/unexpected addons, or a dangling baseproduct symlink — is logged as a WARNING and the host is kept (the check never aborts a connect). The qa product is ignored, and hosts not listed in refhosts.yml are skipped silently.

Host state

Each connected host has a state, set with set_host_state:

• enabled — runs all issued commands (the default).
• disabled — runs nothing and prints nothing.

Commands designed to run in parallel (like run) always fan out concurrently across enabled hosts.

Locking

Two independent locks coordinate concurrent testers on a shared fleet. Both use the same on-host wire format (timestamp:user:pid[:comment]), so concurrent mtui processes — and any other fleet tooling that reads the same files — cooperate rather than clobber one another.

• Operation / zypper lock (/var/lock/mtui.lock, PID-based) — guards serialized repository transactions (enabling/disabling the test repo, install/update/prepare/downgrade). Set it explicitly with lock; it is applied automatically around the flows that need it. Enabled locks are removed when the session exits. To make the lock effective for other sessions too, give it a comment (lock -c "<why>"). Such a reservation is never rewritten or released by run, update, install or prepare on the host; only unlock removes it. Templates loaded in the same process, and the sessions of one mtui-mcp server, hold the operation lock against each other like separate users: a second template’s run or update waits for (or, at [lock] wait = 0, refuses on) the first one’s hold.
• Pool-claim lock (/var/lock/mtui-pool.lock, RRID-based) — taken during host-pool selection to reserve a host for a template. List and remove them with the -p/--pool flag on list_locks / unlock.

By default, lock/unlock act on the whole active template’s hosts; pass -t <host> (repeatable) to scope either to a subset. Plain unlock releases only the operation lock this session took: a lock it never took — another user’s, or a live sibling template’s on a refhost shared inside the same process — is left alone and reported skipped. A foreign lock points at list_locks or --force; a sibling’s is named as held by this mtui process and is released from the template that took it. unlock -f force-removes a lock held by another user or session regardless of who took it. mtui also reaps a pre-existing lock older than [lock] stale_age on connect (almost always left over from a crashed session); see Configuration for reap_stale, stale_age, wait, and wait_poll. The one exception is list_refhosts --free: its connect is a read-only survey, so it never reaps. Its lock column reports one of free, locked, pool (free but pool-claimed), locked+pool, unreachable (the host could not be probed), or unknown (the probe never completed for it); --json keeps this vocabulary and adds a pool key: the claiming RRID, or null when unclaimed.

Product-Increment autolock

When testing a Product Increment (PI) and [lock] pi_autolock is enabled (the default), mtui locks each reference host — with a comment naming the request — as it connects, for the life of the loaded report; the lock is released when the report is unloaded or the session quits, matching how every non-PI report’s pool claims already behave. A host added later with add_host (including a pool-selected host, with no -t) is locked too, since the lock comment is seeded on load and every connect path applies it. Review actions (assign/unassign/approve/reject) do not lock or unlock hosts themselves. A reboot clears /var/lock, so the per-host testing lock is re-applied after the host comes back.

Assignment context on assign

After the assignment succeeds, assign prints best-effort context from the TeReGen report API: the update’s live priority and deadline and, when the report already carries assignment state, who currently holds or has decided each review group (one line per group). This is informational only and never blocks the action — its absence can also mean the lookup simply failed, and the server caches the state for a few minutes, so a line may lag by up to ~5 minutes.

Kernel export file names

A kernel export saves each openQA job’s install log as <host>-zypper-<arch>.log and each LTP result as <host>-<arch>-<test>.json. When several jobs of one export would write the same file, each keeps its own file with the job ID inserted before the extension (<host>-zypper-<arch>-job-<id>.log); a name only one job claims never changes. Files from an earlier export are not deleted, so a plain <host>-zypper-<arch>.log written before this naming existed can remain next to the job-ID files.