Phase 3: SCMI binds, the CPUs get cpufreq (2026-11-03)

Attempt 2026-11-03-8. Image ca51a59382dff6c8, kernel #7 (2026-11-03 4:07 PM ET), commit ca81e32. Sources: the netlog dumps/firstboot/2026-11-03-netlog.txt and the AP console log dumps/firstboot/2026-11-03-ap.log (both gitignored).

Why

Every boot up to attempt -7 ended with one deferred device: firmware:scmi … supplier 17430000.mailbox not ready, then arm-scmi: failed to setup channels and failed to setup channel for protocol:0x10. The CPUs stayed at whatever clocks the firmware left them on.

• mailbox@17430000 is qcom,x1e80100-cpucp-mbox. Its driver is QCOM_CPUCP_MBOX (drivers/mailbox/Kconfig:343, qcom-cpucp-mbox.c:205), =m in defconfig (arch/arm64/configs/defconfig:1614), and the initramfs loads no modules.
• The stock tree drives the same block: cpucp: qcom,cpucp@18830000 (qcom,cpucp-v2) with the same two register ranges (0x17430000, 0x18830000), an arm,scmi node with mboxes = <cpucp 0 cpucp 2> and shmem in mmio-sram@18b4e000 (entry00-base.dts:384-402, :908-926, :10534-10540). hamoa.dtsi describes the same, and each CPU takes its perf domain from scmi_dvfs (hamoa.dtsi:78-79). So no DTS change. The stock tree also lists a vendor protocol@80 that hamoa.dtsi does not have.

Changes (commit ca81e32)

• kernel/mica-scmi.config: QCOM_CPUCP_MBOX=y. It also checks that the mailbox, SCMI core, SCMI mailbox transport, SCMI perf domain and SCMI cpufreq options are =y after the merge (they were already).
• /init: one cpufreq line per policy (CPUs, driver, governor, current, min and max kHz) and an scmi-dmesg capture.

Result: pass

• SCMI binds over the mailbox: arm-scmi arm-scmi.0.auto: Using scmi_mailbox_transport at 0.358 s, SCMI Protocol v2.0 'Qualcomm:' Firmware version 0x20000, scmi-perf-domain scmi_dev.4: Initialized 3 performance domains at 0.376 s.

• cpufreq on all three clusters (schedutil governor):

  Policy	CPUs	Driver	Current at ~110 s	Range
  policy0	0-3	scmi	806.4 MHz	710.4 MHz to 3417.6 MHz
  policy4	4-7	scmi	2188.8 MHz	710.4 MHz to 3417.6 MHz
  policy8	8-11	scmi	2188.8 MHz	710.4 MHz to 3417.6 MHz

• Nothing deferred: deferred through deferred5 are all empty, for the first time on mica.

• Everything else unchanged: both DSPs running at the end, Type-C, PCIe, Wi-Fi (6 GHz, DHCP), eDP 2560x1600, GPU firmware and GMU loaded, netlog sent (1379 lines). reboot: Restarting system at 174.8 s (about 4:11 PM ET).

• Earlier milestones than attempt -7: CDSP up at 0.79 s (was 1.99 s), ADSP at 6.13 s (was 7.41 s, still after the known 5 s SSCTL wait, cdsp-2026-11-03.md), PCIe Gen4 link at 1.11 s (was 2.28 s). Inference: the CPUs now run faster than the firmware’s default clocks. Not measured directly.

Log lines checked

• [Firmware Bug]: Failed to add opps_by_lvl at 3417600 for NCC1 - ret:-16 (twice each for NCC1 and NCC2). This is a dev_info from drivers/firmware/arm_scmi/perf.c:381-383 (and :403-405): xa_insert() returned -EBUSY, so the firmware lists the 3.4176 GHz level more than once for those domains. The level still registers as max=3417600. Inference: harmless.
• region@80a00000 … overlaps with cpucp-log@80e00000 at 0.000 s is not new: it is in every boot’s log on 2026-11-03 (7 of 7) and in 2026-11-02-2. region@80a00000 is not in the mica DTB. Inference: the boot firmware adds it. Not traced further.

Update: attempt -9 (2026-11-03, 4:20 PM ET): SCMI timed out, no cpufreq

Same SCMI config as attempt -8 (the only kernel change was kernel/patches/0001, panel-edp). Sources as above.

• arm-scmi bound over scmi_mailbox_transport at 0.367 s as before. From 0.437 s on, 10 timed out in resp(caller: do_xfer+0xdc/0x180), each about 36 ms after the previous one (the 30 ms max-rx-timeout).
• Late answers arrive for older requests: scmi_protocol scmi_dev.1: Message for 7 type 0 is not expected!, then for 9 and for 32. Inference: the CPUCP firmware answered, but after the kernel had given up on each request.
• Results: SCMI Protocol v2.0 'Qualcomm:' Firmware version 0x0 (was 0x20000, so the version query failed and the quirk_perf_level_get_fc_force quirk was not applied), Failed to query supported version for protocol 0x13, scmi-cpufreq scmi_dev.3: probe with driver scmi-cpufreq failed with error -110, and /init saw cpufreq=none. scmi-perf-domain still initialized 3 performance domains at 0.806 s.
• Nothing else failed: deferred stayed empty, both DSPs ran, display, GPU and Wi-Fi as in -8.
• Upstream has seen the same timeout on x1e80100. Review of the series that enabled X1E cpufreq reports arm-scmi firmware:scmi: timed out in resp on the CRD (PATCH V7 0/2 “qcom: x1e80100: Enable CPUFreq”, 2024-11). The mailbox driver at a804c57 already clears each channel’s interrupt under the channel lock (drivers/mailbox/qcom-cpucp-mbox.c, qcom_cpucp_mbox_irq_fn).

Open: with one good boot (-8) and one bad (-9), the cause is not known. Next step: more boots of the same image to get a failure rate, then compare the first SCMI exchange’s timing across good and bad boots.

Update: cold boot vs warm reset (2026-11-03, 5:25 PM ET)

Attempts -10, -11 and -12, plus the EC console log dumps/firstboot/2026-11-03-ec.log (gitignored).

Matching the EC times to boots: each mainline boot’s reboot comes its uptime plus ~9 s of firmware later (-8: 174.8 s uptime, 184 s; -9: 189.5 s, 198 s; -11: 129.5 s, 138 s).

• Result: SCMI worked in both cold boots and failed in every warm reset from running stock Android, with and without the panel patch (-11 used the plain pin, GBOOK_SKIP_PATCHES=1). The same image failed warm (-11) and worked cold (-12).
• Inference: stock Android’s kernel leaves the CPUCP in a state that survives a warm reset (the stock tree drives it as qcom,cpucp-v2 with a vendor protocol@80), and in that state it answers mainline’s SCMI late or not at all. How the stock kernel uses it is not traced. With two cold and two logged warm boots, this is a strong pattern, not proof.
• The failures grew worse from -9 to -11. Inference: each failed boot may leave the firmware in a worse state for the next warm reset (the research note’s quirk hazard, scmi-cpucp-timeouts.md).

Procedure from now on: power stock off fully before each attempt. attempt.sh verify prints that step, attempt.sh watch shows the EC’s G3->S5 line for a cold start, and /init reports the firmware version as the scmi-fw key so compare flags a warm-reset boot. arm,max-rx-timeout-ms is not needed for cold boots; it stays an option if a warm boot must work (for example after a crash reboot).

Update: cold starts fail too (2026-11-03, 5:59 PM ET)

Three later boots each followed an EC G3->S5 (a cold start; EC uptime 15984, 17025 and 17236 s in dumps/firstboot/2026-11-03-ec.log), with the EC work’s images (ec-2026-11-03.md):

So a cold start does not always bring SCMI up: the “cold boot only” rule above holds for the warm side (no warm reset has worked) but not as a guarantee on the cold side. Open lead 2 in the handoff (SCMI after a warm reset) widens to SCMI in general; docs/research/scmi-cpucp-timeouts.md ranks arm,max-rx-timeout-ms first.

Update: the 250 ms window and the trace (2026-11-03, about 8 PM ET)

arm,max-rx-timeout-ms = <250> (commit d555a07, unverified diagnostic) took effect on both boots (SCMI max-rx-timeout: 250ms).

• 2026-11-03-20 (cold; the EC itself reset at power-on): xfers_response_timeout=7 of sent_ok=78 (response_ok=74, err_protocol=1), firmware 0x0, no quirk_perf_level_get_fc_force (the hazard path, scmi-cpucp-timeouts.md section 4), cpufreq on 2 of 3 clusters.
• 2026-11-03-21 (cold, tracing on): firmware 0x20000, the FC quirk, 3 of 3 clusters, 0 timeouts; the trace’s scmi_xfer_begin to scmi_rx_done latency over 78 replies peaked at 4.0 ms.
• So when CPUCP answers, it answers within milliseconds; on a bad boot some messages get no reply even in 250 ms. A longer window does not help (inference from 2 boots); the next lever is why CPUCP stops answering (research options 3-4: the guard, arm,no-completion-irq, cpuidle.off=1).

Update: reply latency per message, attempt -22 (2026-11-03, 8:14 PM ET)

Cold start, 250 ms window, tracing on: firmware 0x20000, the FC quirk, 3 of 3 clusters, and 4 timeouts. The trace (scmi_xfer_begin to scmi_rx_done, netlog “trace at probe”):

• 67 of 75 replies in under 1 ms.
• The base protocol’s discovery messages (pt=10, msg_id 01 to 06) took 23 to 36 ms each, one after another. msg_id 05 (implementation version, the “Firmware version” line) took 36.0 ms. With the driver’s default 30 ms window that reply is missed, the version stays 0x0 and the FC quirk does not match (inference: it fits every bad boot’s timed out in resp lines about 36 ms apart and Firmware version 0x0).
• Slower still: base msg_id 07 at 222.4 ms, perf (pt=13) msg_id 02 at 155.2 ms and 0B at 67.9 ms.
• Timed out at 250 ms: base msg_id 06, perf msg_id 04 and 02 (twice).
• So the window matters: 250 ms turned this boot good where 30 ms would have failed it. -20 still failed at 250 ms, so some replies take longer or never come. Two good boots (-21, -22) also both had tracing on; that it shifts the timing is unproven (inference).

Update: 1000 ms window, attempts -23 to -25 (2026-11-03, 8:31 PM ET)

arm,max-rx-timeout-ms = <1000> (commit after 927480c), tracing on, three cold starts of one image:

• Since the window went to 250 ms and tracing on: 5 good cold starts in a row (-21 to -25), against 2 of 7 before with the default 30 ms (-20 at 250 ms failed).
• Reply time varies by boot: some boots answer everything within milliseconds (-21, -25), others have a tail of single messages at 0.6 to 1 s. Why CPUCP stalls is still open; the window only absorbs it.
• Not a fix: the property is local and unverified (no stock counterpart), and a boot can still lose a message past 1 s (-23’s perf 04 at 1014 ms). It costs nothing on a fast boot.

Update: cluster_cl5 A/B, attempt -27 (2026-11-03, 9:50 PM ET)

Attempt 2026-11-03-27: kernel #17 (as -26), 1000 ms window, tracing on, plus the diagnostic fragment dts/diag/hamoa-no-cluster-cl5.dtsi (commit 5aa73de): domain-idle-states = <&cluster_cl4> on cluster_pd0-2, so the three CPU clusters never enter cluster_cl5. Source: Jens Glathe’s “arm64: dts: qcom: hamoa: Drop cluster_cl5 idle state from CPU clusters” (https://lore.kernel.org/all/20260604-dc_zva_mitigation-v1-1-d1148c1c0259@oldschoolsolutions.biz/), carried by Omarchy as linux-omarchy-aarch64 patch 2151. Image a563a107aeedf282, DTB 4fa8ee79b09d1ea9; three cold starts of one image (stock powered off, EC G3 before each):

• Against the same 1000 ms window with cluster_cl5 (-23 to -26): 2 of 4 boots stalled (-23, -24), 2 answered within 4 ms (-25, -26). Without cluster_cl5: 2 of 3 stalled. No improvement: the A/B failed. The stalls do not come from the cluster_cl5 idle state (3 boots; a small sample, but the worst probe of the day, boot 3, ran without it).
• All three boots came up fully, so the 1000 ms window keeps absorbing the stalls; 0 timeouts here.
• The stalls fall between 0.37 s and 4.1 s of kernel time, during the SCMI probe and before /init runs, so they do not depend on anything done at the keyboard.
• The fragment is deleted (its text is in commit 5aa73de), per dts/diag/README.md. Open: the other idle-state lever in scmi-cpucp-timeouts.md (cpuidle.off=1, which also covers the CPU idle states the fragment left alone).

Update: cpuidle.off=1 A/B, attempt -28 (2026-11-03, 10:02 PM ET)

Attempt 2026-11-03-28: kernel #17, the board DTB as -26 (f30182513670c4e3, no dts/diag), 1000 ms window, tracing on, plus GBOOK_CMDLINE_EXTRA=cpuidle.off=1 (cpuidle_register_driver() returns -ENODEV while off is set, drivers/cpuidle/driver.c:225 at a804c57, so the idle loop falls back to plain WFI and no PSCI idle state is requested: not the CPU ret state, the cluster states or domain_ss3 on system_pd; the last part is inference from the PSCI cpuidle driver being the only requester). Image 21db583f38e653a3; three cold starts of one image (stock powered off, EC G3 before each):

The trace’s only flagged rows are base DISCOVER_SUB_VENDOR and DISCOVER_AGENT with s=-13 (not supported, 0.0 ms), as on every boot; the SCMI probe ends by 0.38 s.

• With idle states on (1000 ms window, -23 to -27): 4 of 7 cold boots stalled (-23, -24, -27 boots 1 and 3; replies of 0.5 to 1 s), and even the clean ones peaked at 3.6 to 4.0 ms (-21, -25, -26, -27 boot 2). With cpuidle.off=1: 0 of 3 stalled and every reply within 0.1 ms, a floor 40 times lower.
• So the CPUCP’s slow replies follow the AP’s idle states. That matches Sibi Sankar’s statement that the CPUCP firmware bug “happens when idle states are enabled” (scmi-cpucp-timeouts.md section 4). Three boots is a small sample: with 4 of 7 stalling, three clean boots in a row by chance is about 8 % (inference).
• cluster_cl5 alone is ruled out (-27). Left are the CPU ret state (cluster_c4), cluster_cl4 and domain_ss3; the stock tree has the same four states and topology (entry00-base.dts: cluster_pd0-2 with cl4 and cl5, system_pd with system_ss3), so stock Android boots with them (inference: they are not wrong for this board).
• Not a config to keep: cpuidle.off=1 costs idle power. It is a diagnostic; the 1000 ms window stays.

Update: what the trace cannot tell (2026-11-03, 10:20 PM ET)

The latency in the tables above runs from scmi_xfer_begin to scmi_rx_done, and rx_done is traced in the mailbox interrupt’s handler (driver.c:1164 at a804c57), so it cannot separate a late CPUCP reply from an on-time reply whose interrupt woke an idle AP CPU late. “The slow replies follow the AP’s idle states” (-28) holds either way; which side is slow is open. Sibi Sankar’s “idle states” statement is about PERF_LEVEL_GET hard resets, not reply latency, so it does not decide this (inference). The discriminator is arm,no-completion-irq with idle states on (dts/diag/scmi-no-completion-irq.dtsi): polling skips the interrupt; trace_report.py times a polled reply from begin to end. Stock wires the mailbox the same way (qcom,cpucp-v2 at 0x17430000/0x18830000, GIC SPI 28 level-high, no PDC; SCMI channels 0 and 2, no arm,max-rx-timeout-ms), with a downstream mailbox driver.

Update: arm,no-completion-irq A/B, attempt -29 (2026-11-03, 10:24 PM ET)

Attempt 2026-11-03-29: kernel #17, 1000 ms window, tracing on, idle states on, plus dts/diag/scmi-no-completion-irq.dtsi (polling for every SCMI reply; dmesg: Enabled polling mode TX channel). Image aa0661198276109c; three cold starts of one image (EC G3 before each):

(trace_report.py times a polled reply from scmi_xfer_begin to scmi_xfer_end; the two s=-13 replies count as replies after a fix the same night, so /init’s on-device line on this image still says 76.)

• The CPUCP is not slow. With idle states on, it frees the shared-memory channel within 0.1 ms of every request on all three boots, where 4 of 7 interrupt-mode boots stalled for 0.5 to 1 s. The slow part is the completion interrupt.
• The IRQ count is 0 because a polled message asks for no interrupt (shmem.c:123: the channel’s flag SCMI_SHMEM_FLAG_INTR_ENABLED is written as 0 when poll_completion is set). In interrupt mode every boot of the day counted one apss_cpucp_mbox IRQ per reply (75 to 78), all on CPU 0 (GBOOK: irq-counts=, netlog), stalled boots included: the interrupts arrive, late.
• Which half is late, the CPUCP raising it or CPU 0 waking for it, is still open. A lead (inference): the stall ceiling tracks the rx window (timeouts about 36 ms apart at 30 ms, a 222 ms reply at 250 ms, 940 to 1014 ms at 1000 ms), which fits a CPU 0 that sleeps through the interrupt until its own timeout timer wakes it, then takes the pending interrupt just before or just after the timeout. GIC SPI 28 has no PDC pin (qcom,pdc-ranges in hamoa.dtsi:6028 maps SPIs 251-255, 480-573, 609-640, 717-728, 816-834), so it cannot wake the SoC from a state that needs the PDC; whether domain_ss3 on system_pd is such a state is unverified.
• The fragment stays (status: proven as a diagnostic) until the owner decides between polling, the window, or a narrower idle-state change.

Update: what -29 shows and does not (2026-11-03, 10:28 PM ET)

Polling keeps one AP core spinning through the whole SCMI probe, and cpuidle.off=1 (-28) keeps every core out of the PSCI states, so both fast cases share one property: the SoC never reaches its deepest state during the probe. -29 shows that the CPUCP answers within 0.1 ms while one AP core is awake; whether it also does when all cores sleep is untested. Two readings fit: CPU 0 sleeps through an on-time IRQ (SPI 28 has no PDC pin) until a timer wakes it, or the CPUCP defers its reply while the AP is in a deep state. The window-tracking lead favours the first (inference). Next A/B: domain_ss3 dropped from system_pd alone (dts/diag/hamoa-no-system-ss3.dtsi).

Update: domain_ss3 A/B, attempt -30 (2026-11-03, 10:34 PM ET)

Attempt 2026-11-03-30: kernel #17, 1000 ms window, interrupt mode, CPU and cluster idle states as stock, plus dts/diag/hamoa-no-system-ss3.dtsi (system_pd without domain-idle-states; the built DTB’s /psci/power-domain-system has none). Image 8296ea098da5e5b3. One cold start (EC G3 before it):

• Pass (0x20000, FC quirk, cpufreq 3 of 3), but the worst SCMI boot of the day: 1 timeout (perf DESCRIBE_LEVELS, 1012.1 ms, s=-110), base DISCOVER_AGENT 587.0 ms; 76 replies, 76 apss_cpucp_mbox IRQs, all on CPU 0.
• Failed: dropping the system-level state does not stop the stalls. One stalled boot settles it, so boots 2 and 3 were skipped. With -27, the cluster cl5 state and the system ss3 state are each ruled out alone; left are the CPU ret state and cluster_cl4, or a combination.
• Also on this boot, unexplained: the touchscreen probe failed with -110 at 12.8 s (usually -22), with GENI I2C errors on a80000.i2c: 4 transaction timeouts, plus NACK, arbitration-lost and bus-protocol errors (attempt.sh trace i2c touchscreen). The NACK and arbitration errors point at the bus or the device, not at interrupt delivery (inference); see touch-tpm-2026-11-03.md.
• The fragment is deleted (its text is in commit 9fa5a2d), per dts/diag/README.md.

Update: the wake trace, attempts -31 and -32 (2026-11-03, 11:00 PM ET)

Board DTB, interrupt mode, idle states on, plus trace_event=irq:irq_handler_entry,power:cpu_idle,timer:hrtimer_expire_entry (debug-tools.md, “the wake trace”). /init keeps these events for CPU 0 only.

• -31 (image 053a4bced558aa41): the default 1.4 MB per-CPU ring wrapped before the probe snapshot and lost the SCMI probe (replies=0). The boot itself stalled (SCMI 1.26 s to 2.30 s), 0 timeouts.
• -32 (image 5e565f64ce0e9d30, trace_buf_size=8M): the ring still wrapped (2061301 of 2260937 events kept) but kept all 78 SCMI replies. One slow reply: perf PROTOCOL_MESSAGE_ATTRIBUTES, 534.3 ms (request at 0.434542 s, reply interrupt at 0.968882 s); 0 timeouts. Cold start after an EC reset (Reset cause: sysjump hard efs).

What CPU 0 did during those 534 ms (attempt.sh trace wake, and the raw netlog lines):

• It entered idle state=1 (cpuidle index 1, the PSCI ret state cluster_c4) and left it 26694 times, once every ~20 µs, and took only 3 interrupts: 2 IPIs and 1 arch_timer. Per 100 ms bin: about 4960-5000 idle exits and 0-6 interrupts during the stall; in normal periods (0.0-0.4 s, 2.4-4.0 s) the exits roughly match the interrupts.
• The stall ended with the arch_timer interrupt at 0.968876 s; it expired four pm_suspend_timer_fn hrtimers, and 6 µs later the apss_cpucp_mbox interrupt ran and the reply was read.
• The same 20 µs loop with no interrupts covers 1.2 to 2.0 s, when no SCMI message was pending: the loop is not caused by the SCMI request (inference from one boot).

Reading (inference, not proven): CPU 0’s ret entries return early every 20 µs without the kernel taking any interrupt, so something wakes the core that the GIC does not deliver as an interrupt, or delivers only after another interrupt is acknowledged. With cpuidle.off=1 (-28, plain WFI) the replies take 0.1 ms. The other CPUs were not traced (the capture keeps CPU 0 only), so whether they were idle is unknown. Neither tree marks an idle state local-timer-stop, and both have the memory-mapped timer (arm,armv7-timer-mem), so a missing tick broadcast is not the explanation.

Update: precision on the -32 loop (2026-11-03, 11:08 PM ET)

• state=1 is not only the ret state: its enter callback is psci_enter_domain_idle_state (cpuidle-psci.c:271 at a804c57), which can escalate to a cluster or system domain state when CPU 0 is the last CPU of its domain. The cpu_idle event cannot tell which was requested; power:psci_domain_idle_enter/_exit (power.h:89, :96) can.
• Each cycle spends ~20 µs inside the state and ~0 µs outside: a PSCI call that returns without holding the state. Two readings fit: the firmware refuses (or immediately ends) the state while an SCMI command is in flight, or something wakes the core that the kernel does not see as an interrupt. Not decided.
• Two IPIs ran on CPU 0 mid-stall (0.502444 s, 0.538444 s) and the mailbox interrupt still did not follow them: SGIs reached CPU 0 while the SCMI reply’s SPI did not (or was not yet raised).
• No I2C or GENI events fall in 0.43-2.1 s on -32, so this boot cannot say whether other SPIs stall the same way.
• Next capture (one boot, no fragment): cpuidle/state*/{usage,rejected,time} per CPU (rejected exists, drivers/cpuidle/sysfs.c:266), pm_genpd_summary (CONFIG_DEBUG_FS=y, CONFIG_PM_GENERIC_DOMAINS=y in the build), the PSCI domain tracepoints, and per-CPU idle-exit counts over the SCMI window.

Update: idle counters, attempt -33 (2026-11-03, 11:21 PM ET)

Attempt 2026-11-03-33 (image fb75f74a4a79d29e, commit 68667c5): as -32 plus power:psci_domain_idle_enter/exit and the /init idle counters, read at the probe snapshot (cumulative since boot, about 95 s in by the genpd times). Cold start after an EC reset. SCMI stalled: firmware version at 1.03 s, a timeout at 4.11 s, perf domains at 4.11 s. The trace ring wrapped (2185344 of 3404814 events kept) and lost the SCMI lines, so the per-window counts are missing (idle-window=no SCMI window); the cumulative counters are complete.

cpuidle/state1 (cpu-sleep-0, the deepest cpuidle state, whose enter can request a domain state):

genpd idle_states (debugfs; Usage, Rejected, Above):

• power-domain-system S0 (domain_ss3): usage 2402, rejected 598159, above 599516.
• power-domain-cpu-cluster1 S1 (cl5): usage 252401, rejected 7, above 251683; cluster2 S1: 347327, 4, 346622.
• power-domain-cpu-cluster0 S0/S1: usage 1593/3565, rejected 74/18.

What the counters mean at a804c57: cpuidle’s rejected counts an enter that returned an error, and a PSCI domain state’s genpd rejected counts a CPU_SUSPEND for that composite state that returned an error (pm_genpd_inc_rejected(), cpuidle-psci.c:102; core.c:816). genpd’s above counts a stay shorter than the state’s latency plus residency (genpd_reflect_residency(), core.c:331).

• The firmware refused the system state about 598000 times. The refusals sit on the last CPU of clusters 1 and 2 (cpu4-11’s cpuidle rejections add up to 598174, matching system_pd’s 598159): each time one of them is the last CPU awake, it asks for ss3, PSCI returns an error at once, and it tries again.
• CPU 0 is not refused (37): it enters state1 and is woken, 218498 times, the 20 µs loop of -32.
• Clusters 1 and 2 entered cl5 about 250000 and 350000 times, almost all shorter than its residency.
• Reading (inference): a storm of refused system-state requests and early wakes runs while the SoC should be idle. A busy CPU (polling, -29) or no PSCI idle (-28) never gets all CPUs idle, so no storm, and the replies take 0.1 ms. That the storm delays the CPUCP’s completion interrupt is not shown. The counters are cumulative to about 95 s, so they do not prove the storm runs during the SCMI stall; -30 (no ss3) still stalled, with no counters to show what ran.

Update: precision on -33 (2026-11-03, 11:30 PM ET)

• The refused CPUs are not the stalled one. The mailbox interrupt lands on CPU 0, and CPU 0 is entered-and-woken (state1 usage 218498, rejected 37); the refusals are on clusters 1 and 2 asking for ss3. Two coupled loops, causality unknown either way: CPU 0 bouncing could be why the firmware refuses ss3 (cluster 0 is not down), or the refusals could be what wakes CPU 0.
• The counters are cumulative over about 95 s; -32 showed CPU 0’s loop at about 50000 exits/s during stalls against 1000-6000/s otherwise, so the cumulative numbers understate the rate inside a stall.
• Mode: psci: OSI mode supported and CPUidle PSCI: Initialized CPU PM domain topology using OSI mode (-33 dmesg).
• Ruled out as a link: the stock tree’s four qcom,cpucp-enabled flags sit in qcom,memlat monitors (the memory latency governor), not in an idle-state node. The stock psci node matches mainline’s (arm,psci-1.0, smc).
• Not yet tested: no composite requests at all (domain-idle-states removed from cluster_pd0-2 and system_pd, CPU ret kept). -27 removed only cl5, -30 only ss3.

Decisions (2026-11-03, 11:27 PM ET, owner, grilling round)

1. One more boot before choosing the fix: every PSCI domain idle state removed (dts/diag/hamoa-no-domain-idle.dtsi), CPU ret kept, interrupt mode, idle counters read early and at the snapshot.
2. The board DTS keeps arm,max-rx-timeout-ms = <1000> until the cause is known; polling stays a dts/diag fragment (it would hide the stall from every later trace).
3. Nothing goes to a mailing list until the cause is known and a fix is tested on mica; then one report to linux-arm-msm (SCMI and PSCI maintainers on Cc) with the numbers, the cause and a patch, after the owner reviews it.
4. Time box: this boot plus at most one follow-up, then the next handoff item (the never-bound devices, then the touchscreen).

Update: no PSCI domain idle states, attempt -34 (2026-11-03, 11:41 PM ET)

Attempt 2026-11-03-34 (image 30354ef5b4fcf76c, commit d458b08): kernel #17, interrupt mode, 1000 ms window, dts/diag/hamoa-no-domain-idle.dtsi (no domain-idle-states on cluster_pd0-2 or system_pd; the 12 per-CPU domains keep cluster_c4, the CPU ret state). Three cold starts of one image (EC G3 before each):

• No stall on any boot, and no PSCI refusal anywhere. With domain idle states present, 8 of 11 interrupt-mode cold boots stalled (-23, -24, -27 boots 1 and 3, -30, -31, -32, -33); three clean boots in a row by chance at that rate is about 6 % (inference). Removing cl5 alone (-27) or ss3 alone (-30) was not enough; removing every composite state is.
• CPU 0 entered state1 about 700 to 2700 times a second up to the early read, against about 50000 a second inside -32’s stall: no loop.
• power-domain-system’s default state still counts rejected (23796, 38188, 40386 at the snapshot). These are vetoes from the RPMh controller’s genpd notifier, which refuses GENPD_NOTIFY_PRE_OFF while it is busy or cannot flush (rpmh_rsc_pd_callback(), rpmh-rsc.c:949-951; counted at core.c:1014): kernel-side, no PSCI call.
• Reading (inference): the stall needs the composite cluster or system states that PSCI refuses in OSI mode (-33); why the firmware refuses them, and how that delays the CPUCP’s interrupt to CPU 0, is still open. What the change costs: the clusters and the SoC never power down when idle (more idle power; suspend is untested either way).

Update: ADR 0008 is not enough, attempts 2026-11-04-1 and -2 (2026-11-04, 12:26 AM ET)

Two more cold boots on the ADR 0008 board DTS (no PSCI domain idle states), kernel #18 then #19 (the second adds only kernel/patches/0008, a GENI I2C timing test for the touchscreen): both stalled.

• 2 of 6 stalled, so removing the domain idle states lowers the rate (8 of 11 before) but does not stop it. -34’s 3 of 3 was the lucky case. The 2026-11-04-1 stall was missed live: the touch monitor was not watching SCMI.
• The stall goes with CPU 0’s state1 loop, not with PSCI refusals. Stalled boots reach 130000-156000 CPU 0 state1 entries by the early read, against 4700-19000 on clean ones (the early read also comes later, because the stall delays the boot). Every CPU’s rejected is 0, and no composite state exists. -33’s refusal storm ran alongside the stall, not as its cause (inference).
• What is left fits all the data (inference): CPU 0 enters the PSCI ret state (cluster_c4, 0x4), wakes within ~20 µs again and again, and the SCMI completion interrupt is not taken until another interrupt runs (-32). Plain WFI (cpuidle.off=1, -28) and a busy CPU (polling, -29) never stalled (0 of 3 each).
• ADR 0008 gets a dated update; the board DTS change stays until the owner decides.

Update: precision and two dead ends for the next test (2026-11-04, 12:31 AM ET)

• The rate claim was too strong. 2 of 6 stalled boots without domain idle states, against 8 of 11 with them, is not a significant difference at these sample sizes (one-sided p about 0.14, inference). Read it as “may reduce the rate; the sample cannot tell”, not as “cut it to 2 of 6”.
• The window held on the 2-timeout boot. 2026-11-04-1 (2 timeouts, max 1019.2 ms) still came up fully: firmware 0x20000, quirk_perf_level_get_fc_force, cpufreq on all three clusters (netlog, boot 2 of 3 in 2026-11-04-netlog.txt).
• When the loop starts (-32): the request left at 0.434542 s and CPU 0’s first loop exit came at 0.434606 s, 64 µs later, about when the reply’s interrupt would be asserted (polling replies take ~0.1 ms, -29). Mid-loop CPU 0 took two SGIs (IPIs) but not the SPI, and a PPI (arch_timer) released it. Reading (inference): a pending SPI for CPU 0 wakes the core from ret but is not delivered to its CPU interface; the core re-enters, is woken again every ~20 µs, until a PPI runs. It also fits the 1.2-2.0 s loop with no SCMI pending (another SPI) and -30’s GENI I2C -110 timeouts (that controller’s interrupt is an SPI on CPU 0 too).
• irqaffinity= cannot move SPI 28 off CPU 0: irq_affinity_setup() always adds the boot CPU to the mask (kernel/irq/irqdesc.c:36 at a804c57).
• CPU 0 alone cannot lose ret by DT: with no state for one CPU, psci_idle_init_cpu() returns -ENODEV (cpuidle-psci.c:399-401) and psci_cpuidle_probe() fails the whole driver (:437-447), which is cpuidle.off=1 for every CPU. Isolating CPU 0 or the interrupt needs a kernel patch.

Update: the stall follows the interrupt’s CPU, attempt 2026-11-04-3 (2026-11-04, 12:44 AM ET)

Attempt 2026-11-04-3: kernel #20 (kernel/patches/0009, commit cee735d), the ADR 0008 board DTS, interrupt mode, 1000 ms window. The patch calls irq_set_affinity(irq, cpumask_of(1)) in the CPUCP mailbox probe; dmesg shows qcom_cpucp_mbox 17430000.mailbox: gbook: irq 15 affinity to CPU 1: 0 at 0.226519 s. One cold start (EC G3 first).

• All 78 mailbox interrupts ran on CPU 1: irq-counts= 15: 0 78 0 0 0 0 0 0 0 0 0 0 GICv3 60 Level apss_cpucp_mbox.
• SCMI stalled: max reply 872.0 ms, 0 timeouts, cpufreq on all three clusters.
• The loop moved with it. At the early read (9.63 s), CPU 1’s state1 count was 134862. CPU 0 had 6131 and every other CPU had 79 to 471.
• So CPU 0 is not special. The core that takes the reply’s SPI loops in and out of PSCI ret without taking it, wherever the SPI is routed (one stalled boot; it moved exactly as the reading predicted). With -28 (plain WFI, 0 of 3 stalled) and -29 (polling, 0 of 3), the mechanism reads (inference): on mica, a core in the PSCI ret state (cluster_c4, arm,psci-suspend-param 0x4) is woken by a pending SPI but does not take it until a PPI runs. That fits the other SPIs seen stalling too (-30’s GENI I2C timeouts, -32’s 1.2-2.0 s loop with no SCMI pending).
• kernel/patches/0009 stays until the owner decides the fix, then goes (a diagnostic, ADR 0007).

Update: ADR 0009 confirmed, attempt 2026-11-04-4 (2026-11-04, 12:57 AM ET)

Attempt 2026-11-04-4: kernel #21 (patches 0001-0007, no diagnostics), the board DTS with ADR 0009 (cluster_c4 deleted from cpu_pd0-11; cluster and system states as stock), interrupt mode, 1000 ms window. Image 161b32cde0296c3c, commit 1f67420. One cold start.

• No cpuidle driver: every CPU’s cpuidle-early= and cpuidle= line is empty (/sys/devices/system/cpu/cpuN/ cpuidle/ has no states), so cores idle in WFI. The PSCI domain driver still reports CPUidle PSCI: Initialized CPU PM domain topology using OSI mode. That is cpuidle-psci-domain, not the cpuidle driver.
• SCMI clean: 78 replies, max 0.1 ms, 0 timeouts; the SCMI probe ran 0.376-0.391 s. Firmware 0x20000, FC quirk, cpufreq on all three clusters. All 78 mailbox interrupts were on CPU 0.
• With -28 (cpuidle.off=1, 0 of 3), plain WFI is now 4 of 4 cold boots without a stall, every reply within 0.1 ms.

Update: the counts from attempt.sh tally (2026-11-04, 2:31 AM ET)

dumps/firstboot/attempts.tsv now carries machine-read boot health (retro 6), backfilled for -20 to 2026-11-04-4. The counts above were hand-made; the tally reads:

• Domain idle states present, interrupt mode (-23 to -27, -30 to -33): 7 stalled of 10 known boots, plus -31 unknown (its trace ring lost the SCMI lines; its stall is from dmesg timing only). Against no PSCI cpuidle (-28, 2026-11-04-4): 0 of 4, one-sided Fisher p = 0.035. “8 of 11” above counted -31 as a stall.
• ADR 0008’s DT (-34, -35, 2026-11-04-1 to -3): 3 of 7. Against no PSCI cpuidle: 0 of 4, p = 0.21, not significant on its own. ADR 0009 rests on the mechanism (the stall followed the SPI to CPU 1, 2026-11-04-3) together with the first comparison, not on the boot count against ADR 0008.
• Polling (-29): 0 of 3. Commands: scripts/attempt.sh tally '^2026-11-03-(2[3-7]|3[0-3])$' '^2026-11-03-28$|^2026-11-04-4$' and scripts/attempt.sh tally '^2026-11-03-3[45]$|^2026-11-04-[123]$' '^2026-11-03-28$|^2026-11-04-4$'.

Update: the tally at ten boots without PSCI cpuidle (2026-11-04, 4:40 AM ET)

The suspend-test boots (2026-11-04-5 to -10, suspend-2026-11-04.md) all ran the no-PSCI-cpuidle board DT and probed SCMI before suspending. Every one was clean at 0.1 ms. With '^2026-11-03-28$|^2026-11-04-([4-9]|10)$' as group 2 in the two commands above, no PSCI cpuidle is 0 of 10 known boots. Against domain states present (7 of 10), the one-sided Fisher p = 0.0015. Against ADR 0008’s DT (3 of 7), p = 0.051, just above 0.05.