Rejected Is Not Dead: The Liveness Probe That Killed Healthy Tunnels

The heal tick that murdered a working tunnel

I had a reverse SSH tunnel that was fine. Port 2222 was bound on the relay. A cold global login landed on my laptop. Then my own self-healing daemon, tether-heal.sh, ran its sixty-second tick, decided the tunnel was STALE, killed the session, and kicked the client. The tunnel I was watching die was the one keeping me reachable.

The symptom showed up the afternoon I moved the relay off a residential Verizon box and onto an AWS VPS at 98.80.177.159. The migration itself was the right move. The residential relay had been drifting WAN IPs and dropping UPnP forwards for weeks. What the migration exposed was not one bug in heal. It was four, stacked, all latent until the login identity on the far side stopped matching the identity heal assumed.

The conceptual core is the second of those four, and it is the one I keep finding in other systems: a liveness probe that treats "I was refused" as "nothing is there." Rejected is not dead. A refusal is a response. A response means the path carried bytes end to end.

What the probe was actually asking

Heal's job is simple on paper. Every minute it checks three things: is the reverse tunnel bound on the relay, is the data channel still live end to end, and does public DNS still point at the right relay IP. If the bind is missing, kick the client. If the bind is present but the channel is dead, kill the stale sshd-session and kick. If DNS drifted, upsert.

The E2E check was the dangerous one. From the relay's loopback, heal would SSH back through the forwarded port to the roaming machine and look for a magic string:

  # (b) E2E probe: ssh through the tunnel from the relay's loopback to m4.
  # Uses BatchMode (no prompts) + short timeout. We do NOT trust ssh's known_hosts
  # on the relay for 127.0.0.1:2222 since multiple respawns rotate host keys.
  local probe_out
  probe_out="$(relay_ssh "ssh -F /dev/null -o BatchMode=yes -o ConnectTimeout=${PROBE_TIMEOUT} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -p ${REMOTE_PORT} ${TUNNEL_USER}@127.0.0.1 'echo TUNNEL_E2E_OK' 2>&1" 2>/dev/null)"
  if printf '%s' "$probe_out" | grep -q TUNNEL_E2E_OK; then
    log "tunnel: healthy (bind + E2E ok)"
    return 0
  fi

  log "tunnel: STALE — port bound but E2E failed (${probe_out//$'\n'/ | }) — killing stale sshd-session + kicking client"
  # Kill any nick-owned sshd-session children on the relay. Cannot touch the
  # root-owned [priv] half over ssh, but killing the child closes the channel
  # and root reaps the priv side automatically.
  relay_ssh "pkill -u ${TUNNEL_USER} -f 'sshd-session: ${TUNNEL_USER}\$' 2>/dev/null; pkill -u ${TUNNEL_USER} -f 'sshd-session: ${TUNNEL_USER}@notty' 2>/dev/null; true" 2>/dev/null || true
  sleep 2
  launchctl kickstart -k "system/${LAUNCHD_LABEL}" 2>>"$LOG_FILE" || true
  sleep 6
  probe_out="$(relay_ssh "ssh -F /dev/null -o BatchMode=yes -o ConnectTimeout=${PROBE_TIMEOUT} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -p ${REMOTE_PORT} ${TUNNEL_USER}@127.0.0.1 'echo TUNNEL_E2E_OK' 2>&1" 2>/dev/null)"
  if printf '%s' "$probe_out" | grep -q TUNNEL_E2E_OK; then
    log "tunnel: E2E recovered after stale-kill + kickstart"
  else
    log "tunnel: E2E STILL DOWN after stale-kill + kickstart (${probe_out//$'\n'/ | })"
  fi
}

That looks reasonable until you watch what BatchMode actually returns. BatchMode means "never prompt." No password dialog, no keyboard-interactive dance, no agent unlock. If the inner SSH cannot authenticate non-interactively, OpenSSH prints something like nick@127.0.0.1: Permission denied (publickey,password,keyboard-interactive). and exits non-zero. Heal grepped only for TUNNEL_E2E_OK. Anything else, including a clean auth rejection from a live sshd, became STALE.

I reproduced the exact probe against the live tunnel after the VPS cutover:

nick@127.0.0.1: Permission denied (publickey,password,keyboard-interactive).

The tunnel had carried the TCP connection from the relay loopback, through the reverse forward, onto my laptop's sshd. My laptop's sshd had answered with its banner and its auth methods. The only thing that failed was authentication. Heal treated that answer as proof of death and scheduled a kill.

Why Permission denied is a positive signal

SSH has two failure classes that matter for liveness, and they are not the same class.

Transport failure means the bytes never reached an sshd that could speak the protocol. Connection refused. Connection reset. ConnectTimeout. EOF before the banner. Those are the signals that say "this port is bound to nothing useful" or "the session that held the forward is a zombie." Those are the cases where killing the session and kickstarting the client is the right move.

Auth failure means the bytes did reach an sshd. The remote end chose a user, offered publickey/password/keyboard-interactive, evaluated the credentials it got (or did not get, under BatchMode), and refused login. That refusal is an application-layer response on top of a working transport. If you are only asking "is the data channel alive," the answer is yes. You do not need a successful shell. You need proof that the far sshd spoke.

BatchMode makes this distinction load-bearing. Heal deliberately cannot, and should not, hold credentials for an interactive login to my laptop from the relay. The probe is not a login test. It is a path test. Demanding TUNNEL_E2E_OK turns a path test into an auth test, and then punishes the path for failing an auth it was never set up to pass.

The fix renames the question the probe asks:

  # (b) E2E probe: ssh through the tunnel from the relay's loopback back to m4's
  # sshd, logging in as M4_USER (the roaming laptop's user — NOT the relay user).
  # heal has no way to actually AUTHENTICATE to m4 here, and it does not need to:
  # any real SSH response from m4's sshd — including "Permission denied" — proves
  # the tunnel data channel is alive end to end. Only a transport-level failure
  # (connection refused / reset / timeout / EOF before banner) means the port is
  # bound to a dead session. Treating auth-rejection as STALE was the bug that
  # made heal kill a perfectly healthy tunnel after the VPS-relay migration.
  tunnel_e2e_healthy() {
    local out
    out="$(relay_ssh "ssh -F /dev/null -o BatchMode=yes -o ConnectTimeout=${PROBE_TIMEOUT} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -p ${REMOTE_PORT} ${M4_USER}@127.0.0.1 'echo TUNNEL_E2E_OK' 2>&1" 2>/dev/null)"
    printf '%s' "$out"   # surface for logging
    # healthy signals: explicit OK, or m4's sshd answering with an auth rejection
    printf '%s' "$out" | grep -qiE 'TUNNEL_E2E_OK|Permission denied|Authentication failed|publickey|password' && return 0
    return 1
  }

  local probe_out
  probe_out="$(tunnel_e2e_healthy)" && { log "tunnel: healthy (bind + E2E ok)"; return 0; }

  log "tunnel: STALE — port bound but E2E dead (${probe_out//$'\n'/ | }) — killing stale sshd-session + kicking client"
  # Kill any RELAY_USER-owned sshd-session children on the relay. Cannot touch the
  # root-owned [priv] half over ssh, but killing the child closes the channel
  # and root reaps the priv side automatically.
  relay_ssh "pkill -u ${RELAY_USER} -f 'sshd-session: ${RELAY_USER}\$' 2>/dev/null; pkill -u ${RELAY_USER} -f 'sshd-session: ${RELAY_USER}@notty' 2>/dev/null; true" 2>/dev/null || true
  sleep 2
  launchctl kickstart -k "system/${LAUNCHD_LABEL}" 2>>"$LOG_FILE" || true
  sleep 6
  probe_out="$(tunnel_e2e_healthy)" \
    && log "tunnel: E2E recovered after stale-kill + kickstart" \
    || log "tunnel: E2E STILL DOWN after stale-kill + kickstart (${probe_out//$'\n'/ | })"
}

After the change, the same live probe still printed Permission denied, and the new classifier marked it HEALTHY. The tunnel survived the next heal tick.

Three identities, one variable

The auth-rejection bug did not stand alone. It was armed by identity conflation.

On the old self-hosted relay, one human name had covered three roles by accident. The macOS user that owned the tunnel key was nick. The login on the relay was nick. The login on the roaming laptop was nick. One TUNNEL_USER variable looked fine.

The AWS box is different. The relay login is ubuntu. The key still lives under my local nick home directory. The inner probe still has to target nick on the laptop, because that is the only account whose sshd is listening behind the forward. Heal was doing all three jobs with one name:

1. drop privileges locally so ssh can read ~/.ssh
2. log into the relay
3. log into the laptop through the tunnel

After the cutover, step 2 correctly wanted ubuntu, so TETHER_TUNNEL_USER=ubuntu went into .env. The inner probe then became ssh ubuntu@127.0.0.1:2222. My laptop has no ubuntu user. sshd answered Permission denied. Heal called that stale and ran pkill -u ubuntu on the relay, which is exactly the user owning the healthy forward.

The durable split is three names with three jobs:

#   LOCAL_USER  — the macOS user that OWNS the tunnel key
#   RELAY_USER  — the login on the relay box (ubuntu on AWS, nick on self-hosted)
#   M4_USER     — the login on the roaming machine itself
LOCAL_USER="${TETHER_LOCAL_USER:-nick}"
RELAY_USER="${TETHER_TUNNEL_USER:-nick}"
M4_USER="${TETHER_M4_USER:-nick}"

relay_ssh drops to LOCAL_USER and connects as RELAY_USER@${RELAY_HOST}. The E2E probe targets ${M4_USER}@127.0.0.1. The stale-kill pkill targets RELAY_USER. Once those are separate, the false STALE path loses both its wrong user and its wrong interpretation of the rejection.

Parse-time config under launchd

Even with the probe fixed, DNS kept flipping. I would set m4.hack.ski to 98.80.177.159, watch four public resolvers agree, then seven minutes later find it back on the dead residential IP 173.68.237.146. The heal log told on itself:

dns: m4.hack.ski -> 98.80.177.159, expected 173.68.237.146 — upserting

.env already said TETHER_RELAY="98.80.177.159". The running daemon still expected the old home hostname. The bug was ordering, not a missing file.

Near the top of the script, config was bound once at parse time:

RELAY_HOST="${TETHER_RELAY:-home.hack.ski}"
REMOTE_PORT="${TETHER_REMOTE_PORT:-2222}"

load_env() ran later and sourced .env. In an interactive shell that already had exported overrides, or in a one-shot sudo invocation where I had just sourced the file into the parent environment, the parse-time defaults looked correct enough to hide the trap. Under launchd they did not. Launchd starts heal with a minimal process environment. There is no TETHER_RELAY in that environment when bash parses the script. So RELAY_HOST freezes to home.hack.ski before .env is ever opened. Sourcing .env sets TETHER_RELAY in the shell, but nothing reassigns RELAY_HOST. Every tick then does dig home.hack.ski, treats that answer as the expected relay IP, and "heals" public DNS back to the zombie residential box.

Interactive shells forgive this because humans often export the same variables before running a script, or because they run the script after manually sourcing .env in the same shell. launchd does neither. The process environment is the only environment at parse time, and it is empty of project config by design.

The fix is mechanical and total: after sourcing .env, re-derive every config value the rest of the script will read.

load_env() {
  if [ -f "$ENV_FILE" ]; then
    set -a
    # shellcheck source=/dev/null
    source "$ENV_FILE"
    set +a
  fi
  # Re-derive config now that .env is in scope (overrides frozen parse-time defaults)
  RELAY_HOST="${TETHER_RELAY:-home.hack.ski}"
  REMOTE_PORT="${TETHER_REMOTE_PORT:-2222}"
  LOCAL_USER="${TETHER_LOCAL_USER:-nick}"
  RELAY_USER="${TETHER_TUNNEL_USER:-nick}"
  M4_USER="${TETHER_M4_USER:-nick}"
  ...
}

Because launchd re-execs the script from disk each tick, that edit self-applied within sixty seconds. No plist rewrite required for the config-ordering fix to take effect.

dig on a literal IP returns empty

The fourth bug was smaller and just as quiet. With RELAY_HOST finally equal to 98.80.177.159, check_dns still would not hold the record:

relay_ip="$(dig +short "$RELAY_HOST" @1.1.1.1 2>/dev/null | tail -1)"
if [ -z "$relay_ip" ]; then
  log "dns: cannot resolve relay ${RELAY_HOST} — skipping DNS sync"
  return 0
fi

dig +short 98.80.177.159 returns empty. A literal address is not a name. Heal logged "cannot resolve relay" and skipped the upsert. That left the field open for any competing writer, including the old home box's five-minute cron that auto-detected its own public IP and pushed m4.hack.ski back to 173.68.237.146.

The repair is a one-branch special case: if RELAY_HOST already matches an IPv4 shape, use it directly as the target. Only call dig for hostnames. After that change, heal stopped skipping and started winning. The log flipped from skip to assert:

tunnel: healthy (bind + E2E ok)
dns: m4.hack.ski -> 173.68.237.146, expected 98.80.177.159 — upserting

A seven-minute monitor across the old home-cron window then held m4.hack.ski → 98.80.177.159 with zero reverts. Cold logins from a fresh known_hosts file landed on m4-max-959.local. The client launchd plist and the heal plist both pointed at the AWS relay. The residential box was no longer in the path.

The durable lesson

A self-healing system is only as honest as the signals it treats as failure. Heal's E2E probe had been answering the wrong question with the wrong identity, under a config model that only worked in interactive shells, while its DNS path went silent on the exact form a VPS address takes. Each bug was latent on the old single-user residential relay. The migration did not create them. It removed the coincidences that had been hiding them.

The line I keep from the session is the one in the probe comment: heal has no way to authenticate to the laptop here, and it does not need to. Any real SSH response, including Permission denied, proves the tunnel data channel is alive. Only a transport-level failure means the port is bound to a dead session. If your health check cannot tell those two outcomes apart, it will eventually destroy the thing it was written to protect.