Posture — tech-neck coaching on the glasses, + smart-insole second modality

Requested by Sandon 2026-07-27: "use the glasses frame being parallel to the floor vs. tilted down as an indication of poor posture, and build in a posture-correcting app service" + "if bluetooth shoe inserts to detect weight-distribution variations via pressure sensors and correlate w/ posture would help, can also design those."

Two complementary sensors, one posture service:

Glasses (top-down): frame pitch → head/neck angle → tech-neck (built now, on the existing Mentra glasses app).
Smart insoles (bottom-up): foot pressure distribution → stance/lean/balance → standing & seated postural load (a new hardware design; specced here, not built).

Fused they read posture from both ends of the spine; either works alone.

1. Glasses posture add-on — BUILT (OFF by default)

Where: apps/glasses/src/posture.ts (+ posture.test.ts), wired into apps/glasses/src/session.ts (attachJqSession). Product name stays "Downstream Glasses"; JQ is the voice.

The signal

The glasses frame's pitch. Frame parallel to the floor (gaze level) = good; frame tilted down for a sustained stretch — head dropped over a phone or desk — is the tech-neck slouch we coach out of. Head tilt is a proxy for spine posture, not spine posture itself (you can look down with a long neck), so the whole design leans conservative: per-session calibration, a sustained-hold requirement, hysteresis, and a long cooldown — it forgives brief glances down and speaks rarely.

How it works

Calibrate a per-session upright baseline from the median of the first ~8 s of pitch readings (robust to a stray glance).
Detect a slouch when pitch stays ≥ thresholdDeg (18° default) below baseline for ≥ sustainMs (10 s default).
Cue once — a gentle, rotating spoken line ("Ease your head up a touch — let your eyes do the looking down.") via plain TTS (a system receipt, like the lever/aisle acks; not worth a per-event cloned-voice render).
Hysteresis + cooldown: a slouch is entered at 18° but only cleared within recoverDeg (8°) of baseline — no flapping at the boundary — and no two cues fire within cooldownMs (90 s default), even across separate slouches.
Recovery is tracked (and reported) only for a slouch that was actually cued.
Quiet hours (POSTURE_QUIET_START/END, wraps midnight) suppress cues at rest.

Each cue is logged best-effort to /api/glasses/events (kind:'posture') so the wearer can audit it on /dashboard/glasses, same as Cairn arrivals. Privacy contract holds: we report what the glasses said, never what the wearer said, and posture is derived on-device — no raw head-motion stream leaves the session.

Built to be tuned on hardware (the seam)

@mentra/sdk isn't installed in this repo (dist builds in CI), so the SDK's exact head-position shape and pitch sign are unconfirmed. The design isolates that risk:

createPostureMonitor — the pure detection state machine — is fully unit-tested (calibration, sustain, cooldown, recovery, hysteresis) with no SDK and no clock of its own (every sample carries its own tMs).
getHeadPitchStream is the one place that touches the SDK. It degrades to a no-op if the device exposes no head-position stream (the app just runs posture-inert, never breaks), and normalizeDownTilt handles the likely payload shapes (continuous pitch/pitchDeg/euler.pitch, or discrete up/level/down).
On-device correction without a code change: POSTURE_INVERT_PITCH=1 flips the sign if up/down come out reversed.
Desk/CI validation without hardware: POSTURE_SIM_SEQUENCE=0,0,0,25,25,25,25,25,25,25,3,3 (emitted one per POSTURE_SIM_INTERVAL_MS) drives the entire calibrate→slouch→cue→recover loop. Setting it implies POSTURE_ENABLED.

Camera-based pitch — no IMU required (BUILT: geometry; perception = seam)

The seam above assumes an IMU/head-position stream that Mentra Live may not have (it's camera-first). So we can recover pitch a second way — from the camera, which is definitely there and is already used for AisleAsk scans. apps/glasses/src/posture-vision.ts:

The geometry (pure, unit-tested, no ML). The camera is rigidly fixed to the frame, so the camera's pitch is the frame's pitch. Any surface we can assume is horizontal — floor, table, counter, desk — has receding parallel edges whose vanishing point lies on the image horizon. For a pinhole camera with vertical focal length fy (px) and principal row cy, a downward pitch θ lifts the horizon by exactly fy·tan(θ):

θ_down = atan( (cy − v_horizon) / fy )

posture-vision.ts implements the whole chain — vanishingPoint / bestFitVanishingPoint (least-squares over ≥2 edges), pitchFromVanishingPoint, and poseFromHorizonLine (full pitch + roll from two horizontal directions). It's proven by a round-trip test: a synthetic camera projects known horizontal world lines at a chosen pitch/roll and the estimator recovers them to 1e-9 (posture-vision.test.ts). Intrinsics come from intrinsicsFromFov(w, h, hfovDeg) when a full calibration isn't available — and FOV error largely cancels because posture keys off the relative change from a calibrated-upright baseline, not an absolute angle.

The perception step (photo → reference edge segments) is a PerceptionProvider seam. Two ways to fill it:

Classical — line detection (LSD) + vanishing-point clustering (Manhattan-world). No per-photo cost; more code, and heavier without OpenCV in Node.
Vision-LLM — reuse the AisleAsk vision path: ask the model for the pixel endpoints of a horizontal surface's edges; our tested geometry turns those points into the angle (clean split: model for perception, our code for math). Costs per call. ← WIRED (BUILT).

End-to-end, WIRED (user-triggered only):

Say "check my posture" / "how's my posture" / "am I slouching" (intent in apps/glasses/src/intent.ts, gated on POSTURE_ENABLED).
The glasses photograph the scene → POST /api/glasses/posture/reference-edges (apps/backend/src/services/posture-vision.ts, gpt-4o-mini vision, mirrors scanSectionSign) returns the normalized edges of a horizontal surface.
The glasses scale to pixels + run the tested geometry locally (session.ts checkPostureFromCamera) → speak a gentle line ("Your head's about 22 degrees down — ease your chin up"). Logged as kind:'posture' (source:'camera'). Camera model via POSTURE_CAM_HFOV_DEG/POSTURE_CAM_ASPECT.
Degrades: no horizontal surface found → "I couldn't find a flat surface to gauge from — face a table, counter, or the floor"; no OPENAI_API_KEY → 400.
kind:'posture' DB CHECK: migration 595 widens glasses_spoken_events.kind to include cairn + posture (491's CHECK omitted both; reportSpoken is fire-and-forget so a rejected insert failed silently). Apply 595 before relying on the audit log.

Honest role: calibration + fallback, not continuous. A vision call per posture sample is too slow/costly/privacy-heavy for real-time monitoring — so it's user-triggered (no autonomous spend), for (1) calibration — one "sit up straight, look ahead" check learns the frame's mounting offset + validates the pitch sign (improves the IMU path too), and (2) an IMU-absent spot-check. The estimate feeds the same createPostureMonitor (its downTiltDeg is exactly this pitch). Precision caveat: vision LLMs aren't pixel-exact, so treat the angle as approximate.

UX — working backward from the experience

Designed surface-first (the ideal experience, then the build behind it). The five moments:

Discover — a public entry on /features under "On your glasses (Mentra Live) — hands-free JQ": what it is, that it needs no extra hardware, and the privacy line (posture is derived on the spot; only the reminder is logged). DONE.
Understand + enable — a Posture coaching panel on /dashboard/glasses: plain explainer, the "how's my posture" command, the privacy contract, and how to turn it on (+ a link here). DONE.
Ask — say "how's my posture" → a spoken read from a photo. DONE (the camera-pitch flow above).
Ambient watch (opt-in) — occasional auto-captured photos analyzed for tilt, a gentle cue only on a sustained slouch. Photo analyzed then discarded (angle only); hard-gated + disclosed. DONE — see "Ambient posture watch" below.
Reflect + control — posture cues appear in the /dashboard/glasses log (kind posture, teal chip). DONE for the log. Backlog: per-user settings (sensitivity, quiet hours, enable) instead of server env; good-posture streaks / daily upright-time (the monitor already accumulates goodMs/slouchMs/cues — it just isn't persisted yet); a sensitivity slider mapping to thresholdDeg/sustainMs.

Why the deeper build waits: posture detection is still untested on real hardware, so persisting a stats/streaks pipeline now would be building on an unproven signal. Ship the surfaces + discoverability + the honest privacy story first; add per-user settings + streaks once the on-device signal is validated.

Ambient posture watch (opt-in auto-capture) — BUILT

The middle path between "only when you ask" and "continuous" (which is too costly/privacy-heavy): when posture watch is on, the glasses take an occasional, jittered photo, run the same camera-pitch pipeline, and — only after a sustained slouch (≥2 consecutive slouch samples) — speak one gentle cue. No user prompt each time. attachPostureWatch in apps/glasses/src/session.ts (reuses estimateCameraDownTilt + pickCue).

Hard-gated, because it's autonomous camera capture + spend:

Its own opt-in, stricter than the ask-anytime path: POSTURE_WATCH_ENABLED (OFF by default) and POSTURE_ENABLED. Only attaches when a wearer is resolvable (attribution).
Jittered cadence — POSTURE_WATCH_INTERVAL_SEC (default 480s) ± 50%, so it never snaps on a predictable clock. First capture is one interval after connect, never immediately.
Firm daily cap (POSTURE_WATCH_MAX_PER_DAY, default 24) + a 10-min cue cooldown + quiet hours (checked before any photo, so quiet-time costs nothing). Absolute slouch threshold POSTURE_WATCH_THRESHOLD_DEG (default 20° — camera pitch is vs detected horizontal surfaces, so no per-session baseline needed).
Photo is analyzed then discarded — only the derived tilt (and the fact a cue fired, source:'camera_watch') is kept, never the image or a movement stream. Same privacy contract as the ask path.
Disclosed on /dashboard/glasses so the wearer knows occasional photos are being taken and why.

Each auto-capture is a vision call, so the cadence is deliberately sparse and the daily cap firm. Stays OFF until the wearer explicitly turns it on.

Feedback loop / calibration (the wearer dials it in)

The sensor measures an angle; the wearer knows how their alignment actually feels. This loop closes that gap and is the tool for tuning an as-yet-unvalidated signal — the more you report, the better it fits.

Report a felt alignment grade (0-100, or a letter that maps to a percent), paired with the tilt measured at that moment. Two surfaces, both built: a /dashboard/glasses slider ("Dial it in"), and by voice on the glasses — "grade my posture 70" / "my posture feels like 80 percent" / "rate my posture a B" (parsePostureGrade → reportPostureGrade, which takes a fresh reading so the felt grade pairs with a measured tilt).
Fit — services/posture-feedback-core.ts (pure, jest-tested): deriveCalibration(pairs) anchors uprightDeg (the tilt you feel is ~perfect → grade 100) from your high-grade reports and poorDeg (the tilt you feel is bad) from your low-grade ones; gradeFromTilt is the personalized 0-100 grade; cueThresholdDeg is where a sustained slouch becomes cue-worthy. Confidence scales with how many pairs you've given.
Close the loop (BUILT) — once calibrated, "check my posture" speaks your grade ("you're at about 82%") instead of raw degrees; checkPostureFromCamera fetches the personalized grade from GET /api/glasses/posture/grade?tilt= (the backend is the single source of truth — gradeFromTilt is never duplicated on-glasses). The voice report ack also reads back the wearer's calibrated grade for the tilt they just reported ("by your calibration that reads about 78%"). Two people at the same 15° get different grades if one rests more upright than the other.
Storage — migration 596: posture_feedback (each report) + posture_calibration (one derived row/user, refit on every report). Backend-only (service role), scoped to the wearer. Endpoints: POST /api/glasses/posture/feedback, GET /api/glasses/posture/calibration, GET /api/glasses/posture/grade?tilt=.

This is the calibration mechanism the earlier backlog deferred to "after on-device validation" — it is how that validation happens, wearer-in-the-loop.

Config (env, all optional)

POSTURE_ENABLED, POSTURE_QUIET_START/END, POSTURE_INVERT_PITCH, POSTURE_SIM_SEQUENCE, POSTURE_SIM_INTERVAL_MS. Detection thresholds are DEFAULT_POSTURE_CONFIG in posture.ts (override in code; expose via the dashboard later if wearers want sensitivity control).

On-device validation checklist (when the glasses are in hand)

Set POSTURE_ENABLED=1, wear the glasses, hold a level gaze ~10 s (baseline calibrates).
Drop your head to look at a phone and hold ~10 s → expect one spoken cue.
Look level ~3 s → the slouch clears silently; a fresh sustained slouch after the cooldown cues again.
If a look up triggers the cue (sign reversed), set POSTURE_INVERT_PITCH=1.
If the device logs [posture] no head-position stream on this device, Mentra Live doesn't expose head pitch — fall back to the phone-IMU path (see Open questions) or the insole modality.

Phase 2 (not built)

Cloned-voice cues (fetch a few posture lines once per session, play those instead of TTS).
Good-posture streaks / daily upright-time surfaced on /dashboard/glasses (the monitor already accumulates goodMs/slouchMs/cues).
A dashboard sensitivity slider mapping to thresholdDeg/sustainMs.

2. Smart insoles — HARDWARE DESIGN (specced, not built)

A new hardware bet in the family of the Mentra/AisleAsk owned-hardware plays — not an off-the-shelf SDK add-on. Complementary to the glasses: glasses read the top of the spine (head/neck, tech-neck); insoles read the base (stance, lean, weight-shift), which the glasses can't see.

What it senses

A BLE pressure-sensing insole pair — an array of force sensors (FSR or capacitive) under heel, lateral/medial forefoot, and toe, per foot. From the pressure map you derive:

Anterior/posterior lean — weight forward on the toes vs. back on the heels (forward lean pairs with the tech-neck the glasses catch — a cross-check).
Left/right imbalance — chronically loading one leg (hip/pelvis asymmetry, "hip-hike" standing slouch).
Center-of-pressure sway — restlessness / fatigue over a standing session.
Sit vs. stand vs. walk context — so cues fire in the right posture (a forward COP means something different seated vs. standing).

Architecture (mirrors the existing service)

Insole FSR array → tiny MCU (nRF52-class, BLE) → phone/hub → the same posture service

Insole: flexible PCB or discrete FSRs on a lasered insole, coin-cell or micro-LiPo, an nRF52-class BLE SoC. Sample a few Hz (posture is slow — no high-rate IMU needed), stream a compact pressure vector.
Hub: the phone (BLE central) or, for a wearer already on glasses, the glasses' phone companion — so one app fuses both streams.
Service: identical shape to the glasses monitor — a pure detector (calibrate a per-session neutral stance → sustained-deviation + hysteresis + cooldown) feeding the same cue/report path. Reuse createPostureMonitor's structure with a stance-deviation metric instead of pitch.

Why design our own vs. buying

Smart insoles are a proven category (gait labs, running-form insoles, fall-risk research) to reference for sensor placement and BLE profile — but the off-the-shelf ones are closed and export raw gait data, not a posture-coaching signal in our voice, on our consent terms, fused with the glasses. Owning the insole = owning the second modality end to end, same rationale as the Mentra/AisleAsk hardware bets. Bigger effort than the glasses add-on (real hardware: PCB, firmware, enclosure, battery, comfort/durability, BLE pairing UX).

Fusion (the payoff)

Glasses + insoles together read posture from both ends: head-drop (top) and the forward-lean/weight-shift that drives it (bottom). Fused, a cue can be smarter and rarer — e.g. only nudge on head-drop when the insoles also show a forward-collapsed stance (true slouch), and stay silent when you're just reading something low with a sound base. Either sensor is useful alone; the pair is a far fuller picture than either.

Posture coaching is body-sensing, so it stays opt-in, calm, and non-punitive — a gentle occasional cue, never a scold or a streak-shame. It's health-adjacent, not medical: coach a habit, don't diagnose a spine. Posture is derived on-device/locally; only the fact that a cue was given (not a raw motion/pressure stream) is logged for the wearer's own audit. If this ever moves toward clinical framing, it goes behind the same counsel gate as the Speech Mirror.

Open questions / next

Does Mentra Live expose head pitch? It's camera-first; the SDK head-position stream is assumed, not confirmed. The seam degrades gracefully if absent — and if it IS missing, the camera-based pitch above (posture-vision.ts) is the real fallback (the camera is always there), needing only a perception provider wired in. The insole modality remains the third path.
Expose sensitivity + quiet-hours on /dashboard/glasses.
Insole hardware: build only on Sandon's go — it's a hardware-cost commitment (like Mentra/AisleAsk).