← product

Immersive 3D — The Stream & The Dome — Handoff

Status: shipped to prod, in active tuning. Last updated 2026-07-09.

This is the technical handoff for the WebXR work. The vision/product narrative lives in AR_GLASSES_LIVING_VOICE.md (Living Voice Track B). Read this one to pick the work back up; read that one for why it exists.

Built almost entirely blind — the agent could not see the scenes render; the user sends screenshots and the agent iterates. Every visual number (positions, scale, colors, camera) is a best-guess unless a screenshot confirmed it. The Adjust / Edit tooling exists precisely so the user can dial it in without the agent guessing.

The two surfaces

1. The Stream — /dashboard/stream/immersive

An interactive WebXR "river" of your journal notes. Stand at the water's edge (or sit in a vessel / on a dock); notes drift past; catch one → it lifts out and opens in full space, reading aloud in voice; release it → it casts back into the current.

Confirmed working in-browser (user screenshots). Features:

  • Cards typed by data: text note, voice (waveform + in-ear playback), comic/photo (attached image), drop (encouragement), mood (glowing orb), plus non-journal stream types: song (Signal Union), track (Veneer, has audio), veneer, graphene, glass (Looking Glass, "N years ago today"), book, reel.
  • Curated current + a selector (For you / On this day / Recent / Resurfaced).
  • Opened note shows its dashboard background image + its font (PremiumUltra .ttf via troika; Google fonts fall back).
  • Atmosphere: procedural star sky dome, drifting fireflies, a moon + moonglade reflection, opt-in Web-Audio ambient river, weather (clear/mist/rain + ripple rings), vessels (canoe/kayak/rowboat/paddleboard/dock), cards bump around an open panel.
  • Social: dome houses on the hillsides (you + family + AI companions), tap → "about them", visit an AI companion's stream (their shared entries flow instead of yours). Tapping your own house → enters your dome.
  • Persistence: current + vessel + weather in localStorage.

2. The Dome — /dashboard/dome

Your spatial "home" (Quest-Home-style). A geodesic dome interior whose décor mirrors your DreamPro goal adherence. Reached from the stream ("◇ Enter your dome") or by tapping your own house on the hill.

  • The model is real: public/models/dome.glb — the commissioned SketchUp interior, exported to glTF and compressed 17 MB → 3 MB (Draco geometry + WebP textures via @gltf-transform). Loaded with useGLTF(url, true).
  • Goal-reactive layer (rides over the static model): a floating goal hologram (one glowing tile per DreamPro dream, brightening with progress_percentage), and the whole space's warmth/lighting lift with overall adherence.
  • Hubs: a "Write at the Café" desk invite (→ /write-cafe; a dedicated Write-Café 3D is the intended next step) and a corkboard of cross-pollination pins (Write Café, Veneer, Graphene, Odessa, Lovio, DreamPro).
  • World outside (DomeExterior.tsx): star sky, dark hills, other domes glowing, water on one side — visible when the shell is glassed via "See out".
  • First-person controls (LookControls): eye fixed at room centre at your height, drag to look. NOT orbit (orbit dipped under the floor — see landmines).

⚠️ THE OPEN BLOCKER — bake the dome layout defaults

The dome layout is not yet dialed in. The agent's default positions/scale/orientation are guesses. The user must set them via the in-scene tools, then hand the values back to bake as DEFAULT_ADJUST.

The handoff loop:

  1. User opens /dashboard/dome → ⚙ Adjust (bottom-right).
  2. Sets model orientation (Tip/Spin/Roll), Scale, Height (range ±8 — the model is a full geodesic sphere so its centre ≈ the deck; Height drops it onto the user's feet), Eye height, and See out (glass the shell).
  3. Checks "Edit layout — drag in the scene" → clicks the goal shelf / desk / corkboard → drags the TransformControls move-gizmo to place each.
  4. Hits Copy settings → pastes the JSON to the agent.
  5. Agent bakes that JSON into DEFAULT_ADJUST in components/dome/DomeScene.tsx so every user gets the correct dome with nothing to configure.

Until step 5, new visitors see the un-tuned default. Known-wrong-ish defaults as of writing: model floats a bit / fixtures are guessed positions. DomeAdjust shape (all persisted under localStorage key dome:adjust):

{ rx, ry, rz, scaleMul, yOff, camY, seeOut, glassTop, glassOpacity,
  holo:{x,y,z}, desk:{x,y,z}, cork:{x,y,z}, corkRotY }

If "Edit layout" or new controls aren't visible: it's almost always Vercel deploy lag (~2–3 min after merge) or the Adjust panel needs scrolling (it's scrollable; controls are below See-out). Hard-refresh.

Architecture & files

Frontend, Next 15 App Router. All 3D is @react-three/fiber v9 + drei + @react-three/xr v6

  • three, loaded client-only (dynamic(() => import(...), { ssr: false })).

Stream

  • apps/frontend/src/app/dashboard/stream/immersive/page.tsx — data fetch, overlay UI, selectors, toNotes() mapping (journal → StreamNote), handleVisit.
  • apps/frontend/src/components/stream-vr/StreamXRScene.tsx — the scene (river, cards, sky, moon, weather, vessels, open panel, bump physics). Big file.
  • apps/frontend/src/components/stream-vr/Neighborhood.tsx — hillsides, dome houses, AboutPanel.
  • apps/frontend/src/lib/streamExtras.ts — fetch non-journal stream types.
  • apps/frontend/src/lib/useAmbientRiver.ts — Web-Audio ambience.

Dome

  • apps/frontend/src/app/dashboard/dome/page.tsx — data fetch, adherence calc, the Adjust + Edit panel (localStorage dome:adjust, Copy-settings).
  • apps/frontend/src/components/dome/DomeScene.tsx — model load/normalize, goal hologram, lights, LookControls, TransformControls edit mode, DomeAdjust + DEFAULT_ADJUST.
  • apps/frontend/src/components/dome/DomeFixtures.tsx — DeskInvite + Corkboard (interactive prop gates nav in edit mode).
  • apps/frontend/src/components/dome/DomeExterior.tsx — sky/hills/water/other-domes.
  • apps/frontend/public/models/dome.glb — the 3 MB model.

Shared type shim

  • apps/frontend/src/types/react-three-fiber.d.ts — re-applies fiber's JSX element map.

⚠️ Landmines (learned the hard way — do not re-discover)

  1. Next 15's App Router runtime is React 19 (vendored). @react-three/fiber MUST be v9+ (React-19 reconciler). v8 builds clean but crashes in-browser with Cannot read properties of undefined (reading 'ReactCurrentBatchConfig'). (#1102→#1103)
  2. @types/react pinned to 18.3.27 (root package.json devDep + overrides) so the three-stack's peer resolution can't drag a conflicting root v19 alongside the frontend's v18 (dual-copy 'Link' cannot be used as JSX errors).
  3. @iwer/devui stubbed (src/stubs/iwer-devui.js + next.config.js webpack alias). It's the XR emulator xr pulls in; has an undeclared zustand import that breaks the build.
  4. fiber alias in next.config.js — fiber v9 nests in the frontend workspace; xr/@pmndrs/xr hoist to the repo root and can't resolve it, so a webpack alias pins @react-three/fiber to the one copy (also guarantees the single instance r3f needs).
  5. Keep all @react-three/* imports out of a page module — load the scene via dynamic(ssr:false) so the reconciler chain never hits the server prerender.
  6. The Draco GLB loads its decoder from the gstatic CDN at runtime (drei useGLTF(url,true) default). Fine in-app.
  7. SketchUp is Z-up; three is Y-up → the model needs rotation.x = -π/2 (the Tip control). It's a full sphere, so anchor its centre (≈ deck) to the floor, not its bottom.
  8. Interior camera must be first-person, not orbit — orbit circles the room centre and dips the camera under the floor / outside the dome.
  9. Math.random()/Date.now() are fine in the app (that restriction is Workflow-scripts only). But r3f useFrame gives (state, delta) — use delta, don't call Date.now.

Build gate before any PR touching .tsx: cd apps/frontend && npm run check (lint+typecheck). 3D pages don't use useSearchParams/dynamic params, so a full next build isn't strictly required, but it's the only thing that catches prerender/runtime-shape issues — run it for dome changes. Node is via nvm: export PATH="$HOME/.nvm/versions/node/v22.14.0/bin:$PATH".

Data sources (all real endpoints, mapped before building)

  • Stream notes: GET /api/journal/stream-curated?limit=&mode= (modes foryou|onthisday|recent| resurfaced) — blends on-this-day/pinned/voice_stream_enabled/recent/resurfaced, tags each with stream_reason. Attached photo: getFirstImageForEntries (signed thumbnails).
  • Neighbors (houses): GET /api/journal/stream-neighbors — you (profiles), family (family_members), a few AI companions (ai_personas, with visitUserId).
  • Visit a stream: GET /api/journal/stream-visit?userId= — filtered through filterEntryForViewer FOR THE CALLER (only what they've shared with you). Works for personas now; humans need a per-user visitUserId + share-settings gate (not built).
  • Non-journal types: internal to streamExtras.ts (signal-union, veneer, story-seasons, looking-glass, cafe/books, fb-reels endpoints).
  • Dome goals (books): GET /api/dreampro/dreams (progress_percentage, category.color).
  • Dome adherence: GET /api/coaching/me momentum.current_streak (2-week streak = "full"), blended 50/50 with average goal progress.

Backend routes: apps/backend/src/routes/journal.ts (stream-curated, stream-neighbors, stream-visit), apps/backend/src/services/journal-entry-attachments.ts (getFirstImageForEntries).

Roadmap / what's pending

Immediate (blocks polish):

  • Bake dome DEFAULT_ADJUST from the user's Copy-settings JSON (see blocker above).
  • Tune stream blind-visuals from user notes (moon scale, firefly density, vessel proportions, rain, etc. — all quick number tweaks).

Next features (in rough priority):

  • Write-Café 3D — the desk invite should open a dedicated 3D writing scene, not just link to /write-cafe.
  • Map goals to the model's real shelves — instead of a floating hologram, drive the model's own bookshelf/plant/lights by goals (needs the model's mesh coords — inspect in-scene).
  • Human stream visiting — give family/friends a visitUserId once relationship + share-settings permit (the stream-visit read path already handles them).
  • VR controller grab for Edit mode (TransformControls is desktop-mouse only).
  • Corkboard = real cross-pollination — post/pin actual items, two-way with services.
  • Presence cues — a house/dome lights when someone shares; eventually a Lovio voice at the door (read-only bright line intact).

Assets: the user has more from the 99designs project (Dome.skp, interior renders, a hive-dome-cropped-1.png). .skp can't be transcoded by the agent — needs a SketchUp→glTF export. The spheres/.webloc files are dead (Google Poly shut down 2021).

PR history (this arc)

Stream: #1102 (scene) · #1103 (fiber v9 fix) · #1104 (docs) · #1105 (sky+cards) · #1106 (curation) · #1107 (current selector) · #1108 (ambient) · #1109 (catch) · #1110 (vessels) · #1111 (fireflies+ persist) · #1112 (cast-back) · #1113 (moon) · #1114 (weather) · #1115 (attached images) · #1116 (dome houses) · #1117 (visit) · #1118 (note bg+font) · #1119 (card bump) · #1120 (non-journal types). Dome: #1121 (goal-gated interior) · #1122 (you-house→dome) · #1123 (real model) · #1124 (adjust panel) · #1125 (brighter+world) · #1126 (camera fix) · #1127 (see-out) · #1128 (first-person + desk/corkboard) · #1129 (admin placement + center-anchor) · #1130 (in-scene Edit/drag).

IMMERSIVE 3D HANDOFF — Docs | HiveJournal