The architecture behind a little world for two

Start with the person. Work backward to the system.

A person opens Luvu, chooses a few photos, and meets a little version of themselves. Their partner joins. They share what they’re doing, how they feel, or a little love. Elsewhere, a creator makes a pair of birthday glasses in a local Studio, publishes it, and puts it on a character.

That is the intended experience. Nobody should need to understand an expression sheet, an image mask, an object generation precondition, or a WebSocket admission ticket to enjoy it.

The user’s task is small.
The engineering has to keep it small.

Luvu’s architecture follows that constraint across two audiences: the person using the app and the person making assets for it. Both need clear choices, useful previews, reversible edits, and an honest account of what happened when something fails.

This article traces the current implementation from photo discovery to rendering, from local authoring to cloud publication, and from a private account to a shared moment. Luvu is a personal project in beta, designed with love by Idalis Nunez and Taimur Khaliq. The interesting engineering is in the boundaries: where one responsibility ends, what must be true before the next begins, and how those decisions make the experience easier.

A native app, an authoring tool, and services with different jobs.

The iPhone app is SwiftUI on iOS 18 and later, with Swift 6 domain code in LuvuCore. It owns interaction, local storage, editing, and rendering. A TypeScript backend handles verified accounts, pairing, durable shared state, and asynchronous avatar generation.

The Asset Studio is a separate web application running on the creator’s Mac. It can use a local Qwen Image 2.1 runtime through ComfyUI. The app consumes the Studio’s output contract; it does not contain the Studio, its JavaScript dependencies, or its image-generation runtime.

Firestore holds durable account and relationship state. Private Cloud Storage holds user artwork and temporary generation inputs. A separate public bucket holds deliberately published, reusable accessories. Cloudflare Workers and Durable Objects coordinate live heart interactions. APNs delivers native notifications. Sharing a cloud provider does not make these data classes interchangeable.

A person, their character, and today’s feeling are different things.

We keep authentication identity separate from product identity and visual representation. An internal user ID identifies the account. A profile supplies the name a partner sees. AvatarIdentity identifies the selected character and its rendering source. Appearance edits refer to that artwork without replacing the person’s identity.

Business state and repositories live outside SwiftUI views. Views express intent—select, save, invite, share—while services own persistence and network operations. This is especially important at launch: a restoring session is its own state, so a returning user doesn’t briefly see a signed-out screen before their credentials load.

Local ownership metadata lets an authenticated account adopt an existing local profile. It doesn’t require moving every image or regenerating a character. Sign-out removes credentials while retaining local avatar files; account boundaries govern which private data becomes visible in a session.

“Choose three photos” should be a small request.

Finding useful reference photos can be more work than pressing Generate. “Help me choose” therefore begins with an explanation and an explicit permission action. PhotoKit supplies candidate metadata and small previews; Vision ranks technical portrait suitability on the phone. Manual PhotosPicker selection remains available if access is limited, denied, slow, or simply unwanted.

The scanner prefers a clear, prominent face, useful framing, reasonable lighting and detail, then uses pose and image similarity signals to avoid three nearly identical suggestions. It does not identify the person. The confirmation step says to make sure the suggestions are all you. A good portrait of someone else is an expected result to replace, not evidence that the model knows your identity.

Initial analysis uses thumbnail-sized images, one submitted Vision operation at a time on a utility executor, and a bounded scan budget. More expensive landmarks and quality analysis are reserved for finalists. Workload responds to thermal state, memory pressure, Low Power Mode and observed timing rather than a hard-coded list of iPhone models. Partial suggestions can arrive before the scan finishes. Cancellation propagates to PhotoKit and Vision where supported.

Only confirmed photos reach preparation: orientation normalization, appropriate cropping, downsampling and metadata stripping, without modifying the originals in Photos. The normal Gemini path uses three references and asks for one complete character expression sheet, with nine expressions including sleeping. OpenAI remains behind a provider adapter for development; provider-specific output processing still matters, even with a shared job interface.

The phone does not own the lifetime of generation.

The Cloud Run API validates the request, records the job and places work in Cloud Tasks. An IAM-protected worker makes the provider call and stores results. Firestore job claims and request IDs prevent a duplicate task delivery from casually becoming another paid generation. A scheduled sweep handles interrupted work and temporary-source cleanup.

There is no distributed transaction with an external image provider. If a worker disappears after starting a billable call, silently repeating it could double the cost. Recovery must distinguish a retrievable result from a failed job requiring another explicit generation decision. Reopening the screen is not permission to spend again. Provider usage is measured separately from cloud hosting and storage costs.

One character interface. Honest capabilities underneath.

Features render through AvatarView, supplying an AvatarIdentity and semantic state. They do not choose file paths or know how a particular character was made. A renderer resolves the representation and its available expression and motion capabilities.

The distinction prevents a common trap: designing an animation around a capability only one avatar has. Love can use a small affectionate movement, a heart and a broad warm glow even when a character cannot independently bend its elbows. Reduced Motion keeps acknowledgment and warmth while removing traveling or bouncing choreography.

Rendering readiness also matters. A view can have a size before its image has loaded. The presentation tracks actual content readiness rather than treating an empty placeholder as a visible participant. Heart placement follows the rendered characters’ bounds and actual sender/recipient roles. We replaced a visible dotted trajectory with an independent floating heart. It communicates direction while warm light carries the emotional payoff around both characters.

The most useful abstraction was keeping the artwork intact.

An earlier approach generated bald heads and separate hair, facial hair and glasses. It offered theoretical editability but introduced practical problems: poor registration, mismatched styling, visible seams and facial hair that stopped matching the mouth during an expression. Strict quality gates then rejected parts, leaving users with incomplete characters and a confusing sequence of “remake” requests.

The current consumer path preserves the complete generated character—hair, beard, clothing, body and expression—as source artwork. Customization adds targeted operations to that artwork. It does not ask the user to repair the output of an automatic decomposition pipeline.

Recolor a region; preserve the rest.

A local MediaPipe hair segmenter proposes a selection on the avatar image. It runs off the main actor with bounded inference dimensions. Cartoon hair, gray temples and fine wisps are imperfect inputs, so an add/erase brush lets the person correct the mask. The recoloring operation preserves pixels outside the selected region and retains transparency. Clothing details and a familiar logo survive because the editor does not regenerate the whole image.

Carry edits across expressions.

Saving only a customized neutral image caused another regression: excited, tired and other expressions disappeared. The current editor renders a derivative for every available source expression. Apple Vision optical flow estimates correspondence between the neutral frame and each other frame using bounded preview resolution. The mask transfers through that alignment; the expression’s own original pixels remain the source.

Glasses use transferred placement—center, scale and tilt—rather than per-pixel warping that would distort frames or lettering. Placement is authored on Neutral, with expression previews before saving. Registration is an estimate, not a guarantee: inconsistent poses or art between generated frames can still need attention. An older saved edit may need an explicit expression upgrade; viewing it should not silently replace its files.

This is deliberately narrower than arbitrary editing. New glasses can be overlaid and removed; glasses already painted into an original cannot simply be “unequipped.” Recoloring hair is not replacing its shape. Shoes or shirts baked into a full image require a different replacement strategy. Keeping those limits explicit protects both visual quality and the user’s expectations.

A Studio should feel like a workbench, not a cockpit.

The Studio began with technical authoring controls. The useful workflow turned out to be much smaller: My assets → Create → Make changes → Preview → Publish. Advanced contract and fitting tools remain available underneath that flow.

The local generator runs Qwen Image 2.1 through a pinned ComfyUI runtime on the Mac. It is separate from Ollama and from the Gemini provider used for photo avatars. One is an authoring tool for reusable accessories; the other creates a person’s character from confirmed references.

A creator can generate glasses, inspect them, then say “front only” if the model includes side arms that look wrong on a frontal character. “Make changes” uses the selected result as an image reference. Editing a prompt for a fresh variant is a distinct operation. Each completed attempt retains its original PNG and generation metadata in the Studio’s local library, with the relationship to its parent revision where applicable.

The UI reports actual generation stages and elapsed time. Cancellation is requested from the local runtime, though in-flight model operations may take time to yield. A connection failure does not trigger a hidden repeat generation.

The Studio stays on loopback with origin checks and a local session token. Its cloud publisher uses the developer’s short-lived credentials on the local server. Neither the browser UI nor the iPhone receives cloud publishing credentials. This is a private development tool, not a publicly deployed multi-user service.

Packaging enforces that boundary too: Studio lives under tools/asset-studio, outside the iOS target. Release build checks compare the application with and without that directory. The shared schema and a valid catalog remain sufficient inputs; editor dependencies must not inflate the app bundle.

A PNG says what it looks like. A manifest says how to use it.

Artwork alone cannot tell a renderer whether it is glasses, where it attaches, what order to draw it in, or whether its files are complete. The reusable asset contract makes those assumptions explicit and versioned.

LuvuAssetManifest describes asset identity and version, category, compatible rigs and body fits, a canvas, reference anchors, files, layers, masks, tint roles, preview and validation metadata. The catalog is a lightweight discovery document pointing to exact immutable asset versions and pinned manifest hashes. A character package, a reusable accessory and the catalog have related but distinct schemas.

{
  "assetSchemaVersion": 1,
  "id": "birthday-glasses",
  "assetVersion": 1,
  "category": "glasses",
  "displayName": "Birthday glasses",
  "referenceAnchors": {
    "eyesCenter": { "x": 0.5, "y": 0.3 }
  },
  "tintRoles": []
}

Illustrative excerpt, not a complete importable manifest. Coordinates are example values. The full contract also requires compatibility, canvas, files, layers, masks, preview and validation fields.

Layer metadata specifies a file, role, slot, anchor, bounds, z-order and opacity, with placement transforms. These are rendering instructions as data, not downloadable executable code. Category and role must agree; an accessory cannot quietly replace arbitrary identity layers. The canonical canvas and normalized anchors separate attachment geometry from screen pixels, so placement is not hard-coded for a particular phone size.

JSON Schema checks structure, while runtime validation checks cross-field relationships and the actual image bytes: dimensions, supported format, hashes, path safety and compressed/decoded memory budgets. A small compressed PNG can expand into a large allocation, so file size alone is insufficient. A declared “compatible” flag is not a substitute for validating geometry or reviewing visual fit.

The definitions live in shared/avatar-schema. Studio and native validation fixtures exercise the same contract. The contract is what allows a second authoring tool—or a manually designed asset—to participate without coupling the phone to Qwen, ComfyUI or a particular editor.

Publish the catalog last.

For the creator, publishing should be one coherent action with a fitting review. Underneath, it is a sequence with a clear visibility boundary.

Asset files use immutable versioned paths. New objects are created with preconditions that prevent overwriting an existing version. After read-back verification, the publisher updates the catalog using a Cloud Storage generation-match precondition. If another publisher changed the catalog, it rereads and merges within a bounded retry policy instead of losing the other publication. An uncertain commit response is checked before another attempt.

Hashes provide integrity relative to the catalog; they do not independently authenticate a compromised publisher. The trust boundary includes the configured HTTPS origin, publishing permissions and the catalog itself. Private character references, generation prompts and local edit masks are excluded from public publication.

On the phone, discovery is bounded to a 512 KB catalog. Thumbnails load with limited concurrency; asset installation is serialized. The downloader rejects unexpected origins and redirects, incompatible content, invalid hashes and exceeded budgets before selection. Verified files are cached locally, and saved avatar derivatives remain usable when the catalog cannot be reached.

Today, glasses are the complete cloud-to-original-avatar path. The broader schema supports more categories, and Studio can author them, but a schema enum alone does not implement their behavior on every avatar. That separation prevents us from presenting an unsupported capability as a working product.

A small shared space needs a precise ownership model.

Native Sign in with Apple begins authentication; the backend finishes it. The server verifies Apple’s token signature through its public signing keys, along with issuer, audience, expiry and nonce, and handles the authorization-code exchange. It finds or creates a stable internal Luvu user. Decoding a token or accepting a client-supplied Apple identifier is not authentication.

Luvu sessions use short-lived opaque access credentials and rotating refresh credentials with server-side revocation. Sensitive session material lives in Keychain, not UserDefaults. Apple name and email are not assumed to reappear on every sign-in; the product asks for a first name or nickname when needed.

Pairing is an explicit invitation accepted by another signed-in person. Firestore transactions enforce the two-person relationship. Opening a link or installing the app does not perform acceptance. The invitation website is a static fallback; the secret can remain in the URL fragment rather than being sent as a page request. A manual code handles cases where the universal-link or installation journey is awkward.

Sharing an avatar means publishing the selected rendering and its available expressions. The app uploads immutable, hashed artifacts before committing the shared descriptor. Partner access is authorized through current relationship membership, and the client cache is scoped by account, couple and revision. The partner receives finished artwork, not the source selfie library or the local editing recipe.

Account deletion and sign-out have different semantics. Server deletion revokes the account and schedules owned server data cleanup, including the Apple authorization revocation flow. Local avatar retention is described separately. Neither operation should claim to have erased data it has actually retained.

A saved status, a push, and a live heart are not the same event.

Most shared state is durable and revisioned. Activities and feelings have independent content and expiry. Foreground refresh reconciles current server state, with scene-aware polling and backoff. The app can render cached state while connectivity recovers; a previous connection error must not become a permanent caption.

Commit state before delivering an alert.

A product mutation records its notification event in a transactional outbox. Queued delivery runs separately so a transient APNs failure does not undo the person’s saved update. Before dispatch, the worker rechecks relationship membership, device binding, preferences, revision, expiry and acknowledgment as appropriate.

Extending a duration is not a new activity or feeling, so it should not renotify. Rapid changes are coalesced. Activity and feeling details can appear in the native notification, subject to recipient preferences. APNs accepting a request is not proof the phone displayed it, and a crash around delivery can create ambiguity; client event deduplication still matters.

On return to the app, a backlog becomes current state rather than a parade of stale animations. If love was missed, the app preserves one full love experience instead of replaying every duplicate. We removed the persistent “Send love back” and “Again?” prompts: the ordinary heart control is the natural way to begin another interaction.

Put the live interaction in one authoritative room.

Live hearts need tighter coordination than polling a document. The existing authenticated API issues a short-lived, scoped admission ticket. A Cloudflare Worker validates it and checks canonical membership before admitting the connection to a per-couple Durable Object. The Worker does not receive Firestore administrative credentials.

The room sequences inputs, deduplicates retries, persists confirmed outcomes and owns deadlines. Clients reject stale snapshots and render timed effects against server timing estimates. An interrupted participant causes an honest hold or end rather than letting an offline phone accumulate score. Revocation epochs fence obsolete membership; a bounded freshness policy closes the gap when explicit revocation cannot be delivered immediately.

This avoids turning each tap into a Firestore write or a push notification. Native animation uses bounded particles and haptic rates, with gentler behavior under pressure and a reduced-motion presentation. The room owns game truth; the phone owns how that truth feels.

Simplicity depends on failure being contained.

Cloud Run keeps the public API and IAM-protected worker separate. Cloud Tasks gives longer jobs a delivery mechanism independent of a phone’s connection. Firestore holds durable coordination, Cloud Storage holds artifacts, Secret Manager keeps provider and APNs credentials server-side, and scheduled recovery handles work that did not finish cleanly. Cloud Build and deployment scripts produce backend revisions independently of the iOS distribution process.

The website and these engineering pages are static HTML on Firebase Hosting. They require no account session or client-side application runtime. Studio deployment is a separate concern; its private local interface is never published as part of the public website.

Resource limits are part of correctness, not merely optimization: thumbnail-first scanning, serial expensive image operations, bounded image decode budgets, small catalogs, limited downloads, capped live effects and cancellation checkpoints. The goal on an older supported iPhone is a slower operation with a usable interface, rather than a faster path that exhausts memory.

Tests follow the failure boundaries.

• Preservation fixtures: an existing generated avatar survives account adoption, relaunch, selection and local editing with its identity and original files intact.
• Generation orchestration: explicit consent, one intended submission, recoverable job state and no paid provider calls in ordinary automated tests.
• Image and contract tests: mask transfer, preserved unselected pixels, expression outputs, mismatched hashes, invalid dimensions and incompatible asset metadata.
• Publishing fault injection: partial uploads, concurrent catalog changes and ambiguous commit responses never advertise incomplete content.
• Relationship and event tests: revoked sessions, ended pairings, duration-only updates, duplicate notifications and stale live snapshots.
• Native interaction checks: small-screen layouts, keyboard behavior, cancellation, expression preview, offline reuse and Release builds that exclude Studio.

These checks do not establish a universal frame-rate, availability or latency guarantee. Provider variation, old-device memory pressure, cold starts, network partitions and operating-system background limits remain real. Useful instrumentation records stage durations, retries, resource bounds and safe identifiers rather than photos, credentials or private message content.

Expand capabilities without expanding the burden.

The working path is concrete: generate a complete character, recolor its hair locally, publish reviewed glasses from Studio, fit them on the phone, and preserve available expressions in the saved rendering. That is more valuable than promising arbitrary editing of every pixel.

The next accessory categories need explicit attachment and occlusion rules. A shirt pin can be an overlay attached to a torso region. Earrings may need to sit behind hair. A watch follows a wrist. Shoes and belts can overlap or replace existing artwork. They cannot all be solved by adding another name to a category list.

Each extension should add a validated capability to the shared contract, a fitting preview to the authoring workflow, and a supported consumer behavior to the app. Manual adjustment is a useful escape hatch, not a substitute for sensible default placement. Automatic visual quality judgments should assist review without hiding every imperfect but usable result.

Other boundaries are equally important: boards and earlier local history are not fully cloud-synchronized; the public catalog is not a private multi-user creator marketplace; original-image editing is not a universal layered rig. An honest implementation map gives us room to improve these without pretending they already work.