Project and command format

Markdown · Agent index

Integrator-owned. The JSON transport and C ABI remain v1; project documents may use schema 1 to 6. JSON keys are camelCase. Unknown command variants reject. All times serialize as reduced {value: integer, timescale: positive integer}; safe unreduced input is normalized immediately. Numeric integers must be in JavaScript’s safe range. Intervals are half-open. No floats are stored as media time. Positions use finite doubles; transform scale is in (0,16].

Records

type Time = { value: number; timescale: number };
type Range = { start: Time; end: Time };
type SphericalView = { yaw: number; pitch: number; roll: number; verticalFov: number };
type Transform = { x: number; y: number; scale: number; fit: "fit" | "fill"; sphere?: SphericalView; reframe?: "subject" };
type Locator = { kind: "external" | "relative"; path: string };
type Asset = {
  id: string; name: string; kind: "video" | "image"; locator: Locator;
  duration: Time; width: number; height: number; hasAudio: boolean;
  contentHash?: string;
  projection?: "equirectangular" | "insta360_dual_fisheye";
};
type MainSegment = {
  id: string; occurrenceId: string; assetId: string; source: Range;
  originalSource: Range; omissions: Omission[]; pauses: PauseEdit[];
  showVideo: boolean; playAudio: boolean; volume: number; transform: Transform;
};
type Omission = { id: string; source: Range; transcriptId: string; atomIds: string[] };
type PauseEdit = { transcriptId: string; atomId: string; retained: Range };
type OverlayContent =
  | { kind: "media"; assetId: string; sourceStart: Time }
  | { kind: "text"; text: string; fontSize: number; maxWidth: number; elapsed: Time; typewriterCps?: number };
type Overlay = {
  id: string; parentId: string; anchor: Range; content: OverlayContent;
  showVideo: boolean; playAudio: boolean; volume: number; transform: Transform;
};
type Atom = { id: string; text: string; range: Range; kind: "word" | "pause" };
type Transcript = {
  id: string; assetId: string; generation: number; contentHash?: string;
  engine: string; model: string; boundaryPolicy: string; language: string;
  atoms: Atom[];
};
type Correction = { id: string; transcriptId: string; atomIds: string[]; text: string };
type CaptionSettings = { enabled: boolean; fontSize: number; x: number; y: number; maxWidth: number };
type CaptionOverride = { key: string; x?: number; y?: number; hidden?: boolean };
type CaptionBreak = { transcriptId: string; atomId: string };
// One of four presets; fps and audioSampleRate never change.
type Output = (
  | { width: 1080; height: 1920 }   // portrait (default)
  | { width: 1080; height: 1350 }   // vertical45
  | { width: 1080; height: 1080 }   // square
  | { width: 1920; height: 1080 }   // landscape
) & { fps: 30; audioSampleRate: 48000 };
type OutputPreset = "portrait" | "vertical45" | "square" | "landscape";
type Project = {
  schemaVersion: 1 | 2 | 3 | 4 | 5 | 6; id: string; title: string; revision: number;
  assets: Asset[]; assetTitles?: Record<string, string>; main: MainSegment[]; overlays: Overlay[];
  transcripts: Transcript[]; corrections: Correction[];
  captionSettings: CaptionSettings; captionOverrides: CaptionOverride[];
  captionBreaks?: CaptionBreak[]; assetColors?: Record<string, number>;
  output: Output;
};
type RenderSpan = {
  id: string; segmentId: string; occurrenceId: string; assetId: string;
  source: Range; timeline: Range;
  showVideo: boolean; playAudio: boolean; volume: number; transform: Transform;
};
type RenderOverlay = {
  id: string; overlayId: string; parentId: string; assetId: string; source: Range; timeline: Range;
  showVideo: boolean; playAudio: boolean; volume: number; transform: Transform;
};
type ProjectedWord = {
  id: string; occurrenceId: string; segmentId: string; transcriptId: string;
  atomIds: string[]; text: string; kind: "word" | "pause";
  source: Range; timeline: Range; partial: boolean;
};
type RenderClip = { id: string; occurrenceId: string; assetId: string; originalSource: Range; source: Range; timeline: Range; colorIndex: number };
type TranscriptElement = Omit<ProjectedWord, "kind"> & {
  kind: "word" | "pause" | "deleted"; cutId?: string; pauseBounds?: Range; sentenceId: string;
};
type RenderSentence = {
  id: string; label: string; occurrenceId: string; segmentId: string; transcriptId: string;
  index: number; timeline: Range;
};
type RenderText = {
  id: string; overlayId: string; parentId: string; text: string; graphemes: string[];
  source: Range; timeline: Range; elapsedAtStart: Time; typewriterCps?: number;
  fontSize: number; maxWidth: number; transform: Transform; showVideo: boolean;
};
type CaptionReveal = { at: Time; graphemeCount: number };
type CaptionCue = { id: string; text: string; timeline: Range; x: number; y: number; fontSize: number; maxWidth: number; reveals: CaptionReveal[] };
type RenderPlan = {
  revision: number; duration: Time; output: Output; assets: Asset[];
  clips: RenderClip[]; spans: RenderSpan[]; overlays: RenderOverlay[]; texts: RenderText[]; captions: CaptionCue[];
  words: ProjectedWord[]; elements: TranscriptElement[]; sentences: RenderSentence[];
};
type Snapshot = { project: Project; renderPlan: RenderPlan; canUndo: boolean; canRedo: boolean; undoAction?: string; redoAction?: string };
type Request = { projectId: string; expectedRevision: number; operationId: string; commands: Command[]; undoGroup?: string; actionName?: string };
type Reply = { ok: true; value: Snapshot } | { ok: false; error: { code: string; message: string } };

Defaults: transform (0.5,0.5,1,fill); captions enabled, size 64, anchor (0.5,0.82), maxWidth 0.84. Caption x/y denote the cue rectangle’s center in top-left-origin normalized portrait coordinates. These coordinates also apply to picture transforms. An image’s asset duration is zero, and its placement has a real duration. RenderOverlay source range for an image is zero to the placement duration; it is only a duration carrier, not a sampled movie.

Immutable transcripts are embedded in project.json for v1 so one atomic replacement captures a complete portable snapshot; raw engine output may be cached separately. This deliberately simplifies the initially planned sidecar layout. Runtime job state and caches are not serialized into Project.

Projection or spherical-view fields require project schema 2 (video titles require schema 3, below). A command introducing these fields promotes the document atomically; the store preserves its exact prior bytes under .yapwing/backups/schema-1-<sha256>.json before writing the upgrade. Older apps cannot open schema 2; restore the backup into a separate project to use one. Ordinary flat projects remain schema 1. Spherical angles are degrees: yaw/roll in [-180,180], pitch in [-90,90], verticalFov in [1,179], all finite. Prepared equirectangular assets use a default (0,0,0,75) view when absent. Raw dual-fisheye assets may be registered but cannot be placed as main clips or overlays.

assetTitles maps an existing asset ID to a Yapwing-only display title. It never renames the source file or changes Asset.name, which Locate and relink still match; an asset without an entry shows its file name without extension. A stored title is trimmed, nonempty, at most 256 UTF-8 bytes and free of control characters; at most 10,000 entries. A nonempty assetTitles requires project schema 3, and the field is omitted when empty. The reader accepts schemas 1 to 5. After any command the document is promoted atomically to the lowest schema that can hold its content, and never downgraded: undoing or clearing the last title leaves it at schema 3. The store preserves the exact prior bytes under .yapwing/backups/schema-<old>-<sha256>.json before writing any upgrade, e.g. schema-2-<sha256>.json for 2 to 3. Older apps cannot open schema 3. Titles never affect the render plan.

renderPlan.sentences lists the spoken sentences of every placed recording in timeline order, one entry per clip a sentence appears in. A sentence is a run of the current transcript’s atoms that ends after a word ending in ., ?, ! or … (before closing quotes or parentheses); the pauses after it belong to it. index counts sentences within the transcript from zero, and id is derived from the occurrence, transcript and index, so it survives cuts, restores, splits, reorders, trims, caption edits and removals of other clips; a new transcript generation or a new placement gives new ids. label is the short handle agents use (s1, s2, … numbered by recording order: asset order, then each recording’s earliest source; parts of a sentence divided by a split are lettered s4a, s4b in timeline order). Labels are unique in one plan but can renumber when a recording leaves the reel. Every transcript element names its sentence in sentenceId, taken from the element’s first atom. Sentences are derived; they are never stored, so documents of every schema open unchanged.

A text anchor overlapping a media overlay anchor in the same parent requires project schema 4; schema 1 to 3 documents always kept one child lane, so they open unchanged and need no rewrite. The promoting command backs up the prior bytes as above. Older apps cannot open schema 4. An update_overlay may change parentId (moving text into another clip); the anchor is then in the new parent’s source clock.

captionBreaks lists forced caption cue boundaries, each before one word atom of an existing transcript; sorted, no duplicates, at most 100,000. Captions then start a new cue at that word in addition to the usual grouping (6 words, 42 characters, 3 s, cuts and long gaps). A break before the first atom of a corrected group applies to the shown word; one on any later atom of a merged group has no effect. A break on a superseded transcript stays valid and inert. Breaks never change the video or transcript elements, and unaffected cue ids stay stable.

assetColors maps a placed or previously placed asset ID to a clip palette index 0 to 7. RenderClip.colorIndex is every clip’s effective color: a stored entry wins; any other placed source gets, in reel order of first appearance, the lowest index no other placed source uses, or with all eight in use the least-used index (ties lowest). Compilation never mutates the project. When an append leaves (or finds) the reel mixing sources, or the map is already nonempty, the core stores the effective color of every placed source lacking an entry, both before and after the insertion, so later splits, reorders, removals and re-adds never shift a color. Entries are never recolored and are restored by undo. A single-source reel stores nothing and keeps its schema.

A nonempty captionBreaks or assetColors requires project schema 5; both are omitted when empty, and promotion and backups follow the rules above. Older apps cannot open schema 5.

output is one of four presets: portrait 1080x1920 (the default), vertical45 1080x1350, square 1080x1080 and landscape 1920x1080, all at 30 fps and 48000 Hz. Any other size rejects with invalid_output. transform.reframe: "subject" asks the host to follow the clip’s main subject on top of x/y/scale; the path is derived from the source and never stored. Reframe cannot combine with a spherical view or an equirectangular source, and text cannot reframe. A non-portrait output or any reframe requires project schema 6; a portrait project without reframe keeps its schema and bytes. Older apps cannot open schema 6. Stored text and caption font sizes stay valid when the output changes, up to 1920 (the longest edge of any preset); the render plan clamps fontSize to the current output’s longest edge, and a new or changed size must fit it.

library.json is portable source membership and prepared-media association. .yapwing/cache/library/ contains derived metadata only, keyed by source hash and geometry. These caches do not change authored history. Native track timestamps in metadata encode Int64 values/epochs as decimal strings, separately from the safe-integer edit-time contract above.

Commands

Every variant has a type tag. A batch creates one authored undo entry and one monotone revision; an authored batch that leaves the project unchanged is accepted with no new revision or undo entry. No partial batch commits. Receipt payload hashes and responses are persisted by the store for idempotent retry after reopening.

type	Fields
register_asset	asset: Asset (metadata-only; no placement or authored undo entry)
append	asset: Asset, segmentId: string, occurrenceId: string, index?: number, stillDuration?: Time
trim	segmentId: string, source: Range
split	segmentId: string, at: Time, newSegmentId: string (source time)
remove	id: string (main or overlay)
reorder	segmentId: string, toIndex: number (index after removing segment)
cut_units	unitIds: string[] (exact visible ProjectedWord.id values)
cut_words	occurrenceId: string, atomIds: string[]
restore_cut	segmentId: string, cutId: string
set_pause_edge	segmentId: string, transcriptId: string, atomId: string, edge: "left" or "right", at: Time (source time)
correct_words	occurrenceId: string, atomIds: string[], text: string
reset_words	transcriptId: string, atomIds: string[]
set_flags	id: string, showVideo: boolean, playAudio: boolean
set_volume	id: string, volume: number
add_overlay	asset?: Asset, overlay: Overlay
add_text	id: string, timeline: TimeRange, content: OverlayContent, transform: Transform (authored, undo name “Add Text”; timeline is a positive range inside the reel and may cross clips; content must be text; becomes one text overlay per clip it covers, the first with id, later ones with derived ids, each later piece’s elapsed is the previous piece’s elapsed plus its anchor’s source duration, as a split makes; no schema promotion)
update_overlay	overlay: Overlay
set_text	id: string, text: string
set_typewriter	id: string, charactersPerSecond: number or null
set_text_style	id: string, fontSize?: number, maxWidth?: number (authored, undo name “Text Style”; a text overlay; at least one field; fontSize in output pixels, (0, current output’s longest edge], maxWidth a fraction of output width (0,1]; omitted fields are kept; no schema promotion)
set_transform	id: string, transform: Transform
set_captions	settings: CaptionSettings (a changed fontSize must fit the current output’s longest edge)
set_output	preset: OutputPreset (authored, undo name “Format”; only the four preset names parse; the current preset is a no-op without a revision; stored text and caption sizes are kept; a non-portrait preset promotes to schema 6)
set_caption_override	override: CaptionOverride
attach_transcript	transcript: Transcript (metadata-only request)
identify_asset	assetId: string, contentHash: string (metadata-only request)
relink_asset	assetId: string, locator: Locator, contentHash: string (metadata-only; matching bound identity)
adopt_external	project: Project (sole command; same identity/revision/assets/transcripts/output)
rename	title: string
set_asset_title	assetId: string, title: string or null (authored, undo name “Rename Video”; trimmed; null, empty or whitespace clears; promotes to schema 3)
set_caption_break	transcriptId: string, atomId: string, enabled: boolean (authored, undo name “Add Caption Break” or “Remove Caption Break”; atom must be a word; unchanged is a no-op without a revision; promotes to schema 5)
undo / redo	no extra fields; the only command in its request

projectId must equal the live project before any retry or mutation. register_asset, identify_asset, relink_asset and attach_transcript are metadata-only; they may share a batch, but cannot mix with authored commands. Metadata advances revision and preserves authored undo/redo. A request normally holds 1–128 commands. To collect a larger library atomically, 129–10,000 commands are accepted only when every command is relink_asset and each target asset ID occurs once. All targets must exist with the same bound content identity; one invalid relink rejects the entire transaction. The 32 MiB request limit still applies. Other metadata, authored or mixed request shapes remain limited to 128 commands. A measured lowercase SHA-256 may bind an unidentified asset exactly once; changing an existing identity rejects. Identify before ASR. Transcript replacement rejects when corrections, omissions or pause edits in current state or undo/redo depend on its old generation. Transcript text/timing never changes in place.

Every main record is a durable logical clip. originalSource is its recoverable source domain and source its current trim. omissions retain exact original word coverage and identity; pauses retain source-bounded pause handles. Compilation subtracts omissions and removed pause intervals without replacing the logical clip. Even a fully cut clip remains in main and clips with a zero-duration timeline. Still main clips use a virtual clock; append defaults to 5 seconds or accepts a positive stillDuration. An image Asset still has zero source duration/no audio. Image trims may expand their synthetic originalSource domain beyond the default five seconds; image source time does not represent finite recorded samples.

Volume is a finite linear gain in [0,1], independent of playAudio. adopt_external validates an authored-state candidate while preserving the exact asset/transcript evidence; it creates one undo entry. New assets or source identity changes must use typed import/metadata paths, not external adoption.

cut_units targets distinct positive visible ProjectedWord.id values in the inspected revision, across clips if desired. It creates exact omissions for words and collapses selected pauses to their current retained midpoint. It preserves unselected words, pauses, and unrepresented gaps. Correction groups are indivisible. Unknown/duplicate/empty selection rejects atomically. cut_words is the compatibility contiguous atom selector; ambiguous partial atoms reject. It now preserves gaps by the same exact-unit rule. restore_cut removes exactly the named omission after any other edits; it does not undo subsequent edits or restore a removed logical clip. set_pause_edge changes only one handle within the original pause/clip domain. A zero pause is retained and editable. An edge may equal the other; crossing rejects. No source outside the original pause can be added.

elements contains visible words, every intersecting pause (including zero-length handles), and deleted-word placeholders in clip/source order. It reuses the word identity fields. Deleted source is the omitted original interval, timeline a zero-length point, cutId names its recoverable omission, and text contains the original immutable transcript words. Pause source is the retained handle range, pauseBounds its original bounded range. words contains only positive surviving units for rendering/selection. Deleted content is never captioned or played. Corrections render only where an entire group survives inside one logical clip; partial groups fall back to original words marked partial. Explicitly splitting inside a corrected group therefore shows original partial units on each side.

Overlays are owned by exactly one parentId, anchored in that parent’s source clock. A child is media or text, and its kind picks its lane: media overlays and text are separate lanes, derived from content (no stored lane field). Anchors must fit originalSource; overlapping anchors of one lane in one parent reject even when temporarily hidden by cuts/trims. Reorder carries children, removal removes them, and cuts/trims hide source coverage without losing it. Restore re-exposes that exact child coverage. Split partitions parent domains, omissions, pauses, and children; right media sourceStart/text elapsed advance by the exact split offset, and right pieces of crossing children receive new IDs. No free-floating overlay can cross ownership boundaries. A media child requires an existing or supplied matching asset; text needs none. Text has no audio. Text supports up to 2000 grapheme clusters, finite fontSize (0,1920] (new text up to the current output’s longest edge), maxWidth (0,1], and optional typewriterCps (0,240].

Caption cues carry full phrase layout plus ordered absolute reveals; at time t, show the prefix from the latest reveal with at <= t. graphemeCount includes spaces. A corrected group reveals its first written word at its first original timestamp; further written words reveal at equal steps across the group’s own timeline range (no media time is invented). A corrected group with empty text is hidden from captions only; its media still plays and the phrase continues across it. Lay out the complete phrase once so revealing words does not move previously visible words. Text similarly lays out full content: if animated, at a displayed time t show min(graphemes.length, 1 + floor(typewriterCps * (elapsedAtStart + t - timeline.start))) graphemes. elapsedAtStart includes source time skipped by cuts, preserving reveal continuity. Preview and export must follow the identical rule and grapheme sequence.

C ABI

uint32_t yapwing_abi_version(void) returns 1. void *yapwing_session_new(const char *project_json) uses null for a new project. It returns null for invalid input. yapwing_last_error() returns an owned UTF-8 JSON error buffer after a failed creation. Each session has one serialized caller. char *yapwing_snapshot(void *session) and char *yapwing_apply(void *session, const char *request_json) return owned, NUL-terminated Reply JSON. yapwing_buffer_free(char *) releases every buffer. yapwing_session_free(void *) releases the session. Catch panics at the boundary. No video frames or frame-rate calls cross this boundary.

Swift’s Codable records are transport types, not a second validation authority. Project imports must enter yapwing_session_new before the app consumes a decoded snapshot or passes a render plan to media. The shared negative fixtures fixtures/contracts/invalid-unsafe-revision.json and fixtures/contracts/invalid-reversed-range.json must reject through both the Rust project/session entry points and the Swift core bridge.

Persistence and agents

Direct JSON edits are supported while closed. When open, inspect/check/apply routes through the app’s owner-only local endpoint with revision checks. Unexpected file changes suspend autosave and preserve an observed candidate beside the in-memory state; arbitrary uncooperative concurrent raw writers are not supported. Never overwrite a conflict automatically.

The agent bridge (yapwing_agent, the CLI’s agent calls) takes an optional signals member beside method: host-measured source analysis, never authored and never written to the project or to a compiled request.

type SourceSignals = { assets?: Record<string, {
  silences?: Range[];
  shots?: Range[];
  quality?: { sharpness: number; faceQuality?: number; eyesOpen?: number; stability: number }[];
}> };

Ranges are in that asset’s source time, ordered and nonoverlapping. quality[i] describes shots[i], so when present it has one entry per shot; each score is from 0 to 1 with higher better, and the face scores are absent for a shot without a face. Signals for assets not in the project are ignored and a malformed member rejects the call. outline lists the longest silent stretches that play in the edited video (“Silent stretches … in c1”) and, on a “Shots” line, the shots that play with their scores; without signals the outline is unchanged. The planner reads them for edits in the same call.