Skip to content
helmstudio
Contents

Hand off to the timeline

Sequencing is not one studio's job. A sequence cut from a video studio's take, another studio's clip, a still and a voice line is none of those studios' documents, so the timeline belongs to helmstudio: studios hand it clips, and it keeps the sequence, its revisions and its exports. A studio never renders a track or runs ffmpeg itself.

The timeline model: tracks, clips as references, and revisions At the top, a sequence drawn as tracks. One video track, V1, holding three clips laid end to end. One audio track, A1, holding two. A third strip notes that there are up to eight audio tracks in all. Below, three statements. First, a clip is a reference: it names an asset with in and out points, and the source is never modified or copied. Second, the rules: clips on one track never overlap, and sound never runs past the picture. Third, every edit is a patch: it writes a new revision, and undo writes an old one forward rather than rewinding, so nothing is lost by undoing. a sequence V1 A1 up to eight audio tracks in all a clip is a reference it names an asset, with in and out points the source is never modified and never copied the rules clips on one track never overlap and sound never runs past the picture every edit is a patch it writes a new revision, and undo writes an old one forward rather than rewinding, so nothing is lost
A sequence owns no footage. Every clip points at an asset, so an edit writes a new revision and never touches a file — and a sequence using an asset counts as a reference, so nothing can offer to delete footage it is still using.

This studio holds timeline, assets and gallery:

timeline/timeline.py
"""Put outputs in a sequence the framework owns, then export it.

Run under helmstudio or helm dev by a studio holding `timeline`, `assets` and
`gallery`. Exporting needs ffmpeg on the machine running helmstudio.
"""

import os
import struct
import time
import wave
import zlib

from helm_runtime_sdk import from_env

helm = from_env()


def still(name, rgb, width=640, height=360):
    """Adopt a single-colour still, standing in for a generated frame."""
    row = b"\x00" + bytes(rgb) * width
    def chunk(kind, data):
        body = kind + data
        return struct.pack(">I", len(data)) + body + struct.pack(">I", zlib.crc32(body) & 0xFFFFFFFF)
    png = (b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0))
           + chunk(b"IDAT", zlib.compress(row * height)) + chunk(b"IEND", b""))
    path = os.path.join(os.environ["HELM_STAGE_DIR"], name)
    with open(path, "wb") as f:
        f.write(png)
    return helm.assets.adopt({"path": path, "kind": "image"})


def tone(name, seconds=3, rate=48000):
    """Adopt a short sound: silence, standing in for a generated voice line."""
    path = os.path.join(os.environ["HELM_STAGE_DIR"], name)
    with wave.open(path, "wb") as w:
        w.setnchannels(1)
        w.setsampwidth(2)
        w.setframerate(rate)
        w.writeframes(b"\x00\x00" * rate * seconds)
    # A sound's length is a hint the studio gives; an export measures it again.
    return helm.assets.adopt({"path": path, "kind": "audio", "duration_s": float(seconds)})


dawn = still("dawn.png", (230, 160, 90))
dusk = still("dusk.png", (60, 50, 120))

# A sequence is a document the framework keeps, not a file the studio writes.
# Clips given without positions are laid end to end, video and stills on V1 and
# sound on A1; a still is held.
sequence = helm.timeline.create({
    "name": "a day at the lighthouse",
    "target": {"width": 640, "height": 360, "fps": 24},
    "clips": [{"asset_id": dawn["id"], "hold": 2}, {"asset_id": dusk["id"], "hold": 2}],
})
assert sequence["duration_s"] == 4.0, sequence["duration_s"]

# Adding to the end of a track needs no etag, so a studio can hand things over
# as it makes them. It takes video and sound; a still needs a hold, which
# append has no way to give.
voice = tone("voice.wav")
sequence = helm.timeline.append({"asset_id": voice["id"], "timeline_id": sequence["id"]})
assert [t["name"] for t in sequence["tracks"]] == ["V1", "A1"], sequence["tracks"]

# Every other edit is a merge patch against the revision read, and each one is
# a revision that can be reverted to. Here, the voice comes down 6 dB.
tracks = sequence["tracks"]
tracks[1]["clips"][0]["gain_db"] = -6
edited = helm.timeline.update(sequence["id"], {"tracks": tracks}, if_match=sequence["etag"])
assert edited["revision"] == sequence["revision"] + 1, edited["revision"]

# Undo is writing the earlier revision back, as the newest one.
reverted = helm.timeline.revert(edited["id"], {"revision": sequence["revision"]}, if_match=edited["etag"])
assert "gain_db" not in reverted["tracks"][1]["clips"][0], reverted["tracks"][1]
sequence = helm.timeline.update(reverted["id"], {"tracks": tracks}, if_match=reverted["etag"])

# The plan says whether the picture can be copied or must be re-encoded, and
# why. A still is always drawn, so this sequence is re-encoded.
plan = helm.timeline.plan(sequence["id"], preset="h264")
assert plan["mode"] == "conform", plan

job = helm.timeline.export(sequence["id"], {"preset": "h264"})
while True:
    job = next(j for j in helm.timeline.exports(sequence["id"])["items"] if j["id"] == job["id"])
    if job["state"] not in ("queued", "running"):
        break
    time.sleep(0.2)
assert job["state"] == "succeeded", job

# The export is an item in the gallery, labelled with the sequence it came from.
exported = helm.gallery.query(kind="video")
assert any(i.get("timeline_id") == sequence["id"] for i in exported["items"]), exported

print("ok")

Exporting runs ffmpeg on the machine running helmstudio, found through HELM_FFMPEG or on the PATH. helmstudio does not bundle it yet.

A sequence is a document

Edits are revisions

append adds an asset to the end of a track, and needs no etag, so a studio can hand things over as it makes them. Every other edit is a merge patch against the revision you read, passed as if_match. Every edit keeps the document it replaced, and reverting writes an earlier revision forward as the newest one, so undoing an undo loses nothing.

Sources are never modified or copied: a clip is a reference with in and out points. An asset a sequence uses counts as used, so reclaiming disk never offers to delete footage a sequence needs.

Editing one, from the page or the server

The host keeps every sequence and its revisions. Neither the page nor the studio's server holds one.

Export says what it will do

plan says, before anything runs, whether the picture can be copied — every clip already matches the target and the others, and is used whole — or must be conformed, and names the reason for each clip that must be. A still is always drawn, so a sequence with one is conformed. The sound is always re-encoded.

An export is a job. Its progress and its cancellation are under timeline, so a studio need not also hold jobs to watch what it started. When it succeeds, the file is a gallery item labelled with the sequence it came from.

Opening the editor

POST /timeline/{id}:open asks helmstudio to show a sequence in its editor. The launcher's own timeline screen is not built yet, so today it answers 501, under helmstudio and under helm dev alike. A studio hides its Open button when it does; the sequence is still made, edited and exported.

Known problems

Three things do not work as the design says, and the sample above avoids them: