Hand off to the timeline
Sequencing is not one studio's job. A sequence cut from a video studio's take, another studio's clip, a still and a voice line is none of those studios' documents, so the timeline belongs to helmstudio: studios hand it clips, and it keeps the sequence, its revisions and its exports. A studio never renders a track or runs ffmpeg itself.
This studio holds timeline, assets and gallery:
"""Put outputs in a sequence the framework owns, then export it.
Run under helmstudio or helm dev by a studio holding `timeline`, `assets` and
`gallery`. Exporting needs ffmpeg on the machine running helmstudio.
"""
import os
import struct
import time
import wave
import zlib
from helm_runtime_sdk import from_env
helm = from_env()
def still(name, rgb, width=640, height=360):
"""Adopt a single-colour still, standing in for a generated frame."""
row = b"\x00" + bytes(rgb) * width
def chunk(kind, data):
body = kind + data
return struct.pack(">I", len(data)) + body + struct.pack(">I", zlib.crc32(body) & 0xFFFFFFFF)
png = (b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0))
+ chunk(b"IDAT", zlib.compress(row * height)) + chunk(b"IEND", b""))
path = os.path.join(os.environ["HELM_STAGE_DIR"], name)
with open(path, "wb") as f:
f.write(png)
return helm.assets.adopt({"path": path, "kind": "image"})
def tone(name, seconds=3, rate=48000):
"""Adopt a short sound: silence, standing in for a generated voice line."""
path = os.path.join(os.environ["HELM_STAGE_DIR"], name)
with wave.open(path, "wb") as w:
w.setnchannels(1)
w.setsampwidth(2)
w.setframerate(rate)
w.writeframes(b"\x00\x00" * rate * seconds)
# A sound's length is a hint the studio gives; an export measures it again.
return helm.assets.adopt({"path": path, "kind": "audio", "duration_s": float(seconds)})
dawn = still("dawn.png", (230, 160, 90))
dusk = still("dusk.png", (60, 50, 120))
# A sequence is a document the framework keeps, not a file the studio writes.
# Clips given without positions are laid end to end, video and stills on V1 and
# sound on A1; a still is held.
sequence = helm.timeline.create({
"name": "a day at the lighthouse",
"target": {"width": 640, "height": 360, "fps": 24},
"clips": [{"asset_id": dawn["id"], "hold": 2}, {"asset_id": dusk["id"], "hold": 2}],
})
assert sequence["duration_s"] == 4.0, sequence["duration_s"]
# Adding to the end of a track needs no etag, so a studio can hand things over
# as it makes them. It takes video and sound; a still needs a hold, which
# append has no way to give.
voice = tone("voice.wav")
sequence = helm.timeline.append({"asset_id": voice["id"], "timeline_id": sequence["id"]})
assert [t["name"] for t in sequence["tracks"]] == ["V1", "A1"], sequence["tracks"]
# Every other edit is a merge patch against the revision read, and each one is
# a revision that can be reverted to. Here, the voice comes down 6 dB.
tracks = sequence["tracks"]
tracks[1]["clips"][0]["gain_db"] = -6
edited = helm.timeline.update(sequence["id"], {"tracks": tracks}, if_match=sequence["etag"])
assert edited["revision"] == sequence["revision"] + 1, edited["revision"]
# Undo is writing the earlier revision back, as the newest one.
reverted = helm.timeline.revert(edited["id"], {"revision": sequence["revision"]}, if_match=edited["etag"])
assert "gain_db" not in reverted["tracks"][1]["clips"][0], reverted["tracks"][1]
sequence = helm.timeline.update(reverted["id"], {"tracks": tracks}, if_match=reverted["etag"])
# The plan says whether the picture can be copied or must be re-encoded, and
# why. A still is always drawn, so this sequence is re-encoded.
plan = helm.timeline.plan(sequence["id"], preset="h264")
assert plan["mode"] == "conform", plan
job = helm.timeline.export(sequence["id"], {"preset": "h264"})
while True:
job = next(j for j in helm.timeline.exports(sequence["id"])["items"] if j["id"] == job["id"])
if job["state"] not in ("queued", "running"):
break
time.sleep(0.2)
assert job["state"] == "succeeded", job
# The export is an item in the gallery, labelled with the sequence it came from.
exported = helm.gallery.query(kind="video")
assert any(i.get("timeline_id") == sequence["id"] for i in exported["items"]), exported
print("ok")
Exporting runs ffmpeg on the machine running helmstudio, found through HELM_FFMPEG or on the PATH. helmstudio does not bundle it yet.
A sequence is a document
- The target. Width, height and a frame rate: 23.976, 24, 25, 29.97, 30, 48, 50, 59.94 or 60. Every time in the document lands on a whole frame of the target, and a sound's trim on a whole sample, so two editors never cut in different places.
- The tracks. One video track,
V1, contiguous from zero, and up to eight audio tracks,A1onwards. A video clip's own sound plays under it unless it saysaudio: false. A still is held for itshold, and never trimmed. Clips on a track never overlap, and sound never runs past the picture. - Creating one. Clips given without positions are laid end to end: video and stills on
V1, sound onA1.
Edits are revisions
append adds an asset to the end of a track, and needs no etag, so a studio can hand things over as it makes them. Every other edit is a merge patch against the revision you read, passed as if_match. Every edit keeps the document it replaced, and reverting writes an earlier revision forward as the newest one, so undoing an undo loses nothing.
Sources are never modified or copied: a clip is a reference with in and out points. An asset a sequence uses counts as used, so reclaiming disk never offers to delete footage a sequence needs.
Export says what it will do
plan says, before anything runs, whether the picture can be copied — every clip already matches the target and the others, and is used whole — or must be conformed, and names the reason for each clip that must be. A still is always drawn, so a sequence with one is conformed. The sound is always re-encoded.
An export is a job. Its progress and its cancellation are under timeline, so a studio need not also hold jobs to watch what it started. When it succeeds, the file is a gallery item labelled with the sequence it came from.
Opening the editor
POST /timeline/{id}:open asks helmstudio to show a sequence in its editor. The launcher's timeline screen ships with the Mac app, so today it answers 501, under helmstudio and under helm dev alike. A studio hides its Open button when it does; the sequence is still made, edited and exported.
Known problems
Three things do not work as the design says, and the sample above avoids them:
appendcannot add a still. A still needs a hold, andappendhas no field for one, so it is refused. Add stills withcreate, or withupdate.- A dissolve short of handle is accepted. A dissolve is centred on the cut and takes half its length from each side's material beyond the cut. It should be refused when either side has too little; today it is not.
- A dissolve into a still exports short. The exported file is shorter than the sequence: in the case found, four seconds of timeline exported as 3.75 seconds of video.