Everything that broke in an unattended video pipeline

Since early September 2026 a scheduled job on a Mac has rendered ten short videos a night for Tetty's Photo Studio, and since September 23 it has also scheduled three of them to TikTok. How it picks and cuts the footage is a separate post. This one is about running it.

The reason to automate a daily task is that nobody has to look at it. That is also the risk. When a person skips a day, they know. When a midnight job skips a day, the only evidence is a folder that is emptier than it should be. Every failure below was silent in that way.

1. A file that existed and could not be played

The source footage and the output folder both live on a shared Google Drive, mounted on the Mac. At first FFmpeg wrote the finished video straight into the Drive folder. On September 24 the batch finished normally, the posting step opened one of the new files, and FFmpeg could not read it. Drive had been syncing the file while it was still being written. A post was missed that night.

The fix was to stop trusting a successful exit code. The render now goes to local temporary storage, is checked, is copied to Drive, and is checked again. "Checked" means the file has a duration and its last second decodes.

    run(cmd)    if not video_ok(local_out):        raise SystemExit(f"render failed (broken file): {local_out}")    shutil.copyfile(local_out, out_silent)    if not video_ok(out_silent):   # retry once if the copy to Drive is incomplete        shutil.copyfile(local_out, out_silent)        if not video_ok(out_silent):            raise SystemExit(f"write to Drive was corrupted: {out_silent}")

2. Two nights with nothing at all

Drive for desktop does not keep every file on disk. A file can be "dataless": it has a name and a size, and the bytes arrive when something reads it. The pipeline deals with this by reading each chosen file from start to end before FFmpeg touches it.

On October 2 that read raised an operating-system error, "Resource deadlock avoided", on one of the files chosen for the first video. The exception was not caught. The generator exited with zero videos. The checker then reported zero files and zero broken, which is true and useless, and the posting step said there was nothing to post. The whole run took one minute. The same thing happened the next night.

The cause was a design assumption: that a file listed in the folder can be read. The fix has three parts. Reads are retried with doubling waits. A file that still fails becomes a typed exception, not a crash. And the caller treats that as "this material is unavailable tonight": it removes the file from the pool and draws the video again.

def warm_file(path, retries=5):    """Pull a Google Drive placeholder file down to disk (reading it is enough).    Drive sometimes does not answer just after midnight, so retry up to 5 times,    waiting 3/6/12/24/48 seconds."""    import time    if not is_dataless(path):        return True    last = None    for attempt in range(retries):        try:            with open(path, "rb") as f:                while f.read(8 << 20):                    pass            return True        except OSError as e:            last = e            time.sleep(3 * (2 ** attempt))    raise FileNotReady(f"{os.path.basename(path)}: {last}")

        bad = None        for x in chosen + photos:            try:                warm_file(x)            except FileNotReady as e:                bad = x; break        if bad is not None:            if bad in clip_pool: clip_pool.remove(bad)            if bad in photo_pool: photo_pool.remove(bad)            redraws += 1            if redraws > 20:                raise SystemExit("material keeps failing to download; check the Drive connection")            k -= 1; continue

3. The fix had a cost I did not measure

The same commit added a step that downloads new files before generation starts. On October 10 that step found 45 files it could not read, and so did the analysis pass after it. The night still produced ten valid videos and scheduled its posts, which is the behaviour I wanted. It also took an hour and fifty minutes, and the next night did the same with the same 45 files. Five other nights in the same log finished in six to nine minutes.

The arithmetic explains most of it. A file that never becomes readable costs 45 seconds of waiting in the pre-download step and 93 seconds in the analysis pass, and the files are tried one after another. Forty-five of them is about 100 minutes. I have not profiled the run, so this is a calculation that matches the log, not a measurement. The retry policy is right for one stuck file and wrong for forty-five. It needs a budget for the whole night, not per file.

4. A learning loop that was reading stale numbers

The pipeline is meant to learn. A sync script reads seven-day results for the studio's Instagram reels from our analytics database, links each posted reel to the generated file it came from, and after 30 linked results it starts weighting footage by how its videos performed.

On September 27 I asked why it had never linked a single reel. There were two independent reasons. The Instagram connection in the analytics dashboard had been failing every day since September 6 with an expired token, so no new numbers had arrived for three weeks, and the sync script had been reporting the old ones without complaint. And the reels in that data had been posted by hand with the studio's standard caption, not the generated one, so matching by caption could not work: 0 of 236.

Two changes came out of that. The script now prints a warning with the age of the newest number and the stored connection error whenever the data is more than two days old. And videos posted through the pipeline's own Instagram path are linked by their exact post URL, with caption matching kept as a fallback. The first is a fix. The second only matters once that path is switched on, and it is still off.

The outage also had a cost that cannot be repaired. Once the connection was restored, the newer reels arrived and 22 of them matched generated captions. But a seven-day result needs a reading taken between day 4 and day 14 after posting, and for those reels that window fell inside the outage. In the October 11 log the count is still 22 linked, 0 with a seven-day result.

5. Sorting timestamps as text

One rule says a clip or photo may appear in at most one of any six consecutive TikTok posts. To apply it, the code takes the last five posts in order of scheduled time. Scheduled posts come back from the posting service in UTC with a Z suffix. Posts sent immediately were recorded locally in Japan time with +09:00. Sorted as strings, a post at 00:30 on the 28th in Japan sorts after one at 16:00Z on the 27th, although it happened half an hour earlier. The window could contain the wrong five posts.

class MixedTimezoneOrderTest(unittest.TestCase):    def test_orders_by_real_time_not_string(self):        # 2026-09-28T00:30+09:00 is 2026-09-27T15:30Z: earlier than 16:00Z, but later as a string        posted = {"posts": [post("early", "2026-09-28T00:30:00+09:00", ["early"]),                            post("late", "2026-09-27T16:00:00.000Z", ["late"])]}        self.assertEqual(cd.recent_asset_sets(posted, window=2), [{"late"}])

6. A filter that hid two thirds of the photos

Source material was limited to files modified in the last 120 days, to keep the first analysis run short. That is reasonable for clips. For photos it is wrong: the date on a file says nothing about whether the photo is good. Only 38 of 116 retouched photos were ever eligible, and the same faces came up day after day. Photos now have no date filter. Repeats are controlled by the usage weights and the six-post rule, which is what those were built for.

What the fixes have in common

• Check the result, not the exit code. A finished process and a playable file are different facts.
• Degrade, do not abort. One missing file should cost one redraw, not the night. In the nightly script a failed step does not stop the steps after it.
• When a number is old, say so next to the number. A stale figure presented as current is worse than no figure.
• Write the rule as a test. The timezone bug and the banned-wording rules are both pinned by tests now; the suite has 74.

What is still not fixed

There is no alert. Every failure above was found by reading a log or noticing a missing post, and that is still how the next one will be found. The whole system also depends on one Mac. It is set to wake itself five minutes before midnight, but if it is off, nothing runs and nothing says so. Those two gaps are the same gap, and closing it is the next piece of work.

An unattended job needs one more output than the work it does: a way to tell you it did not happen.