video asset deduplication: separate identical bytes from similar creative work
Video asset deduplication has at least two meanings. Exact-byte deduplication finds files with identical contents. Creative similarity review identifies clips that look alike but may have different frames, edits, or approvals. Keep those operations separate so storage cleanup does not erase useful production history.
Genace creates and tracks generation jobs. Your application owns collected files, duplicate detection, storage references, and editorial grouping. Deduplicating stored media is not the same as making a generation POST idempotent or undoing the cost of an already accepted job.
Hash verified collected files
Collect the complete output and validate the video before computing its digest. A partial transfer has its own bytes but is not a valid archive candidate. Use a streaming hash so a large file does not need to fit in one memory buffer.
import { createHash } from 'node:crypto';
import { createReadStream } from 'node:fs';
const digest = createHash('sha256');
for await (const chunk of createReadStream('validated-video.mp4')) {
digest.update(chunk);
}
console.log(digest.digest('hex'));This uses Node.js’s documented hashing interface. Store the digest with file length and media inspection results in your asset manifest.
Preserve distinct relationships
| Relationship | Can share stored bytes? | Must remain distinct? |
|---|---|---|
| Two downloads of one output | Usually yes | Collection attempts |
| Two jobs with identical output | Possibly | Job and accounting history |
| Different encodings of one edit | Not byte-identical | Encoding derivatives |
| Similar scenes | Not automatically | Creative candidates and approvals |
| One file used in two campaigns | Possibly | Usage and publication records |
An identical file does not imply identical permissions. Keep access rules and customer ownership on the usage records rather than treating the content digest as public authorization.
Handle visual similarity cautiously
A perceptual comparison can help reviewers find near-duplicates, but it should not silently delete one. A small crop, corrected label, or single repaired frame can be exactly the reason a new derivative was created.
Group similar candidates for human review and retain the reviewer decision. State whether the goal is reducing redundant options, finding a preferred edit, or cleaning storage. Those goals require different deletion or archival choices.
Make cleanup reference-aware
Before removing a stored object, check all source, derivative, approval, and publication references. An unused working duplicate may be safe to retire under your policy, while an identical object still referenced by another campaign must remain available.
Keep a recoverable record of the cleanup decision. If the application consolidates two storage objects, update their references transactionally or through a verified migration process so no published link points to a missing file.
Do not trigger another generation when duplicate collection is detected. Reuse the verified stored bytes where authorization permits, preserving the separate production records.
Acceptance checklist
Test exact duplicates, re-encoded files, visually similar alternatives, and a shared object used by two campaigns. Verify that cleanup preserves approvals and active publication references. Confirm that generation accounting remains attached to every job regardless of storage consolidation.
See content-hash caching, duplicate requests, and archive policy.
Continue with media delivery
Where these facts come from
- codebase: src/ai/providers/types.ts
- codebase: src/ai/jobs/http.ts
- https://nodejs.org/api/crypto.html#cryptocreatehashalgorithm-options
- Application-owned workflow recommendations; human review and post-processing are outside the generation gateway
- Editorial keyword hypothesis; search volume not measured