Joining Clips from Cameras & Recorders

Some cameras and external recorders use older versions of filesystems which have file size limitations that are a bit out of data for todays content. To work around that, they split a clip into what is sometimes a huge number of subclips. One that I regularly use is the Odyssey 7Q+ in combination with the Sony FS7 camera in 4K mode. At that resolution it can only store 40s of footage per file before it has to create a new subclip.

The app that comes from Convergent Design has the ability to re-join these clips into a single file during copy. But today's workflows often rely on other data copy apps like ShotPut Pro, Hedge, and others that make exactl replicate of the camera cards and create transfer logs and checksums.

Importing these subclips into the NLE is cumbersome unless the NLE recognizes subclips and virtually joins them. Resolve is not one of them. It has the ability to manually create a compound clip, but then the source timecode is lost, which creates a new set of problems.

For a while I had been using EditReady to join files among other meta data operations. But the latest version has a small race condition that can kill the software if it's run in pass-through mode.

When dealing with another 1.5TB of footage from yesterday's shoot I was determined to find a more efficient solution and so I figured the right settings for ffmpeg out. ffmpeg is quite fast because it also supports the options of passthrough (i.e. not re-encoding the video, but just rewrapping it in a new container). 

The result is the following short python script. It's run in the folder where the clip exists, creates a subclip list and then runs ffmpeg to create a new joined clip at the path indicated by the first argument. On my older MacPro it runs at about 3x speed, or about 300MB/s which is decent enough.

Not a super polished script, still has hardcoded paths and doesn't check commandline argument errors. But you get the gist :-)


import os
import glob
import sys

print "Joining clips in current folder"

filelist = open("files.txt","w")
clips = list(glob.glob(os.path.join('.','*.mov')))
for clip in clips:
  filelist.write("file '%s'\n" % os.path.basename(clip))

cmd = "/Users/janklier/Desktop/AllKlier\ Data/Tools/ffmpeg -f concat -i files.txt -c copy " + sys.argv[1]


Importing Odyssey 7Q+ Metadata into Avid Media Composer

I've been using the Odyssey 7Q+ as an external recorder for quite some time and with much success. The reasons for using it are a separate conversation for another time.

One of the things though that has always bothered me that the 7Q+ has menu options to record a variety of meta data for each take, such as Reel #, Scene, good/bad take, etc. But none of these data get stored into the ProRes files the recorder produces. Instead these data get saved into an FCP 7 XML file on a per clip basis in a subfolder on the SSD. That's allo good if you use an NLE where you can import these XML files. I don't have FCP itself. I do work with Resolve, Avid, and Premiere in that order. The only one of these that can interpret the XML files from the recorder is Premeiere. Resolve complains that the files do not contain a timeline (as they're per clip files). And Avid doesn't read them at all, since Avid has standardized around the ALE file format.

After a bit of coding I came up with a Python script that can ingest a folder of Odyssey 7Q+ XML files and translate them into a single ALE file that Avid consumes happily.

I've successfully used the script with the latest version of the 7Q+ firmware, the 3.0 version of the transfer utility, and MC 8.9.x. I've only tested this on a small set of files, so further refinement for special cases and debugging may be in order. But for anyone willing to give it a try and report back any success or issues, there's a link for the Python script at the end of the article. This is written for Phyton 3 which is readily available for Mac or Windows.

To use the script, run the script and point it at the FCP 7 XML folder after the media has been transferred with the CD utility:

cd /Volumes/Jobs/TestJob/Camera\ Files/Card\ 001
/usr/local/bin/python3 /Volumes/Workspace/ALE/fcp_to_ale.py -d fcp\ 7\ xml/

This should process all the XML files for each clip and then producer one CD.ALE file. The syntax for ale files is described in the MC documentation.

It will translate the following fields:

  • In and Out mark if set in the Odyssey Play mode for the clip
  • Good/Bad flag for the clip
  • Description (derived from the Project field)
  • Camera
  • Reel #
  • Scene #
  • Take #
  • Shoot Day
  • LUT Name

Then on the Avid side, first import all the .mov files through the usual means (linking them to a bin in the source browser).

In a second step, select all the clips in the bin, go back to the source browser, select 'import' rather than 'link', and in the import options, go to the 'Shot Log' tab and select 'Merge events with known master clips'. Then select the newly create ALE file and import. This will read the ALE file and merge any new meta data with the selected clips that where just imported previously.

Here is my test ALE file:

FPS	23.98

Name	Tracks	Start	End	Tape ID	Source File	Source Path	Description	Comments	Camera	Reel	Scene	Take	Shoot Day	LUT	Mark IN	Mark OUT	Good

CLIP0000001	VA1A2	00:00:00:01	00:00:07:23		CLIP0000001.mov	/Volumes/Workspace/ALE/Test Data	1674    		A	R001	S1      	008	001	View-SONY_EE_SL3C_L709A-1			
CLIP0000002	VA1A2	00:00:07:19	00:00:31:16		CLIP0000002.mov	/Volumes/Workspace/ALE/Test Data	1674    		A	R001	S1      	009	001	View-SONY_EE_SL3C_L709A-1			
CLIP0000003	VA1A2	00:00:31:12	00:04:43:09		CLIP0000003.mov	/Volumes/Workspace/ALE/Test Data	1674    		A	R001	S1      	010	001	View-SONY_EE_SL3C_L709A-1			
CLIP0000004	VA1A2	00:04:43:05	00:09:43:22		CLIP0000004.mov	/Volumes/Workspace/ALE/Test Data	1674    		A	R001	S1      	011	001	View-SONY_EE_SL3C_L709A-1			
CLIP0000005	VA1A2	00:09:43:18	00:10:07:03		CLIP0000005.mov	/Volumes/Workspace/ALE/Test Data	1674    		A	R001	S1      	012	001	View-SONY_EE_SL3C_L709A-1			
CLIP0000006	VA1A2	00:10:06:23	00:11:08:11		CLIP0000006.mov	/Volumes/Workspace/ALE/Test Data	1674    		A	R001	S1      	013	001	View-SONY_EE_SL3C_L709A-1			
CLIP0000007	VA1A2	00:11:08:07	00:12:10:00		CLIP0000007.mov	/Volumes/Workspace/ALE/Test Data	1674    		A	R001	S1      	014	001	View-SONY_EE_SL3C_L709A-1			
CLIP0000008	VA1A2	00:12:09:21	00:12:33:13		CLIP0000008.mov	/Volumes/Workspace/ALE/Test Data	1674    		A	R001	S1      	015	001	View-SONY_EE_SL3C_L709A-1			
CLIP0000009	VA1A2	00:12:33:09	00:13:28:02		CLIP0000009.mov	/Volumes/Workspace/ALE/Test Data	1674    		A	R001	S1      	016	001	View-SONY_EE_SL3C_L709A-1			
CLIP0000010	VA1A2	00:13:27:22	00:14:18:06		CLIP0000010.mov	/Volumes/Workspace/ALE/Test Data	1674    		A	R001	S1      	017	001	View-SONY_EE_SL3C_L709A-1	00:13:43:00	00:13:54:02	
CLIP0000011	VA1A2	00:14:18:03	00:15:09:14		CLIP0000011.mov	/Volumes/Workspace/ALE/Test Data	1674    	FEATURE	A	R001	S1      	018	001	View-SONY_EE_SL3C_L709A-1			yes
CLIP0000012	VA1A2	00:15:09:10	00:17:15:06		CLIP0000012.mov	/Volumes/Workspace/ALE/Test Data	1674    		A	R001	S1      	019	001	View-SONY_EE_SL3C_L709A-1			
CLIP0000013	VA1A2	00:17:15:02	00:18:10:16		CLIP0000013.mov	/Volumes/Workspace/ALE/Test Data	1674    		A	R001	S1      	020	001	View-SONY_EE_SL3C_L709A-1			

And this is how the bin looks after a successful meta data import of this file:

Link to the Python script: https://drive.google.com/open?id=1h95nnBZ_RquNODVa54o8u_z8TZjtpG-Z