I writing an application using OpenCV 2.2 under VC++. I am getting videos from different network streams and write frame by frame to an AVI file each in separate thread. The video streams are in the hundreds and my application is writing hundreds of files to disk which is very heavy. Can someone advise me the optimized way to do this?
Oh dear. I hope you have plenty of RAM.
Writing multiple files is a real pain. The best you can do is to mitigate the write seeks by always writing as large a chunk of AVI-frames, (preferably a multiple of sector size), as reasonably possible. Maybe:
1) A 'FrameBuf' frame-buffer class. Create a shitload of *FrameBuf at startup and pool them on a producer-consumer queue.
2) A 'FrameVec' container class for multiple *FrameBuf instances. You may need to pool these as well.
3) A threadpool for writing the contents of a *FrameVec to the disk system. This will contain very few threads, possibly only one, for best disk-write performance with few seeks. Best make the number of threads configurable/changeable at runtime to optimize overall throughput. Best make it all configurable - depth of *FrameBuf pool, number of *FrameBuf in each *FrameVec - everything.
If possible, use an SSD. If the system has any 'quiet' time, it could move the accumulated avi's to a big spinner, or networked disks, to free up the SSD for the next 'busy' time.
When moving your various instances about, remember these mantras:
'Stack-objects, copy ctors, any template class with no * bad', and 'pointers, pools, pointer containers good'.
Good luck..