Why is it that 5 separate command prompts crop 30 images faster on my PC than a single batch mogrify command on all 150?

Questions and postings pertaining to the usage of ImageMagick regardless of the interface. This includes the command-line utilities, as well as the C and C++ APIs. Usage questions are like "How do I use ImageMagick to create drop shadows?".
Locked
werneck
Posts: 2
Joined: 2020-03-11T07:41:16-07:00
Authentication code: 1152

Why is it that 5 separate command prompts crop 30 images faster on my PC than a single batch mogrify command on all 150?

Post by werneck »

Hi! I can't make sense of the behavior described in the title, and I thought a few of you might offer some insight into it.

I'm not sure if this is something expected and obvious, or weird and unusual - but given that I do a lot of batch operations and speed is an important factor to me, I thought I might at least try to find out why it happens, so that I'm better at coming up with faster solutions in the future. Please bear with me. 🐻


Here's some context:

I'm working on a tool that allows you to crop a video, but instead of showing the same section of the original video, the crop "moves" - taking a static video and generating something like a close-up shot of a subject.

In order to do so, I have the tool extract each frame of a video with FFMPEG as a BMP, crop a specific section of each frame individually, then stitch it back together into a video.

The thing is, since each crop command has different coordinates, I can't just do a batch mogrify using *.bmp - that would give me a static crop of a single region of the original video. The solution I found is to have my tool generate and run a batch file with one crop command for each frame, like this:

Code: Select all

magick mogrify -path cropped/ -crop 1920x1080+0+0 frame000.bmp
magick mogrify -path cropped/ -crop 1920x1080+0+1 frame001.bmp
magick mogrify -path cropped/ -crop 1920x1080+0+2 frame002.bmp
magick mogrify -path cropped/ -crop 1920x1080+0+3 frame003.bmp
magick mogrify -path cropped/ -crop 1920x1080+0+4 frame004.bmp
magick mogrify -path cropped/ -crop 1920x1080+0+5 frame005.bmp
magick mogrify -path cropped/ -crop 1920x1080+0+6 frame006.bmp
magick mogrify -path cropped/ -crop 1920x1080+0+7 frame007.bmp
magick mogrify -path cropped/ -crop 1920x1080+0+8 frame008.bmp
magick mogrify -path cropped/ -crop 1920x1080+0+9 frame009.bmp
magick mogrify -path cropped/ -crop 1920x1080+0+10 frame010.bmp
(In this example, I'm cropping a 1920x1920 video into 1920x1080, and having the crop section start at the top of the original video and move downward)



It takes about 19s to do the mogrify crop of each frame individually, using the method above. For comparison, a single batch mogrify on all 150 frames takes slightly less - about 16 seconds. It's a small difference, and one that makes sense to me.

But if I divide the 150 frames into 5 batch files with the individual crop instructions for 30 frames each, and run them all at the same time, I can process them all in under 8 seconds!

What I would like to know is: Why does this happen? Is there something silly I'm doing or failing to do? Maybe something to do with multithreading? Is it normal to be able to get much better speeds by running a few instances of the command prompt and processing multiple images at the same time? Am I going mad and probably measuring it wrong somehow? Should I just divide stuff up like this every time I do a batch operation to get faster speeds?

As you can see, I am very confused, hehe. Any insight is appreciated. Thanks!

werneck
Posts: 2
Joined: 2020-03-11T07:41:16-07:00
Authentication code: 1152

Re: Why is it that 5 separate command prompts crop 30 images faster on my PC than a single batch mogrify command on all

Post by werneck »

I said nothing about my system and the current version I'm using. Sorry.

I'm on Windows 10. I did a identify -version but I don't now what might be relevant, so here's all of it:
Image

snibgo
Posts: 13034
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Why is it that 5 separate command prompts crop 30 images faster on my PC than a single batch mogrify command on all

Post by snibgo »

When evaluating performance, it is helpful to see what CPU, RAM and disk resources are used. With Windows 7.1, I can run taskmgr and/or perfmon etc. Questions to ask are: are my CPUs 100% busy? If not, a different job configuration may give better throughput. Is RAM over-used? (If it is, there may be disk swapping.) Is disk over-used, so CPUs are having to wait? And so on. These metrics are indicators only. The critical metric is throughput: how many seconds does it take to process 150 frames, or whatever.

Starting and ending programs takes time, so instead of running magick 30 times for one frame each, it is always quicker to run magick one with a long command that reads one frame, process it, writes it, deletes it from the list and moves on to the next frame.

Your IM has OpenMP, so individual operations can use multiple threads. I often mention that this may not give the fastest throughput of multiple images, eg video frames. It may be better to "-limit thread 1" at each IM command, and run multiple commands simultaneously.

No multithreading is perfect. Resources are needed to start and stop threads, and coordinate results. Reading and writing files isn't parallelised, and nor are other housekeeping tasks such as shuffling metadata. Some pixel operations use GPU, which adds complications. IM won't multi-thread across images.

Running multiple commands simultaneously can do things that IM itself can't. For example, one command might be reading an image into memory (disk intensive) while another command is copying pixels (CPU intensive). The more commands you have, the more likely this is to happen. However, sadly, more commands are also more likely to compete for resources such as disk access.

Another hint: if your work is mostly on 8 bit/channel/pixel images, you might build a Q8 version of IM, and use that. If you do this, you might build it without OpenMP, and without HDRI.
snibgo's IM pages: im.snibgo.com

Locked