How to detect defective images automatically (bash script)?

Questions and postings pertaining to the usage of ImageMagick regardless of the interface. This includes the command-line utilities, as well as the C and C++ APIs. Usage questions are like "How do I use ImageMagick to create drop shadows?".
Locked
MostlyHarmless
Posts: 3
Joined: 2020-03-22T02:19:03-07:00
Authentication code: 1152

How to detect defective images automatically (bash script)?

Post by MostlyHarmless »

I have tens of GB of images in directories and sub-directories.
Unfortunately, some of them seem to be defective:

* some can not be opened at all, example file (10 MB): I had to upload the image as a zip archive: https://file.io/tTFYug

* some are displayed with a more or less large rectangular grey area (see example **screenshot** below) and GIMP says "premature end of jpg file"
Image

here is the original image (12 MB):
https://ibb.co/5hjNsxq

I want to replace the corrupted files by a backup, but I do _not_ want to replace the files which are ok. Sorting them out by hand would be a lot of work.

So I'd like to generate a bash script which would be able to go through all sub-directories of a path and list the paths and names of the defective files.

I found this code in another [discussion][2], however it only finds those images which can not be opened at all, but not e. g. DSC_0127.jpg :

for f in *.JPG ; do identify $f > /dev/null || echo $f >> /tmp/fail ; done ; cat /tmp/fail

How could I also find the ones like DSC_0127.jpg with the grey rectangle?



[1]: https://i.stack.imgur.com/hP5pJ.png
[2]: https://stackoverflow.com/a/4780731/772434

snibgo
Posts: 13034
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: How to detect defective images automatically (bash script)?

Post by snibgo »

1. Check whether IM says the file is corrupt. The text is sent to stderr.

Code: Select all

> magick DSC-0127.jpg info:

magick: Premature end of JPEG file `DSC-0127.jpg' @ warning/jpeg.c/JPEGWarningHandler/399.
magick: Corrupt JPEG data: premature end of data segment `DSC-0127.jpg' @ warning/jpeg.c/JPEGWarningHandler/399.
DSC-0127.jpg JPEG 3872x2592 3872x2592+0+0 8-bit sRGB 3092480B 0.703u 0:00.834
2. To check for areas of flat colour (Windows CMD syntax):

Code: Select all

magick hP5pJ.png -alpha off -statistic standardDeviation 5x5 -threshold 0 -scale "1x1^!" -format %[fx:mean] info:
The result is a number between 0.0 and 1.0. It is the proportion of the image that has "ordinary photographic" variation, so should be close to 1.0. If the result is only 0.8 then this means 20% if the image is flat colour.
snibgo's IM pages: im.snibgo.com

MostlyHarmless
Posts: 3
Joined: 2020-03-22T02:19:03-07:00
Authentication code: 1152

Re: How to detect defective images automatically (bash script)?

Post by MostlyHarmless »

Hi snibgo,
thank you very much - this looks extremely promising and could save me hours of clicking...

However I have some problems here due to outdated hard- and software.
On my MacBook with OS X 10.11.6, I have installed IM with homebrew and got the following version:
convert -version
Version: ImageMagick 6.9.5-10 Q16 x86_64 2016-09-21

On the PC where the defective images are saved, Windows XP is running and in Cygwin I have an even older version of IM installed.

The first command works fine, but if I try the second with "convert", then I get an error message:

Code: Select all

convert DSC_0127.JPG -alpha off -statistic standardDeviation 5x5 -threshold 0 -scale "1x1^!" -format %[fx:mean] info
-bash: !": event not found

snibgo
Posts: 13034
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: How to detect defective images automatically (bash script)?

Post by snibgo »

Sorry, I should have said (now corrected) the command I showed is Windows CMD syntax. "!" is a special character to Windows, so I escaped it with "^". For bash, don't escape it.
snibgo's IM pages: im.snibgo.com

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: How to detect defective images automatically (bash script)?

Post by fmw42 »

You have very old versions and really should update all of them.

MostlyHarmless
Posts: 3
Joined: 2020-03-22T02:19:03-07:00
Authentication code: 1152

Re: How to detect defective images automatically (bash script)?

Post by MostlyHarmless »

@snibgo: thanks for explaining!

@fmw42: I would like to, but I did not find an easy way to install the current version on MacOS X 10.11.6.
I installed it with homebrew today and the result was 6.9.5 (or did I do something wrong?)

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: How to detect defective images automatically (bash script)?

Post by fmw42 »

You should be able to install IM 7 with Homebrew. Here is what I do: I install all my delegates with MacPorts and then install IM from sources. See https://www.imagemagick.org/discourse-s ... =1&t=29100

Locked