PAID: Problem with IMagick, Huge temp files, LF a developer

Do you need consulting from ImageMagick experts and are willing to pay for their expertise? Or are you well versed in ImageMagick and offer paid consulting? If so, post here otherwise post elsewhere for free assistance.
AnavPT
Posts: 33
Joined: 2020-04-15T04:01:15-07:00
Authentication code: 1152

PAID: Problem with IMagick, Huge temp files, LF a developer

Post by AnavPT »

Problem:
https://imagemagick.org/discourse-serve ... 65#p174165

To recap:

I'm into run a stock photo site which does the resizing
I'm on a shared hosting environment with MDDHosting.
8000 x 8000+ pixel images are processed
EPS (encapsulated postscripts) also processed when a user purchases

Problems

- Huge temp files are created (per MDDHosting, a few of them is not a problem if they gets deleted) -- This problem happened while uploading, if everything goes on upload only resizing to thumbnails happens.
- Temp files aren't deleted
- ImageMagick eats up all the server resources. I'M on a smaller plan now, when I was on their biggest plan, it ate up those resoources too
- I like to have my site responsive when conversions happen, in every lets say 4-5 minutes - Lower IM's resource usage, while not let the process die.

I'm looking for paid help.

There has been already pointed out in the above thread where to look for a solution.

Please PM me or send your message here: supwphu gmail com

Thank you!

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: PAID: Problem with IMagick, Huge temp files, LF a developer

Post by fmw42 »

If huge temp files are left when uploading, then it has nothing to do with Imagemagick. Check your uploading software.

If those temp files are not deleted, then you need to consult with your hosting company about that or search solutions for your OS.

Temp files can be left from an Imagemagick command, if the system stops or you run out of temp space during the command. Be sure you have enough space in your /tmp directory.

AnavPT
Posts: 33
Joined: 2020-04-15T04:01:15-07:00
Authentication code: 1152

Re: PAID: Problem with IMagick, Huge temp files, LF a developer

Post by AnavPT »

I think the process (?system) stops. That's the problem IMO but I'm not sure or it uses disk space not RAM

My tmp folder had growth to 1.7 TB so I think it doesn't have a space limit.

Do a Google search, many ppl had this issue: "imagemagick temp files"

User avatar
magick
Site Admin
Posts: 11254
Joined: 2003-05-31T11:32:55-07:00

Re: PAID: Problem with IMagick, Huge temp files, LF a developer

Post by magick »

ImageMagick removes all temporary files when the process exits. It cannot remove them if it receives a signal that it cannot catch such as the dreaded OOM killer or if it receives a SIGBUS because there is not enough available space in the /tmp partition. That is why we encourage a security policy-- it not only keeps your environment secure-- it can help ensure ImageMagick always exits gracefully and properly cleans up its resources. Take a look @ https://imagemagick.org/script/security ... ynchronize. Enable this and a SIGBUS will never be issued. Set your memory resources so that all large images are cached to disk to ensure ImageMagick is not killed by OOM.

AnavPT
Posts: 33
Joined: 2020-04-15T04:01:15-07:00
Authentication code: 1152

Re: PAID: Problem with IMagick, Huge temp files, LF a developer

Post by AnavPT »

Well my hosting said they would limit to 1 GB max, and said every image created a 19 GB cache file in the TMP directory. So currently dont know where to look for a solution

User avatar
magick
Site Admin
Posts: 11254
Joined: 2003-05-31T11:32:55-07:00

Re: PAID: Problem with IMagick, Huge temp files, LF a developer

Post by magick »

1 GB max? We assume memory? Not a problem, ImageMagick can process the images on disk by setting a 1GB memory limit on your security policy. If your hosting company is uncooperative, find another. There are hundreds to choose from. Our online image web site @ https://imagemagick.org/MagickStudio/ is backed by ImageMagick and has been running flawlessly since 1995. The security policy we use is the one we recommend @ https://imagemagick.org/script/security-policy.php. We run on a 8GB VM with 250GB of disk and limit the upload image max dimensions to 8192x8192. We don't track statistics but the site has likely processed 10 of millions of images over the year with no monitoring or maintenance required.

AnavPT
Posts: 33
Joined: 2020-04-15T04:01:15-07:00
Authentication code: 1152

Re: PAID: Problem with IMagick, Huge temp files, LF a developer

Post by AnavPT »

Can you suggest a few? I tried Bluehost, Hawkhost, MDDHosting, and there one other I can't recall.

AnavPT
Posts: 33
Joined: 2020-04-15T04:01:15-07:00
Authentication code: 1152

Re: PAID: Problem with IMagick, Huge temp files, LF a developer

Post by AnavPT »

Maybe there is something with the software. Its called Symbiostock.

Also I create generative art and compared to others I have much greater file number and in addition I always saved 70-100 megapixel images wheres 15-20 MP is the standard.

Now the software should process all my graphics to date and it failed 4 times on diff hosting companies

User avatar
magick
Site Admin
Posts: 11254
Joined: 2003-05-31T11:32:55-07:00

Re: PAID: Problem with IMagick, Huge temp files, LF a developer

Post by magick »

We certainly cannot debug your workflow unless its provably related to ImageMagick. Perhaps you can discuss the problem with the Symbiostock developers. We can't recommend a hosting provider to you other than to say ImageMagick is hosted by Network Redux for over 10 years now and we've never encountered a problem. See https://imagemagick.org/script/support.php for a link.

AnavPT
Posts: 33
Joined: 2020-04-15T04:01:15-07:00
Authentication code: 1152

Re: PAID: Problem with IMagick, Huge temp files, LF a developer

Post by AnavPT »

This is what my hosting sent:

"
What exactly are you converting?
I am actually curious how ghost script is coming into play as this command I feel is important to your issues:

gs -q -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000 -dAlignToPixels=0 -dGridFitTT=2 -sDEVICE=pamcmyk32 -dTextAlphaBits=4 -dGraphicsAlphaBits=4 -r1500x1500 -g54187x37500 -sOutputFile=/tmp/magick-25269EjNy90NAfnnR%d -f/tmp/magick-25269ENWpWU313ARs -f/tmp/magick-25269UQuoD7BLHow4
"

Any idea what is this? .. this is what I noticed -g54187x37500 , I can't really understand the other parts

AnavPT
Posts: 33
Joined: 2020-04-15T04:01:15-07:00
Authentication code: 1152

Re: PAID: Problem with IMagick, Huge temp files, LF a developer

Post by AnavPT »

https://www.ghostscript.com/ ... Oh this is another software

User avatar
magick
Site Admin
Posts: 11254
Joined: 2003-05-31T11:32:55-07:00

Re: PAID: Problem with IMagick, Huge temp files, LF a developer

Post by magick »

ImageMagick leverages Ghostscript to render Postscript and PDF image files-- which suggests our theory is correct. Your large temporary image is likely an attempt to convert a multi-page (perhaps a hundred pages) Postscript or PDF that consumed all the resources on your system. Again, the security policy will prevent that from happening. You can also prevent it by only asking for the first page as we suggested, e.g. convert 'image.pdf[0]' image.png'

AnavPT
Posts: 33
Joined: 2020-04-15T04:01:15-07:00
Authentication code: 1152

Re: PAID: Problem with IMagick, Huge temp files, LF a developer

Post by AnavPT »

19 Gigabyte temp files are created in every 5 second as the cron job runs.

I checked the files and they are not multi-page

-rw------- 1 stockvec stockvec 19G Apr 14 22:30 magick-9752kE_tWUipraPF
-rw------- 1 stockvec stockvec 19G Apr 14 22:42 magick-19150DHZp5R1ZAx2A
-rw------- 1 stockvec stockvec 19G Apr 14 22:54 magick-27734ECwv7QpmqTcl
-rw------- 1 stockvec stockvec 19G Apr 14 23:06 magick-6277DkjPX0l6zKUV
-rw------- 1 stockvec stockvec 19G Apr 14 23:18 magick-16339OrOcaIR8AHu4
-rw------- 1 stockvec stockvec 19G Apr 14 23:30 magick-23769gVWWSVOYns6A
-rw------- 1 stockvec stockvec 19G Apr 14 23:42 magick-1556CCt1AayfCoo5
-rw------- 1 stockvec stockvec 19G Apr 14 23:54 magick-10632mH0LGhRkP8Jz

snibgo
Posts: 13034
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: PAID: Problem with IMagick, Huge temp files, LF a developer

Post by snibgo »

AnavPT wrote:-r1500x1500 -g54187x37500
On Ghostscript options, see https://www.ghostscript.com/doc/current/Use.htm#Options

Those are large numbers. A more typical "-r" option would be "150x150".

Please show the ImageMagick command, and input PDF, that creates that Ghostscript command.
snibgo's IM pages: im.snibgo.com

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: PAID: Problem with IMagick, Huge temp files, LF a developer

Post by fmw42 »

What is your version of Ghostscript? If old, perhaps upgrade it.

Locked