Heat map with lots of events

Discuss digital image processing techniques and algorithms. We encourage its application to ImageMagick but you can discuss any software solutions here.
snibgo
Posts: 13034
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Heat map with lots of events

Post by snibgo »

The important message is the first one. Don't worry about warnings that it can't files in your home directory.

I suspect you haven't adjusted your script for v7, which isn't documented yet. Use "-script" instead of "@". Try something like this:

Code: Select all

%IMGF%magick -script heatmap.mgk
Where heatmap.mgk is:

Code: Select all

# This is a script file for ImageMagick v7.
# IM ignores lines beginning with #.
# Blank lines are ignored.
# As IM directly reads this file, there is no line-continuation character etc.
# Unlike v6, we need to "-write" the final image, then "-exit".

# Execute this script with the command:
#   %IMGF%magick -script heatmap.mgk
# where %IMGF% is the directory of magick.exe, and this file is named "heatmap.mgk".

# See also http://www.imagemagick.org/Usage/bugs/IMv7_Scripting.txt

# The output file is PNG, so it is 16-bit integer.
# Intermediate processing is HDRI (32-bit float).

# To write the output as 32-bit float, replace last two lines with:
#   -depth 32 -define quantum:format=floating-point
#   -write heatmap_coloured.tiff
#   -exit

-size 1000x1000
xc:black
( -size 100x100 radial-gradient:  ) -gravity center -geometry +300+300 -compose Plus -composite
( -size 100x100 radial-gradient:  ) -gravity center -geometry +180+80 -compose Plus -composite
( -size 100x100 radial-gradient:  ) -gravity center -geometry +180+80 -compose Plus -composite
-auto-level
 (
   -size 1x250
   gradient:#004-#804
   gradient:#804-#f00
   gradient:#f00-#f80
   gradient:#f80-#ff0
   gradient:#ff0-#ffa
   -append
 ) -clut
-write heatmap_coloured.png
-exit
In theory you can move options out of the script and into the main command. I don't know if we can call scripts from scripts (yet).
snibgo's IM pages: im.snibgo.com

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Heat map with lots of events

Post by fmw42 »

I was under the impression that IM 7 still supports "convert". Were you having trouble using that rather than "magick"?

You may want to review http://imagemagick.org/script/porting.php#cli for changes relative to IM 6.

DrEvil
Posts: 36
Joined: 2014-01-05T16:59:55-07:00
Authentication code: 6789

Re: Heat map with lots of events

Post by DrEvil »

Still doesn't work.

Installed from ImageMagick-7.0.0-0-Q16-x64-dll.exe

Code: Select all

C:\Users\DrEvil\Desktop\heatmap>magick -script heatmap.mgk
magick: unable to open image 'black': No such file or directory @ error/blob.c/OpenBlob/2635.
magick: unable to open module file 'C:\Users\DrEvil\.magick\IM_MOD_RL_XC_.dll': No such file or directory @ warning/modu
le.c/GetMagickModulePath/816.
magick: no decode delegate for this image format `black' @ error/constitute.c/ReadImage/558.
If I get rid of the xc line it doesn't like the radial gradient lines with the : at the end before the parenthesis. If I remove that and run it I get no errors but it never finishes.

Code: Select all

C:\Users\DrEvil\Desktop\heatmap>magick -version
Version: ImageMagick 7.0.0-0 Q16 x64 2014-01-09 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2014 ImageMagick Studio LLC
Features: DPC Modules HDRI OpenMP
Delegates: bzlib cairo freetype jbig jng jp2 jpeg lcms lqr pangocairo png ps rsvg tiff webp xml zlib

DrEvil
Posts: 36
Joined: 2014-01-05T16:59:55-07:00
Authentication code: 6789

Re: Heat map with lots of events

Post by DrEvil »

There was an option to install 'legacy' apps for convert but I didn't as I figured magick was the proper way to do it.

I can eliminate the errors with the following script

Code: Select all

# This is a script file for ImageMagick v7.
# IM ignores lines beginning with #.
# Blank lines are ignored.
# As IM directly reads this file, there is no line-continuation character etc.
# Unlike v6, we need to "-write" the final image, then "-exit".

# Execute this script with the command:
#   %IMGF%magick -script heatmap.mgk
# where %IMGF% is the directory of magick.exe, and this file is named "heatmap.mgk".

# See also http://www.imagemagick.org/Usage/bugs/IMv7_Scripting.txt

# The output file is PNG, so it is 16-bit integer.
# Intermediate processing is HDRI (32-bit float).

# To write the output as 32-bit float, replace last two lines with:
#   -depth 32 -define quantum:format=floating-point
#   -write heatmap_coloured.tiff
#   -exit

-gravity center

-print 'Generating Image Buffer\n'
-size 1000x1000

-print 'Rendering Gradient 1\n'
( -size 100x100 -radial-gradient ) -geometry +300+300 -compose Plus -composite
-print 'Rendering Gradient 2\n'
( -size 100x100 -radial-gradient ) -geometry +180+80 -compose Plus -composite
-print 'Rendering Gradient 3\n'
( -size 100x100 -radial-gradient ) -geometry +180+80 -compose Plus -composite

-print 'Colorizing Heatmap\n'
-auto-level
(
	-size 1x250
	gradient:#004-#804
	gradient:#804-#f00
	gradient:#f00-#f80
	gradient:#f80-#ff0
	gradient:#ff0-#ffa
	-append
) -clut

-print 'Writing Heatmap\n'
-write heatmap_coloured.png
-exit
Note it appears to expect a - before the radial-gradient

Even though that eliminates the error(such as unbalanced parenthesis(even tho the dash doesnt have anything obvious to do with the parenths), the script still doesn't work, and appears to hang forever at the first gradient.

Code: Select all

C:\Users\DrEvil\Desktop\heatmap>magick -script heatmap.mgk
Generating Image Buffer
Rendering Gradient 1
I don't see much help on the porting page. Seems pretty broken at the moment. I guess I'll go back to version 6

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Heat map with lots of events

Post by fmw42 »

Note it appears to expect a - before the radial-gradient
That is odd. I have not heard of that changing for IM 7.

grayscale gradients and radial-gradients in IM 7 will have only one channel, whereas in IM 6 they have 3 channels though all the same. There could be some command that does not understand the single channel images.

These issues should be brought up in the Bugs forum, if real.

I think it is risky using IM 7 (alpha). You would be better off using IM 6 in HDRI mode, if you can compile from source (along with any delegates you may need).

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Heat map with lots of events

Post by fmw42 »

-radial-gradient
Did you try

radial-gradient:

with the colon (:) at the end?

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Heat map with lots of events

Post by fmw42 »

You could try reinstalling in legacy mode and use convert rather than magick and see if that works better. But then the script likely needs to change and calling it with @scriptfile. Scripting in IM 7 is very new. You may be better using legacy mode with convert and using IM 6 style bat files as you had tried in IM 6 earlier.

I am a Mac person, so know little about Windows scripting.

DrEvil
Posts: 36
Joined: 2014-01-05T16:59:55-07:00
Authentication code: 6789

Re: Heat map with lots of events

Post by DrEvil »

Yes and it still hangs. Without the - in front of radial-gradient it errors about parenthetis

Code: Select all

magick: unbalanced parenthesis `(eof)' in "heatmap.mgk" at line 27,column 17 @ error/magick-cli.c/ProcessScriptOptions/272.
I posted a thread under bugs.


As an aside to this problem. When a script does a long series of images, as I am doing with these gradients, are each one of these gradients being pushed onto the 'image stack' ? If so then presumably that is why I am running out of memory with huge event sets. If so, is there a way to pop the image stack after I combine each gradient into the main image so as not to creep up the memory usage and error out?

DrEvil
Posts: 36
Joined: 2014-01-05T16:59:55-07:00
Authentication code: 6789

Re: Heat map with lots of events

Post by DrEvil »

The old legacy convert is just as broken. Back to v6 it is.

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Heat map with lots of events

Post by fmw42 »

DrEvil wrote:
As an aside to this problem. When a script does a long series of images, as I am doing with these gradients, are each one of these gradients being pushed onto the 'image stack' ? If so then presumably that is why I am running out of memory with huge event sets. If so, is there a way to pop the image stack after I combine each gradient into the main image so as not to creep up the memory usage and error out?
No, not that I know. The only way is to run a loop over convert or magick for each gradient. But that means you have to read the background each time and then save it. IM 7 may be able to do better memory management with -script, but I have not real experience with doing so and do not really know if it can. Anthony would be the one who could answer that as he wrote that code.

DrEvil
Posts: 36
Joined: 2014-01-05T16:59:55-07:00
Authentication code: 6789

Re: Heat map with lots of events

Post by DrEvil »

Surely the library wouldn't be set up so that every single little image fragment is maintained in memory while a script is running? At least not without a mechanism to delete some manually at least. I would like to not have to jump through hoops partitioning my data when I'm processing large data sets. I should just be able to generate a gradient, add it to the large image, throw that gradient away and generate the next as long as necessary.

Now that I finally got 7.0 working by copying DLLs into the main folder I can test it out in the huge event set.

The huge data set I was attempting also generated an almost 2GB tif file as the intermediate file. The final colorized png was still pretty small since most of the events were condensed and there was still lots of background color and such.

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Heat map with lots of events

Post by fmw42 »

fmw42 wrote:
DrEvil wrote:
As an aside to this problem. When a script does a long series of images, as I am doing with these gradients, are each one of these gradients being pushed onto the 'image stack' ? If so then presumably that is why I am running out of memory with huge event sets. If so, is there a way to pop the image stack after I combine each gradient into the main image so as not to creep up the memory usage and error out?
No, not that I know. The only way is to run a loop over convert or magick for each gradient. But that means you have to read the background each time and then save it. IM 7 may be able to do better memory management with -script, but I have not real experience with doing so and do not really know if it can. Anthony would be the one who could answer that as he wrote that code.
I think I may have been wrong about this. Since each gradient is composited, it should be relinquished. But I am not really an expert on memory management.

Perhaps a better approach would be to create one gradient: and write to MPR (in memory format) rather than creating it over and over again. Then use the mpr image in each composite.

-size 1000x1000
xc:black
-size 100x100 gradient -write mpr:gradient +delete
mpr:gradient -gravity center -geometry +300+300 -compose Plus -composite
mpr:gradient -gravity center -geometry +180+80 -compose Plus -composite
mpr:gradient -gravity center -geometry +180+80 -compose Plus -composite
.
.
.
-auto-level

rest of your code

See http://www.imagemagick.org/Usage/files/#mpr


If you are still running out of memory, then look at
http://www.imagemagick.org/Usage/files/#massive
http://www.imagemagick.org/script/archi ... tera-pixel
http://www.imagemagick.org/script/resou ... nvironment


Perhaps also ?
http://www.imagemagick.org/Usage/files/#image_streams

DrEvil
Posts: 36
Joined: 2014-01-05T16:59:55-07:00
Authentication code: 6789

Re: Heat map with lots of events

Post by DrEvil »

Is it possible to generate the gradients with a more direct control over the numeric values of the colors? When I generate the huge event streams, there is a ton of duplicate events that end up mapping to the same pixel, like thousands of them, so it would be a big boon to processing and reduction of event count if I could merge them before processing them. So if I had 5000 events of the same radius on the same pixel value, I would like to merge them if possible, but that is only possible if I can generate a radius with a configurable numeric color values. Instead of 5000 1.0-0.0 gradient values, I need to do 1 gradient of 5000-0 color values. Is this possible?

*EDIT - would this accomplish what I'm talking about? Would this allow multiplication without clamping?

Code: Select all

-evaluate Multiply 5000.0

DrEvil
Posts: 36
Joined: 2014-01-05T16:59:55-07:00
Authentication code: 6789

Re: Heat map with lots of events

Post by DrEvil »

The multiply appears to work for what I'm talking about.

snibgo
Posts: 13034
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Heat map with lots of events

Post by snibgo »

Yes, that's how to do it.
snibgo's IM pages: im.snibgo.com

Locked