Heat map with lots of events

Discuss digital image processing techniques and algorithms. We encourage its application to ImageMagick but you can discuss any software solutions here.
User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Heat map with lots of events

Post by fmw42 »

snibgo wrote:

Code: Select all

convert ^
  heatmap.png ^
  -auto-level ^
  ( -size 1x250 ^
    gradient:#004-#804 ^
    gradient:#804-#f00 ^
    gradient:#f00-#f80 ^
    gradient:#f80-#ff0 ^
    gradient:#ff0-#ffa ^
    +append ^
  ) ^
  -clut ^
  heatmap_coloured.png

Just curious, but do not each gradient need to be placed at different locations via -geometry or -page and composited?

snibgo
Posts: 13034
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Heat map with lots of events

Post by snibgo »

Oops, that should be "-append", not "+". Thanks.

It gives quite a nice clut from dark blue to light yellow. It isn't quite perfect in the sense that #804 and the other transitions are repeated. If each segment was 51 pixels wide the clut image would look like this:
Image
We can see the small kinks. When each segment is 250 pixels wide, I don't think the kinks matter.
snibgo's IM pages: im.snibgo.com

DrEvil
Posts: 36
Joined: 2014-01-05T16:59:55-07:00
Authentication code: 6789

Re: Heat map with lots of events

Post by DrEvil »

Again thanks a lot. Is there a way to specify the geometry of the images based on their center or is it always upper left?

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Heat map with lots of events

Post by fmw42 »

DrEvil wrote:Again thanks a lot. Is there a way to specify the geometry of the images based on their center or is it always upper left?
If you use -gravity center -geometry ..., then if you are placing different gradients at different locations, it will position the centers of the gradients relative to the center of the background image.

Otherwise, the default is top left corner of the gradient relative to the top left corner of the background.

DrEvil
Posts: 36
Joined: 2014-01-05T16:59:55-07:00
Authentication code: 6789

Re: Heat map with lots of events

Post by DrEvil »

Is there a way to create gradients off of line strips in a way that doesn't over saturate the overlapping pixels of neighboring lines(such as if you did each line segments separately)?

snibgo
Posts: 13034
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Heat map with lots of events

Post by snibgo »

I don't understand the question. Do you mean the colours in the clut? My chart above shows the actual colours in the bottom strip. They can be any colours you want. My script has 5 gradients, but you can have any number you like, from one upwards. Each transition can be any colour you want.
snibgo's IM pages: im.snibgo.com

DrEvil
Posts: 36
Joined: 2014-01-05T16:59:55-07:00
Authentication code: 6789

Re: Heat map with lots of events

Post by DrEvil »

Suppose you have a series of line segments that make up a line strip of arbitrary length. The desire is to draw this line strip from point to point representing heat map events such as a persons locational history(where they moved around the environment, etc). Like the radial gradients, I would ideally be able to render these paths with a width and radial gradient falloff.

I would think that the way to accomplish this is to render the line sequence itself into an image with a gradient falloff 'away' from the line, as if rendering the line with a width but the width having a gradient falloff perpendicular to the line. It's that line gradient in particular that I'm asking about. Perhaps that would just be a polygon of the width of the event with a gradient along the polygon direction so that it falls off to the sides?

Drawing the lines appears pretty easy, the part I'm not sure of is the gradient falloff away from the lines based on a configurable width

User avatar
fmw42
Posts: 26383
Joined: 2007-07-02T17:14:51-07:00
Authentication code: 1152
Location: Sunnyvale, California, USA

Re: Heat map with lots of events

Post by fmw42 »

How wide are the lines. If only one pixel, then the best you can do is just anti-alias or linear blur the line, but the blurring would thicken the line. There is no way to make a one-pixel thick line have a gradient in the direction perpendicular to the line direction and keep it one pixel wide.

For blurring perpendicular to the line direction, see -morphology convolve blur:...

http://www.imagemagick.org/Usage/convolve/#blur

But that only works if your lines are horizontal or vertical. You have to choose the angle for -blur as 0 or 90 as appropriate to the direction of your line.

You can convert the gaussian shape to linear by replacing 0xsigma with radiusx65535

DrEvil
Posts: 36
Joined: 2014-01-05T16:59:55-07:00
Authentication code: 6789

Re: Heat map with lots of events

Post by DrEvil »

It seems that I'll need to generate a batch file and such with individual commands for each event to be accumulated into the heatmap prior to colorization? That's fine, but if they are individual commands, does that mean convert.exe has to open, parse, draw, save and close the file for each event? I'm concerned with the performance of that for event counts up in the tens of thousands. Is there a way to feed those commands into an instance of convert.exe that can avoid tends of thousands of basic I/O operations and batch all the event drawing into 1 opening of the file ? I may ultimately be processing large files with which the file IO of opening and closing it tens of thousands of times could become a significant performance bottleneck.

snibgo
Posts: 13034
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Heat map with lots of events

Post by snibgo »

You can create a gradient from a line by blurring it. For example, draw a white line on a black background and "-blur 0x15", perhaps followed by "-auto-level". This will blur in all directions, creating rounded ends. To blur just in the direction perpendicular to the line, you can rotate the line so it is horizontal, blur vertically, and rotate back.

What is your platform? Unix, Windows or what? As I said upthread, you can have one long command that processes all the events, or perhaps a hundred at a time. Unix (or possibly Cygwin on Windows) might give you other piping options.
snibgo's IM pages: im.snibgo.com

DrEvil
Posts: 36
Joined: 2014-01-05T16:59:55-07:00
Authentication code: 6789

Re: Heat map with lots of events

Post by DrEvil »

windows at the moment, but this may end up running on a web backend at some point

snibgo
Posts: 13034
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Heat map with lots of events

Post by snibgo »

On Windows, Powershell or Cygwin might be the geekiest options. I would just take the easy way: put the IM options for the thousands of events in an "at-file". This method is portable (as far as I know) across all environments.

For this method, create a file I will call heatmap.inc containing:

Code: Select all

  ( -size 100x100
    radial-gradient:
    -evaluate Multiply 0.25
  )
  -geometry +180+80
  -compose Plus -composite
  ( -size 100x100
    radial-gradient:
    -evaluate Multiply 0.50
  )
  -geometry +380+80
  -compose Plus -composite
I show just two events. Note there are no line continuation characters. Any percentage signs should not be doubled.

Then create heatmap.bat:

Code: Select all

convert ^
  -size 600x400 ^
  xc:black ^
  @heatmap.inc ^
  heatmap.png
snibgo's IM pages: im.snibgo.com

DrEvil
Posts: 36
Joined: 2014-01-05T16:59:55-07:00
Authentication code: 6789

Re: Heat map with lots of events

Post by DrEvil »

perfect. thanks

DrEvil
Posts: 36
Joined: 2014-01-05T16:59:55-07:00
Authentication code: 6789

Re: Heat map with lots of events

Post by DrEvil »

Ok I'm trying to get the floating point part to work properly and I'm not sure what I'm doing wrong. Specifically, the accumulation isn't working how I expect it to. In the inc file, the 2nd event is double in order to test the accumulation and normalization. The expectation is that overlapping events will accumulate their values and then the auto level will then level based on the new min/max values. The documentation for auto_level sounds correct. Perhaps I am not accumulating the values correctly? I'm trying to set it to use a 32 bit floating point format so that there really isn't a ceiling to the accumulation.

Here's my batch file

Code: Select all

@echo off

echo Blending Events into Accumulation Image
convert -depth 32 -define quantum:format=floating-point -size 1000x1000 xc:black @heatmap.inc heatmap.tif

echo Applying Colorization
convert heatmap.tif -auto-level ( ^
	-size 1x250 ^
	gradient:#004-#804 ^
	gradient:#804-#f00 ^
	gradient:#f00-#f80 ^
	gradient:#f80-#ff0 ^
	gradient:#ff0-#ffa ^
	-append ) -clut heatmap_coloured.png

pause
And here's my inc file

Code: Select all

( -size 100x100 radial-gradient:  ) -gravity center -geometry +300+300 -compose Plus -composite

( -size 100x100 radial-gradient:  ) -gravity center -geometry +180+80 -compose Plus -composite
( -size 100x100 radial-gradient:  ) -gravity center -geometry +180+80 -compose Plus -composite
The 2nd event is doubled to test the accumulation and leveling. The end result should be that the more you stack an event, you should see the intensity of other events diminish.

snibgo
Posts: 13034
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Heat map with lots of events

Post by snibgo »

Please put up the resulting image (eg put it on dropbox.com and paste the URL here) and say what it is wrong with it.

Incidentally, if you want "-gravity center" you can put that just once rather than repeating it for every event.
snibgo's IM pages: im.snibgo.com

Locked