Image resizing with LCD sub-pixel awareness (aka ClearType)

Discuss digital image processing techniques and algorithms. We encourage its application to ImageMagick but you can discuss any software solutions here.
rnbc
Posts: 109
Joined: 2010-04-11T18:27:46-07:00
Authentication code: 8675308

Image resizing with LCD sub-pixel awareness (aka ClearType)

Post by rnbc »

Hi!

I was testing how LCD sub-pixel awareness could enhance images when they are resampled to smaller sizes. This is like ClearType, but for photos. For that effect I used the following commands, you might find useful.

For normal resizing:

Code: Select all

convert input.tif -depth 16 -gamma 0.5 -filter Mitchell -resize 960x640\! -gamma 2.0 -depth 8 -quality 100 -sampling-factor 1x1 output_yyy.jpg
And considering the LCD as a silkscreen of RGB triplets with triple the horizontal resolution:

Code: Select all

convert input.tif -define showkernel=0 -depth 16 -gamma 0.5 -filter Mitchell -resize 2880x640\! -morphology Convolve '3x1: 0.0, 1.0, 0.0' -channel Red -morphology Convolve '3x1: 0, 0, 1' -channel Green -morphology Convolve '3x1: 0, 1, 0' -channel Blue -morphology Convolve '3x1: 1, 0, 0' +channel -filter Point -resize 960x640\! -gamma 2.0 -depth 8 -quality 100 -sampling-factor 1x1 output_rgb.jpg
This results in ugly aliasing in some images, due to the "Point" sampling. There you should either use some horizontal blur, changing the first matrix into '0.25, 0.5, 0.25', or use Mitchell instead of Point. The results are less sharp, but more tempered so to say. The test image Rings1.jpg renders beautifully with a full Mitchell pipeline, and delivers a ugly rainbow otherwise.

Normal photos seem to work better with Point, or Point with a bit of horizontal blurring.

Here is a sub-pixel-resize example, with Point and no blurring ...

Image

... and the same, with normal Mitchell resize.

Image

The difference is subtle, but if you put each image in a browser tab and switch between them it should be apparent. The sub-pixel resampled shows more detail.

Of course, it only works on LCDs (or Plasmas) when 1 pixel in the image equals 1 pixel in the screen, that is, only at 1:1. Otherwise the sub-pixel sampled image is actually worse.

User avatar
anthony
Posts: 8884
Joined: 2004-05-31T19:27:03-07:00
Authentication code: 8675308
Location: Brisbane, Australia

Re: Image resizing with LCD sub-pixel awareness (aka ClearTy

Post by anthony »

This has been discussed before in the forums (a long time ago!)

IM does not currently understand or use sub-pixel LCD rendering. It is also a patented technique making it difficult for any FOSS (free use, open source) program to use.
Anthony Thyssen -- Webmaster for ImageMagick Example Pages
https://imagemagick.org/Usage/

rnbc
Posts: 109
Joined: 2010-04-11T18:27:46-07:00
Authentication code: 8675308

Re: Image resizing with LCD sub-pixel awareness (aka ClearTy

Post by rnbc »

Well, I was not asking for any "support", since it's already supported with the use of current operators :) I was only discussing how to do it, and optimize it, within the current API ;)

BTW, I know it's patented, but that's a bunch of specific implementations, not the idea in general. Both Apple, Adobe, Microsoft and FreeType feature their own implementations, and there is no patent dispute I am aware of right now.

User avatar
anthony
Posts: 8884
Joined: 2004-05-31T19:27:03-07:00
Authentication code: 8675308
Location: Brisbane, Australia

Re: Image resizing with LCD sub-pixel awareness (aka ClearTy

Post by anthony »

Afetr re-organising your command so it is more readable (step by step)

Code: Select all

  convert input.tif -gamma 0.5 -filter Mitchell -resize 2880x640\! \ 
          -morphology Convolve '3x1: 0.0, 1.0, 0.0' \
          -channel Red -morphology Convolve '3x1: 0, 0, 1' \
          -channel Green -morphology Convolve '3x1: 0, 1, 0' \
          -channel Blue -morphology Convolve '3x1: 1, 0, 0' \
          +channel \ 
          -filter Point -resize 960x640\! -gamma 2.0 \ 
          -depth 8 -quality 100 -sampling-factor 1x1 output_rgb.jpg
The first convolve is useless, it does nothing. But you have it there so you can add horizontal blur, that is understandable.

The -depth 16 also is of no use, as it is only an input and output setting.

I would whoever save using PNG not JPEG. Even -quality 100 is lossy with JPEG, unless you use the J2 (JPEG2000) format specifically.

I see you are making correct use of gamma in the handling too. which is good to see, something that I would like to see more automatic in IMv7, (to look at in late stage development).

I am not certain exactly how it is working, but I don't think -filter point (whcih simply removes columns) is right. To me the sharpness may only be caused by the aliasing from using Point filter. But I would need to review exactly how LCD sub-sampling works to know for certain what that operation should be (it has been some time).

Do you have a link to the specific technique detail?

Untill I do review LCD sub-sampling that is all I can say for the moment.

Perhaps others would like to comment.
Anthony Thyssen -- Webmaster for ImageMagick Example Pages
https://imagemagick.org/Usage/

rnbc
Posts: 109
Joined: 2010-04-11T18:27:46-07:00
Authentication code: 8675308

Re: Image resizing with LCD sub-pixel awareness (aka ClearTy

Post by rnbc »

Hi anthony!

As you said the first convolve is there only to add horizontal blur if needed, otherwise it's useless.

Depth 16 is critical for gamma 0.5 followed by gamma 2.0, or otherwise ugly posterization will occur. 8 bits is far from enough for a transformation as aggressive as y=x^2 followed by y=sqrt(x). That is unless you use the hdri enabled version of imagemagick, which uses floating point. I presume most folks don't have hdri enabled (I have) so I included that.

I concur Point filter is not correct from a digital signal processing perspective. The effect is as if the LCD matrix was overlaid directly over an higher resolution image without any anti-alias filter, like a Bayer-array without anti-alias filter. It is well known that results in color moiré in some situations (basically fabrics with a sharp lens...), but also results in sharper output in most cases. For those borderline cases were it fails miserably I recommend either full Mitchell pipeline (theoretically correct, but only eliminates LCD color fringing without increasing apparent resolution) or a bit of horizontal blur (works in practice, and is sharper).

Please note that in the general case (theoretically correct, shall we say) the resolution is not increased, only the slight color fringing normally seen at sharp edges in LCD images is eliminated, resulting in a very subtle effect. That slight fringing is the result of not accounting for the fact that color channels are in average shifted 1/3 pixel in relation to each other in an LCD. Resolution cannot be increased, of course, since your screen still has the same number of pixels in the end, but by allowing moiré you can increase the apparent resolution in a controlled way, just as in a bayer array without anti alias filter does.

I would like to see others comment on this. I don't have a link to the specific technique detail since I came up with it myself yesterday. Of course I have previous generic knowledge in the field of image processing and practical digital photography, otherwise I couldn't come up with all this ideas and interpretations in such a short time.

Jason S
Posts: 103
Joined: 2010-12-14T19:42:12-07:00
Authentication code: 8675308

Re: Image resizing with LCD sub-pixel awareness (aka ClearTy

Post by Jason S »

Here's my writeup about a simple form of subpixel-aware resizing. It may not be the best you can do, but it was surprisingly easy to implement the "offset" feature that makes it possible.

User avatar
anthony
Posts: 8884
Joined: 2004-05-31T19:27:03-07:00
Authentication code: 8675308
Location: Brisbane, Australia

Re: Image resizing with LCD sub-pixel awareness (aka ClearTy

Post by anthony »

Depth 16 is critical for gamma 0.5 followed by gamma 2.0, or otherwise ugly posterization will occur.
But it is 16 bit by default -- in memory which is where it counts. The -depth does not change in memory bit quality, it is only for IO not for memory which is the compile time quality.

And thanks for the re-cap as to how LCD pixel sub-sampling works.

Hmm from my bookmarks from when I looked at the previously, these are my links..

Email detailing the method... (it starts with a scale up!)
http://lists.nongnu.org/archive/html/fr ... 00087.html

Freetype code for it...
http://git.savannah.gnu.org/cgit/freety ... ftlcdfil.c

From http://gidden.net/tom/2006/05/29/lcd-resampling/
from a Blog by Tom Gidden, on LCD sub-sampling wrote:Lossy compression breaks any results, leaving us with large images when JPEG would be more appropriate. Since the antialiasing manifests as a colour fringe, the lossy JPEG algorithm munges the detail completely. Worst still, the "extra" colour in the image can "stain" the normal JPEG artifacts
So I was right -- don't use JPEG for results!

Toms example 'testbed' page
http://gidden.net/junk/lcdresample/
provides a nice way of 'flipping' between photoshop resize and sub-pixel sampling resize. Looking at the sharp edge of the almost perfect vertical lines shows clearly the sharpening that it achieves. Makes a good testing image for this!

Though I somehow have the feeling that the photoshop resize did its resize linearly, without gamma being taken into account. But then IM does not take gamma into account by default either. Basically the edges of the photoshop resize just do not seem as 'smooth' as they should be!

And while it works well for the image, it fails badly for lines. Probably because Tom used various interpolated resize, rather than a better Mitchell resize filter.

Thanks Jason S for the pointer to your write up on this. It is better than anything I previously found.
Anthony Thyssen -- Webmaster for ImageMagick Example Pages
https://imagemagick.org/Usage/

rnbc
Posts: 109
Joined: 2010-04-11T18:27:46-07:00
Authentication code: 8675308

Re: Image resizing with LCD sub-pixel awareness (aka ClearTy

Post by rnbc »

Hello all!

Jason, your method, which I saw a few hours after starting experimenting myself, basically eliminates color fringing. I'm not so sure it increases perceived resolution. In fact it may decrease it a little bit. An improvement still over no-lcd-aware rendering imho.

Meanwhile I found a way to only use decent resize methods and keeping good sharpness, while still color-defringing. It's not as sharp at Point resampling in the end, but much more fool proof:

Code: Select all

convert SDIMa_24396.jpg -gamma 0.5 -filter Mitchell -resize 2700x1800\! -channel Red -morphology Convolve '3x1: 0, 0, 1' -channel Green -morphology Convolve '3x1: 0, 1, 0' -channel Blue -morphology Convolve '3x1: 1, 0, 0' +channel -filter Lanczos -resize 900x600\! -gamma 2.0 -depth 8 -quality 100 -sampling-factor 1x1 SDIMa_24396_rgb.jpg
Here is a result:

Image

Rationale for the changes:

You need a 3x larger image to use small & simple shift matrices, but Mitchell, which is a good theoretical window for sinc with little losses, does not deliver visually sharp output, so let's not fight it and let's use a 3x larger image on both axis instead of horizontal only. Too many pixels never hurt, cpu and memory being cheap. This allows us to later Lanczos for final downsampling, which keeps things far sharper and is still robust for all images I've thrown at it.

Concerning the use of JPEG:

Anthony, while JPEG basically sucks it's standard. In the past I did a study to evaluate JPEG adequacy for high quality image transmission, so I know a bit about this, and I can share some useful facts here.

In the study I used a few jpeg encoders, including ImageMagick and cjpeg. At 100% quality with no color subsampling you get rounding errors only, since all frequency components are transmited, which translates into some added noise.

This noise keeps signal averages and basically does not introduce any statistical bias, and this holds even when you lower quality. That's because of a specific property in DCT I can't recall now (linearity?...).

At 100% quality those errors generated noise at a similar level to the errors already introduced by using 8 bits quantization anyway. Summing both errors it's as if you got precision similar to using 7 bit sampling, but without the posterizing effects.

This noise (8 bit + jpeg_100) also happens to be similar (in terms of magnitude) to shot noise level with a sensor featuring a well depth of ~32k electrons if you encode with gamma 2.0. Analysis was simplified by the fact that the square root transform applied to poisson distributed countings (shot noise...) has the nice property of making noise a constant, since in those countings noise=sqrt(signal).

The noise introduced if also of a magnitude similar to the dithering needed to properly encode using 8 bits anyway.

To keep things simple, by using JPEG at 100% quality with no color subsampling you basically introduce a very subtle noise at a magnitude similar to noise already present in any properly encoded 8 bit image. It's also lower than noise already present in most photographic images. So it's adequate even for very high quality transmission. Don't bother, you won't be able to see the difference from a tiff or a PNG. Well... except maybe in a high brightness DICOM calibrated medical display, but even then I have sincere doubts.

When not to use JPEG: in intermediate steps of image editing, when you don't want to introduce errors at each saving of your work. You should probably also use 16 bit sampling, because otherwise you get similar errors by just using 8 bits.

As a footnote I may add that jpeg supports 12 bits per channel, but I've never seen it used that way. 12 bit / channel jpeg with gamma 2.0 encoding would be way better than any existing image sensor in terms of noise, featuring noise similar to a well depth of 8 million electrons, while even very large sensors with very large pixels have at most 1 million electrons or so.

rnbc
Posts: 109
Joined: 2010-04-11T18:27:46-07:00
Authentication code: 8675308

Re: Image resizing with LCD sub-pixel awareness (aka ClearTy

Post by rnbc »

It just occurred to me you can also combine both methods, Point and Lanczos, in the final step.

This is again not fool proof, requiring horizontal blur sometimes, so for general use, not requiring human evaluation of the results, I still think the method I stated in the previous post is the best.

This new one does deliver nice and sharp results with most images though.

Code:

Code: Select all

convert SDIMa_24396.jpg -gamma 0.5 -filter Mitchell -resize 2700x1800\! -channel Red -morphology Convolve '3x1: 0, 0, 1' -channel Green -morphology Convolve '3x1: 0, 1, 0' -channel Blue -morphology Convolve '3x1: 1, 0, 0' +channel -filter Lanczos -resize 2700x600\! -filter Point -resize 900x600\! -gamma 2.0 -depth 8 -quality 100 -sampling-factor 1x1 SDIMa_24396_rgb_3.jpg
Example:

Image

rnbc
Posts: 109
Joined: 2010-04-11T18:27:46-07:00
Authentication code: 8675308

Re: Image resizing with LCD sub-pixel awareness (aka ClearTy

Post by rnbc »

Even sharper, while accepting aliasing in both axis.

This aliasing can be controlled using blur and keeping the final ratio at 2:1 vertically and 3:1 horizontally. A different blur factor for each axis might be needed for best results. This must me tested by trial and error, since best results vary from image to image.

With this final method, and a bit of intermediate blur if needed, I presume we are getting quite close to the maximum detail per pixel you can express in a LCD.

Code:

Code: Select all

convert SDIMa_24396.jpg -gamma 0.5 -filter Mitchell -resize 2700x1200\! -channel Red -morphology Convolve '3x1: 0, 0, 1' -channel Green -morphology Convolve '3x1: 0, 1, 0' -channel Blue -morphology Convolve '3x1: 1, 0, 0' +channel -filter Point -resize 900x600\! -gamma 2.0 -depth 8 -quality 100 -sampling-factor 1x1 SDIMa_24396_rgb_4.jpg 
Image

SmudgeToolKitteh
Posts: 1
Joined: 2011-09-21T22:09:12-07:00
Authentication code: 8675308

Re: Image resizing with LCD sub-pixel awareness (aka ClearTy

Post by SmudgeToolKitteh »

Many apologies for the lateness of my response, but what may I ask has been patented? LCD rendering? Who actually has the patent on this? I figured that rendering was vague enough of a concept that no one could have "invented it."

Anyway, I digress. . .I'm very new to the world of photo retouching -- let alone LCD sub-pixel rendering. I'm technologically inept in some areas. Can someone help me out? Feel free to PM me. Thanks, all! :)
Dezi
Anyone know anything more about patent infringement? =)

rnbc
Posts: 109
Joined: 2010-04-11T18:27:46-07:00
Authentication code: 8675308

Re: Image resizing with LCD sub-pixel awareness (aka ClearTy

Post by rnbc »

SmudgeToolKitteh, quite frankly I've never seen this applied to images except for Jason S.

My method is different since it also accepts a controllable amount of aliasing (just use more or less gaussian blur, or a specific kernel for filtering in the intermediate steps), benefiting sharpness. It's also quite practical and easy to implement. ImageMagick already supports it, as you can see :)

I couldn't care less about patents... Otherwise I would have patented this myself :D

What has been patented is IIRC specifically Microsoft's ClearType font rendering. Other subpixel rendering technologies, like the ones used in the X server from x.org and FreeType, are available for free under an open source license, are not patented. Don't bother, just use it ;)

rnbc
Posts: 109
Joined: 2010-04-11T18:27:46-07:00
Authentication code: 8675308

Re: Image resizing with LCD sub-pixel awareness (aka ClearTy

Post by rnbc »

This guys describe something similar to what I'm doing, and was published in 2002.

http://ics.ele.tue.nl/~dehaan/pdf/90_jsid2003.pdf

Including the aliasing vs sharpness tradeoffs. Very interesting... I'm reinventing the wheel because someone forgot it long ago... I know of no programs actively doing this to enhance image output, except for fonts ans simple line art.

They also mention the real resolution does not improve, just the perception. Aliasing must be accepted for the sharpness to increase, and moves more into colors, away from luminance. Just as I said... oh well!

I don't know if there is any related patent.

towolf
Posts: 3
Joined: 2012-01-31T14:07:55-07:00
Authentication code: 8675308

Re: Image resizing with LCD sub-pixel awareness (aka ClearTy

Post by towolf »

Rodrigo, this is really sweet and works well. Very interesting!

Im a bit stymied now how I would convert this into a general purpose shell script without hard coding the sizes. Is this possible with a single command? I just want to give it the final size of the largest dimension, similar to "-resize 640".

rnbc
Posts: 109
Joined: 2010-04-11T18:27:46-07:00
Authentication code: 8675308

Re: Image resizing with LCD sub-pixel awareness (aka ClearTy

Post by rnbc »

towolf, it should be trivial to make a perl script invoking imagemagick (or using perlmagick) with the right parameters. Do you want me to do it? :)

Say, such a script could take screen dimensions and downsample an entire image collection for slideshow viewing with the best quality possible, fitting the images into the screen. This only makes sense at the output stage, or near output, since unless you are looking at images 1:1 it only makes things worse!

What I was thinking the other day is if this thing could be accelarated using OpenGL or something like that... making possible real time downsampling of ultra-high-definition-tv, or gaming... but I'm no expert in the relevant tecnologies.

Locked