Free solution for evaluating lens sharpness and AF fine tuning

[fvdbergh]

Yes, I am promoting my own software, but there is no easy way to do this, so here goes ...

Looking for tool to perform quantitative lens sharpness measurements? Not quite willing to spend between $300 and $5000 for Imatest? Then you should give MTF Mapper a try:

http://sourceforge.net/projects/mtfmapper/

The tool is free, and the source code is provided under a BSD license.

The blurb on the sourceforge page covers most of the interesting points, but I would like to emphasise the following for the MyBB audience:

1. MTF Mapper allows you to perform AF fine tuning (or micro-adjust in Canon jargon) in a systematic, quantitative way. Just take some photos of the included test charts, and use MTF Mapper to automatically calculate where the plane of sharpest focus was relative to the intended point of focus. (Full procedure is detailed in the user documentation).
2. You can measure things like focus plane shift on stopping a lens down.
3. You can visualise the sharpness of your lens as it varies across the focal plane. This is a great tool for studying the effects of stopping a lens down on image sharpness across the whole image.

I would appreciate any feedback and comments. I am also committed to adding features and fixing bugs that may pop up.

 

[Edduck]

one word, AWESOME! This is some piece of programming!!! I just downloaded it and I will do a test tomorrow. I want put this on my photoblog, can I? At what stage is the program? Do you think it is ready for some international coverage? I can contact the people over at Petapixel.com and see if I can get some exposure for you!

 

[fvdbergh]

one word, AWESOME! This is some piece of programming!!! I just downloaded it and I will do a test tomorrow. I want put this on my photoblog, can I? At what stage is the program? Do you think it is ready for some international coverage? I can contact the people over at Petapixel.com and see if I can get some exposure for you!

Thanks! You are welcome to spread the word in any way you see fit.

The program is still in beta, but all features are implemented, and work on my machines
It can definitely do with some more independent testing, though, which justifies the "Beta" tag.

I would certainly appreciate international coverage, but I would prefer that you test the software a bit first, and let me know if you think it is ready for said coverage.

My primary concern is with my documentation, though. I am not quite sure if I have explained things well enough for other people to be able to use the software. Feedback in this regard will be appreciated.

 

[hilton]

Does the camera have to support MA or not?

 

[fvdbergh]

Does the camera have to support MA or not?

The software does not directly interact with the camera. This means that it will work with any camera, even those that do not support MA. If your camera does not support MA, you will only be able to measure whether your lens/ body combination require adjustment, which is still useful to know, though.

 

[hilton]

I load a couple of *.cr2 files.

Then when I try and click on them in the right panel I get a message bleating about "Cannot load IMG_4393.tiff"

I'm running Windows 7 64bit.

 

[fvdbergh]

I load a couple of *.cr2 files.

Then when I try and click on them in the right panel I get a message bleating about "Cannot load IMG_4393.tiff"

I'm running Windows 7 64bit.

Firstly, thanks for trying the software.

It sounds like the GUI (mtf_mapper_gui) could not successfully execute dcraw.exe.
Quick fix:
Go to settings, and click the "browse" button under the "dcraw executable" label. Locate "dcraw.exe" --- there should be a copy in <path-to-mtf-mapper>/dcraw.

Long explanation:
When the MTF Mapper GUI runs for the first time it will set up the locations of these helper programs (gnuplot, dcraw and exiv2). These settings are saved in the registry (yuck!). If you subsequently move/rename the MTF Mapper folder, these paths will be incorrect. The settings are persistent to allow the user to specify custom versions of these tools (typically something a Linux user would have to do until exiv2 binary releases catch up with the development branches), but I realize now that moving the MTF Mapper install directory is likely to be a common thing under Windows.

I have thought of a way to fix this, and I will put it into the next release.

 

[hilton]

I tried that but it's very slow. I found that if the folder name has any spaces in it, the program complains bitterly so I changed it to just MTF and then re-assigned the links to the three programs.

However I still can't get it to do anything other than show me a picture in black and white.

I can't upload images here or else I could show you what I'm doing.

I'm happy to take this off-line if you need further testing.

 

[fvdbergh]

I'm happy to take this off-line if you need further testing.

Ok. I've PM'ed you my email address.

 

[fvdbergh]

I found that if the folder name has any spaces in it, the program complains bitterly so I changed it to just MTF and then re-assigned the links to the three programs.

Hilton, many thanks for finding that bug. Turns out my code did not support spaces in any file/path names. Whoops.

I have uploaded 0.4.8 that addresses this problem.

 

[hilton]

Thanks Frans, I will check it out and send you screen shots.

 

[mic_y]

hey man, thanks for this... thanks to your tool. My only suggestion would be to include a bit of a paragraph on the command line options for generating the test chart.

Thanks to your tool, I now have evidence of the fact that my newly purchased 35mm f1.8 is consistently front-focusing (even after having set the adjustment to +20)... so will need to get that swapped out...

 

[fvdbergh]

My only suggestion would be to include a bit of a paragraph on the command line options for generating the test chart.

Thanks to your tool, I now have evidence of the fact that my newly purchased 35mm f1.8 is consistently front-focusing (even after having set the adjustment to +20)... so will need to get that swapped out...

Thanks for the feedback. I have been thinking of a way to include the Unix man pages in the user manual.

Anyhow, really glad you found the software useful. Many thanks for giving it a try!

 

[hilton]

Frans, seeing that you're still online, I'll give the program another shot tomorrow.

 

[mic_y]

Thanks for the feedback. I have been thinking of a way to include the Unix man pages in the user manual.

Anyhow, really glad you found the software useful. Many thanks for giving it a try!

and another suggestion from my dad (dont know how possible this would be) but possibly to include the option of using multiple photos for data points. I was using 16 shots (8 from inf-actual focus, 8 from min-actual focus) and the graph generated by the individual photos was quite different (more so for the 18-200mm than the 35mm). All the shots were taken using a remote release with the camera on a tripod, so they were fairly consistent set-up wise. Maybe this could be rectified by using a number of shots as input, and averaging the values?

Any case, once again, thanks for the great tool.

 

[fvdbergh]

and another suggestion from my dad (dont know how possible this would be) but possibly to include the option of using multiple photos for data points. I was using 16 shots (8 from inf-actual focus, 8 from min-actual focus) and the graph generated by the individual photos was quite different (more so for the 18-200mm than the 35mm). All the shots were taken using a remote release with the camera on a tripod, so they were fairly consistent set-up wise. Maybe this could be rectified by using a number of shots as input, and averaging the values?

The variability that you observe shot-to-shot is due to several factors.

Firstly, although MTF mapper is reasonably consistent, you may see variations of around 5% in individual readings (exact same edge in the "annotated.png" views) owing to image noise. This will improve with averaging, since the noise is supposed to be random, independent, and zero-mean.

The second source of variability is AF repeatability. Is sounds as if you took 8 shots starting from each focus limit, thus effectively 8 samples of AF for a given set-up. I am somewhat hesitant to average these results automatically, but it is certainly possible.

In the meantime, you can use temporary files created by MTF Mapper (specifically, "profile.txt" and "profile_peak.txt") directly to compare these different shots (see figure 11 on p16 of the user docs). You will have to either learn the gnuplot syntax (see "profile.gnuplot" in same place as "profile.txt" for inspiration), or you can pull the data into Excel, or whatever plotting tool you prefer. I have had a request to add the option to save a set of results, but in the meantime you can simply grab the relevant files from the temp directory while MTF Mapper is still open.

For some reason I personally prefer to average the profile plots "by eye", as illustrated in figure 11, but maybe a median plot would produce clean, consistent results. I will certainly look into this.

A metric that can be averaged is the "peak_shift" value reported by the "mtf_mapper" command line tool. The GUI does not currently display this value, but it does appear in a console that opens on my Win7 box. Personally, I actually script the command line tool to extract this value The only problem is that the command line tool does not directly support raw files, so you must do the raw conversion in a script too.
The "peak_shift" value is simply the difference between the position of sharpest focus and position of expected focus, i.e., a direct measurement of the focus error.

I have been toying with the idea of adding "projects" to the GUI, which will create a framework for aggregating the results over multiple images. Once I have this implemented, it should be fairly simple to compute such averaged results. I am slightly worried that this might complicate the user interface, so I will have to experiment a bit.

 

[mic_y]

Afternoon Frans,

thanks for the quick reply. So far, the "by eye" method is what I have been using...

another question that I have is what the height of the blue bar in the graph means (if anything)...?

 

[fvdbergh]

Afternoon Frans,

thanks for the quick reply. So far, the "by eye" method is what I have been using...

another question that I have is what the height of the blue bar in the graph means (if anything)...?

The height of the blue bar represents the MTF50 value across the reference edge (the one you focused on). I have found this value to be unreliable, since the longer edges tend to be affected negatively by lens distortion. Even by limiting the MTF calculation to a length of 200 pixels around the centre of long edges does not appear to be enough to avoid the effects of lens distortion. I am considering just scaling the blue bar to the height of the peak ...

 

[fvdbergh]

I tried that but it's very slow.

@Hilton: I have uploaded a new version on sourceforge, with a complete re-implementation of the core sampling algorithm. The new algorithm is twice as fast, at least in terms of CPU time. Actual runtime gains will depend on the number of CPU cores, since some of the other more expensive steps are not easily parallelizeable. On some of my 16MP test shots, this means wall-clock time is now down to 3-4 seconds on an 8-thread Sandy Bridge machine.

Even better news is that the I have implemented further refinements, and accuracy has improved by about 20%, with far fewer outliers.

All in all, thanks for motivating me to go and work on that part of the code again.

 

[hilton]

The one on the site is dated 7 December 2011.

I tried that version.