Meet the Ladies!
By Dave Etchells
(Updated Indoor/Sunlit test subjects; May, 2008)
| More about our testing |
| New Indoor/Outdoor Portrait Tests (5/23/2008) |
| New Daylight Simulator (5/23/2008) |
| "Sunlit" Portrait - What to look for (5/23/2008) |
| Indoor Portrait - What to look for (5/23/2008) |
| Still Life and Multi Targets (Updated 11/29/2010) |
| HMI Studio Lighting (5/22/2006) |
| Performance Timing (5/22/2006) |
| New Review Format (5/22/2006) |
Longtime readers will be well-familiar with Dave's wife Marti, the subject of countless thousands of test shots on the Imaging Resource website, over the last 10+ years. After a long and fabled career though, Marti is finally retiring from the modeling business. In her place, we'll be using two lifelike mannequins, fitted with human hair wigs. At the same time, we've also upgraded the Daylight Simulator light source, used for our "sunlit" test shots. The result is our new "INB" and "OUTB" test subjects: Everyone please welcome Lauren and Jody!
| New Indoor Test Scene | New "Sunlit" Test Scene |
 |
 |
| Meet "Lauren," our new model for the Indoor test shot. She's not much on personality, but at least seems interested in digital photography. (That's a digital photo primer by O'Reilley she's reading.) | "Jody" is our new model for the simulated sunlight test. She seems sweet, but doesn't say much. At least she looks a little happier than Lauren does. |
In setting up the new test scenes, we considered a number of issues, both pro and con. Here's a brief look at some of the things we took into consideration in arriving at the new test setups:
Key things that are the same or similar to the prior shots
"Sunlit" scene: Contrast Ratio
While it can be useful for looking at other things, the primary intent of the "Sunlit" scene has always been to show how cameras handle very contrasty lighting. Even relative novices know that full sunlight is a lighting condition to be avoided if at all possible. But it often isn't possible, and even in many well-controlled shooting situations, a camera's ability to handle a wide range of subject brightness is critical to achieving the best results. While we've upgraded some aspects of the light source we use for this shot, the contrast ratio between primary and fill illumination is very similar to that which we've used in the past. The main/fill ratio is slightly greater (about a half stop), to better match typical conditions under harsh noonday sunlight, but it's overall quite close. The new scene includes a bit more texture in the model's blouse, so it should act as a better test-bed for evaluating the handling of highlight detail. At the shadow end of the scale, we've tweaked the arrangement of the flowers and the distribution of fill lighting to provide a good range of shadow values that will give a good idea of just how well cameras do with preserving detail and managing noise in deep shadow areas.
Indoor scene: Color Temperature
Where the Sunlit test is mostly about contrast and dynamic range, the primary purpose of the Indoor scene has always been simply to evaluate how well digital cameras handle the incandescent lighting that's so common in US homes. This is very warm-hued lighting, with color temperatures in the range of 2,700K. (Color temperature varies with wattage, lower-wattage bulbs generally producing lower Kelvin temperatures. In the home environment, the use of lampshades often produces a further shift toward lower Kelvin numbers.) From the beginning of our testing in 1998 through the present day, many cameras have had a very difficult time handling this strong-hued light source. In setting up the new scene, we used incandescent bulbs typical of those found in many US homes. For the sake of convenience, in some cases we used higher wattage bulbs, but then dialed down the voltage to them to bring the color temperature back down to the 2,700K that matches the light on our old set, which primarily came from 60-watt light bulbs.
Indoor scene: Subtle hair detail for evaluating noise-reduction impact
While the indoor scene was primarily set up to check white-balance capability, it turned out that it was also an excellent place to look for the effects of noise-reduction processing. The warm-hued light source presents cameras with a worst-case scenario for image noise, as it requires the gain on the (already noisy) blue to be cranked up to get back to a neutral white balance. At the same time, we've found that the fine detail and subtle gradations in human hair present camera noise suppression systems with a particularly difficult challenge.
Cameras that deliver excellent, finely-rendered detail on more contrasty subjects often fall completely apart when faced with human hair as a subject. This is somewhat inherent in the processing that they're doing: Noise reduction systems generally look at the amount of contrast in a local area, flattening it out if it's below a given threshold. This tends to preserve detail along high-contrast edges, so the images will still appear sharp to the unschooled eye. At the same time, this approach cleans up image noise in large areas of flat tint (the sky, for example), where it would be particularly evident.
In human hair though, there tends to be a lot of fine detail present, characterized by relatively subtle contrast between the elements. As a result, noise reduction algorithms often mistake this subtle subject detail for noise, and the results of their efforts often look more like impressionist paintings than digital photographs.
In our new test scenes, both models have human-hair wigs, carefully dyed and bleached (by Marti's hair stylist) to produce the sort of subtle tonal variation you've seen in Marti's hair over the years. Since this was only done once (with second wigs having been prepared at the same time as safety backups), the model's hair will be exactly the same from shot to shot. (A few strands may get shifted a bit one way or the other, but the overall pattern, color, and texture will remain absolutely consistent from shot to shot and camera to camera.)
Improvements over the prior test shots
Much more consistent over time
The lighting on these shots has always been very carefully controlled, so there's always been essentially no variation from shot to shot or camera to camera in that respect. Being a human though, Marti has always been subject to change. Over the long term, time respects no person, and Marti's not an exception. There have thus been numerous changes in Marti's appearance over the years that simply weren't avoidable. While we've tried to maintain her hair color more or less consistent over time, every visit to the hair stylist has meant minor variations. What's more, on a daily basis (and even from shot to shot), her pose has been slightly different, making it difficult or impossible to find exactly comparable points in images captured with different cameras. Even things like skin color and texture changed over time and with the seasons and Marti's level of exposure to the sun. Lauren and Jody have no such issues: Bottom line, the new test shots will be much more consistent over the long haul.
Better daylight simulation
It seems much of our history in testing cameras has been tied up in finding the best light sources to photograph under. Finding a good approximation of "daylight" has been particularly vexing. Full details will be the subject of another article, but suffice to say that natural daylight covers an incredibly wide range of color temperatures, spectra, and contrast ratios. In the "Sunlit" shot, we've tried to simulate noon sunlight as best we can, taking into consideration fill illumination from the sky and surrounding objects.
In this latest round, we've taken our light source up another notch, with better sky simulation, somewhat more collimated "sunlight", and more control of ambient light in the studio, both in the form of inadvertent spill from the light source, as well as light coming from elsewhere in the studio. While our light source still isn't as well-collimated as sunlight, and the "sky" still doesn't wrap around from horizon to horizon, the overall simulation is even better than we've had in the past.
Slightly more contrast and "shape" in the indoor lighting
Indoor lighting is even more widely variable than daylight, so there's no real standard to pursue here. We did feel though, that our previous light setup was a little too "flat" (too even) in its illumination. This new setup maintains the same color temperature of roughly 2600-2700 K, but is less evenly distributed, giving the model's features more shape, and providing a bit more contrast within the scene. The way the model's hair falls across her forehead also provides a wider range of micro-contrast, a help in evaluating camera noise reduction processing. (In a happy accident, we also found that the texture of the cloth in her green jacket is exceptionally revealing of noise reduction processing artifacts.)
Scenes constructed to ease shooting, particularly focusing
This is of less consequence to our readers, but it sure makes life easier in the lab. We built the new test scenes with a clear focus target dead center in each of them, positioned to be in the same plane as critical detail we use in our evaluations. For the Indoor shot, the target is obvious: The alignment reticle held aloft on a thick wire in the middle of the scene. The focus target on the Sunlit shot is less obvious: It's the brooch hanging from the model's neck.
In each case, the focus target is positioned so it's in the same plane as critical detail elsewhere in the frame. In the case of the Indoor scene, the plane of focus also contains the area of the model's forehead where we look for noise processing artifacts, and the label of the wine bottle, a good reference for fine detail. On the Sunlit shot, we fastened a couple of pieces of half-inch foamcore behind the brooch, to bring it into the same plane as the model's eyes and forehead.
The net result is that we no longer have to aim the camera at a critical focal point, half-press the shutter button, and then reframe the image before shooting. This greatly speeds the testing process. An ancillary benefit is that the clear focus target in the Indoor shot eliminates all manner of problems we had in the past with getting cameras (especially SLRs) to focus on Marti's hair. With its low-contrast and very fine detail, Marti's hair proved to be an ongoing focus challenge for a great many cameras. With the new targets, focus lock is much easier to achieve.
Minor front- or back-focusing is also much easier to deal with. In the case of the Indoor shot, we can move the focus target forward or back as much as a couple of inches as needed, to insure that objects in the plane of desired focus are indeed critically focused. In the case of the Sunlit shot, the inanimate model lets us switch to manual focus after focusing automatically, at which point we can move the camera front or back by up to a couple of inches, using a macro slider to effect the change.
(A note here about front/back focusing on SLRs: One could argue that we should show the test images "warts and all", including possible effects due to focus error. We refrain from doing this for two reasons. First, for the sake of maximum consistency from camera to camera, we're shooting these tests with Sigma 70mm f/2.8 Macro lenses. This lens design is one of the sharpest we've tested to date, so goes about as far as we're able toward taking the lens out of the equation, showing just what the camera and sensor themselves are capable of. Since focus accuracy can vary from lens to lens with the same camera body, it wouldn't really be fair to the body under test to report focus error with the Sigma lenses. The second reason we've not traditionally reported focus errors is that we're seeing only a single sample of a camera, so there's no way to tell whether the focus behavior we're seeing is actually representative of all samples of that model. We're considering changing this policy, as it seems some level of focus inaccuracy is almost universal among the SLRs we've tested. When we do report focus error though, we feel it should be based on measurements made with the camera manufacturer's own lenses.)
Issues raised by the new test scenes
While the new subjects bring dramatically improved repeatability to these particular test scenes, there are at least two concerns raised by our adoption of them:
Discontinuity with prior test shots
From time to time throughout the history of IR, we've had to make changes in our test targets to improve our results and keep up with advancing camera technology. As necessary or desirable as these changes are, any change in our test targets introduces a discontinuity with previous test shots. That is, while the targets may have a very long life into the future, we won't have images of the new targets shot with earlier cameras to compare new results against.
Ultimately, there's nothing to be done about this: Time and technology march on, test targets occasionally have to be upgraded, so just get over it. ;-) What we can do though, is to shoot the new targets with as many previously-tested cameras as possible, providing a reasonably extensive library of reference shots right from the beginning. As of this writing in early May, 2008, we have something on the order of 40 different cameras either in-house or immediately accessible to us that we'll be shooting these new test subjects with. This includes most of the current and recent crop of SLRs, so there'll be a considerable basis for comparison, quickly built up over the first month or so that the new targets are in use.
No human skin anymore
A second concern is that none of our ongoing test subjects contain human skin. While we've always been limited to the example of Caucasian skin, it could at least be argued that we've always had some representation of skin in our test lineup. Human skin is a fairly unique photographic subject, as light doesn't simply reflect from its surface, but also penetrates some distance below the surface, from where it's also reflected back out. This translucency is rather unique, and gives human skin a photographic character unlike almost any other subject.
While human skin is a unique subject, we felt that none of our previous test conditions were really set up to show its special character in the first place. Lighting in the "Sunlit" scene was deliberately extremely harsh, designed as it was to test cameras' response to the worst-case contrast of noonday sunlight. While you'd certainly expect to see differences between cameras' rendering of human skin and a mannequin's painted complexion under these conditions, in reality, you'd never photograph a person under such conditions if you cared the least bit about the rendering of their skin tones. Likewise, at the other end of the scale, the dim, strongly-hued lighting of the Indoor test scene also would never be a choice for fine portraiture.
Given these considerations, we felt that we really weren't losing anything essential by moving to non-human subjects: The gains certainly outweighed any minor disadvantages.
Interpreting the Results
We've prepared two guides to interpreting the results from these test scenes. See our Guide to the Indoor Portrait Test and Guide to the "Sunlit" Test for more information
The Bottom Line
At the end of the day, we're very pleased with these new test subjects. They'll significantly improve the repeatability of their respective tests, and should change not the slightest over the next 10-20 years. Changing test subjects is always a difficult decision to make; hopefully we'll not have to change either of these two subjects ever again.