Digital Camera Home > The Imaging Resource Solar Simulator - A new "Outdoor" test shot

The Imaging Resource Solar Simulator - A new "Outdoor" test shot.

The original
"Outdoor Portrait"
The new
"Harsh Portrait"
The original "Outdoor Portrait" shot did a good job of providing an acid test for cameras' tonal capabilities, but was subject to random variation from weather conditions. (Not to mention that it was a royal pain to deal with the limited shooting time that was possible with it.) The new "Harsh Portrait" shot has been designed to duplicate the color temperature and contrast ratio of the local sunlight here on an "average" day, providing an equally challenging test of camera capabilities. Shot in the studio though, it will be 100% consistent and repeatable, as well as much more convenient to work with. (The color difference you see here is likely the result of different white balance response to the background.)

After nearly six years of using the original test setup, I'm changing my "Outdoor Portrait" standard test shot, in order to improve the repeatability of that test, and elevate it to the level of a true test standard. This may not yet be the final form of this test, but any changes from this point on should be pretty minor.

This page is a brief explanation of the new "Outdoor Portrait" test setup, the lighting used, the rationale behind its configuration, and in part, the rationale behind my standardized tests in general. This won't by any means be a full treatment of the subject -- In response to reader emails over the last year or so, I'm working on a more complete discussion of my overall test philosophy and procedures in "background mode," in and around the perpetual backlog of tests and reviews waiting to be completed. Given that I've just changed the "Outdoor" test shots though, it seemed important to describe what was going on, what this particular subject is intended to test, how I've set up the new arrangement, and in the process also describe how to use this shot to reveal common digicam limitations. (It really does make a great "torture" test for cameras, able to reveal subtle noise problems, as well as more obvious ones of tone, contrast, and color.)

Why the change?
I'm sure that the first question many of you will have is "Why change?" - The old test was long established on the site, and provided a basis for comparing the capabilities of cameras stretching back over the last six years now. Why take the drastic step of changing a standardized test setup?

The answer is twofold, but both issues come down to issues of practicality. This particular test has been a thorn in my side for years now, because there's only about an hour or so in the early afternoon each day when the sun is at the right angle for the test to be shot. Here in Atlanta, it turns out that the magical hour happens to be just at the time of day when afternoon thunderheads balloon up out of nowhere, making it a dicey affair to grab the necessary shots while the sun is shining through gaps in the clouds. This problem has been particularly severe over the last year. - Atlanta has come out of a multi-year drought, with the result that we've had solid weeks at a time with no sunshine at the right time of day for this shot to be taken.

Even when the sun was shining, its position in the sky varied quite a bit with the change of seasons, making exact comparisons difficult. Even more than the angle of the sun though, the color balance and contrast levels of the lighting varied all over the place, depending on atmospheric conditions. In the summertime, high humidity and haze typically made for more diffuse light, with a warmer color temperature. - Under these conditions, the sky ends up being almost white, rather than blue, and the proportion of light coming from the sky itself is much greater than when the air is clear and dry. Conversely, in the fall and winter time, crisp, dry air meant no haze providing unwanted "fill" illumination, and the intense blue sky shifted the color balance toward the blue end of the spectrum.

More about our testing
New Indoor/Outdoor Portrait Tests (5/23/2008)
New Daylight Simulator (5/23/2008)
"Sunlit" Portrait - What to look for (5/23/2008)
Indoor Portrait - What to look for (5/23/2008)
Still Life and Multi Targets (Updated 11/29/2010)
HMI Studio Lighting (5/22/2006)
Performance Timing (5/22/2006)
New Review Format (5/22/2006)

Bottom line, this "standard" wasn't a very true standard at all, despite my best efforts to control for sun angle and appropriate shooting conditions. It was still quite useful for evaluating how well cameras handled scenes with a wide range of light-to-dark (dynamic range), but it wasn't something that lent itself well to the sort of direct A/B comparisons that are one of the real strengths of this site. This is a tradeoff that's necessary any time you want to use a "real" subject for a test shot, vs a completely artificial one, constructed in the studio or lab. I've been willing to accept the compromises in this particular test in the past, wanting to have more "natural" subjects in my test suite. Now that I'm routinely shooting and posting "galleries" of random photos for most of the cameras I test though, the need to keep the standardized test subjects "natural" is less of a factor than it once was.

The straw that broke this particular camel's back was that we moved about 15 minutes away from our old house at the end of 2003, making it even more difficult to play hide-and-seek with the sun and clouds. (Even though the folks who bought our house have been very accommodating about our continued use of it as a test subject.) This was finally the impetus I needed to come up with an in-studio version of this test.

What is "Daylight?"
Looking to create a suitable substitute in the studio led me to look a bit more closely at what constitutes "Daylight." My experience with the old setup showed me that there's really no such thing as "Standard Daylight," the ISO 7589 specification notwithstanding. Color temperature and contrast ratio obviously vary all over the place, depending on the season, latitude, and local atmospheric conditions. So what's a reasonable average?

As noted, there's actually an ISO standard specification (7589) that sets out what "standard" daylight is, right down to a specific spectral curve. This standard was settled upon after much debate, by measuring the actual characteristics of sunlight at a particular location and time that was determined to be a reasonably representative middle ground relative to the range of variation across the face of the globe. Here's an example of what typical daylight color spectra look like. (From Testing linear models on spectral daylight measurements by Javier Hernandez-Andres, Javier Romero, Antonio Garc&Mac245;´a-Beltran, and Juan L. Nieves):

Boiled down to a single number, that spectrum comes out to a correlated color temperature of 5500K. While I'm sure that the ISO committee had very good reasons for coming up with the 5500K standard for daylight, it's worth noting that the widely accepted standard for daylight color temperature in the graphic arts industry is 5000K, a noticeably warmer shade of "white." In the interest of providing the greatest compatibility with my own prior tests though, the real question was, what was the color temperature of the light that I'd been shooting under myself all these years?

Taking my trusty Minolta color meter in hand, I ventured out to measure the color temperature of the local Atlanta-area sunlight over a period of several months. I found that most of the time, the color of the local sunlight during the 1-3 PM time window in which I shot this test was in the range of 4700-4900K, rather than the more blue-tinted 5500K of the ISO spec. (I should note that I don't know the absolute accuracy of my color meter, so it's possible that the light I was measuring was in fact closer to 5500K. - But what matters here isn't the absolute value, but rather that I match what I'd been seeing in previous test shots before adopting the new standard.)

So... As measured by my color meter, 4700-4900K is the color that I aimed for in setting up the lighting for my new test. - But there's more involved than just the sun itself, there's the sky, and reflection from surrounding objects as well. Here's what I did to mimic all three:

The Light Source(s)
While it's neither practical nor necessary to precisely match the lighting of any given natural scene (overall color balance and repeatability are the key criteria for a test standard), I did want the lighting for this test to have some of the second-order characteristics of natural daylight. Thus, I've arranged the lighting to include three different components: 1) Direct "sunlight", 2) Diffuse "sky light", and 3) Secondary "fill" light.

"Sunlight"
The "sunlight" for this test comes from an array of 24 Solux 4700K incandescent lights in a 4x6 array, running at about 15 volts apiece, yielding a color temperature of about 4850K, measured at the subject. Natural sunlight is pretty collimated, which means that the light rays fall more or less parallel to each other, as from a distant point source. Given the limits of my current studio (ceiling height of 9 - 10 feet, and a practical limit for light mounting of around 8 feet), I couldn't take the most obvious approach of simply hanging a single, very bright light 20 or 30 feet above the subject to mimic the sun. The solution I came up with was to construct a rectangular array of lights at a height of about 8 feet, with a baffle (an aperture plate, really) positioned in front of them to limit side-spill from the individual bulbs. This produced reasonably well-collimated light, with a very high contrast ratio. (Limiting side-spill also helped avoid the warmer-hued light that the Solux bulbs emit off-axis.)

"Skylight"
For the diffuse "Sky" illumination, I used another array of 16 Solux bulbs, running at ~16 volts, and gelled to produce a color temperature of about 7000K. This bounces off the face of the baffle surrounding the main light array, mimicking a blue sky. This will produce a slight bluish component in shadows, similar to that found in outdoor shots on clear days. The "sky" light is about four stops down from the main "Sun" light, roughly matching typical conditions here in Atlanta in the late spring and early fall. (A reasonable compromise between summer and winter values.)

"Fill light"
The lights described so far produce about 3 KW of very high-contrast light, with a color balance that pretty closely approximates that of late Spring and early Fall sunlight here in Georgia. Shooting in the studio though, there wasn't nearly as much "fill" illumination as there would be outside, where the sunlight reflects off surrounding buildings, trees, grass, people, etc. As a result, the shadows were a little darker than they'd be in a natural environment, even under conditions of maximum contrast. (As seen in wintertime shots, when the sky is a very dark blue, and there's no atmospheric haze to act as a diffuser.) To open the shadows a bit, I set up two 1,000 watt PhotoFlex "StarLights" in 2x3 foot softboxes, at about 45 degree angles from the subject, gelled to match the roughly 3900K color temperature that I found typical of the fill light bouncing off the surrounding trees, grass, houses, etc of our original location. This "fill" light is about four and a half stops down from the main "sun" light, approximating the typical main/fill ratios I found outdoors.

The net of all this is that, while it's by no means an exact duplication of the sunlight I've shot under in the past, the color temperatures and sun/sky/fill intensity ratios are a pretty good approximation of natural daylight, at least here in Georgia.

What's not the same, and how much does it matter?
Of course, there's no question that the light from the above setup isn't identical to the light outdoors, at my prior shooting location. (But then, as noted above, that light was also subject to variation with season and weather.) So what's different about it, and what impact (if any) do those differences have on the test shots themselves?

Overall Brightness
Despite the almost 5 KW of lighting, the light from my "Solar Simulator" is still a good three and a half to four stops dimmer than typical sunlight. Fortunately, this really doesn't matter, for the purposes of the test. The issue isn't whether a camera has a fast enough shutter speed and/or small enough minimum aperture to properly expose sunlit scenes (you can safely count on that capability, as it'd be a pretty useless digicam that wasn't able to shoot outdoors in sunlight.). Rather, the issue is how well a camera handles extreme contrast ratios between highlights and shadows. - And for that, this test subject is ideal, and a fair representation of what you'll encounter out in the "real world." The verdict: No big deal.

The "Sun" is actually multiple light sources
This is probably the most visually distracting factor when looking at photos shot with this setup. Because the "Sun" is actually a 4x6 array of individual light bulbs, sharp edges in the subject cast multiple shadows. Depending on the camera in use (and how it handles highlight detail), and the specific position of the collar of Marti's shirt, you may see shadows that look like they have "echoes" to them, with multiple fringes of decreasing shadow depth as you move away from the edge that's casting them. While this is a little distracting from an aesthetic standpoint though, it really doesn't have any bearing on the primary purpose of the test. The verdict: No big deal.

No ultraviolet light in the "sunlight."
In most of their applications for general scene illumination, one of the advantages of the Solux bulbs used here is that, while they have a spectrum that's pretty close to typical sunlight, there's no ultraviolet component in their spectrum. (This is generally an advantage, as UV will cause objects to fade over time.) While digicams aren't generally sensitive to UV (as indeed they should not be), there is some possibility that objects being photographed may fluoresce slightly, effectively changing their visible characteristics as a function of how much UV is present. It remains to be seen whether this is an issue for the particular subject content of my "Outdoor Portrait" test. It won't likely affect the rendering of Marti's hair or skin tones, but it could possibly affect the colors of the flowers. (I've long wondered whether this could account for some of the difficulty many digicams seem to have with the blue flowers in this shot, as some organic dyes are known to be particularly sensitive to UV content in photographic lighting.) Only time will tell, as I shoot this test with many different digicams. Since I didn't control for UV content in the previous outdoor shots though, the ultimate result will simply be that, once again, results from the new test setup should be more consistent from one shooting session to the next. The verdict: This light source may not show some color problems that might occur under daylight, but it will be absolutely consistent from test to test.

Different scene content = different white-balance response
This is visible in the example images I included at the top of this page. The spectrum of light reflecting from Marti and (more significantly) the wall of the house in the old shot tended to push cameras' auto white balance systems toward a more blue-hued white balance than does the current setup. So images of the new test may generally be a little warmer-toned than those from the old one. Here again though, the main issue is how consistent the test is from shot to shot, and the new setup is near-perfect in that regard. The verdict: Scene content is different from the original, but has no impact on the validity of the test. (Better repeatability with the new setup.)

Different main-light angle
The limited ceiling height in my current studio and practical limitations in the size of the light array meant that I was pretty much restricted to having the main light be above and slightly in front of Marti. This is more directly overhead when compared to the typical position of the sun in the old shot, which of course varied depending on the season of the year. (Closer to overhead in the summertime, angled quite a bit in the winter, when the sun was lower in the sky.) This is the one area in which the lighting for the current setup is most visibly different from that of the original test. Here again though, while this difference makes it more difficult to directly compare results from the old setup with those from the new one, it really has no impact on the value of the new test. While the shadow on the left side of Marti's face (the viewer's left that is, her right) was a good place to look to see how cameras handled detail in deep shadows, the new setup produces fairly deep shadows with lots of detail in her eyes, and in the shadow areas beneath the bouquet. Looking at the results from the first dozen or so cameras I've shot the new test with, it looks like the results are all I could have hoped for, as there's a good range of shadow density present. Having a range of shadow density will let us evaluate the performance of cameras with a wide range of capabilities. The verdict: The main-light angle is different from the original, but has no impact on the validity of the test. (Better repeatability with the new setup.)

"Sky" illumination doesn't cover as wide an angle as it does outdoors
This is another case of a difference between the new and old test setups that, while it makes it difficult to compare results from the two tests, has essentially no effect on the value of the new test itself. With the old test, in the fall and winter of the year, the "sky" light had a moderate effect on the color balance of the image, particularly in the shadows. This is because the sun angle is lower in the fall and winter (reducing its relative intensity, as well as causing it to cast larger shadows), and because the lower humidity means less atmospheric haze, resulting in skies that are much more intensely blue than in the summertime. Given that shadow color isn't really a key component of this test, I probably could have left out the "sky" illumination entirely, simply relying on the fill illumination from the softboxes to open the shadows and produce the desired contrast ratio. Aiming for the greatest verisimilitude possible (or at least, practical) though, I went ahead and constructed the light setup to provide at least some "sky" lighting, even though it didn't cover as broad an area as in nature. The verdict: Different, but no impact on the validity of the test. (Better repeatability with the new setup.)

Fill illumination comes primarily from two discrete sources
Finally, yet another difference that affects the similarity between the old and new tests, but has no impact on the value of the new version. In the original test scene, the "fill" illumination came from sunlight reflecting from objects all around the subject (albeit not uniformly, as there were dark and light areas in the surrounding buildings and landscaping). In the new setup, while some light reflects from a white wall behind the camera, the majority of the fill illumination comes from two large softboxes located about 4 feet in front of the subject, at roughly 45 degree angles relative to the centerline between the subject and the camera's lens. As a result, the fill lighting in the new version is somewhat more concentrated. One possible effect of insufficiently diffuse fill lighting could be that the fill lights themselves might cast secondary shadows on the subject. Fortunately, this doesn't appear to remotely be the case with the current setup, thanks to the relative size of the softboxes, their relative proximity to the subject, and the fact that, taken separately, the light they cast on the subject is a full five stops down from the level of the main light. The verdict: Different, but no impact on the validity of the test. (Better repeatability with the new setup.)

 

What does this test show? What should I look for?
This is the section that I'm most hoping to come back to update, but I wanted to include at least some notes here in the meantime. This subject is actually very revealing of a whole range of digicam failings and limitations, so it's well worth studying.

Dynamic range - highlights and shadow detail.
This was really the primary aim of this test in the first place. One of the first things all the how-to books tell novice photographers is to avoid harsh direct sunlight of the sort this test is set up to simulate. This is because the range of light to dark tones is well beyond what typical cameras (film or digital) can reproduce, let alone that the harsh lighting is ugly and unflattering to your subjects. Since most consumers judge a photo's exposure level based on the midtones, I generally choose the exposure for this shot that puts Marti's skin tones in more or less the right range. With most cameras (even those having a "low contrast" option), this means that the strong highlights on her shirt are going to be blown out, and that the deepest shadows beneath the flowers will be pretty badly plugged. You can tell a lot about a camera's practical dynamic range by looking at how good a job it does of holding onto detail at both extremes of the tonal range. A lot of consumer digicams get their bright, punchy color by jacking up the contrast. This makes bright, pretty-looking (if not terribly accurate) images when you're shooting under relatively even lighting, but loses horrific amounts of detail when faced with a contrasty scene. How-to advice to the contrary, lots of amateurs shoot under harsh sunlight, so this is an important test. Of course pros want cameras that hold onto highlight detail, because it gives them more to work with in Photoshop, so this test is equally important for them as well.

Color - skin tones, colors in flowers
It turns out that Caucasian skin tones are tough for cameras to get right. For one thing, our eye is very accustomed to seeing these colors (at least, here in North America), so we can tell immediately when something is amiss. At the same time, even slight amounts of oversaturation can make your subjects' complexions look very blotchy. Since skin tones are such an important part of consumer (and professional) photography, this is an important subject. (Caucasian skin also seems to be something of a worst-case subject as well. Black and Asian skin is much more forgiving, in that it's much less prone to radical hue and tonal shifts if the color hue or saturation is slightly awry.) The other colors in this shot cover a useful range. Green foliage is a common element in landscape photographs, so the green leaves in the bouquet will give a good idea of how a camera will do with landscape shots. The blue flowers appear to be a very difficult hue for digicams to get right, as many render them with varying degrees of purple tints in them. The red flowers appear to be right on the edge of the sRGB color gamut, so many cameras have a hard time holding "shape" in them. All in all, there's a wealth of color-rendering data available here.

Fine detail - Marti's hair, fabric in flowers, "Peachfuzz" on her cheeks.
This subject is also great for evaluating fine detail, in Marti's hair, the texture of the fabric in the flowers and in the backdrop, right down to the delicate "peachfuzz" on Marti's cheeks.

Image noise - look in shadows on Marti's face, underneath the bouquet
As camera makers have crammed ever-more pixels onto the same size chips, image noise has steadily crept upward. This shot is great for seeing image noise in shadows, particularly if you play with the brightness and contrast in your imaging program. (If you have Adobe Photoshop or Photoshop Elements, the "Levels" adjustment is great for playing with this. You can check detail in both highlights and shadows, just by sliding the midtone (gamma) slider back and forth.)

Impact of anti-noise processing on fine detail - Look in Marti's hair
Faced with more and more noise coming from the image sensor itself, many manufacturers have been cranking up their anti-noise processing in the camera to hide the noise in the final images. The only problem is that this tactic tends to lose a lot of image detail in the process. Most anti-noise algorithms look at how much contrast there is in local areas of the image. Noise will be most noticeable in areas where there's relatively little fine detail in the subject, and these are generally also the areas of the image where people will be less aware of the subject detail in the first place. Consequently, anti-noise algorithms tend to set a local contrast threshold, and just flatten-out the detail whenever the local contrast drops below that level. The extent to which this trick is visible will depend a lot on the particular subject matter in a photo, but you'll tend to see it a lot in grass, other foliage, and hair. Because hair texture tends to be so adversely affected by anti-noise processing, this is a great shot to use to look for its effects. Look in areas where the strands of Marti's hair are close to the same color/shade, and see whether you can actually see individual strands, or if they all blur together into an undifferentiated mush. - The worst cameras produce hair and foliage images that look almost like watercolors. Very revealing of a syndrome that's becoming all too common in consumer digicams these days. (And is even prevalent in a few unfortunate pro SLRs as well.)

 

Summary
Phew! This ended up being a lot more words than I intended, but there was a lot of ground to cover, and hopefully the end result is that I've answered the questions most people had. The bottom line is that the new "Solar Simulator" lighting setup gives me an absolutely consistent test setup that closely mimics the most important characteristics of natural daylight. While photos shot with it can't be directly compared to those from the old arrangement, the test conditions should be equally challenging for the cameras to deal with, and the lighting will be absolutely invariant from day to day and month to month.