Pixels for Geeks: A peek inside Nikon’s super-secret sensor design lab

I’ve known for some time that Nikon actually designs their own sensors, to a fairly minute level of detail. I think this is almost unknown in the photo community; most people just assume that “design” in Nikon’s case simply consists of ordering-up different combinations of specs from sensor manufacturers, picking a feature from column “A”, another from column “B” and so on, like a Chinese restaurant menu. In actuality, they have a full staff of sensor engineers who design cutting-edge sensors like those in the D5 and D850 from the ground up, optimizing their designs to work optimally with NIKKOR lenses and Nikon’s EXPEED image-processor architecture.

Given my own background in the semiconductor industry (my Master’s Degree is in semiconductor physics, although I haven’t practiced as an engineer for many years now), I’ve always wanted to do a story about sensor design, and it seemed like telling the largely-unknown story of Nikon’s sensor design abilities would be a good backdrop for it.

A chance to peek behind the curtain?

I’ve approached other sensor makers in the past about doing a story on sensor design, but it’s such a proprietary area that I’ve always been turned down flat. In Nikon’s case, though, I thought they might be interested in having the story told, because it’s a unique ability of theirs, compared to many of their competitors. I thought it would be beneficial to them to have more people be aware of it, and it might be an angle for me to finally get sensor engineers to peel back the curtain, if only a little.

And from our reader’s side, given that the D850 won our Camera of the Year award for 2017 and its sensor recently receiving the first DxO Mark score of 100 for an SLR, I thought there’d be a lot of interest in hearing some of what went into its making.

My host for the tour of Nikon’s sensor design operations was Shinsuke Sanbongi, Department Manager, 3rd Development Department, Development Sector, Imaging Business Unit. As usual with Japanese business card titles, that’s quite a mouthful, but the short version is that Sanbongi-san oversees Nikon’s entire sensor-design effort. He and his team’s proudest accomplishment is the D850’s sensor.

So I pitched Nikon on the idea of doing a sponsored article about their sensor-design efforts. As with others I’d approached in the past, they had a lot of concern about the deeply proprietary information and technology involved. It took a good four or five months of discussion and back and forth, and the strong support of Nikon’s US PR and marketing team, but in the end they agreed, resulting in the story you see here.

There’s a lot to talk about, so this will be a very long article, but I hope readers will find it as interesting to read as I did to research and write it.

We obviously won’t be able to get all the way down into the details, but hopefully this overview will shed some light on what it takes to make a great-performing sensor, and show some of the surprising (to me, at least) ways in which Nikon optimizes their sensor designs.

What exactly do Nikon’s sensor designers do?

Even though I knew that Nikon had its own sensor design operation, I was surprised to learn just how deep the process runs.

Companies designing chips of whatever sort generally rely on standard processes established by the “foundry” company that does the actual manufacturing. In these cases, the design process is “just” a matter of defining the layout of the devices on the chip. I say “just” though, because it’s far from a routine process to do this. The size and shape of transistors, photodiodes, resistors and capacitors determines their electrical characteristics, and there are loads of tradeoffs involved in balancing light-gathering efficiency, noise levels, readout speeds, on and on. A big trick is designing the pixels and readout circuitry so there’s as little “dead” (non-light-sensitive) area as possible, while maintaining readout speeds and minimizing power consumption.

A micrograph of a sensor from the D5. You can see one corner of the pixel array on the lower left, with the RGB color filters reflecting the light from the microscope. The circuitry around the edges is for reading out and digitizing the pixel data and transmitting it to the camera’s image processor.

This is what Nikon does, but their design and simulation extends to an incredible level of detail.

var axel = Math.random() + “”;
var a = axel * 10000000000000;
document.write(” + ‘iframe>’);

Here’s a much higher magnification microscope image of the D5 sensor seen above, clearly showing the RGB color filters for each pixel. Note how the pixels appear square, with no gaps between them; this is the result of Nikon’s “gapless” microlens technology. Traditional microlenses would appear as circles with significant wasted space between the pixels.

Prior to my tour, I had no idea Nikon’s engineering extended as deep as it does. Some of that is just a result of the many years since I was actively involved in the semiconductor industry, but the level of design, simulation and testing I saw was frankly astonishing.

Why design an original sensor?

It’s is a fair question: Commodity sensors (ones you can just buy from a sensor manufacturer’s catalog) are really quite good these days, so why would Nikon make the huge investment to design their own from the ground up?

I asked about this, and it turns out there are a number of reasons, many having to do with being able to optimize the camera system as a whole, in ways that you can’t if you’re just using off-the shelf parts.

In answering my question, Sanbongi-san talked in part about the need to match sensor designs to the capabilities of Nikon’s EXPEED image processing engines (which are also designed in-house), and about being able to tailor performance to the level of camera each sensor will be used in.

Designing their own sensors lets Nikon optimize how they work with their own in-house designed EXPEED image processors.

Another surprise for me, though, was the extent to which he talked about matching sensor designs to the characteristics of NIKKOR lenses. As we’ll see later on, they place a lot of emphasis on this, and do a lot of testing to optimize the overall combination of lens, camera and sensor from an optical standpoint.

Finally of course, doing their own R&D lets them create sensors like the one in the D850, with its low-ISO ability (a true ISO 64), and the one in the D5, with its exceptional light sensitivity and quantum efficiency. (The efficiency of converting incoming light to an electrical signal.) Throughout all our discussions, image quality was a recurring theme, whether expressed in terms of dynamic range, noise levels, interactions with NIKKOR lenses or color accuracy.

Image Sensors 101

Before we get into the details of what Nikon does, it’d help to understand a bit of what goes into an image sensor, what its component parts are, and how it connects to the outside world.

(Note that much the following is greatly simplified. For our more knowledgeable readers, feel free to add your thoughts in the comments at the bottom, but please understand that I’ve done my best to tell the basics without getting too mired in the details…)

This illustration gives a general idea of the parts of a pixel. The photodiode is what does the actual work of converting light to electrons, while the microlens helps insure that as much light as possible makes it to the photodiode.

Light is photons

Most people have at least heard of photons, the elemental particles of light. We’re never aware of them in daily life, because they’re far too small for our eyes to see individually. What we perceive as light is many (many) billions of photons hitting our eyes at once. (One estimate is that when we’re looking at a clear blue sky, about 3 x 10^14 photons per second are entering each eye. That’s 300,000,000,000,000, or 300,000 billion photons per second!)

Up until the early 1900s, physicists thought that light was made up of waves (think of waves in the ocean) and that matter was composed of discrete particles. A theory by Albert Einstein in 1905, later confirmed by an experiment by Robert Millikan in 1914, showed that light behaved not just as waves, but as particles as well. It turns out that light behaves differently, depending on how you’re looking at it, and particles of matter act as waves, too. This is the so-called wave-particle duality. So what is light, exactly? The short answer is “we don’t know”; no one has really figured out how light can somehow be both things at once. Fortunately, we don’t need to understand what it really is to make use of it. Image sensors turn “particles” of light (photons) into little bits of electrical energy (electrons) that we can count and measure.

Brighter or dimmer light just means more or fewer photons. As we reduce the number of photons/second, at some point we see nothing at all. But far beneath our perception of smoothly-varying brightness are individual light particles, the photons.

To record light, image sensors need to somehow convert photons to electrical signals, measure those signals digitally, and pass the information on to the camera’s processor. This involves five elements; the microlenses, photodiode, readout circuitry, analog to digital (A/D) converters and data transfer circuits. Let’s look at each one separately.

Catch all the light: The microlens

The sensor can only “see” photons that hit its light-sensitive areas, and some of the space in each pixel has to be dedicated to other functions, to measure and read out the light collected. (Even in so-called backside-illuminated sensors like the one in the Nikon D850, there are still isolation areas between pixels that aren’t light-sensitive.)

This shot gives some idea of just how hard the engineers sweat the details, when it comes to getting light into the sensor. On the left, you can see a simulation that’s tracing the paths of individual light rays as they pass through the microlenses, through the intervening material, and into the surface of the silicon. If you look closely, you can see the way the light is reflecting off the metallization layers. I had no idea that they could simulate the optical behavior of the sensors to this fine a level of detail.

To make the most of the available light, modern image sensors have “microlenses” above each pixel, that collect the incoming light and focus it on the light-sensitive area beneath. There are a lot of tricks to optimizing microlens design, including so-called “gapless” layout, the position and even shapes of the microlenses themselves.

Sanbongi-san drew this little diagram, to show how the microlenses are offset near the edges of the array, so light hitting them at an angle will still be directed onto the surface of the photodiode, labeled PD. (Apologies for the blurring at the top; I actually shot this across the conference table, upside-down and at an angle, then de-skewed it in Photoshop 🙂

As part of matching their sensors to NIKKOR optics, Nikon’s sensor designers pay a lot of attention to the microlenses and the structures between them and the silicon surface. Like essentially all camera sensors today, Nikon’s microlenses are offset a variable amount relative to the pixels beneath, to compensate for light rays arriving at oblique angles near the edges of the array.

Interestingly, Sanbongi-san noted that it’s not just current-day NIKKOR optics that are considered; they extend their optimization a little outside the parameters required by their current lenses, to allow for future development.

Photons in, electrons out: The photodiode

Once we’ve collected as much of the light falling on the sensor as possible, the next step is to convert photons into electrons (the fundamental particle of electric charge), so they can be measured electronically. This is the job of the photodiode.

The underlying process is pretty basic: Whenever a photon is absorbed in silicon (the material the sensor is made of), it kicks out an electron. At the heart of every pixel is a photodiode, an electrically-isolated area that collects the electrons generated during an exposure.

Because everything begins with the photodiode, its characteristics have a profound effect on image quality. In simple terms, we want two things from the photodiode.

The challenge of low ISO, and why it’s important

First, at the bright end of things, we want the photodiode to be able to hold lots of charge without “saturating” (ceasing to respond to more light). This basically sets the minimum ISO we can use a sensor at; if the photodiodes fill up quickly (that is, when they’re exposed to less light), the minimum ISO is higher. This is very similar to how film behaved: Once you hit maximum exposure, the negative won’t get any darker or the transparency won’t get any clearer. This happens at lower light levels with higher-ISO films than with lower-ISO ones. The same is true of sensors.

It’s probably no surprise that, generally speaking, big photodiodes can store more charge than tiny ones. This is the reason that we now often see minimum ISO ratings of ISO 200; the tiny pixels of modern sensors can’t store as much charge as the larger ones of days gone by. (Again, this is greatly simplified; I ask our more knowledgeable readers to bear with me here 🙂

Modern simulation software can model the performance of image sensor pixels by tracing the paths of individual photons through the microlens and overlying structures into the silicon itself. By simulating the travel of tens of thousands of virtual photons through the structure, from many different angles, sensor behavior can be well-characterized before a design is committed to (expensive) silicon. The output shown here is from Silvaco’s Device 3D(tm) package. Device 3D is a so-called physics-based simulator, incorporating material properties for most common semiconductor materials in use today. It uses fundamental semiconductor equations to model things like photon absorption and emission, bulk and interface traps, tunneling effects and much (much) more.

While a minimum ISO of 200 isn’t an issue if you’re shooting at smaller apertures or in lower light, it’s a real issue if you’re trying to shoot at wide apertures in bright light, or to use slower shutter speeds to achieve motion blur. Of course, you can always use an ND filter in such situations, but they’re a nuisance, especially if you’re using multiple lenses, and want to switch back and forth between tack-sharp and creatively blurred shots.

This is one area where Nikon’s dedicated sensor design has paid off for them: The D850 has the first true ISO 64 capability in an SLR. (Other cameras have special “Lo” ISO settings that will get there, but those come at the expense of poor tonal qualities and blown highlights.) Sanbongi-san told me that they developed the D850’s true ISO 64 capability in response to requests from motor-sports shooters, who wanted to shoot at large apertures and slow shutter speeds, so they could pan to follow the race cars while dramatically motion-blurring the background.

I don’t know enough about sensor design to understand the details of what’s involved, but when I asked Sanbongi-san what his team’s proudest achievements were, he mentioned the D850’s true ISO 64 first.

No Snow: Leaky pixels = stuck pixels!

At the other end of the scale, when there’s very little light, we want to make sure that the only electrons in the photodiode got there as a result of incoming photons. It turns out that photodiodes “leak” charge, letting random electrons creep in from the surrounding silicon. This shows up as “hot” pixels that look bright whether light is falling on them or not. Even if a given pixel doesn’t stand out on its own, random charge leaking into a sensor’s pixels from this so-called “dark current” can be a significant cause of image noise.

We had to dig way back into the IR archives (to 2002) to find this clear an example of “hot pixels”; where charge leakage made some pixels look bright, even though there was no more light falling on them than any of the others. No modern sensor would look this bad, but it’s still a challenge to reduce the impact of so-called “dark current”.

(Again, to our more knowledgeable readers, yes, shot noise is a big part of high-ISO noise, and there’s nothing to be done about that at a sensor level; it’s just math. But minimizing both bulk and surface leakage while maximizing well capacity and maintaining high readout speed is a significant challenge.)

Sanbongi-san pointed to the D5’s sensor as a case where they’d maintained excellent quantum efficiency and very low leakage, while still maintaining high readout speeds.

Charge to volts: The readout circuitry

After a shot’s been taken, the next step is to convert the electrons collected by the photodiode into a voltage that can be read out. This is the job of the readout circuitry. The readout circuits are basically little electronic amplifiers that turn the collected charge into a voltage, with additional electrical switches and wires to bring the resulting voltages to the next step of the process, which is…

For the electronically-minded among you, this is a simplified circuit diagram for a typical pixel’s readout circuitry. The photodiode collects charge from arriving photons during the exposure. Once the shutter’s closed, the transfer transistor is turned on, letting the readout circuitry “see” the charge that’s been collected. The transistor labeled “pixel amplifier” converts the charge to a voltage. To read out the voltages, the “enable transistor” is turned on. All the enable transistors in a given row are connected to the same control signal, and all the transistors in a column have their outputs connected to a single “column line”. So when the row-enable signal is sent, each column line has a voltage on it from the pixel in the row that’s been turned on. A/D converters at the edge of the chip can then digitize these voltages and send them on to the processor. Once all the rows of pixels have been read out, the “reset” line is triggered, emptying the charge out of all the pixels, to prepare for the next exposure.

Volts to bits: The A/D converter

Computers understand 1s and 0s, not voltages, so we need to convert the voltage from the readout circuit into a number that the image processor can understand. This is the job of the analog to digital (A/D) converter or ADC.

This is a particularly critical part of the process, as the analog to digital conversion needs to happen very quickly, as well as very accurately. The A/D converters in Nikon’s sensors measure the pixel voltages with a resolution of 14 bits. That translates to an accuracy of one part in 16,384!

Analog to digital conversion (ADC) is a critical part of capturing an image. A/D converters translate analog signals from each pixel to digital numbers the camera’s processor can deal with. A/D conversion needs to be both very fast and very accurate. This illustration shows combined red, green and blue numbers for each pixel, but on the sensor itself, each pixel captures only red, green or blue light – it’s the job of the camera’s processor to interpolate the separate pixel data into combined RGB values for each one.

This is an area where there have been huge strides in sensor technology over the last 10 years or so. Once upon a time, sensors just fed analog voltages to external A/D converters in the body of the camera. This was not only slow, but a source of noise, as the wiring between the sensor and the camera could pick up a lot of electrical noise from other parts of the camera’s circuitry. Eventually, A/D conversion moved on-chip, avoiding noise pickup from the analog wiring to the camera. This wasn’t simple to do, though, as the semiconductor processing that’s ideal for a sensor is far from ideal for A/D and digital circuitry. It was hard to get fast, accurate A/D conversion built into the same chip as the sensor.

Once the A/D circuitry was moved onto the sensor chip, the next innovation was massively parallel A/D conversion, with a separate A/D converter for every column of pixels in the array(!) This is where we’re at today. For obvious reasons, Nikon would prefer not to discuss fine details, but as far as I know, most sensors today use per-column A/D conversion.

Bits out to bits in: The data transfer circuitry

The final step is to transmit the numbers measured by the A/D converters to the image processing engine. The trick here is getting the data to the processor fast. In the case of the D850, with 45 million pixels, 14 bits per pixel and 9 frames/second with the accessory grip, that amounts to 5.7 billion bits per second, or 716 megabytes/second. That’s a serious load of data.

Achieving these kinds of data rates requires special technology, called LVDS, for “Low-Voltage Differential Signaling”.

When you need to move data really fast, the less you have to wiggle the signals (that is, the lower the voltages you use) the better. The downside is that low signal levels are easy prey for electrical noise from other parts of the system. The solution is to use differential signalling. Each signal travels on a pair of wires, right next to each other, and the signal itself is the difference in voltage between them. External noise affects both wires equally, and so cancels itself out at the receiver. Without LVDS technology, modern cameras would have much slower continuous shooting speeds.

Image provided by Dave at ti under a CC BY-SA 3.0 license via Wikimedia Commons.

(Knowledgeable readers: Yes, LVDS refers to a specific interface standard for flat panel displays, but it’s also a generic term, which is the sense in which I’m using it here.) Signal levels in LVDS are very small, so it’s easy to drive connections at extremely high speed. The “differential” part means the information travels on a pair of wires, and only the difference in voltage is significant. This minimizes noise effects from the signal transmission.

Here was another big surprise for me: I’d just assumed that the connection between chip and processor would be a simple matter of 1s and 0s. In practice, the data transmission rates are so close to the limit of what’s possible that Nikon uses proprietary encoding to make sure that all the bits make it across with 100% reliability. I had no idea that camera sensors pushed the technology so hard in this area.

This ultra high-speed communication is part of why Sanbongi-san made such a point of matching their sensors to the EXPEED processors. The number, speed, and characteristics of the communication channels between the sensor and processor are integral parts of the overall system design.

Sensors have to be very precisely placed within their packages, else there’d be framing problems or optical problems if the sensor isn’t seated perfectly flat. Here, a D5’s sensor is being measured by a computerized video microscope system. The numbers on the display are in millimeters, so the last digit is 0.1 micron(!) For reference, a typical human hair is about 60-70 microns in diameter; this system is measuring in units 700 times smaller than that. Talk about splitting hairs! (Sorry, I couldn’t resist 🙂

Life in the trenches

I was curious how Nikon organizes its sensor-design efforts. Do they have experts that focus on specific areas (pixel design, A/D, etc), consulting with teams for individual sensor designs, or do they do something different?

It turns out that all of Nikon’s sensor engineers work in teams on specific sensors. Some people are experts or have more experience in certain areas, so they’ll help out other teams that might be having a problem in their area of expertise. But on a day to day basis, everyone is working on the design of a specific sensor, along with others on that chip’s team. (So there was a D850 sensor team, a D7500 one, etc.)

Nikon’s sensor engineers work in teams dedicated to individual sensors. Everyone gets together once a week to set priorities, but the rest of the time work in smaller groups.

Once a week, there’s an all-hands meeting of the entire team, where the prior week’s progress is discussed, and new objectives are set for the week to come. From there, the individual teams among themselves on whatever the priorities are for that week.

Development: It starts with a camera

I was curious what sensor development looks like at Nikon – how does it get started? Do they work on developing new circuits and technology, and then just build them into new sensors as they’re available? Do they design future cameras based on what they believe is coming in terms of sensor capabilities. What’s the starting point?

Usually, engineers collaborate in small teams. No surprise; everyone has at least two monitors on their desk 🙂

Unlike a commodity sensor company, Nikon begins by deciding what sort of camera they’re going to build, what market it will serve, and the price point they need to hit. Feedback from their real-world users drive camera development, and the sensor designs flow from there.

While their efforts are always organised around specific camera models, Nikon’s sensor R&D goes on more or less continuously. I asked Sanbongi-san how long it takes to develop a sensor, and he said that, while it’s hard to point to a specific moment in time as the starting point for any given sensor, the process takes multiple years for advanced sensors like those in the D5 and D850.

Early stages: process and device simulation

As noted earlier, sensor fabrication takes a long time (typically months from start to finish), and is crazy-expensive, so you have to really pick your shots.

Modern “TCAD” software can simulate the operation of the pixel circuitry to a minute level of detail, tracking how the electrons move through the chip on a nanosecond time scale. This illustration shows charge transfer out of a CMOS sensor pixel, before and after the processor has told the sensor to read out the charge accumulated in the pixel. Charge-transfer problems can directly impact sensitivity (maximum ISO) and dynamic range.

It’s been years since I was involved in the semiconductor business myself, so I was completely blown away by how capable and detailed the simulation software has become. You can basically set up an entire semiconductor factory in the CAD software, with the computer simulating every step of the fabrication process down to an atomic level. Decades of development by third-party companies has resulted in simulation software of unprecedented capability. (I’m not sure which company’s simulation software Nikon uses; there are no less than four different companies marketing a product called “TCAD” 🙂

It’s hard to convey just how insanely detailed the CAD simulations of the semiconductor processing are. One minor example: To make semiconductor devices, you introduce tiny amounts of impurities into the silicon, to change its electrical properties. These days, this is done by using room-sized particle accelerators to smash impurity atoms into the silicon at very high energies. In sensor manufacturing, it’s not uncommon for these “ion implanters” to operate at energies of a million volts or more(!)

This has no relationship to Nikon’s sensor design (at least as far as I know), but is an illustration of a room-sized ion implanter, made by a company named Axcelis. Ion implanters smash impurity atoms into silicon wafers to change the silicon’s electrical properties. The faster they can accelerate the atoms, the deeper they go into the silicon; sensors rely on very deep structures to achieve large well capacities and high dynamic range. This particular machine accelerates the impurity ions using voltages as high as 4.5 million volts!

But a lot can happen on the way from the accelerator into the silicon, so the simulation software traces the paths of individual impurity atoms, as they smash through the photoresist, refract through the edges of it, and reflect off other structures. Trace a hundred thousand particles or so and you’ll have a pretty good idea of what things look like in the silicon afterwards. But this is just one small part of the overall process; the level of detail is truly mind-boggling.

Modern CAD software can simulate processes like ion implantation in almost unbelievable detail. This illustration shows an example from Silvaco’s SSuprem 4™, a 2D process simulator that’s widely used in the semiconductor industry for design, analysis and optimization of different fabrication technologies. Here, it’s simulating an ion implant process by blasting simulated dopant ions at a simulated wafer with photoresist on top, that’s meant to define the implant area. As you can see, it’s not a straightforward process; dopant atoms bounce off of the photoresist molecules, and end up in places you don’t want them.

Once the process-simulation software has created a three-dimensional model of the sensor in the computer, you can then simulate its operation, again to an unbelievable level of detail. As we saw in one of

Earlier this year, IR founder and publisher Dave Etchells and senior editor William Brawley attended the CP+ tradeshow in Japan, where they were fortunate to be able to meet with executives from many of the most significant companies in the imaging industry. As we work our way through a backlog of transcription and editing, our interview series continues with a visit to the booth of Sony Corp.

Several Sony staffers kindly made time to answer our questions, but since we weren’t always able to clearly identify speakers as our audio was transcribed, we’ve attributed the answers to Sony staff throughout. With that said, most answers were graciously provided by Kenji Tanaka, Senior General Manager, Business Unit 1, Digital Imaging Group, Imaging Products and Solutions Sector, Sony Corp. Topics for discussion included sensor and processor technology, the future of the mirrorless market, and how artificial intelligence might drive growth in camera sales, among others.

Without any further ado, let’s hand over to Dave for the interview:

Dave Etchells/Imaging Resource: OK, so my first question is about image quality at the high ISO end. It seems we’ve gotten to basic limits for silicon, and the noise at very high ISO is just statistics for, you know, the arrival of small numbers of photons. Short of making much lower-resolution cameras with big pixels, is it true that any further advancements in high ISO performance will have to come from image processing?

Kenji Tanaka
Senior General Manager
Business Unit 1
Digital Imaging Group
Imaging Products and Solutions Sector
Sony Corp.

Sony Staff: Thinking about noise, before we answer the question, we have to know what type of noise to measure. You are talking about the high ISO. In that case, the noise is optical shot noise, a kind of a white noise.

DE: Yes, white noise and shot noise.

Sony Staff: Yeah, each pixel has an optical shot noise, and this noise is random.

DE: Right, yes.

Sony Staff: That kind of thing our engineers know very well. So this noise is very easy to reduce some, with some techniques. But I cannot say details and Ueda-san cannot say details, because this is our core technology. [Ed. Note: Ueda-san is a Sony staff member who is generally the person who briefs us on technology details in Sony’s new products.]

DE: Yes.

Sony Staff: So it is possible to increase the picture quality in the high ISO.

DE: Mmm, so we will see further improvements using processing techniques, hai. [Ed. Note: “Hai” is Japanese for “yes”; I catch myself saying it to people in the US after I’ve been in Japan for an extended period. Ditto “neh”, which means more or less “right” or “right?” You hear both very frequently in Japanese :-)]

Sony Staff: Yes, yes.

DE: And then at the other end of the scale, it might have been in another interview you discussed the need for not just high ISO sensitivities, but also having a greater well depth at low ISO sensitivity. On a practical basis, how do you go about achieving a greater well depth? You have unique sensor technology, so how does your sensor technology let you do that? Is it like, do you stack capacitors on top of each other, or you have to somehow store more charge, no?

Sony Staff: I cannot say the details. Sorry. But you are semiconductor guy, so you understood the electron well. So talking about the electron well, the most important thing is surface area, not the volume.

DE: Hai.

Sony Staff: Surface is very important, so that is one key point.

DE: And that is where a deep implant, a deep structure can give you more area? … Ah, you can’t say…

Sony Staff: <laughs>

DE: Yeah.

Sony Staff: Our semiconductor team already has these kinds of techniques, so we can improve both low ISO and high ISO more.

DE: Yes, at both ends.

Sony Staff: Yes.

DE: Hai. So some of these improvements or technology you have to increase the well depth are still in engineering or development, right? So we will see more improvements in the next year or two years? Or are they already, is it like, are we already seeing all of that effect?

Sony Staff: That kind of the technique is not only one, but many.

DE: Many techniques.

Sony Staff: So, one by one, we can implement in a new sensor.

DE: Ah.

Sony Staff: But with old sensors, of course, we cannot do that because this is hardware issue, so…

DE: Yeah. [Ed. Note: Tanaka-san is noting that improvements of the sort he’s talking about here aren’t a matter of just new firmware, but of the sensor technology itself.]

Sony Staff: But as you know well, Sony has a semiconductor team, and we intentionally use a new sensor in the Alpha 7 Mark III. We [set out to create] a new sensor, so we had a lot of chances to increase the low ISO and high ISO imaging quality.

DE: Mmm, and so there are many things, so the A7 III has some technology that is helping with well depth and dynamic range, but then we will see more evolution still to come.

Sony Staff: Well I think this is OK, because this kind topic’s too deep for your readers.

<laughter>

Sony Staff: Can you make this an article for the proper… the right article for your readers? It’s a little too detailed.

DE: Oh yes, we have a wide range of readers, and some will say something like “Oh no, in semiconductor processing, you need to watch the surface leakage and you’re…” So some people will go even further. But I try to make things understandable to everyone, so when we’re talking about the surface area, maybe we will put a diagram and say look, this is where you store the charge and so more charge means more well depth. We try to explain things to people. [Ed. Note: Sorry for the lack of the diagram that I just promised in this article or for that matter a deeper explanation of sensor technology, but stay tuned for a much more in-depth article on sensor design, coming in the next couple of months ;-)]

Sony Staff: Nobody knows well depth and nobody knows that the most important thing is not the volume, it’s the surface. But… <laughs>

DE: No, no, and so that’s why we do what we do. So we will explain, that’s our unique position.

Sony Staff: Yeah, that is your job, OK. <laughs>

DE: People are interested. And then, then we can explain about a deep structure or something. I know you can’t say that that’s what you’re doing, but… I think, if we could explain about that, that more people would understand that that’s a benefit for you, that Sony has that technology or that capability.

Sony Staff: The point is that only Sony has the technology, so I believe we can continue to drive the growth of the mirrorless camera, the mirrorless market.

DE: The growth of the market overall. Not just with your own cameras, but your technology in other people’s cameras…

Sony Staff: And that is most important thing. Yes.

Sony’s sensor factory in Kumamoto, Japan.

DE: I understand. This is a point you made on my last trip with you, to the Kumamoto factory, and in the discussion at your headquarters later. Sony is concerned with growing the overall market, not just your own cameras’ places in it.

So a question about autofocus, this is really a bit of a deep image processing question. Besides semiconductors, I also worked in image processing architecture many years ago. And it seems like the really tough problem with autofocus is the object tracking. And in particular, what I saw back then as a fundamentally difficult problem in image processing was when you would go from very low-level operators — edge detection, point clouds, that sort of thing — and then at some point you have to transition from the pixel-based spatial-level data to an object-oriented data.

Sony Staff: Yeah.

DE: In a paper I wrote at the time, I called it the iconic to symbolic transition. And I wonder, with the way you’re able to tightly integrate the sensor and processor and memory, will that help with that kind of processing for autofocus, to be able to identify objects and areas of interest? Is that kind of segmentation something that happens at that level, or only in the higher-level processing?

Sony Staff: This is a very good question. Always, the object recognition, for example we’re already doing the eye recognition. So we call it eye AF. That kind of recognition technology is very, very important.

DE: Yes.

Sony Staff: Only mirrorless can do that, because mirrorless cameras capture the light directly with the imaging sensor, not just with the AF sensor.

DE: Yes, yes. You have the image data and the AF data right together.

Sony Staff: Yes. So at the point, we can do object recognition, maybe using AI. That kind of technology is our future. But the part that performs such a process is the processor. You know, the image sensor doesn’t have a brain.

DE: Right, right.

Sony Staff: So the processor is very important. So we have both techniques, the image sensor technology and processing technology.

DE: Yeah. So I wonder if, as is the case in the human brain and eye/retina, there is a lot of processing that happens right at the sensor? What gets fed to higher levels of our own visual systems is already abstracted some, and I wonder can that happen — or is that happening — at the level where you have the processing right on the chip, for things like eye-detect AF? Can it detect eye-like objects (as in “we’ve got some white parts that are lighter than the surrounding area, with a darker iris and pupil at the center”), can it do image processing at that level?

Sony Staff: We can do many things. Talking about AI, as you know well, AI computing can be either in the cloud or on the edge. Both types of AI exist, but we’re talking about the edge type of AI, edge meaning for our case the camera.

DE: Edge means local, yeah.

Sony Staff: Local, yeah, yeah. That kind of technology is improving very fast. For example, our current processor has some general computing unit, but maybe in the next generation, the general computing CPU should become more designed for AI.

DE: For AI processing, and…

Sony Staff: Yeah, yeah. Of course we already, we’re already seeing that sort of a future.

DE: Yes.

Sony Staff: So maybe you know more detail, but I can’t …

DE: Ah, of course, you can’t say too much, I understand. It seems like at that point, though, we’re still talking about the computer chip, the general-purpose processor still taking in fairly low-level information from the sensor.

Sony Staff: Yes, yes.

DE: So still not very not object-structured at that point.

Changing topics slightly, we’re seeing higher and higher frame rates with still cameras, and in case of the A9, very high resolution combined with great speed. It seems that still frame rates are getting closer and closer to video frame rates, and a logical extension of that is that at some point, the still and video become kind of merged, and you just extract still images from a video stream. Panasonic has made a feature of this, they have 4K Photo, 6K Photo, etc. Sony has been such a big player in broadcast and high-end video for so long, we’re a little surprised we haven’t seen that sort of convergence or that extraction from video in your products. How do you see that split between still and video, and how does that shape your product development?

Sony Staff: We can do that, technically speaking. But we don’t have a user interface [like that for our] customers. That technique of extracting from a movie to make a still, it is OK. But usually, photographer has a meaning [that is different from videographer.]

DE: Yeah, yeah.

Sony Staff: In that case, of course, continuous shooting from video, if you use one shot it’s totally different. To create a single shot, the photographer must think and configure many settings and try, and try, and try again, so [it’s a] totally different world, I think.

WB: The mindsets are different.

Sony Staff: Yeah, yeah, totally.

WB: Video versus stills.

Sony Staff: Of course, that kind of technique, I mean continuous shooting, or to extract from a movie, it works, of course it works, but in many, many areas it doesn’t work [in terms of user experience].

DE: Right, hai.

Sony Staff: Most photographers focus on making a great composition or…

DE: A single-image composition, yeah.

Sony Staff: …some kind of camera settings, yeah. So it is very important for them.

DE: Yeah, I did tend to think that there was that very bright line between the two types of photography. Back when Canon introduced the 5D Mark IV, though, we went to a high-end fashion photographer’s studio in Atlanta, and for the first time he was playing with the idea of video and then taking a frame out. And it was interesting, because there he was creating a very specific composition, but then the model would move or she would go through different poses, and he found it very, helpful…

WB: To get all those different frames, and then just…

DE: …and as the model was moving, he would see just the right moment… it was all part of a carefully-constructed composition, but still, within that, he found it useful to have individual frames to choose from… But maybe that is also a very small segment… You see the two as being very different types of photography. Again, I know you can’t comment too much about internal development processes, but we’re very interested that your autofocus has advanced so rapidly, that advance must have been very difficult. It must have required a lot of engineering. I’m wondering if you can comment at all about what that process looked like? And part of what I’m thinking here is we had a very interesting conversation last year with Ogawa-san from Olympus, and he was spoke about the way they evaluate autofocus, in that you have different scenarios, you know, a bird against a sky, a sports runner with someone moving in front… He claimed that one of their strengths was they used more different scenarios, and then they look at thousands of images, and scored the tracking, and tried to advance their autofocus. What does autofocus development look like inside Sony? Can you make any general comments that readers might find interesting?

Sony Staff: Of course our autofocus engineering team has many types of scenarios, we call it “AF hyakkei”. “Hyakkei” means 100 type of scenario.

DE: Oh, a hundred types of scenario, so “hyakkei” means 100?

Sony Staff: Hyaku is 100.

DE: Hyaku is 100, and so you say hyakkei and it’s like 100 scenes…

Sony Staff: It’s not necessarily like 100 exactly, but…

DE: It means large, like many scenes, yes.

Sony Staff: So still, the hyakkei, 100 scenes, is increasing because recently we developed the Alpha 9, last year, and the target [audience] is expanding to the sport professional shooter or the nature photographer. Nature means OK, we are using the…

DE: Long telephotos, yeah.

Sony Staff: At that case, we have to increase the hyakkei.

DE: Mmm, mmm!

Sony Staff: Maybe 110 or 120 or 130…

DE: <laughs> Yeah, yeah.

Sony Staff: So we need, many scenarios. And the point is how to tell the the customers [what setting to use]. Generally, we have an AF setting, but if the setting is wrong, of course our customers cannot create a beautiful image.

DE: They have to know how to work with the AF system, and it’s complicated.

Sony Staff: So, we think the next step of our user interface is very, very important.

DE: Yeah, we ourselves at IR are not professional sports shooters, and it’s challenging for us if we want to test a camera’s AF. I’m thinking in particular Canon or Nikon, too, have many different settings in terms of how much the camera tracks and how much it allows for subject movement and interruptions in its view of the subject. We to try to figure out the best settings to use, and we might say a particular camera was good or not so good – but then some reader will say “Oh no, you should have used this setting, on submenu four etc…” So yeah, there’s a question of user interface and maybe this is also where AI comes in, is for the camera to observe what is going on and decide how to track the subject.

Sony Staff: You’re right, you’re right.

DE: But that’s an interesting point, that as you developed a camera, and were really focusing on professional sports or nature, you had to expand your hyakkei to have more test cases.

Sony Staff: Yep.

DE: Ah, interesting. Lenses like the upcoming 400mm f/2.8 and last year’s 100-400mm show that you’re making more of a commitment to long lenses. Was that more in response to demands from existing users, or was it part of a larger effort — along with the A9 and A7R III — where you are now consciously saying that you need products for sports photographers or wildlife?

Sony Staff: I think both, because 400mm f/2.8 is a generally capable lens for a sports shooter [as well as for nature photographers], so of course we get requests from sports photographers, so they are for both.

DE: So it was really both, that you had existing users…

Sony Staff: It’s not only a matter of meeting [user’s] requests, but we have a business direction…

DE: Yes, so it was both, there was existing demand but it was also strategic for you to move in that direction.

Sony Staff: Yeah, this is true.

DE: We somewhat had this conversation last time we spoke, I think, about the body design and body size. Especially now as we’re getting longer lenses, on the one hand, for many users a very compact body and smaller lenses is nice, the whole system is more compact. But with a lens like the 100-400mm, the bigger body versus smaller body doesn’t matter much in terms of overall size, but having a bigger body with a nice grip is more desirable. I remember you saying “Well, yes that’s true but many A7R III users are not that group [which wants to shoot long lenses]. How do you see balancing the demands from the casual users, the enthusiasts, pro users? Because they may have very different desires for body shape.

Sony Staff: It doesn’t depend on the customer’s profile, whether they are professionals or enthusiasts or entry-level, you know. It depends on how they want to use, how to manipulate it or a favorable design.

DE: Its use case.

Sony Staff: For example, already you saw the [unintelligible] comments for the 400mm f/2.8, you didn’t see that?

DE: I don’t think, no.

Sony Staff: They are saying the 400mm f/2.8 is a very light…

DE: A very light lens, mmm.

Sony Staff: Easy to handle.

Sony Staff: One most important thing is the size and light weight. Another one is [center of gravity].

WB: How it balances on the body.

Sony Staff: So we try to make our lenses easy to manipulate (meaning that the size is very compact), easy to use (meaning that the weight is light), well-balanced… So please, after we launch the 400mm f/2.8, try it. Maybe you’ll appreciate what I’m saying.

DE: Yes. We’re almost out of time, one last question, and this is going back to APS-C again. I remember when we were here for the Kumamoto trip, you all were shocked…

Sony Staff: [Paraphrasing a longer answer that was a little difficult to transcribe, Tanaka-san acknowledged that the executives we met with were very surprised to hear that many people were feeling that Sony’s strategy was to abandon APS-C in favor of full-frame cameras. *Very* surprised…]

DE: Yeah, yeah, you were very strong in your answer, essentially saying “APS-C Forever”. And you just recently came out with the 18-135 lens, which has gotten a lot of interest from our readers. But still, you know, you’ve announced more and more full-frame models but nothing has been happening in APS-C, so I think APS-C users are feeling concerned. It’s like “What about me?”, you know?

Sony Staff: I haven’t forgotten APS-C and, I believe in the APS-C market very deeply, so please wait.

Sony Staff: Please wait. Don’t worry, APS-C users! <laughs>

DE: Don’t worry, be happy, please wait.

<laughter>

DE: We are out of time I think, that’s probably all we have time for. As always, I really, really appreciate your time. I always enjoy talking with you!

<laughter>

Sony Staff: I also am very happy to talk with you and Imaging Resource members, so any time.

DE: Yeah, yeah, I’m always happy to talk with another engineer, when I can talk with another geek, right? <laughs> Thank you very much, it was very, very interesting.

Sony Staff: Yeah, thank you very much.