Sony @ CP+: A.I. is the future of mirrorless cameras; only we have the technology to drive market growth

Earlier this year, IR founder and publisher Dave Etchells and senior editor William Brawley attended the CP+ tradeshow in Japan, where they were fortunate to be able to meet with executives from many of the most significant companies in the imaging industry. As we work our way through a backlog of transcription and editing, our interview series continues with a visit to the booth of Sony Corp.

Several Sony staffers kindly made time to answer our questions, but since we weren't always able to clearly identify speakers as our audio was transcribed, we've attributed the answers to Sony staff throughout. With that said, most answers were graciously provided by Kenji Tanaka, Senior General Manager, Business Unit 1, Digital Imaging Group, Imaging Products and Solutions Sector, Sony Corp. Topics for discussion included sensor and processor technology, the future of the mirrorless market, and how artificial intelligence might drive growth in camera sales, among others.

Without any further ado, let's hand over to Dave for the interview:

 

Dave Etchells/Imaging Resource: OK, so my first question is about image quality at the high ISO end. It seems we've gotten to basic limits for silicon, and the noise at very high ISO is just statistics for, you know, the arrival of small numbers of photons. Short of making much lower-resolution cameras with big pixels, is it true that any further advancements in high ISO performance will have to come from image processing?

Sony Staff: Thinking about noise, before we answer the question, we have to know what type of noise to measure. You are talking about the high ISO. In that case, the noise is optical shot noise, a kind of a white noise.

DE: Yes, white noise and shot noise.

Sony Staff: Yeah, each pixel has an optical shot noise, and this noise is random.

DE: Right, yes.

Sony Staff: That kind of thing our engineers know very well. So this noise is very easy to reduce some, with some techniques. But I cannot say details and Ueda-san cannot say details, because this is our core technology. [Ed. Note: Ueda-san is a Sony staff member who is generally the person who briefs us on technology details in Sony's new products.]

DE: Yes.

Sony Staff: So it is possible to increase the picture quality in the high ISO.

DE: Mmm, so we will see further improvements using processing techniques, hai. [Ed. Note: "Hai" is Japanese for "yes"; I catch myself saying it to people in the US after I've been in Japan for an extended period. Ditto "neh", which means more or less "right" or "right?" You hear both very frequently in Japanese :-)]

Sony Staff: Yes, yes.

DE: And then at the other end of the scale, it might have been in another interview you discussed the need for not just high ISO sensitivities, but also having a greater well depth at low ISO sensitivity. On a practical basis, how do you go about achieving a greater well depth? You have unique sensor technology, so how does your sensor technology let you do that? Is it like, do you stack capacitors on top of each other, or you have to somehow store more charge, no?

Sony Staff: I cannot say the details. Sorry. But you are semiconductor guy, so you understood the electron well. So talking about the electron well, the most important thing is surface area, not the volume.

DE: Hai.

Sony Staff: Surface is very important, so that is one key point.

DE: And that is where a deep implant, a deep structure can give you more area? ... Ah, you can't say...

Sony Staff: <laughs>

DE: Yeah.

Sony Staff: Our semiconductor team already has these kinds of techniques, so we can improve both low ISO and high ISO more.

DE: Yes, at both ends.

Sony Staff: Yes.

DE: Hai. So some of these improvements or technology you have to increase the well depth are still in engineering or development, right? So we will see more improvements in the next year or two years? Or are they already, is it like, are we already seeing all of that effect?

Sony Staff: That kind of the technique is not only one, but many.

DE: Many techniques.

Sony Staff: So, one by one, we can implement in a new sensor.

DE: Ah.

Sony Staff: But with old sensors, of course, we cannot do that because this is hardware issue, so...

DE: Yeah. [Ed. Note: Tanaka-san is noting that improvements of the sort he's talking about here aren't a matter of just new firmware, but of the sensor technology itself.]

Sony Staff: But as you know well, Sony has a semiconductor team, and we intentionally use a new sensor in the Alpha 7 Mark III. We [set out to create] a new sensor, so we had a lot of chances to increase the low ISO and high ISO imaging quality.

DE: Mmm, and so there are many things, so the A7 III has some technology that is helping with well depth and dynamic range, but then we will see more evolution still to come.

Sony Staff: Well I think this is OK, because this kind topic's too deep for your readers.

<laughter>

Sony Staff: Can you make this an article for the proper... the right article for your readers? It's a little too detailed.

DE: Oh yes, we have a wide range of readers, and some will say something like "Oh no, in semiconductor processing, you need to watch the surface leakage and you're..." So some people will go even further. But I try to make things understandable to everyone, so when we're talking about the surface area, maybe we will put a diagram and say look, this is where you store the charge and so more charge means more well depth. We try to explain things to people. [Ed. Note: Sorry for the lack of the diagram that I just promised in this article or for that matter a deeper explanation of sensor technology, but stay tuned for a much more in-depth article on sensor design, coming in the next couple of months ;-)]

Sony Staff: Nobody knows well depth and nobody knows that the most important thing is not the volume, it's the surface. But... <laughs>

DE: No, no, and so that's why we do what we do. So we will explain, that's our unique position.

Sony Staff: Yeah, that is your job, OK. <laughs>

DE: People are interested. And then, then we can explain about a deep structure or something. I know you can't say that that's what you're doing, but... I think, if we could explain about that, that more people would understand that that's a benefit for you, that Sony has that technology or that capability.

Sony Staff: The point is that only Sony has the technology, so I believe we can continue to drive the growth of the mirrorless camera, the mirrorless market.

DE: The growth of the market overall. Not just with your own cameras, but your technology in other people's cameras...

Sony Staff: And that is most important thing. Yes.

DE: I understand. This is a point you made on my last trip with you, to the Kumamoto factory, and in the discussion at your headquarters later. Sony is concerned with growing the overall market, not just your own cameras' places in it.

So a question about autofocus, this is really a bit of a deep image processing question. Besides semiconductors, I also worked in image processing architecture many years ago. And it seems like the really tough problem with autofocus is the object tracking. And in particular, what I saw back then as a fundamentally difficult problem in image processing was when you would go from very low-level operators -- edge detection, point clouds, that sort of thing -- and then at some point you have to transition from the pixel-based spatial-level data to an object-oriented data.

Sony Staff: Yeah.

DE: In a paper I wrote at the time, I called it the iconic to symbolic transition. And I wonder, with the way you're able to tightly integrate the sensor and processor and memory, will that help with that kind of processing for autofocus, to be able to identify objects and areas of interest? Is that kind of segmentation something that happens at that level, or only in the higher-level processing?

Sony Staff: This is a very good question. Always, the object recognition, for example we're already doing the eye recognition. So we call it eye AF. That kind of recognition technology is very, very important.

DE: Yes.

Sony Staff: Only mirrorless can do that, because mirrorless cameras capture the light directly with the imaging sensor, not just with the AF sensor.

DE: Yes, yes. You have the image data and the AF data right together.

Sony Staff: Yes. So at the point, we can do object recognition, maybe using AI. That kind of technology is our future. But the part that performs such a process is the processor. You know, the image sensor doesn't have a brain.

DE: Right, right.

Sony Staff: So the processor is very important. So we have both techniques, the image sensor technology and processing technology.

DE: Yeah. So I wonder if, as is the case in the human brain and eye/retina, there is a lot of processing that happens right at the sensor? What gets fed to higher levels of our own visual systems is already abstracted some, and I wonder can that happen -- or is that happening -- at the level where you have the processing right on the chip, for things like eye-detect AF? Can it detect eye-like objects (as in "we've got some white parts that are lighter than the surrounding area, with a darker iris and pupil at the center"), can it do image processing at that level?

Sony Staff: We can do many things. Talking about AI, as you know well, AI computing can be either in the cloud or on the edge. Both types of AI exist, but we're talking about the edge type of AI, edge meaning for our case the camera.

DE: Edge means local, yeah.

Sony Staff: Local, yeah, yeah. That kind of technology is improving very fast. For example, our current processor has some general computing unit, but maybe in the next generation, the general computing CPU should become more designed for AI.

DE: For AI processing, and...

Sony Staff: Yeah, yeah. Of course we already, we're already seeing that sort of a future.

DE: Yes.

Sony Staff: So maybe you know more detail, but I can't ...

DE: Ah, of course, you can't say too much, I understand. It seems like at that point, though, we're still talking about the computer chip, the general-purpose processor still taking in fairly low-level information from the sensor.

Sony Staff: Yes, yes.

DE: So still not very not object-structured at that point.

Changing topics slightly, we're seeing higher and higher frame rates with still cameras, and in case of the A9, very high resolution combined with great speed. It seems that still frame rates are getting closer and closer to video frame rates, and a logical extension of that is that at some point, the still and video become kind of merged, and you just extract still images from a video stream. Panasonic has made a feature of this, they have 4K Photo, 6K Photo, etc. Sony has been such a big player in broadcast and high-end video for so long, we're a little surprised we haven't seen that sort of convergence or that extraction from video in your products. How do you see that split between still and video, and how does that shape your product development?

Sony Staff: We can do that, technically speaking. But we don't have a user interface [like that for our] customers. That technique of extracting from a movie to make a still, it is OK. But usually, photographer has a meaning [that is different from videographer.]

DE: Yeah, yeah.

Sony Staff: In that case, of course, continuous shooting from video, if you use one shot it's totally different. To create a single shot, the photographer must think and configure many settings and try, and try, and try again, so [it's a] totally different world, I think.

WB: The mindsets are different.

Sony Staff: Yeah, yeah, totally.

WB: Video versus stills.

Sony Staff: Of course, that kind of technique, I mean continuous shooting, or to extract from a movie, it works, of course it works, but in many, many areas it doesn't work [in terms of user experience].

DE: Right, hai.

Sony Staff: Most photographers focus on making a great composition or...

DE: A single-image composition, yeah.

Sony Staff: ...some kind of camera settings, yeah. So it is very important for them.

DE: Yeah, I did tend to think that there was that very bright line between the two types of photography. Back when Canon introduced the 5D Mark IV, though, we went to a high-end fashion photographer's studio in Atlanta, and for the first time he was playing with the idea of video and then taking a frame out. And it was interesting, because there he was creating a very specific composition, but then the model would move or she would go through different poses, and he found it very, helpful...

WB: To get all those different frames, and then just...

DE: ...and as the model was moving, he would see just the right moment... it was all part of a carefully-constructed composition, but still, within that, he found it useful to have individual frames to choose from... But maybe that is also a very small segment... You see the two as being very different types of photography. Again, I know you can't comment too much about internal development processes, but we're very interested that your autofocus has advanced so rapidly, that advance must have been very difficult. It must have required a lot of engineering. I'm wondering if you can comment at all about what that process looked like? And part of what I'm thinking here is we had a very interesting conversation last year with Ogawa-san from Olympus, and he was spoke about the way they evaluate autofocus, in that you have different scenarios, you know, a bird against a sky, a sports runner with someone moving in front... He claimed that one of their strengths was they used more different scenarios, and then they look at thousands of images, and scored the tracking, and tried to advance their autofocus. What does autofocus development look like inside Sony? Can you make any general comments that readers might find interesting?

Sony Staff: Of course our autofocus engineering team has many types of scenarios, we call it "AF hyakkei". "Hyakkei" means 100 type of scenario.

DE: Oh, a hundred types of scenario, so "hyakkei" means 100?

Sony Staff: Hyaku is 100.

DE: Hyaku is 100, and so you say hyakkei and it's like 100 scenes...

Sony Staff: It's not necessarily like 100 exactly, but...

DE: It means large, like many scenes, yes.

Sony Staff: So still, the hyakkei, 100 scenes, is increasing because recently we developed the Alpha 9, last year, and the target [audience] is expanding to the sport professional shooter or the nature photographer. Nature means OK, we are using the...

DE: Long telephotos, yeah.

Sony Staff: At that case, we have to increase the hyakkei.

DE: Mmm, mmm!

Sony Staff: Maybe 110 or 120 or 130...

DE: <laughs> Yeah, yeah.

Sony Staff: So we need, many scenarios. And the point is how to tell the the customers [what setting to use]. Generally, we have an AF setting, but if the setting is wrong, of course our customers cannot create a beautiful image.

DE: They have to know how to work with the AF system, and it's complicated.

Sony Staff: So, we think the next step of our user interface is very, very important.

DE: Yeah, we ourselves at IR are not professional sports shooters, and it's challenging for us if we want to test a camera's AF. I'm thinking in particular Canon or Nikon, too, have many different settings in terms of how much the camera tracks and how much it allows for subject movement and interruptions in its view of the subject. We to try to figure out the best settings to use, and we might say a particular camera was good or not so good - but then some reader will say "Oh no, you should have used this setting, on submenu four etc..." So yeah, there's a question of user interface and maybe this is also where AI comes in, is for the camera to observe what is going on and decide how to track the subject.

Sony Staff: You're right, you're right.

DE: But that's an interesting point, that as you developed a camera, and were really focusing on professional sports or nature, you had to expand your hyakkei to have more test cases.

Sony Staff: Yep.

DE: Ah, interesting. Lenses like the upcoming 400mm f/2.8 and last year's 100-400mm show that you're making more of a commitment to long lenses. Was that more in response to demands from existing users, or was it part of a larger effort -- along with the A9 and A7R III -- where you are now consciously saying that you need products for sports photographers or wildlife?

Sony Staff: I think both, because 400mm f/2.8 is a generally capable lens for a sports shooter [as well as for nature photographers], so of course we get requests from sports photographers, so they are for both.

DE: So it was really both, that you had existing users...

Sony Staff: It's not only a matter of meeting [user's] requests, but we have a business direction...

DE: Yes, so it was both, there was existing demand but it was also strategic for you to move in that direction.

Sony Staff: Yeah, this is true.

DE: We somewhat had this conversation last time we spoke, I think, about the body design and body size. Especially now as we're getting longer lenses, on the one hand, for many users a very compact body and smaller lenses is nice, the whole system is more compact. But with a lens like the 100-400mm, the bigger body versus smaller body doesn't matter much in terms of overall size, but having a bigger body with a nice grip is more desirable. I remember you saying "Well, yes that's true but many A7R III users are not that group [which wants to shoot long lenses]. How do you see balancing the demands from the casual users, the enthusiasts, pro users? Because they may have very different desires for body shape.

Sony Staff: It doesn't depend on the customer's profile, whether they are professionals or enthusiasts or entry-level, you know. It depends on how they want to use, how to manipulate it or a favorable design.

DE: Its use case.

Sony Staff: For example, already you saw the [unintelligible] comments for the 400mm f/2.8, you didn't see that?

DE: I don't think, no.

Sony Staff: They are saying the 400mm f/2.8 is a very light...

DE: A very light lens, mmm.

Sony Staff: Easy to handle.

Sony Staff: One most important thing is the size and light weight. Another one is [center of gravity].

WB: How it balances on the body.

Sony Staff: So we try to make our lenses easy to manipulate (meaning that the size is very compact), easy to use (meaning that the weight is light), well-balanced... So please, after we launch the 400mm f/2.8, try it. Maybe you'll appreciate what I'm saying.

DE: Yes. We're almost out of time, one last question, and this is going back to APS-C again. I remember when we were here for the Kumamoto trip, you all were shocked...

Sony Staff: [Paraphrasing a longer answer that was a little difficult to transcribe, Tanaka-san acknowledged that the executives we met with were very surprised to hear that many people were feeling that Sony's strategy was to abandon APS-C in favor of full-frame cameras. *Very* surprised...]

DE: Yeah, yeah, you were very strong in your answer, essentially saying "APS-C Forever". And you just recently came out with the 18-135 lens, which has gotten a lot of interest from our readers. But still, you know, you've announced more and more full-frame models but nothing has been happening in APS-C, so I think APS-C users are feeling concerned. It's like "What about me?", you know?

Sony Staff: I haven't forgotten APS-C and, I believe in the APS-C market very deeply, so please wait.

Sony Staff: Please wait. Don't worry, APS-C users! <laughs>

DE: Don't worry, be happy, please wait.

<laughter>

DE: We are out of time I think, that's probably all we have time for. As always, I really, really appreciate your time. I always enjoy talking with you!

<laughter>

Sony Staff: I also am very happy to talk with you and Imaging Resource members, so any time.

DE: Yeah, yeah, I'm always happy to talk with another engineer, when I can talk with another geek, right? <laughs> Thank you very much, it was very, very interesting.

Sony Staff: Yeah, thank you very much.