Department of Intelligent Media, ISIR, Osaka Univ.

Computational Imaging

Pioneering novel optical sensing systems exploiting computation

What is computational imaging?

Imaging is foundational to various sciences including medicine, life science, and robotics. Although imaging technology, primarily centered around lens design, has evolved over a long history, recent advancements in information science are fostering innovations at a fundamental level. We are exploring next-generation computational imaging technologies, merging cutting-edge optics and information science. Our research themes include coded-sensing methods utilizing optical design, image-reconstruction methods based on mathematical optimization and machine learning, and the development of cooperative design methods for these. Our goal is to create revolutionary new technologies that can transform all imaging devices, from cameras to scanners, endoscopes, microscopes, and telescopes, by combining advanced knowledge in optics, information science, applied physics, electronics, metrology, materials science, and applied mathematics.

Extended depth-of-field lensless imaging using an optimized radial coded mask

The lensless camera is an ultra-compact imaging device composed of an encoded optical system and a decoding computational system. However, one limitation in practical applications is that a single decoding operation only focuses at a certain distance, known as the depth of field limitation. While it’s possible to repeatedly adjust the distance variable and perform multiple decoding operations to stitch together in-focus areas, this approach increases computation time linearly. In this study, we optimized the design of a radiating encoding aperture with a frequency response suitable for imaging. Using this, we constructed an optical system that expanded the depth of field while maintaining superior imaging characteristics for the lensless camera. By building a prototype of the lensless camera and conducting imaging experiments, we demonstrated that it’s possible to obtain images that are simultaneously in focus over a deep range with a single shot and a single decoding operation.

Award

The 13th Japan-Korea Workshop on Digital Holography and Information Photonics (DHIP2023) Student Presentation Award (2023.12, J. Neto).

Publication

J. Neto, T. Nakamura, Y. Makihara, Y. Yagi, “Extended Depth-of-Field Lensless Imaging using an Optimized Radial Mask,” IEEE Transactions on Computational Imaging, Vol. 9, pp. 857-868, Sep. 2023. [link] [news]

Snapshot super-resolution 3D imaging using compressive sensing with a diffraction grating

Time-of-Flight (ToF) cameras, which measure light travel time to capture distance images, have been applied in various applications as three-dimensional measurement devices. However, a challenge has been the low spatial resolution due to their unique sensor structure. In this study, we designed a new ToF camera system that combines optical encoding measurement of super-resolution information using a diffraction grating with decoding computation that actively utilizes sparsity (compressed sensing) to acquire super-resolution distance images in a snapshot, surpassing the sensor’s spatial resolution limit (Nyquist limit). Through experiments using an optical system constructed by attaching a tilted diffraction grating to the lens surface of an existing ToF camera, combined with computational processing, we demonstrated that it’s possible to obtain distance images with spatial resolution exceeding the original resolution limit of the camera.

Award

The 13th Japan-Korea Workshop on Digital Holography and Information Photonics (DHIP2023) Student Presentation Award (2023.12, H. Kawachi).

Publications
  1. H. Kawachi, T. Nakamura, K. Iwata, Y. Makihara, and Y. Yagi, “Snapshot super-resolution indirect time-of-flight camera using a grating-based subpixel encoder and depth-regularizing compressive reconstruction,” Optics Continuum, Vol. 2, No. 6, pp. 1368-1383 (2023). [link]
  2. H. Kawachi, T. Nakamura, K. Iwata, Y. Makihara, and Y. Yagi,  “Snapshot super-resolution depth measurement using depth regularization and a diffraction grating,” OSK-OPTICA-OSJ Joint Symposia on Optics 2023, W1B-XI.03, (Jeju, oral, 2023.8).
  3. H. Kawachi, T. Nakamura, K. Iwata, Y. Makihara, and Y. Yagi, “Snapshot super-resolution time-of-flight imaging by PSF engineering and untrained deep neural-network prior,” 5th International Workshop on Image Sensors and Imaging Systems (IWISS2022), 5, (Hamamatsu, poster, 2022.12.12).

Wide-field lensless camera with a sparse image sensor and compressive sensing

In this study, we propose a wide-field lens-less camera that combines image capture by a “sparse image sensor” containing many randomly arranged micro-apertures and image restoration by compressed sensing (sparse restoration). The optical system with two sparse image sensors with their photosensitive surfaces facing each other enables us to measure sparse lens-less encoded images on the front and back sides at a time. After the measurement, a dense object image can be recovered from each encoded image by applying a compressed sensing-type image reconstruction process. This enables snapshot wide-angle imaging with an ultra-thin optical system consisting only of an image sensor. We have so far succeeded in a proof-of-principle demonstration using simulated optical experiments, and are currently developing a prototype sparse image sensor.

Award
  1. IWISS2018 ITE Open Poster Session Award 1st place (2018.11).
Publications
  1. F. Hosokawa, K. Kagawa, K. Sasagawa, J. Ohta, and T. Nakamura, “Design of a linkable self-encoding CMOS image sensor for a compact lensless camera with an ultra-wide field of view,” 5th International Workshop on Image Sensors and Imaging Systems (IWISS2022), 7, (Hamamatsu, poster, 2022.12). 
  2. T. Nakamura, K. Kagawa, S. Torashima, and M. Yamaguchi, “Super Field-of-View Lensless Camera by Coded Image Sensors,” Sensors, Vol 19, No. 6, 1329 (2019). [PDF]
  3. T. Nakamura, K. Kagawa, S. Torashima, and M. Yamaguchi, “Lensless imaging by coded image sensors,” 4th International Workshop on Image Sensors and Imaging Systems (IWISS2018), 16 (Tokyo, demo, 2018.11).

High-speed 3D laser scanning microscopy using depth-spatial coordinate linear transformation optics

Laser scanning microscopy is a microscopy technique that realizes high spatial resolution and deep measurement depth and is widely used for biological sample observation. In this study, we propose a new three-dimensional laser scanning microscopy method that can be implemented with two-dimensional scanning by combining a wide-depth batch illumination system using optical needles and a measurement system via a depth-to-spatial coordinate linear transformation optical system using computer-synthesized holographic optical elements. Optical experiments demonstrate the realization of depth-spatial coordinate linear transformation imaging and the resulting high-speed three-dimensional microscopic imaging of the mouse brain.

Publications
  1. Y. Kozawa, T. Nakamura, Y. Uesugi, and S. Sato, “Wavefront engineered light needle microscopy for axially resolved rapid volumetric imaging,” Biomedical Optics Express, Vol. 13, No. 3, pp. 1702-1717, (2022). [PDF]
  2. Y. Kozawa, T. Nakamura, and S. Sato, “Volumetric Imaging Utilizing Linear-Shift Point-Spread Function Based on Multiplexed Computer-Generated Hologram,” Focus on Microscopy (FOM2021), SU-PAR1-E (online, oral, 2021.03).
  3. T. Nakamura, S. Igarashi, Y. Kozawa, and M. Yamaguchi, “Non-diffracting linear-shift point-spread function by focus-multiplexed computer-generated hologram,” Optics Letters, Vol. 43, No. 24, pp. 5949-5952 (2018). [PDF]

All-in-focus lensless camera with a radial coded mask

Camera out-of-focus is often preferred in art photography to making the subject stand out beautifully, but in measurement applications, it is desirable to eliminate it as much as possible because it lacks information content. Recently studied lens-less cameras, which are based on post-measurement reconstruction processing, can realize an all-in-focus camera with no out-of-focus images in principle because the optical design without a lens is allowed. In this study, a radial encoding mask is used as an encoding optical element of the lens-less camera to realize snapshot all-in-focus imaging by physically implementing an optical system response function that is independent of the object’s distance. We have confirmed the numerical demonstration of the proposed all-in-focus lensless imaging principle and the experimental demonstration in a simplified setup.

Publication
  1. T. Nakamura, S. Igarashi, S. Torashima, and M. Yamaguchi, “Extended depth-of-field lensless camera using a radial amplitude mask,” Computational Optical Sensing and Imaging (COSI2020), CW3B.2 (online, oral, 2020.6). [PDF]

Improvement of resolution of FZA lensless camera by using wavelength dependence of optical transfer function

Fresnel zone aperture lens-less cameras have advantages in terms of high-speed image reconstruction processing and analytical solution of the optical transfer function, but the spatial resolution is limited by the zero-crossing of the optical transfer function (OTF) caused by the aperture structure and light diffraction. In this study, we propose an image processing technique to recover high-resolution image information by compensating the cutoff frequency information of a specific color channel from the information of other color channels, taking advantage of the strong wavelength dependence of this zero-crossing property. Optical experiments based on the prototype construction quantitatively demonstrate the effect of the proposed method on the high resolution.

Publication
  1. T. Nakamura, T. Watanabe, S. Igarashi, X. Chen, K. Tajima, K. Yamaguchi, T. Shimano, and M. Yamaguchi, “Superresolved image reconstruction in FZA lensless camera by color-channel synthesis,” Optics Express, Vol. 28, No. 26, pp. 39137-39155 (2020). [PDF]

Non-line-of-sight imaging using a holographic off-axis mirror

A holographic optical element (HOE) is a type of diffractive optical element that modulates light waves based on the principle of holography and can record and reproduce any light wave input/output response that can be implemented by optical diffraction. The HOE can record and reproduce arbitrary light wave input/output responses that can be implemented by optical diffraction. This enables us to implement a “see-through unequal angle reflector”, which cannot be realized by refractive optical elements, and to configure an imaging system through it, it is possible to take a frontal view of an object from a non-line-of-sight direction through a transparent orthorhombic element. In this study, we developed a large homogeneous mounting method of the unequal-angle reflector HOE, proposed a method of optical compensation of wavelength dispersion, full colorization, removal of unwanted ambient light, and demonstrated its application to image communication systems. The effectiveness of each research item was demonstrated based on optical experiments and prototype construction.

Award
  1. IDW’17 Best Paper Award (2018.1)
Publications
  1. S. Kimura, Y. Aburakawa, F. Watanabe, S. Torashima, S. Igarashi, T. Nakamura, and M. Yamaguchi, “Holographic Video Communication System Realizing Virtual Image Projection and Frontal Image Capture,” ITE Transactions on Media Technology and Applications, Vol. 9, No. 1, pp. 105-112 (2021). [PDF]
  2. F. Matsui, F. Watanabe, T. Nakamura, and M. Yamaguchi, “Unmixing of the background components in an off-axis holographic-mirror-based imaging system using spectral image processing,” Optics Express, Vol. 28, No. 26, pp. 39998-40012 (2020). [PDF]
  3. F. Watanabe, T. Nakamura, S. Torashima, S. Igarashi, S. Kimura, Y. Aburakawa, and M.Yamaguchi, “Dispersion compensation for full-color virtual-imaging systems with a holographic off-axis mirror,” SPIE Photonics West, 11306-3 (San Francisco, oral, 2020.2). [LINK]
  4. T.Nakamura, S. Kimura, K. Takahashi, Y. Aburakawa, S. Takahashi, S. Igarashi, S. Torashima, and M. Yamaguchi, “Off-axis virtual-image display and camera by holographic mirror and blur compensation,” Optics Express, Vol. 26, No. 19, pp. 24864-24880 (2018). [PDF]

Projection-type see-through 4D light field display and aerial image-touch detection

A 4D light field display is a type of stereoscopic display based on the light ray reproduction method using lens arrays and can reproduce optical real images in the air. In the projection-type configuration where the projector and lens array are placed far from each other, it is important to align the optical elements with the projected image on the lens array surface. In the case of the holographic optical element (HOE), the lens array surface can be made see-through and the environment-integrated application is possible, but the alignment is not easy due to the transparency of the HOE. In this study, we proposed a fast positioning method based on binary sinusoidal pattern projection and experimentally demonstrated 4D image reproduction based on it. In addition, we propose a fast and stable method for touch detection of 4D images by using scattered light detection and color information identification for 3D touch user interface applications and experimentally demonstrate the effectiveness of the proposed method.

Publications
  1. I. A. S. S. Chavarría, T. Nakamura, and M. Yamaguchi, “Interactive optical 3D-touch user interface using a holographic light-field display and color information,” Optics Express, Vol. 28, No. 24, pp. 36740-36755 (2020). [PDF]
  2. T. Nakamura and M. Yamaguchi, “Simple geometrical calibration procedure for a projection-type holographic light-field display,” Digital Holography & 3-D Imaging (DH2018), DTh3D.5 (Orlando, oral, 2018.6).
  3. T. Nakamura and M. Yamaguchi, “Rapid calibration of a projection-type holographic light-field display using hierarchically upconverted binary sinusoidal patterns,” Applied Optics, Vol. 56, No. 34, pp. 9520-9525 (2017). [PDF]

Gigapixel camera autofocus

The gigapixel camera, which combines a ball lens and a micro-camera array, can take snapshots of images that are two orders of magnitude higher in resolution than ordinary cameras. Because they combine a wide field of view with high resolution and can faithfully record every corner of a large physical space, they are expected to find applications in surveillance and other applications. On the other hand, the optical design of individual micro cameras is designed for high magnification, so the depth range of focus is narrow and a fast autofocus mechanism is required. In this study, a fast autofocus system using a hierarchical algorithm is proposed and implemented in a gigapixel camera. The effectiveness of the proposed method was demonstrated by an actual experiment using a prototype.

Publications
  1. T. Nakamura, D. S. Kittle, S. H. Youn, S. D. Feller, J. Tanida, and D. J. Brady, “Autofocus for a multiscale gigapixel camera,” Applied Optics, Vol. 52, No. 33, pp. 8146-8153 (2013). [PDF]
  2. S. H. Youn, H. S. Son, D. L. Marks, A. Pendleton, P. O. McLaughlin, T. Nakamura, D. J. Brady, and J. Kim, “Thru-focus Optical Analysis of Microcameras in a Gigapixel Camera,” US-Korea Conference on Science, Technology, and Entrepreneurship (UKC2013), K-1029 (New York, oral, 2013.8).

Compact wide-field-of-view imaging system via a designed multiple-scattering medium based on the transmission-matrix method

Since light passing through multiple scatterers is strongly disturbed physically, it is naturally impossible to see object image information (object image) in the incident space from the transmitted light (scattered image). On the other hand, since linearity holds for optical phenomena in general, the object image can be mathematically reconstructed from the scattered image by matrix operations. In this study, we implemented an ultra-compact wide-angle lens-less imaging system by placing an artificial scatterer in front of the image sensor, which is optimally designed for wide-angle imaging. The imaging principle is demonstrated and its effectiveness is quantitatively verified.

Publication
  1. T. Nakamura, R.Horisaki, and J. Tanida, “Compact wide-field-of-view imager with a designed disordered medium,” Optical Review, Vol. 22, No. 1, pp. 19-24 (2015). [PDF]

Imaging with active use of compound eye optics

Compound eye optics in insects is widely known for its advantage of being thin, lightweight, and wide-angle compared with the monocular optics corresponding to human visual systems and cameras, but it is also valuable from the viewpoint of the feasibility of 3D measurement and improvement of freedom of optical design in engineering applications. In this study, we investigated 3D image reconstruction and its parallel processing in the configuration where individual eyes are separated (serial image eye). In addition, we proposed an imaging system with an extended field of view and depth of field inspired by the configuration in which individual eyes are combined (duplex eye) and showed that the system can be applied not only to sensing but also to projection. We have also shown that the system can be applied not only to sense but also to a projector. The results were demonstrated through experiments.

Publications
  1. T. Nakamura, R. Horisaki, and J. Tanida, “Computational phase modulation in light field imaging,” Optics Express, Vol. 21, No. 24, pp. 29523-29543 (2013). [PDF]
  2. T. Nakamura, R. Horisaki, and J. Tanida, “Computational superposition projector for extended depth of field and field of view,” Optics Letters, Vol. 38, No. 9, pp. 1560-1562 (2013). [PDF]
  3. T. Nakamura, R. Horisaki, and J. Tanida, “Computational superposition compound eye imaging for extended depth-of-field and field-of-view,” Optics Express, Vol. 20, No. 25, pp. 27482-27495 (2012). [PDF]