Computer Vision in the Infrared Spectrum: Challenges and Approaches

Human visual perception is limited to the visual-optical spectrum. Machine vision is not. Cameras sensitive to the different infrared spectra can enhance the abilities of autonomous systems and visually perceive the environment in a holistic way. Relevant scene content can be made visible especially in situations, where sensors of other modalities face issues like a visual-optical camera that needs a source of illumination. As a consequence, not only human mistakes can be avoided by increasing the level of automation, but also machine-induced errors can be reduced that, for example, could make a self-driving car crash into a pedestrian under difficult illumination conditions. Furthermore, multi-spectral sensor systems with infrared imagery as one modality are a rich source of information and can provably increase the robustness of many autonomous systems. Applications that can benefit from utilizing infrared imagery range from robotics to automotive and from biometrics to surveillance. In this book, we provide a brief yet concise introduction to the current state-of-the-art of computer vision and machine learning in the infrared spectrum. Based on various popular computer vision tasks such as image enhancement, object detection, or object tracking, we first motivate each task starting from established literature in the visual-optical spectrum. Then, we discuss the differences between processing images and videos in the visual-optical spectrum and the various infrared spectra. An overview of the current literature is provided together with an outlook for each task. Furthermore, available and annotated public datasets and common evaluation methods and metrics are presented. In a separate chapter, popular applications that can greatly benefit from the use of infrared imagery as a data source are presented and discussed. Among them are automatic target recognition, video surveillance, or biometrics including face recognition. Finally, we conclude with recommendations for well-fitting sensor setups and data processing algorithms for certain computer vision tasks. We address this book to prospective researchers and engineers new to the field but also to anyone who wants to get introduced to the challenges and the approaches of computer vision using infrared images or videos. Readers will be able to start their work directly after reading the book supported by a highly comprehensive backlog of recent and relevant literature as well as related infrared datasets including existing evaluation frameworks. Together with consistently decreasing costs for infrared cameras, new fields of application appear and make computer vision in the infrared spectrum a great opportunity to face nowadays scientific and engineering challenges.

Michael Teutsch received his diploma degree in computer science and his PhD degree from the Karlsruhe Institute of Technology (KIT) in 2009 and 2014, respectively. From 2009 to 2016, he worked as a research scientist and a postdoc at the Fraun- hofer IOSB, Karlsruhe, Germany. Since 2016, he has been with Hensoldt Optron- ics, Oberkochen, Germany. His research interests include computer vision, visual surveillance, object detection, object tracking, and machine learning. Michael has been organizing and co-chairing the annual IEEE International Workshop on Percep- tion Beyond the Visible Spectrum (PBVS) in conjunction with the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) since 2018. He is active as lecturer in computer vision currently at the Baden-Wuerttemberg Coop- erative State University (DHBW) Heidenheim, Germany. Michael serves as reviewer for several journals and conferences such as IEEE Transactions on Pattern Analysis and Machine Intelligences (TPAMI), IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), or IEEE Transactions on Geoscience and Remote Sens- ing (TGRS). He has authored or co-authored more than 30 scientific publications.

Angel D. Sappa received his Electro Mechanical Engineering degree (1995) from National University of La Pampa, Argentina, and his PhD degree in Industrial Engi- neering (1999) from Polytechnic University of Catalonia, Barcelona, Spain. In 2003, after research positions in France (LAAS-CNRS), the UK (UK Advanced Robotics) and Greece (ITI-CERTH), he joined the Computer Vision Center, Barcelona, Spain, where he currently holds a Senior Scientist position. Since 2016 he is a full professor at the ESPOL Polytechnic University, Guayaquil, Ecuador, where he leads the com- puter vision team at CIDIS research center; he is the director of the Electrical Engi- neering PhD program. His research interests include cross-spectral image processing and representation; 3D data acquisition, processing, and modeling; and computer vi- sion applications. He published about 200 papers in international journals and con- ference proceedings and served as program committee member in several interna- tional conferences. He has been involved in several national, regional and interna- tional research projects and several technological transfer projects; he has been the cofounder of VINTRA Inc. (San Francisco, USA) and Crowdmobile S.L. (Barcelona, Spain). He is a Senior Member of the Institute of Electrical and Electronics Engineers (IEEE).

Riad I. Hammoud received a MS degree in Controls of Systems and a PhD in Com- puter Vision and Robotics from UTC and INRIA (France) late 1997 and early 2001, respectively. He did his postdoc at Indiana University in 2002. Since early 2003, he is been working on several projects involving infrared imaging for defense, au- tomotive, and robotics applications. Early 2019, he joined TuSimple to develop au- tonomous driving systems. From 2012 to 2019, he worked at BAE Systems (Boston, MA, USA), on DARPA, AFRL and other US government agencies advanced research projects as principal investigator (PI), team lead and research scientist. Before joining BAE Systems, Riad was at Tobii-Dynavox (Pittsburgh, PA, USA) and Delphi Automo- tive Systems (Kokomo, IN, USA) working on Assistive Technologies and Active Safety Systems. He joined Seth Teller's team at MIT as a collaborating Researcher to work on the DARPA Robotics Challenge (2012-2015). Dr. Riad Hammoud served as guest editor of several special issues of top journals in computer vision including CVIU and IJCV. He authored several edited book including the Springer book on Augmented Vision Perception in Infrared. Since 2004, he has been organizing and chairing a workshop series in conjunction with the IEEE CVPR on Perception Beyond the Visible Spectrum (PBVS). He also serves as the general chair of SPIE Automatic Target Recognition conference (2018-2021).