The shortwave infrared (SWIR) domain of the electromagnetic spectrum has maintained a niche existence in remote sensing applications for decades. Specifically, its use for spectral imaging applications on UAVs was previously very limited. Therefore, in 2019, we developed a unique dual-band multicamera system for drones, featuring advanced SWaP InGaAs sensors for the VNIR-SWIR spectrum. This innovation enabled the first-ever capture of 2D images in multiple narrowband wavelengths in parallel, specifically over various agricultural fields. To date, the system has completed nearly 150 flights and gathered valuable spectral information on various types of vegetation, leading to further scientific publications as proof of validation. It has even demonstrated its capability to withstand the harsh environment of the Atacama Desert in Chile during a field campaign in early 2023. However, a major shortcoming of this system is that recording more than two spectral bands of a given scene requires several consecutive flights with a change of the applied filter pair between flights. Despite meticulous calibration on the ground and in the air after each filter pair change, particularly in rapidly changing and unstable lighting conditions, this can lead to spectral intensity mismatches among the individual channels. In addition, the time window for optimum light conditions (solar noon) can pass more quickly due to the extended time span as a result of filter changes and flight time. Additional disadvantages included the outdated analog CameraLink interfaces, which complicated the system design of the backend and the overall software design, consequently affecting usability.

Considering these disadvantages and in light of the rapid technological progress in the past five years since the creation of the predecessor system, an active pursuit of an all-encompassing redesign was undertaken. As a result, a 6-channel VNIR-SWIR multicamera system was designed, surpassing the existing two-band system in various aspects. As a consequence, the complete system was redesigned from the ground up. Now, the latest InGaAs camera technology as well as the latest interfacing and computing hardware are used. The new camera system, which is 200 grams lighter than its predecessor, now features six camera sensor elements, each equipped with an individually selectable narrowband bandpass filter and a corresponding lens. These elements not only provide four times the resolution per channel but also enhance the signal-to-noise ratio and dynamic range. Additionally, a modern digital interface improves interoperability.

The fully functional prototype has already been set up. It is in the process of being tested under laboratory conditions for parameters, to name only a few, such as temperature stability, optical quality, and spectral stability. Results of this first evaluation will be presented, as well as plans on UAV integration and an extended validation in UAV-based applications for the upcoming growing season in 2024. These real-world scenarios will be tested during several field campaigns on agricultural experimental fields and will be accompanied by intensive spectral and destructive ground truth data sampling. This new development paves the way for more sophisticated and simplified UAV-based remote sensing solutions to investigate the SWIR domain of vegetation more thoroughly.

Neural Radiance Fields (NeRFs)[1] have emerged in the field of computer
graphics for the task of novel view synthesis, i.e. the rendering of images from
new perspectives based on existing images with poses as training data. Besides
its convincing rendering quality, NeRFs hold potential for image-based scene re-
construction as geometry is derived implicitly in the form of a three-dimensional
density field encoded in the learned weights of a small multi-layer perceptron.
This implicit neural scene representation has proven to be well-suited for inte-
grating different forms of data and sensing modalities such as active distance
measurements from RGB-D cameras [2] or LiDARs [3], image-derived scene se-
mantics [4] or additional radiometric data, e.g., from a multi-spectral camera
[5]. Thus, the application of NeRFs for multi-sensor mobile-mapping platforms
such as mapping backpacks [6] is a potentially fruitful subject of research.
In this context, Poggi et al. [5] proposed to determine relative poses between
different sensors as additional learned parameters during the NeRF optimiza-
tion. However, this is only demonstrated to work for front-facing cameras with
parallel viewing direction. The aim of this work is to evaluate the effect of differ-
ent camera constellations in a multi-camera setup w.r.t. relative pose regression,
scene rendering and scene reconstruction. To this aim, the Microsoft HoloLens [7] is used as an exemplary off-the-shelf multi-sensor mobile-mapping platform
comprising five different camera sensors with different relative orientations as
depicted in Figure 1. The relative constellations range from a stereo configura-
tion with near complete overlap and small baseline in the center to divergent
views without any overlap at all in the side cameras. Besides investigating the
influence of these different relative camera constellations on pose regression and
NeRF results, we also evaluate the effect of mapping the different cameras to
one and the same radiometric channel vs. using separate per-camera channels
in the resulting radiance field.

References
[1] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron,
Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In European Conference on Computer Vision (ECCV), pages 405–421, 2020.
[2] Yu-Jie Yuan, Yu-Kun Lai, Yi-Hua Huang, Leif Kobbelt, and Lin Gao. Neural Radiance Fields from Sparse RGB-D Images for High-Quality View Synthe-
sis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45
(7):8713–8728, 2022.
[3] Alexandra Carlson, Manikandasriram S. Ramanagopal, Nathan Tseng,
Matthew Johnson-Roberson, Ram Vasudevan, and Katherine A. Skinner.
CLONeR: Camera-Lidar Fusion for Occupancy Grid-Aided Neural Repre-
sentations. IEEE Robotics and Automation Letters, 6(5):2812–2819, 2023.
[4] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davi-
son. In-Place Scene Labelling and Understanding With Implicit Scene Rep-
resentation. In IEEE/CVF International Conference on Computer Vision
(ICCV), pages 15838–15847, 2021.
[5] Matteo Poggi, Pierluigi Zama Ramirez, Fabio Tosi, Samuele Salti, Stefano
Mattoccia, and Luigi Di Stefano. Cross-Spectral Neural Radiance Fields.
In International Conference on 3D Vision (3DV), volume 2209.00648, pages
606–616, 2022.
[6] Mona Goebel and Dorota Iwaszczuk. Backpack System for Capturing 3D
Point Clouds of Forests. ISPRS Annals of the Photogrammetry, Remote
Sensing and Spatial Information Sciences, in print, 2023.
[7] Patrick Hübner, Kate Clintworth, Qingyi Liu, Martin Weinmann, and Sven
Wursthorn. Evaluation of HoloLens Tracking and Depth Sensing for Indoor
Mapping Applications. Sensors, 20(4):1021:1–24, 2020.

Introduction

In photo bathymetry, the bottom of a water body is observed with cameras in the air through the open and dynamic water surface. When entering water, the image rays are bended at the media boundary according to Snell's law of refraction and blurring occurs due to scattering in the water column. While the mathematical treatment of ray refraction is solved and substantial research for multimedia bundle block adjustment was already carried out, the main limiting factor for obtaining higher accuracy in photo bathymetry is the ability to reconstruct or model the dynamic, wave-induced water surface. One option to approach the problem is using a camera rig for capturing both the water surface and bottom strictly at the same time with synchronized oblique and nadir images. In this contribution, we present the setup and first results of a feasibility study carried out in the measurement lab of TU Wien.

Camera rig

We have borrowed a complete camera rig from IPF Stuttgart. This setup is composed of four cameras and lenses, an Arduino Leonardo and the associated cabling. The Arduino serves as controller and synchronizes the cameras by sending a trigger signal in user-definable intervals via a cabled USB connection. Two cameras are used to capture the water surface, looking obliquely from the side, and the other two to capture the water bottom, looking nadir from above.

Lab experiment

For testing the idea of simultaneous image acquisition, we designed and conducted an experiment at the 4D measurement lab of TU Wien. As a prerequisite, we first installed an array of coded photogrammetric targets on the floor, walls, and measurement pillars in the corner of the lab and measured the 3D coordinates with sub-mm precision with a total station. These targets served as control and check points in the bundle block adjustment. In a second step, we installed a 200 l mortar bucket and covered the bottom with gravel stones. Then we measured the topography of the stones with a conventional image block using a Structure-from-Motion and Dense Image Matching approach. After that, we filled the bucket with clear water, and installed and measured additional coded targets on the top and side of the bucket. Finally, we arranged the four cameras as explained above around the bucket. After these preparation steps, we took a series of synchronized images while creating moderate waves. The entire setup is shown in Figure 1.

First results

An exemplary set of synchronized images is plotted in Figure 2. The entire image block can be oriented and georeferenced in high accuracy via the photogrammetric targets. One can see from Figure 2 that only the bottom is visible in the nadir images but not the water surface. From the nadir image pairs we repeatedly derived the bottom topography both for still or wave-induced water surfaces. For the undulated water surfaces, this resulted in the expected drop of accuracy. At the time of writing this abstract, data processing to reconstruct the water surface in 3D from the oblique images is in progress. In these images, the water surface is at least faintly visible. Depending on the final success of the lab experiment, we further plan to use a drone squadron for capturing real world scenes with the same concept.

This paper aims at a quantitative comparison of the sensors resolving capabilities as well as detailed description of their advantages and disadvantages for aerial survey applications.

The sensor technology employed in contemporary aerial camera systems can be primarily classified into two categories: Panchromatic and Bayer pattern sensors. This categorization is applicable to sensors based on Charge-Coupled Device (CCD) and Complementary Metal Oxide Semiconductor (CMOS) based sensor technologies. While the practical work presented in this paper was carried out with CCD sensors, the received results are valid for CMOS based sensors as well.

The resolution potential of the Bayer pattern and the panchromatic sensor was verified with the help of siemens stars projected on a high-resolution color calibrated monitor. This set up enables an effective solution to display siemens stars in multiple colors. In order to verify that the screens emitted light spectrum has some overlap between the primary color’s red, green and blue, which can assist some debayering algorithms, measurements with a calibrated Avantes AVASPEC-ULS2048-USB2 spectrometer were conducted. The distance between the camera cone and the screen was set up in a way, that the optical system cannot out resolve the individual pixels (0.156mm) of the screen. The distance of 5310 mm together with 40 mm lens on the panchromatic and the Bayer pattern sensor, which both featured a 5.2-micron pixel pitch, resulted in a ground sampling distance of 0.69mm. The received images delivered results, which were well suitable for both visual interpretation as well as automatic measurements. Past theoretical studies indicate varying resolution losses of up to 50 percent compared to panchromatic sensors when the colors blue or red are used on the resolution targets. The rather sparse distribution of 25 percent of the blue and red pixel is the underlying technical reason for this degradation. These claims were investigated using the identical lens for both the panchromatic and Bayer pattern sensor. The quantization of the sensor’s resolution was based on an automatic detection of the minimum circle of resolution. This metric has been used over many years within Vexcel Imaging to adjust lenes based on collimator images of siemens stars for both panchromatic as well as Bayer pattern equipped camera cones. Furthermore, real life examples are presented showcasing the resolution differences in real world aerial scenarios.

The individual characteristics and challenges of the sensor principles are given based on common scientific literature, accompanied with practical examples from real world test flights.

Motivation

Mit der fortschreitenden Digitalisierung in Archiven stehen immer mehr historische Daten für die Forschung zur Verfügung. Dazu gehören auch historische Luftbilder, die detaillierte Informationen über die abgebildeten Gebiete enthalten. Anwendungen, die diese Bilder ermöglichen sind beispielsweise die Erkennung von Veränderungen der Landnutzung , der Bodenbedeckung, von Gletschern oder Küstengebieten sowie die Beobachtung von Bodendegradation und Naturgefahren. Die Untersuchung der abgebildeten Gebiete und der auftretenden 3D-Verformungen erfordert die Erstellung genauer digitaler Oberflächenmodelle (DOM), welche in der Regel durch Structure-from-Motion (SfM) gewonnen werden. Herkömmliche SfM-Workflows scheitern jedoch häufig bei der Registrierung historischer Luftbilder aufgrund ihrer durch die Digitalisierung bedingten radiometrischen Eigenschaften, der ursprünglichen Bildqualität oder der großen zeitlichen Veränderungen zwischen den Epochen. Wir zeigen, dass insbesondere der Schritt der Merkmalszuordnung (feature matching) in der SfM-Pipeline von hoher Relevanz ist, um DOMs mit guter Qualität zu erhalten.

Untersuchung

Um eine robuste Merkmalszuordnung für historische Luftbilder zu erreichen, wenden wir die beiden synergetischen neuronalen Netzwerkmethoden SuperGlue und DISK an. Dies erfordert mehrere Modifikationen, um Rotationsinvarianz zu ermöglichen und die hohe Auflösung von Luftbildern zu nutzen (Abb. 1). Im Gegensatz zu anderen Studien erfordert unser Arbeitsablauf keine Vorabinformationen wie bereits existierende DOMs, Flughöhe, Brennweiten oder Scanauflösung, die in Archiven oft nicht mehr vorhanden sind.

Der neu entwickelte Arbeitsablauf wird auf zwei Untersuchungsgebiete angewendet. Der erste Datensatz besteht aus mono-temporalen Luftbildern, die den tropischen Regenwald im Kongo in Zentralafrika zeigen. Diese Aufnahmen konnten aufgrund ihrer Qualität bisher nicht prozessiert werden. Der zweite Datensatz zeigt ein Gebiet in Südfrankreich und umspannt den Zeitraum von 1971-2001 in vier Epochen. Dieses Gebiet ist geprägt durch eine starke Veränderung der Landnutzung und wurde bereits in anderen Publikationen detailliert untersucht.

Ergebnisse

Es wird gezeigt, dass unsere Methoden unter Verwendung angepasster Parametereinstellungen in der Lage sind, mit quasi texturlosen Bildern umzugehen (Abb. 2).

Dies ermöglicht die gleichzeitige Verarbeitung verschiedener Arten von mono- und multitemporalen Daten in einem einzigen Arbeitsablauf von der Datenaufbereitung über den Merkmalsabgleich bis hin zur Schätzung der Kameraparameter und der Erzeugung einer dünnen Punktwolke. Es übertrifft herkömmliche Strategien in der Anzahl der korrekten Merkmalszuordnungen (Abb. 3), der Anzahl der registrierten Bilder und der berechneten 3D-Punkte und ermöglicht die Erzeugung multitemporaler DOMs mit hoher Qualität.

Die Flexibilität des Verfahrens ermöglicht die automatische Verarbeitung von bisher unbrauchbaren oder nur interaktiv zu verarbeitenden Daten, z.B. Luftbilder mit unbekannter Flugroute oder verschiedenen radiometrischen Eigenschaften. Dies ermöglicht es, noch weiter in die Vergangenheit zurückzugehen, wo die Datenqualität in der Regel abnimmt, und ermöglicht eine ganzheitliche Überwachung und einen Vergleich von Umgebungen von hohem Interesse (Abb. 4).