The aim of this work package is to classify historical geodata using methods of artificial intelligence (AI). This is to be done on the basis of object representations in accordance with the specified object type catalogue. This means that historical map data must be segmented and thus converted into an object-based representation. For historical image data, the goal is to predict label images (classification), in which each pixel is assigned to a class label representing an object type.
As the classification is to be carried out using deep learning methods, an important point is to first collect training data for the corresponding test areas and make it available in the database.
This work package requires the solution of three tasks: (1) the classification of aerial and satellite images, (2) the classification of historical maps, and (3) the joint classification of maps and images.
Georeferencing of historical maps and classification of selected objects in these maps (LUH-ikg)
Historical maps are available in analogue form and, after digitisation, in raster format. They differ from today's maps and digital datasets in various ways - on the one hand in content and form of presentation and on the other hand in geometric projection; moreover, they may show distortions and different degrees of generalisation. In order to be able to use historical maps for time series analyses, it is therefore necessary, on the one hand, for the data to be available in a uniform reference system and, on the other hand, for the data to be available in vectorised form, with semantic content that is comparable to the one available in current maps. Therefore, two tasks are tackled in this context: (1) the interpretation of historical maps with current semantics and (2) georeferencing of historical maps.
- In the context of this project, the interpretation will be concentrated on selected objects of high relevance. For this purpose, deep learning methods (in particular Convolutional Neural Networks - CNNs) will be used. Initially, the focus will be on objects represented by point-based cartographic symbols, as experience has shown that they can be determined more reliably than others using standard classifiers. These objects include symbols related to settlements or vegetation. In addition, objects that can be used for automatic georeferencing will also be considered. To overcome the problem of the large quantities of reference objects required for training, self-supervised learning will be used, which has proven successful in preliminary work. Due to the very different cartographic design specifications of the various maps, it may be necessary to use hierarchical approaches in order to be able to deal with different symbol sizes. The results of this step are areas with specific semantic interpretations in maps, which are stored in the database.
- The goal of automatic georeferencing of historical maps is to be achieved by matching procedures in which the system to be developed searches for comparable objects in both datasets, assigns them to each other and establishes the geometric relationship between the datasets using parametrically given transformations. As matching candidates, objects are to be used which have a great significance over a longer period of time and which can also be extracted using the procedures developed from 1), for example settlements, churches, bridges, or road crossings. The resultant method will be demonstrated and evaluated on the basis of various historical maps, thus demonstrating its transferability.
Classification of Images (LUH-IPI)
The aim of this part of the work package is to develop a methodology for the classification of aerial and / or satellite imagery using the previously defined object type catalogue. This is primarily intended to answer the research question of what information can be derived from historical data using AI and to which extent such methods can be transferred to other data with regard to aerial and satellite images of a single point in time. For this purpose, a CNN-based method for the pixel-wise classification of georeferenced aerial images will be implemented and tested. First, based on existing CNN architectures for pixel-wise classification of images, a suitable encoder-decoder architecture is selected and adapted to the problem at hand. For the training, standard procedures for unbalanced class distributions are first used and adapted to the data available here. The images used are high-resolution multispectral aerial images. If necessary, the resolution is reduced compared to the original data (e.g. to approx. 1.0-1.5 m) in order to reduce the computational efforts and to achieve an adaptation to the resolution of the label images to be predicted (approx. 2.5 - 5 m). The first tests of the developed CNN will be carried out using the most recent and best available data with regard to the image quality. Experimental tests will investigate which of the selected object types can be reliably detected using these data. The results of these initial experiments serve as a best-case scenario and provide the framework for further investigations that address the following questions:
- Suitability of different data sources: For the more recent datasets of each test area, image data of different origins will be available. This allows an analysis of the quality of the classification depending on the geometric resolution and the spectral configuration of the input data. In particular, the suitability of freely available remote sensing data, such as Sentinel-2, but also data from other satellites for the classification can be investigated.
- Learning with training data affected by label noise: Another issue to be investigated concerns the need for the availability of training data for all epochs. In the sense of a potential reduction of the effort for the provision of training data, it has to be examined whether existing digital topographic datasets (ATKIS, i.e. the official German land use database) can be used without correction to provide the training labels for the classification of images of other epochs. For this purpose, the training procedure has to be adapted in such a way that errors in these training labels ("label noise") can be compensated.
- Generalisation ability of the classifiers: In further experiments, the variants of the developed methods which have proven to be the most promising ones in the investigations mentioned under points 1) and 2) are to be analysed with regard to their ability to generalise to unseen data. Different subsets of training and test data are compiled from the data of different areas in order to investigate the transferability of the methods to data of different areas and acquired using different sensors with or without new training data.
In all cases, the experiments follow the usual protocols in which accuracy measures for the predictions are derived. In the course of the project, the data basis for the experiments expands, which ultimately allows further analyses with regard to the quality of the results as a function of the properties of the data; in addition, the dependence of the quality on land cover can also be investigated.
Joint classification of historical maps and images (LUH-IPI)
Following the development of a classification method per modality (map, image), multi-modal methods are developed with the help of which image and map data of the same area and representing the same point in time can be used for a pixel-wise classification. The object types to be distinguished correspond to those for image classification. The underlying research hypothesis is that these different datasets can support each other in the classification process, either by the CNN-derived image features complementing the information from the map in areas without cartographic symbols or background colour, or by certain cartographic symbols available in the map helping to distinguish classes that cannot be separated based on the information available the images alone. It is assumed that scanned maps are available for all points in time, containing the information to be predicted only implicitly.
To combine the data, the CNN developed for image classification is extended in such a way that the different input data (scanned maps and images) are first processed in separate encoders; the fusion of the two branches of the CNN takes place before the transition to the joint decoder (late fusion). Subsequently, it will be investigated to what extent the results generated by the map classification approach, which, in contrast to the goals of this joint process, are based on a selection of objects only, can be integrated into and support the joint classification process. This could be done, for example, by presenting raster maps with probabilities for the occurrence of certain objects to the corresponding encoder together with the digitised maps.
A further point of the investigations is the role of digital topographic data, which are available for more recent points in time and already explicitly represent the information to be predicted from the images or scanned maps. These data can be used for the classification in two ways: on the one hand, they can be used to provide the labels for the training for the points in time at which they are available; on the other hand, they can be used as additional observations in the classification process. In both cases, however, deviations of the content of the digital topographic data from the reality represented in the images are to be expected. With regard to training, the experience gained in the context of image classification in dealing with the use of data affected by label noise for training can be directly transferred to the multi-modal case at hand. In the context of joint classification, essentially the same questions are investigated as in the context of image classification, but only for the most promising of the approaches pursued there and with a focus on the change in the quality of classification that is achieved by the joint processing of images and maps or topographic datasets as well as by the integration of the results from the semantic interpretation of maps.
The resulting methods for classification, together with those based solely on maps, represent the final result of AI-assisted classification of historical data. The results obtained provide the answer to the research question of what information can be reliably derived from historical maps and aerial or satellite images.
