Scene Understanding for Mobile Robots exploiting Deep Learning Techniques

Abstract

Every day robots are becoming more common in the society. Consequently, they must have certain basic skills in order to interact with humans and the environment. One of these skills is the capacity to understand the places where they are able to move. Computer vision is one of the ways commonly used for achieving this purpose. Current technologies in this field offer outstanding solutions applied to improve data quality every day, therefore producing more accurate results in the analysis of an environment. With this in mind, the main goal of this research is to develop and validate an efficient object-based scene understanding method that will be able to help solve problems related to scene identification for mobile robotics. We seek to analyze state-of-the-art methods for finding the most suitable one for our goals, as well as to select the kind of data most convenient for dealing with this issue. Another primary goal of the research is to determine the most suitable data input for analyzing scenes in order to find an accurate representation for the scenes by meaning of semantic labels or point cloud features descriptors. As a secondary goal we will show the benefits of using semantic descriptors generated with pre-trained models for mapping and scene classification problems, as well as the use of deep learning models in conjunction with 3D features description procedures to build a 3D object classification model that is directly related with the representation goal of this work. The research described in this thesis was motivated by the need for a robust system capable of understanding the locations where a robot usually interacts. In the same way, the advent of better computational resources has allowed to implement some already defined techniques that demand high computational capacity and that offer a possible solution for dealing with scene understanding issues. One of these techniques are Convolutional Neural Networks (CNNs). These networks have the capacity of classifying an image based on their visual appearance. Then, they generate a list of lexical labels and the probability for each label, representing the likelihood of the present of an object in the scene. Labels are derived from the training sets that the networks learned to recognize. Therefore, we could use this list of labels and probabilities as an efficient representation of the environment and then assign a semantic category to the regions where a mobile robot is able to navigate, and at the same time construct a semantic or topological map based on this semantic representation of the place. After analyzing the state-of-the-art in Scene Understanding, we identified a set of approaches in order to develop a robust scene understanding procedure. Among these approaches we identified an almost unexplored gap in the topic of understanding scenes based on objects present in them. Consequently, we propose to perform an experimental study in this approach aimed at finding a way of fully describing a scene considering the objects lying in place. As the Scene Understanding task involves object detection and annotation, one of the first steps is to determine the kind of data to use as input data in our proposal. With this in mind, our proposal considers to evaluate the use of 3D data. This kind of data suffers from the presence of noise, therefore, we propose to use the Growing Neural Gas (GNG) algorithm to reduce noise effect in the object recognition procedure. GNGs have the capacity to grow and adapt their topology to represent 2D information, producing a smaller representation with a slight noise influence from the input data. Applied to 3D data, the GNG presents a good approach able to tackle with noise. However, using 3D data poses a set of problems such as the lack of a 3D object dataset with enough models to generalize methods and adapt them to real situations, as well as the fact that processing three-dimensional data is computationally expensive and requires a huge storage space. These problems led us to explore new approaches for developing object recognition tasks. Therefore, considering the outstanding results obtained by the CNNs in the latest ImageNet challenge, we propose to carry out an evaluation of the former as an object detection system. These networks were initially proposed in the 90s and are nowadays easily implementable due to hardware improvements in the recent years. CNNs have shown satisfying results when they tested in problems such as: detection of objects, pedestrians, traffic signals, sound waves classification, and for medical image processing, among others. Moreover, an aggregate value of CNNs is the semantic description capabilities produced by the categories/labels that the network is able to identify and that could be translated as a semantic explanation of the input image. Consequently, we propose using the evaluation of these semantic labels as a scene descriptor for building a supervised scene classification model. Having said that, we also propose using semantic descriptors to generate topological maps and test the description capabilities of lexical labels. In addition, semantic descriptors could be suitable for unsupervised places or environment labeling, so we propose using them to deal with this kind of problem in order to achieve a robust scene labeling method. Finally, for tackling the object recognition problem we propose to develop an experimental study for unsupervised object labeling. This will be applied to the objects present in a point cloud and labeled using a lexical labeling tool. Then, objects will be used as the training instances of a classifier mixing their 3D features with label assigned by the external tool.