Multi-modal sensor based high frequency non linear model predictive control for robots

Humans are very adept at carrying out their daily tasks from picking up a bottle to constructing buildings. They constantly make many small decisions needed to achieve their goals and often make them quick enough to react to changes in their environment. Humans are also good at leveraging their five senses while making these decisions. On the other hand, robots barely do any of these things. The algorithms that govern robot movements are often slow and can not be deployed online. At the same time, they are unable to use multi-modal sensor data directly in their decision making. In this thesis, I address these challenges faced in robotics. I introduce a family of algorithms that enable online decision making for robot movements and contact planning while leveraging high dimensional raw sensor data such as images. First, I propose tailored optimization methods that exploit the structure in optimal control problems to quickly solve them. These algorithms are then deployed online as model predictive controllers on various robots. Secondly, I discuss the idea of formulating contact planning as a ranking problem of possible contact points. Through this formulation, I show how we can generate cyclic and acyclic contact plans online for any robot. Finally, I show how to integrate multi-modal sensors into standard numerical model predictive algorithms. The key idea lies in using the current sensor data to adapt the cost function of the control algorithm, which would then change the generated optimal behaviour. By combining all these three ideas, I show that the gap between human capability and robots is reduced. Throughout this thesis, I substantiate the capability of the proposed algorithms by deploying and testing them on different real robots. All the proposed algorithms in this thesis are supported with open source software to facilitate future research.