Computer Science ETDs

Publication Date



Robots are often highly non-linear dynamical systems with many degrees of freedom, making solving motion problems computationally challenging. One solution has been reinforcement learning (RL), which learns through experimentation to automatically perform the near-optimal motions that complete a task. However, high-dimensional problems and task formulation often prove challenging for RL. We address these problems with PrEference Appraisal Reinforcement Learning (PEARL), which solves Preference Balancing Tasks (PBTs). PBTs define a problem as a set of preferences that the system must balance to achieve a goal. The method is appropriate for acceleration-controlled systems with continuous state-space and either discrete or continuous action spaces with unknown system dynamics. We show that PEARL learns a sub-optimal policy on a subset of states and actions, and transfers the policy to the expanded domain to produce a more refined plan on a class of robotic problems. We establish convergence to task goal conditions, and even when preconditions are not verifiable, show that this is a valuable method to use before other more expensive approaches. Evaluation is done on several robotic problems, such as Aerial Cargo Delivery, Multi-Agent Pursuit, Rendezvous, and Inverted Flying Pendulum both in simulation and experimentally. Additionally, PEARL is leveraged outside of robotics as an array sorting agent. The results demonstrate high accuracy and fast learning times on a large set of practical applications.




Reinforcement learning, Motion planning, Robotics, Artificial Intelligence, Unmanned Aerial Vehcile, Systems control

Document Type


Degree Name

Computer Science

Level of Degree


Department Name

Department of Computer Science

First Advisor

Tapia, Lydia

First Committee Member (Chair)

Estrada, Trilce

Second Committee Member

Fierro, Rafael

Third Committee Member

Moses, Melanie

Fourth Committee Member

Williams, Lance

Project Sponsors

Sandia National Laboratories and New Mexico Space Grant

afaustCh3.mp4 (49556 kB)
movie for Chapter 3

afaustActa.mp4 (10898 kB)
movie for Chapter 5