In addressing the household robotic manipulation challenges associated with tray and table organization, a variety of tasks emerge, ranging from rearranging stacked objects on a tray to clearing and stacking different items on a table. These tasks typically involve everyday kitchen items such as cutlery, plates, bowls, and cereal boxes. Symbolic planning methodologies, extensively explored for long-horizon robot manipulation tasks, have proven effective in generating task plans for these scenarios. However, the translation of these plans into executable actions requires the integration of a low-level planner. To bridge this gap, our work introduces a novel hierarchical task and motion planning framework tailored for the household kitchen environment, emphasizing efficient and practical solutions for tray and table organization tasks. This approach holds significant promise for enhancing the capabilities of household robots, contributing to streamlined domestic routines and improved daily living.
This was a course project for Prof Maxim Likhachev’s CMU course 16-782 Planning and Decision making in Robotics. My key contributions to this project were as follows:
- Developed a hierarchical Task and Motion Planner (TAMP) leveraging STRIPS representation and A* graph search for high-level task planning and Dynamic Movement Primitives (DMPs) for low-level motion planning.
- Integrated and tested Task and Motion Planner with ROS control and CoppeliaSim simulation of Universal Robots UR5 robotic manipulator with industrial gripper.
Hierarchical Planning
A hierarchical planning strategy for household manipulation decouples the task and motion planning problems. This ensures that our planner generalizes for other robotic manipulation problems. Using the STRIPS representation for symbolic planning allows the proposed planner to work for a variety of problems as the only change that one needs to implement is write the appropriate environment file for the given task. Dynamic Movement Primitives(DMPs) can generate smooth, obstacle-free geometric trajectories for grounded actions generated by the Symbolic planner. In the case where the DMP fails to generate a feasible trajectory, we employ 2 re-planning strategies for the Symbolic planner. The figure below shows the proposed Hierarchical Planner pipeline.
Symbolic Planning
Symbolic planning involves solving complex task-level planning problems by representing them as symbols and defining a set of rules governing the task. The solution to a problem can be represented as a sequence of actions that transform the initial state of the problem into the desired goal state. We use the Stanford Research Institute Problem Solver (STRIPS) representation for defining our symbolic planning problem. This representation uses a set of propositions, or statements, to describe the initial state of a problem and the goal state that we want to reach. For example, a proposition might be “the robot is at location A” or “the box is on the table”. These propositions are then used to define a set of actions comprised of effects and preconditions, where an action will be executed if the state meets the preconditions and effects represent the modifications induced by the action. For example, an action might be “move the robot from location A to location B” or “pick up the box from the table”. To solve the symbolic planning problem, we create a graph using these propositions and use A* graph search to solve for the path.
Dynamic Motion Primitives
Dynamic movement primitive (DMP) is a method of trajectory generation from a “primitive” demonstration. The DMPs can be used to generate trajectory from different starting location and different goal locations with a single demonstrated “primitive”. The DMPs are formulated as a spring- damper system or point-attractor dynamics. In our method, we use the DMPs to execute skills that are generated by the task planner. We initially use predefined linear and super eclipse curves to generate initial DMP demonstrations to execute different pick and place skills.
Re-planning strategy
The symbolic start and goal states are sent to the symbolic planner, along with the corresponding environment file containing the list of symbols, and actions with preconditions and effects. The Symbolic planner generates a sequence of grounded actions to be taken by the DMP planner. The DMP interprets the symbolic grounded actions and generates the sequence of feasible geometric trajectories to be taken by the manipulator. If the DMP is unable to find a feasible trajectory for a given action, we note the object that the trajectory collides with and add the negation of its state as the precondition for that grounded action and re-plan. In case this re-planning strategy fails to generate a feasible trajectory, we remove the infeasible grounded action from the exhaustive list of actions and re-plan. The figure describes the proposed re-planning strategy as a flowchart.
Results
To test our approach, we have carried out two sets of experiments. First, we defined 2 additional initial states with different number of objects for the symbolic planner to solve. From the table, we can see that the symbolic planning times reduce from environment A to C as we reduce the number of objects. Additionally, we also experimented with the DMP to note the planning times for different lengths of the geometric paths. As DMP solves a differential equation, the planning times increase with increase in the number of integration steps due to the higher length of the path. The same can be seen in the DMP results table where we have shown the time taken by DMP to generate paths for 4 different path lengths.
Here is a demo video showing the full framework in action:
Conclusion
In this project, we introduced a hierarchical approach to address a robotics manipulation task, showcasing the integration of a task-level symbolic planner with Dynamic Movement Primitives (DMPs). Our demonstration focused on planning and executing a sequence for decluttering a breakfast tray, efficiently placing various objects onto designated shelves. Moreover, we proposed a replanning strategy for the symbolic planner to handle cases where DMPs encounter difficulties in executing collision-free paths. The results highlight the effectiveness of our planner in household robotic manipulation, successfully organizing the tray items. Throughout this endeavor, I gained insights into the capabilities of symbolic planning for tackling intricate tasks and the complexities associated with Task and Motion Planning. Additionally, I acquired practical knowledge on developing and integrating a hierarchical planner into the Robot Operating System (ROS) and the CoppeliaSim V-REP simulation environment.