JapanResearchers at the University of Tokyo, in collaboration with Alternative Machine, have made a breakthrough by developing a robot that can directly map natural language commands to robot actions.Humanoid RobotSystem Alter3. Its background model adopts GPT-4Technology that can complete a range of complex tasks, such as taking selfies or playing ghost.
It’s one of a growing number of studies that combine basic models with robotic systems. While these systems haven’t yet reached scalable commercial solutions, they have advanced robotics research in recent years and show great potential.
Alter3 uses GPT-4 technology as a background model, receiving natural language instructions that describe actions or situations in which the robot needs to react. First, the model uses the "agent framework" to plan a series of action steps required for the robot to complete its goals. Second, by encoding the agent, it generates the commands required for the robot to perform each step. Since GPT-4 was not trained with Alter3's programming commands, the researchers used its contextual learning capabilities to adapt its behavior to the robot's API.
Therefore, the prompt contains a list of commands and a set of examples of how to use each command. The model then maps each step to one or more API commands to send to the robot for execution.
The researchers added functionality so that humans can provide feedback, such as "raise your arm a little higher." These instructions are sent to another GPT-4 agent, which reasons about the code, makes necessary corrections, and returns the action sequence to the robot. The improved action recipes and code are stored in a database for future use.
The researchers ran several tests on Alter3, including everyday actions, such as taking selfies and drinking tea, and imitating actions, such as playing a ghost or a snake. They also tested the model's ability to cope with situations that require carefully planned actions. GPT-4's extensive understanding of human behavior and actions allows for the creation of more realistic behavior plans for humanoid robots such as Alter3. The researchers' experiments also showed that they were able to imitate emotions such as shame and joy in the robot.