Humanoid Standup#
This environment is part of the Mujoco environments. Please read that page first for general information.
Action Space 
Box(0.4, 0.4, (17,), float32) 
Observation Shape 
(376,) 
Observation High 
[inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf] 
Observation Low 
[inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf inf] 
Import 

Description#
This environment is based on the environment introduced by Tassa, Erez and Todorov in “Synthesis and stabilization of complex behaviors through online trajectory optimization”. The 3D bipedal robot is designed to simulate a human. It has a torso (abdomen) with a pair of legs and arms. The legs each consist of two links, and so the arms (representing the knees and elbows respectively). The environment starts with the humanoid laying on the ground, and then the goal of the environment is to make the humanoid standup and then keep it standing by applying torques on the various hinges.
Action Space#
The agent take a 17element vector for actions.
The action space is a continuous (action, ...)
all in [1, 1]
, where action
represents the numerical torques applied at the hinge joints.
 Num  Action  Control Min  Control Max  Name (in corresponding XML file)  Joint  Unit  —–——————————————————————————————————————————————– ——————–  0  Torque applied on the hinge in the ycoordinate of the abdomen  0.4  0.4  hip_1 (front_left_leg)  hinge  torque (N m)   1  Torque applied on the hinge in the zcoordinate of the abdomen  0.4  0.4  angle_1 (front_left_leg)  hinge  torque (N m)   2  Torque applied on the hinge in the xcoordinate of the abdomen  0.4  0.4  hip_2 (front_right_leg)  hinge  torque (N m)   3  Torque applied on the rotor between torso/abdomen and the right hip (xcoordinate)  0.4  0.4  right_hip_x (right_thigh)  hinge  torque (N m)   4  Torque applied on the rotor between torso/abdomen and the right hip (zcoordinate)  0.4  0.4  right_hip_z (right_thigh)  hinge  torque (N m)   5  Torque applied on the rotor between torso/abdomen and the right hip (ycoordinate)  0.4  0.4  right_hip_y (right_thigh)  hinge  torque (N m)   6  Torque applied on the rotor between the right hip/thigh and the right shin  0.4  0.4  right_knee  hinge  torque (N m)   7  Torque applied on the rotor between torso/abdomen and the left hip (xcoordinate)  0.4  0.4  left_hip_x (left_thigh)  hinge  torque (N m)   8  Torque applied on the rotor between torso/abdomen and the left hip (zcoordinate)  0.4  0.4  left_hip_z (left_thigh)  hinge  torque (N m)   9  Torque applied on the rotor between torso/abdomen and the left hip (ycoordinate)  0.4  0.4  left_hip_y (left_thigh)  hinge  torque (N m)   10  Torque applied on the rotor between the left hip/thigh and the left shin  0.4  0.4  left_knee  hinge  torque (N m)   11  Torque applied on the rotor between the torso and right upper arm (coordinate 1)  0.4  0.4  right_shoulder1  hinge  torque (N m)   12  Torque applied on the rotor between the torso and right upper arm (coordinate 2)  0.4  0.4  right_shoulder2  hinge  torque (N m)   13  Torque applied on the rotor between the right upper arm and right lower arm  0.4  0.4  right_elbow  hinge  torque (N m)   14  Torque applied on the rotor between the torso and left upper arm (coordinate 1)  0.4  0.4  left_shoulder1  hinge  torque (N m)   15  Torque applied on the rotor between the torso and left upper arm (coordinate 2)  0.4  0.4  left_shoulder2  hinge  torque (N m)   16  Torque applied on the rotor between the left upper arm and left lower arm  0.4  0.4  left_elbow  hinge  torque (N m) 
Observation Space#
The state space consists of positional values of different body parts of the Humanoid, followed by the velocities of those individual parts (their derivatives) with all the positions ordered before all the velocities.
The observation is a ndarray
with shape (376,)
where the elements correspond to the following:
Num 
Observation 
Min 
Max 
Name (in corresponding XML file) 
Joint 
Unit 

0 
xcoordinate of the torso (centre) 
Inf 
Inf 
root 
free 
position (m) 
1 
ycoordinate of the torso (centre) 
Inf 
Inf 
root 
free 
position (m) 
2 
zcoordinate of the torso (centre) 
Inf 
Inf 
root 
free 
position (m) 
3 
xorientation of the torso (centre) 
Inf 
Inf 
root 
free 
angle (rad) 
4 
yorientation of the torso (centre) 
Inf 
Inf 
root 
free 
angle (rad) 
5 
zorientation of the torso (centre) 
Inf 
Inf 
root 
free 
angle (rad) 
6 
worientation of the torso (centre) 
Inf 
Inf 
root 
free 
angle (rad) 
7 
zangle of the abdomen (in lower_waist) 
Inf 
Inf 
abdomen_z 
hinge 
angle (rad) 
8 
yangle of the abdomen (in lower_waist) 
Inf 
Inf 
abdomen_y 
hinge 
angle (rad) 
9 
xangle of the abdomen (in pelvis) 
Inf 
Inf 
abdomen_x 
hinge 
angle (rad) 
10 
xcoordinate of angle between pelvis and right hip (in right_thigh) 
Inf 
Inf 
right_hip_x 
hinge 
angle (rad) 
11 
zcoordinate of angle between pelvis and right hip (in right_thigh) 
Inf 
Inf 
right_hip_z 
hinge 
angle (rad) 
12 
ycoordinate of angle between pelvis and right hip (in right_thigh) 
Inf 
Inf 
right_hip_y 
hinge 
angle (rad) 
13 
angle between right hip and the right shin (in right_knee) 
Inf 
Inf 
right_knee 
hinge 
angle (rad) 
14 
xcoordinate of angle between pelvis and left hip (in left_thigh) 
Inf 
Inf 
left_hip_x 
hinge 
angle (rad) 
15 
zcoordinate of angle between pelvis and left hip (in left_thigh) 
Inf 
Inf 
left_hip_z 
hinge 
angle (rad) 
16 
ycoordinate of angle between pelvis and left hip (in left_thigh) 
Inf 
Inf 
left_hip_y 
hinge 
angle (rad) 
17 
angle between left hip and the left shin (in left_knee) 
Inf 
Inf 
left_knee 
hinge 
angle (rad) 
18 
coordinate1 (multiaxis) angle between torso and right arm (in right_upper_arm) 
Inf 
Inf 
right_shoulder1 
hinge 
angle (rad) 
19 
coordinate2 (multiaxis) angle between torso and right arm (in right_upper_arm) 
Inf 
Inf 
right_shoulder2 
hinge 
angle (rad) 
20 
angle between right upper arm and right_lower_arm 
Inf 
Inf 
right_elbow 
hinge 
angle (rad) 
21 
coordinate1 (multiaxis) angle between torso and left arm (in left_upper_arm) 
Inf 
Inf 
left_shoulder1 
hinge 
angle (rad) 
22 
coordinate2 (multiaxis) angle between torso and left arm (in left_upper_arm) 
Inf 
Inf 
left_shoulder2 
hinge 
angle (rad) 
23 
angle between left upper arm and left_lower_arm 
Inf 
Inf 
left_elbow 
hinge 
angle (rad) 
24 
xcoordinate velocity of the torso (centre) 
Inf 
Inf 
root 
free 
velocity (m/s) 
25 
ycoordinate velocity of the torso (centre) 
Inf 
Inf 
root 
free 
velocity (m/s) 
26 
zcoordinate velocity of the torso (centre) 
Inf 
Inf 
root 
free 
velocity (m/s) 
27 
xcoordinate angular velocity of the torso (centre) 
Inf 
Inf 
root 
free 
angular velocity (rad/s) 
28 
ycoordinate angular velocity of the torso (centre) 
Inf 
Inf 
root 
free 
angular velocity (rad/s) 
29 
zcoordinate angular velocity of the torso (centre) 
Inf 
Inf 
root 
free 
angular velocity (rad/s) 
30 
zcoordinate of angular velocity of the abdomen (in lower_waist) 
Inf 
Inf 
abdomen_z 
hinge 
angular velocity (rad/s) 
31 
ycoordinate of angular velocity of the abdomen (in lower_waist) 
Inf 
Inf 
abdomen_y 
hinge 
angular velocity (rad/s) 
32 
xcoordinate of angular velocity of the abdomen (in pelvis) 
Inf 
Inf 
abdomen_x 
hinge 
angular velocity (rad/s) 
33 
xcoordinate of the angular velocity of the angle between pelvis and right hip (in right_thigh) 
Inf 
Inf 
right_hip_x 
hinge 
angular velocity (rad/s) 
34 
zcoordinate of the angular velocity of the angle between pelvis and right hip (in right_thigh) 
Inf 
Inf 
right_hip_z 
hinge 
angular velocity (rad/s) 
35 
ycoordinate of the angular velocity of the angle between pelvis and right hip (in right_thigh) 
Inf 
Inf 
right_hip_y 
hinge 
angular velocity (rad/s) 
36 
angular velocity of the angle between right hip and the right shin (in right_knee) 
Inf 
Inf 
right_knee 
hinge 
angular velocity (rad/s) 
37 
xcoordinate of the angular velocity of the angle between pelvis and left hip (in left_thigh) 
Inf 
Inf 
left_hip_x 
hinge 
angular velocity (rad/s) 
38 
zcoordinate of the angular velocity of the angle between pelvis and left hip (in left_thigh) 
Inf 
Inf 
left_hip_z 
hinge 
angular velocity (rad/s) 
39 
ycoordinate of the angular velocity of the angle between pelvis and left hip (in left_thigh) 
Inf 
Inf 
left_hip_y 
hinge 
angular velocity (rad/s) 
40 
angular velocity of the angle between left hip and the left shin (in left_knee) 
Inf 
Inf 
left_knee 
hinge 
angular velocity (rad/s) 
41 
coordinate1 (multiaxis) of the angular velocity of the angle between torso and right arm (in right_upper_arm) 
Inf 
Inf 
right_shoulder1 
hinge 
angular velocity (rad/s) 
42 
coordinate2 (multiaxis) of the angular velocity of the angle between torso and right arm (in right_upper_arm) 
Inf 
Inf 
right_shoulder2 
hinge 
angular velocity (rad/s) 
43 
angular velocity of the angle between right upper arm and right_lower_arm 
Inf 
Inf 
right_elbow 
hinge 
angular velocity (rad/s) 
44 
coordinate1 (multiaxis) of the angular velocity of the angle between torso and left arm (in left_upper_arm) 
Inf 
Inf 
left_shoulder1 
hinge 
angular velocity (rad/s) 
45 
coordinate2 (multiaxis) of the angular velocity of the angle between torso and left arm (in left_upper_arm) 
Inf 
Inf 
left_shoulder2 
hinge 
angular velocity (rad/s) 
46 
angular velocity of the angle between left upper arm and left_lower_arm 
Inf 
Inf 
left_elbow 
hinge 
angular velocity (rad/s) 
Additionally, after all the positional and velocity based values in the table, the state_space consists of (in order):
cinert: Mass and inertia of a single rigid body relative to the center of mass (this is an intermediate result of transition). It has shape 14*10 (nbody * 10) and hence adds to another 140 elements in the state space.
cvel: Center of mass based velocity. It has shape 14 * 6 (nbody * 6) and hence adds another 84 elements in the state space
qfrc_actuator: Constraint force generated as the actuator force. This has shape
(23,)
(nv * 1) and hence adds another 23 elements to the state space.cfrc_ext: This is the center of mass based external force on the body. It has shape 14 * 6 (nbody * 6) and hence adds to another 84 elements in the state space. where nbody stands for the number of bodies in the robot and nv stands for the number of degrees of freedom (= dim(qvel))
The (x,y,z) coordinates are translational DOFs while the orientations are rotational DOFs expressed as quaternions. One can read more about free joints on the Mujoco Documentation.
Note: There are 47 elements in the table above  giving rise to (378,)
elements in the state space. In practice (and Gym implementation), the first two
positional elements are omitted from the state space since the reward function is
calculated based on the xcoordinate value. This value is hidden from the algorithm,
which in turn has to develop an abstract understanding of it from the observed rewards.
Therefore, observation space has shape (376,)
instead of (378,)
and the table should not have the first two rows.
Note: HumanoidStandupv4 environment no longer has the following contact forces issue. If using previous HumanoidStandup versions from v4, there have been reported issues that using a MujocoPy version > 2.0 results in the contact forces always being 0. As such we recommend to use a MujocoPy version < 2.0 when using the Humanoid environment if you would like to report results with contact forces (if contact forces are not used in your experiments, you can use version > 2.0).
Rewards#
The reward consists of three parts:
uph_cost: A reward for moving upward (in an attempt to stand up). This is not a relative reward which measures how much upward it has moved from the last timestep, but it is an absolute reward which measures how much upward the Humanoid has moved overall. It is measured as (z coordinate after action  0)/(atomic timestep), where z coordinate after action is index 0 in the state/index 2 in the table, and atomic timestep is the time for one frame of movement even though the simulation has a framerate of 5 (done in order to inflate rewards a little for faster learning).
quad_ctrl_cost: A negative reward for penalising the humanoid if it has too large of a control force. If there are nu actuators/controls, then the control has shape
nu x 1
. It is measured as 0.1 x sum(control^{2}).quad_impact_cost: A negative reward for penalising the humanoid if the external contact force is too large. It is calculated as min(0.5 * 0.000001 * sum(external contact force^{2}), 10).
The total reward returned is reward = uph_cost + 1  quad_ctrl_cost  quad_impact_cost
Starting State#
All observations start in state (0.0, 0.0, 0.105, 1.0, 0.0 … 0.0) with a uniform noise in the range of [0.01, 0.01] added to the positional and velocity values (values in the table) for stochasticity. Note that the initial z coordinate is intentionally selected to be low, thereby indicating a laying down humanoid. The initial orientation is designed to make it face forward as well.
Episode Termination#
The episode terminates when any of the following happens:
The episode duration reaches a 1000 timesteps
Any of the state space values is no longer finite
Arguments#
No additional arguments are currently supported (in v2 and lower), but modifications can be made to the XML file in the assets folder (or by changing the path to a modified XML file in another folder)..
env = gym.make('HumanoidStandupv2')
There is no v3 for HumanoidStandup, unlike the robot environments where a v3 take gym.make kwargs such as xml_file, ctrl_cost_weight, reset_noise_scale etc.
There is a v4 version that uses the mujocobindings
env = gym.make('HumanoidStandupv4')
Version History#
v4: all mujoco environments now use the mujoco bindings in mujoco>=2.1.3
v3: support for gym.make kwargs such as xml_file, ctrl_cost_weight, reset_noise_scale etc. rgb rendering comes from tracking camera (so agent does not run away from screen)
v2: All continuous control environments now use mujoco_py >= 1.50
v1: max_time_steps raised to 1000 for robot based tasks. Added reward_threshold to environments.
v0: Initial versions release (1.0.0)