|
| 1 | +# Create Your Model and Agent |
| 2 | + |
| 3 | +## Development Overview |
| 4 | +The main architecture of the evaluation code adopts a client-server model. In the client, we specify the corresponding configuration (*.cfg), which includes settings such as the scenarios to be evaluated, robots, models, and parallelization parameters. The client sends requests to the server, which then make model to predict and response to the client. |
| 5 | + |
| 6 | +The InternNav project adopts a modular design, allowing developers to easily add new navigation algorithms. |
| 7 | +The main components include: |
| 8 | + |
| 9 | +- **Model**: Implements the specific neural network architecture and inference logic |
| 10 | + |
| 11 | +- **Agent**: Serves as a wrapper for the Model, handling environment interaction and data preprocessing |
| 12 | + |
| 13 | +- **Config**: Defines configuration parameters for the model and training |
| 14 | + |
| 15 | +## Custom Model |
| 16 | +A Model is the concrete implementation of your algorithm. Implement model under `baselines/models`. A model ideally would inherit from the base model and implement the following key methods: |
| 17 | + |
| 18 | +- `forward(train_batch) -> dict(output, loss)` |
| 19 | +- `inference(obs_batch, state) -> output_for_agent` |
| 20 | + |
| 21 | +## Create a Custom Config Class |
| 22 | + |
| 23 | +In the model file, define a `Config` class that inherits from `PretrainedConfig`. |
| 24 | +A reference implementation is `CMAModelConfig` in [`cma_model.py`](../internnav/model/cma/cma_policy.py). |
| 25 | + |
| 26 | +## Registration and Integration |
| 27 | + |
| 28 | +In [`internnav/model/__init__.py`](../internnav/model/__init__.py): |
| 29 | +- Add the new model to `get_policy`. |
| 30 | +- Add the new model's configuration to `get_config`. |
| 31 | + |
| 32 | +## Create a Custom Agent |
| 33 | + |
| 34 | +The Agent handles interaction with the environment, data preprocessing/postprocessing, and calls the Model for inference. |
| 35 | +A custom Agent usually inherits from [`Agent`](../internnav/agent/base.py) and implements the following key methods: |
| 36 | + |
| 37 | +- `reset()`: Resets the Agent's internal state (e.g., RNN states, action history). Called at the start of each episode. |
| 38 | +- `inference(obs)`: Receives environment observations `obs`, performs preprocessing (e.g., tokenizing instructions, padding), calls the model for inference, and returns an action. |
| 39 | +- `step(obs)`: The external interface, usually calls `inference`, and can include logging or timing. |
| 40 | + |
| 41 | +Example: [`CMAAgent`](../internnav/agent/cma_agent.py) |
| 42 | + |
| 43 | +For each step, the agent should expect an observation from environment. |
| 44 | + |
| 45 | +For the vln benchmark under internutopia: |
| 46 | + |
| 47 | +``` |
| 48 | +action = self.agent.step(obs) |
| 49 | +``` |
| 50 | +**obs** has format: |
| 51 | +``` |
| 52 | +obs = [{ |
| 53 | + 'globalgps': [X, Y, Z] # robot location |
| 54 | + 'globalrotation': [X, Y, Z, W] # robot orientation in quaternion |
| 55 | + 'rgb': np.array(256, 256, 3) # rgb camera image |
| 56 | + 'depth': np.array(256, 256, 1) # depth image |
| 57 | +}] |
| 58 | +``` |
| 59 | +**action** has format: |
| 60 | +``` |
| 61 | +action = List[int] # action for each environments |
| 62 | +# 0: stop |
| 63 | +# 1: move forward |
| 64 | +# 2: turn left |
| 65 | +# 3: turn right |
| 66 | +``` |
| 67 | + |
| 68 | +## Create a Trainer |
| 69 | + |
| 70 | +The Trainer manages the training loop, including data loading, forward pass, loss calculation, and backpropagation. |
| 71 | +A custom trainer usually inherits from the [`Base Trainer`](../internnav/trainer/base.py) and implements: |
| 72 | + |
| 73 | +- `train_epoch()`: Runs one training epoch (batch iteration, forward pass, loss calculation, parameter update). |
| 74 | +- `eval_epoch()`: Evaluates the model on the validation set and records metrics. |
| 75 | +- `save_checkpoint()`: Saves model weights, optimizer state, and training progress. |
| 76 | +- `load_checkpoint()`: Loads pretrained models or resumes training. |
| 77 | + |
| 78 | +Example: [`CMATrainer`](../internnav/trainer/cma_trainer.py) shows how to handle sequence data, compute action loss, and implement imitation learning. |
| 79 | + |
| 80 | +## Training Data |
| 81 | + |
| 82 | +The training data is under `data/vln_pe/traj_data`. Our dataset provides trajectory data collected from the H1 robot as it navigates through the task environment. |
| 83 | +Each observation in the trajectory is paired with its corresponding action. |
| 84 | + |
| 85 | +You may also incorporate external datasets to improve model generalization. |
| 86 | + |
| 87 | +## Evaluation Data |
| 88 | +In `raw_data/val`, for each task, the model should guide the robot at the start position and rotation to the target position with language instruction. |
| 89 | + |
| 90 | +## Set the Corresponding Configuration |
| 91 | + |
| 92 | +Refer to existing **training** configuration files for customization: |
| 93 | + |
| 94 | +- **CMA Model Config**: [`cma_exp_cfg`](../scripts/train/configs/cma.py) |
| 95 | + |
| 96 | +Configuration files should define: |
| 97 | +- `ExpCfg` (experiment config) |
| 98 | +- `EvalCfg` (evaluation config) |
| 99 | +- `IlCfg` (imitation learning config) |
| 100 | + |
| 101 | +Ensure your configuration is imported and registered in [`__init__.py`](../scripts/train/configs/__init__.py). |
| 102 | + |
| 103 | +Key parameters include: |
| 104 | +- `name`: Experiment name |
| 105 | +- `model_name`: Must match the name used during model registration |
| 106 | +- `batch_size`: Batch size |
| 107 | +- `lr`: Learning rate |
| 108 | +- `epochs`: Number of training epochs |
| 109 | +- `dataset_*_root_dir`: Dataset paths |
| 110 | +- `lmdb_features_dir`: Feature storage path |
| 111 | + |
| 112 | +Refer to existing **evaluation** config files for customization: |
| 113 | + |
| 114 | +- **CMA Model Evaluation Config**: [`h1_cma_cfg.py`](../scripts/eval/configs/h1_cma_cfg.py) |
| 115 | + |
| 116 | +Main fields: |
| 117 | +- `name`: Evaluation experiment name |
| 118 | +- `model_name`: Must match the name used during training |
| 119 | +- `ckpt_to_load`: Path to the model checkpoint |
| 120 | +- `task`: Define the tasks settings, number of env, scene, robots |
| 121 | +- `dataset`: Load r2r or interiornav dataset |
| 122 | +- `split`: Dataset split (`val_seen`, `val_unseen`, `test`, etc.) |
0 commit comments