[Community] Add IDOL, VITA to project

[pixeli99]

No description provided.

[pixeli99]

There is a bug about data reading.

Under repair.

[SupreethN]

Hi,

Thank you for the fantastic work on developing Mask2Former, IDOL, and VITA using mmtracking. I can successfully run Mask2Former. However, I run into some issues running IDOL and VITA.

For IDOL, I get errors during the training stage. And I get the following error:

Traceback (most recent call last):
  File "tools/train.py", line 119, in <module>
    main()
  File "tools/train.py", line 115, in main
    runner.train()
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1678, in train
    model = self.train_loop.run()  # type: ignore
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/runner/loops.py", line 264, in run
    self.run_iter(data_batch)
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/runner/loops.py", line 287, in run_iter
    outputs = self.runner.model.train_step(
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/model/base_model/base_model.py", line 114, in train_step
    losses = self._run_forward(data, mode='loss')  # type: ignore
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/model/base_model/base_model.py", line 314, in _run_forward
    results = self(**data, mode=mode)
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/supreeth/mmtracking_vis_sota/mmtrack/models/mot/base.py", line 106, in forward
    return self.loss(inputs, data_samples, **kwargs)
  File "/home/supreeth/mmtracking_vis_sota/projects/VIS_SOTA/IDOL/idol_src/idol.py", line 53, in loss
    losses = self.track_head.loss(x, data_samples)
  File "/home/supreeth/mmtracking_vis_sota/projects/VIS_SOTA/IDOL/idol_src/models/idol_query_track_head.py", line 488, in loss
    losses = self.loss_by_feat(*loss_inputs)
  File "/home/supreeth/mmtracking_vis_sota/projects/VIS_SOTA/IDOL/idol_src/models/idol_query_track_head.py", line 534, in loss_by_feat
    losses_mask, losses_dice, min_cost_qids = multi_apply(
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmdet/models/utils/misc.py", line 219, in multi_apply
    return tuple(map(list, zip(*map_results)))
  File "/home/supreeth/mmtracking_vis_sota/projects/VIS_SOTA/IDOL/idol_src/models/idol_query_track_head.py", line 681, in loss_by_feat_single
    loss_dice = self.loss_dice(
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmdet/models/losses/dice_loss.py", line 137, in forward
    loss = self.loss_weight * dice_loss(
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmdet/models/losses/dice_loss.py", line 47, in dice_loss
    a = torch.sum(input * target, 1)
RuntimeError: The size of tensor a (8) must match the size of tensor b (600) at non-singleton dimension 0

---

Regarding VITA, I can train the model successfully. However, during evaluation, I get the following error:

Traceback (most recent call last):
  File "tools/test.py", line 88, in <module>
    main()
  File "tools/test.py", line 84, in main
    runner.test()
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1724, in test
    metrics = self.test_loop.run()  # type: ignore
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/runner/loops.py", line 417, in run
    self.run_iter(idx, data_batch)
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/runner/loops.py", line 436, in run_iter
    outputs = self.runner.model.test_step(data_batch)
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/model/base_model/base_model.py", line 145, in test_step
    return self._run_forward(data, mode='predict')  # type: ignore
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/model/base_model/base_model.py", line 314, in _run_forward
    results = self(**data, mode=mode)
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/supreeth/mmtracking_vis_sota/mmtrack/models/mot/base.py", line 108, in forward
    return self.predict(inputs, data_samples, **kwargs)
  File "/home/supreeth/mmtracking_vis_sota/projects/VIS_SOTA/VITA/vita_src/vita.py", line 106, in predict
    _frame_queries, _mask_features = self.seg_head.predict(feats)
  File "/home/supreeth/mmtracking_vis_sota/projects/VIS_SOTA/VITA/vita_src/models/vita_seg_head.py", line 217, in predict
    all_frame_queries, mask_features = self(x)
ValueError: too many values to unpack (expected 2)

---

Can you please help me with this @pixeli99?

Thanks!

[pixeli99]

I'm glad you are interested in this part of code, but unfortunately, the current code support for VITA is not complete, which will be repaired after the GenVIS reproduction is completed. If you need to do research, you can try the official repo first. The reproduction is expected to be completed in two weeks ✅ In addition, if you have any suggestions on the code or want to join me, I am also very welcome ！😺

[SupreethN]

Thank you for the swift reply. That's great to know! I will checkout the official code repo once. I will let you know if I have any suggestions on the code!

[SupreethN]

Hi @pixeli99 ! Is there any update on the IDOL/ VITA code yet? I am looking forward to trying them! Thanks!