1st Workshop on Maritime Computer Vision (MaCVi)

SeaDronesSee Multi-Object Tracking


TLDR

Download the SeaDronesSee Multi-Object Tracking (all-objects-in-water) dataset on the dataset page. Upload a json-file with predictions on the test set on the upload page.

Overview

The SeaDronesSee benchmark was recently published in WACV 2022. The goal of this benchmark is to advance computer vision algorithms in maritime search and rescue missions. It is aimed at detecting humans, boats and other objects in open water. See the Explore page for some annotation examples. The task of airborne multi-object tracking in the maritime domain is far from solved. The small sizes of objects in conjunction with hardly visible objects due to waves and sun reflections make the detection as well as tracking hard. What is more, gimbal movement and altitude change cause objects to move quickly within the video. The occasional partial occlusion of objects are on top of the other challenges. We provide tracking id labels alongside the bounding box labels for several videos where the task is to track swimmers and boats. This is a short-term tracking task, i.e. objects that disappear from the scene do not to be tracked anymore.

Task

Create a multi-object tracker that, given a video, outputs bounding boxes and tracking ids for every object (boats, swimmer, floater). We group all classes into a single class. Also see the dataset page for an overview. This task is identical to the MOT-All-Objects-In-Water benchmark introduced in the original paper with minor changes in the annotation file. This is a short-term tracking task, i.e. objects that disappear from the scene need not be tracked anymore.

Dataset

We provide 21 clips in the train set, 17 clips in the validation set and 19 clips in the test set with a total of 54,105 frames and 403,192 annotated instances. Note that the video clip ids in the annotations files are not necessarily increasing integers (some integers are missing). Furthermore, there are some images having a video id that is not contained in the train,val and set annotation files. However, this applies to few images only. Each image has precise meta data labels about altitude, angles of the UAV and the gimbal, GPS, and more.
Admittedly, the annotation files are not super clean and you might have to adapt a little bit here and there. Feel free to contact me any time if you find some irregularities (benjamin.kiefer@uni-tuebingen.de).

Evaluation Metrics

We evaluate your predictions on HOTA, MOTA, IDF1, MOTP, MT, ML, FP, FN, Recall, Precision, ID Switches, Frag. The determining metric for winning is HOTA. In case of tie, MOTA is the tiebreaker.

Furthermore, we require every participant to submit information on the speed of their method measured in frames per second walltime. Please also indicate the hardware that you used. Lastly, you should indicate which data sets (also for pretraining) you used during training and whether you used meta data.

Participate

In order to participate, you can perform the following steps:
  1. Download the dataset SeaDronesSee Multi-Object Tracking on the dataset page.
  2. Visualize the bounding boxes using these Python scripts or Google colabs. You should adapt the paths for that.
  3. Train a multi-object trackinger of your choice on the dataset. We try to provide you with a sample multi-object tracking pipeline on Git soon. We also provide public detections that you can use so that you do not need to train your own detector. It's from a Yolov7 model trained on SeaDronesSee-MOT train set for 8 epochs with a AP of roughly 0.5. For reference, the same model (except for the number of class outputs) has a AP of 0.4181 on Object Detection v2. So, it's not the best, but solid.
  4. You need to create a json-prediction file for upload. In the following, we will illustrate the structue of the json based on an example:
    See the MOT_example_submission.json. You can load it in Python via

    import json
    file=open('MOT_example_submission.json','r')
    data=json.load(file)

    This yields you a list where you find that this list is of length = number of frames in entire test set (all videos), i.e. 18,253. You can compare that to the length of the list in the corresponding test-json on Nextcloud.
    The frames of the videos should be ordered according to the following list:

    • DJI_0001.mov
    • DJI_0051.MP4
    • DJI_0065.MP4
    • DJI_0001_d3.mov
    • DJI_0039.MP4
    • DJI_0003.mov
    • DJI_0064.MP4
    • DJI_0069.MP4
    • DJI_0011_d3.mov
    • DJI_0057.MP4
    • DJI_0032.MP4
    • DJI_0001.MOV
    • DJI_0010_d3.mov
    • DJI_0063.MP4
    • DJI_0059.MP4
    • DJI_0006_d3.mov
    • DJI_0055.MP4
    • DJI_0041.MP4
    • DJI_0038.MP4
    These videos were previously not listed but ARE included.
    Also find the respective information on the video names in the test set json-annotation. Note the difference between DJI_0001.MOV and DJI_0001.mov. Furthermore note, that the number of videos is 19, not 22 or 20!

    Each entry in the above list contains a single list. This list contains a single list or multiple lists. If you found no object on that frame, then there is only a single empty list, so that the entry is:

    [[]]

    For example, in the sample submission the first frame does not contain any prediction, i.e.

    data[0]=[[]]

    If you found objects, then there is one or more lists contained. For example, in the sample submission the 101th frame contains three predicted objects:

    data[100]=[[[1.0, 473.14544677734375, 288.2177429199219, 595.2658081054688, 410.7237548828125, 0.9918640851974487],
               [4.0, 3776.576171875, 331.8674621582031, 3847.074951171875, 403.09637451171875, 0.7565602660179138],
               [3.0, 2151.9912109375, 24.94463348388672, 2346.65087890625, 79.3317642211914, 0.5624877214431763]]]


    Each object is itself encoded as a list. For example, see the first object in the 101th frame:

    data[100][0][0]=[1.0, 473.14544677734375, 288.2177429199219, 595.2658081054688, 410.7237548828125, 0.9918640851974487]

    The numbers mean the following:

    object id,bbox_left,bbox_top,bbox_right,bbox_bottom,confidence

    The object id must be an integer (not the datatype but after rounding to the next integer it should be the same value). The confidence should be a float between 0 and 1. The coordinates should be self-explanatory. Let us know if you need further information.
  5. Upload your json-file along with all the required information in the form here. You need to register first.
  6. If you provided a valid file and sufficient information, you should see your method on the leaderboard. Now, you can reiterate. You may upload at most three times per day (independent of the challenge track). Note that the upload can take a few minutes. You can leave the page although the loading sign of your browser indicates otherwise.

Terms and Conditions

In case of any questions regarding the challenge modalities or uploads, please direct them to Benjamin Kiefer.