2^nd Workshop on Maritime Computer Vision (MaCVi)

BoaTrack - Multi-Object Tracking of Boats

Quick links: Dataset download Submit Leaderboard Ask for help

TLDR

Download the BoaTrack dataset on the dataset page. Upload a json-file with predictions on the test set on the upload page.

Overview

Task

Create a multi-object tracker that, given a video, outputs bounding boxes and tracking ids for every object (boat). This is a short-term tracking task, i.e. objects that disappear from the scene need not be re-identified if they reapper in the same video clip again at a later point.

Dataset

We provide 3 clips in the test set. We withold the annotations. On the server-side, we compare your predictions against the annotations. Please note, that we do not provide any train or val videos. This means, you need to train a detector on other datasets, such as LaRS (from the other competition). You are free to use any other publicly available dataset as long as you provide the link/source to it in the upload. Please contact me any time if you find some irregularities (benjamin.kiefer@uni-tuebingen.de).

Evaluation Metrics

We evaluate your predictions on HOTA, MOTA, IDF1, MOTP, MT, ML, FP, FN, Recall, Precision, ID Switches, Frag. The determining metric for winning is HOTA. In case of tie, MOTA is the tiebreaker.

Furthermore, we require every participant to submit information on the speed of their method measured in frames per second walltime. Please also indicate the hardware that you used. Lastly, you should indicate which data sets (also for pretraining) you used during training and whether you used meta data.

Participate

In order to participate, you can perform the following steps:

1. Download the dataset BoaTrack (aka the three videos) on the dataset page.
2. Train a detector+multi-object tracker of your choice on any publicly available dataset (e.g. LaRS).
3. You need to create a json-prediction file on the test set for upload. In the following, we will illustrate the structure of the json based on an example:
   See the BoaTrack_example_submission.json. You can load it in Python via
   
   import json
file=open('BoaTrack_example_submission.json','r')
data=json.load(file)

   This yields you a list where you find that this list is of length = number of frames in entire test set (all videos), i.e. 10,656.
   The frames of the videos should be ordered according to the following list:
   
   
   • 366_cut.avi
   • 75.avi
   • 189_cut.avi
   
   Each entry in the above list contains a single list. This list contains a single list or multiple lists. If you found no object on that frame, then there is only a single empty list, so that the entry is:
   
   [[]]
   
   For example, in the sample submission the first frame does not contain any prediction, i.e.
   
   data[0]=[[]]
   
   If you found objects, then there is one or more lists contained. For example, in the sample submission the 901th frame contains three predicted objects:
   
   data[900]=[[[4, 372.8047619047619, 398.5, 393.4, 429.5, 1.0],
           [5, 189.05714285714268, 374.0000000000001, 216.57142857142858, 408.42857142857144, 1.0],
           [6, 131.71428571428578, 354.0, 151.39999999999998, 400.9, 1.0]]]
   
   Each object is itself encoded as a list. For example, see the first object in the 901th frame:
   
   data[900][0][0]=[4, 372.8047619047619, 398.5, 393.4, 429.5, 1.0]
   
   The numbers mean the following:
   
   object id,bbox_left,bbox_top,bbox_right,bbox_bottom,confidence
   
   The object id must be an integer. The confidence should be a float between 0 and 1. The coordinates should be self-explanatory. Let us know if you need further information.
4. Upload your json-file along with all the required information in the form here. You need to register first.
5. If you provided a valid file and sufficient information, you should see your method on the leaderboard. Since it only has three detections, it will be at the bottom of the leaderboard. Now, you can reiterate. You may upload at most once per day per challenge. Note that the upload can take a few minutes.

Terms and Conditions

• Submissions must be made before the deadline as listed on the dates page
• You may submit at most once per day per challenge
• The winners are determined by the HOTA (tiebreaker MOTA) metric
• You are allowed to use any publicly available data for training but you must list them at the time of upload. This also applies to pretraining.
• Note that we (as organizers) may upload models for this challenge BUT we do not compete for a winning position (i.e. our models do not count on the leaderboard and merely serve as references). Thus, if your method is worse (in any metric) than one of the organizer's, you are still encouraged to submit your method as you might win.

In case of any questions regarding the challenge modalities or uploads, please direct them to Benjamin Kiefer.