3^rd Workshop on Maritime Computer Vision (MaCVi)

Challenges / USV-based Embedded Obstacle Segmentation

USV-based Embedded Obstacle Segmentation

Quick links: Dataset download Validate ONNX Submit Leaderboards Ask for help

Quick Start

1. Download the LaRS dataset on the dataset page.
2. Train your model on the LaRS training set.
3. Export the model as an ONNX file.
4. Submit the ONNX model through the upload page.

🏅 Prizes

The top team of this challenge stands to win $500 worth of Luxonis devices.

Overview

This challenge is an extension of the USV-based Obstacle Segmentation challenge. For an overview of the task and datasets refer to the challenge page.

The aim of this challenge is to develop obstacle segmentation methods suitable for deployment on embedded devices. For this reason your methods will be run, benchmarked and evaluated on a real-world device. The device is an upcoming next-gen device from Luxonis based on Robotic Vision Core 4 (RVC4).

Task

Create a semantic segmentation method that classifies the pixels in a given image into one of three classes: sky, water or obstacle. An obstacle is everything that the USV can crash into or that it should avoid (e.g. boats, swimmers, land, buoys). Refer to the main challenge page for more details.

Evaluation metrics

In the evaluation we will consider the same obstacle detection quality metrics as in the main challenge. In addition, we will also evaluate the throughput of the methods in terms of frames-per-second (FPS) processed on the evaluation device.

To be considered for the challenge, the method must run faster than a set threshold of 30 FPS with 384x768 input shape. The throughput will be evaluated in regular (balanced) mode.

To determine the winner of the challenge the aggregate Q (Quality) metric will be used. In case of a tie, FPS of the methods will be used to determine a winner (faster is better).

Participate

To participate in the challenge follow these steps:

1. Download the LaRS dataset ( LaRS webpage).
2. Train a semantic segmentation model on the LaRS training set. You can also use additional publicly available training data, but must disclose it during submission.
   • Note: the mmsegmentation-macvi may be a good starting point for the development of you model. The repository contains the scripts for training and inference on LaRS.
3. Export your model as ONNX. See below for more details.
   • Note: You may use the tools/pytorch2onnx.py script in the mmsegmentation-macvi repository as a starting point.
4. Validate your .onnx file to check for potential errors here.
   • We will run a shorter version of model conversion to make sure everything is OK. You will receive the results in the dashboard, along with potential errors.
   • This step is recommended to quickly identify errors, without having to wait for full evaluation.
   • The model validation process has no daily upload limit.
5. Upload your .onnx file along with all the required information here.
   • After submission, your method will be run and evaluated on device. This may take some time (up to an hour) so be patient. Results of the evaluation will appear on your dashboard. The dashboard will also display an message in case of failed evaluation (hover over the error icon). You may evaluate at most one submission per day (per challenge track). Failed attempts do not count towards this limit.

Model Guidelines

Since hardware might present certain limitations, we present guidelines for model development. This section describes the allowed operations, model requirements, input definitions, good practices, and expected throughput and accuracy drop.

Allowed Operations

Before running the model on the device, it gets compiled and quantized to appropriate format. While the quantization to INT8 and part of compilation will be executed automatically, users are still required to perform the first step - conversion of the trained Pytorch model to ONNX.

Below is a short example of Pytorch to ONNX conversion, that would create a model.onnx with fixed input shape, which could be submitted to MaCVi:

Some general guidelines to ensure that your model will be exportable:

• Refer to official documentation to see which operators are supported (PyTorch, ONNX).
• Avoid custom operations and rely on torch operators.
• Make sure your model graph is static (inference process should be the same for each image, input and intermediate tensor shapes are fixed).
• Use opset version up to 13 during the export process.

Feel free to refer to the official Pytorch documentation for more instructions on export, limitations, and good practices. Please note that custom operations are not supported.

Furthermore, there might be some additional limitations on the device itself, where certain ONNX operations might not be supported. Please follow this spreadsheet for un/supported ONNX operations. If a certain operation is missing, it is suggested to check whether the model compiles and executes correctly through a submission of an un-trained model.

Inputs and outputs of the model

Inputs

Because models can expect differently processed inputs and evaluation on the device does not expose the pre-processing options to the participants, it is important that model expects the following input:

• A single RGB image (scaled to [0-1] range before normalization) normalized by ImageNet mean and standard deviation:
  • mean = [0.485, 0.456, 0.406]
  • std = [0.229, 0.224, 0.225]
• Input shape of the exported model must be 1x3x384x768 (N, C, H, W), while it can be arbitrary during the training.

Since images in the training set can have different sizes, they will be reshaped (using bilinear interpolation) to fit the above shape (with preserved aspect ratio), centered into the shape, and padded using mirror padding.

Example of Python code using PIL and Numpy to read and normalize the image:

Such images will be fed to the model on device, the outputs will be automatically evaluated in the same manner as the classic segmentation track. Note that example does not include the padding and resizing.

Outputs

The output should be:

• Single output node
• 1x1x384x768 tensor of predictions (argmaxed logits).

Please note that as part of the postprocessing step, the predicted logits will be argmaxed to obtain segmentation mask, then cropped and upscaled (using nearest interpolation) to the original input shape.

Good Practices

While the model will be quantized to INT8, it is recommended to use FP32 during the training, as it is more stable and allows for better convergence. It is also recommended to use the same input shape during the training as it will be during inference.

Expected Throughput and Accuracy Drop

In the table bellow, we report the F1 detection score performance drop and FPS for several common segmentation methods. These benchmarks show what kind of performance drop and throughput to expect from your methods.

The models have been exported using the MMSegmentation toolbox. Input shape of 768x384 was used for all deployed models.

Variants:

• orig: the original model using original input resolution (2048x1024)
• orig (768x384): the original model running inference with target input resolution (768x384)
• quantized: the final quantized model running on device, uses mirror padding

Note: Most of the performance drop is not due to quantization, but due to using lower input resolution during inference. To improve the performance we suggest training specifically for the target resolution.

Terms and Conditions

• Submissions must be made before the deadline as listed on the dates page. Submissions made after the deadline will not count towards the final results of the challenge.
• Submissions are limited to one per day per challenge. Failed submissions do not count towards this limit.
• The winner is determined by the F1 metric (FPS in case of tie).
• You are allowed to use additional publicly available data for training but you must disclose them at the time of upload. This also applies to pre-training.
• In order for your method to be considered for the winning positions and included in the results paper, you will be required to submit a short report describing your method. More information in regards to this will be released towards the end of the challenge.
• Note that we (as organizers) may upload models for this challenge, BUT we do not compete for a winning position (i.e. our models do not count on the leaderboard and merely serve as references). Thus, if your method is worse (in any metric) than one of the organizer's, you are still encouraged to submit your method. Methods that were submitted as part of the MaCVi 2024 challenge will be marked on the leaderboards.

In case of any questions regarding the challenge datasets or submission, please join the MaCVi Support forum.