Challenges / Generalist Meta Challenge
Quick links: Dataset download Submit Leaderboard Ask for help
The Generalist Meta Challenge evaluates cross-task generalization across various maritime computer vision tasks. Participants are encouraged to develop models that can perform well on multiple tasks simultaneously or adapt quickly to new tasks within the maritime domain. Specifically, the current Generalist Meta challenge focuses on the active challenges (e.g., Vision-to-Chart Data Association, Thermal Object Detection Challenge, LaRS Panoptic Segmentation Challenge, Embedded Semantic Segmentation Challenge and Multimodal Semantic Segmentation Challenge).
A Maritime Generalist model is defined as a model capable of performing effectively across all sub-problems within the maritime domain. There are multiple approaches to developing such a generalist model, and the challenge allows flexibility in the chosen methodology. However, participants must clearly justify why their proposed approach qualifies as a generalist model. This requirement encourages participants to identify and articulate the most effective strategy for building a truly generalist solution for the maritime domain. The challenge outlines several acceptable strategies, but participants are not restricted to the approaches explicitly listed.
Participants must develop and submit a method that competes in at least two currently active challenges. Each submission must clearly justify why the proposed approach qualifies as a generalist method. The report must also include a list of the challenges in which the method was evaluated. Submissions are limited to four double-column pages (excluding references). The top three ranked teams will be required to provide code for verification via GitHub. Private repositories are permitted; however, the code must be fully accessible to the organizers for validation.
Leaderboards differ in scale and not every model appears on every benchmark, making raw scores incomparable across them. This metric normalizes ranks to $[0, 1]$ and aggregates via the arithmetic mean. Models absent from a leaderboard are assigned a rank one below last place (contributing a score of 0), which discourages selective participation and pulls the average down for models with incomplete coverage.
Let there be $M$ leaderboards indexed by $i \in \{1, \ldots, M\}$. Leaderboard $i$ contains $N_i \geq 2$ ranked entries (rank 1 is best). Let models be indexed by $j$.
First, we define the per-leaderboard rank $r_{ij}$ for model $j$ on leaderboard $i$ as: $$r_{ij} = \begin{cases} \text{rank of model } j \text{ on leaderboard } i & \text{if present} \\ N_i + 1 & \text{if missing} \end{cases}$$
We convert each rank to an inverted normalized score $s_{ij} \in [0, 1]$ such that: $$r_{ij} = 1 \;\mapsto\; s_{ij} = 1; \qquad r_{ij} = N_i \;\mapsto\; s_{ij} = 0; \qquad r_{ij} = N_i + 1 \;\mapsto\; s_{ij} = 0$$ The normalized score for each model $j$ in each leaderboard $i$ is calculated as: $$s_{ij} = \max\!\left(0,\ 1 - \frac{r_{ij} - 1}{N_i - 1}\right)$$
The Consistency score $C_j$ is the arithmetic mean of per-leaderboard scores. $$C_j = \frac{1}{M} \sum_{i=1}^{M} s_{ij}$$
Missing leaderboards still contribute a score of 0 (via the rank $N_i + 1$ penalty), which pulls the average down and discourages selective participation.
If you have any questions, please join the MaCVi Support forum.
