【Kaggle Method】ISIC2024 3rd Solution explained
This article is an explanation of 3rd solution for Kaggle's ISIC2024 competition.
Original Solution Link:
0. Highlights
・Chapter1: Creating a reliable CV and maximizing the CV score.
・Chapter2: Due to the diversity of inputs to the GBDT, we created a model that captures intermediate values between positive and negative.
1. Overview
He focused on 2 things.
- Developing various image models to boost the GBDT score(used the public notebook as a reference for the table model).
- Creating a reliable CV and maximizing the CV score. (His final CV is 0.183099)
・Model overview
2. Image Model
2x2 models with different positivity criteria.
Positive Condition: "target" = 1
・convnextv2_nano.fcmae_ft_in22k_in1k (CV 0.1599389)
・vit_tiny_patch16_224.augreg_in21k_ft_in1k (CV 0.1612504)
Positive: "target"=1 or "iddx_1"=Indeterminate or "iddx_2"!=nan
vit_tiny_patch16_224.augreg_in21k_ft_in1k (CV 0.1460650)
vit_small_patch16_224.augreg_in21k_ft_in1k (CV 0.1486832)
The second models are utilized to catch data that may or may not be positive. If iddx_1 is indeterminate or iddx_2 is nan, it will be considered suspicious even if target=0.
The last 2 models above are aimed to give conservative predictions so that GBDT models can have more variety of features.
Overall, as many have mentioned, smaller models performed better in my case as well.
3. Model Configurations
・scheduler: CosineAnnealing with warmup
・batch_size: 32
・learning rate: 1e-4
・optimizer: AdamW
・weight decay: 0.001 for weight param
・positive: negative = 1:1
・Classification head: fc(64 or 32) + relu + dropout
・Change negative samples per epoch
・Augumentation based on 1st solution in prev comp 1st place solution for SIIM-ISIC Melanoma Classification
・Ensure the same pos/neg ratio in every batch
4. CV Strategy
・GroupStratified 5Fold by patient_id
・Same split is used for both image and tabular models.
5. Tabler Model
His tabular model is almost identical to the following great notebooks. And my final solution uses LGBM(w/o image) + LGBM(w image) + Catboost(w image) + XGB(w image).
6. Didn't work
・Layerwise learning rate decay
・Models like Eva02, Swin, and EfficientNet
・Stacking of GBDT models
・More image models(>4)
It's over. Good solution and congrats to him!
Discussion