【Error】 Was asked to gather along dimension 0, but all

1. Error

This is an error handling to this:

Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.

I think you're not wrong, try to run code with a single GPU(set CUDA_VISIBLE_DEVICES="0").

This error occurs when using multiple GPU in PyTorch. I also read that this error could be resolved by using DistributedDataParallel (DDP), but in my environment another error occurred and it did not work properly.

The easy solution is to use a single GPU, but if you know of a way to use multiple GPUs, I would appreciate it if you could let me know.

Reference

[1] https://github.com/huggingface/transformers/issues/14128