Mirroring the Unreal Engine Repository to GitHub

Overview

The procedure I followed when mirroring the Unreal Engine source code to a repository in a different organization or a personal account.
Normally, it is better to fork the original UE repository and submit a Pull Request if you want to make changes, but that approach is not suitable when you want to manage it yourself.

Why it became necessary

As you may know, Unreal Engine has a long history, and that history is recorded on GitHub.
If you try to mirror it along with its entire history, you will run into the restriction where a single push to GitHub must be kept within 2GB, causing the push to fail.
source: Troubleshooting the 2 GB push limit - GitHub Docs

As of now (2024/05/19), the latest version 5.4.1 hit this limit, so it will likely continue to be an issue in the future.
I struggled quite a bit to mirror it without hitting that 2GB limit, so I am sharing the method I used.

⚠️ This is not necessarily the "correct" procedure, but rather a sharing of what worked for me! Please keep that in mind. ⚠️

Steps

Clone

First, clone it normally.
You might want to add --mirror.

git clone {Unreal Engine source URL} UnrealEngine

It should be cloned into the UnrealEngine folder.

Changing the Remote

A repository that has just been cloned naturally has its fetch and push targets pointing to Epic's UE repository, so you need to change the remote.
(Of course, please prepare the target repository in advance.)

git remote set-url origin {new-url}

Check if the remote has been changed correctly with the following command.

git remote -v

Pushing Tags

Unreal Engine has tags for each Preview and Release, so we will push these first.
Pushing all tags at once will hit the 2GB limit, so we will push each tag one by one.

Prepare a Shell script like the following and place it in the cloned UnrealEngine folder. I named it tag-push.sh.

#!/bin/bash

tags=$(git tag)

for tag in $tags
do
    echo "Pushing tag $tag"
    git push origin $tag
    echo "Tag $tag pushed successfully"
done

Run the prepared script.

sh tag-push.sh

Once the tags have been successfully pushed, create a main branch from 5.2.1-release and push it.
Anything after that point will likely fail.

Pushing Tags after 5.2.1-release

After 5.2.1-release, the next version as of now (2024/05/19) is 5.3.0-release. If you try to push the 5.3.0-release tag, you will hit the aforementioned 2GB limit.

Therefore, from here on, we will push in small commit units. Just like before, place a script like the following directly under the cloned repository. I named it small-commit-push.sh.

start_tag=$1
end_tag=$2

# Get commits between the start tag and end tag
commits=$(git log --reverse --pretty=format:'%H' $start_tag..$end_tag)

# Get commits between the start tag and end tag
readarray -t commits < <(git log --reverse --pretty=format:'%H' $start_tag..$end_tag)

chunk=()
count=0

for commit in "${commits[@]}"; do
    chunk+=("$commit")
    ((count++))

    if (( count % 2000 == 0 )); then
        echo "Creating branch with 2000 commits"
        git checkout -b temp_branch "${chunk[0]}"
        git push -f origin temp_branch
        git checkout main
        git branch -D temp_branch
        echo "Branch with 2000 commits created successfully"
        chunk=()
    fi
done

# Process the last chunk
if (( ${#chunk[@]} > 0 )); then
    echo "Creating branch with remaining commits"
    git checkout -b temp_branch "${chunk[0]}"
    git push -f origin temp_branch
    git checkout main
    git branch -D temp_branch
    echo "Branch with remaining commits created successfully"
fi

Use this script in the following way:

sh small-commit-push.sh 5.2.1-release 5.3.0-release

This should have pushed the commits up to the 5.3.0-release tag. Once finished, push the 5.3.0-release tag.

Repeat this for each necessary tag. If you need up to 5.4.1-release, do this for all tags that have been created.

Actually, it would probably be better to create a script to automate this as well, but since there weren't that many, I did it manually. If the frequency or number of tags increases, I recommend automating it.

Pushing Branches

Finally, check out the location of the necessary tag, create a branch from there, and push it to finish. If you need 5.4.1-release, create it from there; if you need 5.3.2-release, create it from there.

You should now have a repository with one branch and the tags pushed. A temp_branch might still exist, but as the name suggests, it is temporary and can be deleted.

That's it.