Repository storage paths

Introduced in GitLab 8.10.

GitLab allows you to define multiple repository storage paths to distribute the storage load between several mount points.

Notes:

You must have at least one storage path called default.

The paths are defined in key-value pairs. The key is an arbitrary name you can pick to name the file path.

The target directories and any of its subpaths must not be a symlink.

Configure GitLab

Warning: In order for backups to work correctly, the storage path must not be a mount point and the GitLab user should have correct permissions for the parent directory of the path. In Omnibus GitLab this is taken care of automatically, but for source installations you should be extra careful.

The thing is that for compatibility reasons gitlab.yml has a different structure than Omnibus. In gitlab.yml you indicate the path for the repositories, for example /home/git/repositories, while in Omnibus you indicate git_data_dirs, which for the example above would be /home/git. Then, Omnibus will create a repositories directory under that path to use with gitlab.yml.

This little detail matters because while restoring a backup, the current contents of /home/git/repositories are moved to /home/git/repositories.old, so if /home/git/repositories is the mount point, then mv would be moving things between mount points, and bad things could happen. Ideally, /home/git would be the mount point, so then things would be moving within the same mount point. This is guaranteed with Omnibus installations (because they don't specify the full repository path but the parent path), but not for source installations.

Now that you've read that big fat warning above, let's edit the configuration files and add the full paths of the alternative repository storage paths. In the example below, we add two more mountpoints that are named nfs and cephfs respectively.

For installations from source

Edit gitlab.yml and add the storage paths:

repositories:
  # Paths where repositories can be stored. Give the canonicalized absolute pathname.
  # NOTE: REPOS PATHS MUST NOT CONTAIN ANY SYMLINK!!!
  storages: # You must have at least a 'default' storage path.
    default:
      path: /home/git/repositories
    nfs:
      path: /mnt/nfs/repositories
    cephfs:
      path: /mnt/cephfs/repositories

Restart GitLab for the changes to take effect.

Note: The gitlab_shell: repos_path entry in gitlab.yml will be deprecated and replaced by repositories: storages in the future, so if you are upgrading from a version prior to 8.10, make sure to add the configuration as described in the step above. After you make the changes and confirm they are working, you can remove the repos_path line.

For Omnibus installations

Edit /etc/gitlab/gitlab.rb by appending the rest of the paths to the default one:

git_data_dirs({
  "default" => { "path" => "/var/opt/gitlab/git-data" },
  "nfs" => { "path" => "/mnt/nfs/git-data" },
  "cephfs" => { "path" => "/mnt/cephfs/git-data" }
})

Note that Omnibus stores the repositories in a repositories subdirectory of the git-data directory.

Choose where new project repositories will be stored

Once you set the multiple storage paths, you can choose where new projects will be stored via the Application Settings in the Admin area.

Beginning with GitLab 8.13.4, multiple paths can be chosen. New projects will be randomly placed on one of the selected paths.

Handling failing repository storage

Introduced in GitLab 9.5.

When GitLab detects access to the repositories storage fails repeatedly, it can gracefully prevent attempts to access the storage. This might be useful when the repositories are stored somewhere on the network.

The configuration could look as follows:

For Omnibus installations

Edit /etc/gitlab/gitlab.rb:

git_data_dirs({
  "default" => {
    "path" => "/mnt/nfs-01/git-data",
    "failure_count_threshold" => 10,
    "failure_wait_time" => 30,
    "failure_reset_time" => 1800,
    "storage_timeout" => 5
   }
})

Save the file and reconfigure GitLab for the changes to take effect.

For installations from source

Edit config/gitlab.yml:

repositories:
  storages: # You must have at least a `default` storage path.
    default:
      path: /home/git/repositories/
      failure_count_threshold: 10 # number of failures before stopping attempts
      failure_wait_time: 30 # Seconds after last access failure before trying again
      failure_reset_time: 1800 # Time in seconds to expire failures
      storage_timeout: 5 # Time in seconds to wait before aborting a storage access attempt

Save the file and restart GitLab for the changes to take effect.

failure_count_threshold: The number of failures of after which GitLab will completely prevent access to the storage. The number of failures can be reset in the admin interface: https://gitlab.example.com/admin/health_check or using the api to allow access to the storage again.

failure_wait_time: When access to a storage fails. GitLab will prevent access to the storage for the time specified here. This allows the filesystem to recover without.

failure_reset_time: The time in seconds GitLab will keep failure information. When no failures occur during this time, information about the mount is reset.

storage_timeout: The time in seconds GitLab will try to access storage. After this time a timeout error will be raised.

When storage failures occur, this will be visible in the admin interface like this:

To allow access to all storages, click the Reset git storage health information button.