GitLab experiencing issues with erased production data

GitLab, which started as a start-up in Utrecht, currently has an unreachable site. The company has disclosed that the issues were caused by erased production data and backup mechanisms that were not working or were set up incorrectly.

GitLab, which allows developers to collaborate on projects, detailed the incident in a public text file and provides occasional updates via Twitter. This shows that the problems affected the database, including the issues and merge requests components. However, the repositories and wikis would not have been affected. The company is currently in the process of copying the database, but say that the process is slow.

Around midnight, GitLab announced that part of its production database had been accidentally deleted and that data might be restored from backups. An employee tried to delete an empty directory, but did so with db1.cluster.gitlab.com instead of db2.cluster.gitlab.com. When he found out, only 4.5GB of the total 310GB of data was left. In addition, several backup methods did not work.

GitLab writes that out of a total of five backup methods, none worked properly or were configured correctly. It is not yet clear when the service will be fully available again.