spamemailscreenstock100664935orig

GitLab database goes out after spam attack

Code-hosting site GitLab has suffered an outage after sustaining a "serious" incident on Tuesday with one of its databases that has required emergency maintenance.

The company today said it lost six hours of database data, including issues, merge requests, users, comments, and snippets, for GitLab.com and was in the process restoring data from a backup. Data was accidentally deleted, according to a Twitter message.

"Losing production data is unacceptable, and in a few days we'll post the five whys of why this happened and a list of measures we will implement," GitLab said in a bulletin this morning. Git.wiki repositories and self-hosted installations were unaffected.

The restoration means any data between 17:20 UTC and 23:25 UTC from the database is lost by the time GitLab.com goes live again. Providing a chronology of events, GitLab said it detected Monday that spammers were hammering its database by creating snippets and rendering it unstable. GitLab blocked the spammers based on an IP address and removed a user from using a repository as a form of CDN. This resulted in 47,000 IPs signing in using the same account and causing a high database load, and GitLab removed the users for spamming.

The company provided a statement this morning: "This outage did not affect our Enterprise customers or the wide majority of our users. As part of our ongoing recovery efforts, we are actively investigating a potential data loss. If confirmed, this data loss would affect less than one percent of our user base, and specifically peripheral metadata that was written during a six-hour window," the company said. "We have been working around the clock to resume service on the affected product, and set up long-term measures to prevent this from happening again. We will continue to keep our community updated through Twitter, our blog and other channels."

While dealing with the problem, GitLab found database replication lagged far behind, effectively stopping. "This happened because there was a spike in writes that were not processed on time by the secondary database." GitLab has been dealing with a series of database issues, including a refusal to replicate.

GitLab.com went down at 6:28 pm PST on Tuesday and was back up at 9:57 am PST today, said Tim Anglade, interim vice president of marketing at the company.

IDG Insider

PREVIOUS ARTICLE

« Everything we think we know about the LG G6

NEXT ARTICLE

Cisco amps-up Tetration platform with better security, reduced footprint, AWS cloud option »
author_image
IDG News Service

The IDG News Service is the world's leading daily source of global IT news, commentary and editorial resources. The News Service distributes content to IDG's more than 300 IT publications in more than 60 countries.

  • Mail

Recommended for You

International Women's Day: We've come a long way, but there's still an awfully long way to go

Charlotte Trueman takes a diverse look at today’s tech landscape.

Trump's trade war and the FANG bubble: Good news for Latin America?

Lewis Page gets down to business across global tech

20 Red-Hot, Pre-IPO companies to watch in 2019 B2B tech - Part 1

Martin Veitch's inside track on today’s tech trends

Poll

Do you think your smartphone is making you a workaholic?