//TODO: professional stuff of software engineer 1001010
Fixing GitLab 502 on my Synology

I run a dockerized GitLab (9.4) on my synology & it started giving me the same 502 error on every page load.

Symptoms

  • 502 Errors on all GitLab pages telling me of to reach out to the system administrator (which is me)
  • Synology > Docker > Container(s) > ‘synology_gitlab’ > Details > Terminal
    Shows “Unicorn” starting and stopping every second or two

Problem

The “unicorn.pid” file was not deleted during a previous GitLab shutdown.  This file is used to block two or more instances of GitLab from running at the same time.  Now when GitLab starts, it erroneously thinks that GitLab is already running and immediately shuts down.  This leads to an endless loop of the scheduler restarting GitLab every few seconds.

Fix

This is a workaround to get back up and running quickly.  In a better world, we would never have to do this dumb maintenance task.

  • ssh to your synology
  • connect as root
  • get a bash shell inside the container
    $ docker exec -it synology_gitlab bash
  • remove the bogus pid file
    $rm /home/git/gitlab/tmp/pids/unicorn.pid
  • Verify the 502 error is gone.  May take a few moments as it starts up for real this time

Solution

File a bug with GitLab so you don’t have to do this ever again.

When the application checks for another running instance at startup, it shouldn’t blindly trust that the existance of a pid file as proof that it’s already running.  The previous instance may have crashed and never had the chance to clean up after itself.  The startup code should check if the pid listed is valid and delete the file if it is not.

It’s sort of like checking for a signed certificate on a file.  It’s not sufficient to check that it is signed, you need to take the extra step and verify the signature is valid for THAT exact file.  I mention this because many corporate A/V products had this bug in the past.

About

Categories: GitLab, Linux, Synology