Posts about Incidents

May 13 post-incident report

On May 13 we received the cronjob message: cannot delete snapshot and cannot snapshot read only file system. We discussed the issue in our Daily and assumed a problem with the Filesystem Btrfs. After reading the error logs and some initial research...

May 13/14 incident (resolved)

At 07.54am UTC on May 13 during routine maintenance we discovered an issue in our main data storage. Because customer data is of critical importance to us, we paused the automatic GitHub repository daily backup until we could resolve the issue. We...