Some systems are experiencing issues

Stickied Incidents

27th November 2019

Dwellers Dwellers need to be reprovisioned. Affects Mastodon.

Still down: Mastodon, RabbitMQ, Notifications center, tools.nasqueron.org

2020-02-15 Development facilities like Docker build are ready to use

2020-02-14 A fresh CentOS 8.1 has been reinstalled through Salt.

2020-02-13 Upgrade RAM, new 1 To disk attached to avoid previous Mastodon issues.

2019-11-27 According to the servers log: 07:13 dereckson: Disk drives look corrupted. Attach a new 1 To drive to the machine. Install a clean CentOS 7 on this drive to recover data.

Past Incidents

9th June 2016

No incidents reported

8th June 2016

Dwellers Dwellers partial outage following a fork bomb

We're currently improving our SSL infrastructure, automating Let's encrypt renewals on Dwellers.

This automatic certificate renewal means we need to restart web server. The script to restart the webserver contained a logic error, leading to spawn a lot of processes (a fork bomb) during a test. As such, Dwellers experienced some issues.

19:36 UTC — Perturbations on Dwellers, some services are lagging or not reachable 19:44 UTC — Fork bomb is defused. CPU and schudeling back to normal. 19:44 Retour à la normale 19:46 UTC — MySQL container has been restarted. All services excepted Etherpad are back to normal

21:54 UTC — Production environment tests are launched. We detect Etherpad container was down too, restarted. Production environment tests pass again.

7th June 2016

No incidents reported

6th June 2016

No incidents reported

5th June 2016

No incidents reported

4th June 2016

No incidents reported

3rd June 2016

No incidents reported