Some systems are experiencing issues

Stickied Incidents

27th November 2019

Dwellers Dwellers need to be reprovisioned. Affects Mastodon.

Still down: Mastodon, RabbitMQ, Notifications center, tools.nasqueron.org

2020-02-15 Development facilities like Docker build are ready to use

2020-02-14 A fresh CentOS 8.1 has been reinstalled through Salt.

2020-02-13 Upgrade RAM, new 1 To disk attached to avoid previous Mastodon issues.

2019-11-27 According to the servers log: 07:13 dereckson: Disk drives look corrupted. Attach a new 1 To drive to the machine. Install a clean CentOS 7 on this drive to recover data.

Past Incidents

17th June 2017

No incidents reported

16th June 2017

No incidents reported

15th June 2017

No incidents reported

14th June 2017

Mastodon Mastodon upgrade issue

After the Docker engine issue, we switched Mastodon to use persistent storages on disk for db volume.

In parallel, we upgraded to current master. There are some unidentified JS issues on Google Chrome, so we're rolling back to 1.4 (commit 4289ed1d13b9c3b91663581c44635105c4bc0412).

21:31: Mastodon images rebuild process launch

21:39: Precompiling JS / CSS / etc. assets (this operation generally takes a long time)

21:50: Assets compiled, trying to restart the server

21:55: All clear. Home feeds are lost, but will be regenerated on the fly (db has been restored, cache not). We switched to a permanent storage mode, so cache will be saved forever from now.

Dwellers Docker engine issue on Dwellers

Our Docker engine on Dwellers weren't responsive and spawned zombie processes.

Server has been restarted to solve the zombie process issue.

13th June 2017

No incidents reported

12th June 2017

No incidents reported

11th June 2017

No incidents reported