[00:22:41] <wikibugs>	 10Tool-paulina, 13Patch-For-Review: search authors for a specific death date year - https://phabricator.wikimedia.org/T409205#11342907 (10Oluwatumininu.m) Hi @Mind_Booster_Noori @Pepe_piton @Nat_WDU  I fixed this issue. Kindly have a look to check if I'm on the right track.  Here is the link to merge request:...
[00:59:04] <wikibugs>	 06cloud-services-team, 10Toolforge: Could not connect to database: User s51080 already has more than 'max_user_connections' active connections - https://phabricator.wikimedia.org/T409244 (10Bamyers99) 03NEW
[01:19:29] <wikibugs>	 06cloud-services-team, 10Toolforge: Could not connect to database: User s51080 already has more than 'max_user_connections' active connections - https://phabricator.wikimedia.org/T409244#11343053 (10Zache) It seems that toolsdb is generally down. Here is ticket for montage db connection [https://github.com/hat...
[05:05:56] <jinxer-wm>	 FIRING: SystemdUnitDown: The service unit hdfs_rsync_cirrus_index_dumps.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[05:10:56] <jinxer-wm>	 FIRING: [2x] SystemdUnitDown: The service unit hdfs_rsync_cirrus_index_dumps.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown  - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[05:33:09] <wikibugs>	 06cloud-services-team, 10Toolforge: Could not connect to database: User s51080 already has more than 'max_user_connections' active connections - https://phabricator.wikimedia.org/T409244#11343298 (10Chlod) Also affects: [fountain](https://fountain.toolforge.org/), [PetScan](https://petscan.wmcloud.org/)
[06:56:36] <wikibugs>	 06cloud-services-team, 10Toolforge: Could not connect to database: User s51080 already has more than 'max_user_connections' active connections - https://phabricator.wikimedia.org/T409244#11343376 (10Zache) p:05Triage→03High
[06:57:15] <wikibugs>	 06cloud-services-team, 10Toolforge: Could not connect to database: User s51080 already has more than 'max_user_connections' active connections - https://phabricator.wikimedia.org/T409244#11343377 (10Zache) I triaged this high as this is still going on and impacts to multiple tools.
[07:00:56] <jinxer-wm>	 FIRING: SystemdUnitDown: The systemd unit hdfs_rsync_cirrus_index_dumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[07:05:56] <jinxer-wm>	 FIRING: [2x] SystemdUnitDown: The systemd unit hdfs_rsync_cirrus_index_dumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown  - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[07:19:23] <wikibugs>	 06cloud-services-team, 10Toolforge: Could not connect to database: User s51080 already has more than 'max_user_connections' active connections - https://phabricator.wikimedia.org/T409244#11343456 (10fgiunchedi) tools-db-4 storage volume is out of space, I'll use this task for tracking
[07:26:45] <wikibugs>	 06cloud-services-team, 10Toolforge: Could not connect to database: User s51080 already has more than 'max_user_connections' active connections - https://phabricator.wikimedia.org/T409244#11343478 (10fgiunchedi) disk space free trend for tools-db-4 over the last 30d  {F69903693}  and over the last couple of day...
[07:41:37] <wikibugs>	 06cloud-services-team, 10Toolforge: Could not connect to database: User s51080 already has more than 'max_user_connections' active connections - https://phabricator.wikimedia.org/T409244#11343501 (10fgiunchedi) Doing a comparison with the replica on tools-db-6, there's ~800G free there:  ` root@tools-db-6:/srv...