[00:01:35] 10Scoring-platform-team, 10VPS-project-icinga2, 10User-Zppix: Have ORES-web[01,02] been removed? - https://phabricator.wikimedia.org/T232921 (10Peachey88) [04:02:11] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 0.035 second response time https://wikitech.wikimedia.org/wiki/ORES [04:06:57] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - 512 bytes in 0.021 second response time https://wikitech.wikimedia.org/wiki/ORES [06:26:13] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES [10:01:33] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 979 bytes in 2.814 second response time https://wikitech.wikimedia.org/wiki/ORES [10:06:15] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - 512 bytes in 0.018 second response time https://wikitech.wikimedia.org/wiki/ORES [12:31:39] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 1009 bytes in 0.046 second response time https://wikitech.wikimedia.org/wiki/ORES [12:36:23] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - 512 bytes in 0.022 second response time https://wikitech.wikimedia.org/wiki/ORES [13:31:43] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 977 bytes in 0.242 second response time https://wikitech.wikimedia.org/wiki/ORES [13:36:27] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - 512 bytes in 0.027 second response time https://wikitech.wikimedia.org/wiki/ORES [16:38:54] 10Scoring-platform-team, 10VPS-project-icinga2, 10User-Zppix: Have ORES-web[01,02] been removed? - https://phabricator.wikimedia.org/T232921 (10Halfak) Yes. I killed ores-web-01,2,3 in favor of ores-web-04,5,6. Sorry for the lack of notice! I forgot about icinga checks because we're dealing with an instab... [17:15:34] 10Scoring-platform-team, 10VPS-project-icinga2, 10User-Zppix: Have ORES-web[01,02] been removed? - https://phabricator.wikimedia.org/T232921 (10Zppix) @Halfak do you want to us to add ores[04,05,06] to Icinga2? [18:01:20] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 981 bytes in 4.045 second response time https://wikitech.wikimedia.org/wiki/ORES [18:05:54] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - 512 bytes in 0.023 second response time https://wikitech.wikimedia.org/wiki/ORES [18:31:55] 10Scoring-platform-team, 10Wikilabels: labels.wmflabs.org still shows srwiki in available wikis - https://phabricator.wikimedia.org/T232958 (10Zoranzoki21) [20:15:56] 10Scoring-platform-team, 10editquality-modeling, 10Growth-Team (Current Sprint), 10Patch-For-Review, and 2 others: Update ORES filter thresholds for huwiki - https://phabricator.wikimedia.org/T230031 (10Tgr) Also, goodfaith/likelybad and goodfaith/verylikelybad are barely different for anons ([[https://doc...