[00:01:01] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [01:01:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [01:16:03] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [02:11:04] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:55:45] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:11:04] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [09:11:28] 10Release-Engineering-Team-TODO (201908), 10Lexicographical data, 10MediaWiki-Core-Testing, 10Wikidata, 10ci-test-error: WikibaseLexeme test broken by refactor of MediaWiki's Language class - https://phabricator.wikimedia.org/T231103 (10Simetrical) WikibaseLexeme tests were accessing a private property o... [10:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [10:41:04] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [11:21:04] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [13:55:14] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 15616 bytes in 1.309 second response time [13:55:17] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 15616 bytes in 1.447 second response time [13:55:30] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Wikipedia' not found on 'https://en.m.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 16427 bytes in 2.014 second response time [13:56:05] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [13:56:52] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Wikipedia' not found on 'https://en.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 16180 bytes in 0.526 second response time [14:20:49] 10Beta-Cluster-Infrastructure: Beta cluster is down - https://phabricator.wikimedia.org/T231162 (10Urbanecm) p:05Triage→03Unbreak! [14:21:01] 10Beta-Cluster-Infrastructure: Beta cluster is down - https://phabricator.wikimedia.org/T231162 (10Urbanecm) [16:20:05] Project beta-update-databases-eqiad build #36246: 04FAILURE in 4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/36246/ [16:30:01] ^^that is me ^^ [16:30:13] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 48245 bytes in 3.588 second response time [16:30:17] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 48249 bytes in 0.445 second response time [16:30:28] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 36973 bytes in 0.641 second response time [16:31:53] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 48785 bytes in 0.630 second response time [16:34:26] and ^^that's also me ^^, caused by 45831e6, which I reverted temporarily to test a possible cause of beta outage [16:35:32] Project beta-scap-eqiad build #264015: 04FAILURE in 1 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/264015/ [16:38:49] 10Beta-Cluster-Infrastructure: Table 'enwiki.blobs' doesn't exist thrown on beta after 45831e6 - https://phabricator.wikimedia.org/T231162 (10Urbanecm) [16:43:28] !log Revert 'Revert 45831e6 locally on beta cluster to test possible root cause of T231162' [16:43:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:43:31] T231162: DBQueryError from ExternalStoreDB::fetchBlob: Table 'enwiki.blobs' doesn't exist - https://phabricator.wikimedia.org/T231162 [16:46:24] Yippee, build fixed! [16:46:25] Project beta-scap-eqiad build #264016: 09FIXED in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/264016/ [17:21:30] Yippee, build fixed! [17:21:31] Project beta-update-databases-eqiad build #36247: 09FIXED in 1 min 29 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/36247/ [18:11:04] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [18:20:21] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-jhuneidi is CRITICAL: HTTP CRITICAL: HTTP/1.1 404 Not Found - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 319 bytes in 0.010 second response time [19:03:21] (03CR) 10MJL: [C: 03+2] Edit Project Config [extensions/SoftRedirector] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/528884 (owner: 10MJL) [19:06:29] (03CR) 10MJL: [V: 03+2 C: 03+2] Edit Project Config [extensions/SoftRedirector] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/528884 (owner: 10MJL) [20:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [21:27:53] !log centralauth.globalblocks `ALTER TABLE /*_*/globalblocks MODIFY gb_reason varbinary(767) NOT NULL; [21:27:53] ` T231172 [21:27:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:27:55] T231172: Alter gbw_reason/gb_reason/gbw_by_text on WMF production - https://phabricator.wikimedia.org/T231172 [21:39:55] 10Project-Admins, 10Core Platform Team, 10Performance-Team (Radar): Narrow scope of MediaWiki-Database workboard - https://phabricator.wikimedia.org/T228360 (10Krinkle) 05Open→03Resolved a:05Krinkle→03None [22:11:05] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]