[00:05:27] 10Release-Engineering-Team, 10Code-Stewardship-Reviews, 10MediaWiki-extensions-UserMerge, 10Stewards-and-global-tools: Undeploy UserMerge Extension from WMF production - https://phabricator.wikimedia.org/T216089 (10Jrbranaa) p:05Triage→03Normal [00:07:50] 10Release-Engineering-Team, 10Code-Stewardship-Reviews, 10MediaWiki-extensions-UserMerge, 10Stewards-and-global-tools: Remove UserMerge extension entry from developers/maintainers page - https://phabricator.wikimedia.org/T216091 (10Jrbranaa) p:05Triage→03Normal [00:08:34] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews, 10MediaWiki-extensions-UserMerge, 10Stewards-and-global-tools: UserMerge: Code Stewardship Review - https://phabricator.wikimedia.org/T204747 (10Jrbranaa) a:03Jrbranaa [00:09:23] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews, 10MediaWiki-extensions-CodeReview: CodeReview extension: Code stewardship review - https://phabricator.wikimedia.org/T205482 (10Jrbranaa) a:03Jrbranaa [00:10:12] 10Release-Engineering-Team, 10MediaWiki-extensions-UserMerge, 10Stewards-and-global-tools: Remove UserMerge extension entry from developers/maintainers page - https://phabricator.wikimedia.org/T216091 (10Jrbranaa) [00:10:28] 10Release-Engineering-Team, 10MediaWiki-extensions-UserMerge, 10Stewards-and-global-tools: Undeploy UserMerge Extension from WMF production - https://phabricator.wikimedia.org/T216089 (10Jrbranaa) [00:15:48] (03PS1) 10Dduvall: maintenance: Cleanup old Docker images at a lower threshold [integration/config] - 10https://gerrit.wikimedia.org/r/490505 [00:16:56] (03PS2) 10Dduvall: maintenance: Cleanup old Docker images at a lower threshold [integration/config] - 10https://gerrit.wikimedia.org/r/490505 (https://phabricator.wikimedia.org/T177867) [00:22:50] (03PS3) 10Dduvall: maintenance: Cleanup old Docker images at a lower threshold [integration/config] - 10https://gerrit.wikimedia.org/r/490505 (https://phabricator.wikimedia.org/T177867) [00:37:13] (03PS2) 10Krinkle: Update Quibble containers to Node 10 with npm 6 [integration/config] - 10https://gerrit.wikimedia.org/r/487896 (https://phabricator.wikimedia.org/T211784) (owner: 10Hashar) [00:37:28] (03CR) 10Krinkle: "(updated subject for my dash-boarding convenience)" [integration/config] - 10https://gerrit.wikimedia.org/r/487896 (https://phabricator.wikimedia.org/T211784) (owner: 10Hashar) [00:37:45] (03CR) 10jerkins-bot: [V: 04-1] Update Quibble containers to Node 10 with npm 6 [integration/config] - 10https://gerrit.wikimedia.org/r/487896 (https://phabricator.wikimedia.org/T211784) (owner: 10Hashar) [01:00:37] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 48898 bytes in 6.552 second response time [01:03:14] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 40571 bytes in 3.484 second response time [01:13:15] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Mail, 10Operations, and 2 others: Convert Phabricator mail config to use cluster.mailers - https://phabricator.wikimedia.org/T212989 (10Dzahn) deployed in production (phab1001) , please test that mail still works [01:23:27] 10Project-Admins: Requesting Personal project tag User-crusnov - https://phabricator.wikimedia.org/T215821 (10Dzahn) please excuse me, this is a test [01:33:30] 10MediaWiki-Codesniffer, 10MediaWiki-General-or-Unknown, 10Technical-Debt: Encourage type hints for function parameters and return after moving MediaWiki to PHP 7 - https://phabricator.wikimedia.org/T178136 (10Tgr) I would support making it compulsory for new code. Old code has all kinds of compatibility / d... [01:42:32] 10MediaWiki-Codesniffer: Split ScalarTypeHintUsageSniff per PHP version - https://phabricator.wikimedia.org/T216099 (10Tgr) [01:46:53] Project beta-update-databases-eqiad build #31939: 04STILL FAILING in 5.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31939/ [01:59:07] Project beta-scap-eqiad build #237779: 04STILL FAILING in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237779/ [02:00:30] Project beta-scap-eqiad build #237780: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237780/ [02:13:52] Project beta-scap-eqiad build #237781: 04STILL FAILING in 9 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237781/ [02:20:04] Project beta-update-databases-eqiad build #31940: 04STILL FAILING in 3.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31940/ [02:24:24] Project beta-scap-eqiad build #237782: 04STILL FAILING in 9 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237782/ [02:34:43] Project beta-scap-eqiad build #237783: 04STILL FAILING in 9 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237783/ [02:42:54] woo finally got https://gerrit-review.googlesource.com/c/gerrit/+/214432/ done! Per request from a gerrit maintainer. [02:45:04] Project beta-scap-eqiad build #237784: 04STILL FAILING in 9 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237784/ [02:55:15] Project beta-scap-eqiad build #237785: 04STILL FAILING in 8 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237785/ [03:05:27] Project beta-scap-eqiad build #237786: 04STILL FAILING in 8 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237786/ [03:15:34] Project beta-scap-eqiad build #237787: 04STILL FAILING in 8 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237787/ [03:20:04] Project beta-update-databases-eqiad build #31941: 04STILL FAILING in 3.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31941/ [03:25:48] Project beta-scap-eqiad build #237788: 04STILL FAILING in 9 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237788/ [03:36:02] Project beta-scap-eqiad build #237789: 04STILL FAILING in 9 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237789/ [03:46:20] Project beta-scap-eqiad build #237790: 04STILL FAILING in 9 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237790/ [03:56:36] Project beta-scap-eqiad build #237791: 04STILL FAILING in 9 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237791/ [04:06:05] (03PS1) 10Gergő Tisza: Drop PHP 7.1 compatibility for Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/490528 [04:06:43] Project beta-scap-eqiad build #237792: 04STILL FAILING in 8 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237792/ [04:07:39] (03PS2) 10Gergő Tisza: Drop PHP 7.0 compatibility for Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/490528 [04:08:50] (03CR) 10Gergő Tisza: [C: 04-1] "Not sure if this is good to go or needs more discussion, -1-ing for now." [integration/config] - 10https://gerrit.wikimedia.org/r/490528 (owner: 10Gergő Tisza) [04:17:00] Project beta-scap-eqiad build #237793: 04STILL FAILING in 9 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237793/ [04:20:04] Project beta-update-databases-eqiad build #31942: 04STILL FAILING in 3.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31942/ [04:20:22] (03CR) 10Subramanya Sastry: [C: 03+1] "+1 from me ... but, yes, we should figure out what the PHP7 deployment plan is .. i.e. what version will actually be in production when it" [integration/config] - 10https://gerrit.wikimedia.org/r/490528 (owner: 10Gergő Tisza) [04:22:52] (03CR) 10Gergő Tisza: [C: 04-1] "Also at some point Parsoid will be merged into MediaWiki core (for some value of "merged"). Will MediaWiki drop 7.0 support by then? (I gu" [integration/config] - 10https://gerrit.wikimedia.org/r/490528 (owner: 10Gergő Tisza) [04:27:12] Project beta-scap-eqiad build #237794: 04STILL FAILING in 8 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237794/ [04:37:24] Project beta-scap-eqiad build #237795: 04STILL FAILING in 9 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237795/ [04:40:14] (03PS3) 10Gergő Tisza: Drop PHP 7.0 compatibility for Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/490528 [04:40:54] (03CR) 10Gergő Tisza: [C: 04-1] "Filed T216102 so there's a nicer place to discuss this." [integration/config] - 10https://gerrit.wikimedia.org/r/490528 (owner: 10Gergő Tisza) [04:47:35] Project beta-scap-eqiad build #237796: 04STILL FAILING in 8 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237796/ [04:57:47] Project beta-scap-eqiad build #237797: 04STILL FAILING in 8 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237797/ [05:08:04] Project beta-scap-eqiad build #237798: 04STILL FAILING in 9 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237798/ [05:18:17] Project beta-scap-eqiad build #237799: 04STILL FAILING in 8 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237799/ [05:20:04] Project beta-update-databases-eqiad build #31943: 04STILL FAILING in 3.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31943/ [05:28:30] Project beta-scap-eqiad build #237800: 04STILL FAILING in 9 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237800/ [05:38:42] Project beta-scap-eqiad build #237801: 04STILL FAILING in 8 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237801/ [05:48:49] Project beta-scap-eqiad build #237802: 04STILL FAILING in 8 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237802/ [05:59:04] Project beta-scap-eqiad build #237803: 04STILL FAILING in 9 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237803/ [06:09:35] Project beta-scap-eqiad build #237804: 04STILL FAILING in 9 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237804/ [06:19:51] Project beta-scap-eqiad build #237805: 04STILL FAILING in 9 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237805/ [06:20:04] Project beta-update-databases-eqiad build #31944: 04STILL FAILING in 3.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31944/ [06:20:16] (03PS3) 10Smalyshev: Add WikibaseCirrusSearch extension [tools/release] - 10https://gerrit.wikimedia.org/r/489362 (https://phabricator.wikimedia.org/T215684) [06:30:05] Project beta-scap-eqiad build #237806: 04STILL FAILING in 8 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237806/ [06:40:13] Project beta-scap-eqiad build #237807: 04STILL FAILING in 8 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237807/ [06:40:44] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Testing, 10Patch-For-Review: ExtensionJsonValidationTest does not support @note - https://phabricator.wikimedia.org/T215650 (10Smalyshev) 05Open→03Resolved a:03Smalyshev [06:43:45] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10Marostegui) As per {P8078} data looks in a pretty bad state :( Probably the best approach would be to re-clone that new instance directly from the master - how big is the data s... [06:50:38] Project beta-scap-eqiad build #237808: 04STILL FAILING in 9 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237808/ [07:00:57] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10jcrespo) I don't have anything else to add to what marostegui said about disk being in a bad state, except > and set mariadb::config::basedir: /opt/wmf-mariadb101 That shouldn... [07:01:17] Project beta-scap-eqiad build #237809: 04STILL FAILING in 9 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237809/ [07:06:39] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10jcrespo) > I intend to attempt https://wikitech.wikimedia.org/wiki/Setting_up_a_MySQL_replica#Transferring_Data If there was a working cumin installation on labs (or any other... [07:11:34] Project beta-scap-eqiad build #237810: 04STILL FAILING in 9 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237810/ [07:20:12] Project beta-update-databases-eqiad build #31945: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31945/ [07:21:44] Project beta-scap-eqiad build #237811: 04STILL FAILING in 8 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237811/ [07:23:46] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10ArielGlenn) @Marostegui /srv/sqldata has 39G on it on db04, presumably that's pretty close to the amount of data on the master. @jcrespo There is deployment-cumin.deployment-pr... [07:31:59] Project beta-scap-eqiad build #237812: 04STILL FAILING in 9 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237812/ [07:42:16] Project beta-scap-eqiad build #237813: 04STILL FAILING in 9 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237813/ [07:52:27] Project beta-scap-eqiad build #237814: 04STILL FAILING in 8 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237814/ [08:02:42] Project beta-scap-eqiad build #237815: 04STILL FAILING in 9 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237815/ [08:12:51] Project beta-scap-eqiad build #237816: 04STILL FAILING in 8 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237816/ [08:20:09] Project beta-update-databases-eqiad build #31946: 04STILL FAILING in 8.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31946/ [08:23:04] Project beta-scap-eqiad build #237817: 04STILL FAILING in 8 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237817/ [08:33:15] Project beta-scap-eqiad build #237818: 04STILL FAILING in 8 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237818/ [08:38:49] (03PS1) 10Alexandros Kosiaris: Move cxserver to deployment pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/490559 (https://phabricator.wikimedia.org/T213195) [08:43:28] Project beta-scap-eqiad build #237819: 04STILL FAILING in 9 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237819/ [08:53:48] Project beta-scap-eqiad build #237820: 04STILL FAILING in 9 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237820/ [09:04:03] Project beta-scap-eqiad build #237821: 04STILL FAILING in 9 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237821/ [09:14:24] Project beta-scap-eqiad build #237822: 04STILL FAILING in 9 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237822/ [09:15:38] Project beta-scap-eqiad build #237823: 04STILL FAILING in 1.1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237823/ [09:20:08] Project beta-update-databases-eqiad build #31947: 04STILL FAILING in 7.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31947/ [09:33:11] Project beta-scap-eqiad build #237824: 04STILL FAILING in 8 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237824/ [09:43:26] Project beta-scap-eqiad build #237825: 04STILL FAILING in 9 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237825/ [09:53:35] Project beta-scap-eqiad build #237826: 04STILL FAILING in 8 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237826/ [10:03:59] Project beta-scap-eqiad build #237827: 04STILL FAILING in 9 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237827/ [10:14:09] Project beta-scap-eqiad build #237828: 04STILL FAILING in 8 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237828/ [10:18:45] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10Krenair) >>! In T216067#4953493, @jcrespo wrote: >> and set mariadb::config::basedir: /opt/wmf-mariadb101 > > That shouldn't be necessary, I believe under stretch it defaults t... [10:20:04] Project beta-update-databases-eqiad build #31948: 04STILL FAILING in 3.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31948/ [10:20:24] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10Marostegui) >>! In T216067#4953498, @ArielGlenn wrote: > @Marostegui /srv/sqldata has 39G on it on db04, presumably that's pretty close to the amount of data on the master. > >... [10:23:35] god morning kren air [10:23:38] *good [10:24:20] Project beta-scap-eqiad build #237829: 04STILL FAILING in 8 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237829/ [10:28:30] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10Legoktm) [10:34:36] Project beta-scap-eqiad build #237830: 04STILL FAILING in 9 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237830/ [10:34:43] thanks for the lang fix, lego [10:37:49] :) [10:44:46] Project beta-scap-eqiad build #237831: 04STILL FAILING in 8 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237831/ [10:54:56] Project beta-scap-eqiad build #237832: 04STILL FAILING in 8 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237832/ [11:05:05] Project beta-scap-eqiad build #237833: 04STILL FAILING in 8 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237833/ [11:15:14] Project beta-scap-eqiad build #237834: 04STILL FAILING in 8 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237834/ [11:16:38] (03CR) 10Giuseppe Lavagetto: [C: 03+2] Fix invalidate_opcache [tools/scap] - 10https://gerrit.wikimedia.org/r/489668 (owner: 10Giuseppe Lavagetto) [11:18:39] (03Merged) 10jenkins-bot: Fix invalidate_opcache [tools/scap] - 10https://gerrit.wikimedia.org/r/489668 (owner: 10Giuseppe Lavagetto) [11:19:27] (03CR) 10jenkins-bot: Fix invalidate_opcache [tools/scap] - 10https://gerrit.wikimedia.org/r/489668 (owner: 10Giuseppe Lavagetto) [11:20:04] Project beta-update-databases-eqiad build #31949: 04STILL FAILING in 3.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31949/ [11:25:29] Project beta-scap-eqiad build #237835: 04STILL FAILING in 9 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237835/ [11:35:35] Project beta-scap-eqiad build #237836: 04STILL FAILING in 8 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237836/ [11:45:56] Project beta-scap-eqiad build #237837: 04STILL FAILING in 9 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237837/ [11:56:03] Project beta-scap-eqiad build #237838: 04STILL FAILING in 8 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237838/ [12:06:10] Project beta-scap-eqiad build #237839: 04STILL FAILING in 8 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237839/ [12:09:33] (03PS1) 10Santhosh: Generate coverage for ExternalGuidance [integration/config] - 10https://gerrit.wikimedia.org/r/490590 (https://phabricator.wikimedia.org/T215979) [12:16:23] Project beta-scap-eqiad build #237840: 04STILL FAILING in 9 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237840/ [12:20:04] Project beta-update-databases-eqiad build #31950: 04STILL FAILING in 3.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31950/ [12:26:37] Project beta-scap-eqiad build #237841: 04STILL FAILING in 8 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237841/ [12:37:01] Project beta-scap-eqiad build #237842: 04STILL FAILING in 9 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237842/ [12:47:16] Project beta-scap-eqiad build #237843: 04STILL FAILING in 9 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237843/ [12:47:33] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10Krenair) >>! In T216067#4953699, @Krenair wrote: >>>! In T216067#4953495, @jcrespo wrote: >>> I intend to attempt https://wikitech.wikimedia.org/wiki/Setting_up_a_MySQL_replica#... [12:48:28] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10Krenair) >>! In T216067#4953700, @Marostegui wrote: >>>! In T216067#4953498, @ArielGlenn wrote: >> @Marostegui /srv/sqldata has 39G on it on db04, presumably that's pretty close... [12:51:22] (03PS1) 10Pwirth: Edit Project Config [extensions/AdvancedMeta] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/490599 [12:51:47] (03Abandoned) 10Pwirth: Edit Project Config [extensions/AdvancedMeta] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/490599 (owner: 10Pwirth) [12:52:08] Krenair: where have you got transfer.py on deployment-cumin? I want to try something (withou actually running the script though) [12:52:36] apergos, I cloned the repo to /home/krenair/wmfmariadbpy [12:52:45] ok lemme go play there [12:52:47] and pulled down the commit from gerrit that was linked [12:52:49] sure [12:52:51] or maybe take a copy [12:52:58] best take a copy and I'll report back in a minute [12:53:00] k [12:53:48] I'm busy for the next 2-4 hours [12:55:01] ImportError: No module named cumin [12:55:05] that would be the problem [12:55:23] ok, see you around then maybe [12:57:27] Project beta-scap-eqiad build #237844: 04STILL FAILING in 8 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237844/ [12:59:10] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10jcrespo) > Strange since there is CuminExecution.py in the same directory as transfer.py So WMFMariadbpy is WIP, and has not yet been productionized. It will likely search for... [13:00:17] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10jcrespo) PS: I accept contributors, as this will help us dbas, but also other people manage and transfer files and databases. [13:04:42] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10ArielGlenn) When I go to the directory and do the import manually I get: `ariel@deployment-cumin:~/wmfmariadbpy/wmfmariadbpy$ python Python 2.7.9 (default, Sep 25 2018, 20:42:1... [13:06:33] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10Krenair) >>! In T216067#4954038, @ArielGlenn wrote: > When I go to the directory and do the import manually I get: > > `ariel@deployment-cumin:~/wmfmariadbpy/wmfmariadbpy$ pyth... [13:07:44] Project beta-scap-eqiad build #237845: 04STILL FAILING in 9 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237845/ [13:20:04] Project beta-update-databases-eqiad build #31951: 04STILL FAILING in 3.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31951/ [13:20:53] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10jcrespo) check `/usr/lib/python3/dist-packages/cumin/__init__.py` exist, it does on production. [13:21:20] Yippee, build fixed! [13:21:20] Project beta-scap-eqiad build #237846: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237846/ [13:24:40] !log rearm keyholder on deployment-deploy01: sudo keyholder arm, passwords on https://wikitech.wikimedia.org/wiki/Keyholder [13:24:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:35:58] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews, 10Graphoid, 10Operations, and 2 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10Jhernandez) Given the SCB deprecation and the incoming work load to re-architect this to work on Kubernetes to whoever t... [13:46:43] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10ArielGlenn) Correct, it needed python3, the directory exists , with python3 the error is different and it's likely the error jcrespo mentioned earlier. [14:00:18] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews, 10MediaWiki-extensions-UserMerge, 10Stewards-and-global-tools: UserMerge: Code Stewardship Review - https://phabricator.wikimedia.org/T204747 (10MarcoAurelio) >>! In T204747#4952853, @Jrbranaa wrote: > @MarcoAurelio, we're going to move forwa... [14:02:22] 10Release-Engineering-Team, 10MediaWiki-extensions-UserMerge, 10Stewards-and-global-tools: Undeploy UserMerge Extension from WMF production - https://phabricator.wikimedia.org/T216089 (10MarcoAurelio) We should probably figure a way to keep `Special:Log/usermerge` entries even after the extension is undeploy... [14:20:05] Project beta-update-databases-eqiad build #31952: 04STILL FAILING in 3.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31952/ [14:28:32] (03PS1) 10MarcoAurelio: Archive the UploadLocal extension [integration/config] - 10https://gerrit.wikimedia.org/r/490606 (https://phabricator.wikimedia.org/T213011) [14:29:07] (03PS2) 10MarcoAurelio: Archive the UploadLocal extension [integration/config] - 10https://gerrit.wikimedia.org/r/490606 (https://phabricator.wikimedia.org/T213011) [14:32:25] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews, 10Graphoid, 10Operations, and 2 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10dr0ptp4kt) Hi all - I was aware of this task but hadn't been following it. But it was brought to my attention as having... [15:20:09] Project beta-update-databases-eqiad build #31953: 04STILL FAILING in 8.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31953/ [15:24:02] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10Krenair) Did someone fix deployment-db04? It appears to have MySQL running now... [15:26:09] (03CR) 10Subramanya Sastry: [C: 03+1] "> Patch Set 3:" [integration/config] - 10https://gerrit.wikimedia.org/r/490528 (owner: 10Gergő Tisza) [15:26:12] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10ArielGlenn) Not I. Anyone else? [15:26:45] I have been doing my other work (read: dumps stuff that must be fixed) for a few hours, so did not continue on the db05 path... [15:28:05] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10Krenair) I'm wondering if it's possible that the evacuation of db04 from the old broken hypervisor to the new one might have solved the problem. [15:30:17] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews, 10Graphoid, 10Operations, and 2 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10akosiaris) >>! In T211881#4952874, @Jrbranaa wrote: >> In any case, and with the risk of repeating myself, the service i... [15:37:26] interesting, scap repos we haven't deployed since tin -> deploy1001 switch still have git_server: tin in their DEPLOY_HEAD [15:37:32] $ find /srv/deployment -name DEPLOY_HEAD | xargs grep git_server:.*tin [15:37:53] found out when reinstalling a logstash host [15:38:09] I guess it is ok to change now tho? [15:46:35] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10Andrew) >>! In T216067#4954457, @Krenair wrote: > Did someone fix deployment-db04? It appears to have MySQL running now... All I did was move it and run fsck. If the db is up... [16:20:05] Project beta-update-databases-eqiad build #31954: 04STILL FAILING in 4.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31954/ [16:22:33] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews, 10MediaWiki-extensions-CodeReview: CodeReview extension: Code stewardship review - https://phabricator.wikimedia.org/T205482 (10Jrbranaa) @greg @kunal, @Krinkle and I were chatting about moving this forward. Sounds like the next step is to hav... [16:37:34] (03CR) 10Subramanya Sastry: [C: 03+1] "> Patch Set 3:" [integration/config] - 10https://gerrit.wikimedia.org/r/490528 (owner: 10Gergő Tisza) [16:47:08] (03CR) 10Jforrester: [C: 03+2] Add WikibaseCirrusSearch extension [tools/release] - 10https://gerrit.wikimedia.org/r/489362 (https://phabricator.wikimedia.org/T215684) (owner: 10Smalyshev) [16:47:57] (03Merged) 10jenkins-bot: Add WikibaseCirrusSearch extension [tools/release] - 10https://gerrit.wikimedia.org/r/489362 (https://phabricator.wikimedia.org/T215684) (owner: 10Smalyshev) [16:54:21] good morning / day releng folks .. can i get a review / +2 from some someone on https://gerrit.wikimedia.org/r/c/integration/config/+/490528 ? [17:09:11] 10Beta-Cluster-Infrastructure: Beta cluster is broken: No working replica DB server: Unknown error (172.16.5.5:3306)) - https://phabricator.wikimedia.org/T216045 (10dcausse) 05Open→03Resolved [17:20:09] Project beta-update-databases-eqiad build #31955: 04STILL FAILING in 9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31955/ [17:28:00] godog: yeah, there is a task about making the kind of change automated.... but it's pretty low prio [17:29:13] greg-g: ah! thanks, good to know [17:29:36] I meant literally changing it now on deploy1001, although of course having scap do it would be best [17:29:46] right right [17:30:07] I changed logstash/plugins since I'm reinstalling a few VMs though there's another few repos [18:01:57] 10Project-Admins: Create component project for Status tool - https://phabricator.wikimedia.org/T216082 (10abian) [18:14:36] 10Beta-Cluster-Infrastructure: Puppet failures on deployment-deploy01.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T216164 (10bd808) [18:14:56] is it expected that the beta cluster is still experiencing problems? I thought the VPS failure was more or less resolved [18:16:52] (beta wikidata is apparently in automatic read-only mode due to replication lag) [18:17:14] 10Beta-Cluster-Infrastructure: Puppet failures on deployment-deploy01.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T216164 (10Jdforrester-WMF) Also the case on deployment-deploy02.deployment-prep.eqiad.wmflabs and probably other Beta Cluster servers? [18:20:09] Project beta-update-databases-eqiad build #31956: 04STILL FAILING in 8.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31956/ [18:31:33] (03PS4) 10Dduvall: maintenance: Cleanup old Docker images at a lower threshold [integration/config] - 10https://gerrit.wikimedia.org/r/490505 (https://phabricator.wikimedia.org/T177867) [18:37:49] (03PS5) 10Dduvall: maintenance: Cleanup old Docker images at a lower threshold [integration/config] - 10https://gerrit.wikimedia.org/r/490505 (https://phabricator.wikimedia.org/T177867) [18:39:35] 10Beta-Cluster-Infrastructure: Puppet failures on deployment-deploy01.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T216164 (10bd808) From the prod hieradata/common/profile/services_proxy.yaml: `lang=yaml # This list is mostly to be used inside mediawiki. profile::services_proxy::services:... [18:39:54] thcipriani awesome @ your patch! [18:40:01] paladox: thanks :) [18:45:17] Lucas_WMDE: sadly yes, due to the cloudvirt failures it wasn't just vpses disappearing, but disk curroption. so bringing back the dbs online is taking a while. [18:45:32] Lucas_WMDE: see also: https://phabricator.wikimedia.org/T216067 [18:51:02] alright, thanks greg-g [18:51:34] 10Beta-Cluster-Infrastructure: Puppet failures on deployment-deploy01.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T216164 (10bd808) I made a quick and ugly hack to get Puppet working here again by [[https://wikitech.wikimedia.org/w/index.php?title=Hiera:Deployment-prep&diff=1816153&oldid=18... [18:58:51] (03CR) 10Jforrester: [C: 03+1] Drop PHP 7.0 compatibility for Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/490528 (owner: 10Gergő Tisza) [19:05:55] (03PS10) 10Dduvall: Use blubberoid.wikimedia.org to process blubber.yaml [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/480689 (https://phabricator.wikimedia.org/T212247) [19:06:31] (03CR) 10Dduvall: "check experimental" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/480689 (https://phabricator.wikimedia.org/T212247) (owner: 10Dduvall) [19:20:10] Project beta-update-databases-eqiad build #31957: 04STILL FAILING in 9.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31957/ [19:21:26] oh https://gerrit-review.googlesource.com/q/topic:%22checkers%22+(status:open%20OR%20status:merged) wow [19:23:38] (03CR) 10Dduvall: "check experimental" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/480689 (https://phabricator.wikimedia.org/T212247) (owner: 10Dduvall) [19:35:57] (03PS1) 10Clarakosi: Setup Basic CI for mediawiki/services/kask [integration/config] - 10https://gerrit.wikimedia.org/r/490678 [20:06:53] 10MediaWiki-Codesniffer, 10MediaWiki-General-or-Unknown, 10Technical-Debt: Encourage type hints for function parameters and return after moving MediaWiki to PHP 7 - https://phabricator.wikimedia.org/T178136 (10thiemowmde) I'm not sure I understand the idea presented in this tickets description. Is the sugges... [20:31:29] 10Phabricator: Git hashes for SyntaxHighlight_GeSHi not recognised in Phabricator (due to Pygments fork repo import) - https://phabricator.wikimedia.org/T119588 (10mmodell) So which repo should we nuke? [20:34:40] 10Phabricator: Git hashes for SyntaxHighlight_GeSHi not recognised in Phabricator (due to Pygments fork repo import) - https://phabricator.wikimedia.org/T119588 (10Krinkle) mediawiki/extensions/SyntaxHighlight_Pygments, which was created and only used for a very short time before we realised the mistake and then... [20:37:15] Project beta-scap-eqiad build #237881: 15ABORTED in 52 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237881/ [20:37:19] Project beta-update-databases-eqiad build #31958: 04STILL FAILING in 4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31958/ [20:37:35] I aborted beta-scap-eqiad by the by [20:38:30] Project beta-scap-eqiad build #237882: 04FAILURE in 0.63 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237882/ [20:39:08] oh good [20:40:23] (03CR) 10Krinkle: [C: 03+2] Drop PHP 7.0 compatibility for Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/490528 (owner: 10Gergő Tisza) [20:40:34] 10Phabricator, 10Mail, 10Operations: DomainKeys Identified Mail (DKIM) for phabricator.wikimedia.org - https://phabricator.wikimedia.org/T116805 (10mmodell) I've never seen this but I don't routinely use gmail for following phabricator. Is this really still an issue? [20:42:26] 10Phabricator, 10Mail, 10Operations: DomainKeys Identified Mail (DKIM) for phabricator.wikimedia.org - https://phabricator.wikimedia.org/T116805 (10mmodell) afaik the outbound email path was completely redone since this task was filed. [20:43:07] (03Merged) 10jenkins-bot: Drop PHP 7.0 compatibility for Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/490528 (owner: 10Gergő Tisza) [20:43:17] 10Phabricator, 10Operations, 10Operations-Software-Development, 10Technical-Debt: Update Puppet repo code that uses deprecated maniphest.update/.createtask/.query Conduit API - https://phabricator.wikimedia.org/T159045 (10mmodell) [20:45:56] (03PS1) 10Thcipriani: Revert "lockfile: python2/3 compat" [tools/scap] - 10https://gerrit.wikimedia.org/r/490694 [20:49:57] (03CR) 10Krinkle: "@Hashar Is this something you can prioritise next week? Let me know if not and if you'd like my help as this is blocking T133646." [integration/config] - 10https://gerrit.wikimedia.org/r/487896 (https://phabricator.wikimedia.org/T211784) (owner: 10Hashar) [20:50:45] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/490528 / T133646 [20:50:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:50:48] T133646: Run performance test on commits (Fresnel) - https://phabricator.wikimedia.org/T133646 [20:50:57] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/490528 / T216102 [20:50:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:50:59] T216102: Determine which PHP version to target with Parsoid - https://phabricator.wikimedia.org/T216102 [20:51:46] (03CR) 10Thcipriani: [C: 03+2] "Works for scap3 and for various other scap subcommands, just not scap mediawiki subcommands. Needs additional tests." [tools/scap] - 10https://gerrit.wikimedia.org/r/490694 (owner: 10Thcipriani) [20:52:54] (03Merged) 10jenkins-bot: Revert "lockfile: python2/3 compat" [tools/scap] - 10https://gerrit.wikimedia.org/r/490694 (owner: 10Thcipriani) [20:53:49] (03CR) 10jenkins-bot: Revert "lockfile: python2/3 compat" [tools/scap] - 10https://gerrit.wikimedia.org/r/490694 (owner: 10Thcipriani) [21:19:35] 10Phabricator (Upstream), 10Release-Engineering-Team (Kanban), 10Upstream, 10User-MModell, 10User-Ryasmeen: 404 on workboard for an existing project (due to custom filter applied which did not exist in database) - https://phabricator.wikimedia.org/T199207 (10mmodell) I think this is fixed upstream and I'... [21:20:04] Project beta-update-databases-eqiad build #31959: 04STILL FAILING in 3.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31959/ [21:43:01] (03CR) 10Jforrester: [C: 03+1] maintenance: Cleanup old Docker images at a lower threshold [integration/config] - 10https://gerrit.wikimedia.org/r/490505 (https://phabricator.wikimedia.org/T177867) (owner: 10Dduvall) [21:46:28] (03CR) 10Dduvall: "check experimental" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/480689 (https://phabricator.wikimedia.org/T212247) (owner: 10Dduvall) [22:20:04] Project beta-update-databases-eqiad build #31960: 04STILL FAILING in 3.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31960/ [23:07:16] Yippee, build fixed! [23:07:17] Project beta-scap-eqiad build #237883: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/237883/ [23:08:15] github gets a WIP indicator: https://github.blog/2019-02-14-introducing-draft-pull-requests/ [23:20:04] Project beta-update-databases-eqiad build #31961: 04STILL FAILING in 4.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/31961/ [23:20:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): contint1001 and contint2001 should allow port 9418 from their ipv6 addresses - https://phabricator.wikimedia.org/T216204 (10dduvall) [23:31:31] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): contint1001 and contint2001 should allow port 9418 from their ipv6 addresses - https://phabricator.wikimedia.org/T216204 (10dduvall) p:05Triage→03Normal a:03dduvall