[00:20:02] Project beta-update-databases-eqiad build #32059: 04STILL FAILING in 1.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32059/ [00:25:01] (03CR) 10Krinkle: [C: 03+2] jjb: Update node10-test* images in jobs to version 0.3.3 [integration/config] - 10https://gerrit.wikimedia.org/r/491380 (https://phabricator.wikimedia.org/T211784) (owner: 10Krinkle) [00:27:48] (03Merged) 10jenkins-bot: jjb: Update node10-test* images in jobs to version 0.3.3 [integration/config] - 10https://gerrit.wikimedia.org/r/491380 (https://phabricator.wikimedia.org/T211784) (owner: 10Krinkle) [00:51:49] (03CR) 10Krinkle: [C: 03+2] Add missing node-gyp deps to fresnel image [integration/config] - 10https://gerrit.wikimedia.org/r/491381 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [00:53:43] (03Merged) 10jenkins-bot: Add missing node-gyp deps to fresnel image [integration/config] - 10https://gerrit.wikimedia.org/r/491381 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [00:54:25] !log Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/491381 / bc8e4198961cb73 / T133646 [00:54:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:54:28] T133646: Run performance test on commits (Fresnel) - https://phabricator.wikimedia.org/T133646 [00:56:51] (03CR) 10Krinkle: [C: 03+2] jjb: Update Fresnel job to quibble-fresnel:0.0.28-2 [integration/config] - 10https://gerrit.wikimedia.org/r/491382 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [00:59:41] (03Merged) 10jenkins-bot: jjb: Update Fresnel job to quibble-fresnel:0.0.28-2 [integration/config] - 10https://gerrit.wikimedia.org/r/491382 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [01:02:58] (03PS1) 10Krinkle: fresnel: Fix syntax error [integration/config] - 10https://gerrit.wikimedia.org/r/491394 (https://phabricator.wikimedia.org/T133646) [01:03:16] (03CR) 10Krinkle: [C: 03+2] "(Same version, given it was never built)" [integration/config] - 10https://gerrit.wikimedia.org/r/491394 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [01:05:04] (03Merged) 10jenkins-bot: fresnel: Fix syntax error [integration/config] - 10https://gerrit.wikimedia.org/r/491394 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [01:20:07] Project beta-update-databases-eqiad build #32060: 04STILL FAILING in 6.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32060/ [01:26:07] (03PS1) 10Krinkle: Set CHROMIUM_FLAGS in fresnel image [integration/config] - 10https://gerrit.wikimedia.org/r/491396 (https://phabricator.wikimedia.org/T133646) [02:20:02] Project beta-update-databases-eqiad build #32061: 04STILL FAILING in 1.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32061/ [03:00:59] 10Project-Admins, 10MediaWiki-Documentation, 10Documentation, 10Proposal: Consider archiving #MediaWiki-Documentation in favour of single #Documentation - https://phabricator.wikimedia.org/T142345 (10Tgr) >>! In T142345#3656012, @Legoktm wrote: > That works for search, yes. But is it possible to subscribe... [03:20:02] Project beta-update-databases-eqiad build #32062: 04STILL FAILING in 1.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32062/ [03:50:37] (03PS2) 10Krinkle: Set CHROMIUM_FLAGS in fresnel image [integration/config] - 10https://gerrit.wikimedia.org/r/491396 (https://phabricator.wikimedia.org/T133646) [03:50:39] (03PS1) 10Krinkle: jjb: Update Fresnel job to quibble-fresnel:0.0.28-3 [integration/config] - 10https://gerrit.wikimedia.org/r/491403 (https://phabricator.wikimedia.org/T133646) [03:51:16] (03CR) 10Krinkle: [C: 03+2] Set CHROMIUM_FLAGS in fresnel image [integration/config] - 10https://gerrit.wikimedia.org/r/491396 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [03:52:21] (03CR) 10Krinkle: [C: 03+2] jjb: Update Fresnel job to quibble-fresnel:0.0.28-3 [integration/config] - 10https://gerrit.wikimedia.org/r/491403 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [03:55:27] !log Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/491403 [03:55:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:14:58] !log Disabled spam account on Phab - https://phabricator.wikimedia.org/people/manage/18915/ [04:14:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:20:10] Project beta-update-databases-eqiad build #32063: 04STILL FAILING in 9.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32063/ [04:47:20] (03PS1) 10Krinkle: add firefox-esr to fresnel, to make chromium work (!) [integration/config] - 10https://gerrit.wikimedia.org/r/491405 (https://phabricator.wikimedia.org/T133646) [04:47:22] (03PS1) 10Krinkle: jjb: Update Fresnel job to quibble-fresnel:0.0.28-4 [integration/config] - 10https://gerrit.wikimedia.org/r/491406 (https://phabricator.wikimedia.org/T133646) [04:47:29] Sorry legoktm [04:47:33] (03CR) 10Krinkle: [C: 03+2] add firefox-esr to fresnel, to make chromium work (!) [integration/config] - 10https://gerrit.wikimedia.org/r/491405 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [04:49:20] (03CR) 10Krinkle: [C: 03+2] jjb: Update Fresnel job to quibble-fresnel:0.0.28-4 [integration/config] - 10https://gerrit.wikimedia.org/r/491406 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [04:49:30] (03Merged) 10jenkins-bot: add firefox-esr to fresnel, to make chromium work (!) [integration/config] - 10https://gerrit.wikimedia.org/r/491405 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [04:50:02] !log Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/491405 [04:50:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:52:10] (03Merged) 10jenkins-bot: jjb: Update Fresnel job to quibble-fresnel:0.0.28-4 [integration/config] - 10https://gerrit.wikimedia.org/r/491406 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [04:55:22] !log Removed stale cherry-pick of Ic7e726768701fefdee68622b08e3f2995779fe5a from deployment-puppetmaster03 that was blocking rebase on origin/production [04:55:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:02:20] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Puppet failures on deployment-deploy01.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T216164 (10bd808) @joe that patch is apparently not quite sufficient. New output with the dummy settings from T216164#4955058 removed: ` Error: Could not r... [05:12:37] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Puppet failures on deployment-deploy01.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T216164 (10bd808) Actually it might be related after all. The failure is from an attempt to install the niginx-full package which seems to be triggered by... [05:20:02] Project beta-update-databases-eqiad build #32064: 04STILL FAILING in 1.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32064/ [06:05:39] (03CR) 10Giuseppe Lavagetto: Rewrite the concurrency logic of OpcacheManager (035 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/490888 (https://phabricator.wikimedia.org/T211964) (owner: 10Giuseppe Lavagetto) [06:19:28] (03PS2) 10Giuseppe Lavagetto: Rewrite the concurrency logic of OpcacheManager [tools/scap] - 10https://gerrit.wikimedia.org/r/490888 (https://phabricator.wikimedia.org/T211964) [06:20:07] Project beta-update-databases-eqiad build #32065: 04STILL FAILING in 6.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32065/ [07:20:02] Project beta-update-databases-eqiad build #32066: 04STILL FAILING in 1.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32066/ [08:07:32] 10Continuous-Integration-Config, 10ExternalGuidance: Add MobileFrontend dependency for ExternalGuidance - https://phabricator.wikimedia.org/T213166 (10Petar.petkovic) [08:13:39] 10Continuous-Integration-Config: phpcs at jenkins is ignoring .phpcs.xml file and only checking files modified in the commit - https://phabricator.wikimedia.org/T216417 (10hashar) Our `mediawiki/mediawiki-codesniffer` ships with an utility that can be used as a bootstrapper to alter the configuration, that is us... [08:20:02] Project beta-update-databases-eqiad build #32067: 04STILL FAILING in 2.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32067/ [08:23:05] (03PS1) 10Giuseppe Lavagetto: Remove functionality to talk to conftool [tools/scap] - 10https://gerrit.wikimedia.org/r/491412 [08:25:06] (03CR) 10jerkins-bot: [V: 04-1] Remove functionality to talk to conftool [tools/scap] - 10https://gerrit.wikimedia.org/r/491412 (owner: 10Giuseppe Lavagetto) [08:45:17] Krinkle: why sorry? [08:45:35] (03CR) 10Hashar: "HTML5 is magic. Thank you!" [integration/docroot] - 10https://gerrit.wikimedia.org/r/489106 (owner: 10Krinkle) [08:46:05] https://gerrit.wikimedia.org/r/c/integration/config/+/491405 is pretty hilarious [08:46:29] probably worth a bug report up to debian though [08:54:07] 10Continuous-Integration-Config, 10Patch-For-Review: phpcs at jenkins is ignoring .phpcs.xml file and only checking files modified in the commit - https://phabricator.wikimedia.org/T216417 (10Bawolff) > The reason the change started failing is the patch got rebased without taking in account that mediawiki/medi... [08:55:41] !log Cleaning contint1001 / partition [08:55:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:20:10] Project beta-update-databases-eqiad build #32068: 04STILL FAILING in 9.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32068/ [09:21:31] 10Continuous-Integration-Config, 10Patch-For-Review: phpcs at jenkins is ignoring .phpcs.xml file and only checking files modified in the commit - https://phabricator.wikimedia.org/T216417 (10Bawolff) 05Open→03Resolved a:03Bawolff Thanks. [09:25:47] 10Continuous-Integration-Infrastructure: CI permission errors causing tests to fail - https://phabricator.wikimedia.org/T215992 (10hashar) 05Open→03Resolved When it happens, it is due to a faulty job which has the log directory created within docker and thus is owned by root. That causes the next build to fa... [09:28:45] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Backlog): Move scap-vagrant to Gerrit - https://phabricator.wikimedia.org/T216483 (10hashar) p:05Triage→03Low [09:28:56] 10Beta-Cluster-Infrastructure: Database locked on beta.wmflabs.org sites - https://phabricator.wikimedia.org/T216484 (10RhinosF1) [09:30:53] 10Beta-Cluster-Infrastructure: Database locked on beta.wmflabs.org sites - https://phabricator.wikimedia.org/T216484 (10RhinosF1) [09:31:33] 10Beta-Cluster-Infrastructure: Database locked on beta.wmflabs.org sites - https://phabricator.wikimedia.org/T216484 (10RhinosF1) [09:41:22] (03CR) 10Volans: [C: 03+1] "From the little context-knowledge I have of scap codebase... LGTM" [tools/scap] - 10https://gerrit.wikimedia.org/r/490888 (https://phabricator.wikimedia.org/T211964) (owner: 10Giuseppe Lavagetto) [10:20:02] Project beta-update-databases-eqiad build #32069: 04STILL FAILING in 1.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32069/ [10:29:58] 10Beta-Cluster-Infrastructure: Database locked on beta.wmflabs.org sites - https://phabricator.wikimedia.org/T216484 (10Bawolff) After doing some poking, looks like the master db (deployment-db03) might be down. [10:30:33] 10Beta-Cluster-Infrastructure: Database locked on beta.wmflabs.org sites - https://phabricator.wikimedia.org/T216484 (10Bawolff) Arguably, the read only error message for this situation could probably be improved. [10:30:51] 10Beta-Cluster-Infrastructure: Database locked on beta.wmflabs.org sites (deployment-db03 down?) - https://phabricator.wikimedia.org/T216484 (10Bawolff) [10:30:55] Project beta-scap-eqiad build #238468: 04FAILURE in 2.7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/238468/ [10:44:24] Yippee, build fixed! [10:44:24] Project beta-scap-eqiad build #238469: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/238469/ [11:02:47] 10Beta-Cluster-Infrastructure: Database locked on beta.wmflabs.org sites (deployment-db03 down?) - https://phabricator.wikimedia.org/T216484 (10RhinosF1) >>! In T216484#4963932, @Bawolff wrote: > Arguably, the read only error message for this situation could probably be improved. Arguably??? I think it does ne... [11:06:23] 10Beta-Cluster-Infrastructure: Database locked on beta.wmflabs.org sites (deployment-db03 down?) - https://phabricator.wikimedia.org/T216484 (10Krenair) [11:06:28] 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 10cloud-services-team (Kanban): deployment-db03.deployment-prep.eqiad.wmflabs instance can not start - https://phabricator.wikimedia.org/T216404 (10Krenair) [11:18:37] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10Krenair) -bash: innobackupex-1.5.1: command not found -bash: innobackupex: command not found [11:20:02] Project beta-update-databases-eqiad build #32070: 04STILL FAILING in 1.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32070/ [11:22:48] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10Krenair) And of course I can't find the package from db03/db04 because their apt status is corrupted... [11:28:11] legoktm: I installed firefox to satisfy chrome's needed. [11:28:57] (03PS1) 10Thiemo Kreuz (WMDE): Make the Translate extensions coverage report public [integration/config] - 10https://gerrit.wikimedia.org/r/491446 [11:31:54] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10jcrespo) If they are using our wmf-mariadb101 package, the binary is called "mariabackup", it was put on the path on the latest versions. [11:54:06] 10Release-Engineering-Team (Kanban), 10ContentTranslation, 10Patch-For-Review, 10User-zeljkofilipin: The first Selenium test for ContentTranslation - https://phabricator.wikimedia.org/T216424 (10zeljkofilipin) a:05zeljkofilipin→03None [12:00:07] (03PS3) 10MarcoAurelio: Archive the UploadLocal extension [integration/config] - 10https://gerrit.wikimedia.org/r/490606 (https://phabricator.wikimedia.org/T213011) [12:16:13] 10Gerrit, 10Operations, 10serviceops: Gerrit loads very slowly - https://phabricator.wikimedia.org/T215855 (10hashar) From P8071 , it is often helpful to split by queues with `--by-queue` which would shows the number of worker threads and items for each queues. Example showing `ReplicateTo-slaves` and `WorkQ... [12:16:35] (03CR) 10BBlack: [C: 03+1] "Ready to go, please merge/deploy when able, thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/490880 (owner: 10BBlack) [12:20:07] Project beta-update-databases-eqiad build #32071: 04STILL FAILING in 6.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32071/ [12:26:25] 10Project-Admins: Create project for "Extension:WebDAV" - https://phabricator.wikimedia.org/T216502 (10MarcoAurelio) [12:29:52] 10Project-Admins, 10User-MarcoAurelio: Create project for "Extension:WebDAV" - https://phabricator.wikimedia.org/T216502 (10MarcoAurelio) 05Open→03Resolved a:03MarcoAurelio #mediawiki-extensions-webdav created. Repository in gerrit and maintainer willing to use phabricator to track issues/development. Done. [12:30:33] 10Gerrit, 10Operations, 10serviceops: Gerrit loads very slowly - https://phabricator.wikimedia.org/T215855 (10hashar) tools.ci first request was on 2019-02-12T01:12:07 . Maybe it is doing the requests to fast for Gerrit :/ Maybe https://gerrit.wikimedia.org/r/monitoring can give us some hint. Maybe we ca... [13:20:02] Project beta-update-databases-eqiad build #32072: 04STILL FAILING in 2.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32072/ [14:20:07] Project beta-update-databases-eqiad build #32073: 04STILL FAILING in 6.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32073/ [15:02:56] 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 10cloud-services-team (Kanban): deployment-db03.deployment-prep.eqiad.wmflabs instance can not start - https://phabricator.wikimedia.org/T216404 (10Addshore) p:05Normal→03High I'm not sure why this is only "Normal", I'm going to at least bump this to high. We a... [15:03:31] 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 10User-Addshore, 10cloud-services-team (Kanban): deployment-db03.deployment-prep.eqiad.wmflabs instance can not start - https://phabricator.wikimedia.org/T216404 (10Addshore) [15:03:42] 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 10Wikidata, 10User-Addshore, 10cloud-services-team (Kanban): deployment-db03.deployment-prep.eqiad.wmflabs instance can not start - https://phabricator.wikimedia.org/T216404 (10Addshore) [15:09:50] (03PS3) 10Zoranzoki21: Add Jeblad at CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/487543 [15:09:58] (03CR) 10Zoranzoki21: "Anyone?" [integration/config] - 10https://gerrit.wikimedia.org/r/487543 (owner: 10Zoranzoki21) [15:20:02] Project beta-update-databases-eqiad build #32074: 04STILL FAILING in 1.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32074/ [15:20:28] (03PS1) 10Krinkle: Set MW_SERVER/MW_SCRIPT_PATH for fresnel [integration/config] - 10https://gerrit.wikimedia.org/r/491513 (https://phabricator.wikimedia.org/T133646) [15:25:25] 10Continuous-Integration-Infrastructure, 10cloud-services-team, 10puppet-compiler: compiler1002.puppet-diffs.eqiad.wmflabs instance is down - https://phabricator.wikimedia.org/T216513 (10hashar) [15:26:47] 10Continuous-Integration-Infrastructure, 10cloud-services-team, 10puppet-compiler: compiler1002.puppet-diffs.eqiad.wmflabs instance is down - https://phabricator.wikimedia.org/T216513 (10hashar) 05Open→03Resolved a:03hashar I started the instance via Horizon and it got attached to Jenkins again. [15:34:50] (03PS1) 10Krinkle: jjb: Update Fresnel job to quibble-fresnel:0.0.28-5 [integration/config] - 10https://gerrit.wikimedia.org/r/491516 (https://phabricator.wikimedia.org/T133646) [15:40:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10cloud-services-team: Restore Jenkins CI slave integration-slave-jessie-android 172.16.6.183 - https://phabricator.wikimedia.org/T216517 (10hashar) [15:41:06] James_F: https://phabricator.wikimedia.org/T140664 [15:42:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10cloud-services-team: Restore Jenkins CI slave integration-slave-jessie-android 172.16.6.183 - https://phabricator.wikimedia.org/T216517 (10hashar) I can not find the instance in Horizon/OpenStack anymore :-( [15:42:22] 10Release-Engineering-Team (Watching / External), 10Scap, 10serviceops, 10User-jijiki: Allow scap sync to deploy gradually - https://phabricator.wikimedia.org/T212147 (10jijiki) [15:46:48] !log Creating integration-slave-jessie-android m1.large it got deleted last week | T216517 [15:46:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:46:51] T216517: Restore Jenkins CI slave integration-slave-jessie-android 172.16.6.183 - https://phabricator.wikimedia.org/T216517 [15:52:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Restore Jenkins CI slave integration-slave-jessie-android 172.16.6.183 - https://phabricator.wikimedia.org/T216517 (10bd808) [15:52:52] (03CR) 10Krinkle: [C: 03+2] Set MW_SERVER/MW_SCRIPT_PATH for fresnel [integration/config] - 10https://gerrit.wikimedia.org/r/491513 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [15:53:21] 10Continuous-Integration-Infrastructure, 10puppet-compiler: compiler1002.puppet-diffs.eqiad.wmflabs instance is down - https://phabricator.wikimedia.org/T216513 (10bd808) [15:54:49] (03Merged) 10jenkins-bot: Set MW_SERVER/MW_SCRIPT_PATH for fresnel [integration/config] - 10https://gerrit.wikimedia.org/r/491513 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [15:55:36] !log Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/491513 [15:55:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:56:04] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Restore Jenkins CI slave integration-slave-jessie-android 172.16.6.183 - https://phabricator.wikimedia.org/T216517 (10thcipriani) @dduvall removed that last week after one of the cloudvirt machines had issues. Rather than rebuild the machine... [15:56:06] (03CR) 10Krinkle: [C: 03+2] jjb: Update Fresnel job to quibble-fresnel:0.0.28-5 [integration/config] - 10https://gerrit.wikimedia.org/r/491516 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [15:58:58] (03Merged) 10jenkins-bot: jjb: Update Fresnel job to quibble-fresnel:0.0.28-5 [integration/config] - 10https://gerrit.wikimedia.org/r/491516 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [15:59:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Restore Jenkins CI slave integration-slave-jessie-android 172.16.6.183 - https://phabricator.wikimedia.org/T216517 (10Mholloway) Yes, the CI job that it's my understanding integration-slave-jessie-android only exists to support (apps-android-... [16:01:42] James_F https://github.com/Krinkle/dotfiles#read-only-wmf, [16:07:31] 10Release-Engineering-Team, 10Analytics, 10EventBus, 10MediaWiki-Core-Testing, and 4 others: Flaky quibble-vendor-mysql-hhvm-docker test in Jenkins - https://phabricator.wikimedia.org/T216069 (10Pchelolo) This is getting in our way since we can't merge an otherwise ready patchset [16:09:28] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: Restore Jenkins CI slave integration-slave-jessie-android 172.16.6.183 - https://phabricator.wikimedia.org/T216517 (10hashar) Eventually searching Phabriator for the job name brings: {T198495} I will delete the agent I... [16:09:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: Restore Jenkins CI slave integration-slave-jessie-android 172.16.6.183 - https://phabricator.wikimedia.org/T216517 (10hashar) 05Open→03Declined [16:10:07] 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 10Wikidata, 10User-Addshore, 10cloud-services-team (Kanban): deployment-db03.deployment-prep.eqiad.wmflabs instance can not start - https://phabricator.wikimedia.org/T216404 (10bd808) a:05GTirloni→03None >>! In T216404#4964728, @Addshore wrote: > I'm not su... [16:13:27] 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 10Wikidata, 10User-Addshore, 10cloud-services-team (Kanban): deployment-db03.deployment-prep.eqiad.wmflabs instance can not start - https://phabricator.wikimedia.org/T216404 (10Addshore) > Who are you trying to push to move faster? What is the desired outcome o... [16:16:22] (03PS26) 10Paladox: zuul: convert to scap [wip] [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/488579 [16:16:37] (03CR) 10jerkins-bot: [V: 04-1] zuul: convert to scap [wip] [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/488579 (owner: 10Paladox) [16:20:06] Project beta-update-databases-eqiad build #32075: 04STILL FAILING in 6.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32075/ [16:25:47] 10Phabricator (Search), 10Release-Engineering-Team (Kanban), 10User-MModell: make phabricator search match projects based on alterenate hashtags - https://phabricator.wikimedia.org/T215085 (10mmodell) a:05mmodell→03None Not currently working on this but I need to figure out what the issue is with #mediaw... [16:51:13] 10Release-Engineering-Team (Kanban), 10TimedMediaHandler, 10User-zeljkofilipin: The first Selenium test for TimedMediaHandler - https://phabricator.wikimedia.org/T214480 (10zeljkofilipin) a:05brion→03zeljkofilipin [16:53:36] 10Phabricator, 10Mail, 10Operations: DomainKeys Identified Mail (DKIM) for phabricator.wikimedia.org - https://phabricator.wikimedia.org/T116805 (10Niedzielski) This is only a single datapoint but I noticed a Phab comment email notification from gerritbot was mistakenly marked as spam on Friday in my Gmail. [17:04:02] 10Release-Engineering-Team (Watching / External), 10MW-1.32-notes, 10MW-1.32-release: Release 1.32.1 as a maintenance release - https://phabricator.wikimedia.org/T213595 (10mmodell) @legoktm: I can do it if nobody else wants to ;) [17:06:53] With the new Docker/k8s pipeline, what will become convention for supplying configuration? [17:07:14] For services? [17:07:39] I guess; For whatever is containerized and needs config [17:07:58] That's been scap AFAIK so far [17:08:02] I think the idea is that you'd keep the config and code in the same repo to be injected together in the same repo into the container? [17:08:40] I see, so a change in configuration means a new container? [17:09:02] I think so? I'm the newbie in this, though. :-) [17:11:04] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10puppet-compiler: compiler1002.puppet-diffs.eqiad.wmflabs instance is down - https://phabricator.wikimedia.org/T216513 (10hashar) Joe said that it is /var/lib/catalog-differ being corrupted. That might be easy to recover :) [17:15:00] 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 10Wikidata, 10User-Addshore, 10cloud-services-team (Kanban): deployment-db03.deployment-prep.eqiad.wmflabs instance can not start - https://phabricator.wikimedia.org/T216404 (10Andrew) (For broad context: https://phabricator.wikimedia.org/T215217) [17:20:02] Project beta-update-databases-eqiad build #32076: 04STILL FAILING in 1.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32076/ [17:22:06] 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 10Wikidata, 10User-Addshore, 10cloud-services-team (Kanban): deployment-db03.deployment-prep.eqiad.wmflabs instance can not start - https://phabricator.wikimedia.org/T216404 (10bd808) [17:22:09] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Puppet failures on deployment-deploy01.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T216164 (10bd808) [17:22:15] 10Beta-Cluster-Infrastructure, 10Code-Stewardship-Reviews: deployment-prep: Code stewardship request - https://phabricator.wikimedia.org/T215217 (10bd808) [17:26:54] 10Beta-Cluster-Infrastructure, 10Code-Stewardship-Reviews: deployment-prep: Code stewardship request - https://phabricator.wikimedia.org/T215217 (10Bawolff) > Deployment-prep, also known as the beta cluster, is a Cloud VPS project originally created by technical volunteers {{cn}} Pretty sure that's not true.... [17:33:10] 10Deployments, 10Phabricator, 10Developer Productivity, 10User-MModell: Create a permalink which always redirects to the current week's train blocker task - https://phabricator.wikimedia.org/T207669 (10mmodell) a:05mmodell→03None I'd like to work on this some day soon but it's not currently a priority. [17:33:52] (03CR) 10Hashar: "\o/ Thank you for the patch! Will review soonish (tm) ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/490492 (https://phabricator.wikimedia.org/T198495) (owner: 10Mholloway) [17:34:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10puppet-compiler: compiler1002.puppet-diffs.eqiad.wmflabs instance is down - https://phabricator.wikimedia.org/T216513 (10herron) I've re-created compiler1002 from scratch and am working to bring the puppet compiler service up on... [17:36:37] 10Phabricator, 10Release-Engineering-Team (Kanban), 10media-storage, 10Patch-For-Review, 10User-MModell: Connect Phabricator to swift for storage of git-lfs and file uploads. - https://phabricator.wikimedia.org/T182085 (10mmodell) a:05mmodell→03None There is a patch to phabricator to make this work,... [17:53:15] 10Beta-Cluster-Infrastructure, 10Code-Stewardship-Reviews: deployment-prep: Code stewardship request - https://phabricator.wikimedia.org/T215217 (10greg) > TBH: I'm kind of surprised to see this task. I don't follow betalabs too much, but I thought it was pretty well accepted that it was a RelEng responsibilit... [18:20:02] Project beta-update-databases-eqiad build #32077: 04STILL FAILING in 1.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32077/ [18:31:45] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10Wikipedia-Android-App-Backlog: Disable/remove Jenkins Jobs for Android app - https://phabricator.wikimedia.org/T198862 (10Charlotte) [18:35:24] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Parsoid: Parsoid fails to deploy to beta cluster - https://phabricator.wikimedia.org/T216534 (10Arlolra) [19:20:06] Project beta-update-databases-eqiad build #32078: 04STILL FAILING in 6.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32078/ [19:57:30] (03PS2) 10Hashar: Remove apps-android-wikipedia-periodic-test job and related bits [integration/config] - 10https://gerrit.wikimedia.org/r/490492 (https://phabricator.wikimedia.org/T198495) (owner: 10Mholloway) [20:11:51] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Regression, 10Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640 (10Kipcool) 05Resolved→03Open Hi, it seems that I am also affected by the same issue: - Cannot assign user name "kipcool" to account 6839; nam... [20:16:41] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Jenkins: Jenkins plugins security update 2019-02-19 - https://phabricator.wikimedia.org/T216550 (10hashar) [20:17:14] and I am going to upgrade the jenkins plugins [20:17:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Jenkins: Jenkins plugins security update 2019-02-19 - https://phabricator.wikimedia.org/T216550 (10thcipriani) [20:18:07] oh tyler did :) [20:18:12] :) [20:18:37] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Jenkins: Jenkins plugins security update 2019-02-19 - https://phabricator.wikimedia.org/T216550 (10hashar) 05duplicate→03Resolved a:03thcipriani 19:57 restarting ci-jenkins for plugin update :) [20:18:40] I too was just about to double check :) [20:18:44] arf ;D [20:18:51] hashar: oh! I didn't see cloudfoundry [20:19:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Jenkins: Jenkins plugins security update 2019-02-19 - https://phabricator.wikimedia.org/T216550 (10hashar) [20:19:24] thcipriani: seems we are all coverd :) [20:20:02] Project beta-update-databases-eqiad build #32079: 04STILL FAILING in 2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32079/ [20:22:57] (03CR) 10Hashar: [C: 03+2] "Seems to catch all whistles and bells that were involved. I will also archive the Gerrit repository and disable the plugin in Jenkins :) " [integration/config] - 10https://gerrit.wikimedia.org/r/490492 (https://phabricator.wikimedia.org/T198495) (owner: 10Mholloway) [20:23:19] !log Deleting Jenkins job apps-android-wikipedia-periodic-test | T198495 [20:23:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:23:22] T198495: Fix or delete apps-android-wikipedia-periodic-test from CI - https://phabricator.wikimedia.org/T198495 [20:26:54] (03Merged) 10jenkins-bot: Remove apps-android-wikipedia-periodic-test job and related bits [integration/config] - 10https://gerrit.wikimedia.org/r/490492 (https://phabricator.wikimedia.org/T198495) (owner: 10Mholloway) [20:27:33] !log gerrit: archived repository integration/jenkinsci/android-emulator-plugin | T198495 [20:27:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:28:37] !log jenkins: disable Android Emulator plugin | T198495 [20:28:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:28:39] T198495: Fix or delete apps-android-wikipedia-periodic-test from CI - https://phabricator.wikimedia.org/T198495 [20:35:59] thcipriani: https://phabricator.wikimedia.org/F28247287 :D [20:36:05] for the zuul status page [20:36:23] courtesy of paladox ! [20:36:46] hashar: paladox yes please! very awesome. [20:36:52] :) [20:36:58] and from a chat we had with tyler [20:37:01] (already merged) and updated wmf/stable-2.16 [20:37:14] we gotta solve the Zuul madness before being able to upgrade to gerrit 2.16 [20:37:28] https://gerrit-review.googlesource.com/c/plugins/zuul-status/+/214812 [20:37:28] tentatively switch to use scap which will makes thing easier [20:37:35] yeh [20:39:35] hashar only needs https://github.com/wikimedia/integration-zuul/blob/debian/jessie-wikimedia/debian/patches/0013-WMF-pin-requirements-to-what-is-deployed.patch#L25 upgraded and https://github.com/wikimedia/integration-zuul/blob/debian/jessie-wikimedia/zuul/connection/gerrit.py#L388 to be stripped of /p/ [20:39:38] (03PS1) 10Krinkle: zuul: Enable fresnel job on coverage pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/491577 (https://phabricator.wikimedia.org/T133646) [20:39:47] seeing as /p/ is optional according to upstream [20:40:05] i've tested it in the scap version https://gerrit.wikimedia.org/r/#/c/integration/zuul/+/488579/25..26/zuul/connection/gerrit.py [20:40:27] ;))) [20:40:32] ah yeah you did all the patches already [20:40:46] well guess it is going to be easy to review / bump changelog and rebuild the package [20:40:55] I will check what I can do on that front this week [20:41:17] and https://gerrit.wikimedia.org/r/#/c/integration/zuul/+/488579/26/requirements.txt@11 [20:41:58] hashar though i did do the gitpython upgrade in the debian version here https://gerrit.wikimedia.org/r/#/c/integration/zuul/+/486702/ [20:42:06] though all those reuqirements.txt changes are cumbersome and having a frozen set of modules would be easier to handle ;) [20:43:18] E: zuul changes: bad-distribution-in-changes-file jessie+wikimedia [20:43:21] bah lintian ... :/ [20:43:30] yeh, it failed on your merged change too [20:44:57] I swear I have fixed that at some point [20:45:03] ah yeah https://phabricator.wikimedia.org/T186587 ;) [20:46:40] (03PS2) 10Krinkle: zuul: Enable fresnel job in a new 'patch-performance' pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/491577 (https://phabricator.wikimedia.org/T133646) [20:46:52] (03PS3) 10Krinkle: zuul: Enable fresnel job in a new 'patch-performance' pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/491577 (https://phabricator.wikimedia.org/T133646) [20:48:33] hashar though we will need to modify the scap patches (if your happy with how it looks) for prod. [20:48:42] Seeing as i did the config so i could test in labs :) [20:48:59] just scap/scap.cfg && the file containing the hosts. [20:49:07] (03PS4) 10Krinkle: zuul: Enable fresnel job in a new 'patch-performance' pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/491577 (https://phabricator.wikimedia.org/T133646) [20:49:35] hashar i also created integration/zuul/wheels && integration/zuul/build [20:50:57] (03PS5) 10Krinkle: zuul: Enable fresnel job in a new 'patch-performance' pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/491577 (https://phabricator.wikimedia.org/T133646) [20:52:02] (03PS6) 10Krinkle: zuul: Enable fresnel job in a new 'patch-performance' pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/491577 (https://phabricator.wikimedia.org/T133646) [20:52:07] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10puppet-compiler: compiler1002.puppet-diffs.eqiad.wmflabs instance is down - https://phabricator.wikimedia.org/T216513 (10herron) Compiler1002 is back online and successfully ran through a few local test catalog compiles. `popula... [20:52:11] (03CR) 10Krinkle: [C: 03+2] zuul: Enable fresnel job in a new 'patch-performance' pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/491577 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [20:53:21] (03CR) 10jerkins-bot: [V: 04-1] zuul: Enable fresnel job in a new 'patch-performance' pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/491577 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [20:53:31] (03CR) 10jerkins-bot: [V: 04-1] zuul: Enable fresnel job in a new 'patch-performance' pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/491577 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [21:06:02] gyp ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable. [21:06:06] (03PS7) 10Krinkle: zuul: Enable fresnel job in a new 'patch-performance' pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/491577 (https://phabricator.wikimedia.org/T133646) [21:06:07] * hauskatze don't understand [21:06:29] 10Beta-Cluster-Infrastructure, 10User-Ryasmeen: Beta cluster is broken: No working replica DB server: Unknown error (172.16.5.5:3306)) - https://phabricator.wikimedia.org/T216045 (10Ryasmeen) Hey @dcausse/@jcrespo, it seems Beta cluster is still broken.I see it's in read-only mode and also sometimes getting t... [21:07:33] (03CR) 10jerkins-bot: [V: 04-1] zuul: Enable fresnel job in a new 'patch-performance' pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/491577 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [21:15:47] > MultipleInvalid: expected str for dictionary value @ data['pipelines'][4]['failure-message'] [21:15:51] anyone know what's that about? [21:16:00] I don't see how my entry differs from the others.. [21:16:41] (03PS8) 10Krinkle: zuul: Enable fresnel job in a new 'patch-performance' pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/491577 (https://phabricator.wikimedia.org/T133646) [21:19:52] (03CR) 10Krinkle: [C: 03+2] "I guess it hates emojs, how boring." [integration/config] - 10https://gerrit.wikimedia.org/r/491577 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [21:20:12] Project beta-update-databases-eqiad build #32080: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32080/ [21:21:21] (03Merged) 10jenkins-bot: zuul: Enable fresnel job in a new 'patch-performance' pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/491577 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [21:23:34] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/491577, also deploys https://gerrit.wikimedia.org/r/490492 / T198495 [21:23:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:23:38] T198495: Fix or delete apps-android-wikipedia-periodic-test from CI - https://phabricator.wikimedia.org/T198495 [21:25:27] (03PS1) 10Hashar: debian-glue: avoid mangling $distribution and $DIST [integration/config] - 10https://gerrit.wikimedia.org/r/491639 [21:25:52] (03CR) 10Hashar: "Example issue https://gerrit.wikimedia.org/r/#/c/integration/zuul/+/486702/" [integration/config] - 10https://gerrit.wikimedia.org/r/491639 (owner: 10Hashar) [21:26:07] (03CR) 10Hashar: "The lintian failure should be fixed by https://gerrit.wikimedia.org/r/#/c/integration/config/+/491639/" [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/486702 (https://phabricator.wikimedia.org/T214739) (owner: 10Paladox) [21:26:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10puppet-compiler: compiler1002.puppet-diffs.eqiad.wmflabs instance is down - https://phabricator.wikimedia.org/T216513 (10hashar) Wonderful, thank you :-] [21:26:59] hashar thanks! [21:30:31] hi hashar : https://gerrit.wikimedia.org/r/#/c/integration/config/+/490606/ the extension is already archived, can this be approved? tnx [21:34:32] (03PS1) 10Paladox: Remove /p/ from /info/refs call [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/491641 [21:39:45] (03PS2) 10Paladox: Remove /p/ from /info/refs call [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/491641 [21:39:59] (03PS3) 10Paladox: Remove /p/ from /info/refs call [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/491641 (https://phabricator.wikimedia.org/T214807) [21:40:09] hashar ^^ [21:40:11] :) [21:41:19] (03CR) 10jerkins-bot: [V: 04-1] Remove /p/ from /info/refs call [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/491641 (https://phabricator.wikimedia.org/T214807) (owner: 10Paladox) [21:41:47] (03CR) 10Paladox: [C: 03+1] debian-glue: avoid mangling $distribution and $DIST [integration/config] - 10https://gerrit.wikimedia.org/r/491639 (owner: 10Hashar) [21:49:55] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10puppet-compiler: compiler1002.puppet-diffs.eqiad.wmflabs instance is down - https://phabricator.wikimedia.org/T216513 (10herron) That finished a bit faster than I was expecting! Ready to re-enable in the morning. And FWIW here'... [22:09:19] 10Release-Engineering-Team (Kanban), 10MediaWiki-Authentication-and-authorization, 10MediaWiki-Core-Testing, 10MW-1.33-notes (1.33.0-wmf.17; 2019-02-12), and 4 others: wdio browser tests fail locally due to session not being persisted before 2nd stage of login star... - https://phabricator.wikimedia.org/T214471 [22:20:02] Project beta-update-databases-eqiad build #32081: 04STILL FAILING in 1.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32081/ [22:30:33] 10Release-Engineering-Team (Watching / External), 10Operations, 10Scoring-platform-team: Contact number of some WMDE staff should be avalible to SRE/RelEng - https://phabricator.wikimedia.org/T210721 (10Halfak) p:05High→03Triage [23:03:10] 10Continuous-Integration-Config, 10Fresnel, 10Performance-Team: Add "Console section" pattern to Jenkins to make Fresnel results easy to find - https://phabricator.wikimedia.org/T216572 (10Krinkle) [23:08:48] 10Beta-Cluster-Infrastructure: de.wikipedia.beta.wmflabs.org is currently in readonly mode - https://phabricator.wikimedia.org/T216573 (10Morten_Haan) p:05Triage→03Unbreak! [23:08:50] 10Beta-Cluster-Infrastructure: de.wikipedia.beta.wmflabs.org is currently in readonly mode - https://phabricator.wikimedia.org/T216573 (10Luke081515) Exact message: Warning: The database has been locked for maintenance, so you will not be able to save your edits right now. You may wish to copy and paste your tex... [23:09:17] 10Continuous-Integration-Config, 10Fresnel, 10Performance-Team: For the Fresnel job, distinguish system failure from assert failure - https://phabricator.wikimedia.org/T216574 (10Krinkle) [23:13:00] 10Beta-Cluster-Infrastructure: de.wikipedia.beta.wmflabs.org is currently in readonly mode - https://phabricator.wikimedia.org/T216573 (10Paladox) This is because the DB was corrupted during last week cloud outage. See T216404 [23:14:16] 10Beta-Cluster-Infrastructure: de.wikipedia.beta.wmflabs.org is currently in readonly mode - https://phabricator.wikimedia.org/T216573 (10greg) [23:14:29] 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 10Wikidata, 10User-Addshore, 10cloud-services-team (Kanban): deployment-db03.deployment-prep.eqiad.wmflabs instance can not start - https://phabricator.wikimedia.org/T216404 (10greg) [23:20:01] Project beta-update-databases-eqiad build #32082: 04STILL FAILING in 1.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32082/ [23:25:00] 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 10Wikidata, 10User-Addshore, 10cloud-services-team (Kanban): deployment-db03.deployment-prep.eqiad.wmflabs instance can not start - https://phabricator.wikimedia.org/T216404 (10greg) [23:25:03] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Puppet failures on deployment-deploy01.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T216164 (10greg) [23:25:05] 10Beta-Cluster-Infrastructure, 10Code-Stewardship-Reviews: deployment-prep: Code stewardship request - https://phabricator.wikimedia.org/T215217 (10greg) [23:25:22] 10Beta-Cluster-Infrastructure, 10Code-Stewardship-Reviews: deployment-prep: Code stewardship request - https://phabricator.wikimedia.org/T215217 (10greg) p:05Triage→03Normal [23:32:46] 10Beta-Cluster-Infrastructure: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10Krenair) It doesn't appear to work the same way: ` krenair@deployment-db05:~$ sudo mariabackup --apply-log --use-memory=12G /srv/sqldata Info: Using unique option prefix 'apply-...