[00:09:22] <wikibugs>	 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T263182 (10Krinkle)
[00:10:06] <wikibugs>	 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T263182 (10Krinkle)
[05:23:54] <wikibugs>	 10LibUp, 10User-DannyS712: LibUp should remove empty `rules` from eslint - https://phabricator.wikimedia.org/T267178 (10DannyS712)
[05:45:07] <wikibugs>	 10LibUp, 10Gerrit-Privilege-Requests: LibUp cannot vote +1 on VisualEditor/VisualEditor and unicodejs - https://phabricator.wikimedia.org/T257731 (10DannyS712) @Legoktm some more repos where the bot didn't vote V+1 Extensions: NavigationTiming - https://gerrit.wikimedia.org/r/c/mediawiki/extensions/NavigationT...
[09:14:45] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)): CI job mw-tools-codesniffer-mwcore-testrun fails on clean run - https://phabricator.wikimedia.org/T267150 (10hashar) a:03hashar The job is configured to arch...
[09:14:57] <wikibugs>	 (03PS1) 10Hashar: jjb: allow empty log for phpcs testrun [integration/config] - 10https://gerrit.wikimedia.org/r/639003 (https://phabricator.wikimedia.org/T267150)
[09:16:01] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] jjb: allow empty log for phpcs testrun [integration/config] - 10https://gerrit.wikimedia.org/r/639003 (https://phabricator.wikimedia.org/T267150) (owner: 10Hashar)
[09:16:50] <wikibugs>	 (03PS2) 10Hashar: jjb: allow empty log for phpcs testrun [integration/config] - 10https://gerrit.wikimedia.org/r/639003 (https://phabricator.wikimedia.org/T267150)
[09:17:09] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] jjb: allow empty log for phpcs testrun [integration/config] - 10https://gerrit.wikimedia.org/r/639003 (https://phabricator.wikimedia.org/T267150) (owner: 10Hashar)
[09:18:29] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: allow empty log for phpcs testrun [integration/config] - 10https://gerrit.wikimedia.org/r/639003 (https://phabricator.wikimedia.org/T267150) (owner: 10Hashar)
[09:27:04] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Patch-For-Review: CI job mw-tools-codesniffer-mwcore-testrun fails on clean run - https://phabricator.wikimedia.org/T267150 (10hashar) I have updated the j...
[10:10:14] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Zuul, and 4 others: Improve scheduling of CI jobs invoked by zuul - https://phabricator.wikimedia.org/T258630 (10hashar) I have raised awareness of...
[13:55:43] <wikibugs>	 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T263182 (10ssastry) https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/638520 should ride the train (has a parsoid bug fix).
[14:22:23] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Operations, 10Traffic, 10HTTPS: contint.wikimedia.org: add TLS termination - https://phabricator.wikimedia.org/T263830 (10Nintendofan885)
[14:39:03] <wikibugs>	 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Sustainability (Incident Followup), 10User-brennen: Avoid unfinished train deploys over holidays, weekends, or other stretches of no-deploy days - https://phabricator.wikimedia.org/T260401 (10Joe) I think...
[14:42:39] <wikibugs>	 (03PS1) 10Hashar: Job to run npm run test:size inside Quibble env [integration/config] - 10https://gerrit.wikimedia.org/r/639166 (https://phabricator.wikimedia.org/T255149)
[14:51:10] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Patch-For-Review: Add a CI Job for running bundlesize tests in Vector - https://phabricator.wikimedia.org/T255149 (10hashar) a:03hashar Looks like I forg...
[14:52:38] <wikibugs>	 (03CR) 10Hashar: "The job should theoretically do what is intended though I haven't tested it. Hence I am just adding it to the experimental pipeline for no" [integration/config] - 10https://gerrit.wikimedia.org/r/639166 (https://phabricator.wikimedia.org/T255149) (owner: 10Hashar)
[16:05:43] <cscott>	 the wmf/1.36.0-wmf.16 branch of mediawiki-core doesn't seem to have a .gitmodules file in it
[16:05:53] <cscott>	 unlike wmf.14 and all previous
[16:06:26] <cscott>	 is that intentional?  trying to figure out if I need to cherry-pick the latest mediawiki-vendor to mediawiki-core to fix T259832
[16:06:27] <stashbot>	 T259832: mediawiki-vendor submodule doesn't get automatically bumped on release branches - https://phabricator.wikimedia.org/T259832
[16:07:12] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Parsoid, 10Patch-For-Review, 10User-brennen: mediawiki-vendor submodule doesn't get automatically bumped on release branches - https://phabricator.wikimedia.org/T259832 (10cscott) 1.36.0-w...
[16:11:05] <cscott>	 brennen, dancy ^
[16:12:09] <brennen>	 cscott: hrm.  i have no idea re: .gitmodules.
[16:12:41] <dpifke>	 !log Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/639197 in beta, should only affect deployment-webperf11.
[16:12:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:14:01] <wikibugs>	 (03PS10) 10Brennen Bearnes: Split php-fpm and apache2 containers [releng/dev-images] - 10https://gerrit.wikimedia.org/r/622575 (https://phabricator.wikimedia.org/T245444)
[16:14:11] <brennen>	 (investigating.)
[16:15:00] <cscott>	 brennen: thanks. seems pretty odd to me.  i wouldn't have noticed if I weren't checking whether our latest mediawiki-vendor branch cherry-pick got included.
[16:15:10] <wikibugs>	 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T263182 (10cscott) The `wmf/1.36.0-wmf.16` branch of mediawiki-core appears to be missing `.gitmodules` at the moment.  When it's crea...
[16:34:24] <brennen>	 cscott: .gitmodules is here: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/638236
[16:34:55] <brennen>	 i just haven't merged that yet.
[16:38:17] <cscott>	 brennen: ah.  i added a gerrit comment about the needed/currently-missing update to the vendor submodule
[16:39:06] <cscott>	 parsoid -a15 fixes a potential wikitext corruption bug on edit, so we're being extra attentive to make sure it gets in wmf.16
[16:39:39] <brennen>	 ack
[16:40:32] <cscott>	 (the wikitext corruption bug we introduced in -a14, unreleased except temporarily to beta, so we're trying not to release -a14 to any more places.)
[16:43:36] <brennen>	 cscott: gah, ok, +2'd that before thinking it through...  i guess a follow-up patch is going to be the easiest thing here
[16:43:56] <cscott>	 yeah, probably.
[16:44:31] <brennen>	 kk.  sorry, i haven't touched the train in a bit and it was a long night.
[16:46:52] <subbu>	 brennen, no worries .. it has been a stressful night indeed and probably more this week. thanks cscott for working with brennen to get all the parsoid pieces in place.
[16:50:13] <wikibugs>	 10LibUp: Libup should mention failing eslint rules when adding profiles like wikimedia/mediawiki - https://phabricator.wikimedia.org/T267236 (10Umherirrender)
[16:51:50] <wikibugs>	 10LibUp: Libup should mention failing eslint rules when adding profiles like wikimedia/mediawiki - https://phabricator.wikimedia.org/T267236 (10Umherirrender) If you need an example: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/TocTree/+/639188   ` 16:41:50 /src/modules/ext.toctree.js 16:41:50    6:0  w...
[16:55:24] <wikibugs>	 10Release-Engineering-Team, 10Cloud-VPS (Project-requests), 10cloud-services-team (Kanban): Request creation of pipelinelib-experimental VPS project - https://phabricator.wikimedia.org/T266174 (10nskaggs) 05Open→03Resolved Project has now been created and all users have been added as admins.
[16:58:18] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Patch-For-Review: Add a CI Job for running bundlesize tests in Vector - https://phabricator.wikimedia.org/T255149 (10Jdlrobson) >  Since all repositories d...
[17:04:46] <wikibugs>	 10Release-Engineering-Team, 10Cloud-VPS (Project-requests), 10cloud-services-team (Kanban): Request creation of pipelinelib-experimental VPS project - https://phabricator.wikimedia.org/T266174 (10dduvall) Thank you, @nskaggs!
[17:05:44] <wikibugs>	 (03CR) 10Ahmon Dancy: [C: 04-1] chore: clean up shell quoting, other things found by linters (033 comments) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/638672 (owner: 10Lars Wirzenius)
[17:06:52] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Patch-For-Review: Add a CI Job for running bundlesize tests in Vector - https://phabricator.wikimedia.org/T255149 (10hashar) That is a good point, I will l...
[17:12:25] <brennen>	 cscott: branch commit's merged into wmf/1.36.0-wmf.16
[17:19:00] <wikibugs>	 (03CR) 10Lars Wirzenius: "Pushing new patch set soon." (033 comments) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/638672 (owner: 10Lars Wirzenius)
[17:42:39] <wikibugs>	 10Release-Engineering-Team (Logspam), 10CommonsMetadata, 10Structured Data Engineering, 10Structured-Data-Backlog, 10User-DannyS712: CommonsMetadata bad wfTimestamp call - https://phabricator.wikimedia.org/T267033 (10Aklapper) @TJH2018: Please avoid posting commonplaces which don't help tracking down the...
[17:43:56] <dpifke>	 !log Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/639216 in beta.
[17:43:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:50:25] <dpifke>	 !log Reverting previous cherry-pick for now; requires minor code change first.
[17:50:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:56:44] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10serviceops, 10Patch-For-Review: Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn)
[18:05:15] <dpifke>	 !log Cherry-picking https://gerrit.wikimedia.org/r/c/analytics/statsv/+/639223 in beta.
[18:05:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[18:18:28] <dpifke>	 deploy_service is not armed in keyholder on deployment-deploy01.  Can someone point me to where the passphrase is stored?
[18:21:47] <bd808>	 dpifke: I don't think there is a passphrase? The keys there at least used to be unencrypted. But it sill needs manual arming
[18:22:34] <dpifke>	 It prompts for a password.  Tried just hitting enter and it's still not armed.
[18:23:16] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Patch-For-Review: Gerrit link to Sonar cloud reports are broken - https://phabricator.wikimedia.org/T267028 (10hashar) p:05Triage→03Medium
[18:23:24] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)): CI job mw-tools-codesniffer-mwcore-testrun fails on clean run - https://phabricator.wikimedia.org/T267150 (10hashar) p:05Triage→03Medium
[18:23:37] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Patch-For-Review: Add a CI Job for running bundlesize tests in Vector - https://phabricator.wikimedia.org/T255149 (10hashar) p:05Triage→03Medium
[18:23:42] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)): Create a runbook for switching CI master - https://phabricator.wikimedia.org/T256396 (10hashar) p:05Triage→03High
[18:24:23] <bd808>	 dpifke: `keyholder status` on deployment-deploy01 shows 3 keys loaded and the proxy as active
[18:25:00] <bd808>	 ah. "/etc/keyholder.d/phabricator is not an acceptable key. Is it an RSA or ED25519 key with passphrase?"
[18:25:55] <dpifke>	 I'm trying to scap deploy something which depends on /etc/keyholder.d/deploy_service (not mwdeploy, dumpsdeploy, or eventlogging), and getting SSH authentication failure.
[18:26:18] <bd808>	 it looks like the keys /etc/keyholder.d/analytics_deploy and /etc/keyholder.d/deploy_service also are wanting passwords
[18:26:22] <dpifke>	 Yeah, saw that, but assumed it wouldn't break the other keys (esp. since some seem to have loaded).
[18:26:49] <bd808>	 sorry all I had was bad info  :/
[18:27:06] <wikibugs>	 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10MediaWiki Train Development Environment: train-dev: Include sample security patches in deploy VM - https://phabricator.wikimedia.org/T260079 (10mmodell) p:05Triage→03Medium
[18:27:18] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Elasticsearch: Set up data storage to collect loosely structured data from CI - https://phabricator.wikimedia.org/T211904 (10mmodell) p:05Medium→...
[18:27:36] <dpifke>	 No worries.  Hopefully someone knows where the passphrase is kept.  Wikitech says it might be somewhere on the puppetmaster.   I looked there but couldn't figure out where.
[18:28:52] <dpifke>	 Oh wait, I didn't read far enough down on that page.
[18:29:02] <bd808>	 I really don't think any ssh keys used by scap in the deployment-prep project should be using pass phrases as a matter of policy, but it has been a very, very long time since I tried to keep things working in that project.
[18:30:27] <dpifke>	 Found passphrase in deployment-puppetmaster04.deployment-prep.eqiad.wmflabs:/var/lib/git/labs/private/files/ssh/tin/servicedeploy_rsa.passphrase
[18:32:41] <wikibugs>	 (03PS5) 10Dduvall: experimental: Support buildkit [blubber] - 10https://gerrit.wikimedia.org/r/504651
[18:42:03] <dpifke>	 scap was still broken, complaining about host key.  Manually editing /etc/ssh/ssh_host_keys to replace "deployment-webperf11.deployment-prep.eqiad1.wikimedia.cloud" with "deployment-webperf11.deployment-prep.eqiad.wmflabs" fixed it.
[18:42:37] <dpifke>	 I assume the correct fix is to tell scap the updated hostname; not sure where it gets it from.
[18:50:31] <dpifke>	 !log Re-cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/639216 in beta.
[18:50:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[18:53:07] <wikibugs>	 (03PS6) 10Dduvall: experimental: Support buildkit [blubber] - 10https://gerrit.wikimedia.org/r/504651
[18:54:20] <bd808>	 dpifke: a reasonable place to look would be the dsh_targets of the scap3 config for the repo you deployed
[18:57:28] <dpifke>	 Looks like it's using a group (/etc/dsh/group/webperf), which in theory comes from puppet, same as /etc/ssh/ssh_known_hosts.  My guess is something is overridden in the manual hiera config which shouldn't be.
[18:57:59] <dpifke>	 Will have to figure it out later; I'm unstuck on the current task for now.
[19:21:27] <wikibugs>	 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T263182 (10brennen)
[19:42:48] <wikibugs>	 (03CR) 10Hashar: [C: 04-1] "Per Jdlrobson feedback on the task, we should run it for mediawiki/core as well." [integration/config] - 10https://gerrit.wikimedia.org/r/639166 (https://phabricator.wikimedia.org/T255149) (owner: 10Hashar)
[19:59:54] <hashar>	 !log operations/software/gerrit : pushed upstream git tag, synced the master branch  # T262241
[19:59:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[19:59:57] <stashbot>	 T262241: Upgrade Gerrit to 3.3 - https://phabricator.wikimedia.org/T262241
[20:04:59] <wikibugs>	 (03PS1) 10Hashar: Review access change [software/gerrit] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/639121
[20:08:07] <wikibugs>	 (03PS2) 10Hashar: Change access to a dedicated group [software/gerrit] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/639121
[20:08:41] <wikibugs>	 (03CR) 10Hashar: "The repository being owned by Administrators / Gerrit Managers does not make much sense so I have created a new group :)" [software/gerrit] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/639121 (owner: 10Hashar)
[20:15:55] <wikibugs>	 (03CR) 10Thcipriani: "> Patch Set 2:" [software/gerrit] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/639121 (owner: 10Hashar)
[20:33:14] <hashar>	 !log devtools: fixed java version on gerrit-prod-1001.devtools.eqiad.wmflabs  by setting java::profile hieradata variable at https://horizon.wikimedia.org/project/instances/48c5ff1c-2885-410d-beeb-d5a57a0a91c7/
[20:33:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[20:36:02] <hashar>	 https://gerrit.devtools.wmflabs.org/r/ is up again (poke paladox )
[20:36:10] <hashar>	 I have no idea how it is deployed though ;)
[20:36:55] <wikibugs>	 (03CR) 10Umherirrender: Report particularly confusing assertNotEquals( false ) (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/576276 (owner: 10Thiemo Kreuz (WMDE))
[20:39:34] <wikibugs>	 (03CR) 10Hashar: "> Patch Set 2:" [software/gerrit] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/639121 (owner: 10Hashar)
[21:09:49] <paladox>	 Hi hashar
[21:09:50] <paladox>	 We use scap, though at the moment I’m just git pulling
[21:18:44] <hashar>	 paladox: ah perfect!
[21:19:00] <hashar>	 I might use that instance to test out a couple pending patches
[21:19:11] <hashar>	 to update javamelody and the other to install the rename plugin
[21:19:25] <hashar>	 I have fixed the gerrit system unit
[21:33:01] <hashar>	 !log devtools upgraded gerrit da40615f726..9275b308ffd
[21:33:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:33:16] <hashar>	 paladox: thank you very much :]  I have upgraded  Gerrit on devtools to catchup
[21:33:47] <hashar>	 I am off now, it is getting late :]
[21:37:26] <paladox>	 :D
[21:40:25] <wikibugs>	 10Phabricator, 10Developer-Advocacy (Oct-Dec 2020): Review the default options in Phab's Favorites menu; offer dedicated forms for bug reports vs feature requests? - https://phabricator.wikimedia.org/T253744 (10Aklapper) I've been thinking of "replacing" the default form (but currently thinking that it's likel...
[22:04:43] <wikibugs>	 10Phabricator, 10Developer-Advocacy (Oct-Dec 2020): Review the default options in Phab's Favorites menu; offer dedicated forms for bug reports vs feature requests? - https://phabricator.wikimedia.org/T253744 (10Aklapper) Rough strawman proposal (might have to rework): * Create a new form "Report Software Bug"....
[22:08:08] <wikibugs>	 10LibUp, 10Gerrit-Privilege-Requests: LibUp cannot vote +1 on VisualEditor/VisualEditor and unicodejs - https://phabricator.wikimedia.org/T257731 (10Umherirrender) This can found also patch sets: https://gerrit.wikimedia.org/r/q/committer:tools.libraryupgrader%2540tools.wmflabs.org+status:merged+-label:Verifie...
[23:04:37] <wikibugs>	 (03CR) 10Jeena Huneidi: [C: 04-1] "Looks good to me. Just one thing: Should we now assert the submoduleOption in PatchsetTest.groovy?" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/635886 (owner: 10Dduvall)
[23:09:04] <wikibugs>	 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T263182 (10brennen) We've hit today's 15:00 PST deployment cutoff.  Train can resume again in US morning.  To summarize current state:...
[23:29:51] <wikibugs>	 10Release-Engineering-Team (Logspam), 10CommonsMetadata, 10Structured Data Engineering, 10Structured-Data-Backlog, 10User-DannyS712: CommonsMetadata bad wfTimestamp call - https://phabricator.wikimedia.org/T267033 (10cscott) I'm probably the guilty party: 916a1777ef8796605aa7eab20c97694aee99e553 added a...