[01:34:15] 10Gerrit, 10Browser-Support-Microsoft-Edge, 10Regression: Microsoft Edge does not display review details - https://phabricator.wikimedia.org/T257186 (10Dzahn) ` HTTP404: NICHT GEFUNDEN: Der Server hat keine Übereinstimmungen für den angeforderten URI (Uniform Resource Identifier) gefunden. GET - https://gerr... [01:38:51] 10Gerrit, 10Browser-Support-Microsoft-Edge, 10Regression: Microsoft Edge does not display review details - https://phabricator.wikimedia.org/T257186 (10Dzahn) a:03Paladox Boldly assigning to Paladox since I know he is an Edge user and involved with upstream ticket regarding font loading. Yes, https://gerr... [02:04:07] 10Gerrit, 10Browser-Support-Microsoft-Edge, 10Regression: Microsoft Edge does not display review details - https://phabricator.wikimedia.org/T257186 (10Paladox) Oh I don’t use Microsoft edge and haven’t done so in a long while. [02:14:05] 10Gerrit: Make Gerrit test instance at https://gerrit.git.wmflabs.org work again or update docs - https://phabricator.wikimedia.org/T256563 (10Dzahn) @Zoranzoki21 I would really recommend requesting access to the existing devtools project if you want to test something. It is MUCH easier than repeating all the st... [03:52:44] * DannyS712 is manually deleting a row from enwiki betacluster `pagetriage_page` table [03:52:54] T234577 [03:52:55] T234577: Delete a row from pagetriage_page on beta cluster - https://phabricator.wikimedia.org/T234577 [03:56:09] !log beta cluster enwiki: ran `DELETE FROM pagetriage_page WHERE ptrp_page_id = 194182 LIMIT 1;` (T234577) [03:56:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:57:34] * DannyS712 is done [06:49:50] 10Continuous-Integration-Config, 10Gerrit, 10Release-Engineering-Team-TODO, 10Developer Productivity, 10Patch-For-Review: Jenkins comments no longer formatted on Gerrit 3 - https://phabricator.wikimedia.org/T256575 (10QChris) 05Open→03Resolved a:03QChris After some hacking on the plugin mentioned b... [06:49:53] 10Gerrit, 10Patch-For-Review: Gerrit 3.2 upgrade - https://phabricator.wikimedia.org/T254158 (10QChris) [06:50:25] 10Gerrit, 10Upstream: gerrit: submitting a topic requires push option (.../master%topic=my-topic), appending to branch (.../master/my-topic) is no longer supported in Gerrit 3.2 - https://phabricator.wikimedia.org/T256653 (10QChris) [06:53:57] 10Gerrit, 10DBA, 10Patch-For-Review: Make sure both `reviewdb-test` (used forgerrit upgrade testing) and `reviewdb` (formerly production) databases get torn down - https://phabricator.wikimedia.org/T255715 (10QChris) >>! In T255715#6275884, @QChris wrote: > [...] then I think we won't need the actual databa... [06:54:00] PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%) [07:02:01] 10Continuous-Integration-Config, 10Gerrit, 10Release-Engineering-Team-TODO, 10Developer Productivity, 10Patch-For-Review: Jenkins comments no longer formatted on Gerrit 3 - https://phabricator.wikimedia.org/T256575 (10Legoktm) This is absolutely fantastic and so much better than what we had previously \o... [07:04:00] RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found) [08:19:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Jenkins: Upgrade Jenkins instances to 2.235.1 - https://phabricator.wikimedia.org/T256978 (10hashar) [08:19:23] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Jenkins: Upgrade Jenkins instances to 2.235.1 - https://phabricator.wikimedia.org/T256978 (10hashar) 05Open→03Re... [08:21:09] 10Gerrit, 10DBA, 10Patch-For-Review: Make sure both `reviewdb-test` (used forgerrit upgrade testing) and `reviewdb` (formerly production) databases get torn down - https://phabricator.wikimedia.org/T255715 (10Marostegui) I would suggest we start by renaming the tables first, to make sure nothing really break... [08:21:49] 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO (201908), 10Graphoid, 10Operations, and 2 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10Legoktm) For reference, all of the current discussion seems to be taking place in {T249419}. [08:28:25] Yippee, build fixed! [08:28:25] Project beta-scap-eqiad build #307728: 09FIXED in 3 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/307728/ [08:30:14] 10Gerrit, 10Upstream: gerrit: submitting a topic requires push option (.../master%topic=my-topic), appending to branch (.../master/my-topic) is no longer supported in Gerrit 3.2 - https://phabricator.wikimedia.org/T256653 (10hashar) 05Open→03Resolved I haven't spotted any other use case beside mediawiki/to... [08:30:18] 10Gerrit, 10Patch-For-Review: Gerrit 3.2 upgrade - https://phabricator.wikimedia.org/T254158 (10hashar) [08:30:31] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Upstream: gerrit: submitting a topic requires push option (.../master%topic=my-topic), appending to branc... - https://phabricator.wikimedia.org/T256653 [09:18:09] 10Beta-Cluster-Infrastructure: Beta cluster has reached its quota - https://phabricator.wikimedia.org/T257118 (10hnowlan) [10:07:32] 10Beta-Cluster-Infrastructure, 10Wikidata: Beta cluster wikidata fails to load scripts with "OutOfBoundsException from line 60 of /srv/mediawiki/php-master/extensions/Wikibase/lib/includes/SettingsArray.php: Attempt to get non-existing setting "repoScriptPath" - https://phabricator.wikimedia.org/T257296 (10Maja... [10:08:02] addshore: you've been messing with wikidata config so any idea what is causing https://phabricator.wikimedia.org/T257296? [10:08:51] hmmmm, that might be due to some other Wikibase development [10:08:54] I’ll take a look later [10:09:24] thanks! [10:25:16] 10Beta-Cluster-Infrastructure, 10Wikidata: Beta cluster wikidata fails to load scripts with "OutOfBoundsException from line 60 of /srv/mediawiki/php-master/extensions/Wikibase/lib/includes/SettingsArray.php: Attempt to get non-existing setting "repoScriptPath" - https://phabricator.wikimedia.org/T257296 (10Dann... [10:27:44] 10Beta-Cluster-Infrastructure, 10Wikidata: Beta cluster wikidata fails to load scripts with "OutOfBoundsException from line 60 of /srv/mediawiki/php-master/extensions/Wikibase/lib/includes/SettingsArray.php: Attempt to get non-existing setting "repoScriptPath" - https://phabricator.wikimedia.org/T257296 (10Maja... [11:06:45] 10Gerrit: Editting long files is very slow on Gerrit's new UI - https://phabricator.wikimedia.org/T257300 (10MarcoAurelio) [11:10:14] 10phan-taint-check-plugin: Create a web demo - https://phabricator.wikimedia.org/T257301 (10Daimona) [12:09:28] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10git-protocol-v2: Update Stretch CI containers to use a git that supports ... - https://phabricator.wikimedia.org/T256844 [12:16:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10git-protocol-v2: Update Stretch CI containers to use a git that supports ... - https://phabricator.wikimedia.org/T256844 [12:20:13] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.40 deployment blockers - https://phabricator.wikimedia.org/T256668 (10Jdforrester-WMF) Two scaps and forty minutes later, testwikis seem initially... [12:23:31] 10Beta-Cluster-Infrastructure, 10Wikidata: Beta cluster wikidata fails to load scripts with "OutOfBoundsException from line 60 of /srv/mediawiki/php-master/extensions/Wikibase/lib/includes/SettingsArray.php: Attempt to get non-existing setting "repoScriptPath" - https://phabricator.wikimedia.org/T257296 (10Jdfo... [12:23:45] 10Beta-Cluster-Infrastructure, 10Wikidata: Beta cluster wikidata fails to load scripts with "OutOfBoundsException from line 60 of /srv/mediawiki/php-master/extensions/Wikibase/lib/includes/SettingsArray.php: Attempt to get non-existing setting "repoScriptPath" - https://phabricator.wikimedia.org/T257296 (10Jdfo... [12:23:48] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.40 deployment blockers - https://phabricator.wikimedia.org/T256668 (10Jdforrester-WMF) [12:24:31] 10Beta-Cluster-Infrastructure, 10Wikidata: Beta cluster wikidata fails to load scripts with "OutOfBoundsException from line 60 of /srv/mediawiki/php-master/extensions/Wikibase/lib/includes/SettingsArray.php: Attempt to get non-existing setting "repoScriptPath" - https://phabricator.wikimedia.org/T257296 (10Jdfo... [12:24:52] 10Beta-Cluster-Infrastructure, 10Wikidata: Beta cluster wikidata fails to load scripts with "OutOfBoundsException from line 60 of /srv/mediawiki/php-master/extensions/Wikibase/lib/includes/SettingsArray.php: Attempt to get non-existing setting "repoScriptPath" - https://phabricator.wikimedia.org/T257296 (10Jdfo... [12:26:44] 10Beta-Cluster-Infrastructure, 10Wikidata: Beta cluster wikidata fails to load scripts with "OutOfBoundsException from line 60 of /srv/mediawiki/php-master/extensions/Wikibase/lib/includes/SettingsArray.php: Attempt to get non-existing setting "repoScriptPath" - https://phabricator.wikimedia.org/T257296 (10Jdfo... [12:27:28] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.40 deployment blockers - https://phabricator.wikimedia.org/T256668 (10Jdforrester-WMF) Naturally, spoke too soon. :-( [12:41:54] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10ChangeProp, 10Operations, and 7 others: Migrate cpjobqueue to kubernetes - https://phabricator.wikimedia.org/T220399 (10hnowlan) 05Open→03Resolved [12:41:57] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Operations, 10Release Pipeline, and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10hnowlan) [12:51:33] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Scap, 10Sustainability (Incident Prevention): Make scap skip restarting php-fpm when using --force - https://phabricator.wikimedia.org/T243009 (10jijiki) @thcipriani @LarsWirzenius would it be too... [13:27:34] 10Phabricator, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Quality-and-Test-Engineering-Team (QTE), 10User-Vidhi-Mody, 10User-zeljkofilipin: Error in pushing code to Gerrit for the phab-deployment repository - https://phabricator.wikimedia.org/T255761 (10zeljkofilipin) [13:47:13] 10Release-Engineering-Team (CI & Testing services), 10User-Demian: Evaluate the workflows and workload of managing a Node package repository - https://phabricator.wikimedia.org/T257061 (10Demian) [13:47:57] 10Release-Engineering-Team (Development services), 10TechCom-RFC, 10Developer Productivity, 10Proposal, 10User-Demian: Draft: RFC: Evaluate alternative Node package managers for improved package security - https://phabricator.wikimedia.org/T257068 (10Demian) [13:48:24] 10Release-Engineering-Team (Development services), 10TechCom-RFC, 10Developer Productivity, 10Proposal, 10User-Demian: Draft: RFC: Evaluate alternative Node package managers for improved package security - https://phabricator.wikimedia.org/T257068 (10Demian) [13:54:36] 10Release-Engineering-Team, 10Operations, 10Scoring-platform-team (Current): Production shell access for Chris Albon - https://phabricator.wikimedia.org/T256412 (10ssingh) a:05ssingh→03None The only remaining item on this task is the "cloud (labs) groups: deployment-prep", that is better suited for the r... [13:59:28] 10Beta-Cluster-Infrastructure: Beta cluster has reached its quota - https://phabricator.wikimedia.org/T257118 (10Mholloway) [14:01:32] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10Technical-Debt: Move integration/docroot.git deployment to scap - https://phabricator.wikimedia.org/T256005 (10hashar) And of cours... [14:19:15] 10Beta-Cluster-Infrastructure: Beta cluster has reached its quota - https://phabricator.wikimedia.org/T257118 (10Mholloway) [14:19:21] 10Deployments: scap::target failed to clone a new deployment repository - https://phabricator.wikimedia.org/T257316 (10hashar) [14:20:51] 10Deployments: scap deploy --init on deployment server fails on first puppet run - https://phabricator.wikimedia.org/T257317 (10hashar) [14:21:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10Technical-Debt: Move integration/docroot.git deployment to scap - https://phabricator.wikimedia.org/T256005 (10hashar) [14:21:28] 10Scap: scap deploy --init on deployment server fails on first puppet run - https://phabricator.wikimedia.org/T257317 (10hashar) [14:21:31] 10Deployments: scap::target failed to clone a new deployment repository - https://phabricator.wikimedia.org/T257316 (10hashar) [14:21:33] 10Scap: scap deploy --init on deployment server fails on first puppet run - https://phabricator.wikimedia.org/T257317 (10greg) [14:21:35] 10Scap: scap::target failed to clone a new deployment repository - https://phabricator.wikimedia.org/T257316 (10greg) [14:25:11] 10Scap: scap deploy --init on deployment server fails on first puppet run - https://phabricator.wikimedia.org/T257317 (10hashar) One of the issue is that we only know that git exited with return code 128 and loose the useful stderr output. It seems to be swallowed by #scap. [14:31:37] 10Scap: scap deploy --init fails on the sapre deployment server due to a global scap lock being held - https://phabricator.wikimedia.org/T257319 (10hashar) [14:59:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10Technical-Debt: Move integration/docroot.git deployment to scap - https://phabricator.wikimedia.org/T256005 (10hashar) After much t... [15:06:34] 10Release-Engineering-Team (CI & Testing services), 10User-Demian: Determine Node package auditing workflows - https://phabricator.wikimedia.org/T257072 (10Demian) Initial thoughts republished from wikitech-l: 100% review wouldn't be sustainable IMO (causing developer burnout very quickly), but looking for sp... [15:08:41] PROBLEM - Host deployment-maps05 is DOWN: CRITICAL - Host Unreachable (172.16.4.168) [15:27:13] 10Beta-Cluster-Infrastructure: Beta cluster has reached its quota - https://phabricator.wikimedia.org/T257118 (10Mholloway) [15:28:44] PROBLEM - Host deployment-chromium02 is DOWN: CRITICAL - Host Unreachable (172.16.4.14) [15:53:13] 10Scap: scap deploy --init fails on the sapre deployment server due to a global scap lock being held - https://phabricator.wikimedia.org/T257319 (10hashar) [15:53:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10Technical-Debt: Move integration/docroot.git deployment to scap - https://phabricator.wikimedia.org/T256005 (10hashar) [15:53:24] 10Scap: scap deploy --init fails on the spare deployment server due to a global scap lock being held - https://phabricator.wikimedia.org/T257319 (10hashar) [15:55:30] 10Scap: scap deploy --init fails on the spare deployment server due to a global scap lock being held - https://phabricator.wikimedia.org/T257319 (10hashar) 05Open→03Resolved a:03hashar The primary server was not fully provisioned and no deployment occurred yet. The suspicion is that Puppet tried to init th... [15:55:33] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10Technical-Debt: Move integration/docroot.git deployment to scap - https://phabricator.wikimedia.org/T256005 (10hashar) [15:55:54] 10Release-Engineering-Team, 10Prod-Kubernetes, 10serviceops, 10Kubernetes: CI pipeline/job to build and release helm chart artifacts - https://phabricator.wikimedia.org/T257333 (10JMeybohm) [15:56:41] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments: 1.35.0-wmf.41 deployment blockers - https://phabricator.wikimedia.org/T256669 (10Lucas_Werkmeister_WMDE) [15:57:02] 10Scap: scap::target failed to clone a new deployment repository - https://phabricator.wikimedia.org/T257316 (10hashar) 05Open→03Resolved a:03hashar The primary server was not fully provisioned and no deployment occurred yet. The repository was not initialized on the primary server and thus the target coul... [15:57:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10Technical-Debt: Move integration/docroot.git deployment to scap - https://phabricator.wikimedia.org/T256005 (10hashar) [16:34:43] 10Phabricator, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Quality-and-Test-Engineering-Team (QTE), 10User-Vidhi-Mody, 10User-zeljkofilipin: Error in pushing code to Gerrit for the phab-deployment repository - https://phabricator.wikimedia.org/T255761 (10mmodell) a:03mmodell [16:38:25] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Scap, 10User-brennen: Build a development / test environment for deployment tooling - https://phabricator.wikimedia.org/T250322 (10mmodell) [16:38:28] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Scap, 10Developer Productivity, and 2 others: Create a development & testing environment for scap + train deployment... - https://phabricator.wikimedia.org/T255695 [16:48:07] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.40 deployment blockers - https://phabricator.wikimedia.org/T256668 (10Majavah) [16:48:36] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments: 1.35.0-wmf.41 deployment blockers - https://phabricator.wikimedia.org/T256669 (10Majavah) [17:10:38] (03PS3) 10Hashar: Add filter_scripts volume mount to logstash-filter-verifier job [integration/config] - 10https://gerrit.wikimedia.org/r/602730 (https://phabricator.wikimedia.org/T222826) (owner: 10Cwhite) [17:20:06] (03CR) 10Hashar: [C: 03+2] "I had missed Cole reply made immediately after my review, I apologize things some time disappear. We clarified over IRC whether we could " [integration/config] - 10https://gerrit.wikimedia.org/r/602730 (https://phabricator.wikimedia.org/T222826) (owner: 10Cwhite) [17:21:24] (03Merged) 10jenkins-bot: Add filter_scripts volume mount to logstash-filter-verifier job [integration/config] - 10https://gerrit.wikimedia.org/r/602730 (https://phabricator.wikimedia.org/T222826) (owner: 10Cwhite) [17:26:25] 10phan-taint-check-plugin: Foreach loops hide the "outer" source of taintedness in caused-by lines - https://phabricator.wikimedia.org/T257348 (10Daimona) [17:26:48] 10phan-taint-check-plugin, 10Patch-For-Review: Foreach loops hide the "outer" source of taintedness in caused-by lines - https://phabricator.wikimedia.org/T257348 (10Daimona) [17:26:50] 10phan-taint-check-plugin, 10Patch-For-Review: Fix caused-by lines in phan-taint-check - https://phabricator.wikimedia.org/T203652 (10Daimona) [17:29:25] (03PS1) 10Hashar: jjb: volume mount for logstash must be absolute path [integration/config] - 10https://gerrit.wikimedia.org/r/610119 (https://phabricator.wikimedia.org/T222826) [17:30:45] (03CR) 10Hashar: [C: 03+2] "Deployed and confirmed to work hurrah" [integration/config] - 10https://gerrit.wikimedia.org/r/610119 (https://phabricator.wikimedia.org/T222826) (owner: 10Hashar) [17:32:06] (03Merged) 10jenkins-bot: jjb: volume mount for logstash must be absolute path [integration/config] - 10https://gerrit.wikimedia.org/r/610119 (https://phabricator.wikimedia.org/T222826) (owner: 10Hashar) [17:32:24] 10phan-taint-check-plugin: Make caused-by lines longer - https://phabricator.wikimedia.org/T257350 (10Daimona) [17:33:23] 10phan-taint-check-plugin, 10Patch-For-Review: Make caused-by lines longer - https://phabricator.wikimedia.org/T257350 (10Daimona) [17:33:26] 10phan-taint-check-plugin, 10Patch-For-Review: Fix caused-by lines in phan-taint-check - https://phabricator.wikimedia.org/T203652 (10Daimona) [17:36:11] 10phan-taint-check-plugin: Caused-by lines include lines for unrelated types of taintedness - https://phabricator.wikimedia.org/T257351 (10Daimona) [17:37:14] 10phan-taint-check-plugin, 10Patch-For-Review: Fix caused-by lines in phan-taint-check - https://phabricator.wikimedia.org/T203652 (10Daimona) [17:37:16] 10phan-taint-check-plugin, 10Patch-For-Review: Caused-by lines apparently picking up everything that is concatenated - https://phabricator.wikimedia.org/T250679 (10Daimona) [17:37:25] 10phan-taint-check-plugin, 10Patch-For-Review: Caused-by lines include lines for unrelated types of taintedness - https://phabricator.wikimedia.org/T257351 (10Daimona) [17:37:27] 10phan-taint-check-plugin, 10Patch-For-Review: Fix caused-by lines in phan-taint-check - https://phabricator.wikimedia.org/T203652 (10Daimona) [18:09:30] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release Pipeline: Make pipelinelib able to update deployment chart image tags - https://phabricator.wikimedia.org/T255835 (10jeena) [18:10:00] mutante: Let's continue over here. [18:10:40] ok, hi qchris and paladox [18:10:44] hi [18:10:50] so there were 2 comments on the first patch, that made it -1 [18:10:51] Hi :-) [18:11:02] the first one was the nitpick [18:11:06] i believe that qchris addressed his nitpick [18:11:15] and that you've answered the question he had [18:11:38] regarding the second comment .. i checked the private repo [18:11:47] 10Release-Engineering-Team, 10Prod-Kubernetes, 10serviceops, 10Kubernetes: CI pipeline/job to build and release helm chart artifacts - https://phabricator.wikimedia.org/T257333 (10jeena) Hi, we also thought it would be helpful for developers to not have to create a patchset to deployment-charts every time... [18:11:55] at first i expected we want ssh_host_key and ssh_host_rsa_key to be identical [18:12:01] The files differ, but do they contain the same RSA values? [18:12:03] but they are in different formats [18:12:16] ssh_host_key: Java serialization data, version 5 [18:12:16] ssh_host_rsa_key: OpenSSH private key [18:12:29] That difference is expected. [18:13:00] The question though is: How did ssh_host_rsa_key get generated? [18:13:03] well, let's see how we compare the actual values? [18:13:21] * mutante goes back to the old ticket [18:14:05] ssh-keygen -q -t rsa -N "" -C "gerrit-code-review@gerrit1001.wikimedia.org" -f /var/lib/gerrit2/review_site/etc/ssh_host_rsa_key [18:14:13] it seems [18:14:29] Ok. That looks like it contains a new RSA key. [18:14:50] So every user will get prompted about mismatching keys, if we use that file :-/ [18:14:51] yea, we just made a new one [18:15:21] so we should instead just convert the existing key to new format? [18:15:45] so "how to convert from Java to OpenSSH private key" [18:15:48] That would be best. And ideally someone knowledgable has a command handy for that. [18:15:54] well they would be getting it because of the new ed25519 key. [18:16:12] yea, i was thinking that too. there are also the other new keys [18:16:32] paladox: I tested that on Debian buster. And there, ssh kept using the rsa key, because ssh knew that already. [18:16:37] ticket says to do it "while at it" [18:16:41] oh [18:16:51] not always a fan of "while at it" but it depends [18:16:58] Not sure other ssh implementations (or different versions) will behave alike. [18:17:10] That's why I mentioned announcing the key update on mailing lists. [18:17:21] -O lets you convert formats [18:17:34] * qchris bows to Platonides. [18:17:35] hmm, not really [18:17:43] :-D [18:17:46] * Platonides actually checks the man page [18:18:04] so you have a rsa key and wants it in openssh format? [18:18:25] -m key_format [18:18:25] Specify a key format for the -i (import) or -e (export) conver‐ [18:18:28] we have "Java serialization data, version 5" binary [18:18:31] sion options. [18:18:38] 10Gerrit: Editing long files is very slow on Gerrit's new web UI - https://phabricator.wikimedia.org/T257300 (10Aklapper) [18:18:38] and we want "OpenSSH private key" ascii [18:19:06] try ssh-keygen -i -f [18:19:39] if that fails, it will need an explicit -m [18:20:23] I just got news that we actually want a new key. [18:20:27] So ignore my comments. [18:20:36] But I think we should announce it on the mailing lists then. [18:20:38] Platonides: yea, thanks a lot but we need a new one [18:21:00] use a new one, then :P [18:21:19] ¯\_(ツ)_/¯ [18:23:47] Thinking about it ... [18:24:01] As we want a new key ... this change will be disruptive for users. [18:24:11] yup [18:24:29] We should give users a few days warning. [18:24:31] don't we? [18:25:02] it would probably avoid a couple pings at least [18:25:05] yup [18:25:12] Doesn't mean they'll read it though :P [18:25:20] but you've at least something to point them at [18:25:31] what are you going to break this time? [18:25:39] EVERYTHING [18:25:42] * Majavah is too lazy to read scrollback [18:25:48] If you're not breaking stuff, are you even changing anything? [18:26:16] heh [18:26:36] yea, let's announce it before we do it instead of after we do it :p [18:26:37] I have a hammer ... in case something should survive without breaking. [18:26:58] so we can now instead continue with further cleanup changes [18:27:03] Cool. [18:27:12] That's less controversial I guess :) [18:27:18] and then we have to schedule something on deployment calendar, i'd say [18:27:22] * qchris puts away his hammer again. Meh. [18:27:34] maybe the "remove firewall hole from db servers" should wait too :p [18:27:44] but maybe not.. looking at the low hanging fruit first [18:29:33] starting with .... https://gerrit.wikimedia.org/r/c/operations/puppet/+/609883 ? [18:30:04] LGTM :-) [18:30:16] * mutante randomly switches to -operations, kind of, because now it's about merges :p [18:30:27] Yup. Let's move over. [19:20:09] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.40 deployment blockers - https://phabricator.wikimedia.org/T256668 (10mmodell) No sign of deprecation warnings for the revision class. I only see d... [19:29:15] 10Gerrit, 10Operations, 10vm-requests, 10Patch-For-Review: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `gerrit1002.wikimedia.org` - gerrit1002.wikimedia.org (**PASS**) - Downtim... [19:43:45] 10Gerrit, 10Patch-For-Review: Gerrit 3.2 upgrade - https://phabricator.wikimedia.org/T254158 (10Dzahn) [19:43:47] 10Gerrit, 10Operations, 10vm-requests, 10Patch-For-Review: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 (10Dzahn) 05Open→03Resolved The VM has been removed from all places in the repos, puppet and DNS. The decom cookbook destroyed it and removed it from monitoring, pup... [19:43:49] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Developer Productivity, and 2 others: Upgrade to Gerrit 2.16.13 - https://phabricator.wikimedia.org/T200739 (10Dzahn) [20:42:24] 10Gerrit: Make Gerrit test instance at https://gerrit.git.wmflabs.org work again or update docs - https://phabricator.wikimedia.org/T256563 (10Dzahn) 05Open→03Resolved https://gerrit-test.wikimedia.org has been removed today. It was always meant to be a temp setup for this specific major upgrade. A new test... [20:50:32] 10Release-Engineering-Team (Code Health), 10Core Platform Team, 10MediaWiki-extensions-ShortUrl, 10Code-Stewardship-Reviews: Code Stewardship Review: ShortUrl Extension - https://phabricator.wikimedia.org/T187045 (10eprodromou) OK, I'll take a look at this. I'm all for reducing complexity in production. [20:54:18] 10MediaWiki-Releasing, 10MW-1.35-notes, 10MW-1.35-release: Write and send release announcements for MediaWiki 1.35.0 - https://phabricator.wikimedia.org/T256380 (10eprodromou) [20:57:27] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release Pipeline: Make pipelinelib able to update deployment chart image tags - https://phabricator.wikimedia.org/T255835 (10JMeybohm) That sounds nice! I would suggest to... [20:57:58] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:08] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release Pipeline: Make pipelinelib able to update deployment chart image tags - https://phabricator.wikimedia.org/T255835 (10jeena) >>! In T255835#6287321, @JMeybohm wrote... [21:05:12] 10Gerrit, 10Browser-Support-Microsoft-Edge, 10Regression: Microsoft Edge does not display review details - https://phabricator.wikimedia.org/T257186 (10QChris) >>! In T257186#6284109, @Dzahn wrote: > Yes, https://gerrit.wikimedia.org/fonts/robotomono/RobotoMono-Regular.ttf is 404 but we also don't expect /f... [21:06:51] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release Pipeline: Make pipelinelib able to update deployment chart image tags - https://phabricator.wikimedia.org/T255835 (10jeena) [21:07:06] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release Pipeline: Make pipelinelib able to update deployment chart image tags - https://phabricator.wikimedia.org/T255835 (10jeena) [21:12:50] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 97328 bytes in 2.027 second response time [21:15:02] 10Phabricator, 10Editing-Team-Request, 10Editing-team (Tracking): Custom task form for #Editing-team-request - https://phabricator.wikimedia.org/T256123 (10JTannerWMF) Hey @Aklapper perhaps would it be easier to hop on a call to share what we are hoping to do? [21:44:51] 10Gerrit: Make Gerrit test instance at https://gerrit.git.wmflabs.org work again or update docs - https://phabricator.wikimedia.org/T256563 (10Zoranzoki21) Thank you @Dzahn! [21:45:08] Majavah: https://developer.github.com/changes/2020-04-15-replacing-the-installation-and-installation-repositories-events/ [21:45:52] 10Gerrit: Make Gerrit test instance at https://gerrit.git.wmflabs.org work again or update docs (replaced by https://gerrit.devtools.wmflabs.org/) - https://phabricator.wikimedia.org/T256563 (10Dzahn) [21:47:00] 10Gerrit, 10Browser-Support-Microsoft-Edge, 10Regression: Microsoft Edge does not display review details - https://phabricator.wikimedia.org/T257186 (10Dzahn) a:05Paladox→03None [22:20:49] 10LibUp: Update libup for Gerrit's new ssh host keys - https://phabricator.wikimedia.org/T257382 (10Legoktm) [22:37:05] 10Gerrit, 10Browser-Support-Microsoft-Edge, 10Regression: Microsoft Edge does not display review details - https://phabricator.wikimedia.org/T257186 (10QChris) @Xqt I missed one aspect of your report. You mention "since today" (which was Monday) ... did it work before Monday? [22:38:12] 10LibUp: Update libup for Gerrit's new ssh host keys - https://phabricator.wikimedia.org/T257382 (10Dzahn) [23:26:00] PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%) [23:36:01] RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)