[00:01:16] (03CR) 10Dduvall: [C: 04-1] "I really like what you've done with the different credential binding classes, and I have nothing to add for the implementation in general." (034 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/668245 (https://phabricator.wikimedia.org/T269902) (owner: 10Jeena Huneidi) [00:03:21] James_F: it looks like there's a flaky browser test... :| [00:03:50] 15:24:11 [0-1] Error in "Page should be undoable" [00:03:50] 15:24:11 Node is either not visible or not an HTMLElement [00:11:34] 10Release-Engineering-Team, 10MW-on-K8s, 10serviceops: Containers on releases hosts cannot update apt cache from non-WMF sources - https://phabricator.wikimedia.org/T277109 (10dduvall) Notes from `#mw-on-k8s` for one possible solution. 1. add `apt.proxy` support to blubber 2. add policy for our blubberoid i... [00:11:46] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10MW-on-K8s, 10serviceops: Containers on releases hosts cannot update apt cache from non-WMF sources - https://phabricator.wikimedia.org/T277109 (10dduvall) [01:42:06] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.34 deployment blockers - https://phabricator.wikimedia.org/T274938 (10brennen) End-of-day notes: - Log triage meeting was largely uneventful, we reported a fe... [04:01:51] 10Phabricator: Spaces request for Technical-Program-Management - https://phabricator.wikimedia.org/T277107 (10Aklapper) 05Open→03Stalled Hi, thanks for filing this. Is this a request to create one Space or four Spaces? And/or is this also a request to create some project tags? Note that Spaces are independen... [04:52:44] 10Project-Admins: Create project tag for Slow-DB-Query - https://phabricator.wikimedia.org/T277124 (10tstarling) [05:01:35] 10Project-Admins: Create project tag for Slow-DB-Query - https://phabricator.wikimedia.org/T277124 (10Reedy) Seems reasonable to me. It's basically a subset of #performance_issue; so can we create it as subproject of it? [05:12:07] PROBLEM - Gerrit JSON on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Gerrit%23Monitoring [05:13:13] PROBLEM - Gerrit Health Check on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [05:23:49] RECOVERY - Gerrit JSON on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 21657 bytes in 0.543 second response time https://wikitech.wikimedia.org/wiki/Gerrit%23Monitoring [05:27:25] RECOVERY - Gerrit Health Check on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 905 bytes in 5.905 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [05:31:05] PROBLEM - Gerrit JSON on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Gerrit%23Monitoring [05:33:17] RECOVERY - Gerrit JSON on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 21657 bytes in 0.030 second response time https://wikitech.wikimedia.org/wiki/Gerrit%23Monitoring [05:34:18] 10Release-Engineering-Team (Pipeline), 10Release Pipeline, 10Wikidata, 10Wikidata-Query-Service: Allow timeout override for pipeline k8s deployments to ci cluster - https://phabricator.wikimedia.org/T277125 (10jeena) [05:57:34] 10Release-Engineering-Team (Pipeline), 10Release Pipeline, 10Wikidata, 10Wikidata-Query-Service: Allow timeout override for pipeline k8s deployments to ci cluster - https://phabricator.wikimedia.org/T277125 (10jeena) [06:00:38] (03PS1) 10Jeena Huneidi: Add helm deploy timeout override [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/670635 (https://phabricator.wikimedia.org/T277125) [06:03:38] 10Gerrit, 10Release-Engineering-Team: Gerrit Apache out of workers - https://phabricator.wikimedia.org/T277127 (10thcipriani) [06:09:02] (03PS10) 10Jeena Huneidi: PipelineRunner: allowedCredentials [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/668245 (https://phabricator.wikimedia.org/T269902) [06:09:13] (03CR) 10Jeena Huneidi: PipelineRunner: allowedCredentials (034 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/668245 (https://phabricator.wikimedia.org/T269902) (owner: 10Jeena Huneidi) [06:10:35] (03PS11) 10Jeena Huneidi: PipelineRunner: allowedCredentials [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/668245 (https://phabricator.wikimedia.org/T269902) [06:11:03] (03PS12) 10Jeena Huneidi: PipelineRunner: allowedCredentials [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/668245 (https://phabricator.wikimedia.org/T269902) [06:12:42] 10Project-Admins: Create project tag for Slow-DB-Query - https://phabricator.wikimedia.org/T277124 (10Aklapper) The project desc for #Wikimedia-database-error at https://phabricator.wikimedia.org/project/profile/4182/ mentions `timeouts` (see also T228360 for previous discussion). IMHO scope isolation between th... [06:27:16] 10Project-Admins: Create project tag for Slow-DB-Query - https://phabricator.wikimedia.org/T277124 (10tstarling) Is it necessary for tags to be non-overlapping? A query can be slow without being an error, and there can be errors without slowness. [06:32:50] 10Project-Admins: Create project tag for Slow-DB-Query - https://phabricator.wikimedia.org/T277124 (10Marostegui) I think having a way to tag slow queries is definitely useful. A few years ago we created #mariadb-optimizer-bug which is not exactly for slow queries but for queries where the optimizer goes mad (wh... [07:27:47] 10Phabricator: Herald rule to suppress Task activity? (and just receive Task creation) - https://phabricator.wikimedia.org/T277132 (10valerio.bozzolan) [07:39:12] 10Project-Admins: Create project tag for Slow-DB-Query - https://phabricator.wikimedia.org/T277124 (10Aklapper) >>! In T277124#6903441, @tstarling wrote: > Is it necessary for tags to be non-overlapping? A query can be slow without being an error, and there can be errors without slowness. It is not necessary; g... [07:57:07] 10Phabricator (Upstream), 10Upstream: Herald rule to suppress Task activity? (and just receive Task creation) - https://phabricator.wikimedia.org/T277132 (10Aklapper) p:05Triage→03Lowest This is basically https://secure.phabricator.com/T9161 and maybe https://secure.phabricator.com/T5791 . The available op... [08:37:09] (03CR) 10Mainframe98: "Maybe I'm missing something, but shouldn't @doesNotPerformAssertions be added too?" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/670120 (https://phabricator.wikimedia.org/T276971) (owner: 10DannyS712) [08:56:46] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Patch-For-Review, 10ci-test-error (WMF-deployed Build Failure): Many WikibaseLexeme tests suddenly failing in quibble-vendor-mysql-php72-{selenium,noselenium}-docker - https://phabricator.wikimedia.org/T277061 (10Lucas_Werkmeister_WMDE) [08:56:48] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.35 deployment blockers - https://phabricator.wikimedia.org/T274939 (10Lucas_Werkmeister_WMDE) [08:58:45] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Patch-For-Review, 10ci-test-error (WMF-deployed Build Failure): Many WikibaseLexeme tests suddenly failing in quibble-vendor-mysql-php72-{selenium,noselenium}-docker - https://phabricator.wikimedia.org/T277061 (10Lucas_Werkmeister_WMDE) p:05Triage→0... [09:12:08] !log Updated plugins on https://releases-jenkins.wikimedia.org/ [09:12:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:37:18] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review: Switch Jenkins servers to Java 11 - https://phabricator.wikimedia.org/T269354 (10hashar) @MoritzMuehlenhoff and I ha... [09:43:14] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Patch-For-Review, 10ci-test-error (WMF-deployed Build Failure): Many WikibaseLexeme tests suddenly failing in quibble-vendor-mysql-php72-{selenium,noselenium}-docker - https://phabricator.wikimedia.org/T277061 (10Tarrow) p:05High→03Unbreak! Since t... [09:59:29] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Patch-For-Review, 10ci-test-error (WMF-deployed Build Failure): Many WikibaseLexeme tests suddenly failing in quibble-vendor-mysql-php72-{selenium,noselenium}-docker - https://phabricator.wikimedia.org/T277061 (10Lucas_Werkmeister_WMDE) a:03Lucas_Werk... [10:07:03] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Patch-For-Review, 10ci-test-error (WMF-deployed Build Failure): Many WikibaseLexeme tests suddenly failing in quibble-vendor-mysql-php72-{selenium,noselenium}-docker - https://phabricator.wikimedia.org/T277061 (10Lucas_Werkmeister_WMDE) >>! In T277061#... [10:11:22] (03PS1) 10Filippo Giunchedi: Add operations/alerts tox [integration/config] - 10https://gerrit.wikimedia.org/r/670782 (https://phabricator.wikimedia.org/T272977) [10:14:17] hashar: Hey! Would you have a few minutes for CI questions? [10:14:38] Trying to add sonar analysis for python with zpapierski [10:14:40] https://meet.google.com/bmx-pwta-wsf [10:17:09] marxarelli: ah, cool! thanks [10:21:38] (03PS6) 10ZPapierski: Add sonar scanner to discolytics [integration/config] - 10https://gerrit.wikimedia.org/r/669770 (https://phabricator.wikimedia.org/T264877) [10:22:15] hashar: zpapierski is going to send you the CR first, ping us if you have time to go over it at some point [10:23:11] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Patch-For-Review, 10ci-test-error (WMF-deployed Build Failure): Many WikibaseLexeme tests suddenly failing in quibble-vendor-mysql-php72-{selenium,noselenium}-docker - https://phabricator.wikimedia.org/T277061 (10Addshore) @Lucas_Werkmeister_WMDE If it... [10:23:40] (03CR) 10jerkins-bot: [V: 04-1] Add sonar scanner to discolytics [integration/config] - 10https://gerrit.wikimedia.org/r/669770 (https://phabricator.wikimedia.org/T264877) (owner: 10ZPapierski) [10:38:14] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Patch-For-Review, 10ci-test-error (WMF-deployed Build Failure): Many WikibaseLexeme tests suddenly failing in quibble-vendor-mysql-php72-{selenium,noselenium}-docker - https://phabricator.wikimedia.org/T277061 (10Lucas_Werkmeister_WMDE) >>! In T277061#... [10:51:19] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Patch-For-Review, 10ci-test-error (WMF-deployed Build Failure): Many WikibaseLexeme tests suddenly failing in quibble-vendor-mysql-php72-{selenium,noselenium}-docker - https://phabricator.wikimedia.org/T277061 (10Lucas_Werkmeister_WMDE) Meh, but I have... [10:53:43] marxarelli: I'm not sure that option works unfortunately on second thought [11:14:07] (03PS7) 10ZPapierski: Add sonar scanner to discolytics [integration/config] - 10https://gerrit.wikimedia.org/r/669770 (https://phabricator.wikimedia.org/T264877) [11:24:52] (03PS8) 10ZPapierski: Add sonar scanner to discolytics [integration/config] - 10https://gerrit.wikimedia.org/r/669770 (https://phabricator.wikimedia.org/T264877) [11:25:16] (03CR) 10ZPapierski: Add sonar scanner to discolytics (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/669770 (https://phabricator.wikimedia.org/T264877) (owner: 10ZPapierski) [11:25:52] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Patch-For-Review, 10ci-test-error (WMF-deployed Build Failure): Many WikibaseLexeme tests suddenly failing in quibble-vendor-mysql-php72-{selenium,noselenium}-docker - https://phabricator.wikimedia.org/T277061 (10Lucas_Werkmeister_WMDE) I can confirm t... [11:29:03] (03CR) 10Lars Wirzenius: [C: 03+2] Demote a _runcmd log message to debug level [tools/scap] - 10https://gerrit.wikimedia.org/r/670564 (owner: 10Ahmon Dancy) [11:29:49] (03Merged) 10jenkins-bot: Demote a _runcmd log message to debug level [tools/scap] - 10https://gerrit.wikimedia.org/r/670564 (owner: 10Ahmon Dancy) [11:40:28] (03CR) 10Lars Wirzenius: [C: 03+2] Use train-dev branch of operations/mediawiki-config [tools/train-dev] - 10https://gerrit.wikimedia.org/r/670316 (owner: 10Ahmon Dancy) [11:40:40] (03CR) 10Lars Wirzenius: [V: 03+2 C: 03+2] Use train-dev branch of operations/mediawiki-config [tools/train-dev] - 10https://gerrit.wikimedia.org/r/670316 (owner: 10Ahmon Dancy) [11:42:33] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review: Switch Jenkins servers to Java 11 - https://phabricator.wikimedia.org/T269354 (10hashar) [12:11:45] (03PS1) 10Hashar: analytics/udplog: make debian glue voting [integration/config] - 10https://gerrit.wikimedia.org/r/670805 (https://phabricator.wikimedia.org/T276422) [12:12:06] (03CR) 10Hashar: [C: 03+2] analytics/udplog: make debian glue voting [integration/config] - 10https://gerrit.wikimedia.org/r/670805 (https://phabricator.wikimedia.org/T276422) (owner: 10Hashar) [12:13:12] (03Merged) 10jenkins-bot: analytics/udplog: make debian glue voting [integration/config] - 10https://gerrit.wikimedia.org/r/670805 (https://phabricator.wikimedia.org/T276422) (owner: 10Hashar) [12:14:39] 10Continuous-Integration-Config, 10Gerrit, 10Patch-For-Review: Unarchive analytics/udplog repository - https://phabricator.wikimedia.org/T276422 (10hashar) 05Open→03Resolved p:05Triage→03Medium The debian-glue job passed on https://gerrit.wikimedia.org/r/c/analytics/udplog/+/668451 which targets `b... [12:27:07] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Patch-For-Review, 10ci-test-error (WMF-deployed Build Failure): Many WikibaseLexeme tests suddenly failing in quibble-vendor-mysql-php72-{selenium,noselenium}-docker - https://phabricator.wikimedia.org/T277061 (10Lucas_Werkmeister_WMDE) I’ve created {T... [12:32:36] reminder: we will be performing beta cluster database maintenance including some read-only time in one hour (13:30 UTC) [13:03:12] 10Gerrit, 10Release-Engineering-Team: Gerrit Apache out of workers - https://phabricator.wikimedia.org/T277127 (10hashar) I can't tell what went wrong. I have notice there is some bot from WMCS which hammers gitiles as fast as it can. The requests are using `format=TEXT` which emits some base64 payload. They... [13:18:31] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.35 deployment blockers - https://phabricator.wikimedia.org/T274939 (10Lucas_Werkmeister_WMDE) [13:19:41] 10Continuous-Integration-Infrastructure, 10Wikidata, 10MW-1.36-notes (1.36.0-wmf.35; 2021-03-16), 10Patch-For-Review, 10ci-test-error (WMF-deployed Build Failure): Many WikibaseLexeme tests suddenly failing in quibble-vendor-mysql-php72-{selenium,noselenium}-dock... - https://phabricator.wikimedia.org/T277061 [13:31:37] hi Majavah! [13:31:55] hi [13:32:03] so...let's switch the masters :-) [13:32:04] I'll guess let's start with mw-level ro? [13:32:08] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/670804 [13:32:20] Majavah: can you try to set DB level RO on master? [13:32:25] I'm wondering whether MW detects that properly [13:32:50] !log set deployment-db06 as read only T277070 [13:32:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:32:58] T277070: Beta cluster master switchover to deployment-db07 - https://phabricator.wikimedia.org/T277070 [13:33:15] "The Wikipedia database is temporarily in read-only mode" [13:33:17] looks correct [13:33:31] cool [13:33:51] so let's switch masters now :) [13:34:41] !log stop and reset slave on deployment-db07 T277070 [13:34:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:37:47] !log make deployment-db06 and deployment-db08 be replicas of deployment-db07 T277070 [13:37:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:38:10] master changed, https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/670803 next I believe [13:38:39] Majavah: shouldn't we decomission db06? it has old mariadb [13:39:31] Urbanecm: I was thinking of doing that separately, but I guess we can do that at the same time too [13:40:02] Majavah: in https://phabricator.wikimedia.org/T276968#6900225, the (prod) DBAs say that master should have the oldest version [13:40:13] so i think we should do it now [13:40:30] ah fair point, didn't see that [13:40:34] sure, let's do that now [13:40:42] I'll amend my patch [13:40:47] thanks [13:40:52] and also stop replication there i guess [13:41:06] !log stop slave on deployment-db06 T276968 [13:41:08] done [13:41:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:41:11] T276968: deployment-db05 needs replacing following disk corruption - https://phabricator.wikimedia.org/T276968 [13:41:14] thanks for noticing that [13:41:53] patch updated [13:42:34] reviewing [13:42:55] patch sound ok, merging [13:43:26] thanks [13:44:53] https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/341279/ has the config patch in it [13:45:09] cool [13:47:02] Majavah: new master is still RO, right? [13:47:17] Urbanecm: yes, as it was a replica that was ro [13:47:41] cool [13:47:44] MW patch is deployed [13:47:48] Majavah: please set new master to RW [13:48:01] !log stop mariadb to ensure reads have stopped on deployment-db06 T277070 [13:48:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:48:05] T277070: Beta cluster master switchover to deployment-db07 - https://phabricator.wikimedia.org/T277070 [13:48:26] !log set deployment-db07 as r/w T277070 [13:48:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:49:27] and beta is now RW again [13:49:32] thanks Majavah :) [13:50:04] replication seems to still be working, at least based on master positions [13:50:32] let's shutdown db06 and delete after a day or two? [13:50:48] sure! [13:51:06] !log shut down deployment-db06, now unused T277070 [13:51:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:51:16] thanks :) [13:52:22] 10Beta-Cluster-Infrastructure, 10Quality-and-Test-Engineering-Team (QTE): Upgrade deployment-prep-db hosts to buster/MariaDB 10.4 - https://phabricator.wikimedia.org/T268628 (10Majavah) [13:52:52] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10User-Majavah, and 2 others: Beta cluster master switchover to deployment-db07 - https://phabricator.wikimedia.org/T277070 (10Majavah) 05Open→03Resolved a:03Majavah This was done with... [13:53:20] 10Beta-Cluster-Infrastructure, 10Quality-and-Test-Engineering-Team (QTE), 10User-Majavah: Upgrade deployment-prep-db hosts to buster/MariaDB 10.4 - https://phabricator.wikimedia.org/T268628 (10Majavah) 05Open→03Resolved a:03Majavah [14:14:30] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.35 deployment blockers - https://phabricator.wikimedia.org/T274939 (10Krinkle) [14:44:06] 10MediaWiki-Codesniffer: False positive for Java-style array types - https://phabricator.wikimedia.org/T277176 (10thiemowmde) [14:56:27] (03PS1) 10Thiemo Kreuz (WMDE): Fix false positive for Java-style array types [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/670832 (https://phabricator.wikimedia.org/T277176) [14:56:48] 10Phabricator (Upstream), 10Upstream: Herald rule to suppress Task activity? (and just receive Task creation) - https://phabricator.wikimedia.org/T277132 (10valerio.bozzolan) [14:57:44] 10Phabricator (Upstream), 10Upstream: Herald rule to suppress Task activity? (and just receive Task creation) - https://phabricator.wikimedia.org/T277132 (10valerio.bozzolan) [15:24:12] 10Release-Engineering-Team (Logspam), 10JavaScript, 10MW-1.36-notes (1.36.0-wmf.35; 2021-03-16), 10MediaWiki-Interface (Tables), and 2 others: Some tables cannot be sorted (TypeError: $nextRows[i] is undefined in table sorting / Uncaught TypeError: Cannot read pro... - https://phabricator.wikimedia.org/T265503 [15:32:46] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10serviceops, 10Patch-For-Review: Replace production deployment servers and update them to Buster - https://phabricator.wikimedia.org/T265963 (10Cmjohnson) [15:32:49] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Cmjohnson) [15:33:32] (03PS1) 10Jforrester: CI access list: Add ThisDot [integration/config] - 10https://gerrit.wikimedia.org/r/670842 [15:40:23] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)): Fix or delete integration-config-qa - https://phabricator.wikimedia.org/T245037 (10hashar) That is intended to show up a dashboard of things that should be fix... [15:41:40] 10Continuous-Integration-Config: CI should ensure all files are listed in PECL package.xml - https://phabricator.wikimedia.org/T277068 (10hashar) p:05Triage→03Medium [15:42:14] 10Continuous-Integration-Config, 10Analytics, 10Event-Platform: Jenkins-bot does not submit changes on passing gate-and-submit for /schemas/event/* repos - https://phabricator.wikimedia.org/T277051 (10hashar) You are welcome, thank you to have confirmed the fix! [15:50:42] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Code-Health, 10MW-1.35-notes (1.35.0-wmf.15; 2020-01-14), 10Platform Team Initiatives (API Integration Tests): In CI, configure MediaWiki to enable API testing - https://phabricator.wikimedia.org/T243978 (1... [15:50:45] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Code-Health, 10MW-1.35-notes (1.35.0-wmf.19; 2020-02-11), 10Platform Team Initiatives (API Integration Tests): Enable API integration tests in CI for MediaWiki cor... - https://phabricator.wikimedia.org/T236680 [15:55:33] 10Phabricator (Upstream), 10Upstream: Herald rule to suppress Task activity? (and just receive Task creation) - https://phabricator.wikimedia.org/T277132 (10epriestley) > When: > [ Projects ][ include any of ][ X, Y, Z, ... ] > Take these actions [ only the first time this rule matches ]: > [ Send me an email... [16:00:55] 10phan, 10Documentation, 10Doxygen, 10MW-1.35-notes (1.35.0-wmf.20; 2020-02-18), and 2 others: Doxygen: Ignore `@phan` notation (and more) - https://phabricator.wikimedia.org/T241182 (10hashar) This task apparently depended on an upgrade of Doxygen. CI now runs Doxygen 1.8.19. So we should now be able to... [16:30:50] Majavah: hey, nice work on the db master migration! [16:31:28] thank you marxarelli! [16:33:38] (03PS4) 10DannyS712: FunctionAnnotationsSniff: recognize more phpunit annotations [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/670120 (https://phabricator.wikimedia.org/T276971) [16:33:43] (03CR) 10DannyS712: "> Patch Set 3:" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/670120 (https://phabricator.wikimedia.org/T276971) (owner: 10DannyS712) [16:38:53] Well done Majavah [16:39:01] Nice buster progress too! [16:47:55] RhinosF1: https://os-deprecation.toolforge.org/jessie/deployment-prep.html "by June 2020" [16:48:34] Majavah: you still have done a lot in a week or so [16:48:40] 9 months ain't that bad tbh [16:49:07] !log delete deployment-etcd-01 T276462 [16:49:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:49:12] T276462: Replace deployment-etcd-01 with a Buster host - https://phabricator.wikimedia.org/T276462 [16:49:18] 10Beta-Cluster-Infrastructure, 10User-Majavah: Replace deployment-etcd-01 with a Buster host - https://phabricator.wikimedia.org/T276462 (10Majavah) 05Open→03Resolved [16:49:20] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Jessie Deprecation): Migrate deployment-prep away from Debian Jessie to Debian Stretch/Buster - https://phabricator.wikimedia.org/T218729 (10Majavah) [16:49:28] RhinosF1: I just have the hardest ones remaining... [16:49:55] I'm sure you'll get it done [16:52:57] (03PS1) 10Hashar: jjb: set DEBEMAIL in debian-glue jobs [integration/config] - 10https://gerrit.wikimedia.org/r/670863 (https://phabricator.wikimedia.org/T273896) [16:53:14] (03CR) 10Hashar: [C: 03+2] jjb: set DEBEMAIL in debian-glue jobs [integration/config] - 10https://gerrit.wikimedia.org/r/670863 (https://phabricator.wikimedia.org/T273896) (owner: 10Hashar) [16:53:48] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Patch-For-Review, 10cloud-services-team (Kanban): labs/toollabs fails debian-glue-unstable for lintian errors caused by the config - https://phabricator.wikimedia.org/T273896 (10hashar) 05Open→03Resolved a:03hashar... [16:54:51] (03Merged) 10jenkins-bot: jjb: set DEBEMAIL in debian-glue jobs [integration/config] - 10https://gerrit.wikimedia.org/r/670863 (https://phabricator.wikimedia.org/T273896) (owner: 10Hashar) [16:57:58] !log copy a tarball of deployment-fluorine02 /home to deployment-mwlog01 root home dir, delete deployment-fluorine02 T276419 [16:58:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:58:03] T276419: Replace deployment-fluorine02 with a Buster host - https://phabricator.wikimedia.org/T276419 [17:04:08] (03CR) 10Hashar: [C: 03+2] CI access list: Add ThisDot [integration/config] - 10https://gerrit.wikimedia.org/r/670842 (owner: 10Jforrester) [17:05:33] (03Merged) 10jenkins-bot: CI access list: Add ThisDot [integration/config] - 10https://gerrit.wikimedia.org/r/670842 (owner: 10Jforrester) [17:07:12] (03PS2) 10Zabe: Add Artrix in the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/669192 [17:08:17] 10Continuous-Integration-Infrastructure, 10Quibble: Can we make the PHP inbuilt webserver used in CI log the time it has taken to execute calls to the API? - https://phabricator.wikimedia.org/T277202 (10Addshore) [17:08:36] 10Continuous-Integration-Infrastructure, 10Quibble: Can we make the PHP inbuilt webserver used in CI log the time it has taken to execute calls to the API? - https://phabricator.wikimedia.org/T277202 (10Addshore) [17:10:44] (03PS3) 10Hashar: Add Artrix in the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/669192 (owner: 10Zabe) [17:11:18] (03CR) 10Hashar: [C: 03+2] Add Artrix in the CI whitelist (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/669192 (owner: 10Zabe) [17:13:07] (03CR) 10Zabe: Add Artrix in the CI whitelist (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/669192 (owner: 10Zabe) [17:14:00] (03Merged) 10jenkins-bot: Add Artrix in the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/669192 (owner: 10Zabe) [17:15:20] 10Continuous-Integration-Infrastructure, 10Quibble, 10Browser-Tests: Can we make the PHP inbuilt webserver used in CI log the time it has taken to execute calls to the API? - https://phabricator.wikimedia.org/T277202 (10Addshore) [17:15:42] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Browser-Tests: Centrally look for flakey browser tests - https://phabricator.wikimedia.org/T277205 (10Addshore) [17:15:46] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.34 deployment blockers - https://phabricator.wikimedia.org/T274938 (10brennen) [17:16:37] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.35 deployment blockers - https://phabricator.wikimedia.org/T274939 (10brennen) [17:22:52] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Instrument-ClientError, 10Documentation, 10User-brennen: Deploy process should be updated so JavaScript errors block the train - https://phabricator.wikimedia.org/T274000 (10brennen) Updat... [17:26:55] 10Beta-Cluster-Infrastructure, 10User-Majavah: Replace deployment-fluorine02 with a Buster host - https://phabricator.wikimedia.org/T276419 (10Majavah) 05Open→03Resolved [17:26:58] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Jessie Deprecation): Migrate deployment-prep away from Debian Jessie to Debian Stretch/Buster - https://phabricator.wikimedia.org/T218729 (10Majavah) [17:36:53] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Browser-Tests: Centrally look for flakey browser tests - https://phabricator.wikimedia.org/T277205 (10Addshore) I very briefly looked at the main jenkins node, for a single example job and identified one of the ✖s that indicates a failed test in 124 logs.... [17:37:50] (03PS1) 10Lars Wirzenius: fix: increase git VM disk size; make mirror-gits quieter [tools/train-dev] - 10https://gerrit.wikimedia.org/r/670896 [17:39:04] (03CR) 10Ahmon Dancy: [C: 03+2] fix: increase git VM disk size; make mirror-gits quieter [tools/train-dev] - 10https://gerrit.wikimedia.org/r/670896 (owner: 10Lars Wirzenius) [17:39:21] 10Continuous-Integration-Infrastructure, 10Quibble, 10Browser-Tests: Can we make the PHP inbuilt webserver used in CI log the time it has taken to execute calls to the API? - https://phabricator.wikimedia.org/T277202 (10hashar) Those messages come from `php -S` stderr being relayed. PHP does not seem to hav... [17:39:25] (03CR) 10Ahmon Dancy: [V: 03+2 C: 03+2] fix: increase git VM disk size; make mirror-gits quieter [tools/train-dev] - 10https://gerrit.wikimedia.org/r/670896 (owner: 10Lars Wirzenius) [17:43:06] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Browser-Tests: Centrally look for flakey browser tests - https://phabricator.wikimedia.org/T277205 (10Addshore) my bash foo isn't up to figuring out how to tie these greps back to the individual runs that had the errors to allow for deep diving into the l... [17:45:43] 10Beta-Cluster-Infrastructure: Beta cron jobs seem broken - https://phabricator.wikimedia.org/T277206 (10Tgr) [17:46:33] 10Continuous-Integration-Infrastructure, 10Quibble, 10Browser-Tests: Can we make the PHP inbuilt webserver used in CI log the time it has taken to execute calls to the API? - https://phabricator.wikimedia.org/T277202 (10Addshore) > Those messages come from php -S stderr being relayed. PHP does not seem to ha... [17:47:08] hashar: I guess the move to apache would have to haappen within quibble itself? [17:47:21] 10Beta-Cluster-Infrastructure: Beta cron jobs seem broken - https://phabricator.wikimedia.org/T277206 (10Tgr) I have seen the same status=2 exit last week, although I didn't investigate then, so this is probably not new. [18:00:39] 10Continuous-Integration-Infrastructure, 10Quibble, 10Browser-Tests: Can we make the PHP inbuilt webserver used in CI log the time it has taken to execute calls to the API? - https://phabricator.wikimedia.org/T277202 (10hashar) > I did lots of searching and couldn't find any :( Maybe in php8.0 source code t... [18:02:23] (03CR) 10Ahmon Dancy: [C: 03+1] Add helm deploy timeout override [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/670635 (https://phabricator.wikimedia.org/T277125) (owner: 10Jeena Huneidi) [18:03:17] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Instrument-ClientError, 10Documentation, 10User-brennen: Deploy process should be updated so JavaScript errors block the train - https://phabricator.wikimedia.org/T274000 (10Jrbranaa) I th... [18:09:02] Not sure if this is the right channel so feel free to redirect but who do I talk to about GitHub -> Gerrit possible mirroring issues. [18:12:25] addshore: sorry was looking elsewhere. [18:12:42] Np [18:12:50] addshore: awight went with the CI image providing the apache config and using supervisord to spawn apache [18:13:24] addshore: quibble is then started with something like --web-url=http://127.0.0.1:8082 (whatever Apache is listening on) , and does not spawn the php built in webserver [18:14:06] clarakosi: it is the other way around? Gerrit replicates to Github , those are usually tracked using #github-mirrors + #gerrit in Phabricator [18:14:41] addshore: that got sprinted by Adam and Kosta if you know them ;]]]]]]]]]]]] [18:15:36] youre 3 of my favourite people :P [18:15:42] is there any phab ticket? [18:15:45] * addshore food [18:15:47] well [18:15:55] there are phab tickets for everything already :] [18:16:22] i am off as well for dinner etc [18:16:37] clarakosi: I guess file a task and progress from there [18:16:49] hashar: no its GH to Gerrit. In particular the RESTBase repo. We do our dev on GH and I assumed the Gerrit repo was automatically updated [18:16:59] oh [18:17:02] no for sure no [18:17:04] ;] [18:17:13] I guess people manually sync it from their machine [18:17:39] Really? I didn't realize that [18:17:57] I don't really know to be honest [18:18:33] I dont even think mediawiki/services/restbase.git in Gerrit is used for anything [18:18:52] We use it for a script to update the deploy/restbase repo [18:19:19] But maybe I can just do it manually. Will check [18:20:01] Petr would know for sure, Hugh might as well [18:20:34] I am just as confused if not moreso :) [18:20:45] it is all for historicaly reasons really [18:21:09] I think the grand idea was to have the micro services to be developed on github to involve 3rd parties and give the projects more exposure [18:21:33] the deploy repo still uses https://gerrit.wikimedia.org/r/mediawiki/services/restbase as a submodule so the gerrit one is still authoritative [18:21:52] while on oeprations side we only deploy from Gerrit , cause we (at time) at least have some guarantee about what we are going to deploy and know that our git is fixable by ourselve [18:22:11] so I guess the github repositories are then manually pushed to the equivalent gerrit repo [18:22:33] the /deploy repo has the gerrit repo as a submodule which just need to be bumped for deployment [18:22:41] yeah [18:22:50] it just the sync up from github to gerrit has to be done manually [18:23:02] one can think about it as a manual snapshot of the github repo [18:26:33] I am off for dinner [18:27:13] (03CR) 10Dduvall: [C: 04-1] Add helm deploy timeout override (033 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/670635 (https://phabricator.wikimedia.org/T277125) (owner: 10Jeena Huneidi) [18:28:43] Thanks hasharDinner ! Enjoy your dinner 👋 [18:34:49] (03CR) 10Jeena Huneidi: Add helm deploy timeout override (033 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/670635 (https://phabricator.wikimedia.org/T277125) (owner: 10Jeena Huneidi) [18:35:34] (03CR) 10Dduvall: [C: 04-1] "Just a few more nits. Barring those, I'm good for a merge." (035 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/668245 (https://phabricator.wikimedia.org/T269902) (owner: 10Jeena Huneidi) [18:38:56] (03CR) 10Dduvall: [C: 04-1] Add helm deploy timeout override (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/670635 (https://phabricator.wikimedia.org/T277125) (owner: 10Jeena Huneidi) [18:40:45] (03CR) 10Dduvall: [C: 04-1] Add helm deploy timeout override (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/670635 (https://phabricator.wikimedia.org/T277125) (owner: 10Jeena Huneidi) [18:55:12] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10MW-on-K8s, 10serviceops: Containers on releases hosts cannot update apt cache from non-WMF sources - https://phabricator.wikimedia.org/T277109 (10dduvall) Another option might just be to set an `http_proxy` envir... [18:57:21] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10MW-on-K8s, 10serviceops: Containers on releases hosts cannot update apt cache from non-WMF sources - https://phabricator.wikimedia.org/T277109 (10dduvall) >>! In T277109#6906099, @dduvall wrote: > Another option... [19:01:54] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10MW-on-K8s, 10serviceops: Containers on releases hosts cannot update apt cache from non-WMF sources - https://phabricator.wikimedia.org/T277109 (10dduvall) [19:01:56] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10MW-on-K8s, 10Patch-For-Review: Multi-version MediaWiki image is built and published - https://phabricator.wikimedia.org/T274182 (10dduvall) [19:02:31] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10MW-on-K8s, 10serviceops: Containers on releases hosts cannot update apt cache from non-WMF sources - https://phabricator.wikimedia.org/T277109 (10dduvall) p:05Triage→03Medium a:03dduvall [19:03:52] (03PS2) 10Jeena Huneidi: Add helm deploy timeout override [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/670635 (https://phabricator.wikimedia.org/T277125) [19:08:38] (03PS13) 10Jeena Huneidi: PipelineRunner: allowedCredentials [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/668245 (https://phabricator.wikimedia.org/T269902) [19:09:08] (03CR) 10Jeena Huneidi: PipelineRunner: allowedCredentials (035 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/668245 (https://phabricator.wikimedia.org/T269902) (owner: 10Jeena Huneidi) [19:11:16] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T271342 (10brennen) 05Declined→03Resolved [19:11:17] (03CR) 10Dduvall: [C: 03+2] PipelineRunner: allowedCredentials [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/668245 (https://phabricator.wikimedia.org/T269902) (owner: 10Jeena Huneidi) [19:12:47] (03Merged) 10jenkins-bot: PipelineRunner: allowedCredentials [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/668245 (https://phabricator.wikimedia.org/T269902) (owner: 10Jeena Huneidi) [19:14:02] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T271342 (10brennen) 05Resolved→03Declined [19:18:45] 10Release-Engineering-Team (Logspam), 10JavaScript, 10MW-1.36-notes (1.36.0-wmf.35; 2021-03-16), 10MediaWiki-Interface (Tables), and 2 others: Some tables cannot be sorted (TypeError: $nextRows[i] is undefined in table sorting / Uncaught TypeError: Cannot read pro... - https://phabricator.wikimedia.org/T265503 [19:18:58] (03CR) 10Dduvall: [C: 04-1] "Just one outstanding issue and now it looks like a merge conflict. Fix that and I can merge!" (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/670635 (https://phabricator.wikimedia.org/T277125) (owner: 10Jeena Huneidi) [19:19:18] 10Release-Engineering-Team (Logspam), 10JavaScript, 10MW-1.36-notes (1.36.0-wmf.35; 2021-03-16), 10MediaWiki-Interface (Tables), and 2 others: Some tables cannot be sorted (TypeError: $nextRows[i] is undefined in table sorting / Uncaught TypeError: Cannot read pro... - https://phabricator.wikimedia.org/T265503 [19:22:43] (03CR) 10Jeena Huneidi: Add helm deploy timeout override (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/670635 (https://phabricator.wikimedia.org/T277125) (owner: 10Jeena Huneidi) [19:25:07] (03CR) 10Dduvall: [C: 03+1] Add helm deploy timeout override (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/670635 (https://phabricator.wikimedia.org/T277125) (owner: 10Jeena Huneidi) [19:28:49] (03PS3) 10Jeena Huneidi: Add helm deploy timeout override [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/670635 (https://phabricator.wikimedia.org/T277125) [19:30:59] (03CR) 10Dduvall: [C: 03+2] Add helm deploy timeout override [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/670635 (https://phabricator.wikimedia.org/T277125) (owner: 10Jeena Huneidi) [19:32:02] (03Merged) 10jenkins-bot: Add helm deploy timeout override [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/670635 (https://phabricator.wikimedia.org/T277125) (owner: 10Jeena Huneidi) [19:35:19] 10Continuous-Integration-Config: Introduce non-voting jobs with quibble+apache - https://phabricator.wikimedia.org/T276428 (10Krinkle) > The major difference is that the apache job can serve requests concurrently, which will expose some issues with our code and require fixes to stabilize tests. Have we seen thi... [19:37:43] 10Continuous-Integration-Config: Introduce non-voting jobs with quibble+apache - https://phabricator.wikimedia.org/T276428 (10Addshore) >>! In T277202#6905923, @hashar wrote: >>> But that opens some other issues such as race conditions between queries (and potentially locking issue with Sqlite). >> Hmm, where is... [19:42:46] (03PS1) 10Jeena Huneidi: PipelineRunner: Correct comment for timeout var [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/670919 [19:43:35] (03CR) 10Jeena Huneidi: [C: 03+2] PipelineRunner: Correct comment for timeout var [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/670919 (owner: 10Jeena Huneidi) [19:44:12] (03Merged) 10jenkins-bot: PipelineRunner: Correct comment for timeout var [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/670919 (owner: 10Jeena Huneidi) [19:49:38] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release Pipeline: Better way to restrict credentials available to pipelinelib - https://phabricator.wikimedia.org/T267699 (10jeena) [19:49:40] 10Release-Engineering-Team (Pipeline), 10Release Pipeline, 10Patch-For-Review: Allow overrides to pipeline builder for passing allowed credentials list - https://phabricator.wikimedia.org/T269902 (10jeena) 05Open→03Resolved [19:54:57] 10Continuous-Integration-Config: Introduce non-voting jobs with quibble+apache - https://phabricator.wikimedia.org/T276428 (10kostajh) >>! In T276428#6906218, @Krinkle wrote: >> The major difference is that the apache job can serve requests concurrently, which will expose some issues with our code and require fi... [20:05:36] 10Continuous-Integration-Infrastructure, 10Quibble, 10Browser-Tests: Can we make the PHP inbuilt webserver used in CI log the time it has taken to execute calls to the API? - https://phabricator.wikimedia.org/T277202 (10kostajh) I don't know the answer to your question, but if you know a certain test is havi... [20:13:18] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "Sure!" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/670120 (https://phabricator.wikimedia.org/T276971) (owner: 10DannyS712) [20:22:08] 10Phabricator (Upstream), 10Upstream: Herald rule to suppress Task activity? (and just receive Task creation) - https://phabricator.wikimedia.org/T277132 (10valerio.bozzolan) >>! In T277132#6905271, @epriestley wrote: >> When: >> [ Projects ][ include any of ][ X, Y, Z, ... ] >> Take these actions [ only the f... [20:53:13] 10Continuous-Integration-Infrastructure, 10Quibble, 10Browser-Tests: Can we make the PHP inbuilt webserver used in CI log the time it has taken to execute calls to the API? - https://phabricator.wikimedia.org/T277202 (10Addshore) >>! In T277202#6906417, @kostajh wrote: > I don't know the answer to your quest... [21:21:15] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.34 deployment blockers - https://phabricator.wikimedia.org/T274938 (10brennen) [21:24:17] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.34 deployment blockers - https://phabricator.wikimedia.org/T274938 (10Jdlrobson) [21:33:20] 10Continuous-Integration-Config, 10Add-Link, 10Growth-Team, 10Release Pipeline: java.lang.RuntimeException: Invalid Credential: 'SONAR_API_KEY' - https://phabricator.wikimedia.org/T277236 (10kostajh) [21:33:38] 10Continuous-Integration-Config, 10Add-Link, 10Growth-Team, 10Release Pipeline: java.lang.RuntimeException: Invalid Credential: 'SONAR_API_KEY' - https://phabricator.wikimedia.org/T277236 (10kostajh) @jeena or @hashar do you have an idea of what may have changed in our config to cause this? [21:38:05] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Instrument-ClientError, 10Documentation, 10User-brennen: Deploy process should be updated so JavaScript errors block the train - https://phabricator.wikimedia.org/T274000 (10Jdlrobson) Aft... [21:38:56] 10Continuous-Integration-Config, 10Add-Link, 10Growth-Team, 10Release Pipeline: java.lang.RuntimeException: Invalid Credential: 'SONAR_API_KEY' - https://phabricator.wikimedia.org/T277236 (10kostajh) p:05Triage→03High [21:49:25] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Patch-For-Review: l10n process for MW containerization - https://phabricator.wikimedia.org/T260827 (10dancy) 05Open→03Resolved [21:49:29] 10Release-Engineering-Team, 10MW-on-K8s, 10Release Pipeline, 10serviceops-radar, 10Epic: Deploy MediaWiki to Wikimedia production in containers - https://phabricator.wikimedia.org/T238770 (10dancy) [21:50:25] 10Continuous-Integration-Config, 10Add-Link, 10Growth-Team, 10Release Pipeline: java.lang.RuntimeException: Invalid Credential: 'SONAR_API_KEY' - https://phabricator.wikimedia.org/T277236 (10hashar) The `SONAR_API_KEY` credential is still in the CI Jenkins. I guess that comes from https://gerrit.wikimedia... [21:51:04] longma: marxarelli: looks like the pipelinelib / credentials stuff broke something https://phabricator.wikimedia.org/T277236 . kostajh filed a task about SONAR_API_KEY not being reachable [21:51:11] I am off, night time ! [21:51:50] Sorry! we need to merge this patch and then it should work https://gerrit.wikimedia.org/r/c/integration/config/+/668199 [21:52:12] ^ marxarelli [21:53:31] gah [21:53:34] * marxarelli looks [21:54:05] (03CR) 10Dduvall: [C: 03+1] Define and pass allowed credentials to pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/668199 (https://phabricator.wikimedia.org/T269900) (owner: 10Jeena Huneidi) [22:00:59] longma: i'm going to update the `research-mwaddlink-pipeline-test` and trigger job first [22:01:17] so we can re-run/test it [22:01:24] before updating all the pipeline jobs [22:01:24] marxarelli what do you need to add? [22:01:36] add? [22:01:36] SONAR_API_KEY is allowed by default [22:01:57] sorry I mean why do you need to update the job? [22:02:14] no i mean i'm going to run `tox -e jenkins-jobs -- update` to update the jobs in jenkins [22:02:21] against your patch [22:02:24] ah ok [22:02:28] got it [22:02:39] kostajh: ^ [22:03:41] !log running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb/ trigger-research-mwaddlink-pipeline-test research-mwaddlink-pipeline-test` to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/668199 for select jobs (T277236) [22:03:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:03:45] T277236: java.lang.RuntimeException: Invalid Credential: 'SONAR_API_KEY' - https://phabricator.wikimedia.org/T277236 [22:03:58] k. updated [22:04:57] oh boy `Caused: java.io.NotSerializableException: groovy.json.internal.LazyMap` [22:05:05] :( [22:05:10] we should revert the pipelinelib patch for now [22:05:15] * marxarelli does [22:05:15] okay [22:06:25] (03PS1) 10Dduvall: Revert "PipelineRunner: allowedCredentials" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/670963 [22:06:27] when I ran tox to update the jobs locally I didn't see any errors like that [22:07:00] yeah it's not the job generation that failed. it's a runtime exception, no doubt related to CPS [22:07:07] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.34 deployment blockers - https://phabricator.wikimedia.org/T274938 (10Jdlrobson) [22:07:08] ahh [22:07:48] interesting that it was on the pipelinelib side since the systemtests didn't indicate that [22:08:13] we probably need to add a systemtest for the credentials functionality [22:08:26] I added in some credentials and tried it [22:08:52] so maybe it's the map that's being passed through from the integration/config side that is funky [22:09:11] hmm, possibly. not sure! needs a closer look [22:09:36] (03CR) 10Dduvall: [C: 03+2] Revert "PipelineRunner: allowedCredentials" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/670963 (owner: 10Dduvall) [22:10:40] !log running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb/ trigger-research-mwaddlink-pipeline-test research-mwaddlink-pipeline-test` to revert https://gerrit.wikimedia.org/r/c/integration/config/+/668199 for select jobs (T277236) [22:10:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:10:45] T277236: java.lang.RuntimeException: Invalid Credential: 'SONAR_API_KEY' - https://phabricator.wikimedia.org/T277236 [22:10:48] (03Merged) 10jenkins-bot: Revert "PipelineRunner: allowedCredentials" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/670963 (owner: 10Dduvall) [22:11:22] !log reverted https://gerrit.wikimedia.org/r/670963 and re-running failed job(T277236) [22:11:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:12:13] I think I need to convert the jsonSlurper output to a hashmap [22:13:41] oh yeah. that makes sense [22:16:07] you should be able to do `fooLazyMap as Map` [22:16:10] sorry for the trouble kostajh marxarell.i ! [22:16:42] no worries [22:16:53] i reviewed it :) [22:17:10] shared responsibility [22:21:24] I forget how to make a jenkins job non voting, any pointers? [22:22:08] addshore: you can tweak voting for jobs in integration/config/zuul/layout.yaml [22:22:18] (03PS5) 10Jeena Huneidi: Define and pass allowed credentials to pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/668199 (https://phabricator.wikimedia.org/T269900) [22:22:21] aaah, its in zuul! not on jjb! ty! [22:22:21] or rename the job to {name}-non-voting [22:22:26] i think thats the poke I needed [22:22:34] (03PS1) 10Addshore: Add php72-apache flavour to quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/670965 (https://phabricator.wikimedia.org/T276428) [22:22:34] https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/refs/heads/master/zuul/layout.yaml#20 [22:23:09] legoktm: oh neat. i didn't know that! https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/refs/heads/master/zuul/layout.yaml#801 [22:23:36] i like that convention [22:25:11] (03PS1) 10Addshore: non voting quibble-apache job for Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/670967 (https://phabricator.wikimedia.org/T276428) [22:25:49] * addshore would be super happy is someone would review this non voting job ;) [22:26:20] im excited as it potentially could speed up browser tests across the board if it all works out :D [22:26:28] (03CR) 10jerkins-bot: [V: 04-1] non voting quibble-apache job for Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/670967 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [22:26:33] 10Continuous-Integration-Config, 10Add-Link, 10Growth-Team, 10Release Pipeline: java.lang.RuntimeException: Invalid Credential: 'SONAR_API_KEY' - https://phabricator.wikimedia.org/T277236 (10dduvall) We hadn't yet merged a [[ https://gerrit.wikimedia.org/r/c/integration/config/+/668199 | patchset to integr... [22:26:33] oh :( [22:26:38] rejected! [22:26:42] ;_; [22:26:56] oh wait, thats to be expected, as it is in 2 parts, i always forget [22:27:14] :D [22:27:33] > 14:26:21 'non voting jobs are not allowed in gate-and-submit') [22:27:40] *facepalm* [22:27:53] (03CR) 10Legoktm: [C: 04-1] non voting quibble-apache job for Wikibase (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/670967 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [22:28:07] (03PS2) 10Addshore: non voting quibble-apache job for Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/670967 (https://phabricator.wikimedia.org/T276428) [22:29:00] postmerge? [22:29:35] (03CR) 10jerkins-bot: [V: 04-1] non voting quibble-apache job for Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/670967 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [22:35:10] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release Pipeline: Better way to restrict credentials available to pipelinelib - https://phabricator.wikimedia.org/T267699 (10jeena) [22:35:20] 10Release-Engineering-Team (Pipeline), 10Release Pipeline: Allow overrides to pipeline builder for passing allowed credentials list - https://phabricator.wikimedia.org/T269902 (10jeena) 05Resolved→03Open reverted due to a bug in the related integration/config patch [22:44:59] Jdlrobson do you have a minute to talk? [22:46:06] DannyS712: Not right now. I'm just stepping out for my afternoon walk. [22:46:14] okay. Enjoy [22:46:25] Apisandbox or global search out of interest? [22:46:44] Apisandbox - wondering how to move forward. I added two other reviewers who I believe are familiar with OOUI [22:46:52] i should have some time tomorrow if we want to prearrange a chat. [22:47:09] not sure about my time tomorrow, sorry. Its not urgent [22:48:35] ApiSandbox is one of those things that lacks a proper maintenance team. I really like what you're doing there but I don't know enough about the codebase to help you out quicker. I'm sorry it's taken so much time to get review for. I wonder if the platform team clinic duty could be used here to speed things up [22:48:43] anyway chat tomorrow or next week :) see ya in a bit [22:49:54] thanks [23:16:05] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.34 deployment blockers - https://phabricator.wikimedia.org/T274938 (10brennen)