[00:07:45] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: "MediaWiki 1.35 internal error" upon login - https://phabricator.wikimedia.org/T234491 (10Jdforrester-WMF) 05Open→03Resolved a:03Krenair All praise be unto Krenair. [00:24:28] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [00:37:40] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [00:53:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Fundraising-Backlog: Create composer-test-php70 docker image for fundraising tech's crm tests - https://phabricator.wikimedia.org/T230446 (10Ejegg) Yep, the payments clus... [01:53:05] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: Move CI selenium/qunit tests of mediawiki repository to a standalone job - https://phabricator.wikimedia.org/T232759 (10Krinkle) 05Open→03Resolved a:03hashar >>! In T232759#5535626, @ha... [01:53:07] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO: CI mediawiki/core run times have increased since July 26th, 2019 - https://phabricator.wikimedia.org/T232626 (10Krinkle) [01:53:12] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, and 5 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Krinkle) [01:54:24] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, and 4 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Krinkle) [03:11:28] 10Continuous-Integration-Config, 10HHVM: HHVM checks only happen on submit - https://phabricator.wikimedia.org/T226324 (10MaxSem) 05Open→03Invalid Now that we're not supporting HHVM anymore, this is moot. [03:17:53] (03CR) 10MaxSem: "That's it, MW doesn't support HHVM anymore." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/538726 (https://phabricator.wikimedia.org/T231710) (owner: 10MaxSem) [03:33:40] 10Scap (Scap3-MediaWiki-MVP), 10releng-201617-q4, 10scap2, 10Operations: Make scap able to depool/repool servers via the conftool API - https://phabricator.wikimedia.org/T104352 (10Krinkle) [03:34:19] 10Continuous-Integration-Infrastructure, 10HHVM: CI slaves should have HHVM call the exception user handler so we have useful stack trace on fatal errors - https://phabricator.wikimedia.org/T126473 (10Krinkle) 05Open→03Resolved a:03Krinkle This was fixed for both PHP and HHVM but forgot to close the tas... [04:52:43] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10MediaWiki-Core-Testing, 10Browser-Tests, and 3 others: Make MediaWiki Wdio tests less slow (Sept 2019) - https://phabricator.wikimedia.org/T234002 (10Krinkle) [05:51:27] 10Deployments, 10MediaWiki-SWAT-deployments: Figure out what to do with `fatalmonitor` script - https://phabricator.wikimedia.org/T234345 (10Joe) A few observations: - We have logstash dashboards that should help; however, sometimes logstash is hosed and lagging behind more than the on-disk counterpart - If we... [06:08:06] (03CR) 10Daimona Eaytoy: [C: 03+1] "> That's it, MW doesn't support HHVM anymore." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/538726 (https://phabricator.wikimedia.org/T231710) (owner: 10MaxSem) [08:02:55] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Developer-Advocacy, 10wikimedia.biterg.io: biterg.io Gerrit crawling probably stresses the server too much - https://phabricator.wikimedia.org/T234328 (10hashar) Indeed https://gerrit.wikimedia.org/r/... [08:17:46] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Wikimedia-Logstash, and 3 others: Look into shoving gerrit logs into logstash - https://phabricator.wikimedia.org/T141324 (10hashar) It seems the message parsing has been enhanced recently. At least on Sept 23rd... [08:32:25] 10Project-Admins: Archive HHVM project - https://phabricator.wikimedia.org/T234512 (10Bugreporter) [09:23:16] 10Project-Admins: Archive HHVM project - https://phabricator.wikimedia.org/T234512 (10Peachey88) 05Open→03Stalled Stalled till parent task is resolved. [09:23:30] 10Project-Admins: Archive HHVM project - https://phabricator.wikimedia.org/T234512 (10Peachey88) [09:27:01] 10Project-Admins: Archive HHVM project - https://phabricator.wikimedia.org/T234512 (10Aklapper) p:05Triage→03Lowest [10:35:57] (03PS1) 10Daimona Eaytoy: Release 27.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/540575 [10:37:46] 10Phabricator: Herald rule: KaiOS -> Inuka - https://phabricator.wikimedia.org/T234217 (10SBisson) @Aklapper I think I would like it to behave like H26 so it doesn't have anything to do if "#Inuka-Team (Kanban)" is already applied. [11:07:22] !log jenkins: removing global environment variable: HHVM_REPO_CENTRAL_PATH=$WORKSPACE/central.hhbc | https://integration.wikimedia.org/ci/configure | T234384 [11:07:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:07:26] T234384: Drop HHVM from CI - https://phabricator.wikimedia.org/T234384 [11:09:03] !log integration-castor03: pruning HHVM jobs caches: rm -fR /srv/jenkins-workspace/caches/*/*/*hhvm* # T234384 [11:09:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:35:00] (03CR) 10Hashar: "The Quibble local_settings.php file is prepended to LocalSettings.php in order to have the settings taken in account by extension/skins." [integration/quibble] - 10https://gerrit.wikimedia.org/r/540118 (https://phabricator.wikimedia.org/T190829) (owner: 10Awight) [11:45:39] (03CR) 10Hashar: "That highlights a potential issue with --skip-deps I am find it confusing since 'npm install' will still be run by other commands when run" [integration/quibble] - 10https://gerrit.wikimedia.org/r/540387 (https://phabricator.wikimedia.org/T225008) (owner: 10Awight) [11:46:26] (03CR) 10Hashar: [C: 04-1] "Haven't tested, but pretty sure wgEnableUpload will not be enabled. Probably can be confirmed by running something like:" [integration/quibble] - 10https://gerrit.wikimedia.org/r/540118 (https://phabricator.wikimedia.org/T190829) (owner: 10Awight) [11:54:45] (03CR) 10Hashar: Log command versions (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/540413 (https://phabricator.wikimedia.org/T181942) (owner: 10Awight) [12:37:41] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Quibble, and 3 others: CI: Create a way to share a secret between MediaWiki and the testing framework. - https://phabricator.wikimedia.org/T233092 (10LarsWirzenius) I continue to comment... [13:09:21] 10Gerrit: Lots of "Skipping change xxx because the corresponding repository was not found" in the logs - https://phabricator.wikimedia.org/T233989 (10hashar) > 07:00 gerrit: forcing reindex of changes # T233989 My hope was to have the complete reindexing of changes to delete the missing repositories... [13:13:14] 10Gerrit: Lots of "Skipping change xxx because the corresponding repository was not found" in the logs - https://phabricator.wikimedia.org/T233989 (10hashar) The error is triggered when doing a query which is logged as well: ` [2019-10-01 13:42:27,917] [SSH gerrit query limit:10 (status:open OR status:closed) AN... [13:15:31] 10Gerrit: Lots of "Skipping change xxx because the corresponding repository was not found" in the logs - https://phabricator.wikimedia.org/T233989 (10thcipriani) >>! In T233989#5543982, @hashar wrote: > The error is triggered when doing a query which is logged as well: > ` > [2019-10-01 13:42:27,917] [SSH gerrit... [13:22:11] oh ym god [13:22:15] gerrit dieing again [13:22:19] will try to take some traces [13:24:49] PROBLEM - Gerrit JSON on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Gerrit%23Monitoring [13:24:53] PROBLEM - Gerrit Health Check on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [13:29:39] RECOVERY - Gerrit JSON on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 26094 bytes in 0.066 second response time https://wikitech.wikimedia.org/wiki/Gerrit%23Monitoring [13:29:45] RECOVERY - Gerrit Health Check on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 865 bytes in 0.069 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [13:30:01] Project beta-code-update-eqiad build #266325: 04FAILURE in 7 min 0 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/266325/ [13:32:06] PROBLEM - Puppet errors on integration-agent-stretch-1001 is CRITICAL: CRITICAL: 4.49% of data above the critical threshold [3.0] [13:34:25] Yippee, build fixed! [13:34:26] Project beta-code-update-eqiad build #266326: 09FIXED in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/266326/ [13:41:13] 10Gerrit: Lots of "Skipping change xxx because the corresponding repository was not found" in the logs - https://phabricator.wikimedia.org/T233989 (10hashar) I tried creating a couple of missing repositories `maps/kartotherian` and `maps/tilerator`. The warning then disappear, but when I delete the repositories... [13:50:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10serviceops-radar: Proposal: simplify set up of basic CI jobs for new projects - https://phabricator.wikimedia.org/T233298 (10LarsWirzenius) I too like this proposal, and as Tyler said, the plans for future CI inclu... [13:56:28] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10observability, 10serviceops-radar: Gerrit: Alert when deadlock occur ahead of Gerrit being unresponsive - https://phabricator.wikimedia.org/T230138 (10hashar) 05Invalid→03Open [13:56:32] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10serviceops-radar, 10Upstream: Gerrit account cache has a faulty reentrant lock causing http/sendemail threads to stall completely - https://phabricator.wikimedia.org/T224448 (10hashar) [13:56:34] hashar: oh, I thought you reindexed changes? [13:56:35] Didn’t realise you created the repo after [13:57:20] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10observability, 10serviceops-radar: Gerrit: Alert when deadlock occur ahead of Gerrit being unresponsive - https://phabricator.wikimedia.org/T230138 (10hashar) From discussion with Tyler I have reopene... [13:58:22] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10observability, 10serviceops-radar: Gerrit: Alert when deadlock occur ahead of Gerrit being unresponsive - https://phabricator.wikimedia.org/T230138 (10elukey) We keep restarting Gerrit after it gets u... [14:01:22] hashar: I filed https://bugs.chromium.org/p/gerrit/issues/detail?id=11650&q=&colspec=ID%20Opened%20Modified%20Type%20Stars%20Status%20Priority%20Triaged%20Reporter%20Owner%20Summary [14:11:51] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10serviceops-radar, 10Upstream: Gerrit account cache has a faulty reentrant lock causing http/sendemail threads to stall completely - https://phabricator.wikimedia.org/T224448 (10hashar) Traces from today incide... [14:17:13] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10observability, 10serviceops-radar: Gerrit: Alert when deadlock occur ahead of Gerrit being unresponsive - https://phabricator.wikimedia.org/T230138 (10hashar) We do have the alarms for HTTP/SSH. The... [14:17:39] paladox: thank you :] [14:17:42] I am off! [14:17:57] :) [14:47:44] (03CR) 10Awight: Log command versions (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/540413 (https://phabricator.wikimedia.org/T181942) (owner: 10Awight) [15:03:35] 10Gerrit, 10Release-Engineering-Team-TODO: gerrit shows merged patch as pending - https://phabricator.wikimedia.org/T234533 (10CDanis) [15:03:45] 10Gerrit, 10Release-Engineering-Team-TODO: gerrit shows merged patch as pending - https://phabricator.wikimedia.org/T234533 (10CDanis) p:05Triage→03High [15:05:03] (03PS2) 10Awight: Log command versions [integration/quibble] - 10https://gerrit.wikimedia.org/r/540413 (https://phabricator.wikimedia.org/T181942) [15:05:52] (03CR) 10jerkins-bot: [V: 04-1] Log command versions [integration/quibble] - 10https://gerrit.wikimedia.org/r/540413 (https://phabricator.wikimedia.org/T181942) (owner: 10Awight) [15:07:36] (03PS1) 10Ayounsi: Modify access rules [homer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/540614 [15:08:04] (03PS3) 10Awight: Log command versions [integration/quibble] - 10https://gerrit.wikimedia.org/r/540413 (https://phabricator.wikimedia.org/T181942) [15:20:15] 10Gerrit, 10Release-Engineering-Team-TODO: gerrit shows merged patch as pending - https://phabricator.wikimedia.org/T234533 (10thcipriani) My question is: how was this merged? Gerrit doesn't seem to think it was merged through the gerrit interface. I get a 404 on the page for https://gerrit.wikimedia.org/r/ch... [15:26:05] 10Gerrit, 10Release-Engineering-Team-TODO: gerrit shows merged patch as pending - https://phabricator.wikimedia.org/T234533 (10Marostegui) >>! In T234533#5544167, @thcipriani wrote: > My question is: how was this merged? Gerrit doesn't seem to think it was merged through the gerrit interface. > > I get a 404... [15:31:01] Well mysterious behaviour [15:31:17] Thcipriani: gitiles shows no contents in the commit like gerrit [15:33:03] 10Gerrit, 10Release-Engineering-Team-TODO: gerrit shows merged patch as pending - https://phabricator.wikimedia.org/T234533 (10thcipriani) hrm, yes, I see this in the review notes as well: ` commit 8bcb37652c245276d023720e3cac27609d8cd1b9 Author: Manuel Arostegui Date: Wed Sep 4 1... [15:33:05] The only way I could think is if it was pushed directly? [15:33:19] Oh [15:34:50] that's what I thought too [15:34:54] but evidently not [15:35:04] and git notes was updated [15:35:05] 10Gerrit, 10Release-Engineering-Team-TODO: gerrit shows merged patch as pending - https://phabricator.wikimedia.org/T234533 (10Paladox) The url is different in that @thcipriani (Reviewed-on: https://gerrit.wikimedia.org/r/534392) compared to the url in description https://gerrit.wikimedia.org/r/c/operations/pu... [15:36:08] 10Gerrit, 10Release-Engineering-Team-TODO: gerrit shows merged patch as pending - https://phabricator.wikimedia.org/T234533 (10Marostegui) The URL https://gerrit.wikimedia.org/r/c/operations/puppet/+/534389 is the one that just showed up on my Gerrit "Outgoing reviews", with that same patch that was merged a m... [15:37:22] 10Gerrit, 10Release-Engineering-Team-TODO: gerrit shows merged patch as pending - https://phabricator.wikimedia.org/T234533 (10thcipriani) >>! In T234533#5544222, @Paladox wrote: > The url is different in that @thcipriani (Reviewed-on: https://gerrit.wikimedia.org/r/534392) compared to the url in description h... [15:41:38] 10Phabricator: Reset 2FA for Phabricator account `Apap04` - https://phabricator.wikimedia.org/T234233 (10Aklapper) @Apap04: Please explain where there is already information available about the "real life identity" of https://www.mediawiki.org/w/index.php?title=User:Apap04 , to compare with. [15:48:18] I wonder if this showed up after the restart? [15:48:19] thcipriani: ^ [15:50:06] 10Gerrit, 10Release-Engineering-Team-TODO: gerrit shows merged patch as pending - https://phabricator.wikimedia.org/T234533 (10thcipriani) OK. So this change was submitted twice. * https://gerrit.wikimedia.org/r/534392 at 3:15 * https://gerrit.wikimedia.org/r/534389 at 3:12 The diffs are identical, as are th... [15:50:54] 10Gerrit, 10Release-Engineering-Team-TODO: gerrit shows merged patch as pending - https://phabricator.wikimedia.org/T234533 (10thcipriani) >>! In T234533#5544228, @Marostegui wrote: > The URL https://gerrit.wikimedia.org/r/c/operations/puppet/+/534389 is the one that just showed up on my Gerrit "Outgoing revie... [15:51:16] 10Gerrit, 10Release-Engineering-Team-TODO: gerrit shows merged patch as pending - https://phabricator.wikimedia.org/T234533 (10Marostegui) 3:12 vs 3:15? I only hit a rebase today, I didn't merge anything [15:51:40] 10Gerrit, 10Release-Engineering-Team-TODO: gerrit shows merged patch as pending - https://phabricator.wikimedia.org/T234533 (10Marostegui) >>! In T234533#5544293, @thcipriani wrote: >>>! In T234533#5544228, @Marostegui wrote: >> The URL https://gerrit.wikimedia.org/r/c/operations/puppet/+/534389 is the one tha... [15:52:05] Well [15:52:15] I think it’s the index I guess then? [15:52:21] Maybe some race condition? [15:52:32] both is my guess [15:52:54] I can say that the commit doesn't exist as a loose object [15:54:47] Ok [15:54:48] Should we try a reindex? Though I’m guessing that may not fix it as it’s gone and created a new change [15:54:49] But it may show it as merged [15:55:13] it's on disk so a reindex wouldn't change anything I think [15:55:34] (since that would just re-read from the canonical source: the disk) [15:55:40] 10Release-Engineering-Team-TODO (201910): Set up an Argo instance - https://phabricator.wikimedia.org/T234542 (10LarsWirzenius) [15:56:52] Ah ok [16:03:12] 10Release-Engineering-Team-TODO (201910): Learn Go - https://phabricator.wikimedia.org/T234543 (10LarsWirzenius) [16:34:12] 10Gerrit, 10Release-Engineering-Team-TODO: gerrit shows merged patch as pending - https://phabricator.wikimedia.org/T234533 (10thcipriani) the change numbers should increase, i.e., 534389 was pushed up before 534392. This is supported by the git timestamps for the changes first being created in Gerrit: * 5343... [16:36:28] 10Gerrit, 10Release-Engineering-Team-TODO: gerrit shows merged patch as pending - https://phabricator.wikimedia.org/T234533 (10thcipriani) >>! In T234533#5544444, @thcipriani wrote: > The fix is: we should just delete the unmerged change from disk. This will trigger a reindex. Clarification: delete the unmerg... [16:50:01] 10Release-Engineering-Team, 10Contributors-Team, 10Epic, 10Readers-Web-Backlog (Tracking), and 2 others: [EPIC] Wikimedia should use a standard set of tools for managing code quality - https://phabricator.wikimedia.org/T111396 (10Jdlrobson) [17:17:18] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10MediaWiki-Core-Testing, 10Browser-Tests, and 3 others: Make MediaWiki Wdio tests less slow (Sept 2019) - https://phabricator.wikimedia.org/T234002 (10Krinkle) [17:44:35] 10Gerrit, 10Release-Engineering-Team-TODO: gerrit shows merged patch as pending - https://phabricator.wikimedia.org/T234533 (10thcipriani) 05Open→03Resolved a:03thcipriani >>! In T234533#5544460, @thcipriani wrote: >>>! In T234533#5544444, @thcipriani wrote: >> The fix is: we should just delete the unmer... [18:12:11] (03CR) 10Jforrester: [C: 03+2] "This'll give us a nice clean base-line. Thank you." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/540575 (owner: 10Daimona Eaytoy) [18:12:37] (03CR) 10Jforrester: Remove prohibitions on new PHP features (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/538726 (https://phabricator.wikimedia.org/T231710) (owner: 10MaxSem) [18:13:16] (03CR) 10Daimona Eaytoy: [C: 03+1] Remove prohibitions on new PHP features (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/538726 (https://phabricator.wikimedia.org/T231710) (owner: 10MaxSem) [18:13:28] (03Merged) 10jenkins-bot: Release 27.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/540575 (owner: 10Daimona Eaytoy) [18:15:40] (03CR) 10Daimona Eaytoy: [C: 03+2] Remove prohibitions on new PHP features [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/538726 (https://phabricator.wikimedia.org/T231710) (owner: 10MaxSem) [18:16:23] (03PS1) 10Daimona Eaytoy: Require PHPUnit 6+ [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/540646 [18:17:54] (03PS3) 10Daimona Eaytoy: Remove prohibitions on new PHP features [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/538726 (https://phabricator.wikimedia.org/T231710) (owner: 10MaxSem) [18:18:11] (03CR) 10Jforrester: [C: 03+2] Require PHPUnit 6+ [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/540646 (owner: 10Daimona Eaytoy) [18:18:18] (03CR) 10Daimona Eaytoy: [C: 03+2] "> Patch Set 3: Patch Set 2 was rebased" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/538726 (https://phabricator.wikimedia.org/T231710) (owner: 10MaxSem) [18:18:30] (03PS11) 10Daimona Eaytoy: Improve checks for variargs [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539177 (https://phabricator.wikimedia.org/T231710) [18:18:38] (03PS7) 10Daimona Eaytoy: Add rules for PHP71 nullable types [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539308 (https://phabricator.wikimedia.org/T231710) [18:18:52] (03Merged) 10jenkins-bot: Require PHPUnit 6+ [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/540646 (owner: 10Daimona Eaytoy) [18:18:59] James_F: would you have time to adjust CI for the codesniffer repo? [18:19:13] Sure, what does it need? [18:19:35] Dropping PHP<7.2 [18:19:52] Unless it was covered by the big HHVM drop [18:20:17] (03PS1) 10Daimona Eaytoy: Require PHP 7.2+ in composer.json [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/540648 [18:20:23] (03PS2) 10Daimona Eaytoy: Require PHP 7.2+ in composer.json [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/540648 [18:20:29] It wasn't, one sec. [18:20:33] Sure, ty [18:20:46] (03PS4) 10Daimona Eaytoy: Don't suggest to use a temporary variable for &$this [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539529 [18:20:48] (03PS1) 10Jforrester: layout: [mediawiki/tools/codesniffer] Move to PHP7.2+ [integration/config] - 10https://gerrit.wikimedia.org/r/540649 [18:21:03] (03CR) 10Jforrester: [C: 03+2] layout: [mediawiki/tools/codesniffer] Move to PHP7.2+ [integration/config] - 10https://gerrit.wikimedia.org/r/540649 (owner: 10Jforrester) [18:21:05] (03PS4) 10Daimona Eaytoy: Remove prohibitions on new PHP features [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/538726 (https://phabricator.wikimedia.org/T231710) (owner: 10MaxSem) [18:21:12] (03CR) 10Daimona Eaytoy: [C: 03+2] "> Patch Set 4: Patch Set 3 was rebased" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/538726 (https://phabricator.wikimedia.org/T231710) (owner: 10MaxSem) [18:21:26] (03CR) 10jerkins-bot: [V: 04-1] Require PHP 7.2+ in composer.json [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/540648 (owner: 10Daimona Eaytoy) [18:22:04] (03Merged) 10jenkins-bot: Remove prohibitions on new PHP features [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/538726 (https://phabricator.wikimedia.org/T231710) (owner: 10MaxSem) [18:22:23] (03CR) 10Jforrester: "Won't this whine when running composer on php70 in phan? Or do we skip that there? See T234062." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/540648 (owner: 10Daimona Eaytoy) [18:22:44] (03Merged) 10jenkins-bot: layout: [mediawiki/tools/codesniffer] Move to PHP7.2+ [integration/config] - 10https://gerrit.wikimedia.org/r/540649 (owner: 10Jforrester) [18:22:54] (03CR) 10jenkins-bot: Release 27.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/540575 (owner: 10Daimona Eaytoy) [18:23:20] (03CR) 10jenkins-bot: Require PHPUnit 6+ [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/540646 (owner: 10Daimona Eaytoy) [18:23:57] (03CR) 10Daimona Eaytoy: "> Won't this whine when running composer on php70 in phan? Or do we" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/540648 (owner: 10Daimona Eaytoy) [18:24:03] (03CR) 10jerkins-bot: [V: 04-1] Add rules for PHP71 nullable types [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539308 (https://phabricator.wikimedia.org/T231710) (owner: 10Daimona Eaytoy) [18:24:24] (03CR) 10jenkins-bot: Remove prohibitions on new PHP features [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/538726 (https://phabricator.wikimedia.org/T231710) (owner: 10MaxSem) [18:25:30] Daimona: Your optimism bias is showing. ;-) [18:25:55] !log Zuul: [mediawiki/tools/codesniffer] Move to PHP7.2+ [18:25:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:25:59] Heh, I'm an eternal optimist :) [18:26:11] (03PS3) 10Daimona Eaytoy: Require PHP 7.2+ in composer.json [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/540648 [18:26:27] (03CR) 10Daimona Eaytoy: "recheck" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539308 (https://phabricator.wikimedia.org/T231710) (owner: 10Daimona Eaytoy) [18:26:29] hello releng, any idea why I can't submit https://gerrit.wikimedia.org/r/c/operations/homer/public/+/540619 ? [18:27:23] XioNoX you need to grant "submit" [18:27:24] XioNoX: Access is locked down at https://gerrit.wikimedia.org/r/admin/projects/operations/homer/public,access [18:27:24] in the acl [18:27:26] * paladox does [18:27:30] And it's… yeah. [18:27:37] It's so locked down it's un-submittable. [18:27:40] (03PS1) 10Paladox: Modify access rules [homer/public] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/540651 [18:27:45] XioNoX ^ [18:27:55] thanks [18:28:06] what should I change? [18:28:17] Merge that patch. [18:28:19] ^ [18:28:23] (c+2 and v+2) [18:29:07] Perl: Everything's a string. Java: Everthing's an Exception. Gerrit: Everything's a git change. [18:29:16] heh [18:29:27] I don't understand [18:29:38] XioNoX https://gerrit.wikimedia.org/r/540651 [18:29:43] i guess you ignore wikibugs? [18:30:34] but this one https://gerrit.wikimedia.org/r/admin/projects/operations/homer/mock-private,access has the same permissions and I was able to submit it [18:30:49] that's because [18:30:52] it inherit's from https://gerrit.wikimedia.org/r/#/admin/projects/labs/private,access [18:30:57] which grants the submit [18:30:58] Yeah. [18:31:10] ah [18:31:14] Maybe it shouldn't inherit from there? Different ACL tree… [18:31:33] XioNoX maybe we should grant submit to https://gerrit.wikimedia.org/r/#/admin/projects/operations/homer,access [18:31:42] and change the inheritency of https://gerrit.wikimedia.org/r/admin/projects/operations/homer/mock-private,access [18:31:52] paladox: yeah that looks better [18:32:11] and delete the groups `operations-homer` and similar [18:32:14] (03PS1) 10Paladox: Modify access rules [homer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/540652 [18:32:18] XioNoX https://gerrit.wikimedia.org/r/#/c/operations/homer/+/540652/ [18:32:24] oh [18:32:33] XioNoX should we change it to ldap/ops? [18:32:42] yeah, ldap/ops [18:33:09] ok [18:33:16] (03Abandoned) 10Paladox: Modify access rules [homer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/540652 (owner: 10Paladox) [18:33:40] (03PS1) 10Paladox: Modify access rules [homer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/540653 [18:33:43] XioNoX https://gerrit.wikimedia.org/r/#/c/operations/homer/+/540653/ [18:37:46] (03CR) 10Jforrester: [C: 03+2] Require PHP 7.2+ in composer.json [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/540648 (owner: 10Daimona Eaytoy) [18:38:26] (03Merged) 10jenkins-bot: Require PHP 7.2+ in composer.json [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/540648 (owner: 10Daimona Eaytoy) [18:39:00] (03CR) 10Ayounsi: [C: 03+1] Modify access rules [homer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/540653 (owner: 10Paladox) [18:39:06] paladox: LGTM, thx! [18:39:15] XioNoX you should be able to merge that :) [18:39:37] paladox: nah I can't even do +2 [18:39:38] oh [18:39:39] https://gerrit.wikimedia.org/r/#/admin/groups/1671,members [18:39:47] XioNoX i have +2 but cannot submit :P [18:39:56] thcipriani around? :) [18:39:58] this is funny [18:40:17] (03CR) 10jenkins-bot: Require PHP 7.2+ in composer.json [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/540648 (owner: 10Daimona Eaytoy) [18:41:14] * thcipriani looks [18:41:46] (03CR) 10Jforrester: [C: 03+1] Add rules for PHP71 nullable types [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539308 (https://phabricator.wikimedia.org/T231710) (owner: 10Daimona Eaytoy) [18:41:53] thcipriani thanks! [18:42:01] indeed, thanks :) [18:42:09] also all those groups can be deleted: https://gerrit.wikimedia.org/r/admin/groups/q/filter:homer [18:42:25] as permissions should be granted to ldap/ops [18:42:51] groups carn't be deleted yet [18:42:58] but upstream have recently kick started that [18:43:09] (03CR) 10Thcipriani: [V: 03+2 C: 03+2] Modify access rules [homer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/540653 (owner: 10Paladox) [18:43:11] by recently i mean this week :P [18:43:20] done [18:43:28] (03Abandoned) 10Paladox: Modify access rules [homer/public] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/540651 (owner: 10Paladox) [18:43:45] XioNoX you should have submit now [18:44:10] indeed! [18:44:12] thx! [18:49:00] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Developer-Advocacy, 10wikimedia.biterg.io: biterg.io Gerrit crawling probably stresses the server too much - https://phabricator.wikimedia.org/T234328 (10Aklapper) Bitergia currently run into: ` $ ssh... [18:51:54] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10serviceops-radar, 10Upstream: Gerrit account cache has a faulty reentrant lock causing http/sendemail threads to stall completely - https://phabricator.wikimedia.org/T224448 (10hashar) I have found a similar b... [18:55:24] 10Release-Engineering-Team, 10MediaWiki-User-management, 10MediaWiki-extensions-FlaggedRevs, 10User-DannyS712: Pending changes: autoreview randomly fails - https://phabricator.wikimedia.org/T233561 (10Neolexx) [[https://ru.wikipedia.org/wiki/Википедия:Форум/Технический#Идея_Neolex|I got an idea]] of what m... [19:07:50] (03PS1) 10Jforrester: jjb: Provide wikimedia-fundraising-crm-composer-php70-docker [integration/config] - 10https://gerrit.wikimedia.org/r/540662 (https://phabricator.wikimedia.org/T230446) [19:07:52] (03PS1) 10Jforrester: layout: [wikimedia/fundraising/crm] Add …-composer-php70-docker as experimental [integration/config] - 10https://gerrit.wikimedia.org/r/540663 (https://phabricator.wikimedia.org/T230446) [19:07:54] (03PS1) 10Jforrester: layout: [wikimedia/fundraising/crm] Make …-composer-php70-docker a full job [integration/config] - 10https://gerrit.wikimedia.org/r/540664 [19:08:35] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/540662 (https://phabricator.wikimedia.org/T230446) (owner: 10Jforrester) [19:10:19] (03CR) 10Jforrester: [C: 03+2] layout: [wikimedia/fundraising/crm] Add …-composer-php70-docker as experimental [integration/config] - 10https://gerrit.wikimedia.org/r/540663 (https://phabricator.wikimedia.org/T230446) (owner: 10Jforrester) [19:10:56] (03Merged) 10jenkins-bot: jjb: Provide wikimedia-fundraising-crm-composer-php70-docker [integration/config] - 10https://gerrit.wikimedia.org/r/540662 (https://phabricator.wikimedia.org/T230446) (owner: 10Jforrester) [19:12:17] (03Merged) 10jenkins-bot: layout: [wikimedia/fundraising/crm] Add …-composer-php70-docker as experimental [integration/config] - 10https://gerrit.wikimedia.org/r/540663 (https://phabricator.wikimedia.org/T230446) (owner: 10Jforrester) [19:12:51] !log [wikimedia/fundraising/crm] AAdd experimental php70 job T230446 [19:12:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:12:54] T230446: Create composer-test-php70 docker image for fundraising tech's crm tests - https://phabricator.wikimedia.org/T230446 [19:13:19] double AA :P [19:13:26] Oops. [19:32:00] 10Release-Engineering-Team (Deployment services), 10Release, 10Train Deployments: 1.34.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T220750 (10dduvall) 05Open→03Resolved [19:42:58] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10Performance-Team, and 3 others: Usage instructions in tests/selenium/README.md are confusing - https://phabricator.wikimedia.org/T214708 (10Krinkle) 05Open→03Resolved a:03Kri... [19:43:03] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10MediaWiki-Core-Testing, 10Browser-Tests, and 3 others: Make MediaWiki Wdio tests less slow (Sept 2019) - https://phabricator.wikimedia.org/T234002 (10Krinkle) [19:44:49] thcipriani: paladox so the Gerrit sendEmail of doom [19:44:53] I don't have any theory really :-\ [19:44:56] yup [19:45:09] ^ [19:45:11] beside there might be a racecondition somewhere when sending emails in parallel maybe [19:45:33] i'm hopping gerrit1001 will fix this [19:45:40] but I could not find out when the issue started first [19:46:11] I think it's more likely that there's some misuse of guava's concurrent hashmap for the gerrit account cache and that we hit it because we use ldap while upstream uses oauth [19:46:11] I was looking for a held lock [19:46:14] hashar when "sendEmail" started first hapening today or when it first started happening in general? [19:46:27] but now I am more tempted to think that two sendEmail threads end up using the same object [19:46:30] such as the queue of tasks to do [19:46:37] and would end up blocking on each otehr as a result [19:46:41] eh, we had this problem when we only had 1 sendemail thread [19:46:46] paladox: when that first started in general [19:46:54] ah :-\ [19:46:56] hashar i firsted noticed it earlier this year [19:47:00] we added another thread in a failed attempt to stop this issue :) [19:47:02] january [19:47:06] cause T224448 got filled end of may [19:47:07] T224448: Gerrit account cache has a faulty reentrant lock causing http/sendemail threads to stall completely - https://phabricator.wikimedia.org/T224448 [19:47:14] january is the first time I saw it as well. [19:47:16] it was around the time of the all-hands. [19:47:22] and we bumped sendEmail threads to 2 early in may ( https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/507852/ ) [19:47:24] ok ok [19:47:28] :-\\\ [19:47:46] so well hmm [19:48:00] I guess we need a core dump or whatever is the equivalent in the java world [19:48:06] i raised that hopping it would fix the threads issue [19:48:14] in the hope that maybe we can find which object is used twice / the lock [19:49:13] https://github.com/google/guava/issues/3602#issuecomment-530457714 seems like the most relevant comment I've seen anywhere upstream [19:50:00] tl;dr: buried somewhere deep in the depths of gerrit core there be dragons [19:51:22] lol [19:51:27] thcipriani yup [19:51:35] i think caffeine exposed that [19:52:01] someone, somewhere said they fixed the issue by replacing Guava with Caffine [19:52:03] since merging caffeine up to 2.15, upstream found some issue with some patch thing (i forgot the name of) [19:52:04] Paladox filled a task to guava [19:52:13] thcipriani yup [19:52:16] but someone replied with an answer I cant even understand [19:52:25] marco from ericson said they still have that running [19:52:29] it like being an ant trying to understand what an airplane is [19:52:43] cafeine [19:52:48] (ah that was the comment Tyler pointed at) [19:52:49] :( [19:52:50] heh [19:52:59] it'd be nice if marco would upstream that :) [19:53:12] they did [19:53:17] but was reverted [19:53:25] they need to look at porting this to 2.15 [19:53:33] fixing the issues that caused them to revert it [19:53:42] (so they are basically running a forked 2.14 site) [19:53:42] and https://github.com/google/guava/issues/2743#issuecomment-303643038 by same person also states ConcurrentHashMap can livelock under jdk8 :\ [19:54:23] paladox: Ericson effectively hold upgrading due to that one? [19:54:24] so java 11 looks like it may fix it? [19:54:34] i guess so [19:54:45] * paladox finds the conversation [19:55:08] marco: "Ericsson has a fork that included replacing guava with caffeine, AFAIK. I will need to revisit that whole fork though, soon or before upgrading to 2.15 (from 2.14). I have to review that fork myself first, as it was mostly built while I was away from the team." [19:55:24] ben-manes seems to have a pretty deep understand threadsafe data structures/be the person everyone defers to in these matter/says lots of smart things I don't quite grasp [19:55:51] that's probably why cafeine is the better choice :P [19:55:58] hmm [19:56:57] now i realize I am not smart AT ALL [19:56:59] re:caffeine -- yeah, probably [19:57:28] also, I guess, google folks say it's "better maintained" than guava: https://github.com/google/guava/issues/3602#issuecomment-530912005 [19:57:55] which is not very comforting, really :) [19:58:28] so in short, the root cause is way above our league [19:58:42] but migrating to caffeine might be something we can push for possibly with help of Ericsson ? [19:59:53] that's been the conclusion I've come to: I can't fix it, someone has a fix, it'll still require a lot of work to get someone else to fix it [19:59:58] hashar thcipriani i just wrote https://github.com/google/guava/issues/3602#issuecomment-538104189 [20:00:28] * paladox pushing for him to go with caffeine [20:01:00] Caffeine might well be the best course of action yeah [20:01:11] I will reply on the task to thank the people that quickly triaged the email [20:01:16] s/the email/the issue/ [20:01:36] it's also faster so performance improvements too [20:01:39] and then I guess we can propose to dismiss that issue? [20:01:44] in favor of a Caffeine migration [20:01:53] then 2.15 is EOL so .. [20:02:15] yup [20:09:35] paladox: thcipriani agreement to close this? [20:09:40] I mean [20:09:46] to close the upstream task to Guava? [20:09:58] I'll close it in a few, waiting for a reply by him. [20:10:25] ok [20:10:26] Since if we can do a workaround, around https://github.com/GerritCodeReview/gerrit/blob/stable-2.15/gerrit-server/src/main/java/com/google/gerrit/server/account/AccountCacheImpl.java#L85 [20:10:28] replying as well [20:10:32] then that'll be awesome. [20:12:07] https://github.com/google/guava/issues/3602#issuecomment-538108956 :] [20:12:10] my reply [20:13:29] thanks! [20:17:35] hashar thcipriani https://github.com/google/guava/issues/3602#issuecomment-538110691 [20:18:51] paladox: well I don't think they have much willingness to debug / code for Gerrit :-] [20:18:58] yup [20:19:13] but his knowledge may say that will help prevent a deadlock [20:19:16] he says: [20:19:23] / Avoid locking on a cache hit [20:19:23] V value = cache.getIfPresent(key); [20:19:23] if (value != null) { [20:19:23] return value; [20:19:24] } [20:20:12] in the detailed comment he left [20:21:18] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10serviceops-radar, 10Upstream: Gerrit account cache has a faulty reentrant lock causing http/sendemail threads to stall completely - https://phabricator.wikimedia.org/T224448 (10hashar) After a chat with @Palad... [20:22:11] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10serviceops-radar, 10Upstream: Gerrit account cache has a faulty reentrant lock causing http/sendemail threads to stall completely - https://phabricator.wikimedia.org/T224448 (10hashar) [20:22:27] with https://gerrit-review.googlesource.com/c/gerrit/+/154130 being the nicest entry point we have [20:22:35] links to various issues [20:23:03] paladox: maybe that could work. I don't know really :-] [20:23:13] hashar it definitly worked! [20:23:17] at least for ericson [20:23:37] I thought Ericson was stuck to 2.14 [20:23:46] it is [20:23:56] but have caffeine on 2.14 [20:25:49] I wish I could subscribe to ben-manes comments. They are nice "In LRU every read is a write by mutating a doubly-linked list" [20:25:59] remembers me of university courses [20:26:09] trying to parse what the teachers were saying :-^ [20:26:16] heh [20:39:22] (03PS2) 10Jforrester: layout: [wikimedia/fundraising/crm] Make …-composer-php70-docker a full job [integration/config] - 10https://gerrit.wikimedia.org/r/540664 (https://phabricator.wikimedia.org/T230446) [20:39:26] (03PS1) 10Jforrester: dockerfiles: Rename php to php70 and add intl [integration/config] - 10https://gerrit.wikimedia.org/r/540681 (https://phabricator.wikimedia.org/T230446) [20:39:28] (03PS1) 10Jforrester: dockerfiles: Bump things over to use php70 which adds intl [integration/config] - 10https://gerrit.wikimedia.org/r/540682 (https://phabricator.wikimedia.org/T230446) [20:39:30] (03PS1) 10Jforrester: jjb: Move jobs over to php70-inherited images [integration/config] - 10https://gerrit.wikimedia.org/r/540683 (https://phabricator.wikimedia.org/T230446) [20:40:07] well I am done for tonight ;] *wave* [20:40:11] and thank you paladox ! [20:40:21] :) [20:44:30] ohhh [20:44:38] heh hashar left just as ben wrote his comment [20:44:41] thcipriani https://github.com/google/guava/issues/3602#issuecomment-538119157 [20:44:45] we should try that [20:49:28] 10Release-Engineering-Team, 10MediaWiki-User-management, 10MediaWiki-extensions-FlaggedRevs, 10User-DannyS712: Pending changes: autoreview randomly fails - https://phabricator.wikimedia.org/T233561 (10Ladsgroup) FlaggedRevs currently doesn't have a clear maintainer (T185664) making things even more complic... [20:49:58] 10Release-Engineering-Team (Code Health), 10Code-Stewardship-Reviews, 10MediaWiki-extensions-FlaggedRevs: FlaggedRevs: code stewardship review - https://phabricator.wikimedia.org/T185664 (10Ladsgroup) Another case of rather high important bug without a clear maintainer: {T233561} [20:54:28] 10Continuous-Integration-Config: Add `ext-mysqli` to the CI container - https://phabricator.wikimedia.org/T226585 (10Jdforrester-WMF) OK, here are the list of debian packages we're currently installing in production and in CI images, I think: | package | Prod | CI php72 | CI quibble-stretch-php72 | | -... [20:54:45] (03CR) 10Jforrester: [C: 03+2] dockerfiles: Rename php to php70 and add intl [integration/config] - 10https://gerrit.wikimedia.org/r/540681 (https://phabricator.wikimedia.org/T230446) (owner: 10Jforrester) [20:56:26] (03Merged) 10jenkins-bot: dockerfiles: Rename php to php70 and add intl [integration/config] - 10https://gerrit.wikimedia.org/r/540681 (https://phabricator.wikimedia.org/T230446) (owner: 10Jforrester) [20:56:37] 10Release-Engineering-Team (Code Health), 10Code-Stewardship-Reviews, 10MediaWiki-extensions-FlaggedRevs: FlaggedRevs: code stewardship review - https://phabricator.wikimedia.org/T185664 (10Tgr) Although as noted there it is likely to be a bug with Wikimedia configuration and not the extension itself. (Not s... [20:57:48] !log Docker: Rename php to php70 and add intl [20:57:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:59:37] (03CR) 10Jforrester: [C: 03+2] dockerfiles: Bump things over to use php70 which adds intl [integration/config] - 10https://gerrit.wikimedia.org/r/540682 (https://phabricator.wikimedia.org/T230446) (owner: 10Jforrester) [21:01:42] (03Merged) 10jenkins-bot: dockerfiles: Bump things over to use php70 which adds intl [integration/config] - 10https://gerrit.wikimedia.org/r/540682 (https://phabricator.wikimedia.org/T230446) (owner: 10Jforrester) [21:03:18] !log Docker: Bump things over to use php70 which adds intl [21:03:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:44:22] 10Release-Engineering-Team-TODO, 10Wikimedia-Site-requests, 10CPT Initiatives (PHP7 (TEC4)), 10Core Platform Team Workboards (Clinic Duty Team), and 2 others: Perf regression from ExtensionRegistry::loadFromQueue on PHP7 - https://phabricator.wikimedia.org/T187154 (10Krinkle) [22:03:46] (03PS2) 10Jforrester: jjb: Move jobs over to php70-inherited images [integration/config] - 10https://gerrit.wikimedia.org/r/540683 (https://phabricator.wikimedia.org/T230446) [22:03:48] (03PS1) 10Jforrester: dockerfiles: Drop php and php-compile, replaced by php70 and php70-compile [integration/config] - 10https://gerrit.wikimedia.org/r/540694 [22:05:07] (03CR) 10Jforrester: [C: 04-1] "Meh, only some of the images have inherited the new set of php extensions." [integration/config] - 10https://gerrit.wikimedia.org/r/540683 (https://phabricator.wikimedia.org/T230446) (owner: 10Jforrester) [22:05:24] (03CR) 10Jforrester: [C: 04-1] "Not working yet." [integration/config] - 10https://gerrit.wikimedia.org/r/540664 (https://phabricator.wikimedia.org/T230446) (owner: 10Jforrester) [22:07:12] (03CR) 10Jforrester: [C: 03+2] Don't suggest to use a temporary variable for &$this [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539529 (owner: 10Daimona Eaytoy) [22:08:30] (03Merged) 10jenkins-bot: Don't suggest to use a temporary variable for &$this [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539529 (owner: 10Daimona Eaytoy) [22:13:36] (03CR) 10jenkins-bot: Don't suggest to use a temporary variable for &$this [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539529 (owner: 10Daimona Eaytoy) [22:14:09] 10Continuous-Integration-Config: Add `ext-mysqli` to the composer-test CI container - https://phabricator.wikimedia.org/T226585 (10Krinkle) [22:16:16] 10Release-Engineering-Team (Code Health), 10Code-Stewardship-Reviews, 10MediaWiki-extensions-FlaggedRevs: Code stewardship review: FlaggedRevs - https://phabricator.wikimedia.org/T185664 (10Krinkle) [22:16:37] 10Release-Engineering-Team (Code Health), 10Code-Stewardship-Reviews, 10Discovery-Search, 10GeoData: Code steward process for GeoData extension - https://phabricator.wikimedia.org/T188434 (10Krinkle) [22:24:47] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO, 10Developer Productivity, 10local-charts, 10Epic: Add pipeline config publish step for parsoid docker image - https://phabricator.wikimedia.org/T234578 (10jeena) [22:26:16] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO, 10Developer Productivity, 10local-charts, 10Epic: Add pipeline config and publish step for parsoid dev docker image - https://phabricator.wikimedia.org/T234578 (10jeena) [22:26:53] (03PS1) 10Brennen Bearnes: layout: add publication jobs for parsoid dev image [integration/config] - 10https://gerrit.wikimedia.org/r/540699 (https://phabricator.wikimedia.org/T234578) [23:00:25] thcipriani https://gerrit-review.googlesource.com/c/gerrit/+/239436 \o/ [23:00:45] it passes the tests so that must be a good sign! [23:03:49] paladox: neat. Is this what the caffeine maintainer suggested? [23:03:55] yup [23:04:17] thcipriani see https://github.com/google/guava/issues/3602#issuecomment-538119157 [23:05:50] i won't be able to truly confirm this will fix the issue until we deploy it, but what i need to do is test locally to make sure you can still sign in. [23:06:04] (e.g making sure it dosen't break anything else) [23:36:43] thcipriani: gerrit is running on gerrit1001 and it is buster [23:36:53] we need to init gerrit [23:37:24] paladox: maybe this is the perfect moment to take a break [23:37:29] yup [23:37:31] we finally got the IPs fixed [23:37:33] the role works [23:37:34] the service runs [23:37:36] \o/ [23:37:46] and here we wait to setup the replication [23:37:50] tomorrow maybe [23:38:16] we can combine it with restart of cobalt [23:38:21] mutante did we restart gerrit on cobalt? [23:38:22] which got the new replication config [23:38:28] since replication will be broken atm [23:38:36] due to a bug in replication when auto reloading the config [23:39:08] was it broken before the merge too? [23:39:26] nope [23:39:32] only happens when the config is touched [23:39:49] since replication auto reloads the config (but that seems broken) [23:39:53] upstream are aware of that [23:40:32] uhm, so if i don't restart .. it is broken.. if i restart it i use new options we did not use before and if i revert it is also broken ?:) [23:40:44] not ideal when i wanted to soon leave [23:41:06] yeh [23:44:02] i will do the restart [23:44:08] without reverting [23:51:44] paladox: restarted. let's test something replicates to github. per chat [23:52:02] ok! [23:52:37] commit-received and then some errors .. hmm [23:54:17] i guess they belong into the "normal" category [23:54:20] mutante https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/540727/ [23:54:23] lucene search indexer [23:55:56] i see tasks in ssh -p 29418 paladox@gerrit.wikimedia.org gerrit show-queue -w [23:55:58] mutante [23:56:01] (for replication) [23:56:15] paladox: test change merged [23:56:20] thanks [23:56:21] and good [23:56:35] 39dcf189 waiting .... 23:51:40.362 [99b39d65] push git@github.com:wikimedia/operations-puppet [23:57:34] confirmed. i also see them [23:58:04] down from 4147 to 3978 tasks. looking fine [23:58:10] it does stuff [23:59:13] it's waiting for all those index changes in mw/core [23:59:49] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO, 10Developer Productivity, 10local-charts, 10Epic: Add pipeline config and publish step for restbase dev docker image - https://phabricator.wikimedia.org/T234580 (10jeena)