[00:31:11] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201909), 10Release Pipeline, 10Maps (Kartotherian): Deployment Pipeline fails with CPS error for Kartotherian - https://phabricator.wikimedia.org/T233316 (10Mathew.onipe) Post merge builds seems to fail. https://gerrit.wikimedia.org/r/... [02:13:54] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10MediaWiki-Core-Testing, 10Browser-Tests, and 3 others: Make MediaWiki Wdio tests less slow (Sept 2019) - https://phabricator.wikimedia.org/T234002 (10Krinkle) [02:13:58] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Core-Testing, 10Browser-Tests, and 2 others: Usage instructions in tests/selenium/README.md are confusing - https://phabricator.wikimedia.org/T214708 (10Krinkle) [02:14:05] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10MediaWiki-Core-Testing, 10Browser-Tests, and 3 others: Make MediaWiki Wdio tests less slow (Sept 2019) - https://phabricator.wikimedia.org/T234002 (10Krinkle) [02:14:09] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Core-Testing, 10Browser-Tests, and 2 others: Usage instructions in tests/selenium/README.md are confusing - https://phabricator.wikimedia.org/T214708 (10Krinkle) [02:14:15] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10MediaWiki-Core-Testing, 10Browser-Tests, and 3 others: Make MediaWiki Wdio tests less slow (Sept 2019) - https://phabricator.wikimedia.org/T234002 (10Krinkle) [02:14:19] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10MediaWiki-Core-Testing, 10Browser-Tests, and 2 others: Usage instructions in tests/selenium/README.md are confusing - https://phabricator.wikimedia.org/T214708 (10Krinkle) [04:45:58] (03CR) 10Krinkle: Dedicated Selenium job for gated repos (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/539630 (https://phabricator.wikimedia.org/T232759) (owner: 10Hashar) [04:53:26] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909): Move CI selenium/qunit tests of mediawiki repository to a standalone job - https://phabricator.wikimedia.org/T232759 (10Krinkle) @hashar To confirm, we need only one of those, right?... [06:01:58] 10Gerrit: Gerrit workflow: "Merge review" AND "Open push" for BlueSpice? - https://phabricator.wikimedia.org/T234224 (10Osnard) [06:06:58] 10Gerrit: Gerrit workflow: "Merge review" AND "Open push" for BlueSpice? - https://phabricator.wikimedia.org/T234224 (10Osnard) I detail it says ` remote: Branch refs/heads/REL1_31: remote: You are not allowed to perform this operation. remote: To push into this reference you need 'Push' rights. remote: User: r... [06:14:23] 10Gerrit: Gerrit workflow: "Merge review" AND "Open push" for BlueSpice? - https://phabricator.wikimedia.org/T234224 (10Osnard) Okay, I just set the `Push` permission to group `bluespice` on https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/BlueSpiceSignHere,access and now it works. Thank you... [06:57:58] 10Gerrit: Gerrit workflow: "Merge review" AND "Open push" for BlueSpice? - https://phabricator.wikimedia.org/T234224 (10hashar) > This will result in dozens or hundreds of commits to be pushed to origin. We have enabled "Merge review" for our repositories, so `git push` results in Do not push? You can send for... [07:05:57] 10Gerrit: Gerrit workflow: "Merge review" AND "Open push" for BlueSpice? - https://phabricator.wikimedia.org/T234224 (10Osnard) Yes, I actually want to `push` instead of `review`, as I just want to synchronize branches `REL1_31` and `REL1_31_dev`. In `REL1_31_dev` everything has already been reviewed, so I don't... [07:56:09] 10Gerrit, 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit GC thrashing during branch cut - https://phabricator.wikimedia.org/T231872 (10hashar) @thcipriani mentioned high CPU occurring Sep 28 which solved after a reboot: {F30517408} This morning (Oct 1st 07:00) I forced a reindex of all changes... [08:19:54] Hmm, https://gerrit.wikimedia.org/r/monitoring still shows the “change cannot be found” :( [08:19:55] Most recently logged at 8:19 utc [09:20:21] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Update existing Selenium documentation - https://phabricator.wikimedia.org/T232598 (10zeljkofilipin) [09:21:19] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Update existing Selenium documentation - https://phabricator.wikimedia.org/T232598 (10zeljkofilipin) [09:31:20] 10Release-Engineering-Team, 10MediaWiki-User-management, 10MediaWiki-extensions-FlaggedRevs, 10User-DannyS712: Pending changes: autoreview randomly fails - https://phabricator.wikimedia.org/T233561 (10Tgr) p:05Triage→03High I see all the groups I should be seeing (`["checkuser", "sysop", "*", "user", "... [09:32:40] (03CR) 10Awight: [C: 03+2] Rephrase a comment in quibble.zuul.clone_worker [integration/quibble] - 10https://gerrit.wikimedia.org/r/539537 (owner: 10Hashar) [09:33:28] (03Merged) 10jenkins-bot: Rephrase a comment in quibble.zuul.clone_worker [integration/quibble] - 10https://gerrit.wikimedia.org/r/539537 (owner: 10Hashar) [09:37:35] (03CR) 10Awight: "I like this, note that we took a contrasting approach in I875743f2019b2c5b5befd2d2ae9886f0ec4ee47d by implementing `--skip-install`. Ther" [integration/quibble] - 10https://gerrit.wikimedia.org/r/438084 (owner: 10Legoktm) [09:46:12] 10Release-Engineering-Team, 10MediaWiki-User-management, 10MediaWiki-extensions-FlaggedRevs, 10User-DannyS712: Pending changes: autoreview randomly fails - https://phabricator.wikimedia.org/T233561 (10Neolexx) To the best of my knowledge en-wiki doesn't use [[https://www.mediawiki.org/wiki/Extension:Flagge... [09:52:20] 10Release-Engineering-Team, 10MediaWiki-User-management, 10MediaWiki-extensions-FlaggedRevs, 10User-DannyS712: Pending changes: autoreview randomly fails - https://phabricator.wikimedia.org/T233561 (10Zache) >>! In T233561#5537071, @Neolexx wrote: > To the best of my knowledge en-wiki doesn't use [[https:/... [10:02:04] 10Release-Engineering-Team, 10MediaWiki-User-management, 10MediaWiki-extensions-FlaggedRevs, 10User-DannyS712: Pending changes: autoreview randomly fails - https://phabricator.wikimedia.org/T233561 (10Tgr) Review (FlaggedRevs) log for ISS: https://en.wikipedia.org/wiki/Special:Log?type=review&user=&page=In... [10:19:00] 10Release-Engineering-Team, 10MediaWiki-User-management, 10MediaWiki-extensions-FlaggedRevs, 10User-DannyS712: Pending changes: autoreview randomly fails - https://phabricator.wikimedia.org/T233561 (10Neolexx) >>! In T233561#5537079, @Zache wrote: > In enwiki it is named as "Pending changes" and in flagged... [10:25:39] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10MW-1.34-notes (1.34.0-wmf.25; 2019-10-01), and 2 others: Upgrade webdriverio to version 5 - https://phabricator.wikimedia.org/T213268 (10zeljkofilipin) 05Open→03Resolved a:03Krinkle [10:25:45] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10MediaWiki-Core-Testing, 10Browser-Tests, and 3 others: Make MediaWiki Wdio tests less slow (Sept 2019) - https://phabricator.wikimedia.org/T234002 (10zeljkofilipin) [10:28:41] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10MW-1.34-notes (1.34.0-wmf.25; 2019-10-01), and 2 others: Upgrade webdriverio to version 5 in mediawiki/core - https://phabricator.wikimedia.org/T213268 (10zeljkofilipin) [10:30:21] 10Release-Engineering-Team, 10MediaWiki-User-management, 10MediaWiki-extensions-FlaggedRevs, 10User-DannyS712: Pending changes: autoreview randomly fails - https://phabricator.wikimedia.org/T233561 (10Zache) I tried to check yesterday if the patrolling is working or not too, but there are not many wikis wh... [10:33:44] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Upgrade webdriverio to version 5 for all repositories - https://phabricator.wikimedia.org/T234314 (10zeljkofilipin) [10:34:02] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Upgrade webdriverio to version 5 for all repositories - https://phabricator.wikimedia.org/T234314 (10zeljkofilipin) p:05Triage→03Normal [10:37:25] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Upgrade webdriverio to version 5 for all repositories - https://phabricator.wikimedia.org/T234314 (10zeljkofilipin) [10:37:56] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10MW-1.34-notes (1.34.0-wmf.25; 2019-10-01), and 2 others: Upgrade webdriverio to version 5 in mediawiki/core - https://phabricator.wikimedia.org/T213268 (10zeljkofilipin) [10:37:59] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Upgrade webdriverio to version 5 for all repositories - https://phabricator.wikimedia.org/T234314 (10zeljkofilipin) [10:38:03] 10Phabricator: Herald rule: KaiOS -> Inuka - https://phabricator.wikimedia.org/T234217 (10Aklapper) 05Open→03Resolved a:03Aklapper Created H333: * When all of these conditions are met: ** Project tags include any of #KaiOS-Wikipedia-app * Take these actions every time this rule matches: ** Add projects: #I... [10:38:16] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201909), 10User-zeljkofilipin: Upgrade webdriverio to version 5 for all repositories - https://phabricator.wikimedia.org/T234314 (10zeljkofilipin) [10:40:46] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (201910), 10local-charts, 10User-zeljkofilipin: Error: error installing: the server could not find the requested resource - https://phabricator.wikimedia.org/T233960 (10zeljkofilipin) [10:40:51] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10User-zeljkofilipin: Upgrade webdriverio to version 5 for all repositories - https://phabricator.wikimedia.org/T234314 (10zeljkofilipin) [10:46:40] 10Phabricator: Herald rule: KaiOS -> Inuka - https://phabricator.wikimedia.org/T234217 (10SBisson) Thanks @Aklapper [10:49:00] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10TCB-Team, 10Two-Column-Edit-Conflict-Merge, and 3 others: Fix and restore daily browser tests for TwoColConflict - https://phabricator.wikimedia.org/T234311 (10zeljkofilipin) [10:51:41] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10TCB-Team, 10Two-Column-Edit-Conflict-Merge, and 3 others: Fix and restore daily browser tests for TwoColConflict - https://phabricator.wikimedia.org/T234311 (10zeljkofilipin) Let me know when you're ready... [11:01:27] (03PS1) 10Awight: Reenable TwoColConflict browser tests [integration/config] - 10https://gerrit.wikimedia.org/r/540098 (https://phabricator.wikimedia.org/T234311) [11:02:28] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10TCB-Team, 10Two-Column-Edit-Conflict-Merge, and 5 others: Fix and restore daily browser tests for TwoColConflict - https://phabricator.wikimedia.org/T234311 (10awight) [11:08:41] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "> The ampersand in "&$this" is redundant. Remove it if you only want the callee to modify properties of $this, or assign $this to a tempor" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539529 (owner: 10Daimona Eaytoy) [11:11:42] (03CR) 10WMDE-Fisch: [C: 03+1] Reenable TwoColConflict browser tests [integration/config] - 10https://gerrit.wikimedia.org/r/540098 (https://phabricator.wikimedia.org/T234311) (owner: 10Awight) [11:13:24] !log shutting down integration-castor03 # T232646 [11:13:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:13:27] T232646: Move integration-castor03.integration.eqiad.wmflabs to a newer cloudvirt machine - https://phabricator.wikimedia.org/T232646 [11:17:33] PROBLEM - Host integration-castor03 is DOWN: CRITICAL - Host Unreachable (172.16.5.161) [11:19:44] (03CR) 10Daimona Eaytoy: [C: 04-1] "> > The ampersand in "&$this" is redundant. Remove it if you only" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539529 (owner: 10Daimona Eaytoy) [11:20:59] (03PS2) 10Daimona Eaytoy: Don't suggest to use a temporary variable for &$this [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539529 [11:22:37] RECOVERY - Host integration-castor03 is UP: PING OK - Packet loss = 0%, RTA = 2.22 ms [11:24:47] (03CR) 10jerkins-bot: [V: 04-1] Don't suggest to use a temporary variable for &$this [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539529 (owner: 10Daimona Eaytoy) [11:25:34] (03CR) 10Thiemo Kreuz (WMDE): "> I'm just more used to […] programming languages such as C# or Java […]" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/492634 (https://phabricator.wikimedia.org/T216971) (owner: 10Mainframe98) [11:25:50] I'm running into an exciting browser test failure which would benefit from zeljkof's insight... https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php72-docker/22025/console [11:26:09] Why would browser.options be undefined? And... how does it normally get defined? [11:26:18] 10Release-Engineering-Team, 10MediaWiki-User-management, 10MediaWiki-extensions-FlaggedRevs, 10User-DannyS712: Pending changes: autoreview randomly fails - https://phabricator.wikimedia.org/T233561 (10Tgr) >>! In T233561#5537181, @Zache wrote: > In de.wiktionary.org there was edit by user with `autoreview... [11:26:42] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] Reenable TwoColConflict browser tests [integration/config] - 10https://gerrit.wikimedia.org/r/540098 (https://phabricator.wikimedia.org/T234311) (owner: 10Awight) [11:26:42] awight__: I got something similar this morning :/ [11:26:46] (locall) [11:26:48] locally [11:26:48] oh noes :-) [11:27:06] okay, it's helpful to know that I'm not alone! [11:27:42] awight: I think it's connected to https://gerrit.wikimedia.org/r/c/mediawiki/core/+/539441 [11:28:01] but I'm not sure yet [11:28:17] https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/539441/9/tests/selenium/wdio-mediawiki/Page.js [11:28:43] That makes sense. Maybe updating your local mediawiki-core will help? [11:29:12] I did [11:29:15] no luck [11:29:28] I'll debug it in a few minutes, in the middle of something else [11:29:37] kk no rush here [11:31:45] zeljkof: FYI, looks like all tests will have to rewrite `browser.options` as `browser.config`. Maybe core should include backwards-compatibility to ease this update. [11:33:20] other repos should only use code in wdio-mediawiki, so no backwards compatibility is needed in core [11:33:32] but something _is_ broken [11:33:48] hmm, yeah I think you're right, my extension is being naughty. [11:35:41] I want to run the same version of npm on my laptop as is run on CI e.g. on mwgate-node10-docker. I think in the past I was fine using docker-registry.wikimedia.org/nodejs10-devel but I think these diverged recently. Is there a devel image that is kept in sync with regard to node and npm version run in CI? [11:36:15] hashar might know ^  [11:37:25] tarrow: whatever releng/xxxx container that is being used by the job ? [11:38:03] (03PS3) 10Daimona Eaytoy: Don't suggest to use a temporary variable for &$this [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539529 [11:38:12] https://integration.wikimedia.org/ci/job/mwgate-node10-docker/ gives me docker-registry.wikimedia.org/releng/node10-test:0.5.0 [11:38:16] hashar: I also need to be able to run apt etc... and I guess the releng container is locked down [11:38:43] $ docker run --rm -it --entrypoint=npm docker-registry.wikimedia.org/releng/node10-test:0.5.0 --version [11:38:43] 6.5.0 [11:38:52] it runs as the "nobody" user [11:39:03] so yeah you would need to run it as root: docker run --user root [11:39:14] then apt update ... apt instal whatever [11:39:38] and nodejs is 10.15.2 [11:39:39] tarrow: It's probably easy to port those updated dependencies to add*shore's container, if that's what you're developing with. [11:40:19] yeah, the nodejs10-devel one is a little ahead of CI [11:40:56] (03CR) 10Zfilipin: Reenable TwoColConflict browser tests (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/540098 (https://phabricator.wikimedia.org/T234311) (owner: 10Awight) [11:41:52] "should" the releng test image and devel image be in sync? are they both maintained by the same people? [11:42:21] 10Release-Engineering-Team, 10MediaWiki-User-management, 10MediaWiki-extensions-FlaggedRevs, 10User-DannyS712: Pending changes: autoreview randomly fails - https://phabricator.wikimedia.org/T233561 (10Neolexx) Long time ago just for fun I wrote a patrolling script that works straight through the wrapping i... [11:42:22] or are they actually for totally different things and we just got lucky that for some time they were overlapping [11:42:30] tarrow: I think the devel image is a personal project maintained by add*shore? [11:42:34] awight: I think I've found the problem to the failing job [11:42:51] awight: I think it's an official wmf image? [11:43:03] its on docker-registry.wikimedia.org [11:43:23] awight: looks like twocolconflict is reaching directly into core here? https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/TwoColConflict/+/master/tests/selenium/pageobjects/editconflict.page.js#2 [11:43:30] could that be causing the problem? [11:44:06] /o\ indeed [11:45:00] tarrow: Looks like you're right, there's a wiki page and everything, https://www.mediawiki.org/wiki/MediaWiki-Docker-Dev [11:45:24] +1 I think this is the right direction to go in, just surprised it actually happened. [11:54:30] 10Release-Engineering-Team-TODO (201910): Arrange intra-team PGP keysigning at Atlanta offsite - https://phabricator.wikimedia.org/T232990 (10LarsWirzenius) p:05Triage→03Normal [11:55:03] 10Release-Engineering-Team-TODO (201910): Update architecture document draft for future CI - https://phabricator.wikimedia.org/T229258 (10LarsWirzenius) [12:03:01] (03PS2) 10Awight: Reenable TwoColConflict browser tests [integration/config] - 10https://gerrit.wikimedia.org/r/540098 (https://phabricator.wikimedia.org/T234311) [12:19:30] Darn, I've cleaned up our extension to use the wdio-mediawiki API now, but browser.config seems to be undefined. [12:27:32] PROBLEM - Host integration-castor03 is DOWN: CRITICAL - Host Unreachable (172.16.5.161) [12:31:57] PROBLEM - Gerrit Health Check on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [12:32:23] PROBLEM - Gerrit JSON on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Gerrit%23Monitoring [12:33:07] PROBLEM - SSH access on cobalt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Gerrit [12:33:23] oh gerrit... [12:34:35] RECOVERY - SSH access on cobalt is OK: SSH OK - GerritCodeReview_2.15.14-16-g855b179b5f (SSHD-CORE-1.6.0) (protocol 2.0) https://wikitech.wikimedia.org/wiki/Gerrit [12:34:39] (03PS1) 10Hashar: disable castor temporarily [integration/config] - 10https://gerrit.wikimedia.org/r/540108 (https://phabricator.wikimedia.org/T232646) [12:35:01] RECOVERY - Gerrit Health Check on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 865 bytes in 0.345 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [12:35:11] rxy_: offline reindex [12:35:27] RECOVERY - Gerrit JSON on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 26356 bytes in 0.036 second response time https://wikitech.wikimedia.org/wiki/Gerrit%23Monitoring [12:35:49] back up for me [12:35:57] back :D [12:36:03] (03CR) 10Hashar: [C: 03+2] disable castor temporarily [integration/config] - 10https://gerrit.wikimedia.org/r/540108 (https://phabricator.wikimedia.org/T232646) (owner: 10Hashar) [12:38:58] (03Merged) 10jenkins-bot: disable castor temporarily [integration/config] - 10https://gerrit.wikimedia.org/r/540108 (https://phabricator.wikimedia.org/T232646) (owner: 10Hashar) [12:41:35] mutante: re https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/539676/ I'm not able to make that hash commit display without putting the whole gitiles url in the commit message [12:42:35] RECOVERY - Host integration-castor03 is UP: PING OK - Packet loss = 0%, RTA = 0.89 ms [12:44:20] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "Cool for me – obviously. ;-) I will merge this within the next days if nobody objects." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539529 (owner: 10Daimona Eaytoy) [12:44:59] (03PS1) 10Hashar: Revert "disable castor temporarily" [integration/config] - 10https://gerrit.wikimedia.org/r/540110 (https://phabricator.wikimedia.org/T232646) [12:45:16] (03CR) 10Hashar: [C: 03+2] Revert "disable castor temporarily" [integration/config] - 10https://gerrit.wikimedia.org/r/540110 (https://phabricator.wikimedia.org/T232646) (owner: 10Hashar) [12:47:18] 10Continuous-Integration-Infrastructure: castor rsync's taking 3-5 minutes for mwgate-npm jobs - https://phabricator.wikimedia.org/T188375 (10hashar) [12:47:20] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Cloud-VPS, and 2 others: Move integration-castor03.integration.eqiad.wmflabs to a newer cloudvirt machine - https://phabricator.wikimedia.org/T232646 (10hashar) 05Open→... [12:48:21] (03Merged) 10jenkins-bot: Revert "disable castor temporarily" [integration/config] - 10https://gerrit.wikimedia.org/r/540110 (https://phabricator.wikimedia.org/T232646) (owner: 10Hashar) [12:50:39] 10Continuous-Integration-Infrastructure: castor rsync's taking 3-5 minutes for mwgate-npm jobs - https://phabricator.wikimedia.org/T188375 (10hashar) The instance has been moved to a faster host (cloudvirt1021). So potentially we could reenable `--compress`. The default is 6 from a bracket of 0 - 9, so maybe... [12:51:48] (03CR) 10Daimona Eaytoy: "> Cool for me – obviously. ;-) I will merge this within the next days" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539529 (owner: 10Daimona Eaytoy) [13:20:05] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10TCB-Team, 10Two-Column-Edit-Conflict-Merge, and 5 others: Fix and restore daily browser tests for TwoColConflict - https://phabricator.wikimedia.org/T234311 (10awight) a:03awight [13:26:50] 10Release-Engineering-Team, 10MediaWiki-User-management, 10MediaWiki-extensions-FlaggedRevs, 10User-DannyS712: Pending changes: autoreview randomly fails - https://phabricator.wikimedia.org/T233561 (10Zache) >>! In T233561#5537057, @Tgr wrote: > I see all the groups I should be seeing (`["checkuser", "syso... [13:43:04] !log Gerrit: disabling bot account "owl" , see whether that helps on the JVM gc issue [13:43:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:47:56] (03CR) 1020after4: [V: 03+2 C: 03+2] Don't filter the version out of stack traces. [releng/phatality] - 10https://gerrit.wikimedia.org/r/538947 (owner: 1020after4) [13:48:30] (03PS5) 1020after4: Don't filter the version out of stack traces. [releng/phatality] - 10https://gerrit.wikimedia.org/r/538947 [13:48:34] (03CR) 1020after4: [V: 03+2 C: 03+2] Don't filter the version out of stack traces. [releng/phatality] - 10https://gerrit.wikimedia.org/r/538947 (owner: 1020after4) [13:50:04] hashar: heh, I think I saw that account several times in the logs [13:51:54] (03PS1) 10Awight: Allow uploads to support FileImporter tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/540118 (https://phabricator.wikimedia.org/T190829) [13:54:03] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Developer-Advocacy, 10wikimedia.biterg.io: biterg.io Gerrit crawling probably stresses the server too much - https://phabricator.wikimedia.org/T234328 (10hashar) [13:54:55] 10Gerrit: Lots of "Skipping change xxx because the corresponding repository was not found" in the logs - https://phabricator.wikimedia.org/T233989 (10hashar) I have disabled the bot account since I suspect it causes gc/memory pressure on Gerrit: T234328 [13:55:12] !log Gerrit: disabled bot account "owl" , see whether that helps on the JVM gc issue T234328 [13:55:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:55:15] T234328: biterg.io Gerrit crawling probably stresses the server too much - https://phabricator.wikimedia.org/T234328 [14:02:22] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Developer-Advocacy, 10wikimedia.biterg.io: biterg.io Gerrit crawling probably stresses the server too much - https://phabricator.wikimedia.org/T234328 (10hashar) Looking up for `owl` queries made over... [14:17:25] godog: good morning ;] Since you are around I have a few connections regarding the gerrit / java melody prometheus monitoring ;) [14:17:28] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Developer-Advocacy, 10wikimedia.biterg.io: biterg.io Gerrit crawling probably stresses the server too much - https://phabricator.wikimedia.org/T234328 (10thcipriani) >>! In T234328#5537622, @Stashbot... [14:17:46] I am trying to figure out how the metrics get collected and wether some more could be added :] [14:19:50] found it https://gerrit.wikimedia.org/r/monitoring?format=prometheus :] [14:19:57] hashar: hi! it is indeed morning here (working PDT hours for the next two weeks) the tl;dr is that prometheus does an http call [14:20:03] yeah that [14:20:04] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata, 10Wikidata-Campsite, 10Patch-For-Review: [Task] Add Scribunto to extension-gate in CI - https://phabricator.wikimedia.org/T125050 (10alaa_wmde) [14:20:12] I wanted to find some metrics related to the jvm heap [14:20:18] adding more I'm not sure, I'm not familiar with javamelody [14:20:27] which do not show up at https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody?orgId=1 [14:20:31] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata, 10Wikidata-Campsite, 10Patch-For-Review: [Task] Add Scribunto to extension-gate in CI - https://phabricator.wikimedia.org/T125050 (10alaa_wmde) Need to figure out the current status of this and why/when we should do it [14:20:38] I have no idea how that grafana board is generated [14:20:41] heh, my recommendation would be to add jvm_exporter in addition to java melody [14:20:47] seems it just magically happens based on whatever is exposed on https://gerrit.wikimedia.org/r/monitoring?format=prometheus [14:20:55] ah yeah [14:21:00] the jvm exporter, we talked about that [14:21:01] hmm [14:21:53] godog: oh and my last comment shows I have been confused by all those plugins :-\ [14:23:01] yeah it is confusing alright, personally I'm not sure what java melody adds to the picture but I'm not familiar with it either [14:24:28] hashar: re: jvm_exporter a dashboard that uses its metrics is https://grafana.wikimedia.org/d/000000537/jvm-overview-work-in-progress-gehel?orgId=1 for example [14:25:09] yeah [14:25:12] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Add prometheus exporter to Gerrit - https://phabricator.wikimedia.org/T184086 (10hashar) Sorry indeed, I have been confused by all those plugins. Turns out today I lacked some m... [14:25:24] and my last comment on T184086 confused it with the javamelody exporter [14:25:25] T184086: Add prometheus exporter to Gerrit - https://phabricator.wikimedia.org/T184086 [14:25:30] different thing [14:26:58] godog: I am afraid we will need a hand to setup the jmx exporter :/ [14:29:55] hashar: for sure! puppet has some examples already and LMK how we can help [14:32:17] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 3 others: MediaWiki web installer do not show extension when their dependency is missing - https://phabricator.wikimedia.org/T220514 (10eprodromou) We're going to deal... [14:32:48] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Add prometheus exporter to Gerrit - https://phabricator.wikimedia.org/T184086 (10hashar) [14:33:23] godog: I am more suggesting for you guys to set it up for us : ] So we don't have to learn all the puppet config that is necessary :\ [14:37:55] I'm happy to review patches to add the exporter, the work should be similar to what happened with javamelody [14:37:58] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.32-release, and 3 others: MediaWiki web installer does not show extension when their dependency is missing - https://phabricator.wikimedia.org/T220514 (10daniel) [14:38:38] note that I've been suggesting that since jan 2018 on task [14:43:24] 10Continuous-Integration-Infrastructure, 10Core Platform Team, 10MediaWiki-Installer, 10MW-1.32-release, and 3 others: MediaWiki web installer does not show extension when their dependency is missing - https://phabricator.wikimedia.org/T220514 (10CCicalese_WMF) We discussed this in Clinic Duty. The related... [14:50:39] (03Abandoned) 10Thiemo Kreuz (WMDE): Split cast operator logic from "forbidden functions" sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/493656 (https://phabricator.wikimedia.org/T216971) (owner: 10Thiemo Kreuz (WMDE)) [14:51:05] 10Release-Engineering-Team, 10MediaWiki-User-management, 10MediaWiki-extensions-FlaggedRevs, 10User-DannyS712: Pending changes: autoreview randomly fails - https://phabricator.wikimedia.org/T233561 (10JJMC89) I also saw this on enwiki (protection configuration) on September 23. As a sysop, it told me that... [14:51:15] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, 10Patch-For-Review: Add prometheus exporter to Gerrit - https://phabricator.wikimedia.org/T184086 (10hashar) I do not have any idea from where to start though there are a few leads in the previous... [14:54:21] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] Allow uploads to support FileImporter tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/540118 (https://phabricator.wikimedia.org/T190829) (owner: 10Awight) [14:56:14] (03PS9) 10Daimona Eaytoy: Improve checks for variargs [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539177 (https://phabricator.wikimedia.org/T231710) [15:00:38] (03PS10) 10Daimona Eaytoy: Improve checks for variargs [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/539177 (https://phabricator.wikimedia.org/T231710) [15:05:29] PROBLEM - Host integration-agent-docker-1015 is DOWN: CRITICAL - Host Unreachable (172.16.6.203) [15:17:00] !log Pooled unused Jessie agent https://integration.wikimedia.org/ci/computer/integration-agent-jessie-docker-1001/ [15:17:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:17:12] !log Deleted integration-agent-docker-1015 (was too slow) [15:17:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:21:39] (03PS1) 10Hashar: Tie mwcore-phpunit-coverage-master to a Jessie docker [integration/config] - 10https://gerrit.wikimedia.org/r/540152 [15:22:22] (03CR) 10Hashar: [C: 03+2] Tie mwcore-phpunit-coverage-master to a Jessie docker [integration/config] - 10https://gerrit.wikimedia.org/r/540152 (owner: 10Hashar) [15:25:06] (03Merged) 10jenkins-bot: Tie mwcore-phpunit-coverage-master to a Jessie docker [integration/config] - 10https://gerrit.wikimedia.org/r/540152 (owner: 10Hashar) [15:41:26] 10Release-Engineering-Team-TODO (201910), 10MediaWiki-General, 10Release: Mark MediaWiki core as 1.35.0-alpha once REL1_34 is branched - https://phabricator.wikimedia.org/T232030 (10Jdforrester-WMF) [15:41:41] 10Continuous-Integration-Config, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201910), 10Scap, and 5 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) [15:44:15] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Quibble: Quibble should fatal out on clone/fetch failure"ERROR:zuul.Repo:Unable to initialize repo for npm-test.git" - https://phabricator.wikimedia.org/T233143 (10hashar) [15:44:30] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team-TODO (201910), 10Zuul: Zuul cancels all changes when a change is manually merged - https://phabricator.wikimedia.org/T203846 (10hashar) [15:44:41] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201910), 10Zuul, 10Documentation: Document Zuul problems caused by force merge - https://phabricator.wikimedia.org/T225955 (10hashar) [15:45:39] hashar: Is https://phabricator.wikimedia.org/T232706 done or not? [15:46:17] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: Move mwcore-codehealth-master-non-voting to a dedicated Jenkins label / agent - https://phabricator.wikimedia.org/T234259 (10hashar) [15:46:20] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Jenkins: Enable ansicolor globally in Jenkins when plugin is released - https://phabricator.wikimedia.org/T233688 (10hashar) [15:46:22] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Test-Coverage: Switch mediawiki code coverage from xdebug to phpdbg or pcov - https://phabricator.wikimedia.org/T234020 (10hashar) [15:46:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Documentation: Document how to deploy a new Quibble version to CI - https://phabricator.wikimedia.org/T231251 (10hashar) [15:46:26] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: Move CI selenium/qunit tests of mediawiki repository to a standalone job - https://phabricator.wikimedia.org/T232759 (10hashar) [15:46:28] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO, 10Wikimedia-Logstash, 10observability: logstash-beta.wmflabs.org does not receive any mediawiki events - https://phabricator.wikimedia.org/T233134 (10hashar) [15:46:30] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Jenkins: integration-config-zuul-layout-validate-docker takes too long in Jenkins due to huge output - https://phabricator.wikimedia.org/T232287 (10hashar) [15:47:16] hashar: Also, I guess that you're the best person to respond to T233092? [15:47:17] T233092: CI: Create a way to share a secret between MediaWiki and the testing framework. - https://phabricator.wikimedia.org/T233092 [15:47:36] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201910), 10Patch-For-Review, 10phan: ci-src-setup job (used by mediawiki-core-php72-phan-docker) is still running on PHP 7.0.33 - https://phabricator.wikimedia.org/T234062 (10Jdforrester-WMF) [15:48:03] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10phan: ci-src-setup job (used by mediawiki-core-php72-phan-docker) is still running on PHP 7.0.33 - https://phabricator.wikimedia.org/T234062 (10hashar) [15:48:05] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Quibble, and 3 others: CI: Create a way to share a secret between MediaWiki and the testing framework. - https://phabricator.wikimedia.org/T233092 (10hashar) [15:48:07] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO: CI mediawiki/core run times have increased since July 26th, 2019 - https://phabricator.wikimedia.org/T232626 (10hashar) [15:48:09] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10CPT Initiatives (API Integration Tests), and 2 others: Set up CI for mediawiki/tools/api-testing - https://phabricator.wikimedia.org/T230340 (10hashar) [15:48:52] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201909), 10Performance-Team (Radar): Reduce static asset time on disk from five trains' worth to two - https://phabricator.wikimedia.org/T140921 (10Jdforrester-WMF) [15:49:13] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201910), 10Performance-Team (Radar): Reduce static asset time on disk from five trains' worth to two - https://phabricator.wikimedia.org/T140921 (10Jdforrester-WMF) Clearer title. [16:01:50] 10Continuous-Integration-Infrastructure: castor rsync's taking 3-5 minutes for mwgate-npm jobs - https://phabricator.wikimedia.org/T188375 (10Jdforrester-WMF) Current builds seem to take ~20s. Is this fast enough? [16:11:16] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10User-zeljkofilipin: Upgrade webdriverio to version 5 for all repositories - https://phabricator.wikimedia.org/T234314 (10Krinkle) This should probably wait for T234002 to settle first so that we can push a n... [16:11:18] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201910), 10Patch-For-Review, 10phan: ci-src-setup job (used by mediawiki-core-php72-phan-docker) is still running on PHP 7.0.33 - https://phabricator.wikimedia.org/T234062 (10Jdforrester-WMF) [16:22:34] 10Deployments, 10MediaWiki-SWAT-deployments: Figure out what to do with `fatalmonitor` script - https://phabricator.wikimedia.org/T234345 (10Lucas_Werkmeister_WMDE) [16:23:26] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201910), 10MediaWiki-extensions-CentralAuth, 10User-zeljkofilipin: [betalabs] memcached listens solely on 127.0.0.1 (was: Cannot create a new user account) - https://phabricator.wikimedia.org/T232796 (10Jdforrester-WMF) [16:27:42] paladox: re gerrit1001, I guess the 'replication' function will also take into count this new host? [16:27:58] hauskater i'm updating it to do that (see https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/540164/) :) [16:28:05] I mean the gerrit replicate start --wait [16:28:34] Perfect [16:28:42] yup [16:28:44] Is cobalt going down or something? [16:30:06] hauskater it's being replaced with a more powerful server [16:30:08] since we need more ram :) [16:30:41] Good, good to see some progress :) [16:34:17] yup! [16:34:38] * paladox hopping to get the gerrit class applied to it [16:34:41] *tonight [16:36:47] and the upgrading of gerrit? :) [16:37:15] That should come after i imagine. [16:37:18] I feel we'll go straight to gerrit v.3 if we delay it more and more [16:38:49] * hauskater remembers when Chad had to update Gerrit, such fun learning all kinds of 'bad words' :P [16:39:13] lol [16:39:20] hey. Is there a way for me, to run shell.php with prod env? I'd like to check couple config variables values for couple wikis [16:39:30] gerrit v3 is quite breaking in terms of ui. [16:39:40] there is a thing that should work, but it doesn't and I have no idea why ;/ [16:39:43] gerrit 3.1 is around the corner! [16:39:57] raynor: mwscript shell.php --wiki=dbname ? [16:40:23] 'tho I think we use eval.php [16:40:35] * hauskater defers the question to somebody else [16:40:40] on which machine? deployment? [16:41:00] deployment has the beta config, so most probably I need to ssh to some apache, [16:41:39] I'm not sure about that raynor. I'd wait for someone else more experienced to answer :) [16:42:00] kk, thx hauskater [16:42:20] I'm not sure who's online atm that can help you though [16:42:38] I usually ssh to mwdebug1002 for quick shell.php checks [16:42:55] ah, right, mwdebug1002 should be good [16:43:44] used to add PHP=php7.2 to work around T186936 [16:43:44] T186936: shell.php broken on most production hosts - https://phabricator.wikimedia.org/T186936 [16:43:55] might not be necessary anymore, haven’t tried it yet since the PHP7 migration finished [16:46:55] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201910), 10Performance-Team (Radar): Reduce static asset time on disk from five trains' worth to two - https://phabricator.wikimedia.org/T140921 (10Jdforrester-WMF) a:03dduvall [16:49:04] Lucas_WMDE, when you do shell.php cheks on mwdebug1002, do you get prod configs? [16:49:15] yes? [16:49:18] ok [16:49:24] could you check one thing with me please [16:50:01] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201910), 10Performance-Team (Radar): Reduce static asset time on disk from five trains' worth to two - https://phabricator.wikimedia.org/T140921 (10Jdforrester-WMF) [[https://wikitech.wikimedia.org/w/index.php... [16:50:19] hauskater: heh, is it you? is that link to another gerrit change? [16:53:15] mutante: I'm here, what's up? [16:54:21] the issue with the commit message, okay, the change exists, but it was done by the system while renaming the project so it has no change-id [16:54:48] see the comments, it can be fetched via the full gitiles url [16:55:46] https://gerrit.wikimedia.org/r/plugins/gitiles/All-Projects/+/10107cf5056eb6ef3903f49d59cd27387831c5b5 [16:56:37] (resolved raynor’s question out-of-channel btw) [16:57:31] yup, we found out that there is `mobile.php` file that overrides some mobile frontend configs [16:59:02] raynor: Yeah, I'm planning to scrap that eventually. [16:59:03] hauskater: i see now. well, kind of:) [16:59:15] raynor: Config is too much of a mess right now. :-( [16:59:22] hauskater: ok, i got the link. it was a bit of a nitpick anyways. thanks! [16:59:54] mutante: I understand, but I was not aware the link would be innefective [17:00:41] James_F, what's the procedure if I want to change sth there? [17:00:47] same as InitializeSettings.php ? [17:01:38] raynor: It's more like CommonSettings.php (no ability to alter per-wiki), but yes. [17:02:31] ok, thx [17:24:08] hauskater: merged it now. thanks for finding it [17:24:38] mutante: Thank you sir :) puppet-merge too? [17:24:59] sudo puppet merge I guess [17:25:07] yes, it is running as we speak [17:25:10] unless you operate as root :) [17:25:15] sudo puppet-merge [17:26:19] config changed. but no service restart done. [17:26:48] I think hashar wanted to restart gerrit at some point today, or was thcipriani [17:26:58] they made some java changes and reindexes [17:27:12] let me know when I can test the change :) [17:27:19] I wanted to do a restart [17:27:20] usually i find it better later in the (US) day [17:27:24] creating projects via the command line is a PITA [17:27:24] but either works [17:27:26] marxarelli is running the branch cut now. [17:27:33] So restarting gerrit is contra-indicated. ;-) [17:27:40] Certainly [17:27:40] after merging: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/540114/ [17:28:25] oh yea, that was next in the list [17:28:32] already glanced [17:28:40] thcipriani: I've been creating projects via the gerrit create-project ssh command line, but if the change I did makes the UI not assing "Administrators" as default owners of newly created projects it'd save me a lot of time :D [17:28:42] nice, thank you :) [17:29:10] hauskater: thanks for spotting the problem, I appreciate it [17:29:21] I ain't sure that'd solve the issue [17:29:31] I always ask qchris when in doubt [17:29:36] "unlock experimental options" :) [17:29:42] sounds promising [17:30:03] mutante: ssh -p 29148 gerrit make-sandwich :P [17:30:10] hehee [17:30:10] don't tell anyone [17:32:33] merging the G1GC tuning change [17:32:36] not restarting anything [17:35:59] thcipriani: `make-wmf-branch` is working on core now (just about to finish) if you want to do the restart between this and `scap prep` [17:36:03] fwiw, puppet does trigger "refresh" for systemd::unit[gerrit]. but that is not like a real restart [17:37:42] marxarelli: sure! let me know when all's clear [17:38:10] mutante: yeah, IIRC that's just refreshing of systemd's understanding of the unit files. Because systemd :) [17:38:10] 10Release-Engineering-Team, 10Scap, 10Wikimedia-Incident: Scap should delete files after other updates - https://phabricator.wikimedia.org/T233769 (10Krinkle) Hm.. yeah, `--delete-delay` might not be the right answer here. I think, the "perfect" answer is "Don't do this", because we're supposed to split thi... [17:38:29] oh whoops. i misread the output. we're in extensions/W* atm [17:38:45] still... then will be now soon [17:39:24] heh, the "W*" extensions is a big group around here :) [17:39:49] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Developer-Advocacy, 10wikimedia.biterg.io: biterg.io Gerrit crawling probably stresses the server too much - https://phabricator.wikimedia.org/T234328 (10Jdforrester-WMF) @Aklapper, can you fix the co... [17:40:49] thcipriani: ack. and i think we want to keep it this way and not have automatic restart after merge [17:42:32] +1 [17:45:50] thcipriani lol [17:46:43] hauskater the UI allows you to configure whos the owner now :) [17:47:08] * paladox done that [17:49:40] thcipriani: branch cut is done [17:50:27] marxarelli: thanks [17:50:27] mutante wondering if we can do https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/539204/ today please? (and also https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/540164/) :) [17:51:00] I spoke to thcipriani who said we should rsync the repos to reduce delta's. [17:51:20] ^ open to other thoughts there, but that was my thinking offhand [17:51:49] so we merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/539204/ , make sure gerrit is stoped. Rsync the repos. Run https://gerrit-review.googlesource.com/Documentation/pgm-reindex.html and then merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/540164/ [17:52:32] the last patch to merge will require a restart instantly otherwise replication will stop working until a restart occurs. [17:54:35] Project beta-code-update-eqiad build #266064: 04FAILURE in 1 min 35 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/266064/ [17:55:05] Code Review - Error [17:55:05] Class$S655 [17:55:05] Download of https://gerrit.wikimedia.org/r/gerrit_ui/deferredjs/51A1AC2D5D6D649F7AB4FAE855D4FD72/7.cache.js?autoRetry=3 failed with status 404(Script Tag Failure - no status available) [17:55:42] And now gerrit is down [17:55:58] paladox: ^ [17:56:04] oh [17:56:13] someone reloaded gerrit [17:56:15] see -operations [17:56:47] Ah, I see [17:56:52] I restarted for the config changes [17:56:52] Yup, back up again [17:56:55] which should be live now [17:57:08] thanks! [17:58:41] oh [17:58:50] thcipriani heh your log was hidden in the other logs :P [17:59:00] even though your name is purple to me, i couldn't spot it [17:59:07] so thought godog was restarting gerrit [17:59:39] heh, sometimes -operations channel is a bit...chaotic :) [17:59:44] :P [17:59:48] paladox: just one thing to add. the "rsync the repos" part also is a gerrit change. i dont want to manually hack it [18:00:07] because that means stopping or hacking ferm and later having to cleanup rsyncd remnants and more [18:00:20] thcipriani also the "Skipping change 468692 because the corresponding repository was not found" seem to have stopped on https://gerrit.wikimedia.org/r/monitoring since the online reindex this morning done by hashar [18:00:21] easier to do it in puppet to me [18:00:36] took a while (since i was wondering why it was still happening, stopped at around 1pm) [18:00:38] mutante ah [18:00:46] paladox: "quickdatacopy" :) [18:00:50] paladox: ah, that's good to hear. [18:00:53] mutante would you be able to do that change please? :) [18:01:01] paladox: yes [18:01:07] thanks! [18:04:11] paladox: FWIW what I'm doing on irssi is turn bot messages to look like they were NOTICE, easier to tell apart [18:04:29] Yippee, build fixed! [18:04:29] Project beta-code-update-eqiad build #266065: 09FIXED in 1 min 28 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/266065/ [18:04:32] oh, heh. [18:05:00] thcipriani username colour is similar to stashbot's [18:06:52] lol [18:20:04] rsync data _from_ cobalt to gerrit1001. https://gerrit.wikimedia.org/r/c/operations/puppet/+/540190 [18:20:20] let us _push_ from cobalt into gerrit1001 /srv/gerrit/ whatever we want [18:20:28] thanks mutante ! [18:20:40] https://puppet-compiler.wmflabs.org/compiler1001/18700/gerrit1001.wikimedia.org/ [18:20:47] no change on cobalt [18:21:23] also that upgrades it from 'spare::system' to a "standard" profile [18:21:45] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Developer-Advocacy, 10wikimedia.biterg.io: biterg.io Gerrit crawling probably stresses the server too much - https://phabricator.wikimedia.org/T234328 (10Aklapper) a:03Aklapper Thanks; I've raised a... [18:21:46] this is before we apply the actual gerrit role then [18:22:07] well.. jerkins dont like.. fixing i guess [18:23:42] paladox: strange, like the issue yesterday where it was just variable assignment [18:23:51] oh [18:24:35] oh, it's "rysnc" and i once added variations of that to the typo file.. [18:24:38] maybe that :) [18:24:58] mutante fixed! [18:25:02] oh [18:25:07] heh you caught it too :) [18:25:09] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Developer-Advocacy, 10wikimedia.biterg.io: biterg.io Gerrit crawling probably stresses the server too much - https://phabricator.wikimedia.org/T234328 (10thcipriani) >>! In T234328#5538670, @Aklapper... [18:25:15] puppet repo "typos" has [18:25:16] rysnc [18:25:16] rsnyc [18:25:52] hrm, thinking about how I remember chad doing this for gerrit2001 he might have just used tar and scp [18:26:24] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO, 10Developer Productivity: FY201819 TEC12:O1 Outcome – Local development is unified with testing and production - https://phabricator.wikimedia.org/T222237 (10brennen) [18:26:26] 10Release-Engineering-Team-TODO (201909), 10Developer Productivity, 10dev-images, 10local-charts, 10Patch-For-Review: Move dev-images PHP image from php -S to Apache + php-fpm - https://phabricator.wikimedia.org/T222494 (10brennen) 05Open→03Resolved [18:26:26] rsync will use a lot of cpu calculating checksums and the like [18:26:31] not sure if that matters or not [18:26:33] thcipriani: we did the exact same thing when migrating to cobalt https://gerrit.wikimedia.org/r/c/operations/puppet/+/314726 [18:26:44] ah, ok [18:26:46] paladox found the old change [18:26:50] must've misremembered [18:26:57] maybe that was before cobalt even [18:27:17] or something else he was doing all together :P [18:27:22] heh [18:27:23] in that case: change lgtm! [18:27:23] when migrating to 'lead' [18:27:30] ok, thanks [18:27:37] * paladox remembers lead [18:27:46] i like the part that we dont change stuff on the prod server but just push from there to new [18:28:06] yeah, too me a minute to realize what was happening :) [18:28:13] paladox: in California there always has to be a sign to warn about lead [18:28:21] heh [18:28:24] lead in the water? [18:28:33] in paint [18:28:36] oh [18:29:36] https://en.wikipedia.org/wiki/Lead_paint#Regulation [18:30:28] 10Gerrit: Lots of "Skipping change xxx because the corresponding repository was not found" in the logs - https://phabricator.wikimedia.org/T233989 (10Paladox) Filed this upstream at https://bugs.chromium.org/p/gerrit/issues/detail?id=11650 [18:30:33] i should just rename my commit. it's pushing to cobalt [18:30:41] arg. "from" [18:30:43] lol [18:33:46] next: style check hates the role :) brb [18:34:05] heh [18:40:50] 10Release-Engineering-Team-TODO (201910), 10MediaWiki-General, 10Release: Mark MediaWiki core as 1.35.0-alpha once REL1_34 is branched - https://phabricator.wikimedia.org/T232030 (10Jdforrester-WMF) 05Open→03Resolved Done in https://gerrit.wikimedia.org/r/c/mediawiki/core/+/539917. [18:41:11] 10MediaWiki-Releasing, 10Release-Engineering-Team-TODO (201909), 10Core Platform Team, 10MW-1.34-notes, 10MW-1.34-release: Branch REL1_34 for MediaWiki and deployed extensions - https://phabricator.wikimedia.org/T232024 (10Jdforrester-WMF) 05Stalled→03Open [18:41:13] 10Release-Engineering-Team-TODO (201910), 10MediaWiki-General, 10Release: Mark MediaWiki core as 1.35.0-alpha once REL1_34 is branched - https://phabricator.wikimedia.org/T232030 (10Jdforrester-WMF) [18:41:16] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.34-notes, 10MW-1.34-release: Release MW 1.34 - https://phabricator.wikimedia.org/T232023 (10Jdforrester-WMF) [18:47:08] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Operations, 10Core Platform Team Legacy (Watching / External), and 3 others: FY2017/18 Program 6: Streamlined Service delivery - https://phabricator.wikimedia.org/T170453 (10Pchelolo) [19:45:05] 10Release-Engineering-Team-TODO (201910), 10Code-Review-Workgroup: Define code review metrics - https://phabricator.wikimedia.org/T229510 (10Jdforrester-WMF) [19:58:49] thcipriani: paladox: rsync running in a screen session on cobalt now. pushes /srv/gerrit/git up to gerrit1001 [19:58:57] awesome! [19:58:58] going for lunch [19:59:02] thanks mutante! [20:08:57] ./git dir already done [20:09:00] bbiaw [20:22:43] kudos! thank you! [20:54:55] thcipriani do we want to copy the ssh keys for gerrit or generate new keys? [21:05:24] hopefully ORES repo cloning ain't broken again... [21:05:30] K18/K19 [21:17:47] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (201910), 10dev-images: MediaWiki pipeline config: Correctly tag development images with dev - https://phabricator.wikimedia.org/T234379 (10brennen) [21:27:57] 10Release-Engineering-Team: gate-and-submit-wmf might have redundant quibble jobs (Oct 2019) - https://phabricator.wikimedia.org/T234381 (10Krinkle) [21:31:04] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (201910): gate-and-submit-wmf might have redundant quibble jobs (Oct 2019) - https://phabricator.wikimedia.org/T234381 (10Jdforrester-WMF) I think this is an interstitial state (?); @hashar can give more details. [21:34:41] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Developer-Advocacy, 10wikimedia.biterg.io: biterg.io Gerrit crawling probably stresses the server too much - https://phabricator.wikimedia.org/T234328 (10hashar) @Aklapper I have disabled the account... [21:44:07] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10HHVM: Drop HHVM from CI - https://phabricator.wikimedia.org/T234384 (10Jdforrester-WMF) [21:44:38] (03PS3) 10Jforrester: [DNM] layout: Drop HHVM jobs from -wmf branches [integration/config] - 10https://gerrit.wikimedia.org/r/534520 (https://phabricator.wikimedia.org/T234384) [21:45:06] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10HHVM, 10Patch-For-Review: Drop HHVM from CI - https://phabricator.wikimedia.org/T234384 (10Jdforrester-WMF) [21:45:20] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10HHVM, 10Patch-For-Review: Drop HHVM from CI - https://phabricator.wikimedia.org/T234384 (10Jdforrester-WMF) [21:45:34] (03CR) 10jerkins-bot: [V: 04-1] [DNM] layout: Drop HHVM jobs from -wmf branches [integration/config] - 10https://gerrit.wikimedia.org/r/534520 (https://phabricator.wikimedia.org/T234384) (owner: 10Jforrester) [21:45:37] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10HHVM, 10Patch-For-Review: Drop HHVM from CI - https://phabricator.wikimedia.org/T234384 (10Jdforrester-WMF) 05Open→03Stalled [21:47:48] (03PS4) 10Jforrester: [DNM] layout: Drop HHVM jobs from -wmf branches [integration/config] - 10https://gerrit.wikimedia.org/r/534520 (https://phabricator.wikimedia.org/T234384) [21:49:08] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (201910): gate-and-submit-wmf might have redundant quibble jobs (Oct 2019) - https://phabricator.wikimedia.org/T234381 (10Krinkle) [21:50:00] (03PS4) 10Jforrester: [DNM] layout: Drop HHVM testing from all quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/534521 (https://phabricator.wikimedia.org/T192166) [21:52:41] (03PS5) 10Jforrester: [DNM] layout: Drop HHVM jobs from -wmf branches [integration/config] - 10https://gerrit.wikimedia.org/r/534520 (https://phabricator.wikimedia.org/T234384) [21:52:43] (03PS5) 10Jforrester: [DNM] layout: Drop HHVM testing from all quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/534521 (https://phabricator.wikimedia.org/T234384) [21:52:45] (03PS4) 10Jforrester: layout: Collapse -nohhvm jobs into their base as we've dropped that [integration/config] - 10https://gerrit.wikimedia.org/r/534522 (https://phabricator.wikimedia.org/T234384) [21:52:47] (03PS3) 10Jforrester: layout: Drop all HHVM testing except for Fundraising [integration/config] - 10https://gerrit.wikimedia.org/r/534523 (https://phabricator.wikimedia.org/T234384) [22:09:38] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (201910), 10dev-images, 10Patch-For-Review: MediaWiki pipeline config: Correctly tag development images with dev - https://phabricator.wikimedia.org/T234379 (10brennen) p:05Triage→03Normal [22:11:44] paladox: i still need to fix the ferm hole for rsync .. ugh [22:11:51] hmm [22:11:57] and then it was 2 different pathes [22:12:09] but we can use the same "rsync module" for it i think [22:16:35] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (201910), 10dev-images, 10local-charts: Point deployment-charts/mediawiki-dev at latest dev image published by pipeline - https://phabricator.wikimedia.org/T234391 (10brennen) [22:17:22] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (201910), 10dev-images, 10local-charts: Point deployment-charts/mediawiki-dev at latest dev image published by pipeline - https://phabricator.wikimedia.org/T234391 (10brennen) [22:17:24] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (201910), 10dev-images, 10Patch-For-Review: MediaWiki pipeline config: Correctly tag development images with dev - https://phabricator.wikimedia.org/T234379 (10brennen) [22:17:44] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (201910), 10dev-images, 10local-charts: Point deployment-charts/mediawiki-dev at latest dev image published by pipeline - https://phabricator.wikimedia.org/T234391 (10brennen) p:05Triage→03Normal [22:17:52] paladox: oh. there is no ferm snippet for that at all on gerrit1001.. even though it has all the rules from "standard" now [22:18:02] and even though we use ferm::service in the role [22:18:07] oh [22:22:54] (03CR) 10jerkins-bot: [V: 04-1] [DNM] layout: Drop HHVM jobs from -wmf branches [integration/config] - 10https://gerrit.wikimedia.org/r/534520 (https://phabricator.wikimedia.org/T234384) (owner: 10Jforrester) [22:42:13] 10Gerrit, 10Operations: setup/install gerrit1001 - https://phabricator.wikimedia.org/T231046 (10Dzahn) [22:43:57] 10Gerrit, 10Operations: setup/install gerrit1001 - https://phabricator.wikimedia.org/T231046 (10Dzahn) There were/are a bunch of changes in a common topic branch not linked to this ticket that were needed: https://gerrit.wikimedia.org/r/q/topic:%22gerrit1001%22+(status:open%20OR%20status:merged) I just close... [22:44:24] 10Gerrit, 10Operations: setup/install gerrit1001 - https://phabricator.wikimedia.org/T231046 (10Dzahn) 05Open→03Resolved [22:44:33] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Gerrit Hardware Upgrade (+ upgrade from jessie to stretch or buster) - https://phabricator.wikimedia.org/T222391 (10Dzahn) [22:46:03] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Gerrit Hardware Upgrade (+ upgrade from jessie to stretch or buster) - https://phabricator.wikimedia.org/T222391 (10Dzahn) p:05Normal→03High [23:07:26] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Gerrit Hardware Upgrade (+ upgrade from jessie to stretch or buster) - https://phabricator.wikimedia.org/T222391 (10Dzahn) @thcipriani You and the other members of gerrit-roots admin gr... [23:11:15] paladox: awww man.. actually it's still not working (ferm) [23:11:23] but ssh access is [23:11:55] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Gerrit Hardware Upgrade (+ upgrade from jessie to stretch or buster) - https://phabricator.wikimedia.org/T222391 (10thcipriani) >>! In T222391#5539841, @Dzahn wrote: > @thcipriani You a... [23:12:02] oh :( [23:13:01] it's as if that ferm::service wouldnt be there [23:13:31] maybe it's the lack of base::firewall [23:13:52] i just assumed i dont have to do it anymore now because standard.. but maybe wrong [23:14:39] mutante i doin't see firewall in https://github.com/wikimedia/puppet/blob/production/modules/standard/manifests/init.pp [23:14:46] yea [23:14:59] but it does get standard rules [23:15:03] it's not like it has no rules [23:15:12] isnt it in base.. and base is in standard [23:15:18] oh [23:15:38] it is not all over site.pp anymore as it used to be [23:17:10] "just compile it" [23:23:29] paladox: heh, yea. https://puppet-compiler.wmflabs.org/compiler1002/18708/gerrit1001.wikimedia.org/ ... [23:23:42] File[/etc/ferm/conf.d/10_gerrit-migration-rsync] [23:23:47] ah [23:23:54] fixed by https://gerrit.wikimedia.org/r/c/operations/puppet/+/540244 [23:23:59] it should [23:24:28] great! [23:26:01] yes, it works now :) [23:26:07] also the part that we just use host name [23:26:12] in rsyncd config [23:26:33] dzahn@cobalt:/srv/gerrit/git$ rsync -avp /srv/gerrit/git/ rsync://gerrit1001.wikimedia.org/gerrit-data/git/ [23:26:42] this works now [23:26:58] thcipriani: ^ we can now run this anytome [23:27:07] paladox: tell me one more time the second dir :p [23:27:17] /srv/gerrit/plugins [23:27:22] (contains lfs objects) [23:27:59] uploading plugins dir from cobalt to gerrit1001 [23:29:09] thanks! [23:29:32] so in the command line above, see "gerrit-data" [23:29:51] that is not a real path in the file system. it's the name of the rsyncd "module" [23:30:05] but it translates to /srv/gerrit/ ..which can be configured in Hiera now [23:30:31] and it works to append real pathes to it .. /git/ or /plugins/ [23:30:46] so there needs to be only one module and we can rsync any subdir of /srv/gerrit/ [23:30:49] if needed [23:31:02] rsync of plugins is done now [23:31:07] :) [23:31:09] awesome! [23:32:19] mutante so we are ready to apply the gerrit role? :) [23:32:32] permissions are still not right [23:32:45] the old issue with the uid not matching [23:32:50] oh [23:34:31] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Gerrit Hardware Upgrade (+ upgrade from jessie to stretch or buster) - https://phabricator.wikimedia.org/T222391 (10Dzahn) >>! In T222391#5539856, @thcipriani wrote: > Confirmed that I... [23:34:54] i'll fix it, but not right now [23:35:03] because i also need to get another thing going [23:35:21] ok [23:35:22] and per arie.l's law it never took 5 minutes. more like most of the day, heh [23:35:35] good progress nevertheless.. that one ticket is closed [23:35:37] right [23:35:53] yup! [23:48:50] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO, 10Developer Productivity, 10dev-images, 10local-charts: Handle multiversion in local-charts MediaWiki deployment - https://phabricator.wikimedia.org/T222488 (10brennen) p:05Triage→03Low