[00:32:02] is releases-jenkins.wikimedia.org still being used? [05:11:54] 10Phabricator, 10DBA: Upgrade m3 to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T259589 (10Marostegui) >>! In T259589#6374543, @mmodell wrote: > @marostegui: That works for me. Thanks - I have updated the calendar event. [07:31:58] (03CR) 10Thiemo Kreuz (WMDE): Add PropertyCommentSniff to validate @var on class properties (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/619304 (owner: 10Umherirrender) [07:42:14] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+2] Fix handling of comment string for MissingReturnType [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/619307 (owner: 10Umherirrender) [07:43:56] (03Merged) 10jenkins-bot: Fix handling of comment string for MissingReturnType [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/619307 (owner: 10Umherirrender) [08:01:25] (03PS5) 10Thiemo Kreuz (WMDE): Add (optional) FullQualifiedClassNameSniff from WikibaseCodeSniffer [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/603496 (https://phabricator.wikimedia.org/T164653) [08:09:40] !log Gerrit: marked operations/software/prometheus_jmx_exporter read-only/archive since we now use a Debian package. T181728 [08:09:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:09:43] T181728: Stop using jmx_exporter deployed via scap in favour of Debian package - https://phabricator.wikimedia.org/T181728 [08:21:18] (03Abandoned) 10Hashar: dockerfiles: [castor] Move from wikimedia-stretch to ci-stretch [integration/config] - 10https://gerrit.wikimedia.org/r/589650 (owner: 10Jforrester) [08:28:27] PROBLEM - Host deployment-changeprop is DOWN: CRITICAL - Host Unreachable (172.16.5.21) [08:28:27] PROBLEM - Host deployment-cpjobqueue is DOWN: CRITICAL - Host Unreachable (172.16.4.124) [08:28:27] PROBLEM - Host deployment-chromium02 is DOWN: CRITICAL - Host Unreachable (172.16.4.14) [08:46:08] (03Abandoned) 10Hashar: Avoid duplicate codecoverage report [integration/config] - 10https://gerrit.wikimedia.org/r/617453 (https://phabricator.wikimedia.org/T242444) (owner: 10Hashar) [08:52:41] (03PS1) 10Hashar: Revert "Publish coverage reports from extension-codehealth" [integration/config] - 10https://gerrit.wikimedia.org/r/619370 (https://phabricator.wikimedia.org/T242444) [08:52:55] (03CR) 10jerkins-bot: [V: 04-1] Revert "Publish coverage reports from extension-codehealth" [integration/config] - 10https://gerrit.wikimedia.org/r/619370 (https://phabricator.wikimedia.org/T242444) (owner: 10Hashar) [09:11:09] (03PS2) 10Hashar: Revert "Publish coverage reports from extension-codehealth" [integration/config] - 10https://gerrit.wikimedia.org/r/619370 (https://phabricator.wikimedia.org/T242444) [09:12:29] (03CR) 10Hashar: Revert "Publish coverage reports from extension-codehealth" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/619370 (https://phabricator.wikimedia.org/T242444) (owner: 10Hashar) [09:50:26] 10Gerrit, 10Patch-For-Review: Use WMF Gerrit's logo as favicon - https://phabricator.wikimedia.org/T257218 (10Lucas_Werkmeister_WMDE) 05Open→03Resolved Shame about T257218#6342783, I really liked it (and didn’t vote in the poll for that reason). But okay. [11:00:37] PROBLEM - Host deployment-mcs01 is DOWN: CRITICAL - Host Unreachable (172.16.5.64) [11:16:48] 10Beta-Cluster-Infrastructure, 10The-Wikipedia-Library, 10Patch-For-Review: Email spam from Betawiki - https://phabricator.wikimedia.org/T256297 (10Urbanecm) >>! In T256297#6372256, @Samwalton9 wrote: > @Catrope Would it be possible for you to disable this extension on Beta Wiki? Done. [12:07:47] 10VPS-project-codesearch, 10Maps (Kartotherian): Missing service in codesearch - https://phabricator.wikimedia.org/T260124 (10TheDJ) [12:08:00] 10VPS-project-codesearch, 10Maps (Kartotherian): Missing Kartotherian service in codesearch - https://phabricator.wikimedia.org/T260124 (10TheDJ) [12:42:34] 10Beta-Cluster-Infrastructure, 10VPS-Projects, 10Product-Infrastructure-Team-Backlog (Kanban), 10Puppet: Puppet failures on deployment-docker-changeprop01, deployment-docker-cpjobqueue01, deployment-push-notifications01, deployment-docker-mobileapps01, and deployme... - https://phabricator.wikimedia.org/T259812 [13:02:31] (03PS1) 10Hashar: zuul: ensure MediaWiki repos have proper queue name [integration/config] - 10https://gerrit.wikimedia.org/r/619463 [13:05:29] (03CR) 10Hashar: [C: 03+2] "Manually aborted zuul diff since it is so large that the diff highlighter fails." [integration/config] - 10https://gerrit.wikimedia.org/r/619463 (owner: 10Hashar) [13:06:23] (03Merged) 10jenkins-bot: zuul: ensure MediaWiki repos have proper queue name [integration/config] - 10https://gerrit.wikimedia.org/r/619463 (owner: 10Hashar) [13:06:39] 10Phabricator, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Operations, 10Traffic, and 2 others: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10Vgutierrez) hmm that's interest... [13:20:10] 10Phabricator: Change my Phabricator username to GeneralNotability - https://phabricator.wikimedia.org/T260135 (10creffett) [13:44:17] (03PS2) 10Umherirrender: Add PropertyDocumentationSniff to validate @var on class properties [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/619304 [13:44:26] (03CR) 10Umherirrender: Add PropertyDocumentationSniff to validate @var on class properties (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/619304 (owner: 10Umherirrender) [13:48:58] 10Phabricator, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Operations, 10Traffic, and 2 others: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10CDanis) The Envoy TLS terminato... [14:19:19] 10Release-Engineering-Team, 10Product-Infrastructure-Team-Backlog, 10Recommendation-API, 10Services, and 2 others: Migrate recommendation-api to kubernetes - https://phabricator.wikimedia.org/T241230 (10Mholloway) PI will want to track this since #recommendation-api contains a couple of endpoints we mainta... [14:42:54] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): Old MediaWiki versions are not purged from staging/app server - https://phabricator.wikimedia.org/T260145 (10hashar) [14:48:55] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): Old MediaWiki versions are not purged from staging/app server - https://phabricator.wikimedia.org/T260145 (10hashar) [14:48:58] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T257972 (10hashar) [14:51:34] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): scap clean -delete does not have permission to delete old l10n caches - https://phabricator.wikimedia.org/T260146 (10hashar) [14:57:54] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T257972 (10hashar) Had to abort the deployment to testwiki due to some disk space issue (... [14:59:15] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): scap clean -delete does not have permission to delete old l10n caches - https://phabricator.wikimedia.org/T260146 (10Jdforrester-WMF) Yeah, I thoug... [15:11:18] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Scap: Scap's error message when preppping a non-existent branch is unhelpful - https://phabricator.wikimedia.org/T259699 (10dancy) [15:12:39] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Scap, 10User-dancy: scap prints error information twice - https://phabricator.wikimedia.org/T260151 (10dancy) [15:13:33] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Scap: Scap's error message when preppping a non-existent branch is unhelpful - https://phabricator.wikimedia.org/T259699 (10dancy) Also T260151 [15:13:45] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Scap: Scap's error message when prepping a non-existent branch is unhelpful - https://phabricator.wikimedia.org/T259699 (10dancy) [15:35:24] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Scap, 10User-dancy: scap prints error information twice - https://phabricator.wikimedia.org/T260151 (10thcipriani) Related: {T222372} [15:39:27] 10Beta-Cluster-Infrastructure, 10Product-Infrastructure-Team-Backlog, 10VPS-Projects, 10Puppet: Puppet failures on deployment-docker-changeprop01, deployment-docker-cpjobqueue01, deployment-push-notifications01, deployment-docker-mobileapps01, and deployment-docker... - https://phabricator.wikimedia.org/T259812 [15:41:49] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): Old MediaWiki versions are not purged from staging/app server - https://phabricator.wikimedia.org/T260145 (10thcipriani) Updated docs: https://wikitech.wikimedia.org/w/index.php?title=Heterogeneous_deployment/Train_d... [15:42:52] 10Phabricator, 10Wikipedia-Android-App-Backlog: Add project descriptions to Android App Phab projects which have no project description - https://phabricator.wikimedia.org/T253516 (10Dbrant) 05Open→03Resolved a:03Dbrant Those projects are not in use, and have now been archived. [15:49:27] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T257972 (10hashar) [15:53:18] 10Phabricator, 10Wikipedia-Android-App-Backlog: Add project descriptions to Android App Phab projects which have no project description - https://phabricator.wikimedia.org/T253516 (10Aklapper) Thanks! [16:08:16] 10Beta-Cluster-Infrastructure, 10cloud-services-team (Kanban): non-wdqs VMs sometimes getting scheduled on wdqs hardware - https://phabricator.wikimedia.org/T259542 (10nskaggs) p:05Triage→03Medium [16:22:02] 10Gerrit, 10Toolforge, 10Documentation, 10cloud-services-team (Kanban): Document and update Gerrit groups for repositories related to Toolforge - https://phabricator.wikimedia.org/T159051 (10nskaggs) p:05Triage→03Low [16:55:48] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T257972 (10hashar) [17:05:39] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T257972 (10hashar) **status** * We had to clear out very old mediawiki versions from the... [17:24:00] PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%) [17:27:46] 10MediaWiki-Codesniffer: Sniff SpaceyParenthesis reports wrong error on trailing space about spacing of parenthesis - https://phabricator.wikimedia.org/T260177 (10Umherirrender) [17:34:00] RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found) [17:48:36] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): Old MediaWiki versions are not purged from staging/app server - https://phabricator.wikimedia.org/T260145 (10thcipriani) 05Open→03Resolved a:03hashar Assigning to @hashar since he fixed the actual issue. I've u... [17:48:39] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T257972 (10thcipriani) [17:52:22] (03PS2) 10Dduvall: WIP: Support scratch images [blubber] - 10https://gerrit.wikimedia.org/r/619072 [17:54:26] (03CR) 10Lars Wirzenius: [C: 03+1] "Thanks, the use cases make sense." [blubber] - 10https://gerrit.wikimedia.org/r/619072 (owner: 10Dduvall) [17:58:26] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Scap, 10User-dancy: Scap fails when checking train version number - https://phabricator.wikimedia.org/T259706 (10dancy) p:05Triage→03Medium a:05LarsWirzenius→03dancy [18:07:16] hi. so if every Tuesday the new MW release is built automatically, has that already happened today? [18:11:45] mutante: it's already on group0 wikis [18:15:37] (03PS1) 10Ahmon Dancy: scap prep: Fix version_parser [tools/scap] - 10https://gerrit.wikimedia.org/r/619538 (https://phabricator.wikimedia.org/T259706) [18:16:22] (03CR) 10Ahmon Dancy: "No unit test.. not sure where to add that in. Advice is welcome!" [tools/scap] - 10https://gerrit.wikimedia.org/r/619538 (https://phabricator.wikimedia.org/T259706) (owner: 10Ahmon Dancy) [18:17:04] RhinosF1: thanks, but i mean specifically when the build is made by jenkins, not when it's installed on appservers [18:17:15] but if that covers both, yes :) thx [18:18:17] mutante: hang on, 03:05 was the commit from branch bot https://gerrit.wikimedia.org/r/c/mediawiki/core/+/619397/ [18:18:35] That'll be 02:00 UTC [18:19:19] thanks RhinosF1 [18:20:07] mutante: https://phabricator.wikimedia.org/T242446#6186639 [18:21:48] RhinosF1: ok, i am not sure if that is the same thing as when the image is built. i should have said the context is what releases-jenkins.wikimedia.org does in this case [18:22:27] mutante: not sure about that [18:41:09] 10Project-Admins, 10User-greg: Request to create #User-ANEUN personal project - https://phabricator.wikimedia.org/T243720 (10Aklapper) @A-NEUN: Are there any plans to use this project? [18:49:34] hmm.. scrolling back in this channel i don't see many humans talking at all. is this not the right place anymore to contact releng? [18:53:04] (03CR) 10Ahmon Dancy: [C: 03+1] WIP: Support scratch images (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/619072 (owner: 10Dduvall) [19:18:17] 10Deployments, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): scap clean -delete does not have permission to delete old l10n caches - https://phabricator.wikimedia.org/T260146 (10hashar) [19:37:28] bah [19:37:46] I might have broken deployment-prep deploy server : / [19:38:59] ah no [19:39:01] works [19:52:09] hashar: could you maybe help with configuring jenkins some time? [19:53:44] Hmm, how do I figure out what went wrong here? No logs? https://integration.wikimedia.org/ci/job/trigger-wikispeech-server-pipeline-test/1/console [19:54:02] 8) [19:54:41] mutante: yeah sure [19:55:07] kalle: so the trigger-* jobs are a bit annoying. They do nothing by themselves but trigger another jenkins job [19:55:11] 00:00:05.805 wikispeech-server-pipeline-test #1 completed. Result was FAILURE [19:55:42] so hmm https://integration.wikimedia.org/ci/job/wikispeech-server-pipeline-test/1/ [19:55:49] Is that all I get to know? No output from execution? [19:55:56] then on left click "Open blue ocean" [19:56:00] that is a revamped UI for Jenkins [19:56:02] which leads you to https://integration.wikimedia.org/ci/blue/organizations/jenkins/wikispeech-server-pipeline-test/detail/wikispeech-server-pipeline-test/1/pipeline [19:56:15] hashar: it is about configuring the jenkins on releases-jenkins when it is on a new machine. to copy the config from the old machine. there is a browser-based wizard i used to create an admin user. i have no idea how easy or hard that is yet [19:57:08] mutante: ohhhh. Well that is the same thing we did for the contint* upgrades. Merely just copy the whole /var/lib/jenkins from the old machine to the new one [19:57:24] kalle: bad example sorry :( [19:57:33] hashar: So it reads the pipelinelib config. Should't there be some error or something, or perhaps an error from Blubber? [19:57:49] kalle: so yeah looking at the actual job, in the console log at https://integration.wikimedia.org/ci/job/wikispeech-server-pipeline-test/1/console [19:57:54] java.lang.RuntimeException: cycle detected in graph (digraph { run-test -> null; null -> null; setup -> run-test; teardown }) [19:58:17] so there is some dependency issue in the pipeline file I guess [19:58:38] Ah there we go, thanks! [19:58:45] hashar: if that's what needs to be done and we can forget about the wizard thing then i got it and will care about the rsync. the thing was just that once you apply the puppet role on something new you end up with a public jenkins that lets you configure an admin.. :p [19:58:49] I'll try to figure that one our. [19:58:51] out [20:00:06] kalle: I believe longma or marxarelli would be able to assist you [20:01:14] looking [20:01:19] Thanks! [20:02:59] mutante: with the web ui wizard one would end up having to click a myriad of options so yeah not ideal. We would want to adjust the IP of the attached agent to have the jenkins master on the new server to connet to the new server [20:03:06] but beside that should be straightforward [20:05:44] hashar: so copying all of /var/lib/jenkins over is the right thing to do? i will do that then [20:06:34] longma: hashar: Oh wait, could it be that this patch is not rebased and thus lacking .pipeline? [20:06:41] kalle: I think we need to make some changes to your .pipeline/config file. It looks like you've added some stages that are supposed to actually be properties of the run-test stage [20:06:42] Let me try to push in a rebased [20:08:17] Nah, that did nothing. [20:08:34] kalle: do you mind If I upload a patch to your config file? [20:08:46] Feel free! [20:09:26] mutante: oh. And we have a weekly scheduled job that cut the production wmf branches. So if we copy the job, it will run TWICE (once on the current server, and again on the new one). [20:09:27] Perhaps in a new change-id though? [20:09:29] and yeah I got disconnected :-( [20:10:39] 10Phabricator, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Operations, 10Traffic, and 2 others: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10Dzahn) test comment [20:12:36] ooh sorry kalle [20:13:04] I can undo and make a new patch [20:13:04] No problems [20:13:10] twentyafterfour: weeeee.. aphlict is back !:) [20:13:13] Just leave it. [20:13:21] So thankful! <3 [20:14:53] I've been batteling thing pipeline/blubber thing for so long now. Starting to get a grip of it and think I might see the light at the end of the tunnel. [20:15:25] (this) [20:15:53] YAY [20:16:17] 10Phabricator, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Operations, 10Traffic, and 2 others: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10mmodell) Will it blend? [20:16:17] :) happy to help [20:16:19] Again, thanks! So I have a couple of services I need to update with that change. [20:17:40] If anyone feels up to it, I have a couple of rather wicked blubber solutions implemented in the build stage that could use a review. :) [20:18:43] In a couple of projects I ended up using reuqires [ . ] in builder to copy everything to /srv and then execute a script that does all sort of crazy things. [20:19:04] Feels like I'm overriding the whole copy-section there. Probably not ment to be solved that way, [20:19:13] but hey, it works :) [20:19:30] 10Phabricator, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Operations, 10Traffic, and 2 others: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10mmodell) it appears to blend. [20:20:16] This is a good example of that: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/wikispeech/mary-tts/+/refs/heads/master/.pipeline/blubber.yaml [20:20:33] twentyafterfour: what are you trying to blend? Some fruit? [20:20:37] :-) [20:20:42] And this is the prepping-script: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/wikispeech/mary-tts/+/refs/heads/master/blubber-prepare.sh [20:20:47] hauskatze: aphlict is back!:) [20:20:52] it was not easy [20:20:56] sorry, who dat? [20:21:06] sure I will take a look. marxarelli might like to as well [20:21:09] hauskatze: having realtime pop-up notifications in phab [20:21:17] if you enable it [20:21:17] oh, nice, kudos [20:21:37] btw kalle you have probably seen this already, but there is a reference here for pipelineLib https://wikitech.wikimedia.org/wiki/PipelineLib [20:21:42] oder Vielen Dank für ihre Arbeit [20:24:35] mutante: for the release jenkins, I guess we can pair it next week (I am on train this week). And I can ask releng team whether someone would be interested in joining [20:25:22] hauskatze: if you have a phab workboard and someone moves tasks to another column.. you should see it without reloading the page [20:25:45] Yup, I remember that [20:26:09] * hauskatze eyes xSavitar [20:26:25] hashar: ok, thanks. let me first just sync the config. i learned that is only used once a week, each Tuesday anyways. so i think i could just switch it over. also revert is super easy, only DNS change with 5min TTL [20:30:29] mutante: or do what we did for contint. Keep Jenkins shutdown on the new server, sync the data once [20:30:31] 10Phabricator, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Developer Productivity: Enhance phabricator project metrics and reporting - https://phabricator.wikimedia.org/T255703 (10mmodell) [20:30:33] 10Phabricator: Phab feature request: Cycle time for a task entering a column to resolution, with support for wildcards - https://phabricator.wikimedia.org/T148805 (10mmodell) [20:30:42] then on the migration time. Stop jenkins, sync again, start jenkins on the new machine [20:30:53] I still have a task to write down the instructions for the contint switch over bah :-\ [20:33:09] hashar: ok! one last question for now. does it actually matter that releases-jenkins is on the same host as releases.wikimedia.org because it copies files to releases.wm.org after building it or is it theoretically unrelated [20:35:28] mutante: I have no idea at all [20:36:10] the original use case for releases-jenkins was to automatize the mediawiki tarball creation [20:36:22] 10Phabricator, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Operations, 10Traffic, and 2 others: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10Dzahn) >>! In T238593#6376059,... [20:36:26] hashar: ok, fair enough [20:36:28] and ultimately have it published on releases.wikimedia.org (with a manual step to gpg sign the tarball before publish) [20:37:10] 10Phabricator, 10DBA, 10Operations: replace phabricator db passwords with longer passwords - https://phabricator.wikimedia.org/T250361 (10mmodell) ugh. @jcrespo, I apologize, I let the ball drop on this one. It wouldn't take much effort on my part, we already have the puppet scaffolding to support separati... [20:37:22] so the only job being used there is the one to cut the wmf branch [20:37:31] so probably they could be split on different systems [20:37:48] I mean [20:37:56] hauskatze: some real-time notifications in phabricator [20:38:00] aphlict is back online [20:38:07] mutante: we could have releases.wikimedia.org and releases-jenkins on different VM. And maybe we actually should [20:38:14] twentyafterfour: awesome <3 [20:38:20] it only took 9 months [20:39:16] well, it could be worse, I'm still waiting for > 1 year for a medical appointment [20:40:21] hashar: great! i could easily switch just releases.wm.org to buster and wait with the jenkins part [20:44:02] mutante: yup hopefully that will work fine ;] [20:44:07] 10Phabricator, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Operations, 10Traffic, and 2 others: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10Dzahn) 05Open→03Resolved a:... [20:44:12] 10Phabricator, 10RelEng-Archive-FY201718-Q2, 10Patch-For-Review: Enable notification server (real-time pop-up notifications) in Phabricator - https://phabricator.wikimedia.org/T765 (10Dzahn) [20:46:13] mutante: oh. And I still have to deal with the contint* document root. Can we schedule a time to do it next week? [20:47:00] hashar: yea, but first the gerrit host key?:) [20:47:05] or both [20:47:11] and yeah the gerrit host key [20:47:35] we can do both after another in one window.. i dont mind [20:47:43] the task we had about supporting ecdsa is rather old. Looks like the paramiko we use should work fine [20:48:14] the document root ones, I can send you a list of the patches if you want to review them in advance. Then it should be quite fast [20:48:19] yea, it "should be" very quick. but i don't want to jinx it [20:48:26] yeah [20:48:34] for the gerrit change key, I guess we just have to announce it a day in advance maybe [20:48:42] and it seems straightforward as well [20:48:42] ok, yes, please send the list or just put them in the same topic branch [20:48:46] then it's one link [20:48:56] * hashar digs in Gerrit [20:49:08] yea, and since we already announced it on 3 lists we just have to reply to the old threads [20:49:16] like "now for real" and that' sit [20:49:17] https://gerrit.wikimedia.org/r/q/topic:%22ci-docroot-scap%22+(status:open%20OR%20status:merged) :] [20:49:30] got it [20:49:34] which I rolled once but failed cause of some issues in the php code that serves it [20:49:56] ok, will review those [20:51:52] mutante: and I have added the ci-docroot-scap topic to the actual change https://gerrit.wikimedia.org/r/c/operations/puppet/+/611369/ (the others are parent changes but they are merely just cleanup) [20:52:07] I will have another one later, but can't figure it out yet :-\ so that will have to wait [20:52:42] for the Gerrit key / zuul. I guess it is all about adding the key to known_hosts and restart zuul/zuul-merger process. [20:57:02] (03CR) 10Hashar: [C: 03+1] "Xqt, looks like we can drop this job entirely right? or is it waiting for something else?" [integration/config] - 10https://gerrit.wikimedia.org/r/610295 (https://phabricator.wikimedia.org/T213287) (owner: 10Xqt) [20:57:13] on those good words. I am off ;] [20:57:22] see you tomorrow for the rest of the mw train [20:58:03] hashar: it will be 2 steps, rename RSA host key variable and add ECDSA keys https://gerrit.wikimedia.org/r/q/topic:%22gerrit-hostkeys%22+(status:open%20OR%20status:merged) [20:58:09] bye [20:58:28] mutante: ahhh cool ;) [21:00:33] * hashar vanishes [21:06:22] anyone else having issues with Phab search? e.g. https://phabricator.wikimedia.org/search/query/plU5U7u76z5j/#R [21:06:23] Unhandled Exception ("PhutilAggregateException") [21:10:08] musikanimal: this is why we can't have nice things [21:10:15] hehe [21:10:17] i can confirm that when following your link but not when i just put a random word in the search box [21:10:20] All of the configured Fulltext Search services failed. [21:10:21] - AphrontSchemaQueryException: #1054: Unknown column 'ft_rank.rawCorpus' in 'field list' [21:10:49] I got it for a few searches, all I did was put in a subscriber and set the document status [21:11:06] but I also tried doing a plain search for just "test" (no other filters) and it timed out [21:12:30] searching for "groar" works. test is a bad test, heh [21:13:03] okay, so I think it happens only if there is no search query, just filters [21:14:54] musikanimal: can i ask you an unrelated thing about phab. did you ever have the realtime notifications enabled? [21:15:29] I don't think I ever enabled it myself, but yes. Isn't it disabled now? [21:16:19] musikanimal: it's re-enabled since an hour or something [21:16:29] if you like it please try it [21:16:50] there is now a dedicated VM to handle it [21:16:53] will do! [21:17:37] there is "Web", "Desktop" or "Web and Desktop" [21:18:19] mutante: it shows connected at the bottom of my notification list in phab [21:19:01] RhinosF1: in your profile -> settings -> notifications (drop down menu). what do you have selected [21:19:22] eh, profile->applications->notifications actually [21:19:39] 10Phabricator: Phabricator search fails with PhutilAggregateException when there's no search query - https://phabricator.wikimedia.org/T260197 (10MusikAnimal) [21:19:54] there's my bug report ^ [21:19:59] 3rd attempt:) settings->applications->notifications actually [21:21:17] musikanimal: i searched for PhutilAggregateException and yea, it sounded familiar. was fixed though https://phabricator.wikimedia.org/T161772 [21:21:25] linking [21:22:17] mutante: I got them before it was disabled so should still be [21:22:36] well, that's a different issue nevertheless [21:23:09] RhinosF1: ok, so the Desktop-level kind of notification (vs. browser?) [21:23:33] 10Gerrit, 10Patch-For-Review: Use WMF Gerrit's logo as favicon - https://phabricator.wikimedia.org/T257218 (10QChris) >>! In T257218#6375473, @Lucas_Werkmeister_WMDE wrote: > Shame about T257218#6342783, I really liked it (and didn’t vote in the poll for that reason). But okay. I also liked the T257218#634278... [21:23:52] mutante: I used to get the little pop up at the corner of my screen with changes. [21:24:00] It shows that real time is connected [21:24:55] RhinosF1: ok, thanks [21:38:38] 10VPS-project-codesearch, 10Maps (Kartotherian), 10Patch-For-Review, 10User-Ladsgroup: Missing Kartotherian service in codesearch - https://phabricator.wikimedia.org/T260124 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup https://codesearch.wmcloud.org/search/?q=Kartotherian&i=nope&files=&repos= [21:50:40] PROBLEM - Parsoid on deployment-parsoid11 is CRITICAL: connect to address 172.16.1.115 and port 8000: Connection refused [22:01:47] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Wikispeech-Jobrunner, and 3 others: Tell Jenkins and Zuul about the Speechoid pip... - https://phabricator.wikimedia.org/T259911 [22:03:14] (03PS1) 10Karl Wettin (WMSE): Fixed typo in Wikispeech service Mishkal [integration/config] - 10https://gerrit.wikimedia.org/r/619579 (https://phabricator.wikimedia.org/T259911) [22:09:09] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments: 1.36.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T257973 (10thcipriani) p:05Triage→03Medium a:03mmodell [22:10:14] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments: 1.36.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T257974 (10thcipriani) p:05Triage→03Medium a:03dduvall [22:11:07] kalle: just chiming in about https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/wikispeech/mary-tts/+/refs/heads/master/.pipeline/blubber.yaml [22:11:35] not knowing too much about your build process, i can't say to what degree your build step is crazy or not :) [22:11:53] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments: 1.36.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T257976 (10thcipriani) p:05Triage→03Medium a:03jeena [22:12:00] but i do think the production variant could probably be more streamlined [22:14:07] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments: 1.36.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T257977 (10thcipriani) p:05Triage→03Medium a:03LarsWirzenius [22:14:54] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments: 1.36.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T257978 (10thcipriani) p:05Triage→03Medium a:03brennen [22:15:08] as of now, it `includes: [build]` which is going to result in a prod image that contains all of your build dependencies and toolchain. it's likely possible to start with a more minimal base image for the production variant and use `copies` to selectively copy over distributable files from your build environment [22:15:49] if you loop me in on the tasks i can probably be more helpful [22:28:13] (03PS1) 10Ejegg: Run CiviCRM tests on PHP 7.3 only [integration/config] - 10https://gerrit.wikimedia.org/r/619583 [22:29:16] (03CR) 10jerkins-bot: [V: 04-1] Run CiviCRM tests on PHP 7.3 only [integration/config] - 10https://gerrit.wikimedia.org/r/619583 (owner: 10Ejegg) [22:34:18] (03PS2) 10Ejegg: Run CiviCRM tests on PHP 7.3 only [integration/config] - 10https://gerrit.wikimedia.org/r/619583 [22:36:00] hi Releng! We've upgraded our CiviCRM server to php 7.3 and are now trying to update CI to match [22:36:43] I'm not sure if https://gerrit.wikimedia.org/r/619583 will do that, or if the wikimedia-fundraising-civicrm-docker job is going to run on 7.0 until we get docker-registry.wikimedia.org/releng/civicrm updated to use php7.3 [23:29:23] 10Phabricator: Searching for "gerrit" in global Phabricator search leads to upstream request timeout - https://phabricator.wikimedia.org/T258803 (10mmodell) so I am able to reproduce this when logged out in both firefox and chrome. [23:44:45] legoktm: hmm.. looks like this might be broken? [23:44:46] https://codesearch-beta.wmcloud.org/deployed/?files=&i=nope&q=mediawiki.skinning.interface [23:44:53] panicked at 'called `Option::unwrap()` on a `None` value', src/view.rs:266:27 [23:46:08] mmm [23:46:09] heh [23:46:13] there's a FIXME on that line just for this case [23:46:18] // FIXME: what if config.json and hound get out of sync? [23:47:07] Question: T258306 was just claimed by https://phabricator.wikimedia.org/p/Bodhi_Tri/ [23:47:08] T258306: Security Readiness Review For Suggestor - https://phabricator.wikimedia.org/T258306 [23:47:46] disabled [23:47:54] (they're already blocked on meta) [23:48:20] yeah, so I was curious. Removed them from the task [23:51:19] Krinkle: fixed now. I have a long-term plan to fix it without manual intervention [23:51:54] (I need to fix the lack of caching headers on https://codesearch.wmcloud.org/search/api/v1/repos) [23:54:19] PROBLEM - Host deployment-sentry01 is DOWN: CRITICAL - Host Unreachable (172.16.5.16) [23:55:23] legoktm: awesome, working now yeah [23:55:42] legoktm: btw, I see that highlights are now on entire lines rather than intraline, should I file a task? [23:55:52] noticed it when comparing this to https://codesearch.wmcloud.org/deployed/?files=&i=nope&q=mediawiki.skinning.interface [23:58:08] Krinkle: if you'd like that changed, then yes, but at least right now it's intentional on my part. the original hound code would re-run the regex on each match to identify which portion to highlight, which just seemed inefficient/extra to me [23:58:39] Hm.. I see, perhaps this is a rare query. I see both sides yeah [23:59:00] for this one, it just seemed confusing as it looked like it was serving non-matching results and not obvious where to look or why they matched [23:59:11] I don't suppose the information is preserved from the backend then? [23:59:42] the backend doesn't provide it