[00:11:53] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.34 deployment blockers - https://phabricator.wikimedia.org/T274938 (10brennen) End of day status: - On all wikis since 22:41 UTC - Things seem pretty stable... [00:15:29] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.34 deployment blockers - https://phabricator.wikimedia.org/T274938 (10Jdlrobson) I missed a full stop. Will deal with this Monday. We will have to live with the... [00:18:20] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.34 deployment blockers - https://phabricator.wikimedia.org/T274938 (10brennen) No worries - and thanks for the assistance this week. I'm signing off for the day... [00:30:08] 10Deployments, 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Sustainability (Incident Followup), 10User-brennen: Avoid unfinished train deploys over holidays, weekends, or other stretches of no-deploy days - https://phabricator.wikimedia.org/T260401 (10thcipriani) 05Open→03Declined... [01:25:13] (03PS2) 10Jforrester: jjb: Add php72-apache flavour to quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/670965 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [01:25:15] (03PS3) 10Jforrester: Zuul: [mediawiki/extensions/Wikibase] Add non-voting quibble-apache job [integration/config] - 10https://gerrit.wikimedia.org/r/670967 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [01:25:17] (03PS1) 10Jforrester: dockerfiles: Rename quibble-apache to quibble-stretch-php72-apache [integration/config] - 10https://gerrit.wikimedia.org/r/670993 [01:26:29] (03CR) 10Jforrester: [C: 04-1] "You're trying to use wmf-quibble-vendor-mysql-php72-apache-docker but that's not generated; I've edited your patch to list the jobs actual" [integration/config] - 10https://gerrit.wikimedia.org/r/670967 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [01:26:49] (03CR) 10jerkins-bot: [V: 04-1] Zuul: [mediawiki/extensions/Wikibase] Add non-voting quibble-apache job [integration/config] - 10https://gerrit.wikimedia.org/r/670967 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [01:41:02] (03CR) 10Jforrester: [C: 04-1] "Also, why not just an experimental job instead of the complexity of a non-voting one? :-)" [integration/config] - 10https://gerrit.wikimedia.org/r/670967 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [02:00:48] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.34 deployment blockers - https://phabricator.wikimedia.org/T274938 (10Krinkle) [02:02:11] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.34 deployment blockers - https://phabricator.wikimedia.org/T274938 (10Krinkle) [02:06:00] (03CR) 10Jforrester: "(Not merging alone, as this is mostly Antoine's call now. :-))" [integration/config] - 10https://gerrit.wikimedia.org/r/670993 (owner: 10Jforrester) [02:23:23] James_F thanks for reviewing so much! Anything of yours you want reviewed? [08:14:35] 10Continuous-Integration-Config, 10Add-Link, 10Growth-Team, 10Release Pipeline: java.lang.RuntimeException: Invalid Credential: 'SONAR_API_KEY' - https://phabricator.wikimedia.org/T277236 (10kostajh) 05Open→03Resolved a:03kostajh >>! In T277236#6906912, @dduvall wrote: > We hadn't yet merged a [[ htt... [08:44:32] kostajh: isn't it great to have american colleagues to fix up stuff for us while we sleep? :] [08:45:08] 10Continuous-Integration-Config, 10Add-Link, 10Growth-Team, 10Release Pipeline: java.lang.RuntimeException: Invalid Credential: 'SONAR_API_KEY' - https://phabricator.wikimedia.org/T277236 (10hashar) @dduvall nice! Thank you for the detailed fix up report. [08:45:22] hashar: \o/ [08:45:38] gehel: hi, I kind of lost track of the java / sonarqube polishing effort you are leading. Any stuff I should have a look at or any pairing required? [08:49:57] hashar: not on the java side, but zpapierski has started to look at analyzing python [08:50:27] if you have time to look at https://gerrit.wikimedia.org/r/c/integration/config/+/669770 [08:50:31] gehel: yeah I noticed that change, I don't know what to do about it ;) [08:50:52] we should probably jump in a meet and discuss :) [08:51:10] in 15' ? zpapierski would you be around as well? [08:51:25] * gehel needs to help get Augustin ready [08:51:32] no :( my wife needs to go out, I'll be looking after my son [08:51:57] maybe later this morning, or early this afternoon? [08:52:10] second option sounds ok [08:52:28] anything between 13:30 CET to 16:30 CET works for me [08:52:28] 10:30 UTC would be fine for me, I think [08:52:34] or that, too [08:52:43] or 10:30 yeah works too [08:53:00] let's stay with 10:30, then [08:53:04] my only constraint today are lunch and kids coming back from school :] [08:53:09] deal [08:53:30] "coming back from school" - I don't remember those times anymore [08:54:13] (if one day someone would have told me I would arrange a meeting with a Polish and a Swiss over IRC, I would have rejected that as a pure fantasy) [08:54:28] :D [09:18:54] I'll try to be there, but there's an friend of France coming over for lunch. [09:19:09] frog legs? ;] [09:19:35] Strange story: she hasn't seen her for over 10 years, and just learned that she has kids just the same age as ours, and the second one is deaf as well [09:19:59] He got his implants a few weeks ago and had the first tuning session today [09:20:08] lot to discuss! [09:30:25] 10phabricator maintenance bot: Remove #Patch-For-Review when patch is abandoned in Gerrit - https://phabricator.wikimedia.org/T276390 (10Ladsgroup) The code at least in paper supports abandoned patches too: https://github.com/Ladsgroup/Phabricator-maintenance-bot/blob/master/patchforreview_remover.py#L43 Might... [09:30:53] gehel: zpapierski: I have created https://meet.google.com/jxo-zetk-oqp [09:31:12] wiat, I meant UTC [09:31:16] didn't we say UTC? [09:31:16] s/wiat/wait [09:31:25] oh men [09:31:30] ;D [09:31:37] yep, that's still 1h from now [09:31:38] yeah, it will always be confusing :) [09:31:49] we're all in CET, we should use that :) [09:32:00] we are all in CET, but still refer to UTC to setup meeting. We have been brain washed! [09:32:05] see you in an hour haha [09:32:08] :) [09:33:25] (03CR) 10Addshore: "the phab task said non voting, and I kind of agree, as this will result in more job runs rather than someone having to run around asking f" [integration/config] - 10https://gerrit.wikimedia.org/r/670967 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [09:34:47] (03CR) 10Addshore: [C: 04-1] jjb: Add php72-apache flavour to quibble jobs (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/670965 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [09:52:46] 10phabricator maintenance bot, 10User-Ladsgroup: Remove #Patch-For-Review when patch is abandoned in Gerrit - https://phabricator.wikimedia.org/T276390 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup Done. https://github.com/Ladsgroup/Phabricator-maintenance-bot/commit/2cbe8ea3af24b57df224ee701ea269b4b86b6... [10:09:48] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Jessie Deprecation): Migrate deployment-prep away from Debian Jessie to Debian Stretch/Buster - https://phabricator.wikimedia.org/T218729 (10Majavah) >>! In T218729#6897099, @MoritzMuehlenhoff wrote: > I think we can simply remove deployment-sca01/sca02? The r... [10:20:03] 10Gerrit, 10Accessibility, 10Patch-For-Review: "CR" and "V" columns are not colorblind-friendly - https://phabricator.wikimedia.org/T256615 (10hashar) I have lost the dynamic on this task and forgot about it completely. Probably cause I was busy with other things immediately after the Gerrit upgrade happened... [10:23:41] 10Gerrit, 10Release-Engineering-Team, 10Accessibility: Gerrit red-green diff make the code hard to read (impossible for color-blind users) - https://phabricator.wikimedia.org/T232893 (10hashar) I have revisited this task after T256615 which is about the label voting chips being hard to read. I really like t... [10:24:59] gehel,hashar - my SO's back, we can meet [10:25:12] +1 [10:25:54] https://meet.google.com/jxo-zetk-oqp [10:32:45] (03CR) 10Gehel: Add sonar scanner to discolytics (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/669770 (https://phabricator.wikimedia.org/T264877) (owner: 10ZPapierski) [10:46:02] (03PS9) 10Hashar: Add sonar scanner to discolytics [integration/config] - 10https://gerrit.wikimedia.org/r/669770 (https://phabricator.wikimedia.org/T264877) (owner: 10ZPapierski) [10:48:20] (03CR) 10Hashar: [C: 03+2] "Lets go for it! ;)" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/669770 (https://phabricator.wikimedia.org/T264877) (owner: 10ZPapierski) [10:50:45] (03Merged) 10jenkins-bot: Add sonar scanner to discolytics [integration/config] - 10https://gerrit.wikimedia.org/r/669770 (https://phabricator.wikimedia.org/T264877) (owner: 10ZPapierski) [11:11:10] gehel: more or less a success and I have brain dumped a lot of context to zpapierski (origin of Gerrit, talked a bit about Zuul etc ) ;) [11:11:31] seems we are just missing the sonarqube property file at the root of the repository cause sonar:scanner tries to reach localhost:9000 [11:11:41] sounds like easily fixable. Will check after lunch [11:16:18] I to be precise - it's not missing, but it isn't located - I'm assuming some paths are messed up [11:16:25] I'm checking that now [11:20:05] I think I found it [11:20:08] the workdir is wrong [11:20:23] the sonar-scanner image defaults to /workspace/src [11:20:38] and it should be /src ? [11:20:40] but we run the image with -v src:/src [11:20:41] yeah [11:20:57] I thought that's that [11:21:41] (03PS1) 10Hashar: Update workdir for tox sonar scanner [integration/config] - 10https://gerrit.wikimedia.org/r/671122 (https://phabricator.wikimedia.org/T264877) [11:21:46] zpapierski: ^ :) [11:21:49] thanks! [11:21:58] I should really noramlize all the images/jobs to always use /src [11:22:17] (03CR) 10ZPapierski: [C: 03+1] Update workdir for tox sonar scanner [integration/config] - 10https://gerrit.wikimedia.org/r/671122 (https://phabricator.wikimedia.org/T264877) (owner: 10Hashar) [11:22:31] I have deployed th ejob [11:22:35] manually ran a build: https://integration.wikimedia.org/ci/job/wikimedia-discovery-analytics-patch-tox-docker-with-sonar/3/console [11:22:40] we will see :] [11:23:24] fingers crossed [11:26:35] it failed again :( [11:30:11] need to take care of my son, again - be back in half an hour [11:30:41] zpapierski: dont worry :] [11:30:45] I will have lunch soon [11:30:48] we can resume later this afternoon [11:32:59] my ci config change is wrong ;) [11:33:55] (03PS2) 10Hashar: Update workdir for tox sonar scanner [integration/config] - 10https://gerrit.wikimedia.org/r/671122 (https://phabricator.wikimedia.org/T264877) [11:34:24] next try https://integration.wikimedia.org/ci/job/wikimedia-discovery-analytics-patch-tox-docker-with-sonar/4/console [11:47:34] INFO: Project root configuration file: /src/sonar-project.properties [11:47:35] better [11:51:10] (03CR) 10Hashar: "My first patch was completely wrong, it did not change anything. The second patchset overrides docker run --workdir when executing the re" [integration/config] - 10https://gerrit.wikimedia.org/r/671122 (https://phabricator.wikimedia.org/T264877) (owner: 10Hashar) [12:02:51] hashar: how do you feel about the way I did the apache jobs for Wikibase to try them out in https://phabricator.wikimedia.org/T276428 ? [12:03:03] if I get some vauge sense of approval I'll go ahead and deploy them and try them out :) [12:05:07] addshore: lunch break here! But yeah I planned to look at those this afternoon since that spammed my email inbox ;] [12:05:31] there was something about renaming the releng/quibble-apache image as well [12:05:48] ack! that might be why james changed it in the patch then! [12:05:52] yeah, I'd be pro that [12:05:56] also going for lunch! [12:05:59] enjoy your tonatoes! [12:06:00] ;D [12:06:10] lets poke each other after lunch! [12:06:14] sounds grand! [13:00:32] so A coffee [13:00:34] B poke folks [13:04:38] zpapierski: gehel: that almost works now. But we got: "Could not find a default branch to fall back on." :D [13:04:41] https://integration.wikimedia.org/ci/job/wikimedia-discovery-analytics-patch-tox-docker-with-sonar/4/console [13:07:52] I'm back [13:08:23] huh, that's a sonar thing [13:08:37] let me verify my setttings [13:11:40] gehel: should I preconfigure project on sonarqube? [13:12:44] Nope, the first analysis should create it [13:13:10] You might need to have a first analysis for the main branch before you can analyse CRs [13:13:34] I think that's what hashar actually did, zuul branch is master [13:13:46] I tried to build the other job [13:14:10] https://integration.wikimedia.org/ci/job/wikimedia-discovery-analytics-master-tox-docker-with-sonar/1/console [13:14:14] 10Gerrit, 10Accessibility, 10Patch-For-Review: "CR" and "V" columns are not colorblind-friendly - https://phabricator.wikimedia.org/T256615 (10Daimona) I also forgot about this task. Since I created it, I was also able to hack-inject my own stylesheet into gerrit, which I'm copying below for reference. Thing... [13:14:27] which used -Dsonar.branch.target=master -Dsonar.branch.name=master [13:14:42] and still leads to the same error "Could not find a default branch to fall back on." :-\ [13:14:43] yep, it didn't work either [13:14:51] or maybe it is because the project isn't created yet? [13:15:05] Daimona: :]]]] [13:15:36] in any case, I made a typo in projectName [13:15:41] I need to correct it [13:15:49] Not sure, my expectation is that projects are auto created at first analysis. But maybe we need an analysis without any branch information first ? [13:15:52] but I doubt that's the issue [13:15:55] hashar: thank you, I will try your version later, I recognize that mine might now work for everyone [13:16:58] Daimona: my change is the very bare minimum :-] I don't mind expanding it though [13:17:55] Daimona: and I found an old task about changing the diff view colors. Apparently that got lost in the Gerrit 3 upgrade , cause I can't find a way to change the diff theme :D [13:18:17] Well, things sure changed a lot [13:18:29] Although TBH I like the red-green version better [13:22:07] gehel: I manually created the project in sonarqube and apparently it works now [13:22:34] (03CR) 10Hashar: [C: 03+2] "I have did the rename using a docker push:" [integration/config] - 10https://gerrit.wikimedia.org/r/670993 (owner: 10Jforrester) [13:22:48] it alwso found zero issues which makes a bit suspicious ;) [13:23:43] Daimona: if you can give me a review about the bold / 1.1em change at https://gerrit.wikimedia.org/r/c/operations/puppet/+/671101 that would be wonderful. I will hunt for an SRE person to get it deployed for us :] [13:23:49] zpapierski: you can run sonarlint locally and compare results [13:24:03] Let me try it [13:24:11] (03Merged) 10jenkins-bot: dockerfiles: Rename quibble-apache to quibble-stretch-php72-apache [integration/config] - 10https://gerrit.wikimedia.org/r/670993 (owner: 10Jforrester) [13:24:22] zpapierski: the report should list the files analysed, even if there are no violations [13:24:31] and there are none [13:24:40] even though it says it analyzed 38 files... [13:24:42] Sounds suspicious to me ;) [13:24:52] isn't that the issue we had with some java sonar jobs? [13:25:08] aka the analyze somehow going on but the report being more or less empty [13:25:10] Oh, no violations, but s list of files analysed ? That sounds good then ! [13:25:30] hashar: yep, we still have an issue with java [13:25:38] no violations, and no files [13:25:42] at least https://sonarcloud.io/dashboard?id=org.wikimedia.gearman%3Agearman-java shows some bugs and issues ;] [13:25:50] number of files come from the build log [13:25:52] Hmmmm I'm not seeing any difference [13:26:11] Daimona: then my patch is wrong :\ [13:26:44] gehel: but I haven't analyzed main branch yet, I'll try to do so [13:26:46] hashar: we get reports on the main branch, but Cars never report any new issue [13:27:10] s/Cars/CRs/ [13:28:39] o/ [13:30:41] zpapierski: looks like the main branch hasn't been analyzed yet https://sonarcloud.io/project/branches?id=discovery-analytics [13:30:51] I know, I'm doing that now [13:31:31] Daimona: I have used my browser inspector tool to edit the style for the the voting chip element [13:31:44] But the code list for the branches seems empty, that's suspicious to me https://sonarcloud.io/code?branch=671136-1&id=discovery-analytics [13:32:25] I think I did the same a few months ago, but the class in my case seems to be just "chip"? [13:32:33] https://www.irccloud.com/pastebin/WKVNwy1t/ [13:32:46] we have an incorrect configuration for a main branch [13:33:07] apparently it only makes sense for branches [13:34:45] so hashar, those jobs? ;) [13:34:55] I can probably figure out how to rename the docker image probably [13:35:52] addshore: I have done it for releng/quibble-apache > releng/quibble-stretch-php72-apache https://gerrit.wikimedia.org/r/c/integration/config/+/670993 :] [13:36:05] docker tag then sudo docker-pusher [13:39:44] <3 [13:39:56] (03CR) 10Hashar: "will amend" (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/670965 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [13:40:19] addshore: it is a bit of a mess :D [13:43:17] addshore: james patch generates so many different variants we probably don't care about. and only triggers wmf-quibble-vendor-mysql-php72-apache-docker [13:43:23] which I am tempted to mark experimental [13:43:57] so, im against being experimental, as then im just gonna have to run around to patchests triggering it all the time :/ [13:44:15] ideally in a few days we might be able to make it the voting job for Wikibase [13:44:31] I guess all the other varients will be needed if we are to migrate? [13:45:01] yeah eventually [13:45:09] but I also need to first move from Stretch to Buster [13:45:23] which is like a month late or so [13:45:24] :-\ [13:45:42] hmm, as in move within the images? or within CI nodes? [13:46:56] actually both are needed [13:47:08] the CI nodes that can wait for sure, they are dumb enough (we just need a docker daemon there) [13:47:23] there are quibble images using buster instead of stretch [13:47:34] which also mean new firefox/chromium and new whatever libraries :-\ [13:49:22] addshore: and I got the point about using a non voting job. More builds, more races conditions caught! [13:51:53] new browsers tooo, oooh :P [13:51:59] (03PS3) 10Hashar: jjb: wikimedia Quibble job with Apache [integration/config] - 10https://gerrit.wikimedia.org/r/670965 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [13:52:37] addshore: ^ simplified James change to only add one job [13:53:35] (03CR) 10Addshore: [C: 03+1] jjb: wikimedia Quibble job with Apache [integration/config] - 10https://gerrit.wikimedia.org/r/670965 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [13:53:43] looks good to me! [13:55:36] hashar: should this be merged - https://gerrit.wikimedia.org/r/c/integration/config/+/671122/2 ? [13:56:02] (03CR) 10Hashar: [C: 03+2] "Job deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/670965 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [13:56:20] zpapierski: yeah apparently ;) [13:56:29] (03PS4) 10Hashar: Zuul: [mediawiki/extensions/Wikibase] Add non-voting quibble-apache job [integration/config] - 10https://gerrit.wikimedia.org/r/670967 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [13:56:40] (03CR) 10Hashar: [C: 03+2] "That fixed the workdir issue. The project properties file is now found properly." [integration/config] - 10https://gerrit.wikimedia.org/r/671122 (https://phabricator.wikimedia.org/T264877) (owner: 10Hashar) [13:56:46] thanks! [13:56:57] too many stuff in parallel [13:57:02] yeah, sorry for that :) [13:57:11] (03Merged) 10jenkins-bot: jjb: wikimedia Quibble job with Apache [integration/config] - 10https://gerrit.wikimedia.org/r/670965 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [13:58:22] (03Merged) 10jenkins-bot: Update workdir for tox sonar scanner [integration/config] - 10https://gerrit.wikimedia.org/r/671122 (https://phabricator.wikimedia.org/T264877) (owner: 10Hashar) [14:00:04] hashar: I'll alter the other PS to make wikibase use the job [14:00:38] oh, you beat me [14:00:47] (03CR) 10Addshore: [C: 03+1] Zuul: [mediawiki/extensions/Wikibase] Add non-voting quibble-apache job [integration/config] - 10https://gerrit.wikimedia.org/r/670967 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [14:00:58] i can deploy it if you would like :) [14:01:48] addshore: I am looking at the next change that adds the Zuul config [14:01:50] cause well [14:01:52] filters are not fun [14:01:59] :P [14:02:06] gehel: if I skip setting the branch name for sonar scanner - can I assume main branch will be analyzed? [14:02:56] zpapierski: I think it's safer to explicitly define the branch [14:03:00] addshore: bah the zuul-layout-diff output is wrong somehow, based on some other change :\ [14:03:12] I will then [14:03:43] (03PS5) 10Hashar: Zuul: [mediawiki/extensions/Wikibase] Add non-voting quibble-apache job [integration/config] - 10https://gerrit.wikimedia.org/r/670967 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [14:06:10] addshore: https://gerrit.wikimedia.org/r/c/integration/config/+/670967 looks good. Guess we can deploy it [14:06:31] yyyeaaahhhh [14:06:47] (03PS1) 10ZPapierski: Do not set target on master sonar analysis [integration/config] - 10https://gerrit.wikimedia.org/r/671142 (https://phabricator.wikimedia.org/T264877) [14:06:54] (03CR) 10Hashar: [C: 03+1] "rebased due to zuul-layout-diff being broken somehow. That looks fine, the job is non voting and we need it to trigger often to gather p" [integration/config] - 10https://gerrit.wikimedia.org/r/670967 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [14:07:18] addshore: I guess help your self and deploy! [14:07:25] will do! [14:08:16] (03CR) 10Addshore: [C: 03+2] Zuul: [mediawiki/extensions/Wikibase] Add non-voting quibble-apache job [integration/config] - 10https://gerrit.wikimedia.org/r/670967 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [14:09:36] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Wikibase] Add non-voting quibble-apache job [integration/config] - 10https://gerrit.wikimedia.org/r/670967 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [14:10:53] !log reload zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/670967/ T276428 [14:10:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:10:57] T276428: Introduce non-voting jobs with quibble+apache - https://phabricator.wikimedia.org/T276428 [14:11:35] addshore: aslo those wmf-quibble jobs are running wayy too many test suite. I htink they run some tests that are solely for Wikibase and might not need to be running for every other repositories [14:11:44] addshore: but that is a different topic [14:14:25] kostajh: gehel: sorry for adding you to a wrong CR, too many open tabs... [14:14:49] (03CR) 10Gehel: Do not set target on master sonar analysis (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/671142 (https://phabricator.wikimedia.org/T264877) (owner: 10ZPapierski) [14:16:06] (03CR) 10Kosta Harlan: Do not set target on master sonar analysis (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/671142 (https://phabricator.wikimedia.org/T264877) (owner: 10ZPapierski) [14:16:14] (03PS2) 10ZPapierski: Do not set target on master sonar analysis [integration/config] - 10https://gerrit.wikimedia.org/r/671142 (https://phabricator.wikimedia.org/T264877) [14:16:46] (03CR) 10ZPapierski: Do not set target on master sonar analysis (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/671142 (https://phabricator.wikimedia.org/T264877) (owner: 10ZPapierski) [14:18:22] (03CR) 10Gehel: Do not set target on master sonar analysis (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/671142 (https://phabricator.wikimedia.org/T264877) (owner: 10ZPapierski) [14:18:32] (03CR) 10Kosta Harlan: [C: 03+1] Do not set target on master sonar analysis (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/671142 (https://phabricator.wikimedia.org/T264877) (owner: 10ZPapierski) [14:18:46] zpapierski: sorry to be a pain. I didn't read the first version well enough [14:19:36] no worries - I needed to override args so that I'm not setting target [14:19:45] I am off for kids etc [14:19:51] will be back later though [14:20:08] (03CR) 10Gehel: Do not set target on master sonar analysis (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/671142 (https://phabricator.wikimedia.org/T264877) (owner: 10ZPapierski) [14:21:00] bah, the quibble-apache job for some reason still uses the quibble-stretch-php72:0.0.46 [14:21:06] addshore: :-\ [14:23:24] addshore: I think the issue may be that you're not setting --web-backend=external ? [14:23:44] (03PS1) 10Hashar: jjb: use proper image for quibble apache job [integration/config] - 10https://gerrit.wikimedia.org/r/671143 [14:23:44] zpapierski: quick meet? meet.google.com/onz-vrbw-bmq [14:23:52] addshore: I screwed it up sorry [14:24:20] (03CR) 10Kosta Harlan: jjb: wikimedia Quibble job with Apache (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/670965 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [14:24:38] kostajh: the image entrypoint takes care of spawning apache via supervisord then does invokes quibble --web-backend=external --web-url=http://127.0.0.1:9413 [14:24:41] kostajh: so should be fine [14:24:47] ah ok [14:24:58] (03CR) 10Addshore: jjb: wikimedia Quibble job with Apache (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/670965 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [14:25:33] addshore: I have deployed https://gerrit.wikimedia.org/r/c/integration/config/+/671143 [14:25:50] (03CR) 10Hashar: "Job deployed. Sorry for the screw up :-\" [integration/config] - 10https://gerrit.wikimedia.org/r/671143 (owner: 10Hashar) [14:25:50] :D [14:25:59] and that job does not run the selenium tests [14:26:05] so we would need an extra one just for selenium :D [14:26:38] woo! I see quibble-stretch-php72-apache:0.0.46 in the job config now [14:27:19] hmm, browser tests do / should run as part of this job afaik..? [14:28:07] as they run as part of the wmf-quibble-vendor-mysql-php72-docker job [14:28:37] pretty sure they skip selenium [14:28:41] there is another one for selenium [14:28:58] basically do the same logic we have just did but for the wmf-quibble.*selenium job [14:29:07] I am rushing to kids school! [14:29:13] byyyyeee! [14:29:14] and given it is raining I guess I will be back soon [14:29:15] :) [14:30:09] bah, yes, perhaps I switched job names at some point... heh [14:30:34] (03CR) 10Addshore: [C: 03+2] jjb: use proper image for quibble apache job [integration/config] - 10https://gerrit.wikimedia.org/r/671143 (owner: 10Hashar) [14:30:39] (03CR) 10Addshore: [C: 03+2] "already deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/671143 (owner: 10Hashar) [14:32:30] (03PS3) 10ZPapierski: Do not set target on master sonar analysis [integration/config] - 10https://gerrit.wikimedia.org/r/671142 (https://phabricator.wikimedia.org/T264877) [14:32:40] (03Merged) 10jenkins-bot: jjb: use proper image for quibble apache job [integration/config] - 10https://gerrit.wikimedia.org/r/671143 (owner: 10Hashar) [14:36:10] (03PS1) 10Addshore: jjb: Add quibble apache job for selenium too [integration/config] - 10https://gerrit.wikimedia.org/r/671167 (https://phabricator.wikimedia.org/T276428) [14:37:28] (03PS1) 10Addshore: zuul: Use non voting wmf-quibble-apache-selenium-php72-docker for Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/671168 (https://phabricator.wikimedia.org/T276428) [14:49:34] addshore: AND BACK [14:49:44] oooh, i just sat back down too :P [14:49:49] I made 2 more patches ;) [14:50:08] the quibble with apache that doesnt run browser tests seems to be fine [14:50:29] but also, i guess we only want to spawn apache when we actually need it xD [14:50:36] guess it is only used for Qunit [14:50:59] aah, perhaps qunit too, and maybe api integration tests (but they might have their own job?) [14:51:56] (03CR) 10Hashar: [C: 03+2] "blah bla deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/671167 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [14:52:35] (03CR) 10Hashar: [C: 03+2] zuul: Use non voting wmf-quibble-apache-selenium-php72-docker for Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/671168 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [14:52:43] addshore: it is a bit of a mess really [14:52:55] ideally we would have a first step that clone repo / setup dependencies [14:53:01] capture a snapshot of the result [14:53:03] (03Merged) 10jenkins-bot: jjb: Add quibble apache job for selenium too [integration/config] - 10https://gerrit.wikimedia.org/r/671167 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [14:53:04] i guess most of this will be changing this year as gitlab happens? [14:53:10] THEN spawn all the jobs we want reusing that snapshot [14:53:16] unlikely [14:53:17] yeah [14:53:26] oh gitlab is likley to still be jenkins powered? [14:53:37] unknown , but probably not ;D [14:53:45] (03Merged) 10jenkins-bot: zuul: Use non voting wmf-quibble-apache-selenium-php72-docker for Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/671168 (https://phabricator.wikimedia.org/T276428) (owner: 10Addshore) [14:54:26] deployed ^ [14:54:31] whoooo! [14:54:33] in short the gitlab CI is unknown to me [14:54:51] I guess for jobs that just clone a single repo, it is probably not too complicated [14:55:05] a lot of jobs would just clone the merge request then invoke the preexisting containers [14:55:10] so that is probably straightforward [14:55:13] but [14:55:44] we entirely loose the centrally managed CI system we have or easiness to maintain CI accross a large fleet of repositories [14:56:04] I don't think gitlab CE has a way to share some CI definition accross a set of repositories (haven't looked up) [14:56:17] github actions does now, so I imagine gitlab might, but i dont know [14:56:20] for the mediawiki jobs which clone multiple repo and ensure they are working well together: future unknown [14:56:29] I think it'll all just need a bunch of automation to keep things in check and in sync [14:56:42] yeah "just" [14:56:42] ;D [14:56:45] ;) [14:56:55] I'll be excited to help out when we get there ;) [14:57:15] then with all the infra work I am doing, and covid taking its toll (lockdown, kids at home etc) [14:57:28] + the various ci related stuff reviews and new stuff coming in [14:57:44] I can't say I had much time to look at the future of CI with gitlab in [14:58:07] but others in the team at least looked a bit at it and are experimenting. So it is not all dark [14:58:43] we also have the PipelineLib thing, which is implemented on top of Jenkins capabilities. That one would need a full rewrite (or we hook gitlab with a jenkins) [15:03:38] hashar: when you have the time - https://gerrit.wikimedia.org/r/c/integration/config/+/671142/ [15:06:25] zpapierski: so get the branch from the properties file rather than hardcoding it right? [15:06:26] i wonder if this apache image has apc :P [15:07:01] addshore: you can docker run it with --entrypoint=php ... -m ? [15:07:04] hashar: yep, I was convinced it makes more sense. Mostly the patch is about not setting a target on main branch analysis, though [15:07:17] apparently sonar isn't a fan of that [15:07:18] me will check [15:09:07] zpapierski: I am looking at a way to keep the sonar parameters in the job template. They sound generic [15:09:37] oh no [15:09:40] I was looking at that, too - in the end decided to do the same thing that mwcore-codehealth does [15:09:47] that aligns with the other mwext-codehealth jobs [15:09:51] yeah [15:09:53] I couldn't figure out anything better [15:09:53] it is bad [15:09:55] but consistent [15:09:56] so ;) [15:10:01] yeah :D [15:10:25] so I then just look at the diff https://integration.wikimedia.org/ci/job/integration-config-jjb-diff-docker/3238/console [15:10:40] which looks legit [15:11:09] yep, already verified this locally, too [15:11:41] (03CR) 10Hashar: [C: 03+2] Do not set target on master sonar analysis [integration/config] - 10https://gerrit.wikimedia.org/r/671142 (https://phabricator.wikimedia.org/T264877) (owner: 10ZPapierski) [15:12:01] and one day I will need to work on opening up the rights to do some CI config [15:12:13] there are more folks familiar with zuul/jjb etc [15:12:37] oh so many selenium failures :D [15:12:43] ;D [15:13:06] (03Merged) 10jenkins-bot: Do not set target on master sonar analysis [integration/config] - 10https://gerrit.wikimedia.org/r/671142 (https://phabricator.wikimedia.org/T264877) (owner: 10ZPapierski) [15:13:06] so many its not even funny [15:13:09] and I cant remember whether we still have sqlite jobs [15:13:25] but surely parallel requests and sqlite as a backend would not play well [15:13:35] hashar: also have some time to deploy it ;) ? [15:13:49] hashar: I guess that depends on what the parallel requests are doing [15:14:10] (03CR) 10Hashar: "Jobs updated \o/" [integration/config] - 10https://gerrit.wikimedia.org/r/671142 (https://phabricator.wikimedia.org/T264877) (owner: 10ZPapierski) [15:14:26] hashar: awesome, thanks! [15:14:35] now, the moment of truth [15:14:35] 10Continuous-Integration-Config, 10Patch-For-Review: Introduce non-voting jobs with quibble+apache - https://phabricator.wikimedia.org/T276428 (10Addshore) Here is a lovely run with many failures! :D https://integration.wikimedia.org/ci/job/wmf-quibble-apache-selenium-php72-docker/2/console [15:20:42] gehel: yay, we have violations :D - https://sonarcloud.io/dashboard?id=discovery-analytics&branch=master [15:21:03] AWESOME! [15:21:07] best failure ever ;] [15:21:08] like kostajh mentioned, nothing on code coverage yet, but I'll investigate [15:21:21] yeah, I can go with failing to succed :) [15:21:43] it would be awesome if you can report what you did somewhere (maybe a short phabricator blog post) and explain the added value of sonarqube for python project [15:21:44] now for branch build [15:21:58] then I guess we can work with the various python repos owner to migrate and benefit from sonar? [15:22:12] sure! [15:22:34] I am happy to try integration/quibble (it is a python project). We can probably get volans (not in this channel) to be interested he wrote several python tools for SRE [15:23:04] and for the record [15:23:30] I originally had doubt about usefulness of SonarQube compared to having a fleet of linters and parsing the console output [15:23:44] or failed to see a point in running some checks a second time [15:24:03] but I must say the SonarQube error reporting is gold with very nice explanations for the faults it detects [15:24:08] so I guess I am sold ;] [15:24:37] yeah, that and seeing progress over time [15:25:07] I've used it for years in different projects, in some places we basically had a sonar dashboard running on some screen in the room 100% of the time [15:25:26] working in a team room sounds a bit abstract nowadays :) [15:27:48] zpapierski: cool! If you figure out CR analysis, you're welcomed to have a look at T264873 as well! [15:27:49] T264873: Ensure that SonarQube is commenting on gerrit code reviews of the Search Platform team - https://phabricator.wikimedia.org/T264873 [15:27:58] yes, no good deed goes unpunished! [15:28:17] hold your horses, I ain't nearly done ;) [15:28:30] I still want/need to investigate coverage and there's still mjolnir [15:29:48] hmm, branch analysis still shows nothing under the code tab [15:29:53] maybe that's normal? [15:30:16] did you re-run that branch analysis? [15:30:20] yeah [15:30:43] branches reports should only report new violations, not existing ones. So if that CR did not generate any new violation, the report should be empty [15:30:52] even the code? [15:30:57] ok, I need to violate that code [15:31:06] I think there should still be a list of files analyzed and the code modified, but I'm not entirely sure [15:31:15] looks like the same issue we have for Java projects [15:31:33] also it asked me to define some new code policy for master? [15:31:46] I have no idea what that is, but I've set it to previous version [15:31:52] I'll rebuild all [15:31:57] I think most projects use 30 days for new code [15:33:25] addshore: if the non selenium apache job is working fine, I guess we can promote it on monday? [15:33:47] yeah it seems to be working fine for wikibase [15:34:06] i'll spend a few more mins thinking about the selenium one, but i don't imediatly spot anything [15:34:39] that sounds like a lot of effort [15:34:53] who knows what ends up being broken when requests are layed in parallel [15:35:18] well, the web is made to work with them in parallel :P [15:35:24] yea [15:35:30] but some tests might not take that in account [15:35:35] like 3x load.php calls at once, or an app making 3 api calls at once on pourpose [15:35:59] it looks like the tests are most just blanket failing, and I dont know why [15:36:12] well, some pass actually! :) [15:36:56] (03CR) 10Michael Große: "I think this should now be ready to be merged code-wise. But I'm not exactly sure about the order of things. Do we need to create the Gerr" [integration/config] - 10https://gerrit.wikimedia.org/r/670898 (https://phabricator.wikimedia.org/T277060) (owner: 10Michael Große) [15:36:56] ironically most of the failure are timeouts of waiting for things to appear, so maybe this ends up making browser tests slower :/ [15:37:36] (03CR) 10Addshore: "> Patch Set 3:" [integration/config] - 10https://gerrit.wikimedia.org/r/670898 (https://phabricator.wikimedia.org/T277060) (owner: 10Michael Große) [15:37:52] hmm [15:38:09] maybe php cli and php-fpm behave differently [15:38:18] like opcache/jit or something like that not being used for php cli [15:38:23] but being used for php-fpm [15:38:27] I don't know really :-\ [15:38:37] I'll re run it and see if it doesn anything different [15:38:39] addshore: running them in mediawiki-docker-dev I assume works? [15:38:49] at lesat we have the apache log captured [15:39:00] https://integration.wikimedia.org/ci/job/wmf-quibble-apache-selenium-php72-docker/4/artifact/log/apache-access.log [15:39:06] kostajh: running browser tests aginst mwdd? yes that works [15:39:49] and potentially we could get the apache log to be formatted to add the time it took for the query to be served [15:43:05] Hello there, I'd like to discuss a deployment on friday if anybody is around, see -ops [15:44:29] addshore: does this recording look correct? https://integration.wikimedia.org/ci/job/wmf-quibble-apache-selenium-php72-docker/2/artifact/log/if-the-client-page-is-editable-for-the-user_show-the-editlink.mp4 [15:44:48] for "if the client page is editable for the user... show the editlink" [15:45:12] no :/ [15:46:20] let me find a successfull one [15:47:15] oh i think the wikibase ones only get recorded on error [15:47:18] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.34 deployment blockers - https://phabricator.wikimedia.org/T274938 (10Daimona) [15:48:53] I mean actually that vid could look totally fine :/ [15:49:06] DataBridgePage.open( title ); [15:49:06] assert.ok( DataBridgePage.overloadedLink.isDisplayed() ); [15:49:17] addshore: I would try to run quibble locally and set a ~60 wait somewhere and check the browser console [15:49:24] the page is open, and I can visibly see the edit link, but I'm guessing the JS didn't run on it to apply the magic [15:49:28] but getting setup for all that is kind of a pain [15:49:47] the apache log seems fine [15:50:33] let me line up the test with the apache log [15:52:17] https://phabricator.wikimedia.org/P14819 [15:52:45] that looks fine, and I see the JS being loaded etc [15:53:17] I do notice requests are being made to http://127.0.0.1:9413//index.php so doulbe // [15:53:48] at a glance I don't see anything funny in mw-debug-www.log either [15:54:23] This appears 34 times in the context of ApiMain->executeAction [GlobalTitleFail] MessageCache::parse called with no title set but doesn't seem the culprit [15:54:28] beside the spam of mwbot/1.0.10 requests ;D [15:55:15] so those requests are the creation of a page to test with the infobox, (so the page we are looking at) [15:57:11] I mean, the element being waited for is `$( `a=${DataBridgePage.EDIT_LINK_ANCHOR}` )` where the anchor is `Edit this on Wikidata` [15:57:26] i can literally see that link :P [15:57:47] aah no, wait, `Failed to wait for wikibase.client.data-bridge.app to be ready after 10000 ms.` is a different wait [15:58:26] https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Wikibase/+/ffc260767f54cd4866400caeb13fab4ed540f458/client/data-bridge/tests/selenium/pageobjects/dataBridge.page.js#125 [15:58:40] so its waiting for resourceloader to say that the module wikibase.client.data-bridge.app is ready [15:59:17] maybe prepapring the module is too slow? [15:59:40] like if we have several load.php requests in parallel, that might trigger the preparation of several RL modules [15:59:51] which would starve cpu/io or whatever and cause them to be slightly longer [15:59:56] I don't know really [16:00:17] it is not like I know anything about mediawiki anymore :-\ [16:00:17] addshore: I suspect there is a JS error on the page, not sure though [16:00:29] kostajh: I also suspect that, but not sure how to debut that here :/ [16:00:47] aren't we capturing the browser console at the end of each test? [16:00:53] addshore: would it be easy to deploy a non voting job for another extension, like GrowthExperiments or something with just a handful of tests (and that is also a lot easier to setup locally) [16:00:55] are we? :D [16:01:00] I thought we did [16:01:03] hashar: not that I know of [16:01:09] we might be, but maybe i dont know where to look [16:01:12] or maybe I hacked something like that at some point in the past to investigate an issue [16:01:32] kostajh: a whole bunch of tests poass in this run, which is posotive :) [16:01:43] but this 1 wait condition breaks in every single case that it is called [16:01:58] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/426087/1/tests/selenium/wdio.conf.js ! [16:02:08] actually, it looks like browser tests for non js uis work, and tests for js uis break :P [16:02:11] (abandoned patch: Selenium: dump browser console on failure [16:02:11] ) [16:02:40] maybe i try that in a test patch in wikibase :D [16:03:58] and from Fresnel (the utlity to do browser performance collection): https://gerrit.wikimedia.org/r/plugins/gitiles/performance/fresnel/+/refs/heads/master/src/probes/ [16:04:08] there are a bunch of trivial probes there that might give more details [16:04:42] eg for transfers: performance.getEntriesByType( 'navigation' ); [16:04:52] but for a js error, I guess dumping the console output might be sufficient [16:06:02] I am off, might be back later tongiht [16:06:11] need some groceries before curfew [16:07:25] I made a https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/671188 WIP DNM: data-bridge: dump browser console logs on failure [WIP] [NEW] [16:07:30] and will wait to see if this helps with the output :) [16:11:48] bah family is out for groceries shopping ahah [16:13:33] * kostajh watches Jenkins TV [16:13:54] a.k.a. https://integration.wikimedia.org/ci/job/wmf-quibble-apache-selenium-php72-docker/7/console [16:18:33] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Scap, 10User-brennen: Applying security patches should be robust and also give some useful output - https://phabricator.wikimedia.org/T269153 (10LarsWirzenius) I'm sketching a scenario for... [16:19:41] hehe, jenkins tv always gives me 431 errors ;) [16:28:57] well thats fun, i change the config and all the bridge ones passed now [16:29:05] see [16:29:11] it just needed some config tuning! :D [16:29:26] if that is perf related [16:29:37] maybe it only explodes when several jobs are running on the same jenkins agent [16:30:05] yeah, other tests still fail, but not the bridge ones [16:31:41] Looking at https://integration.wikimedia.org/ci/job/wmf-quibble-apache-selenium-php72-docker/7/artifact/log/can-add-a-statement-using-the-keyboard.png now, seemingly the JS has not finished running for it [16:32:25] 10Phabricator: Spaces request for Technical-Program-Management - https://phabricator.wikimedia.org/T277107 (10MBinder_WMF) Hi, @Aklapper . Thanks for helping refine this request. The tags are made, for now. I've added you to the tags to make sure you can see them. The end goal is to make the tags and tasks wi... [16:36:35] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Scap, 10User-brennen: Applying security patches should be robust and also give some useful output - https://phabricator.wikimedia.org/T269153 (10hashar) There are a few typo here and there... [16:54:40] 10Continuous-Integration-Infrastructure, 10Quibble, 10Browser-Tests: Can we make the PHP inbuilt webserver used in CI log the time it has taken to execute calls to the API? - https://phabricator.wikimedia.org/T277202 (10hashar) @Addshore rolled the Quibble apache jobs today. There is a non voting one runnin... [16:55:11] addshore: about logging response time, that can be done by Apache. I wrote some bits at https://phabricator.wikimedia.org/T277202#6908996 [16:55:33] hashar thcipriani brennen: is one of you around? it looks like the WikimediaEvents 1.36.0-wmf.34 is in a weird state [16:55:37] addshore: just need a few copy paste to the image apache cconfig file, changelog bump, build image, update jjb and done (hopefully) [16:56:07] Jdlrobson: log-wise or behavior-wise? [16:56:37] here [16:56:47] thcipriani: so i debugged the error spike yesterday with Sam [16:56:53] hashar: ack! [16:57:09] I am off for real now. See you all on monday! [16:57:31] my assumption was that when we cherry picked the patch yesterday (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/670879) we would have the current state of master [16:57:52] but when i inspect the production code it doesn't match [16:58:26] hrm [16:58:42] before digging into this, i'll say it's entirely possible i screwed up somewhere along the way. [16:59:43] for clarity, does the code on 670879 not appear to match what's in production? [17:00:04] brennen: that patch is live [17:00:12] i think the problem is we backported an older version of another patch [17:00:18] im trying to work out where and how [17:00:42] one solution might be to fast forward 1.36.0-wmf.34 to current master for WikimediaEvents [17:01:02] another option would be to find the bad/missing backport and apply it as well [17:01:10] i need a bit more time to do that [17:03:00] I think the issue is these 2 are different > https://gerrit.wikimedia.org/r/q/I7093ed4b170c5b42ad25f3e6da9ea68f30992dc0 [17:04:32] but it seems i cant cleanly revert and reapply it [17:04:35] without manual changes [17:06:24] So option 1: fast forward 1.36.0-wmf.34 to current master [17:06:44] option 2: Revert https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/671160 and reapply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/670585 [17:07:57] option 3: live with the status quo [17:08:11] Jdlrobson: i guess: i'll defer on the appropriate solution, but it being a friday, does this constitute an emergency? [17:08:40] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release, 10Train Deployments: 1.36.0-wmf.35 deployment blockers - https://phabricator.wikimedia.org/T274939 (10Jdlrobson) [17:08:52] what's the user impact of the patch being out of date? [17:08:57] er, the repo rather [17:09:40] here's the diff between what (should be) deployed and master: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/WikimediaEvents/+/refs/heads/wmf/1.36.0-wmf.34..refs/heads/master [17:09:57] and it mostly looks like type annotations [17:10:20] there is no user impact. the only impact here is that our eventlogging pipeline is getting a larger amount of traffic than expected and what that volume over the weekend will be is unknown. That said traffic to error logging is still within safe boundaries. [17:11:03] ah [17:11:09] https://grafana.wikimedia.org/d/000000566/overview?orgId=1&from=1615384920669&to=1615583705027 demonstrates the volume difference [17:11:39] heh, so like an order of magnitude more events [17:11:45] if analytics and product infrastructure are okay with this, I'm fine. [17:12:03] I can live with a broken graph for a week (I would rather not establish a new baseline) [17:12:26] sure, what is the fix? what's causing so amny more events? [17:12:59] L276 on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/670585 never got deployed [17:13:27] the fix we deployed yesterday made the assumption that line was present [17:13:31] without it, the fix did nothing [17:14:00] The errors are anonymized script errors that we have no way of debugging. Previously we didn't log these because they are inactionable, but now we are [17:14:00] ah, yes, I see that in the diff I posted. This also requires the normalizeErrorMesssageFunction to be deployed [17:14:18] so addressing this problem would be more about system health than user experience health [17:15:03] but as I said.. our systems should be able to handle it [17:15:03] i think if we don't do it today on the basis that it's friday, a backport on monday would be entirely reasonable. [17:15:03] I guess also the function name "shouldNotLogFileUrl" became "shouldIgnoreFileUrl" -- I don't know if that matters [17:15:16] it's just a case of something is wrong on the internet :) [17:15:49] thcipriani: yeh that matters less, but yeh the code is an inconsistent/never verified state [17:28:06] thcipriani: brennen do you have a test for whether something needs a friday deploy? TBH my preferred solution would be to make wmf34 identical to current master for that extension, but I'm not sure how to do that in gerrit. I'd be happy for us to do that monday if that's not a friday-deploy-kinda-thing [17:28:15] (03PS1) 10Dduvall: apt: Support configuration of http/https proxies [blubber] - 10https://gerrit.wikimedia.org/r/671199 (https://phabricator.wikimedia.org/T277109) [17:28:43] if we can't do a make wmf34 identical to current master kind of thing, i'd suggest just leaving the branch as a broken branch and wait for next week's train rather than to duct tape it some more [17:29:18] https://wikitech.wikimedia.org/wiki/Deployments/Emergencies is the litmus [17:29:21] https://wikitech.wikimedia.org/wiki/Deployments/Emergencies has our guidelines. [17:29:44] I'm trying to find some opinions from observability about whether this will hurt availability [17:30:10] (also: jinx!) [17:31:57] 10Beta-Cluster-Infrastructure: Broken puppet on deployment-mediawiki servers - https://phabricator.wikimedia.org/T277069 (10dancy) This started happening after https://gerrit.wikimedia.org/r/c/operations/puppet/+/666787 was merged. [17:32:22] i'll note here that there's also https://phabricator.wikimedia.org/T277302 pending some action - which hashar evaluated earlier as being worth a friday revert, i'd say since it's a user-facing issue and a simple revert. [17:34:59] Jdlrobson: after checking in with observability folks it seems like the system is fine with the additional messages and we can try to backport on Monday [17:36:30] k. backport what? [17:36:49] can we make wmf34 the same as master? is there a special git commit we can prepare to do that? [17:36:50] whatever you were going to have us backport today ;) [17:37:39] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10Release Pipeline, 10serviceops, 10Patch-For-Review: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10dduvall) Success! I was able to build and push an image via docker-pusher on releases1... [17:37:48] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10MW-on-K8s: Security patch workflow for MediaWiki on k8s - https://phabricator.wikimedia.org/T271274 (10dduvall) [17:37:52] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10Release Pipeline, 10serviceops, 10Patch-For-Review: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10dduvall) 05Open→03Resolved [17:39:05] thcipriani: is `git push --force origin/wmf/1.36.0-wmf.34 master` an option? [17:39:08] sounded like it was adding the "normalizeErrorMessage" function and changing fileUrl to use "|| 'undefined'" was the meat of what's causing this [17:39:40] heh, well that's what'll happen with wmf.35 roughly [17:39:55] (03PS2) 10Dduvall: apt: Support configuration of http/https proxies [blubber] - 10https://gerrit.wikimedia.org/r/671199 (https://phabricator.wikimedia.org/T277109) [17:40:07] I guess I'd make a merge commit and backport that if that's the desired state [17:41:09] i guess an old fashioned copy pasta will do it [17:41:12] i.e., git fetch; git checkout wmf/1.36.0-wmf.34; git merge master; git push origin HEAD:refs/for/wmf/1.36.0-wmf.34 [17:41:21] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/671202 Use master version of clientError.js [NEW] [17:41:22] or copy-pasta a diff [17:42:48] thcipriani: okay it's on the calendar [17:42:54] thanks for working through this :) [17:43:43] +1'd -- for sure! thanks for raising the problem :) [17:46:42] (03PS3) 10Dduvall: apt: Support configuration of http/https proxies [blubber] - 10https://gerrit.wikimedia.org/r/671199 (https://phabricator.wikimedia.org/T277109) [18:11:13] 10Continuous-Integration-Infrastructure, 10Quibble, 10Browser-Tests: Can we make the PHP inbuilt webserver used in CI log the time it has taken to execute calls to the API? - https://phabricator.wikimedia.org/T277202 (10Jdforrester-WMF) Neat! [18:14:51] (03PS1) 10Jforrester: dockerfiles: [quibble-stretch-php72-apache] Add custom log to time requests [integration/config] - 10https://gerrit.wikimedia.org/r/671207 (https://phabricator.wikimedia.org/T277202) [18:16:11] 10Beta-Cluster-Infrastructure: Broken puppet on deployment-mediawiki servers - https://phabricator.wikimedia.org/T277069 (10dancy) The affected hosts are * deployment-parsoid11.deployment-prep.eqiad.wmflabs * deployment-jobrunner03.deployment-prep.eqiad.wmflabs * deployment-mediawiki-09.deployment-prep.eqiad.wm... [18:25:24] 10Beta-Cluster-Infrastructure: Broken puppet on deployment-mediawiki servers - https://phabricator.wikimedia.org/T277069 (10Legoktm) a:03Legoktm Yeah, we just need to truncate or pad the `profile::mediawiki::php::monitoring::salt` hiera so its 8 characters. I'll do that. This has been silently broken since whe... [18:26:18] 10Beta-Cluster-Infrastructure: Broken puppet on deployment-mediawiki servers - https://phabricator.wikimedia.org/T277069 (10dancy) Thanks Legoktm! [18:28:45] Majavah, dancy: do you know where deployment-prep keeps its private hiera [18:28:48] ?* [18:29:00] legoktm: deployment-puppetmaster04, /var/lib/git/labs/private [18:29:27] aha! That was something I was trying to dig up earlier. Thanks Majavah [18:29:31] legoktm: I already grepped for that hiera variable at some point but only found 8 char values, thought I commented that already on that task [18:29:41] hmm [18:29:45] yeah, same [18:29:49] Could it be that the salt is blank? [18:30:05] (that might be worth its own check in the the code)( [18:30:07] what abotu hacking the message to show the salt that was passed in? [18:30:14] (03PS1) 10Dduvall: Provide httpProxy context variables if a proxy is configured [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/671211 (https://phabricator.wikimedia.org/T277109) [18:31:09] that would be helpful [18:31:29] (03PS2) 10Dduvall: Provide `setup.httpProxy` context variable if a proxy is configured [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/671211 (https://phabricator.wikimedia.org/T277109) [18:32:40] * legoktm brb in 5 [18:35:12] it has a longer value that's coming from /var/lib/git/labs/private-back/hieradata/common/profile/mediawiki/php/monitoring.yaml [18:35:25] what is private-back and why does puppet include it [18:43:51] heh [18:43:58] sounds like a backup? [18:44:52] drwxr-xr-x 8 root root 4096 Mar 12 08:36 .git [18:44:55] I guess that was you just now? [18:45:03] rest of the repo looks untouched since June 2020 [18:46:43] Majavah: ^ [18:46:57] maybe we move it out of /var/lib/ entirely so puppet doesn't include it? [18:47:21] legoktm: probably yes [18:48:13] moving it to /root or something sounds sensible [18:48:37] if that's picked up from there is something else? would moving it break things? [18:49:59] maybe! [18:50:27] I'd mostly be worried about what local commits are in that repo that aren't in the correct labs/private folder [18:51:08] diffing the two folders will be useless because private-back hasn't been rebased on Gerrit for nearly a year [18:51:20] June 2020 sounds like https://wikitech.wikimedia.org/wiki/Incident_documentation/20200604-cloud-private-repo [18:53:39] oh, I remember that day :( [18:56:04] https://phabricator.wikimedia.org/P14820 [18:57:13] (03PS3) 10Dduvall: Provide `setup.httpProxy` context variable if a proxy is configured [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/671211 (https://phabricator.wikimedia.org/T277109) [18:57:28] ok, all the local commits in private-back are also in private (based on commit message) [18:59:44] Majavah: I'm going to move it unless you have any outstanding concerns [19:00:45] legoktm: probably fine, I just wish we had a puppetboard on beta to see which hosts have changes made after this and which dont [19:01:45] !log legoktm@deployment-puppetmaster04:/var/lib/git/labs$ sudo mv private-back /root/private-back-2020-06 [19:01:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:03:21] how long does it take for changes to get reflected? [19:03:50] deployment-mediawiki-07.deployment-prep.eqiad.wmflabs is still erroring... [19:04:01] after run-puppet-agent? [19:04:28] i think like half an hour between prod puppetmaster and beta puppetmaster [19:04:33] unless someone manually pulls [19:06:28] we changed something in the local /var/lib/git/labs/private repo in this case [19:07:39] it should be reflected immediately after changing files I think [19:08:02] in that case, yes [19:08:12] :| [19:09:24] Majavah: i think you have cumin, you could run puppet on all at once and see the logs at least [19:09:39] how was private-back included in the first place? [19:09:47] do we have cumin in deployment-prep? /me looks [19:12:06] we do, but not sure how much puppet will like me if I run it on 72 hosts at once [19:12:24] I just saw https://wikitech.wikimedia.org/wiki/Help:Cumin_master but not sure about depl-prep specific now [19:12:31] you can stagger it, one sec [19:12:38] we have deployment-cumin [19:12:47] Majavah: -b for batch and -s for sleep [19:13:01] -b batch [19:13:03] :) [19:13:22] https://wikitech.wikimedia.org/wiki/Cumin#Run_Puppet_keeping_the_output [19:13:27] ^ keeping the output and -b usage [19:13:55] !log taavi@deployment-cumin:~$ sudo cumin -b 1 -s 5 'wdqs2*' 'run-puppet-agent -q' [19:13:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:14:10] bah, not wdqs2* but * [19:14:31] also not -q, copying the right thing is hard [19:15:04] it failed after root@deployment-acme-chief03.deployment-prep.eqiad1.wikimedia.cloud: Permission denied (publickey) [19:16:55] /root/.ssh and /root/.ssh/authorized_keys does not exist, should them? or are cumin keys stored somewhere else? [19:18:14] FOUND IT [19:18:32] it was in the horizon puppet [19:18:50] you can't be serious [19:19:20] https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/e908a3584339ebed4fdc5e6abdacf775a8bb7212%5E%21/#F0 [19:19:26] legoktm: can we get that somehow in codesearch? [19:19:40] yes [19:20:39] puppet passed on mediawiki07 [19:20:47] running it on the other failures now [19:21:46] so I think the private-back was entirely a red herring [19:22:36] yeah I don't think that was the cause at all [19:23:01] if you run cumin once on * . it then has an option to run again "only on recent failures" [19:23:32] I would like some clear documentation on which hiera values override which? ie I've seen ops/puppet override horizon, but in this case horizon was overriding ops/puppet [19:23:55] 10Beta-Cluster-Infrastructure: Broken puppet on deployment-mediawiki servers - https://phabricator.wikimedia.org/T277069 (10Legoktm) 05Open→03Resolved After trying to figure out where it was set...eventually found it in Horizon: https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/e908a3584... [19:23:55] no, it was overriding labs/puppet [19:23:59] er, labs/private* [19:24:00] mutante: I don't think deployment-cumin works at all, the key isn't valid [19:24:59] Majavah: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Hiera#Precedence_of_hiera_keys [19:25:03] for the hiera question [19:25:27] https://github.com/wikimedia/puppet/blob/production/modules/puppetmaster/files/labs.hiera.yaml [19:25:57] thanks mutante [19:26:50] can't say much about the cumin key part.. all I knew was that it somehow exists in cloud [19:27:04] it's used by wmcs across projects [19:27:57] but https://wikitech.wikimedia.org/wiki/Help:Cumin_master talks about setting one up for a project [19:29:30] in other cloud projecs when I see stuff in Horizon hiera and it's not just a very short lived temp test.. always prefer to move it into the actual repo [19:29:49] can someone tell me how tags are created in the docker registry? if I create a tag for a repository in gerrit, should that trigger the creation of a tag in the registry? [19:29:50] and then clean out Horizon so there isn't even a question next time [19:30:32] looking at that guide it's setup correctly on horizon [19:31:23] the annoying thing about setting them on ops/puppet is that only SREs can edit them, and I (and others) have to always bug you to merge them or cherry pick them to local puppet (which isn't backed up or searchable and causes some occasional merge conflicts) [19:32:20] I thought that was solved by having the local puppetmaster but that means cherry-picking, I understand [19:34:18] maybe something like a regular sync from cherry-picked back to prod that does not involve having to ping people on IRC [19:34:24] either cherry picking or committing them to labs/private, both of them have the same downsides [19:34:27] dunno though, it's an old problem [19:35:32] yea, but labs/private is not the right place to put normal hiera [19:36:01] currently deployment-puppetmaster04 labs/private has 31 local commits [19:36:31] Majavah: second best thing is project/prefix hiera, at least it's better than using instance names [19:36:42] I'd imagine most of them are secrets that can't be made public and we don't have a better place for them [19:36:43] the worst is ../hosts/ [19:36:45] imho [19:37:22] Majavah: fyi https://gerrit.wikimedia.org/r/c/labs/codesearch/+/671223/ [19:37:23] I dunno about that. the entire labs cant have secrets [20:01:18] (03CR) 10Umherirrender: [C: 04-1] "What about a comment on the both const in PhpunitAnnotationsSniff that each annotation usable on function comments should be added to the " (035 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/670120 (https://phabricator.wikimedia.org/T276971) (owner: 10DannyS712) [20:09:49] (03PS5) 10DannyS712: FunctionAnnotationsSniff: recognize more phpunit annotations [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/670120 (https://phabricator.wikimedia.org/T276971) [20:09:57] (03CR) 10DannyS712: FunctionAnnotationsSniff: recognize more phpunit annotations (034 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/670120 (https://phabricator.wikimedia.org/T276971) (owner: 10DannyS712) [20:13:57] (03CR) 10jerkins-bot: [V: 04-1] FunctionAnnotationsSniff: recognize more phpunit annotations [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/670120 (https://phabricator.wikimedia.org/T276971) (owner: 10DannyS712) [20:14:27] Hey releng: I forget if an official decision was made about this, but is it planned that the wikimedia installation of gitlab core will support both oauth2 and PATs? [20:15:03] (03PS6) 10DannyS712: FunctionAnnotationsSniff: recognize more phpunit annotations [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/670120 (https://phabricator.wikimedia.org/T276971) [20:50:57] (03CR) 10Krinkle: FunctionAnnotationsSniff: recognize more phpunit annotations (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/670120 (https://phabricator.wikimedia.org/T276971) (owner: 10DannyS712) [20:51:01] (03CR) 10Krinkle: [C: 04-1] FunctionAnnotationsSniff: recognize more phpunit annotations [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/670120 (https://phabricator.wikimedia.org/T276971) (owner: 10DannyS712) [20:51:41] urandom: which repository are you trying to publish an image for? [20:52:00] longma: mediawiki/services/kask [20:53:48] I don't think we have a way to tag just one specific image, but marxarelli may be able to confirm whether that is the case. However you can add tags here https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/kask/+/refs/heads/master/.pipeline/config.yaml#26 [20:54:42] longma: I don't think that's the same thing [20:55:11] longma: if you look at https://docker-registry.wikimedia.org/wikimedia/mediawiki-services-kask/tags/ (scroll to the bottom), you can see tags that correspond with git tags [20:55:31] or maybe that's the result of something legacy, those were all created quite some time ago [20:55:39] but it used to work [20:56:55] (03CR) 10DannyS712: FunctionAnnotationsSniff: recognize more phpunit annotations (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/670120 (https://phabricator.wikimedia.org/T276971) (owner: 10DannyS712) [20:58:21] urandom: yeah, unfortunately I don't see that the pipelinelib does any tagging like that [21:01:35] we could probably add that feature [21:03:57] but I'm not sure how it would work if the git tag is not part of a commit [21:06:30] I thought those docker tags were commit hashes though [21:07:24] oh, maybe you're talking about the "v1.0.0" ones [21:09:55] urandom, longma: i'm probably missing context. are you wanting to have an image built when a new git tag is pushed and have the image tagged with the same value (git tag) [21:09:58] ? [21:10:15] I was also wondering that [21:14:41] if so, the first step is to schedule a job under the `publish` (weird name) zuul pipeline https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/refs/heads/master/zuul/layout.yaml#771 [21:15:01] then we'd have to make sure those PipelineBuilder based jobs work with ref updates [21:16:04] i'm not sure we've tested that, but i see no reason why we couldn't make it work. iirc, the statically defined (non `.pipeline/config.yaml`) service-pipeline jobs did not, which is perhaps what urandom is referring to [21:16:15] did *that* [21:16:59] marxarelli: umm... I'm not sure it needs to result in a new image necessarily. If it's a tag to a git commit that corresponds with an image built for that commit, it could just be a tag. [21:17:17] OTH, it wouldn't hurt if it generated an image [21:17:30] marxarelli: but how did the tags that are there get created? [21:17:37] it will effectively do that if the build is already cached [21:17:47] but yeah, it's not super smart about existing images [21:18:18] oh... do you mean if the tag comes after a build recent enough to still be cached? [21:18:20] the tags are created according to the publish configuration https://wikitech.wikimedia.org/wiki/PipelineLib/Reference#Publish [21:18:39] and there's a default tag based on the timestamp at build start and the stage name [21:19:17] marxarelli: I think he means the ones that got created before with the version tags [21:19:23] marxarelli: I'm not sure that's the same thing; how were the semver style tags created here https://docker-registry.wikimedia.org/wikimedia/mediawiki-services-kask/tags/ [21:19:32] v1.0.0 to v1.0.6 [21:19:36] those correspond to git tags [21:19:52] urandom: re: cached, i mean if the same commit was built previously on the same node, the computed cache keys from the working directory will match the previous layer in the docker cache [21:19:56] i think [21:19:59] maybe :) [21:20:10] I assume that was prior to migrating kask to use the .pipeline/config [21:20:11] theoretically? [21:20:20] docker always surprises [21:20:39] longma: i believe so, yes. prior to use of .pipeline/config.yaml and PipelineBuilder [21:21:01] what created docker-registry.wikimedia.org/wikimedia/mediawiki-services-kask:v1.0.6 ? [21:21:10] on my end I pushed a tag [21:21:14] pushed it to gerrit [21:21:33] urandom: https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/refs/heads/master/jjb/service-pipeline.groovy#72 [21:21:55] !? [21:22:09] but that code isn't used for projects that have a `.pipeline/config.yaml` [21:22:15] ooohh [21:22:47] all of that was refactored into a new `PipelineBuilder` class that dynamically defines the Jenkins pipeline based on the repo's .pipeline/config.yaml [21:22:58] and i think the handling of tag ref updates didn't make it in [21:23:05] well damn :( [21:23:07] OK [21:23:08] so we just need to add that [21:23:15] shouldn't be too hard [21:23:37] cool, that's pretty useful [21:24:16] yeah, i recommend checking out the PipelineLib docs. it's pretty flexible! [21:25:08] * marxarelli needs to update the docs with examples of new features, stage output processing and list comprehensions= [21:25:53] marxarelli: btw lmk when you are ready to merge my pipelinelib/config changes [21:26:01] so...update a tag in config.yaml whenever making a release? [21:26:03] for sure [21:26:15] is that the recommended approach? [21:26:21] urandom: you could do that for now, yeah [21:26:25] k [21:26:51] i'd like to get a variable in there that has the value of any pushed tag ref [21:27:10] then you'd be able to do `tags: ['${setup.git_tag}']` or some such [21:27:23] yeah, it'd be nice to do that sort of thing in one place (tagging), and git seems like the right place [21:27:25] need to file a task for that [21:27:40] git ops ftw [21:28:41] longma: yes, i'm tweaking one thing in my blubber `apt.proxies` patch and then i'll be ready [21:28:50] cool [21:29:25] (03PS4) 10Dduvall: apt: Support configuration of http/https proxies [blubber] - 10https://gerrit.wikimedia.org/r/671199 (https://phabricator.wikimedia.org/T277109) [21:36:06] 10Gerrit, 10Accessibility: "CR" and "V" columns are not colorblind-friendly - https://phabricator.wikimedia.org/T256615 (10hashar) Turns out GerritSite.css is only used for the login page nowadays. It is a legacy stylesheet from Gerrit 2.15 and is no more included in Gerrit 3.2. @Paladox and I tried to inject... [21:37:56] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release Pipeline: PipelineBuilder based jobs need to handle git tags and other ref updates - https://phabricator.wikimedia.org/T277346 (10dduvall) [21:38:34] longma: k! ready. i'll send ya a meet link [21:48:02] (03PS6) 10Jeena Huneidi: Define and pass allowed credentials to pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/668199 (https://phabricator.wikimedia.org/T269900) [21:54:14] (03PS1) 10Dduvall: Revert "Revert "PipelineRunner: allowedCredentials"" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/671270 [22:13:46] (03PS7) 10Jeena Huneidi: Define and pass allowed credentials to pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/668199 (https://phabricator.wikimedia.org/T269900) [22:20:38] (03CR) 10Jeena Huneidi: "recheck" [blubber-doc/example/helloworldoid] (kubernetes-tutorial) - 10https://gerrit.wikimedia.org/r/612410 (owner: 10Jeena Huneidi) [22:22:48] (03CR) 10Dduvall: [C: 03+2] Revert "Revert "PipelineRunner: allowedCredentials"" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/671270 (owner: 10Dduvall) [22:23:04] (03CR) 10Dduvall: [C: 03+1] Define and pass allowed credentials to pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/668199 (https://phabricator.wikimedia.org/T269900) (owner: 10Jeena Huneidi) [22:23:55] (03Merged) 10jenkins-bot: Revert "Revert "PipelineRunner: allowedCredentials"" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/671270 (owner: 10Dduvall) [22:28:19] !log running `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb '*-pipeline-*'` to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/668199 [22:28:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:30:25] (03CR) 10Dduvall: [C: 03+2] Define and pass allowed credentials to pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/668199 (https://phabricator.wikimedia.org/T269900) (owner: 10Jeena Huneidi) [22:31:30] (03CR) 10Jeena Huneidi: "recheck" [blubber-doc/example/helloworldoid] (kubernetes-tutorial) - 10https://gerrit.wikimedia.org/r/612410 (owner: 10Jeena Huneidi) [22:32:03] (03Merged) 10jenkins-bot: Define and pass allowed credentials to pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/668199 (https://phabricator.wikimedia.org/T269900) (owner: 10Jeena Huneidi) [22:48:19] (03PS1) 10Dduvall: zuul: Run trigger-helloworldoid-pipeline-publish for tag updates [integration/config] - 10https://gerrit.wikimedia.org/r/671295 [22:51:15] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release Pipeline: PipelineBuilder based jobs need to handle git tags and other ref updates - https://phabricator.wikimedia.org/T277346 (10dduvall) p:05Triage→03Medium a:03dduvall [22:51:42] (03PS2) 10Dduvall: zuul: Run trigger-helloworldoid-pipeline-publish for tag updates [integration/config] - 10https://gerrit.wikimedia.org/r/671295 (https://phabricator.wikimedia.org/T277346) [22:54:17] (03CR) 10Dduvall: [C: 03+2] zuul: Run trigger-helloworldoid-pipeline-publish for tag updates [integration/config] - 10https://gerrit.wikimedia.org/r/671295 (https://phabricator.wikimedia.org/T277346) (owner: 10Dduvall) [22:56:07] (03Merged) 10jenkins-bot: zuul: Run trigger-helloworldoid-pipeline-publish for tag updates [integration/config] - 10https://gerrit.wikimedia.org/r/671295 (https://phabricator.wikimedia.org/T277346) (owner: 10Dduvall) [22:57:13] !log reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/671295 [22:57:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:02:18] 10Release-Engineering-Team (Pipeline), 10Release Pipeline, 10Patch-For-Review: Pass allowed credentials parameters to pipeline builder via jjb job templates - https://phabricator.wikimedia.org/T269900 (10jeena) 05Open→03Resolved [23:02:20] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release Pipeline: Better way to restrict credentials available to pipelinelib - https://phabricator.wikimedia.org/T267699 (10jeena) [23:02:40] 10Release-Engineering-Team (Pipeline), 10Release Pipeline: Allow overrides to pipeline builder for passing allowed credentials list - https://phabricator.wikimedia.org/T269902 (10jeena) 05Open→03Resolved [23:02:43] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release Pipeline: Better way to restrict credentials available to pipelinelib - https://phabricator.wikimedia.org/T267699 (10jeena) [23:05:14] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Release Pipeline, 10WVUI, 10Vue.js (Vue.js-Search): Add ability to use ssh-key credentials in pipelinelib pipelines - https://phabricator.wikimedia.org/T267702 (10jeena) 05Open→03Resolved Done in... [23:05:52] 10Release-Engineering-Team (Pipeline), 10Release Pipeline, 10Wikidata, 10Wikidata-Query-Service: Allow timeout override for pipeline k8s deployments to ci cluster - https://phabricator.wikimedia.org/T277125 (10jeena) 05Open→03Resolved [23:06:35] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Release Pipeline: Make Pipelinelib promote step use service name instead of repo name - https://phabricator.wikimedia.org/T264236 (10jeena) 05Open→03Resolved