[00:04:21] <wikibugs>	 (03open) 10addbot: Bump docker-registry.wikimedia.org/wikimedia/mediawiki-libs-shellbox image group to 2024-10-07-154620 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/584
[00:50:03] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T375657#10216082 (10Jdlrobson)
[00:52:53] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T375657#10216093 (10Jdlrobson) I'm cautiously adding T376814 as a blocker. This is pretty visible and impactful to our mobile editors and if we can roll it ba...
[01:04:55] <wikibugs>	 10Phabricator, 10Tool-ldap: https://ldap.toolforge.org/ integration assumes that `cn` and `uid` are equivalent - https://phabricator.wikimedia.org/T376769#10216109 (10Legoktm) Deployed https://gitlab.wikimedia.org/toolforge-repos/ldap/-/commit/99594edfa46508acc8d3f286472e6f2dbd8c08e9  ` $ curl -I 'https://ldap...
[01:07:40] <wikibugs>	 (03open) 10legoktm: Fix ldap.toolforge.org link to use new /cn/ route [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/41 (https://phabricator.wikimedia.org/T376769)
[01:28:48] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T375657#10216136 (10Ladsgroup) I'd wait a bit until @Ebrahim can see the issue and might be able to find a solution quickly. If it's not possible, we can merg...
[02:52:25] <icinga-wm>	 PROBLEM - gerrit process on gerrit2003 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/lib/jvm/java-17-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site https://wikitech.wikimedia.org/wiki/Gerrit
[02:53:25] <icinga-wm>	 RECOVERY - gerrit process on gerrit2003 is OK: PROCS OK: 1 process with regex args ^/usr/lib/jvm/java-17-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site https://wikitech.wikimedia.org/wiki/Gerrit
[04:04:25] <icinga-wm>	 PROBLEM - gerrit process on gerrit2003 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/lib/jvm/java-17-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site https://wikitech.wikimedia.org/wiki/Gerrit
[04:05:25] <icinga-wm>	 RECOVERY - gerrit process on gerrit2003 is OK: PROCS OK: 1 process with regex args ^/usr/lib/jvm/java-17-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site https://wikitech.wikimedia.org/wiki/Gerrit
[05:12:35] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T375657#10216216 (10Ebrahim) I reproduced the issue on testwiki. As markup there are too different from regular use and the page is doing things too different...
[05:19:25] <icinga-wm>	 PROBLEM - gerrit process on gerrit2003 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/lib/jvm/java-17-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site https://wikitech.wikimedia.org/wiki/Gerrit
[05:20:25] <icinga-wm>	 RECOVERY - gerrit process on gerrit2003 is OK: PROCS OK: 1 process with regex args ^/usr/lib/jvm/java-17-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site https://wikitech.wikimedia.org/wiki/Gerrit
[06:04:25] <icinga-wm>	 PROBLEM - gerrit process on gerrit2003 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/lib/jvm/java-17-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site https://wikitech.wikimedia.org/wiki/Gerrit
[06:05:25] <icinga-wm>	 RECOVERY - gerrit process on gerrit2003 is OK: PROCS OK: 1 process with regex args ^/usr/lib/jvm/java-17-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site https://wikitech.wikimedia.org/wiki/Gerrit
[06:52:32] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T375657#10216433 (10Ebrahim) A fix uploaded at https://gerrit.wikimedia.org/r/1079146 and here is the screenshots https://phabricator.wikimedia.org/T376814#10...
[07:14:01] <wikibugs>	 06Release-Engineering-Team, 06Infrastructure-Foundations: Spicerack's tox config times out all the time after T342019 - https://phabricator.wikimedia.org/T372485#10216506 (10elukey) 05Open→03Resolved a:03elukey
[07:16:19] <awight>	 Cross-posting from operations: the SSH host key fingerprints for deploy2002 don't match wikitech.  Please advise...
[07:18:38] <wikibugs>	 10Gerrit, 06collaboration-services, 10VPS-project-Codesearch, 10VPS-project-Extdist, 13Patch-For-Review: Move clients off of gerrit-replica.wikimedia.org back to gerrit.wikimedia.org - https://phabricator.wikimedia.org/T336710#10216523 (10hashar) >>! In T336710#10215974, @Ladsgroup wrote: > BTW, last tim...
[07:19:33] <hashar>	 awight: here is what I have:  deploy2002.codfw.wmnet,deploy2002,10.192.32.7,2620:0:860:103:10:192:32:7 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBY+38EuejOMROC+TbiReqgSMiiKX5SKJASBs4fwvRfLdtgqgTtNTigHuLWFcw2o9cTo5ewBDvuFASQKeysCat4=
[07:22:08] <awight>	 hashar: Thanks for the sanity check—seems that the wikitech page needs an update, then!  I'll mention on the talk page.
[07:24:17] <hashar>	 it got reimaged on July 30 and I guess updating the fingerprint is not part of the automated process. The task was https://phabricator.wikimedia.org/T371282
[07:24:28] <hashar>	 I don't know how to securely update them :/
[07:27:33] <awight>	 Makes perfect sense, then!  https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/deploy2002.codfw.wmnet was last edited in 2021.
[07:29:53] <wikibugs>	 10Continuous-Integration-Infrastructure: Drop xdebug from all RelEng images - https://phabricator.wikimedia.org/T319495#10216547 (10ArthurTaylor) Thanks for the explanation @hashar. I understand the point that XDebug shouldn't be used in CI, but there are cases where it's useful to be able to connect a debug...
[07:31:33] <awight>	 wild.  My talk page edit with the new fingerprint was blocked by rule "common vandalistic edits"
[07:32:27] <icinga-wm>	 PROBLEM - gerrit process on gerrit2003 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/lib/jvm/java-17-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site https://wikitech.wikimedia.org/wiki/Gerrit
[07:34:03] <wikibugs>	 (03CR) 10Arthur taylor: [C:03+2] "Makes sense - thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/1078978 (owner: 10Hashar)
[07:34:49] <wikibugs>	 (03CR) 10Arthur taylor: [C:03+2] "sure - let's do it :)" [integration/config] - 10https://gerrit.wikimedia.org/r/1078981 (owner: 10Hashar)
[07:35:29] <icinga-wm>	 RECOVERY - gerrit process on gerrit2003 is OK: PROCS OK: 1 process with regex args ^/usr/lib/jvm/java-17-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site https://wikitech.wikimedia.org/wiki/Gerrit
[07:35:50] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: grep from log dir to have short names [integration/config] - 10https://gerrit.wikimedia.org/r/1078978 (owner: 10Hashar)
[07:36:24] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: force grep color when searching mw error logs [integration/config] - 10https://gerrit.wikimedia.org/r/1078981 (owner: 10Hashar)
[07:38:15] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Cloud-VPS (Quota-requests): Quota increase for Integration project (Jenkins CI runners) - https://phabricator.wikimedia.org/T376847#10216566 (10dcaro) Is this going to be freed after the test?  Also, it seems you are requesting more cpu and ram than 12 instances of th...
[07:43:28] <wikibugs>	 (03CR) 10Isabelle Hurbain-Palatin: README.md: doc loading a plugin from the browser (032 comments) [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1078735 (owner: 10Hashar)
[07:46:27] <icinga-wm>	 PROBLEM - gerrit process on gerrit2003 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/lib/jvm/java-17-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site https://wikitech.wikimedia.org/wiki/Gerrit
[07:46:37] <hashar>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1079206 Disable gerrit monitoring on gerrit2003
[07:47:27] <icinga-wm>	 RECOVERY - gerrit process on gerrit2003 is OK: PROCS OK: 1 process with regex args ^/usr/lib/jvm/java-17-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site https://wikitech.wikimedia.org/wiki/Gerrit
[08:03:59] <wikibugs>	 (03CR) 10Hashar: [C:04-1] "**Thanks Isabelle**, I assumed Chromium would behave the same way as Firefox and it different.  So I will rework the text:" [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1078735 (owner: 10Hashar)
[08:06:28] <hashar>	 codders: have you updated the Jenkins jobs after +2ing https://gerrit.wikimedia.org/r/c/integration/config/+/1078978 and https://gerrit.wikimedia.org/r/c/integration/config/+/1078981/  ? :)
[08:06:32] <hashar>	 it is not done automatically
[08:08:51] <andre>	 Hmm, train blocked on https://phabricator.wikimedia.org/T376814 though I personally would not consider that a train blocker. hashar: Got any opinions? :-/
[08:09:04] <codders>	 I have not done - what do I need to do for that?
[08:09:50] <RhinosF1>	 andre: there is a patch on it though already that should fix
[08:09:59] <andre>	 RhinosF1, yes but unreviewed
[08:10:29] <RhinosF1>	 could someone review quick enough
[08:10:58] <RhinosF1>	 https://wikitech.wikimedia.org/wiki/Deployments/Holding_the_train#Major_stylistic_problems_affecting_all_pages doesn't really help
[08:11:05] <RhinosF1>	 it says avoid individual decisions
[08:13:30] * RhinosF1 goes off to see if he can poke anyone from a useful team
[08:18:14] <hashar>	 andre: let me check
[08:18:33] <hashar>	 ah that is frontend thing :/
[08:19:11] <hashar>	 so yeah the web team would block the train on major visual regression, it is up to them really, they have the data/metrics/knowledge to know when something is major
[08:19:23] <andre>	 I'm even like... where to find some way how to contact that team in real time? WMF, sigh
[08:19:54] <hashar>	 try their Skype room?
[08:19:58] * hashar giggles
[08:20:32] <andre>	 I already tried ICQ
[08:20:40] <hashar>	 TOUCHÉ
[08:20:53] * hashar FALLS ON THE FLOOR LAUGHING
[08:21:48] <hashar>	 more seriously
[08:21:52] <hashar>	 jon proposed a rollback https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1079044
[08:22:05] * hashar which reverts https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1077790
[08:22:12] <hashar>	 written by Ebrahim
[08:22:23] <hashar>	 who in turns propose the fix at https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1079146
[08:27:01] <andre>	 hmm I guess I'll wait then, meh
[08:35:23] <hashar>	 I think the fix would have to wait for Jon
[08:36:09] <hashar>	 an alternative would be to revert solely in the wmf branch
[08:36:36] <hashar>	 that unlocks the train right now
[08:36:48] <hashar>	 and save us from potentially having to postpone it to monday
[08:40:09] <andre>	 hashar: I'll drop a message to wikitech-l@
[08:40:21] <hashar>	 andre: I am sending a revert patch for wmf branch :)
[08:40:47] <andre>	 hashar, okay in that case I probably shouldn't message wikitech-l@
[08:40:49] <andre>	 thx
[08:41:22] <hashar>	 https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1079218 Revert "Use HTML markup instead of bidi control chars in wiki changes" 
[08:41:27] <hashar>	 andre: for your CR+2 power
[08:42:05] <jelto>	 GitLab needs a short maintenance restart in one hour (at 9:45 UTC)
[08:42:24] <andre>	 hashar: okay, I think I need some handholding for this, sorry. Cannot CR+2 that one myself, no such button, hmm
[08:42:43] <hashar>	 WHAT
[08:43:06] <hashar>	 ah I know
[08:43:13] <andre>	 maybe our team onboarding lacks some permission steps? hummm
[08:43:16] <hashar>	 you might not be in the cool kids user group
[08:43:38] <andre>	 I'm not in https://gerrit.wikimedia.org/r/admin/groups/11,members , heh
[08:44:07] <hashar>	 the parent permissions are on https://gerrit.wikimedia.org/r/admin/repos/mediawiki,access
[08:44:17] <hashar>	 under `refs/heads/wmf/*`  (that is for the deployment branches)
[08:44:42] <hashar>	 which allos CR+2 to CI, wmf-deployment, that group 11 you mention above and who ever is owning the project (that is Gerrit admins)
[08:45:00] <andre>	 ah, so I guess I could try to add myself to https://gerrit.wikimedia.org/r/admin/groups/3fdcf8fd0d569e90a3e9b39788a29f2c50d33be9,members ?
[08:45:05] <hashar>	 yeah
[08:45:14] <andre>	 okay, done
[08:45:27] <andre>	 yes, I can +2 now. Thanks
[08:45:27] <hashar>	 can you file form X-803-Y with two copies and mail it to the Headquarter legal department?
[08:45:34] <hashar>	 hmm
[08:45:40] <hashar>	 that groups includes release engineering..
[08:45:51] <hashar>	 ah
[08:45:52] <hashar>	 https://gerrit.wikimedia.org/r/admin/groups/86c7c9e8fe1436b71bba8433ac31a908fe43cc5b,members
[08:46:27] <andre>	 eheh
[08:46:47] <andre>	 okay, also added myself to the RelEng gerrit group now
[08:47:14] <andre>	 hashar: I'll submit/merge your revert
[08:47:34] <hashar>	 but
[08:47:49] <hashar>	 now I wonder how you added yourself to the group? 
[08:47:53] <hashar>	 are you a gerrit admin?
[08:47:54] <hashar>	 :D
[08:47:58] <andre>	 hashar, looks like :D
[08:48:13] <hashar>	 https://ldap.toolforge.org/group/gerritadmin
[08:48:19] <hashar>	 indeed
[08:48:21] <hashar>	 \o/
[08:48:32] <andre>	 I lost track of my hats, need a bigger cupboard for them
[08:49:10] <andre>	 okay, merged
[08:49:15] <andre>	 Do I need to log that?
[08:51:28] <hashar>	 hmm
[08:51:37] <hashar>	 no I don't think so
[08:51:43] <hashar>	 well you can also !log it here
[08:51:59] <hashar>	 else there is the audit log https://gerrit.wikimedia.org/r/admin/groups/86c7c9e8fe1436b71bba8433ac31a908fe43cc5b,audit-log
[08:52:03] <hashar>	 I think that is good enough
[08:52:37] <hashar>	 you need the change to be pulled on the deployment server, I guess `scap backport` would do it
[08:54:12] <andre>	 thanks, I'll `scap backport 1079218` on deploy2002 now
[09:03:31] <wikibugs>	 10Continuous-Integration-Infrastructure: Drop xdebug from all RelEng images - https://phabricator.wikimedia.org/T319495#10216783 (10hashar) 05Resolved→03Open I agree yes, it is rather small compared to the size of the Quibble instance. I guess we can partially revert https://gerrit.wikimedia.org/r/c/integrat...
[09:12:14] <andre>	 Thanks. Backport finished. Now going to deploy .26 to group2
[09:13:16] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T375657#10216799 (10Aklapper)
[09:16:34] <hashar>	 andre: congrats!
[09:16:41] <hashar>	 codders: sorry I got caught with that train thing :)
[09:16:49] <andre>	 hashar, thanks for the help. Took some additional notes
[09:17:02] <codders>	 no worries - seems important
[09:17:05] <hashar>	 codders: you'd need to setup Jenkins Job Builder locally https://www.mediawiki.org/wiki/CI/JJB
[09:17:09] <hashar>	 with the auth token
[09:17:36] <codders>	 k. i'll take a look at that
[09:17:39] <hashar>	 there is then an helper at the root of integration/config to update jobs: `./jjb-update [list of jobs to update]`
[09:17:43] <hashar>	 I have run it and updating the jobs
[09:18:07] <hashar>	 we can probably create a dummy job to let you train :]
[09:18:42] <codders>	 I'll probably get to that between 12h and 13h. A dummy job would be helpful
[09:48:30] <hashar>	 codders: https://gerrit.wikimedia.org/r/c/integration/config/+/1079227 :)
[09:49:18] <hashar>	 and the wiki doc should work as well
[09:50:12] <jelto>	 GitLab maintenance done
[09:53:00] <hashar>	 jelto: congrats
[09:53:50] <jelto>	 thanks :)
[10:07:38] <andre>	 yay
[10:43:17] <codders>	 @hashar: https://integration.wikimedia.org/ci/view/All/job/arthur-hello/ seems to work!
[11:20:08] <hashar>	 codders: well done, and one can see the diff at https://integration.wikimedia.org/ci/view/All/job/arthur-hello/jobConfigHistory/
[11:20:33] <hashar>	 you can delete it using the web interface (Delete Project) in the sidebar
[11:20:38] <hashar>	 and abandon the change
[11:21:18] <hashar>	 to verify the diff and obtain the list of affected jobs, you can run ./utils/jjb-diff.sh
[11:21:46] <hashar>	 the list of jobs can be pasted in the commit message, though I usually do not do it when the list of jobs long
[11:25:21] <codders>	 so you would usually run jjb-diff, then commit, then run jjb-update? Or what would your process be?
[11:27:38] <hashar>	 hmm pretty much
[11:28:05] <hashar>	 I would start hacking the yaml files,  run `./utils/jjb-diff.sh` to check the resulting modifications
[11:28:21] <hashar>	 that script accepts a job filter as argument which speed it up
[11:29:04] <hashar>	 ./utils/jjb-diff.sh arthur-hello
[11:29:10] <hashar>	 rinse and repeat until I am happy with the result
[11:29:31] <hashar>	 I then craft the commit and would pass what jjb-diff generated for me:
[11:29:43] <hashar>	 New jobs:
[11:29:43] <hashar>	  - arthur-hello
[11:30:09] <hashar>	 or not paste it depending on my mood. Ie for a single job, it is usually mentioned in the commit message so that above text is useless
[11:30:24] <hashar>	 if a change modifies a lot of jobs, that ends up overloading the commit message with a lot of lines so I would skip
[11:30:25] <codders>	 fair
[11:30:35] <hashar>	 then git-review
[11:31:02] <hashar>	 I would additionally check the output of the integration-config-jjb-diff but that is the same output as the local `./utils/jjb-diff.sh`
[11:31:16] <hashar>	 then ./jjb-update <the list of jobs>
[11:31:23] <hashar>	 CR+2
[11:31:53] <codders>	 got it.
[11:31:55] <hashar>	 alternative:  update just one or two jobs, check they are working then CR+2 and update all jobs
[11:32:33] <hashar>	 at least, you managed to deploy ONE job, so I guess everything is properly setup
[11:34:33] <codders>	 so should I try deploy the updates from earlier? Which jobs are actually impacted by those changes? just quibble?
[11:35:16] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Decrease number of open Phab tickets with assignee field set for more than two years (aka cookie licking) (Q4/2024 edition) - https://phabricator.wikimedia.org/T365183#10217231 (10Aklapper) 05Stalled→03Open a:03Aklapper
[11:38:44] <hashar>	 codders: yeah quibble jobs. I have deployed it though :)
[11:38:52] <hashar>	 using pretty much the above
[11:39:04] <hashar>	 I have updated my copy of integration/config to point to the change
[11:39:14] <hashar>	 ran ./utils/jjb-diff.sh to get the list of affected jobs
[11:39:23] <hashar>	 then ./jjb-update <the list of jobs>
[11:40:20] <codders>	 sounds straightforward
[11:40:30] <wikibugs>	 (03PS1) 10Arthur taylor: Add xdebug back to Quibble PHP images [integration/config] - 10https://gerrit.wikimedia.org/r/1079250 (https://phabricator.wikimedia.org/T319495)
[11:41:16] <hashar>	 and hopefully we haven't broke everything
[11:41:28] <codders>	 for something like ^^, would it make sense to +2 that first, then make a separate change for the job config
[11:41:40] <codders>	 (besides that I probably don't have permission to push docker images)
[11:43:17] <hashar>	 ah
[11:44:17] <hashar>	 when connected on `contint.wikimedia.org` , sudo -l should give you:
[11:44:17] <hashar>	     (dockerpkg-builder) NOPASSWD: ALL
[11:44:46] <hashar>	 the `./fab deploy_docker` command ssh to that hostname and run docker-pkg as `dockerpkg-builder`
[11:45:02] <hashar>	 well or should
[11:45:13] <codders>	 confirmed - I have dockerpkg-builder
[11:45:34] <codders>	 but the 'fab' command is new to me
[11:46:00] <hashar>	 I think dockerpkg-builder is only used for pushing
[11:46:32] <hashar>	 so the `fab` script is named like that because at some point we used https://www.fabfile.org/
[11:46:58] <codders>	 got it
[11:47:04] <hashar>	 reworked as a shell script T236689
[11:47:04] <stashbot>	 T236689: Upgrade integration/config to use Fabric 2.x / python3 - https://phabricator.wikimedia.org/T236689
[11:47:10] <hashar>	 that is one less level of overhead :]
[11:48:00] <hashar>	 so that `./fab deploy_docker`  ssh to the host, invoke docker-pkg there which will eventually publish the images
[11:48:49] <codders>	 based on code on master? Or based on a patch?
[11:49:13] <hashar>	 the scripts pull  integration/config on the contint host
[11:49:17] <hashar>	 so whatever is master is deployed
[11:49:23] <codders>	 k.
[11:49:45] <codders>	 so it would be +2 the docker image changes, `./fab deploy_docker`, then stage the jjb changes and jjb_diff etc.etc.
[11:49:56] <hashar>	 correct!
[11:50:59] <codders>	 great - so far so understood! That sounds like an adventure for tomorrow-me
[11:51:13] <codders>	 thanks for the explanations
[11:52:23] <hashar>	 well gotta try once :]
[11:52:48] <hashar>	 I am going offline, I have to rewrite a git repo history
[11:52:51] <hashar>	 and gotta focus
[11:53:14] <codders>	 have fun - cya tomorrow!
[12:12:07] <wikibugs>	 (03CR) 10Jforrester: [C:04-1] "There is extra cost to these in configuring, build time, image size, and reasoning complexity. I don't think this is a good trade-off." [integration/config] - 10https://gerrit.wikimedia.org/r/1079250 (https://phabricator.wikimedia.org/T319495) (owner: 10Arthur taylor)
[12:19:23] <wikibugs>	 (03CR) 10Jforrester: "It doesn't look like you deployed this change, or the one before it? It's not live on contint.wikimedia.org from what I can see, and there" [integration/config] - 10https://gerrit.wikimedia.org/r/1078981 (owner: 10Hashar)
[12:32:46] <wikibugs>	 (03CR) 10Hashar: "I ran the update this morning. Although I did not `!log` it, I reached out to Arthur over IRC to let him know and we exchanged to have him" [integration/config] - 10https://gerrit.wikimedia.org/r/1078981 (owner: 10Hashar)
[12:33:15] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Priority Backlog 📥), 10Scap: Automate Gerrit deployment steps - https://phabricator.wikimedia.org/T317412#10217411 (10Aklapper)
[12:37:37] <James_F>	 !log Zuul: Sync'ed to contint.wikimedia.org to push new jjb patches to machine, already live on jenkins itself.
[12:37:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[12:42:11] <wikibugs>	 (03CR) 10Jforrester: "Aha, awesome." [integration/config] - 10https://gerrit.wikimedia.org/r/1078981 (owner: 10Hashar)
[12:49:18] <wikibugs>	 10Scap, 10MW-on-K8s, 06serviceops: Evaluate the performance improvements brought in by prefetching MW images on WikiKube hosts - https://phabricator.wikimedia.org/T366778#10217508 (10akosiaris) 05Open→03Resolved a:03akosiaris I 'll resolve this one. Things overall are OK deployment times wise. In f...
[13:09:18] <wikibugs>	 10Phabricator: Logging in - https://phabricator.wikimedia.org/T376803#10217596 (10Hdicu) 05Stalled→03Open I'm using Edge on Mac Screenshots from devtools: After trying to click button {F57604488} After. reloading and pressing button and going to mediawiki login {F57604489}  Works in Safari. Same problem in c...
[14:00:13] <hashar>	 !log gerrit: reindex all changes (`gerrit index start changes --force`)
[14:00:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:37:05] <Reedy>	 I guess that explains why gerrit is down
[14:51:18] <wikibugs>	 (03update) 10dancy: Don't load plugins from user's home directory or current directory [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/466
[14:53:21] <wikibugs>	 (03merge) 10dancy: Don't load plugins from user's home directory or current directory [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/466
[15:25:54] <wmf-insecte>	 Yippee, build fixed!
[15:25:54] <wmf-insecte>	 Project beta-code-update-eqiad build #516867: 09FIXED in 2 min 54 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/516867/
[15:28:11] <wikibugs>	 (03update) 10dancy: WIP: spiderpig auth [repos/releng/scap] (master-I6b866887f1fb30397d9cb409a4f0844cc11235c2) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/472
[15:28:29] <wikibugs>	 (03update) 10dancy: WIP: spiderpig auth [repos/releng/scap] (master-I6b866887f1fb30397d9cb409a4f0844cc11235c2) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/472
[15:28:30] <wikibugs>	 (03update) 10dancy: WIP: Spiderpig HTTP API [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/451
[15:28:35] <wikibugs>	 (03update) 10dancy: WIP: SpiderPig Web UI [repos/releng/scap] (master-I0a50f91db8c9aa9147afdd0468e37e001af86167) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/471
[15:29:52] <wikibugs>	 (03open) 10dancy: Release 4.110.0-1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/473
[15:31:43] <wikibugs>	 (03merge) 10dancy: Release 4.110.0-1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/473
[15:31:45] <wikibugs>	 (03update) 10dancy: WIP: spiderpig auth [repos/releng/scap] (master-I6b866887f1fb30397d9cb409a4f0844cc11235c2) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/472
[15:34:33] <wikibugs>	 (03update) 10dancy: WIP: spiderpig auth [repos/releng/scap] (master-I6b866887f1fb30397d9cb409a4f0844cc11235c2) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/472
[15:44:38] <wikibugs>	 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#10218318 (10JTweed-WMF) Thanks @Ladsgroup, confirmed as working.
[16:12:40] <wikibugs>	 10Phabricator: Logging into Phabricator via SUL works only on second time with Chrome only - https://phabricator.wikimedia.org/T376803#10218538 (10Aklapper)
[16:17:23] <wikibugs>	 10Gerrit, 06collaboration-services, 10VPS-project-Codesearch, 10VPS-project-Extdist, 13Patch-For-Review: Move clients off of gerrit-replica.wikimedia.org back to gerrit.wikimedia.org - https://phabricator.wikimedia.org/T336710#10218571 (10Dzahn) I had already clicked +2 on that change yesterday and it ha...
[16:31:24] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T375657#10218631 (10Aklapper) 05Open→03Resolved
[16:39:57] <James_F>	 Hmm, the longest-running gate-and-submit job is currently at 2.5 hours but no alerting in here. Did it break?
[16:41:12] <James_F>	 (The alerting, that is. We all know that CI can flake some times.)
[16:53:21] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Quibble: Speed up git clone operation in wmf-quibble job - https://phabricator.wikimedia.org/T374717#10218709 (10kostajh) >>! In T374717#10206214, @hashar wrote: >>>! In T374717#10152684, @kostajh wrote: >> @ArthurTaylor and I had noticed...
[17:58:07] <wikibugs>	 (03update) 10dancy: WIP: spiderpig auth [repos/releng/scap] (master-I6b866887f1fb30397d9cb409a4f0844cc11235c2) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/472
[17:58:11] <wikibugs>	 (03update) 10dancy: WIP: Spiderpig HTTP API [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/451
[17:58:14] <wikibugs>	 (03update) 10dancy: WIP: SpiderPig Web UI [repos/releng/scap] (master-I0a50f91db8c9aa9147afdd0468e37e001af86167) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/471
[18:52:27] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Codex, 06Design-System-Team, 07Epic: [Spike/Timebox] DST Support for Codex-ifying SpiderPig UI - https://phabricator.wikimedia.org/T376932 (10egardner) 03NEW
[18:57:15] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Codex, 10Design-System-Team (DST-Sprint-33 (2024-09-30 to 2024-10-11)), 07Epic: [Spike/Timebox] DST support for web app scaffolding & initial setup - https://phabricator.wikimedia.org/T375826#10219200 (10egardner) 05Open→03Resolved Filed...
[19:03:19] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Codex, 06Design-System-Team, 07Epic: [Spike/Timebox] DST Support for Codex-ifying SpiderPig UI - https://phabricator.wikimedia.org/T376932#10219246 (10egardner) Hey @dancy – what do you think about showing only the last X number of jobs on the ma...
[19:05:54] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Codex, 06Design-System-Team, 07Epic: [Spike/Timebox] DST Support for Codex-ifying SpiderPig UI - https://phabricator.wikimedia.org/T376932#10219250 (10dancy) Here are screenshots of the current prototype:  Login page: {F57604892} Overview page af...
[19:07:33] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Codex, 06Design-System-Team, 07Epic: [Spike/Timebox] DST Support for Codex-ifying SpiderPig UI - https://phabricator.wikimedia.org/T376932#10219269 (10dancy) >>! In T376932#10219246, @egardner wrote: > Hey @dancy – what do you think about showing...
[19:11:27] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Codex, 06Design-System-Team, 07Epic: [Spike/Timebox] DST Support for Codex-ifying SpiderPig UI - https://phabricator.wikimedia.org/T376932#10219272 (10dancy)
[19:14:35] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Codex, 06Design-System-Team, 07Epic: [Spike/Timebox] DST Support for Codex-ifying SpiderPig UI - https://phabricator.wikimedia.org/T376932#10219277 (10dancy)
[19:39:49] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Codex, 06Design-System-Team, 07Epic: [Spike/Timebox] DST Support for Codex-ifying SpiderPig UI - https://phabricator.wikimedia.org/T376932#10219416 (10mwilliams) I'd be happy to support this from the design side. Maybe we start with getting the p...
[19:43:25] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Codex, 06Design-System-Team, 07Epic: [Spike/Timebox] DST Support for Codex-ifying SpiderPig UI - https://phabricator.wikimedia.org/T376932#10219421 (10egardner) >>! In T376932#10219416, @mwilliams wrote: > I'd be happy to support this from the de...
[20:01:38] <wikibugs>	 (03update) 10dancy: WIP: spiderpig auth [repos/releng/scap] (master-I6b866887f1fb30397d9cb409a4f0844cc11235c2) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/472
[20:01:45] <wikibugs>	 (03update) 10dancy: WIP: SpiderPig Web UI [repos/releng/scap] (master-I0a50f91db8c9aa9147afdd0468e37e001af86167) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/471
[20:06:18] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Cloud-VPS (Quota-requests): Quota increase for Integration project (Jenkins CI runners) - https://phabricator.wikimedia.org/T376847#10219529 (10bd808) >>! In T376847#10216566, @dcaro wrote: > Is this going to be freed after the test?  I would expect the quota to stay...
[20:31:46] <wikibugs>	 10Gerrit, 06collaboration-services, 10VPS-project-Codesearch, 10VPS-project-Extdist, 13Patch-For-Review: Move clients off of gerrit-replica.wikimedia.org back to gerrit.wikimedia.org - https://phabricator.wikimedia.org/T336710#10219597 (10Dzahn) This is now deployed.   I did run these commands. but after...
[20:50:21] <wikibugs>	 10Gerrit, 10Wikidata, 10Wikidata Analytics: Clean up old and unused WMDE Gerrit groups - https://phabricator.wikimedia.org/T376951 (10WMDE-leszek) 03NEW
[21:07:09] <mutante>	 !log gerrit - removed all members from all WMDE gerrit groups listed on T376951
[21:07:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:07:12] <stashbot>	 T376951: Clean up old and unused WMDE Gerrit groups - https://phabricator.wikimedia.org/T376951
[21:20:06] <wikibugs>	 (03PS1) 10Eric Gardner: Add Codex PHP doc and source code link [integration/docroot] - 10https://gerrit.wikimedia.org/r/1079360 (https://phabricator.wikimedia.org/T375939)
[21:28:53] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T375658#10219820 (10Jdlrobson)
[21:49:56] <wikibugs>	 10Gerrit, 06collaboration-services, 10Wikidata, 10Wikidata Analytics: Clean up old and unused WMDE Gerrit groups - https://phabricator.wikimedia.org/T376951#10219909 (10Dzahn)
[21:50:38] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] Add Codex PHP doc and source code link [integration/docroot] - 10https://gerrit.wikimedia.org/r/1079360 (https://phabricator.wikimedia.org/T375939) (owner: 10Eric Gardner)
[21:51:06] <wikibugs>	 (03Merged) 10jenkins-bot: Add Codex PHP doc and source code link [integration/docroot] - 10https://gerrit.wikimedia.org/r/1079360 (https://phabricator.wikimedia.org/T375939) (owner: 10Eric Gardner)
[22:54:38] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10220077 (10Umherirrender) F...
[23:40:57] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥): Draft and get approval for next hypothesis to follow WE6.2.1 - https://phabricator.wikimedia.org/T375145#10220264 (10bd808) Back channel discussions between @bd808 and @thcipriani are trending towards the next necessary step being work to define a strategy and tac...