[00:16:51] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Release, 10Train Deployments: 1.41.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T340247 (10tstarling) ##### Risky Patch! πŸš‚πŸ”₯ * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/936132 * **Summary**: ** Diff cache keys are ch... [02:09:01] 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MediaWiki-Core-Tests, 10Code-Health, and 8 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Krinkle) I've uploaded the raw log files from @Ladsgroup's high-res ex... [04:01:51] (03PS2) 10Subramanya Sastry: Few more fixes based on visual diff testing [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/940262 [06:45:16] 10Phabricator, 10Release-Engineering-Team (Radar), 10DBA, 10Data-Persistence-Backup, and 2 others: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10Marostegui) I have added DROP (which gives you TRUNCATE), CREATE and CREATE TEMPORARY TABLES. [06:49:32] 10Phabricator, 10wikimedia.biterg.io: Closed tickets in Bugzilla migrated without a closing date - https://phabricator.wikimedia.org/T107254 (10valerio.bozzolan) Let's join this lovely pastebin with our scraped stuff [07:33:31] PROBLEM - Gerrit JSON on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [07:33:45] PROBLEM - Gerrit Health Check SSL Expiry on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [07:33:47] PROBLEM - Gerrit Health Check on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [07:34:53] RECOVERY - Gerrit JSON on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 74583 bytes in 0.040 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [07:35:07] RECOVERY - Gerrit Health Check SSL Expiry on gerrit.wikimedia.org is OK: OK - Certificate gerrit.wikimedia.org will expire on Sun 08 Oct 2023 09:52:13 PM GMT +0000. https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [07:35:09] RECOVERY - Gerrit Health Check on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 973 bytes in 0.028 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [07:45:24] 10GitLab (Auth & Access), 10Release-Engineering-Team, 10collaboration-services, 10Patch-For-Review, 10User-brennen: Create bot to sync LDAP groups with related GitLab groups - https://phabricator.wikimedia.org/T319211 (10Jelto) The script fails during the daily restore with ` requests.exceptions.HTTPErr... [07:48:18] 10Phabricator: Herald rule broken for CommTech - https://phabricator.wikimedia.org/T342414 (10KSiebert) [07:48:45] 10Phabricator: URGENT: Herald rule broken for CommTech - https://phabricator.wikimedia.org/T342414 (10KSiebert) p:05Triageβ†’03Unbreak! [07:51:09] 10Phabricator: URGENT: Herald rule broken for CommTech - https://phabricator.wikimedia.org/T342414 (10RhinosF1) @KSiebert: is this really everyone should drop everything now to look at this? [07:57:12] 10Phabricator: URGENT: Herald rule broken for CommTech - https://phabricator.wikimedia.org/T342414 (10RhinosF1) @samwilson and @MusikAnimal within your team have access to fix it if it's urgent. They just need to add the Kanban to list of projects not matching the rule. In regards to access, that should probabl... [08:08:30] 10Phabricator: URGENT: Herald rule broken for CommTech - https://phabricator.wikimedia.org/T342414 (10Aklapper) 05Openβ†’03Resolved p:05Unbreak!β†’03High a:03Aklapper Hi, I edited H260 and added "Project tags include none of: Community-Tech (CommTech-Kanban)" as a condition. If I understand correctly, this... [08:19:47] 10Phabricator: URGENT: Herald rule H260 should not remove CommTech Kanban project - https://phabricator.wikimedia.org/T342414 (10Aklapper) [08:22:17] 10Phabricator: URGENT: Herald rule H260 should not remove CommTech Kanban project - https://phabricator.wikimedia.org/T342414 (10Samwilson) @RhinosF1 I don't have access to edit that rule (clicking edit just gives a permissions error: "You do not have permission to edit this object."). [08:24:49] 10Phabricator: URGENT: Herald rule H260 should not remove CommTech Kanban project - https://phabricator.wikimedia.org/T342414 (10Aklapper) @Samwilson: Added you to the ACL group, feel free to try again [08:29:01] 10Continuous-Integration-Config, 10Math, 10phan, 10phan-taint-check-plugin, and 2 others: Phan is stopping for unknown reason in automated checks - https://phabricator.wikimedia.org/T342100 (10Stegmujo) Thanks for the quick help ! [08:34:35] 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MediaWiki-Core-Tests, 10Code-Health, and 8 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Lucas_Werkmeister_WMDE) >>! In T225730#9032655, @Daimona wrote: > The... [08:41:11] 10Phabricator: URGENT: Herald rule H260 should not remove CommTech Kanban project - https://phabricator.wikimedia.org/T342414 (10RhinosF1) >>! In T342414#9033451, @Samwilson wrote: > @RhinosF1 I don't have access to edit that rule (clicking edit just gives a permissions error: "You do not have permission to edit... [08:42:01] 10Phabricator: Herald rule H260 should not remove CommTech Kanban project - https://phabricator.wikimedia.org/T342414 (10RhinosF1) [10:30:52] 10Phabricator: Herald rule H260 should not remove CommTech Kanban project - https://phabricator.wikimedia.org/T342414 (10KSiebert) Thanks for looking at this, yeah, maybe it wasn't that urgent, I just knew it take another 6 hours until Leon will be around and we might have accidentally lost track of some tickets... [10:37:57] 10Phabricator: Herald rule H260 should not remove CommTech Kanban project - https://phabricator.wikimedia.org/T342414 (10KSiebert) Is there a template to request the permissions so I can do it correctly? [10:41:27] 10Phabricator: Herald rule H260 should not remove CommTech Kanban project - https://phabricator.wikimedia.org/T342414 (10RhinosF1) >>! In T342414#9033779, @KSiebert wrote: > Thanks for looking at this, yeah, maybe it wasn't that urgent, sorry for that, I just knew it takes another 6 hours until Leon will be arou... [11:01:01] (03CR) 10Hashar: "I am super happy to see that feature being followed up! I have added a few hints, mostly suggestions for later. There are two though tha" [releng/phatality] - 10https://gerrit.wikimedia.org/r/940265 (https://phabricator.wikimedia.org/T342400) (owner: 10Dduvall) [12:14:19] 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MediaWiki-Core-Tests, 10Code-Health, and 8 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Daimona) [13:07:53] 10Phabricator: Herald rule H260 should not remove CommTech Kanban project - https://phabricator.wikimedia.org/T342414 (10Aklapper) >>! In T342414#9033801, @KSiebert wrote: > Is there a template to request the permissions so I can do it correctly? See https://phabricator.wikimedia.org/project/profile/1081/ itself. [15:14:21] hashar: wonder if you or maybe someone else here can help me track down what happened to https://gerrit.wikimedia.org/r/c/operations/puppet/+/940366 [15:14:45] its shown as merged but when i look in git the ;ast copmmit is from Jun 7 [15:20:46] jbond: I see 940366 as "active" so not merged, where are you seeing Jun 7? [15:21:30] thc sorry wrong paste https://gerrit.wikimedia.org/r/c/operations/puppet/+/938894 [15:22:04] getting jun 7 from [15:22:05] git log -- modules/vrts/files/install_vrts.sh [15:24:38] looks like that commit only touched upgrade_vrts.sh? https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/5f3078d0386f1e28d0143a1abeeba23c1e58ca29 [15:25:48] thcipriani: yes thats correct but im not seeing that commit with the above log command [15:26:42] right, but you should see it with: git log -- modules/vrts/files/upgrade_vrts.sh [15:26:54] oh doh! [15:26:56] :D [15:27:02] thanks yuo for the stupid stick :) [15:27:57] heh, if I had a nickle for every time I did roughly the same thing, I'd have a lot of nickles :) [15:28:11] tangled up in git is my default state [15:28:40] yes i realize daily that i know nothing about how git works :) [15:28:56] although this one is jjst a case of opening my eyes ;) [15:30:44] I learn new things about git constantly. Even though I'd assumed that would stop at some point :P [15:31:03] glad to know im not alone :) [15:41:04] 10Release-Engineering-Team (Deployment Training Requests): Deployment Training Request for jebe - https://phabricator.wikimedia.org/T341559 (10thcipriani) Hi @JEbe-WMF Thanks for requesting deployment training, sorry for the delayed response. The early-UTC window is 7:00 UTC on Thursdays. The next one is 202... [15:41:43] 10Release-Engineering-Team (Deployment Training Requests): Deployment Training Request for jebe - https://phabricator.wikimedia.org/T341559 (10thcipriani) p:05Triageβ†’03Medium [15:42:50] 10Release-Engineering-Team (Deployment Training Requests): Deployment Training Request for jebe - https://phabricator.wikimedia.org/T341559 (10thcipriani) (Also, CCing @ArielGlenn as the usual suspect in UTC-early training, in case of conflicts) [15:44:47] 10Release-Engineering-Team (Deployment Training Requests): Deployment Training Request for xcollazo - https://phabricator.wikimedia.org/T341377 (10thcipriani) Hi @xcollazo thanks for requesting deployment training, apologies for the late reply. The late-UTC training is at 20UTC and the next one is on 2023-07-27... [15:45:34] 10Release-Engineering-Team (Deployment Training Requests): Deployment Training Request for xcollazo - https://phabricator.wikimedia.org/T341377 (10thcipriani) p:05Triageβ†’03Medium [16:17:55] 10GitLab (Auth & Access), 10Release-Engineering-Team, 10collaboration-services, 10Patch-For-Review, 10User-brennen: Create bot to sync LDAP groups with related GitLab groups - https://phabricator.wikimedia.org/T319211 (10dancy) >>! In T319211#9033400, @Jelto wrote: > But we should think about what happen... [16:33:15] 10Phabricator, 10Security-Team, 10Security: Create #security-bug tag to specifically classify security bugs within phabricator - https://phabricator.wikimedia.org/T342449 (10sbassett) [16:34:35] 10Phabricator, 10Security-Team, 10Security: Create #security-bug tag to specifically classify security bugs within phabricator - https://phabricator.wikimedia.org/T342449 (10sbassett) [16:35:11] 10Phabricator, 10Security-Team, 10Security: Create #security-bug tag to specifically classify security bugs within phabricator - https://phabricator.wikimedia.org/T342449 (10sbassett) [16:43:10] 10Project-Admins, 10Security-Team, 10Security: Create #security-bug tag to specifically classify security bugs within phabricator - https://phabricator.wikimedia.org/T342449 (10Aklapper) Removing #Phabricator (the software and its config); adding #Project-Admins (creating a project tag) [16:43:58] 10Project-Admins, 10Security-Team, 10Security: Create #security-bug tag to specifically classify security bugs within phabricator - https://phabricator.wikimedia.org/T342449 (10Aklapper) Hmm, I'd say that's ideally what the "Security issue" task subtype is for, or a column on a workboard. But I see how this... [16:46:11] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T340246 (10dancy) 05Openβ†’03Resolved [17:07:28] Hi all, beta seems to be terribly broken: T342452, looks like it's inheriting all settings from production [17:07:29] T342452: Beta is inheriting prod value for $wgCampaignEventsProgramsAndEventsDashboardInstance - https://phabricator.wikimedia.org/T342452 [17:08:45] I'm still looking into this to better understand what's broken, but if someone has any ideas, I'd be happy to hear them [17:09:42] (03CR) 10Ahmon Dancy: "Looking forward to this!" [releng/phatality] - 10https://gerrit.wikimedia.org/r/940265 (https://phabricator.wikimedia.org/T342400) (owner: 10Dduvall) [17:30:12] (03CR) 10Subramanya Sastry: [C: 03+2] "Lets get this in." [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/940262 (owner: 10Subramanya Sastry) [17:31:04] (03Merged) 10jenkins-bot: Few more fixes based on visual diff testing [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/940262 (owner: 10Subramanya Sastry) [19:00:36] 10Beta-Cluster-Infrastructure, 10Scap, 10Patch-For-Review, 10Technical-Debt: Reconcile "Scap: scap_source correct gid" Beta Cluster hot patch - https://phabricator.wikimedia.org/T342320 (10dancy) @Krinkle: I think we should merge https://gerrit.wikimedia.org/r/361796 as-is. I update the commit message tod... [20:05:00] 10Project-Admins, 10Security-Team, 10Security: Create #production-risk-assessment Phabricator project/tag - https://phabricator.wikimedia.org/T342466 (10sbassett) [20:07:11] 10Gitlab-Application-Security-Pipeline, 10Security: Design AppSec Pipeline metrics approach - https://phabricator.wikimedia.org/T342467 (10sbassett) [20:07:56] 10Gitlab-Application-Security-Pipeline, 10Security: Design AppSec Pipeline metrics approach - https://phabricator.wikimedia.org/T342467 (10sbassett) [20:07:58] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10Security: Application Security Pipeline Components for Gitlab - Phase 2 Work - https://phabricator.wikimedia.org/T342177 (10sbassett) [20:08:26] 10Gitlab-Application-Security-Pipeline: Craft more mediawiki-specific and php semgrep rule sets - https://phabricator.wikimedia.org/T342468 (10sbassett) [20:09:05] 10Gitlab-Application-Security-Pipeline: Craft more mediawiki-specific and php semgrep rule sets - https://phabricator.wikimedia.org/T342468 (10sbassett) [20:09:08] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10Security: Application Security Pipeline Components for Gitlab - Phase 2 Work - https://phabricator.wikimedia.org/T342177 (10sbassett) [20:10:31] 10Gitlab-Application-Security-Pipeline: Establish a more specific policy/best practices around security include failures - https://phabricator.wikimedia.org/T342469 (10sbassett) [20:11:08] 10Gitlab-Application-Security-Pipeline: Establish a more specific policy/best practices around security include failures - https://phabricator.wikimedia.org/T342469 (10sbassett) [20:11:11] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10Security: Application Security Pipeline Components for Gitlab - Phase 2 Work - https://phabricator.wikimedia.org/T342177 (10sbassett) [21:35:33] (03CR) 10Ahmon Dancy: Link to git blames for each of the stacktrace frames (031 comment) [releng/phatality] - 10https://gerrit.wikimedia.org/r/940265 (https://phabricator.wikimedia.org/T342400) (owner: 10Dduvall) [22:40:16] (03CR) 10Cwhite: "LGTM overall!" [releng/phatality] - 10https://gerrit.wikimedia.org/r/940265 (https://phabricator.wikimedia.org/T342400) (owner: 10Dduvall) [22:54:59] jbond, thcipriani: I find that using tigΒΉ significantly simplifies git and helps me understand it better at the same time. [1] https://jonas.github.io/tig/ [22:58:05] 10Phabricator: Herald rule H260 should not remove CommTech Kanban project - https://phabricator.wikimedia.org/T342414 (10MusikAnimal) I'm confused what the problem is, but if I caused it, allow me to apologize! > Hi, I edited H260 and added "Project tags include none of: Community-Tech (CommTech-Kanban)" as a c... [23:13:56] twentyafterfour: quality footnote usage