[04:16:20] 06Release-Engineering-Team (Doing 😎), 06Abstract Wikipedia team, 06DBA, 10Wikifunctions, and 2 others: @Krazy69guero - https://phabricator.wikimedia.org/T406185#11244632 (10Krazy69guero.og) 05Resolved→03In progress [05:15:30] 06Release-Engineering-Team (Doing 😎), 06Abstract Wikipedia team, 06DBA, 10Wikifunctions, and 2 others: Wikimedia\Rdbms\DBQueryError: Error 1146: Table '[wiktionary].wikifunctionsclient_usage' doesn't exist - https://phabricator.wikimedia.org/T406185#11244673 (10A_smart_kitten) 05In progress→03Resolv... [05:38:06] 06Release-Engineering-Team, 06collaboration-services, 07Security: ProbeDown - https://phabricator.wikimedia.org/T406403#11244683 (10ABran-WMF) p:05Triage→03High a:03ABran-WMF After a new [[ https://grafana.wikimedia.org/goto/_C8CywqHg?orgId=1 | attack pattern ]] I'll [[ https://mod-qos.sourceforge.net... [05:39:55] 06Release-Engineering-Team, 06collaboration-services, 07Security: ProbeDown - gerrit1003:443 - https://phabricator.wikimedia.org/T406403#11244693 (10ABran-WMF) [07:44:12] 10Phabricator (Upstream), 07Upstream: Record in task histories when subscribers are added during a task's creation - https://phabricator.wikimedia.org/T405245#11244911 (10Aklapper) I personally prefer not to change the current behavior as I have not yet come across another similar request, and as the task... [08:00:53] 10Phabricator (Upstream), 07Upstream: Record in task histories when subscribers are added during a task's creation - https://phabricator.wikimedia.org/T405245#11244940 (10Aklapper) Oh wait, reading the task description again: > or auto-subscribed because of a project I am a part of. Could you elaborate w... [08:29:08] 06Project-Admins, 07Chinese-Sites: Create project tag for Sinitic-Sites - https://phabricator.wikimedia.org/T406456 (10Winston_Sung) 03NEW [09:12:34] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-maps-master02 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [09:23:13] 10GitLab, 10bacula, 06collaboration-services, 10Data-Persistence-Backup, 13Patch-For-Review: Split repository backups (gerrit and gitlab) into its own dedicated storage daemons - https://phabricator.wikimedia.org/T403946#11245334 (10Jelto) >>! In T403946#11241419, @jcrespo wrote: > @Jelto @Dzahn The back... [09:26:48] 06Project-Admins, 07Chinese-Sites: Create project tag for Sinitic-Sites - https://phabricator.wikimedia.org/T406456#11245374 (10Aklapper) Hi, I am usually reluctant to add more such language tags as I am always worried about people spending time on categorizing for the sake of categorizing. This is a bit simil... [09:30:25] 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review, 07Security: ProbeDown - gerrit1003:443 - https://phabricator.wikimedia.org/T406403#11245396 (10Jelto) [09:38:05] 10GitLab, 10bacula, 06collaboration-services, 10Data-Persistence-Backup, 13Patch-For-Review: Split repository backups (gerrit and gitlab) into its own dedicated storage daemons - https://phabricator.wikimedia.org/T403946#11245418 (10jcrespo) No problem with scheduling. [10:09:44] (03CR) 10Hashar: [C:03+2] Zuul: add VisualEditor dependency explicitly [integration/config] - 10https://gerrit.wikimedia.org/r/1193486 (https://phabricator.wikimedia.org/T406376) (owner: 10Hashar) [10:11:29] (03Merged) 10jenkins-bot: Zuul: add VisualEditor dependency explicitly [integration/config] - 10https://gerrit.wikimedia.org/r/1193486 (https://phabricator.wikimedia.org/T406376) (owner: 10Hashar) [10:12:55] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/1193486 , add VisualEditor as a dependency for: BlueSpiceProDistributionConnector, Chart, CognitiveProcessDesigner, PDFCreator, Sanctions and SimpleTasks [10:12:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:17:09] (03CR) 10Finchgold: [C:03+1] Zuul: add VisualEditor dependency explicitly [integration/config] - 10https://gerrit.wikimedia.org/r/1193486 (https://phabricator.wikimedia.org/T406376) (owner: 10Hashar) [10:20:59] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#11245578 (10Ladsgroup) >>! In T706#11236647, @DMburugu wrote: > Hi, > > Please add me to ACL Project Admin > > I'd like to become a member so that I can crea... [11:29:46] 06Project-Admins, 07Chinese-Sites: Create project tag for Sinitic-Sites - https://phabricator.wikimedia.org/T406456#11245749 (10Winston_Sung) [11:31:33] Project mediawiki-core-doxygen build #14147: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/14147/ [11:47:40] 06Project-Admins, 07Chinese-Sites, 07Community-consensus-needed: Create project tag for Sinitic-Sites - https://phabricator.wikimedia.org/T406456#11245785 (10Winston_Sung) //**Problem**// Hard to find/monitor non-Mandarin Sinitic languages wiki tasks //**Considerations**// Not all non-Mandarin Sinitic langu... [11:48:57] 06Project-Admins, 07Chinese-Sites, 07Community-consensus-needed: Create project tag for Sinitic-Sites - https://phabricator.wikimedia.org/T406456#11245793 (10Winston_Sung) [11:49:56] 06Project-Admins, 07Chinese-Sites, 07Community-consensus-needed: Create project tag for Sinitic-Sites - https://phabricator.wikimedia.org/T406456#11245796 (10Winston_Sung) Solution 2 seems to be better. [12:12:58] PROBLEM - jenkins_service_running on contint1002 is CRITICAL: PROCS CRITICAL: 0 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [12:20:32] 10Phabricator, 06Release-Engineering-Team (Seen), 07Developer Productivity, 07SecTeam-Processed: Some very specific Maniphest search queries by RelEng, Sec Team and WMCS are global and shown for all users - https://phabricator.wikimedia.org/T214579#11245877 (10fgiunchedi) I'm cc'ing #sre-onfire (and #o... [12:27:27] self note for after the Gerrit upgrade: [12:27:27] ssh contint.wikimedia.org sh -c 'sudo systemctl unmask jenkins && sudo systemctl start jenkins' [12:27:27] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1193408 would need a new +2 [12:27:27] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1193824 will need a recheck [12:38:58] RECOVERY - jenkins_service_running on contint1002 is OK: PROCS OK: 1 process with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [12:46:40] 06Project-Admins, 07Chinese-Sites, 07Community-consensus-needed: Create project tag for Sinitic-Sites - https://phabricator.wikimedia.org/T406456#11245900 (10Aklapper) My point was that the problem and the resources will not change magically by creating more project tags. It needs people to constantly triage... [12:52:07] 06Project-Admins, 07Chinese-Sites, 07Community-consensus-needed: Create project tag for Sinitic-Sites - https://phabricator.wikimedia.org/T406456#11245917 (10Winston_Sung) Yeah. I know that. We do have the resources to maintain the board, and the reason to have this tag is that we can more easily have a ove... [12:53:04] 06Project-Admins, 07Chinese-Sites: Create project tag for Sinitic-Sites - https://phabricator.wikimedia.org/T406456#11245929 (10Winston_Sung) [12:57:31] Yippee, build fixed! [12:57:31] Project beta-scap-sync-world build #226908: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/226908/ [13:27:13] Hi! I have a patch ready for deployment: [13:27:13] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Scribunto/+/1191861 [13:27:14] It updates namespace names for diqwiki (Bug:T328207) [13:27:15] T328207: Change Namespace Aliases on diq.wikipedia - https://phabricator.wikimedia.org/T328207 [13:28:42] cappybaraa: this is the wrong channel for that, please join #wikimedia-operations :) [13:29:51] Yippee, build fixed! [13:29:51] Project mediawiki-core-doxygen build #14148: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/14148/ [13:35:33] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T405678#11246080 (10jnuche) [14:04:49] 10GitLab, 06collaboration-services, 10VPS-project-devtools: Puppet failure on gitlab-1002.devtools.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T406234#11246223 (10Jelto) p:05Triage→03Medium a:03Jelto [14:59:03] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T388025) [14:59:09] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T388025) [15:06:03] 10Continuous-Integration-Config, 06Experimentation Lab (Experiment Platform Sprint 13), 07OKR-Work: Configure MetricsPlatform to depend on GrowthExperiments in CI - https://phabricator.wikimedia.org/T405715#11246509 (10Milimetric) 05Open→03Resolved [15:34:48] 10Gerrit, 06collaboration-services: Gerrit switchover between secondary instances - https://phabricator.wikimedia.org/T406334#11246628 (10ABran-WMF) [15:42:06] 10Gerrit, 06collaboration-services: Gerrit switchover between secondary instances - https://phabricator.wikimedia.org/T406334#11246673 (10LSobanski) In scope: decide and document whether we keep the second replica around permanently. [15:42:15] 10Gerrit, 06collaboration-services: Gerrit switchover between secondary instances - https://phabricator.wikimedia.org/T406334#11246675 (10LSobanski) p:05Triage→03Medium [15:43:31] 10Gerrit, 06collaboration-services: gerrit: config tweaks - https://phabricator.wikimedia.org/T406333#11246679 (10LSobanski) p:05Triage→03Low [15:45:50] 10Gerrit: sync-with-gerrit causes issues when run on macos - https://phabricator.wikimedia.org/T406503 (10Reedy) 03NEW [16:31:26] 10Continuous-Integration-Config, 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team: Add CentralAuth to gated extensions - https://phabricator.wikimedia.org/T333541#11246933 (10Jdforrester-WMF) >>! In T333541#11243876, @Tgr wrote: > Isn't that the same thing as adding it to the gated extensions? O... [16:40:11] 10Gerrit: sync-with-gerrit causes issues when run on macos - https://phabricator.wikimedia.org/T406503#11246972 (10hashar) > Because we have LDAPGroups and LdapGroups and HoverGallery and Hovergallery... I don't know from from where you get `LdapGroups` or `Hovergallery`: * the `.gitmodules` file has `LDAPGrou... [16:46:22] 10Gerrit: sync-with-gerrit causes issues when run on macos - https://phabricator.wikimedia.org/T406503#11246988 (10Reedy) {F66734062 size=full} {F66734063 size=full} [16:47:48] 10Gerrit: sync-with-gerrit causes issues when run on macos - https://phabricator.wikimedia.org/T406503#11246993 (10Reedy) ` reedy@Sams-Mac-mini extensions % ./sync-with-gerrit.py INFO:root:Fetching projects from gerrit (prefix: mediawiki/extensions/) INFO:root:Checking modules INFO:root:mediawiki/extensions/APC... [17:16:29] 10Gerrit: sync-with-gerrit causes issues when run on macos - https://phabricator.wikimedia.org/T406503#11247113 (10hashar) With the screenshots I know understand what you meant by //Because we have LDAPGroups and LdapGroups and HoverGallery and Hovergallery...//. There are indeed ones that are read-only: ` lang=... [17:19:49] 10Gerrit: sync-with-gerrit causes issues when run on macos - https://phabricator.wikimedia.org/T406503#11247118 (10hashar) I think the repos were originally created with the wrong name and were kept to keep a copy of the changes that were created. If the read-only repositories share the same history, I think I... [18:22:30] (03open) 10dancy: deploy: Add blast-databases utility script [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/164 [18:22:32] (03update) 10dancy: deploy: Add blast-databases utility script [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/164 [18:23:29] (03merge) 10dancy: deploy: Add blast-databases utility script [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/164 [19:21:54] huh, weird. The phpunit-coverage-patch job seems to be failing due to wrong version of parsoid installed [19:24:22] 06Release-Engineering-Team (Doing 😎), 13Patch-For-Review: Add a staging kubernetes cluster to train-dev - https://phabricator.wikimedia.org/T402350#11247523 (10dancy) 05Open→03Resolved [19:30:25] Oh, i guess i just needed to rebase [19:36:39] 06Release-Engineering-Team (Radar), 10Cloud-VPS (Quota-requests): Grant gitlab-runners-staging access to fast-iops volume type and a 4xiops instance flavor - https://phabricator.wikimedia.org/T406271#11247592 (10dcaro) +1 [21:32:38] 10Phabricator: "Trying to access array offset on value of type null" in server pull logs for Phabricator Diffusion repository - https://phabricator.wikimedia.org/T406528 (10Aklapper) 03NEW p:05Triage→03Low [21:37:06] 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎): "Trying to access array offset on value of type null" in server pull logs for Phabricator Diffusion repository - https://phabricator.wikimedia.org/T406528#11247912 (10Aklapper) a:03Aklapper https://gitlab.wikimedia.org/repos/phabricator/... [21:37:36] 10Diffusion, 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎): "Trying to access array offset on value of type null" in server pull logs for Phabricator Diffusion repository - https://phabricator.wikimedia.org/T406528#11247915 (10Aklapper) [21:53:32] 10GitLab (CI & Job Runners), 06serviceops: failed to push docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.10.5-py3.11-2025-10-03-192132-3003d4328df66a0086a350fdd2ba1dbd80a235c5: unknown: blob upload invalid - https://phabricator.wikimedia.org/T406392#11247933 (10Scott_French) @ama... [22:23:27] 10GitLab (Project Migration), 06Community-Tech, 10CopyPatrol: Migrate CopyPatrol repo from GitHub to GitLab - https://phabricator.wikimedia.org/T395401#11247984 (10bd808) >>! In T395401#11232164, @Samwilson wrote: > @brennen @bd808 Is there a preference for VPS tools to have their repos under `/cloudvps-repo... [22:26:40] 10GitLab (Project Migration), 06Community-Tech, 10CopyPatrol: Migrate CopyPatrol repo from GitHub to GitLab - https://phabricator.wikimedia.org/T395401#11247996 (10bd808) >>! In T395401#11232222, @JJMC89 wrote: > If it were up to me, the repo wouldn't be migrated to GitLab since GitLab is an inferior experie... [22:43:30] !log Unblock 185.73.168.0/22 (T406091) [22:43:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:57:33] !log Unblock 159.196.168.0/22 (T406076) [22:57:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:11:55] !log Unblock 37.120.0.0/17 (T405451) [23:11:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:43:16] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T405547#11248182 (10bd808) 05Open→03Invalid This was probably a stale cherry-pick that someone cleaned up witho... [23:46:32] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-puppetserver-1 on project deployment-prep - https://phabricator.wikimedia.org/T405788#11248203 (10bd808) 05Open→03Invalid `lang=shell-session bd808@deployment-puppetserver-1.deployment-prep.eqiad1:~$ sudo -i puppet agent -tv... [23:51:06] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-puppetserver-1 on project deployment-prep - https://phabricator.wikimedia.org/T405788#11248211 (10Dzahn) likely related to https://gerrit.wikimedia.org/r/c/operations/puppet/+/1190300 / T399891 [23:52:09] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-maps-master02 in project deployment-prep - https://phabricator.wikimedia.org/T405888#11248216 (10bd808) `lang=shell-session,counterexample bd808@deployment-maps-master02:~$ sudo -i puppet agent -tv Info: Using environment 'produc... [23:53:21] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-maps-master02 in project deployment-prep - https://phabricator.wikimedia.org/T405888#11248218 (10bd808) Caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1191680