[01:47:32] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-echostore02 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [06:33:16] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Cloud-Services, 06collaboration-services: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479 (10hashar) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https:/... [06:34:46] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 06collaboration-services, 10Data-Services: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12063721 (10hashar) [06:35:16] 10Continuous-Integration-Infrastructure (Zuul upgrade): upload-logs-swift role fails due to ModuleNotFoundError: No module named 'requestsexceptions' - https://phabricator.wikimedia.org/T430458#12063723 (10hashar) 05Openβ†’03Resolved Fixed after I have synced our integration/zuul-jobs fork with upstream.... [06:41:39] (03CR) 10Hashar: [C:03+2] "> PageImages is a soft runtime dependency (guarded by ExtensionRegistry)," [integration/config] - 10https://gerrit.wikimedia.org/r/1305998 (owner: 10Eric Gardner) [06:43:22] (03Merged) 10jenkins-bot: Add PageImages to MultimediaViewer's phan (and test) dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1305998 (owner: 10Eric Gardner) [06:45:17] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/Chart] Add Scribunto dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1305909 (https://phabricator.wikimedia.org/T430324) (owner: 10Aude) [06:47:01] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Chart] Add Scribunto dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1305909 (https://phabricator.wikimedia.org/T430324) (owner: 10Aude) [07:07:35] 06Project-Admins: Create project tag for Open Data Ivoir - https://phabricator.wikimedia.org/T430467#12063771 (10Aklapper) 05Openβ†’03Declined Hi, I am not sure why you reopened this. Please see T430467#12063510 and create the project tag - thank you. :) Tu peux creer le Phab projet toi-meme; il y a pas d... [07:10:45] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Rename "Nekomoto" Phabricator account to "Vulpomoto" - https://phabricator.wikimedia.org/T430472#12063781 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Done. [07:30:34] 06Project-Admins, 10Reconciliation: Reconciliation project - Subproject to manage reconciliation service - https://phabricator.wikimedia.org/T426541#12063864 (10Aklapper) 05Stalledβ†’03Open If you are really sure that [all current members of Reconciliation](https://phabricator.wikimedia.org/project/members/5... [07:42:15] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 06collaboration-services, 10Data-Services: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12063909 (10hashar) From zuul1002.eqiad.wmnet I can reproduce using curl. I have tried two app... [08:16:38] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 06collaboration-services, 10Data-Services: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12064055 (10hashar) If I try https://openstack.eqiad1.wikimediacloud.org:25000/ from contint10... [08:19:33] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T430421#12064076 (10ABran-WMF) [08:19:41] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T430421#12064081 (10ABran-WMF) [08:19:46] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T430421#12064086 (10ABran-WMF) [08:20:50] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 06collaboration-services, 10Data-Services: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12064092 (10hashar) Using webproxy (rather than url-downloader), the tunneling is rejected: `... [08:20:52] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: allow new zuul executor VMs in prod to talk to cloud VPS - https://phabricator.wikimedia.org/T394838#12064095 (10hashar) [08:20:57] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 06collaboration-services, 10Data-Services: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12064096 (10hashar) [08:27:13] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit2003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://phabricator.wikimedia.org/T430421#12064124 (10ABran-WMF) [08:27:16] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit2003 - https://phabricator.wikimedia.org/T430421#12064125 (10LSobanski) [08:29:44] 10Continuous-Integration-Infrastructure (Zuul upgrade): Artifact storage for updated Zuul CI - https://phabricator.wikimedia.org/T397385#12064130 (10hashar) [08:29:51] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 06collaboration-services, 10Data-Services: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12064131 (10hashar) [08:43:37] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 06collaboration-services, 10Data-Services, 13Patch-For-Review: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12064239 (10hashar) a:03hashar [08:51:43] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Gerrit backups are growing - https://phabricator.wikimedia.org/T411583#12064312 (10ABran-WMF) I guess before reducing granularity, we could try and exclude [[ https://gerrit.wikimedia.org/r/1306164 | caches ]] and [[ https://gerrit.wikimedia.org/r/13061... [09:10:55] 10Phabricator, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 13Patch-For-Review: Remove "Deadline" task subtype - https://phabricator.wikimedia.org/T335395#12064387 (10Aklapper) [09:25:08] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit2003 - https://phabricator.wikimedia.org/T430421#12064407 (10ABran-WMF) a:03ABran-WMF [09:28:29] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit2003 - https://phabricator.wikimedia.org/T430421#12064417 (10ABran-WMF) Gerrit was [[ https://grafana-rw.wikimedia.org/d/nX8li17Sk/gerrit-overview-sre-collab?forceLogin=true&from=2026-06-28T01:41:45.088Z&orgId=1&refresh=1m&timezone=utc&to=... [09:31:56] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit2003 - https://phabricator.wikimedia.org/T430421#12064423 (10LSobanski) p:05Triageβ†’03High [09:32:10] !log gerrit: deleted repository phabricator/extensions/BurnDownCharts , created in July 2014, had no commit/changes [09:32:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:36:28] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit2003 - https://phabricator.wikimedia.org/T430421#12064436 (10ABran-WMF) On [[ https://w.wiki/RtWw | Turnilo, JA4H ]] `ge20cr200000_1ad82be8b01f_b54fc47906f4` seem to be the top hitter Even more visible, [[ https://w.wiki/RtX3 | splitting... [09:39:03] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Archive https://gerrit.wikimedia.org/g/phabricator/extensions/security - https://phabricator.wikimedia.org/T430502 (10Aklapper) 03NEW p:05Triageβ†’03Low [09:45:07] 10Phabricator, 06Release-Engineering-Team (Doing 😎), 10Wikimedia-Phabricator-Extensions: Archive https://gerrit.wikimedia.org/g/phabricator/extensions/security - https://phabricator.wikimedia.org/T430502#12064505 (10Aklapper) 05Openβ†’03Resolved * Changed repository Status from Active to Read-Only * Ad... [09:52:38] 10Gerrit, 10Phabricator Antivandalism Extension, 06Release-Engineering-Team (Priority Backlog πŸ“₯): Delete the "master" branch of the Phabricator Antivandalism extension repo - https://phabricator.wikimedia.org/T413890#12064565 (10Aklapper) TIL from hashar that the Gerrit UI allows pointing HEAD to a different... [09:53:48] 10Gerrit, 10Phabricator Translations Extension, 06Release-Engineering-Team (Priority Backlog πŸ“₯): Delete the "master" branch from the Phabricator Translations repo - https://phabricator.wikimedia.org/T412641#12064571 (10Aklapper) TIL from hashar that I can do that in the Gerrit UI via https://gerrit.wikimedia... [10:42:32] 10Phabricator, 10CAS-SSO, 06Infrastructure-Foundations, 06SRE: Add logout.d script for Phabricator - https://phabricator.wikimedia.org/T286904#12064836 (10MoritzMuehlenhoff) >>! In T286904#12041211, @Aklapper wrote: > Hi, I myself am not sure what to add apart from T286904#11091482. Please elaborate if any... [10:44:06] 10MediaWiki-Releasing, 07Security: Write and send pre-release announcements for MediaWiki 1.43.7/1.44.4/1.45.2 - https://phabricator.wikimedia.org/T411387#12064847 (10Reedy) [10:46:42] 10MediaWiki-Releasing, 07Security: Write and send pre-release announcements for MediaWiki 1.43.7/1.44.4/1.45.2 - https://phabricator.wikimedia.org/T411387#12064857 (10Reedy) [10:47:52] hashar: bonjour! whenever you have a moment, we're ready for https://gerrit.wikimedia.org/r/c/integration/config/+/1305645 to be merged :) [11:11:47] eugh castor [11:20:03] maintenance-disconnect-full-disks build 819816 integration-agent-docker-1082 (/: 28%, /srv: 95%, /var/lib/docker: 24%): OFFLINE due to disk space [11:25:03] maintenance-disconnect-full-disks build 819817 integration-agent-docker-1082 (/: 28%, /srv: 88%, /var/lib/docker: 24%): RECOVERY disk space OK [11:30:36] Project mediawiki-core-doxygen build #21328: 04FAILURE in 15 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21328/ [11:41:36] (03open) 10btullis: Add data-engineering/refinery to the trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/179 (https://phabricator.wikimedia.org/T383417 https://phabricator.wikimedia.org/T429316) [11:49:21] Yippee, build fixed! [11:49:21] Project mediawiki-core-doxygen build #21329: 09FIXED in 18 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21329/ [12:18:59] 06Project-Admins: Create project tag for Open Data Ivoir - https://phabricator.wikimedia.org/T430467#12065103 (10poro26) >>! Dans T430467#12063771, @Aklapper a Γ©crit : > Hi, I am not sure why you reopened this. Please see T430467#12063510 and create the project tag - thank you. :) > > Tu peux creer le Phab... [13:25:12] jakob_WMDE: Hi! I was on some focus work :) [13:26:00] (03CR) 10Hashar: [C:03+2] "Lets fly! :) Once deployed you will want to recheck a change on WikibaseLexeme and verify the api testing job is working!" [integration/config] - 10https://gerrit.wikimedia.org/r/1305645 (https://phabricator.wikimedia.org/T430038) (owner: 10Kim.pham) [13:38:51] (03Merged) 10jenkins-bot: Zuul: Add extension-apitests to WikibaseLexeme extension [integration/config] - 10https://gerrit.wikimedia.org/r/1305645 (https://phabricator.wikimedia.org/T430038) (owner: 10Kim.pham) [13:40:09] (03CR) 10Hashar: [C:03+2] "I have deployed the change" [integration/config] - 10https://gerrit.wikimedia.org/r/1305645 (https://phabricator.wikimedia.org/T430038) (owner: 10Kim.pham) [13:41:37] hashar: thank you! [13:47:12] 06Release-Engineering-Team, 06collaboration-services: install a service on phab1005 - https://phabricator.wikimedia.org/T377889#12065689 (10Arnoldokoth) a:03Arnoldokoth [13:56:58] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-devtools, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#12065795 (10Arnoldokoth) a:03Arnoldokoth [14:07:24] 06Release-Engineering-Team, 10Scap: https://versions.toolforge.org/ shows group2 as partially-deployed - https://phabricator.wikimedia.org/T430159#12065917 (10Jdforrester-WMF) 05Openβ†’03Resolved a:03Krinkle Let's provisionally claim this is fixed, then. [14:27:48] (03CR) 10Eric Gardner: "Great, thanks! I'll post a revert patch if this ends up being a dead end." [integration/config] - 10https://gerrit.wikimedia.org/r/1305998 (owner: 10Eric Gardner) [14:31:08] (03PS1) 10JMeybohm: helm-linter: Update istioctl package to 1.29.4-2 [integration/config] - 10https://gerrit.wikimedia.org/r/1306321 (https://phabricator.wikimedia.org/T427401) [14:31:11] (03PS1) 10JMeybohm: helm-linter: Use version 0.8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1306322 (https://phabricator.wikimedia.org/T427401) [14:32:38] hashar: ^ would you be so kind if you can find 5min? You may recheck https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1306280/2 after which should fail in ~30min :D [14:33:04] jayme: I'll build it ;) [14:33:10] <3 [14:34:09] (03PS2) 10Hashar: helm-linter: Update istioctl package to 1.29.4-2 [integration/config] - 10https://gerrit.wikimedia.org/r/1306321 (https://phabricator.wikimedia.org/T427401) (owner: 10JMeybohm) [14:34:39] (03CR) 10Hashar: [C:03+2] "I have changed the changelog entry to have the Phabricator task on the same line and between parentheses, for consistency with other entri" [integration/config] - 10https://gerrit.wikimedia.org/r/1306321 (https://phabricator.wikimedia.org/T427401) (owner: 10JMeybohm) [14:38:24] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-echostore02 in project deployment-prep - https://phabricator.wikimedia.org/T430384#12066161 (10dancy) ` dancy@deployment-echostore02:/$ sudo run-puppet-agent Error: No space left on device @ fptr_finalize_flush - /var/lib... [14:40:51] (03Merged) 10jenkins-bot: helm-linter: Update istioctl package to 1.29.4-2 [integration/config] - 10https://gerrit.wikimedia.org/r/1306321 (https://phabricator.wikimedia.org/T427401) (owner: 10JMeybohm) [14:41:34] (03PS2) 10JMeybohm: helm-linter: Use version 0.8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1306322 (https://phabricator.wikimedia.org/T427401) [14:42:33] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-echostore02 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [14:43:54] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-echostore02 in project deployment-prep - https://phabricator.wikimedia.org/T430384#12066196 (10dancy) 05Openβ†’03Resolved p:05Triageβ†’03Low a:03dancy [14:44:42] 06Project-Admins: Create project tag for the MediaWiki Media Platform Team (new team within MediaWiki, starting FY26/27) - https://phabricator.wikimedia.org/T430544 (10SLopes-WMF) 03NEW [14:44:50] (03merge) 10dancy: Add data-engineering/refinery to the trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/179 (https://phabricator.wikimedia.org/T383417 https://phabricator.wikimedia.org/T429316) (owner: 10btullis) [14:47:15] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 10Puppet-Core, 06SRE: [DRAFT][RfC] Deployment of python applications in production - https://phabricator.wikimedia.org/T180023#12066258 (10LSobanski) 05Openβ†’03Resolved a:03LSobanski Resolving. @Joe please reopen if this is still a... [14:49:54] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Remove buildkit helper image docker/dockerfile-copy from build pipeline - https://phabricator.wikimedia.org/T321316#12066284 (10Jelto) [14:49:59] (03PS3) 10Hashar: jjb: [helm-lint] Use version 0.8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1306322 (https://phabricator.wikimedia.org/T427401) (owner: 10JMeybohm) [14:50:58] (03CR) 10Hashar: [C:03+2] "I slightly rephrased the commit message to match previous entry and to add the image changelog entry. The helm-lint job has been reconfig" [integration/config] - 10https://gerrit.wikimedia.org/r/1306322 (https://phabricator.wikimedia.org/T427401) (owner: 10JMeybohm) [14:51:01] jayme: done! [14:51:19] hashar: thanks! [14:52:46] (03Merged) 10jenkins-bot: jjb: [helm-lint] Use version 0.8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1306322 (https://phabricator.wikimedia.org/T427401) (owner: 10JMeybohm) [14:56:32] 10Beta-Cluster-Infrastructure, 06Growth-Team, 10GrowthExperiments-NewcomerTasks, 06Discovery-Search (2026.06.01 - 2026.07.03): [beta-cluster] Fetching task suggestions failed: cirrussearch-backend-error - https://phabricator.wikimedia.org/T427196#12066350 (10dcausse) @bking thanks! I confirm that I can sea... [15:35:50] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Gerrit backups are growing - https://phabricator.wikimedia.org/T411583#12066679 (10ABran-WMF) @jcrespo while we [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1306166/comments/5b48b203_440fd774 | sort out ]] if we want to backup indexes or not,... [16:02:32] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [16:02:43] 10Beta-Cluster-Infrastructure: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://phabricator.wikimedia.org/T430562 (10wmcs-alerts) 03NEW [16:04:55] 10Continuous-Integration-Infrastructure (Zuul upgrade): Zuul web API calls are cached - https://phabricator.wikimedia.org/T430462#12066888 (10Dzahn) We can easily turn off all caching if you want that. [16:08:15] 10Continuous-Integration-Infrastructure (Zuul upgrade): Zuul web API calls are cached - https://phabricator.wikimedia.org/T430462#12066902 (10Dzahn) But is that really the goal here? Or is this an action that only happens often at the beginning and as long as you know what's happening and you just have to wait... [16:09:41] 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Zuul web API calls are cached - https://phabricator.wikimedia.org/T430462#12066913 (10Dzahn) As it currently stands the ticket is just pointing out a fact. Is it a call to action though? [16:10:58] 06Project-Admins: Create project tag for the MediaWiki Media Platform Team (new team within MediaWiki, starting FY26/27) - https://phabricator.wikimedia.org/T430544#12066918 (10Aklapper) Hi, where to find out what the "PP3 Responsible Reuse objective" is? Is there a public team wiki page to point to which provid... [16:12:35] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Eliminate sudo rm -rf /var/lib/puppet/ssl step in new deployment-prep WMCS project (and others) - https://phabricator.wikimedia.org/T429413#12066921 (10dancy) >>! In T429413#12033569, @Andrew wrote: >>>! In T429413#12027023, @dancy wrote: >> @Andrew Do you ant... [16:12:36] 10GitLab: Gitlab replica-b being indexed by Duck Duck Go - https://phabricator.wikimedia.org/T430563 (10TheDJ) 03NEW [16:13:26] 10GitLab: Gitlab replica-b being indexed by Duck Duck Go - https://phabricator.wikimedia.org/T430563#12066949 (10Dzahn) Does it cause a problem? [16:14:38] 10GitLab: Gitlab replica-b being indexed by Duck Duck Go - https://phabricator.wikimedia.org/T430563#12066958 (10Dzahn) Oh, well, the part that we see "HTTP 502: Waiting for GitLab to boot" on replica-b is unexpected, at least to me. But seems entirely separate from the indexing question. [16:14:54] 10GitLab, 06collaboration-services: Gitlab replica-b being indexed by Duck Duck Go - https://phabricator.wikimedia.org/T430563#12066959 (10Dzahn) [16:15:58] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Eliminate sudo rm -rf /var/lib/puppet/ssl step in new deployment-prep WMCS project (and others) - https://phabricator.wikimedia.org/T429413#12066960 (10dancy) TODO: Update https://wikitech.wikimedia.org/wiki/Help:Project_puppetserver after https://gerrit.wikim... [16:35:31] 10GitLab, 06collaboration-services: Gitlab replica-b being indexed by Duck Duck Go - https://phabricator.wikimedia.org/T430563#12067044 (10TheDJ) well if this is the canonical url that should be indexed, I'd expect it to work indeed :) if it's not the canonical url, it should score lower than the canonical url... [16:35:42] 10GitLab, 06collaboration-services: Gitlab replica-b being indexed by Duck Duck Go - https://phabricator.wikimedia.org/T430563#12067045 (10Dzahn) Now it's back up. So you can access https://gitlab-replica-b.wikimedia.org/hartman/vtt-vivid just fine. [16:36:29] 10GitLab, 06collaboration-services: Gitlab replica-b being indexed by Duck Duck Go - https://phabricator.wikimedia.org/T430563#12067052 (10Dzahn) ACK, let's get more opinions from others on the team. [16:45:42] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Gerrit backups are growing - https://phabricator.wikimedia.org/T411583#12067103 (10jcrespo) Let me check tomorrow, as it takes sometimes a few hours for the change to take effect. [16:46:12] 10GitLab, 06collaboration-services: Gitlab replica-b being indexed by Duck Duck Go - https://phabricator.wikimedia.org/T430563#12067105 (10RhinosF1) >>! In T430563#12066949, @Dzahn wrote: > Does it cause a problem? It indexing the wrong site is a problem, yes. It means those visting from search engines will g... [16:47:34] 10GitLab, 06collaboration-services: Gitlab replica-b being indexed by Duck Duck Go - https://phabricator.wikimedia.org/T430563#12067107 (10RhinosF1) I see a canonical tag on gitlab-replica-b, it should probably include gitlab.wikimedia.org instead of the replica domain [16:48:46] 10Beta-Cluster-Infrastructure: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://phabricator.wikimedia.org/T430562#12067110 (10bd808) 05Openβ†’03Resolved a:03bd808 `lang=shell-session bd808@deployment-puppetserver-1.deployment-prep.eqiad1:~$ sudo /usr/lo... [16:51:10] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Gerrit backups are growing - https://phabricator.wikimedia.org/T411583#12067125 (10jcrespo) I don't have yet the data for you, but I have an additional data point: **Full backups take 120-150GB**, it sounds weird that we need a backup of a third of that... [16:52:32] RESOLVED: PuppetStaleCertificates: Found non-revoked Puppet certificates for 3 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [17:58:00] (03update) 10jforrester: build: Upgrade eslint-config-wikimedia from 0.26.0 to 0.28.2 and qunit from ^2.20.0 to ^2.22.0 [repos/ci-tools/browserslist-config-wikimedia] - 10https://gitlab.wikimedia.org/repos/ci-tools/browserslist-config-wikimedia/-/merge_requests/14 [17:58:33] (03update) 10jforrester: build: Upgrade eslint-config-wikimedia from 0.26.0 to 0.32.4 and qunit from ^2.20.0 to ^2.26.0 [repos/ci-tools/browserslist-config-wikimedia] - 10https://gitlab.wikimedia.org/repos/ci-tools/browserslist-config-wikimedia/-/merge_requests/14 [18:01:50] 10GitLab, 06collaboration-services: Gitlab replica-b being indexed by Duck Duck Go - https://phabricator.wikimedia.org/T430563#12067454 (10Dzahn) >>! In T430563#12067105, @RhinosF1 wrote: >>>! In T430563#12066949, @Dzahn wrote: >> Does it cause a problem? > It indexing the wrong site is a problem, yes. It mean... [18:01:52] (03open) 10jforrester: [BREAKING CHANGE] Drop modern-es6-only, deprecated in v0.5.0 [repos/ci-tools/browserslist-config-wikimedia] - 10https://gitlab.wikimedia.org/repos/ci-tools/browserslist-config-wikimedia/-/merge_requests/16 [18:03:07] 10GitLab, 06collaboration-services: Gitlab replica-b being indexed by Duck Duck Go - https://phabricator.wikimedia.org/T430563#12067458 (10RhinosF1) >>! In T430563#12067454, @Dzahn wrote: >>>! In T430563#12067105, @RhinosF1 wrote: >>>>! In T430563#12066949, @Dzahn wrote: >>> Does it cause a problem? >> It inde... [18:19:29] PROBLEM - Host contint1003 is DOWN: PING CRITICAL - Packet loss = 100% [18:20:27] I rebooted this machine. [18:20:32] But it should be faster. [18:20:55] sigh, should be back by now.. have to look [18:21:06] RECOVERY - Host contint1003 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [18:21:20] PROBLEM - jenkins_service_running on contint1003 is CRITICAL: PROCS CRITICAL: 0 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [18:23:12] ffs, it is masked after reboot. why does this remain being a problem so much. [18:23:22] either not masked when it should or masked when it should not. [18:23:51] at this point I really want to drop that class that is trying to be smart about it [18:24:04] and just edit Hiera manually and that's it [18:25:04] or the monitoring downtime was deleted when the host went down? [18:25:41] yea, the mystery is more like why it started alerting in that moment [18:26:18] ah, maybe because the reboot fixed the firewall and monitoring host can SSH to it again, heh [18:29:42] PROBLEM - Host contint2003 is DOWN: PING CRITICAL - Packet loss = 100% [18:30:08] now doing the other host. under control. [18:31:10] RECOVERY - Host contint2003 is UP: PING OK - Packet loss = 0%, RTA = 31.64 ms [18:34:56] 10Phabricator: allow trusted contributors to delete gerritbot comments - https://phabricator.wikimedia.org/T430582#12067551 (10neriah) [18:42:24] 10Phabricator: allow trusted contributors to delete gerritbot comments - https://phabricator.wikimedia.org/T430582#12067579 (10Aklapper) 05Openβ†’03Declined Hi, I believe that allowing random folks to delete random things is much more harmful and dangerous than any potential advantages which are unfortunat... [18:44:19] 10Phabricator: allow trusted contributors to delete gerritbot comments - https://phabricator.wikimedia.org/T430582#12067591 (10Dzahn) In my experience deleting things generally causes more problems than just leaving things as is unless there is a hard legal reason or something. [18:49:29] 10Phabricator: allow trusted contributors to delete gerritbot comments - https://phabricator.wikimedia.org/T430582#12067613 (10neriah) Okay, I understand. Thank you. [18:50:31] thcipriani: tbh I'm unsure what to reply to the latest Bitergia emails. (In case you have thoughts more sorted out then me.) :-/ [18:50:55] what do they want?:) [18:51:23] 10Phabricator: allow trusted contributors to delete gerritbot comments - https://phabricator.wikimedia.org/T430582#12067618 (10Aklapper) For future reference, please use the feature request form (linked from the top of the task creation page) to create feature requests, and fill in the sections in the templa... [18:57:29] mutante: no, we want things :) [18:59:36] andre: I doubt I have my thoughts any more together, but I can take a try at a reply that pretty much points to your previous email. What do you think of that? [19:00:19] 10Beta-Cluster-Infrastructure: Eliminate sudo rm -rf /var/lib/puppet/ssl step in new deployment-prep WMCS project (and others) - https://phabricator.wikimedia.org/T429413#12067638 (10dancy) >>! In T429413#12066960, @dancy wrote: > TODO: Update https://wikitech.wikimedia.org/wiki/Help:Project_puppetserver after h... [19:00:57] thcipriani: That would probably be lovely, thanks. (I wondered if I should just re-send my email to the other addresses? but that feels... so weird) [19:01:52] I'll do that without doing it :) [19:03:33] "per my last email" ;) [19:07:24] 10Beta-Cluster-Infrastructure: Eliminate sudo rm -rf /var/lib/puppet/ssl step in new deployment-prep WMCS project (and others) - https://phabricator.wikimedia.org/T429413#12067672 (10dancy) 05In progressβ†’03Resolved [x] On the new node: sudo rm -rf /var/lib/puppet/ssl Resolved via https://gerrit.wikimedia... [19:16:00] thanks [20:02:23] 10Phabricator, 06Release-Engineering-Team (Doing 😎): "Additional Hashtags" field is inconsistently placed - https://phabricator.wikimedia.org/T413801#12067864 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Should be more consistent now across all Forms listed on https://phabricator.wikimedia.org/transac... [20:13:01] 10Phabricator (Upstream), 07Upstream: Mass-editing task status can create Closed tasks with "Unknown status" - https://phabricator.wikimedia.org/T125803#12067914 (10Aklapper) 05Openβ†’03Stalled p:05Mediumβ†’03Low `SELECT objectPHID, transactionType, oldValue, newValue, contentSource, FROM_UNIXTIME(dateCrea... [20:30:57] (03update) 10jforrester: build: Upgrade eslint-config-wikimedia from 0.26.0 to 0.32.4 and qunit from ^2.20.0 to ^2.26.0 [repos/ci-tools/browserslist-config-wikimedia] - 10https://gitlab.wikimedia.org/repos/ci-tools/browserslist-config-wikimedia/-/merge_requests/14 [21:28:36] FIRING: ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:28:38] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T430597 (10phaultfinder) 03NEW [21:33:31] RESOLVED: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:15:41] 10WikimediaDebug, 10API Platform: WikimediaDebug makes api.wikimedia.org always 301 redirect - https://phabricator.wikimedia.org/T430600 (10DLynch) 03NEW [22:17:13] 10WikimediaDebug, 10API Platform: WikimediaDebug makes api.wikimedia.org always 301 redirect - https://phabricator.wikimedia.org/T430600#12068296 (10DLynch) I discovered this while backporting a change to a #editcheck that had to access an API hosted on api.wikimedia.org, which seems to be fundamentally untest... [22:29:10] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.47.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T423918#12068328 (10Jdlrobson-WMF) [22:46:53] 10Beta-Cluster-Infrastructure: Eliminate sudo rm -rf /var/lib/puppet/ssl step in new deployment-prep WMCS project (and others) - https://phabricator.wikimedia.org/T429413#12068363 (10bd808) >>! In T429413#12067672, @dancy wrote: > Resolved by setting `profile::puppetserver::autosign: /usr/local/sbin/validate... [23:05:27] 10Phabricator maintenance bot, 10VisualEditor, 13Patch-For-Review: Should some additional VisualEditor-related projects be added to the @Maintenance_bot rule that auto-tags the #VisualEditor umbrella project? - https://phabricator.wikimedia.org/T422362#12068413 (10Ladsgroup) 05In progressβ†’03Resolved