[01:21:27] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:07:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [05:21:27] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:54:10] (03update) 10raymond-ndibe: [lima-kilo] cache container images [repos/cloud/toolforge/lima-kilo] (refactor_in_preparation_for_cache) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/196 [05:55:16] (03update) 10raymond-ndibe: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] (add_cache_registry) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) [06:07:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:09:12] 10Tool-itwiki: Catch unmanaged exceptions to report their time - https://phabricator.wikimedia.org/T377081#10224595 (10valerio.bozzolan) 05Open→03Resolved Closed in: https://gitlab.wikimedia.org/valeriobozzolan/itwiki-deletions-info-bot/-/commit/0d4bff70f40f85ee1f93c0d5a6af88ae4b9c1d89 [06:18:43] 10Tool-itwiki: Understand why BotCancellazioni is stuck (after every forced wiki read-only mode) apparently for cURL reasons - https://phabricator.wikimedia.org/T375937#10224602 (10valerio.bozzolan) I'm kind of 99.99% sure that the bot was stuck for 12 hours here: https://gitpull.it/source/boz-mw/browse/master/... [07:02:35] (03update) 10raymond-ndibe: Draft: [lima-kilo] test k8s Installation [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [07:04:06] (03update) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [07:09:35] (03update) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [07:10:46] (03update) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [07:12:42] (03update) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [07:13:24] (03update) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [07:23:06] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 [07:23:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [07:24:34] !log dcaro@urcuchillay tools END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 [07:24:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [07:24:59] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 [07:25:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [07:26:07] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 [07:26:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:16:33] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-74 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [08:21:33] RESOLVED: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-74 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProce [08:24:07] (03PS1) 10Giuseppe Lavagetto: Add api tokens for requestctl web ui [labs/private] - 10https://gerrit.wikimedia.org/r/1079930 [08:25:37] (03CR) 10Giuseppe Lavagetto: [V:03+2 C:03+2] Add api tokens for requestctl web ui [labs/private] - 10https://gerrit.wikimedia.org/r/1079930 (owner: 10Giuseppe Lavagetto) [08:27:18] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 06Tech-Docs-Team, 07Documentation: WMCS: Document different types of root and admin privileges - https://phabricator.wikimedia.org/T375113#10224770 (10aborrero) >>! In T375113#10222009, @fnegri wrote: > The structure is now fairly symmetrical: > * [Po... [08:43:11] (03open) 10dcaro: builds-builder: add script to migrate pipelineruns from v1beta1 to v1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/554 (https://phabricator.wikimedia.org/T376710) [08:45:41] (03open) 10kbach: Update #wikimedia-techdocs ops [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/9 (https://phabricator.wikimedia.org/T375854) [08:46:14] (03update) 10dcaro: builds-builder: add script to migrate pipelineruns from v1beta1 to v1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/554 (https://phabricator.wikimedia.org/T376710) [08:46:53] 06cloud-services-team, 10Cloud-VPS: openstack designate: introduce some as-code mechanism to instrument DNS data - https://phabricator.wikimedia.org/T338929#10224827 (10aborrero) →14Duplicate dup:03T374338 [08:51:23] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: tofu-infra: extend coverage to Designate DNS data - https://phabricator.wikimedia.org/T374338#10224830 (10aborrero) [08:52:02] 06cloud-services-team, 10Toolforge: toolforge: consider relocating core k8s components out of puppet into its own repository - https://phabricator.wikimedia.org/T328539#10224875 (10aborrero) [08:54:22] 06cloud-services-team, 10Toolforge: [cookbook,infra] automate docker image caching workflow - https://phabricator.wikimedia.org/T359816#10224846 (10aborrero) 05Open→03Resolved see https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1052286 [08:54:31] 06cloud-services-team, 10Toolforge: toolforge: consider relocating core k8s components out of puppet into its own repository - https://phabricator.wikimedia.org/T328539#10224876 (10aborrero) 05Open→03Resolved a:03aborrero [08:57:18] (03approved) 10aborrero: builds-builder: add script to migrate pipelineruns from v1beta1 to v1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/554 (https://phabricator.wikimedia.org/T376710) (owner: 10dcaro) [09:21:27] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:30:20] 10Cloud-VPS (Quota-requests): Temporary (1-2 weeks) quota increase for disaster recovery exercise - https://phabricator.wikimedia.org/T375977#10225001 (10Slst2020) a:05fnegri→03Slst2020 [09:32:08] 10Cloud-VPS (Quota-requests): Temporary (1-2 weeks) quota increase for disaster recovery exercise - https://phabricator.wikimedia.org/T375977#10225023 (10Slst2020) 05Stalled→03In progress [09:36:49] 10Cloud-VPS (Quota-requests): Temporary (1-2 weeks) quota increase for disaster recovery exercise - https://phabricator.wikimedia.org/T375977#10225055 (10Slst2020) @Audiodude I interpreted your request as needing another 4GB of RAM for Trove. If this was not the correct quota increase, please let me know. ` sst... [09:45:33] 06cloud-services-team, 10wikitech.wikimedia.org, 06Trust-and-Safety: Account recovery help needed for Developer account jgiannelos - https://phabricator.wikimedia.org/T377117 (10Jgiannelos) 03NEW [09:47:38] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 06Tech-Docs-Team, 07Documentation: WMCS: Document different types of root and admin privileges - https://phabricator.wikimedia.org/T375113#10225135 (10fnegri) > I would put them on the portal. I moved Cloud_roots_and_Cloud_admins to Portal:Cloud_VPS... [09:54:08] (03PS1) 10JMeybohm: Merge worker_containerd values back into worker [labs/private] - 10https://gerrit.wikimedia.org/r/1079955 (https://phabricator.wikimedia.org/T362408) [09:57:46] (03PS2) 10JMeybohm: Merge worker_containerd back to worker role [labs/private] - 10https://gerrit.wikimedia.org/r/1079955 (https://phabricator.wikimedia.org/T362408) [10:02:33] (03CR) 10JMeybohm: [V:03+2 C:03+2] Merge worker_containerd back to worker role [labs/private] - 10https://gerrit.wikimedia.org/r/1079955 (https://phabricator.wikimedia.org/T362408) (owner: 10JMeybohm) [10:03:59] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for rskwiki - https://phabricator.wikimedia.org/T375016#10225195 (10Ladsgroup) wiki is live now. About to create five more. [10:07:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [10:15:57] (03PS1) 10JMeybohm: Remove role kubernetes::staging::worker_containerd [labs/private] - 10https://gerrit.wikimedia.org/r/1079961 (https://phabricator.wikimedia.org/T362408) [10:17:14] (03CR) 10JMeybohm: [V:03+2 C:03+2] Remove role kubernetes::staging::worker_containerd [labs/private] - 10https://gerrit.wikimedia.org/r/1079961 (https://phabricator.wikimedia.org/T362408) (owner: 10JMeybohm) [10:21:14] (03update) 10dcaro: builds-builder: add script to migrate pipelineruns from v1beta1 to v1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/554 (https://phabricator.wikimedia.org/T376710) [10:43:15] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for nrwiki - https://phabricator.wikimedia.org/T375101#10225303 (10Ladsgroup) wiki is live [10:46:56] (03update) 10dcaro: builds-builder: add script to migrate pipelineruns from v1beta1 to v1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/554 (https://phabricator.wikimedia.org/T376710) [10:50:52] (03update) 10dcaro: builds-builder: add script to migrate pipelineruns from v1beta1 to v1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/554 (https://phabricator.wikimedia.org/T376710) [10:54:33] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for rskwiki - https://phabricator.wikimedia.org/T375016#10225340 (10Ladsgroup) >>! In T375016#10225195, @Ladsgroup wrote: > wiki is live now. About to create five more. for tddwiki the ticket has not been created. Consider it too :D [11:00:28] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for annwiki - https://phabricator.wikimedia.org/T377118#10225376 (10Ladsgroup) wiki is live [11:10:42] (03approved) 10dcaro: builds-builder: add script to migrate pipelineruns from v1beta1 to v1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/554 (https://phabricator.wikimedia.org/T376710) [11:10:49] (03merge) 10dcaro: builds-builder: add script to migrate pipelineruns from v1beta1 to v1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/554 (https://phabricator.wikimedia.org/T376710) [11:12:22] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 15), 13Patch-For-Review: [builds-builder,builds-api] After the upgrade build actions take >10s - https://phabricator.wikimedia.org/T376710#10225423 (10dcaro) All the pipelineruns were upgraded to v1 storage version, we are back t... [11:12:52] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for ibawiki - https://phabricator.wikimedia.org/T376571#10225412 (10Ladsgroup) wiki is live [11:14:03] 10Tool-Global-user-contributions, 10Special:GlobalContributions, 06Stewards-and-global-tools, 07Epic, and 2 others: [Epic] Implement global contributions feature - https://phabricator.wikimedia.org/T337089#10225429 (10kostajh) [11:15:25] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 15), 13Patch-For-Review: [builds-builder,builds-api] After the upgrade build actions take >10s - https://phabricator.wikimedia.org/T376710#10225421 (10dcaro) 05Open→03Resolved [11:19:23] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for bclwikisource - https://phabricator.wikimedia.org/T377087#10225442 (10Ladsgroup) wiki is live [11:41:48] 10cloud-services-team (Hardware): replace cloudlb2001-dev with cloudlb2004-dev - https://phabricator.wikimedia.org/T377126 (10aborrero) 03NEW [11:42:31] 10cloud-services-team (Hardware): replace cloudlb2001-dev with cloudlb2004-dev - https://phabricator.wikimedia.org/T377126#10225523 (10aborrero) p:05Triage→03Medium [11:45:27] 10Tool-Global-user-contributions, 10Special:GlobalContributions, 06Stewards-and-global-tools, 10Temporary accounts (Blockers to minor pilot wiki deployment), 10Trust and Safety Product Sprint (Sprint Cello (Oct 7 - 18)): Implement Global Contributions a... - https://phabricator.wikimedia.org/T376612#10225529 [11:53:26] 10cloud-services-team (Hardware): replace cloudlb2001-dev with cloudlb2004-dev - https://phabricator.wikimedia.org/T377126#10225540 (10aborrero) created https://netbox.wikimedia.org/ipam/ip-addresses/18168/ [12:10:51] 10cloud-services-team (Hardware), 13Patch-For-Review: replace cloudlb2001-dev with cloudlb2004-dev - https://phabricator.wikimedia.org/T377126#10225573 (10aborrero) p:05Medium→03Low [12:24:19] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1080003 (owner: 10L10n-bot) [12:24:35] !log dcaro@urcuchillay catalyst START - Cookbook wmcs.vps.refresh_puppet_certs on k3s.catalyst.eqiad1.wikimedia.cloud [12:24:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Catalyst/SAL [12:24:41] !log dcaro@urcuchillay catalyst END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on k3s.catalyst.eqiad1.wikimedia.cloud [12:24:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Catalyst/SAL [12:26:32] !log dcaro@urcuchillay catalyst START - Cookbook wmcs.vps.refresh_puppet_certs on k3s.catalyst.eqiad1.wikimedia.cloud [12:26:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Catalyst/SAL [12:26:36] !log dcaro@urcuchillay catalyst END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on k3s.catalyst.eqiad1.wikimedia.cloud [12:26:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Catalyst/SAL [12:27:15] !log dcaro@urcuchillay catalyst START - Cookbook wmcs.vps.refresh_puppet_certs on k3s.catalyst.eqiad1.wikimedia.cloud [12:27:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Catalyst/SAL [12:27:19] !log dcaro@urcuchillay catalyst END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on k3s.catalyst.eqiad1.wikimedia.cloud [12:27:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Catalyst/SAL [12:27:36] !log dcaro@urcuchillay catalyst START - Cookbook wmcs.vps.refresh_puppet_certs on k3s.catalyst.eqiad1.wikimedia.cloud [12:27:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Catalyst/SAL [12:28:13] !log dcaro@urcuchillay catalyst END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on k3s.catalyst.eqiad1.wikimedia.cloud [12:28:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Catalyst/SAL [12:34:53] (03open) 10sstefanova: deployment: add mock storage support for tokens [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/25 (https://phabricator.wikimedia.org/T362066) [12:56:28] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10225742 (10aborrero) >>! In T375847#10217807, @cmooney wrote: >>>! In T375847#10195673, @aborrero wrote: >> `lang=shell-session >> roo... [13:03:55] (03update) 10dcaro: Add superuser support [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/43 (https://phabricator.wikimedia.org/T362066) [13:21:28] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:29:18] (03update) 10dcaro: Draft: Add the creation of a single continuous job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/23 (https://phabricator.wikimedia.org/T362066) [13:49:12] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10225985 (10cmooney) We definitely want to use DHCPv6 (stateful) for address assignment. So OpenStack is in control of what IPs are us... [14:01:38] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10226051 (10cmooney) So.... maybe this is normal for DHCPv6? Re-reading the reddit post and looking at the setup on the VM it seems li... [14:07:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [14:38:20] (03update) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [14:59:40] (03merge) 10jjmc89: Update #wikimedia-techdocs ops [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/9 (https://phabricator.wikimedia.org/T375854) (owner: 10kbach) [15:31:53] (03open) 10sstefanova: dev: local pre-commit hook for exporting opanapi spec [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/26 [15:33:40] (03open) 10aborrero: codfw1dev: default secgroup: refresh IPv6 settings [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/96 (https://phabricator.wikimedia.org/T374714) [15:38:17] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: clarify IPv6 firewalling - https://phabricator.wikimedia.org/T374714#10226475 (10aborrero) [15:42:04] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: clarify IPv6 firewalling - https://phabricator.wikimedia.org/T374714#10226479 (10aborrero) 05Open→03In progress a:03aborrero [15:43:08] (03update) 10dcaro: Draft: Add the creation of a single continuous job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/23 (https://phabricator.wikimedia.org/T362066) [15:44:52] (03update) 10aborrero: codfw1dev: default secgroup: refresh IPv6 settings [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/96 (https://phabricator.wikimedia.org/T374714) [15:44:55] (03update) 10aborrero: codfw1dev: default secgroup: refresh IPv6 settings [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/96 (https://phabricator.wikimedia.org/T374714) [15:47:57] (03update) 10dcaro: Draft: Add the creation of a single continuous job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/23 (https://phabricator.wikimedia.org/T362066) [16:57:09] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for TheProtonade - https://phabricator.wikimedia.org/T375007#10226803 (10theprotonade) Done :) [17:14:10] (03update) 10dcaro: Draft: Add the creation of a single continuous job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/23 (https://phabricator.wikimedia.org/T362066) [17:14:51] (03update) 10dcaro: Draft: Add the creation of a single continuous job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/23 (https://phabricator.wikimedia.org/T362066) [17:21:28] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:25:19] (03update) 10dcaro: Draft: Add the creation of a single continuous job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/23 (https://phabricator.wikimedia.org/T362066) [17:34:37] 10VPS-project-Wikistats: Add rskwiki to wikistats - https://phabricator.wikimedia.org/T375021#10226915 (10Dzahn) a:03Dzahn [17:39:07] 10VPS-project-Wikistats: Add bclwikisource to wikistats - https://phabricator.wikimedia.org/T377092#10226926 (10Dzahn) a:03Dzahn [17:39:47] 10VPS-project-Wikistats: Add ibawiki to wikistats - https://phabricator.wikimedia.org/T376576#10226929 (10Dzahn) 05Stalled→03Open a:03Dzahn [17:41:18] 10VPS-project-Wikistats: Add tddwiki to wikistats - https://phabricator.wikimedia.org/T375428#10226934 (10Dzahn) 05Stalled→03Open a:03Dzahn [17:42:11] 10VPS-project-Wikistats: Add nrwiki to wikistats - https://phabricator.wikimedia.org/T375106#10226940 (10Dzahn) a:03Dzahn [18:07:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [18:44:17] 10VPS-project-Wikistats: Add annwiki to wikistats - https://phabricator.wikimedia.org/T377163 (10Pppery) 03NEW [20:28:34] 10VPS-project-Wikistats: Add annwiki to wikistats - https://phabricator.wikimedia.org/T377163#10227229 (10Dzahn) a:03Dzahn [21:21:28] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:07:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [22:47:34] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate mwv-builder-03.mediawiki-vagrant.eqiad.wmflabs is about to expire in 10d 23h 58m 34s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire