[00:15:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [00:18:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [00:29:00] FIRING: [4x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [00:43:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [00:48:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [01:03:04] FIRING: ObjectStorageSizeQuotaFull: Object storage quota by 'size' is 80.17% full for project tools-logging - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageSizeQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageSizeQuotaFull [01:13:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [01:18:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [01:24:00] FIRING: [5x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [01:43:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [01:49:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [01:54:00] FIRING: [6x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [02:04:00] FIRING: [7x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [02:14:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [02:19:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [02:20:00] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [02:34:00] FIRING: [8x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [02:39:00] FIRING: [9x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [02:44:01] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment eqiad1 [02:44:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [02:46:17] FIRING: JobUnavailable: Reduced availability for job rabbitmq in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:47:09] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=0) on deployment eqiad1 [02:47:22] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services [02:48:49] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 5.799% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [02:48:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [02:51:17] RESOLVED: JobUnavailable: Reduced availability for job rabbitmq in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:51:43] FIRING: [2x] NodeTextfileStale: Stale textfile for cloudcumin1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [03:00:19] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services [03:11:04] RESOLVED: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.743% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [03:11:30] RESOLVED: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [03:21:26] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:03:04] FIRING: ObjectStorageSizeQuotaFull: Object storage quota by 'size' is 80.16% full for project tools-logging - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageSizeQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageSizeQuotaFull [06:51:44] FIRING: [2x] NodeTextfileStale: Stale textfile for cloudcumin1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:03:04] FIRING: ObjectStorageSizeQuotaFull: Object storage quota by 'size' is 80.02% full for project tools-logging - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageSizeQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageSizeQuotaFull [09:46:28] FIRING: [2x] NodeTextfileStale: Stale textfile for cloudcumin1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:51:28] RESOLVED: NodeTextfileStale: Stale textfile for cloudcumin1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:57:40] !log dcaro@cloudcumin1001 pki START - Cookbook wmcs.vps.refresh_puppet_certs on pki-pm.pki.eqiad1.wikimedia.cloud [10:00:48] !log dcaro@cloudcumin1001 pki END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on pki-pm.pki.eqiad1.wikimedia.cloud [10:04:30] FIRING: [6x] PuppetCertificateAboutToExpire: Puppet CA certificate pki-db.pki.eqiad1.wikimedia.cloud is about to expire in 21d 0h 58m 33s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [10:07:00] !log dcaro@cloudcumin1001 pki START - Cookbook wmcs.vps.refresh_puppet_certs on pki-db.pki.eqiad1.wikimedia.cloud [10:08:50] !log dcaro@cloudcumin1001 pki END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on pki-db.pki.eqiad1.wikimedia.cloud [10:09:03] !log dcaro@cloudcumin1001 pki START - Cookbook wmcs.vps.refresh_puppet_certs on pki-root.pki.eqiad1.wikimedia.cloud [10:10:24] !log dcaro@cloudcumin1001 pki END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on pki-root.pki.eqiad1.wikimedia.cloud [10:11:13] !log dcaro@cloudcumin1001 pki START - Cookbook wmcs.vps.refresh_puppet_certs on pki-intermediate.pki.eqiad1.wikimedia.cloud [10:12:04] 06cloud-services-team, 10Data-Services, 10Datasets-General-or-Unknown, 06Traffic, 13Patch-For-Review: Move dumps.wikimedia.org HTTP service behind CDN edge - https://phabricator.wikimedia.org/T306550#11595657 (10ayounsi) [10:13:10] !log dcaro@cloudcumin1001 pki END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on pki-intermediate.pki.eqiad1.wikimedia.cloud [10:14:28] !log dcaro@cloudcumin1001 pki START - Cookbook wmcs.vps.refresh_puppet_certs on pki-proxy.pki.eqiad1.wikimedia.cloud [10:15:56] !log dcaro@cloudcumin1001 pki END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on pki-proxy.pki.eqiad1.wikimedia.cloud [11:02:49] RESOLVED: ObjectStorageSizeQuotaFull: Object storage quota by 'size' is 80% full for project tools-logging - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageSizeQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageSizeQuotaFull [11:08:42] 06cloud-services-team, 10Cloud-VPS: Jacob Thwaites - unable to log into Cloud VPS Bastion - https://phabricator.wikimedia.org/T416689#11596101 (10Jacob_WMDE) Thanks for your help @Andrew, this is working now. [11:08:58] 06cloud-services-team, 10Cloud-VPS: Jacob Thwaites - unable to log into Cloud VPS Bastion - https://phabricator.wikimedia.org/T416689#11596102 (10Jacob_WMDE) 05Open→03Resolved [11:31:09] 10Toolforge, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11596159 (10fgiunchedi) Following up from Lisbon: there are essentially two options available wrt network implementation, @cmooney... [12:26:54] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/34 [12:26:55] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/31 [12:44:30] FIRING: [5x] PuppetCertificateAboutToExpire: Puppet CA certificate pki-db.pki.eqiad1.wikimedia.cloud is about to expire in 20d 22h 18m 33s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [12:49:30] FIRING: [5x] PuppetCertificateAboutToExpire: Puppet CA certificate pki-db.pki.eqiad1.wikimedia.cloud is about to expire in 20d 22h 17m 33s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [12:54:30] RESOLVED: [5x] PuppetCertificateAboutToExpire: Puppet CA certificate pki-db.pki.eqiad1.wikimedia.cloud is about to expire in 20d 22h 16m 33s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [13:06:53] (03update) 10raymond-ndibe: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 [13:10:55] (03update) 10raymond-ndibe: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 [13:11:50] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: resolve “jobs-framework” and “web” image variants to the same image - https://phabricator.wikimedia.org/T414978#11596510 (10fnegri) > I decided to assign the open MRs about this issue to the task https://phabricator.wikimedia.org/T415322, they are m... [13:12:49] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: resolve “jobs-framework” and “web” image variants to the same image - https://phabricator.wikimedia.org/T414978#11596512 (10fnegri) →14Duplicate dup:03T415322 [13:12:58] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: Replace job image variants with webservice image variants - https://phabricator.wikimedia.org/T415322#11596514 (10fnegri) [14:06:46] (03update) 10mercy-o: Fix bugs and improve UX in microtask generator [toolforge-repos/microtask-generator] - 10https://gitlab.wikimedia.org/toolforge-repos/microtask-generator/-/merge_requests/3 [14:18:31] (03update) 10raymond-ndibe: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 [14:29:25] (03update) 10raymond-ndibe: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 [14:29:41] (03CR) 10Andrew Bogott: "I used this yesterday after puppet started to drop messages, and it worked great." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1236841 (owner: 10Andrew Bogott) [14:45:51] (03update) 10isaacj: Fix bugs and improve UX in microtask generator [toolforge-repos/microtask-generator] - 10https://gitlab.wikimedia.org/toolforge-repos/microtask-generator/-/merge_requests/3 (owner: 10mercy-o) [14:46:17] (03merge) 10isaacj: Fix bugs and improve UX in microtask generator [toolforge-repos/microtask-generator] - 10https://gitlab.wikimedia.org/toolforge-repos/microtask-generator/-/merge_requests/3 (owner: 10mercy-o) [15:02:51] 06cloud-services-team, 06collaboration-services, 06SRE Observability, 13Patch-For-Review: Move https://status.wikimedia.org/ away from rackspace - https://phabricator.wikimedia.org/T414098#11596979 (10Jelto) >>! In T414098#11591965, @Dzahn wrote: > The repo for this that was already created: > > https://g... [15:40:30] 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: codfw: Upgrade cloudsw1-b1-codfw (2026) - https://phabricator.wikimedia.org/T416443#11597118 (10ayounsi) p:05Triage→03Medium [17:01:44] (03update) 10dcaro: change permission of home dir to 0701 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/305 (owner: 10raymond-ndibe) [17:34:34] (03open) 10sfaci: Renamed MetricsPlatform with TestKitchen [toolforge-repos/codex-usage] - 10https://gitlab.wikimedia.org/toolforge-repos/codex-usage/-/merge_requests/2 (https://phabricator.wikimedia.org/T414459) [17:43:32] 06cloud-services-team, 06collaboration-services, 06SRE Observability, 13Patch-For-Review: Move https://status.wikimedia.org/ away from rackspace - https://phabricator.wikimedia.org/T414098#11598147 (10Dzahn) Yay:) I was going to ask about this (comments over on gitlab) and you guys already fixed it, thanks! [17:47:36] 06cloud-services-team, 06collaboration-services, 06SRE Observability, 13Patch-For-Review: Move https://status.wikimedia.org/ away from rackspace - https://phabricator.wikimedia.org/T414098#11598179 (10Dzahn) publish image pipeline job worked: **docker-registry.discovery.wmnet/repos/sre/miscweb/status.wiki... [17:49:00] (03PS1) 10Santiago Faci: InitialiseSettings.php.txt: Updated configuration to use TestKitchen [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1237962 (https://phabricator.wikimedia.org/T414466) [17:58:31] (03open) 10sfaci: Removed MetricsPlatform as a dependency. No longer used by WikimediaMessages... [toolforge-repos/abstract-wiki-prototype] - 10https://gitlab.wikimedia.org/toolforge-repos/abstract-wiki-prototype/-/merge_requests/1 (https://phabricator.wikimedia.org/T414458) [18:29:29] 10VPS-project-Wikistats: Add kaiwiki to wikistats - https://phabricator.wikimedia.org/T414242#11598441 (10Dzahn) I am going to manually link T414234 here then so that I can see the status. [18:38:46] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/31 (owner: 10l10n-bot) [18:38:49] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/31 (owner: 10l10n-bot) [18:42:22] 10VPS-project-Codesearch: Codesearch down/unreachable (2026-02-05) - https://phabricator.wikimedia.org/T416614#11598498 (10brennen) > Can you share your ldap username? I'll add you in horizon. "Brennen Bearnes" / brennen [18:42:53] 10VPS-project-Codesearch: Codesearch down/unreachable (2026-02-05) - https://phabricator.wikimedia.org/T416614#11598499 (10brennen) [19:02:36] 06cloud-services-team, 06collaboration-services, 06SRE Observability: Move https://status.wikimedia.org/ away from rackspace - https://phabricator.wikimedia.org/T414098#11598603 (10Dzahn) deployed cert addition to admin_ng per https://wikitech.wikimedia.org/wiki/Kubernetes/Remove_a_service#Deploy_changes_to_... [19:20:12] 06Toolforge-standards-committee: Share/publish source code of "request" tool - https://phabricator.wikimedia.org/T416678#11598642 (10LucasWerkmeister) I don’t remember anything ^^ but T389540#10774116 is probably most of what I found at the time (sadly with less “methodology” than I’d like now). [19:23:49] 10VPS-project-Codesearch: Codesearch down/unreachable (2026-02-05) - https://phabricator.wikimedia.org/T416614#11598654 (10Ladsgroup) >>! In T416614#11598498, @brennen wrote: >> Can you share your ldap username? I'll add you in horizon. > > "Brennen Bearnes" / brennen Done. Here is the doc: https://www.med... [19:45:45] 10wikitech.wikimedia.org, 06serviceops-radar, 06SRE, 13Patch-For-Review, 07SRE-Unowned: Redesign wikitech-static - https://phabricator.wikimedia.org/T376400#11598756 (10Andrew) I'm pretty sure I have images getting rolled into the static site now. Going to wait a few days for @taavi to find another issue... [19:52:49] 06Toolforge-standards-committee: Share/publish source code of "request" tool - https://phabricator.wikimedia.org/T416678#11598772 (10LucasWerkmeister) There are a bunch of separate Git repositories in `FNBot/` (along with more directories that aren’t Git repos and which will just go into the main request-legacyc... [20:38:10] 10Cloud-VPS (Project-requests): Request creation of cd.io-teeW9or8 VPS project - https://phabricator.wikimedia.org/T416738#11598957 (10Andrew) Hello @Otcenas11 -- I have a few questions about your request. 1) The license terms listed for changedetection.io look a bit inconsistent to me. On the one hand, github... [20:41:14] 10PAWS, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T415828#11598963 (10LibUp-bot) A new upstream version of Pywikibot is now available: 11.0.0. * https://gerrit.wikimedia.org/g/pywikibot/core/+/refs/tags/11.0.0 * https://doc.wikimedia.org/pywikibot/stable/chan... [20:41:19] 10Toolforge, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T415829#11598964 (10LibUp-bot) A new upstream version of Pywikibot is now available: 11.0.0. * https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Pywikibot_image * https://gerrit.wikimedia.org/g/p... [20:51:46] (03CR) 10Krinkle: "Does the MegaTable still work after this? It takes a bit of setting up, and I'd rather not spend time on that. The file should not be upda" [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1237962 (https://phabricator.wikimedia.org/T414466) (owner: 10Santiago Faci) [21:48:43] (03CR) 10Santiago Faci: "I'm sorry. I didn't remember that comment. We were trying to update all the remaining references to `MetricsPlatform` because the extensio" [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1237962 (https://phabricator.wikimedia.org/T414466) (owner: 10Santiago Faci) [21:48:51] (03Abandoned) 10Santiago Faci: InitialiseSettings.php.txt: Updated configuration to use TestKitchen [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1237962 (https://phabricator.wikimedia.org/T414466) (owner: 10Santiago Faci) [22:36:42] 06cloud-services-team, 10Cloud-VPS: Jacob Thwaites - unable to log into Cloud VPS Bastion - https://phabricator.wikimedia.org/T416689#11599466 (10bd808) a:03Andrew