[00:40:09] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 6 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11721374 (10Ladsgroup) >>! In T414805#11675775, @Ladsgroup wrote: >>>! In T414805#11668230, @Ladsgroup wrote: >> Top "file formats" f... [03:07:20] FIRING: DnsboxServiceMismatch: Service ntp-b state mismatch on dns2005:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=codfw&var-instance=dns2005:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [03:17:20] RESOLVED: DnsboxServiceMismatch: Service ntp-b state mismatch on dns2005:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=codfw&var-instance=dns2005:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [04:34:43] FIRING: [2x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [04:39:43] FIRING: [5x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [04:44:43] FIRING: [5x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [04:49:43] RESOLVED: [5x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [07:24:49] 06Traffic, 06collaboration-services, 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: Gerrit: Debug connection re-use on Gerrit's httpd causing Gerrit interface to be very slow - https://phabricator.wikimedia.org/T420189#11721740 (10ABran-WMF) [07:28:34] 06Traffic, 06collaboration-services, 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: Gerrit: Debug connection re-use on Gerrit's httpd causing Gerrit interface to be very slow - https://phabricator.wikimedia.org/T420189#11721745 (10ABran-WMF) Testing connection re-use with the new tweaks had th... [07:58:53] 06Traffic, 06collaboration-services, 10Gerrit, 06Release-Engineering-Team: Gerrit: Debug connection re-use on Gerrit's httpd causing Gerrit interface to be very slow - https://phabricator.wikimedia.org/T420189#11721787 (10ABran-WMF) The [[ https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&ref... [08:55:42] 06Traffic, 06collaboration-services, 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use - https://phabricator.wikimedia.org/T417998#11721863 (10hashar) 05Resolved→03Stalled Reopening as {nav Stalled} given the ATS connecti... [09:08:51] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: Update ULSFO LVS service IP's - https://phabricator.wikimedia.org/T418971#11721899 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=b34081b8-989e-49aa-91c7-56b4548775e2) set by slyngshede@cumin1003 for 4:... [09:25:38] FIRING: [14x] LVSRealserverMSS: Unexpected MSS value on 198.35.26.240:443 @ cp4045 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_upload - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS [09:25:38] FIRING: [21x] LVSRealserverMSS: Unexpected MSS value on 198.35.26.224:443 @ cp4037 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_text - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS [09:30:38] FIRING: [16x] LVSRealserverMSS: Unexpected MSS value on 198.35.26.240:443 @ cp4045 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_upload - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS [09:30:38] FIRING: [24x] LVSRealserverMSS: Unexpected MSS value on 198.35.26.224:443 @ cp4037 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_text - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS [09:35:38] RESOLVED: [16x] LVSRealserverMSS: Unexpected MSS value on 198.35.26.240:443 @ cp4045 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_upload - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS [09:35:38] RESOLVED: [24x] LVSRealserverMSS: Unexpected MSS value on 198.35.26.224:443 @ cp4037 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_text - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS [09:47:11] 06Traffic, 10Liberica: Replace ping offload servers with eBPF - https://phabricator.wikimedia.org/T367973#11721970 (10ayounsi) @ssingh @Vgutierrez I was wondering if you could prioritize this at some point or agree to drop the current ping offload servers. With the progress to the new network design we won't b... [09:55:45] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06ServiceOps new, 07Kubernetes: Migrate Wikikube k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420436 (10JMeybohm) 03NEW [09:57:05] 06Traffic, 06Data-Platform-SRE, 10Liberica, 10Prod-Kubernetes, 07Kubernetes: Migrate DSE k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420437 (10JMeybohm) 03NEW [09:57:59] 06Traffic, 10Liberica, 06Machine-Learning-Team, 10Prod-Kubernetes, 07Kubernetes: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438 (10JMeybohm) 03NEW [09:59:01] 06Traffic, 06Infrastructure-Foundations, 10Liberica, 10Prod-Kubernetes, 07Kubernetes: Migrate AUX k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420439 (10JMeybohm) 03NEW [10:01:54] 06Traffic, 10Prod-Kubernetes, 06ServiceOps new, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#11722069 (10JMeybohm) I've created subtasks for the remaining clusters/teams, so this "analyze and prepare"-... [10:02:08] 06Traffic, 10Prod-Kubernetes, 06ServiceOps new, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#11722075 (10JMeybohm) 05Open→03Resolved [10:02:47] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06Data-Platform-SRE (2026-03-06 - 2026-03-27), 07Kubernetes: Migrate DSE k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420437#11722078 (10BTullis) a:03BTullis [10:07:09] 06Traffic, 10Liberica, 10Prod-Kubernetes, 07Kubernetes, 06ServiceOps new (Next quarter): Migrate Wikikube k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420436#11722086 (10JMeybohm) [10:07:41] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: Update ULSFO LVS service IP's - https://phabricator.wikimedia.org/T418971#11722088 (10SLyngshede-WMF) 05Open→03Resolved @Papaul Done :-) [11:44:43] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 6 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11722551 (10MatthewVernon) >>! In T414805#11721374, @Ladsgroup wrote: >>>! In T414805#11682308, @MatthewVernon wrote: >> @Ladsgroup t... [11:56:40] 06Traffic, 10Liberica: Replace ping offload servers with eBPF - https://phabricator.wikimedia.org/T367973#11722626 (10Vgutierrez) this is a by-product of switching to Katran and not a feature that we can deploy independently at the moment, so in eqiad and codfw this is currently blocked till we can switch ever... [12:04:08] XioNoX: I could develop a standalone ping offload eBPF program and deploy that on lvs instances in eqiad/codfw if really needed [12:04:42] vgutierrez: no real rush, why is it not possible to only deploy for the text + upload VIPs? [12:05:08] we don't want a mixed environment where the active LB runs one technology and the secondary one another [12:05:19] I see [12:05:35] I think we're in good way to have it all on katran in the next months, no? [12:05:45] yes [12:53:30] 06Traffic, 06SRE Observability, 13Patch-For-Review: Move wikimediastatus.net 301 to ncredir - https://phabricator.wikimedia.org/T419887#11722879 (10taavi) [13:09:31] 06Traffic, 06collaboration-services, 10Gerrit, 06Release-Engineering-Team: Gerrit: Debug connection re-use on Gerrit's httpd causing Gerrit interface to be very slow - https://phabricator.wikimedia.org/T420189#11722913 (10ABran-WMF) After a conversation with @hashar where I was thinking about bumping `http... [14:22:44] 06Traffic, 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 07OKR-Work, 13Patch-For-Review, 06Test Kitchen (Experiment Platform Sprint 21): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11723298 (10Sfaci) @ssingh As far as I und... [14:35:20] FIRING: DnsboxServiceMismatch: Service ntp-b state mismatch on dns1005:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=eqiad&var-instance=dns1005:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [14:35:29] yeah that's fine, reboot [14:35:44] should it fired now? no, but the downtiming thing is all over the place so I am not going to bother looking [14:40:20] RESOLVED: [4x] DnsboxServiceMismatch: Service authdns-ns0 state mismatch on dns1005:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=eqiad&var-instance=dns1005:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [14:54:26] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11723442 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp3074.esams.wmnet with OS trixie [14:55:00] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11723446 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp3075.esams.wmnet with OS trixie [15:39:47] 06Traffic, 06DC-Ops, 10decommission-hardware, 10ops-codfw: Decommission codfw cp hosts cp2027-cp2040 - https://phabricator.wikimedia.org/T419753#11723690 (10BCornwall) [15:51:40] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11723747 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp3074.esams.wmnet with OS trixie completed: - cp3074 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled Pup... [16:00:14] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11723764 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp3076.esams.wmnet with OS trixie [16:00:25] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11723769 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp3075.esams.wmnet with OS trixie completed: - cp3075 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled Pup... [16:33:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp2045 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2045 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [16:36:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp2046 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2046 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [16:38:43] RESOLVED: HaproxyKafkaExporterDown: HaproxyKafka on cp2045 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2045 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [16:41:43] RESOLVED: HaproxyKafkaExporterDown: HaproxyKafka on cp2046 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2046 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [16:41:48] 06Traffic, 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 07OKR-Work, 13Patch-For-Review, 06Test Kitchen (Experiment Platform Sprint 21): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11724127 (10ssingh) >>! In T407570#1172329... [16:47:37] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11724154 (10CDobbins) [16:55:37] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11724192 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp3076.esams.wmnet with OS trixie completed: - cp3076 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled Pup... [17:01:53] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11724218 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp3077.esams.wmnet with OS trixie [17:03:07] 06Traffic, 05MW-1.46-notes (1.46.0-wmf.19; 2026-03-10), 07OKR-Work, 13Patch-For-Review, 06Test Kitchen (Experiment Platform Sprint 21): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11724223 (10Sfaci) > Things seem to be fin... [17:03:10] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11724225 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp3078.esams.wmnet with OS trixie [17:08:52] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11724282 (10BCornwall) [17:10:14] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11724289 (10BCornwall) [17:10:44] 06Traffic, 06SRE: Deprecate low-traffic proxoid service and O:hcaptcha_proxy for the older hcaptcha proxy setup - https://phabricator.wikimedia.org/T411097#11724294 (10Raine) [17:16:19] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11724371 (10CDobbins) [17:30:28] 06Traffic, 10observability: Factor in pooled status for SLO measurements - https://phabricator.wikimedia.org/T420498 (10BCornwall) 03NEW [17:30:46] 06Traffic, 10observability: Factor in pooled status for SLO measurements - https://phabricator.wikimedia.org/T420498#11724502 (10BCornwall) [17:56:28] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11724704 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp3077.esams.wmnet with OS trixie completed: - cp3077 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled Pup... [18:00:01] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11724738 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp3078.esams.wmnet with OS trixie completed: - cp3078 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled Pup... [18:13:46] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11724811 (10CDobbins) [18:17:12] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11724847 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp5018.eqsin.wmnet with OS trixie [18:18:08] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11724852 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp5017.eqsin.wmnet with OS trixie [19:05:32] 06Traffic: Add new Auth DNS IPv6 addresses to ns_group firewall group - https://phabricator.wikimedia.org/T420361#11725057 (10ssingh) 05Open→03Resolved a:03ssingh [19:09:26] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11725062 (10BCornwall) [19:26:14] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11725168 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp5018.eqsin.wmnet with OS trixie completed: - cp5018 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled Pup... [19:29:09] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11725171 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp5020.eqsin.wmnet with OS trixie [19:36:32] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11725176 (10CDobbins) [19:39:36] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11725199 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp5017.eqsin.wmnet with OS trixie executed with errors: - cp5017 (**FAIL**) - Downtimed on Icinga/Alertmanager - D... [20:05:42] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11725284 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp5017.eqsin.wmnet with OS trixie [20:23:58] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11725337 (10BCornwall) [20:50:38] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11725426 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp5020.eqsin.wmnet with OS trixie executed with errors: - cp5020 (**FAIL**) - Downtimed on Icinga/Alertmanager - D... [21:16:51] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11725522 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp5017.eqsin.wmnet with OS trixie completed: - cp5017 (**PASS**) - Removed from Puppet and PuppetDB if present and d... [21:49:20] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 6 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11725676 (10TheDJ) I was testing File:High_quality_skull.stl locally via instantcommons. And i'm not sure why, but it seems my setup... [22:48:36] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 6 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11725889 (10Ladsgroup) STL is basically the only file handler left that is not following thumb steps yet (everything else from T41480... [23:05:38] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11725916 (10BCornwall) [23:05:55] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11725918 (10BCornwall)