[00:49:04] 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Rollout Phase 3] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11231732 (10Krinkle) [01:42:33] 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Main Rollout] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11231812 (10Krinkle) [01:51:09] 06Traffic, 06MediaWiki-Platform-Team (Radar): Report progress on rollout of unified mobile routing (Hadoop query) - https://phabricator.wikimedia.org/T405429#11231816 (10Krinkle) ## Report: 30 Sep 2025 - Performance recovery I'd like share some early performance report from the rollout so far. Back in Februa... [01:54:54] 06Traffic, 06MediaWiki-Platform-Team (Radar): Report progress on rollout of unified mobile routing (Hadoop query) - https://phabricator.wikimedia.org/T405429#11231819 (10Krinkle) [04:03:51] FIRING: FermMSS: Unexpected MSS value on 10.2.2.27:80 @ ms-fe1009 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [04:08:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.2.27:80 @ ms-fe1009 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [04:24:51] FIRING: FermMSS: Unexpected MSS value on 10.2.2.27:80 @ ms-fe1009 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [04:29:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.2.27:80 @ ms-fe1009 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [07:01:30] 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Main Rollout] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11232058 (10Krinkle) [07:01:45] 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Main Rollout] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11232059 (10Krinkle) p:05Triage→03Medium [07:12:25] 06Traffic, 10DNS, 06SRE, 06Traffic-Icebox, and 2 others: Many misc wikis lack mobile domains - https://phabricator.wikimedia.org/T152882#11232084 (10Krinkle) [07:22:23] 06Traffic, 10DNS, 06SRE, 06Traffic-Icebox, and 2 others: Many misc wikis lack mobile domains - https://phabricator.wikimedia.org/T152882#11232096 (10Krinkle) 05Open→03Resolved a:03Krinkle I think we can call this done. All wiki listed here, plus a dozen more that I found, have been fixed so that... [07:54:12] 06Traffic, 10Page Content Service, 06serviceops: Block traffic to RESTBase /page/talk endpoint and sunset it - https://phabricator.wikimedia.org/T401895#11232159 (10akosiaris) [10:46:03] 06Traffic: Move HTTP/1.0 requests rejections at HAProxy level - https://phabricator.wikimedia.org/T365456#11232704 (10Fabfur) 05Stalled→03Declined This has been superseded by more refined actions to exclude a broader class of "invalid" requests. As HTTP/1.0 requests are not per-se invalid we can consider... [14:37:42] 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#11233546 (10elukey) I managed to have an idrac upgrade triggered by the cookbook, but it then failed when checking the state of the idrac (that was down because... [15:36:28] 06Traffic, 06Data-Platform-SRE: Disable LVS paging for WDQS - https://phabricator.wikimedia.org/T406141 (10LSobanski) 03NEW [15:38:55] 06Traffic, 06Data-Platform-SRE: Disable LVS paging for WDQS - https://phabricator.wikimedia.org/T406141#11233772 (10LSobanski) [15:42:00] 06Traffic, 06Data-Platform-SRE (2025.09.26 - 2025.10.17): Disable LVS paging for WDQS - https://phabricator.wikimedia.org/T406141#11233778 (10Gehel) p:05Triage→03High [15:45:13] 06Traffic, 06Data-Platform-SRE (2025.09.26 - 2025.10.17): Disable LVS paging for WDQS - https://phabricator.wikimedia.org/T406141#11233796 (10Gehel) For the record, our [[ https://wikitech.wikimedia.org/wiki/SLO/WDQS#Service_Level_Objectives | current SLO is 95% uptime ]], which allows for 4.5 days of downtime... [15:51:03] 10Acme-chief: Suggestion on how to setup acme-chief in the dcl testing environment - https://phabricator.wikimedia.org/T364511#11233820 (10jhathaway) ## DCL & Pebble ### Working hacks! 1. Run pebble on `acme-chief` server: PEBBLE_VA_ALWAYS_VALID=1 pebble 2. `acme-chief` chokes on EC certs, instead... [16:09:50] 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Main Rollout] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11233889 (10Krinkle) [16:10:05] 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Main Rollout] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11233892 (10Krinkle) [16:19:51] 06Traffic, 06Data-Platform-SRE (2025.09.26 - 2025.10.17): Disable LVS paging for WDQS - https://phabricator.wikimedia.org/T406141#11233945 (10ssingh) a:03ssingh [16:27:50] 06Traffic, 06Data-Engineering, 13Patch-For-Review: improved x-analytics data on Edge Uniques status - https://phabricator.wikimedia.org/T405783#11233996 (10Ottomata) @JAllemandou to review patches druid ingestion patch? [16:28:22] 06Traffic, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: improved x-analytics data on Edge Uniques status - https://phabricator.wikimedia.org/T405783#11234002 (10Ottomata) [16:42:43] 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Main Rollout] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11234039 (10Krinkle) #tech-news draft: > Mobile devices will receive mobile articles directly on the standard domain... [16:50:42] 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Main Rollout] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11234165 (10BCornwall) [17:45:40] 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Main Rollout] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11234452 (10BCornwall) [17:45:58] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad: 2 VM request for hCaptcha - https://phabricator.wikimedia.org/T406166#11234464 (10ssingh) [17:48:07] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad: 2 VM request for hCaptcha - https://phabricator.wikimedia.org/T406166#11234487 (10ssingh) [17:48:25] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: codfw: 2 VM request for hCaptcha - https://phabricator.wikimedia.org/T406167#11234491 (10ssingh) [19:03:13] 06Traffic, 10DNS, 13Patch-For-Review: Setting up Wikimedia Trust and Safety Help Center with Zendesk product: Seeking Guidance on host mapping - https://phabricator.wikimedia.org/T400952#11234743 (10ssingh) @BCornwall from Traffic is handling this and will sync with @JAbrams and @jrbs for when the change is... [19:18:39] 06Traffic, 10DNS, 13Patch-For-Review: Setting up Wikimedia Trust and Safety Help Center with Zendesk product: Seeking Guidance on host mapping - https://phabricator.wikimedia.org/T400952#11234803 (10BCornwall) I sent an email to hopefully coordinate some time for us all to work on this synchronously. [19:44:29] 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Main Rollout] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11234874 (10BCornwall) [20:12:21] 06Traffic, 10DNS, 06SRE, 10wikimediafoundation.org, 07IPv6: wikimediafoundation.org does not support IPv6 - https://phabricator.wikimedia.org/T403269#11234952 (10BCornwall) 05Open→03In progress p:05Triage→03Low a:03BCornwall I've contacted @Varnent to try and get the right IPv6 address. [20:13:26] 06Traffic, 10DNS, 10wikimediafoundation.org, 07IPv6: wikimediafoundation.org does not support IPv6 - https://phabricator.wikimedia.org/T403269#11234957 (10BCornwall)