[00:35:59] 06Traffic: Googlebot Commons 429 throttling - https://phabricator.wikimedia.org/T398668#11018910 (10Krinkle) Since our limits (afaik) aren't per-domain, I wanted to check whether and how much this may've impacted other domains in Google Search during the period where it was hitting the limit a lot. I suppos... [04:45:32] 06Traffic, 10Community-Tech (Sea Lion Squad), 10MediaWiki-Platform-Team (Radar), 07SEO: Suppress mobile redirect for Googlebot Smartphone on Commons - https://phabricator.wikimedia.org/T397267#11019062 (10tstarling) [09:27:28] 06Traffic: Create alert for low haproxykafka message rate (traffic) - https://phabricator.wikimedia.org/T400039 (10Fabfur) 03NEW [10:25:09] 06Traffic, 10HaproxyKafka: Create better alerts for HAProxyKafka - https://phabricator.wikimedia.org/T400039#11019838 (10Fabfur) p:05Triage→03High [10:34:27] 06Traffic, 10HaproxyKafka: Haproxykafka silently stops sending request data to kafka - https://phabricator.wikimedia.org/T400039#11019874 (10Vgutierrez) [11:28:00] FIRING: AnycastHealthcheckerRestarted: anycast-healthchecker service restarted on doh7003:9100 - https://wikitech.wikimedia.org/wiki/Anycast#Anycast_healthchecker_not_running - https://grafana.wikimedia.org/d/dxbfeGDZk/anycast?orgId=1&var-protocol=BGP&var-site=magru&var-cluster=All&var-ip_version=All - https://alerts.wikimedia.org/?q=alertname%3DAnycastHealthcheckerRestarted [11:32:25] FIRING: SystemdUnitFailed: anycast-healthchecker.service on doh7003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:48:44] 10netops, 06Infrastructure-Foundations, 06SRE: BGP: Support receipt of graceful-shutdown community and set local-pref - https://phabricator.wikimedia.org/T399931#11020341 (10cmooney) 05Open→03Resolved a:03cmooney [14:28:25] 06Traffic, 06Movement-Insights, 10Data-Engineering (Q4 2025 April 1st - June 30th): NEW BUG REPORT: Investigate rise in May 2025 Reader metrics - https://phabricator.wikimedia.org/T395934#11020480 (10Ahoelzl) a:03mforns [16:35:38] FIRING: [4x] LVSRealserverMSS: Unexpected MSS value on 195.200.68.224:443 @ cp7001 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=magru&var-cluster=cache_text - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS [16:35:48] ^ reboot, expected [16:40:38] RESOLVED: [4x] LVSRealserverMSS: Unexpected MSS value on 195.200.68.224:443 @ cp7001 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=magru&var-cluster=cache_text - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS [17:07:06] 06Traffic, 10DNS, 06SRE: Verify wikimediafoundation.org for Visual Studio Marketplace. - https://phabricator.wikimedia.org/T400089 (10Seddon) 03NEW [17:30:51] 06Traffic, 10DNS, 06SRE, 13Patch-For-Review: Verify wikimediafoundation.org for Visual Studio Marketplace. - https://phabricator.wikimedia.org/T400089#11021515 (10BCornwall) 05Open→03In progress p:05Triage→03Low a:03BCornwall [17:56:35] 06Traffic, 10DNS, 06SRE, 13Patch-For-Review: Verify wikimediafoundation.org for Visual Studio Marketplace. - https://phabricator.wikimedia.org/T400089#11021595 (10BCornwall) @Seddon This has been merged. Let me know if it works! [18:42:49] 06Traffic, 10DNS, 06SRE: Verify wikimediafoundation.org for Visual Studio Marketplace. - https://phabricator.wikimedia.org/T400089#11021750 (10Seddon) Thank you! Will keep you in the loop. [19:18:38] FIRING: [4x] LVSRealserverMSS: Unexpected MSS value on 195.200.68.224:443 @ cp7001 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=magru&var-cluster=cache_text - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS [19:23:48] ^ brett restarted again? (checking) [19:24:59] Yeah [19:25:33] thanks [20:16:33] 06Traffic, 06SRE, 07affects-Kiwix-and-openZIM: Rate limiting/status code 429 for mwclient? - https://phabricator.wikimedia.org/T400018#11022094 (10Pppery) [20:18:28] mutante: What's the status of the websocket rollout? [20:28:38] RESOLVED: [4x] LVSRealserverMSS: Unexpected MSS value on 195.200.68.224:443 @ cp7001 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=magru&var-cluster=cache_text - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS [20:33:32] 06Traffic, 06SRE, 07affects-Kiwix-and-openZIM: Rate limiting/status code 429 for mwclient? - https://phabricator.wikimedia.org/T400018#11022145 (10ssingh) Thanks for sharing; we are looking into it. [20:43:22] brett: reverted and there will be a new attempt [20:44:05] to our surprise just editing the .erb template had no effect [20:44:13] now: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1171263 [21:01:32] 06Traffic, 06SRE, 07affects-Kiwix-and-openZIM: Rate limiting/status code 429 for mwclient? - https://phabricator.wikimedia.org/T400018#11022178 (10Scott_French) @Audiodude - At least at the version of `mwclient` that you appear to be using (0.9.3 per your [[ https://github.com/openzim/wp1/blob/main/Pipfile |... [21:24:26] 06Traffic, 06SRE, 07affects-Kiwix-and-openZIM: Rate limiting/status code 429 for mwclient? - https://phabricator.wikimedia.org/T400018#11022205 (10Audiodude) Thanks for taking the time and looking into what mwclient version we use in production. Upgrading to 0.11.0 was the first thing I did when attempting t... [21:32:45] 06Traffic, 06SRE, 07affects-Kiwix-and-openZIM: Rate limiting/status code 429 for mwclient? - https://phabricator.wikimedia.org/T400018#11022221 (10Scott_French) Thanks for the follow-up @Audiodude. So, from a quick look at 0.11.0 it looks like [[ https://github.com/mwclient/mwclient/blob/v0.11.0/mwclient/cli... [23:18:24] 06Traffic, 06SRE, 07affects-Kiwix-and-openZIM: Rate limiting/status code 429 for mwclient? - https://phabricator.wikimedia.org/T400018#11022400 (10Audiodude) Thanks for that. I think you might have incorrect assumptions about the wp1 code, though. We do not attempt to set any custom "WP 1.0 Bot" user agent,... [23:21:00] 06Traffic, 06SRE, 07affects-Kiwix-and-openZIM: Rate limiting/status code 429 for mwclient? - https://phabricator.wikimedia.org/T400018#11022401 (10Audiodude) Oh nevermind, we do use a connection pool in order to re-use the login cookies! So it seems according to your analysis (which I just confirmed), becaus... [23:59:43] 06Traffic, 06SRE, 07affects-Kiwix-and-openZIM: Rate limiting/status code 429 for mwclient? - https://phabricator.wikimedia.org/T400018#11022420 (10Scott_French) Thanks for taking a closer look. Indeed, what's likely happening is that, in the absence of User-Agent being explicitly set, the `requests` library...