[00:10:17] 10netops, 06Infrastructure-Foundations, 06SRE: Management routers: use BGP instead of OSPF - https://phabricator.wikimedia.org/T294845#11130589 (10Papaul) @ayounsi thank you I will get back on this next week since i am off tomorrow and Monday is a U.S holiday. [00:34:26] 06Traffic, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: [Rollout Phase 1] Implement redirect-less mobile routing and enable for wikitech.wikimedia.org - https://phabricator.wikimedia.org/T401595#11130636 (10Krinkle) @Jdlrobson-WMF Do we want `mobileaction` params to remain in the address bar after... [04:38:00] FIRING: [3x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [04:43:00] FIRING: [27x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [04:48:01] RESOLVED: [29x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [05:52:24] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Remove static routes for LVS VIPs from core routers - https://phabricator.wikimedia.org/T300877#11130890 (10ayounsi) > the idea is that static routes should help save us in that situation That would only be the case for lvs1016, 1018, 1019 and 1020... [06:39:40] FIRING: VarnishHighThreadCount: Varnish's thread count on cp5030:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=eqsin&var-instance=cp5030 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [06:43:01] FIRING: PurgedHighBacklogQueue: Large backlog queue for purged on cp5030:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=eqsin%20prometheus/ops&var-instance=cp5030 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [06:44:40] FIRING: VarnishHighThreadCount: Varnish's thread count on cp5030:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=eqsin&var-instance=cp5030 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [07:03:00] RESOLVED: PurgedHighBacklogQueue: Large backlog queue for purged on cp5030:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=eqsin%20prometheus/ops&var-instance=cp5030 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [07:03:02] 06Traffic: anycast-healthchecker fails to start after a reboot and before a puppet run - https://phabricator.wikimedia.org/T314457#11130947 (10fgiunchedi) 05Open→03Invalid Good to close if that's expected/wanted [07:54:40] FIRING: [2x] VarnishHighThreadCount: Varnish's thread count on cp5030:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=eqsin&var-instance=cp5030 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [08:14:40] RESOLVED: VarnishHighThreadCount: Varnish's thread count on cp5030:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=eqsin&var-instance=cp5030 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [09:00:18] 06Traffic, 10Maps, 06SRE: Allow Wikimedia Maps usage on Wikidata for Firefox (Browser extension) - https://phabricator.wikimedia.org/T398588#11131131 (10TheDJ) Do extensions set a referrer header ? As volunteer im supportive in spirit. i don't think when the policy was designed any of us had considered brow... [09:02:39] 06Traffic, 10Maps, 06SRE: Allow Wikimedia Maps usage on Wikidata for Firefox (Browser extension) - https://phabricator.wikimedia.org/T398588#11131135 (10TheDJ) [09:04:00] FIRING: [2x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [09:04:03] 06Traffic, 10Maps, 06SRE: Allow Wikimedia Maps usage on Wikidata for Firefox (Browser extension) - https://phabricator.wikimedia.org/T398588#11131140 (10TheDJ) > The extension can also display maps from Wikibase instances other than Wikidata. In those cases I can still display the standard openstreetmap tile... [09:08:15] 06Traffic, 06Data-Engineering-Radar, 10HaproxyKafka, 10Observability-Logging, 13Patch-For-Review: Shutdown varnishkafka webrequest instances - https://phabricator.wikimedia.org/T393772#11131144 (10Fabfur) >>! In T393772#11129887, @ssingh wrote: > @Fabfur: I think this all done and the alerts have been re... [09:09:00] RESOLVED: [5x] PurgedHighEventLag: High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [11:58:24] 06Traffic, 06SRE, 10API Platform (RESTBase Deprecation Roadmap): Block non-browser requests that use generic user agent (UA) headers - https://phabricator.wikimedia.org/T319423#11131560 (10hashar) In deed, as part of T400119 that was done by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1181639 . The... [11:58:41] 06Traffic, 06SRE, 10API Platform (RESTBase Deprecation Roadmap): Block non-browser requests that use generic user agent (UA) headers - https://phabricator.wikimedia.org/T319423#11131565 (10hashar) →14Duplicate dup:03T400119 [11:58:53] 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11131562 (10hashar) [12:57:35] 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#11131759 (10elukey) It seems that Virtualization cannot be disabled in the BIOS processor settings, see the Web UI: {F65930889} This seems to follow something... [13:56:36] 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11131904 (10CKoerner_WMF) @joe Could I ask for a two week exemption for diff.wikimedia.org until we have our next sprint with our devs? Right now folks can... [14:29:55] 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#11132111 (10ssingh) >>! In T392851#11131759, @elukey wrote: > It seems that Virtualization cannot be disabled in the BIOS processor settings, see the Web UI: >... [14:32:26] 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#11132121 (10elukey) @wiki_willy hi! We stumbled upon an issue in the Dell BIOS configs, namely it doesn't seem possible in this configuration to disable the Pro... [14:48:25] 06Traffic, 10DNS, 06SRE, 10wikimediafoundation.org, 07IPv6: wikimediafoundation.org does not support IPv6 - https://phabricator.wikimedia.org/T403269#11132222 (10ssingh) `wikimediafoundation.org`'s `A` record is set in our zone files, while `techblog` is a `CNAME` to `techblog-wikimedia-org.go-vip.net.`... [15:08:19] 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11132283 (10Vgutierrez) >>! In T400119#11131904, @CKoerner_WMF wrote: > @joe Could I ask for a two week exemption for diff.wikimedia.org until we have our... [15:34:22] 06Traffic, 10Maps, 06SRE: Allow Wikimedia Maps usage on Wikidata for Firefox (Browser extension) - https://phabricator.wikimedia.org/T398588#11132405 (10ssingh) Thanks for the feedback, @TheDJ. It's been a while since I wrote a Firefox extension so I may be mistaken if a referrer header can be set (that is,... [15:50:05] 06Traffic, 06SRE: Setting up Wikimedia Trust and Safety Help Center with Zendesk product: Seeking Guidance on host mapping - https://phabricator.wikimedia.org/T400952#11132472 (10ssingh) Hi @JAbrams: Sorry for the lack of a response. This and a bunch of other related tickets were not addressed and we are makin... [16:22:57] 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11132534 (10Rohitverma9625) Hello everyone, I'm a contributor to the Wikimedia Commons Android App. The Commons app recently started facing some API issues... [16:26:31] 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11132540 (10CDanis) >>! In T400119#11132534, @Rohitverma9625 wrote: > Hello everyone, I'm a contributor to the Wikimedia Commons Android App. The Commons a... [16:35:59] 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11132563 (10Rohitverma9625) Hi @CDanis, thanks for the quick response. We are facing this issue for some URLs as other parts just fetch data correctly. One... [16:46:50] 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11132642 (10Rohitverma9625) This is one more URL that fails with same error: <-- 403 https://upload.wikimedia.org/wikipedia/commons/thumb/d/d9/Beautiful_C... [16:49:40] 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11132651 (10CKoerner_WMF) >>! In T400119#11132283, @Vgutierrez wrote: >>>! In T400119#11131904, @CKoerner_WMF wrote: >> @joe Could I ask for a two week exe... [17:05:42] 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11132702 (10CDanis) Thank you, that helps a lot. I notice that there's two different HTTP clients used inside the app -- and only one of them sets User-Ag... [17:10:46] 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11132723 (10Rohitverma9625) > I notice that there's two different HTTP clients used inside the app -- and only one of them sets User-Agent on outbound requ... [17:14:55] 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#11132747 (10RobH) I'll open a ticket with support about this next week and followup with both support and our account team. [17:22:26] 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#11132767 (10wiki_willy) Thanks @RobH. Our account team has changed quite a bit, but you can follow up with Hossam and Dawn after creating the support ticket >... [17:29:52] 06Traffic, 06SRE, 10Wikidata, 10Wikidata-Query-Service: Find a solution for SPARQL federation that is blocked by stricter user agent policy enforcement - https://phabricator.wikimedia.org/T402959#11132802 (10CDanis) Hi @Lydia_Pintscher , SRE can make some exception here. It seems warranted given the statu... [17:44:58] 06Traffic, 10Beta-Cluster-Infrastructure: Add allowlist to make poking holes in abuse_networks:blocked_nets:networks easier - https://phabricator.wikimedia.org/T393481#11132887 (10bd808) The business logic that needs to be changed has moved recently as part of a larger project to improve blocking capabilities... [20:10:43] 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11133146 (10tfmorris) Note that returning a plain text error message to a JSON API request is going to make it very likely to get swallowed by a JSON parse... [20:27:36] 06Traffic, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: [Rollout Phase 1] Implement redirect-less mobile routing and enable for wikitech.wikimedia.org - https://phabricator.wikimedia.org/T401595#11133178 (10Jdlrobson-WMF) I need a bit more time to read through what your proposing (and reminding mys... [20:28:05] 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11133180 (10Tgr) >>! In T400119#11133146, @tfmorris wrote: > Note that returning a plain text error message to a JSON API request is going to make it very... [21:53:29] 06Traffic, 10Beta-Cluster-Infrastructure: Add allowlist to make poking holes in abuse_networks:blocked_nets:networks easier - https://phabricator.wikimedia.org/T393481#11133324 (10bd808) I wondered if it all might be as easy as adding another section to the `abuse_networks` config that would end up generating... [23:17:06] 06Traffic, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: [Rollout Phase 1] Implement redirect-less mobile routing and enable for wikitech.wikimedia.org - https://phabricator.wikimedia.org/T401595#11133443 (10Krinkle) Feel free to review if you have the time, though note that Tim is my main collab on...