[02:48:17] 06Traffic, 06Commons: Varnish cache purges of file assets sometimes seem to be failing, at least sometimes - https://phabricator.wikimedia.org/T435283#12249763 (10Bawolff) > IMHO, the right fix for this is to fix the MW side of this to purge the base URL of the original, without the excess params (I assume it... [05:22:37] 06Traffic, 10Internet-Archive, 06SRE, 07Regression: archive.org cannot reach Wikimedia - https://phabricator.wikimedia.org/T435743#12249997 (10Cyberpower678) >>! In T435743#12246795, @TheDJ wrote: > Maybe @cyberpower678 can help find the right contacts I’ve relayed this to the engineers. [07:59:20] 06Traffic, 06SRE, 10Thumbor, 07affects-Kiwix-and-openZIM: MWoffliner scrapes slowed down by Thumbor failure throttling 429s - https://phabricator.wikimedia.org/T304814#12250223 (10SLyngshede-WMF) The infrastructure around thumbnails have been significantly reworked since 2022. While the 429 remains in plac... [08:08:10] 06Traffic, 10Internet-Archive, 06SRE, 07Regression: archive.org cannot reach Wikimedia - https://phabricator.wikimedia.org/T435743#12250256 (10AlexisJazz) [08:15:53] 06Traffic, 10Internet-Archive, 06SRE, 07Regression: archive.org cannot reach Wikimedia - https://phabricator.wikimedia.org/T435743#12250269 (10Fabfur) a:03Fabfur [08:34:24] 06Traffic, 07Sustainability (Incident Followup): Rate limiting for hotlinked images - https://phabricator.wikimedia.org/T317799#12250331 (10SLyngshede-WMF) @RLazarus / @CDanis do we want to keep this task open, or do we now view this as "mostly under control"? [08:36:49] 06Traffic, 10RESTBase-API, 07Documentation: I am hitting a rate limit on REST API endpoint - https://phabricator.wikimedia.org/T307610#12250333 (10SLyngshede-WMF) 05Stalled→03Resolved a:03SLyngshede-WMF No follow up for three years. Closing. Documentation for rate-limiting as of today: * Generic R... [08:49:54] 10netops, 06Infrastructure-Foundations, 06SRE: Missing series for BGP session_state from eqiad CRs since upgrade to 23.4R2-S8.7 - https://phabricator.wikimedia.org/T435909 (10cmooney) 03NEW p:05Triage→03Medium [08:52:20] 10netops, 06Infrastructure-Foundations, 06SRE: Missing series for BGP session_state from eqiad CRs since upgrade to 23.4R2-S8.7 - https://phabricator.wikimedia.org/T435909#12250416 (10cmooney) [08:53:07] 10netops, 06Infrastructure-Foundations, 06SRE: Missing series for BGP session_state from eqiad CRs since upgrade to 23.4R2-S8.7 - https://phabricator.wikimedia.org/T435909#12250417 (10cmooney) [09:23:59] 06Traffic, 06SRE, 13Patch-For-Review: netconsole being used for cache hosts? - https://phabricator.wikimedia.org/T427646#12250507 (10MoritzMuehlenhoff) >>! In T427646#11967900, @ssingh wrote: > I meant we set `profile::netconsole::client::ensure: absent` in `hieradata/role/common/cache/upload.yaml` so it sho... [09:32:40] 10netops, 06Infrastructure-Foundations, 10Observability-Metrics, 06SRE: Expand blackbox icmp probes to ping specific router interfaces/circuits - https://phabricator.wikimedia.org/T435855#12250560 (10cmooney) [09:35:57] 10netops, 06Infrastructure-Foundations, 10Observability-Metrics, 06SRE: Expand blackbox icmp probes to ping specific router interfaces/circuits - https://phabricator.wikimedia.org/T435855#12250572 (10cmooney) [09:43:10] 06Traffic, 10Internet-Archive, 06SRE, 07Regression: archive.org cannot reach Wikimedia - https://phabricator.wikimedia.org/T435743#12250594 (10Fabfur) I've noticed that archiving a new page from Commons, Web Archive completes the task successfully: it fetches the page and the image (and thumbnails) as expe... [10:38:03] 06Traffic: Generate some test data for cache hosts - https://phabricator.wikimedia.org/T435918 (10Fabfur) 03NEW [10:38:48] 06Traffic, 05WMF-NDA: Generate some test data for cache hosts - https://phabricator.wikimedia.org/T435918#12250811 (10Fabfur) [10:53:15] 06Traffic, 13Patch-For-Review: Kapow score support on cache hosts - https://phabricator.wikimedia.org/T435799#12250930 (10Fabfur) [11:05:48] 06Traffic, 13Patch-For-Review: Kapow score support on cache hosts - https://phabricator.wikimedia.org/T435799#12251032 (10Fabfur) [11:08:50] 06Traffic, 13Patch-For-Review: Kapow score support on cache hosts - https://phabricator.wikimedia.org/T435799#12251044 (10Fabfur) 05Open→03In progress p:05Triage→03Medium [11:48:52] 06Traffic, 10Internet-Archive, 06SRE, 07Regression: archive.org cannot reach Wikimedia - https://phabricator.wikimedia.org/T435743#12251206 (10Cyberpower678) The Wayback Machine engineers have responded with the following: The hostname en.wikipedia.org was classified as "dead host". This happens when we ge... [11:57:48] 06Traffic, 10Internet-Archive, 06SRE, 07Regression: archive.org cannot reach Wikimedia - https://phabricator.wikimedia.org/T435743#12251241 (10Cyberpower678) The takeaway here is the live web checker seemed to have gotten a large number of 503s or 999s. [12:12:49] 06Traffic, 10Maps (Kartographer): Geoshape rate limiting breaks the live maps - https://phabricator.wikimedia.org/T435741#12251275 (10TheDJ) The wgKartographerLiveData fragment that is loaded by Kartographer is: P96250 [12:15:28] 10netops, 06Infrastructure-Foundations, 10Observability-Metrics, 06SRE, 13Patch-For-Review: Expand blackbox icmp probes to ping specific router interfaces/circuits - https://phabricator.wikimedia.org/T435855#12251284 (10cmooney) [12:29:24] 10netops, 06Infrastructure-Foundations, 10Observability-Metrics, 06SRE, 13Patch-For-Review: Expand blackbox icmp probes to ping specific router interfaces/circuits - https://phabricator.wikimedia.org/T435855#12251323 (10cmooney) So thinking about this further, after rubber-ducking it with Hugh earlier, I... [13:18:32] 06Traffic, 06Commons, 10MediaWiki-File-management, 06MediaWiki-Core-Platform-Team (Kanban): Varnish cache purges of file assets sometimes seem to be failing, at least sometimes - https://phabricator.wikimedia.org/T435283#12251542 (10OWresch-WMF) [13:19:13] 06Traffic, 06Commons, 06DC-Ops, 10MediaWiki-Core-Revision-backend, and 3 others: ESAMS and others serving older revisions of overwritten files - https://phabricator.wikimedia.org/T425216#12251545 (10Krinkle) The report date (2 May) aligns well with after the first rollout of this to Commons: >>! In T41433... [13:19:17] 06Traffic, 06Commons, 10MediaWiki-File-management, 06MediaWiki-Core-Platform-Team (Kanban): Varnish cache purges of file assets sometimes seem to be failing, at least sometimes - https://phabricator.wikimedia.org/T435283#12251551 (10Krinkle) →14Duplicate dup:03T425216 [13:19:23] 06Traffic, 06Commons, 06DC-Ops, 10MediaWiki-Core-Revision-backend, and 3 others: ESAMS and others serving older revisions of overwritten files - https://phabricator.wikimedia.org/T425216#12251548 (10Krinkle) [13:20:36] 06Traffic, 06Commons, 10MediaWiki-File-management, 06SRE, 06MediaWiki-Core-Platform-Team (Kanban): Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12251556 (10Krinkle) [13:20:46] 06Traffic, 06Commons, 10MediaWiki-File-management, 06SRE, 06MediaWiki-Core-Platform-Team (Kanban): Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12251559 (10Krinkle) p:05Triage→03High a:03Krinkle [13:23:29] 10netops, 06SRE Observability: Prometheus rule evaluation failures (instance titan1001) - https://phabricator.wikimedia.org/T435494#12251579 (10tappof) I can confirm it's working properly now. Thank you. [13:23:43] 06Traffic, 06Commons, 10MediaWiki-File-management, 06SRE, 06MediaWiki-Core-Platform-Team (Kanban): Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12251581 (10Krinkle) Varnish strips query parameters for GET requests to upl... [13:26:17] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06ServiceOps, and 2 others: Migrate Wikikube k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420436#12251587 (10Jelto) [13:26:44] Puppet is temporarily disabled on cp:upload nodes [13:38:15] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12251659 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1003 for host dns2005.wikimedia.org with OS trixie [13:48:16] 06Traffic, 10bot-traffic-requests, 06Future-Audiences, 06MediaWiki-Core-Platform-Team (Radar), 07SEO: Link preview from iMessage on macOS blocked - https://phabricator.wikimedia.org/T432373#12251763 (10SLyngshede-WMF) @Krinkle Did it work? [13:49:03] 06Traffic, 06SRE: netconsole being used for cache hosts? - https://phabricator.wikimedia.org/T427646#12251765 (10SLyngshede-WMF) 05Open→03Resolved p:05Triage→03Low a:03SLyngshede-WMF [13:50:17] Puppet is re-enabled for cp:upload [13:52:07] 06Traffic, 10Internet-Archive, 06SRE, 07Regression: archive.org cannot reach Wikimedia - https://phabricator.wikimedia.org/T435743#12251774 (10Fabfur) Thanks for the update! Can you still confirm the archive.org is remained the same and/or no changes has been made regard to the network from where these req... [13:52:46] 06Traffic, 10Thumbor, 07affects-Kiwix-and-openZIM, 07Wikimedia-Performance-recommendation: Thumbor response lacks "ETag" HTTP header (fixed once expired from Varnish and re-filled from Swift) - https://phabricator.wikimedia.org/T295556#12251777 (10Krinkle) [13:56:45] 06Traffic, 10Internet-Archive, 06SRE, 07Regression: archive.org cannot reach Wikimedia - https://phabricator.wikimedia.org/T435743#12251805 (10Cyberpower678) I'm not aware of any network changes. I'll double check. [14:06:44] 06Traffic, 06SRE: Investigate / Fix upload.wikimedia.org lack of Cache-Control headers - https://phabricator.wikimedia.org/T431621#12251894 (10Krinkle) > We do emit `ETag` […] See also {T295556} and {T256217}, which together make it hard or impossible for clients to avoid re-downloading images when they reval... [14:08:43] 06Traffic, 06Commons, 10MediaWiki-File-management, 06SRE: Thumbnail urls should be versioned and sent with Cache-Control max-age headers - https://phabricator.wikimedia.org/T19577#12251911 (10Krinkle) [14:43:45] Hi, an update that I'm planning to switch eswiki tomorrow, that'll be mostly magru but next week I'm going to all wikis except enwiki which would mean half of traffic will start to shift. Do you need more time? [14:44:20] Amir1: we will be discussing this in the Traffic meeting today and can follow up after that :> [14:44:51] sure thing. Note that it'll take a while for things to shift [14:50:14] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12252265 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1003 for host dns2005.wikimedia.org with OS trixie completed: - dns2005 (**WARN**) - Downtimed on Icinga/Ale... [15:58:07] Amir1: We're good for moving eswiki tomorrow [16:16:55] \o. [16:16:58] \o/ [16:22:38] 06Traffic, 06Data-Persistence, 06MediaWiki-Media-Platform-Team, 13Patch-For-Review: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465#12252840 (10Ladsgroup) [16:46:25] 10Domains, 07HTTPS, 10DNS, 06SRE, 06Traffic-Icebox: Merge Wikipedia subdomains into one, to discourage censorship - https://phabricator.wikimedia.org/T215071#12253004 (10BCornwall) 05Open→03Declined Thanks for the clarification. Closing. [17:01:58] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12253073 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1003 for host dns2004.wikimedia.org with OS trixie [17:06:04] 10netops, 06Infrastructure-Foundations, 10Observability-Metrics, 06SRE, 13Patch-For-Review: Expand blackbox icmp probes to ping specific router interfaces/circuits - https://phabricator.wikimedia.org/T435855#12253094 (10cmooney) p:05High→03Medium The stats for the magru HE circuits are working well,... [18:23:49] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12253522 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1003 for host dns2004.wikimedia.org with OS trixie completed: - dns2004 (**WARN**) - Downtimed on Icinga/Ale... [18:49:40] 06Traffic: Wikipedia pages returning "upstream connect error or disconnect/reset before headers" - https://phabricator.wikimedia.org/T436004#12253654 (10Aklapper) [18:50:18] 06Traffic: Wikipedia pages returning "upstream connect error or disconnect/reset before headers" - https://phabricator.wikimedia.org/T436004#12253656 (10-sche) I am also encountering this on en.Wiktionary ("Original error: upstream connect error or disconnect/reset before headers. reset reason: connection timeou... [18:57:15] 06Traffic: Wikipedia and Wiktionary pages returning "upstream connect error or disconnect/reset before headers" - https://phabricator.wikimedia.org/T436004#12253688 (10-sche) [19:01:49] 06Traffic: Wikipedia and Wiktionary pages returning "upstream connect error or disconnect/reset before headers" - https://phabricator.wikimedia.org/T436004#12253719 (10-sche) Poking around, is #sre a relevant group to tag? [19:02:45] 06Traffic, 06SRE: Wikipedia and Wiktionary pages returning "upstream connect error or disconnect/reset before headers" - https://phabricator.wikimedia.org/T436004#12253722 (10-sche) [19:04:38] 06Traffic, 06SRE, 07Wikimedia-Incident: Wikipedia and Wiktionary pages returning "upstream connect error or disconnect/reset before headers" - https://phabricator.wikimedia.org/T436004#12253732 (10AntiCompositeNumber) This is a known incident being actively responded to. https://www.wikimediastatus.net/incid... [19:10:36] 06Traffic, 06SRE, 07Wikimedia-Incident: Wikipedia and Wiktionary pages returning "upstream connect error or disconnect/reset before headers" - https://phabricator.wikimedia.org/T436004#12253759 (10Jdforrester-WMF) [19:13:54] 06Traffic, 06SRE, 07Wikimedia-Incident: Wikipedia and Wiktionary pages returning "upstream connect error or disconnect/reset before headers" - https://phabricator.wikimedia.org/T436004#12253790 (10RhinosF1) [20:14:39] 06Traffic, 06DC-Ops, 10ops-drmrs: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12254073 (10RobH) DHL closed our case stating: > Hello again Rob > > The delivery station has sent the message below. Please reach out to your shipper for further help.... [20:53:28] 06Traffic, 06DC-Ops, 10ops-drmrs: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12254225 (10BCornwall) Bless you, Rob. - {F100099161, layout=link}