[07:52:48] 10Traffic, 10Community-Relations, 10Operations, 10Performance, and 2 others: Sometimes pages load slowly for users routed to the Amsterdam data center (due to some factor outside of Wikimedia cluster) - https://phabricator.wikimedia.org/T226048 (10Zache) Reported again in [[ https://fi.wikipedia.org/wiki/W... [07:58:24] 10Traffic, 10Community-Relations, 10Operations, 10Performance, and 2 others: Sometimes pages load slowly for users routed to the Amsterdam data center (due to some factor outside of Wikimedia cluster) - https://phabricator.wikimedia.org/T226048 (10ArielGlenn) Can we get approximate times for these last rep... [08:14:29] 10Traffic, 10Community-Relations, 10Operations, 10Performance, and 2 others: Sometimes pages load slowly for users routed to the Amsterdam data center (due to some factor outside of Wikimedia cluster) - https://phabricator.wikimedia.org/T226048 (10Ejs-80) @ArielGlenn, two fiwiki users reported about this b... [08:21:00] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 2 others: Thumbnail rendering of complex SVG file leads to Error 500 or Error 429 instead of Error 408 - https://phabricator.wikimedia.org/T226318 (10ema) Just bumped into another very frequent one: ` 08:18:54 ema@cp5005.eqsin.wmnet:~ $... [08:21:41] 10Traffic, 10Operations, 10Performance-Team: Send peering requests to AS with the worst TTFB - https://phabricator.wikimedia.org/T219486 (10Gilles) a:05Gilles→03ayounsi [08:44:02] 10Traffic, 10Community-Relations, 10Operations, 10Performance, and 2 others: Sometimes pages load slowly for users routed to the Amsterdam data center (due to some factor outside of Wikimedia cluster) - https://phabricator.wikimedia.org/T226048 (10ArielGlenn) The 03-03:35 incidents were likely related to n... [08:56:48] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 2 others: Thumbnail rendering of complex SVG file leads to Error 500 or Error 429 instead of Error 408 - https://phabricator.wikimedia.org/T226318 (10Gilles) I've seen this one stuck in poolcounter throttling for a while, it's definitely... [09:44:45] 10Traffic, 10Operations: rack/setup/install ganeti400[123] - https://phabricator.wikimedia.org/T226444 (10akosiaris) 05Open→03Stalled p:05Normal→03Low OK, good to know. Moving to Low priority and Stalled status until then. [10:21:29] 10Traffic, 10Operations, 10CommRel-Specialists-Support (Apr-Jun-2019), 10Performance, and 2 others: Sometimes pages load slowly for users routed to the Amsterdam data center (due to some factor outside of Wikimedia cluster) - https://phabricator.wikimedia.org/T226048 (10Qgil) [10:38:49] 10Traffic, 10Operations, 10CommRel-Specialists-Support (Apr-Jun-2019), 10Performance, and 2 others: Sometimes pages load slowly for users routed to the Amsterdam data center (due to some factor outside of Wikimedia cluster) - https://phabricator.wikimedia.org/T226048 (10Elitre) >>! In T226048#5279710, @Kri... [12:17:04] 10Traffic, 10Operations, 10CommRel-Specialists-Support (Apr-Jun-2019), 10Performance, and 2 others: Sometimes pages load slowly for users routed to the Amsterdam data center (due to some factor outside of Wikimedia cluster) - https://phabricator.wikimedia.org/T226048 (10Trizek-WMF) >>! In T226048#5288560,... [14:05:56] 10netops, 10Operations: RPKI Validation - https://phabricator.wikimedia.org/T220669 (10ayounsi) All our San Francisco POP now have a `validation-state` on its received prefixes. Next step is to push it to all the sites. [14:17:47] 10Traffic, 10Operations, 10Patch-For-Review: Replace Varnish backends with ATS on cache upload nodes in codfw - https://phabricator.wikimedia.org/T226637 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts: ` ['cp2002.codfw.wmnet'] ` The log can be found in `... [14:48:18] 10Traffic, 10Operations, 10Patch-For-Review: Replace Varnish backends with ATS on cache upload nodes in codfw - https://phabricator.wikimedia.org/T226637 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts: ` ['cp2002.codfw.wmnet'] ` The log can be found in `... [14:55:12] I suspect I'll be ~3-5 late for meeting by the time I round up everything here and get back in business [14:56:01] bblack: ack! [15:00:34] ack [15:10:33] 10Traffic, 10Operations, 10CommRel-Specialists-Support (Apr-Jun-2019), 10Performance, and 2 others: Sometimes pages load slowly for users routed to the Amsterdam data center (due to some factor outside of Wikimedia cluster) - https://phabricator.wikimedia.org/T226048 (10Elitre) To clarify, Trizek isn't ask... [15:12:27] 10Traffic, 10Operations, 10Patch-For-Review: Replace Varnish backends with ATS on cache upload nodes in codfw - https://phabricator.wikimedia.org/T226637 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts: ` ['cp2002.codfw.wmnet'] ` The log can be found in `... [16:16:42] 10Traffic, 10Operations, 10Patch-For-Review: Replace Varnish backends with ATS on cache upload nodes in codfw - https://phabricator.wikimedia.org/T226637 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2002.codfw.wmnet'] ` and were **ALL** successful. [17:26:27] 10netops, 10Analytics, 10Operations, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10Ottomata) [21:39:06] 10Traffic, 10Operations: mobile commons GET dying in Varnish layer(?) under oddly specific conditions - https://phabricator.wikimedia.org/T226776 (10CDanis) [21:56:21] 10Traffic, 10Operations: mobile commons GET dying in Varnish layer(?) under oddly specific conditions - https://phabricator.wikimedia.org/T226776 (10CDanis) So, it looks like that this 500 did in fact come from the application layer... but shouldn't we still be getting more response headers from the edge?