[00:27:03] 10Traffic, 10DNS, 10Mail, 10Operations: SPF for Greenhouse - https://phabricator.wikimedia.org/T189065#4029896 (10tstarling) [00:42:08] why you banned me? [02:08:04] 10Traffic, 10netops, 10Operations, 10ops-eqsin: replace eqsin SFP-T/SFP+ - https://phabricator.wikimedia.org/T188923#4030533 (10Papaul) 05Open>03Resolved Complete [05:51:34] 10Traffic, 10Analytics, 10Operations: Investigate and fix odd uri_host values - https://phabricator.wikimedia.org/T188804#4020291 (10Nuria) Ok, was about to say same, this is basically reproducible with a curl in which you override host header. [08:57:18] !log cp3033: restart varnish-be, backend connections piling up (~12k) [08:57:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:09:51] 10Traffic, 10Operations, 10Performance: Resources and pages occasionally take seconds to respond or fail - https://phabricator.wikimedia.org/T189085#4031273 (10jcrespo) This has been handled by the traffic engineers and no further problems should happen in the next hours/days (apparently one edge server was... [14:19:58] 10Traffic, 10DNS, 10Mail, 10Operations: DNS Change for GreenHouse - https://phabricator.wikimedia.org/T103893#4031999 (10faidon) [14:20:00] 10Traffic, 10DNS, 10Mail, 10Operations: SPF for Greenhouse - https://phabricator.wikimedia.org/T189065#4029896 (10faidon) This has been discussed in bigger requests a couple of times before (T103893, T84201) for Greenhouse specfically, plus a bunch of other times for other third-party services. The TL;DR i... [15:10:47] 10Traffic, 10netops, 10Operations, 10ops-eqsin: Setup eqsin RIPE Atlas anchor - https://phabricator.wikimedia.org/T179042#4032161 (10faidon) I believe this was blocked until today on an SFP replacement (T188923). It seems that the IP of the Atlas is responding now, and we even receive an SSH banner. So I j... [15:10:59] 10Traffic, 10netops, 10Operations, 10ops-eqsin: Setup eqsin RIPE Atlas anchor - https://phabricator.wikimedia.org/T179042#4032163 (10faidon) [15:41:44] 10Traffic, 10Operations, 10Performance: Resources and pages occasionally take seconds to respond or fail - https://phabricator.wikimedia.org/T189085#4031089 (10ema) The host was close to its weekly varnish-be scheduled restart, and I've handled it by manually restarting the varnish backend instance. We used... [15:44:53] 10Traffic, 10Operations, 10Performance: Resources and pages occasionally take seconds to respond or fail - https://phabricator.wikimedia.org/T189085#4031089 (10BBlack) I suspect this ticket, the above-referenced T175803, and T181315 are all inter-related or possibly pointing at the same underlying issue, jus... [16:04:01] 10Traffic, 10Operations, 10ops-eqsin: cp5006 unresponsive - https://phabricator.wikimedia.org/T187157#4032343 (10RobH) There was a bit of confusion on this. I've opened a self dispatch (SR961821650) to try to get the part sent out. However, as its the first Singapore dispatch, it will likely fail. The Net... [19:32:07] 10Traffic, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar): Varnish HTTP response from app servers taking 160s (only 0.031s inside Apache) - https://phabricator.wikimedia.org/T181315#4033104 (10Krinkle) [19:33:06] 10Traffic, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar): Varnish HTTP response from app servers taking 160s (only 0.031s inside Apache) - https://phabricator.wikimedia.org/T181315#3786217 (10Krinkle) [19:33:33] 10Traffic, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar): Varnish HTTP response from app servers taking 160s (only 0.031s inside Apache) - https://phabricator.wikimedia.org/T181315#3786217 (10Krinkle) [20:51:58] 10Traffic, 10Operations, 10Patch-For-Review: Content purges are unreliable - https://phabricator.wikimedia.org/T133821#4033339 (10daniel) [22:54:23] 10Traffic, 10Operations, 10wikitech.wikimedia.org, 10cloud-services-team (Kanban): How to purge misc-web varnishes for wikitech changes? - https://phabricator.wikimedia.org/T189168#4033759 (10Andrew) @bblack, I suspect you're the one who would know an answer for this [22:54:27] 10Traffic, 10Operations, 10wikitech.wikimedia.org, 10cloud-services-team (Kanban): How to purge misc-web varnishes for wikitech changes? - https://phabricator.wikimedia.org/T189168#4033761 (10bd808) [23:36:44] 10Traffic, 10Operations, 10wikitech.wikimedia.org, 10cloud-services-team (Kanban): How to purge misc-web varnishes for wikitech changes? - https://phabricator.wikimedia.org/T189168#4033862 (10Krenair) hieradata/role/common/cache/misc.yaml:profile::cache::base::purge_multicasts: ['239.128.0.115'] hieradata/...