[07:51:15] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10ema) Vanilla 6.0.1 was performing worse than 5.1.3 and similarly to 6.0.7 when we tested it in January: >>! In T264398#673141... [07:52:56] 10netops, 10SRE: mr1 port utilization alerts shouldn't mention hash page in their IRC logs - https://phabricator.wikimedia.org/T281055 (10fgiunchedi) >>! In T281055#7034863, @CDanis wrote: > Moving to AM sounds good to me. But if needed, in the interim we could change the magic string we use in `check_librenm... [08:06:02] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10Gilles) It's great that we narrowed this down and confirmed it, excellent work! The change's claimed behaviour is definitely c... [12:39:05] 10Traffic, 10MediaWiki-General, 10SRE, 10Browser-Support-Apple-Safari: File:Chessboard480.svg WEBP thumbnail version not visible on safari when size is fixed at 208px - https://phabricator.wikimedia.org/T280439 (10Daimona) 05Open→03Resolved a:03ema Working as expected now, thank you! [14:29:52] codfw-C2 time! :) [14:30:01] (well, pre-downtime time, anyways) [14:31:14] dns2001, lvs2009, cp2035, cp2036 [14:33:59] bblack: I'm around if needed! [14:34:34] I'll just do it, it's a good morning brain wakeup exercise :) [14:35:49] ack then I'll continue being outraged at the difference between commit messages and patch contents [14:36:21] ema: lol, varnish patches? [14:36:35] how did you know?!? [14:37:12] 🙃 [14:38:51] "varnishd: fix one typo in a comment" -- 82 files changed, 1807 insertions(+), 3490 deletions(-) [14:39:48] Please note that comparing this change using diff -u does not make much sense, I believe the simplicity of the reshuffling change becomes clear when looking at the old and new code side-by-side [14:42:57] at least there is _a_ commit message, half of all imagemagick commits only have "..." as the commit message, like https://github.com/ImageMagick/ImageMagick6/commit/f2a5d8646472e4bc5cd0a18546db0a29d3b7795c [14:43:55] I'm not entirely sure what's worse between silence and lies [14:44:09] "..." is developer-speak for "I accidentally fixed what are probably 3 serious security bugs that nobody noticed yet, and I hope nobody notices now" [14:44:13] I think ImageMagick wins here, it provides RCEaaS very well [14:54:09] 10netops, 10SRE, 10ops-codfw: Multiple host down alerts from rack C2 - https://phabricator.wikimedia.org/T279457 (10BBlack) Traffic stuff (lvs/cp/dns) is depooled, downtimed, and ready for the network fixups. [16:32:34] 10netops, 10SRE, 10ops-codfw: Multiple host down alerts from rack C2 - https://phabricator.wikimedia.org/T279457 (10BBlack) Note to our future selves: we forgot to consider the cross-row LVS connections in this downtime: lvs2008 and lvs2010 do not live in row C at all, but had cross-row connections via C2 to... [16:52:08] moritzm: the best part is that the GitHub title of the committer is "ImageMagick Principle [sic] Architect" [17:55:44] 10Traffic, 10SRE, 10Patch-For-Review: Implement machine-local forwarding DNS caches - https://phabricator.wikimedia.org/T171498 (10mmodell) [18:37:55] 10netops, 10SRE, 10ops-codfw: Multiple host down alerts from rack C2 - https://phabricator.wikimedia.org/T279457 (10BBlack) Traffic lvs/cp/dns are all repooled, un-downtimed, and green. Waiting until the other C2 hosts are fully reconfigured (network ports) before re-pooling codfw at the public traffic level. [21:32:45] 10Traffic, 10SRE, 10decommission-hardware: decommission cescout1001.eqiad.wmnet - https://phabricator.wikimedia.org/T275696 (10ssingh) p:05Medium→03High [22:22:22] 10netops, 10SRE, 10ops-codfw: Multiple host down alerts from rack C2 - https://phabricator.wikimedia.org/T279457 (10Papaul) switch replace, onsite work complete and Netbox updated. Will be shipping the faulty switch tomorrow. [22:22:40] 10netops, 10SRE, 10ops-codfw: Multiple host down alerts from rack C2 - https://phabricator.wikimedia.org/T279457 (10Papaul) p:05High→03Low