[00:48:00] 10Traffic, 10Platform Engineering, 10SRE, 10cloud-services-team (Kanban): Get platform engineering team green light for Cloud NAT to wikis change - https://phabricator.wikimedia.org/T273738 (10tstarling) I commented on the parent task. [07:27:59] 10Traffic, 10Commons, 10SRE, 10Patch-For-Review: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10Joe) According to data in turnilo, the flow of requests has gone down significantly after the app authors fixed the issue... [07:37:21] 10Traffic, 10Commons, 10SRE, 10Patch-For-Review: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10Sdkb) Just a quick heads up, apparently //Vice// found this phab ticket worthy of an article: https://www.vice.com/en/arti... [08:33:04] https://engineering.skroutz.gr/blog/uncovering-a-24-year-old-bug-in-the-linux-kernel/ [09:51:17] 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10jcrespo) I've setup higher MTU on backup1002 and backup2001 as per @ayounsi suggestion, and will do a bac... [10:02:09] 10Traffic, 10DNS, 10SRE: Apple Business Manager: verify ownership of wikimedia.org - https://phabricator.wikimedia.org/T274592 (10Vgutierrez) p:05Triage→03Medium [10:12:42] 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10jcrespo) It is very early to say, but if it has any impact at all, so far it looks negative (codfw->eqiad... [10:16:36] 10Traffic, 10DNS, 10SRE, 10Patch-For-Review: Apple Business Manager: verify ownership of wikimedia.org - https://phabricator.wikimedia.org/T274592 (10Vgutierrez) I've created the patch that adds the TXT record (https://gerrit.wikimedia.org/r/c/operations/dns/+/663794), could you review it @bblack? [13:16:52] networking is hard :-) [13:58:05] protocols are hard :) [13:59:06] ema would love that bug write-up using systemtap for debugging purposes :) [14:26:52] I've also been meaning to write something about bpftrace, but I've forgotten most of it since I last used it [16:17:00] 10Traffic, 10Commons, 10SRE, 10Patch-For-Review: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10Mvolz) >>! In T273741#6820298, @Joe wrote: >>>! In T273741#6816099, @Joe wrote: >>>>! In T273741#6815874, @Majavah wrote:... [16:37:25] 10Traffic, 10Commons, 10SRE, 10Patch-For-Review: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10Dzahn) >>! In T273741#6826701, @Mvolz wrote: > The sort of suggests the developer community needs an example image to use...