[04:51:48] 10Traffic: OpenSSL < 1.1.0 compatibility issues with new LE issuance chain - https://phabricator.wikimedia.org/T283165 (10Marostegui) [04:52:21] 10Traffic, 10observability, 10User-fgiunchedi: Port traffic/netops grafana alerts to AM - https://phabricator.wikimedia.org/T282806 (10Marostegui) [07:14:47] 10netops, 10SRE, 10observability: Create RIPE Atlas measurements against our authoritative DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10ayounsi) Indeed the nerd-sniping gravitational pull is strong on that one and deserves its own task. In the short term, understanding and contro... [09:05:46] 10netops, 10SRE, 10observability: Create RIPE Atlas measurements against our authoritative DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10cmooney) Indeed some fascinating results there alright thanks for sharing Chris. While I agree with Arzhel, these results do serve to remind me... [09:18:19] 10Acme-chief, 10Traffic: Let's Encrypt chain size increase - https://phabricator.wikimedia.org/T283061 (10Marostegui) [12:29:51] 10netops, 10Continuous-Integration-Infrastructure, 10serviceops: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ) - https://phabricator.wikimedia.org/T283582 (10Marostegui) p:05Triage→03Medium Tagging also #netops in case they can help out [13:38:14] 10netops, 10Continuous-Integration-Infrastructure, 10serviceops: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ) - https://phabricator.wikimedia.org/T283582 (10ayounsi) I'd defer to DCops mgmt is connected to un-managed switches, so we don't have much visibility on this side. Either the... [13:41:29] 10netops, 10Continuous-Integration-Infrastructure, 10DC-Ops, 10serviceops: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ) - https://phabricator.wikimedia.org/T283582 (10Marostegui) [14:48:53] 10netops, 10SRE, 10netbox: Stage drmrs in Netbox - https://phabricator.wikimedia.org/T283594 (10ayounsi) p:05Triage→03Medium [14:49:45] 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10jcrespo) You may be upon something. UDP transmission speed seems equivalent in both ways: (there was the... [14:56:18] 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10jcrespo) Compare if I run the above with TCP (minus the -u), where the difference can be appreciated in o... [15:04:37] 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10jcrespo) > @jcrespo, if you are ok with the proposed tests can you advise a newbie on what the best way t... [15:23:50] 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10jcrespo) backups hosts happen to have a generous scratching area, I have left on backup1003:/srv and back... [15:32:01] 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10cmooney) Ok thanks for the update, and confirmation that it's ok to to add those temp iptables rules if n... [15:50:40] 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10jcrespo) > I might run one or two tests between these hosts also if that is ok. Absolutely no problem, o... [15:52:25] 10netops, 10SRE, 10netbox, 10Patch-For-Review: Stage drmrs in Netbox - https://phabricator.wikimedia.org/T283594 (10Volans) I would suggest to start adding things with `Status=Planned` and adapt our automation and tooling to exclude them if they're not doing that already. [16:45:17] 10Traffic: RIPE Atlas monitoring of reachability & latency towards anycasted Wikidough IP - https://phabricator.wikimedia.org/T283614 (10CDanis) [16:47:09] 10netops, 10SRE, 10observability: Create RIPE Atlas measurements against our authoritative DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10CDanis) Totally agree with all of the above! Filed a placeholder task: {T283614} [18:09:12] 10Traffic, 10SRE, 10Patch-For-Review: Offer Wikidough as an anycasted service - https://phabricator.wikimedia.org/T283027 (10cmooney) [18:09:14] 10Traffic, 10SRE: RIPE Atlas monitoring of reachability & latency towards anycasted Wikidough IP - https://phabricator.wikimedia.org/T283614 (10cmooney) [18:09:55] 10netops, 10SRE, 10observability: Create RIPE Atlas measurements against our authoritative DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10cmooney) Great, I'm working with Sukhbir to get that online in next day or two so we should be able to progress it any time after that. [18:40:31] 10Traffic, 10Advanced Mobile Contributions, 10SRE, 10User-Joe: AMC – Opt-in for logged out users - https://phabricator.wikimedia.org/T215624 (10Jdlrobson) @ovasileva @phuedx is it useful to still have this ticket open? If so, should it be in #readers-web-backlog or tracking? [19:47:51] 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10cmooney) Thanks Jamie I've been digging into this. Looking at the PCAPs, and even the iperf cli output,... [20:08:07] 10netops, 10SRE, 10observability: Create RIPE Atlas measurements against our authoritative DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10jbond) > ! And from a cursory glance we can spot a number of interesting things that I am holding myself back from getting nerdsniped on: this i...