[01:03:19] 10Traffic, 10Operations, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs: Wikipedia iOS apps sending harmful bursts of traffic synchronized to the top of the hour, especially at 22:00 UTC - https://phabricator.wikimedia.org/T264881 (10Ladsgroup) I don't know if this has been considered or not and I admit I don... [01:10:23] 10Traffic, 10Operations, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs: Wikipedia iOS apps sending harmful bursts of traffic synchronized to the top of the hour, especially at 22:00 UTC - https://phabricator.wikimedia.org/T264881 (10CDanis) >>! In T264881#6527486, @Ladsgroup wrote: > I don't know if this has... [06:46:37] 10Traffic, 10Operations, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10Gilles) I've added a dropdown to pick the percentile on https://grafana.wikimedia.org/d/M7xQ_BeWk/response-time-by-host Here's what it looks... [07:45:50] 10netops, 10Operations, 10Patch-For-Review, 10Security, 10User-jbond: Review default ferm INPUT policy - https://phabricator.wikimedia.org/T264888 (10akosiaris) Overall, I am willing to test this out, couples of points though: * Since it's recommended by various standards to do the default DROP thing, w... [08:23:13] 10Traffic, 10Operations, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10ema) >>! In T264398#6524473, @Gilles wrote: > I really don't understand what I did wrong here It was never my intention to offend you, and g... [09:03:49] 10Traffic, 10Operations, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10Gilles) I don't think that this discussion is appropriate in a public forum. An email thread seems like an ok starting point, and/or a meetin... [09:45:52] 10netops, 10Operations, 10Patch-For-Review, 10Security, 10User-jbond: Review default ferm INPUT policy - https://phabricator.wikimedia.org/T264888 (10jbond) >>! In T264888#6528076, @akosiaris wrote: > Overall, I am willing to test this out, couples of points though: > > * Since it's recommended by vario... [10:02:32] 10netops, 10Operations, 10Patch-For-Review, 10Security, 10User-jbond: Review default ferm INPUT policy - https://phabricator.wikimedia.org/T264888 (10jbond) > This would also mean that a malicious actor could use us to reflect RST packets however the 40b rst packet comes at a cost of a 60b syn This is n... [10:28:53] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10ayounsi) Done! And confirmed with kafkacat, eg: `"comms": "2914:420_2914:1008_2914:2000_2914:3000_14907:4"` As well as no dr... [11:52:41] 10Traffic, 10Operations, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10mark) Hi all, I recommend we limit the conversations on this task to the technical aspects of this particular regression and its investigati... [12:13:05] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10mforns) Awesome! The size of the events has increased in about 25-30%, which is considerable, but I believe sustainable for now. When we sanitize... [12:17:09] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10ayounsi) Wow, that 's more then expected indeed! If it's an issue down the road we could think of filtering out some communities (for example only... [13:38:03] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10mforns) After discussing with the team, we think it's fine for now. If we want to add more fields or increase the sampling ratio, then we should i... [13:44:32] 10netops, 10Operations, 10Patch-For-Review, 10Security, 10User-jbond: Review default ferm INPUT policy - https://phabricator.wikimedia.org/T264888 (10BBlack) FWIW, I am in general a fan of `REJECT` over `DROP`, especially when there's not even a great obscurity argument, as is the case here. It will be... [15:06:23] 10Traffic, 10Operations, 10Technical-blog-posts: Blog post series: the evolution of Wikimedia's Content Delivery Network - https://phabricator.wikimedia.org/T264729 (10ema) >>! In T264729#6526259, @srodlund wrote: > I made some minor grammar suggestions. Can you accept / reject them Done, thank you! I chang... [16:05:39] 10netops, 10Operations, 10Patch-For-Review, 10Security, 10User-jbond: Review default ferm INPUT policy - https://phabricator.wikimedia.org/T264888 (10jbond) Thanks all quick update. I have deployed the firewall change to idp-test1001 and the scan time about 3x faster with the new rule (see below). howe... [17:19:21] bblack: just FYI after some more digging I found various corner cases and with Xio.NoX we came up with a smaller consolidation, basically only /30 /31 it's the safest option at the moment and gives us quite some gain already (-48 zonefiles) [17:19:46] all the patches are out and reviewed, the exposed surface is much smaller too,so I'll probably deploy them in a bit [17:23:36] volans: ack [17:54:04] bblack, chaomodus (et al.): coordinated/emergency dns changes are now possible and described in wikitech (see https://phabricator.wikimedia.org/T264846#6530108 for direct pointers) [17:54:23] awesome thanks [17:55:01] lmk if you have questions/comments [17:56:34] nice [17:57:49] 🎉 [18:06:44] would it be expected that puppet class "profile::cache::ssl::unified" is not used (anymore)? [18:12:27] possibly! [18:12:49] it's probably still referenced by some config-driven conditionals and just doesn't happen to be configured for those cases anymore [18:13:41] hmmm maybe not even that. in some grepping, it seems most references to the profile are just self-reference [18:13:46] s/most/all/ [18:14:07] still, I wouldn't yank it out just yet [18:14:21] $ sudo cumin 'P:cache::ssl::unified' [18:14:21] No hosts found that matches the query [18:14:31] we're due for more churn in this area in the coming quarter or two, and I don't know if we might end up re-using some parts of it [18:14:37] bblack: it was more that i wanted to make some small fixes inside it and knowing it's not used ..makes the merge so much easier :) [18:14:49] ok :) [18:14:51] lol [18:15:21] yea, i asked after the compiler came up empty..and thought i'd mention it. thanks [18:16:28] the higher-level why could be summarized as "upgrade to puppet6 before EOL of buster" :P [18:16:35] hiera() needs to go [20:46:19] and with --skip-authdns-update I think I'm able to deploy the consolidation without even having duplicate records at any time [20:52:14] merge consolidation with backward compatibility, run the cookbook with --skip-authdns-update, merge the INCLUDEs change, run authdns-update [20:52:37] and finally merge the removal of backward compatible code and re run the cookbook this time normally [21:18:52] FYI it's all done as of ~15m ago [21:46:33] 10Traffic, 10Operations, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs: Wikipedia iOS apps sending harmful bursts of traffic synchronized to the top of the hour, especially at 22:00 UTC - https://phabricator.wikimedia.org/T264881 (10LGoto) [21:52:33] 10Traffic, 10Operations, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs: Wikipedia iOS apps sending harmful bursts of traffic synchronized to the top of the hour, especially at 22:00 UTC - https://phabricator.wikimedia.org/T264881 (10JMinor) We are publishing v6.7.2 (1780) of the app as I write, which has our... [21:53:16] 10Traffic, 10Operations, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs: Wikipedia iOS apps sending harmful bursts of traffic synchronized to the top of the hour, especially at 22:00 UTC - https://phabricator.wikimedia.org/T264881 (10JMinor) a:05Dmantena→03None [21:54:26] 10Traffic, 10Operations, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs: Wikipedia iOS apps sending harmful bursts of traffic synchronized to the top of the hour, especially at 22:00 UTC - https://phabricator.wikimedia.org/T264881 (10CDanis) Really great to see this happen so quickly! Thanks so much :) I'll... [22:21:50] 10Traffic, 10Operations, 10Platform Team Initiatives (API Gateway), 10Story: Client Developer has a cookie-free API call - https://phabricator.wikimedia.org/T258748 (10eprodromou) [22:37:15] 10Traffic, 10Operations, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs: Wikipedia iOS apps sending harmful bursts of traffic synchronized to the top of the hour, especially at 22:00 UTC - https://phabricator.wikimedia.org/T264881 (10CDanis) When the 22:00 traffic spike happened today, it was a bit more impac...