[07:11:56] 10Traffic, 10Operations, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs: Wikipedia iOS apps sending harmful bursts of traffic synchronized to the top of the hour, especially at 22:00 UTC - https://phabricator.wikimedia.org/T264881 (10Joe) I think some form of ratelimiting for that should be present in restbas... [07:31:54] 10netops, 10Operations, 10Patch-For-Review, 10Security, 10User-jbond: Review default ferm INPUT policy - https://phabricator.wikimedia.org/T264888 (10MoritzMuehlenhoff) >>! In T264888#6529767, @jbond wrote: >however i, like @BBlack, prefer reject to drop if possible. As such it would be nice to be good... [08:33:36] 10Traffic, 10Operations, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10Gilles) I would like to know what the Traffic team is doing or planning on doing about this investigation at the moment. Now that I've narrow... [09:07:03] 10Traffic, 10Operations, 10Performance-Team, 10Wikimedia-Incident: 15% response start regression as of 2019-11-11 (Varnish->ATS) - https://phabricator.wikimedia.org/T238494 (10Gilles) @bblack when we last discussed the subject of this task in a meeting recently, you mentioned that replacing ats-tls (the "p... [14:08:07] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10mforns) Hi @ayounsi, can you help me? I have some more questions: * What is the field that we want to extract the AS name for? I see as_src, as_d... [14:38:06] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10ayounsi) > * What is the field that we want to extract the AS name for? I see as_src, as_dst, peer_as_src, peer_as_dst? Ideally all of them, but a... [15:13:16] 10Traffic, 10Operations, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10ema) With [[ https://github.com/rakyll/hey | hey ]] on cp3052 (Varnish 5.1.3-1wm15) and cp3054 (6.0.6-1wm1) I obtained the following two late... [16:14:49] 10Traffic, 10Operations, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs: Wikipedia iOS apps sending harmful bursts of traffic synchronized to the top of the hour, especially at 22:00 UTC - https://phabricator.wikimedia.org/T264881 (10Tsevener) @CDanis we didn't remove fetches against `/api/rest_v1/page/random... [16:22:51] 10Traffic, 10Operations, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs: Wikipedia iOS apps sending harmful bursts of traffic synchronized to the top of the hour, especially at 22:00 UTC - https://phabricator.wikimedia.org/T264881 (10CDanis) >>! In T264881#6532868, @Tsevener wrote: > @CDanis we didn't remove... [16:52:23] hi all i was wondering if we have any physical hosts i can run an nmap scan from. i want to rule out he hyper visor from any performance issues. [16:54:21] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10mforns) > > What is the field that we want to extract the AS name for? I see as_src, as_dst, peer_as_src, peer_as_dst? > Ideally all of them, but... [16:59:02] jbond42: cescout1001.eqiad.wmnet is a physical host. it doesn't run anything critical per se, so feel free to use it (we use it to fetch censorship-related data) [17:01:14] sukhe: thanks ill install nmap and run the task in tmux session under my user. if it dose cause any issues just kill it [17:20:58] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10ayounsi) > Or maybe I misunderstood what needs to be done here... I assumed we want to determine whether the given IP is v4 or v6. But which IP w... [17:29:29] 10Traffic, 10Operations, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs: Wikipedia iOS apps sending harmful bursts of traffic synchronized to the top of the hour, especially at 22:00 UTC - https://phabricator.wikimedia.org/T264881 (10Tsevener) @CDanis Thanks! That response is working fine in my testing, feel... [17:32:59] 10Traffic, 10Operations, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs: Wikipedia iOS apps sending harmful bursts of traffic synchronized to the top of the hour, especially at 22:00 UTC - https://phabricator.wikimedia.org/T264881 (10CDanis) Great! Thanks @Tsevener ! [18:05:35] 10netops, 10Operations, 10Patch-For-Review, 10Security, 10User-jbond: Review default ferm INPUT policy - https://phabricator.wikimedia.org/T264888 (10jbond) no sure why but some host take a looong time ` Completed SYN Stealth Scan against 208.80.153.45 in 50099.78s (63 hosts left) Completed SYN Stealth... [18:27:21] 10Traffic, 10Operations, 10Technical-blog-posts: Blog post series: the evolution of Wikimedia's Content Delivery Network - https://phabricator.wikimedia.org/T264729 (10srodlund) It looks good to me! I am copying it over to the blog for publication next week. (Tuesday 13 Oct) [19:37:15] bblack: any chance you're around? [19:47:33] if bblack isn't around, then I think rzl has the most context? but I would appreciate a review on https://gerrit.wikimedia.org/r/c/operations/puppet/+/633229 [19:49:21] I do want to push this out today, it will very likely save us from getting paged multiple times over the weekend [20:15:38] happy to review, but all else being equal, I'll leave it for a real trafficologist