[01:37:26] 10netops, 10Operations, 10Wikimedia-Incident: Investigate Juniper storm control - https://phabricator.wikimedia.org/T245192 (10Papaul) @ayounsi please see below for the configuration i just added the first interface for now. If all looks good I will add the other interfaces ` [edit interfaces ge-0/0/0 unit... [08:47:38] 10Traffic, 10Operations, 10Phabricator: Phabricator is inaccessible from Egypt: HTTP 501 error - https://phabricator.wikimedia.org/T246923 (10akosiaris) >>! In T246923#5943203, @Urbanecm wrote: >>>! In T246923#5943004, @Krenair wrote: >> If it is that I would not expect HTTP 501 responses. > >>>! In T246923... [08:48:05] 10Traffic, 10Operations, 10Phabricator: Phabricator is inaccessible from Egypt: HTTP 501 error - https://phabricator.wikimedia.org/T246923 (10akosiaris) [08:49:30] 10Traffic, 10Operations: switch to irate() instead of rate() for traffic graphs - https://phabricator.wikimedia.org/T246902 (10akosiaris) p:05Triage→03Lowest Lowest priority until it is better phrased and work on it scheduled. But +1 on premise. [08:49:55] 10Traffic, 10netops, 10Operations: BGP: Investigate isolating codfw and eqiad - https://phabricator.wikimedia.org/T246721 (10akosiaris) p:05Triage→03Medium [08:56:04] 10netops, 10Operations: can aggregated netflow data include the router it was sampled from? - https://phabricator.wikimedia.org/T246186 (10akosiaris) p:05Triage→03Medium [09:06:03] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown: Different age of history logged in and out when from the EU (but not SF office) - https://phabricator.wikimedia.org/T246185 (10akosiaris) p:05Triage→03Medium I can not reproduce this either. Is it still a problem? [12:08:46] 10netops, 10Operations, 10cloud-services-team (Kanban): CloudVPS: enable BGP in the neutron transport network - https://phabricator.wikimedia.org/T245606 (10aborrero) Fixed package is neutron-bgp-dragent_11.0.0-2~bpo9+1 and friends. [12:33:23] 10netops, 10Operations, 10cloud-services-team (Kanban): CloudVPS: enable BGP in the neutron transport network - https://phabricator.wikimedia.org/T245606 (10aborrero) hey @ayounsi do you have to enable anything in your side for BGP to work? I see something weird, I get a no route to host error here: `lang=s... [12:49:44] 10netops, 10Operations, 10cloud-services-team (Kanban): CloudVPS: enable BGP in the neutron transport network - https://phabricator.wikimedia.org/T245606 (10aborrero) The ARP reply is produced and reach my server, just I don't know yet what's going on, or which interface is this reply packet using: `lang=sh... [12:57:39] 10netops, 10Operations, 10cloud-services-team (Kanban): CloudVPS: enable BGP in the neutron transport network - https://phabricator.wikimedia.org/T245606 (10aborrero) Forget last 2 comments. I think I can assign the address to the bridge device instead of the vlan device and everything should work as expected. [13:09:20] 10netops, 10Operations, 10cloud-services-team (Kanban): CloudVPS: introduce filtering for neutron BGP addresses - https://phabricator.wikimedia.org/T246887 (10aborrero) [14:09:11] 10netops, 10Operations, 10cloud-services-team (Kanban): CloudVPS: enable BGP in the neutron transport network - https://phabricator.wikimedia.org/T245606 (10ayounsi) `lang=diff [edit protocols bgp] group Netflow { ... } + /* T245606 */ + group Cloud { + import BGP_Cloud_in; + family... [15:01:15] 10netops, 10Operations, 10Wikimedia-Incident: Investigate Juniper storm control - https://phabricator.wikimedia.org/T245192 (10ayounsi) Looks good! Instead of manually applying the profile to each interface I think we should refactor them and use interface-range like we do on access switches. That range coul... [16:40:31] 10netops, 10Operations, 10fundraising-tech-ops: DHCP routing issue with civi2001 - https://phabricator.wikimedia.org/T246812 (10Papaul) 05Stalled→03Resolved This is fixed. eth0 cable was connected to payments2002 eth1 [16:42:31] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission lvs2004.codfw.wmnet - https://phabricator.wikimedia.org/T246669 (10Papaul) [16:43:00] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission lvs2005.codfw.wmnet - https://phabricator.wikimedia.org/T246666 (10Papaul) [16:43:23] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission lvs2006.codfw.wmnet - https://phabricator.wikimedia.org/T246329 (10Papaul) [16:45:01] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission lvs2003.codfw.wmnet - https://phabricator.wikimedia.org/T246334 (10Papaul) [16:45:25] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission lvs2006.codfw.wmnet - https://phabricator.wikimedia.org/T246329 (10Papaul) [17:09:46] 10Traffic, 10Operations, 10Wikimedia-Logstash, 10observability, and 3 others: Changing Kibana filters is ridiculously slow - https://phabricator.wikimedia.org/T189333 (10Krinkle) [17:11:09] 10Traffic, 10Operations, 10Wikimedia-Logstash, 10observability, and 3 others: Changing Kibana filters is ridiculously slow - https://phabricator.wikimedia.org/T189333 (10Krinkle) This is still an issue. Editing Kibana dashboards: * In Safari, crashes the tab. * In Firefox, times out after 30 seconds, the... [17:38:52] 10netops, 10Operations, 10Patch-For-Review: can aggregated netflow data include the router it was sampled from? - https://phabricator.wikimedia.org/T246186 (10ayounsi) a:03elukey The new field is being exported properly to Kafka, see: `"peer_ip_src": "103.102.166.129"` In Turnilo it would be convenient to... [19:17:47] 10netops, 10Operations, 10Wikimedia-Incident: Add graceful-restart to cr2-esams - https://phabricator.wikimedia.org/T246338 (10CDanis) a:05ayounsi→03CDanis [19:29:26] 10Traffic, 10MediaWiki-ResourceLoader, 10Operations, 10Performance-Team, 10Wikimedia-Incident: load.php?modules=startup miss rate tripped on 2012-02-05 - https://phabricator.wikimedia.org/T247020 (10Krinkle) [19:40:14] 10netops, 10Operations, 10Wikimedia-Incident: Add graceful-restart to cr2-esams - https://phabricator.wikimedia.org/T246338 (10CDanis) Will do this tonight, at or after 00:00 UTC. Currently there are just a few (AMSIX peer) BGP sessions down: `cdanis@re0.cr2-esams> show bgp summary | match "(Active|Idle|Con... [19:51:33] bblack: an idle question for you, re: T246338 -- it's work scheduled offpeak for esams (deepest nadir 03:00-04:00 UTC depending on the day, but 00:00 is already around 1/3rd of peak traffic. do you think there's still a need to use the esams-offline map at such a time? [19:51:34] T246338: Add graceful-restart to cr2-esams - https://phabricator.wikimedia.org/T246338 [19:52:14] (I was going to default to doing such, because it's certainly safer and sloshing some other traffic around isn't really bad, but the question still seemed interesting to me :) [20:24:15] 10Traffic, 10MediaWiki-ResourceLoader, 10Operations, 10Performance-Team, 10Wikimedia-Incident: load.php?modules=startup miss rate tripped on 2012-02-05 - https://phabricator.wikimedia.org/T247020 (10Jdforrester-WMF) The `Accept-Encoding` changes seem most suspect? [20:29:35] 10Traffic, 10MediaWiki-ResourceLoader, 10Operations, 10Performance-Team, 10Wikimedia-Incident: load.php?modules=startup miss rate tripped on 2012-02-05 - https://phabricator.wikimedia.org/T247020 (10Krinkle) Indeed. I suspect that maybe it is now getting both gzip and no-gzip traffic and/or no longer sha... [20:30:15] 10Traffic, 10MediaWiki-ResourceLoader, 10Operations, 10Performance-Team, 10Wikimedia-Incident: load.php?modules=startup miss rate tripped on 2020-02-05 - https://phabricator.wikimedia.org/T247020 (10Mholloway) [20:32:34] 10Traffic, 10MediaWiki-ResourceLoader, 10Operations, 10Performance-Team, 10Wikimedia-Incident: load.php?modules=startup miss rate trippled on 2020-02-05 - https://phabricator.wikimedia.org/T247020 (10Krinkle) [21:02:27] 10netops, 10Operations, 10Wikimedia-Incident: Investigate Juniper storm control - https://phabricator.wikimedia.org/T245192 (10Papaul) I created the interface range mgmt-switches, added interfaces ge-0/0/0 to ge-0/0/31 to it and bind the storm control profile wmf-mgmt-storm to it. ` papaul@msw1-codfw# show |... [22:16:53] 10Traffic, 10DNS, 10Operations, 10Technical blog, 10cloud-services-team (Kanban): Setup DNS to direct techblog.wikimedia.org to new Wordpress VIP hosting - https://phabricator.wikimedia.org/T246507 (10bd808) Upstream instructions https://wpvip.com/documentation/vip-go/managing-domains-and-dns/#external-d... [23:11:19] 10Traffic, 10DNS, 10Operations, 10Technical blog, and 2 others: Setup DNS to direct techblog.wikimedia.org to new Wordpress VIP hosting - https://phabricator.wikimedia.org/T246507 (10bd808) [23:18:32] 10Traffic, 10DNS, 10Operations, 10Technical blog, and 2 others: Setup DNS to direct techblog.wikimedia.org to new Wordpress VIP hosting - https://phabricator.wikimedia.org/T246507 (10bd808)