[06:27:45] 10Traffic, 10Cloud-VPS, 10Operations, 10serviceops: Difficulties to create offline version of Wikipedia because of HTTP 429 response - https://phabricator.wikimedia.org/T213475 (10Kelson) I'm not sure to fully understand the technical explanation. Is the problem confirmed? If "yes", what is the plan to sol... [07:54:44] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: cp1075-90 - bnxt_en transmit hangs - https://phabricator.wikimedia.org/T203194 (10Vgutierrez) cp1088 has been affected as well after the kernel upgrade [13:03:43] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: cp1075-90 - bnxt_en transmit hangs - https://phabricator.wikimedia.org/T203194 (10BBlack) >>! In T203194#4880054, @Vgutierrez wrote: > on the Dell community forum there is a [[ https://www.dell.com/community/PowerEdge-Hardware-General/Critical-netwo... [13:09:01] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: cp1075-90 - bnxt_en transmit hangs - https://phabricator.wikimedia.org/T203194 (10BBlack) See also this email thread where Michael Chan (broadcom driver dev) asks for firmware level output, sees the same numbers we have on cp1088, and tells them to... [13:27:51] bblack: regarding bnxt_en moritzm also suggested testing stretch-backports (4.19) kernel.. let's upgrade fws first and if that's not enough try with the kernel? [13:37:36] sure [13:37:50] I'm not even sure what firmware is avail, I didn't look [13:38:10] I'll take care of it after the physiotherapy appointment [13:38:38] ok! [13:39:08] and I'll ask robh too see how I can update them myself to avoid locking them upgrading 15 servers [17:03:05] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster - https://phabricator.wikimedia.org/T207321 (10Ottomata) Great meeting! We realized that the same problem we are encountering with the available Cloud VPS infrastructu... [19:09:04] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: Offload pings to dedicated server - https://phabricator.wikimedia.org/T190090 (10ayounsi) Diff for option 3 in eqiad is: `lang=diff [edit interfaces ae1 unit 1017 family inet] + filter { + output private-out4; + } [edit interface... [20:00:03] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster - https://phabricator.wikimedia.org/T207321 (10bd808) >>! In T207321#4885112, @Ottomata wrote: > A. Bryan noted that we won't want to use users regular LDAP accounts fo... [20:05:42] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster - https://phabricator.wikimedia.org/T207321 (10Ottomata) > I was trying to figure out how this will actually look for the end user It'd be in the JDBC connection, e.g... [21:26:29] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster - https://phabricator.wikimedia.org/T207321 (10bd808) >>! In T207321#4885829, @Ottomata wrote: >> I was trying to figure out how this will actually look for the end use... [21:35:05] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster - https://phabricator.wikimedia.org/T207321 (10Ottomata) > That works for a java developer, but nobody else (including Quarry). Quarry is a Python app which is why I wa... [22:13:24] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293 (10ayounsi) [23:05:44] 10Certcentral, 10Patch-For-Review: certcentral is incompatible with the current python3-acme version shipped in stretch-backports - https://phabricator.wikimedia.org/T213820 (10Krenair) @Vgutierrez: this is done now right? [23:32:58] 10netops, 10Operations, 10Patch-For-Review: Detect IP address collisions - https://phabricator.wikimedia.org/T189522 (10ayounsi) a:05ayounsi→03faidon Note that I also created an LibreNMS alert to monitor explicitly the mgmt network: `%syslog.msg ~ "KERN_ARP_ADDR_CHANGE" && %devices.hostname ~ "mr" && %de... [23:37:21] 10netops, 10Operations, 10Patch-For-Review: IGMP snooping breaks IPv6 ND on Junos 14.1X53-D46 - https://phabricator.wikimedia.org/T201039 (10ayounsi) p:05High→03Normal [23:46:27] 10netops, 10Operations, 10Patch-For-Review: IGMP snooping breaks IPv6 ND on Junos 14.1X53-D46 - https://phabricator.wikimedia.org/T201039 (10ayounsi) Reducing priority as the situation is stable. At this point I don't think the cost of upgrading the switch stacks of row B and C (full row down for ~15min) is... [23:53:08] 10Traffic, 10netops, 10Operations: Free up 185.15.59.0/24 - https://phabricator.wikimedia.org/T211254 (10ayounsi) [23:56:28] 10Traffic, 10netops, 10Operations: Free up 185.15.59.0/24 - https://phabricator.wikimedia.org/T211254 (10ayounsi) [23:56:30] 10netops, 10Operations: esams/knams: advertise 185.15.58.0/23 instead of 185.15.56.0/22 - https://phabricator.wikimedia.org/T207753 (10ayounsi)