[03:34:13] 10netops, 10Operations: nfacctd segfaulting on netflow2001 - https://phabricator.wikimedia.org/T256790 (10CDanis) Likely segfaulting on netflow1001 now too. Upstream has a patch; I'll attempt backporting it to our version in the morning. If it works I'll also file in Debian BTS with the patch and see about g... [06:56:03] 10Traffic, 10Operations: Certain links being rejected by caching if opened in Internet Explorer - https://phabricator.wikimedia.org/T256302 (10Urbanecm) @Aklapper: Here you are, this is the complete message. ` Request from (REDACTED) via cp3062 frontend, Varnish XID 675554498 Upstream caches: cp3062 int Error... [07:34:01] 10Traffic, 10Operations, 10Patch-For-Review: purged crashes with "fatal error: concurrent map read and map write" - https://phabricator.wikimedia.org/T256479 (10ema) 05Open→03Resolved a:03ema Fixed by deploying https://gerrit.wikimedia.org/r/c/operations/software/purged/+/608045, the issue hasn't occur... [08:41:12] 10Traffic, 10Operations: Certain links being rejected by caching if opened in Internet Explorer with a HTTP 400 error - https://phabricator.wikimedia.org/T256302 (10Aklapper) [08:52:45] 10Traffic, 10Operations, 10Patch-For-Review: Current codfw caches have wrong NVME format - https://phabricator.wikimedia.org/T256655 (10ops-monitoring-bot) Icinga downtime for 0:30:00 set by vgutierrez@cumin1001 on 2 host(s) and their services with reason: kernel upgrade ` cp[2039-2040].codfw.wmnet ` [09:42:33] 10Traffic, 10Operations, 10Patch-For-Review: Current codfw caches have wrong NVME format - https://phabricator.wikimedia.org/T256655 (10ops-monitoring-bot) Icinga downtime for 0:30:00 set by vgutierrez@cumin1001 on 2 host(s) and their services with reason: kernel upgrade ` cp[2041-2042].codfw.wmnet ` [09:44:24] 10Traffic, 10Operations, 10Patch-For-Review: Current codfw caches have wrong NVME format - https://phabricator.wikimedia.org/T256655 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez [12:29:47] 10netops, 10Operations, 10ops-eqiad: (Need by: 2019-09-30) upgrade msw1-eqiad from EX4200 to EX4300 - https://phabricator.wikimedia.org/T225121 (10ayounsi) I cleaned up some default config leftovers as well as refactored the interfaces the same way we have them in codfw (with storm control). It's ready to b... [13:05:14] 10netops, 10Operations: nfacctd segfaulting on netflow2001 - https://phabricator.wikimedia.org/T256790 (10CDanis) 05Open→03Resolved a:03CDanis Backport deployed, all seems well (fortunately netflow1001 was still receiving the triggering BGP data when I went to test, so verified old version still crashed,... [17:10:36] 10netops, 10Operations: Configure management-instance on routers with Junos > 17.3 - https://phabricator.wikimedia.org/T247073 (10ayounsi) [17:37:50] 10netops, 10Operations, 10ops-eqsin, 10Patch-For-Review: cr3-eqsin to production - https://phabricator.wikimedia.org/T255766 (10ayounsi) Replacement went smooth! Last step is to update Netbox.