[06:01:37] 10Traffic, 10Operations, 10Patch-For-Review: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts: ` cp2038.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage... [06:26:56] 10Traffic, 10Operations: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2038.codfw.wmnet'] ` Of which those **FAILED**: ` ['cp2038.codfw.wmnet'] ` [06:31:08] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2012.codfw.wmnet - https://phabricator.wikimedia.org/T249080 (10Vgutierrez) [06:38:58] 10Traffic, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: decommission cp2012.codfw.wmnet - https://phabricator.wikimedia.org/T249080 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by vgutierrez@cumin1001 for hosts: `cp2012.codfw.wmnet` - cp2012.codfw.wmnet (**PASS**)... [07:34:05] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2012.codfw.wmnet - https://phabricator.wikimedia.org/T249080 (10Vgutierrez) a:05Vgutierrez→03Papaul [07:34:26] 10Traffic, 10Operations: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10Vgutierrez) [07:43:21] 10Traffic, 10Operations, 10Patch-For-Review: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts: ` cp2039.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage... [07:58:50] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2017.codfw.wmnet - https://phabricator.wikimedia.org/T249084 (10Vgutierrez) [08:09:08] 10Traffic, 10Operations, 10Patch-For-Review: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2039.codfw.wmnet'] ` Of which those **FAILED**: ` ['cp2039.codfw.wmnet'] ` [08:30:40] 10Traffic, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: decommission cp2017.codfw.wmnet - https://phabricator.wikimedia.org/T249084 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by vgutierrez@cumin1001 for hosts: `cp2017.codfw.wmnet` - cp2017.codfw.wmnet (**PASS**)... [08:31:13] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2017.codfw.wmnet - https://phabricator.wikimedia.org/T249084 (10Vgutierrez) a:05Vgutierrez→03Papaul [08:31:43] 10Traffic, 10Operations, 10Patch-For-Review: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10Vgutierrez) [08:46:38] 10Traffic, 10Operations, 10Patch-For-Review: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts: ` cp2040.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage... [08:59:20] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2013.codfw.wmnet - https://phabricator.wikimedia.org/T249088 (10Vgutierrez) [09:12:04] 10Traffic, 10Operations: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2040.codfw.wmnet'] ` Of which those **FAILED**: ` ['cp2040.codfw.wmnet'] ` [11:50:20] 10netops, 10Analytics, 10Operations: Move netflow to TLS encryption/authentication via librdkafka - https://phabricator.wikimedia.org/T248980 (10elukey) We could start with TLS authentication only, with: ` security.protocol=SSL ssl.ca.location=/etc/ssl/certs/Puppet_Internal_CA.pem ssl.cipher.suites=ECDHE-EC... [12:20:00] 10Traffic, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: decommission cp2013.codfw.wmnet - https://phabricator.wikimedia.org/T249088 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by vgutierrez@cumin1001 for hosts: `cp2013.codfw.wmnet` - cp2013.codfw.wmnet (**PASS**)... [12:23:22] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2013.codfw.wmnet - https://phabricator.wikimedia.org/T249088 (10Vgutierrez) a:05Vgutierrez→03Papaul [12:23:40] 10Traffic, 10Operations: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10Vgutierrez) [12:50:04] 10Traffic, 10Operations, 10Patch-For-Review: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts: ` cp2041.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage... [13:15:51] 10Traffic, 10Operations: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2041.codfw.wmnet'] ` Of which those **FAILED**: ` ['cp2041.codfw.wmnet'] ` [13:53:13] 10Traffic, 10Operations, 10Patch-For-Review: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts: ` cp2042.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage... [13:56:33] 10Traffic, 10Operations, 10Patch-For-Review: check_trafficserver_log_fifo: false positives when changing log format - https://phabricator.wikimedia.org/T248067 (10ema) 05Open→03Resolved a:03ema [14:00:25] 10Traffic, 10Analytics, 10Operations, 10Performance-Team: Only serve debug HTTP headers when x-wikimedia-debug is present - https://phabricator.wikimedia.org/T210484 (10ema) 05Open→03Resolved Done: ` $ curl -v -H "X-Wikimedia-Debug: mwdebug1001.eqiad.wmnet" https://en.wikipedia.org/wiki/Main_Page 2>&1... [14:00:29] 10Traffic, 10Operations, 10SRE-swift-storage, 10Patch-For-Review, 10Performance-Team (Radar): Reduce amount of headers sent from web responses - https://phabricator.wikimedia.org/T194814 (10ema) [14:04:09] 10Traffic, 10Operations, 10SRE-swift-storage, 10Patch-For-Review, 10Performance-Team (Radar): Reduce amount of headers sent from web responses - https://phabricator.wikimedia.org/T194814 (10ema) 05Open→03Resolved [14:18:35] 10Traffic, 10Operations: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2042.codfw.wmnet'] ` Of which those **FAILED**: ` ['cp2042.codfw.wmnet'] ` [14:34:07] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp20[18,20,22,24-26].codfw.wmnet - https://phabricator.wikimedia.org/T249115 (10Vgutierrez) [14:34:47] 10Traffic, 10Operations: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10Vgutierrez) [14:43:45] !log depool && decommission cp[2018,2020,2022,2024-2026].codfw.wmnet - T249115 [14:43:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:43:51] T249115: decommission cp20[18,20,22,24-26].codfw.wmnet - https://phabricator.wikimedia.org/T249115 [14:50:04] 10Traffic, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: decommission cp20[18,20,22,24-26].codfw.wmnet - https://phabricator.wikimedia.org/T249115 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by vgutierrez@cumin1001 for hosts: `cp2018.codfw.wmnet` - cp2018.codfw.wmne... [14:53:10] 10Traffic, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: decommission cp20[18,20,22,24-26].codfw.wmnet - https://phabricator.wikimedia.org/T249115 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by vgutierrez@cumin1001 for hosts: `cp[2020,2022,2024-2026].codfw.wmnet` -... [14:59:38] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission cp2007.codfw.wmnet - https://phabricator.wikimedia.org/T248941 (10Papaul) ` [edit interfaces interface-range disabled] member xe-2/0/5 { ... } + member xe-2/0/3; [edit interfaces] - xe-2/0/3 { - description cp200... [15:00:06] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission cp2007.codfw.wmnet - https://phabricator.wikimedia.org/T248941 (10Papaul) [15:04:05] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission cp2010.codfw.wmnet - https://phabricator.wikimedia.org/T249002 (10Papaul) ` [edit interfaces interface-range disabled] member xe-2/0/3 { ... } + member xe-7/0/3; [edit interfaces] - xe-7/0/3 { - description cp201... [15:04:26] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission cp2010.codfw.wmnet - https://phabricator.wikimedia.org/T249002 (10Papaul) [15:08:02] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp20[18,20,22,24-26].codfw.wmnet - https://phabricator.wikimedia.org/T249115 (10Vgutierrez) a:05Vgutierrez→03Papaul [15:15:06] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp20[16,19,23,27].codfw.wmnet - https://phabricator.wikimedia.org/T249125 (10Vgutierrez) [15:15:37] 10Traffic, 10Operations: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10Vgutierrez) [15:33:44] 10Traffic, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: decommission cp20[16,19,23,27].codfw.wmnet - https://phabricator.wikimedia.org/T249125 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by vgutierrez@cumin1001 for hosts: `cp[2016,2019,2023,2027].codfw.wmnet` - cp2... [15:42:27] 10netops, 10Operations: roll out sensible flow-table-sizes to Juniper core routers with sampling enabled - https://phabricator.wikimedia.org/T248394 (10CDanis) 05Open→03Resolved Homer-ized and done. [15:43:04] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission cp2011.codfw.wmnet - https://phabricator.wikimedia.org/T248950 (10Papaul) ` [edit interfaces interface-range disabled] member xe-7/0/3 { ... } + member xe-7/0/4; [edit interfaces] - xe-7/0/4 { - description cp201... [15:43:12] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2007.codfw.wmnet - https://phabricator.wikimedia.org/T248941 (10RobH) [15:43:51] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission cp2011.codfw.wmnet - https://phabricator.wikimedia.org/T248950 (10Papaul) [15:45:10] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2012.codfw.wmnet - https://phabricator.wikimedia.org/T249080 (10Papaul) ` [edit interfaces interface-range disabled] member xe-7/0/4 { ... } + member xe-7/0/5; [edit interfaces] - xe-7/0/5 { - description cp2012; - e... [15:46:58] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp20[16,19,23,27].codfw.wmnet - https://phabricator.wikimedia.org/T249125 (10Vgutierrez) a:05Vgutierrez→03Papaul [15:47:52] 10Traffic, 10Operations: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez [15:51:06] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2012.codfw.wmnet - https://phabricator.wikimedia.org/T249080 (10Papaul) [15:51:26] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp20[16,19,23].codfw.wmnet - https://phabricator.wikimedia.org/T249125 (10Vgutierrez) [15:52:01] 10Traffic, 10Operations: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10Vgutierrez) 05Resolved→03Open [15:54:25] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2012.codfw.wmnet - https://phabricator.wikimedia.org/T249080 (10RobH) [15:59:07] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2013.codfw.wmnet - https://phabricator.wikimedia.org/T249088 (10Papaul) ` [edit interfaces interface-range disabled] member xe-2/0/46 { ... } + member xe-2/0/3; [edit interfaces] - xe-2/0/3 { - description cp2013; -... [15:59:39] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2013.codfw.wmnet - https://phabricator.wikimedia.org/T249088 (10Papaul) [16:00:53] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission cp2014.codfw.wmnet - https://phabricator.wikimedia.org/T249009 (10Papaul) ` [edit interfaces interface-range disabled] member xe-2/0/3 { ... } + member xe-2/0/4; [edit interfaces] - xe-2/0/4 { - description cp201... [16:01:37] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission cp2014.codfw.wmnet - https://phabricator.wikimedia.org/T249009 (10Papaul) [16:06:00] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2017.codfw.wmnet - https://phabricator.wikimedia.org/T249084 (10Papaul) ` [edit interfaces interface-range disabled] member xe-2/0/4 { ... } + member xe-7/0/4; [edit interfaces] - xe-7/0/4 { - description cp2017; - e... [16:06:29] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2017.codfw.wmnet - https://phabricator.wikimedia.org/T249084 (10Papaul) [16:07:48] 10Traffic, 10Operations, 10Patch-For-Review: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts: ` cp2027.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage... [16:24:28] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Move netflow to TLS encryption/authentication via librdkafka - https://phabricator.wikimedia.org/T248980 (10elukey) p:05Triage→03High [16:37:45] 10Traffic, 10Operations: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp2027.codfw.wmnet'] ` and were **ALL** successful. [16:40:29] 10Traffic, 10Operations: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10Vgutierrez) [16:40:39] 10Traffic, 10Operations: Replace cp20[01-26] with cp20[27-42] - https://phabricator.wikimedia.org/T248816 (10Vgutierrez) 05Open→03Resolved [16:57:20] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2017.codfw.wmnet - https://phabricator.wikimedia.org/T249084 (10RobH) [17:06:01] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp20[16,19,23].codfw.wmnet - https://phabricator.wikimedia.org/T249125 (10Papaul) ` [edit interfaces interface-range disabled] member xe-7/0/4 { ... } + member xe-7/0/3; [edit interfaces] - xe-7/0/3 { - description cp2016; -... [17:07:18] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp20[16,19,23].codfw.wmnet - https://phabricator.wikimedia.org/T249125 (10Papaul) [17:08:02] 10Traffic, 10Operations, 10decommission, 10ops-eqiad: Decommission old eqiad caches - https://phabricator.wikimedia.org/T208584 (10RobH) [17:11:43] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp20[16,19,23].codfw.wmnet - https://phabricator.wikimedia.org/T249125 (10RobH) [17:25:15] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2002.codfw.wmnet - https://phabricator.wikimedia.org/T248818 (10RobH) [17:25:35] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2004.codfw.wmnet - https://phabricator.wikimedia.org/T248824 (10RobH) [17:25:54] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2001.codfw.wmnet - https://phabricator.wikimedia.org/T248815 (10RobH) [17:26:52] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2005.codfw.wmnet - https://phabricator.wikimedia.org/T248848 (10RobH) [17:27:02] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission cp2006.codfw.wmnet - https://phabricator.wikimedia.org/T248856 (10RobH) [17:27:30] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2007.codfw.wmnet - https://phabricator.wikimedia.org/T248941 (10RobH) [17:27:39] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2007.codfw.wmnet - https://phabricator.wikimedia.org/T248941 (10RobH) [17:27:55] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission cp2008.codfw.wmnet - https://phabricator.wikimedia.org/T248864 (10RobH) [17:28:14] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2011.codfw.wmnet - https://phabricator.wikimedia.org/T248950 (10RobH) [17:28:30] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission cp2010.codfw.wmnet - https://phabricator.wikimedia.org/T249002 (10RobH) [17:28:46] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2012.codfw.wmnet - https://phabricator.wikimedia.org/T249080 (10RobH) [17:29:04] 10Traffic, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: decommission cp2014.codfw.wmnet - https://phabricator.wikimedia.org/T249009 (10RobH) [17:30:11] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2014.codfw.wmnet - https://phabricator.wikimedia.org/T249009 (10RobH) [17:30:27] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2017.codfw.wmnet - https://phabricator.wikimedia.org/T249084 (10RobH) [17:30:47] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp2013.codfw.wmnet - https://phabricator.wikimedia.org/T249088 (10RobH) [17:37:20] 10Traffic, 10Operations, 10decommission, 10ops-eqiad: Decommission old eqiad caches - https://phabricator.wikimedia.org/T208584 (10RobH) [17:38:37] 10Traffic, 10Operations, 10decommission, 10ops-codfw: decommission cp20[18,20,22,24-26].codfw.wmnet - https://phabricator.wikimedia.org/T249115 (10RobH) [17:42:25] 10Traffic, 10Operations, 10decommission, 10ops-eqiad: Decommission lvs1007-1012 - https://phabricator.wikimedia.org/T208586 (10RobH) [21:32:08] 10Traffic, 10Operations, 10ops-esams: cp3057 crash (was: network down) - https://phabricator.wikimedia.org/T244127 (10faidon) [21:32:11] 10Traffic, 10Operations: servers freeze across the caching cluster - https://phabricator.wikimedia.org/T238305 (10faidon) [21:32:39] 10Traffic, 10Operations: cp3055 crashed - https://phabricator.wikimedia.org/T240425 (10faidon) [21:32:43] 10Traffic, 10Operations: servers freeze across the caching cluster - https://phabricator.wikimedia.org/T238305 (10faidon) [21:32:49] 10Traffic, 10Operations: cp3051 crashed - https://phabricator.wikimedia.org/T241306 (10faidon) [21:32:52] 10Traffic, 10Operations: servers freeze across the caching cluster - https://phabricator.wikimedia.org/T238305 (10faidon) [21:40:04] 10Traffic, 10Operations: servers freeze across the caching cluster - https://phabricator.wikimedia.org/T238305 (10faidon) What's the latest here? I haven't heard about these crashes lately but it may just be that I missed it. Do we know more about this now? Also, it's great to hear that we have a traceback no... [21:40:32] 10Traffic, 10DC-Ops, 10Operations, 10ops-esams: Upgrade BIOS and IDRAC firmware on R440 cp systems - https://phabricator.wikimedia.org/T243167 (10faidon) [21:40:38] 10Traffic, 10Operations: servers freeze across the caching cluster - https://phabricator.wikimedia.org/T238305 (10faidon) [21:41:01] 10Traffic, 10Operations, 10ops-esams: cp3057 crash (was: network down) - https://phabricator.wikimedia.org/T244127 (10faidon) [21:41:06] 10Traffic, 10Operations: servers freeze across the caching cluster - https://phabricator.wikimedia.org/T238305 (10faidon) [21:41:20] 10Traffic, 10Operations, 10ops-esams: cp3057 crash (was: network down) - https://phabricator.wikimedia.org/T244127 (10faidon) [21:41:23] 10Traffic, 10DC-Ops, 10Operations, 10ops-esams: Upgrade BIOS and IDRAC firmware on R440 cp systems - https://phabricator.wikimedia.org/T243167 (10faidon) [22:06:05] 10netops, 10Operations: netflow2001 kafkatee-webrequest restart loop - https://phabricator.wikimedia.org/T249176 (10Volans) p:05Triage→03Medium [22:07:44] 10netops, 10Operations: netflow2001 kafkatee-webrequest restart loop - https://phabricator.wikimedia.org/T249176 (10Volans) I've then stopped and restarted the systemd unit and it was able to start properly, but it should be investigated. My understanding is that the systemd Icinga check was already alarming f... [22:13:22] 10netops, 10Operations: netflow hosts spamming /var/log - https://phabricator.wikimedia.org/T249177 (10Volans) p:05Triage→03Medium [22:52:34] 10Traffic, 10Operations: Servers freezing across the caching cluster (November 2019) - https://phabricator.wikimedia.org/T238305 (10Krinkle)