[01:03:04] 10netops, 10Operations: mr1-esams flowd logs flood - https://phabricator.wikimedia.org/T238174 (10ayounsi) p:05Triage→03High [01:13:12] 10netops, 10Operations: mr1-esams flowd logs flood - https://phabricator.wikimedia.org/T238174 (10ayounsi) Opened case 2019-1112-0803 [08:34:31] 10Traffic, 10Operations, 10Wikimedia-Logstash: varnishlog consumers http request/response logging field explosion - https://phabricator.wikimedia.org/T238089 (10fgiunchedi) 05Open→03Invalid Confirmed only a subset of headers are whitelisted for sending, closing [10:51:11] 10Traffic, 10Operations, 10Puppet, 10User-jbond: In valid byte sequence: File[/etc/update-ocsp.d/hooks/trafficserver-tls-ocsp] - https://phabricator.wikimedia.org/T238198 (10jbond) p:05Triage→03Normal [10:56:32] 10Traffic, 10Operations, 10Patch-For-Review: ats-tls-restart failed on cp4027 - https://phabricator.wikimedia.org/T237425 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez This has been fixed with the conditional reload: `vgutierrez@cp5007:~$ sudo -i ats-tls-restart eqsin/cache_text/nginx/cp5007.eqsin.wm... [11:06:25] 10Traffic, 10Operations, 10Patch-For-Review, 10Puppet, 10User-jbond: In valid byte sequence: File[/etc/update-ocsp.d/hooks/trafficserver-tls-ocsp] - https://phabricator.wikimedia.org/T238198 (10jbond) This was actully fixed by the following change https://gerrit.wikimedia.org/r/c/operations/puppet/+/5506... [11:08:37] 10Traffic, 10Operations: debmonitor TLS termination - https://phabricator.wikimedia.org/T238200 (10ema) [11:08:53] 10Traffic, 10Operations: debmonitor TLS termination - https://phabricator.wikimedia.org/T238200 (10ema) p:05Triage→03Normal [11:31:36] godog: also among SOSP accepted papers -> https://research.fb.com/wp-content/uploads/2019/09/Taiji-Managing-Global-User-Traffic-for-Large-Scale-Internet-Services-at-the-Edge.pdf [12:16:15] 10Traffic, 10Operations: debmonitor TLS termination - https://phabricator.wikimedia.org/T238200 (10Volans) As we discussed a while ago about this, the easiest solution is to pick another port for the public TLS server on the debmonitor servers as the 443 is already taken for the internal clients to report the... [12:24:47] ema: fascinating! thanks for the link [12:26:07] *from reading the 'introduction' that is [13:27:47] 10Traffic, 10Operations: Renew and deploy GlobalSign unified cert (2019) - https://phabricator.wikimedia.org/T237650 (10Seb35) A certificate warning here https://social.imirhil.fr/@aeris/103126273693383568 the user still had the cert 2018 at 2019-11-12 17:07 UTC although the sites served the cert 2019 at that... [14:13:18] 10Traffic, 10Operations: Renew and deploy GlobalSign unified cert (2019) - https://phabricator.wikimedia.org/T237650 (10BBlack) >>! In T237650#5660016, @Seb35 wrote: > A certificate warning here https://social.imirhil.fr/@aeris/103126273693383568 the user still had the cert 2018 at 2019-11-12 17:07 UTC althoug... [15:39:05] 10Traffic, 10Operations, 10Patch-For-Review: Replace Varnish backends with ATS on cache text nodes - https://phabricator.wikimedia.org/T227432 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts: ` ['cp3054.esams.wmnet'] ` The log can be found in `/var/log/wm... [15:42:26] 10netops, 10Operations, 10ops-esams: mr1-esams flowd logs flood - https://phabricator.wikimedia.org/T238174 (10ayounsi) > Upon internally checking information about the logs you provided, we have found out that this is a hardware issue, please fill out the shipping form to proceed replacing this unit: > [...] [16:13:34] 10Traffic, 10Operations, 10Patch-For-Review: Replace Varnish backends with ATS on cache text nodes - https://phabricator.wikimedia.org/T227432 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp3054.esams.wmnet'] ` and were **ALL** successful. [18:47:52] 10netops, 10Operations, 10ops-esams: mr1-esams flowd logs flood - https://phabricator.wikimedia.org/T238174 (10ayounsi) a:05ayounsi→03RobH Over to Rob for the RMA. [19:21:22] 10netops, 10Operations, 10ops-esams: mr1-esams flowd logs flood - https://phabricator.wikimedia.org/T238174 (10ayounsi) [19:48:57] 10netops, 10Operations, 10ops-esams: setup new MX204 in knams - https://phabricator.wikimedia.org/T237030 (10ayounsi)