[03:07:32] 10Traffic, 10ContentSecurityPolicy, 10Gerrit, 10Operations, and 2 others: Add gerrit.wikimedia.org to the Phabricator CSP - https://phabricator.wikimedia.org/T218308 (10Paladox) The CSP header is being added here https://github.com/phacility/phabricator/blob/241f06c9ffc63909ec04e29f67f53cb310fb1953/src/aph... [03:08:38] 10Traffic, 10ContentSecurityPolicy, 10Gerrit, 10Operations, and 2 others: Add gerrit.wikimedia.org to the Phabricator CSP - https://phabricator.wikimedia.org/T218308 (10Dzahn) Looks like the headers are coming from "Aphront". https://github.com/phacility/phabricator/blob/241f06c9ffc63909ec04e29f67f53cb310... [03:22:26] 10Traffic, 10ContentSecurityPolicy, 10Gerrit, 10Operations, and 2 others: Add gerrit.wikimedia.org to the Phabricator CSP - https://phabricator.wikimedia.org/T218308 (10Dzahn) So.. the value for $csp is set to the $altdom / $alt_host setting which is in Hiera as `hiera('phabricator_altdomain', 'phab.wmfuse... [03:23:31] 10netops, 10Operations: fastnetmon fired for routine text-lb.esams traffic - https://phabricator.wikimedia.org/T241281 (10ayounsi) This is most likely a false positive triggered on the ICMP: threshold_icmp_pps = 2000 Incoming icmp pps: 2124 packets per second I created https://gerrit.wikimedia.org/r/#/c/opera... [14:18:50] 10netops, 10Operations: fastnetmon fired for routine text-lb.esams traffic - https://phabricator.wikimedia.org/T241281 (10CDanis) Thanks! I had been confused by `Attack protocol: tcp` in the reports. It fired several more times and every time it was for >2000 ICMP pps, so I merged this. [14:40:17] 10netops, 10Operations: fastnetmon fired for routine text-lb.esams traffic - https://phabricator.wikimedia.org/T241281 (10CDanis) a:05ayounsi→03CDanis I'll keep an eye on this and close if there's no other noise. [22:44:53] 10Traffic, 10Operations: cp3051 crashed - https://phabricator.wikimedia.org/T241306 (10Volans) [23:04:31] 10Traffic, 10Operations: cp3051 crashed - https://phabricator.wikimedia.org/T241306 (10Volans) Nothing in `racadm`, checked both `getsel` and `lclog view`. Nothing in syslog & co. FYI in `dmesg` during the end of the boot process it logged a bunch of `kvm: disabled by bios`. [23:12:10] 10Traffic, 10Operations: cp3055 crashed - https://phabricator.wikimedia.org/T240425 (10Volans) The host crashed again today, nothing in `racadm`, checked both `getsel` and `lclog view`. [23:24:52] 10Traffic, 10Operations: cp3055 crashed - https://phabricator.wikimedia.org/T240425 (10Volans) Nothing on the host logs either. For the record it crashed 7 minutes after cp3051 (see T241306) and both are part of the upload esams cluster. Like cp3051 this one too logged a bunch of `kvm: disabled by bios` durin... [23:27:04] 10Traffic, 10Operations: cp3051 crashed - https://phabricator.wikimedia.org/T241306 (10Volans) p:05Triage→03Normal [23:27:39] 10Traffic, 10Operations: servers freeze across the caching cluster - https://phabricator.wikimedia.org/T238305 (10Volans)