[00:19:00] 10Traffic, 6Operations, 5Patch-For-Review: openssl-1.0.2f introduced minor bug with nginx - https://phabricator.wikimedia.org/T126616#2030023 (10ori) p:5Triage>3Unbreak! /var/log/nginx/unified.error.log has not been rotated since July 1. Since Feb 8, it has been growing at a rate of ~550M a day. On sever... [01:46:55] 10Traffic, 6Operations, 5Patch-For-Review: openssl-1.0.2f introduced minor bug with nginx - https://phabricator.wikimedia.org/T126616#2030137 (10BBlack) I've fixed the ones that were close (above 79% full on rootfs) for now. I'll keep an eye and we can fix it right tomorrow morning... [09:11:22] 10Traffic, 6Operations, 5Patch-For-Review: openssl-1.0.2f introduced minor bug with nginx - https://phabricator.wikimedia.org/T126616#2030486 (10ema) [16:01:37] 7HTTPS, 6Research-and-Data, 10The-Wikipedia-Library, 10Wikimedia-General-or-Unknown, and 4 others: Set an explicit "Origin When Cross-Origin" referer policy via the meta referrer tag - https://phabricator.wikimedia.org/T87276#2031615 (10BBlack) >>! In T87276#1997286, @Johan wrote: > I'll mention it in Tech... [17:30:52] 10Traffic, 6Operations, 5Patch-For-Review: Decrease max object TTL in varnishes - https://phabricator.wikimedia.org/T124954#2032116 (10BBlack) I took a look at another small sample of data today, over on the cache_upload clusters, which we'd expect to behave very differently. This was a single 10-minute run... [17:33:22] 10Traffic, 10Deployment-Systems, 6Operations, 6Performance-Team, 5Patch-For-Review: Make Varnish cache for /static/$wmfbranch/ expire when resources change within branch lifetime - https://phabricator.wikimedia.org/T99096#2032126 (10Krinkle) [17:35:36] 10Traffic, 10Deployment-Systems, 6Operations, 6Performance-Team, 5Patch-For-Review: Make Varnish cache for /static/$wmfbranch/ expire when resources change within branch lifetime - https://phabricator.wikimedia.org/T99096#2032143 (10Krinkle) `$wgLocalStylePath` was using `/w/static/{wmfbranch}` instead o... [17:52:52] 10Traffic, 6Operations, 10ops-codfw: ulsfo possible downtime - PDU swaps in both cabinets - https://phabricator.wikimedia.org/T127094#2032198 (10BBlack) And to be clear, what I expect we'll do on our end is depool ulsfo from users in DNS in our `config-geo` ahead of the window (let's say 2 hours ahead? the T... [17:54:02] 10Traffic, 6Operations, 10ops-codfw: ulsfo possible downtime - PDU swaps in both cabinets - https://phabricator.wikimedia.org/T127094#2032212 (10emailbot) **`Rob Halsell`** replied via email on `Tue, 16 Feb 2016 09:53:06 -0800` `Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failure` > Phu/Support, >... [18:00:59] 10Traffic, 6Operations, 5Patch-For-Review: Decrease max object TTL in varnishes - https://phabricator.wikimedia.org/T124954#2032264 (10BBlack) I re-ran the parsing script over the exact same input data as the last results, with finer-grained detail on the 4-12h range (I had captured the output at an intermed... [18:01:21] 10Traffic, 10Deployment-Systems, 6Operations, 6Performance-Team, 5Patch-For-Review: Make Varnish cache for /static/$wmfbranch/ expire when resources change within branch lifetime - https://phabricator.wikimedia.org/T99096#2032268 (10Krinkle) Rollout to group1 and group2 is blocked on Apache config being... [18:04:03] 10Traffic, 10Deployment-Systems, 6Operations, 6Performance-Team, 5Patch-For-Review: Make Varnish cache for /static/$wmfbranch/ expire when resources change within branch lifetime - https://phabricator.wikimedia.org/T99096#2032287 (10Stashbot) {nav icon=file, name=Mentioned in SAL, href=https://tools.wmfl... [18:30:32] 10Traffic, 6Operations, 5Patch-For-Review: Decrease max object TTL in varnishes - https://phabricator.wikimedia.org/T124954#2032419 (10BBlack) next upload datapoint is this. This is an esams backend instance (pulls from eqiad, gets requests from esams frontends): ``` Total: 704980 1s+: 100.00% 1m+: 99.95... [19:01:44] 10Traffic, 6Operations, 10ops-codfw: ulsfo possible downtime - PDU swaps in both cabinets - https://phabricator.wikimedia.org/T127094#2032508 (10emailbot) **`UnitedLayer Support Ticket System`** replied via email on `Tue, 16 Feb 2016 10:59:45 -0800` `Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failur... [19:03:27] 7Varnish, 10MobileFrontend, 3Reading-Web-Sprint-67-i: Stop default redirecting Samsung Smart TVs to mobile web - https://phabricator.wikimedia.org/T127021#2032538 (10Jdlrobson) p:5Triage>3Low [19:57:56] 10Traffic, 6Operations, 10ops-codfw: ulsfo possible downtime - PDU swaps in both cabinets - https://phabricator.wikimedia.org/T127094#2032729 (10emailbot) **`Rob Halsell`** replied via email on `Tue, 16 Feb 2016 11:57:31 -0800` `Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failure` > Tomorrow, 2016-0... [20:01:49] 10Traffic, 6Operations, 10ops-codfw: ulsfo possible downtime - PDU swaps in both cabinets - https://phabricator.wikimedia.org/T127094#2032740 (10RobH) [20:16:03] 10Traffic, 6Operations, 10ops-codfw: ulsfo possible downtime - PDU swaps in both cabinets - https://phabricator.wikimedia.org/T127094#2032842 (10RobH) a:5RobH>3BBlack I'm reassigning this from myself to @bblack (and adding the #netops project tag for the traffic move.) [20:37:49] 10Traffic, 6Operations, 5Patch-For-Review: Decrease max object TTL in varnishes - https://phabricator.wikimedia.org/T124954#2032965 (10BBlack) (note I've edited some of my cache_upload commentary above to remove questions/mysteries that turned out to mostly be my own braindeadness) [22:51:07] 10Traffic, 6Operations, 10ops-codfw: ulsfo possible downtime - PDU swaps in both cabinets - https://phabricator.wikimedia.org/T127094#2033453 (10emailbot) **`UnitedLayer Support Ticket System`** replied via email on `Tue, 16 Feb 2016 14:51:01 -0800` `Re: [UnitedLayer #118704] SF8 - Wikimedia: PDU nic failur... [23:03:28] 10Traffic, 6Operations, 10ops-codfw: ulsfo possible downtime - PDU swaps in both cabinets - https://phabricator.wikimedia.org/T127094#2033469 (10BBlack) Will depool at ~9AM Pacific, 17:00 UTC.