[08:13:31] bblack: (for when you're around): zerofetch is failing on a bunch of legacy esams cp* that have the text-lb (= zero.wm.org) IP bound to lo, but with no varnish running [09:47:36] 7HTTPS, 10Traffic, 6Operations, 10Wikimedia-Blog: Switch blog to HTTPS-only - https://phabricator.wikimedia.org/T105905#2154394 (10Tbayer) Update: After some more detours, all posts that embedded HTTP content should now have been converted to HTTPS. In the end Automattic ran two conversion scripts for us,... [13:16:39] paravoid: oops :) [13:21:55] fixed [13:46:25] 10Traffic, 10Beta-Cluster-Infrastructure, 6Operations, 13Patch-For-Review, 7Puppet: Fix puppet on deployment-cache* hosts in beta cluster - https://phabricator.wikimedia.org/T129270#2154574 (10Ottomata) Thanks yall! [13:50:40] 10Traffic, 6Operations, 10ops-eqiad: investigate radon crash - https://phabricator.wikimedia.org/T131053#2154606 (10BBlack) [16:02:36] 10Traffic, 6Operations, 7Mobile, 13Patch-For-Review: Investigate if login.m.wikimedia.org needs to stay around - https://phabricator.wikimedia.org/T123431#2154832 (10csteipp) The only danger would be if MobileFrontend suddenly started hooking earlier in the login process, and redirected mobile users from l... [16:09:13] 7Varnish, 10MobileFrontend, 13Patch-For-Review, 3Reading-Web-Sprint-68-"Java and JavaScript are basically the same", 3Reading-Web-Sprint-69-k: Stop default redirecting Samsung Smart TVs to mobile web - https://phabricator.wikimedia.org/T127021#2154851 (10dr0ptp4kt) [17:27:27] 10Traffic, 6Operations, 10ops-eqiad: investigate radon crash - https://phabricator.wikimedia.org/T131053#2154594 (10Cmjohnson) I see CPU heat warnings...could that be a cause or just a symptom? [17:32:26] 10Traffic, 6Operations, 10ops-eqiad: investigate radon crash - https://phabricator.wikimedia.org/T131053#2155164 (10BBlack) note: it crashed a second time just now. we're looking into dealing with the hardware issue now. @faidon has re-routed the requests to baham in codfw (ns1), so from an external perspe... [17:42:52] 10Traffic, 6Operations: Set up LVS for current AuthDNS - https://phabricator.wikimedia.org/T101525#2155228 (10BBlack) Bump - T131053 reminds us that it's uncomfortable to lose even 1/3 authdns servers when hardware fails, as that places us within 1 more random hardware failure of "oh crap". We really should r... [18:06:04] 10Traffic, 6Operations, 10ops-eqiad: investigate radon crash - https://phabricator.wikimedia.org/T131053#2155361 (10Cmjohnson) Not sure if it's the culprit yet but re-applied thermal paste to the cpu. It was dry and brittle. [18:22:31] 10Traffic, 6Operations, 7Mobile, 13Patch-For-Review: Investigate if login.m.wikimedia.org needs to stay around - https://phabricator.wikimedia.org/T123431#2155461 (10Dzahn) also see comment from bblack on: T111967#1622278 and i checked [oxygen:/srv/log/webrequest] $ jq .uri_host /srv/log/webrequest/sample... [18:23:53] 10Traffic, 6Operations, 7Mobile: Investigate if www.m.wikipedia.org needs to stay around - https://phabricator.wikimedia.org/T120143#2155477 (10Dzahn) [18:23:55] 10Traffic, 6Operations, 7Mobile, 13Patch-For-Review: Investigate if login.m.wikimedia.org needs to stay around - https://phabricator.wikimedia.org/T123431#2155475 (10Dzahn) 5Open>3Resolved https://gerrit.wikimedia.org/r/#/c/276385/ merged [18:25:30] 7HTTPS, 10Traffic, 6Operations: Preload HSTS for select hostnames within wikimedia.org - https://phabricator.wikimedia.org/T111967#1621104 (10Dzahn) login.m.wikimedia.org has been removed now with T123431 / https://gerrit.wikimedia.org/r/#/c/276385/. [18:29:27] 10Traffic, 6Operations, 7Mobile: Investigate if login.m.wikimedia.org needs to stay around - https://phabricator.wikimedia.org/T123431#2155508 (10Dzahn) [19:43:44] 10Traffic, 6Operations, 10ops-eqiad: investigate radon crash - https://phabricator.wikimedia.org/T131053#2155703 (10BBlack) Status update: radon's been back up for 1h40m now and is theoretically functional. Waiting a bit to see if it seems stable. ns0 traffic is still routed to baham. If for some reason w... [20:00:39] 10Traffic, 6Operations, 6Performance-Team, 13Patch-For-Review: Support HTTP/2 - https://phabricator.wikimedia.org/T96848#2155759 (10BBlack) 1.9.12 porting/packaging was fairly trivial so far. Testing on cp1008/pinkunicorn to happen later today or tomorrow. [23:47:55] 7HTTPS, 10Traffic, 6Operations, 10Wikimedia-Fundraising, 13Patch-For-Review: delete links.email.donate.wikimedia.org (and all other email.donate.*?) from DNS - https://phabricator.wikimedia.org/T130414#2156614 (10CCogdill_WMF) @Dzahn I've confirmed it's fine on our side to delete all the subdomains you...