[01:13:43] Might be a good idea, that troll is still around. [03:32:51] 10Traffic, 06Operations, 10ops-eqiad: cp1052 ethernet link down 2016-10-22 14:11 - https://phabricator.wikimedia.org/T148891#3056828 (10Cmjohnson) @ema the sfp has been replaced I see a link light now. LMK if that fixes the problem. [08:41:03] 07HTTPS, 10Traffic, 06Discovery, 06Operations, and 2 others: compile number of http uses for http://www.wikidata.org/entity - https://phabricator.wikimedia.org/T154017#3057021 (10Esc3300) @Lydia_Pintscher can you arrange for this to be done? [09:43:41] 07HTTPS, 10Traffic, 06Discovery, 06Operations, and 2 others: compile number of http uses for http://www.wikidata.org/entity - https://phabricator.wikimedia.org/T154017#3057250 (10Lydia_Pintscher) I currently don't have the time, sorry. [11:40:14] 10Traffic, 06Operations, 10ops-eqiad: cp1052 ethernet link down 2016-10-22 14:11 - https://phabricator.wikimedia.org/T148891#3057526 (10ema) 05Open>03Resolved @Cmjohnson looks good. Thanks! [13:08:59] 10Traffic, 10DNS, 06Operations, 07Mobile, 13Patch-For-Review: Many misc wikis lack mobile domains - https://phabricator.wikimedia.org/T152882#3057738 (10ema) p:05Triage>03Normal [13:30:33] 10Traffic, 06Operations, 15User-Elukey: certspotter: Error retrieving STH from log - https://phabricator.wikimedia.org/T159137#3057816 (10ema) [13:37:16] 10Traffic, 06Operations: certspotter: Error retrieving STH from log - https://phabricator.wikimedia.org/T159137#3057819 (10elukey) [13:37:29] 10Traffic, 06Operations: certspotter: Error retrieving STH from log - https://phabricator.wikimedia.org/T159137#3057797 (10faidon) Yeah, WoSign's CT server seems to be occasionally flaky, I saw someone else complaining about this somewhere. Not sure what we can do about that :/ [17:14:40] <_joe_> bblack: I might need some help transforming the discovery thing in a standalone service vs modifying authdns itself [17:17:05] <_joe_> it would be easier for me to make it optional in the class ::authdns, then to add the role::authdns to a new server just for testing [17:19:59] <_joe_> but my tendons are aching, maybe tomorrow :) [18:14:29] had a chat with elukey, we decided to bump timeout_idle once more on _misc to get sc_rx_timeout to 0 and see if that has an impact on piwik 503s [18:14:52] https://grafana.wikimedia.org/dashboard/db/varnish-machine-stats?var-server=cp1045&panelId=26&fullscreen&var-datasource=eqiad%20prometheus%2Fops [18:15:08] backend errors should go to 0 soon [18:17:38] aaand zero :) [18:25:15] _joe_: we've put the authdns class on cp1008 before too [18:25:40] (also, there's some requisite changes in the DNS repo to restructure those templates) [18:29:46] _joe_: uploaded here (splits config-geo into two sub-portions): https://gerrit.wikimedia.org/r/#/c/340154/ [18:33:03] _joe_: maybe I can split this up a bit and do the structure change as a no-op first [18:58:57] 10Traffic, 06Analytics-Kanban, 06Operations, 06Wikipedia-iOS-App-Backlog, and 2 others: Periodic 500s from piwik.wikimedia.org - https://phabricator.wikimedia.org/T154558#2915651 (10ema) >>! In T154558#3058745, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-operations), href=https://t... [19:26:26] <_joe_> bblack: that was my plan [19:30:17] _joe_: I broke it up already, so we can deploy no-op structural template changes first that cross the puppet+DNS repo boundary (tricky) [19:30:36] _joe_: (also, those changes necessitated some changes to authdns-local-update + authdns-lint, just finished those) [19:31:35] so we can do the dns commit and the puppet structure commit "together" (puppet disabled on real authdns, check no-op + functional on cp1008) first [19:31:43] and then worry about the complex one for the actual discovery bits [19:31:53] (with the conditional thing I guess on cp1008) [21:53:45] 10Domains, 10Traffic, 10Education-Program-Dashboard: Create short link for outreachdashboard.wmflabs.org - https://phabricator.wikimedia.org/T146332#3059379 (10Ragesoss)