[01:17:35] 10netops, 06Operations: Packet loss from Voxel to text load balancers - https://phabricator.wikimedia.org/T153998#2927180 (10faidon) 05Open>03stalled This is impossible to debug further without more information. Can we get a complete traceroute (ICMP or UDP, although TCP in addition to those won't hurt) a... [01:18:19] 10netops, 06Operations, 10ops-ulsfo: lvs4002 power supply failure - https://phabricator.wikimedia.org/T151273#2927187 (10faidon) a:03RobH @RobH what's the status of this? [01:18:50] 10netops, 06Operations: Investigate why disabling an uplink port did not deprioritize VRRP on cr2-eqiad - https://phabricator.wikimedia.org/T119759#2927190 (10faidon) 05Open>03Resolved a:03faidon Resolving as per the above. [01:19:33] 10netops, 10Monitoring, 06Operations, 13Patch-For-Review: Juniper monitoring - https://phabricator.wikimedia.org/T83992#2927193 (10faidon) [11:52:12] 10Traffic, 10Citoid, 06Operations, 10RESTBase, and 4 others: Set-up Citoid behind RESTBase - https://phabricator.wikimedia.org/T108646#2927705 (10Mvolz) [18:05:06] 10Traffic, 06Operations: convert wikitech.wikimedia.org from globalsign to letsencrypt certificate (deadline 2017-02-24) - https://phabricator.wikimedia.org/T154913#2928272 (10RobH) [18:08:38] 10Traffic, 06Operations: convert wikitech.wikimedia.org from globalsign to letsencrypt certificate (deadline 2017-02-24) - https://phabricator.wikimedia.org/T154913#2928308 (10RobH) [18:09:09] 10Traffic, 06Operations: convert wikitech.wikimedia.org from globalsign to letsencrypt certificate (deadline 2017-02-24) - https://phabricator.wikimedia.org/T154913#2928272 (10RobH) [18:21:16] 10Traffic, 06Operations, 10Wikimedia-Mailing-lists: convert lists.wikimedia.org certificate to LetsEncrypt (deadline:2017-03-02) - https://phabricator.wikimedia.org/T154917#2928370 (10RobH) [18:22:57] 10Traffic, 10Citoid, 10ContentTranslation-CXserver, 10MediaWiki-extensions-ContentTranslation, and 4 others: Decom legacy ex-parsoidcache cxserver, citoid, and restbase service hostnames - https://phabricator.wikimedia.org/T133001#2928388 (10mobrovac) [18:23:03] 10Traffic, 10Citoid, 06Operations, 10RESTBase, and 4 others: Set-up Citoid behind RESTBase - https://phabricator.wikimedia.org/T108646#2928385 (10mobrovac) 05Open>03Resolved a:03mobrovac This can now be considered done. There is still {T133001} to deal with, though. [18:40:36] 10Traffic, 06Operations: convert librenms.wikimedia.org from GS to LE cert (expires: 2017-09-11) - https://phabricator.wikimedia.org/T154919#2928434 (10RobH) [18:41:35] 10Traffic, 06Operations: convert librenms.wikimedia.org from GS to LE cert (expires: 2017-02-11) - https://phabricator.wikimedia.org/T154919#2928449 (10RobH) [19:08:02] 10Traffic, 06Operations: convert wikitech.wikimedia.org from globalsign to letsencrypt certificate (deadline 2017-02-24) - https://phabricator.wikimedia.org/T154913#2928540 (10RobH) [20:33:14] 10Traffic, 06Operations: convert wikitech.wikimedia.org from globalsign to letsencrypt certificate (deadline 2017-02-24) - https://phabricator.wikimedia.org/T154913#2928975 (10Krenair) a:03Krenair [20:36:42] 10Traffic, 06Operations, 13Patch-For-Review: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2240497 (10RobH) [20:36:56] 10Traffic, 06Operations, 13Patch-For-Review: convert wikitech.wikimedia.org from globalsign to letsencrypt certificate (deadline 2017-02-24) - https://phabricator.wikimedia.org/T154913#2929003 (10Dzahn) [20:37:00] 10Traffic, 06Operations, 13Patch-For-Review: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2929004 (10Dzahn) [20:39:57] 10Traffic, 06Operations, 13Patch-For-Review: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2929006 (10Dzahn) [20:40:02] 10Traffic, 06Operations, 10Wikimedia-Mailing-lists: convert lists.wikimedia.org certificate to LetsEncrypt (deadline:2017-03-02) - https://phabricator.wikimedia.org/T154917#2929005 (10Dzahn) [20:41:43] 10Traffic, 06Operations: convert tendril to use Letsencrypt for SSL cert - https://phabricator.wikimedia.org/T154938#2929008 (10Dzahn) [20:42:31] 10Traffic, 06Operations: convert tendril to use Letsencrypt for SSL cert (deadline 2017-03-17) - https://phabricator.wikimedia.org/T154938#2929008 (10Dzahn) [20:49:39] 10Traffic, 06Operations: convert ganglia to use Letsencrypt for SSL cert (deadline: 2017-02-07) - https://phabricator.wikimedia.org/T154939#2929027 (10Dzahn) [20:51:58] 10Traffic, 06Operations, 13Patch-For-Review: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2929049 (10Dzahn) [20:54:12] 10Traffic, 06Operations: convert dumps to use Letsencrypt for SSL cert (deadline: 2017-04-26) - https://phabricator.wikimedia.org/T154940#2929052 (10Dzahn) [21:01:54] 10Traffic, 06Operations: convert archiva to use Letsencrypt for SSL cert (deadline 2017-05-08) - https://phabricator.wikimedia.org/T154942#2929084 (10Dzahn) [22:25:36] 10Traffic, 06Operations, 07Wikimedia-Incident: Investigate varnishd child crashes when multiple nodes get depooled/pooled concurrently - https://phabricator.wikimedia.org/T154801#2929228 (10greg) [22:25:50] 10Traffic, 10Monitoring, 06Operations, 07Wikimedia-Incident: Plot number of cached objects on a per-server per-DC basis - https://phabricator.wikimedia.org/T154864#2929229 (10greg) [22:47:49] 10Domains, 10Traffic, 10DNS, 06Operations: Donate wiktionary.pl to the Foundation - https://phabricator.wikimedia.org/T154826#2929270 (10tomasz) The domain is now the property of the Foundation, and MarkMonitor have set the DNS to ns[0, 1, 2].wikimedia.org. Does anything else need to be done here? [23:35:01] 10Traffic, 06Operations, 13Patch-For-Review: convert ganglia to use Letsencrypt for SSL cert (deadline: 2017-02-07) - https://phabricator.wikimedia.org/T154939#2929401 (10Dzahn) {F5263611} [23:35:44] 10Traffic, 06Operations, 13Patch-For-Review: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2929403 (10Dzahn) [23:35:47] 10Traffic, 06Operations, 13Patch-For-Review: convert ganglia to use Letsencrypt for SSL cert (deadline: 2017-02-07) - https://phabricator.wikimedia.org/T154939#2929402 (10Dzahn) 05Open>03Resolved [23:36:05] 10Traffic, 06Operations: convert ganglia to use Letsencrypt for SSL cert (deadline: 2017-02-07) - https://phabricator.wikimedia.org/T154939#2929027 (10Dzahn) [23:36:30] 10Traffic, 06Operations, 13Patch-For-Review: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2240497 (10Dzahn) [23:41:34] 10Traffic, 06Operations, 10Wikimedia-Mailing-lists: convert lists.wikimedia.org certificate to LetsEncrypt (deadline:2017-03-02) - https://phabricator.wikimedia.org/T154917#2929416 (10RobH) So the old call for the certificate file also set group ownership to debian-exim. I had to remove that call from the L... [23:41:52] 10Traffic, 06Operations, 13Patch-For-Review: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2929417 (10RobH) [23:47:58] 10netops, 06Operations: cr2-esams<->cr2-eqiad link down - https://phabricator.wikimedia.org/T154952#2929438 (10faidon)