[04:18:50] 10Traffic, 06Operations, 13Patch-For-Review: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2922470 (10Dzahn) [04:19:08] 10Traffic, 06Operations, 13Patch-For-Review: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2240497 (10Dzahn) Icinga switched to LE just now. [09:42:03] 10netops, 06Operations, 10ops-codfw: asw-a7-codfw is down - https://phabricator.wikimedia.org/T154758#2922945 (10faidon) [09:48:34] 10Traffic, 06Operations, 10Pybal: Pybal not happy with DNS delays - https://phabricator.wikimedia.org/T154759#2922958 (10faidon) [09:49:49] 10netops, 06Operations, 10ops-codfw: asw-a7-codfw is down - https://phabricator.wikimedia.org/T154758#2922945 (10faidon) [10:46:11] 10netops, 06Operations, 10ops-codfw: asw-a7-codfw is down - https://phabricator.wikimedia.org/T154758#2923142 (10ema) The impact on varnish errors has been minimal. In codfw we've had two hiccups, one at 8:30 and another smaller one at 8:40 {F5241163} In ulsfo we had a small 503 spike at 8:51 {F5241168} We... [14:13:03] 10netops, 06Operations, 10ops-codfw, 05Security: asw-a7-codfw is down - https://phabricator.wikimedia.org/T154758#2923393 (10Reedy) [15:40:45] 10netops, 06Operations, 10ops-codfw, 05Security: asw-a7-codfw is down - https://phabricator.wikimedia.org/T154758#2923547 (10mark) The switch has been brought back up with a hard power cycle. We don't have a real indication yet of why it crashed, there's nothing concrete in the logs (local or otherwise),... [15:41:01] 10netops, 06Operations, 10ops-codfw, 05Security: asw-a7-codfw is down - https://phabricator.wikimedia.org/T154758#2923548 (10mark) p:05Unbreak!>03Normal [18:07:29] 10Traffic, 06Operations, 10media-storage: Cache and media (images) issues on all Wikimedia wikis - creates problems on upload, display and generation of thumbnails and files - https://phabricator.wikimedia.org/T154780#2923652 (10jcrespo) [18:08:02] 10Traffic, 06Operations, 10media-storage: Cache and media (images) issues on all Wikimedia wikis - creates problems on upload, display and generation of thumbnails and files - https://phabricator.wikimedia.org/T154780#2923652 (10jcrespo) @Aklapper The problems have been minimized, but this is still WIP. [18:08:53] 10Traffic, 06Operations, 10media-storage: Cache and media (images) issues on all Wikimedia wikis - creates problems on upload, display and generation of thumbnails and files - https://phabricator.wikimedia.org/T154780#2923953 (10jcrespo) [18:10:42] 10Traffic, 06Operations, 10media-storage: Cache and media (images) issues on all Wikimedia wikis - creates problems on upload, display and generation of thumbnails and files - https://phabricator.wikimedia.org/T154780#2923652 (10jcrespo) [18:15:08] 10Traffic, 06Operations, 10media-storage: Cache and media (images) issues on all Wikimedia wikis - creates problems on upload, display and generation of thumbnails and files - https://phabricator.wikimedia.org/T154780#2923982 (10Elisfkc) 05Open>03Resolved a:03Elisfkc Got upload wizard and Flickr2Common... [18:39:24] 10Traffic, 06Operations, 10media-storage: Cache and media (images) issues on all Wikimedia wikis - creates problems on upload, display and generation of thumbnails and files - https://phabricator.wikimedia.org/T154780#2924063 (10czar) Special:Upload is back up for me after being down for a while (T154790). I... [19:03:52] 10Traffic, 06Operations, 10media-storage: Cache and media (images) issues on all Wikimedia wikis - creates problems on upload, display and generation of thumbnails and files - https://phabricator.wikimedia.org/T154780#2924104 (10jcrespo) @czar There will be a post mortem, as usual, on https://wikitech.wikime... [19:07:22] 10Traffic, 06Operations, 10media-storage: Cache and media (images) issues on all Wikimedia wikis - creates problems on upload, display and generation of thumbnails and files - https://phabricator.wikimedia.org/T154780#2924114 (10czar) That's great but I meant that as a largely non-technical user, I didn't kn... [20:42:12] 10Traffic, 06Operations: Investigate varnishd child crashes when multiple nodes get depooled/pooled concurrently - https://phabricator.wikimedia.org/T154801#2924351 (10ema) [20:42:17] 10Traffic, 06Operations: Investigate varnishd child crashes when multiple nodes get depooled/pooled concurrently - https://phabricator.wikimedia.org/T154801#2924367 (10ema) p:05Triage>03Normal [23:29:21] 10Traffic, 10DNS, 06Operations: Donate wiktionary.pl to the Foundation - https://phabricator.wikimedia.org/T154826#2924998 (10tomasz) [23:29:59] 10Domains, 10Traffic, 10DNS, 06Operations: Donate wiktionary.pl to the Foundation - https://phabricator.wikimedia.org/T154826#2925024 (10hashar) [23:40:46] 10Domains, 10Traffic, 10DNS, 06Operations: Donate wiktionary.pl to the Foundation - https://phabricator.wikimedia.org/T154826#2924998 (10CRoslof) Send me an email and I can help you get the process started: croslof@wikimedia.org Please note that a domain generally cannot be transferred until 60 days after... [23:41:09] 10Domains, 10Traffic, 10DNS, 06Operations: Donate wiktionary.pl to the Foundation - https://phabricator.wikimedia.org/T154826#2925039 (10CRoslof) a:03CRoslof