[06:59:55] 10netops, 10Operations, 10Patch-For-Review, 10Wikimedia-Incident: Juniper HA audit - https://phabricator.wikimedia.org/T191667 (10Krinkle) [07:01:02] 10Traffic, 10Operations, 10Wikimedia-Incident: Investigate 2018-04-10 global traffic drop - https://phabricator.wikimedia.org/T191940 (10Krinkle) 05Open>03Resolved [07:01:05] 10Traffic, 10Operations, 10Wikimedia-Incident: Investigate 2018-04-10 global traffic drop - https://phabricator.wikimedia.org/T191940 (10Krinkle) [07:01:14] 10Traffic, 10Operations, 10Wikimedia-Incident: Investigate 2018-04-10 global traffic drop - https://phabricator.wikimedia.org/T191940 (10Krinkle) [08:43:09] 10Wikimedia-Apache-configuration, 10Operations, 10Chinese-Sites, 10Patch-For-Review, 10User-Urbanecm: All "zh-my" variant page views get 404 Not Found on zh.wikipedia.org - https://phabricator.wikimedia.org/T198371 (10RazeSoldier) @Urbanecm Any progress? [09:25:58] 10Wikimedia-Apache-configuration, 10Operations, 10Chinese-Sites, 10Patch-For-Review, 10User-Urbanecm: All "zh-my" variant page views get 404 Not Found on zh.wikipedia.org - https://phabricator.wikimedia.org/T198371 (101233thehongkonger) Should be deployment problem? [10:26:36] [cross-posting] as heads up we're planning to upgrade cumin on sarin shortly, so use neodymium instead if you need to run anything cumin-related. We'll advice when we're about to upgrade neodymium too [10:57:34] (advise) [11:05:07] indeed :) [11:32:37] 10Traffic, 10Operations, 10TemplateStyles, 10Wikimedia-Extension-setup, and 4 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410 (10Tgr) [11:32:42] 10Traffic, 10Operations, 10TemplateStyles, 10Wikimedia-Extension-setup, and 4 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410 (10Deskana) [13:10:28] so the new cumin version is deployed on sarin together with the upgrade scripts ported to python3 [13:10:51] I'll test wmf-upgrade-and-reboot w/ pinkunicorn [13:13:08] thanks ema! [13:14:34] ha, the upgrade process failed installing /var/cache/apt/archives/linux-image-4.9.0-0.bpo.6-amd64-dbg_4.9.88-1+deb9u1~bpo8+1_amd64.deb [13:15:04] and the script stopped as expected, so all good on the cumin side [13:15:46] ahahaha [13:16:08] (No space left on device) [13:17:34] volans: good that we're testing the edge cases! :) [13:18:10] ehehe, true! [13:18:29] let's see if the normal case works too :D [13:18:59] let's do that, yes [13:19:34] kernel dbg package removed, trying again [13:22:24] ema: elukey moved a log of yours to stat1006(?), should rm that if you don't need it [13:22:46] not sure if you saw that :) [13:22:59] paravoid: yes thank you! [13:23:05] already removed :) [13:23:07] ah perfect [13:25:20] volans: the normal case works as well :) [13:25:29] nice! thanks for testing it [13:41:20] ema: the upgrade_varnish one I guess is not testable right? (or too complicated) [13:41:38] volans: testing it now (without actually downgrading varnish first) [13:42:06] oh nice! I thought it was necessary to downgrade and might be painful [13:44:35] nice to follow along `journalctl -f` and see all the varnish "ecosystem" reacting to things (varnishkafka, vhtcpd, varnishmtail, ...) :) [13:45:16] volans: all good! [13:46:45] volans: is this urllib3 info message expected? [13:46:48] 2018-07-19 13:44:22 [INFO] (ema) urllib3.connectionpool::_new_conn: Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet [13:46:58] I don't recall seeing it in the past [13:47:12] ema: yeah it's new in the sense that now urllib3 by default logs that [13:47:20] I was in doubt if silencing it or not [13:47:44] also bare in mind that we're using an older version of urllib3 for strange reasons ;) [13:48:23] python3-urllib is older than python-urllib3 on those hosts, and we decided to not upgrade them right away today because it might have unforeseen consequences [13:49:38] we'll be migrating those hosts to stretch anyway (hardware replacement) soon [13:51:37] volans: I'm confused :) [13:51:49] so urllib3 would be urllib2 but better [13:52:05] python-urllib3 is the python2 version of this "better urllib2" [13:52:20] python3-urllib3 is the python3 version [13:52:31] but I don't see any python3-urllib(2)? [13:53:37] you have all the rights to be confused! [13:53:41] so urllib is part of stdlib [13:54:04] urllib2 *was* part of stdlib in py2 [13:54:22] and has been splitted into urllib.request and urllib.error in py3, still stdlib [13:54:29] urllib3 is 3rd party [13:54:41] so has it's external package both python- and python3- [13:56:41] I knew I didn't want to know [13:56:48] so urllib2 has been renamed to urllib? [13:56:57] merged into, yes [13:57:17] remember, you asked for it! :D [13:59:38] ah maybe I got it [13:59:41] 15:48 < volans> python3-urllib is older than python-urllib3 [13:59:56] you meant python3-urllib3 here, right? ^ [14:00:37] for a second I thought there was an external py3 version of urllib2 too [14:01:40] there's also the confusing situation that python-urllib3 1.19.1-1 was backported to jessie-wikimedia without reflecting that in the version number [14:01:54] no no, the installed version on sarin/neo is from jessie, while the python2 version is from backport [14:02:06] and the person making that backport only backported python-urllib3, but not python3-urllib3 [14:02:38] it's not from debian-backports, then it would have the bpo suffix [14:03:17] see jessie-wikimedia/backports/p/python-urllib3 [14:03:47] on install1002.wikimedia in /srv/wikimedia [14:06:01] [cross posting] so far all good on sarin, we're planning to upgrade cumin/reimage/debdeploy/etc. on neodymium too, any blocker in progress? [14:06:47] volans: go ahead [14:07:10] thx [14:25:26] [cross-posting] cumin/debdeploy/reimage/other-tools upgrade completed, let me know if you encounter any issue [15:58:16] 10Traffic, 10Operations, 10Patch-For-Review: Pick up a suitable ACME library for certcentral - https://phabricator.wikimedia.org/T199717 (10Vgutierrez) I've been playing a little bit with the ACME library from certbot and it looks promising. on https://gerrit.wikimedia.org/r/#/c/operations/software/certcentr... [23:13:09] 10Traffic, 10Operations, 10ops-eqsin: rack/setup scs-eqsin.mgmt.eqsin.wmnet - https://phabricator.wikimedia.org/T181569 (10BBlack) 05stalled>03Resolved Assuming the above is true :) [23:38:04] 10Traffic, 10Operations, 10Patch-For-Review: WP Zero workarounds for eqsin - https://phabricator.wikimedia.org/T189250 (10BBlack) Thanks @DFoy for chasing down the contracts and issues here, we're clear to remove these workaround and close up this ticket. \o/ [23:46:37] 10Traffic, 10Operations: Enable Service in Asia Cache DC - https://phabricator.wikimedia.org/T156026 (10BBlack) [23:47:02] 10Traffic, 10Operations, 10Patch-For-Review: WP Zero workarounds for eqsin - https://phabricator.wikimedia.org/T189250 (10BBlack) 05Open>03Resolved [23:54:22] 10Traffic, 10Operations: Backend naming in VCL needs to use fqdn+port - https://phabricator.wikimedia.org/T138546 (10Krinkle)