[06:39:39] hello everybody [06:39:56] tomorrow it starts the apachecon NA and there is a "CDN" track https://apachecon.com/acah2020/tracks/content.html [08:06:58] elukey: interesting, I didn't know that Disney also uses ATS [08:08:35] and exactly one month from now is the ATS summit too, lots of virtual presentations at horrible times for me on the horizon! [08:08:55] :D [08:24:35] 10netops, 10Operations, 10decommission-hardware, 10ops-eqiad: Decommission asw-c-eqiad - https://phabricator.wikimedia.org/T208734 (10ayounsi) I'm seeing interfaces down on asw2-c-eqiad, and I'm not able to ssh to asw-c-eqiad, so I guess some of those steps have been done? As they are now alerting I'm del... [08:26:52] 10netops, 10Operations, 10decommission-hardware, 10ops-eqiad: Decommission asw-c-eqiad - https://phabricator.wikimedia.org/T208734 (10ayounsi) [08:44:32] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: eqiad row D switch upgrade - https://phabricator.wikimedia.org/T172459 (10ayounsi) 05Stalled→03Declined Forgot about that old task! Not needed anymore as we're not using multicast anymore. [08:46:53] 10netops, 10Operations, 10ops-eqiad: eqiad row D switch fabric recabling - https://phabricator.wikimedia.org/T256112 (10Marostegui) [08:46:57] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: eqiad row D switch upgrade - https://phabricator.wikimedia.org/T172459 (10Marostegui) [08:50:30] 10netops, 10DBA, 10Operations, 10ops-eqiad, and 3 others: Upgrade eqiad rack D4 to 10G switch - https://phabricator.wikimedia.org/T196487 (10ayounsi) @Cmjohnson the console port is still not responding, could you please have a look before today's maintenance? As we still need to configure the switch (and m... [09:33:28] 10netops, 10Operations: Configure BGP route damping on Anycast sessions - https://phabricator.wikimedia.org/T262372 (10ayounsi) 05Resolved→03Open When bird restarts on the centrallog servers it causes bird to bounce a few times: ` Sep 28 09:06:18 centrallog2001 bird: Shutting down Sep 28 09:06:18 centrallo... [09:34:33] 10netops, 10Operations: Configure BGP route damping on Anycast sessions - https://phabricator.wikimedia.org/T262372 (10ayounsi) p:05Medium→03High [09:35:12] 10Traffic, 10Operations, 10serviceops: puppetmaster[12]001: add TLS termination - https://phabricator.wikimedia.org/T263831 (10ArielGlenn) p:05Triage→03Medium [09:35:47] 10Traffic, 10Continuous-Integration-Infrastructure, 10Operations: contint.wikimedia.org: add TLS termination - https://phabricator.wikimedia.org/T263830 (10ArielGlenn) p:05Triage→03Medium [09:39:59] 10Domains, 10Traffic, 10Analytics-Radar, 10Operations, and 2 others: Blocking all third-party storage access requests - https://phabricator.wikimedia.org/T262996 (10ArielGlenn) p:05Triage→03Medium [09:47:54] 10Traffic, 10Operations: backport ipvsadm>=1.30 to buster-wikimedia or buster-backports - https://phabricator.wikimedia.org/T263788 (10ArielGlenn) p:05Triage→03Medium [09:48:04] 10Traffic, 10Operations: Switch to Maglev hashing ('mh') on LVS hosts - https://phabricator.wikimedia.org/T263797 (10ArielGlenn) p:05Triage→03Medium [09:48:49] 10Traffic, 10MediaWiki-REST-API, 10Operations: Route requests to the REST MediaWiki API to the api cluster - https://phabricator.wikimedia.org/T263729 (10ArielGlenn) p:05Triage→03Medium [09:54:49] 10Traffic, 10netops, 10Analytics, 10Operations: Turnilo: per-second rates for wmf_netflow bytes + packets - https://phabricator.wikimedia.org/T263290 (10ArielGlenn) p:05Triage→03Medium [09:55:45] 10Traffic, 10netops, 10Operations: experiment with reënabling compression between applayer's TLS terminators and edge caches - https://phabricator.wikimedia.org/T263288 (10ArielGlenn) p:05Triage→03Medium [09:59:05] 10Traffic, 10Operations, 10observability, 10Patch-For-Review, 10User-fgiunchedi: Aggregated metrics for ats-tls <-> clients ttfb percentiles - https://phabricator.wikimedia.org/T263536 (10fgiunchedi) [10:05:01] <_joe_> would it be crazy to run tcpdump on a load-balancer? [10:05:10] <_joe_> low-traffic one, not the high-traffic one [10:05:21] <_joe_> and I'm going to look for a specific destination port [10:09:39] 10Domains, 10Traffic, 10Analytics-Radar, 10Operations, 10Wikimedia-General-or-Unknown: WMF third-party cookies rejected - https://phabricator.wikimedia.org/T262882 (10ArielGlenn) p:05Triage→03Medium [10:32:52] _joe_: I've done it, be sure to be as specific as possible, but it works fine [10:33:18] <_joe_> XioNoX: I just needed to wait a bit it seems [12:04:56] 10Traffic, 10Operations, 10conftool, 10serviceops: confd's watch functionality appears to be partially broken when interacting with etcd 3.x - https://phabricator.wikimedia.org/T260889 (10ArielGlenn) [12:16:03] 10Traffic, 10Operations: cache_upload varnish-fe exhausting transient memory - https://phabricator.wikimedia.org/T249809 (10ArielGlenn) [12:28:05] 10netops, 10Operations: Configure BGP route damping on Anycast sessions - https://phabricator.wikimedia.org/T262372 (10ayounsi) The issue is that `ss -lun | fgrep -q :10514` often take more than 2s to complete and we don't let it retry. As it happen regularly, it sometimes happen right after the bird restart,... [12:33:57] 10Traffic, 10Operations, 10Wikimedia-Incident: Memory leak on ats-tls 8.0.6 - https://phabricator.wikimedia.org/T249335 (10ArielGlenn) [12:35:39] 10Traffic, 10Operations, 10serviceops, 10Performance-Team (Radar), 10Sustainability: Make CDN purges reliable - https://phabricator.wikimedia.org/T133821 (10ArielGlenn) [12:56:45] ema: bblack: I tried to split up my awful patches into sensible chunks, let me know if it makes sense or not [13:12:06] 10netops, 10DBA, 10Operations, 10ops-eqiad, and 2 others: Upgrade eqiad rack D4 to 10G switch - https://phabricator.wikimedia.org/T196487 (10Cmjohnson) @ayounsi I am not able to get the console to work on the new switch, it's plugged in, I verfied it worked by connecting to the current asw in d4 and get th... [13:30:59] cdanis: what happens if ISP_FILE isn't available? [13:32:06] ema: we'll syslog(LOG_CRIT, ...) and then carry on with our lives [13:32:36] excellent [13:34:22] ema: the VTCs test that case btw (and I did some extra testing locally of the actual functionality by copying the needed geoip files into my vagrant machine) [13:36:14] ack [13:36:58] cdanis: s/backend response/backend request/ in the commit log on https://gerrit.wikimedia.org/r/c/operations/puppet/+/630316 [13:38:54] ahh good catch! [13:46:02] cdanis: maybe you can add a couple of VTC tests expecting that the various headers are indeed set at the origin server level? [13:46:13] ema: you need the geoip files to do that :) [13:47:46] that makes sense :) [13:47:55] not sure if there are dummy ones available [13:48:19] is there an equivalent of the free GeoIP db for those files? [13:49:34] it's also fine to mock the values, but it would be nice to be able to test that the headers are actually there [13:50:01] yeah, I agree [13:50:17] I guess worst case we could MMDB-encode our own faux data [13:50:56] oh, what's this 👀 https://github.com/maxmind/MaxMind-DB/tree/master/test-data [13:51:38] aaawwww <3 [14:11:14] geolite needs an account, but is free [14:11:21] https://db-ip.com/ is free and doesn't need an account I believe [14:11:24] and is mmdb as well [14:15:17] I'm not even sure I want 'real' data here though, that can change over time [14:15:27] also, the test data is CC-BY-SA-3.0, so that's nice [14:20:18] ema: dumb question, what creates the /vagrant path on the VM? [14:21:54] cdanis: vagrant does, I think [14:22:23] ah, yes, you are right [14:23:44] 10netops, 10DBA, 10Operations, 10ops-eqiad, and 2 others: Upgrade eqiad rack D4 to 10G switch - https://phabricator.wikimedia.org/T196487 (10ayounsi) [14:43:57] https://etherpad.wikimedia.org/p/Traffic-2020-09-28 [14:53:51] thanks for doing what I thought we should have a browser extension for :P [15:02:08] sukhe: I just run `google-chrome https://etherpad.wikimedia.org/p/Traffic-$(date -I -d 'Monday')` ;) [15:02:43] haha [15:02:44] close enough [15:23:21] sorry sukhe I misspoke, it isn't `dpkg-source` that will download everything given a URL to a .dsc, it's `dget` (part of the devscripts package) [15:28:05] got it, thanks [16:37:24] 10netops, 10Operations, 10ops-eqiad: (Need by: 2019-09-30) upgrade msw1-eqiad from EX4200 to EX4300 - https://phabricator.wikimedia.org/T225121 (10Cmjohnson) 05Open→03Resolved Both have been updated [16:45:48] 10netops, 10Operations, 10fundraising-tech-ops: Automate diff and commit of frack ACL - https://phabricator.wikimedia.org/T260655 (10Jgreen) a:03Jgreen [16:58:53] 10Traffic, 10Analytics, 10Operations: ~1 request/minute to intake-logging.wikimedia.org times out at the traffic/service interface - https://phabricator.wikimedia.org/T264021 (10CDanis) [16:59:01] 10netops, 10Operations, 10ops-eqiad, 10Patch-For-Review: Rack and setup new eqiad row D switch stack (EX4300/QFX5100) - https://phabricator.wikimedia.org/T148506 (10Cmjohnson) [16:59:19] 10Traffic, 10Analytics, 10Operations: ~1 request/minute to intake-logging.wikimedia.org times out at the traffic/service interface - https://phabricator.wikimedia.org/T264021 (10CDanis) p:05Triage→03Low Clients will retry automatically so this isn't a huge deal, but it does merit investigation at some po... [17:31:33] 10Traffic, 10Analytics, 10Operations: ~1 request/minute to intake-logging.wikimedia.org times out at the traffic/service interface - https://phabricator.wikimedia.org/T264021 (10JAllemandou) Idea: Could missing-revisions (T215001) be related to this?