[00:36:52] 10Traffic, 10Operations, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs, 10iOS-app-Bonefish-On-A-Balloon: Wikipedia iOS apps sending harmful bursts of traffic synchronized to the top of the hour, especially at 22:00 UTC - https://phabricator.wikimedia.org/T264881 (10Tsevener) [08:35:24] look at this beauty here https://cache-tests.fyi/ [09:09:06] nice [09:11:06] heads up if you see on strange traffic logs coming from an internal ip with user agent WMFBackups [09:15:30] jynus: ack [09:17:49] I sent you a meeting invite for Oct 28 [09:19:25] 10HTTPS, 10Traffic, 10Operations: HSTS preload for wmfusercontent.org - https://phabricator.wikimedia.org/T132452 (10Nintendofan885) [09:23:26] ema: I wonder if we should run those caching tests against our varnish/ATS builds and configs [09:26:15] vgutierrez: at least some for sure [09:39:45] 10netops, 10Operations, 10cloud-services-team (Kanban): Enable L3 routing on cloudsw nodes - https://phabricator.wikimedia.org/T265288 (10aborrero) Announcement sent to the community: https://lists.wikimedia.org/pipermail/cloud-announce/2020-October/000331.html [10:37:11] 10netops, 10Operations: cr3-esams linecard diversity issue - https://phabricator.wikimedia.org/T262524 (10ayounsi) a:05ayounsi→03wiki_willy @wiki_willy is that something that could be planned for next Q? It should be trivial enough for the DC remote hands. [10:42:13] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Operations, 10Thumbor: Frequent "Error: 429, Too Many Requests" errors on pages with many (>50) thumbnails - https://phabricator.wikimedia.org/T266155 (10Gilles) [10:45:23] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Operations, 10Thumbor: Frequent "Error: 429, Too Many Requests" errors on pages with many (>50) thumbnails - https://phabricator.wikimedia.org/T266155 (10Gilles) @ema can you confirm that int-front cache status in the response means that the 429 was em... [11:22:40] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Operations, 10Thumbor: Frequent "Error: 429, Too Many Requests" errors on pages with many (>50) thumbnails - https://phabricator.wikimedia.org/T266155 (10ema) >>! In T266155#6574047, @Gilles wrote: > @ema can you confirm that int-front cache status in... [11:56:49] 10Traffic, 10netops, 10DNS, 10Operations, and 2 others: Cloud: define relationship between wikimediacloud.org domain, CIDR prefixes and netbox automation - https://phabricator.wikimedia.org/T266331 (10aborrero) [11:57:11] 10Traffic, 10netops, 10DNS, 10Operations, and 2 others: Cloud: define relationship between wikimediacloud.org domain, CIDR prefixes and netbox automation - https://phabricator.wikimedia.org/T266331 (10aborrero) p:05Triage→03Medium [12:45:05] gilles: ok to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/636012/ now or would you rather wait for Monday? [12:45:17] ema: it's fine to do it now, thanks! [12:49:11] gilles: FYI you can add something like "Hosts: thumbor1001.eqiad.wmnet" to the commit message and then comment with "check experimental" on gerrit to get a jenkins-pcc run on the change [12:49:30] pl [12:49:32] ok [12:49:34] not that we expect anything crazy out of this specific patch, but it's a nice feature I think [12:53:03] right, so now jenkins-bot said "Experimental build succeeded", and at the bottom of https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler-test/588/console there's a link to https://puppet-compiler.wmflabs.org/compiler1003/588/ with the change [12:53:32] merging! [12:57:34] cool [12:57:55] if you can roll-restart thumbor with cumin that'd be great: https://wikitech.wikimedia.org/wiki/Thumbor#Restarting if I do it myself I have to do it host by host [12:57:59] thanks! [12:59:50] sure, I'll force a puppet run first [13:07:26] gilles: done [13:11:38] ema: thank you [17:48:47] 10netops, 10Operations: cr3-esams linecard diversity issue - https://phabricator.wikimedia.org/T262524 (10wiki_willy) a:05wiki_willy→03ayounsi Sounds good @ayounsi, thanks for the heads up. Just shoot open a dc-ops task with the ops-esams project tag, and @RobH will work with you and Iron Mountain on gett... [18:16:02] 10Traffic, 10Operations, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs, 10iOS-app-Bonefish-On-A-Balloon: Wikipedia iOS apps sending harmful bursts of traffic synchronized to the top of the hour, especially at 22:00 UTC - https://phabricator.wikimedia.org/T264881 (10CDanis) Thanks @Tsevener ! Detailed repl... [21:47:34] 10Traffic, 10Operations, 10Patch-For-Review: Update certspotter - https://phabricator.wikimedia.org/T204993 (10Dzahn) What is the status of this nowadays? I ran across it in a different matter while looking for absented crons and found the TODO and link in code that points over here and to the pending patch... [22:08:33] 10netops, 10Operations, 10Sustainability (Incident Followup): Audit Juniper EX snapshots version - https://phabricator.wikimedia.org/T262290 (10Krinkle) [22:08:38] 10netops, 10Operations, 10ops-eqiad, 10Sustainability (Incident Followup): eqiad row D switch fabric recabling - https://phabricator.wikimedia.org/T256112 (10Krinkle) [22:08:43] 10netops, 10Operations, 10Sustainability (Incident Followup): Configure BGP route damping on Anycast sessions - https://phabricator.wikimedia.org/T262372 (10Krinkle) [23:42:40] 10Traffic, 10Desktop Improvements, 10Operations, 10Readers-Web-Backlog, 10Wikimedia-production-error: Connection closed while downloading PDF of articles - https://phabricator.wikimedia.org/T266373 (10Urbanecm) Can reproduce from toolforge: ` urbanecm@tools-sgebastion-07 ~/tmp $ wget https://fr.wikiped...