[03:30:36] 10Traffic, 06Operations, 13Patch-For-Review: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2241887 (10BBlack) [03:31:03] 10Traffic, 06Operations, 13Patch-For-Review: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2240497 (10BBlack) Table at top updated. rt.wikimedia.org is on LE now as our first example with Apache. [03:31:44] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: Sort out letsencrypt puppetization for simple public hosts - https://phabricator.wikimedia.org/T132812#2241889 (10BBlack) Converted rt.wm.o, so now we have 1x apache + 1x nginx converted. Next I'm going to switch ubuntu+mirrors (both on carbon) to a si... [03:50:46] 07HTTPS, 10Traffic, 06Operations: Secure redirect service for large count of non-canonical / junk domains - https://phabricator.wikimedia.org/T133548#2241897 (10BBlack) [03:50:48] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: enable https for (ubuntu|apt|mirrors).wikimedia.org - https://phabricator.wikimedia.org/T132450#2241899 (10BBlack) [03:50:50] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: Sort out letsencrypt puppetization for simple public hosts - https://phabricator.wikimedia.org/T132812#2241893 (10BBlack) 05Open>03Resolved a:03BBlack SAN test worked as well. We'll likely have more refinement and bugfixes to deal with later when... [05:32:43] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: Sort out letsencrypt puppetization for simple public hosts - https://phabricator.wikimedia.org/T132812#2241955 (10Dzahn) {F3935166} :) [05:52:43] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: enable https for (ubuntu|apt|mirrors).wikimedia.org - https://phabricator.wikimedia.org/T132450#2242004 (10Chmarkine) @mark As (ubuntu|mirrors).wikimedia.org now supports HTTPS, could we update Wikimedia's Ubuntu mirror link to https://ubuntu.wikimedia.... [06:31:17] 10Traffic, 10domains, 06Operations, 06WMF-Legal: wikipedia.lol - https://phabricator.wikimedia.org/T88861#2242036 (10Dzahn) 05Resolved>03Open >>! In T88861#1890467, @Mschon wrote: > does wmf support https://letsencrypt.org ? Times have changed. The answer is now Yes. [06:32:03] 10Traffic, 10domains, 06Operations, 06WMF-Legal: wikipedia.lol - https://phabricator.wikimedia.org/T88861#2242041 (10Dzahn) T133548 [08:20:17] 10netops, 06Operations, 13Patch-For-Review: block labs IPs from sending data to prod ganglia - https://phabricator.wikimedia.org/T115330#2242184 (10fgiunchedi) thanks @dzahn ! there's another ~25 labs instances reporting data to misc-eqiad, likely because they are not running puppet or a self-hosted puppet m... [08:42:49] 10Traffic, 06Operations, 10Phabricator: Phabricator needs to expose notification daemon (websocket) - https://phabricator.wikimedia.org/T112765#2242206 (10fgiunchedi) this doesn't seem to be blocked on ops ATM, let us know when the pieces are in place and if we can help [09:25:03] 10Traffic, 06Operations, 13Patch-For-Review: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2242275 (10fgiunchedi) p:05Triage>03Normal [10:23:53] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: enable https for (ubuntu|apt|mirrors).wikimedia.org - https://phabricator.wikimedia.org/T132450#2242379 (10JanZerebecki) @Chmarkine Yes, please do so. [10:25:42] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: enable https for (ubuntu|apt|mirrors).wikimedia.org - https://phabricator.wikimedia.org/T132450#2242386 (10MoritzMuehlenhoff) Let's also update the description while we're at it :-) "Wikimedia's Ubuntu Archive mirror in Tampa, Florida" [10:30:15] 07HTTPS, 10Traffic, 06Operations: Secure redirect service for large count of non-canonical / junk domains - https://phabricator.wikimedia.org/T133548#2242401 (10BBlack) According to [[ https://letsencrypt.org/upcoming-features/ | https://letsencrypt.org/upcoming-features/ ]], they don't yet have [[ https://... [10:48:58] bblack: puppet runs are failing on carbon related to /usr/bin/sbin/acme-setup [10:52:00] 07HTTPS, 10Traffic, 06Operations: Secure redirect service for large count of non-canonical / junk domains - https://phabricator.wikimedia.org/T133548#2242447 (10BBlack) Also, on the SAN list length limits, LE has this to say: https://community.letsencrypt.org/t/sans-per-cert-and-sni-for-hosting-service/5105... [11:00:26] 07HTTPS, 10Traffic, 06Operations: Secure redirect service for large count of non-canonical / junk domains - https://phabricator.wikimedia.org/T133548#2242454 (10BBlack) We also need to decide on a data model, and especially about what kinds of hostnames we're going to support for the redirect domains. We ca... [11:00:51] 07HTTPS, 10Traffic, 06Operations: Secure redirect service for large count of non-canonical / junk domains - https://phabricator.wikimedia.org/T133548#2242455 (10BBlack) [11:02:27] 07HTTPS, 10Traffic, 06Operations: Secure redirect service for large count of non-canonical / junk domains - https://phabricator.wikimedia.org/T133548#2235376 (10BBlack) [11:05:45] moritzm: ack [11:06:05] seems like a bug in the script makes it think the already-OK cert isn't OK, so it keeps making new ones and hit an LE ratelimit [11:14:33] morning :) I merged the maintenance change for misc this morning, shall I merge also the error page? (https://gerrit.wikimedia.org/r/#/c/285363/1) [11:28:50] moritzm: fixed, thanks! (icinga's slow to notice, but puppet's ok now) [11:31:49] nice, thanks [11:33:15] elukey: yeah [11:35:08] ack! [13:12:00] 10Traffic, 06Operations: restrict upload cache access for private wikis - https://phabricator.wikimedia.org/T129839#2242749 (10fgiunchedi) p:05Triage>03Normal [13:13:17] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-Shop: https://store.wikimedia.org doesn't set HSTS header - https://phabricator.wikimedia.org/T128559#2078914 (10fgiunchedi) hi @Ppena, did shopify come back to you with an answer? thanks! [13:13:38] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-Shop: https://store.wikimedia.org doesn't set HSTS header - https://phabricator.wikimedia.org/T128559#2242757 (10fgiunchedi) p:05Triage>03Normal [13:17:18] 10Traffic, 06Operations, 10Phabricator: Phabricator needs to expose notification daemon (websocket) - https://phabricator.wikimedia.org/T112765#2242776 (10BBlack) We've basically never configured any websockets stuff through our #Traffic layer before. Phab isn't the only use-case, either. We also have `str... [13:22:53] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-Shop: https://store.wikimedia.org doesn't set HSTS header - https://phabricator.wikimedia.org/T128559#2242787 (10BBlack) [13:22:55] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: Enforce HTTPS+HSTS on remaining one-off sites in wikimedia.org that don't use standard cache cluster termination - https://phabricator.wikimedia.org/T132521#2242786 (10BBlack) [13:41:54] bblack: I've built 3.0.6+patches from 3.0.7. will add that to cp1008 for a test drive, ok? [13:42:11] ( https://gerrit.wikimedia.org/r/285641 ) [14:35:22] 10Traffic, 06Mobile-Apps, 06Operations: WikipediaApp for Android hits loads.php on bits.wikimedia.org - https://phabricator.wikimedia.org/T132969#2242936 (10fgiunchedi) p:05Triage>03Normal [14:58:09] 07HTTPS, 10Traffic, 10MediaWiki-General-or-Unknown, 06Operations, 10Wikimedia-General-or-Unknown: securecookies - https://phabricator.wikimedia.org/T119570#2242999 (10fgiunchedi) p:05Triage>03Normal [15:06:58] 07HTTPS, 10Traffic, 06Operations: Secure redirect service for large count of non-canonical / junk domains - https://phabricator.wikimedia.org/T133548#2243021 (10fgiunchedi) p:05Triage>03Normal [15:07:04] 10netops, 06Operations: HTCP purges flood across CODFW - https://phabricator.wikimedia.org/T133387#2243023 (10fgiunchedi) p:05Triage>03Normal [15:10:08] bblack: maybe a stupid question but I don't see https://yarn.wikimedia.org/ changing after my merge, so I am wondering if it is cached by Varnish somewhere [15:10:42] should use error_synth as well [16:04:36] <_joe_> bblack: now you can do [16:04:37] <_joe_> oblivian@cp3038:~$ sudo -i pool service=nginx [16:04:37] <_joe_> Pooling cp3038.esams.wmnet from service=nginx... [16:04:37] <_joe_> cp3038.esams.wmnet: pooled changed yes => yes [16:07:05] 10Traffic, 06Mobile-Apps, 06Operations: Millions of request per minute to /.well-known/apple-app-site-association producing 404s - https://phabricator.wikimedia.org/T130647#2243217 (10fgiunchedi) p:05Triage>03Low it looks like this has reduced dramatically since fixing {T111829} but [[ https://grafana.wi... [16:18:14] 10Traffic, 10MobileFrontend, 06Operations, 10Reading-Web: Seeing desktop text cache while browsing mobile sites - https://phabricator.wikimedia.org/T133441#2243266 (10MBinder_WMF) [16:35:37] _joe_: amazing :) [16:37:49] moritzm: yeah that sounds great [16:42:00] <_joe_> bblack: confctl will complain on palladium today because it can't speak with tcpircbot [16:42:07] <_joe_> but I didn't get to configure it [16:42:15] <_joe_> silence it witn --quiet [16:43:57] ok [16:50:02] 10Traffic, 07Varnish, 06Operations: varnishmedia: repeated calls to flush_stats() - https://phabricator.wikimedia.org/T132474#2243981 (10Nuria) [16:54:32] 10Traffic, 07Varnish, 06Operations, 13Patch-For-Review: varnishstatsd crashes with ValueError in vsl_callback without being restarted by systemd - https://phabricator.wikimedia.org/T132430#2243997 (10Nuria) [16:55:36] 10Traffic, 10Analytics, 06Operations: cronspam from cpXXXX hosts related to varnishkafka non existent processes - https://phabricator.wikimedia.org/T132346#2243999 (10Nuria) 05Open>03Resolved [17:08:27] bblack: ok, will install it tomorrow morning [17:10:25] moritzm: ok [17:10:28] 10Traffic, 10MediaWiki-Parser, 06Operations, 06Parsing-Team, and 6 others: Banners fail to show up occassionally on Russian Wikivoyage - https://phabricator.wikimedia.org/T121135#2244039 (10MBinder_WMF) [17:16:15] gehel: btw, the confctl updates _joe_ has been talking about above, those are the ones that get rid of the for loops two days ago :) [17:16:49] I glanced at the conversation and it looked nicer than what we did... [17:17:12] yeah :) [17:17:36] now we'll be able to break things with much simpler commands that look prettier [17:17:52] appearance IS important... [17:28:42] elukey: on your errorpage thing not producing expected results... [17:29:10] elukey: the way our errorpage html currently works, for confusing reasons, direct updates to the HTML won't be reflected until VCL is reloaded for some other reason [17:29:51] elukey: because that HTML file is read from VCL with std.fileread(), and std.fileread() only reads the file once (the first time a varnish thread starts). [17:30:01] elukey: the next time cache_misc VCL is reloaded, it will load new copies of the HTML into memory [17:30:34] elukey: so, if you want to move things along, make some 1-byte change to a misc-common VCL I guess [17:31:48] probably we could fix that somehow... [18:49:55] 10Traffic, 06Commons, 06Operations, 10media-storage, and 2 others: Deleted files sometimes remain visible to non-privileged users if permanently linked - https://phabricator.wikimedia.org/T109331#2244514 (10TerraCodes) >>! In T109331#2207393, @NahidSultan wrote: > Another one: https://upload.wikimedia.org/... [18:58:47] 10Traffic, 06Labs, 06Operations: check_dns needs to be rewritten - https://phabricator.wikimedia.org/T133791#2244564 (10BBlack) Sticking the #Traffic tag on because this affects monitoring of the production DNS authservers too, and that check_dns utility is awful to be relying on for monitoring something so... [19:41:00] 10netops, 06Operations, 13Patch-For-Review: block labs IPs from sending data to prod ganglia - https://phabricator.wikimedia.org/T115330#2244694 (10Dzahn) thanks for the fix. definitely fewer IPs in there now. the remaining ones i see currently: 10.68.16.147 (down) 10.68.17.204 (down) 10.68.16.53 (integrat... [19:52:21] 10netops, 06Operations, 13Patch-For-Review: block labs IPs from sending data to prod ganglia - https://phabricator.wikimedia.org/T115330#2244714 (10Dzahn) host 10.68.16.66 is special, look how many names that has: host 10.68.16.66 | wc -l 27 all in contintcloud.eqiad.wmflabs [20:08:23] 10netops, 06Operations, 13Patch-For-Review: block labs IPs from sending data to prod ganglia - https://phabricator.wikimedia.org/T115330#2244755 (10Dzahn) >>! In T115330#2244714, @Dzahn wrote: > host 10.68.16.66 is special, look how many names that has: hashar> mutante: chasemp relevant task is T126518 [20:13:43] 07HTTPS, 10Traffic, 06Operations, 13Patch-For-Review: enable https for (ubuntu|apt|mirrors).wikimedia.org - https://phabricator.wikimedia.org/T132450#2244769 (10faidon) I'm not convinced https for that is a good idea. apt doesn't support it by default — apt-transport-https isn't installed out of the box ev... [21:22:42] 10Traffic, 10domains, 06Operations, 06WMF-Legal: wikipedia.lol - https://phabricator.wikimedia.org/T88861#2244998 (10Dzahn) a:05Dzahn>03None [21:33:17] 10Traffic, 06Commons, 06Operations, 10media-storage, and 2 others: Deleted files sometimes remain visible to non-privileged users if permanently linked - https://phabricator.wikimedia.org/T109331#2245057 (10Dereckson) I've still a 404 for https://upload.wikimedia.org/wikipedia/commons/7/7f/Sajid-Monkey-Biz... [22:04:47] 10Traffic, 06Commons, 06Operations, 10media-storage, and 2 others: Deleted files sometimes remain visible to non-privileged users if permanently linked - https://phabricator.wikimedia.org/T109331#2245176 (10TerraCodes) It was on a windows mobile phone on LTE data (AT&T), so I don't have a console. When vie... [22:05:52] 10Traffic, 06Commons, 06Operations, 10media-storage, and 2 others: Deleted files sometimes remain visible to non-privileged users if permanently linked - https://phabricator.wikimedia.org/T109331#2245183 (10TerraCodes) >>! In T109331#2245176, @TerraCodes wrote: > It was on a windows mobile phone on LTE dat... [22:21:11] 10Traffic, 10MobileFrontend, 06Operations, 10Reading-Web-Backlog: Seeing desktop text cache while browsing mobile sites - https://phabricator.wikimedia.org/T133441#2232043 (10Jdlrobson) Probably related? https://twitter.com/therealprotonk/status/723963502856142848 [22:26:49] 10Traffic, 10MobileFrontend, 06Operations, 10Reading-Web-Backlog: Seeing desktop text cache while browsing mobile sites - https://phabricator.wikimedia.org/T133441#2245296 (10Jdlrobson) Sorry ignore that. Seems like a different problem. [22:27:27] 10Traffic, 10MobileFrontend, 06Operations, 10Reading-Web-Backlog: Seeing desktop text cache while browsing mobile sites - https://phabricator.wikimedia.org/T133441#2245299 (10BBlack) It looks like the same problem to me... I put this in our operations outbound for SoS, and raised this ticket the other day... [22:29:00] 10Traffic, 10MobileFrontend, 06Operations, 10Reading-Web-Backlog: Seeing desktop text cache while browsing mobile sites - https://phabricator.wikimedia.org/T133441#2245306 (10BBlack) Oh you're right, not the same problem. Still, if we don't understand the problem in this ticket, how do we know it's not go... [22:33:41] 10Traffic, 06Commons, 06Operations, 10media-storage, and 2 others: Deleted files sometimes remain visible to non-privileged users if permanently linked - https://phabricator.wikimedia.org/T109331#2245320 (10TerraCodes) >>! In T109331#2245057, @Dereckson wrote: > I've still a 404 for https://upload.wikimedi... [22:43:34] 10Traffic, 06Operations, 06Performance-Team, 13Patch-For-Review: Support HTTP/2 - https://phabricator.wikimedia.org/T96848#2245349 (10BBlack) So, I had intended to do the quick test and start the 24H test today, but I've run into some issues. Running the kernel object with `staprun` caused cp1065 to get i... [23:07:57] 10Traffic, 06Commons, 06Operations, 10media-storage, and 2 others: upload-lb.ulsfo.wikimedia.org still allow access to some deleted files - https://phabricator.wikimedia.org/T133819#2245416 (10Dereckson) [23:08:52] 10Traffic, 06Commons, 06Operations, 10media-storage, and 2 others: Deleted files sometimes remain visible to non-privileged users if permanently linked - https://phabricator.wikimedia.org/T109331#1546225 (10Dereckson) Thanks, that was useful.. [23:09:39] 10Traffic, 06Commons, 06Operations, 10media-storage, and 2 others: upload-lb.ulsfo.wikimedia.org still allow access to some deleted files - https://phabricator.wikimedia.org/T133819#2245434 (10Dereckson) [23:12:56] 10Traffic, 06Commons, 06Operations, 10media-storage, and 2 others: upload-lb.ulsfo.wikimedia.org still allow access to some deleted files - https://phabricator.wikimedia.org/T133819#2245445 (10Dereckson)