[06:22:11] morning all! I'm upgrading deployment-prep app servers to buster and puppet is failing to install package ttf-ubuntu-font-family, it's not available in debian repos (https://phabricator.wikimedia.org/T103325) but is in {jessie,stretch}-wikimedia, but not in buster-wikimedia, is that expected/should I do something to it? [06:24:46] Majavah: use a different font. The Ubuntu font is considered non-free, see https://en.wikipedia.org/wiki/Ubuntu_(typeface)#Ubuntu_Font_Licence [06:25:01] legoktm: but puppet wants to install it :/ [06:25:02] we shouldn't be using it at all tbh [06:25:33] I think we should just remove it [06:25:39] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/mediawiki/manifests/packages/fonts.pp#33 [06:27:15] package { 'ttf-ubuntu-font-family': ensure => absent } [06:28:24] it's unfortunate that Ubuntu of all projects released a font under a non-free license, but that's out of our control :( [06:31:32] https://gerrit.wikimedia.org/r/675357 [06:37:20] how do the current appservers running buster even work? shouldn't that be failing to install there too? [06:40:02] eh, wtf, it's pulling them from the non-free component [06:40:08] why do we even have that enabled? [06:42:37] firmware I guess [09:00:40] random puppet/historical question, why is mlocate banned/forced uninstall from the machines? [09:02:23] this is all I found (and it's 10 years old xd) https://gerrit.wikimedia.org/r/c/operations/puppet/+/250 [09:13:17] dcaro: at a guess, because having periodic heavy disk i/o outweighs the benefits [09:14:49] hmm, seems a bit blunt to enforce everywhere though, is there a way to bypass the enforcement on selected machines? [09:23:23] not that i can see. it's enforced by `base::standard_packages` [10:24:19] hi, another beta-only hiera patch, could someone merge https://gerrit.wikimedia.org/r/675485 please [10:25:33] Majavah: merging [10:25:39] thankyou [12:53:35] one more: https://gerrit.wikimedia.org/r/675503 [13:09:45] Majavah: looking [13:10:23] merged [13:21:15] just a heads up that the new buster debian-installer image was updated this morning so builds/reimages should work as normal. please let me know if you see any issues [13:23:13] effie: ^^^ [13:24:05] :D thank you both [13:24:17] I will let you know if something goes wrong [14:04:40] is it expected that newly installed parsoid nodes have parsoid/js installed and running? I thought everything was already migrated to parsoid/php [17:27:59] https://gerrit.wikimedia.org/r/c/operations/puppet/+/675559 last deployment-prep hiera change for today, ty in advance [17:36:25] Majavah: merged [21:59:48] Reedy: re https://phabricator.wikimedia.org/T278599 - the 404 for the http one is strange, that seems like a separate issue? [22:00:02] Does it work when you use deployment-mediawiki-07 instead of localhost? [22:00:13] Yeah, I'm not complaining about the HTTP issue particularly [22:00:22] more "I can't use HTTP as a workaround" is an aside [22:01:03] https://wikitech.wikimedia.org/wiki/Debugging_in_production#Locally [22:01:06] https://phabricator.wikimedia.org/T190111 [22:01:19] "localhost" is special and catches into health monitor [22:01:59] reedy@deployment-mediawiki-07:~$ curl -k -H "Host: en.wikipedia.beta.wmflabs.org" https://deployment-mediawiki-07/wiki/Special:Export [22:01:59] curl: (7) Failed to connect to deployment-mediawiki-07 port 443: Connection refused [22:02:11] oh https on mwdebug works with localhost, and actually calls mediawiki? [22:02:20] interesting, maybe the health vhost is port 80-only [22:02:41] two thigns cancel out [22:03:33] Reedy: curl -k -H "Host: en.wikipedia.beta.wmflabs.org" http://deployment-mediawiki-07/wiki/Special:Export -i [22:03:33] HTTP/1.1 302 Found [22:03:39] (http) [22:04:19] oh, well [22:04:21] redirect to https [22:04:24] that's not so useful [22:04:41] that wasn't there in the past [22:30:35] fwiw, to get the correct certificate (no need for -k) one has to use --resolve instead of Host: [22:30:50] I still don't do this by habit yet, but there's increasingly scnearios that call for it [22:30:53] curl -v 'https://test.wikipedia.org/w/load.php' --resolve "test.wikipedia.org:443:$(hostname -i)" -i [22:31:05] has the downside of specifying hostname twice, which bothers me indeed [22:31:21] but seems to be the "right" way [22:31:43] curl uses the hostname of the connection rather than the Host header to validate [22:31:59] which I guess makes sense since technically it's not supposed to care/know what headers will be sent [22:32:13] eventhough it could peek at what -H you specified of course [22:41:54] ah --resolve '*:443:…" is supported! [22:41:55] * Krinkle updates docs [22:46:29] no wildcard for port unfortunately, and it just ignores the whole thing if you use http without any warning [22:46:43] --connect-to "::$(hostname -i)" [22:46:45] that seems to work [23:04:23] ok, now up at https://wikitech.wikimedia.org/wiki/Debugging_in_production#Locally [23:04:35] but remains the same that for Beta, https is broken and http just redirects to https [23:06:53] but.. [23:06:54] curl -i --connect-to "::$HOSTNAME" 'http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page' -H "X-Forwarded-Proto: https" [23:07:10] Thats the currently working way for beta! [23:08:32] That doc is fairly amusing [23:08:33] >Prior to 2015, it was also possible to simply use http://localhost but per T190111 that now uses the VirtuallHost for status-check connections, which doesn't expose MediaWiki. [23:08:41] T190111: VirtualHost for mod_status breaks debugging Apache/MediaWiki from localhost (on jobrunners) - https://phabricator.wikimedia.org/T190111 [23:19:56] hehe yeah, I've left a trail there as external brain [23:23:56] it's still odd it works fine in production [23:25:08] Reedy: yeah, that's because mwdebug doesn't have the added role [23:25:20] the debug app server has everything [23:25:25] s/debug/beta/ [23:25:36] debug + app + api + job + canary [23:25:48] it also fails on some prod hosts [23:53:04] Krinkle: it's on my mailman3 task list to fix the mod_status thing...we just need a newer version of the prometheus stuff