[00:38:13] PROBLEM - Host tools-docker-builder-01 is DOWN: CRITICAL - Host Unreachable (10.68.19.180) [02:29:21] 10Tool-Labs-tools-Zppixbot: Complete config - https://phabricator.wikimedia.org/T148944#2737275 (10Zppix) [02:30:03] 10Tool-Labs-tools-Zppixbot: Complete config - https://phabricator.wikimedia.org/T148944#2737287 (10Zppix) p:05Triage>03High [02:40:45] hi! a question regarding tool labs: how can I force my tool to be https-only? as far as I understand, tool works behind a reverse proxy, so it sees every request as 'http'. are original headers available somehow? maybe there's another server-side way? I'm aware I can do that with JS too [02:49:20] Leloiandudu, original headers wouldn't include whether it's http or https [02:49:56] TLS/SSL takes place above the HTTP layer [02:50:07] Krenair, I meant lightttpd's $HTTP["scheme"] [02:51:05] Leloiandudu, https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/dynamicproxy/templates/urlproxy.conf;HEAD$146 [02:51:24] the tools proxy will set X-Forwarded-Proto [02:51:27] you just need to check that header [03:00:03] Krenair, thanks! is there a way to read that value in lighttpd's config file? I'm not a lighttpd expert [03:00:34] I've never touched lighttpd [03:00:48] what file are you looking at Leloiandudu ? [03:01:15] Krenair, ~/.lighttpd.conf [03:05:32] tools.chie-bot, tools.rfa-tool or tools.fountain ? [03:06:15] looks like fountain [03:07:08] So I see this line [03:07:08] # accesslog.format = "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" [03:11:13] this doesn't look promising: https://redmine.lighttpd.net/boards/2/topics/6028 [03:12:50] Krenair, fountain. this line is commented :) Thanks for your help [03:12:56] I have another question: [03:13:11] I have a database at tools.labsdb [03:13:22] I'm not done with the first yet but okay :) [03:13:37] is it backed up regulary? If so - how often? if not - what's the preffered way to configure backups? [03:13:49] https://stackoverflow.com/questions/2969824/conditional-lighttpd-configuration-based-on-request-headers [03:14:12] Krenair, I googled - looks like there is no way to put a header value into a conditional statement [03:14:18] Yes, exactly lol [03:14:24] that's pretty sad thought [03:14:26] *though [03:14:31] thanks for your help :) [03:15:33] Leloiandudu, re tools.labsdb I'm actually not sure [03:16:36] ok, thanks! maybe someone else will come and answer that :) [03:25:12] I checked the tools he's member of [03:25:23] (the tools admin in https://redmine.lighttpd.net/boards/2/topics/6028 ) [03:25:35] there's nothing that looks like it solves your x-forwarded-proto problem [03:28:17] mod_extforward is 500ish lines of C: https://github.com/lighttpd/lighttpd1.4/blob/master/src/mod_extforward.c [03:29:20] has some code for trusting proxies which might not be as necessary for just x-forwarded-proto [03:29:41] oh, wait: https://redmine.lighttpd.net/issues/1499 [03:30:17] I didn't look through the file fully, it also gained x-forwarded-proto support in that bug [03:34:36] Krenair, I'm not sure if I follow... [03:35:07] also, how is that in https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/dynamicproxy/templates/urlproxy.conf this works: [03:35:09] if ($http_x_wikimedia_debug) { [03:35:17] where does this var come from? [03:36:11] if I try the same with $http_x_forwarded_proto it throws this error: "invalid character in variable name" at first '_' char [03:36:56] that's nginx config [03:36:58] not lighttpd [03:37:09] it's config for the tools proxy [03:37:28] ahhh... damn! [03:37:42] I see now lol, thanks [03:40:26] tools-webgrid-lighttpd-[1401-1416,1418].tools.eqiad.wmflabs (17) [03:40:29] lighttpd/1.4.33 (ssl) - a light and fast webserver [03:40:32] tools-webgrid-lighttpd-[1201-1210].tools.eqiad.wmflabs (10) [03:40:35] lighttpd/1.4.28 (ssl) - a light and fast webserver [03:44:22] are we planning to upgrade them to 1.4.40 anytime soon? :) [03:45:15] !log tools reset host keys for tools-puppetmaster-02 on -01, looks like it was recreated 5-6 days ago [03:45:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [03:45:53] Leloiandudu, dunno [03:50:08] Leloiandudu, looks like we just use the lighttpd package out of nova.clouds.archive.ubuntu.com [03:51:39] which probably just follows debian packaging [03:54:35] I see, thanks [03:55:36] the debian pkg-lighttpd-maintainers list might know more [06:50:25] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:25:24] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [13:37:08] Hi, I was pointed to the -labs channel with my isuee. I have no access to http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-operations/?C=M;O=D (channel archive of -operations). It throws 403. But when I'm connected through my personal proxy server I have no problem. [13:37:22] My no-proxy IP is 213.226.216.2 , my proxy IP is 94.143.232.92 [13:37:45] I think maybe my no-proxy IP is listed in something like blacklist... [13:38:48] I was told that IP shouldn't make any difference but from unknown reason it makes. [13:39:11] PROBLEM - Host tools-flannel-etcd-01 is DOWN: PING CRITICAL - Packet loss = 100% [13:39:14] Can anybody have a look at it? [14:32:46] (03Abandoned) 10Zppix: Add labs/tools/zppixbot [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/317380 (owner: 10Zppix) [14:47:51] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:51:26] !log deployment-prep T142288: Shut off -pdf02 and -conftool [14:51:27] T142288: Consolidate, remove, and/or downsize Beta Cluster instances to help with [[wikitech:Purge_2016]] - https://phabricator.wikimedia.org/T142288 [14:51:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, Master [15:17:06] (03PS1) 10Lokal Profil: Ensure orderby is allways an array [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317524 (https://phabricator.wikimedia.org/T148773) [15:23:40] (03PS1) 10Ladsgroup: Add WikibaseLexeme to #wikidata-feed channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/317526 [15:23:45] (03CR) 10Jean-Frédéric: [C: 032] Ensure orderby is allways an array [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317524 (https://phabricator.wikimedia.org/T148773) (owner: 10Lokal Profil) [15:24:27] (03CR) 10Paladox: [C: 032] Add WikibaseLexeme to #wikidata-feed channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/317526 (owner: 10Ladsgroup) [15:27:54] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:28:24] (03Merged) 10jenkins-bot: Ensure orderby is allways an array [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317524 (https://phabricator.wikimedia.org/T148773) (owner: 10Lokal Profil) [15:28:49] (03Merged) 10jenkins-bot: Add WikibaseLexeme to #wikidata-feed channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/317526 (owner: 10Ladsgroup) [15:30:19] (03CR) 10Jean-Frédéric: [C: 032] Pass positional arguments to nosetests via tox [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/314172 (owner: 10Jean-Frédéric) [15:30:53] (03CR) 10Jean-Frédéric: "I’m confident these issues are due to a corrupted pywikibot install or smth like that." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/314172 (owner: 10Jean-Frédéric) [15:31:17] (03Merged) 10jenkins-bot: Pass positional arguments to nosetests via tox [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/314172 (owner: 10Jean-Frédéric) [15:35:00] !log tools.lolrrit-wm deploying https://gerrit.wikimedia.org/r/317526 [15:35:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lolrrit-wm/SAL, Master [15:36:17] Amir1 ^^ ive deployed it now :) [15:36:38] yeah, already spammed the channel a lot [15:36:55] Oh [15:53:59] (03CR) 10Jean-Frédéric: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309858 (https://phabricator.wikimedia.org/T112460) (owner: 10Lokal Profil) [16:01:20] 06Labs, 10Cognate: Request creation of Cognate labs project - https://phabricator.wikimedia.org/T148889#2738517 (10Andrew) 05Open>03Resolved a:03Andrew @addshore, you are now project admin in the Cognate project. You can add additional users or admins as you see fit. [16:01:23] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#2738521 (10Andrew) [16:01:41] 06Labs: Request creation of hound labs project - https://phabricator.wikimedia.org/T148573#2726640 (10Andrew) This is almost certainly fine but I'm waiting on a definite thumbs up from legal regarding the weird licensing terms for React. [16:03:08] 06Labs: Request creation of status labs project - https://phabricator.wikimedia.org/T148569#2726492 (10Andrew) @matthewrbowker the Labs operators discussed this a bit today. In general we /try/ to provide Labs monitoring tools but they're admittedly not so great, so you are welcomed (and encouraged) to work on... [16:06:59] 06Labs: Request increased quota for services-test labs project - https://phabricator.wikimedia.org/T148869#2735296 (10Andrew) I can grant a temporary quota increase while you debug this problem. HOWEVER, as far as I can tell there is no such project named services-test. Can you clarify? [16:09:45] 06Labs: Request increased quota for services-test labs project - https://phabricator.wikimedia.org/T148869#2738559 (10Eevans) >>! In T148869#2738555, @Andrew wrote: > I can grant a temporary quota increase while you debug this problem. > > HOWEVER, as far as I can tell there is no such project named services-te... [16:11:50] 06Labs, 10Tool-Labs: Different mono versions on "login server" and on grid - https://phabricator.wikimedia.org/T148984#2738562 (10Kotz) [16:22:51] 06Labs, 10Tool-Labs: Different mono versions on "login server" and on grid - https://phabricator.wikimedia.org/T148984#2738645 (10AlexMonk-WMF) 05Open>03Invalid You forgot to specify the correct distribution for jsub. Leaving it unspecified defaults to precise, which is deprecated and will have everything... [16:35:16] 06Labs, 07Tracking: Existing Labs project quota increase requests (Tracking) - https://phabricator.wikimedia.org/T140904#2738715 (10Andrew) [16:35:20] 06Labs: Request increased quota for services-test labs project - https://phabricator.wikimedia.org/T148869#2738714 (10Andrew) 05Open>03stalled [16:36:06] Hey, andrewbogott? Got any ideas for a better project name on T148569 ? I'm at a loss... [16:36:06] T148569: Request creation of status labs project - https://phabricator.wikimedia.org/T148569 [16:36:34] yuvipanda: ^^ [16:37:22] Matthew_: tools-monitoring? :) [16:37:47] Would labs-monitoring be ok? [16:38:14] How can I check if a name is used? [16:38:23] Matthew_: hmm, how about community-monitoring? [16:38:27] 06Labs, 10Cognate: Request creation of Cognate labs project - https://phabricator.wikimedia.org/T148889#2738725 (10Addshore) Thanks! [16:38:29] although I guess that sounds NFA ish [16:38:37] Hi, should be the text table visible in toollabs or not? [16:38:42] Because it isn't. [16:38:46] Matthew_: I'm trying to make sure it doesn't give off the impression that the labs admins are responsible for it and maintain it [16:39:07] I thought this is planned and wanted but at -tech I've got "apergos no part of the text table is visible? interesting" as reply. [16:39:15] yuvipanda: Fair. How about community-labs-monitoring ? [16:39:23] Urbanecm: Can you be more specific? [16:39:25] (as reply to my question how can I run a query which needs text table) [16:39:35] Matthew_: sounds good to me. [16:39:37] Yeah. I wish to run a query which needs this table. [16:39:40] Urbanecm: yup, the text table isn't visible [16:39:45] and probably will never be [16:39:50] I know it isn't. But why or will it be? [16:40:00] we don't store text in the text table in production - use ExternalStore instad [16:40:09] and that isn't really replicatable in the same way for performance reasons [16:40:13] 06Labs: Request creation of community-labs-monitoring labs project - https://phabricator.wikimedia.org/T148569#2738732 (10Matthewrbowker) [16:40:38] I wish a list of articles which was created by changing redirect i.e. redirect was the previous revision of the page. [16:40:42] 06Labs: Request creation of community-labs-monitoring labs project - https://phabricator.wikimedia.org/T148569#2726492 (10Matthewrbowker) @Andrew Re IRC, I've changed the requested name. Thank you :) [16:40:45] But there is nothing like rev_is_redirect... [16:42:37] yuvipanda: Thanks for your explanation. BTW why we aren't granting read-only access directly to prod servers? Maybe due to performance (everyone will be able to slow-down the servers)? Sorry for my silly question. [16:43:38] Urbanecm: no worries! no question is silly :) it's because there is sensitive data there - passwords, IPs, and revdeleted content [16:43:48] so we apply filtering before allowing access to it [16:44:40] Urbanecm: it could also overload the servers in production easily, so that is a reason too [16:45:35] Yeah, I understood. When I hide an edit as a sysop (I mean I hide text of the revision), is it still in the text table? [16:46:06] Urbanecm: I'm not sure, it might be [16:46:21] Urbanecm: legoktm would know better maybe [16:47:49] Yeah, understood. Why the idea of replicating the text table like the others isn't doable? A lot of data which should be replicated? [16:48:35] Urbanecm: yeah, plus the cluster it lives on isn't normal mysql but mysql configured differently for a different use case [16:48:52] yes, suppressed content remains in the database tables. The suppression is done at the MW software view layer [16:49:08] Thanks bd808 [16:49:27] and thanks yuvipanda too of course. [16:49:53] yw Urbanecm [16:49:59] this is needed for several reasons, but the most obvious is so that the content can be un-suppressed if the suppression was over reaching [16:50:40] But it can be moved to another table (like in the page table, the row will go to archive I think) [16:52:09] the best way I know of to get revision content is by reading the dumps using https://pythonhosted.org/mediawiki-utilities/core/xml_dump.htm [16:52:24] 404... [16:52:30] (should be an `l` at the end of that url) [16:52:42] That works, thanks. [16:52:43] https://pythonhosted.org/mediawiki-utilities/core/xml_dump.html [16:53:36] I'll examine it, thank you bd808 ! [16:58:25] is there a way to see my ssh login somewhere? [16:58:33] for tech labs [17:00:24] *waves* [17:00:33] hey addshore! :) [17:00:34] dennyvrandecic: there should be! [17:00:40] I agree with that :) [17:01:21] dennyvrandecic: https://wikitech.wikimedia.org/wiki/Special:Preferences "Instance shell account name" [17:01:52] thanks! I was blind and didn't see it :D [17:02:01] now to figure out my password... [17:02:13] haha, your password to wikitech? :) [17:02:34] I think I have my password to the wiki, but the ssh thing asks me for a password [17:02:39] I thought it would just take the key [17:02:50] it should take a key! not a password [17:02:55] weird [17:03:01] it took a password, and now I am logge din [17:03:09] maybe it is a local password to my private key [17:03:10] Wait [17:03:15] usually ssh first checks the keys, if you don't have the right key it asks for a password. but i don't think there are any valid passwords for that [17:03:16] maybe your key took a password! [17:03:22] that's also possible [17:03:22] The password is your wikitech password [17:03:32] ah well, I am logged in now :) [17:03:37] Great! :D [17:03:48] now I want to figure out how to host a bit of HTML and JS [17:04:02] preparing my wikidata 4th birthday gift :) [17:04:08] Make a tool, become the tool, run a webservice, host some code :) [17:04:19] dennyvrandecic: see [[tools/getting started]] [17:04:22] @link [17:04:22] https://wikitech.wikimedia.org/wiki/tools/getting_started [17:05:31] dennyvrandecic: re the birthday, I just made a new version of the map that we had on the wall, a few new areas have lit up (I'll write a blog post soon). [17:05:43] send me a printout! :D [17:05:44] Also working on making maps for every dump from oct 2014 to now, and making another gif. [17:05:59] thanks that is very cool! [17:06:05] i am looking forward to see the gif! [17:06:12] I am too! :) [17:06:36] My bdays next monday [17:07:11] :D [17:07:58] Zppix, No. [17:08:08] Do not expose your wikitech password to labs instances. [17:08:13] Ab [17:08:38] I thought he meant ssh on login.wmflabs because it always ask for ti [17:08:38] It [17:08:57] It does not ask for it if you set it up correctly. [17:09:07] If it asks for it and you provide it, change your password. [17:09:16] thanks! [17:09:17] will do [17:09:36] evil passwords [17:10:00] *Especially* if you are in any privileged groups that provide access to confidential information using that password [17:10:02] (03PS1) 10Jean-Frédéric: Harvest Wikidata item in es-ct (ca) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317538 (https://phabricator.wikimedia.org/T140795) [17:10:36] I think all my privileges have long been rescinded [17:11:03] It asks before i get an connection to anything it doesnt all the time just on new client sessions (when i get a new client andor dif ip [17:11:57] Don't count on it [17:14:08] Zppix, you must have done something wrong [17:14:32] No because the next session its ok [17:14:35] good point [17:15:18] dennyvrandecic, I've checked your account and there doesn't appear to be any groups providing confidential information access in there [17:15:33] thanks! [17:15:43] that's good [17:15:46] I dont either [17:15:55] one step away from world domination, but still good :) [17:16:03] But in general, I wouldn't trust any offboarding procedures around here [17:16:04] dennyvrandecic: lol [17:16:22] :) understood [17:16:28] Krenair: i thought the pw thing was norm :/ [17:16:37] Zppix, no. [17:17:52] addshore: now I realize that this wasn't some zen koan but actual instructions [17:19:38] Zppix: what you are describing sounds like your ssh-agent prompting you for a password the first time it loads your private key, but I'm not 100% sure of that. You certainly should not be prompted by the Labs/Tool Labs ssh servers for "keyboard interactive" login. [17:20:25] dennyvrandecic: indeed ;) [17:20:32] hah! [17:21:07] Krenair: in regards to offboarding, yeh.... :P [17:23:08] Does tools support ppk keys or just openssh keys [17:24:03] I think OSM checks them for openssh format? I might be wrong [17:24:14] "Make a tool, become the tool, run a webservice, host some code :)" [17:24:17] So ppk not supported [17:24:19] ? [17:24:26] i just keep repeating that until I internalize it [17:25:03] dennyvrandecic: if you need help code wish as long as its not php (i suck at that) let me know [17:25:46] thanks Zppix! :) [17:26:01] No prob [17:27:43] sweet, I figured it out [17:28:19] ok, everything is ready for release this week :) [17:43:10] Hey when someone from the tools team is around... I'm getting 502 errors for xtools-ec, however; the webservice is running fine. Can someone point me to the next troubleshooting step, as I don't know what else to try from my end. [17:44:36] yuvipanda: You would probably be the best person to ask about that, right? [18:16:18] PROBLEM - Host tools-exec-cyberbot is DOWN: CRITICAL - Host Unreachable (10.68.16.39) [18:20:18] (03CR) 10Jean-Frédéric: "Tested locally on Docker − everything works !" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317538 (https://phabricator.wikimedia.org/T140795) (owner: 10Jean-Frédéric) [18:28:39] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 0.74 ms [18:31:04] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [18:56:00] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Update /etc/hosts during labs instance first boot - https://phabricator.wikimedia.org/T120830#2739242 (10Andrew) 05Open>03Resolved Adding entries like breaks hostname -d. Entries like seem to be fine, although I'm not... [19:09:24] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 1.11 ms [19:14:25] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [19:21:07] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 1.02 ms [19:23:32] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [19:23:51] 10Labs-project-Wikistats: W3C wiki updates broken - https://phabricator.wikimedia.org/T149000#2739316 (10Dzahn) [19:48:21] PROBLEM - Puppet run on tools-grid-shadow is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [19:57:14] Is anyone here up on mail configuration? I've got a django app on wmlabs that needs to speak to the mail system - I know mail is running (I can send it to the command line) but I'm not sure where to tell Django to *look* for it. (nothing's listening on the usual port, so my usual smtp config is a nonstarter.) [19:57:58] is it possible that horizon doesn't list all existing web proxies any more? [19:58:02] don't all labs instances have exim running? [19:58:19] gwicke, I don't think so [19:58:23] RECOVERY - Puppet run on tools-grid-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [19:58:24] I have one in the services project (swproxy.wmflabs.org) that still works, but doesn't show up [19:58:25] gwicke, what's missing? [19:58:25] Yes, exim is running [19:58:30] hang on [19:58:34] just created another one, and that does show up [19:59:02] But Django can't find it, and nothing's listening on 25 [19:59:34] (I would normally tell django to go talk to port 25, but that fails if nothing is listening there, so I hoped there was some secret wmflabs config fact I was unaware of) [20:01:40] where the hell is /usr/bin/sqlite3 on novaproxy-02? [20:01:48] * gwicke needs to run, will be back in a bit [20:05:19] Alternately, is there any documentation about how mail works on labs? Was looking for some on wikitech, couldn't find any [20:10:33] gwicke, so you have swproxy-mobile.wmflabs.org -> http://10.68.16.62:8934 but not swproxy.wmflabs.org ? [20:10:45] did something die or is my dns just being odd? [20:11:42] uh [20:12:26] Krenair: do you see anything @ http://enwiktionary-cognate.wmflabs.org ? [20:12:37] yes [20:12:51] good, something just went badly wrong with my dns then... [20:13:07] okay [20:13:10] I just saw issues too [20:13:12] but it's back [20:13:18] that was worrying, I was logged into the proxy box [20:13:52] yeh, http://enwiktionary-cognate.wmflabs.org was working for me 10 mins ago, and it's just vanished for me (cant resolve from here) [20:14:23] addshore: I get MediaWiki 1.28.0-alpha [20:14:23] LocalSettings.php not found. [20:14:38] cool, thats what I was expecting to get! [20:14:46] right, time to kill my ISP again... [20:17:40] Violence isnt the answer addshore [20:21:17] ThatAndromeda: some docs at https://wikitech.wikimedia.org/wiki/Help:Tool_Labs#Mail_from_tools -- that seems to only cover using /usr/sbin/exim though [20:21:54] Yes. exim is running and I have successfully executed that command [20:22:56] But Django usually expects to talk to port 25 and nothing is on port 25, so I need some other way to hook into it, so I was hoping there were more details somewhere :/ [20:22:58] thanks for the pointer though [20:23:05] ThatAndromeda: I think that tools-mail.tools.eqiad.wmflabs is the internal mail server [20:23:36] that seems to be the configured smart host for exim [20:23:38] ooooh, that might be the magic incantation I need - I will try it! thanks [20:24:26] (03PS2) 10Jean-Frédéric: Harvest Wikidata item for es-ct (ca) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317538 (https://phabricator.wikimedia.org/T140795) [20:24:28] (03PS1) 10Jean-Frédéric: Harvest Wikidata item for es-vc (ca) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317576 (https://phabricator.wikimedia.org/T140795) [20:24:30] (03PS1) 10Jean-Frédéric: Harvest Wikidata item for fr (ca) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317577 (https://phabricator.wikimedia.org/T140795) [20:24:32] (03PS1) 10Jean-Frédéric: Harvest Wikidata item for ad (ca) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317578 (https://phabricator.wikimedia.org/T140795) [20:24:34] (03PS1) 10Jean-Frédéric: Harvest Wikidata item for es (ca) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317579 (https://phabricator.wikimedia.org/T140795) [20:24:36] (03PS1) 10Jean-Frédéric: Harvest Wikidata item for de-he (de) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/317580 (https://phabricator.wikimedia.org/T140795) [20:24:57] ThatAndromeda, sorry, dropped the ball on your question. kind of juggling things over here [20:25:43] no problem, I assume we all have multiple things on fire to deal with [20:29:08] ThatAndromeda: it would be more robust to use the local exim to queue and route mail but I'm not finding an obvious Django email backend that sends via shell exec [20:31:03] bd808: I'm getting a new error now, though, so that's exciting! [20:31:12] progress! [20:47:46] andrewbogott: great to see all the disk space that got reclaimed on labs ( top left graph of https://grafana.wikimedia.org/dashboard/db/labs-capacity-planning ) [20:47:49] quite impressive [20:59:23] PROBLEM - Free space - all mounts on tools-bastion-02 is CRITICAL: CRITICAL: tools.tools-bastion-02.diskspace._public_dumps.byte_percentfree (No valid datapoints found)tools.tools-bastion-02.diskspace.root.byte_percentfree (<50.00%) [21:17:58] bd808: yayyyyyy, it sends! here, there is cake in my dining room - please have some! [21:18:31] * bd808 noms on a slice of internet cake [21:20:55] Krenair: yes: both work, but only the newer of them shows up in horizon [21:21:13] yeah [21:21:31] I found the difference is between the normal proxy server and the one we moved over to a few days go [21:21:33] ago [21:22:03] I'm pretty certain it did show up fairly recently (~1 week ago) [21:22:20] nod, that would be consistent [21:22:55] yeah it shows on the normal one [21:23:00] not the one we failed over to [21:23:29] it's not blocking me right now, just wanted to report it [21:35:27] 06Labs, 10Horizon: Add a 'remember me' feature to Horizon - https://phabricator.wikimedia.org/T149036#2740050 (10yuvipanda) [21:38:53] 06Labs, 10Horizon: Add a 'remember me' feature to Horizon - https://phabricator.wikimedia.org/T149036#2740050 (10bd808) For Striker I do this: * `SESSION_EXPIRE_AT_BROWSER_CLOSE = True` to make the default session cookie have browser session scope * [[https://github.com/wikimedia/labs-striker/blob/52bc0f1d92a6... [21:43:02] !log project-proxy move public floating IP back to novaproxy-01 [21:43:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy/SAL, Master [21:46:51] gwicke: we failed it over back - your old proxies should be back now, but the ones you added over the last two days might be gone and require re-adding :( [21:46:52] sorry! [21:47:02] addshore: hey! your cognate proxy also met with the same fate, unfortunately :( can you re-add? ^ [21:47:25] haha, okay! [21:48:27] yuvipanda: it reports duplicate record set :/ [21:48:36] but doesnt appear in the list [21:48:36] (03Draft1) 10Zppix: Edit Project Config [labs/tools/ZppixBot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/317583 [21:48:52] addshore: ah, interesting. [21:49:00] addshore: does the proxy itself work right now? [21:49:46] *checks* [21:49:58] 404, so no :P [21:52:14] addshore: ok, what's the domain name? [21:52:52] enwiktionary-cognate and dewiktionary-cognate [21:52:57] .wmflabs.org [21:52:59] wow, we have 1065 proxies! [21:53:33] addshore: try now? [21:54:12] [23:52:58] wow, we have 1065 proxies! /// in parallel or in series ? :p [21:54:25] in parallel, thankfully [21:54:27] biggest in series is probably PAWS [21:54:28] ^^ [21:54:44] where you have novaproxy -> paws-proxy -> tools-proxy -> configurable-http-proxy -> actual notebook [21:54:47] yuvipanda: still the same message [21:54:51] wah [21:54:52] weird [21:55:08] :D [21:55:17] 4 is already a bunch :/ [21:55:27] i'm sure i've seen more though, can't remember where [21:55:51] (not wmf) [21:57:20] addshore: ah, I think I know it [21:57:33] I made some others for now ;) [21:57:55] andrewbogott: Krenair we need to delete these DNS domains... [21:57:59] so horizon can add them back [21:58:04] but I can't seem to find them in horizon? [21:58:31] scope to wmflabsdotorg project [21:58:55] Krenair: i am [21:59:03] Krenair: oh [21:59:04] nvm [21:59:12] i forgot I want 'manage records' [21:59:30] Krenair: uh, it only lets me 'edit proxies' [21:59:41] oh right [21:59:54] which records do you want deleted? [22:00:12] enwiktionary-cognate and dewiktionary-cognate .wmflabs.org (for me) [22:00:24] enwiktionary-cognate.wmflabs.org, dewiktionary-cognate.wmflabs.org, and swproxy-mobile.wmflabs.org (for gwicke) [22:01:32] one minute [22:02:53] | 702a0cca-6386-4fe7-a642-a7879dcc714c | A | enwiktionary-cognate.wmflabs.org. | 208.80.155.156 | [22:02:53] | 3b6a1e93-4bbf-4012-a3e6-3ccf65207c69 | A | dewiktionary-cognate.wmflabs.org. | 208.80.155.156 | [22:03:05] | 0cb72d37-cceb-4480-9ac8-24327c8c33cb | A | swproxy-mobile.wmflabs.org. | 208.80.155.156 | [22:03:09] krenair@silver:~$ designate record-delete wmflabs.org. 702a0cca-6386-4fe7-a642-a7879dcc714c [22:03:09] krenair@silver:~$ designate record-delete wmflabs.org. 3b6a1e93-4bbf-4012-a3e6-3ccf65207c69 [22:03:12] krenair@silver:~$ designate record-delete wmflabs.org. 0cb72d37-cceb-4480-9ac8-24327c8c33cb [22:03:14] krenair@silver:~$ [22:03:51] Still get duplicate record set here :/ [22:04:01] ugh [22:04:43] hmm [22:09:53] addshore: Krenair maybe it takes a while to propogate? [22:10:10] *tries again* [22:10:14] sometimes [22:10:30] same here right now, I'll keep trying! [22:10:32] I might try it with the recordset api [22:11:34] !log tools.morebots Stopping labslogbot and qa-morebots to test !log to wiki handling by stashbot [22:11:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.morebots/SAL, Master [22:13:44] !log tools.stashbot Restarting to test new config and features [22:18:18] addshore, try again [22:18:38] Krenair: working! :) [22:18:50] musikanimal, hi, i've seen the reports for the pageviews tool and i see no problem [22:19:00] many thanks! [22:19:11] addshore, I changed nothing [22:19:14] it was just absurdly slow [22:19:19] haha, okay! [22:19:22] musikanimal, nothing on console [22:19:27] alchimista: thanks, it's frustrating, seems to be fine for most people [22:19:32] and it's not the RESTBase API [22:19:34] omg working with pip in a venv on tools-bastion-02 is sloooow [22:19:47] musikanimal, ah, and i'm in portugal [22:19:54] ok, good to know! [22:20:01] thank you [22:20:20] bd808 thats ok virtualenv and i have exchanged some very "French" words [22:20:25] yuvipanda, ^ [22:20:29] so if it's a location problem, should be in brasil [22:20:33] should I file the issue about designate being slow to delete records? [22:20:47] musikanimal: checking the 'network' tab in the dev tools (F12 in Chrome and FF) might help [22:21:06] repeat the action, then see if the API request succeeded [22:21:12] bd808: yes, very. there's a bug about it. it's orders of magnitude faster inside kubernetes [22:21:32] and someday I will be able to run bots on k8s ;) [22:21:35] Krenair: yay! thanks for letting us know, addshore [22:21:37] gwicke: yes, all is well on my end, unfortunately these users are not super tech-savvy, and I'm starting to believe it's not any of the APIs that are messing up [22:21:43] bd808: you alraedy can :) just not as easily [22:21:53] they've confirmed the RESTBase API is fine, which is where we get the data that is visualized [22:22:09]