[00:00:19] hey ashwinpp [00:00:23] legoktm ^ do you know? [00:00:50] ashwinpp: I don't think so. Let me check... [00:01:40] ashwinpp: no, they are not resolved. [00:02:45] Are there any SSH login issues now? I got kicked off mid-session and can't log back in. [00:04:09] not with the normal bastion [00:04:13] what are you connecting to? [00:04:31] login.tools.wmflabs.org. Putty says "Server unexpectedly closed network connection" after I type the password for my private key. [00:05:00] oh, tools [00:05:12] yeah tools login is broken [00:05:15] valhallasw`cloud, YuviPanda ^ [00:05:31] alex@alex-laptop:~$ ssh tools-login.wmflabs.org [00:05:31] Connection closed by 208.80.155.130 [00:05:46] Okay thanks. Glad to know it's not something I did. [00:06:26] uh oh [00:06:27] looking [00:07:09] Okay [00:07:15] Krenair: can you try logging in now? [00:07:30] I tried resolving but it resulted in no result. I believe it would be a bug [00:07:35] YuviPanda, still fails [00:11:08] Krenair: ahecht try now? [00:11:16] https://gerrit.wikimedia.org/r/#/c/230257/ was the problem [00:11:17] Yup, works now [00:11:22] sorry about that [00:11:42] do i have to grant project admin to let people ssh into my instance? i'm pretty sure it was that way but recently you told me it's not (anymore)? [00:12:20] mutante: you just have to add people to the project [00:12:27] I'm pretty sure it was never that way :) [00:14:32] YuviPanda: thanks [02:39:16] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108: Evaluate kubernetes for use on Tool Labs - https://phabricator.wikimedia.org/T107993#1520699 (10yuvipanda) So... for the toollabs webservice use case: # Code will still be loaded off of NFS # Each webservice will run in a container in a pod with a replication contr... [02:54:22] There are some rows in the databases that contain text encoded in latin1 [02:54:25] is this intentional? [03:25:08] SigmaWP: probably for older wikis yeah [03:50:24] legoktm: should they be converted to utf8? [04:00:34] I'm going to make a phab bug thingy. [04:09:18] 6Labs: latin1 encoding in sql databases - https://phabricator.wikimedia.org/T108434#1520801 (10Sigma) 3NEW [04:13:35] 6Labs: latin1 encoding in sql databases - https://phabricator.wikimedia.org/T108434#1520810 (10Sigma) It seems to be terribly encoded only for some revisions: https://fr.wikipedia.org/w/index.php?title=Institut_d%27histoire_de_la_R%C3%A9volution_fran%C3%A7aise&diff=prev&oldid=59390655&diffonly=1 is in utf8. [06:00:54] Krenair: looking [06:02:42] Krenair: seems to work now; maybe have been an intermittent thing due to my hba change, sorry. [06:02:58] wait, I'm admin, so it might still bebroken [06:03:09] fsck [06:05:09] Krenair: no, I see other users connecting as well [06:08:56] YuviPanda: Note to self: maintenance windows where we actually deploy stuff.... [08:51:46] noobish question for using Flask on labs: app.run() fails, due to "Address already in use". I assume it just have to change the port e.g app.run(port=MAGIC_WMFLABS_SPECIFIC) [08:52:27] WSGIServer doesn't require special handling [09:27:39] !log mediawiki-core-team rebooting urlshortener instance because it's being weird [09:27:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mediawiki-core-team/SAL, Master [09:28:06] eranroz: are you using the uwsgi webservice setup? [09:29:32] legoktm: already solved the issue by app.run(); => fcgi.WSGIServer(app).run() [09:29:46] [app=Flask(__name__)] [09:30:00] ok, so you're running it as fcgi instead of uwsgi? [09:30:42] the "correct" way is to use uwsgi? [09:31:41] eranroz: yeah. https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web#Python_2_.28uwsgi.29 [09:33:06] 10x, will check it [10:00:41] eranroz: the supported way to run python apps is over uwsgi [10:03:03] i see it requires to run webservice2 with different command, would other non-python (php) still work with this? [10:04:04] No, because it gives full control of the tools.wmflabs.org/yourtoolname namespace to uwsgi/python [10:05:34] since I have multiple web scripts, some in php and some in python I guess I should slpit it to different projects... [10:05:57] or to keep be lazy and use cgi :) [10:06:13] using cgi is an option, but splitting is probably a better option [10:06:24] you might be able to get it to work over fcgi as well [10:06:25] hi [10:06:39] but even though possible, it's not a supported use case :-) [10:07:59] Linedwell: hello [10:08:08] I have a question about a command (from the labs) I importated on my own server (for similar needs), "become". I can't use it without putting "sudo" before [10:08:32] is there any "trick" (other than aliasing) to avoid this ? [10:09:11] ( /usr/bin/become ) [10:09:17] Linedwell: you have to give your current user rights to sudo-as-a-different-user in /etc/sudoers [10:09:37] for the specified group, I did [10:09:52] # Allow members of group botowner to execute "become" as root [10:09:52] %botowner ALL=(root) NOPASSWD:/usr/bin/become [10:09:52] I'm not sure how we have it configured for tool labs [10:10:08] also, become assumes your accounts are named 'tools.XXXXXX', I think [10:10:28] not necessarly, I modified this part [10:10:32] essentially, it runs /usr/bin/sudo -niu [10:10:41] if that doesn't work, there's an issue with your sudo configuration [10:11:00] it just check if 1) the users exists; 2) the user who use the command have the right to become "this" tool [10:11:09] maybe [10:11:42] (or maybe because I only allowed to %botuser the right to use "become", and not any other root command) [10:14:33] Okay, I don't get what you're trying to do exactly, but I would advise you to read up on how sudo works. Become is just a wrapper around sudo, and you should understand what sudo does before attempting to make become do something it wasn't built for [10:15:07] If you misconfigure sudo, you end up giving root rights to the wrong users [10:15:44] Thanks, for the advice, even if I know how does sudo work [10:58:02] YuviPanda: oh, you reverted the hba patch. But I can still login using hba to other hosts :O [10:58:20] YuviPanda: thank you for reverting it, by the way [11:04:35] 6Labs, 10Tool-Labs, 5Patch-For-Review: HBA not configured correctly for tools-bastion-02 - https://phabricator.wikimedia.org/T104613#1521268 (10valhallasw) 5Resolved>3Open [11:13:32] 6Labs, 10Tool-Labs, 5Patch-For-Review: HBA not configured correctly for tools-bastion-02 - https://phabricator.wikimedia.org/T104613#1521279 (10valhallasw) OK, so the hba situation is a bit complicated, because access.conf has different settings for different hosts, and they conflict a bit. Bastions have th... [11:16:15] YuviPanda: https://phabricator.wikimedia.org/T104613#1521279 [11:18:27] sad puppet is sad [16:11:21] I'm trying to follow https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web#Python_2_.28uwsgi.29 - should the example just work or should I install Flask myself? [16:15:42] eranroz: install flask in the venv [16:15:59] or should I use --system-site-packages? [16:16:25] both are possible [19:44:58] bd808: how hard is it to setup a logstash cluster in a project? [19:45:01] are there docs somewhere? [19:45:06] (this is for ORES) [19:48:33] YuviPanda: nontrivial, but if you could figure it out, have my blessing (and then please rinse and repeat for toollabs :-p) [19:49:04] valhallasw`cloud: toollabs has extra problems of authentication unfortunately [19:49:22] YuviPanda: even if it's just for admin logs, it would be useful [19:50:00] I'm fine with sshing in when wikibugs is broken, but I'd like to have a web interface to check why there's a shinken alert [19:50:26] +1 yah [20:15:32] YuviPanda: it's not too hard. The puppet roles do most of the work these days [20:15:45] one trick: you need a trebuchet server [20:16:11] all 3 parts now have trebuchet bits [20:18:42] oh, that could explain why I couldn't get it to work [20:19:56] I've been meaning to split up the deploy server role to make getting a basic trebuchet server setup easier but haven't gotten around to it [20:22:39] bd808: uh uh, that'd be great, I think... [20:24:04] today role::deployment::server is a giant pile of stuff today :/ [20:24:34] yeah [20:24:41] trebuchet, scap, release upload, keyholder, ... [20:24:50] that's partially my fault - it was spread out over like, 6-7 different things and I consolidated into one [20:24:56] but didn't separate out scap and trebuchet [20:25:28] mostly because my attitude towards trebuchet was / is 'oh dear, I got enough stuff to juggle already can I ignore this until it goes away?' which isn't quite healthy but was a sanity saving measure... [20:25:47] lol [20:25:48] so sorry about that :| but it's in better shape than it used to be, I think [20:26:28] trebuchet is fine except for all the ways that salt can break under it [20:26:36] yeah [20:27:24] thcipriani has a WIP patch that could replace a lot of trebuchet usage with a new thing built with scap python guts [20:27:34] yep [20:27:38] that'll be great [20:27:40] and make me very happy [20:27:47] esp. if it lands before the ORES deployment [20:27:59] so I don't have to deal with trebuchet :D [20:28:10] what is ores? [20:28:28] halfak: ^ [20:28:37] bd808: https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service [20:28:43] ah [20:28:44] bd808: I like to call it 'ClueBot as a Service' [20:29:05] I remember a presentation on that [20:29:09] bd808: I've been pushing for it and doing all the deploymnet work (https://phabricator.wikimedia.org/T106867 [20:29:10] ) [20:32:02] 10Quarry: Query counter increases but draft query is not accessible when window is closed and query doesn't have a title - https://phabricator.wikimedia.org/T101394#1521767 (10yuvipanda) Try now? They should show up as queries without titles... [20:32:03] maybe this is a stupid question, but why is there both a manifest /and/ a deploy tool? that seems double? or is that a 'things installed from apt' vs 'our own software' divide? [20:32:34] 10Quarry: The last run time of all queries reports 5 months ago - https://phabricator.wikimedia.org/T87200#1521773 (10yuvipanda) Is this still happeneing? [20:33:37] trebuchet is for deploying things that we are too lazy to turn into debs basically [20:34:20] for the ELK stack, that's Elasticsearch plugins, Logstash plugins and Kibana itself [20:34:55] git::clone{} sort of does the same thing [20:35:13] but trebuchet gives you control of what revision goes out and when [20:35:31] so you can have things in beta cluster before prod, etc [20:36:01] ah, right [20:36:05] thanks, that makes sense [20:36:27] I still dunno what it does that we can't do with tags and git::clone [20:36:32] I guess post actions like restart [20:36:37] but those have historically never worked... [20:36:50] force change on all the servers at once [20:36:57] ah right [20:36:59] that's fair enough [20:37:05] rather than "sometime in the next 20 minutes or so" [20:37:09] puppet as a deployment tool is just asking for trouble anyway [20:37:16] for sure [20:37:27] 22:36 puppet is just asking for trouble anyway <- ftfy [20:37:42] :P [20:37:50] trebuchet could be replaced with a ton of different things but each has had challenges. [20:38:07] but yeah, see adminbot, where the apt-puppet-restart cycle is pretty painful [20:38:36] debs are good for system software but not custom apps IMO [20:38:56] I've done it in the past and always ended up in a tight spot eventually [20:39:19] like "I want to run 2 copies of X with different versions" [20:39:25] valhallasw`cloud: adminbot we can just put in our local repo [20:39:37] YuviPanda: doesn't matter, still needs puppet to run on all exec hosts :P [20:40:14] why doesn't it just run from a virtualenv in the tool home? [20:40:23] because historical reasons, mostly [20:40:28] *nod* [20:40:39] I'm planning to virtualenv + fab-ify it [20:40:54] I think it ran as a service on a prod host originally [20:41:18] those hysterical raisins are everywhere [20:42:36] fab could replace trebuchet, except... it doesn't work with prod ssh these days [20:43:01] * bd808 shakes fist at non-openssh ssh implementations [20:43:08] oh, right, paramiko making life difficult [20:44:02] I should break the ssh loop out of scap and publish it as a package [20:45:22] bd808: please do, I need to replace paramiko in some labs scripts [20:45:33] a parmiko interface compatible thing with openssh backend! [20:49:59] YuviPanda: bastion.wmflabs is using the new ciphers, right, while tools-login is not? [20:50:34] everything is using the new ciphers. tools isn't using the new key exchange algorithms [20:50:39] ciphers were mostly unproblematic [20:51:16] ah, yes, the key exchange was the issue [20:53:00] YuviPanda: the paramiko maintainer is going to look at it 'in the near future' (as said mid-june) [20:53:21] valhallasw`cloud: moritz said there's already a patch for it unmerged [20:53:25] https://github.com/paramiko/paramiko/issues/532 is the specfic issue [20:53:45] oh, #356 [20:55:28] YuviPanda: yes, indeed. https://github.com/zamiam69/paramiko/commit/b3b0f2d0dff3aef736128935302aeb6adb1ee020 solves it [20:56:12] valhallasw`cloud: yup, patch is > 1y old [20:57:21] YuviPanda: but, er, just install that version in your venv and be done with it? [20:57:37] easier than rewriting everything from paramiko to something else [20:57:50] valhallasw`cloud: these are scripts running in prod etc [20:57:53] so no venvs [20:58:40] build a .deb with that version, then? ;-) [20:59:28] moritz considered and discarded that idea as too much work because we'll have to keep our fork up to date [21:00:35] O_o [21:01:02] okay, well, good luck with rewriting then. [21:02:25] valhallasw`cloud: it isn't that difficult in this case. it's a one line call that I can replace. probably won't atm unless someone presses me tho :D we just disabled new kexes on the one host it was required to access [21:02:42] ah, right [21:03:04] so status quo is ok. maintaining a fork is a lot of work, so in terms of work it's 1. leave it be, wait for patch to get merged upstream 2. shell out, 3. maintain fork [21:03:09] so i currently pick (1) [21:03:10] otoh, it's unlikely any stable distro will have their paramiko package updated with the new kex, so if home-packaging is considered too much work... [21:03:24] valhallasw`cloud: that's fine. if it gets merged upstream we have no problems maintaining a package [21:03:31] it's the fact that it's unmerged that's the problem [21:03:44] valhallasw`cloud: we can't cherry-pick just that patch onto the current version either. doesn't apply [21:03:58] so we have to merge it, and then rebase on top of any new changes.... [21:04:01] so be a good open source citizen, yadayada [21:04:19] in my copious spare time... [21:04:23] unfortunately [21:04:27] hehe [21:14:10] 10Quarry: Recurring queries - https://phabricator.wikimedia.org/T101835#1521826 (10yuvipanda) An extra table would be required - schedules. It'll have: (id, query, schedule), where schedule is one of 'daily, weekly, monthly'. And then we'll have celerybeat or something wake up every minute or so, check the sched... [21:16:47] 10Quarry: Recurring queries - https://phabricator.wikimedia.org/T101835#1521836 (10yuvipanda) An alternative is to use crontab like mechanics, but have those be randomly generated by the code when it is selected. This will distribute them randomly, and also give people an accurate estimate of when this gets upda... [21:37:55] 10Quarry: Recurring queries - https://phabricator.wikimedia.org/T101835#1521874 (10valhallasw) The tsreports approach to this is regenerating on access, but showing the cached version while the regeneration is in process (plus an ETA for the new version).