[00:15:52] wikitech isn't letting me manage my labs instances all of a sudden [00:15:59] is this a known thing? [00:17:18] milimetric: let me look [00:17:56] I notice the Special:NovaProxy and NovaInstance aren't linked to from the left anymore and if I go directly to them it doesn't find my projects when I search in the filter [00:18:14] milimetric: looks wrong for me too. [00:18:29] thx andrewbogott, it's nothing urgent for me [00:18:53] but if anyone figures something out, I'd appreciate a ping [02:01:16] milimetric: try now, if you’re still around? [02:01:45] milimetric: I predict that the sidebar will stil be wrong but that the NovaInstance page works correctly [02:01:46] thanks andrewbogott ! I happened to try it just a few seconds ago and it worked [02:01:55] sidebar too? [02:01:58] the sidebar was there, that's what prompted me to trying [02:02:08] * andrewbogott wants his sidebar back [02:02:26] :) i logged out / in again because of a different reason, maybe that was it? [02:02:44] I did too. [09:56:34] 6Labs, 6operations, 10wikitech.wikimedia.org, 7HHVM: Move wikitech to HHVM - https://phabricator.wikimedia.org/T98813#1413134 (10Joe) This ticket makes no sense, IMO. We have no reason to migrate wikitech to HHVM at the moment. I'm resolving this ticket as invalid until someone explains me why should we e... [10:03:17] 6Labs, 6operations, 10wikitech.wikimedia.org, 7HHVM: Move wikitech to HHVM - https://phabricator.wikimedia.org/T98813#1413140 (10Joe) 5Open>3Invalid [10:11:06] Hi [10:11:55] Is there any API available for quarry? [10:45:13] 6Labs, 10Tool-Labs: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413177 (10jcrespo) 3NEW a:3yuvipanda [10:54:20] 6Labs, 10Quarry: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413194 (10valhallasw) [10:58:37] jynus: lab ops are DOS'ing themselves now? Awesome [10:59:58] labs doesn't need help to DOS itself [11:01:10] I just made it faster [11:01:13] True ;) [11:01:31] jynus: I'm not sure how it would DOS the server, though. According to google, mysql doesn't multithread, and quarry kills queries after a certain time [11:01:51] I only need a minute to do that [11:01:53] although I guess you could use multiple queries / accounts / etc [11:02:07] and doesn't seem to have a limit on number of queries per user [11:02:22] right now, disable [11:02:49] we can talk later to enable it with due precautions [11:03:11] jynus: ironically quarry was designed to provide a tool so people wouldn't run their own violating TOS and therefore technical issues :) [11:03:29] well, in that case [11:03:41] and do not get me wrong, I understand [11:03:46] jynus: is it causing issues right now, or is it just that there is the possibility of abuse? [11:03:55] I will refuse to maintain that server [11:05:44] jynus: what would be helpful is stating exactly what (in your opinion) could be done to resolve the issue. Like limiting queries to one per user. [11:06:13] But I understand where you come from as this is making your already difficult job, more difficult [11:06:47] please note that I have not disabled it [11:06:57] I am asking it to force discussion [11:06:58] jynus: hey! I talked with Sean before opening it up and he was totally fine with it. Also I'll note that they can do the exact same thing with just creating an account on tools... [11:07:25] YuviPanda, then ask sean for support [11:07:27] jynus: I have currently the following measures in place: Queries get killed when they run for 20mins. Let's discuss if we need to add more. [11:07:31] 6Labs, 10Quarry: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413228 (10valhallasw) > Disable querry, as it goes against labs usage policy (allowing arbitrary queries to non lab users) Note that this is a Tool Labs policy, not a generic Labs policy. [11:08:09] I supose you can handle labs dbs on your own [11:08:58] I'm very confused now. let me respond on the ticket. [11:13:19] jynus: wait, did your queries break labsdb completely now? [11:16:04] 6Labs, 10Quarry: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413264 (10yuvipanda) So the measures that Quarry has in place to prevent abuse are: # Tied to user accounts on SUL, and the queries themselves have the username attached. We can trout people wh... [11:17:56] 6Labs, 10Quarry: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413266 (10yuvipanda) I'll note that the only person who seems to have run toxic queries has been you :) http://quarry.wmflabs.org/JCrespo%20(WMF) has broken Quarry as well. [11:20:27] 6Labs, 10Quarry: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413276 (10yuvipanda) ('broken' in the sense that it is reporting non-running queries (they were killed too) as running, not that Quarry itself isn't working anymore) [11:20:50] 6Labs, 10Quarry: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413277 (10jcrespo) My suggestions: * queries should be limited to 1 per account * There should be a throttling on new account creation [11:21:27] 6Labs, 10Quarry: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413278 (10jcrespo) [11:21:52] 10Quarry: Limit number of concurrent queries one user can run at a time - https://phabricator.wikimedia.org/T104316#1413282 (10yuvipanda) 3NEW [11:23:29] 6Labs, 10Quarry: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413290 (10yuvipanda) I've filed T104316 for #1. Can you explain what you mean by #2? Quarry's logins are just Wikimedia SUL accounts... I think '1' is too stringent a limit - Quarry has never c... [11:28:12] 6Labs, 10Quarry: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413316 (10jcrespo) Accounts probably should be autoconfirmed, or have a verified email to get in contact with them. Or maybe have a default small query limit (1 min), make it longer for older a... [11:30:45] 6Labs, 10Quarry: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413318 (10yuvipanda) I think we should be as permissible as possible and do restrictions based only on actual abuse than imagined. Autoconfirmed does not work - autoconfirmed on which wiki? meta... [11:49:13] 6Labs, 10Quarry: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413354 (10jcrespo) 5Open>3Invalid [12:24:50] valhallasw: https://wikitech.wikimedia.org/wiki/Help:Shared_storage I wrote that the other day [12:26:40] 10Quarry: Have an easy way to ban users from Quarry - https://phabricator.wikimedia.org/T104322#1413399 (10yuvipanda) 3NEW [12:26:52] 6Labs, 10Quarry: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413409 (10yuvipanda) Filed T104322 as well as follow up. [12:34:44] YuviPanda: lgtm, although I'm not sure why people should not just get /public/dumps mounted by default? [12:35:15] valhallasw: currently am at 'No NFS unless you ask for it' :) most people don't need it, and for those that do it's easy enough to enable. [12:35:21] YuviPanda: also, if this is still possible, I'd create a file in the base image as /data/scratch/README that refers to that doc page, so people understand it's not mounted and why [12:35:48] YuviPanda: well, yeah, but in the case of dumps, you'd rather have them use the shared dumps, not download the dumps themselves [12:35:55] which is a risk you run by adding a barrier [12:36:03] valhallasw: hmm, I wonder if we can do it one better - have a 'new project welcome kit'? [12:36:16] with this info and other info on how to navigate our quagmire. [12:36:31] valhallasw: true re: barriers. [12:36:49] I don't know enough about NFS to see what the impact is of having it available by default [12:37:20] valhallasw: so what I observed during the last outage (and other ones as well) is that if there's any NFS mount that stops responding, then load on the instance skyrockets and it becomes fairly unusable. [12:37:30] and then I get paged in the middle of the night / am already awake can not sleep [12:37:41] we can maybe bring it back on by default once NFS has proved to be less crappy [12:38:00] YuviPanda: are you sure the high load actually causes the instance to be unusable? [12:38:01] good luck with that YuviPanda [12:38:08] i.e. not gonna happen [12:38:14] (nfs getting better) [12:38:22] mobrovac: I think there's lots of low hanging fruit for our NFS setup :) [12:38:54] valhallasw: we weren't able to ssh into them even if they were non-precise, so I"d say yes. I do not have a reason why, however. [12:39:05] YuviPanda: because /home, I think :-) [12:39:22] valhallasw: nope - quarry had only /data/project mounted. [12:39:37] and remember the bastions were unreachable as ewll [12:39:38] *well [12:39:44] and they did not have /home mounted. [12:40:02] also ssh just hung, and we explicitly already (before the outage!) made sure it wasn't looking in /home for keys [12:40:20] anyway, I'm tired of fighting with it as well. We can re-evaluate in a few months :) [12:41:01] YuviPanda: oh, yeah, that was on my to-do list, but I forgot to make a task [12:41:10] which one? [12:41:11] strace the heck out of ssh to see if it's hitting /home and if so why [12:41:15] ah yes [12:41:22] that might be a good thing to check too... [12:41:43] 6Labs, 10Tool-Labs: strace sshd to figure out which filesystems are hit on login - https://phabricator.wikimedia.org/T104327#1413460 (10valhallasw) 3NEW [12:41:57] meh, probably not TL related because we use NFS anyway [12:42:32] 6Labs: strace sshd to figure out which filesystems are hit on login - https://phabricator.wikimedia.org/T104327#1413468 (10valhallasw) [12:43:03] yeah [12:44:10] 6Labs, 3Labs-Sprint-104, 7Puppet: Allow per-host hiera overrides via wikitech - https://phabricator.wikimedia.org/T104202#1413471 (10yuvipanda) A couple of methods: # Pages named Hiera:/host/hostname # a 'host' key in the Hiera: page itself. [13:41:24] next openstack release would be Mita apparently http://civs.cs.cornell.edu/cgi-bin/results.pl?id=E_4983776e190c8dbc :D [13:42:28] 6Labs, 6operations, 10wikitech.wikimedia.org, 7HHVM: Move wikitech to HHVM - https://phabricator.wikimedia.org/T98813#1413617 (10Krenair) 5Invalid>3Open It's an absolute blocker for {T91590}, {T94149}, and perhaps {T75901}. If you want to close this, you'll have to reject at least those first two. [13:44:29] YuviPanda: are you doing file recovery for tools labs ? just wondering [13:44:30] 6Labs, 6operations, 10wikitech.wikimedia.org, 7HHVM: Move wikitech to HHVM - https://phabricator.wikimedia.org/T98813#1413626 (10Reedy) >>! In T98813#1413617, @Krenair wrote: > It's an absolute blocker for {T91590}, {T94149}, and perhaps {T75901}. If you want to close this, you'll have to reject at least t... [13:44:44] hashar: I am, yes. [13:44:53] I should announce that too. thanks for reminding me [13:45:24] I guess I should just file a task ? [13:46:38] 6Labs, 3Labs-Sprint-104: Recover files from old corrupted file system (Tracking) - https://phabricator.wikimedia.org/T104334#1413628 (10yuvipanda) 3NEW a:3coren [13:47:18] hashar: ^ [13:47:22] 6Labs, 10Tool-Labs-tools-meetbot, 10Labs-Infrastructure: Recover meetbot (toolslabs) files after NFS crash - https://phabricator.wikimedia.org/T104335#1413637 (10hashar) 3NEW [13:47:23] hashar: yup, file a task as a subtask of ^ [13:47:38] 6Labs, 10Tool-Labs-tools-meetbot, 10Labs-Infrastructure: Recover meetbot (toolslabs) files after NFS crash - https://phabricator.wikimedia.org/T104335#1413645 (10hashar) [13:47:41] 6Labs, 3Labs-Sprint-104: Recover files from old corrupted file system (Tracking) - https://phabricator.wikimedia.org/T104334#1413644 (10hashar) [13:47:41] YuviPanda: excellent thanks a ton [13:47:54] no hurry on that one [13:48:14] 6Labs, 10Tool-Labs-tools-meetbot, 10Labs-Infrastructure: Recover meetbot (toolslabs) files after NFS crash - https://phabricator.wikimedia.org/T104335#1413647 (10yuvipanda) Can you give the path in the file system where these files would be? [13:48:38] YuviPanda: Hi, Vivek Cherian here. [13:48:57] * VivCheri at work [13:52:16] 6Labs, 3Labs-Sprint-104: Recover files from old corrupted file system (Tracking) - https://phabricator.wikimedia.org/T104334#1413666 (10Magnus) Please recover: * Tool "wikidata-todo", likely in subdir "scripts", a PHP file, probably called "bot"-something, should contain the string "gnd" * Tool "magnustools",... [13:52:56] 6Labs, 3Labs-Sprint-104: GND bot - https://phabricator.wikimedia.org/T104336#1413669 (10Magnus) 3NEW a:3coren [13:53:24] 6Labs, 3Labs-Sprint-104: wikidata.php - https://phabricator.wikimedia.org/T104337#1413676 (10Magnus) 3NEW a:3coren [13:55:05] 6Labs, 3Labs-Sprint-104, 7Puppet: Allow per-host hiera overrides via wikitech - https://phabricator.wikimedia.org/T104202#1413686 (10scfc) I prefer 1. so that it is possible to just copy the project Hiera page to a host's Hiera page to start customizing that. [13:55:42] YuviPanda: got permission to access irc from office :) [13:55:51] 6Labs, 10Tool-Labs-tools-meetbot, 10Labs-Infrastructure: Recover meetbot (toolslabs) files after NFS crash - https://phabricator.wikimedia.org/T104335#1413689 (10hashar) No idea where https://tools.wmflabs.org/meetbot/wikimedia-office/2015/ points to :-/ [14:03:24] 6Labs, 3Labs-Sprint-104, 7Puppet: Allow per-host hiera overrides via wikitech - https://phabricator.wikimedia.org/T104202#1413702 (10yuvipanda) Yeah, I prefer $1 too, since that mirrors closely what we have in the operations/puppet git repository [15:11:55] 6Labs, 10Labs-Infrastructure, 5Continuous-Integration-Isolation, 3Labs-Sprint-103, and 2 others: Instances without a shared NFS storage suffers from a 3 minutes boot delay - https://phabricator.wikimedia.org/T102544#1413950 (10Andrew) The puppet fix for this can be merged as soon as the export daemon is ru... [15:22:33] 6Labs: Cleanup ldap/files/scripts - https://phabricator.wikimedia.org/T104342#1413958 (10yuvipanda) 3NEW [15:33:12] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1201 is CRITICAL 22.22% of data above the critical threshold [0.0] [15:35:24] PROBLEM - Puppet failure on tools-exec-1213 is CRITICAL 60.00% of data above the critical threshold [0.0] [15:37:43] 6Labs, 10Datasets-General-or-Unknown, 10Labs-Infrastructure, 10Wikidata, and 2 others: Add Wikidata json dumps to labs in /public/dumps - https://phabricator.wikimedia.org/T100885#1414000 (10Addshore) Slightly altered in https://gerrit.wikimedia.org/r/#/c/221855/ [15:38:49] YuviPanda: ^^^Error: /Stage[main]/Ldap::Client::Utils/File[/usr/local/sbin/manage-nfs-volumes-daemon]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///modules/ldap/scripts/manage-nfs-volumes-daemon [15:38:58] valhallasw: yes, transient. [15:39:03] ok [15:39:18] valhallasw: I just moved those scripts around, and they're also no longer provisioned on all labs instances >_> [15:39:23] they are used only on the NFS server... [15:39:27] >_< [15:40:14] valhallasw: so puppet catalog gets compiled, refers to them, and when the agent attempts to contact the puppetmaster for these files, they are gone! [15:40:21] because catalog and files have changed in the meantime! [15:40:25] and one puppet failure happens. [15:40:32] yes, makes sense [15:40:37] stupid puppetmaster [15:41:07] which should just know 'oh, this file had this version in the config version you're using' [15:41:36] yeah, or include the files in the catalog compilation... [15:42:09] oh well. [15:45:55] So Labs went down a little while ago and rolled back to before I set up on it, which I guess means I'm going to have to figure out what steps I have to repeat again. [15:48:22] Ok, I see that my tool is still up but that there's nothing in it now. [15:48:56] `virtualenv venv –python=python3.4.3` will work with no problems, right? [15:49:00] ResMar: YuviPanda just mailed labs-l about getting files back from backups [15:49:13] It's fine, I had only the bare essentials. [15:49:23] valhallasw: I had a Python 2 install but I want to make a Python 3 install now. [15:49:33] not sure if python3.4.3 exists, but python3.4 should (and python3 should definitely exist) [15:49:59] Ah, virtualenv venv –python=python3.4, then [15:50:14] As long as it's 3, I don't think I use anything dependent on the minor versioning. [15:52:15] Oh, shoot. I created a folder called "-p/" and now I can't delete it because it think I'm trying to enter a command-line option. [15:52:44] rm -- -p/ [15:53:34] rm -r -- –p/ [15:53:54] ^ that worked, thanks. [15:54:20] > Status: Mostly OK, yell if something is wrong [15:55:15] Right, well my instructions to myself on setting up the environment are to start by punching in `mkdir –p www/python` [15:55:30] But that failed and created a -p/ instead, and I think I know why. [15:56:17] Teach me to store things in Word: Word converted the dash to "–" instead of "-". [15:56:38] TIL. [15:58:11] * CP678|away is away: This is a manual computer virus. Please copy paste me in your away message. I'm not here right now. [15:58:12] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1201 is OK Less than 1.00% above the threshold [0.0] [15:58:21] tools.fc-importer@tools-bastion-01:~/www/python$ virtualenv venv -python=python3 The executable ython=python3 (from --python=ython=python3) does not exist [15:58:41] valhallasw: Surely not. [16:00:06] I must be inputting somethin wrong...no way venv doesn't have python3 installed on Labs. [16:01:28] Can someone give me a `venv` command to run to install python3? [16:01:46] ResMar: I'm on my way out, but I think you have to give it the full path to python3. so virtualenv -python=/usr/bin/python3 env [16:02:19] sitic: Let me try it then [16:02:30] ResMar: as sitic said, virtualenv -p python3 --system-site-packages ENV, worked for me [16:02:48] I prefer to have the system packages available, as I need some of them [16:03:01] ResMar: --python= [16:03:02] What do you mean by "system packages"? [16:03:07] Like, sys, time? [16:03:13] ResMar: not -python [16:03:30] Oh, shoot. I should know better. [16:03:37] ResMar: I need numpy & scipy, for instance [16:03:38] no, those are always included. system packages are packages that are installed system-wide [16:04:10] Does it take those packages from your PC or from whatever Labs has? [16:05:00] valhallasw: That worked. [16:05:20] RECOVERY - Puppet failure on tools-exec-1213 is OK Less than 1.00% above the threshold [0.0] [16:05:51] ResMar: whatever tool labs has [16:06:11] Labs doesn't have damn near everything? [16:06:43] Oh, but it's just pointers, not a seperate data installation. [16:07:14] I can barely get around on my own computer, doing things on a server is a whole new experience for me :) [16:11:52] I want to install some dependencies, but pip is complaining that it can't find a recent enough version of pywikibot. [16:12:41] `No distributions matching the version for pywikibot>=3.3 (from -r requirements.txt (line 4))` [16:13:08] andrewbogott: I deleted a file from my home directory mistakenly, do you know if I can recover it? [16:13:37] Howie: If your home dir is mounted on NFS and if the file was there three weeks ago, then maybe :) [16:13:50] I think it should be [16:14:00] Howie: if so, open a phab ticket and link me to it, I’ll try to get it filed properly. [16:14:06] Or, rather, YuviPanda will :) [16:14:25] 6Labs, 3Labs-Sprint-102, 3Labs-Sprint-104, 5Patch-For-Review: Labs: manage-nfs-volumes-daemon rewrite - https://phabricator.wikimedia.org/T102782#1414193 (10yuvipanda) a:5coren>3yuvipanda [16:16:22] andrewbogott: much thanks! [16:16:36] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1402 is CRITICAL 60.00% of data above the critical threshold [0.0] [16:17:11] Dang, can I not install pywikibot at all? [16:18:30] ResMar: use pip install --pre pywikibot [16:18:55] pywikibot version 3.3 does not exist, so you requirements.txt is wrong [16:19:15] I see it listed here: https://www.mediawiki.org/wiki/Manual:Pywikibot/Overview [16:19:46] thats the python version [16:20:06] if you require pywikibot>=3.3 you want version 3.3 of pywikibot [16:20:19] Oh, shoot. Reading failure. [16:20:24] YuviPanda: I just created a phabricator ticket about my lost file. Thank you if you can help. [16:20:38] *loud groan* [16:20:45] YuviPanda: Thank you even if you can't help. :) [16:21:20] Howie: np :) I'll probably get to it later today. do file a bug however. [16:21:25] andrewbogott: How do I link it to you? [16:21:33] just paste a link right here [16:22:10] https://phabricator.wikimedia.org/T104348 [16:22:22] Yuvipanda: How do I file a bug? [16:22:43] Howie: you just did [16:22:59] ok good [16:23:06] YuviPanda: much thanks! [16:24:03] 6Labs: Lost a file in my home directory "/home/hcohl/purgemathcache.sql" - https://phabricator.wikimedia.org/T104348#1414232 (10Physikerwelt) p:5Triage>3Normal [16:35:13] 6Labs: Lost a file in my home directory "/home/hcohl/purgemathcache.sql" - https://phabricator.wikimedia.org/T104348#1414273 (10yuvipanda) Which project is this in? [16:36:07] 6Labs: Lost a file in my home directory "/home/hcohl/purgemathcache.sql" - https://phabricator.wikimedia.org/T104348#1414278 (10Hcohl) math project. [16:46:36] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1402 is OK Less than 1.00% above the threshold [0.0] [16:50:30] andrewbogott: Hi [16:50:38] andrewbogott: Vivek here, at work [16:50:54] andrewbogott: Got permission to use IRC from work :) [17:11:27] can someone with a tool access to sql at enwiki.labsdb and confirm everything seems normal? [17:13:55] looks ok, jynus ... anything specific to watch out for? [17:14:13] 6Labs, 3Labs-Sprint-103, 3Labs-Sprint-104: In openstack upstream, add project_id to instance metadata - https://phabricator.wikimedia.org/T103384#1414379 (10Andrew) This is now in openstack review: https://review.openstack.org/#/c/197185/ To avoid conflicts, the upstream key is project_id whereas my custom... [17:14:19] Krenair, can you count the databases you see? [17:14:27] SHOW DATABASES; <-- how many lines? [17:14:37] 1858 rows in set (0.02 sec) [17:15:35] Krenair, can you share your mysql user, if not on private? (SELECT current_user()) [17:15:53] jynus, u2170@% [17:19:26] strange, I only see 905 with the same permissions [17:20:30] Krenair, SELECT @@GLOBAL.server_id; ? [17:22:22] we are probably on different servers [17:33:37] jynus, I connected again, ran that and got 171967499 [17:34:05] still the same number of databases and user [17:39:22] Krenair, thank you, I found it [18:13:06] 6Labs, 7HTTPS, 3Labs-Sprint-104: Renew toolserver.org SSL certificate - https://phabricator.wikimedia.org/T104211#1414586 (10RobH) a:5RobH>3Andrew The new certificate and key pair are in Andrew's home directory on iron (ops only bastion). Andrew: Please commit to the repos when you push into service. I... [18:18:55] I have changed some grants on replica db servers [18:19:06] access should not be affected at all [18:19:28] but if for any reason you see something wrong, tell it here [18:20:00] I will provide a summary later, when I have checked that everithing is ok [18:35:41] Shoot. I need to troubleshoot why my lab app isn't starting up. [18:36:06] Ok, how do I go about troubleshooting? [18:37:19] Anyone on that can help me get back to a minimum working example on Python 3? [18:38:55] My error.log is empty and the webservice starts, but going to the URL gives me nothing: https://tools.wmflabs.org/fc-importer/ [18:41:46] Or never mind, it appears the webservice is not starting at all, but `webservice2 uwsgi-python restart` is not generating error messages either. [18:43:35] und dann "&props=279" (oder so) anhängen für die Edges [18:44:01] oops wrong channel [18:48:41] YuviPanda: can you help me figure out what still needs to be done for the mailrelay? [18:48:50] valhallasw: yeah, but not today, sadly :( [18:48:54] hmkay [18:49:05] I've to go and get my hair colored... [18:49:07] apparently. [18:49:08] ResMar: uwsgi.log? [18:49:14] pics or it will not have happened [18:49:14] * YuviPanda goes afk again [18:49:21] indeed, I shall provide [18:49:28] Ooh? [18:49:47] Didn't know that these failures are so surprising [18:49:49] * deadpan * [18:49:51] ResMar: yeah, uwsgi.log has info [18:50:07] but not exactly comprehensible info [18:50:16] I rebuilt a working example on my local virtual environment, so it should work... [18:50:18] * should * [18:50:42] I fixed a bug today where instead of getting ".page" I got ".pages" of a Request object. [18:51:00] The culprit was nothing more and nothing less than my not pressing Ctrl+S hard enough. [18:51:08] did we have any problems with DNS on labs? I'm getting a ton of java.net.UnknownHostException: www.wikidata.org in the logs [18:51:30] ResMar: uwsgi.ini should be in python/, I think [18:51:49] The ini file? [18:51:52] 6Labs, 10Incident-20150617-LabsNFSOutage: Recover data file /data/project/phetools/public_html/data/new_stats.py - https://phabricator.wikimedia.org/T104239#1414753 (10Phe) [18:51:52] yeah [18:51:52] GNU nano 2.2.6 File: uwsgi.ini [uwsgi] wsgi-file=app.py [18:51:54] 6Labs, 3Labs-Sprint-104: Recover files from old corrupted file system (Tracking) - https://phabricator.wikimedia.org/T104334#1414752 (10Phe) [18:52:02] ResMar: it should be in www/python, not in www [18:52:03] Wait [18:52:12] Oh [18:52:24] You peeked into it. [18:52:30] SMalyshev: I'm not aware of any. On which host is that? [18:52:44] Yeah I'm noticing now; setting up is a pain, huh. [18:53:07] valhallasw: wdq-beta.eqiad.wmflabs. But doesn't happen now, happened before [18:53:24] ResMar: it says so in the docs? https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web#Python_.28uwsgi.29 [18:54:32] I wrote down painstaking instructions for myself for the future on how to get everything running when I first did this a few weeks ago. [18:54:47] Not painstaking enough :(. And in the wrong format. [18:55:31] I'll learn though! :) [18:55:39] :-) [18:56:06] ResMar: but let me clarify my tutorial as well, explicitly mentioning paths instead of "cd .." [18:57:11] valhallasw: Still didn't start unfortunately. [18:58:09] Is this correct? The contents of uwsgi.ini are: [18:58:15] > [uwsgi] [18:58:16] > [uwsgi] wsgi-file=app.py [18:58:20] Er. [18:58:25] > [uwsgi] [18:58:26] > wsgi-file=app.py [18:58:28] * [18:58:41] I'm not missing a u there are I? [18:58:45] am* [18:59:35] ResMar: no, looks OK [19:00:57] ohh [19:01:01] app.py should be in src, I think [19:01:02] Here's what I am seeing: https://dpaste.de/Rjw2 [19:01:37] All these folders! [19:01:58] yes, now we're getting somewhere [19:02:01] Set PythonHome to /data/project/fc-importer/www/python/venv [19:02:01] ImportError: No module named site [19:02:02] wat. [19:02:55] But it is there: https://dpaste.de/dpsd [19:03:09] I just moved it to src [19:03:28] Oh [19:03:42] it's using Python version: 2.7.6 (default, Jun 22 2015, 18:01:27) , which is wrong [19:03:59] venv failed to install Python 3? [19:04:28] no, uwsgi fails to understand the venv, it seems [19:04:39] Am I supposed to be activating the venv? [19:04:56] Like `souce activate fc-importer`, which is what I do on my local machine. [19:04:58] no, uwsgi should do that... [19:05:52] Well the only substantial thing it says is `wsgi-file=app.py`, which I can't see as being a trigger for anything. [19:09:07] (http://uwsgi-docs.readthedocs.org/en/latest/Configuration.html) [19:09:40] Well, I suppose I never really had much of a clue what I was doing in the first place, so being stuck now isn't much of a change. :-) [19:12:17] well, the issue is it's loading the wrong plugin (python2.7 rather than python3.4) [19:12:23] but I thought it would just get that from the venv... [19:12:40] I am to understand that 2.7 is the default. [19:12:47] But beyond that I am relatively clueless. [19:13:18] venv/bin has `python3.4` folders [19:15:59] yeah, uwsgi doesn't get the virtualenv for some reason [19:16:51] Hi, although stuff on Labs is supposed to be open-source, is it fine if we use Google Analytics? [19:17:22] L235: no, but mostly because of privacy reasons [19:17:31] ok, thank you [19:17:43] any alternatives you recommend? [19:17:48] Anyone I could ping about that or are we all equally stumped? [19:18:08] (and valhallasw, can I post this response to a public mailing list?) [19:18:16] ResMar: I think the end result is 'python3 is not supported' :( [19:18:31] Hey! [19:18:36] Don't scare me like that! [19:18:38] L235: sure [19:19:21] any alternatives to Google Analytics you (anyone) would recommend for use on Labs? [19:19:36] No way python3 is not supported, come on, it's 2015...ugh. [19:19:39] 6Labs, 10Tool-Labs: support python3 uwsgi apps - https://phabricator.wikimedia.org/T104374#1414842 (10valhallasw) 3NEW [19:19:41] I'll look up a Phab ticket. [19:19:46] 6Labs, 10Tool-Labs: support python3 uwsgi apps - https://phabricator.wikimedia.org/T104374#1414851 (10valhallasw) 5Open>3stalled [19:19:48] ResMar: ^ [19:19:54] Dang it [19:20:45] That doesn't look fixable. [19:20:48] L235: not directly; I think the standard response is 're-consider whether you really need it, as it inherently invades privacy' [19:21:04] right [19:21:06] thanks [19:22:36] L235: see https://wikitech.wikimedia.org/wiki/Wikitech:Labs_Terms_of_use#What_can_and_can.E2.80.99t_be_done_with_user_information.3F for more details [19:23:04] valhallasw: Well I figured it wouldn't hurt so I copied in the .ini file at the end of the comments here into there: https://github.com/unbit/uwsgi/issues/621 [19:23:24] The webserver starts and restarts now, but I still don't see any output. [19:23:31] (no, I have no idea what the contents of that file are) [19:23:51] ResMar: no, it still doesn't start, because command line parameters take precedence over the ini file [19:23:59] ResMar: uwsgi is called with --plugin=python... [19:24:03] valhallasw: ah right... ok, that puts that option off the table [19:24:22] valhallasw: Haha dang it I've not a clue what that means here. [19:24:54] I thought ini files in general were so dangerous because they took precendence over everything else [19:24:56] ResMar: when you start the webserver, uwsgi is started as uwsgi --plugin=python --venv=/.../www/python/venv --etc --ini-file=/.../www/python/uwsgi.ini [19:25:15] ResMar: and that --plugin=python overrides the plugin=python3 in uwsgi.ini [19:25:42] No way to change --plugin=python? [19:27:16] 6Labs, 10Tool-Labs: allow users to override uwsgi parameters - https://phabricator.wikimedia.org/T104375#1414900 (10valhallasw) 3NEW [19:27:18] not easily, no [19:27:27] Wow [19:27:45] Look at all these bugs I am creating! :p [19:29:34] I suppose my only choice in the short term is to rewrite the code to work in Python 2. [19:29:45] That was actually what I did the first time around, but then the labs outage happened and I lost the source file. [19:29:58] And I figured why am I running in Python 2 anyway? I write in Python 3, and it's 2015...whoops. [19:30:56] I'm not sure but those bugs aren't ones that would be fixed in the short term would they? [19:31:19] a.e. I should investigate how to reinstall everything in python 2 now. [19:31:58] ResMar: just run 'deactivate', then virtualenv www/python/venv, and maybe a pip install flask [19:32:25] I wonder if 3to2 can't just handle the code conversion for me; last time I did it by hand by importing a lot of __future__ modules with someone else's help. [19:32:30] probably [19:32:38] you can also write code that works in both [19:32:59] with some help of six, maybe [19:33:13] Yeah but the trouble is [19:33:23] I'm a hobbyist, I don't study this in my classes [19:33:32] I picked up Python because I wanted minimum overhead maximum flexibility [19:33:48] and you chose python? :P [19:33:52] But then there's all this 2 or 3 stuff :/ [19:34:18] ebernhardson: After dealing with Java and C, Python looked super-appealing. Still does, in its good moments. [19:34:39] And then you get to things like having to convert between time objects and datetime objects and you think, come on guys... [19:34:40] ResMar: the simple answer is 'we should have python3 available for uwsgi' [19:34:59] Well, you should, I agree, but since it's not there now, I'm not sure how long it would take for support to be added. [19:35:17] another option is to run it over fcgi with lighttpd instead [19:35:20] Any sense of the priority? [19:35:46] If you ask YuviPanda, you might prod his pride ;-) [19:35:59] Yuvi is always so busy though. [19:36:18] He's been promising to fix the Signpost mobile app for the longest time :-( [19:36:32] ResMar: let me try to set up fcgi for you [19:36:39] Which I didn't even know existed until I got on there, the current editorial board is terrible at promotion. [19:37:05] YuviPanda: Ping? [19:37:23] valhallasw: Roger [19:37:54] If you can make a list of commands to run that'd be helpful too because while I was fiddling around with this one I also wrote a second importer for a different taskset at the paper. [19:38:56] My ultimate goal is to set them up online so that you press a button, it runs a script, bot posts it in, and presto, out pops an autogenerated page. [19:40:25] ResMar: :D that worked in one attempt! [19:40:26] https://tools.wmflabs.org/fc-importer/ [19:40:44] Dang that was fast [19:41:39] Ok so in terms of file structure, everything goes in /src, yes? [19:41:53] hi! one question. I have a script I'd like to run it simultaneously to as many db as possible to obtain results faster. will I saturate my 'user quota' or sth and provoke a "mysql gone away" message or sth? since different languages are in different servers... should I run only one script for a different db server? [19:42:32] ResMar: yes. Note that you have to restart the webservice for changes to take effect [19:42:39] I should change my Maniphest settings; it's a little wierd subscribing to something and then waiting for my phone to buzz on me once I get the email notification. [19:42:42] marmick: all languages are on all servers [19:42:45] valhallasw: Is the restart command the same? [19:43:00] 6Labs, 10Tool-Labs: support python3 uwsgi apps - https://phabricator.wikimedia.org/T104374#1414998 (10valhallasw) Workaround: use lighttpd with fcgi. ``` pip install flipflop ``` create ~/www/python/src/app.fcgi: ``` #!/data/project/fc-importer/www/python/venv/bin/python from flipflop import WSGIServer from... [19:43:01] ResMar: webservice restart, yeah [19:43:15] mmmm...ok [19:43:26] Thanks. [19:44:14] You could make a blog post out of it methinks, so that I (and others) have someplace to refer to in the future on the topic of setting this stuff up using fcgi. [19:44:23] valhallasw: how can I know the number of times I am able to run the script simultaneously? [19:44:35] ResMar: I added it to the bug, https://phabricator.wikimedia.org/T104374#1414998 [19:44:58] Ah, there goes my phone. [19:45:09] Thanks. [19:45:17] marmick: there probably is a connection limit for mysql, but I don't know what it is [19:45:48] that's why i suspected that if i connect using itwiki_p is not the same than enwiki_p [19:46:02] so i could run the script twice without risking both processes [19:46:32] marmick: the connection limit is probably on the order of 10 connections rather than 1 [19:46:51] this is a bit difficult to understand :) [19:47:16] marmick: I don't know the exact number, but it's maybe 10 or 20 connections [19:47:20] marmick: not 1 or 2 [19:47:52] but unless things have to work in parallel, I'd suggest doing it in series, as that reduces the load on the db servers [19:50:43] it's about time. the more parallel the fastest i get the results for several languages. [19:51:35] that's only true for a few queries, and only as long as the servers are doing nothing [19:53:51] valhallasw: Ok, so I coped in a frame-booter that works for me locally that uploads an HTML page called frame.html. Online it gave me an error, and checking the log tells me what it is: (https://dpaste.de/GKn8). Which I find odd, because, well, it's right there: (https://dpaste.de/eiSS). [19:54:27] Am I in the wrong folder again? [20:01:56] ResMar: I don't know. [20:02:37] ResMar: if Google is right, it should be in a templates/ subdir [20:03:08] http://flask.pocoo.org/docs/0.10/quickstart/#rendering-templates [20:04:22] 6Labs, 10Tool-Labs: Provision and test tools-mailrelay-01 - https://phabricator.wikimedia.org/T97574#1415116 (10valhallasw) [20:13:20] 6Labs, 7HTTPS, 3Labs-Sprint-104, 5Patch-For-Review: Renew toolserver.org SSL certificate - https://phabricator.wikimedia.org/T104211#1415147 (10Andrew) 5Open>3Resolved [20:15:46] 6Labs, 6WMF-Legal: Make sure tools can be taken over after they are abandoned - https://phabricator.wikimedia.org/T102066#1415150 (10ZhouZ) That's a good point @Ricordisamoa. I think ideally, we do want developers to keep their OAuth key private only to themselves while providing access to other parts of thei... [20:20:04] 6Labs, 10Tool-Labs: Provision and test tools-mailrelay-01 - https://phabricator.wikimedia.org/T97574#1415157 (10valhallasw) I've braindumped a checklist at https://etherpad.wikimedia.org/p/T97574 assuming the new mail server manifest gets merged. One thing I'm not sure of is how to test the server without act... [20:20:33] YuviPanda: ^ documented some stuff there [20:20:41] time to go offline now :w [20:53:04] 6Labs, 10MediaWiki-extensions-OATHAuth, 10MobileFrontend, 10wikitech.wikimedia.org, 3Reading-Web: MF Special:Login doesn't have a field for 2FA - https://phabricator.wikimedia.org/T103771#1415262 (10Jdlrobson) 5Open>3Resolved a:3Jdlrobson That patch got merged. If you enable $wgMFNoLoginOverride =... [20:53:06] 6Labs, 10Wikimedia-Extension-setup, 10wikitech.wikimedia.org, 7Mobile: Install MobileFrontend on wikitech - https://phabricator.wikimedia.org/T87633#1415265 (10Jdlrobson) [20:56:29] 6Labs, 3Labs-Sprint-103, 3Labs-Sprint-104: Limit available images on horizon - https://phabricator.wikimedia.org/T91782#1415272 (10Andrew) This is now live-hacked on Californium, and there's a pending upstream patch here: https://review.openstack.org/#/c/197306/ [21:04:23] 6Labs, 3Labs-Sprint-104: Recover wikidata.php from magnustools - https://phabricator.wikimedia.org/T104337#1415341 (10Ricordisamoa) [21:04:53] 6Labs, 3Labs-Sprint-104: Recover GND bot from wikidata-todo - https://phabricator.wikimedia.org/T104336#1415344 (10Ricordisamoa) [21:25:19] !ping [21:25:19] !pong