[00:43:47] Could someone please reset my password on beta.wmflabs.org? (username: PiRSquared) [00:44:14] I'd just create a new one, but I already have global sysopship there, and I want to test Flow. [00:44:30] (It's been a while since I used beta.wmflabs.) [00:48:44] (proof: http://deployment.wikimedia.beta.wmflabs.org/wiki/Special:GlobalUsers/PiRSquared?limit=1 - https://meta.wikimedia.org/w/index.php?title=Special:GlobalUsers&limit=1&username=PiRSquared17 ) [02:00:39] Is there any way to get a beta.wmflabs.org password reset without email? [03:52:10] Ryan_Lane: http://ganglia.wikimedia.org/latest/graph.php?r=4hr&z=xlarge&h=labstore4.pmtpa.wmnet&m=cpu_report&s=descending&mc=2&g=cpu_report&c=Labs+NFS+cluster+pmtpa [03:53:36] looks broken to me [03:53:38] ;) [03:53:46] I'm going to give it a while [10:38:41] hashar: around? [10:39:02] yurik_: yup [10:39:22] yurik_: I think you pinged me yesterday, I was busy withJenkins / conf calls all night long [10:39:22] sorry [10:39:29] no worries :) [10:39:43] could you remind me which server to use to run puppets to beta cluster [10:39:55] my IRC log is on another computer and i can't get to it :( [10:40:59] and also - how to execute puppet to add -- https://gerrit.wikimedia.org/r/#/c/89836/2/templates/varnish/mobile-frontend.inc.vcl.erb [10:41:35] the check for cluster_options has already been added there [10:42:16] hashar: and i have this patch to enable it (needs +2) - https://gerrit.wikimedia.org/r/#/c/90095/ [10:42:25] \O/ [10:43:14] yurik_: there is deployment-staging-cache-mobile02.pmtpa.wmflabs with IP address 10.4.0.52 [10:43:38] deployment-staging-cache-mobile02 is a Wikimedia Puppetmaster for itself (puppetmaster). [10:43:38] deployment-staging-cache-mobile02 is a Wikimedia mobile Varnish cache server (role::cache::mobile). [10:43:51] the role::cache::mobile puppet class takes care of installing the varnish + VCL [10:43:53] for mobile [10:44:15] puppetmaster let you have puppet on that instance to point to the local /var/lib/git/operations/puppet instead of querying the labs puppetmaster [10:44:16] SO [10:44:26] you can fetch a change in /var/lib/git/operations/puppet [10:44:30] connected to ssh -A deployment-staging-cache-mobile02.pmtpa.wmflabs [10:44:40] and when running puppet ( puppetd -tv ) , it will kindly apply the change you fetched there [10:45:21] yurik_: do you happen to have a Google plus account so I can share my screen with ya ? [10:45:25] might be easier to demo [10:45:27] yep [10:46:18] yurik_: amusso@wikimedia.org [10:46:32] I am too tired to talk though [10:46:38] but that would be fine to share screen shehe [10:58:44] hello [10:59:12] hi [11:00:33] i want make wikibot with php [11:01:47] for this work how run the bot [11:01:53] maybe use https://en.wikipedia.org/wiki/Wikipedia:Creating_a_bot#PHP [11:02:04] thanks [11:03:10] if you have question how to run it on the labs, read https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help throughly [11:03:26] it's a bit technically involved for those who didn't use clusters before [11:03:50] the documentation tries to make it a bit more intuitive by being sufficiently clear [11:06:53] Coren: https://bugzilla.wikimedia.org/show_bug.cgi?id=54164 - is it difficult to fix? [11:07:03] oh, thanks Svetlana [11:07:20] my bot is currently broken due to this, or should I workaround it? [11:11:47] yurik_: curl deployment-staging-cache-mobile02.instance-proxy.wmflabs.org [12:49:30] liangent: Not very. Lemme check. [13:11:38] Coren: are the public git repositories available locally somewhere on TL? [13:12:20] making cloning over the FS possible instead of over SSH (which is slow, especially for mw-core...) [13:12:45] valhallasw: You mean as files? No, and I recall there was a technical issue that made it difficult to share that way. Ryan_Lane is probably the best person to ask that of though. [13:13:08] I see [13:13:56] But you can clone over http though, which should be much faster. [13:15:43] liangent: Should not be too hard to add; it's just an exception to add to the schema for wikidatawiki. [13:20:10] Coren: isn't it ok for everything? or does it throw an error when the field doesn't exit [13:20:22] I mean, not just for (test)wikidatawiki [13:21:03] Only tables with no redaction at all create views with *; otherwise columns need to be explicitly enumerated. [13:26:38] (redoing S5 views) [13:27:19] liangent: {{done}} [13:43:58] Coren: and still not done for testwikidata see comments on that bug [13:44:08] though I don't need it for now [13:44:52] Hm. Is testwikidata a permanent project? [13:46:33] Coren: no idea [13:48:20] ok I bot seems working now on wikidata [13:51:52] Oh, bah, testwikidatawiki is on s3. It will take some time before the change goes live. [13:53:21] Coren: what's its difference from s5? [14:12:55] liangent: s3 has several hundred databases to go through. s5 has two. :-) [14:15:34] Coren: If you find time, https://bugzilla.wikimedia.org/show_bug.cgi?id=55619 would be nice to have. [14:23:09] anomie: Today is bug hunting day for me, so I'll probably get around to it. [14:23:23] Coren: ok, thanks [15:57:00] I'm getting a home partition r/o error, no idea why: [15:57:00] touch ~/nostril [15:57:02] touch: cannot touch `/home/adamw/nostril': Read-only file system [15:57:06] projectstorage.pmtpa.wmnet:/fundraising-awight-home on /home type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) [16:46:30] awight, that's probably something I can fix. What's the project name? [16:47:17] * YuviPanda blames glusterfs [16:47:23] andrewbogott: thanks! fundraising-dev-awight [16:47:41] awight: um… that is not a project. But I'm assuming it's the 'fundraising' project? [16:49:27] awight: or fundraising-awight? [16:49:49] andrewbogott: I think the latter... I'll go fumble around in Nova... [16:50:04] well, is it fixed already? If so, it was fundraising :) [16:50:49] andrewbogott: nope, the project is fundraising-awight [16:51:47] awight: how's that? [16:52:33] andrewbogott: THANK U [16:54:32] !ping [16:54:32] !pong [16:54:37] goddamit [17:00:11] Coren: do the lighthttpd instances run onthe grid? [17:00:12] or elsewhere? [17:01:19] YuviPanda: On the grid, on a dedicated queue. [17:01:22] ah [17:01:23] nice [17:01:50] (So that I can do slot-based allocation rather than vmem-based one given they all have the same executables) [17:01:56] nice [17:34:11] hrm, what is actually the difference between jsub and qsub? [17:35:01] MartijnH: jsub is a wrapper for qsub [17:35:58] is that a MW thing or an open grid engine thing? [17:36:28] cause I can't find documentation on jsub apart from the interwebs [17:36:39] er, apart from the wikitech page [17:36:48] MartijnH: jsub is toollabs thing [17:36:59] MartijnH: do a 'vim `which jsub`' just a simple warpper script [17:38:29] pretty elaborate wrapper script if you ask me ;) [17:38:56] hehe [17:57:58] MartijnH: It's just meant as a slightly more user-friendly interface that provides sane defaults for most things so that people don't need to understand the quirks of gridengine for most stuff. [17:59:36] Because the reaction of most people upon encountering gridengine - even when familiar with Unix and coding - usually goes along the lines of "omg, what does all this funky terminology even _mean_?! 'hard complex variables?'" [17:59:48] heh [18:00:07] are grid jobs allowed to spawn new grid jobs by the way? [18:01:27] MartijnH: Not atm. I've had a few requests for it but I've yet to have had a compelling argument to convince me to open /that/ can of works. [18:01:29] worms* [18:02:53] I can see an out of control job spawning sub jobs left and right being able to do a ton of hurting [18:02:58] otoh: awesomeness ;) [18:03:52] a usecase can be revive dead processes by the way [18:04:01] or dynamically scaling applications [18:05:22] MartijnH: I mean an actual use case, not an hypothetical one. :-) [18:05:47] Coren, I actually wanted to experiment with the latter [18:05:52] which sort of implies the former [18:06:08] * Coren politely points at toolsbeta [18:06:17] Experiments are thattaway. :-) [18:06:32] * MartijnH looks [18:06:56] Same setup, only no BOFH being all stern and annoying about experimenting with stuff that can bring the gridengine down. :-) [18:07:49] "To request access to Toolsbeta, please visit #wikimedia-labs on IRC" [18:07:58] o/ [18:08:36] Heh. Petan is normally the one who does this; but gimme a sec. What's your wikitech username? [18:08:46] Martijn Hoektra [18:11:55] Successfully added Martijn Hoekstra to toolsbeta. [18:21:47] thanks Coren [18:43:19] Coren: Ryan_Lane http://tools.wmflabs.org/sulinfo/sulinfo.php?username=MZMcBride [18:43:26] this was reported yesterday [18:43:36] MZMcBride was reported yesterday? [18:43:37] : The SQL server s3 is down or having issues. [18:43:50] warning Due to an issue on Wikimedia Labs s7 databases server, these informations cannot been displayed. Please wait that the problem is fixed or request help at #wikimedia-labs. [18:43:56] first it says s3, then it says s7 [18:44:22] sulinfo.php [18:45:51] is it csteipp-related? [18:46:02] because it's sul [18:47:20] It also says that because s7 has an issue, it can't display the information it then displays. [18:49:04] And I see nothing wrong with s3, or any of the enumerated databases. [18:49:10] (from a random sampling) [18:50:18] In other words, I have no idea what MZM is talking about. [18:50:21] told csteipp , maybe he knows more about SUL info [18:50:32] Coren: gotcha, ok [19:01:49] Coren: one of my lab mates finds grid job 1273421 errors out, chdir doesn't work, is that a known issue triggering it? [19:04:05] He's trying to chdir to *his* home, rather than the tool's. Default home permissions don't allow strangers snooping around. :-) [19:04:27] Also, he shouldn't have any dependencies between his tool and his personal home in the first place. [19:04:28] Coren: he's running it from his own user account? [19:04:53] Nettrom: He is. [19:05:26] tsk tsk. [19:05:35] I don't get it [19:06:07] it's as if I tried to run a Python script in my own home directory and it doesn't work? [19:06:11] why would that not work? [19:09:07] Nettrom: because we engineered things to use service users, rather than individual users [19:10:07] individual users are not supported on the gridengine. It may or may not work for non-continuous jobs under certain circumstances (because I haven't been dilligent it closing holes that aren't being actively abused) but is definitely not supported. [19:10:11] people don't ever want to go into another user's home directory and if we need to rescue a tool or bot that's unmaintained it's nice to have it disassociated with users [19:10:41] oh, is that documented anywhere? [19:10:54] !tools-help [19:10:55] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help [19:11:01] I believe it is there... [19:11:33] More importantly: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Rules [19:11:38] Rule 1. :-) [19:11:38] it's not very clear there [19:11:51] ah. it's way more clear there ;) [19:13:44] yeah, the general help doesn't make that clear... might be useful to add a note about grid jobs to Rule 1 as well [19:14:15] thanks for clearing that up! [19:14:37] we've already got a shared tool account set up, I'll make sure we all use it from now on [19:26:03] Perhaps a nice way to put it would be that "One's first command upon logging in should almost always be 'become'" :-) [19:27:21] (But a simple use case I wanted to support without fuss was the ability for a user to log in to do, like, a quick DB query (interactively!) without having to switch) [19:28:42] if I change a gitblit URL to reference HEAD I get errors. E.g. http://git.wikimedia.org/blob/operations%2Fpuppet.git/18f719ef3d297878e198a864f5bb31dd9cb047af/manifests%2Frole%2Fcoredb.pp works, [19:29:17] hmm, gitblit/git.wikimedia.org is probably an operations issue? [19:30:13] anyway, but git.wikimedia.org/blob/operations%2Fpuppet.git/HEAD/manifests%2Frole%2Fcoredb.pp gives error 500. [19:30:33] Clarity isn't helped in the /Help page by having its link to /Rules broken. :-) Fix't. [19:31:04] * Coren has very little knowledge of git black magic. [20:35:27] Coren: Can I have a personal 'tool' account #troll [20:36:03] Nope. [20:36:53] :O [20:38:02] On a similar note - are the tool 'user' database servers accessible outside of tools (internally)? [20:41:26] Can anyone here reset beta.wmflabs.org passwords? [20:53:42] Coren: I had a question about the labs-db /etc/hosts and how it changes [20:54:03] milimetric: Shoot. [20:54:04] in short, I'd like to update my copy of it as soon as possible after the official one changes [20:54:16] what's the best way for me to do that without giving you any extra work? [20:54:28] 1. do you send notifications I can subscribe to? [20:54:35] 2. I could just rsync it periodically [20:54:43] Ah, hm, ostensibly this should go into DNS. rsync is the easiest atm. [20:55:01] ok, cool. thank you! [20:55:03] :) [20:55:08] It should be fairly static now that the bugs with the duplicate DBs have been fixed. [20:55:22] I just fixed a problem by updating to latest - I believe cawiki got moved [20:55:36] but no big deal, it's been pretty stable for the most part [20:59:19] Coren: I see you closed bug 55619 a few hours ago, but /usr/bin/qstat still fails with "No such file or directory" from the script under lighttpd [20:59:55] * Coren checks. [21:07:02] anomie: Ah, there is an error applying puppet. [21:07:14] * Coren grumbles. [21:07:49] Stupid puppet and its prohibition in getting two identical definitions. [21:16:40] anomie: Puppet applying now. [21:17:06] Coren: Seems to work now! Thanks! [21:17:29] * Coren mumbles a few choice words in puppet's general direction. [21:36:53] Coren petan: Got a totally off the wall question for you. [21:37:09] Shoot. [21:38:03] OK, I'm developing a form to allow users to request articles as part of However, one of the requests I've gotten is to have the form on-wiki, and only have labs as the backend. Is that practical? Does that violate the ToS? Will I break stuff? [21:38:40] (for context: ) [21:42:25] If it's on /tool/ labs, then it's okay by the TOS (no need for a disclaimer there) [21:42:44] It is, sorry I wasn't being clear... [21:43:03] What I'm trying to figure out, is can I embed a form on Wikipedia and have it talk to tool labs... [21:44:39] hmmm... I don't think you /can/, at least not directly. You might be able to fake it by placing a tool labs form in an iframe? Or you could do some javascript trickery. [21:45:18] Yeah, but then it'll look out of place, unless I load the stylesheets... [21:46:11] Hmmmm… can I hit the API from tool labs? I assume so, but just making sure. If so, I can just store the text on-wiki, and pull it via the API [21:51:22] Sorry, stupid question I just realized... [21:51:48] Hi Matthew_ [21:51:55] Hi, PiRSquared! [21:54:05] Matthew_: please fix this : http://tools.wmflabs.org/matthewrbowker-dev/blacklist/check.php?value=%3Cbig%3E%3Cbig%3E%3Cbig%3EXSS [21:54:23] (escape the input at least) [21:54:23] PiRSquared: In regard to what? [21:54:36] you could embed a malicious script there [21:54:44] Oh, duh… yeah, that's a dev tool - it does nothing yet. [21:54:57] Speaking of which, that notice is broken... [21:58:38] PiRSquared: OK, that should fix the xss issue… broke the tool however… go figure XD [21:59:05] Sorry about that... [21:59:06] you only have to escape it when you output it, not when you do the useful stuff [21:59:26] PiRSquared: Yeah, it's basically just getting put against a regex... [22:00:02] Does it only check enwiki? [22:00:17] No, it should also do the meta blacklist. Like I said, it's a dev tool tho... [22:00:48] well, the XSS issue is not very important :P [22:02:04] Heh, still good to know about tho. [22:04:32] Hi! I am having troubles ssh-ing into a few instances that I could ssh into before. The error message is "Permission denied (publickey)." [22:04:37] For example I can no longer kraken-namenode-standby (of the analytics project). [22:05:08] PiRSquared: Should be fixed… I think, I'll do some more testing before that tool goes live. [22:05:34] Coren: poke [22:09:03] Does anyone here have shell or database access to beta wmflabs? [22:26:40] * Coren pokes Matthew_ back! [22:26:55] Coren: :D qchris had a question I thought you could answer :3 [22:27:10] * Coren reads scrollback. [22:27:16] Thanks! [22:27:47] ssh -v output is at: http://dpaste.com/1419357/plain/ [22:28:11] Heh. I can't help, qchris, even my root keys don't work. [22:28:22] * Coren points towards the reset button. [22:28:42] :-) [22:28:45] Ok thanks. [22:29:19] And, Coren? Thanks :) [22:29:58] Matthew_: That's what they pay me the big^H^H^Hmediu^Wadequate bucks for. [22:30:31] Coren: Can you access kafka-main1 or kafka-main2 from the analytics project? [22:31:03] (They show the same symptom for me) [22:31:14] Nope. Are they off puppet? This is how the root keys get distributed. [22:31:27] ottomata could log into them when we last tried. [22:31:43] I do not know about their setup :-( [22:32:40] (If they are using puppetmaster::self, they don't get the private repositories) [22:32:40] Anyways ... Thanks! [23:06:46] So I have situation where when I submit lobs to the job queue they don't run plus don't give me any error logs. [23:07:12] The code runs perfectly when I execute interactively in my virtualenv [23:08:02] I have used following convention jsub [23:11:39] auduwage: Define "lots"? because there is a per-tool limit [23:14:14] I am not really sure if I understand what you meant. [23:14:19] can you give and example? [23:14:46] This is how I submit the job jsub -N [23:42:30] beta labs down with 'DatabaseBase::factory no viable database extension found for type 'mysqli'' looks like [23:42:57] chrismcmahon: see discussion in #mediawiki [23:43:05] why would i get something like this: when i submit the job [23:43:10] libgcc_s.so.1 must be installed for pthread_cancel to work [23:44:42] auduwage: you need to increase the amount of memory given to the job [23:44:45] this is all in the docs.. [23:45:05] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Why_am_I_getting_errors_about_libgcc_s.so.1_must_be_installed_for_pthread_cancel_to_work.3F