[00:15:09] spagewmf: sorry, I was eating dinner… any better now? [00:25:16] So... http://ganglia.wmflabs.org/latest/ [00:25:22] Apache index of empty dir [00:25:55] petan: Was it compatible with eqiad, or does it need migration / being migrated? [00:27:01] Krinkle: I'm working on that project right now. [00:27:09] Building a new host since the old one was still running lucid. [00:27:11] cool [00:27:57] I don't think any of the folks who set it up originally are around anymore. I wouldn't mind help if someone is! [00:56:37] when I tar anything up on /data or /home, I get a few "tar: vagrant/.vim: file changed as we read it". The .tgz checksum is different but the contents seem identical. Perhaps the labs mounts don't have -noatime and updating the access time? [00:57:21] I've been seeing those too… I suspect it's something to do with gluster playing fast and loose with access times. [00:58:51] andrewbogott: maybe -noatime would help. Also BTW, the labs-vagrant role seemingly did not create /home/vagrant, or maybe it's on the instance "underneat" the mounted /home [00:59:17] bd808: ^ are you involved with labs-vagrant these days? [01:09:51] What would happen if a script I have malfunctions, and spams sendmail? What's the worst that happens? I send myself email spam? [01:12:38] a930913: Script = cron job? Or script = script that sends mail to "somebody"? [01:13:52] scfc_de: Script loops for events, on event uses sendmail to send me an email. [01:15:31] spagewmf: I am sort of involved with labs-vagrant. And I think there is a pending patch to operations/puppet to take care of creating the /home/vagrant directory. [01:16:24] thanks. I tar'd up most of the pmtpa /home/vagrant and transferred it, and `labs-vagrant provision` is working [01:16:34] spagewmf: https://gerrit.wikimedia.org/r/#/c/118053 [01:17:30] Cool. I know I built an instance in eqiad using it but I had to make the /home/vagrant directory manually [01:18:20] a930913: Then (probably) the only victim would be you. If the frequency is high (let's say, more than one mail per minute), then worst case would be that your mail provider considers Tools a spammer with this or that consequence (between delaying mail delivery to reporting it to a RBL). [01:18:55] When I built my instance I talked Cor-en into adding a vagrant user to labs ldap so that the role would work with the new NFSv4 servers [01:21:30] hm, so maybe spagewmf needs to add that user to the project? [01:22:18] a930913: And speaking of experience with cron jobs running amok when NFS was down or the replicas not accessible, once per minute didn't disturb /my/ ISP, and I think noone else's will complain either. [01:23:08] scfc_de: If it did go wrong, it'd probably be 50-100 per minute :p [01:24:46] a930913: Depending on your provider, that could be Bad™ [01:24:48] a930913: Are you mailing a RC feed? :-) [01:25:06] a930913: gmail, in particular, with throttle your incoming email for several days if that happens. [01:25:43] scfc_de: A filtered version atm (i.e. one page.) [01:26:19] Coren: It is gmail :p That's why I wanted to check that failsafes were in place. [01:26:59] a930913: Then I would probably write a errorlog instead and set up a check that tests whether it has been modified in the last five minutes or so. [01:27:02] a930913: There are none. If you send yourself 100 mail per minute, the relay with gleefully flood you. It would be wise to build some sort of throttle into your own code. :-) [01:27:36] I try to put as few straightjackets on the users as possible. [01:27:38] Coren: Won't gmail mark tools as bad? [01:28:29] a930913: Not unless you (and many other people) then report the flood as spam. [01:28:56] a930913: But it's a bad idea regardless; and the kind of thing that would have me cut off the flood from the source. [01:29:03] * Coren grins evilly. [01:30:32] Coren: How could you track what process is sending? [01:31:54] a930913: I probably wouldn't bother and turn the user off instead until it's fixed. The takeaway from this is: "don't send mail in a loop without some sort of throttling" :-) [01:33:48] I guess I keep it personal until then :p [01:35:35] a930913: its common sense [01:36:17] I send myself emails 12 emails per hour [01:36:23] and thats it [01:36:48] Betacommand: Just have a buffer? [01:37:08] a930913: No its properly spaced cron [01:37:41] a930913: you could output data to a log file and have that emailed every X minutes if not empty [01:37:51] Yeah, a buffer. [01:38:13] You then flush it every five minutes. [01:38:17] a930913: thats more than just a plain buffer [01:38:35] a930913: I would say every 10 minutes [01:49:44] a930913: One thing to consider: What do you do with the mails? If you have to parse errorlogs, having them already as files is a big plus. If on receiving 1 + n mails, you just delete all blindly, having an indicator mail ("something's wrong, log in") once per hour or so is probably enough. [01:53:34] scfc_de: It's emailing me diffs to my talk page atm, so I don't need to check it every few days. [01:58:47] a930913: = diffs of your talk page? You can let MediaWiki mail them to you. [01:59:02] Well, not the diffs, but the link. [01:59:21] scfc_de: Oh joy, a link :p [01:59:49] scfc_de: I get the link on IRC. I still don't check every edit. [02:00:05] scfc_de: My talk page gets more edits than some pumps. [02:08:08] a930913: There's also an RSS feed with diffs. I am not very invested in this issue, but setting up a bot on a remote server to have you sent changes per mail with the potential to snafu when you could achieve the same results with much less risk on your local machine doesn't seem very rational to me :-). [02:09:13] scfc_de: It uses the redis feed though :p [02:54:13] !mysql [02:54:26] !sql [02:54:36] !ping [02:54:36] !pong [03:23:05] Coren, ping [05:04:04] I just created a instance with role::lamp::labs , what is the address to access the web server from outside? [05:04:27] http://i-000002a6.eqiad-proxy.wmflabs.org and http://i-000002a6.eqiad.wmflabs.org don't work [05:04:48] * Jasper_Deng doesn't use Labs, but knows that your instance is behind NAT by default IIRC [05:07:28] At the time of PMTPA I was using .pmtpa-proxy.wmflabs.org ... now I deleted my PPTA instance and recreated one on EQIAD, but I don't know what is the new URL... on PMTPA no special NAT setting was needed, it was working just by running these simple steps: https://wikitech.wikimedia.org/wiki/Help:Lamp_Instance (which need to be updated by the way) [05:08:44] It is probably a URL with "-proxy" somewhere. [05:12:02] legoktm ^ [05:45:33] anyone here that constructs tools and such? [05:46:01] I need a tool that can list indef block IPs on en.wikipedia but also offer filtering based on 3 criteria [05:46:27] if IP has edits, if IP was blocked as an open proxy, if IP is on a dynamic range [05:46:37] perhaps as checkboxes [05:47:21] the second criteria is to be based on block rationale [05:47:31] and third one probably is non-trivial [05:47:46] first two crietria could be sufficient for now [09:24:36] Coren, are the databases still migrating - I can't find my tools-db database (new username s51230) [11:42:05] somone familar with gerrit-patch-uploader around? [11:52:03] Reason: Patch failed (is your patch in unified diff format, and does it patch apply cleanly to master?) (check log above for details) [11:52:03] :/ [11:54:48] Can someone explain to me what my options on Tools are regarding emails? Tim L (IRC nick?) had a go at https://en.wikipedia.org/wiki/User_talk:Jarry1250 but I'm still unclear what I should do if I just don't want my cron'ed qsub generating emails. Is there a flag for that? [11:56:29] jarry1250___: he is scfc_de [11:56:54] matanya: Thanks [11:59:08] jarry1250___, you need to disable your cron from generating emails [11:59:18] jarry1250___: add ">/dev/null 2>&1" to the end of your commands [11:59:52] e.g. [11:59:54] 0 * * * * /home/james/etc/command >/dev/null 2>&1 [12:02:42] and in regards to the mail there at the moment, if you don't need it - ask Tim to move it out to the home directory and then remove it [12:03:16] TheLetterE: But I still want a log output? [12:03:16] I just don't want the "task submitted" emails [12:03:56] if that's the case, use: [12:04:23] 0 * * * * /home/james/etc/command > /home/james/etc/command.log 2>&1 [12:05:25] adjust accordingly, as such jarry1250___ ;-) [12:05:38] TheLetterE: That overrides the default log output? As I say, I'm really quite happy with the default .out stuff, it's just the emails I don't need [12:06:07] I still get .out logs when I have the ">/dev/null 2>&1" at the end [12:06:23] I have another cron to remove these periodically, also [12:18:26] jarry1250___: there is a -quiet flag for jsub [12:19:32] gifti: Ah yes, that looks easy to do [12:21:38] matanya: hi :) do you know how to pull from github using gerrit patch uploader? [12:22:06] Steinsplitter: hi, haven't done it [12:48:44] hi Cyberpower678 [12:48:56] akoopal, hi [12:49:02] * Cyberpower678 yawns [12:50:03] Cyberpower678: when asking about erwin's tools a while back, I also asked about related changes [12:50:35] had a quick look myself, and it is using a personal database: u_erwin85.sc_ from head [12:51:01] I think if that is renamed to the new convention all should start working again [12:55:24] Cyberpower678: time to have a look? Don't think it is worth making a clone to fix it myself [13:43:29] Coren, andrewbogott_afk, or anyone else with the necessary rights: Poke re https://wikitech.wikimedia.org/wiki/New_Project_Request/mediawiki-core-team [13:43:34] (in yesterday's MW Core meeting, I was asked to set up a test instance for something, there was some talk about which project we could stick it in, and we concluded we should have a project designed to stick random instances in instead of abusing some other project) [13:44:28] hehe, create a random project! :) [15:15:48] anomie: created. [15:15:55] andrewbogott: Thanks! [15:19:27] petan: have you worked on the labs ganglia project at all? If so can you give me a hand? [15:19:56] so, uh, my labs instance was turned off, i guess there's a better place to create them now? [15:21:47] andrewbogott: no I didn't touch ganglia [15:21:53] andrewbogott: only icinga and huggle [15:21:55] dang :( [15:21:59] edsu, what project? [15:23:38] Hello all. [15:23:47] * Coren is long overdue for two straight days of sleep. [15:23:56] Catching up on scrollback. [15:24:12] jarry1250___: Hi! So should I move the mailboxes to your tool's directories so that you can delete them there? (And gifti was right about "jsub -quiet"; that's the best solution if you want to silence "has been submitted", but still want to be alerted to errors & Co.) [15:24:18] andrewbogott: http://wikistream.wmflabs.org/ [15:24:42] andrewbogott: https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistream [15:24:53] edsu: are you subscribed to labs-l? Or wikitech-l? [15:25:09] andrewbogott: not anymore [15:25:17] Or do you read your personal email? I've been trying to communicate with yiou about this for many weeks :( [15:25:18] andrewbogott: can't be bothered really [15:25:43] edsu: So, if you can't be bothered, shall I shut down/erase your project? Or are you still using it? [15:25:49] jarry1250___: -quiet to jsub is the simplest way to keep it quiet unless something breaks. :-) [15:25:56] andrewbogott: yes, i read your email ; sorry i didn't reply, but i don't have time at the moment to help with wmf housecleaning [15:26:10] andrewbogott: sure, go ahead [15:26:32] andrewbogott: if you don't think the project has value in the community that's fine [15:26:42] Cyberpower678: are you willing to spend time on related_changes, or give me access to the project to have a look? [15:27:06] andrewbogott: if you aren't sure the instructions to get it running are here https://github.com/edsu/wikistream [15:27:16] related_changes? [15:28:11] the tool from erwin [15:28:20] that is not working at the moment [15:28:33] edsu: Is there another volunteer who might want to take responsibility for the project? Generally we rely on volunteers to maintain those projects that they create. [15:28:44] I can't personally maintain all 60 projects :) [15:28:49] it is likely just a matter of giving the extra table used a correct name [15:29:55] andrewbogott: it's only one :) [15:30:08] andrewbogott: anyway, no -- i don't feel like doing wmf's job anymore on this [15:30:13] andrewbogott: seeya [15:30:20] wtf was that? [15:30:29] * YuviPanda pats andrewbogott and offers cookie [15:31:31] * andrewbogott appreciates cookie, has not yet had breakfast [15:32:37] * YuviPanda faxes andrewbogott some bacon [15:34:38] akoopal, I have no access right now. [15:35:39] andrewbogott, that's what I feel like I'm maintaining. :p [15:35:43] Cyberpower678: huh? you are one of the maintainers for 'erwin85' I though? [15:36:18] With no access to labs right now. [15:36:46] Cyberpower678: I'm behind… what access problem are you having? [15:36:46] Cyberpower678: ahhh [15:37:04] andrewbogott, no access to my keys right now. [15:37:17] Ah, I see. So, not an upstream problem :) [15:37:20] as said, if you trust me, give 'Akoopal' access and I am willing to look myself [15:37:20] aka I'm busy with other things right now. [15:37:26] Cyberpower678: you should be able to use gist to paste your keys and send them to akoopal :) [15:37:30] (when you have access again) [15:37:56] YuviPanda, :p [15:38:11] *cough*Not*cough*happening*cough* [15:38:47] Cyberpower678: so, is a project going to fall through the cracks if you're too busy? I'm happy to migrate things, just trying to weed out the projects that are fully defunct. [15:40:16] andrewbogott, I appear to be a maintainer of someone else's project. So I never migrated it. I finished the migration for those tools, but some appear to be broken. I have migrated everything else though, that I'm aware of. [15:41:02] Thank you for migrating! Please let me know if I can do anything to help. [15:41:15] And, do you know who the 'someone else' is? Do we need to add them to the admin list or something? [15:41:26] andrewbogott, and as you probably may or may not be aware of, I'm working out the kinks in the new tool I wrote. [15:42:08] * andrewbogott nods [15:42:27] andrewbogott, what do you think of it so far. [15:43:56] Cyberpower678: Sorry, which tool? I haven't been following along. [15:44:23] akoopal, I can make you an admin on Cyberpower678's project if that helps... [15:44:35] Cyberpower678: andrewbogott manages to be even /more/ busy than I am for the next week. :-) [15:44:52] We are need vacashiun. [15:45:00] andrewbogott, I thought your nod meant you were aware of it. I decommissioned X!'s Edit Counter, and replaced it with the new User Analysis Tool. [15:45:15] tools.wmflabs.org/supercount [15:45:26] andrewbogott, MY project? [15:45:33] Or erwin's? [15:46:07] Cyberpower678, akoopal, Um… I don't know? :) I just noticed that akoopal was volunteering to help with something but needed access. [15:46:16] which, if an existing admin approves, I can give him that access. [15:47:15] andrewbogott, I'm not one to approve. The tools belong to erwin [15:47:20] Cyberpower678: Supercount has a really good-looking interface! [15:47:22] Cyberpower678: ok [15:47:37] andrewbogott, thanks. [15:47:38] * andrewbogott also likes the name 'supercount' [15:47:57] andrewbogott: Cyberpower678 fwiw, supercount also seems to be loading lots of 3rd party CSS that's probably a privacy policy violation [15:48:05] The goal for the tool is to allow heavy analysis of users. [15:48:35] YuviPanda, I know. I'm going to relink those scripts to static [15:48:43] Cyberpower678: sweet :) thought you should know :) [15:49:01] I should setup a project at some point that just serves these commonly used scripts and such out of a varnish instance directly. [15:49:23] YuviPanda, is bootstrap in there? [15:49:44] Cyberpower678: the project doesn't exist yet :) I shall build it at some point [15:50:31] in static I mean. :p The interface uses bootstrap, and graphael [15:51:18] * Cyberpower678 sees both. :D [16:00:00] * Cyberpower678 reports notconfusing on the account of having a confusing username. [16:01:14] * notconfusing guilty as charged [16:39:41] Hi ! are the labs ready to grant new requests ? [16:40:28] I had made one last month, https://wikitech.wikimedia.org/wiki/New_Project_Request/mediawiki-VERP [16:44:07] tonythomas: I'll try to do that today. [16:44:21] Coren: Thanks a lot for that ! [16:49:28] Coren: While you're there, https://wikitech.wikimedia.org/wiki/New_Project_Request/Security_Tools :) [16:50:00] csteipp: Noted. [16:51:23] csteipp (follow up from half-conversation in #operations): know anything about labs ganglia? I note that you're a project admin. [16:51:57] https://gerrit.wikimedia.org/r/#/c/119256/ can't rebase atm [16:52:19] andrewbogott: Not much. Antoine set most of that up, iirc. I'm a labs admin because I used labs a lot for a couple of projects. [16:52:50] csteipp: you're specifically an admin of /that/ project. [16:52:55] but, ok :) [16:52:59] hashar: labs ganglia? Yours? [16:53:12] csteipp: I mean, the project called 'ganglia'. [16:53:42] Doh, I was thinking beta's ganglia... I actually don't recall ever looking at a ganglia project [16:54:01] Huh. There sure are a lot of people on the admin list who claim not to know about it :( [16:54:05] Oh, except I did a bunch of security fixes to ganglia last year [16:54:31] I bet that's what it was. Py and I were trying to fix some of ganglia web's security issues [16:54:50] andrewbogott: the labs ganglia has been created by a contractor iirc [16:55:09] andrewbogott: it doesn't even receive messages from the eqiad instances :-/ [16:55:20] I guess ganglia is something for post pmtpa migration [16:58:48] andrewbogott: Can I have Admin in the mediawiki-core-team project? [16:59:11] anomie: I think you do...? [16:59:43] Oh, oops, you don't. I'll fix [16:59:46] Thanks [16:59:48] done [17:17:41] !log deployment-prep Created and validated job that updates Parsoid on the EQIAD beta cluster \O/ [17:17:44] Logged the message, Master [17:44:42] * andrewbogott => lunch [18:08:51] andrewbogott_afk: That should be "op { 'andrewbogott': ensure => lunch }" :-) [18:09:07] ETOOMUCHPUPPET [18:09:20] Coren: not ensure => 'lunch'? [18:10:20] I'm presuming here that "lunch" is a natural keyword of the op resource type. :-) [18:10:34] ^^ [18:10:52] Coren: pywikipedia.org is not working yet -- is it something that needs to be deployed at some point? [18:11:42] also, https://gerrit.wikimedia.org/r/#/c/109237/ hasn't been merged yet [18:11:57] so I guess your 'close' was for the tools-end changes that were necessary? [18:13:21] valhallasw: I honestly don't know; the ticket was for repointing the domain name and setting up the redirects, which is done indeed. Lemme see. [18:14:02] Ah, wait, there's a completed patch but it's never been merged. [18:14:50] * Coren does that now so that it's /really/ closed. :-) [18:22:35] Coren: does that still need to be deployed manually, or will that happen automagically? [18:26:09] valhallasw: I'm in the process of doing so. [18:26:38] Coren: you are my hero [18:26:39] I'm doing so carefully; I've not done that often yet. [18:44:11] Cyberpower678: just wondering, who migrated erwin's tools then, was that magnus? [18:44:36] No one. Since it wasn't touched, it was Coren. [18:45:07] The automated process isn't so much "migrated" as "copied blindly" [18:45:29] ok [18:45:42] no, migrated from toolserver I mean [18:45:55] not the internal labs migration [18:50:47] Ah. No idea then. [18:52:36] haven't seen magnus on irc for a while? [18:54:42] valhallasw: Done, but there is some caching that needs to be flushed. [18:55:44] hurrah! [18:56:05] akoopal: I think Magnus volunteered on labs-l or toolserver-l?! But I believe there's even a Bugzilla for it. [18:57:29] probably labs-l [18:57:38] as I didn't read it [18:57:57] There are apparently many still NEW: https://bugzilla.wikimedia.org/buglist.cgi?quicksearch=erwin85&list_id=292105 [18:59:31] Is the web proxy deal for equad supposed to be insanely slow, or did I screw it up somehow? http://mwcore-contactpage-test.wmflabs.org/wiki/Main_Page takes a very long time to load. [19:02:17] ahhh [19:04:55] anomie: it's quite fast. [19:05:18] anomie: is it not loading at all or just really slow? [19:05:54] YuviPanda: Really slow. As in it seems to sit there for a few minutes, then suddenly everything loads. [19:06:14] anomie: curl from localhost? [19:07:02] YuviPanda: Just did a curl now. It took 60 seconds to do anything, then downloaded the whole page immediately. [19:07:17] anomie: ah. not proxy then :D [19:07:26] anomie: did you set this up yourself or used mediawiki_signlenode or labs-vagrant? [19:08:00] YuviPanda: role::mediawiki-install::labs [19:08:16] hmm, I guess that's mediawiki_singlenode. [19:08:19] not sure what's up :( [19:08:27] wait, let me see what that role does [19:09:24] YuviPanda: Ah. Did another curl, it fetched the first bit of the page then paused for a long time then continued. So yeah, probably not proxy. [19:09:32] anomie: :) [19:10:34] anomie: ah, ok :) it's mediawiki_singlenode, so poke andrewbogott :) [19:13:47] matanya: are you associated with the 'catgraph' project? [19:13:56] not at all [19:14:17] hm… jkroll and Kai Nissen [19:14:44] jkroll moved it from 'mothballed' to 'in progress,' just wondering if they need help. [19:15:43] hi andrewbogott. [19:15:43] bd808: You're still working on logstash? Need help with anything? [19:15:55] andrewbogott: I have about an hour or so now to rescue the design project. [19:16:11] andrewbogott: and could perhaps use help if you've the time :) [19:16:37] YuviPanda: sure. It looks like you've already revived the copy in eqiad? [19:17:03] andrewbogott: yeah, but I can't logikn [19:17:04] andrewbogott: at all [19:17:08] andrewbogott: can you see if your root key works? [19:17:10] ok, I'll have a look. [19:17:20] There are always some steps needed after an instance migrates. [19:18:21] Hm, it seems wrong, I'm going to delete it and recopy. [19:18:25] right [19:18:25] ok [19:18:35] it's also running labs-vagrant so might be issues there too [19:23:38] YuviPanda, can you log into the old instance? [19:23:45] andrewbogott: moment [19:23:57] trying [19:24:24] I guess I can, it's just very slow [19:25:59] andrewbogott: stuck at 'debug2: we sent a publickey packet, wait for reply' [19:26:03] on the way from bastion to the instance [19:26:13] if you're using eqiad bastion, that won't work for sure [19:26:28] andrewbogott: checking [19:26:51] andrewbogott: no, pmtpa [19:26:52] debug1: Executing proxy command: exec ssh -e none bastion1.pmtpa.wmflabs exec nc -w 3600 dptypes.pmtpa.wmflabs 22 [19:27:15] andrewbogott: bastion.wmflabs.org is still pmtpa? [19:27:17] or is that eqiad now? [19:27:23] pmtpa [19:27:49] andrewbogott: yeah, then I'm logging in from pmtpa [19:28:07] andrewbogott: worst case I can rebuild the instance. It is only about 2-3 hours of wokr for me [19:28:16] the new ee-flow labs instance in eqiad is really slow at serving PHP (30+ seconds a page). It's fast at showing an image, so it's not Apache. [19:28:44] spagewmf: if it is using mediawiki_singlenode, anomie also seems to have similar issues [19:29:01] YuviPanda: It has slowly gotten faster for some reason [19:29:07] heheh [19:31:21] YuviPanda: thx, just looking for ideas. anomie: an empty load.php was about 20 seconds, now it's quicker. [19:31:53] 404 page request ( http://ee-flow.wmflabs.org/wiki/Testwiki:Community_portal ) took 61 seconds [19:33:46] andrewbogott: if you think the issues might take too long to resolve (>2h) I can just rebuild it tomorrow [19:34:08] YuviPanda: well, I'll try a new migration in a minute, we'll see how it goes. [19:34:20] andrewbogott: ok! :) [19:34:44] YuviPanda: any chance you turned nfs storage on on that instance, and then turned it back off again? Or something like that? [19:35:04] andrewbogott: that might be possible. I don't fully remember. During initial setup you mean? [19:35:12] anytime, I dunno. [19:35:15] Just, $home is messed up [19:35:57] valhallasw: All done, after some fumbling trying to get varnish to forget about it. :-) [19:35:58] hmm [19:36:06] Anyway, I'll try a migrate. [19:38:07] Coren: cool! [19:43:21] !log deployment-prep created jenkins slave deployment-bastion.eqiad [19:43:23] Logged the message, Master [19:49:27] YuviPanda: the new instance is up now, you should be able to log in and tidy up. [19:50:30] andrewbogott: eqiad? [19:50:31] woot [19:50:32] let me check [19:50:35] yep [19:51:34] andrewbogott: woo seems to work. let me verify [19:53:04] YuviPanda: Once you're happy, move that project from 'mothballed' to 'migrated'? [19:53:11] andrewbogott: I will! [19:53:23] thx [19:56:14] marktraceur: still working on 'orgcharts' or can I mark it as migrated? [19:56:20] (Sorry if this is a repeat, I'm losing track :( ) [20:06:09] andrewbogott: I'm still working on it in that I haven't finished, but I *think* all the data is saved [20:06:12] Not sure [20:06:52] andrewbogott: woo, that seems to have worked! :)http://chicken.wmflabs.org/wiki/Main_Page [20:07:01] andrewbogott: i'll test for a couple more minutes and then update page [20:07:04] andrewbogott: thank you very much! [20:09:34] andrewbogott: marked as done. thank you [20:10:05] YuviPanda: great, thanks. Do you have any other migrations in progress? [20:10:48] marktraceur: ok, just trying to clear the list so I know when it's safe to shut things down. Do think you'll wrap things up soon? [20:10:51] andrewbogott: that's it from me. I might want to save some data from data4all but that's tiny. [20:10:59] YuviPanda: ok. [20:13:11] andrewbogott: I said I would work on it today, so maybe [20:13:24] ok [20:13:27] But it remains not a high priority.... [20:33:42] bd808: any clue why scap-recompile is lacking the executable bit ? :-] [20:33:45] in the git repo [20:34:41] https://gerrit.wikimedia.org/r/120867 [20:34:42] tools are dead? [20:34:46] wtf is in topic [20:34:58] who removed these channel links meh [20:35:49] !channels is #wikimedia-labs-nagios #wikimedia-labs-offtopic #wikimedia-labs-requests [20:35:49] This key already exist - remove it, if you want to change it [20:35:53] hashar: Because scap-recomile is dead tech? [20:35:53] !channels [20:35:54] | #wikimedia-labs-nagios #wikimedia-labs-offtopic #wikimedia-labs-requests [20:35:58] oh lol [20:36:01] :) [20:36:25] bd808: using it on beta to compile the ocaml Math system [20:36:31] @seenrx scfc [20:36:31] petan: Last time I saw scfc_de` they were changing the nickname to scfc_de, but scfc_de is no longer in channel because he quitted the network 01:36:28.0219250 ago. The nick change was done in #wikimedia-toolserver at 3/25/2014 6:19:59 PM (2h16m31s ago) (multiple results were found: scfc_de) [20:36:55] hashar: I thought there was a package for that now? You might ping Sam. [20:37:20] * hashar feels crazy :-] [20:37:32] Reedy: any clue how scap-recompile is provided in production nowadays? [20:37:43] hashar: copy paste? :D [20:38:27] bd808: puppet has a link to from scriptpath/scap-recompile to /srv/scap/bin/scap-recompile :-D [20:40:56] bd808: found out. puppet clone and then set a bunch of symlink under /usr/local/bin/scap-recompile :-] Thanks [20:41:03] Reedy: issue solved. [20:51:56] what's the right way to jsub processes which launch new processes? it seems like the new ones aren't being executed at all [20:57:52] Hey folks. I'm trying to look up some docs on creating a new WMF Labs project. [20:58:00] Can anyone help me figure out where to start reading? [20:58:05] !docs [20:58:05] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help [20:58:13] !tooldocs [20:58:13] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help [20:58:18] oh lol [20:58:25] these both point to same :) [20:58:31] not on tools though [20:58:38] yes I see [20:58:46] @search Help [20:58:46] Results (Found 29): ssh, start, security, git, port-forwarding, instance, accountreq, puppetmaster::self, addresses, sudo-policies, sudo-policy, puppetmasterself, requests, single-node-mediawiki, tooldocs, proxy, replicateddb, sudo, docs, wikitech-putty, tdb, mediawiki-instance, toolsvslabs, mediawiki, labs-putty, newweb, self, accessproblems, tools-request, [20:58:59] heh. So I want to start another project -- not join tools or toolsbeta [20:59:21] for whatever reason someone deleted our infobot for documentation [20:59:23] let me find it [20:59:29] thanks petan [20:59:52] halfak: what would the new project be used for? [20:59:54] I once wrote it and they removed it :/ [21:00:01] !labsdocs is https://wikitech.wikimedia.org/wiki/Help:Contents [21:00:02] Key was added [21:00:07] halfak ^ [21:00:21] I'm actually working to help notconfusing at the moment, but I will be creating a new project soon so this is helpful to me too. [21:00:44] mutante, if you are curious, see https://wikimania2014.wikimedia.org/wiki/Submissions/WikiCredit_-_Calculating_%26_presenting_value_contributed_to_Wikipedia [21:01:12] Here's a diagram of the VMs I'll spool up when I get there: https://wikimania2014.wikimedia.org/wiki/File:Content_persistence.system_architecture.diagram.svg [21:01:13] give me a reasonable project name and that it's WMF related, and i'll just create it now [21:01:17] ok [21:01:18] But I'm not quite there yet. [21:01:28] notconfusing, ^ [21:02:19] mutante mutante thanks, lets call 'wikidumpparse' [21:02:45] !log wikicredit - created with halfak as admin per https://wikimania2014.wikimedia.org/wiki/Submissions/WikiCredit_-_Calculating_%26_presenting_value_contributed_to_Wikipedia [21:02:47] Logged the message, Master [21:03:16] mutante, :P Thanks. I guess I don't have to bug you in a few weeks now. :) [21:03:46] notconfusing: do you have a link maybe that goes with that? [21:03:55] halfak: that was the idea:), why not [21:04:37] mutante, you want some proof of what is being worked on? [21:05:17] notconfusing: well.. not really proof, i belive you already because halfak is helping you, it's just nice to have [21:05:17] mutante, i basically want to implement halfak's mediawiki-utilities beta [21:05:18] https://github.com/halfak/mediawiki-utilities [21:05:26] cool [21:05:44] your labs name is like your irc name? [21:06:12] no its maximilianklein@tools-login:~$ [21:07:16] !log wikidumpparse - created with maximilianklein as admin for https://github.com/halfak/mediawiki-utilities [21:07:16] wikidumpparse is not a valid project. [21:07:19] there you go [21:07:23] eh, :) wait [21:07:36] oh i just got an echo notification [21:07:51] it should be a valid project [21:07:52] !log wikidumpparse - created with maximilianklein as admin for https://github.com/halfak/mediawiki-utilities [21:07:55] Logged the message, Master [21:07:57] there we go [21:08:54] halfak: https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikicredit see red "add documentation" link for later:) [21:09:08] notconfusing: same for you https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidumpparse [21:09:21] you both have your own server admin log as well [21:09:24] Thanks mutante. I've got lots of content to dump in. :) [21:09:32] https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikicredit/SAL [21:09:37] thanks mutante [21:09:39] https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidumpparse/SAL [21:09:52] so when you use !log yourproject here like i did, it adds to that [21:10:00] mutante is there documentation on how to log on and use it? [21:10:02] Hmm.. mutante. How difficult is it to rename a project? [21:10:19] WikiCredit will just be one part of the "Content Persistence" project. [21:10:50] notconfusing: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [21:11:14] halfak: not sure, but i guess making a new one is easier than renaming [21:11:32] Sorry for the trouble. :\ [21:11:50] projects are cheap, like instances, i think it's fine if you just use it to throw away later [21:11:55] no worries [21:12:05] Excellent. Could we make the switch now then? [21:12:22] all you have until now are projects, but not instances in it yet [21:12:28] Yup. [21:12:47] ContentPersistence? [21:13:00] Could just be "persistence" [21:13:48] mutante, I am not it seems a bastion member. Does that make sense if I already use tools-labs? [21:14:53] projects should have speaking names [21:14:55] imho [21:15:12] !log wikicredits - deleted in favor of "persistence" project [21:15:13] wikicredits is not a valid project. [21:15:18] gifti, "speaking names"? [21:15:21] !log wikicredit - deleted in favor of "persistence" project [21:15:21] wikicredit is not a valid project. [21:15:31] :) which proofs i removed it, haha [21:16:03] names that show you what it's there for [21:17:41] notconfusing: i don't know, it seems not if you ever ssh'ed into an instance before [21:19:03] Hm, does anyone know if Olexandertimkov or Olexander timkov is the real account? Both were created just now. [21:19:15] !log persistence - created with halfak as project admin [21:19:18] Logged the message, Master [21:19:42] hanks mutante [21:19:56] !project wikidumpparse [21:19:56] There are multiple keys, refine your input: project-access, project-discuss, projects, [21:20:06] !project-discuss wikidumpparse [21:20:07] https://labsconsole.wikimedia.org/wiki/Nova_Resource_Talk:wikidumpparse [21:20:25] SSL cert error.. hrmmbl [21:20:27] gifti, "persistence" is for tracking content persistence patterns in Wikipedia. See https://meta.wikimedia.org/wiki/Research:Content_persistence [21:20:30] i know, i know [21:21:09] andrewbogott: labsconsole.. it's been won't fixed, right [21:21:22] those links are just all over the place , f.e. bots [21:21:54] I don't know what we would fix. Having a second cert seems like overkill. [21:22:40] it has way more users than other things we bought certs for [21:23:22] adding it as alt name on the same cert is an option [21:24:52] mutante: ok… I haven't really done anything with certs. If it's easy to add as an alt name that's fine with me. [21:25:08] i don't know, i thought it already went by RobH [21:25:15] he buys all of them [21:25:35] but what i know is that users will create bugzilla for all of those.. been there.. :P [21:25:44] let me ask [21:26:45] i meant "i know adding it is totally possible, but i don't know how much it costs" [21:28:14] hah, created by JeremyB in RT.. usual suspect:) https://rt.wikimedia.org/Ticket/Display.html?id=6769 [21:28:17] knew it [21:34:43] mutante, what might be going on here [21:35:10] notconfusing@eigenzorg:~$ ssh wikidumpparse.pmtpa.wmflabs [21:35:10] Connection closed by 208.80.153.207 [21:35:10] ssh_exchange_identification: Connection closed by remote host [21:40:48] notconfusing: did you create an instance yet? [21:41:02] notconfusing: all i made was the project, but not an instance yet [21:45:25] notconfusing: so you need to fire up an instance (VM) in your project first [21:45:25] mutante, ok that makes sense, thanks [21:45:28] when it asks you which type, just pick the default [21:51:06] andrewbogott ebernhardson figured out our 61 second MediaWiki page serve times come from Apache contacting an nginx server with a 60 second timeout. [21:51:10] andrewbogott: poking around with strace, it runs: connect(13, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("208.80.155.156")}, 16) and then poll([{fd=13, events=POLLIN|POLLOUT|POLLERR|POLLHUP}], 1, 60000 [21:51:23] andrewbogott: that ip is filtered by the labs firewall, so it has to wait the entire 60s timeout [21:51:52] that ip seems to have an nginx server of some sort, not sure where/why yet [21:52:11] That's a proxy set up for the project, right? [21:52:23] not sure yet, are the proxys nginx? [21:52:33] all i know so far is its nginx/1.5.0 [21:52:46] ebernhardson: the proxies are nginx [21:53:17] ebernhardson: and they are 1.5.0. I don't know why the apache is trying to contact the nginx server? [21:53:19] https://wikitech.wikimedia.org/wiki/Special:NovaProxy is where I added ee-flow.wmflabs.org But image requests, load.php, api.php are all pretty snappy [21:53:43] YuviPanda: i'm not sure yet either, since it never connects i dont get to see the payload [21:53:59] load.php, etc - *everything* goes through the proxy [21:54:03] so, yeah, that should be the proxy connecting to the backend, which is apache. [21:54:18] right, and only index.php of MW pages have the 60 second timeout [21:54:20] So, if apache is trying to talk to the proxy… that suggests that something strange is happening on the backend. [21:55:26] yup, thats the ip returned for ee-flow.wmflabs.org, so its trying to connect to itself [21:55:31] hmm [21:55:31] the proxy (aka 208.80.155.156) is open to all IPs on port 80. [21:56:11] andrewbogott: if i log into the machine(ee-flow.eqiad.wmflabs) and `telnet ee-flow.wmflabs.org 80` it just sits, as if filtered by a firewall [21:56:21] it probably is filtered by a firewall :) [21:56:36] although i would agree, it shouldn't be doing that in the first place(connecting to self) [21:56:38] I think instances inside labs can't reach instances inside labs via public IP [21:56:49] or somesuch which I was told was a nova bug [21:56:52] YuviPanda: They can't, that's known, but why would it even try? [21:56:57] Coren: indeed. [21:57:00] whaaayyy [21:57:03] Not so much "bug" as "not supported" [21:57:16] I think maybe something is using wgServer for something it is not supposed to be using [21:57:56] ebernhardson: you could add an alias of ee-flow.eqiad.wmflabs to 127.0.0.1 in /etc/hosts. That might solve the problem, although it won't explain why it's happening in the first place. [21:58:05] YuviPanda: could be, we use $wgServer = "//" . $_SERVER["SERVER_NAME"]; for some reason [21:58:10] ebernhardson: andrewbogott http://chicken.wmflabs.org/wiki/Main_Page is fast enough, but it is running labs-vagrant and not mediawiki_singlenode [21:58:59] spagewmf: Ah, yeah, that $wgServer hack is needed to support https on some installs. I'd welcome a better solution. [22:01:45] I can also just give you a public IP if you want to stop thinking about this for now. [22:07:53] sendto(13, "POST /w/index.php?title=Special%3ARunJobs&tasks=jobs&maxjobs=1&sigexpiry=1395785232&signature=d09d8df550fc6b2e81381082ce2ad177204b9a67 HTTP/1.1\r\nHost: ee-flow.wmflabs.org\r\nConnection: Close\r\n\r\n", 193, MSG_DONTWAIT, NULL, 0) = 193 [22:08:14] something calling Special:RunJobs is the culprit [22:09:12] MediaWiki::triggerJobs [22:09:59] i could submit a patch to drop the timeout from 60s to something reasonable(5? 1?) but thats just papering over the problem [22:10:56] ebernhardson: try having $wgServer = "//ee-flow.wmflabs.org"; [22:10:57] ? [22:11:05] hmm, nevermind. [22:11:14] only if ee-flow.wmflabs.org could resolve to the internal ip :) [22:11:37] I am trying to understand why labs-vagrant doesn't have this issue :D [22:11:48] ebernhardson: so by default I think mediawiki tries to run a job every page view. [22:11:53] ebernhardson: if you turn it off this might go away [22:12:05] YuviPanda: easiest way is to open up the php debugger and set a breakpoint :) [22:12:22] ebernhardson: wgJobRunRate = 0; [22:12:30] that would do ti [22:12:39] https://www.mediawiki.org/wiki/Manual:Job_queue [22:12:43] I might be totally wrong [22:13:22] well, i'm going to submit a patch to change the timeout, nothing should have a 60s timeout. Other than that not sure, for the moment i have `127.0.0.1 ee-flow.wmflabs.org` in /etc/hosts [22:13:26] which lets it "work" [22:13:48] ebernhardson: it was a much smaller timeout, it was increased to 60s on analytics' request [22:13:54] oh [22:14:03] ebernhardson: they've some uploads coming through that are large enough to need 60s. [22:14:13] It used to be about 10m for a while, and then they fixed their code :P [22:14:29] if we had converted at the same time as the rest of the world we could use fastcgi_end_request() for those things :P [22:14:30] ebernhardson: anyway, ask milimetric. It should be a trivial change in the dynamicproxy module in puppet [22:14:47] s/end/finish/ [22:15:04] (it essentially flushes the output and closes the user connection, and then finishes processing) [22:15:10] ebernhardson: converted to what? [22:15:16] out of apache into a saner system? :) [22:15:22] YuviPanda: fastcgi (php5-fpm) [22:15:27] heheh yeah [22:15:41] nginx + php5-fpm ftw, but I don't think we will see it ever here. [22:16:13] shrug, i'd rather spend my time advocating hacklang in prod :) [22:16:19] ebernhardson: :) [22:16:23] ebernhardson: good luck! [22:17:57] its probably hopeless :P even though it directly solves real issues we have [22:19:31] ebernhardson: we used to synchronously wait for some db work - that was just for a week while we fixed a bug [22:19:34] shortening the timeout doesn't fix, though, does it? It will still fail, just faster? [22:20:01] andrewbogott: correct, but (and this is just imo) nothing should have a user waiting 60s for a timeout [22:20:07] people are uploading potentially large files (csvs with user ids) [22:20:13] thats incredibly long in internet time [22:20:21] ok, but -- that timeout was set for a reason. I don't remember what it was, but I don't want to blindly change it. [22:20:25] so these could have at most like 200k lines [22:20:31] Especially since… it doesn't fix the problem? [22:21:00] since each line is like let's say 20 bytes, then 4MB-ish is the highest size we'd expect [22:21:23] so what's a reasonable time for 4MB to get transferred to a labs instance? [22:21:49] andrewbogott: fair enough, perhaps i just have an unreasonable hate for anyhting that says "a user can wait 60s they wont click away or anything like that" [22:21:50] assuming 3Mbps, let's say 10 seconds right? [22:21:57] so I'd be ok if we made it 15 seconds [22:22:03] andrewbogott: but its unrealted to this specific issue, it just makes it more obvious [22:22:04] It's not always users, though. [22:22:19] milimetric: I've been on a 1mbps connection for the last 3 weeks 90% of the time, fwiw [22:22:36] well, YuviPanda, I guess you should upload no more than 60k user cohorts :) [22:22:47] :D I am distraught, I tell you [22:22:52] I also don't quite understand where the timeout comes in to play. [22:23:02] The issue is that the backend instance can't contact the proxy? [22:23:06] I explained above, but feel free to tell me I'm being stupid [22:23:12] If so, how does the proxies timeout matter? Since we can't contact it? [22:23:19] proxy's [22:23:52] andrewbogott: its not the proxy timeout, its the fsockopen() timeout in the php code that calls it [22:24:07] oh, so the timeout is in the mediawiki config /on/ the backend instance? [22:24:12] andrewbogott: yes [22:24:15] Ah! [22:24:25] Ok, i missed that. In that case I retract all of my earlier objections to changing the timeout :) [22:24:28] ebernhardson: aaah, I completely misunderstood. sorry. [22:24:29] i'm now very confused... [22:24:38] But I still think you should just add something to /etc/hosts [22:24:40] ok, milimetric is confused because *I* was confused. [22:24:46] ok :) [22:24:48] ignore, milimetric :) nothing to do with analytics [22:24:50] good luck [22:24:54] sorry [22:25:20] this is from includes/Wiki.php:654, its the only fsockopen call in there [22:25:40] ok, that's… entirely outside of my area of influence :) [22:54:18] I thought the runJobs happened at the end of a MW request [22:56:52] spagewmf: it does, but with apache mod_php the jobs have to finish before the request can end [22:59:59] ebernhardson: but... but why is the body of my Justin Bizzle request held up for "an HTTP request back to the server to run a specified number of jobs. [23:00:56] spagewmf: really, the problem is mod_php :P php5-fpm has the method php.net/fastcgi_finish_request, which basically says "I'm done sending things to the user, close the request and then let me do cleanup processing" [23:01:25] mod_php has no such thing, so before the web server can close the users connection it has to do everything, like job processing. [23:02:06] ebernhardson: and there's no way to flush the current request output buffer so I see something? [23:02:40] spagewmf: it should be flushed [23:02:50] spagewmf: my wget's were downloading the whole thing, then stalling [23:49:31] can someone add me - User:Maximilianklein - to Bastion Project please, i don't seem to be listed as a member there: https://wikitech.wikimedia.org/wiki/Nova_Resource:Bastion [23:53:20] is it correct that there are only 7 databases at the moment?