[00:38:53] Change on 12mediawiki a page OAuth was modified, changed by Base link https://www.mediawiki.org/w/index.php?diff=828535 edit summary: Marked this version for translation [00:39:03] Change on 12mediawiki a page OAuth/en was created, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?title=OAuth%2fen edit summary: Importing a new version from external source [10:18:56] !proxy [10:18:56] did you know we have a proxy ? :-] https://wikitech.wikimedia.org/wiki/Help:Proxy [10:19:00] !ssh [10:19:00] https://labsconsole.wikimedia.org/wiki/Help:SSH [10:19:04] !redir [10:19:08] !sshproxy [10:19:12] @search proxy [10:19:12] Results (Found 2): socks-proxy, proxy, [10:19:17] !socks-proxy [10:19:18] ssh @bastion.wmflabs.org -D ; # [11:30:45] Hi all [11:30:52] I have a Q [11:32:16] What means " Server unexpectedly closed network connection" ? [11:36:22] No one there? [12:03:53] Elph: exactly what it says [12:04:03] no answer What means " Server unexpectedly closed network connection" ? [12:04:08] the server for some unknown reason closed a connection [12:04:19] for example, it died? or connection died? [12:04:30] I think it is pretty descriptive error [12:04:48] the connection was closed by remote with no explanation why [12:05:26] what's the solution [12:05:38] I don't even know what program gives you this error [12:05:46] putty [12:05:49] aha so ssh [12:05:57] what server is it [12:06:01] no> it's ok [12:06:13] ? [12:06:15] what server is it [12:06:16] tools-login.wmflabs.org [12:06:52] it may be a firewall? [12:06:58] do you see anything before it closes the connection? [12:07:03] no. [12:07:23] did you make putty not close the window when connection edns? [12:07:24] ends [12:07:58] I recommend you to get a better client than putty, for example ssh compiled for windows [12:08:02] it's part of msys [12:08:08] yes [12:08:10] you could just run ssh -vv to get the details [12:08:22] I personally think it's your firewall [12:08:28] or maybe some authentication problem [12:08:38] did it ever work before? [12:08:51] Could also be a timeout on a transparent proxy somewhere [12:09:00] yes i work until 3 days ago [12:09:22] it works until 3 days ago [12:09:39] ok [12:09:47] in that case I think it's connectivity problem? [12:10:05] let me check ssh logs on login [12:10:13] ok [12:10:29] can you try login now [12:11:04] no [12:11:10] i can't [12:11:20] i thin it's a authentication problem [12:11:32] Failed publickey for elph from xxxxxxxxxx port 53391 ssh2 [12:11:38] fatal: Access denied for user elph by PAM account configuration [12:11:47] yes it is [12:11:59] because i run my bot direclly some days ago [12:12:07] hmm [12:12:24] @labs-user Elph [12:12:25] Elph is member of 1 projects: Bastion, [12:12:33] you were removed from tools project [12:12:38] but I don't know who did it [12:12:42] let me check logs [12:12:43] !sal [12:12:43] production: https://wikitech.wikimedia.org/wiki/Server_Admin_Log labs: https://wikitech.wikimedia.org/wiki/Labs_Server_Admin_Log [12:13:09] nothing in there [12:13:19] ok did you receive some e-mail regarding this? [12:13:26] or any other message from someone [12:13:42] I think it's pretty lame to remove someone and don't even notify them :P [12:14:16] I think Coren would know, but he is likely sleeping now [12:14:43] no email & no messages [12:16:22] !log tools petrb: despwapping -login (swapoff -a && swapon -a) [12:16:25] Logged the message, Master [12:17:11] Elph: ok, just wait a moment for Coren let him decide [12:17:47] ok. but where is Coren? [12:22:45] In a timezone that means you will do bes to wait for another ~3 hours. [12:23:56] he is in Canada somewhere I think [12:49:07] Elph: It wasn't the first time I had to kill a large number of bots of yours from tools-login, and every time I left you warnings plastered over every session. I'll return access to tools, but do *not* run noninteractive jobs on the bastions again. It's against the rules but, more importantly, it prevents everyone /else/ from working. [12:50:28] Elph: If you need help with using the grid, just ask. [12:52:08] ok [12:52:12] thank you [14:24:52] thanks for the review mutante :) [14:36:02] Coren: maybe we should setup a -sandbox instance for newbies to learn with grid on, so that when they accidentaly start their bot on bastion, it will at least be a sandbox bastion which wouldn't affect other regular users... but that's just an idea [14:36:58] Coren: also if you are going to recreate all instances, can you make new bastion a larger instance? :P [14:37:05] That'd just move the problem elsewhere. It's not going to be so disruptive in eqiad since the -login and -dev boxen will be bigger. [14:37:10] And yes, that was the plan. :-) [14:37:24] Coren: yes, that's what I mean, move the problem elsewhere, where it doesn't matter :D [14:38:03] but idc... [14:46:07] Quick Mediawiki maintenance question. How can we delete a sub page parent and all its children? [15:11:59] renoirb: You know, I don't think there /is/ a mechanism for that that I know of. [15:12:39] From a database point of view, subpages aren't special. [15:12:41] Thanks Coren. I doubt that it didn't cross the mind of the developers to have such feature [15:12:53] That's what I am considering Coren [15:13:10] I found the nice option in php maintenance/nukePage.php running --profiler=text [15:13:22] it gives some database schema hints [15:13:58] What I do not like is the idea of deleting page and not being sure if I am also bring down revisions [15:14:20] it would require some time just to study the schema. I wonder if somebody actually documented the DB query for doing so [15:30:57] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/List of Toolserver Tools was modified, changed by Frog23 link https://www.mediawiki.org/w/index.php?diff=829238 edit summary: /* Active Tools on the Toolserver */ DeadLinkFinder added [15:35:37] petan: ping? [15:35:37] Hi Danny_B, you just managed to say pointless nick: ping. Now please try again with some proper meaning of your request, something like nick: I need this and that. Or don't do that at all, it's very annoying. Thank you [15:38:07] wm-bot, did you sleep poorly last night? [15:38:08] LOL [16:49:54] andrewbogott: ping [16:50:12] YuviPanda: what's up? [16:52:45] * andrewbogott notes that wm-bot is more polite to yuvi than to Danny [16:54:08] andrewbogott: haha :D [16:54:17] andrewbogott: sending an email to labs-l about the proxy? :D [16:54:55] yeah, I have it written but I just tinkered with DNS and now the api is broken pending propagation [16:55:07] …that or I broke something else unobvious :( [16:55:31] andrewbogott: the DNS API? [16:55:55] no, just, the proxy API is hosted on proxy.wmflabs.org and right now wikitech can't see proxy.wmflabs.org [16:56:01] aaaah [16:56:18] no9t network reachable? [16:56:28] weird because my local system can… [16:56:38] DNS. unknown host proxy.wmflabs.org [16:56:45] propogation [16:57:07] right [16:57:29] Don't want to announce a service /while/ it is broken :) [16:59:23] andrewbogott: heh yeah :D [18:39:59] andrewbogott: thanks for the email! [18:40:16] YuviPanda: yep, hope it's accurate and things stay up for a while :) [18:40:22] :D [19:50:18] * YuviPanda waves at andrewbogott [19:50:32] * andrewbogott waves back [19:50:38] andrewbogott: i can haz new project? [19:50:50] sure, what name? [19:51:11] andrewbogott: 'aargh' [19:51:12] ? [19:51:18] really? [19:51:24] Is that descriptive? [19:51:32] well, in some ways :P [19:52:01] andrewbogott: data4all? [19:52:09] or sql4all [19:52:11] that's a little better :) [19:52:27] i stil like aarg :P [19:53:41] ok, data4all created [19:54:24] lol [19:57:24] andrewbogott: sweet! Thanks! [20:07:05] !log wikimania-support Created wikimania-scholarship.wmflabs.org proxy [20:07:07] Logged the message, Master [20:07:32] bd808: woot! nice :) [20:07:53] YuviPanda: Seems to work and super easy [20:07:59] bd808: yup! [20:08:07] bd808: you also get SPDY for free btw [20:08:45] And SSL! Awesome [20:09:20] yeah that too :) [20:20:03] !log logstash Created logstash.wmflabs.org proxy. [20:20:05] Logged the message, Master [20:21:49] YuviPanda: We don't do anything to prevent name collision, do we? That might be a good feature to add at some point :) [20:22:10] Hello everyone, i'm a french student working on project wanderwiki :http://en.wikipedia.org/wiki/User:FatJagm/WanderWiki, and i have some issues with the database i have created [20:23:17] andrewbogott: ah, yes :D [20:23:32] andrewbogott: we could do that in wikitech, perhaps...? [20:23:55] yeah, by checking the ldap/dns entries. [20:23:59] Maybe that's better anyway. [20:24:12] yeah [20:24:17] actually it /maybe/ does that already. [20:24:18] * andrewbogott tries it [20:24:21] hah :D [20:24:44] Basicaly i'd like to create mysql functions but it seems i don't have the rights to do it so i'd like to know if there is a way to have it? [20:24:53] YuviPanda: Yeah, "Failed to add mwreviewproxy entry for IP address 208.80.153.214. " That's not too bad. [20:24:58] Could be a little more helpful [20:25:03] andrewbogott: :D [20:25:06] But at least it doesn't clobber the existing proxy. [20:25:11] yup that's nice [20:27:22] andrewbogott: need to check which projects are still running off proxy-dammit and ask them to move [20:28:34] Or we can just move them... [20:30:27] andrewbogott: hmm, that's possible too :D [20:30:41] andrewbogott: I'm not projectadmin on analytics. Think you can move theirs? [20:30:48] sure [20:31:10] andrewbogott: let me find what it maps to [20:31:39] halloooo [20:31:49] andrewbogott: :) [20:32:13] hey -- we're talking about migrating your proxies from proxy-dammit to the new, more-official proxy host. [20:32:22] Do you want to do that, or at least tell me in advance which ones I can drop? [20:32:25] proxy-dammit? [20:32:25] hmm [20:32:26] uhh [20:32:45] not sure what that is... [20:32:46] so [20:32:54] we have a public IP on one of my nodes [20:32:59] haproxy is running there manually [20:33:06] and proxies to other hosts [20:33:09] based on name [20:33:37] andrewbogott: metrics.wmflabs.org -> wikimetrics.pmtpa.wmflabs:80 [20:34:01] andrewbogott: that's the only analytics thing running on the proxy [20:34:06] proxy-dammit [20:34:24] Oh, well, then... [20:34:32] that's trivial to move. So, nevermind ottomata, I'll take care of it. [20:34:49] sorry for the confusion ottomata [20:35:45] ottomata: proxy-dammit was the former back-end for the dynamic proxy… now that things are puppetized we're migrating to dynamicproxy-gateway. No reason for you to know (or care) about that though :) [20:35:45] oh [20:35:47] wikimetrics [20:36:03] hm, i think milimetric knows more about that than me [20:37:09] andrewbogott: let me know if you need anything from me [20:37:25] milimetric: still using gingle.wmflabs.org? [20:37:37] it maps to limn0 [20:37:38] YuviPanda had customized the response timeout for me to 10 minutes because I had upload that used to take a long time [20:38:02] that's been made async since then so I'm fine with something shorter, but just curious, what would be the timeout on the "official" proxy? [20:38:07] milimetric: that was just set for everyone, I don't think this change should change [20:38:15] although we might want to reduce it later [20:38:37] andrewbogott: gingle is sort of used right now, but if it's a pain you can kill it for now (I'll stand it up later) [20:38:40] milimetric: so that's a 'yes, still using gingle'? [20:38:42] it just links commits to mingle [20:38:57] yes, but it's nothing important [20:39:48] ok YuviPanda, that's cool. But if you do want to make the timeout something more reasonable though, I'm fine with that. Also related, I think I ran into a POST size limit. Can that be increased? [20:39:59] anyway… moved :) [20:40:13] milimetric: oh, yeah, should be. can you file a bug so I don't forget? [20:40:14] thx andrewbogott [20:40:32] ok YuviPanda, I'll file it if necessary. Thx [20:40:38] I think you also had both metrics.wmflabs.org and wikimetrics.wmflabs.org -- does that sound right? Do you still want both? [20:41:00] YuviPanda: Do you need any help with adapting your happy fun proxy for the tools stuff? It would be cool if I didn't have to mess with replicating tools-webproxy in eqiad. [20:41:06] i don't know what wikimetrics.wmflabs.org is. People just use metrics.wmflabs.org [20:41:14] cool. [20:41:44] Coren: ooo that'll be awesome [20:42:15] milimetric: ok, you should be all set then… please let me know if something breaks int the next hour or so as DNS propagates. [20:42:29] sweet, will do andrewbogott [20:43:06] Coren: it shouldn't be too much work to replace webproxy as it is [20:43:18] Coren: only thing is the portgrabber stuff, will need to be different [20:43:59] Well yeah; that's why I factored it out. All it needs to do is speak to your proxy rather than dump stuff in a file and we're golden. [20:44:48] Adapt portgrabber -> everything Just Works™ with whatever frontend. [20:45:38] That'd be in fact /easier/ than the NFS file locking I have to do now. :-) [20:47:08] And the end result (nginx in front of a horde of lighttpd) >> apache in front of apaches. [20:47:32] Coren: yeah! [20:48:00] Coren: actually I'm going to do it now [20:48:13] Coren: does tools have a public IP quota? [20:48:15] * YuviPanda checks [21:03:46] !log tools created tools-proxy-test instance to play around with the dynamicproxy [21:03:48] Logged the message, Master [21:05:01] YuviPanda: All projects have a quota. [21:05:15] Coren: yeah, I meant does it have IPs left in its quota [21:05:24] english is hard! :D [21:05:45] it does [21:06:15] I might have one or two left. Were you planning on setting up a /distinct/ yuviproxy for tools rather than use the general one? [21:06:46] Coren: yes, because portgrabber will need a different interface [21:06:55] Coren: and the apache stuff would be different [21:07:00] * Coren nods. [21:07:05] You are making a sense. [21:07:58] Coren: good to know :D [21:10:04] puppet is sooo slow [21:13:02] andrewbogott: did you remove metrics.wmflabs.org from proxy-dammit? [21:13:08] yes [21:13:14] sweet [21:16:25] Coren: status/progress of bug 56995? do we need additional packages? htmlgen/xmlgen and cgi i guess? [21:16:59] giftpflanze: Lemme see. [21:17:48] giftpflanze: Ah, that's the custom package. I'll try to do it this week; but I'm very busy with preparing the move to eqiad. [21:18:04] ok [21:24:51] Coren: do you know where to find the two additional packages? should i make a comment mentioning them? [21:25:18] giftpflanze: Alternately, if you can find a sucke^H^H^H^H^H volunteer to make a proper .deb of it it becomes trivial for me to deploy it. :-) [21:25:32] giftpflanze: It'd probably be best if you mention them. [21:25:52] ahem [21:25:55] * YuviPanda pokes Coren and andrewbogott with https://gerrit.wikimedia.org/r/#/c/97625/ [21:26:02] Ryan_Lane too :D [21:27:29] needs a jenkins run [21:37:58] andrewbogott: puppet help? [21:38:09] yeah, just a minute... [21:40:10] YuviPanda: what's up, can I help? [21:40:27] ottomata: probably! [21:40:32] I'm free now too. [21:40:33] ottomata: duplicate definition issue [21:40:34] https://dpaste.de/o9WB [21:41:22] this is on instance tools-proxy-test, which has role::labs::tools::proxy [21:41:53] do I need to have a class in toollabs::proxy that then declares ::dynamicproxy? I feel that won't even fix things [21:42:23] I don't even know why it has a problem with the labsproxy stuff. that rule isn't being applied here [21:42:31] andrewbogott: ottomata ^ [21:42:47] it's also been a month or so since I touched puppet, so maybe I'm missing something... [21:43:21] yeah YuviPanda, you are including this class twice with different parameters [21:43:28] once in labs.pp line 61 [21:43:35] another time in labsproxy.pp on line 66 [21:43:49] do you really want every box with role::labs::tools::config to implement a proxy? [21:43:50] yeah but labsproxy.pp is defining a role that isn't being applied to this instance [21:44:06] looks like both class role::labs::tools::proxy and class [21:44:06] role::dynamicproxy [21:44:08] are being included [21:44:22] h [21:44:23] hm [21:44:48] andrewbogott: role::labs::tools::proxy, not ::config. It just inherits from config [21:45:13] ottomata: yeah but how? I don't have role::dynamicproxy ticked [21:45:49] just double checked [21:46:24] uhhhhn i hate nested classes! [21:48:36] hm [21:49:25] yah, hm, YuviPanda, not sure [21:49:36] i think I've had similar issues in the past…and usually fixed them like it looks you are tryign to, with :: [21:49:37] but [21:49:44] hm, or maybe I fixed them just by renaming the classes :p [21:49:54] ottomata: which classes? [21:50:08] role::dynamicproxy to be something else? [21:50:12] yeah maybe? [21:50:19] that's the only one that would make sense [21:50:24] well, sortof sense [21:50:41] ottomata: hmm, but it is already in use with the proxy project, I think andrewbogott even setup a new instance wit it, and it's fine... [21:50:47] that doesn't get failed puppet runs [21:51:21] I agree that this looks like it should work... [21:52:47] I've verified in ldap that that host only includes ldap::role::client::labs, exim::simple-mail-sender, sudo::labs_project, role::labs::tools::proxy [21:52:57] and base [21:53:13] andrewbogott: yeah [21:53:14] andrewbogott: so... [21:53:18] I'm confused :| [21:53:59] Seems like this could be the result of a missing } somewhere [21:54:06] Although I don't know how that would make it past jenkins [21:54:51] andrewbogott: yeah it prolly wouldn't [21:54:58] andrewbogott: plus this is on tools so a lot would be broken :P [21:58:37] andrewbogott: maybe class { 'toollabs::infrastructure': } [21:59:20] idk why [22:02:54] YuviPanda: I'm investigating still… will let you know if I turn anything up [22:03:23] andrewbogott: I think the patch I just submitted might be removing the toollabs::infrastructure thing, which I think is duplicated [22:06:44] I'm not sure this is related to the changes that have been going on lately, but my *first* connection to labs instances is always really slow [22:06:54] then if I disconnect and reconnect relatively quickly afterwards - fast [22:07:08] but if I disconnect for like ... 24 hours, it's slow to connect again [22:19:16] YuviPanda: try it now? [22:19:32] andrewbogott: need to wait for jenkins-bot [22:20:10] It's merged... [22:23:06] andrewbogott: that works [22:23:09] :D [22:23:10] thanks! [22:23:33] You're welcome, but I wish I had a clearer reason to condemn the old syntax other than 'one time that caused problems' [22:23:40] heh [22:23:47] yeah I hope so too [22:24:00] Ryan_Lane: would you expect this to fix anything? https://gerrit.wikimedia.org/r/#/c/97631/ [22:24:03] andrewbogott: also I guess you can't include something with params? [22:24:23] YuviPanda: Right, it's specifically the param syntax w/out params that gave me the creeps. [22:24:29] And, apparently, puppetmaster as well. [22:24:35] andrewbogott: hmm, why? [22:25:05] Because ugly -- I didn't actually think it would fix anything when I changed it. [22:25:20] Except, since it was hard for me to parse, thought maybe it was hard for puppet too. [22:27:56] andrewbogott: so what if in the future dynamicproxy class has some params? [22:28:02] will adding a param fix it? [22:28:29] Probably? [22:28:35] hmmm [22:28:50] andrewbogott: I remember ori talking about include vs class at some point for vagrant, maybe he'd know? [22:29:40] andrewbogott: so how do I copy the deb for nginx from proxy project to tools? [22:29:50] andrewbogott: also, think you can do that for me? :D [22:29:54] or just tell me how to? [22:31:05] YuviPanda: It's set up using labsdebrepo [22:31:24] andrewbogott: sure, but how do I access /data/project for proxy project from that of tools? [22:31:38] andrewbogott: tools already has labsdebrepo setup, so I just need to copy the deb now [22:31:39] scp? [22:32:13] andrewbogott: ah, hmm. was hoping for a better one :P [22:32:14] fine [22:32:28] what's better than scp? [22:32:36] ice cram? [22:32:37] *craem [22:32:41] *cream [22:32:42] I mean, you can move the file via your local host if you don't want to get the two systems talking to each other [22:33:04] nah it's fine [22:34:45] Coren: does tools already have labsdebrepo setup? [22:34:48] I guess not [22:34:50] * YuviPanda checks [22:35:30] mmm it's per host anyway6 [22:35:32] * YuviPanda does it [22:37:06] andrewbogott: weirdness again [22:37:17] andrewbogott: it is now trying to install the invisible unicorn api there [22:38:04] * YuviPanda investigates [22:38:09] YuviPanda: that may not be what you want, but that's what that puppet module does... [22:38:13] because I made it do that [22:38:16] ah [22:38:20] andrewbogott: I'll just move it to the role [22:38:20] I think... [22:38:23] can't remember. [22:38:30] But, yeah, splitting it out should be trivial. [22:38:39] maybe :) [22:39:36] andrewbogott: yeah, doing it now [22:54:41] YuviPanda: /data/project/.system/deb [22:54:57] Coren: hmm, currently on /data/project/repo [22:55:06] because that's what https://wikitech.wikimedia.org/wiki/Help:Using_debs_in_labs says [22:55:09] anyway [22:55:15] I'm saying how it's setup on Labs. No reason it can't be unified. [22:55:28] But not straight in /data/project/ [22:55:35] hmm right [22:55:38] that's true [22:55:42] Coren: I'll move it in a bit [23:03:44] !self [23:08:16] !self is https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster [23:08:17] Key was added [23:08:34] (fixed link in Puppet coding page) [23:11:26] known issue with ntpd failing to bind on interfaces in labs instances? [23:11:36] Nov 25 23:09:32 muzilla ntpd[5756]: unable to create socket on eth0 (16) for fe80::f816:3eff:fe28:e8e6#123 [23:11:39] Nov 25 23:09:32 muzilla ntpd[5756]: failed to init interface for address fe80::f816:3eff:fe28:e8e6 [23:27:41] Coren: http://toodadoo.wmflabs.org/ [23:28:02] Coren: proxies to the apaches now, and can also proxy to the random domains [23:28:13] Coren: need to add a way for it to proxy based on URL paths, and then portgrabber [23:30:23] now to puppetize it