[01:51:42] it seems I have removed myself from the service group for my tool on tools.wmflabs.org [01:52:03] what is the best way to request someone to fix that? [01:53:06] id [02:00:18] cbm: bugzilla [02:00:38] cbm: But I'm around. What tool is that? [02:01:01] veblenbot. I changed the permissions within the past hour, if there is a log [02:03:16] {{done}}. By the way, you may want to look into specifying output files when you use jobs, you got a lot of files piling up in there. :-) [02:03:35] (You'll have to log off and back on again to get the group back) [02:03:48] thanks for fixing it. I do need to fix the problem with jsub creating files; I have been deleting them manually [02:04:32] cbm: The easiest way to help without loosing data is to specify a '-o somefilename'; it will append rather than create new files for every job. [02:04:48] And since most of your jobs don't actually have output, the file will normally not even grow. [02:07:13] Thanks, I'll go ahead and take care of that. I appreciate the fast fix of the access problem. [02:07:49] cbm: You're lucky I was around. But for the record, the best way is to open a bugzilla - any project admin could have helped. [02:16:39] Will do. Thanks again - i'm logging off [02:18:00] My bot keeps hitting a 503 when it tries to add a large amount of images to a page [02:18:06] is there anything I can do to get around this? [05:35:59] 3Wikimedia Labs / 3tools: deletion queries joined with tokudb replication tables are really slow - 10https://bugzilla.wikimedia.org/68918#c4 (10Sean Pringle) MariaDB 10.0.14 has TokuDB 7.5.0 [1] which now uses a bulk-fetch approach for delete [2][3]. The labs DBs have been upgraded. Please test and report b... [09:49:49] PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools.diskspace._var.byte_avail.value (22.22%) [10:25:29] RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK [10:42:49] 3Wikimedia Labs / 3tools: Move wiki.toolserver.org to WMF - 10https://bugzilla.wikimedia.org/60220#c35 (10nosy) I have a dumpBackup.php --full copied off the server if you need it. [12:13:32] 3Wikimedia Labs / 3wikidata: make instance for wbdoc.wmflabs.org work with normal puppet master - 10https://bugzilla.wikimedia.org/71511 (10Jan Zerebecki) 3NEW p:3Unprio s:3normal a:3Wikidata bugs Make instance for wbdoc.wmflabs.org work with normal puppet master. While you are at it might as well mo... [12:16:03] Are these DB problems tracked somewhere? https://tools.wmflabs.org/wikiviewstats/ [12:16:28] ugh I still have to re-enable puppet [12:19:37] andrewbogott_afk: should be enabled again now; I had disabled it for some live hack but I can let the puppet bulldozer erase everything now [12:28:32] Coren, hey. [13:43:15] 3Wikimedia Labs / 3deployment-prep (beta): deployment-rsync01 20GB hard drive is too small - 10https://bugzilla.wikimedia.org/71431#c4 (10Antoine "hashar" Musso) It seems the root cause of the issue was the LDAP being upgraded/unreacheable intermittently over the past few days. As a result, when puppet run... [13:43:45] 3Wikimedia Labs / 3Infrastructure: Prevent puppet from creating local user when they are defined in LDAP - 10https://bugzilla.wikimedia.org/71480#c1 (10Antoine "hashar" Musso) A second though, maybe the l10nupdate and mwdeploy User definitions in puppet should be given UID/GID that matches the one from LDAP. [14:56:23] hello, anyone there? I cannot resolve amy wmflabs domains in neither 3G or with my regular broadband connection [14:56:29] *connection [14:56:45] nuria_: are you at the WMF office? [14:56:58] andrewbogott no, in seattle [14:58:14] ok, looking... [14:58:29] andrewbogott thank you [14:58:42] nuria_: try now -- any different? [15:00:13] andrewbogott no... [15:00:26] andrewbogott but lemme clear all caches [15:00:44] andrewbogott and try curl [15:01:07] andrewbogott curl not working either [15:02:25] speaking of resolving domains, what's up with otrs-test.wmflabs.org? [15:04:15] I assume that the DNS issues are for wmflabs.org? [15:05:41] nuria_: I saw the outage you were seeing, but it's back for me now. Mind clearing your caches one more time? [15:05:55] andrewbogott doing that [15:08:04] andrewbogott, I created that as a proxy to otrs-test.eqiad.wmflabs a few days ago, but it hasn't been resolving for some people [15:08:34] andrewbogott ; still no luck (with curl cleaning cache) but i can ping the ip directly [15:08:48] andrewbogott like ping 208.80.155.156 works fine [15:09:01] which is http://metrics.wmflabs.org [15:09:36] Krenair: 'for some people' is different from 'for all people'? [15:10:04] It was working for me. I might have added it to my hosts file [15:10:30] 3Wikimedia Labs / 3deployment-prep (beta): deployment-rsync01 20GB hard drive is too small - 10https://bugzilla.wikimedia.org/71431#c5 (10Greg Grossmeier) Created attachment 16640 --> https://bugzilla.wikimedia.org/attachment.cgi?id=16640&action=edit disk space percent free graph So it appears that things... [15:10:35] actually it's working for me now, not in /etc/hosts [15:11:10] if someone can assist: I cant appear to resolve tools-login.wmflabs.org at the moment. Is that a part of the current DNS issues that are occouring or is that another issue? [15:11:32] OK -- so, as we decomission virt0 there are some dns hiccups. I suspect that some local dns caches are caching the empty record. Most likely it will recover when the cache expires in an hour. [15:11:36] JasonDC: probably the same issue [15:12:06] alrighty [15:22:10] I'm freakin' out here guys [15:22:18] No labs servers seem to be working for me [15:22:23] Apparently I'm the only one [15:22:32] 3Wikimedia Labs / 3deployment-prep (beta): deployment-rsync01 20GB hard drive is too small - 10https://bugzilla.wikimedia.org/71431#c6 (10Bryan Davis) 5NEW>3RESO/FIX a:3Bryan Davis (In reply to Antoine "hashar" Musso from comment #4) > It seems the root cause of the issue was the LDAP being > upgraded/... [15:24:02] marktraceur: I had problems with access via bastion.wmflabs.org but switched my ssh config to route through bastion2.eqiad.wmflabs and now things are fine for ssh access. [15:24:48] bd808: No, I'm talking about http access [15:24:52] I can't hit betalabs at all [15:24:58] Or our metrics instance [15:25:00] marktraceur: there have been some dns ups and downs involved with killing off virt0. You can add ips to your /etc/hosts or wait for dns caches to update [15:25:10] marktraceur: can you give me an example url? [15:25:18] marktraceur: probably your DNS cache is poisoned from earlier. [15:25:21] WMF but dns can be a fickle pig [15:25:35] andrewbogott: I was trying to do http://en.wikipedia.beta.wmflabs.org/wiki/Special:Upload [15:25:43] It says "server not found" [15:26:06] That is bad dns cache for sure. I'm looking at the page right now [15:26:17] hehe, WMF instead of WFM [15:26:28] Yup [15:26:35] ha [15:26:43] WMF WFM [15:26:55] marktraceur: yeah, works for me as well [15:27:03] Balls. OK, I'll just wait. [15:27:16] WTF WMF WFM [15:27:22] * bd808 stops [15:31:47] andrewbogott still no luck, will check with you in an hour or so , thank you! [15:31:58] nuria_: thanks, sorry for the outage [15:32:46] andrewbogott working with /etc/hosts for now, so i am not blocked [15:32:56] nuria_: try using google dns? :) [15:33:03] and if you're using google dns, maybe non-google dns [15:33:30] 3Wikimedia Labs / 3Infrastructure: Prevent puppet from creating local user when they are defined in LDAP - 10https://bugzilla.wikimedia.org/71480#c2 (10Bryan Davis) (In reply to Antoine "hashar" Musso from comment #1) > A second though, maybe the l10nupdate and mwdeploy User definitions in > puppet should be... [15:37:38] DNS problem for me also trying to get to beta labs [16:19:15] 3Wikimedia Labs / 3tools: deletion queries joined with tokudb replication tables are really slow - 10https://bugzilla.wikimedia.org/68918#c5 (10Silke Meyer (WMDE)) Ok, cool, Sean! Thanks! Merl, have you tried this, yet? [16:28:47] has anybody else also problem with missing dns entry tools-login.wmflabs.org ? dns for tools-dev.wmflabs.org works for me. [16:30:38] Merlissimo: there was a short dns outage, some dns servers have cached the broken entry [16:31:33] andrewbogott: can you give me the correct ip? [16:31:49] 208.80.155.130 [16:31:54] thx [16:32:52] andrewbogott: Is the DNS problem ongoing? I just got "The server at logstash-beta.wmflabs.org can't be found, because the DNS lookup failed." in my browser. [16:33:15] bd808: I don't know what's happening. Everything continues to look ok on my end [16:35:14] hmmm... works for me now. /me shrugs [16:36:37] Coren: virt1000 syslog keeps saying "virt1000 pdns[7326]: Error sending reply with sendto (socket=6): Invalid argument" <- any ideas? [16:38:12] andrewbogott: EINVAL is really not a normal error for sendto(); it smells like a bad fd, which implies that it tried to open a socket but couldn't. Perhaps it tried to create an IPv6 socket but it's disabled? [16:38:41] Is pdns configured explicitly to listen to one or more IPv6 addresses that aren't there? [16:38:42] It doesn't happen on labcontrol2001, should be the same config [16:38:47] but possibly a different firewall [16:39:06] Coren: Hrm. I have a crontab entry on tools.anomiebot to submit jobs to rotate the bot's logs, and it seems to not have been able to actually submit any of those jobs. "access denied (server host resolves rdata host "tools-submit" as "tools-submit.eqiad.wmflabs")" [16:39:09] That couldn't be firewall related; the kernel wouldn't know. But the ip config might be different [16:39:58] (That is, the network config on the box might be the difference) [16:40:08] sure [16:40:11] anomie: Could you open a bz? [16:40:15] Coren: ok [16:40:23] !log deployment-prep Added Gilles to under_NDA sudoers group [16:40:25] andrewbogott: Does one of the box have v6 and the other not? [16:40:27] Logged the message, Master [16:40:52] Coren: probably, labcontrol2001 is Trusty, virt1000 precise. What should I check for? [16:41:15] Just compare the 'ip addr' of both. [16:41:44] I'll wager 2001 has proper v6 config and virt1000 doesn't. [16:42:33] hm, both look ok to me actually [16:43:15] Coren: https://dpaste.de/VEHv [16:43:51] So they do. Did you check that the pdns config matches those IPs? [16:44:43] Neither has a reference to a v6 address that i can see [16:44:53] * Coren grumbles. [16:45:17] I don't think we can figure it out without stracing pdns from startup and seeing how a bad fd is being passed to sendto() [16:46:39] It doesn't sound like you think this is related to dns outages though [16:46:50] should tools-login be reachable right now? [16:47:04] Coren: because people keep showing up here and saying things like ^ [16:47:10] haha [16:47:36] gifti: try 208.80.155.130 [16:48:08] andrewbogott: I doubt it is; I'd wager that's dnsmasq not answering right for pdns to recurse to. [16:51:38] fingerprint changed, andrewbogott? [16:51:51] gifti: not that I know of [16:51:56] uh [16:51:59] Also, I resolve tools.wmflabs.org without issue [16:52:04] Coren: me too [16:52:11] Doc_Taxon_: ^ [16:52:51] gifti: why would you have a fingerprint for that IP anyway? [16:52:56] gifti: You may never have had the IP associated yet; check it against https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/tools-login.wmflabs.org [16:53:20] andrewbogott: no idea, I'm just a proxy [16:54:53] Coren: FYI, https://commons.wikimedia.org/wiki/Commons:Village_pump#Toollabs_facilitating_MP4 Looks like its still "active" [16:56:06] will the dns servers be reachable by 192.168.198.8:22 later again [16:56:47] Dispenser: Feel free to raise the issue with legal@wikimedia.org; this does not violate the TOU in any way and - absent a request from legal - I see no reason to take it down. [16:58:29] iirc it exists also some other project than commons, commons can do whatsoever they want for their internal policy but they don't drive all projects [16:59:29] phe: Its over software patents, WMF does not have a license to decode H.264 [17:00:18] There are free decodes with the license but only available as binary blobs [17:01:40] That may violate "(3) Proprietary software: Do not use or install any software unless the software is licensed under an Open Source license." [17:02:05] then it's no to comons to decide it, but wmf legalese [17:02:32] Dispenser: if there's any issue at all, it's a patent issue and not a license issue. aconv / x264 are open source [17:02:33] Dispenser: And the WMF is, afaict, doing no such thing. We are also not in the business of policing wether labs users need patent licenses to do something, or whether or not they have one. One thing we *do* know, is that it doesn't violate the terms of use because those libraries are very explicitly open source and not proprietary, and licensed accordingly. [17:03:18] This may be overly flip, but… I fear that if we worry about patents we'll have to stop runing linux altogether. [17:03:19] Dispenser: So that matter is closed. If you feel there is an issue that WMF Legal should be informed of, you are welcome to do so. [17:03:28] Dunno, maybe some patents are more worthy of attention than others [17:04:02] andrewbogott: can you give me the ip of wdq.wmflabs.org ? [17:04:31] Merlissimo: 208.80.155.156 [17:04:39] thx Coren [17:04:50] * JetLaggedPanda adds icinga alerts for our new DNS servers, Coren and andrewbogott :) [17:05:20] JetLaggedPanda: we sort of have them for the servers, just they were broken too until a few minutes ago :( [17:05:52] JetLaggedPanda: irc dns bot would be nice ;-) [17:06:07] andrewbogott: yeah, I realized :) Just that people have been asking you two for IPs here... :) [17:06:10] Coren: So would you like to post a follow up or should I? [17:08:00] Dispenser: I have no follow up /to/ post; I've already stated clearly that there are no TOU issues with that tool. The only valuable added opinion there could be on the matter is Legal's; anything else is just blowing smoke out of orifices. [17:10:28] Coren: Does WMF have any H.264 licenses for Tool Labs (besides the Canonical one)? [17:16:44] coren : fingerprint is okay, but publickey will be refused [17:16:54] jetzt : server refused our key / no support authentication methods available (server sent: publickey,hostbased) [17:17:18] Dispenser: That is neither here nor there. If you worry about Foundation liability, please contact legal@wikimedia.org. If you worry about the tool maintainers' liability, contact /them/ and let them retain legal council if they feel the need. Otherwise, I don't see what this has to do with Labs, or Commons. [17:17:55] übersetz mal [17:18:11] I'll take that as a "No" [17:18:37] Dispenser: WMF doesn't need one, according to MPEG-LA. [17:18:37] Doc_Taxon_: That sounds like a silly question, but are you certain you are doing your ssh from where you are expecting it? That's the kind of error I've seen before when someone is trying to ssh from home keys without realizing they are already ssh'ed somewhere else. :-) [17:19:25] valhallasw`cloud: [citation needed] [17:19:42] Dispenser: http://www.mpegla.com/main/programs/AVC/Pages/Agreement.aspx [17:20:01] Dispenser: you only need a license if you distribute h.264 video [17:20:09] or if you distribute software to en/decode h.264 video [17:20:11] WMF does neither [17:20:51] Dispenser: Take that however you wish. Again, and for the last time, bring any concern you may have to legal@wikimedia.org - they alone are in a position to determine what patent licenses may or may not be required, however they may or may not apply to the foundation. [17:21:30] I am not a patent lawyer, and neither are you. [17:22:23] what are home keys? [17:22:29] anyone here can approve an OAuth application please? It will expire tomorrow. https://www.mediawiki.org/w/index.php?title=Special:OAuthListConsumers/view/22282bcadc4a854a9d8bf5b350b8e4e9 [17:23:05] Dispenser: also, is this actual concern or just a passive-aggressive response to your tools not being allowed on labs? [17:24:22] gifti: I mean the ssh keys you use to log into labs and which are, presumably, on your home computer. :-) [17:24:43] anomie, Deskana|Away, Eloquence, robla I believe one of you may be able to approve the application for the wikimaps warper? https://www.mediawiki.org/w/index.php?title=Special:OAuthListConsumers/view/22282bcadc4a854a9d8bf5b350b8e4e9 [17:25:22] andrewbogott: I'm indeed seeing odd resolution failures. [17:25:37] andrewbogott: very intermittent. Still trying to find a pattern. [17:26:08] coren : i use privkey and publickey like before, no changes [17:26:40] and I trusted to fingerprint [17:28:30] andrewbogott: Hmmm. Okay, I found the pattern; I get empty answers with no error (not even NXDOMAIN) when I hit labs-ns0, but labs-ns1 works fine all the time. [17:28:39] Doc_Taxon_: What is your username? [17:28:49] taxonbot [17:30:02] Coren: ok, so labs-ns0 should be virt1000. But until recently it was virt0, can you verify that you're hitting virt1000? [17:30:40] Doc_Taxon_: Your account is okay, and your keys have not changed since Aug 17. Logs show your attempts, but that you are not presenting a key at all. Can you try your ssh with -v and dpaste the output, please? [17:31:29] I use putty [17:32:24] Doc_Taxon_: There isn't much I can to do help you much further then; I don't know how to get detailed logging out of putty. Double check that your config still points to the correct private key? Betacommand may be able to help you more, I know he uses Windows. [17:33:14] yes, it points to correct private key [17:33:32] the output was: [17:33:39] Doc_Taxon_: have you logged in before? [17:33:39] jetzt : server refused our key / no support authentication methods available (server sent: publickey,hostbased) [17:34:17] i logged in yesterday, yes [17:35:00] by 192.168.198.8:22 , not working today [17:35:02] Then I'm out of my water; from what I can tell everything on this side is okay, your account works and has two public keys available 'rsa-key-20140331' and 'taxonbot', and neither have changed or been touched since Aug 17 08:08 [17:35:43] 192.168.198.8 is not a Labs or WMF address. [17:36:04] Doc_Taxon_: have you installed/updated any programs since them? [17:36:05] but I took it everydays [17:36:07] *Then [17:36:33] Doc_Taxon_: That's an address on /your/ network. Did you have some sort of proxy in place? [17:36:53] valhallasw`cloud: It is unrelated to my banning. If you look I do have an interest in audio/video players + formats. [17:37:01] by 192.168.198.8:22 , it's tools-login.wmflabs.org:22 [17:37:42] i don't use proxies [17:37:48] Doc_Taxon_: What is your connection type? [17:37:58] or is putty a proxy? [17:38:03] No, 192.168.198.8 isn't - and has never been - tools-login. In fact, it's not even a routable address at all. [17:38:11] i connect by putty [17:38:35] why did it work till yesterday? [17:39:36] Doc_Taxon_: vpn? [17:39:41] Doc_Taxon_: odds are you changed something in your config [17:40:30] andrewbogott: I'm not sure I can tell for sure from the outside. All I can tell you is that it points to 208.80.154.19 and doesn't work. :-) [17:40:36] will tools-login.wmflabs.org:22 work again in time? [17:40:51] Doc_Taxon_: it still works [17:40:52] Coren: That's virt1000 [17:41:08] * Coren checks to make sure we don't have that IP on two boxes again. [17:41:20] I can't connect to bastion [17:42:01] andrewbogott: So that syslog message may be related after all. Ima start pdns up under strace. [17:42:12] is it because of DNS issues , tools-login.wmflabs.org:22 does not work [17:42:40] Doc_Taxon_: yeah, probably. [17:42:55] so it will work in future again? [17:43:11] Doc_Taxon_: should, possibly, yes. [17:43:21] okay [17:43:26] Doc_Taxon_: I suspect it will, since the IP you provided seems bogus, and that might be a DNS problem [17:43:51] Coren: so if you do dig tools-login.wmflabs.org labs-ns0.wikimedia.org you get an empty response? [17:43:54] what means "bogus"? [17:43:55] Coren: whats login's external IP? [17:44:02] andrewbogott: Yep. [17:44:14] andrewbogott: I get an empty one as well [17:44:16] NOERROR but empty [17:44:42] Is this empty too? Do I just not know how to read dig output? https://dpaste.de/2wQ0 [17:44:47] andrewbogott: Actually, every answer is empty regardless of the query [17:45:04] Coren: Getting a host does not exist error for login [17:45:08] andrewbogott: Yep. [17:45:23] me too [17:45:27] Betacommand: One of the two DNS is currently having issues, so you have a ~50% chance of resolution at any one time. [17:45:57] Betacommand: tools-login is 208.80.155.130 [17:46:36] Doc_Taxon_: I'm guessing you have a router that does you DNS which gives bogus answers when it can't find a good one. [17:47:05] narf [17:47:27] Coren: ok, I restarted pdns on virt1000 yet again. [17:47:36] Let's see if it falls apart again in a few minutes [17:47:52] Doc_Taxon_: put 208.80.155.130 as a replacement for tools-login.wmflabs.org [17:48:50] i tried many times [17:49:28] and the answer is: [17:49:40] server refused our key / no support authentication methods available (server sent: publickey,hostbased) [17:50:10] but the keys are okay [17:51:22] Doc_Taxon_: You have the tools-login settings saved in putty? [17:51:32] yes [17:51:36] all the same [17:51:47] what do you mean all the same? [17:52:14] I have the same problems, no putty, so I don't think it's putty [17:52:32] the settings are both the same, for tools-login and 208.80.155.130 [17:53:06] gifti2 : and connectbot? [17:53:18] yes [17:53:38] me too [17:54:13] WinSCP doesn't work too [17:54:21] I think it's our provider's dns [17:55:03] es wird aber wieder gehen demnächst [17:55:19] hopefully [17:55:31] i hope too [18:00:54] Our DNS is fixed; so except for bogus answers that may have been cached on your side, there should be no issues. [18:01:16] what is bogus? [18:01:42] Doc_Taxon_: "Bogus" means bad, as in incorrect and random. [18:02:13] ya, it works!!! XD [18:02:13] Like, "192.168.198.8" is a bogus answer to a query about tools-login.wmflabs.org. :-) [18:02:25] valhallasw`cloud, Coren: "MPEG LA’s AVC License Will Not Charge Royalties for Internet Video that is Free to End Users through Life of License" Does that mean we could build an H.264 video player on Labs? Would it still fall under the "Free Content" if the videos can't be sold? [18:02:45] #foodforthought [18:03:08] Dispenser: No. #foodforpatentlawyers [18:03:18] :-) [18:04:42] andrewbogott all in order now, Thank you! [18:05:09] yes, but tools-login only, not by IP [18:06:02] nuria: please let me know if it drops out again, I'm not positive that this is stable yet [18:06:10] will do [18:06:16] okay [18:06:33] Doc_Taxon_: There is something odd going on in the way things are set up on your end which I don't understand I'm afraif. [18:07:09] Coren: back in ~an hour. [18:07:15] kk [18:07:27] no problems, tools-login.wmflabs.org:22 works [18:16:12] Dispenser: that sounds a bit like the personal rights/trademark issue -- we have photos of people and logos on commons. [18:16:47] Dispenser: but you might be right that it can be done for free as long as there are no ads [18:16:57] Dispenser: otherwise there's the $25000 per encoder price tag [18:43:06] hmm, still can't log in to labs ;( [19:30:45] JetLaggedPanda: tell me about 'still can't log in' [19:31:41] andrewbogott: dig tools-login.wmflabs.org gives me an empty response [19:31:47] andrewbogott: status: SERVFAIL [19:32:03] https://gist.github.com/anonymous/9d3fa8eb3cb0010ab76f [19:32:30] JetLaggedPanda: what about if you use labs-ns0.wikimedia.org or labs-ns1.wikimedia.org? [19:32:41] andrewbogott: hmm, how do I do that? [19:32:44] * JetLaggedPanda does man dig [19:33:16] just the 2nd arg [19:33:53] andrewbogott: hmm, gives me two answers [19:34:07] andrewbogott: bah, nevermind, it actually just resolved labs-ns1 as well [19:34:18] andrewbogott: and now dig works fine [19:34:19] wut [19:34:40] probably cache [19:34:42] * andrewbogott puts on shades, dusts off hands [19:35:51] andrewbogott: did you manage to rescue my instances, btw? [19:36:34] JetLaggedPanda: some of them! [19:36:36] * andrewbogott checks [19:43:31] Error: /Stage[main]/Role::Labs::Instance/Mount[/home]: Could not evaluate: Execution of '/bin/mount /home' returned 32: mount.nfs: mounting labstore.svc.eqiad.wmnet:/project/icinga/home failed, reason given by server: [19:43:31] No such file or directory [19:43:38] andrewbogott: ^ I'm trying to figure out why this happens, any clues? [19:43:52] nfs isn't mounted [19:43:53] JetLaggedPanda: you missed one, 'verpverpverp' [19:43:57] Can I kill it? [19:44:01] andrewbogott: oh, that's tonythomas's [19:44:02] nto mine [19:44:05] ok... [19:44:09] well, I created it, but that's pretty much all I did [19:44:21] JetLaggedPanda: are you getting that error on a new instance? There's a race between instance creation and volume creation [19:44:30] andrewbogott: it's a new instance, yeah [19:44:34] andrewbogott: but it's not a new project... [19:44:43] "Have you tried turning it off and back on again?" [19:44:49] :D [19:44:49] no [19:44:53] I shall [19:44:59] I don't particularly care, tho :D [19:45:04] * tonythomas almost got killed :D [19:45:35] tonythomas: verpverpverp is in a nasty puppet state and I'd like to know if it's important before I spend more time trying to rescue it. [19:46:07] andrewbogott: its in a nasty state, but quite serving its need as a mail server though [19:46:20] tonythomas: please subscribe to labs-l [19:46:48] I had to manually include the mail::role::mx to get the mx done - probably puppet never liked it [19:47:01] andrewbogott: ok. something fishy going on ? [19:47:10] what's your email? [19:47:26] * JetLaggedPanda continues to try out emacs [19:47:26] 01tonythomas@gmail.com [19:49:22] andrewbogott: oh. ok. Will apply those rightaway [19:49:27] thanks [19:49:36] :) [19:49:48] !paste [19:49:48] http://tools.wmflabs.org/paste/ [19:49:48] you can verify by checking /etc/ldap.conf and see if the ldap servers are virt1000,virt0 or ldap-eqiad,ldap-codfw [19:50:07] JetLaggedPanda: Jet lag is for pussies! :P [19:50:16] multichill: :D [19:50:32] Jet lag is a hangover without having the fun of a night out [19:50:39] anyone have suggestions for optimizing https://tools.wmflabs.org/paste/view/ea2bc358 [19:55:54] Betacommand: page namespaces are integers, not strings [19:56:40] valhallasw`cloud: thanks, copy/pasted it from php and was adapting it. Missed that [19:57:41] I'm also a bit confused about the first left join -- isn't that one superfluous? [19:58:34] oh, that checks if the page (pl_namespace, pl_title) exists [19:58:35] I see [19:58:44] valhallasw`cloud: its code from Special:WantedPages my query is kinda old [19:59:07] otherwise, EXPLAIN it and see if that helps [19:59:43] *maybe* re-ordering helps (i.e. page_to LEFT JOIN pagelinks LEFT JOIN page_from instead) [19:59:55] as the page table is significantly shorter than pagelinks [20:00:01] valhallasw`cloud: Im trying to help with http://en.wikipedia.org/wiki/Wikipedia%3AVillage_pump_%28technical%29#Unique_red_links [20:01:02] you could also leave out the second join altogether; that does mean you also get non-NS0 pages with just a single link but that might not be a huge issue [20:01:23] although that join should be pretty trivial [20:01:58] Betacommand: You might want to move some stuff to join conditions instead in the where part. Excluding redirects speeds crap up. pagelinks has pl_from_namespace now too, you might to want to use that too [20:02:19] Betacommand: good practive is to prefer using primary key on the is null test (page_id is this case). secondly for "left join pg2 " combining with "pg2.page_namespace = '0'" you could also use inner join which is faster [20:03:14] and an "order by null" wil speed up the query, too [20:04:12] Merlissimo: can you pastebin a copy with those changes so I can see what your talking about? [20:07:27] Betacommand: https://tools.wmflabs.org/paste/view/ab93aa47 [20:10:29] Merlissimo: thanks [20:11:53] Betacommand: forgot the change to "page_id is null" but i don't know if this really help. i only know that this is good practise [20:13:20] andrewbogott: verpververp made ok. :) but had to stash all my changes though [20:13:20] !log deployment-prep updated OCG to version 48c495e3656f528abe636ce0cd7562270505534f [20:13:25] Logged the message, Master [20:14:00] tonythomas: thanks! [20:14:23] np :) [20:32:15] Corren: is it possible to request additional key for replications tables at labs which are not created by mediawiki by default? [20:33:09] andrewbogott, question about emails in labs [20:33:11] made otrs-test.eqiad.wmflabs [20:33:13] can send email [20:33:15] What to do about receiving email? [20:33:17] OTRS does allow you to give it the details of a POP3/IMAP account to pull from. But I'd prefer it to have a @otrs-test.wmflabs.org address rather than @gmail.com [20:33:55] Krenair: I'm not sure. I don't think that receiving mail @wmflabs is currently supported. [20:34:03] JetLaggedPanda: does that sound right? [20:44:06] andrewbogott: Krenair I think it can be made to work in some way. tool labs accounts can receive mail [20:44:36] Coren might know? [20:44:50] he might! [20:45:33] Is he around at the moment? [20:45:43] Krenair: valhallasw`cloud might know as well [20:45:50] wikibugs works by receiving email [20:47:45] Krenair: yes, it's possible [20:48:07] Krenair: anything sent to abc+toolname@tools.wmflabs.org is received and forwarded to the tool authors by default [20:48:13] cool [20:48:16] Can I set that up for my project? [20:48:47] Ah. That, I do not know. There were some demands from legal (mail couldn't be stored on disk, for instance) [20:49:08] Oh. [20:49:15] Well the point of this is that it stores mail on disk. [20:49:35] beta login http://en.wikipedia.beta.wmflabs.org/wiki/Special:UserLogin is 503 right now. known? [20:50:16] people have been complaining about things like that in prod [20:50:52] Krenair: might have been a tools-only thing, not sure. [20:51:26] I'm about to send an email about this project, I believe some LCA people are on the list. Let's see what they say [20:54:48] 3Wikimedia Labs / 3deployment-prep (beta): login busted - 10https://bugzilla.wikimedia.org/71532 (10Chris McMahon) 3NEW p:3Unprio s:3critic a:3None http://en.wikipedia.beta.wmflabs.org/wiki/Special:UserLogin is 503 [20:57:00] 3Wikimedia Labs / 3deployment-prep (beta): login busted - 10https://bugzilla.wikimedia.org/71532#c1 (10Erik Bernhardson) Currently all pages on beta are down, the hhvm instances are in a restart loop. It might be related to https://gerrit.wikimedia.org/r/#/c/164160/ , still investigating. [21:19:02] Krenair: Mail to and from labs in general (as opposed to tools) is not currently supported. By an amusing coincidence, Mark has just asked me to do extactly that (mail relay for labs) today. :-) [21:19:18] Oh, ok. [21:19:25] So will this be able to support my use case? [21:21:44] It will all be handled centrally? Or each project handling mail will need its own setup? [21:43:56] chrismcmahon: beta labs is down [21:45:26] kaldari|2: poke ori, we think it broke from https://gerrit.wikimedia.org/r/#/c/164160/ [21:45:30] kaldari|2: yep, I replied on the mail list [21:47:02] chrismcmahon: thanks [21:47:04] Coren? [21:49:01] ori: every time you break beta labs a kitten dies [21:56:08] kaldari|2: https://commons.wikimedia.org/wiki/File:Server-kitty.jpg [22:03:03] kaldari|2, ebernhardson: fixed. is it too late to revive the kitty? [22:19:38] ori: not too late, thanks! [22:26:15] 3Wikimedia Labs / 3tools: deletion queries joined with tokudb replication tables are really slow - 10https://bugzilla.wikimedia.org/68918#c6 (10merl) Today i tried to run a script (missing articles) which has a lot of those deletion queries with many rows affected. It was not possible the run this script on... [23:30:20] 3Tool Labs tools / 3Erwin's tools: shared.css gives a 404 breaking layout. - 10https://bugzilla.wikimedia.org/71482#c1 (10Krinkle) https://bits.wikimedia.org/static-current/resources/src/mediawiki.legacy/shared.css [23:35:00] 3Tool Labs tools / 3Erwin's tools: Erwin's tools: Broken layour due to shared.css 404 Not Found - 10https://bugzilla.wikimedia.org/71482#c2 (10Krinkle) Loading raw css files hasn't been the standard practice since 2010. Instead of requesting the raw file directly - which is unsupported (MediaWiki can change... [23:44:26] Jackmcbarn you around? [23:44:33] Betacommand: yeah [23:44:44] jackmcbarn: Im already running that red link count [23:44:54] what query are you using? [23:45:17] a variant of the Special:WantedPages query [23:46:47] i thought you weren't allowed to edit on enwiki though [23:47:39] jackmcbarn: I may not be able to edit, doesnt mean I cant run db queries and create tools [23:47:45] jackmcbarn: results will be at http://tools.wmflabs.org/betacommand-dev/reports/single_red_links.txt [23:48:32] should take somewhere around an hour and a half to finish [23:51:22] jackmcbarn: if you need anything/have tool requests let me know, odds are I already have most of the code for it [23:51:34] kk [23:52:57]