[01:17:50] JFTR: I'm seeing intermittent connect failures to the replicas, "Could not connect to s1: Lost connection to MySQL server at 'reading authorization packet', system error: 0". Different replicas, not constant nor any other pattern discernible. [01:39:13] Wow, never noticed before: labsdb1001 has ~ 190 GByte of memory. That's more than twice my hard disk's capacity until a month ago :-). [04:23:14] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/Migration of Toolserver tools was modified, changed by Tim.landscheidt link https://www.mediawiki.org/w/index.php?diff=891793 edit summary: /* Can I import my JIRA bug tracker information to Bugzilla? If so, how? */ Refer to valhallasw's awesomeness. [06:59:48] Anyone around who has a login on tools-exec-02? I would've thought I did, but... [07:22:37] @trusted [07:22:37] I trust: .*@wikimedia/.* (2trusted), .*@mediawiki/.* (2trusted), .*@wikimedia/Ryan-lane (2admin), .*@wikipedia/.* (2trusted), .*@nightshade.toolserver.org (2trusted), .*@wikimedia/Krinkle (2admin), .*@[Ww]ikimedia/.* (2trusted), .*@wikipedia/Cyberpower678 (2admin), .*@wirenat2\.strw\.leidenuniv\.nl (2trusted), .*@unaffiliated/valhallasw (2trusted), .*@mediawiki/yuvipanda (2admin), .*@wikipedia/Coren (2admin), [12:47:27] Coren: do we usually put SSL cert keys into labs/private? [12:47:36] for testing that is [14:55:24] andrewbogott_afk: exec nodes use HBA. You can ssh there from tools-login or tools-dev. [14:55:41] andrewbogott_afk: Although the root key should work from anywhere. [14:57:19] Hi. So, I am added to the members of bastion project, and have the necessary configuration in my ~/.ssh/config -- but, ssh -A SuchetaG@bastion.wmflabs.org, is returning Permission denied anyway. [15:03:12] ireas O_O [15:03:23] Steinsplitter, ? [15:03:26] sucheta: Usernames are case significant. Makesure that this is really the capitalization you need (hint: I'm pretty sure it isn't) :-) [15:03:56] * Steinsplitter wollte nur hallo sagen :-X [15:04:27] hallo Steinsplitter :) I feared that I broke sth :D [15:04:37] :P [15:17:19] !log deployment-prep applying role::beta::fatalmonitor puppet class on deployment-bastion {{bug|60046}} [15:17:21] Logged the message, Master [15:40:21] Coren: around? :) [15:40:34] Coren, But it doesn't work with ssh -A suchetag@bastion.wmflabs.org either. [15:40:35] Sure, what's up? [15:40:43] could we have 1 public IP for scrumbugz please? :) [15:41:04] My shellname is User:SuchetaG [15:41:04] addshore: That's the project name? Sure, gimme a min. [15:41:13] that is indeed the project name :) [15:41:33] addshore: does it need non web access? :P [15:41:34] sucheta: That's not your shell name, that's your wikitech user name; check in your preferences, you should be able to see 'shell account' in there. [15:41:44] YuviPanda: indeed, it needs non web access [15:41:48] ah ok :) [15:42:50] looking at https://wikitech.wikimedia.org/wiki/Special:NovaProxy it might be possible through proxies but right now public ip address is ideal :) [15:45:02] addshore: proxies does http and https [15:45:10] addshore: quota++ [15:45:14] okay, yep defo need public ip then :) [15:45:15] addshore: if you need anything else, then yeah, you need a public IP [15:45:16] cheers! [15:45:20] addshore: :) [15:52:47] https://tools.wmflabs.org/?status is beginning to be downright impressive. [15:53:27] Coren: woah, looks nicer than it did when I last looked at it :) [15:53:59] Cosmetics? Brandon did a revamp some time ago, for great JUSTICE! [15:54:40] Coren: of course! nicely done :) [15:54:46] wow, that does look nice! [15:58:24] yeah [15:58:30] addshore: now if only it were live updating... :D [15:58:35] :> [15:59:09] I havnt done anything with labs in months now! [15:59:21] addshore: me neither [15:59:32] I'll probably have more time when the app comes out... [17:07:09] Coren: on toolserver there are -user and -rr databases. Some years ago when -rr was introduced on toolserver, it was much work to check each script of my script if it needs user databases or not. If i am changing everything to labsdb this info is lost forever. would i make sence to create aliases like s5-rr.labsdb to preserve this info for possible future changes on toollabs, wouldn't it? [17:09:19] Merlissimo: Hm. That's an interesting question. Lemme think on it for a while. [17:59:31] +1 to Merlissimo's idea. Aliases are cheap and low maintenaince. [18:00:28] +1 ^^ [18:00:30] Coren: Did andrewbogott_afk talk with you about (apparently) the NFS load that causes network drops to labsdb*? (That's what I gather from the backscroll of -operations.) [18:01:13] It's not nfs load per say, it's just crazy network load caused by a broken tool. [18:02:37] And culprit identified? [18:03:36] And beaten up with prejudice. [18:03:57] Perfect. [19:28:01] Coren, I need your service. [19:28:33] https://tools.wmflabs.org/xtools/pcount constantly shows No input file specified. [19:29:36] I can't seem to find an issue with the code primarily for debugging purposes, I placed as the first executed statement "die( "Down for mainanence" ); [19:30:01] Coren, what could be causing it? [19:31:01] Cyberpower678: Sorry dude, but I'm leaving for FOSDEM in a couple days, then the facebook thingy so I'm swamped with infrastructure and the migration for the rest of the week and will have very little time to do individual tool debugging. I don't know how pcount works so making hypotheses about what could be wrong would be... unproductive. :-) [19:32:06] Coren, I'm thinking it might be an issue with labs and not the tool since the tool just now broke and it hasn't been touched in months. [19:34:20] Coren, I was wondering if you could take a look with the webservers. [19:34:42] Coren: enjoy FOSDEM! :) [19:34:44] Cyberpower678: Is any other tool showing issues? [19:35:17] No. [19:35:30] YuviPanda, can you possibly help me? [19:35:39] not right now, Cyberpower678. sorry [19:36:00] I placed at the top of the pcount script a die( message ); and that's not even showing. [19:36:03] Cyberpower678: And there is no help at all in the error log? [19:37:10] Coren, these are the last errors recorded in the log. [19:37:12] 2014-01-27 19:27:54: (configfile.c.853) source: /var/run/lighttpd/xtools.conf line: 559 pos: 1 invalid character in variable name [19:37:12] 2014-01-27 19:27:54: (configfile.c.909) configfile parser failed at: (EOL) [19:38:18] Cyberpower678: Seems clear that you've broken something in your lighttpd.conf (it's a bit annoying that the line number refers to the /merged/ file though); check what you most recently changed in there. [19:38:36] Coren, I haven't changed anything. [19:38:55] I have even logged in to xtools for the last several weeks now. [19:39:06] *haven't [19:39:20] But the break occured today. [19:39:22] Cyberpower678: Line 559 is the start of ~/.lighttpd.conf, and that starts with . [19:39:32] Did you restart the webservice today? [19:39:39] Just now yes. [19:39:45] Cyberpower678: Another BOM! [19:39:54] I haven't touched it. [19:39:58] So you changed lighttpd.conf in the mean time, but lighttpd only noticed it at the restart. [19:39:58] I SWEAR. [19:40:10] Cyberpower678: no, but you maybe touched it two weeks ago, without restarting lighttpd? [19:40:37] I haven't touched it all IIRC [19:40:53] Coren: We should remove leading BOMs in ~/.lighttpd.conf because they are legal there, but not mid-file? Should I do that in ... whatever the script's called? [19:41:12] scfc_de: BOM marks aren't legal in UTF-8 period. [19:41:25] Cyberpower678: U+FEFF is perfectly legal in UTF-8 [19:41:28] er, Coren [19:41:41] it just isn't a BOM in the traditional sense [19:41:48] valhallasw, not possible I restarted after the break occured. [19:42:00] valhallasw: It's /specifically/ a forbidden sequence. It is not a valid code point at all. :-) [19:43:08] Coren: No. U+FEFF is just 'ZERO WIDTH NO-BREAK SPACE', and it's perfectly legal: http://www.unicode.org/faq/utf_bom.html#bom5 [19:43:15] * anomie still wonders why lighttpd cats the user conf file instead of using "include" [19:44:07] valhallasw: Oh, FFS! Mark caved in. [19:44:49] anomie: Wouldn't change anything U+FEFF would (correctly) throw an error anywhere outside a string literal. [19:44:49] Mark? [19:45:03] see also http://www.unicode.org/versions/Unicode5.0.0/ch02.pdf @ page 36 [19:45:13] Coren: It would fix the line number in the error message though, wouldn't it? [19:45:22] Coren, so how do I fix this? [19:45:38] YuviPanda: Last I checked, he was still fighting against that broken compatibility idiocy at the Consortium. [19:45:43] Cyberpower678: Remove the BOM, of course [19:45:49] where? [19:45:52] ah! [19:45:57] Cyberpower678: in ~/.lighttpd.conf [19:46:19] If his editor put it there, I doubt that he can see it :-). [19:46:22] * Cyberpower678 doesn't recognize that file, so he couldn't have modified it. [19:46:43] Cyberpower678: Remove all BOM from all your files and fix your fscking text editor for crying out loud. It's like the 50th time you break something by editing it. [19:47:53] Coren, I'm not even using that editor anymore. [19:48:12] Coren: Okay, so party line is: BOM on Tools = no, no, no? [19:48:24] Coren, and I never edited that file. [19:48:35] scfc_de: Well, unless you are using UCS-2 for some strange reason. :-) [19:48:38] scfc_de: BOM in UTF-8 is 'please, for the love of god, don't, because you break a lot of stuff' [19:48:55] typical example is PHP's 'output already started' because there is a BOM before the Cyberpower678: I promise there is no process that runs around your tools and edits its files to insert random BOMs at the beginning. :-) [19:49:50] nice idea though, we should do that [19:49:53] helps build character [19:50:02] valhallasw: I know, and if it's only Cyberpower678, I think that's manageable :-). [19:50:04] also helps build code points :P [19:50:11] YuviPanda: Strictly specing, it helps build codepoint. :-) [19:50:18] So I edited the file and removed the BOMs with N++. [19:50:26] Coren: that was the end of my joke :P [19:50:30] Restarted the webserver, and same results. [19:50:31] i said it right before you did! [19:50:39] Cyberpower678: Are you the only one maintaining xtools? What about tparis? [19:51:00] YuviPanda: No fair time traveling to retroactively ninja my punchlines. [19:51:01] scfc_de, he's not active atm afaik. [19:51:05] Coren: :D [19:51:19] * Coren goes back to work. [19:51:38] Cyberpower678: "Down for maintenance!" => Works for me. [19:52:05] Now it's working. :D [19:52:18] Coren, scfc_de, YuviPanda: Thanks. [19:52:44] Coren, maybe we should have a shell script 'unbom' that does sed -e '1 s/^\xef\xbb\xbf//' -i ? [19:52:47] * Coren suggests that Cyberpower rgreps for BOMs [19:53:29] valhallasw: we should call it 'unabomb' [19:54:05] Coren, I have better idea. I'm just going to toss them into N++ and have it UNBOM all of them. [19:54:40] * hedonil is amused [19:54:44] !BOM [19:54:45] Did Cyberpower678 use his crappy editor again? [19:54:54] !BOManswer [19:54:54] of course he did it again! [19:55:08] * Coren should probably buy the new version of the Unicode Standard, he's two editions behind now. [19:55:11] !info BOM [19:55:12] https://www.mediawiki.org/wiki/WMF_Projects/Wikimedia_Labs [19:55:16] @info BOM [19:55:25] How did that go again? [19:55:31] @keyinfo BOM [19:55:38] !keyinfo BOM [19:55:49] :/ [19:55:58] @help [19:55:58] I am running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.20.2.1 my source code is licensed under GPL and located at https://github.com/benapetr/wikimedia-bot I will be very happy if you fix my bugs or implement new features [19:56:11] Cyberpower678: it goes like this.. [19:56:14] Cyberpower678: what are you looking for? [19:56:23] !somethingstorelax [19:56:25] Look up key info. [19:56:36] !somethingtorelax [19:56:36] http://www.flickr.com/photos/110698835@N04/ [19:57:17] @infobot-detail BOM [19:57:17] Info for BOM: this key was created at 12/27/2013 7:01:08 PM by Cyberpower678, this key was displayed 2 time(s), last time at 1/27/2014 7:54:45 PM (00:02:32.2191820 ago) this key is normal [19:57:33] @infobot-detail BOManswer [19:57:33] Info for BOManswer: this key was created at 12/27/2013 7:02:16 PM by hedonil, this key was displayed 2 time(s), last time at 1/27/2014 7:54:54 PM (00:02:38.9949770 ago) this key is normal [19:57:44] I made the BOM key? :O [19:58:31] Cyberpower678: admit it, you just use BOM as running gag ;) [19:59:02] hedonil, you got me. I use the BOM thing as a gag to drive Coren nuts with. :p [19:59:15] hehe [19:59:18] find . -type f -exec awk 'FNR==1 && /^\xef\xbb\xbf/ { print FILENAME; }; FNR>1 {nextfile}' {} + [19:59:21] the power of google [19:59:52] Google is hard [19:59:59] not terrible fast, but it should find all bombed files [20:00:41] then just | while read $i; do sed -e '1 s/^\xef\xbb\xbf//' -i $i; done [20:01:21] although awk could probably also do that for you [20:02:06] Cyberpower678: but Coren ist a steeled community guy - and former ARBCOM, he surely wears sunglases at night 8-) [20:02:57] Coren, I can imagine Coren blocking me with the reason being, "Pissed of a former Arb with his sheer bot incompetence." :p [20:03:04] err hedonil ^ [20:03:31] hehehe [20:05:18] but dont't mees around w/ BOM's - take a look at v4 Wiki ViewStats instead - hastened with redis now [20:05:22] https://tools.wmflabs.org/wikiviewstats/ [20:09:46] hedonil, I don't intentionally try to mess with BOMs. It just kind of happens. :p [20:10:06] Cyberpower678: you are a good guy ;) [20:10:38] :D [20:11:09] Cyberpower678: bringing some atmosphere in this (sometimes) boring IRC [20:11:53] Hey I have this issue. This Script isn't working. [20:12:01] BOM! [20:12:04] :D [20:14:02] * Cyberpower678 resumes his class attendance. [20:14:46] Coren btw. what caused this weekend's network outages? replica connections timed out in masses. counter reached new records [20:14:52] so i was trying to do a gunzip -c file | perl perlscript.pl but it ofc wouldn't work over jsub, any tips? [20:15:36] to enclose in a bash -c "c1 | c2" doesn't seem to work either [20:15:44] hedonil: As far as I can tell, it was a couple stray tools that completely gummed up the network. I'm still a bit surprised at it, but this is Sean's best bet. Said stray tools have been impounded though. [20:15:53] (hi btw) [20:24:03] fluff: Your best bet is to use a wrapper script, i. e. put "c1 | c2" in a script, and then call that via jsub. The underlying SGE mangles command line arguments, so "fancy" ones get usually beaten to death :-). [20:25:05] scfc_de: thanks [20:27:07] Coren: hiiii [20:27:23] i'm setting up new wikimetrics staging instance in labs [20:27:25] that needs to talk to labsdb [20:27:42] i've got everything in place, are there firewall rules that need to be set in order for my new instance to connect o labsdb? [20:28:30] ottomata: No, but you probably want the happy fun helpful hosts and iptables files. :-) [20:28:52] ottomata: (Otherwise you have to hardcode host/port info per shard in your app, and that is !wise) [20:29:01] oooooof ok, um, iptables files eh? [20:29:17] ottomata: You can simply crib the tools' version in /data/project/.system/{hosts,iptables.conf} [20:29:50] ok i think i can grab it from wikimetrics instance [20:29:50] beacuse it already works [20:30:07] ah ha [20:30:08] found it [20:39:26] Coren, should there be a puppet module to enable labs db? [20:39:32] so that someone who wanted this could just check the box? [20:39:37] and get /etc/hosts and get iptables rules? [20:39:55] +1 ottomata [20:39:57] should be! [20:40:02] ottomata: That actually makes sense. Stop making sense, we're at the WMF! :-) [20:41:02] none of that is private info, right? [20:41:26] its just a host mapping and iptables rules [20:42:58] ottomata: only question is username/password [20:43:02] but I guess that can be obtained manually [20:43:14] yeah [20:56:18] ottomata: https://gerrit.wikimedia.org/r/#/c/107010/2 [20:56:58] (Untested, just a brain dump.) [21:14:48] ah scfc_de awesome [21:15:34] ottomata: Well, it's not working at all (as told by Jenkins' vote :-)). [21:18:15] looks like a simple syntax error [21:18:21] Syntax error at '}'; expected ']' at /srv/ssd/jenkins-slave/workspace/operations-puppet-validate/manifests/role/db.pp:624 [21:21:32] Be careful, though, the default ferm policy is DROP, so if you enable ferm, you need to adjust that (I think). [21:23:40] oo [21:23:40] ok [21:27:04] For some reason my cronjob on tool labs stopped running on December 11, 2013 and hasn't run since. I can still manually run the script, but I can't get the crobjob to run. [21:27:10] I currently have it set up as: [21:27:13] 45 7 * * * jsub -N hotarticlesbot -mem 350m php /data/project/hotarticles/public_html/runbot.php [21:27:58] Coren: ^ [21:28:56] I see nothing wrong with it. What do you see in your hotarticlesbot.err ? [21:34:38] Coren: libgcc_s.so.1 must be installed for pthread_cancel to work [21:35:06] kaldari: It's in the FAQ. You don't have enough -mem :-) [21:36:28] Coren: What is a good level for executing PHP now? 500? [21:37:07] guess I'll just play around with it and see [21:37:10] It depends a lot on what you are doing; 500 has elbow room aplenty. [21:37:33] Coren: Thanks! [21:50:03] I'm trying to execute wikimedia-bot. In monodevelop, this works fine, but how can I compile it on the command line? "mcs Program.cs" gives lots of errors à la "The name `Syslog' does not exist in the current context". [21:57:30] Ah, there's "xbuild" that reads the *.csproj and does its thing. [21:59:26] And afterwards "mono bin/Debug/wmib.exe". [23:30:06] Could the 'page_links_updated' field of the page table be exposed in the labs dbs? [23:34:44] Considering it's just a timestamp [23:34:54] bawolff: Probably; as Coren is packing for FOSDEM, could you file a bug so it is not forgotten? [23:35:03] ok, will do [23:37:38] bawolff: Is that a new column? I don't recall seeing it before. [23:38:03] Coren: yes (relatively) [23:38:19] Its last time refreshlinksjob was ran on the page [23:38:38] I think it was added about a month ago [23:39:14] Filed as bug 60506 [23:39:39] bawolff: Bad timing simply then, the page table isn't redacted so the column will appear next time I run a maintain-replicas (~ every 2 months, I don't do it too often because it locks ALL THE THINGS) [23:40:27] I'll try to squeeze it in before I fly out. [23:41:28] Coren: cool thanks. I'll note I'm not in a rush, so if it would cause a lot of problems, and will happen anyways, I can wait [23:42:04] I need to do it anyways; I think the last time I ran it was in late Nov. anyways so I'm due.