[00:00:14] a930913: the law of halfs logging I find very useful [00:00:35] Betacommand: Isn't that a binary log? :p [00:00:42] If you have 100 lines, save the var in question at lines 1,50,100 [00:01:10] If the var is good at 1 and bad at 50, add a log at line 25 [00:01:27] and repeat until you know where the problem is [00:01:53] Betacommand: Divide and conquer -> log(n) [00:02:59] It may be crude, but it works [00:03:41] Ive got a function that I use that just outputs the current line number [00:03:59] handy for tracking code flow and finding where something stops [01:31:14] Howicus over at #wikipedia-en is asking what the URL for reflinks is. Does anybody know? [01:32:26] There isn't one yet and maybe never, depend on WMF support [01:33:43] You'd think with all the attention it would a WMF pirority [01:37:39] Dispenser: People are saying it's down or something. [01:37:51] http://lists.wikimedia.org/pipermail/toolserver-l/2014-March/006526.html Maintenance: Solaris Updates [01:39:21] Dispenser: https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Reflinks [02:07:29] Coren: why the heck have usernames changed?! [02:07:56] aka why does the migration have to be more difficult than strictly necessary [02:08:01] valhallasw`cloud: You mean tool account usernames? [02:08:09] yes [02:09:25] for tsreports, I use the username to distinguish between production and -dev [02:11:05] valhallasw`cloud: http://permalink.gmane.org/gmane.org.wikimedia.labs/1652 ("Impending change of DOOM!!1!one!") [02:12:03] I'm sorry, *what*? That's an email from september last year [02:12:04] (I don't agree with all assumptions there, but I don't have to make the infrastructure work :-).) [02:13:32] well, I guess there is one advantage to this... [02:13:45] Well, it's the first item at https://wikitech.wikimedia.org/wiki/Tool_Labs/Migration_to_eqiad. [02:13:56] valhallasw`cloud: What advantage? [02:14:36] !log will now sensibly work for tools projects [02:14:36] will is not a valid project. [02:14:50] i.e. !log tools.pywikibot instead of !log local-pywikibot [02:15:57] !log tools.pywikibot Project had been migrated; testing nightly generation. [02:15:57] tools.pywikibot is not a valid project. [02:15:59] oh. [02:16:02] well, that sucks. [02:16:15] Uh, I think we need to tweak morebots for that?! But we need to think about where to put tools information on the wiki. Some tools use {{Nova Resource}} which makes them show up as full-sized projects in SMW queries. [02:16:29] local-etc does work [02:17:29] !log local-pywikibot Project had been migrated to eqiad. Nightly generation is broken due to permission errors. Re-cloning git repos. [02:17:31] Logged the message, Master [02:18:47] Hmmm. Doesn't show up at https://wikitech.wikimedia.org/wiki/Special:RecentChanges [02:19:39] https://wikitech.wikimedia.org/w/index.php?title=Special:RecentChanges&hidebots=0 ;-) [02:20:32] Ah, I saw so many "Hide x", "Hide y", that I missed that one of them was "Show". [02:22:27] One of the "offenders" is https://wikitech.wikimedia.org/wiki/Nova_Resource:Xtools which disguises itself as a project. Would be nice to have a similar layout with "tool" instead of "project" and "Maintainers" instead of "Admins" and "Members". [02:25:37] !log local-pywikibot compat re-cloned and re git gc --aggressive -ed (also subrepos). Nightlies are now working, and started webservice is available via http://tools-eqiad.wmflabs.org/pywikibot/ [02:25:39] Logged the message, Master [02:26:14] now I have a problem. Some people are apparently using /data/project/pywikibot/public_html directly [02:26:28] so that directory must stay on both old and new servers [02:26:36] oh, I know what to do [02:27:31] Does pywikibot have a web presence? I think Coren uses ~/public_html's presence to determine where to proxy to. [02:27:40] yeah, for the nightlies [02:27:57] but I can't kill the tools-login-old one becasue people might depend on the ~/public_html existing [02:29:29] !log local-pywikibot crontab re-activated on old tools-login to make sure nightlies, but same permissions error there. Trying a simple gc --aggressive first before re-cloning. [02:29:30] Logged the message, Master [02:29:51] Can you upload the nightlies to both pmtpa and eqiad? Ah, okay, you're way ahead :-). [02:30:41] !log local-pywikibot pywikibugs bot online in eqiad now [02:30:43] Logged the message, Master [02:30:51] Otherwise I would just nag Coren to add an excemption for pywikibot. [02:34:34] Goedenacht! [02:34:43] Slaap lekker! [02:36:50] !log local-pywikibot nightlies unbroken on ptmpa; symlinks in /shared/pywikibot were already functional in eqiad. Migration complete! (hopefully...) [02:36:51] Logged the message, Master [02:41:39] Coren: you around? [08:17:33] hi! during my first login on tools-login.wmflabs.org my home folder (/home/rotpunkt) has been created with wrong owner/group (root.root instead of rotpunkt.wikidev). Can someone fix (chown) it? [13:33:48] Coren, thanks. [13:35:40] Coren, I assume you were able to fix the DB connection issue. My bot stopped throwing those error messages. [13:52:57] rotpunkt: Use tools-login-eqiad. [13:53:17] rotpunkt: The support for new users or tools on tools-login is broken because of the migration. [13:54:20] oh thanks, I try just now [13:56:44] Coren, how do I turn on error logging again? [13:56:55] @Coren I am logged in, with a non-empty .bash_rc, but I have no replica.my.cnf [13:57:13] rotpunkt: It takes a few minutes for it to appear [13:57:27] nice! thx [13:57:56] ok I have seen it now, thx! [13:57:57] Cyberpower678: debug.log-request-handling = "enable" [14:01:43] thx [14:03:23] Coren, I'm not getting any errors. But http://tools.wmflabs.org/peachy/wiki/api.php isn't working ever since I migrated and dumped the dump into the new DB. [15:22:01] Coren, I know this is low priority, but is there an eta for eqiad ganglia? [15:29:29] Coren: are you around? [15:30:01] Only partially. I /do/ need days off once every so often. [15:31:03] Coren: I got a really odd email [15:31:09] This message was created automatically by mail delivery software. [15:31:10] A message that you sent has not yet been delivered to one or more of its [15:31:12] recipients after more than 24 hours on the queue on relay.tools.wmflabs.org. [15:31:45] Betacommand: The (local) mail queue was clogged, I just unclogged it. [15:31:58] Coren: thanks [15:34:10] Coren, I know this is low priority, but is there an eta for eqiad ganglia? Or does it already exist [15:34:29] Cyberpower678: You are correct, this is low priority. Not eta. [15:35:02] Coren, any way I can monitor memory consumption of the cyberbot node? [15:35:28] ... 'free'? For that matter, you can see useage on the status page too. [15:35:52] ? [15:35:55] I'm lost [15:36:46] marc@tools-exec-cyberbot:~$ free [15:36:47] total used free shared buffers cached [15:36:47] Mem: 4049952 3666656 383296 0 26464 1563252 [15:36:47] -/+ buffers/cache: 2076940 1973012 [15:37:05] I got it. I [15:37:30] I'm trying to seal a serious memory leak in spambot. It's been leaking 1G of memory. [15:38:43] Quite a leak. [15:40:50] I assume 2076940 is the number to be looking at for how much is actually being used. [15:42:27] Coren, ^ I know right? But then again if you've seen TrustedInstaller.exe run Windows updates, it fills up all of your RAM. [15:42:35] I have 16 GB. [15:42:57] Coren: You might need days off, but most volunteers are active on the days you'd normall have off :P [15:43:30] Cyberpower678: Why do you look at the memory consumption of the whole node and not only that of the spambot job? [15:43:50] If you want just the process, check the values from ps or proc [15:44:13] scfc_de, because I can, and want to. I want to see what the total consumption is when all scripts are running. [15:45:02] The problem is the system will quite happily swap memory, thus changing the value you're looking at for all sorts of reasons [15:45:13] Might want to tweak sysctl settings to stop it until you're actually using ram [15:45:45] Cyberpower678: Just a moment ago you wrote about a memory leak in spambot that you wanted to seal. That's already done?! [15:54:55] scfc_de, very likely. It was likely coming from inefficient memory management of open MySQL connections. [15:56:21] scfc_de: Coren (relaxing): /usr/bin/webservice seems to be missing from tools-dev [15:57:19] * hedonil mumbles that Cyberpower678 should use persistant connections to fix that [15:58:17] hedonil, No. It was using an old DB class on Peachy, that still utilized mysql_connect [15:58:53] I deleted that class and replaced connections with mysqli functions. [15:59:45] Cyberpower678: even if that works now, the other solution could even save more resources and connection time [16:01:08] moar moar [16:01:09] usage 1: cpu=00:05:03, mem=315.75155 GBs, io=7.73198, vmem=1.081G, maxvmem=1.303G [16:01:33] mem=315.75155 GBs [16:01:47] Am I not reading that right. That sounds a bit worrying. [16:04:45] I went to created a tool called 'tools' and it didn't reject that. what will happen with mails? :p [16:05:15] Coren: ^ [16:06:17] liangent: Coren ist relaxing his ol' bones. pssst ;) [16:06:28] hedonil: Re webservice, I think an old(er) version of jobutils is still in /data/project/.system/deb. I'll copy it manually to tools-dev for the moment. [16:06:54] scfc_de: fine [16:39:42] liangent: since tools.x and x.* both work, there is not issue. Everything to tools.tools goes to it. :-) [16:39:45] Coren: in your email, is the first dot in .*@tools.wmflabs.org a literal dot or a regex any char dot? [16:40:04] gifti: It's a glob, not a regex. :-) [16:40:10] ok [16:40:47] why not also @tools.wmflabs.org? [16:41:41] gifti: Because there is no guarantee that toolnames are unique and distinct from usernames. (And, in fact, we have a couple tools named after their maintainers already) [16:41:52] oh, right [16:43:02] liangent: Oh, wait, I see your issue. Hm. [16:43:47] liangent: Heh. Well, sorry but your tools tool won't support tools.* :-) [16:53:24] /digh [16:53:32] xtools has hung up. Again. [16:54:15] Coren, 10.68.16.4 tools-webgrid-01:4078 - [09/Mar/2014:16:46:10 +0000] "GET /xtools/pcount/Hall_Monitor/en/wiki HTTP/1.1" 404 345 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" [16:57:04] * Cyberpower678 gets to work on the new edit counter [17:00:36] Coren: fine [17:09:25] Can someone tell me which doxygen I should be using? I want to install it on my vagrant development VM [17:28:58] hi my bot doesn't work [17:29:11] error returns this message: "Fatal Python error: Couldn't create autoTLSkey mapping" [19:58:15] Damian: When would namespace=="Main:", edit_status!="reverted" and edit_score=="N/A"? [19:58:17] Damianz: ^ [20:10:27] Dunno without digging [20:44:16] Damianz: i hear i broke labs, can you please elaborate? [20:44:43] uh, what? [20:44:44] Did you move misc::ircecho to ircecho? [20:44:51] i did [20:46:01] Ok - so it broke some labs instances in 2 ways. a) Wikitech isn't updated (neither the database or ldap) so the puppet ENC is still enforcing misc::irecho on instances so they fail a catalog compile. b) Labs doesn't support maps via the ENC, so you can't use the module (it use to be 2 string vars). [20:46:27] I don't /think/ it's widley used, but where it is nothing has been updated (though the wider issue is workflow etc, which I had a seperate rant about). [21:19:52] hi my bot doesn't work with jstart [21:20:11] the error returns this message: "Fatal Python error: Couldn't create autoTLSkey mapping" [21:22:21] coet: Most issues with odd error messages when running something on the grid is "not enough memory". [21:24:34] ok, Coren. [21:24:54] i had something similar [21:25:09] solution will be an update of a package [21:25:20] openssl had lost a feature [21:25:57] i'm still waiting but i have patience ... must have ^^ [21:26:35] but if I run it with jsub the trouble doesn't occur [21:27:29] coet: That's odd. jsub and jstart are virtually the same; the latter just adds a restart wrapper and checks that no other job with that name exists. [21:27:34] ... that's odd. There's probably a slightly higher memory requirement with jstart vs jsub, but it'd be very small. Unless you happened to be at the very very limit. [21:29:00] Hi all. Project-crontabs on tools-login.eqiad are not working for one of my migrated projects, any obvious things I might have missed? [21:29:30] dschwen: They are copied, but commented out. Did you edit and uncomment it? [21:29:37] yep [21:29:58] I have it working on a different project just fine [21:30:34] HA!!! [21:30:37] NO! [21:30:45] Hm. I have heard of no issues with cron. Did you check the mailbox for your tool see if there are crond complaints in it? [21:30:51] crontab does work, but my python bot throws: "libgcc_s.so.1 must be installed for pthread_cancel to work" [21:30:59] yikes! [21:31:04] dschwen: Not enough memories. :-) [21:31:30] do I nbeed to request more on jsub, or is there just a lack of mem [21:31:35] if so, why? [21:31:38] Just request more. [21:31:49] ok, the bot is not very demanding though [21:32:06] python is (a) a glutton abd (b) deals very badly with malloc() returning no more memory. :-) [21:32:15] I think I did not specify any mem on my jsub line [21:32:23] ha, ok [21:32:42] Python demands >300m of vmem before it even starts. [21:32:56] uhm [21:33:10] but that is shared stuff like libs, or not [21:33:28] Much of it is. [21:33:53] so -mem 500m? [21:34:06] Should be plenty. I think that's what most python users use. [21:34:27] ok, thanks [21:35:10] yeah, seems to run now [21:35:38] have we documented those values somewhere? [21:38:58] Coren: how would you specify a script to process emails in a tool? [21:40:19] gifti: https://wikitech.wikimedia.org/wiki/Nova_Resource_Talk:Tools/Help#300-350M.3F_Really.3F [21:40:52] gifti: And that's not supported quite yet (email to process) because right now it'd run on the mail server (not good). That'll come next week. [21:41:41] so, when it is available, how would the format be? [21:42:24] Almost certainly '|theprogram' though that is traditionally piped whereas I'm almost certainly going to use a temporary file instead. [21:42:42] hm [21:53:04] those values are very theoretic and not from practice [22:19:09] Eh, they are anomie's practical experiences?! [22:20:12] Is there anyone in here that could help me with a one-off database query? I'm trying to produce a list of all the files that are above MediaWiki's 50 megapixel limit. [22:21:45] not [22:24:18] hi. do you know how to make a query with mediawiki api to get all the articles inside a category, and also the articles insite the subcategories? using "categorymembers" [22:24:56] make a recursive function [22:25:23] Dmcdevit: Do you need the SQL query for yourself or the list of files that exceed 50 MP? [22:25:45] Arnaugir: The API doesn't allow that IIRC; you have to use something like CatScan for that. [22:25:47] gifti: I can do that, but I meant directly from the query [22:25:53] scfc_de: thanks [22:26:05] not possible imo [22:26:14] scfc_de: Sorry, that was unclear. I'm asking for the list. I can't run a query myself. [22:26:58] Dmcdevit: On Commons or a specific wiki? [22:28:07] Mostly Commons since that's where I expect most will be found, but if it's possible to make the query across all wikis that's preferable. [22:29:01] Let me see if I can whip up something for Commons; if it works there, I'll schedule a job to collect it for all x hundred wikis :-). [22:30:22] Dmcdevit: Do you have an example image on Commons with more than 50 MP to verify my query works? [22:31:08] That would be awesome. I think these types of thumbnail errors would be a good regular database report for Commons. But what I wanted to do right now is see what the situation is for high-resolution images, and whether it's worth investigating raising the limit again (it's been about a year since it was brought up to the current limit). [22:32:03] scfc_de: https://commons.wikimedia.org/wiki/File:Constitution_of_the_United_States,_page_1.tif [22:34:46] Dmcdevit: Okay, my query works; I'll schedule the jobs. Getting the results may take a while; where can you be reached? [22:35:45] I'm on IRC a lot, or else [[commons:User talk:Dominic]] works. [22:35:52] Thanks so much. :-) [22:36:55] Dmcdevit: Okay, I'll get back to you once they finished. [22:40:08] Hello, I'm having problems accessing the Tools database in Python. When I use the command line it works, but in Python, it throws the exception: ""Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)". Is anyone familiar with such an issue/has a resolution for the issue? I've tried using my default_file, directly using the login vars, changing the port [22:40:08] an none of that works. [22:42:16] ceradon: "Can't connect to *local* MySQL server" doesn't sound right. You need to either connect to tools-db or enwiki.labsdb/s1.labsdb/etc. [22:43:15] *to the DB server at [22:44:29] scfc_de, the full db server adress would be what? tools-db.wikitech.wmflabs.org or something like that? [22:46:04] tools-db should be sufficient; or tools-db.eqiad.wmflabs if you prefer FQDNs (and your tool is migrated to the new datacenter). [22:46:20] scfc_de: If it's possible to to get back the resolution in MP of each of the results, too, that would help . Sorry if you already started the query. [22:47:17] Dmcdevit: No problem, I'll include img_name, img_width, img_height, and img_width * img_height / 1000 / 1000; [22:47:31] Perfect! [23:06:15] Dmcdevit: Done; if I counted correctly, 20802 images with >= 50 MP in all projects, 20206 of those on Commons. In what format do you want the data? CSV? [23:07:28] That works fine. [23:07:36] Or a table on-wiki. [23:07:58] I guess 20,000 is a lot for a wiki page, actually. ;-) [23:10:50] 2152805 bytes minimum -- I wouldn't try it :-). I'll produce a CSV and give you a link in a few minutes. [23:23:07] Dmcdevit: Try importing http://tools.wmflabs.org/wikilint/50mps.csv? [23:25:32] Technically, that was a TSV. ;-) [23:25:45] scfc_de: Thanks so much! Looks great. [23:32:45] I think I haven't actually used commas in a CSV in a *long* time :-). [23:36:30] IPs get redacted before they get to your webscript, right? [23:37:21] a930913: They should be; otherwise there's a mistake in the setup. [23:38:24] Is there a way to track uniques though? I.e. to prevent abuse. [23:59:55] a930913: With PHP you can rely on a session.