[00:25:27] <grrrit-wm>	 (03PS1) 10Yurik: Added ZeroBanner & Portal to mobile channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/136513 
[00:26:37] <YuviPanda>	 yurik: want me to merge and deploy? :)
[00:26:53] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] Added ZeroBanner & Portal to mobile channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/136513 (owner: 10Yurik)
[00:26:56] <grrrit-wm>	 (03Merged) 10jenkins-bot: Added ZeroBanner & Portal to mobile channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/136513 (owner: 10Yurik)
[00:27:03] <yurik>	 YuviPanda, yes please )
[00:27:08] <yurik>	 i have no clue how to depl it
[00:27:47] <YuviPanda>	 yurik: there's a wikitech page, but I wrote the bot so easier for me to do anyway.
[00:28:03] <yurik>	 ) thx
[00:29:01] <grrrit-wm>	 (03CR) 10Yuvipanda: "Thanks for the patch!" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/136513 (owner: 10Yurik)
[00:36:15] <Cyberpower678>	 Coren, ping
[10:15:16] <hedonil>	 Coren: YuviPanda|zzz: Webproxy bug:  If trailing / is missing, proxy adds it but switches then from  https to  http 
[11:35:09] <liangent>	 hedonil: it's not added by the proxy
[11:35:12] <liangent>	 hedonil: see https://bugzilla.wikimedia.org/show_bug.cgi?id=64627
[11:39:03] <hedonil>	 liangent: thx for bz, but well, if it's not added by the proxy - who adds it (someone/-thing does it ;-) )
[11:47:21] <liangent>	 hedonil: the http server running as your user
[11:47:41] <liangent>	 hedonil: the one started by that webservice command
[11:47:56] <hedonil>	 liangent: yes I know, but that's another thing
[11:51:09] <hedonil>	 liangent: it happens somewhere here:  https://git.wikimedia.org/blob/operations%2Fpuppet/69908a8f919e9f6bcae545cf49a38c913bdaa9ed/modules%2Fdynamicproxy%2Ffiles%2Furlproxy.lua
[11:57:46] <wikibugs>	 3Wikimedia Labs / 3tools: lighttpd redirects URLs of directories without a trailing slash from https to http - 10https://bugzilla.wikimedia.org/64627#c5 (10metatron) If trailing / is missing, proxy adds it but switches then from  https to  http   Example: (all but the last entry in top menu have trailing / )...
[12:02:45] <wikibugs>	 3Wikimedia Labs / 3tools: lighttpd redirects URLs of directories without a trailing slash from https to http - 10https://bugzilla.wikimedia.org/64627#c6 (10metatron) And it is webproxy who does this, not lighty. So, the caption of this bz may be misleading.
[12:02:57] <liangent>	 hedonil: hm I didn't actually check it myself
[13:20:13] <grrrit-wm>	 (03PS1) 10Multichill: Some cleanup to move from Toolserver to Toollabs: * Changed hardcoded database servers * Removed a lot of hardcoded paths * Added .gitreview [labs/tools/multichill] - 10https://gerrit.wikimedia.org/r/136536 
[13:22:15] <grrrit-wm>	 (03CR) 10Multichill: [C: 032 V: 032] "ok, that worked. Let's see if this works." [labs/tools/multichill] - 10https://gerrit.wikimedia.org/r/136536 (owner: 10Multichill)
[14:16:57] <abartov>	 hi
[14:17:25] <abartov>	 what is the place to ask for additional packages to be installed on tools machines?
[14:18:00] <abartov>	 (e.g. I need 'gawk libyaml-dev libgdbm-dev libncurses5-dev bison libffi-dev' for Ruby.)
[14:20:37] <Coren>	 abartov: Opening a bugzilla.  If you want extra-fast processing, adding the needed patch against dev_environ and exec_environ helps.
[14:21:13] <abartov>	 thanks.
[14:22:05] <Coren>	 (Look at modules/toollabs/manifests/dev_environ.pp in operations/puppet for instance, it's usually very straightforward)
[14:22:13] <abartov>	 component: infrastructure?
[14:23:06] <Coren>	 component: tools
[14:26:35] <wikibugs>	 3Wikimedia Labs / 3tools: install gawk libyaml-dev libgdbm-dev libncurses5-dev bison libffi-dev - 10https://bugzilla.wikimedia.org/65974 (10Asaf Bartov) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier I'd like the following packages installed, to be able to run Ruby tools:  gawk libyaml-dev libgdbm-dev...
[14:42:23] <IVeertje>	 hi
[14:42:46] <IVeertje>	 http://toolserver.org/~bryan/flickr/upload < hasn't migrated yet
[14:42:58] <IVeertje>	 and Bryan seems to have left the project ~2 years ago
[15:10:25] <Betacommand>	 IVeertje: you have to renew your toolserver account every 6 months, so he is probably around
[15:10:39] <IVeertje>	 ok
[15:32:11] <multichill>	 What do I have to put in my .htaccess again so that the log files are not offered for download at http://tools.wmflabs.org/catbot/logs/ ?
[15:36:10] <Nemo_bis>	 IVeertje: he said that he'd prefer not to migrate it, ideally UploadWizard or magnus' tool should be fixed/enhanced to cover all the use cases 
[15:36:34] <Nemo_bis>	 You can find more info on Commons-l and perhaps bugzilla
[15:37:01] <IVeertje>	 yeah, UploadWizard sort of does Flickr nowadays but I read yesterday that only admin + some other rank have that feature
[15:38:41] <hedonil>	 multichill: try this (untested)  https://tools.wmflabs.org/paste/view/d1ae31e4
[15:39:01] <hedonil>	 multichill: add this lines to your .lighttpd.conf  && restart webservice
[15:39:34] <marktraceur>	 IVeertje: That's correct.
[15:39:42] <hedonil>	 multichill: your .htaccess has no power here ;-)
[15:40:54] <IVeertje>	 I filed this bug report about flickr + UploadWizard yesterday: https://bugzilla.wikimedia.org/show_bug.cgi?id=65941
[15:41:08] <multichill>	 hedonil: Thanks, will try it in a moment
[15:48:41] <hedonil>	 multichill: little modification in regex (now tested)  https://tools.wmflabs.org/paste/view/e2b10fac
[15:49:57] <multichill>	 hedonil: That's not what I meant. Sorry, I meant that you would get it in the browser
[15:50:18] <hedonil>	 multichill:you meant dir-listing?
[15:50:21] <multichill>	 hedonil: Like https://toolserver.org/~multichill/temp/queries/erfgoed/
[15:50:49] <hedonil>	 multichill: ok
[15:50:58] <multichill>	 If you click a file, you view it in your browser. On http://tools.wmflabs.org/catbot/logs/ my browser wants to download the files
[15:51:55] <hedonil>	 multichill: try this one https://tools.wmflabs.org/paste/view/8f2dccbe
[15:52:33] <multichill>	 dir-listing.activate = "enable" ?
[15:52:42] <hedonil>	 multichill: yep
[15:52:44] <multichill>	 That's already working
[15:53:24] <hedonil>	 multichill: ok, it seems I now got what you mean
[15:53:47] <Betacommand>	 multichill: one sec, its with defining encoding/file type
[15:54:05] <hedonil>	 multichill:  add this one     mimetype.assign += ( ".log"  => "text/plain" )
[15:55:00] <Betacommand>	 make it "text/plain;charset=UTF-8"
[15:55:12] <Betacommand>	 otherwise your going to have a headache
[15:56:42] <Betacommand>	 hedonil: are you trying to give multichill a headache?
[15:57:27] <hedonil>	 Betacommand: listet several thing, no headache so far :-)
[15:58:04] <Betacommand>	 hedonil: forgetting to set encoding can cause some really odd things
[15:59:44] <hedonil>	 Betacommand: well, paracetamol is your friend :P
[16:00:25] <multichill>	 Nope, that didn't work
[16:00:55] <Betacommand>	 hedonil: Hell I once had to debug an issue when uploading a script I couldnt find out why it wouldnt run on the server, but worked locally. Ended up being an issue with \r\n in files on Unix systems
[16:01:04] <Betacommand>	 multichill: did you reboot your webservice?
[16:01:10] <hedonil>	 Betacommand: but I agree with you, besides having UTF-8 set anyway as standard, it's ok to do this in the header ;-)
[16:01:10] <multichill>	 Yes, restart
[16:01:30] <multichill>	 Fuck it, enough time wasted on this. Next
[16:02:53] <Betacommand>	 multichill: can you pastebin your config file? I do  the same thing on my tool and dont have issues
[16:03:24] <multichill>	 What's the path? I'll just copy that one :-)
[16:03:57] <hedonil>	 https://tools.wmflabs.org/paste
[16:04:25] <multichill>	 Betacommand: I mean the path on toollabs
[16:04:39] <Betacommand>	 multichill: https://tools.wmflabs.org/paste/view/a1a6486f
[16:04:50] <Betacommand>	 my tools dir is not readable
[16:05:21] <hedonil>	 as it's mutcichil's 
[16:05:44] <Betacommand>	 hedonil: ??
[16:06:09] <hedonil>	 many tool dirs are locked down
[16:06:15] <multichill>	 Betacommand: Just to be sure. What is the full path?
[16:06:24] <hedonil>	 bad habit imho
[16:06:26] <valhallasw>	  /data/project/betacommand-dev/.lighttpd.conf is world-readable
[16:06:39] <Betacommand>	 valhallasw: it shouldnt be
[16:06:42] <Betacommand>	 damit
[16:06:53] <multichill>	 Why shouldn't it be?
[16:07:14] <Betacommand>	 multichill: Security issues
[16:07:19] <hedonil>	 haha
[16:07:22] <multichill>	 LOL
[16:07:38] <Betacommand>	 people being able to read the / subdata can cause issues
[16:08:05] <Betacommand>	 IE stored cookies and what not
[16:08:22] <hedonil>	 just to remind the people here: it's an open source dev platform
[16:08:28] <valhallasw>	 unless you hand-rolled your cookie handling, I doubt that's an issue
[16:09:04] <hedonil>	 with all data complying to TOS and privacay rules of Toollabs
[16:09:07] <multichill>	 So we can steal the cookies of all your blocked accounts? 
[16:09:32] <multichill>	 Anyway, thanks to the open file I now have it working
[16:09:41] <multichill>	 thank you hedonil and Betacommand
[16:10:52] <multichill>	 valhallasw: http://tools.wmflabs.org/catbot/logs/loose_category_from_gallery.log <- hmm, volgens mij moet ik nog ergens een console encoding goed zetten ;-)
[16:12:10] <Betacommand>	 hedonil: you still shouldnt be snooping in other peoples directories
[16:12:13] <valhallasw>	 multichill: wtf. why would *anything* use latin-1 on linux :|
[16:13:35] <multichill>	 Dunno, forced it to utf-8 now
[16:13:39] <multichill>	 console_encoding = u'utf-8'
[16:13:40] <valhallasw>	 ascii, I understand. UTF-8, too, but latin-1?!
[16:13:48] <valhallasw>	 this is on labs via jsub?
[16:14:02] <multichill>	 I think pywikibot/compat might be to blame for that
[16:14:11] <valhallasw>	 ohh.
[16:14:43] <valhallasw>	 nope, that also uses __sys.stdout.encoding
[16:14:44] <valhallasw>	 :/
[16:14:56] <valhallasw>	 oh, wait
[16:15:00] <valhallasw>	 https://github.com/wikimedia/pywikibot-compat/blob/master/config.py#L655
[16:15:25] <multichill>	 right
[16:15:42] <valhallasw>	 which only happens if it can't determine sys.stdout.encoding for some reason
[16:15:55] <multichill>	 Might happen if it's jsubbed
[16:15:59] <valhallasw>	 which happens when you pipe data
[16:16:09] <valhallasw>	 python -c "import sys; print sys.stdout.encoding" | less    returns 'None'
[16:16:18] <multichill>	 Have to use compat otherwise some of the categorization code doesn't really work.....
[16:16:29] <valhallasw>	 it's the same in core
[16:16:34] <multichill>	 ah
[16:16:54] <hedonil>	 Betacommand: no one is "snooping" w/o reason, but as a rule of thumb: the more code being "borrowed" the more locked down the dir 
[16:16:55] <multichill>	 Anyway, yet another bot moved
[16:18:37] <hedonil>	 http://en.wikipedia.org/wiki/Open-source_software
[16:20:25] <Betacommand>	 hedonil: it all depends, and honestly Id rather not have to worry about security issues and leaked information
[16:21:22] <hedonil>	 Betacommand: just saying. In the end it's your gizmo
[16:24:38] * hedonil just took a look at Betacommand's counts  https://tools.wmflabs.org/xtools/ec/?user=Betacommand&lang=en&wiki=wikipedia
[16:38:40] <Mifter>	 Anyone know how to stop a job in-progress on labs?
[16:38:54] <Betacommand>	 Mifter: qdel
[16:38:59] <Mifter>	 Thanks
[17:28:18] <gifti>	 i actually wanted to ask if everything in http://tools.wmflabs.org/giftbot/dwl.sh had to be in the grid, but now i read in tools/help that the whole thing may not be longer than a few seconds?? what's the point? i thought jlocal is for submitting jobs from within a "job"
[17:30:10] <valhallasw>	 gifti: ??
[17:31:04] <valhallasw>	 a) I have no clue where you got the 'not longer than a few seconds', nor where you heard of jlocal, because as far as I can see that's not a command on tool labs
[17:31:58] <gifti>	 https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Creating_a_crontab
[17:32:08] <gifti>	 If your cron entry only includes a brief script that, itself, sends any real work to the grid then you may skip that automatic invocation by prepending jlocal explicitly marking it as a local job. Any script or job invoked with jlocal should not be running more than a few seconds and use minimal resources; misuse of that feature may have severe impact on general reliability for all users and is not allowed. 
[17:32:34] <valhallasw>	 !log local-nlwikibots tvpmelder upped max_vmem from 256M to 1G; hopefully that will prevent further OOMs
[17:32:37] <labs-morebots>	 Logged the message, Master
[17:32:58] * hedonil notes that "jlocal" is the one command that must not be named here
[17:33:15] <wikibugs>	 3Wikimedia Labs / 3tools: lighttpd redirects URLs of directories without a trailing slash from https to http - 10https://bugzilla.wikimedia.org/64627#c7 (10Tim Landscheidt) (In reply to metatron from comment #6) > And it is webproxy who does this, not lighty. So, the caption of this bz may > be misleading....
[17:33:38] <valhallasw>	 ohhh, jlocal is just a placeholder to prevent the crontab-editor from adding jsub to your crontab line. I see
[17:34:32] <valhallasw>	 gifti: looking at that script, the jsub invocations are okay, but the sort is already dubious.
[17:34:58] <valhallasw>	 what's the problem with running that script on the grid?
[17:35:21] <gifti>	 so you suggest to leave it as it is?
[17:36:09] <valhallasw>	 actually, I'd remove the internal jsub invocations, as you're running it with -sync  y anyway
[17:40:11] <valhallasw>	 !log local-nlwikibots archivering / erwin85 and dplinks / erwin85 are still running on TS. First step: moving code from the TS to TL.
[17:40:12] <labs-morebots>	 Logged the message, Master
[17:42:46] <gifti>	 Coren: can you clarify wrt above issue?
[18:12:33] <valhallasw>	 !log local-nlwikibots dplinks needs a table that was generated by bryan; see https://nl.wikipedia.org/wiki/Wikipedia:De_kroeg/Archief_20101212 -- I'll first move archivering.
[18:12:34] <labs-morebots>	 Logged the message, Master
[18:12:53] <valhallasw>	 !log nlwikibots has the new morebots been deployed yet?
[18:12:55] <labs-morebots>	 Logged the message, Master
[18:13:02] <valhallasw>	 \o/
[18:30:02] * valhallasw hits bash with a hammer
[18:32:03] <Betacommand>	 valhallasw at least your DNS didnt decide to die on you :P
[18:40:27] <valhallasw>	 Betacommand: :p well, I don't need my own grid, that helps :-p
[18:41:35] <Betacommand>	 valhallasw: Neither do I, however when your in the middle of patching a bug, and you cannot access any outside site, IE your SVN server......
[18:42:10] <valhallasw>	 8.8.8.8 ftw.
[18:42:15] <valhallasw>	 also, git
[18:42:18] <valhallasw>	 :-p
[18:42:57] <Betacommand>	 valhallasw GIT sucks on windows
[18:43:10] <alkamid>	 Hi, has something happen to pywikibot recently? I'm getting "ImportError: cannot import name catlib" from "from pywikibot import catlib"
[18:43:44] <valhallasw>	 alkamid: not that I know of. What are you trying to do, and what's the full error message you get?
[18:44:22] <alkamid>	 valhallasw, http://pastebin.com/n4nc2kJU
[18:44:34] <alkamid>	 it must have started about a week ago
[18:44:54] <alkamid>	 in every script with "from pywikibot import catlib"
[18:45:34] <alkamid>	 my PYTHONPATH: /shared/pywikipedia/rewrite:/shared/pywikipedia/rewrite/externals/httplib2:/shared/pywikipedia/rewrite/scripts:/data/project/alkamidbot/scripts
[18:46:10] <alkamid>	 I can't remember changing anything in my scripts recently (and the modification dates attest to this)
[18:46:53] <valhallasw>	 alkamid: as far as I know, 'from pywikibot import catlib' never worked with core; try 'from pywikibot.compat import catlib' instead
[18:47:19] <Betacommand>	 valhallasw he is using the old core location, not compat
[18:47:33] <valhallasw>	 Betacommand: I know. Re-read my message.
[18:47:57] <Betacommand>	 Ah
[18:48:01] <valhallasw>	 there's a module 'pywikibot.compat' which provides wrappers to ease porting
[18:50:07] <alkamid>	 valhallasw, I thought I modified my scripts to work with the rewrite/core branch
[18:50:30] <alkamid>	 so I guess changing /shared/pywikipedia/rewrite to /core should do the job?
[18:51:26] <valhallasw>	 no, that path is fine. You need to change the import
[18:51:48] <valhallasw>	 or, alternatively, add /shared/pywikipedia/rewrite/pywikibot/compat to your PYTHONPATH
[18:52:57] <alkamid>	 valhallasw, but why would I want to have anything to do with compat?
[18:53:37] <valhallasw>	 you don't
[18:53:45] <Betacommand>	 alkamid: catlib is from compat
[18:54:10] <valhallasw>	 https://github.com/wikimedia/pywikibot-core/tree/master/pywikibot/compat provides wrappers to ease portign from compat to core
[18:54:37] <valhallasw>	 e.g. https://github.com/wikimedia/pywikibot-core/blob/master/pywikibot/compat/catlib.py
[18:56:38] <alkamid>	 fair enough, so I probably want to change catlib to Category in my scripts, right?
[18:57:01] <valhallasw>	 you can just import Category from pywikibot
[18:58:32] <wikibugs>	 3Wikimedia Labs / 3tools: crontab -e adds bashrc output to crontab file - 10https://bugzilla.wikimedia.org/65980 (10Merlijn van Deen) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier How to reproduce: ------------------ - add "echo something" to .bashrc. - Then run crontab -e.  Expected behavior: ------...
[18:58:56] <valhallasw>	 !log nlwikibots 'archivering' moved to Tool Labs; initial max_vmem at 350M
[18:58:58] <labs-morebots>	 Logged the message, Master
[19:05:26] <alkamid>	 valhallasw, thanks for your help
[19:05:30] <alkamid>	 Betacommand, thanks as well
[19:33:56] <valhallasw>	 !log nlwikibots Op basis van https://nl.wikipedia.org/wiki/Wikipedia:De_kroeg/Archief_20101212 dab.sh opgezet. Reporten naar nlwiki moet nog gebeuren.
[19:33:58] <labs-morebots>	 Logged the message, Master
[19:37:23] <valhallasw>	 multichill: ^ dplinks helemaal omzetten heb ik even geen zin maar in, maar ben daar nu halverwege mee
[19:37:36] <valhallasw>	 multichill: heb iig archivering + dab-database van bryan geport
[19:37:51] <valhallasw>	 dan is dplinks omzetten een kwestie van de goede db table instellen en gaan met die banaan
[19:54:30] <wikibugs>	 3Wikimedia Labs / 3tools: lighttpd redirects URLs of directories without a trailing slash from https to http - 10https://bugzilla.wikimedia.org/64627#c8 (10metatron) Hmm, this need further knowledge on the subject.  1. Lighttpd is configured without ssl engine, and makes rewrite only for tool's root >> ssl.e...
[21:18:01] <wikibugs>	 3Wikimedia Labs / 3tools: install gawk libyaml-dev libgdbm-dev libncurses5-dev bison libffi-dev - 10https://bugzilla.wikimedia.org/65974 (10Sam Reed (reedy)) s:5normal>3enhanc
[21:51:11] <multichill>	 valhallasw: Moet je niet loggen onder tools.nlwikibots?
[21:51:26] <multichill>	 Goed bezig verder. Waar kan ik dag log ook alweer zien?
[22:00:02] <valhallasw>	 multichill: https://wikitech.wikimedia.org/wiki/Nova_Resource:Local-nlwikibots/SAL
[22:00:29] <multichill>	 Die groepen zijn toch allemaal recent hernoemd?
[22:00:32] <valhallasw>	 multichill: service groups are still local-XXXXXX in ldap
[22:00:54] <valhallasw>	 and I submitted a patch to make XXXXX an alias of local-XXXXX if the name itself doesn't exist
[22:01:34] <multichill>	 !log local-catbot Moved all the crap from Toolserver to Toollabs and burned the old account
[22:01:36] <labs-morebots>	 Logged the message, Master
[22:01:52] <multichill>	 hmm, ok
[22:02:21] <valhallasw>	 !log catbot so this also works
[22:02:22] <labs-morebots>	 Logged the message, Master
[22:02:27] <valhallasw>	 !log tools.catbot this doesn't work
[22:02:27] <labs-morebots>	 tools.catbot is not a valid project.
[22:03:02] <multichill>	 Hmm, but you're not member of the catbot group....
[22:03:41] <valhallasw>	 there's no authentication :-p
[22:04:10] <valhallasw>	 there's the beautiful IRC +b flag
[23:29:00] <wikibugs>	 3Wikimedia Labs / 3tools: crontab -e adds bashrc output to crontab file - 10https://bugzilla.wikimedia.org/65980#c1 (10Marc A. Pelletier) This workaround /does/ work, but I should point out that it is generally not recommended to have *any* output whatever in one's .bashrc as this is implicitly sourced in nu...