[00:43:30] Kolossos, Magnus Manske, anyone mind giving me access to geohack project so I can fix the missing CSS? [00:45:37] scfc_de: why revert? [00:46:56] scfc_de: I'm confused. is the problem that infrastructuredoesn't get applied? then it should be included in the proxy role, rather than revert this [01:10:26] YuviPanda|zzzz: Still awake? [04:52:39] (03PS1) 10Spage: Notify mobile and corefeatures of Mantle commits [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/137270 [06:12:30] 3Wikimedia Labs / 3Infrastructure: l10nupdate gid should be 10002 to match production/Puppet - 10https://bugzilla.wikimedia.org/65588#c2 (10Bryan Davis) On deployment-bastion, /home/l10nupdate and /mnt/srv/scap-stage-dir/php-master/cache/l10n contain all of the files owned by the l10nupdate user. /home/l10... [08:50:08] !log integration rebased operations/puppet.git on puppetmaster [08:50:10] Logged the message, Master [09:06:02] 3Wikimedia Labs / 3tools: Moving toolserver domain, mail and redirects - 10https://bugzilla.wikimedia.org/66113 (10nosy) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier At some point after 30th June we will have to start migration of mail, redirects and domain. I'd like to ask you for a work flow for t... [09:12:59] 3Wikimedia Labs / 3tools: Copy contents of https://svn.toolserver.org/ to Wikimedia git - 10https://bugzilla.wikimedia.org/58801#c15 (10nosy) Ping [09:50:32] !log deployment-prep Reset salt caches by running `salt '*' state.clear_cache` from deployment-salt -- deployment-pdf01 now no longer reports errors when returning status for deployment [09:50:34] Logged the message, Master [10:06:59] 3Wikimedia Labs / 3tools: Move wiki.toolserver.org to WMF - 10https://bugzilla.wikimedia.org/60220#c27 (10Silke Meyer (WMDE)) Ping Reedy! [11:40:29] 3Wikimedia Labs / 3tools: Tool Labs: Provide anonymized view of the user_properties table - 10https://bugzilla.wikimedia.org/58196#c43 (10Eran Roz) What is the current status of the bug? It becomes very importantly to fix it in the very soon future, as the toolserver is going to be down in 3 weeks http://li... [11:59:36] @notify scfe_de [12:13:17] Good day. After I opened Putty. I did this: become jbbot, nano task1.sh, #!/bin/bash, Enter, python commonscat.py -start:! -always, Ctrl + x, Y, chmod +x task1.sh, jsub task1.sh. All this I did but the tool did not work. Please help me to fix this problem. [12:13:43] javedbaker: heya! what error were you getting? [12:16:28] no error I saw, there was a message: your job submitted, but when I made qstat the state was qw [12:18:03] javedbaker: did you check your logs to see if it errored out? [12:18:11] javedbaker: they should be in your tool's home folder with a .err extension [12:20:34] YuviPanda:Sorry Im new in tool labs, tell me please how to open error message, you mean in Putty or Wikitech? [12:20:46] javedbaker: via putty [12:21:05] javedbaker: do an ls on your tool's home directory, and see if there's a file that ends with .err there [12:22:17] YuviPanda: Can you please give the order to do it? [12:22:45] javedbaker: sorry, I don't know what you mean by 'order' [12:23:51] YuviPanda: I mean how to open the file? I do not know how [12:24:24] javedbaker: hmm, I am unsure how to do it with Putty either (haven't used Windows in a long time). Sorry! Maybe someone more familiar with putty will come along at some point [12:24:33] petan: ^ [12:28:27] javedbaker: you could use nano as you did with task1.sh [12:31:29] YuviPanda & gifti: OK, I have found this error message:python: can't open file 'commonscat.py': [Errno 2] No such file or directory. So how to fix it?.Note:I already wrote git clone --recursive https://gerrit.wikimedia.org/r/pywikibot/compat.git pywikipedia, and cd pywikipedia [13:28:14] 3Wikimedia Labs / 3tools: Tool Labs: Provide anonymized view of the user_properties table - 10https://bugzilla.wikimedia.org/58196#c44 (10Helder) s:5normal>3major Changing to major per previous comment. [13:35:32] 3Tool Labs tools / 3[other]: Migrate https://toolserver.org/~erwin85/randomarticle.php to Tool Labs - 10https://bugzilla.wikimedia.org/60871#c1 (10Silke Meyer (WMDE)) Am I right in thinking the tool is working in Tool Labs? If so, I would ask Nosy to create a redirect as erwin85's toolserver account has expi... [13:45:31] 3Wikimedia Labs / 3tools: Tool Labs: Provide filtered view of user_properties table containing short list of properties, linked to userID - 10https://bugzilla.wikimedia.org/64115#c2 (10Marc A. Pelletier) 5NEW>3ASSI a:3Sean Pringle Handing off to Sean for the underlying table. [13:51:28] 3Wikimedia Labs / 3tools: tools.wmflabs.org inaccessible via labs instances - 10https://bugzilla.wikimedia.org/54052#c20 (10Marc A. Pelletier) 5PATC>3RESO/FIX There is now a local alias on all instances to point the public name at the private IP (though not done with the proposed patch). [13:55:30] 3Wikimedia Labs / 3tools: Install python module matplotlib - 10https://bugzilla.wikimedia.org/61445 (10Marc A. Pelletier) 5PATC>3RESO/FIX [14:06:38] Coren: good morning! I am wondering why we connect to nova.clouds.archive.ubuntu.com instead of our own mirror? [14:08:19] hashar: ... not by design; that comes from the image default and is not overriden in puppet. It might just be a question of applying the right class. Plz to open a bugzilla? [14:08:30] sure! [14:15:10] filled as https://bugzilla.wikimedia.org/show_bug.cgi?id=66121 [14:15:16] 3Wikimedia Labs / 3Infrastructure: set apt to Wikimedia mirror instead of http://nova.clouds.archive.ubuntu.com/ubuntu/ - 10https://bugzilla.wikimedia.org/66121 (10Antoine "hashar" Musso) 3NEW p:3Unprio s:3normal a:3None I noticed on integration-dev.eqiad.wmflabs (and probably everywhere) that apt is... [14:15:29] Coren: also you are not a default CC to the Wikimedia labs > infrastructure bugzilla compoent [14:17:35] Noted; I know I'm on labs > tools but it'd be good if I were also on >infrastructure [14:17:45] adding you there [14:23:15] 3Tool Labs tools / 3[other]: Migrate MMP steward bots https://toolserver.org/~stewardbots/docs/home - 10https://bugzilla.wikimedia.org/61208#c2 (10Silke Meyer (WMDE)) 5NEW>3RESO/FIX The steward bots have migrated to http://tools.wmflabs.org/stewardbots/. Old MMP on toolserver as expired, so I'm closing t... [14:31:29] 3Tool Labs tools / 3[other]: Migrate steward tools by jyothis https://toolserver.org/~jyothis/tools/stewtools.html - 10https://bugzilla.wikimedia.org/61196#c1 (10Silke Meyer (WMDE)) Still open. Poked jyothis on talk page. [14:36:14] 3Wikimedia Labs / 3tools: tools.wmflabs.org inaccessible via labs instances - 10https://bugzilla.wikimedia.org/54052#c21 (10Tim Landscheidt) 5RESO/FIX>3REOP (In reply to Gerrit Notification Bot from comment #19) > Change 123149 abandoned by coren: > Tools: Alias tools.wmflabs.org to internal webproxy >... [14:40:44] 3Wikimedia Labs / 3tools: tools.wmflabs.org inaccessible via labs instances - 10https://bugzilla.wikimedia.org/54052#c22 (10Marc A. Pelletier) 5REOP>3RESO/FIX Sorry, unmerged forked of it; bringing it in back to git is on my todo. Same idea. :-) [14:42:29] 3Wikimedia Labs / 3tools: Update tcl-trf to version 2.1.4-dfsg-3 - 10https://bugzilla.wikimedia.org/62387#c2 (10Marc A. Pelletier) 5NEW>3ASSI As per the link debian bug, the package does need a fix but the actual files for 2.1.4-dsfg-3 do not seem to be actually available at http://ftp.debian.org/debian... [15:15:08] scfc_de: you poked? :) [15:16:03] YuviPanda: Well, with your zzzs you always have to count them :-). [15:16:27] The basic reasoning is: Previously, tools-webproxy was puppetized rudimentally, now it isn't at all. [15:25:59] 3Wikimedia Labs / 3tools: tools.wmflabs.org inaccessible via labs instances - 10https://bugzilla.wikimedia.org/54052#c23 (10Tim Landscheidt) 5RESO/FIX>3REOP (In reply to Marc A. Pelletier from comment #22) > Sorry, unmerged forked of it; bringing it in back to git is on my todo. > Same idea. :-) Then... [15:26:39] scfc_de: tools-webproxy was pretty much empty, had just a 'TODO' [15:26:49] scfc_de: no actual puppet code there, other than a scaffold [15:27:29] 3Wikimedia Labs / 3tools: tools.wmflabs.org inaccessible via labs instances - 10https://bugzilla.wikimedia.org/54052#c24 (10Yuvi Panda) What Tim said :) [15:31:53] * YuviPanda goes off for food [15:32:31] YuviPanda: I know as I had noted that it needed to be puppetized :-). But your change bulldozed the scaffold instead of building a house :-). [15:33:57] YuviPanda|zzz: Eating while sleeping?! (Yeah, I know about closing the lid.) Enjoy your meal! :-) [16:00:46] bd808: ready for me to break the l10nupdate user? [16:01:22] andrewbogott: I'll get ready [16:01:59] 3Wikimedia Labs / 3tools: Update tcl-trf to version 2.1.4-dfsg-3 - 10https://bugzilla.wikimedia.org/62387#c3 (10Marc A. Pelletier) After a bit of chat with the upstream maintainer, I found the right version to backport. News soon. [16:02:10] andrewbogott: Logged in to the right boxes. Ready when you are [16:05:03] bd808: ok, changed [16:05:38] * bd808 sees: uid=10002(l10nupdate) gid=10002 groups=602(l10nupdate),10002 [16:05:48] I'll check to see if the 602 is a local group [16:07:34] andrewbogott: the `groups=602(l10nupdate),10002` doesn't seem to be coming from a local passwd/groups file on deployment-bastion [16:08:25] hm... [16:08:35] 602 is LDAp [16:08:37] Maybe it's cached? I wouldn't expect that especially [16:09:13] Coren: in theory it isn't anymore since I just switched it to 10002 [16:09:36] Ah, in LDAP? Then you'll probably need to clear the nlscd cache [16:11:12] nscd* [16:11:25] nscd -i passwd && nscd -i group [16:11:42] Coren, that's on the instance in question, right? bd808 ^ [16:11:51] yes [16:12:27] Hmm.. no joy; cleared and restarted nscd -- uid=10002(l10nupdate) gid=10002 groups=602(l10nupdate),10002 [16:12:41] bd808: What instance is this? [16:12:53] Coren: deployment-salt [16:14:13] `chgrp 10002 /home/l10nupdate` -- chgrp: changing group of `/home/l10nupdate': Invalid argument [16:14:17] root@deployment-salt:~# getent group l10nupdate [16:14:17] l10nupdate:*:602:l10nupdate [16:14:25] scfc_de: am back. I also don't see why we need the older apache based proxying system puppetized when it's no longer used anywhere :) [16:15:06] andrewbogott: It's still 602 in LDAP dude. [16:15:27] 151 cn=l10nupdate,ou=groups,dc=wikimedia,dc=org [16:15:27] gidNumber: 602 [16:16:02] is that with ldaplist -l passwd? [16:16:26] andrewbogott: I went and looked directly in ou=groups,dc=wikimedia,dc=org with ldapvi [16:16:52] YuviPanda: Eh? We need to puppetize what's on tools-webproxy? [16:16:52] Ah, so I see. Huh. [16:16:54] Ok, one second... [16:17:13] scfc_de: tools-webproxy is now solely running the nginx proxy, apache stuff is no longer there. [16:17:28] YuviPanda: Yes, and we need to puppetize that. [16:17:38] bd808: better? [16:17:46] scfc_de: you want to puppetize the apache stuff that's no longer there? [16:17:48] * YuviPanda is confused [16:18:05] andrewbogott: Yeah that looks better [16:18:11] ok, sorry [16:18:12] * bd808 tries to chgrp again [16:18:19] I changed the primary gid of the user w/out actually changing the group :( [16:18:27] YuviPanda: What Apache stuff? We need to puppetize the /current/ setup of tools-webproxy. [16:18:42] scfc_de: it *is* puppetized as role::proxy [16:19:02] andrewbogott: No worries. Thanks for the help [16:19:10] scfc_de: most of the code for that is also in the dynamicproxy module rather than toollabs since it shares code with the general labs proxy, but it's fully puppetized. [16:19:23] scfc_de: I even have a test instance (tools-proxy-test.wmflabs.org) that was built solely with puppet :) [16:19:58] scfc_de: role::labs::tools::proxy [16:21:12] YuviPanda: Now you got me confused. Hang on. [16:21:18] scfc_de: Yuvi is correct, the old apache-based webproxy role was obsolete before it was even completed; I never bothered filling it out since we knew it was going to be replaced, not it just dangles uselessly. [16:22:38] YuviPanda: Okay, now I see what you did there: You didn't reuse webproxy, but created a new one and refered to that as "the proxy role". [16:22:48] scfc_de: indeed, I now see the confusion :) [16:25:28] hi. what would be better(performance wise) to convert several images to one pdf(from python tool): using convert command or using a specific pdf conversion module.. [16:26:51] rohit-dua: depends on which module you are going to use [16:27:06] rohit-dua: in general, I'd reccomend going with whichever makes it the easiest to write code for, and then modifying later if performance is a problem [16:27:18] !log deployment-prep Changed uid/git for files owned by l10nupdate user [16:27:21] Logged the message, Master [16:28:05] YuviPanda: ok. then i'll just use the convert command from subprocess for now :) [16:28:20] rohit-dua: cool. also check out the python sh module, which is AMAZINGLY better than subprocess :) [16:31:41] YuviPanda: thank you. It seems simpler to write.. and if I'm not wrong the subprocess module itslf calls /bin/sh. [16:32:09] rohit-dua: unsure about what subprocess does, but sh is rather stable and usable :) [16:46:05] bd808: everything squared away? [16:46:28] andrewbogott: thoughts on replacing the general labs proxy with a trusty box? it gets way less traffic [16:47:12] YuviPanda: I don't want to mess with anything until we have a test to demonstrate the problems we had in Zurich [16:47:20] andrewbogott: hmm, alright. [16:47:36] andrewbogott: I am unsure how to test it. hitting a random url seems too simplistic [16:48:11] andrewbogott: a possible approach is to pick out tool-labs logs for about 2h, and then simulate traffic by re-hitting those same URLs against the test proxy [16:48:30] andrewbogott: and I could hit them from localhost in the proxy itself. thoughts? [16:48:37] Yeah, that seems like a good idea -- might be that even simpler things would work like just hammering on one url [16:48:42] Why not just watch the live log for an increase in various status codes? [16:49:14] Coren, I think we did that and didn't learn much. All the problems were intermittent, and nothing happened that wasn't happening (a little bit) during 'correct' performance [16:49:53] Ah. So the signal gets drowned. Annoying. [16:50:19] andrewbogott: Coren yeah, I am just checking logs again, 499s are still happening at roughly the same rate, so kinda a red herring [16:50:44] andrewbogott: Coren so I suppose the thing to check for is client side disconnects. [16:52:16] andrewbogott: salt commands are still running but I think they will finish soon [16:52:25] Otherwise I think it's good [16:52:32] bd808: cool. I need to change venue in a few minutes but will be back in not to long [17:11:01] !log deployment-prep Unwedged the jenkins jobs to updating beta by stopping the stuck db update job [17:11:04] Logged the message, Master [18:15:14] 3Wikimedia Labs / 3deployment-prep (beta): mwdeploy user has shell /bin/bash in labs LDAP and /bin/false in production/Puppet - 10https://bugzilla.wikimedia.org/65591#c1 (10Bryan Davis) p:5Normal>3High Ori has given a -2 to https://gerrit.wikimedia.org/r/134519 due to the need for a work around for this... [18:15:27] bd808|deploy: thanks [18:15:40] sorry to be a dick about it, hope you understand [18:16:04] I totally do, and the cherry-pick keeps things working until we can fix it right [18:35:44] 3Wikimedia Labs / 3tools: Tool Labs: Provide anonymized view of the user_properties table - 10https://bugzilla.wikimedia.org/58196#c45 (10Luis Villa (WMF Legal)) Current status of the bug is "questions have been posed about privacy concerns". It would be helpful to have feedback from people who use the field... [19:01:31] !log deployment-prep deploying Elasticsearch 1.2.1 and some updated plugins to beta [19:01:33] Logged the message, Master [19:17:22] !log local-heritage Fixed the mysqldump and enabled /data/project/heritage/erfgoedbot/populate_image_table.py [19:17:25] Logged the message, Master [19:17:30] (03Abandoned) 10JanZerebecki: Add dhparam file which will be used by at least nginx. [labs/private] - 10https://gerrit.wikimedia.org/r/133066 (owner: 10JanZerebecki) [19:21:02] (03PS1) 10Multichill: Make pretty images and pretty urls [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/137398 [19:21:36] (03CR) 10Multichill: [C: 032 V: 032] "We all like pretty images" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/137398 (owner: 10Multichill) [19:22:41] !local-heritage https://gerrit.wikimedia.org/r/137398 pretty images live, see http://tools.wmflabs.org/heritage/api/api.php?action=images&imcountry=ad&imid=100&format=html&props=img_name [19:23:21] hashar: Mr Jenkins, would you be so kind to also work on my new repo's ;-) [19:23:30] !log local-heritage https://gerrit.wikimedia.org/r/137398 pretty images live, see http://tools.wmflabs.org/heritage/api/api.php?action=images&imcountry=ad&imid=100&format=html&props=img_name [19:23:31] Logged the message, Master [19:24:45] multichill: helllo [19:24:56] multichill: you can probably do most of the work yourself :-]  I will be happy to guide [19:27:02] ah, that's nice, I'll follow up on that. [19:27:05] !log deployment-prep plugins deployed to beta - time to restart Elasticsearch in beta - should cause not interruption of service [19:27:07] Logged the message, Master [19:27:39] !log deployment-prep sorry, can't do that yet, [19:27:42] Logged the message, Master [19:30:26] hashar: Something else you probably know. The database are still utf-8 in latin1 mysql databases right? I ran into a decoding error on commonswiki_p with python [19:33:00] Or does the view hide that?.... [19:33:00] multichill: I think that is a binary field in mysql [19:33:14] multichill: cause mysql did not support all the unicode space [19:33:25] for the labs part I have no clue, that is probably a binary as well [19:33:53] I am not sure how MediaWiki handles wrong unicode. Ideally it would validate it and reject the text revision being submitted but who knows.. [19:34:39] Retrying, would be nice if I can now just set the charset to utf8 and not having to use the horrible latin1 encoding trick [19:35:44] I just read the Toolserver mailing list. My dev/test instance of the monuments api has 15.000 hits a day :P [19:58:50] !log local-heritage Pointed https://commons.wikimedia.org/wiki/Template:Monuments_database_more_images to the api on labs. Was 15K hits on the Toolserver (?!) [19:58:51] Logged the message, Master [20:04:09] !log local-heritage Managed to get the image database updated by switching latin1 -> utf8. Still have to commit. https://commons.wikimedia.org/wiki/Commons:Monuments_database/Indexed_images/Statistics [20:04:11] Logged the message, Master [20:46:03] !log deployment-prep Fixed file ownership on /data/project/apache/uncommon for beta-recompile-math-texvc-eqiad job [20:46:05] Logged the message, Master [20:48:46] valhallasw: trusty has python3.4 packaged by default! [20:48:47] woo [21:22:44] 3Wikimedia Labs / 3tools: tools.wmflabs.org inaccessible via labs instances - 10https://bugzilla.wikimedia.org/54052#c25 (10Marc A. Pelletier) p:5High>3Normal s:5blocke>3normal The actual problem being fixed; bumping down. [21:49:06] YuviPanda: yeah! :-) [22:11:31] hi, do i need to manually sync the privateSettings on deployment-bastion ? [22:12:21] 3Wikimedia Labs / 3tools: Tool Labs: Provide anonymized view of the user_properties table - 10https://bugzilla.wikimedia.org/58196#c46 (10Eran Roz) Hi Luis, thanks for the update. (In reply to Krinkle from comment #40) > > To avoid strange statistical variance like that, I'd recommend making > timestamp of... [22:21:05] Hello guys. [22:21:46] I'm trying to access analytics1010.eqiad.wmnet, but I get and error. I wonder if I have access rights to analytics1010 anyway. how can I check on that? [22:22:17] HaithamS: you should ask in #wikimedia-operations, since that is a prod db [22:23:35] YuviPanda, thanks, makes sense. wanted to check if it's an analytics issue before escalating to operations. [22:23:57] HaithamS: :) #wikimedia-labs doesn't usually have analytics folks much though [22:24:42] no? may be I'm just confusing channels then :) [22:41:53] HaithamS: :)