[04:26:29] <wikibugs>	 10Wikibugs, 5Patch-For-Review: Wikibugs should ignore changes to the security field - https://phabricator.wikimedia.org/T105625#1570016 (10Legoktm) >>! In T105625#1570011, @gerritbot wrote: > Change 233665 had a related patch set uploaded (by Legoktm):  Ignore, wrong bug.
[04:26:37] <wikibugs>	 10Wikibugs: Wikibugs should ignore changes to the security field - https://phabricator.wikimedia.org/T105625#1570017 (10Legoktm)
[08:18:15] <wikibugs>	 10Tool-Labs-tools-Erwin's-tools: mysql library deprecated - https://phabricator.wikimedia.org/T109591#1570351 (10Nemo_bis) 5Open>3Resolved a:3Nemo_bis Thanks, done; things work again. How much time do we have till this bites us again?
[08:36:23] <wikibugs>	 10Tool-Labs-tools-Erwin's-tools: mysql library deprecated - https://phabricator.wikimedia.org/T109591#1570369 (10valhallasw) We are not planning on removing Precise servers until after we provide Jessie servers. As long as you don't run  ``` webservice stop webservice start ```  (i.e. without --release precise)...
[08:51:08] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore - https://phabricator.wikimedia.org/T110052#1570375 (10hashar) 5Open>3Resolved a:3hashar integration-slave-precise-1014 was affected as wel...
[08:51:39] <wikibugs>	 6Labs, 10Labs-Infrastructure, 6operations: disk space on labvirt1007 - https://phabricator.wikimedia.org/T109752#1570391 (10hashar) Seems some disk space has been reclaimed on labvirt1007. I managed to boot the three CI instances that were paused (T110052).
[09:39:33] <valhallasw`cloud>	 YuviPanda: shinken is still not on irc
[09:39:38] <valhallasw`cloud>	 also, CRITICAL for tools-webgrid-generic-1404/Puppet failure
[09:39:48] <YuviPanda>	 oh right
[09:39:51] <YuviPanda>	 let me look at shinken
[09:40:15] <valhallasw`cloud>	 becuse tools.copyvios has 4x uwsgi with ~600M RSS each
[09:40:36] <YuviPanda>	 should we add more nodes?
[09:40:51] <YuviPanda>	 !log shinken restarted ircecho on shinken-01
[09:40:55] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL, Master
[09:40:55] <valhallasw`cloud>	 eh, actually
[09:40:55] <YuviPanda>	 there we go
[09:41:06] <valhallasw`cloud>	 it wasn't even a memory issue like all the other times
[09:41:06] <valhallasw`cloud>	 ESC[1;31mError: Could not retrieve catalog from remote server: Error 400 on SERVER: Reading data from Tools failed: NoMethodError: undefined method `[]' for nil:NilClass at /etc/puppet/manifests/realm.pp:65 on node tools-webgrid-generic-1404.tools.eqiad.wmflabsESC[0m
[09:41:27] <valhallasw`cloud>	 although the server does only have ~480M free
[09:41:43] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-generic-1404 is OK Less than 1.00% above the threshold [0.0]
[09:41:44] <YuviPanda>	 did we write a new node checklist, btw?
[09:41:56] <valhallasw`cloud>	 YuviPanda: yes
[09:42:03] <valhallasw`cloud>	 it's under https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Admin
[09:42:11] <valhallasw`cloud>	 specifically, https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Admin/new_exec_host
[09:42:53] <valhallasw`cloud>	 also, tools-exec-1207.eqiad.wm and tools-exec-1208.eqiad.wm have queues in alarm state due to 9.5 / 10 load
[09:47:07] <YuviPanda>	 valhallasw`cloud: do you think we shouldn't build more nodes?
[09:49:33] <valhallasw`cloud>	 YuviPanda: webgrid-generic-1401 and -1402 both have > 4GB free, -1403 and -1404 only a few hundred megs
[09:49:48] <shinken-wm>	 PROBLEM - Puppet failure on tools-web-static-01 is CRITICAL 50.00% of data above the critical threshold [0.0]
[09:49:50] <YuviPanda>	 yeah, I guess our vmem stuff is a free for all everywhere basically
[09:49:56] <YuviPanda>	 oh what now. that's probably cdnjs
[09:50:14] <valhallasw`cloud>	 YuviPanda: you win!
[09:50:24] <valhallasw`cloud>	 we should make it   git fetch && git reset --hard   instead of git pull
[09:50:29] <valhallasw`cloud>	 because that does all kinds of weird things
[09:51:12] <YuviPanda>	 yeah
[09:51:24] <YuviPanda>	 maybe switch it from a puppet ensure => latest
[09:51:27] <YuviPanda>	 to a cronjob
[09:51:30] <YuviPanda>	 but then we won't get alerts at all
[09:52:50] <YuviPanda>	 valhallasw`cloud: so assuming that'll recover... objections to building new nodes?
[09:52:59] <valhallasw`cloud>	 no, no objections
[09:53:37] <YuviPanda>	 valhallasw`cloud: ok, I'll build maybe 1 more generic node and maybe 2 precises?
[09:53:44] <valhallasw`cloud>	 eeh
[09:54:12] * YuviPanda attempts to interpret the 'eeh'
[09:54:18] <valhallasw`cloud>	 yeah, tools-exec-12* all has load > 1
[09:54:33] <valhallasw`cloud>	 the -14xx have several idle
[09:54:49] <YuviPanda>	 sigh
[09:54:55] <YuviPanda>	 we should find ways to switch the default I guess
[09:55:06] <valhallasw`cloud>	 or just build more precise
[09:55:16] <YuviPanda>	 yeah
[09:55:20] <valhallasw`cloud>	 I think scfc's idea of not forcing people to switch until we have jessie is sane
[09:55:22] <YuviPanda>	 let's do the latter now
[09:55:25] <YuviPanda>	 oh yeah
[09:55:27] <YuviPanda>	 I totally agree
[09:55:40] <YuviPanda>	 which is why we haven't switched yet, I think
[09:55:48] <YuviPanda>	 for webservices it was possible to leave the current precise ones in precise
[09:55:52] <YuviPanda>	 not sure how to do with for job submission
[09:55:57] <YuviPanda>	 so that only 'new' ones default to trusty
[09:56:52] <valhallasw`cloud>	 -14 webgrids are also fairly loaded
[09:56:58] <valhallasw`cloud>	 YuviPanda:    qhost -q -h `qstat -f -xml | grep name | cut -d"@" -f2 | cut -d. -f1 ` | less
[09:57:13] <valhallasw`cloud>	 actually, just      qhost -h `qstat -f -xml | grep name | cut -d"@" -f2 | cut -d. -f1 ` | less
[09:58:18] <valhallasw`cloud>	 (why do exec hsots have 24G swap?)
[09:58:59] <YuviPanda>	 coren set it up that way.
[09:59:19] <YuviPanda>	 !log tools created tools-exec-1220 and -1221
[09:59:23] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[09:59:29] <YuviPanda>	 let me create a couple of webgrid-14s as well
[09:59:50] * valhallasw`cloud goes back to work
[10:00:08] <YuviPanda>	 valhallasw`cloud: thanks for pointing out!
[10:04:34] <YuviPanda>	 valhallasw`cloud: btw, just merged https://gerrit.wikimedia.org/r/#/c/233184/ - per-host hiera overrides no longer need to be in ops/puppet!
[10:04:50] <YuviPanda>	 !log tools apply exec node puppet roles to tools-exec-1220 and -1221
[10:04:50] <valhallasw`cloud>	 YuviPanda: eh, okay
[10:04:53] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[10:05:02] <valhallasw`cloud>	 I like having them in ops/puppet =p
[10:05:05] <YuviPanda>	 :D
[10:05:17] <YuviPanda>	 well, you can put them on wikitech first and then make a patch to ops puppet
[10:05:22] <valhallasw`cloud>	 yeah, fair enough
[10:05:23] <YuviPanda>	 thus requiring you to be not blocked
[10:06:02] <YuviPanda>	 I just restarted apache on puppetmaster for https://gerrit.wikimedia.org/r/#/c/233184/
[10:06:09] <YuviPanda>	 in case there are puppet failures from that
[10:06:18] <YuviPanda>	 man, the exec node puppet runs take forever
[10:16:16] <YuviPanda>	 !log tools created tools-webgrid-generic-1405
[10:16:19] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[10:24:47] <shinken-wm>	 RECOVERY - Puppet failure on tools-web-static-01 is OK Less than 1.00% above the threshold [0.0]
[11:02:53] <YuviPanda>	 !log marathon deleted marathon-puppetmaster since it was dead anyway
[12:05:57] <valhallasw`cloud>	 YuviPanda: yeah, I should fix labslogbot :P
[12:27:06] <JohnFLewis>	 I can't ssh to any labs machines via bastion.wmflabs.org (or to it either). Any ideas?
[12:28:33] <JohnFLewis>	 nevermind - for some reason my ssh key was not being sent :/
[12:36:33] <JohnFLewis>	 YuviPanda: around?
[12:36:47] <JohnFLewis>	 mailman-jessie.wmflabs.org seems to have lost its MX
[12:38:51] <JohnFLewis>	 it would be great if we [I] can get it back because its not useful testing something mail related when you can't send it mail :)
[13:11:27] <andrewbogott>	 YuviPanda: did you do anything about the puppet issues overnight, or did they clear up on their own?
[13:11:37] <wikibugs>	 10Quarry: Show all published queries in profile - https://phabricator.wikimedia.org/T77948#1570952 (10matej_suchanek)
[13:11:55] <YuviPanda>	 andrewbogott: we are adding new nodes...
[13:12:05] <YuviPanda>	 But I haven't done anything else 
[13:19:54] <JohnFLewis>	 andrewbogott / YuviPanda: either of you able to look at the above MX thing? :)
[13:20:18] <YuviPanda>	 JohnFLewis: hmm, it shouldn't have lost anything
[13:20:53] <YuviPanda>	 JohnFLewis: what was the public hostname again?
[13:21:03] <JohnFLewis>	 mailman-jessie.wmflabs.org
[13:26:07] <andrewbogott>	 hashar_: your instances are happy now?
[13:26:25] <hashar_>	 andrewbogott: good morning. Yeah they all booted just fine this morning. Thank you very much!
[13:26:33] <YuviPanda>	 JohnFLewis: ok, I'm looking at it again now
[13:26:39] <andrewbogott>	 great, sorry for the outage
[13:26:51] <hashar_>	 maybe we could ask folks to clean up unused instances?  Seems there is a lot of them left bit rotting
[13:27:18] <JohnFLewis>	 okay thanks
[13:27:37] <andrewbogott>	 hashar_: Yeah, it might be time for a nag.  Labs as a whole isn’t full, though, it’s just the scheduler being dumb.  I tweaked it a bit yesterday, hopefully it’ll distribute disk usage a bit better.
[13:28:07] <YuviPanda>	 JohnFLewis: I think we only added an MX record for mailman-three.wmflabs.org
[13:28:12] <YuviPanda>	 JohnFLewis: and I see that record still exists
[13:28:46] <JohnFLewis>	 YuviPanda: indeed but -jessie was the original one which was done because mailman has mail stored :)
[13:28:53] * JohnFLewis finds tickets as I think there was one
[13:29:13] <JohnFLewis>	 https://phabricator.wikimedia.org/T91576
[13:29:53] <YuviPanda>	 ah, interesting. not sure how Coren added the record...
[13:29:56] <YuviPanda>	 anyway let me add one now
[13:30:11] <YuviPanda>	 JohnFLewis: what's the IP? 208.80.155.206?
[13:30:19] <JohnFLewis>	 yep
[13:31:38] <YuviPanda>	 JohnFLewis: try now?
[13:33:42] <JohnFLewis>	 YuviPanda: record returns, thanks :)
[13:33:50] <YuviPanda>	 JohnFLewis: yw!
[13:36:17] <godog>	 FYI I'm about to provision a temporary xlarge for some cassandra tests, I'm assuming openstack will just refuse if that's not ok?
[13:37:00] <YuviPanda>	 theoretically :)
[13:37:49] <godog>	 hehe
[13:38:43] <godog>	 indeed large went through, xlarge didn't
[13:39:09] <YuviPanda>	 godog: it might be you're out of quota too
[13:39:41] <godog>	 true that, this is the "monitoring" project, a large will do for now I think anyways
[13:51:59] <YuviPanda>	 godog: cool!
[13:53:52] <Niharika>	 sitic: Hi! I was wondering - are there plans to move crosswatch from it's current labs project or was that the intended location for the watchlist?  
[13:56:33] <YuviPanda>	 Niharika: to where?
[13:56:39] <YuviPanda>	 Niharika: to production?
[13:56:39] <sitic>	 Niharika: hey, not really, http://tools.wmflabs.org/crosswatch/ is the place it's supposed to be (move it where?)
[13:57:13] <Niharika>	 YuviPanda: Yes, production, in case the this was just the demo server. 
[13:58:47] <Niharika>	 sitic: Cool, thanks! 
[13:59:24] <sitic>	 :-)
[14:01:07] <hashar>	 !log meetbot restarted bot according to https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Tools/meetbot#Administration
[14:01:07] <labs-morebots>	 meetbot is not a valid project.
[14:13:30] <Niharika>	 YuviPanda: I created a new tool and want to apt-get some packages on it. It asks for a password - which password is that? Is there a default password? (The one in replica.my.cnf doesn't work)
[14:13:59] <YuviPanda>	 ah
[14:14:24] <YuviPanda>	 Niharika: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs#My_tool_requires_a_package_that_is_not_currently_installed_in_Tool_Labs._How_can_I_add_it.3F
[14:15:08] <Niharika>	 YuviPanda: Oops, missed that. Thanks.
[14:18:13] <YuviPanda>	 !log tools pooled in tools-webgrid-generic-1405
[14:18:16] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[14:18:21] <YuviPanda>	 valhallasw`cloud: btw, tools-exec-1201 seems to be empty all the time
[14:18:34] <valhallasw`cloud>	 YuviPanda: maybe still the queue/dns thing?
[14:18:45] <valhallasw`cloud>	 don't have the phab T by hand
[14:18:55] <YuviPanda>	 the big /etc hosts thing or the aliases thing?
[14:19:04] <valhallasw`cloud>	 aliases (.tools.eqiad vs .eqiad)
[14:19:26] <YuviPanda>	 1201 should've been created way before that happend so it should've been handled
[14:19:30] <YuviPanda>	 but I'll check the aliases file
[14:20:14] <YuviPanda>	 valhallasw`cloud: nope it exists in the aliases file
[14:20:23] <valhallasw`cloud>	 not the point
[14:20:32] <valhallasw`cloud>	 there was a mixum between hostname and queue name
[14:22:05] <YuviPanda>	 oh?
[14:22:09] <YuviPanda>	 I wasn't aware of this...
[14:24:47] <andrewbogott>	 !log staging moving staging-restbase10 to labvirt1009
[14:24:49] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Staging/SAL, dummy
[14:26:13] <YuviPanda>	 !log tools rebooting tools-exec-1220 because NFS wedge...
[14:26:17] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[14:27:58] <addshore>	 YuviPanda: /valhallasw`cloud  i have a QU for you ;)
[14:28:12] <addshore>	 When using the grid engine, parameters with special chars (eg umlauts in wiki page names) get mangled (converted to 2-char iso-latin garbage). Is there any way around that?
[14:28:23] <valhallasw`cloud>	 {{moreinfoneeded}}
[14:28:57] <valhallasw`cloud>	 SGE blissfully ignores encodings and just handles bytes
[14:29:05] <gifti>	 my way is not to use any "special" characters
[14:32:45] <andrewbogott>	 !log staging moving staging-restbase06 to labvirt1009
[14:32:48] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Staging/SAL, dummy
[14:32:49] <valhallasw`cloud>	 addshore: I wouldn't know how you would get utf-8-interpreted-as-latin-1 in any case
[14:33:43] <addshore>	 also valhallasw`cloud is there a way to make pywikibot libs not ask you for a password every time?
[14:34:12] <addshore>	 oh wait, i gues a password can be stored in user-config.py ? (Im just reading the docs for someone) ;)
[14:35:41] <valhallasw`cloud>	 ashley: passwordfile
[14:35:44] <valhallasw`cloud>	 eh, addshore 
[14:36:04] <valhallasw`cloud>	 addshore: oh, you're talking about pywikibot specifically? set console_encoding='utf-8'
[14:36:57] <addshore>	 valhallasw`cloud: where is the docs for passwordfile ? :P
[14:37:04] <valhallasw`cloud>	 addshore: pywikibot/config2.py
[14:37:10] <valhallasw`cloud>	 all config docs are there :>
[14:37:14] <valhallasw`cloud>	 also maybe doc.wm.o
[14:37:19] <valhallasw`cloud>	 but proooobably not
[14:37:23] <addshore>	 ahh, im looking on mw.org
[14:38:46] <valhallasw`cloud>	 addshore: it's also there, hidden in a cupboard, guarded by a dragon: https://www.mediawiki.org/wiki/Manual:Pywikibot/Use_on_third-party_wikis#Bot_doesn.27t_want_to_stay_logged_in
[14:38:56] <addshore>	 there be dragons
[14:42:23] <andrewbogott>	 !log deployment-prep moving deployment-cache-mobile04 to labvirt1004
[14:42:26] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, dummy
[14:42:46] <addshore>	 cheers for the pokes in the right direction valhallasw`cloud :) (and gifti )!
[14:43:27] <wikibugs>	 10Wikibugs, 3Collaboration-Team-Current: Add Collaboration-Team-Current to #wikimedia-collaboration - https://phabricator.wikimedia.org/T110186#1571135 (10Mattflaschen) 3NEW
[14:45:05] <YuviPanda>	 !log tools reboot tools-exec-1221
[14:45:07] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[14:55:59] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure, 6operations: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN - https://phabricator.wikimedia.org/T92351#1571173 (10Krinkle)
[14:56:02] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: Reenable resolv.conf ndots:2 option on CI instances - https://phabricator.wikimedia.org/T105297#1571171 (10Krinkle) 5Open>3declined It seems this is obsolete with the new DNS infrastructure.  See T92351 for details. I've undeployed ou...
[14:58:13] <YuviPanda>	 !log tools pooled in two new instances for the precise exec pool
[14:58:16] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[14:58:19] <YuviPanda>	 valhallasw`cloud: ^
[14:58:27] <valhallasw`cloud>	 \o/
[14:58:29] <YuviPanda>	 am going to edit the page withe special instructions for precise
[14:58:35] <YuviPanda>	 since there's this swap thing that requires a reboot
[14:58:41] <valhallasw`cloud>	 yes please
[14:59:43] <valhallasw`cloud>	 YuviPanda: could you also take a look at the python puppet patch?
[15:00:25] <YuviPanda>	 valhallasw`cloud: yup, right after editing
[15:00:26] <valhallasw`cloud>	 there's probably some things that could be better in actually communicating the result (I'm not too happy with the html file)
[15:00:43] <valhallasw`cloud>	 and I'm wondering whether we should maybe only install python3 packages on trusty
[15:01:43] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore - https://phabricator.wikimedia.org/T110052#1571184 (10hashar) 5Resolved>3Open So apparently some git repos in Jenkins workspace ended up bei...
[15:01:53] <YuviPanda>	 valhallasw`cloud: ok, edited
[15:02:39] <valhallasw`cloud>	 also maybe I should add the packaged version instead of just an 'x'?
[15:03:00] <_joe_>	 valhallasw`cloud: you FOOL!
[15:03:02] <_joe_>	 :P
[15:03:13] <_joe_>	 python3 is a tool of EVIL :)
[15:03:18] <YuviPanda>	 yeah, perl6 instead
[15:03:24] <_joe_>	 lol
[15:03:24] <YuviPanda>	 valhallasw`cloud: I'm looking at the patch now
[15:03:29] <_joe_>	 no, php7 instead
[15:03:31] <valhallasw`cloud>	 yes, let me install perl6-mwparserfromhell for you
[15:03:40] <valhallasw`cloud>	 and haskell-awesome
[15:03:50] <valhallasw`cloud>	 because haskell is basically wikipedia with its cabal
[15:03:57] * _joe_ puts an alias in valhallasw`cloud shell for python='/usr/bin/java'
[15:04:35] <YuviPanda>	 ok, both exec-1220 and 1221 are getting new tasks now
[15:04:37] <YuviPanda>	 so that's good
[15:05:00] <valhallasw`cloud>	 I was going to say '/usr/bin/java: command not found', but I got '/usr/bin/python: can't find '__main__' module in '/usr/share/command-not-found'' instead.
[15:05:35] <YuviPanda>	 hahahaha :P
[15:05:59] <YuviPanda>	 valhallasw`cloud: wow, is that really all the packages we have in python?
[15:06:07] <andrewbogott>	 !log staging moving staging-restbase05 and 09 to labvirt1004
[15:06:09] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Staging/SAL, dummy
[15:06:12] <valhallasw`cloud>	 YuviPanda: except for the home-packaged ones
[15:06:19] <YuviPanda>	 right
[15:06:22] <valhallasw`cloud>	 YuviPanda: I'd also happily just install python-* and python3-* :-p
[15:06:26] <YuviPanda>	 heh
[15:06:29] <YuviPanda>	 virtualenvs!
[15:06:34] <YuviPanda>	 virtualenvs for everyone really
[15:06:41] <valhallasw`cloud>	 yes, please go compile scipy
[15:06:43] <valhallasw`cloud>	 :P
[15:06:46] <YuviPanda>	 wel
[15:06:51] <valhallasw`cloud>	 it might work if we provide wheels
[15:06:53] <YuviPanda>	 except for anything involving compilation
[15:08:49] <YuviPanda>	 PuppetSWAT in a few hours!
[15:09:24] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore - https://phabricator.wikimedia.org/T110052#1571198 (10hashar) integration-slave-trusty-1014 on boot reports:      /bin/sh: 1: exec: cloud-init:...
[15:19:10] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore - https://phabricator.wikimedia.org/T110052#1571237 (10hashar) I have poked the #releng team mailing list to pair the rebuild with someone.
[16:35:07] <wikibugs>	 6Labs, 5Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#1571723 (10scfc)
[16:35:09] <wikibugs>	 6Labs, 3Labs-Sprint-104, 3Labs-Sprint-105, 5Patch-For-Review, 7Puppet: Allow per-host hiera overrides via wikitech - https://phabricator.wikimedia.org/T104202#1571721 (10scfc) 5Open>3Resolved Verified by temporarily setting `"ssh::server::explicit_macs": true` at [[https://wikitech.wikimedia.org/wiki...
[16:48:42] <ebernhardson>	 must have been some blip...working now
[16:49:03] <YuviPanda>	 ebernhardson: which blip?
[16:55:10] <ebernhardson>	 YuviPanda: couldn't ssh into labs
[16:55:16] <YuviPanda>	 ebernhardson: ah ok
[17:13:17] <lbertolotti>	 good day
[17:13:47] <lbertolotti>	 my crontab jobs are not being submited
[17:13:57] <lbertolotti>	 is there anything I can do?
[17:14:09] <YuviPanda>	 lbertolotti: hey! can you give us an example?
[17:14:38] <lbertolotti>	 first of all
[17:15:10] <lbertolotti>	 someone here told me to write like this
[17:15:11] <lbertolotti>	  45 19 * * * jsub python bot.py
[17:15:25] <lbertolotti>	 say a job being submited at 19:45 utc
[17:15:29] <lbertolotti>	 for example
[17:17:00] <lbertolotti>	 But here
[17:17:01] <lbertolotti>	 https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Grid#Specifying_time_zones
[17:17:01] <lbertolotti>	 is a bit different
[17:24:45] <lbertolotti>	 also is there a command at toolabs to view utc time?
[17:26:31] <mutante>	 lbertolotti: what does this output?  date +%Z 
[17:27:47] <mutante>	 lbertolotti: zdump UTC
[17:29:07] <lbertolotti>	 yes UTC
[17:29:30] <mutante>	  zdump UTC
[17:29:30] <mutante>	 UTC  Tue Aug 25 17:27:42 2015 UTC
[17:29:31] <lbertolotti>	 tools.portalbox@tools-bastion-01:~$ date +%Z  UTC
[17:29:45] <mutante>	 lbertolotti: then it's just "date"
[17:29:46] <lbertolotti>	 tools.portalbox@tools-bastion-01:~$ zdump UTC UTC  Tue Aug 25 17:29:38 2015 UTC tools.portalbox@tools-bastion-01:~$ 
[17:29:56] <mutante>	 there you go, or that
[17:30:14] <lbertolotti>	 ok
[17:30:30] <lbertolotti>	 can I add this to the wikitech page?
[17:30:35] <mutante>	 of course
[17:30:48] <mutante>	 please do
[17:32:47] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Nick wintr was created, changed by Nick wintr link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Nick_wintr edit summary: Created page with "{{Tools Access Request |Justification=I'm a developer on the Wiki Education Foundation Dashboard, and will  be a maintainer of the wikiedudashboard tool. |Completed=false |Use..."
[17:33:42] <lbertolotti>	 ok, done
[17:35:07] <lbertolotti>	 ok, so I scheduled
[17:35:22] <lbertolotti>	 # 34 17 * * * jsub python bot.py
[17:35:27] <lbertolotti>	 at crontab -e
[17:35:43] <lbertolotti>	 but nothing happened
[17:36:05] <lbertolotti>	 even if 
[17:36:09] <lbertolotti>	 tools.portalbox@tools-bastion-01:~$ python bot.py
[17:36:12] <lbertolotti>	 works fine
[17:36:14] <lbertolotti>	 why?
[17:36:41] <lbertolotti>	 tools.portalbox@tools-bastion-01:~$ date Ter Ago 25 17:36:32 UTC 2015
[17:37:51] <mutante>	 lbertolotti: maybe because it doesnt run as the same user / in the same environment when started from cron
[17:38:16] <mutante>	 try using full pathes and send the output to a log file
[17:38:22] <mutante>	 then check the log later
[17:41:19] <lbertolotti>	 is there a command to view job submission history
[17:42:08] <lbertolotti>	 qstat only list current jobs
[17:45:57] <lbertolotti>	 ?
[17:46:07] <wikibugs>	 10Wikibugs, 3Collaboration-Team-Current: Add Collaboration-Team-Current to #wikimedia-collaboration - https://phabricator.wikimedia.org/T110186#1572476 (10Quiddity)
[17:59:54] <valhallasw`cloud>	 lbertolotti: qacct
[18:00:43] <valhallasw`cloud>	 lbertolotti: or, more effectively, tail -n 1000 /data/project/.system/accounting | grep -e 'tool.portalbox'
[18:01:28] <grrrit-wm>	 (03PS1) 10Legoktm: Send Collaboration-Team(-.*)? to #wikimedia-collaboration [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/233761 (https://phabricator.wikimedia.org/T110186) 
[18:02:22] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Send Collaboration-Team(-.*)? to #wikimedia-collaboration [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/233761 (https://phabricator.wikimedia.org/T110186) (owner: 10Legoktm)
[18:02:39] <grrrit-wm>	 (03Merged) 10jenkins-bot: Send Collaboration-Team(-.*)? to #wikimedia-collaboration [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/233761 (https://phabricator.wikimedia.org/T110186) (owner: 10Legoktm)
[18:03:58] <lbertolotti>	 tools.portalbox@tools-bastion-01:~$ qacct Total System Usage     WALLCLOCK         UTIME         STIME           CPU             MEMORY                 IO                IOW ================================================================================================================   16691904594 365217448.056 55380721156.580 56358711742.416   137700947049.953        8002822.094              0.000
[18:04:17] <wikibugs>	 !log tools.wikibugs Updated channels.yaml to: a541227fe36479f99a74f8d56586bcf4b8f55108 Send Collaboration-Team(-.*)? to #wikimedia-collaboration
[18:04:20] <wikibugs>	 10Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1572558 (10Legoktm) 1
[18:04:20] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master
[18:04:40] <valhallasw`cloud>	 lbertolotti: man qacct to see options
[18:04:53] <wikibugs>	 10Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1023025 (10Legoktm)
[18:05:01] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Nick wintr was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=175010 edit summary: 
[18:05:20] <wikibugs>	 10Wikibugs, 3Collaboration-Team-Current, 5Patch-For-Review: Add Collaboration-Team-Current to #wikimedia-collaboration - https://phabricator.wikimedia.org/T110186#1572562 (10Legoktm) 5Open>3Resolved a:3Legoktm [11:04:18] <wikibugs> !log tools.wikibugs Updated channels.yaml to: a541227fe36479f99a74f8d56...
[18:11:16] <YuviPanda>	 !log wikilabels restarted uwsgi server on wikilabels-01
[18:11:19] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL, Master
[18:12:47] <lbertolotti>	 tools.portalbox@tools-bastion-01:~$ qacct -j bot.py error: job name bot.py not found
[18:20:42] <valhallasw`cloud>	 lbertolotti: as far as I can see, no jobs have been submitted by tools.portalbox
[18:25:07] <lbertolotti>	 indeed that's what I thought
[18:25:28] <lbertolotti>	 how could that happen
[18:25:35] <lbertolotti>	 it's strange
[18:25:48] <lbertolotti>	 since it was running fine the beggining of the year
[18:26:06] <valhallasw`cloud>	 YuviPanda: packages.ubuntu.com is not available over https :(
[18:26:22] <lbertolotti>	 https://en.wikipedia.org/w/index.php?title=Portal:Business_and_economics/Market_Indices&action=history
[18:26:47] <valhallasw`cloud>	 YuviPanda: and I'll jinja All The Things, yes, good idea.
[18:27:35] <YuviPanda>	 valhallasw`cloud: boo, and yay
[18:28:48] <valhallasw`cloud>	 YuviPanda: ! and I just noticed there are wheel packages in ubuntu!
[18:28:51] <valhallasw`cloud>	 e.g. http://packages.ubuntu.com/trusty-updates/all/python-requests-whl/filelist
[18:28:56] <valhallasw`cloud>	 let me also add those :-)
[18:30:24] <valhallasw`cloud>	 not sure how to use them though...
[18:31:18] <YuviPanda>	 I still don't understand wheels properly
[18:31:23] <YuviPanda>	 mostly on account of not having looked
[18:31:31] <valhallasw`cloud>	 wheels are just zips of compiled packages
[18:31:38] <valhallasw`cloud>	 basically a zip of site-packages/packagename
[18:32:26] <valhallasw`cloud>	 oh, but
[18:32:27] <valhallasw`cloud>	 "A very limited set of wheel packages are available in the archive, but these support the narrow purpose of enabling the pip tool, in a Debian policy compliant way."
[18:32:41] <valhallasw`cloud>	 so it's irrelevant
[18:56:57] <wikibugs>	 6Labs, 10Labs-Infrastructure, 3Labs-Sprint-111, 5Patch-For-Review: Support cold-migration or suspended migration, or something, between labvirt hosts - https://phabricator.wikimedia.org/T109902#1572709 (10Andrew) ok, ~root/cold-migrate now works.  It's stupid since it copies every file twice, but I've migr...
[18:58:09] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore, ended up corrupted. Need rebuild - https://phabricator.wikimedia.org/T110052#1572710 (10hashar)
[18:59:00] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore, ended up corrupted. Need rebuild - https://phabricator.wikimedia.org/T110052#1567858 (10hashar)
[18:59:25] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore, ended up corrupted. Need rebuild - https://phabricator.wikimedia.org/T110052#1567858 (10hashar)
[19:17:06] <valhallasw`cloud>	 YuviPanda: bah, jinja whitespace handling is impossible
[20:16:35] <valhallasw`cloud>	 YuviPanda: https://wikitech.wikimedia.org/wiki/User:Merlijn_van_Deen/test :>
[21:04:49] <grrrit-wm>	 (03CR) 10Jean-Frédéric: [C: 032 V: 032] Support for Wikidata while guessing categories from CommonsCat [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/233340 (https://phabricator.wikimedia.org/T110003) (owner: 10Jean-Frédéric)
[21:22:14] <ankita-ks>	 YuviPanda : around?
[21:22:57] <ankita-ks>	 I setup mediawiki-vagrant inside the labs instance with the role visual editor enabled 
[21:23:21] <ankita-ks>	 i need to setup a new visual editor extension that I created. 
[21:24:07] <ankita-ks>	 I cloned the extension in extensions folder and added "wfLoadExtension( 'LanguageTool' );" to the LocalSettings.php
[21:24:14] <ankita-ks>	 Then I ran a vagrant provision 
[21:24:25] <ankita-ks>	 But I can not see the extension when i go to the url
[21:24:54] <ankita-ks>	 Also, there are two LocalSettings.php, one in /srv/mediawiki-vagrant/
[21:25:10] <ankita-ks>	 and another in /serv/mediawiki-vagrant/mediawiki/
[21:25:27] <ankita-ks>	 which one do i alter? (If at all that is the issue)
[21:28:16] <ankita-ks>	 I have tried both btw, and nothing happened. :/
[21:34:42] <ankita-ks>	 bd808 : around?
[21:35:22] <bd808>	 ankita-ks: 0/ reading backscroll
[21:36:51] <bd808>	 ankita-ks: the LocalSettings.php in /srv/mediawiki-vagrant is the "real" one
[21:37:20] <bd808>	 the one in the mediawiki folder is the stub that is created by the installer. we add a bit to the end of it to read the other
[21:37:32] <ankita-ks>	 yes, i noticed that.
[21:37:55] <bd808>	 a slightly easier way to add an extension (or other block of config) is to make a file in /srv/mediawiki-vagrant/settings.d
[21:37:58] <ankita-ks>	 but even when I load the extension from the "real" LocalSettings.php, it does not seem to work
[21:38:32] <bd808>	 Are you seeing any errors in the logs or just the lack of your extension's features?
[21:40:00] <ankita-ks>	 Just the lack of extension's feature. The button that should get added to the toolBar does not get added
[21:42:41] <bd808>	 ankita-ks: and your extension is using an extension.json file to describe how to load it right?
[21:42:47] <ankita-ks>	 yes
[21:46:19] <bd808>	 hmmm... I'm not sure where to start trying to debug this. You could make sure that the LocalSettings.php you are editing is doing *something* by adding an echo or something to it and reloading your Main_Page
[21:46:33] <bd808>	 changes should be real-time, no provision needed
[21:47:11] <ankita-ks>	 okay, trying it out
[23:56:33] <ashwinpp>	 I have a question regarding running Python2 uwsgi (Flask). I want to return a json response. 
[23:56:41] <ashwinpp>	 return Response("Test",  content_type='text/plain') works fine
[23:56:56] <ashwinpp>	 However jsonify doesn't
[23:57:16] <ashwinpp>	 jsonify works fine locally, but not when using webservice2
[23:57:31] <ashwinpp>	 Any clue why this would be the case?
[23:58:17] <ashwinpp>	 It is not a problem with url routing