[02:50:42] <gry>	 hi, i am trying to install local::lib on tool labs
[02:50:54] <gry>	 cpanm --local-lib=~/perl5 local::lib && eval $(perl -I ~/perl5/lib/perl5/ -Mlocal::lib)
[02:51:24] <gry>	 i get this: http://dpaste.com/2BS9WG7.txt
[02:51:35] <gry>	 this is only the issue after precise migration.. what do i do from there ?
[06:15:32] <wikibugs_>	 (03CR) 10BryanDavis: [C: 04-1] Add rewritten crontab in Python (033 comments) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/336998 (https://phabricator.wikimedia.org/T156174) (owner: 10Zhuyifei1999)
[06:22:45] <bd808>	 gry: looks like we need to know what happened in /data/project/gpy/.cpanm/work/1490928614.25875/build.log that kept MakeMaker from building
[06:26:47] <bd808>	 "undefined symbol: Perl_gv_init at /usr/share/perl/5.18/XSLoader.pm line 68"
[07:10:19] <wikibugs_>	 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Add a server-side caching service for the new XTools - https://phabricator.wikimedia.org/T161057#3145757 (10Samwilson) PR for review: https://github.com/x-tools/xtools-rebirth/pull/14
[07:22:54] <wikibugs>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Convince nova-scheduler to pay attention to CPU metrics - https://phabricator.wikimedia.org/T161006#3145791 (10hashar) Thanks a ton Andrew!  antoine-approve
[09:25:38] <shinken-wm>	 PROBLEM - High iowait on tools-exec-1402 is CRITICAL: CRITICAL: tools.tools-exec-1402.cpu.total.iowait (>44.44%)
[09:35:39] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1402 is OK: OK: All targets OK
[10:13:21] <shinken-wm>	 PROBLEM - High iowait on tools-exec-1409 is CRITICAL: CRITICAL: tools.tools-exec-1409.cpu.total.iowait (>11.11%)
[10:33:22] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1409 is OK: OK: All targets OK
[11:15:08] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Leung Chung-ming was created, changed by Leung Chung-ming link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Leung_Chung-ming edit summary: Created page with "{{Tools Access Request |Justification=To create an interactive information counter for Chinese Wikipedia users based on HTML5. The tool is intended to contain frequently asked..."
[11:35:39] <wikibugs>	 06Labs, 10Labs-Vagrant, 10MediaWiki-Vagrant: labs-vagrant vagrant up fails with 404 at Dropbox - https://phabricator.wikimedia.org/T161891#3146355 (10Nemo_bis)
[12:17:34] <wikibugs_>	 06Labs, 10Labs-Vagrant, 10MediaWiki-Vagrant: labs-vagrant vagrant up fails with 404 at Dropbox - https://phabricator.wikimedia.org/T161891#3146424 (10Nemo_bis)
[13:32:24] <andrewbogott>	 !log analytics deleting shutdown Precise instance limn1
[13:32:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Analytics/SAL
[13:33:49] <andrewbogott>	 !log editor-engagement deleting shutdown Precise instance mwui
[13:33:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Editor-engagement/SAL
[13:34:23] <chasemp>	 andrewbogott: already on the war path? :)
[13:34:40] <andrewbogott>	 !log maps deleting shutdown Precise instance maps-tiles1
[13:34:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Maps/SAL
[13:34:44] <andrewbogott>	 chasemp: yep!
[13:34:55] <chasemp>	 nice man, thanks
[13:35:46] <andrewbogott>	 !log mediahandler-tests deleting shutoff precise instance mediahandler-tests-static
[13:35:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mediahandler-tests/SAL
[13:36:26] <andrewbogott>	 !log openstack deleting shutdown precise instance labs-vmbuilder-precise
[13:36:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL
[13:37:03] <andrewbogott>	 !log otrs deleting shutoff precise instance otrs-test2
[13:37:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Otrs/SAL
[13:37:39] <andrewbogott>	 !log signwriting deleting shutoff precise instance signwriting-ase-wiki
[13:37:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Signwriting/SAL
[13:38:16] <andrewbogott>	 !log visualeditor deleting shutoff precise instance 'towtruck'
[13:38:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Visualeditor/SAL
[13:38:45] <andrewbogott>	 !log wikidata-dev deleting shutoff precise instance wikidata-wdq-mm
[13:38:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL
[13:39:25] <chasemp>	 andrewbogott: I'm hopping in a hangout, I can help mop up in a bit but if you power through it all so much teh better and thanks again
[13:39:36] <andrewbogott>	 I think I can do them all, just a few left
[13:39:55] <andrewbogott>	 !log wikisource-dev deleting shutoff precise instance 'wikisource-dev'
[13:39:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikisource-dev/SAL
[13:40:39] <andrewbogott>	 !log wikisource-tools deleting shutoff precise instance 'wsexport'
[13:40:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikisource-tools/SAL
[13:41:06] <andrewbogott>	 !log wikistream deleting shutoff precise instance 'wikistream-web'
[13:41:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistream/SAL
[13:41:45] <andrewbogott>	 !log wildcat deleting shutoff precise instance 'dannyb'
[13:41:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wildcat/SAL
[13:42:21] <andrewbogott>	 !log wlmjudging deleting shutoff precise instances 'wlm-mysql-master' and 'wlm-apache1'
[13:42:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wlmjudging/SAL
[13:48:15] <wikibugs>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Deprecate precise instances in Labs by 2017-03-31 - https://phabricator.wikimedia.org/T143349#3146576 (10Andrew) 05Open>03Resolved a:03Andrew I've just removed all of the shutdown Precise instances.  A few instances remain that have Precise base images...
[13:59:49] <wikibugs>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Deprecate precise instances in Labs by 2017-03-31 - https://phabricator.wikimedia.org/T143349#3146592 (10hashar) 🥂🍾 ==
[14:02:17] <wikibugs_>	 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3146611 (10jcrespo) a:03jcrespo I am kickstarting the replication right now- this required different pupetization of the repliation "grants". I wi...
[14:22:48] <wikibugs>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Deprecate precise instances in Labs by 2017-03-31 - https://phabricator.wikimedia.org/T143349#3146637 (10chasemp)
[14:23:27] <wikibugs_>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Deprecate precise instances in Labs by 2017-03-31 - https://phabricator.wikimedia.org/T143349#3146642 (10Andrew)
[14:23:29] <wikibugs>	 06Labs, 10Labs-Infrastructure: Ensure that precise instance creation is disabled everywhere - https://phabricator.wikimedia.org/T143359#3146639 (10Andrew) 05Open>03Resolved a:03Andrew I checked again and can't find any active precise base images.
[14:23:37] <wikibugs_>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Deprecate precise instances in Labs by 2017-03-31 - https://phabricator.wikimedia.org/T143349#3146644 (10chasemp)
[14:27:59] <wikibugs_>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Deprecate precise instances in Labs by 2017-03-31 - https://phabricator.wikimedia.org/T143349#3146651 (10chasemp)
[14:28:01] <wikibugs>	 06Labs, 13Patch-For-Review: Disable multiarch support in all Labs precise instances - https://phabricator.wikimedia.org/T111760#3146649 (10chasemp) 05Open>03Resolved a:03chasemp
[14:33:13] <wikibugs_>	 06Labs: iowait alerts for grid engine nodes - https://phabricator.wikimedia.org/T161898#3146695 (10Andrew) p:05Triage>03Normal
[14:38:18] <wikibugs>	 06Labs, 10Labs-Vagrant, 10MediaWiki-Vagrant: labs-vagrant vagrant up fails with 404 at Dropbox - https://phabricator.wikimedia.org/T161891#3146714 (10Reedy) Looks like @ori was hosting this...
[14:38:20] <wikibugs_>	 06Labs, 10Labs-Vagrant, 10MediaWiki-Vagrant: labs-vagrant vagrant up fails with 404 at Dropbox - https://phabricator.wikimedia.org/T161891#3146716 (10Reedy) Looks like @ori was hosting this...
[14:44:32] <wikibugs_>	 06Labs: iowait alerts for grid engine nodes - https://phabricator.wikimedia.org/T161898#3146683 (10chasemp) I don't have anything definite and in spot checking I don't see the issue now unfortunately.    Without catching this in action it is hard to know and it could be one tool floating around causing issues ev...
[15:08:45] <wikibugs>	 06Labs, 10Labs-Infrastructure: Precise instances say "ImportError: No module named cc_power_state_change" on startup - https://phabricator.wikimedia.org/T103808#3146733 (10MoritzMuehlenhoff) 05Open>03Invalid precise is gone
[15:20:03] <wikibugs_>	 06Labs, 06Operations: Investigate ceasing new Trusty instance creation in Labs - https://phabricator.wikimedia.org/T161899#3146739 (10chasemp)
[15:26:52] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#3146758 (10jcrespo) 05Open>03Resolved a:03jcrespo This is done. Some small follups (not related to jessie) at: T157359
[16:17:26] <wikibugs_>	 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3146925 (10jcrespo) Ok, now puppet works, but either it puppet needs more work or it fails silently-this needs more researech. Replication is not wo...
[16:58:01] <mhashemirc>	 oh man, something's not right on labs: ImportError: No module named datetime
[16:58:29] <mhashemirc>	 i'm on vacation, but i think i have just enough time to phab it
[16:59:19] <andrewbogott>	 mhashemirc: 'on labs'?
[17:01:26] <mhashemirc>	 hehe tools.wmflabs to be more specific
[17:06:10] <wikibugs_>	 06Labs, 10Tool-Labs: Python environment weirdness on labs - https://phabricator.wikimedia.org/T161915#3147094 (10mahmoud)
[17:21:52] * madhuvishy looks
[17:26:57] <madhuvishy>	 mhashemirc: how are you running this? webservices on tools expect things to be in a specific directory structure - and i don't think this is set up that way
[17:27:42] <madhuvishy>	 datetime is definitely available
[17:27:46] <madhuvishy>	 https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web/Kubernetes#python_.28uwsgi_.2B_python3.4.29
[17:28:39] <madhuvishy>	 although your tool is currently running on grid engine
[17:44:41] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1016 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[17:47:28] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1017 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[17:47:45] <yuvipanda>	 uh oh
[17:47:46] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1028 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[17:47:58] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1020 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[17:48:16] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1009 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[17:49:00] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1012 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[17:49:20] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1006 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[17:49:28] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1018 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[17:49:48] <yuvipanda>	 I disabled puppet on all workers
[17:50:26] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1011 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[17:51:12] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1021 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[17:53:41] <yuvipanda>	 huzzah, fixed
[17:55:17] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[18:05:18] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:09:04] <wikibugs>	 06Labs, 10Tool-Labs: Python environment weirdness on labs - https://phabricator.wikimedia.org/T161915#3147249 (10madhuvishy) @mahmoud Webservices in tools need to be set up in a predetermined directory structure. You also don't need to define your own wsgi server, since the webservice abstraction already takes...
[18:11:13] <wikibugs>	 06Labs, 10Tool-Labs: Python environment weirdness on labs - https://phabricator.wikimedia.org/T161915#3147259 (10madhuvishy) 05Open>03Invalid Closing as invalid as the problem wasn't on the python environment end.
[18:22:28] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1017 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:23:18] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1009 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:24:28] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1018 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:24:43] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:25:27] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1011 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:27:47] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1028 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:27:57] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:28:59] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1012 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:29:21] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1006 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:31:11] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:44:37] <wikibugs_>	 06Labs, 10Tool-Labs: Virtualenvs slow on tool labs NFS - https://phabricator.wikimedia.org/T136712#3147347 (10madhuvishy) 05Open>03Resolved With lookupcache on across tools, this should be much faster, i've run some of the time commands listed here on this thread, and found at least 2-3 times speed up.
[19:43:29] <wikibugs>	 10Labs-project-other: Successful pilot of Discourse on https://discourse.wmflabs.org/ as an alternative to wikimedia-l mailinglist - https://phabricator.wikimedia.org/T124690#1962887 (10fantasticfears) I contributes to Discourse features continuously as well as an Wikimedia User Group China. If you'd like some c...
[20:32:16] <wikibugs_>	 06Labs, 10Tool-Labs: Python environment weirdness on labs - https://phabricator.wikimedia.org/T161915#3147094 (10valhallasw) For future reference, this error is seen when a virtualenv is created on Precise and then used on Trusty (or vice versa), as the Python versions (or glibc versions, or... something else)...
[21:26:52] <wikibugs>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Audit use of 'novaadmin' within keystone and ldap - https://phabricator.wikimedia.org/T158650#3147694 (10Andrew)
[22:23:55] <wikibugs_>	 06Labs, 10Tool-Labs: Python environment weirdness on labs - https://phabricator.wikimedia.org/T161915#3147746 (10madhuvishy) @valhallasw Ah interesting, TIL - I suppose that may have been the real fix (I created a new venv) - thanks for the note :)
[22:25:01] <yuvipanda>	 !log tools apt-get update && apt-get install kubernetes-node on tools-proxy-01 to upgrade kube-proxy systemd service unit
[22:25:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:25:43] <yuvipanda>	 madhuvishy: works ok!
[22:26:35] <madhuvishy>	 yuvipanda: cool!
[22:27:05] <yuvipanda>	 madhuvishy: now upgrading it across workers
[22:29:19] <yuvipanda>	 done
[22:29:23] <yuvipanda>	 (and bastions and proxies too)
[22:31:20] <madhuvishy>	 yuvipanda: should k8s webservice restarts work now?
[22:31:35] <yuvipanda>	 madhuvishy: yup, I just tried a couple and they seem to work
[22:31:53] <yuvipanda>	 the 'get()' python exception is unrelated race that has an open task, and doesn't actually prevent restarts from completing
[22:32:08] <madhuvishy>	 ah
[22:32:14] <madhuvishy>	 yeah i got that exception 
[22:32:28] <madhuvishy>	 but yes seems to have come up
[22:32:38] <yuvipanda>	 madhuvishy: yeah, if you do 'kubectl get svc' you will see it's new
[22:32:44] <madhuvishy>	 yup
[22:32:45] <yuvipanda>	 and you can just do a 'webservice restart' in a few secs and should be ok
[22:32:46] <shinken-wm>	 PROBLEM - Puppet run on tools-bastion-05 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[22:33:05] <yuvipanda>	 madhuvishy: https://phabricator.wikimedia.org/T156626
[22:33:48] <madhuvishy>	 yuvipanda: okay, i'll look
[22:34:11] <yuvipanda>	 madhuvishy: ok. I'm looking at the puppet failure
[22:34:29] <madhuvishy>	 aah
[22:34:31] <madhuvishy>	 okay
[22:35:12] <yuvipanda>	 madhuvishy: the puppet failure is just a clash between my apt-get command and puppet running one too. is all good now
[22:35:17] <yuvipanda>	 madhuvishy: did a restart work?
[22:35:24] <madhuvishy>	 yuvipanda: cool! yeah it did
[22:35:36] <yuvipanda>	 webservice shell works too
[22:35:37] <madhuvishy>	 tried on ifttt
[22:35:52] <yuvipanda>	 I think that's it then
[22:35:56] <yuvipanda>	 madhuvishy: thank you for verifying :)
[22:36:05] <madhuvishy>	 yuvipanda: np :) thanks for fixing
[22:36:08] <yuvipanda>	 Goodbye, channel! I'll be back in May :)
[22:36:13] <madhuvishy>	 go back to icecream
[22:36:15] <madhuvishy>	 <3
[22:36:16] <yuvipanda>	 have fun, everyone. I'll miss you :)
[22:47:46] <shinken-wm>	 RECOVERY - Puppet run on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0]
[23:12:50] <wikibugs_>	 06Labs, 10Labs-Vagrant, 10MediaWiki-Vagrant: labs-vagrant vagrant up fails with 404 at Dropbox - https://phabricator.wikimedia.org/T161891#3146355 (10bd808) We should probably just backport the puppet install provisioner to the trusty branch. That was all that was different in our lxc box image from the upst...