[00:07:04] RECOVERY - Puppet run on tools-static-02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:18:19] (03Draft1) 10Paladox: Add phabricator.* to the #wikimedia-dev channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/307041 [00:19:20] (03PS2) 10Paladox: Add phabricator.* to the #wikimedia-dev channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/307041 [00:25:53] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:30:50] PROBLEM - Free space - all mounts on tools-static-01 is CRITICAL: CRITICAL: tools.tools-static-01.diskspace._srv.byte_percentfree (<100.00%) [01:44:28] PROBLEM - Free space - all mounts on tools-web-static-01 is CRITICAL: CRITICAL: tools.tools-web-static-01.diskspace._srv.byte_percentfree (<10.00%) [01:48:07] yuvipanda: grafana half died again. [01:48:15] Cyberbot project is dead [01:48:28] No info available. [02:48:27] 06Labs, 10Horizon, 13Patch-For-Review: Create puppet backend with REST api for labs instance configuration - https://phabricator.wikimedia.org/T133412#2587894 (10Andrew) @yuvipanda, the puppet gui running on labtesthorizon should now be fully functional. It needs more code review but what you see there shou... [03:32:01] (03PS1) 10Krinkle: [WIP] Implement "Recent only" feature [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/307050 (https://phabricator.wikimedia.org/T64914) [03:53:35] !log butterfly turned on butterfly-m4m in horizon [03:53:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Butterfly/SAL, Master [04:16:00] PROBLEM - Puppet run on tools-exec-1409 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [04:29:06] 06Labs, 15User-Hydriz: Dumps instances occasionally hammer NFS for temporary storage - https://phabricator.wikimedia.org/T134148#2587910 (10Hydriz) [04:55:58] RECOVERY - Puppet run on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [05:22:13] PROBLEM - Puppet run on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [05:36:01] Hi, when trying to "screen" as my tool @ Tool Labs, I'm getting a "Cannot open your terminal '/dev/pts/76' - please check.". This has never happened before - how can I fix it? [05:37:02] or maybe I'm not allowed to screen as a tool [06:02:15] RECOVERY - Puppet run on tools-webgrid-lighttpd-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [07:38:31] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [08:03:18] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [08:50:54] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [09:48:20] PROBLEM - Puppet staleness on tools-exec-cyberbot is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [10:33:56] (03CR) 10Merlijn van Deen: [C: 04-1] "Everything is already sent to #wikimedia-dev by default" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/307041 (owner: 10Paladox) [10:34:14] (03Abandoned) 10Paladox: Add phabricator.* to the #wikimedia-dev channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/307041 (owner: 10Paladox) [10:46:49] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:48:40] yuvipanda: valhallasw`cloud: Some of the compute nodes are Trusty and some are older. I was wondering what the plans are to complete the move [10:49:11] (had several bugs where the easy fix was just adding -l release=trusty ) [10:52:39] And I think someone is hammering NFS because it appears to be very sluggish (tools.multichill@tools-exec-1408:~$ tail -f logs/nat 1, 2, 3, .....5 seconds, complete) [10:58:46] multichill: bd808 sent an email about the precise tp trusty move approx a week ago [10:59:22] Multichill: currently precise is default, which can be confusing (bastions are trusty) [11:01:45] valhallasw`cloud: I completely missed that. He didn't actually mention Trusty in his email so searching for "Trusty" didn't return it ;-) [11:03:00] Oh wait, different laptop, different default search, doh [11:21:52] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:40:56] PROBLEM - Puppet staleness on tools-exec-1211 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [11:43:46] 10Tool-Labs-tools-Gerrit-Patch-Uploader: failed to upload patch - https://phabricator.wikimedia.org/T144087#2588248 (10Zaher.Kadour) [11:58:22] 10Tool-Labs-tools-Gerrit-Patch-Uploader: failed to upload patch - https://phabricator.wikimedia.org/T144087#2588260 (10valhallasw) ``` access(".git/hooks/commit-msg", X_OK) = -1 EACCES (Permission denied) ``` yet ``` tools.gerrit-patch-uploader@tools-webgrid-lighttpd-1412:/tmp/218464.1.webgrid-lighttpd/tmpn... [11:58:27] 06Labs, 07Tracking: Existing Labs project quota increase requests (Tracking) - https://phabricator.wikimedia.org/T140904#2588265 (10Luke081515) [11:58:29] 06Labs, 15User-Luke081515: Revert: Request increased quota for rcm labs project - https://phabricator.wikimedia.org/T142311#2588261 (10Luke081515) 05stalled>03Open a:05Luke081515>03chasemp I delete that instance now. So you can a) revert my quota, or b) let it as it is, and I won't use it, so I don't s... [12:10:23] 10Tool-Labs-tools-Gerrit-Patch-Uploader: failed to upload patch - https://phabricator.wikimedia.org/T144087#2588266 (10valhallasw) It *hasn't* changed recently. Uh? https://github.com/wikimedia/operations-puppet/commit/dafe707988bfc773659211a023f8c93dc6628fef At the same time, a simple solution is to install t... [12:24:48] [13gerrit-patch-uploader] 15valhallasw pushed 1 new commit to 06master: 02https://git.io/v6pDI [12:24:48] 13gerrit-patch-uploader/06master 1491a1bbe 15Merlijn van Deen: Manually build Change-Id instead of using commit hook... [12:25:07] Phawkes? /me grins [12:25:10] that's a very old pun [12:25:34] 10Tool-Labs-tools-Gerrit-Patch-Uploader: failed to upload patch - https://phabricator.wikimedia.org/T144087#2588269 (10valhallasw) Should be resolved with https://github.com/valhallasw/gerrit-patch-uploader/commit/91a1bbe5207e1e935cf12aea67bfa4aff45f1203 Your patch has been uploaded as https://gerrit.wikimedia.... [12:25:58] 10Tool-Labs-tools-Gerrit-Patch-Uploader: failed to upload patch - https://phabricator.wikimedia.org/T144087#2588270 (10valhallasw) 05Open>03Resolved a:03valhallasw [12:28:20] 2 new issues in valhallasw/gerrit-patch-uploader // in the latest commit to branch origin/master we found 2 new issue(s). 3 issue(s) were fixed. [12:29:02] I have to admit I forgot to document my new function **grin* [14:42:00] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 2.34 ms [15:05:14] PROBLEM - Puppet staleness on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [15:05:22] PROBLEM - Puppet staleness on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [15:11:05] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [16:30:19] (03Draft2) 10Paladox: Add gerrit project to #wikimedia-releng channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/307071 [16:30:22] (03Draft1) 10Paladox: Add gerrit project to #wikimedia-releng channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/307071 [16:57:20] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 0.87 ms [17:03:17] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [17:06:22] 10PAWS: PAWS can not login - https://phabricator.wikimedia.org/T136114#2588510 (10Dvorapa) It was resolved for a while and now it doesn't work again [17:22:24] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 0.64 ms [17:32:23] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [17:48:07] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login - https://phabricator.wikimedia.org/T136114#2588561 (10yuvipanda) I can confirm this happened on both Dvorapa and Dvorapa bot :( I can't reproduce it on my account though. I've added additional OAuth related tags to see if someone... [17:50:23] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login - https://phabricator.wikimedia.org/T136114#2588563 (10Urbanecm) @yuvipanda I can login and pwb.py login -lang:cs works fine. Account UrbanecmBot was used. [18:21:06] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login - https://phabricator.wikimedia.org/T136114#2323722 (10Tgr) Well, what token are you using? ([[https://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/windows_dos_copy.mspx?mfr=true|copy text from a command p... [18:34:27] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login - https://phabricator.wikimedia.org/T136114#2588571 (10yuvipanda) @tgr this is at paws.wmflabs.org, so after the user logs in, we inject the credentials into pywikibot using this code: https://github.com/yuvipanda/paws/blob/master/si... [19:19:27] PROBLEM - Free space - all mounts on tools-web-static-01 is CRITICAL: CRITICAL: tools.tools-web-static-01.diskspace._srv.byte_percentfree (<11.11%) [19:46:30] 06Labs, 10Phabricator: Applying role role::phabricator::main causes errors on instances - https://phabricator.wikimedia.org/T138881#2588672 (10Paladox) @mmodell we could make role::phabricator::main work on labs. By making somethings optional like scap and other things that fail on labs. We can switch the thin... [21:08:38] 06Labs, 10Beta-Cluster-Infrastructure: puppet::self hosts now have two servers set - https://phabricator.wikimedia.org/T144108#2588773 (10Andrew) [21:14:04] (03PS1) 10BryanDavis: [WIP] Install/upgrade via wheels rather than complete venv reload [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/307085 [21:21:29] 06Labs, 10Beta-Cluster-Infrastructure, 13Patch-For-Review, 07Puppet: /etc/puppet/puppet.conf keeps getting double content - first for labs-wide puppetmaster, then for the correct puppetmaster - https://phabricator.wikimedia.org/T132689#2588790 (10hashar) Seems to cause {T144108} :( [21:41:50] 06Labs, 10Beta-Cluster-Infrastructure: puppet::self hosts now have two servers set - https://phabricator.wikimedia.org/T144108#2588773 (10AlexMonk-WMF) I set up a couple of hosts in deployment-prep while that was cherry-picked on deployment-puppetmaster and everything appeared okay. Or do you mean new servers... [22:04:42] 06Labs, 13Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#2588848 (10AlexMonk-WMF) [22:06:53] 06Labs, 13Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#2588850 (10AlexMonk-WMF) [22:21:14] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login - https://phabricator.wikimedia.org/T136114#2323722 (10Framawiki) Hi, I have the same problem for //pwb.py login.py -lang:fr// after disconnet / close / shutdown all ``` pywikibot.exceptions.NoUsername: Failed OAuth authentication fo... [22:59:35] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login - https://phabricator.wikimedia.org/T136114#2588864 (10Urbanecm) Maybe try to force password login using pwb.py login -pass -lang:fr for example as a workaround? [23:00:51] (03PS2) 10Krinkle: [WIP] Implement "Recent only" feature [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/307050 (https://phabricator.wikimedia.org/T64914) [23:08:07] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login - https://phabricator.wikimedia.org/T136114#2588865 (10Framawiki) I've got the same error, with and without good password, preceded by ``` WARNING: The -pass argument is not implemented yet. See: https://phabricator.wikimedia.org/T1... [23:08:57] (03PS3) 10Krinkle: [WIP] Implement "Recent only" feature [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/307050 (https://phabricator.wikimedia.org/T64914) [23:30:40] (03PS4) 10Krinkle: [WIP] Implement "Recent only" feature [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/307050 (https://phabricator.wikimedia.org/T64914) [23:30:42] (03PS1) 10Krinkle: Minor clean up and preparation for "Recent only" feature [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/307104 (https://phabricator.wikimedia.org/T64914) [23:42:20] 06Labs, 10Phabricator, 07Puppet: Phabricator labs puppet role configures phabricator wrong - https://phabricator.wikimedia.org/T131899#2182293 (10Paladox) Actually phabricator did this. It is required for http cloning to fix this all you have to do is '''cd /support/bin''' '''sudo ln -sv /user/li... [23:44:23] 06Labs, 10Phabricator, 07Puppet: Phabricator labs puppet role configures phabricator wrong - https://phabricator.wikimedia.org/T131899#2588885 (10Paladox) But I think we need to instead update the main puppet role to work on both production and labs and remove the labs phabricator role once we have done that. [23:46:39] 06Labs, 10Phabricator: Update phabricator role to support labs - https://phabricator.wikimedia.org/T144112#2588887 (10Paladox) [23:47:21] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login - https://phabricator.wikimedia.org/T136114#2588899 (10Urbanecm) Ok. So try to remove lines from # If OAuth integration is available, take it to the end of file in file named /srv/paws/user-config.py, then try to login normally. This... [23:48:39] (03PS2) 10Krinkle: Clean up to preparate for "Recent only" feature [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/307104 (https://phabricator.wikimedia.org/T64914) [23:51:07] (03PS3) 10Krinkle: Clean up in preparation for the "Recent only" feature [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/307104 (https://phabricator.wikimedia.org/T64914) [23:51:10] (03PS5) 10Krinkle: [WIP] Implement "Recent only" feature [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/307050 (https://phabricator.wikimedia.org/T64914) [23:53:08] (03PS1) 10BryanDavis: Add pretty error pages [labs/striker] - 10https://gerrit.wikimedia.org/r/307107 (https://phabricator.wikimedia.org/T143949) [23:54:42] (03CR) 10jenkins-bot: [V: 04-1] Add pretty error pages [labs/striker] - 10https://gerrit.wikimedia.org/r/307107 (https://phabricator.wikimedia.org/T143949) (owner: 10BryanDavis)