[00:40:45] PROBLEM - Puppet run on tools-puppetmaster-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [02:39:23] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [06:08:15] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [06:37:39] PROBLEM - Puppet run on tools-exec-1416 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:17:39] RECOVERY - Puppet run on tools-exec-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [07:36:39] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [08:54:09] 06Labs, 10Labs-Infrastructure, 06Operations, 10netops, 10wikitech.wikimedia.org: Provide public access to OpenStack APIs - https://phabricator.wikimedia.org/T150092#2774444 (10AlexMonk-WMF) is this a duplicate of T49515? [08:54:25] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Dump instance info as a static file updated periodically - https://phabricator.wikimedia.org/T143136#2774447 (10AlexMonk-WMF) >>! In T143136#2772063, @bd808 wrote: >>>! In T143136#2770926, @yuvipanda wrote: >> The script works, but is disabled right now. I n... [11:35:15] 06Labs, 10Tool-Labs: Perl module problems on 14## exec nodes - https://phabricator.wikimedia.org/T150120#2774533 (10Beetstra) [11:58:42] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [12:06:11] 10Striker, 07Epic, 07I18n: Enable i18n for Striker - https://phabricator.wikimedia.org/T144328#2596433 (10Nemo_bis) Is the Striker l10n stable now? [12:33:40] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:58:25] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 0.54 ms [13:04:02] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [14:46:07] 10Tool-Labs-tools-Xtools, 07I18n: Update Intuition to no longer use deprecated functions - https://phabricator.wikimedia.org/T138527#2774769 (10Nemo_bis) [15:12:48] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 1.90 ms [15:24:04] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [15:39:42] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 200.08 ms [15:46:39] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [15:57:35] Hi, why python ~/pwb/scripts/login.py lasts for 37 seconds? I think this shouldn't considered as normal... [15:58:34] (~/pwb is a symlink to /shared/pywikipedia/core/ [15:58:36] ) [16:12:24] 10PAWS: Re-render index from a Wiki page - https://phabricator.wikimedia.org/T150131#2774853 (10Halfak) [16:53:31] (03CR) 10MarkTraceur: "@Zppix @paladox can you please explain to me how this got CR+2'd when the logging is so inaccurate? Just because we've waited 17 seconds f" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319908 (owner: 10Paladox) [16:54:48] (03CR) 10Paladox: "@MarkTraceur I made it 17 seconds so it can send the message to irc. Do you have any suggestions for improvements please?" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319908 (owner: 10Paladox) [16:56:22] (03CR) 10MarkTraceur: "Yeah, how about we actually wait for the connection to be established before proclaiming that it worked?" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319908 (owner: 10Paladox) [16:57:04] (03CR) 10Paladox: "@MarkTraceur oh yeh but how would I be able to do that please?" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319908 (owner: 10Paladox) [17:00:27] (03CR) 10MarkTraceur: "Well, you see where it says "connected to the event stream!"? That's the "ready" handler for the SSH connection. That's how you're *suppos" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319908 (owner: 10Paladox) [17:09:23] (03CR) 10MarkTraceur: "If I'm honest with you, I'm less interested in seeing the improvement than hearing Zppix explain their +2. It seems like a pretty weird th" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319908 (owner: 10Paladox) [17:10:10] (03CR) 10Paladox: "Sorry" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319908 (owner: 10Paladox) [17:13:00] 10Striker, 07Epic, 07I18n: Enable i18n for Striker - https://phabricator.wikimedia.org/T144328#2774937 (10bd808) >>! In T144328#2774572, @Nemo_bis wrote: > Is the Striker l10n stable now? The app is definitely still in a state of large change, but that will be expected for some time. This particular feature... [17:23:03] 06Labs, 10Labs-Infrastructure, 06Operations, 10netops, 10wikitech.wikimedia.org: Provide public access to OpenStack APIs - https://phabricator.wikimedia.org/T150092#2774942 (10bd808) >>! In T150092#2774143, @Andrew wrote: >>>! In T150092#2773987, @bd808 wrote: >> What about using https://blueprints.launc... [17:45:20] 06Labs: Add access to nova's admin api - https://phabricator.wikimedia.org/T49515#2774959 (10Andrew) [17:45:23] 06Labs, 10Labs-Infrastructure, 06Operations, 10netops, 10wikitech.wikimedia.org: Provide public access to OpenStack APIs - https://phabricator.wikimedia.org/T150092#2774962 (10Andrew) [18:06:37] 06Labs, 10Labs-Infrastructure, 06Operations, 10netops, 10wikitech.wikimedia.org: Provide public access to OpenStack APIs - https://phabricator.wikimedia.org/T150092#2775006 (10Andrew) > in general the point of OAuth would be use easily revoked tokens for > authentication requests coming from any extern... [18:08:04] 06Labs, 10Labs-Infrastructure, 06Operations, 10netops, 10wikitech.wikimedia.org: Provide public access to OpenStack APIs - https://phabricator.wikimedia.org/T150092#2775007 (10Andrew) > Can probably usurp Keystone's own password authentication plugin, subclass > it, add a check against context.remote_ad... [18:08:32] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Research-and-Data, 15User-bd808: 2016 Tool Labs user survey - https://phabricator.wikimedia.org/T147336#2775008 (10bd808) I have turned off the survey form. We received 175 raw responses. One interesting thing I see from a very high level is that there... [18:32:03] 10Labs-project-other, 06Developer-Relations: move WikiApiary to Labs - https://phabricator.wikimedia.org/T149874#2775045 (10Aklapper) [18:33:18] 06Labs, 10Labs-Infrastructure, 06Operations, 10netops, 10wikitech.wikimedia.org: Provide public access to OpenStack APIs - https://phabricator.wikimedia.org/T150092#2775047 (10bd808) >>! In T150092#2775006, @Andrew wrote: >> in general the point of OAuth would be use easily revoked tokens for >> authen... [19:08:32] I've just scheduled a job with jsub. When I qstat the job id, it says that some queue instances dropped. [19:09:05] is it still working? [19:11:50] if there's an "r", yep, it should be [19:12:11] alright, good. [19:12:19] qstat should tell you iirc [19:12:28] or it was qstat jobname? [19:13:32] qstat -j 190590 [19:15:02] just qstat will give you a list of running jobs [19:16:23] I know, but at the -j with jobnumber gives the queue instances dropped issue [19:16:27] I hoped maybe you could see it? [19:17:46] yep, I saw them. In fact I always see some dropped instances but I don't know why they're always there. [19:18:25] maybe a labs admin or a more expert user could provide more light in this issue, I'm just a basic user :( [19:35:23] 10Tool-Labs-tools-Other, 06translatewiki.net, 07I18n, 13Patch-For-Review: [[Intuition:Monumentsapi-title/en]] i18n issue - https://phabricator.wikimedia.org/T137951#2775148 (10Nemo_bis) [20:17:47] DatGuy: dropped instances are not a problem [20:18:03] it just means that no new jobs will be scheduled on those hosts [20:19:11] I think that "scheduling info" data is cluster-wide information. not really specific to a particular job [20:19:22] is that mostly right valhallasw`vecto? [20:20:21] yep [20:20:30] it's the same for all jobs [21:10:27] bd808: need an alternative one? :o [21:42:56] (03CR) 10Paladox: "@MarkTraceur one problem with that is irc is not connected until after ssh is so that will fail in ready since it wont be able to call any" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319908 (owner: 10Paladox) [21:45:44] (03CR) 10MarkTraceur: "@paladox I'm not talking about calling an IRC function, but you could do that too. I'm going to go on faith and say that the IRC client li" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/319908 (owner: 10Paladox)