[00:19:21] if i wrap my pywikbot fetches in multiprocessing on labs, it hangs, any idea why/ [00:19:21] ? [02:35:37] Coren: you about? [06:33:53] PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:38:15] PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:40:53] PROBLEM - Puppet failure on tools-uwsgi-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:47:45] PROBLEM - Puppet failure on tools-redis is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:56:55] PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:58:52] RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0] [07:00:13] hoi [07:00:31] can someone restart all tools from Magnus ? [07:03:16] RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0] [07:05:49] RECOVERY - Puppet failure on tools-uwsgi-01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:12:47] RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0] [07:21:53] RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0] [09:44:59] Hi, how can I regain my access to Phabricator? I've lost my mobile token. [13:00:23] PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:30:23] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:41:47] 3Labs-Team: Storage capacity & redundancy expansion - https://phabricator.wikimedia.org/T85604#950661 (10coren) 3NEW [14:42:23] 3Labs-Team: Set storage service up in codfw - https://phabricator.wikimedia.org/T85605#950667 (10coren) 3NEW [14:44:22] 3Labs-Team: Replicate data between codfw and eqiad - https://phabricator.wikimedia.org/T85606#950673 (10coren) 3NEW [14:44:48] 3Labs-Team: Increase storage available to labs NFS server - https://phabricator.wikimedia.org/T85607#950679 (10coren) 3NEW [14:48:44] 3Labs-Team: Replicate data between codfw and eqiad - https://phabricator.wikimedia.org/T85606#950689 (10coren) [14:50:12] 3Labs-Team: Process for user backups - https://phabricator.wikimedia.org/T85608#950693 (10coren) 3NEW [14:50:50] 3Labs-Team: Process for user backups - https://phabricator.wikimedia.org/T85608#950693 (10coren) [14:51:33] 3Labs-Team: Labs available in the new data centre (with Neutron/IPv6) - https://phabricator.wikimedia.org/T85609#950700 (10coren) 3NEW [14:51:51] 3Labs-Team: Labs available in the new data centre (with Neutron/IPv6) - https://phabricator.wikimedia.org/T85609#950700 (10coren) [14:52:40] 3Labs-Team, Tool-Labs: Distributing tools, deployment-prep to both data centers (availability/redundancy) - https://phabricator.wikimedia.org/T85610#950707 (10coren) 3NEW [14:52:53] 3Labs-Team, Tool-Labs: Distributing tools, deployment-prep to both data centers (availability/redundancy) - https://phabricator.wikimedia.org/T85610#950707 (10coren) [14:54:43] 3Labs-Team: Neutron networking, with IPv6 at eqiad - https://phabricator.wikimedia.org/T85611#950714 (10coren) 3NEW [14:55:35] 3Labs-Team: OpenStack APIs opened to end users - https://phabricator.wikimedia.org/T85612#950720 (10coren) 3NEW [14:56:16] 3Labs-Team: OpenStack manager extension improvements or replacement - https://phabricator.wikimedia.org/T85613#950728 (10coren) 3NEW a:3Andrew [14:58:26] 3Labs-Team: Set storage service up in codfw - https://phabricator.wikimedia.org/T85605#950738 (10coren) p:5Triage>3High a:3coren [14:59:12] 3Labs-Team: Replicate data between codfw and eqiad - https://phabricator.wikimedia.org/T85606#950740 (10coren) Rsync of hourly snapshots pending on both storage clusters being working [15:00:24] 3Labs-Team: Replicate data between codfw and eqiad - https://phabricator.wikimedia.org/T85606#950742 (10coren) p:5Triage>3Normal [15:01:18] 3Labs-Team: Increase storage available to labs NFS server - https://phabricator.wikimedia.org/T85607#950744 (10coren) Hardware ordered. Actual capacity expansion is simply adding the new disks to the LVM volume group. [15:02:25] 3Labs-Team: Labs available in the new data centre (with Neutron/IPv6) - https://phabricator.wikimedia.org/T85609#950745 (10coren) p:5Triage>3High [15:03:39] 3Labs-Team, Tool-Labs: Distributing tools, deployment-prep to both data centers (availability/redundancy) - https://phabricator.wikimedia.org/T85610#950747 (10coren) p:5Triage>3Normal [15:04:10] 3Labs-Team: Neutron networking, with IPv6 at eqiad - https://phabricator.wikimedia.org/T85611#950749 (10coren) [15:05:16] 3Labs-Team: Neutron networking, with IPv6 at eqiad - https://phabricator.wikimedia.org/T85611#950757 (10coren) p:5Triage>3Normal [15:11:40] 3Labs-Team: Replicate data between codfw and eqiad - https://phabricator.wikimedia.org/T85606#950762 (10coren) A point of note that doing so will require rejiggering storage in eqiad to do thin volumes also (for snapshots) and will require extended downtime (24h or so) [15:17:57] 3Labs-Team: Replicate data between codfw and eqiad - https://phabricator.wikimedia.org/T85606#950771 (10coren) [15:32:48] 3Labs-Team: Process for user backups - https://phabricator.wikimedia.org/T85608#950795 (10coren) [15:37:45] 3Labs-Team: Increase storage available to labs NFS server - https://phabricator.wikimedia.org/T85607#950798 (10coren) [15:50:30] 3Tool-Labs: add basic expectations management to docs - https://phabricator.wikimedia.org/T56701#950830 (10Aklapper) I strongly wonder which docs this refers to, and if any docs already exist(ed). Links highly welcome if this task is still valid. [16:31:32] 3Labs-Team: Labs available in the new data centre (with Neutron/IPv6) - https://phabricator.wikimedia.org/T85609#950917 (10coren) Currently, the bigger stumbling block is the architecture of networking with Neutron, which does not support the model we currently use (single, flat, uniform network with DHCP) [16:45:46] (03CR) 10Merlijn van Deen: [C: 032] Move operations projects to #wikimedia-operations [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/182235 (owner: 10Merlijn van Deen) [16:46:05] (03Merged) 10jenkins-bot: Move operations projects to #wikimedia-operations [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/182235 (owner: 10Merlijn van Deen) [16:48:31] !log tools.wikibugs Updated channels.yaml to: 0ba0b2c47cd593b64c4149931bfdaf022dff230c Merge branch 'master' of https://github.com/wikimedia/labs-tools-wikibugs2 [16:48:39] Logged the message, Master [17:13:02] PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:14:10] Help [17:14:12] bastion.wmflabs.org [17:14:17] doesn't worl [17:14:19] work [17:17:28] 3Tool-Labs: add basic expectations management to docs - https://phabricator.wikimedia.org/T56701#950962 (10scfc) 5Open>3Invalid IIRC this referred to [[https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help]], or nowadays [[https://wikitech.wikimedia.org/wiki/Help:Tool_Labs]]. That page has been revam... [17:29:21] 3Phabricator, Tool-Labs-tools-Other: grrrit-wm doesn't include links to phab from the commit msg - https://phabricator.wikimedia.org/T85626#950975 (10Glaisher) [17:31:21] 3Phabricator, Tool-Labs-tools-Other: grrrit-wm doesn't include links to phab from the commit msg - https://phabricator.wikimedia.org/T85626#950991 (10Glaisher) [17:32:24] 3Wikimedia-Git-or-Gerrit, Tool-Labs-tools-Other: lolrrit-wm / grrrit-wm should report Phabricator tasks on patches the way it did for Bugzilla links - https://phabricator.wikimedia.org/T75799#782728 (10Glaisher) [17:42:54] (03PS1) 10Glaisher: Add task link to grrrit-wm irc message [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/182369 [17:43:02] RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [17:47:59] !admin [17:48:22] that doesn't work here, I think [17:48:29] dah, I was afraid of that [18:01:39] Glaisher: Do you know who I can talk to? [18:01:53] about what? [18:02:34] usually it's best if you just ask the question directly without asking whether you can ask or who to ask :) [18:04:11] I wanted to see if an admin could review my tools access request [18:06:27] when did you request it? [18:06:44] "Requests for access are generally dealt with within the day (often faster), though response-time may be longer depending on admin availability" [18:07:04] since it's holidays for most people, it might take longer than usual [18:08:47] coren: ↑ [18:28:21] Coren: Is there documentation of /data/scratch somewhere? [18:29:30] anomie: Not as such. It's defined as "Space you can use for large things but which can go away without warning or be randomly wiped/purged." Think of it as a large, persistent /tmp [19:08:46] PROBLEM - Puppet failure on tools-redis is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:33:31] 3Tool-Labs: add basic expectations management to docs - https://phabricator.wikimedia.org/T56701#951207 (10scfc) 5Invalid>3Open (Sorry, misclicked.) [19:33:46] RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0] [19:47:53] Coren: just a note, the trusty login server doesnt show the MOD [19:48:41] That's actually Trusty in general atm; it handles /etc/motd.d differently and we haven't tweaked the manifest to match yet. [19:49:21] Coren: no big deal, just wanted to give a heads up [19:49:29] that's https://gerrit.wikimedia.org/r/#/c/181789/ [19:50:11] Ah, as usual paravoid Knows All and Sees All. :-) [19:51:13] Coren: Im surprised we dont have a variant of vagrant that works outside of VM [19:51:55] labs-vagrant [19:52:15] It just runs the puppet code from mw-vagrant on any labs instance [19:52:31] bd808: I was thinking for non-labs systems [19:52:35] it would sort of mostly work bare metal anywhere [19:53:13] I've been meaning to work on getting mw-vagrant to work with lxc containers [19:53:28] I got it to work with docker but its a bit flakey [19:53:48] docker and full stack images don't play too well toghether [20:03:56] PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:28:57] RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [20:28:58] Coren: something is screwy with my toolserver redirect [20:29:08] Betacommand: how so? [20:29:25] Coren: http://toolserver.org/~betacommand/SIL.html [20:29:41] switches protocol and adds a / [20:30:27] Ah, yes. The protocol switch is expected (Redirect have compatibility issues unless a protocol is specified in many browsers); the extra / isn't. [20:30:45] Hm. Lemme go see what's up. [20:31:14] Coren: the protocol switch shouldnt happen if one is given [20:32:05] Wait, you specifically want people to be unconditionally redirected to http://? [20:32:13] No [20:32:47] if HTTP is passed in the URL it should be redirected to the HTTP version, if https is passed it should be given HTTPs [20:32:58] I found the cause of your extra /. Trivially fixed. Gimme a sec. [20:34:10] Fixed. There was a half-dozen redirects that were written slightly differently than others; yours was one of them (source and target without and with trailing slash) [20:34:40] Coren: thanks [20:35:19] But yeah, as for the protocol switch, that's unavoidable on a redirect unless we're willing to break som older browsers. A header with "Location: //foo" breaks some of 'em. :-( [20:36:31] IE6 for sure; possibly IE7 [20:36:58] Coren: you cant just change the rewrite rules to follow the same protocol? [20:38:35] I can, with a protocol-relative URL; but older browsers fail to understand those in a 301. What's the issue with redirecting to the SSL termination anyways? [20:39:52] Coren: not all tools support HTTPS [20:40:15] o_O [20:40:30] How so? That's a bug in the proxy if so - the tool shouldn't even need to be aware. [20:41:05] Coren: its nothing in the proxy its in the code of the tools that I just havent gotten around to update [20:41:33] SIL for example, passes the field value to a cgi tool via http [20:41:36] What I mean is that you shouldn't have to update anything at all, as far as I know. [20:41:38] Ah! [20:42:10] and Ive got a few others that work in a similar method [20:42:11] Yeah, I see in the source; action="http://tools.wmflabs.org/betacommand-dev/cgi-bin/SIL" instead of action="//tools.wmflabs.org/betacommand-dev/cgi-bin/SIL" [20:42:29] Well, if you want I can have your tool sent to http:// instead. [20:42:44] Coren: it shouldnt be that big of a deal [20:42:48] I'd rather special case than break compatibility for the general case. [20:42:56] It'd take me all of 10s. :-) [20:43:09] Im just shocked that the rewrite rules cant be smarter [20:43:49] It can; it's the browser that can be dumb. If you use a modernish browser it'd just work if I didn't specify the protocol at all. [20:45:08] Coren: Couldnt the rewrite rules check and identify the current protocol and just pass it? IE if HTTP http:labs if HTTPS https:labs [20:46:34] Well, it's using redirects not rewriterules for efficiency; but it could if it was absolutely required - but I don't see the point if your tool needs http to work right you don't want to preserve https :-) [20:47:28] Coren: my tools are split between them :P Most are PR but not all of them [20:48:05] SIL is one of the minority where its not PR [20:49:49] All of that said, people are not following old links to it at all; I see only one attampt since the redirect exist that isn't me or you. :-) [20:50:18] That's presuming that also wasn't you with a different IP on 28/Dec/2014:16:08:03 UTC [20:50:19] :-) [20:51:09] So it's very much a not-issue in practice. :_) [20:52:27] Coren: I should get around to checking for http/https issues sometime, I just havent been motivated [20:52:59] I've just check and /~betacommand/cgi-bin/cite.py is the only uri of yours that has things pointing to it. [20:54:00] Ah, wrong, there's also a singular access to /~betacommand/reports/CATCSD.html yesterday [20:54:15] Coren: both of those should handle HTTPs fine [20:54:34] !log mobile Added jhobs as a project member [20:54:38] Logged the message, Master [20:56:22] hi Coren are you working today? I replied to your labs-l thread about downtime for the labs filesystem. [20:57:55] chrismcmalunch: I am, and the deployment thing is noted; I'll probably move to a Thu/Fri for that reason, but I want to see a bit more feedback before I commit to a date in case of (OMG no we're doing $GLAM_EVEN on that date) and such. [20:59:41] Coren: sounds good. people get irate when beta isn't usable :-) [21:17:13] Coren: your email about redirects. those are OSM related [21:55:32] (03CR) 10Legoktm: [C: 031] "Looks sane, but Yuvi should take a look." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/182369 (owner: 10Glaisher)