[00:02:56] RECOVERY - Puppet run on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [00:29:16] 06Labs: Replicate watchlist to labs - https://phabricator.wikimedia.org/T93887#2337195 (10MZMcBride) I think @jcrespo may be handling this as part of T59617. It may make sense to assign this task to him or mark this task as a duplicate of T59617. [02:00:00] 06Labs, 10Tool-Labs: toolsbeta grid misconfigured - https://phabricator.wikimedia.org/T136433#2337224 (10scfc) The automatic configuration of hostgroups and queues has never worked (CMIIW; cf. T88711); `modules/gridengine/files/tracker`: ``` […] ### XXX: RIGHT NOW THIS ONLY ECHOS THE COMMANDS TO AVOID ###... [02:04:46] how big of a hassle is it if you change your username? [03:57:49] 06Labs, 10Tool-Labs: Grid job stuck at 't' state - https://phabricator.wikimedia.org/T136508#2337259 (10liangent) [03:58:44] 06Labs, 10Tool-Labs: Grid job stuck at 't' state - https://phabricator.wikimedia.org/T136508#2337272 (10liangent) [06:36:04] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Not Available - string 'Magnus' not found on 'http://tools.wmflabs.org:80/' - 530 bytes in 0.020 second response time [06:56:06] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 3669 bytes in 0.067 second response time [07:02:02] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:02:23] ^ is me [07:05:26] PROBLEM - Puppet run on tools-worker-1003 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:06:08] ^ is transient [08:08:34] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/PetrohsW was modified, changed by PetrohsW link https://wikitech.wikimedia.org/w/index.php?diff=589025 edit summary: [08:15:08] 06Labs, 10Tool-Labs: Grid job stuck at 't' state - https://phabricator.wikimedia.org/T136508#2337259 (10valhallasw) State 't' means 'in process of being transferred to the exec host'. The thing that surprises me is that the start time changes: ``` 05/28/2016 10:04:01 05/29/2016 03:54:01` ``` and now ``` 05/29... [10:05:09] 06Labs, 10Tool-Labs: Grid job stuck at 't' state - https://phabricator.wikimedia.org/T136508#2337412 (10valhallasw) @Avicennasis, @abian, @Emijrp, @Pcoombe, @liangent, I have force-deleted the jobs in the post above; please resubmit the jobs (they should not have started, so it is probably safe to do so). Sorr... [11:57:46] PROBLEM - Free space - all mounts on tools-worker-1004 is CRITICAL: CRITICAL: tools.tools-worker-1004.diskspace.root.byte_percentfree (<10.00%) [12:45:10] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [13:08:13] 06Labs, 10Tool-Labs: wikiviewstats is using 232G on Tools - https://phabricator.wikimedia.org/T136198#2337465 (10Cyberpower678) a:05Cyberpower678>03None I'm not sure why this is assigned to me. I'm not too familiar with wikiviewstats. I can't find this massive directory as mentioned "bak" does not exist. [14:07:09] Coren or petan, I've just made a new tool account (xtools-mab-dev), but now weird things are happening - directories I create are instantly deleted, and then magically reappear, only to be disapear again [14:10:12] blockbowl: how are you creating them? [14:10:36] mkdir dirname [14:11:09] It worked ok before, then after I did various things like try and clone a git repo into the tool directory, it went weird [14:11:24] hrm. [14:12:39] Same with files. Just now did 'touch asdf'. First time I 'ls' it was there, second time it was gone [14:13:15] blockbowl: can you log out for a second? [14:14:17] Ok have done [14:15:48] blockbowl: there's still one `become` session active, although it doesn't seem to be connected to a terminal [14:15:57] it's opening vi over and over again? [14:16:08] vi, git, etc [14:16:48] can I kill that? [14:17:01] Ah yeah I saw that also when I added a custom .bashrc and tried to source it, it opened the file in vi [14:17:04] Yeah go for it [14:18:19] no, there's still weird stuff happening in that directory. What on earth. [14:19:08] blockbowl: have you logged in to other hosts except tools-login? [14:22:56] Just my normal username (mabandalone). Don't have access to anything else on that server [14:24:01] Let me see, it was all fine initially, I created some directories and copied some files from the xtools dir, then started the webservice [14:24:36] Then decided to clone from a git repo instead, so deleted the files, tried to clone, failed, so tried to copy directly from my mabandalone username. [14:24:46] And somewhere around this point the weird stuff started [14:25:02] Oh and I was also using my custom .bashrc [14:26:52] I'm at a loss, to be honest [14:27:08] I'll try chowning the dir to root, and hopefully that will error out whatever is doing this [14:27:26] Great thanks for looking into this [14:28:02] ok, that at least stops the writing. Let's see what happens if I chown it back... [14:28:44] ok, that seems to have solved it [14:29:35] !log tools chowned /data/project/xtools-mab-dev to root and back to stop rogue process that was writing to the directory. I'm still not sure where that process was running, but at least this seems to have solved the issue [14:29:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [14:29:51] Thanks. Ok I've logged in, I'll just try doing the git clone again [14:29:56] ok! [14:32:19] Seems ok so far. Thanks for your help [15:21:32] hi! How do I open a php terminal or php shell in bash prompt? [15:22:32] have it to use rlwrap? [15:24:52] ? [15:25:52] i want to open an interactive php shell [15:26:36] "If you type 'php -a' and get a response of 'Interactive Shell' followed by a 'php>' prompt, you have interactive shell available (PHP was compiled with readline support). If instead you get a response of 'Interactive mode enabled', you DO NOT have interactive shell available and this article does not apply to you." [15:26:58] hm, php5-readline is not installed [15:27:00] that might resolve it [15:28:32] 06Labs, 10Tool-Labs: install php5-readline on bastion and exec hosts - https://phabricator.wikimedia.org/T136519#2337526 (10valhallasw) [15:28:37] i got "Interactive mode enabled" [15:29:59] thanks for the task [15:30:18] but do you have an idea, when it will be done? [15:30:37] When someone has time to write up the puppet manifest and to +2 it [15:30:47] you can do the first part yourself to help it get resolved faster [15:39:24] RECOVERY - Puppet run on tools-worker-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [16:01:39] !log wikilabels pushed wikilabels-wikimedia-config:3f7b958 to labels-staging [16:01:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL, Master [16:28:02] !log wikilabels pushed wikilabels-wikimedia-config:3a3a850 to labels-staging [16:28:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL, Master [16:35:05] !log wikilabels pushed wikilabels-wikimedia-config:9eabc81 to labels-staging [16:35:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL, Master [16:37:37] !log wikilabels pushed wikilabels-wikimedia-config:9eabc81 to production [16:37:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL, Master [16:48:32] !log wikilabels pushed wikilabels-wikimedia-config:6bfc83e to labels-staging [16:48:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL, Master [16:55:21] PROBLEM - SSH on tools-exec-1221 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:57:44] !log wikilabels pushed wikilabels-wikimedia-config:6bfc83e to labels production [16:57:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL, Master [17:17:19] !log wikilabels pushed wikilabels-wikimedia-config:3b0ba53 to labels-staging [17:17:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL, Master [17:29:44] !log wikilabels pushed wikilabels-wikimedia-config:6ca744c to labels-staging [17:29:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL, Master [17:36:27] !log wikilabels pushed wikilabels-wikimedia-config:6ca744c to labels production [17:36:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL, Master [17:36:34] And I think that fixes all of the problems! [17:36:38] ALL OF THEM [17:40:11] RECOVERY - SSH on tools-exec-1221 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [18:58:26] !log tools deleted tools-k8s-bastion-01 for T136496 [18:58:27] T136496: puppet failing on tools-k8s-bastion-01 (kubeconfig lock files) - https://phabricator.wikimedia.org/T136496 [18:58:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [18:58:49] 06Labs, 10Tool-Labs: puppet failing on tools-k8s-bastion-01 (kubeconfig lock files) - https://phabricator.wikimedia.org/T136496#2337682 (10yuvipanda) I've just deleted it instead, since you can now kubectl stuff from regular bastion. [18:58:56] 06Labs, 10Tool-Labs: puppet failing on tools-k8s-bastion-01 (kubeconfig lock files) - https://phabricator.wikimedia.org/T136496#2337683 (10yuvipanda) 05Open>03Resolved a:03yuvipanda [19:00:48] PROBLEM - Host tools-k8s-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.19.200) [19:01:38] Okay, any specific reason I'm getting asked for a password when I ssh into login.tools.wmflabs.org? Command being used is "ssh -i ~/.ssh/Toolserver-Labs.ssh mattherbowker@login.tools.wmflabs.org" [19:03:26] Matthew_: can you do that once again just now? [19:03:52] I just did, still asking for a password. [19:04:10] May 29 19:03:36 tools-bastion-03 sshd[4499]: input_userauth_request: invalid user mattherbowker [preauth] [19:04:15] Matthew_: is that your shell user name? [19:04:33] Matthew_: it's missing a 'w' [19:04:44] Oh... oops. [19:04:48] :) [19:04:54] I checked my username three times and missed it all three. [19:04:55] Thank you. [19:05:08] :D yw! [19:05:09] * YuviPanda goes afk [19:42:09] is mod_rewrite enabled on tool labs? [19:42:54] Matthew_: the tool labs webservices run lighttpd, not apache [19:43:15] I didn't know that, huh. [19:43:18] Matthew_: https://redmine.lighttpd.net/projects/1/wiki/docs_modrewrite should be enabled [19:43:41] Thank you. [19:44:13] Matthew_: and https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web has some examples [19:44:32] well, one example, and info on how to configure lighttpd on tool labs [19:46:05] Perfect, thank you!