[00:00:19] (And a change from short to long tags, 6Labs, 10Labs-Infrastructure, 10Wikimedia-Apache-configuration, 6operations, 10wikitech.wikimedia.org: wikitech-static sync broken - https://phabricator.wikimedia.org/T101803#1384434 (10Dzahn) The whole VM has 40G in / but /var/lib/mysql alone is 24G already, plus the images and dump files, we'll just ne... [00:23:20] 6Labs, 10Tool-Labs: Recover homedir of "ci" tool - https://phabricator.wikimedia.org/T103205#1384473 (10Legoktm) 3NEW a:3coren [00:39:50] remind me again, what's the right process for getting betalabs nodes to see puppet repo updates? I'm pretty sure virt1000:/var/lib/git/operations/puppet was already updated by some cron or whatever [00:40:03] yet my client is still getting failing catalog compilation from something outdated [00:40:58] nevermind, it eventually magically fixed itself [03:31:00] YuviPanda|zzzz: sorry I don't log this channel, now I do. Saw your ping. I have a feeling it has to do with my current issues. Can you drop me a mail as I'm not sure i'll be on in the next 24 hours. i'll pm you my addr [04:19:34] 6Labs, 10Labs-Infrastructure, 10Wikimedia-Apache-configuration, 6operations, 10wikitech.wikimedia.org: wikitech-static sync broken - https://phabricator.wikimedia.org/T101803#1384634 (10Andrew) wikitech-static was syncing a ton of unused image files from silver which I just cleaned up a few days ago -- t... [05:00:21] 10MediaWiki-extensions-OpenStackManager, 7I18n: OpenStackManager has 15 messages without documentation - https://phabricator.wikimedia.org/T103214#1384653 (10Legoktm) 3NEW [05:37:57] 6Labs, 10Tool-Labs: Tools not using jsub on crontab - https://phabricator.wikimedia.org/T101609#1384745 (10Revi) 5Open>3Invalid a:3Revi Converted to jsub after modifying crontab to re-enable from the labs fail this week. Seems it doesn't convert non-edited old crontabs. [07:27:57] Hi all [07:29:38] I have a problem connecting my instance. I think the server dont't mount the partition with ssh keys [07:31:11] But on my other server the login works [07:33:40] the instance is: osmit-cruncher1.osmit.eqiad.wmflabs [10:25:25] 6Labs: Kill NFS in scrumbugz project - https://phabricator.wikimedia.org/T102704#1385034 (10yuvipanda) :) I did remove NFS - let me know if you want that recovered. [10:35:03] (03CR) 10Ricordisamoa: [C: 032 V: 032] "Let's go." [labs/tools/translatemplate] - 10https://gerrit.wikimedia.org/r/215644 (owner: 10Ricordisamoa) [10:42:42] 6Labs, 3Labs-Sprint-102: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1385047 (10yuvipanda) [10:42:44] 6Labs: Investigate and get rid of NFS on the etcd project - https://phabricator.wikimedia.org/T102845#1385046 (10yuvipanda) 5Open>3Resolved [10:43:49] 6Labs, 3Labs-Sprint-102: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1360119 (10yuvipanda) [10:43:51] 6Labs: Turn off NFS for varnish project - https://phabricator.wikimedia.org/T102385#1385048 (10yuvipanda) 5Open>3Resolved a:3yuvipanda [10:44:08] 6Labs: Kill NFS in scrumbugz project - https://phabricator.wikimedia.org/T102704#1385052 (10Christopher) no thanks, NFS was not needed it seems. I am however, having a problem now getting connected to phab08 with ssh. I do not know if it is related to this relatively new error "Warning: the ECDSA host key for... [10:44:28] 6Labs: Remove NFS completely from dynamicproxy project - https://phabricator.wikimedia.org/T102369#1385053 (10yuvipanda) Is currently used for backup [10:44:47] 6Labs: Remove NFS mounts from project Akosiaristests - https://phabricator.wikimedia.org/T102379#1385054 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Done now. [10:44:48] 6Labs, 3Labs-Sprint-102: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1385057 (10yuvipanda) [11:12:01] hi, is there again something broken on labs? [11:12:10] Coren|away: YuviPanda|zzzz and andrewbogott_afk thanks for all your work to get the server and data back [11:12:32] tools is up, stewardbot is working [11:14:25] stewardbot has nothingt to sy [11:14:26] *say [11:14:39] connection from tools --> wmf is higly slow [11:18:48] Stewardbot is speaking fine [11:19:03] wm-bot is working and responding to commands [11:19:48] I can log into bastion and ssh to wm-bot [11:19:54] reasonably quickly [11:21:02] tools takes a login and takes "become" fine [11:21:22] try to wget from wmf slow [11:21:29] try to jpegtran somethin, it takes ages [11:36:17] it is allowed to tunnel over ssh to wmflabs? [11:55:19] 6Labs: Kill NFS in scrumbugz project - https://phabricator.wikimedia.org/T102704#1385120 (10yuvipanda) The bastion warning seems to be just an outcome of https://lists.wikimedia.org/pipermail/labs-l/2015-June/003781.html - remove the old keys and you should be able to log back in? [11:56:15] 6Labs: Kill NFS in scrumbugz project - https://phabricator.wikimedia.org/T102704#1385121 (10yuvipanda) And if puppet has been disabled completely for a long time, that instance might not be recoverable at all. That's not a supported mode of operating in labs, and you might have to recreate the instance. I will a... [12:04:09] ping yuvipanda [12:04:16] 6Labs: Map Warper - No space left on device - https://phabricator.wikimedia.org/T102414#1385123 (10Chippyy) a:3Chippyy I can handle this. The instance has not much space left on the image - almost all of the dynamic files live on NFS. This is currently experiencing issues, and should be up soon. Logs and data... [12:05:00] 6Labs: Map Warper - No space left on device - https://phabricator.wikimedia.org/T102414#1385125 (10yuvipanda) You can increase the amount of space in your instance by selecting the labs::lvm::srv role on wikitech configure page for your instance - should give a big /srv partition [12:14:21] !log staging enable puppetmaster autoupdate for staging project, /cc thcipriani|afk greg-g [12:14:23] Logged the message, Master [12:16:19] 6Labs, 7Tracking: New Labs project requests (Tracking) - https://phabricator.wikimedia.org/T76375#1385130 (10yuvipanda) [13:01:11] !log bots restarted wm-bot not receiving recentchanges data [13:01:14] Logged the message, Master [14:51:24] does anyone use rvm on tool labs? [14:54:41] it works for me on bash, but not on jsub, producing this error: [14:54:51] ruby: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.17' not found (required by /data/project/contribstats/.rvm/rubies/ruby-2.1.2/bin/../lib/libruby.so.2.1) [15:03:53] whym: try adding -l release=trusty [15:03:57] to your jsub command [15:04:25] * YuviPanda|zzzz goes afk again [15:07:48] YuviPanda|zzzz: tx, that did the trick [16:58:22] oh, thanks for at least mostly recovering the tools (so far) [17:00:25] I found Reasonator's absence particularly vexing, and hit at least one or two other missing tools [17:03:17] any labs admins around? one of my instances still has ec2 instance id when i look at https://wikitech.wikimedia.org/wiki/Special:NovaInstance -- pcar-client14.puppet-ca-replacement.eqiad.wmflabs aka i-00000d5a. puppet is broken as a result, so i'd like to get it fixed. [17:05:59] xtools still down? [17:18:19] YuviPanda might be, jgage [17:19:38] help me yuvi-wan, you're my only hope [18:01:56] hi, I'm trying to submit Tools Access Request, but getting "Invalid or virtual namespace -1 given." ? [18:28:49] YuviPanda: killed both in mediawiki-verp [18:28:55] how sinful [18:28:58] :P [18:47:34] kahvel: hey! [18:47:45] kahvel: tell me your wikitech username? [18:48:14] Kahvel [18:52:43] kahvel: added you [18:55:01] thanks [18:55:19] jgage: not today, probably. also for those you need andrewbogott [18:55:35] YuviPanda: looks like proxycommand settings under https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_ProxyCommand_ssh_option_.28recommended.29 is not working for me :( [18:55:43] yuvipanda: ok, no prob. thanks. [18:56:01] I've to go now sadly, tonythomas :( [18:56:02] others might be able to help? [18:56:05] I'll brb later [18:56:07] I removed keys from both bastion and wikitech, and now its not showing up [18:56:11] YuviPanda: okey ;) [18:57:49] tonythomas: you should put in a new key in wikitech and uset hat :) [18:58:19] YuviPanda: my original key in my machine is safe. I generated one for bastion and uploaded in wikitech earlier [18:58:27] ah ok :) [18:58:38] I've to go now [18:58:40] cya [18:58:49] YuviPanda|zzz: oeky ;) [19:08:51] YuviPanda|zzz: Bye ;) [19:09:24] tonythomas: Hi [19:09:37] Vivek: hey ! [19:09:54] tonythomas: Are you Indian ? [19:10:28] Vivek: I am :) - but now in a bit different timezone - so would be up for few more hours [19:11:11] Ok [19:44:13] Hello, since the revert of my tools to a 8 days old copy, there appears to be an issue with 7z. It's creating corrupt archives. [19:44:44] you can grab one here: https://tools.wmflabs.org/oojs-ui/oojsui.7z [19:45:58] the command to create the archive is [19:45:59] 7z a "../oojsui.7z" "demos" "vendor" "php" "i18n" "LICENSE-MIT" "README.md" "AUTHORS.txt" "node_modules/jquery/dist" "node_modules/oojs/dist" "dist" [19:51:05] now that I didn't use jsub it worked .... [19:55:31] so, just confirmed; jsub scheduled jobs produce corrupt archives [19:56:58] OK: https://tools.wmflabs.org/oojs-ui/oojsui.7z CORRUPT: https://tools.wmflabs.org/oojs-ui/oojsui_corrupt.7z [19:57:58] mabe grid engine can't write file bigger than 1.048.608 to my home directory or something is rotten win 7z or .... [19:58:03] *maybe [19:58:39] * SamB_7 grumbles about the crazy ways some people write commas and decimal points [20:00:41] SamB_7 sorry, this is was the localized version of windows [20:01:18] I prefer 1'048'608 - which is also common in Switzerland [20:02:17] looks like I have to open a phabricator ticket [20:04:43] rillke: lol, in the us it's 1,048,608 [20:04:45] interesting [20:14:11] rillke: try adding -l release=trusty to jsub [20:14:19] Gets you newer 7z [20:35:37] (03PS1) 10Catrope: When finding the newest WMF branch, sort numerically, not alphabetically [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/219593 (https://phabricator.wikimedia.org/T103241) [21:43:52] 6Labs: Unable to ssh to dwl - https://phabricator.wikimedia.org/T103245#1385440 (10Giftpflanze) 3NEW [22:18:40] 6Labs: Unable to ssh to dwl - https://phabricator.wikimedia.org/T103245#1385462 (10Matanya) a:3yuvipanda [22:27:59] assuming wildcat projects are not yet fixed? [22:35:04] (03CR) 10Legoktm: [C: 032 V: 032] "PS2: Simplified wmf_number a bit. Thanks!" [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/219593 (https://phabricator.wikimedia.org/T103241) (owner: 10Catrope)