[00:40:08] 6Labs, 6operations, 10wikitech.wikimedia.org, 7Ipv6: Set IPv6 PTR for wikitech-static - https://phabricator.wikimedia.org/T103621#1394899 (10Dzahn) login at https://mycloud.rackspace.com/ (somehow) [01:14:16] Coren, you forgot didn't you. :p [01:23:25] 6Labs, 10Tool-Labs, 10Continuous-Integration-Infrastructure: Recover homedir of "ci" tool - https://phabricator.wikimedia.org/T103205#1394990 (10jayvdb) Is this part of {T100903}? *wishful thinking* [01:25:11] 6Labs, 10Tool-Labs, 10Continuous-Integration-Infrastructure: Recover homedir of "ci" tool - https://phabricator.wikimedia.org/T103205#1394992 (10Legoktm) >>! In T103205#1394990, @jayvdb wrote: > Is this part of {T100903}? *wishful thinking* No, it's a bot that updates (or used to...) https://www.mediawiki.o... [01:51:09] Cyberpower678: I didn't forget; but I am rather busy. I haven't done BAG work in years so I'm looking at how things work nowadays. [04:11:01] what does labs recommend for ssh key options? [04:11:12] rsa, 4096 bit strength? [05:25:36] 10Tool-Labs-xTools: Contribution counting error NehalDaveND - https://phabricator.wikimedia.org/T102460#1395312 (10NehalDaveND) {F182912} 500 {F182913} 500 {F182914} 425 ---------------- Total 1425 but here http://tools.wmflabs.org/supercount/index.php?user=NehalDaveND&project=sa.wikipedia.org&toplim... [06:58:31] 6Labs, 10Labs-Other-Projects, 3Labs-Sprint-103: investigate/clean up 'servermon' project - https://phabricator.wikimedia.org/T103149#1395398 (10akosiaris) Yes, it was indeed. I wasn't using it at the moment and it definitely did not have anything worth keeping so feel free. I might have to recreate it in the... [07:01:26] Submit Tools Access Request seems failing to me [07:01:37] got `Invalid or virtual namespace -1 given.` [07:12:37] fantasticfears: that's odd. [07:13:43] fantasticfears: for now, can you tell me what your wikitech user name is, and what you're planning to use tool labs for? [07:15:30] valhallasw: sure. My wikitech user name is `Fantasticfears`. The plan is "enhance a interwiki links translating tool by interwiki wikidata, origin from Liangent. http://tools.wmflabs.org/liangent-django/link_translator/" The tool is asked privately by a zhwiki_p admin [07:16:55] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Fantasticfears was created, changed by Merlijn van Deen link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Fantasticfears edit summary: Created page with "{{Tools Access Request |Justification=enhance a interwiki links translating tool by interwiki wikidata, origin from Liangent. http://tools.wmflabs.org/liangent-django/link_tra..." [07:17:27] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Fantasticfears was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=167496 edit summary: [07:17:33] fantasticfears: done! [07:18:03] valhallasw: thanks [07:18:11] 6Labs, 10Tool-Labs, 10wikitech.wikimedia.org: "Invalid or virtual namespace -1 given" when submitting access request - https://phabricator.wikimedia.org/T103653#1395487 (10valhallasw) 3NEW [08:35:30] 6Labs, 10Tool-Labs, 10Labs-Infrastructure, 7Regression: Tools Labs proxy should not overwrite error page with generic "400 Bad Request" - https://phabricator.wikimedia.org/T103662#1395671 (10Krinkle) 3NEW [08:58:30] 6Labs, 10Tool-Labs, 10Labs-Infrastructure, 7Regression: Tools Labs proxy should not overwrite error page - https://phabricator.wikimedia.org/T103662#1395689 (10Krinkle) [09:33:34] 6Labs, 10Tool-Labs, 10Labs-Infrastructure: Update meta_p to contain accurate canonical server endpoints - https://phabricator.wikimedia.org/T103663#1395727 (10Krinkle) 3NEW [09:39:57] !log commons-archive test [09:39:58] commons-archive is not a valid project. [09:40:10] !log commonsarchive test [09:40:13] Logged the message, Master [09:43:21] 6Labs, 10Tool-Labs, 10Labs-Infrastructure: Update meta_p to contain accurate canonical server endpoints - https://phabricator.wikimedia.org/T103663#1395766 (10Krinkle) [10:54:44] 6Labs, 10Tool-Labs, 3Labs-Sprint-103: Labs: Move tools-shadow off the same host as tool-master - https://phabricator.wikimedia.org/T103390#1396003 (10yuvipanda) Do remember to cleanup that project / delete when done :)f [11:14:30] Coren: any news regarding the recovery of old files? [11:15:38] 6Labs: Instance osmit.eqiad.wmflabs refuses my public ssh key - https://phabricator.wikimedia.org/T103310#1396043 (10Sbiribizio) Any news? Regards [11:27:18] 6Labs, 5Continuous-Integration-Isolation: Nodepool can't SSH to spawn instances due to wmflabs provided instances due ssh algorithms - https://phabricator.wikimedia.org/T102459#1396075 (10hashar) [12:27:16] 6Labs, 10Tool-Labs, 10Labs-Infrastructure: Update meta_p to contain accurate canonical server endpoints - https://phabricator.wikimedia.org/T103663#1396265 (10coren) This was a simple fix as the list of canonical URL had been updated; a simple run of maintain-replicas updated the meta table with up-to-date i... [12:28:02] 6Labs, 10Tool-Labs, 10Labs-Infrastructure: Update meta_p to contain accurate canonical server endpoints - https://phabricator.wikimedia.org/T103663#1396266 (10coren) 5Open>3Resolved a:3coren [12:34:40] Coren, gotchya. :-) [12:34:59] elee, do you have shell access yet? [12:35:41] Cyberpower678: Yeah sorry CP. I'm leaving for vacation this friday so I've got seven tons of things to do before then leaving me no time for volunteery stuff. [12:36:05] Coren, should I just ask addshore instead? [12:36:19] Cyberpower678: More likely to get a chance, yes. [12:36:25] *waves* [12:36:30] whats occouring? [12:36:51] addshore, can you have a look at my BRFA. I'm sure you'll see which one is mine. :P [12:37:03] give me a link? ;) [12:37:07] #lazy [12:37:25] #workdeprived :p [12:37:50] addshore, https://en.wikipedia.org/wiki/Wikipedia:Bots/Requests_for_approval/Cyberbot_II_5 [12:38:21] Grrr. Ever since labs broke down, my bots have been failing one after another. :( [12:38:42] Code fix here, code fix there. [12:39:09] shurly shouldnt have been that bad? isnt the code all in git / on github? ;p [12:39:55] yes. [12:40:06] whats causing the failures then? [12:40:33] Probably API changes. TallyBot randomly stopped working at the framework level. :/ [12:41:24] PCBot started to obsevilly tag pages that already have tags. [12:41:36] *obsessively [12:42:06] One thing comes to mind, and is that they both use the same function of Peachy. [12:42:38] which is? [12:42:47] The embeddedin function [12:43:41] PCBot and TallyBot started failing the same day. [12:44:05] Stack Traces also point to the embeddedin function [12:54:31] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103: Labs: Make a new backup of the Labs storage to codfw - https://phabricator.wikimedia.org/T103356#1396325 (10coren) Papaul is having some difficulty getting the shelves recognized in codfw. Getting two shelves attached to labstore1001 would allow us... [13:20:02] addshore, so I fixed the bug, now I'm just waiting for you. :D [13:30:30] 6Labs, 10Tool-Labs, 10MediaWiki-extensions-SemanticForms, 10wikitech.wikimedia.org: "Invalid or virtual namespace -1 given" when submitting access request - https://phabricator.wikimedia.org/T103653#1396357 (10Krenair) [13:31:26] addshore, still there? [13:31:56] Nuts!! [13:32:28] * Cyberpower678 considers re-applying to bag. [13:33:29] Cyberpower678: good luck :) [13:41:58] zhuyifei1999, thanks. :) [13:42:59] * Cyberpower678 firsts need to have a word with TParis [13:49:57] 6Labs, 10Tool-Labs, 10MediaWiki-extensions-SemanticForms, 10wikitech.wikimedia.org: "Invalid or virtual namespace -1 given" when submitting access request - https://phabricator.wikimedia.org/T103653#1396390 (10Krenair) I made it throw exceptions in SFAutoeditAPI::logMessage and got this on submission: ```2... [14:03:41] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Alex Monk was created, changed by Alex Monk link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Alex_Monk edit summary: Created page with "{{Tools Access Request |Justification=test |Completed=false |User Name=Alex Monk }}" [14:04:55] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Alex Monk was modified, changed by Alex Monk link https://wikitech.wikimedia.org/w/index.php?diff=167532 edit summary: delete [14:05:07] 6Labs, 10Tool-Labs, 10MediaWiki-extensions-SemanticForms, 10wikitech.wikimedia.org: "Invalid or virtual namespace -1 given" when submitting access request - https://phabricator.wikimedia.org/T103653#1396407 (10Krenair) Error is thrown in the internalAttemptSave call in doStore, even though $editor->getTitl... [14:05:57] 6Labs, 10Tool-Labs, 10MediaWiki-extensions-SemanticForms, 10wikitech.wikimedia.org: "Invalid or virtual namespace -1 given" when submitting access request - https://phabricator.wikimedia.org/T103653#1396408 (10Krenair) Changing the internalAttemptSave call to this: ```$t = RequestContext::getMain()->getTit... [14:18:30] YuviPanda, Coren, I’m looking at the puppetmaster if y’all aren’t already in to that. [14:19:43] andrewbogott: kk [14:20:01] looks like apache just… gave up. [14:22:24] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Merlijn van Deen was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=167534 edit summary: [14:22:38] Krenair: ^ seems to work. Thanks :-) [14:23:35] we should figure out exactly what caused that to break (how long has it been broken?). it's some annoying context problem somewhere I think. [14:23:39] and then commit a proper fix [14:24:05] I can get you an upper limit for 'how long' [14:24:35] that would be helpful in figuring out what broke it [14:24:40] Krenair: https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Tools/Access_Request/DB111&oldid=163853 [14:24:52] so... a week [14:25:25] how often did you usually get one? [14:25:49] Varies. One every few days, typically: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request [14:25:52] okay [14:26:01] next train deploy would be the wmf10, the day after that [14:26:19] so I imagine it's something here: https://www.mediawiki.org/wiki/MediaWiki_1.26/wmf10 [14:26:29] Coren: fixed, might have been something to do with log-rotate, so be alert to that if it happens again. [14:30:21] nothing obvious there though [14:32:22] andrewbogott: would you look on why: puppet seems failing to initialize on commonsarchive-test.commonsarchive.eqiad.wmflabs (already a few hours passed), cant login (publickey), console output has lots of [1;31mError: Could not request certificate: Connection reset by peer - SSL_connect[0m [14:32:29] valhallasw, I can't help but wonder if the turning on of some of the anti-abuse features may be involved [14:32:38] zhuyifei1999: yes, puppet is broken somehow, I’m working on this now. [14:32:52] ok [14:33:53] Krenair: Hm, could be. Although for most of those, I would expect my edits to go through [14:34:07] autoconfirmed etc [14:34:19] Krenair: https://gerrit.wikimedia.org/r/#/c/220181/ CR too when you have time :) [14:35:07] YuviPanda, ewww [14:35:15] hahaha [14:35:18] YuviPanda, it should work though [14:35:24] yeah [14:35:39] Krenair: there's also a hook that attaches to UserRights [14:35:41] but like [14:35:45] YuviPanda: ahhh but [14:35:46] Cyberpower678: its open in my browser [14:35:47] in theory I should tell you to document this magic user group [14:35:49] will loook at it when i get time [14:35:55] YuviPanda: you cannot add someone to a project before they ahve the shell right [14:36:06] oh wait [14:36:07] valhallasw: yes, I removed that check [14:36:07] because that's set up by site config, not the extension itself [14:36:08] you removed that [14:36:12] * valhallasw should learn how to read [14:36:26] but... it's OSM :/ [14:36:26] Krenair: yeah but it's all around the place everywhere. [14:36:32] addshore, :D [14:36:42] Krenair: yeah, that too [14:41:25] Coren: I am wrong about this being fixed :( [14:41:56] Krenair: wanna +2? :) [14:42:01] andrewbogott: anything I can help with? [14:42:09] I don’t know yet [14:42:12] 10Tool-Labs-tools-Other, 10Analytics: Work on Metrics tools wm-metrics and MediaCollectionDB, refactoring and code quality. - https://phabricator.wikimedia.org/T100710#1396448 (10JeanFred) 5Open>3Resolved I am happy with the results in the context of the hackathon. Work will continue but this can be closed. [14:42:49] AH00106: piped log program '/usr/bin/cronolog --once-only --period="1 minute" /var/log/apache2/puppetmaster.log' failed unexpectedly [14:42:57] /usr/bin/cronolog: invalid explicit period specification ((null)) [14:43:01] cryptic! [14:43:24] Krenair: woops, needs rebase. [14:43:35] * Coren loves error messages that give no clue about what the actual error is. [14:43:46] Well, it's a clue, but a really crappy one. [14:45:05] --period="1 minute" is perfectly valid afaict [14:45:46] I would think [14:45:55] Krenair: too late for swat today you think? [14:46:45] andrewbogott: Also, hasn't that been the value in some time? I'd have expected it to fail before that. [14:47:03] Yeah, I’m trying to figure out if anything changed. [14:47:17] I think it might be just that it’s complaining about trying to pipe an empty stream or something [14:47:26] in which case it may be a random symptom [14:47:54] Well, the message says that it tries to pupe to cronolog but cronolog dies. [14:48:04] YuviPanda, why aren't you removing the now unused message? [14:48:09] Krenair: oh, should I? [14:48:13] let me do that [14:48:15] well, yeah [14:48:25] it works on the commandline [14:48:56] Krenair: done [14:49:18] ah, 9bb3219a67a671ee1f5da2d6f833e9a0e3c86eab [14:49:19] 6Labs, 10Tool-Labs, 10Labs-Infrastructure, 7Regression: Tools Labs proxy should not overwrite error page - https://phabricator.wikimedia.org/T103662#1396481 (10scfc) IIRC originally the reasoning was that with default error pages, all users would be pointed to `webmaster@tools.wmflabs.org` which would be u... [14:50:49] 6Labs, 10Tool-Labs, 10Labs-Infrastructure, 7Regression: Tools Labs proxy should not overwrite error page - https://phabricator.wikimedia.org/T103662#1396486 (10yuvipanda) If you pass the X-Wikimedia-Debug header and set it to 1 it'll pass them through as well, in the proxy. [14:50:54] YuviPanda, it's not too late for swat if you feel you can justify it needing swat [14:52:34] Krenair: I've updated it now, so it stops creating the shell request as well [14:53:15] YuviPanda, ...? [14:53:26] Krenair: see the new patchset? [14:53:28] it was already merged? [14:53:31] oh? [14:53:34] lol [14:53:35] ok [14:53:38] let me make that into a new patch [14:53:47] you should have got an error when trying to upload a new patchset [14:54:02] Krenair: I did, wasn't paying attention [14:54:07] git push takes a long time so I just tabbed out [14:57:41] 6Labs, 10Tool-Labs, 10MediaWiki-extensions-SemanticForms, 10wikitech.wikimedia.org: "Invalid or virtual namespace -1 given" when submitting access request - https://phabricator.wikimedia.org/T103653#1396492 (10Krenair) a:3Krenair That patch is live on the WMF cluster (on tin, uncommitted), we need to fig... [14:59:25] 6Labs, 10wikitech.wikimedia.org, 5Patch-For-Review: Automatically grant shell user right to everyone who signs up on wikitech - https://phabricator.wikimedia.org/T97334#1396494 (10scfc) Great! When the automation has been tested, we need to remove the "generate shell request page on account creation" logic... [15:00:21] 6Labs, 10wikitech.wikimedia.org, 5Patch-For-Review: Automatically grant shell user right to everyone who signs up on wikitech - https://phabricator.wikimedia.org/T97334#1396502 (10yuvipanda) Yup, I have a patch for the removal as well. [15:02:37] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103: Labs: Salvage, then remove volumes on labstores' raid6 - https://phabricator.wikimedia.org/T103265#1396508 (10coren) [15:02:40] 6Labs, 10Tool-Labs, 10Continuous-Integration-Infrastructure: Recover homedir of "ci" tool - https://phabricator.wikimedia.org/T103205#1396506 (10coren) 5Open>3Resolved Salvaged files are available in ~tools.ci/restore.tgz [15:05:38] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103: Labs: Salvage, then remove volumes on labstores' raid6 - https://phabricator.wikimedia.org/T103265#1396518 (10coren) [15:05:39] 6Labs, 10Incident-20150617-LabsNFSOutage: Recover cssk tool's log files - https://phabricator.wikimedia.org/T103350#1396515 (10coren) 5Open>3Resolved a:3coren Salvaged files have been put into ~tools.cssk/restore.tgz [15:14:49] 6Labs, 10Incident-20150617-LabsNFSOutage: Recover files for project liangent-php - https://phabricator.wikimedia.org/T103268#1396549 (10coren) There are no files in ~/mw/ that have been modified since the backup. The most recent changes date from June 11, and are all located in ~/mw-log/ and ~/sql/dumps/ [15:16:43] andrewbogott: I still see nothing wrong with that cronolog invokation. [15:17:27] Coren: I expect it’s a bug either in cronolog or in how apache is passing things [15:17:37] I can’t tell what the default period is if I don’t specify it [15:18:00] (03PS1) 10Sitic: Fix fontsize and fix layout in Firefox [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/220477 (https://phabricator.wikimedia.org/T103680) [15:18:17] (03CR) 10Sitic: [C: 032 V: 032] Fix fontsize and fix layout in Firefox [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/220477 (https://phabricator.wikimedia.org/T103680) (owner: 10Sitic) [15:18:34] andrewbogott: Might be quoting. Try --period=1minute on the commandline; if that works, that means the apache config can be changed to that and axe the attempt at quoting. [15:18:45] andrewbogott: I.e., in apache, "1 minute" still ends up being two tokens. [15:18:58] I tried many different quoting schemes, but not that one... [15:21:22] Coren: that seems to work! Good thinking :) [15:22:42] andrewbogott: General paranoia about how apache tries to quote the pipe invokation in CustomLog. :-) [15:26:39] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103: Restore some files for tools-magog - https://phabricator.wikimedia.org/T103689#1396658 (10coren) 3NEW a:3coren [15:29:29] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103: Labs: Salvage, then remove volumes on labstores' raid6 - https://phabricator.wikimedia.org/T103265#1396669 (10coren) [15:29:30] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103: Restore some files for tools-magog - https://phabricator.wikimedia.org/T103689#1396667 (10coren) 5Open>3Resolved Both files could be salvaged and are available in ~tools.magog/restore.tgz [15:31:38] Coren: ok, puppet is working again now, thanks to a hotfix of https://gerrit.wikimedia.org/r/#/c/220476/, pending Ori’s review. [15:31:48] oh, there it goes :) [15:38:05] 6Labs, 10Incident-20150617-LabsNFSOutage: Start regular snapshots of the labs filesystems and cross-server backups - https://phabricator.wikimedia.org/T103691#1396700 (10coren) 3NEW a:3coren [15:48:43] 6Labs, 10Tool-Labs, 10MediaWiki-extensions-SemanticForms, 10wikitech.wikimedia.org: "Invalid or virtual namespace -1 given" when submitting access request - https://phabricator.wikimedia.org/T103653#1396759 (10Yaron_Koren) By coincidence, I just added a way to fix this to the SF documentation two days ago... [15:49:50] 6Labs: Investigate if NFS is needed on the language project - https://phabricator.wikimedia.org/T103130#1396760 (10yuvipanda) Anyone? [15:51:09] 6Labs: Investigate if NFS is needed on the language project - https://phabricator.wikimedia.org/T103130#1396765 (10KartikMistry) Looks good to me ie I don't need my HOME! [15:51:28] 6Labs, 3Labs-Sprint-102, 3Labs-Sprint-103: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1396768 (10yuvipanda) [15:51:31] 6Labs: Investigate and disable NFS on the swift project - https://phabricator.wikimedia.org/T102847#1396766 (10yuvipanda) 5Open>3Resolved Has no instances, done. [15:57:23] 6Labs, 10Tool-Labs, 10MediaWiki-extensions-SemanticForms, 10wikitech.wikimedia.org: "Invalid or virtual namespace -1 given" when submitting access request - https://phabricator.wikimedia.org/T103653#1396793 (10Krenair) So probably broke on wikitech due to T102652 [15:58:46] Coren: can you recover /data/project/home of "bots" project? It was lost during outage and never recovered [15:59:12] I had some code in that [15:59:19] which I didn't yet push to github [16:00:14] petan: I can possibly recover specific files, but not blindly copy over. Please open a ticket with https://phabricator.wikimedia.org/maniphest/task/create/?parent=103265 with what you need. [16:01:20] Ah, the entire directory is new content? [16:01:32] Still open a ticket, I'll see what I can do. [16:01:55] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103: Recover /data/project/home/botbot on bots project - https://phabricator.wikimedia.org/T103696#1396803 (10Petrb) 3NEW a:3coren [16:02:11] yes [16:02:16] it was created after the backup was taken [16:03:34] 6Labs, 10Tool-Labs, 10MediaWiki-extensions-SemanticForms, 10wikitech.wikimedia.org: "Invalid or virtual namespace -1 given" when submitting access request - https://phabricator.wikimedia.org/T103653#1396810 (10Krenair) Relevant commit in ConfirmEdit would be https://gerrit.wikimedia.org/r/#/c/209637/ but w... [16:07:39] 6Labs, 6operations: Recover home folders and /data/project from wikimetrics1 - https://phabricator.wikimedia.org/T103530#1396826 (10yuvipanda) I'm going to: # Bring back /data/project # Copy the old contents of your home folders into /data/project/home This means your actual /home folders will not be on NFS,... [16:09:07] 6Labs, 6operations: Recover home folders and /data/project from wikimetrics1 - https://phabricator.wikimedia.org/T103530#1396827 (10yuvipanda) alright, if you run puppet on instances now you'll get /data/project back. I've copied over the contents of home folders into /data/project/home as well. [16:09:24] 6Labs, 10Tool-Labs, 10MediaWiki-extensions-SemanticForms, 10wikitech.wikimedia.org: "Invalid or virtual namespace -1 given" when submitting access request - https://phabricator.wikimedia.org/T103653#1396828 (10Krenair) I think it's likely to be the SpamBlacklist code. AbuseFilter would've already been on,... [16:10:20] Krinkle_: just wnated to say I intend to respond to your email soon, it's not forgotten, just lost amongst lots of other stuff [16:14:16] 6Labs, 6operations: Recover home folders and /data/project from wikimetrics1 - https://phabricator.wikimedia.org/T103530#1396833 (10mforns) Awesome, thanks! Personally, I don't need shared home folders much. [16:18:10] 6Labs, 3Labs-Sprint-102, 3Labs-Sprint-103: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1396858 (10yuvipanda) [16:18:13] 6Labs, 10Beta-Cluster: Disable NFS home directories on deployment-prep - https://phabricator.wikimedia.org/T102169#1396856 (10yuvipanda) 5Open>3Resolved This is just the homedirs, and they are gone. [16:18:23] 6Labs: Audit get rid of NFS usage in social-tools project - https://phabricator.wikimedia.org/T102391#1396859 (10yuvipanda) this is done now. let me know if you want a copy of your old homedirs somewhere [16:19:40] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Alex Monk was modified, changed by Alex Monk link https://wikitech.wikimedia.org/w/index.php?diff=167571 edit summary: [16:20:12] 6Labs, 3Labs-Sprint-102, 3Labs-Sprint-103: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1396873 (10yuvipanda) [16:20:15] 6Labs: Audit get rid of NFS usage in social-tools project - https://phabricator.wikimedia.org/T102391#1396870 (10yuvipanda) 5Open>3Resolved a:3yuvipanda 'now' as in during the NFS outage. [16:20:31] 6Labs, 10Tool-Labs, 10MediaWiki-extensions-SemanticForms, 10MediaWiki-extensions-SpamBlacklist, 10wikitech.wikimedia.org: "Invalid or virtual namespace -1 given" when submitting access request - https://phabricator.wikimedia.org/T103653#1396874 (10Krenair) Yeah. Disabled the SemanticForms live hack I had... [16:21:08] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Alex Monk was modified, changed by Alex Monk link https://wikitech.wikimedia.org/w/index.php?diff=167573 edit summary: Undo revision 167571 by [[Special:Contributions/Alex Monk|Alex Monk]] ([[User talk:Alex Monk|talk]]) [16:21:12] 6Labs: Investigate wether NFS mounts are really necessary for HHVM project - https://phabricator.wikimedia.org/T102371#1396879 (10yuvipanda) I removed them during the NFS outage, and they're all up now without NFS (both hhvm and hat-imagescalers). Recovered files can be found on labstore1002, and I can copy them... [16:23:31] (03PS1) 10Sitic: Show links to page history and user contribs [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/220491 (https://phabricator.wikimedia.org/T103677) [16:23:48] (03CR) 10Sitic: [C: 032 V: 032] Show links to page history and user contribs [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/220491 (https://phabricator.wikimedia.org/T103677) (owner: 10Sitic) [16:29:16] Hi everybody [16:30:00] The following URL is returning the php source code instead of its output: [16:30:01] https://tools.wmflabs.org/spellcheck/de/checkArticle.php?pageName=Deutschland [16:30:52] looks like php has been disabled for this resource [16:41:37] Martin__: what is it supposed to return? [16:42:10] Martin__: also, is this used in a Gadget on dewiki somehow? [16:42:15] I dont't know - guess a result for the spell check [16:42:32] but definitly not a bunch of php-variables and code [16:42:55] 6Labs, 10Labs-Other-Projects, 3Labs-Sprint-103: investigate/clean up 'servermon' project - https://phabricator.wikimedia.org/T103149#1396968 (10Andrew) 5Open>3Resolved ok, deleted. Thank you! [16:43:06] Yes it is: I found this Error in my Console after opening a de.Wiki-Page [16:43:10] Martin__: yeah, I'm just not sure whether 'an empty page' is a corret result [16:44:25] 10Tool-Labs-tools-Other: "spellcheck" tool uses php short tags - https://phabricator.wikimedia.org/T103707#1396971 (10valhallasw) 3NEW [16:44:28] Yout should ask the person who developed this tool about this [16:46:01] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103: Recovering my Python files in the root - https://phabricator.wikimedia.org/T103708#1396984 (10marcmiquel) 3NEW a:3coren [16:46:37] 6Labs: Investigate wether NFS mounts are really necessary for HHVM project - https://phabricator.wikimedia.org/T102371#1396992 (10yuvipanda) 5Open>3Resolved a:3yuvipanda I removed them during the NFS outage, and they're all up now without NFS (both hhvm and hat-imagescalers). Recovered files can be found o... [16:46:38] 6Labs, 3Labs-Sprint-102, 3Labs-Sprint-103: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1396995 (10yuvipanda) [16:47:20] 6Labs: Disable NFS on the orgcharts project - https://phabricator.wikimedia.org/T103137#1396996 (10yuvipanda) @Dzahn should be a completely separate discussion from this bug, however :) [16:47:46] 10Tool-Labs-tools-Other: "spellcheck" tool only returns empy pages - https://phabricator.wikimedia.org/T103709#1396997 (10valhallasw) 3NEW [16:50:09] thanks (I was just the messenger ;) [16:51:25] 6Labs, 10Tool-Labs: Monitor bigbrother - https://phabricator.wikimedia.org/T90850#1397020 (10yuvipanda) [16:53:27] YuviPanda: Quis custodiet ipsos custodes? [16:53:44] nobody, after you take them out the back and shoot them in the head... [17:13:53] 6Labs, 10wikitech.wikimedia.org, 5Patch-For-Review, 5WMF-deploy-2015-06-23_(1.26wmf11), 5WMF-deploy-2015-06-30_(1.26wmf12): Automatically grant shell user right to everyone who signs up on wikitech - https://phabricator.wikimedia.org/T97334#1397095 (10yuvipanda) So this works, except that the user doesn'... [17:14:04] andrewbogott_afk: do you know if there are existing problems about users not being added to bastion project when granted shell? [17:16:00] YuviPanda: worked this morning for me. Try doing it manually? [17:20:31] 10Tool-Labs-tools-Other: "spellcheck" tool uses php short tags - https://phabricator.wikimedia.org/T103707#1397112 (10APPER) 5Open>3Resolved a:3APPER [17:20:43] 10Tool-Labs-tools-Other: "spellcheck" tool uses php short tags - https://phabricator.wikimedia.org/T103707#1396971 (10APPER) Fixed that issue. [17:22:20] YuviPanda: /data/scratch is the non-nfs storage thing right? I couldn't find much docs about it [17:24:38] valhallasw: ^? [17:24:50] legoktm: scratch is also nfs [17:24:54] just not backed up, I think [17:25:11] so...what's the advantage in using it? [17:27:29] it saves resources because it's not backed up, I think =p [17:28:00] but you shouldn't put huge stuff on nfs anyway, so I'm not sure what the point is [17:28:35] 10Tool-Labs-tools-Other: "spellcheck" tool only returns empy pages - https://phabricator.wikimedia.org/T103709#1397143 (10APPER) 5Open>3Invalid a:3APPER It's a correct behavior, because the article "Deutschland" doesn't have any spelling mistakes. [17:28:54] 6Labs, 6operations, 10ops-eqiad, 5Patch-For-Review, 3ToolLabs-Goals-Q4: Rename virt1000 to labcontrol1002, move to same subnet as labcontrol1001 - https://phabricator.wikimedia.org/T102646#1397149 (10Cmjohnson) Physically moved the server to row/rack C7 Completed dns changes for labcontrol1002 https://g... [17:30:19] hello my name is yusuf ameri. I need help getting access to the bastion. my shell username is whyameri [17:43:02] 6Labs, 10wikitech.wikimedia.org, 5Patch-For-Review, 5WMF-deploy-2015-06-23_(1.26wmf11), 5WMF-deploy-2015-06-30_(1.26wmf12): Automatically grant shell user right to everyone who signs up on wikitech - https://phabricator.wikimedia.org/T97334#1397214 (10scfc) That would be a new problem. Why is the conseq... [17:46:13] 6Labs, 10wikitech.wikimedia.org, 5Patch-For-Review, 5WMF-deploy-2015-06-23_(1.26wmf11), 5WMF-deploy-2015-06-30_(1.26wmf12): Automatically grant shell user right to everyone who signs up on wikitech - https://phabricator.wikimedia.org/T97334#1397228 (10scfc) Sorry, wrong code location. I meant `special/S... [17:46:27] legoktm: it's NFS, just... ephemeral? and not restricted to one project [17:47:11] valhallasw: uh, doing [17:48:34] valhallasw: hmm, doing it manually works [17:48:47] 6Labs, 10wikitech.wikimedia.org, 5Patch-For-Review, 5WMF-deploy-2015-06-23_(1.26wmf11), 5WMF-deploy-2015-06-30_(1.26wmf12): Automatically grant shell user right to everyone who signs up on wikitech - https://phabricator.wikimedia.org/T97334#1397232 (10yuvipanda) Doing it manually works. [18:08:04] 6Labs, 3Labs-Sprint-103, 3ToolLabs-Goals-Q4: virt1000 SPOF - https://phabricator.wikimedia.org/T90625#1397297 (10Andrew) [18:08:07] 6Labs, 6operations, 10ops-eqiad, 5Patch-For-Review, 3ToolLabs-Goals-Q4: Rename virt1000 to labcontrol1002, move to same subnet as labcontrol1001 - https://phabricator.wikimedia.org/T102646#1397295 (10Andrew) 5Open>3Resolved thank you! I can do the install. [18:08:27] 6Labs, 3Labs-Sprint-103, 3ToolLabs-Goals-Q4: Set up labcontrol1002 as hot spare for labcontrol1001. - https://phabricator.wikimedia.org/T103722#1397300 (10Andrew) 3NEW a:3Andrew [18:11:41] Hi, can anyone explain me if/how the tools' list in tools.wmflabs.org reads toolinfo.json? My tool isn't getting linked... [18:13:28] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103: Labs: Salvage, then remove volumes on labstores' raid6 - https://phabricator.wikimedia.org/T103265#1397332 (10coren) [18:13:29] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103: Recovering my Python files in the root - https://phabricator.wikimedia.org/T103708#1397330 (10coren) 5Open>3Resolved All the python files that were in your home have been salvaged and copied to ~/recover.tgz [18:14:34] 6Labs, 10wikitech.wikimedia.org, 5Patch-For-Review, 5WMF-deploy-2015-06-23_(1.26wmf11), 5WMF-deploy-2015-06-30_(1.26wmf12): Automatically grant shell user right to everyone who signs up on wikitech - https://phabricator.wikimedia.org/T97334#1397334 (10scfc) Okay, rereading the code made it now clear to m... [18:17:11] petan: There is over 350M of files in that directory; surely you don't actually have all of that that is irreplacable? [18:17:40] is there a tool that gives page views for labs instances like devhub and living-style-guide.wmflabs.org ? [18:17:52] spagewmf: nope, sadly. [18:18:41] 6Labs, 10wikitech.wikimedia.org, 5Patch-For-Review, 5WMF-deploy-2015-06-23_(1.26wmf11), 5WMF-deploy-2015-06-30_(1.26wmf12): Automatically grant shell user right to everyone who signs up on wikitech - https://phabricator.wikimedia.org/T97334#1397358 (10scfc) I. e., `manageShellAccess()` shouldn't be hooke... [18:19:18] YuviPanda: hello there! thanks, I'll file an enhancement request. Maybe there's a quick and dirty way to use the access logs of a labs-vagrant instance... [18:19:26] YuviPanda for CTO! [18:19:35] spagewmf: don't you mean legoktm for CTO? :D [18:19:55] 6Labs, 10wikitech.wikimedia.org, 5Patch-For-Review, 5WMF-deploy-2015-06-23_(1.26wmf11), 5WMF-deploy-2015-06-30_(1.26wmf12): Automatically grant shell user right to everyone who signs up on wikitech - https://phabricator.wikimedia.org/T97334#1397360 (10yuvipanda) That seems totally possible! [18:21:00] YuviPanda: tough to support a candidate who won't show his or her face https://wikimediafoundation.org/wiki/Staff_and_contractors?showall=1 [18:21:37] when I joined WMF and people urged me to "get a cloak", I thought it was somehow related to "Wearing Cloak of Invisibility" :-) [18:22:09] spagewmf: have you looked at mine? I look homeless [18:23:14] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103: Recover /data/project/home/botbot on bots project - https://phabricator.wikimedia.org/T103696#1397370 (10coren) botbot's home contains well over 300G of files, most of which seem to be packages of some sort. Please specify the exact files (or subse... [18:23:35] I look like a whiteneck, what's your point? Oooh, The Man moved me to Reading Product Management [18:23:46] hahaha [18:24:37] 6Labs, 10Incident-20150617-LabsNFSOutage: Recover files for project liangent-php - https://phabricator.wikimedia.org/T103268#1397375 (10coren) 5Open>3Resolved a:3coren [18:24:38] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103: Labs: Salvage, then remove volumes on labstores' raid6 - https://phabricator.wikimedia.org/T103265#1397377 (10coren) [18:25:16] Hi? [18:28:19] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103: Recover files for taxonbot - https://phabricator.wikimedia.org/T103725#1397394 (10coren) 3NEW a:3coren [18:29:01] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103: Recover files for taxonbot - https://phabricator.wikimedia.org/T103725#1397394 (10coren) [18:30:49] jem: I don't think it's documented anywhere, but it looks at least for ~/toolinfo.json or ~/public_html/toolinfo.json [18:31:15] jem: you probably also need to make sure that it has the permissions to read it [18:31:55] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103: Recover files for taxonbot - https://phabricator.wikimedia.org/T103725#1397404 (10coren) 5Open>3Resolved The files have been salvaged and may be found in ~tools.taxonbot/recover.tgz [18:31:56] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103: Labs: Salvage, then remove volumes on labstores' raid6 - https://phabricator.wikimedia.org/T103265#1397406 (10coren) [18:34:47] jem: it does show info from toolsinfo.json (or something like that), but see hays directory for details on how to set it uo [18:36:01] marmick: hey! do you want files recovered? poke Coren! [18:36:04] he might be able to [18:36:36] thanks for remembering YuviPanda! :) he just did it. i'm checking them. it seems they are all ok! :) great [18:40:52] 6Labs, 12Analytics-Backlog, 10Labs-Infrastructure: Report page views for labs instances - https://phabricator.wikimedia.org/T103726#1397418 (10Spage) 3NEW [18:46:44] 6Labs, 12Analytics-Backlog, 10Labs-Infrastructure: Report page views for labs instances - https://phabricator.wikimedia.org/T103726#1397438 (10Spage) [18:52:59] sitic, valhallasw: Thanks, yes, I have ~/public_html/toolinfo.json and I have the "url" included, but other tools have the same, are not included in Hay's directory wiki page, and the link for that URL does appear in the tools.wmflabs.org index [18:53:20] And yes, the toolinfo.json is readable for all [18:54:14] jem: oh, you mean the link at the name of the project? there's a bug for that, I think [18:54:30] Yes, I mean that [18:54:38] but if you have a correct toolinfo.json, it will show up as a table on the right side anyway [18:55:04] 6Labs, 10Tool-Labs, 10Continuous-Integration-Infrastructure: Recover homedir of "ci" tool - https://phabricator.wikimedia.org/T103205#1397469 (10Legoktm) Thanks! [18:55:35] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103: Labs: Make a new backup of the Labs storage to codfw - https://phabricator.wikimedia.org/T103356#1397470 (10coren) labstore2001 has the new controller, but it's proving to be //amusing// to configure to our needs. In the meantime, and to make sure... [18:56:22] I don't know if I understand, valhallasw [18:56:48] jem: scroll down to 'hay' and you'll see a table on the right [18:56:56] which is read from toolinfo.json aiui [18:57:08] Yes [18:57:29] I know that part of the toolinfo.json works [18:57:56] The strange thing was the link on the tool name [18:58:05] (Or is) [18:59:35] the basic rule was 'there is a public_html', but I think that was scrapped at some point [19:00:42] I see when there is more than one "subtool", the url parameter shows as the link for the tool name [19:00:48] but it looks like some sort of database hack [19:00:53] ?? [19:01:23] But if it is just one tool, the only name is at the left column, and the link shows... I don't know when or how [19:01:49] 6Labs, 10Tool-Labs: tools missing in tools.wmflabs.org / toollabs_p - https://phabricator.wikimedia.org/T100931#1397482 (10valhallasw) The lack of updating also causes weird issues with tools not being linked. [19:01:55] ^ it's this bug. [19:02:35] Ok then, let me try to find the bug [19:02:52] 6Labs, 10Tool-Labs: tools missing in tools.wmflabs.org / toollabs_p - https://phabricator.wikimedia.org/T100931#1397485 (10coren) Ah, this would be a consequence of the cron jobs being suspended on restoration of the filesystem. I will check that system cron jobs are reenabled. [19:02:57] ^^ [19:03:03] Should be trivial to fix. [19:03:25] I think that, too :) [19:03:59] Reenabled cron, doing a manual run now. Let's see if that sufficed. [19:04:14] How's that? [19:04:14] 6Labs, 3Labs-Sprint-102, 3Labs-Sprint-103: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1397491 (10hashar) [19:04:17] 6Labs, 10Beta-Cluster: Disable NFS home directories on deployment-prep - https://phabricator.wikimedia.org/T102169#1397489 (10hashar) 5Resolved>3Open Not yet. ``` hashar@deployment-salt:~$ sudo salt '*' cmd.run 'mount|grep /home' deployment-fluorine.deployment-prep.eqiad.wmflabs: deployment-sca02.deploymen... [19:04:34] :) Great [19:04:58] Link for the tool and for the author also, that was missing [19:05:25] The .description file is obsolete if I use the json, right? [19:05:40] jem: It is; it prefers the .json if you have it [19:05:47] 6Labs, 10Tool-Labs: tools missing in tools.wmflabs.org / toollabs_p - https://phabricator.wikimedia.org/T100931#1397497 (10coren) 5Open>3Resolved a:3coren Reenabled the cron job and did a manual update. That seems to have fixed it. [19:06:00] Ok, I'll remove to not confuse myself in the future [19:07:42] Ah [19:07:59] "Ah"? [19:07:59] In http://tools.wmflabs.org/?tool=jembot only .description is used [19:08:08] So I can't remove it for the moment [19:08:28] Is that going to be changed? [19:08:36] Hm. ?tool is decidedly obsolete and needs to use the same mechanism as the general list [19:09:07] Ok [19:09:34] But I have arrived there somehow... [19:09:45] Yes, in http://tools.wmflabs.org/?status [19:10:01] The link to the tools uses ?tool= [19:18:01] Coren: should that be changed? Is a bug needed? [19:18:30] It should be. It's also not very hard, the php source isn't complicated. It's really just a matter of available time. [19:21:30] Ok, so probably pinging the right person (who?) could be more effective than opening a bug :) [19:24:43] 6Labs, 10Beta-Cluster: Disable NFS home directories on deployment-prep - https://phabricator.wikimedia.org/T102169#1397551 (10hashar) [19:31:37] 6Labs, 10Beta-Cluster: Disable NFS home directories on deployment-prep - https://phabricator.wikimedia.org/T102169#1397564 (10hashar) I have fixed DNS on the i-** instances. * deployment-cache-upload02 fixed up (dns/puppet/certs etc) * deployment-zookeeper01 no more has the /home NFS dir after a reboot * de... [20:06:07] jem: I'm the one who wrote most of that code, but some time ago and my schedule really does not allow me the luxury of playing with it. If you want to take a stab at a patch, though, I can review it. [20:21:00] Coren, don't worry, it isn't so urgent nor important [20:21:37] And my time is also very limited too [20:22:36] For the moment I'll note it for vacances in August, and I'll ask again then [20:49:27] 6Labs, 5Patch-For-Review: Make a labs_storage module - https://phabricator.wikimedia.org/T93781#1397840 (10coren) 5Resolved>3Open After discussion with @faidon, this is going to actually move to a module now (except named 'labstore' for clarity). [20:55:47] 6Labs, 10Tool-Labs, 10MediaWiki-extensions-SemanticForms, 10MediaWiki-extensions-SpamBlacklist, 10wikitech.wikimedia.org: "Invalid or virtual namespace -1 given" when submitting access request - https://phabricator.wikimedia.org/T103653#1397855 (10Yaron_Koren) Okay, it's good to know that this problem al... [21:02:33] YuviPanda: meetbot it dead ? [21:02:47] matanya: dunno? ask its maintainers? [21:02:55] it could very well be... [21:03:03] YuviPanda: in terms of labs outages [21:03:08] oh, right. [21:03:28] it's holding up an RFC meeting [21:03:31] 6Labs, 10Tool-Labs, 10MediaWiki-extensions-SemanticForms, 10MediaWiki-extensions-SpamBlacklist, 10wikitech.wikimedia.org: "Invalid or virtual namespace -1 given" when submitting access request - https://phabricator.wikimedia.org/T103653#1397893 (10Krenair) I'm not sure there is a better way around it in... [21:03:36] bblack: matanya I restarted it [21:03:41] thanks [21:03:48] bblack: matanya let me know if that works [21:03:59] it did [21:04:03] thanks [21:25:05] JohnFLewis: ping [21:25:09] does the wmt project need NFS? [21:26:03] YuviPanda: We use /data/project but not the rest. use of /data/project could probably be re-evaluated as well [21:26:20] JohnFLewis: so how about I 1. file a bug, 2. disable everything else? [21:26:28] so that'll be home and scratch and dumps [21:26:31] that works for me [21:27:39] YuviPanda: I thinj /home/southparkfan may have content in it (I could check right now but busy) if it does, could you move that elsewhere/recover it on wmt-exec? [21:27:56] JohnFLewis: how about I make a backup of home into /data/project? [21:28:04] please :) [21:28:12] JohnFLewis: sweet. on it now. [21:28:34] YuviPanda: do what you need :) [21:28:57] I'm not sure though if I have content in my home dir.. [21:29:04] 6Labs: Re-evaluate use of NFS in WMT project - https://phabricator.wikimedia.org/T103750#1397999 (10yuvipanda) 3NEW a:3yuvipanda [21:29:08] there you go [21:30:15] cool [21:31:47] SPF|Cloud: :) just disabling NFS everywhere except /data/project [21:32:12] SPF|Cloud: JohnFLewis will the instances be ok with a restart? or should I do it without restarts? [21:32:30] Well we don't run anything important [21:32:39] a restart is fine, things have been stalled on implementing so the project is more or less unused atm [21:32:45] So I guess JohnFLewis agrees with a reboot [21:33:25] alright then [21:35:02] !log wmt rebooting all instances, cleansing of NFS [21:35:07] Logged the message, Master [21:36:55] Only global cloudadmins and project admins for project Wmt can edit this pag [21:36:57] yaay. [21:37:17] well idc actually [21:37:51] I actually still had stuff in my home dir but I won't bug you to revert the change since it's not important :) [21:38:20] SPF|Cloud: no, I'm going to make your homedir available in /data/project [21:38:24] so you can get stuff off it [21:38:25] cool..! [21:39:25] !log wmt log test :D [21:39:29] Logged the message, Master [21:39:46] And change reverted on the SAL, cool bot [21:39:50] :) [21:40:08] SPF|Cloud: JohnFLewis hmm, wmt-apache and wmt-exec seem unavailable instances... [21:40:28] YuviPanda: hm? [21:40:29] wmt.wmflabs.org is still working [21:40:43] I see two instances that I can't ssh into even with my root key [21:40:46] (which should be served by wmt-apache) [21:40:46] wmt-apache and wmt-exec [21:41:21] anyway, rebooting via other means! [21:41:45] from my part you may do everything as needed [21:42:16] thank you [21:43:16] JohnFLewis: I seriously wonder why our bastion is bigger than our apache. [21:44:06] SPF|Cloud: the whole things needs to be redone anyway which is why we need to talk about it when you have time [21:44:28] Well you can do now if you want [21:44:45] But if needs much time then somewhere next week [21:45:07] well it needs both of us so next week then :) [21:45:19] 6Labs: Disable NFS on the orgcharts project - https://phabricator.wikimedia.org/T103137#1398024 (10MarkTraceur) orgchart can be murdered, nobody is maintaining it, if I had it to do again I wouldn't do it this way. The data is in mongodb, which can probably be dumped relatively trivially. Thanks! [21:50:45] 6Labs: Disable NFS for 'planet' project - https://phabricator.wikimedia.org/T102695#1398039 (10yuvipanda) Dzahn says ok, and there are no current instances - done! [22:25:16] 6Labs, 10Beta-Cluster: Completely remove Beta Cluster dependency on NFS - https://phabricator.wikimedia.org/T102953#1398133 (10greg) [22:25:44] 6Labs, 10Beta-Cluster: Completely remove Beta Cluster dependency on NFS - https://phabricator.wikimedia.org/T102953#1378487 (10greg) [22:26:10] 6Labs, 10Beta-Cluster: Completely remove Beta Cluster dependency on NFS - https://phabricator.wikimedia.org/T102953#1378487 (10greg) [22:26:44] 6Labs, 10Beta-Cluster: Disable NFS home directories on deployment-prep - https://phabricator.wikimedia.org/T102169#1398147 (10greg) [22:26:46] 6Labs, 10Beta-Cluster: Completely remove Beta Cluster dependency on NFS - https://phabricator.wikimedia.org/T102953#1378487 (10greg) [22:28:00] JohnFLewis: SPF|Cloud can you get into wmt-apache or wmt-exec? [22:28:01] I can't [22:28:36] I'm on my phone [22:29:00] But I have ssh keys everywhere in the most strange places so I'll see if I can find it and ssh [22:29:35] I can't with me new ssh key and I can't get to bastion with my old so nope [22:29:44] ugh [22:30:49] I cant [22:36:22] 6Labs: Investigate and remove NFS from embed-sandbox project - https://phabricator.wikimedia.org/T103755#1398181 (10yuvipanda) 3NEW a:3yuvipanda [22:37:51] 6Labs: Investigate and remove NFS from embed-sandbox project - https://phabricator.wikimedia.org/T103755#1398203 (10brion) Nuke the NFS and the instances, they're obsolete anyway, but leave the project. I'll set up fresh instances as necessary when I get back to this later this year. [22:38:16] 6Labs: Investigate and remove NFS from embed-sandbox project - https://phabricator.wikimedia.org/T103755#1398207 (10yuvipanda) Yessir! [22:45:11] 6Labs, 3Labs-Sprint-102, 3Labs-Sprint-103: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1398220 (10yuvipanda) [22:45:14] 6Labs: Investigate and remove NFS from embed-sandbox project - https://phabricator.wikimedia.org/T103755#1398218 (10yuvipanda) 5Open>3Resolved Done. [22:50:02] 6Labs, 3Labs-Sprint-102, 3Labs-Sprint-103: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1398231 (10yuvipanda) [22:50:05] 6Labs: Investigate and disable NFS on 'dns' project - https://phabricator.wikimedia.org/T102384#1398228 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Instance deleted (@Dzahn was ok with it!) and NFS disabled as well. [22:50:06] 6Labs, 3Labs-Sprint-103, 3ToolLabs-Goals-Q4: virt1000 SPOF - https://phabricator.wikimedia.org/T90625#1398234 (10Andrew) [22:50:08] 6Labs, 6operations, 10ops-eqiad, 5Patch-For-Review, 3ToolLabs-Goals-Q4: Rename virt1000 to labcontrol1002, move to same subnet as labcontrol1001 - https://phabricator.wikimedia.org/T102646#1398232 (10Andrew) 5Resolved>3Open Chris, I turn out to be stumped with partman and also to have trapped this bo... [22:50:26] 6Labs, 6operations, 10ops-eqiad, 5Patch-For-Review, 3ToolLabs-Goals-Q4: Rename virt1000 to labcontrol1002, move to same subnet as labcontrol1001 - https://phabricator.wikimedia.org/T102646#1398235 (10Andrew) a:5Andrew>3Cmjohnson [22:54:35] 6Labs, 3Labs-Sprint-102, 3Labs-Sprint-103: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1398263 (10yuvipanda) [22:54:37] 6Labs, 10Wikimedia-IEG-grant-review: Audit and potentially remove all NFS mounts in grantreview project - https://phabricator.wikimedia.org/T102373#1398260 (10yuvipanda) 5Open>3Resolved a:3yuvipanda @bd808 said it was ok, and it has been made ok! All gone now. [22:58:38] 6Labs, 6Discovery, 10Maps: Investigate and reduce NFS use in maps-team project - https://phabricator.wikimedia.org/T103757#1398271 (10yuvipanda) 3NEW a:3yuvipanda [22:58:54] MaxSem: https://phabricator.wikimedia.org/T103755 [23:03:46] 6Labs: Investigate and remove netflow project's NFS - https://phabricator.wikimedia.org/T103759#1398343 (10yuvipanda) 3NEW a:3yuvipanda [23:08:40] 6Labs: Investigate and remove netflow project's NFS - https://phabricator.wikimedia.org/T103759#1398402 (10JKrauska) ok by me to shut down instances related to netflow -- they are non-operational at the moment. [23:17:38] 6Labs, 3Labs-Sprint-102, 3Labs-Sprint-103: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1398437 (10yuvipanda) [23:17:40] 6Labs: Investigate and remove netflow project's NFS - https://phabricator.wikimedia.org/T103759#1398435 (10yuvipanda) 5Open>3Resolved NFS killed, flow1 in SHUTOFF, and flow-localpuppet deleted (local puppetmaster had fallen behind way too much, does not recognize even my root key). Thanks Joel! [23:31:51] 6Labs: Disable NFS in puppet3-diffs project - https://phabricator.wikimedia.org/T103760#1398458 (10yuvipanda) 3NEW a:3yuvipanda [23:43:02] Wikitech Main_Page: replaced RT link with Phab [23:43:13] made Phab clickable [23:43:39] 6Labs, 6Research-and-Data: Investigate and potentially get rid of NFS usage in shiny-r project - https://phabricator.wikimedia.org/T102678#1398503 (10yuvipanda) @Ironholds I've left your old homedirs in /tmp/home