[00:01:23] (03PS1) 10Sitic: Add coloring to differentiate projects [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/224375 [00:01:41] (03CR) 10Sitic: [C: 032 V: 032] Add coloring to differentiate projects [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/224375 (owner: 10Sitic) [00:53:34] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Hardwigg was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=170005 edit summary: [02:11:04] YuviPanda: quick question, whats the state of yuvipanda/python-mwapi? I'm thinking about using it for a little project... [05:04:04] PROBLEM - SSH on tools-exec-1401 is CRITICAL: Server answer [05:09:07] RECOVERY - SSH on tools-exec-1401 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2 (protocol 2.0) [05:57:47] 6Labs, 10Tool-Labs, 7Database: Replication lag on multiple databases on tool-labs - https://phabricator.wikimedia.org/T105585#1448640 (10Nemo_bis) Lag seems suddenly resolved, mostly under few minutes with few exceptions. [05:58:23] Clearyly 12 < 7. Sob. https://phabricator.wikimedia.org/T105585#1448573 [06:30:07] 10Quarry: quarry.wmflabs.org/robots.txt gives 502 - https://phabricator.wikimedia.org/T105662#1448645 (10Nemo_bis) 3NEW [07:23:54] 6Labs, 10Tool-Labs, 10Pywikibot-OAuth: Add OAuth to https://tools.wmflabs.org/pywikibot-testwiki/ - https://phabricator.wikimedia.org/T104291#1448697 (10VcamX) @Ladsgroup You're not a member of this service group? If so, what should I do next? @jayvdb [08:03:44] 6Labs, 7Database: Tables corrupted or impossible to work with them - https://phabricator.wikimedia.org/T105503#1448726 (10jcrespo) Can you check again? [08:05:20] 6Labs, 10Tool-Labs: Replication lag on multiple databases on tool-labs - https://phabricator.wikimedia.org/T105585#1448727 (10jcrespo) [08:06:39] 6Labs, 10Tool-Labs: Replication lag on multiple databases on tool-labs - https://phabricator.wikimedia.org/T105585#1447265 (10jcrespo) @Betacommand, can you check again? [08:40:51] 6Labs, 7Database: Tables corrupted or impossible to work with them - https://phabricator.wikimedia.org/T105503#1448848 (10marcmiquel) Now it does work.Thanks! Cheers, Marc [09:00:38] 6Labs, 7Database: Tables corrupted or impossible to work with them - https://phabricator.wikimedia.org/T105503#1448874 (10jcrespo) @marcmiquel To clarify, are all issues solved or just the connection? [09:03:48] 6Labs, 7Database: Tables corrupted or impossible to work with them - https://phabricator.wikimedia.org/T105503#1448877 (10marcmiquel) Apparently all issues are solved. I'm a bit surprised by all these changes. I haven't changed any line of code. Thanks for your attention. Marc [09:04:27] 6Labs, 7Database: Tables corrupted or impossible to work with them - https://phabricator.wikimedia.org/T105503#1448878 (10jcrespo) 5Open>3Resolved a:3jcrespo [10:03:54] 6Labs, 10Labs-Infrastructure: New jessie instance can't attach to puppet due to wrong certname - https://phabricator.wikimedia.org/T102108#1449009 (10hashar) I have deleted the two tests instances I created on the integration labs project. [10:46:33] Hello everyboy. [10:46:39] everybody [10:47:25] I was wondering if someone has connect to the sql database enwiki through R? [10:57:47] Can someone restart https://tools.wmflabs.org/blockcalc/ or something, has been down for a few months now... [11:00:15] Please [11:02:28] Somebody out there used R to connect to the sql ? [11:12:17] I think it just if the webservice that has haged/got turned off... [11:22:05] ? [11:25:15] 6Labs, 10Tool-Labs: Replication lag on multiple databases on tool-labs - https://phabricator.wikimedia.org/T105585#1449114 (10Steinsplitter) yes, looks fine now. [11:37:04] Josve05a: Have you tried contacting the two maintainers of the tool? If it has been down for a few months, then the problem could reappear soon if they aren't paying attention [11:37:45] they (one of them) seem to b aware, from the talk page, but is not activly maintaining it... [11:38:15] Well... [11:38:51] If he's aware he should be at least capable of doing a "webservice restart" [11:39:41] Or give permissions to someone who could [12:24:31] I was wondering if someone has connected to the sql database enwiki through R? [13:38:10] (03Abandoned) 10John F. Lewis: add dummy lists.wm.o key [labs/private] - 10https://gerrit.wikimedia.org/r/224208 (owner: 10John F. Lewis) [13:48:25] 6Labs, 6WMF-Legal: Make sure tools can be taken over after they are abandoned - https://phabricator.wikimedia.org/T102066#1449274 (10Ricordisamoa) >>! In T102066#1438355, @csteipp wrote: > they should be aware that by using OAuth, we reserve the right to revoke the Consumer if it seems like it is being misused... [13:55:52] 10Tool-Labs-tools-Other, 7Epic: Convert all Labs tools to use cdnjs for static libraries - https://phabricator.wikimedia.org/T103934#1449282 (10Ricordisamoa) Whoever converts more tools will get a kitten on his/her talk page. Hurry up, spread the word! [14:54:02] I was wondering if someone has connected to the sql database enwiki through R? [15:59:14] YuviPanda: want i should merge https://gerrit.wikimedia.org/r/#/c/223828/1? [15:59:39] andrewbogott: yeah let me do that and babysit... [15:59:44] ‘k [16:00:06] I’m trying to cut down on my list of pending reviews… lots in there that I’ve already +1’d, I wish gerrit would exclude such things from the list. [16:00:50] andrewbogott: I think you can customize it to do so [16:00:56] hm... [16:00:59] * andrewbogott looks [16:02:37] Krenair: can you try to log in to bastion-01? [16:03:15] YuviPanda, works for me [16:03:40] cool thanks [16:06:39] Coren, ping [16:12:55] Cyberpower678: I think he's travelling today so best to email him I believe [16:13:10] JohnFLewis, it's not urgent [16:13:45] well if you want a response naturally, email otherwise hope he sees it eventually [16:27:04] 6Labs, 10Tool-Labs, 10Wikimania-Hackathon-2015, 3Labs-Sprint-106: Conduct a Tool Labs Workshop in Wikimania hackathon - https://phabricator.wikimedia.org/T91061#1449715 (10yuvipanda) [16:27:32] 6Labs, 10Labs-Infrastructure, 3Labs-Sprint-105, 3Labs-Sprint-106: replica.my.cnf creation broken - https://phabricator.wikimedia.org/T104453#1449716 (10yuvipanda) [16:27:48] YuviPanda: have a moment to help me out with a puzzle? See if you can log in to wikidata-suggester.wikidata-dev.eqiad.wmflabs with root key, and then figure out why you can’t log into it with user key [16:27:48] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-102, 3Labs-Sprint-103, and 3 others: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1449717 (10yuvipanda) [16:27:53] it’s up to date and puppetized and happy [16:28:14] bah! YuviPanda nevermind it just started working [16:28:30] 6Labs: Remove pdbhandler project - https://phabricator.wikimedia.org/T105704#1449718 (10yuvipanda) 3NEW a:3yuvipanda [16:28:43] andrewbogott: ok! :) [16:28:55] YuviPanda: third reboot was the charm [16:28:58] andrewbogott: if I delete a project, will instances in it be autodeleted? [16:28:59] heh [16:29:29] YuviPanda: no. got to clean up the project first. [16:29:43] andrewbogott: ok, so that's just deleting the instances, right? [16:29:53] or do I need to delete proxies, security groups, etc? [16:30:22] YuviPanda: I /think/ just instances. [16:30:33] Cleaning up proxies wouldn’t hurt. Security groups are definitely nova’s problem. [16:30:40] ok! [16:30:48] YuviPanda: also wait and double-check that instances are really gone before deleting the project, otherwise I can leave zombies [16:30:53] *it [16:31:04] andrewbogott: I was going to delete them from nova commandline [16:31:56] still worth double-checking with ‘nova list’ [16:32:06] and of course delete the /project/ through wikitech or things will leak [16:32:07] yesh [16:32:10] right [16:32:29] andrewbogott: btw, with agent forwarding disabled do we have an alternative for doing instance migrations? [16:32:36] nope [16:32:45] In theory live migration + suspend/resume except... [16:32:57] I think we’ve established that suspend/resume isn’t a good idea with this kernel [16:33:02] should set one up, I guess. id_labstore on the labstore module does the same thing [16:33:15] right, and the new kernel didn't let you put instances on it? [16:33:23] I’d rather fix live migration than hack up a new cold-migration solution [16:33:26] the new kernel... [16:33:38] Oh, actually, that box turned out to have had virtualization disabled in the bios. [16:33:41] So that was that problem. [16:33:42] ah! [16:33:47] so it still has the new kernel? [16:34:01] You could try a kernel upgrade again if you want. I set that box up to test with labnet1002 but now it’s going to be at least 2 weeks until I get that ready :( [16:34:11] heh [16:34:19] No, I reimaged because I needed it to be similar to the other boxes for my tests that I never got to run [16:35:31] righ [16:35:51] I don't think I'll get to it anytime soon however - lots of things to do :( [16:35:59] need to start the mesos / kubernetes / others thing as well [16:36:31] 6Labs, 10wikitech.wikimedia.org, 3Labs-Sprint-105, 5Patch-For-Review: remove nutcracker from wikitech - https://phabricator.wikimedia.org/T102993#1449754 (10Andrew) Wikitech is no longer using nutcracker for mediawiki. Nutcracker is still installed, though. [16:36:48] YuviPanda: I don’t know what either of those things are :) [16:36:59] andrewbogott: ah, the 'experimental' goal :) [16:37:07] But I guess not being able to migrate is pretty serious, if a host breaks we’ll be in trouble. [16:37:13] Maybe I’ll work on that this week. [16:37:20] we also need to work on the actual goal itself - decide what exactly are the 'services' 'Labs' provides [16:37:29] and put checks into checkpoint [16:37:30] err [16:37:30] No key forwarding = 1001 special-purpose keypairs littered everywhere [16:37:32] catchpoint [16:37:48] which is actually not too bad, really... [16:37:56] also what we need to do is make keyholder more useful [16:38:03] maybe… more complexity = more attack surface [16:38:05] although maybe not in this case, since we're root anyway [16:38:20] yeah but with -A game over if attack succeeds ;) [16:38:35] while with a key that's installed only on labvirt, if compromised, only labvirt is compromised [16:38:50] so less places key is accepted less attack surface? [16:39:26] !log pdbhandler deleting instances before deleting project, per https://phabricator.wikimedia.org/T105704 [16:39:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Pdbhandler/SAL, Master [16:39:44] !log pdbhandler deleted project [16:39:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Pdbhandler/SAL, Master [16:46:13] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-102, 3Labs-Sprint-103, and 3 others: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1449805 (10yuvipanda) [16:46:14] 6Labs, 5Patch-For-Review: Disable NFS in puppet3-diffs project - https://phabricator.wikimedia.org/T103760#1449804 (10yuvipanda) 5Open>3Resolved [16:46:24] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-102, 3Labs-Sprint-103, and 3 others: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1360119 (10yuvipanda) [16:46:26] 6Labs, 5Patch-For-Review: Disable NFS for dwl project - https://phabricator.wikimedia.org/T103864#1449812 (10yuvipanda) 5Open>3Resolved [16:46:47] 6Labs: Investigate and disable NFS on mobile project - https://phabricator.wikimedia.org/T103098#1449816 (10yuvipanda) 5Open>3Resolved This has been done. [16:46:48] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-102, 3Labs-Sprint-103, and 3 others: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1360119 (10yuvipanda) [16:47:16] hello [16:47:32] I cannot see my user database on c2 [16:47:56] just went away two days ago [16:48:26] see http://tools.wmflabs.org/superyetkin/istenen_ogeler.php [16:49:22] anyone can look at this please? [16:50:50] YuviPanda: ^ ? [16:51:04] hi Superyetkin [16:51:10] hello panda [16:51:15] nice to hear from you again [16:51:16] Superyetkin: can you file a bug? what's the name of the db? [16:51:34] db name: s51698__yetkin [16:51:56] YuviPanda: a lot of these complaints have come in over the weekend, Jaime I think is aware but unsure [16:52:12] not a lot but a few I should say :) [16:52:34] JohnFLewis: The over-the-weekend issues are all marked as fixed as best I can tell. [16:52:51] Superyetkin: indeed, I can't find it either [16:52:58] lol [16:53:01] not sure what to do - our DBA isn't around [16:53:06] http://tools.wmflabs.org/superyetkin/istenen_ogeler.php [16:53:18] andrewbogott: probably something done that didn't fix the root issue I guess as they were DBA/1002 related [16:53:20] broken for two days... [16:53:46] JohnFLewis: maybe. I think Jaime is rearranging some things but now he’s out sick. [16:53:58] oh [16:54:05] Superyetkin: best bet is to open a phab task so we can get it in front of Jaime when he’s back from clinic [16:54:07] andrewbogott: sick? bah [16:54:26] when will he be back? [16:54:33] in a day, maybe? [16:54:53] Superyetkin: I dunno, when he’s better. But Sean can also look — he’s in Australia so sleeping now. [16:55:12] yes, so do file a bug either way? [16:55:15] any other DBA? [16:56:11] Superyetkin: nope, just those two [16:58:30] 6Labs, 10Tool-Labs: missing database on replica server - https://phabricator.wikimedia.org/T105713#1449858 (10Superyetkin) 3NEW [16:58:43] https://phabricator.wikimedia.org/T105713?workflow=create [16:59:05] we need to add new DBAs :) [17:00:00] also, I have a question for you [17:00:39] are there any recommendations as to which replica servers we should use for our tools? [17:00:54] for performance concerns, I mean [17:01:07] I am using c2. Should I switch to tohers? [17:01:58] you should ideally just use *wiki.labsdb [17:02:01] I do have 12 tools on labs, fed by c2 [17:02:03] and we'll move ethem around as required [17:02:30] move "them" around? [17:02:39] what are they? [17:02:48] I used to use trwiki.labsdb [17:02:58] 6Labs, 10Tool-Labs: missing database on replica server - https://phabricator.wikimedia.org/T105713#1449872 (10Andrew) a:3jcrespo [17:03:04] but after a major outage last month, I was advised to switch to c2 [17:03:28] ok, so if your user database is on c2 [17:03:34] and you want to join things against it [17:03:36] you have to use c2 [17:03:46] unfortunately c2 has had issues in the past week or so... [17:04:18] 6Labs, 7Mobile: Decide what to do with the android-build machine - https://phabricator.wikimedia.org/T104190#1449875 (10yuvipanda) Anyone should be able to add them to the 'mobile' project on wikitech to give them access :) And yes, nginx should be able to serve the files very easily and then we can switch th... [17:05:01] panda: yes, but someone here advised me to switch to c2 from trwiki.labsdb... [17:05:18] anyway, I will be waiting for DBAs to look at this [17:05:32] thanks [17:08:02] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-102, 3Labs-Sprint-103, and 3 others: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1449888 (10yuvipanda) [17:08:04] 6Labs, 5Patch-For-Review: Remove pdbhandler project - https://phabricator.wikimedia.org/T105704#1449886 (10yuvipanda) 5Open>3Resolved Done [17:15:59] 6Labs: Remove pdbhandler project - https://phabricator.wikimedia.org/T105704#1449918 (10Ricordisamoa) [17:19:11] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Wbm1058 was created, changed by Wbm1058 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Wbm1058 edit summary: Created page with "{{Tools Access Request |Justification=I would like to help support xTools (specifically try to get https://tools.wmflabs.org/xtools-articleinfo/index.php?article=Hardware&proj..." [17:20:07] 6Labs, 10wikitech.wikimedia.org, 3Labs-Sprint-105, 5Patch-For-Review: remove nutcracker from wikitech - https://phabricator.wikimedia.org/T102993#1449933 (10Krenair) Is uninstalling it something we want to do? If so, is it part of this task? Will we want to revert this after {T105131} is fixed? [17:21:07] 6Labs, 10wikitech.wikimedia.org, 3Labs-Sprint-105, 5Patch-For-Review: remove nutcracker from wikitech - https://phabricator.wikimedia.org/T102993#1449941 (10Andrew) I'm ambivalent. Wikitech config differing from prod is bad, wikitech using a component that doesn't do anything is bad; either way we lose :) [18:02:13] YuviPanda: Random query for you... [18:02:20] sure! [18:02:44] YuviPanda: Do you know why this isn't working? https://metrics.wmflabs.org/static/public/dash/#projects=wikidatawiki/metrics=RollingSurvivingNewActiveEditor [18:02:54] YuviPanda: I wondered if it just needs kicking or something. [18:03:09] Deskana: ah, nope. the instance itself is up - analytics would know why that specific URL isn't up [18:03:19] YuviPanda: Okie doke. Thanks! [18:04:03] Coren: Are we successfully getting scripted/automatic backups now? [18:04:22] andrewbogott: https://gerrit.wikimedia.org/r/#/c/224064/ is a patch [18:04:38] So that would be ‘not yet’ [18:14:03] 6Labs: Labs team reliability goal for Q1 2015/16 - https://phabricator.wikimedia.org/T105720#1450103 (10yuvipanda) 3NEW [18:16:47] 6Labs: Identify services labs provides - https://phabricator.wikimedia.org/T105721#1450114 (10yuvipanda) 3NEW [18:17:28] 6Labs: Identify services labs provides - https://phabricator.wikimedia.org/T105721#1450114 (10yuvipanda) [18:18:00] 6Labs: Allow NFS to be enabled / disabled granularly for Labs projects - https://phabricator.wikimedia.org/T98571#1450132 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Done now :) [18:18:41] 6Labs: Find a different backup solution for Wikimetrics - https://phabricator.wikimedia.org/T103001#1450136 (10yuvipanda) /data/project is back and IIRC you guys are using it for now? [18:18:57] YuviPanda: so how do I use MWApi._request in your yuvipanda/python-mwapi project? [18:19:13] Negative24: hmm, not sure if you should consider it maintained or not... :) [18:19:34] oh [18:19:58] I tried to find that out yesterday but I didn't get a response from you :) [18:20:27] yeah was travelling sorry [18:20:32] np [18:20:58] 6Labs: Eliminate SPOFs in Labs infrastructure (Tracking) - https://phabricator.wikimedia.org/T105723#1450142 (10yuvipanda) 3NEW [18:23:05] 6Labs: Find replacements for various things that people were using NFS for but should not have been (Tracking) - https://phabricator.wikimedia.org/T104193#1450161 (10yuvipanda) [18:23:07] 6Labs: Labs team reliability goal for Q1 2015/16 - https://phabricator.wikimedia.org/T105720#1450160 (10yuvipanda) [18:23:17] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-102, 3Labs-Sprint-103, and 3 others: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1450163 (10yuvipanda) [18:23:59] 6Labs: Find replacements for various things that people were using NFS for but should not have been (Tracking) - https://phabricator.wikimedia.org/T104193#1409578 (10yuvipanda) [18:28:46] 6Labs, 10Tool-Labs, 7Database, 3Labs-Q4-Sprint-1, and 4 others: Make sure tools-db is replicated somewhere - https://phabricator.wikimedia.org/T88718#1450206 (10yuvipanda) [18:28:47] 6Labs: Eliminate SPOFs in Labs infrastructure (Tracking) - https://phabricator.wikimedia.org/T105723#1450205 (10yuvipanda) [18:28:48] 6Labs, 10Tool-Labs, 3ToolLabs-Goals-Q4: Make sure tools-db is backed up in some form - https://phabricator.wikimedia.org/T88716#1450207 (10yuvipanda) [18:35:26] 6Labs: Eliminate SPOFs in Labs infrastructure (Tracking) - https://phabricator.wikimedia.org/T105723#1450239 (10Dzahn) Does Beta have poolcounter? [18:41:44] 6Labs: Eliminate SPOFs in Labs infrastructure (Tracking) - https://phabricator.wikimedia.org/T105723#1450261 (10yuvipanda) This is about Labs infrastructure - beta does not count. [18:43:57] 6Labs: Eliminate SPOFs in Labs infrastructure (Tracking) - https://phabricator.wikimedia.org/T105723#1450274 (10yuvipanda) [18:54:52] 6Labs: Labs team reliability goal for Q1 2015/16 - https://phabricator.wikimedia.org/T105720#1450307 (10yuvipanda) Not sure about the NFS sharding one - @mark / @coren is that just the tools / others / maps being on different arrays? Or is there more to that? :) [18:59:53] 6Labs: Make labs domainproxies fully redundant - https://phabricator.wikimedia.org/T98556#1450313 (10yuvipanda) [19:03:16] andrewbogott: do you know how wikitech knows which IP to ping for labs proxy? [19:03:48] YuviPanda: I think it’s in a mw config. [19:03:51] want me to look? [19:06:15] andrewbogott: yeah, that'd be great! [19:06:18] I did a grep and couldn't find it [19:06:20] off hand [19:06:21] ok [19:06:52] hm, so, I’ve been maintaining a bot in #wikimedia-e3 but that room says it forwards to wikimedia-growth but I’m unable to /join wikimedia-growth [19:06:58] What does that mean, I wonder? [19:07:10] andrewbogott: that's because wikimedia-growth no longer exists! [19:07:21] that team was absorbed into the mobile team and then the mobile web team and now the readership team [19:07:26] But how can the /channel/ no longer exist? [19:07:36] But anyway I will kill off that bot forever [19:07:39] 12:08 -!- #wikimedia-growth #wikimedia-dev Forwarding to another channel [19:07:43] andrewbogott: it is a redirect [19:07:45] to -dev :) [19:07:49] ah [19:07:53] ok then [19:07:54] yeah second redirect :) [19:07:59] no more e3-logbot [19:09:05] imagines bots getting caught in a circular redirect for a year [19:09:28] poor bot, etc [19:09:50] mutante: re: your comment about poolcounter, the task / goal is about labs infrastructure only and betacluster doesn't count [19:10:08] YuviPanda: // Key/value pairs like array( 'region1' => '10.4.0.11', 'region2' => '10.68.1.35' ) [19:10:09] $wgOpenStackManagerProxyGateways = array(); [19:10:10] beta is not infra? [19:10:15] YuviPanda: is that enough info? [19:10:35] mutante: no, it's not infrastructure maintained by the Wikimedia Labs team - it's maintained by the Release Engineering team and they're responsible for it [19:10:51] mutante: https://wikitech.wikimedia.org/wiki/Labs_labs_labs :) [19:11:10] andrewbogott: yes, thanks :D [19:11:16] I generally mean ‘infra’ to mean “Things that labs uses, not things that use labs” [19:11:18] YuviPanda: ok, it just seems hard to keep up without .. the infrastructure being up [19:11:46] mutante: well that's true for all the things on labs, but I don't think fixing poolcounter on beta is part of it :) [19:11:51] it = quarterly goal [19:12:15] 6Labs: Make labs domainproxies fully redundant - https://phabricator.wikimedia.org/T98556#1450359 (10yuvipanda) $wgOpenStackManagerProxyGateways in mediawiki-config is responsible for figuring out which host to contact. [19:12:20] !log wikimania-support Made Dduvall a project admin [19:12:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikimania-support/SAL, Master [19:12:40] bd808: thanks! [19:13:03] marxarelli: The instance is bd808-vagrant.wikimania-support.eqiad.wmflabs [19:22:34] YuviPanda: yep, i get it if it's about the goals.. stuff is not used by labs [19:22:56] mutante: yup :) and to actually answer your question, no there's no poolcounter in beta [19:23:22] but there should be [19:23:36] and there should be pybal too [19:24:18] yeah [19:24:28] andrewbogott: btw, https://phabricator.wikimedia.org/T92756 too - or we might be in puppetmaster hell again [19:25:04] yeah, good point. [19:26:16] quick question. how much memory can I allocate using jsub -mem in toollabs and how much is recommendable? [19:28:26] I think max is 4G, marmick [19:28:34] as is recommended - as much as you need :) [19:33:02] YuviPanda: thanks. [19:46:17] 6Labs, 7Puppet: puppetmaster::gitsync should update labs/private repository as well - https://phabricator.wikimedia.org/T92756#1450444 (10Andrew) Attached patch can be much prettier if/when the script is rewritten in a proper language. [20:06:39] YuviPanda: fyi, I’m am removing “puppetClass: base” from every instance in ldap. Base is already pulled in via role::labs::instance so it’s redundant, and new instances don’t include it anyway. [20:06:50] yup! [20:10:32] Hi. Is there someone who could please help me with some basic issues with accessing an instance? [20:12:33] 6Labs, 7Mobile: Decide what to do with the android-build machine - https://phabricator.wikimedia.org/T104190#1450521 (10bearND) @yuvipanda I still don't know what to do. Should I figure out how to add the new android-builder instance to tools? Is that by using the link "Create New Tool" from https://tools.wmf... [20:13:46] 6Labs, 7Mobile: Decide what to do with the android-build machine - https://phabricator.wikimedia.org/T104190#1450532 (10yuvipanda) ah, no. tools has nothing to do with labs (see wikitech.wikimedia.org/wiki/Labs_labs_labs). You can use wikitech.wikimedia.org/wiki/Special:NovaProject to add people to the project... [20:15:02] 6Labs, 3Labs-Sprint-106, 7Puppet: puppetmaster::gitsync should update labs/private repository as well - https://phabricator.wikimedia.org/T92756#1450537 (10yuvipanda) [20:18:27] Huji: hi! what is the problem? [20:19:07] YuviPanda: well, for one thing I cannot seem to be able to SSH into the instance I create. And for another, I cannot seem to be able to add rules to a security group that I created, which I want to use with that instnce [20:19:28] I have Amazon EC2 experience, so the fact that I cannot get this to work is quite surprising to msyelf [20:19:34] andrewbogott: ^ [20:19:38] re: security groups [20:19:47] Huji: this is going to sound terrible, but have you tried logging out and back in? [20:19:49] for the security groups [20:19:58] out of what? [20:19:59] SSH? [20:20:03] or the wiki? [20:20:04] wikitech [20:20:10] for the security groups [20:20:27] Huji: didn’t we sort out ssh with you a couple of days ago? Or am I thinking of someone else? [20:20:35] there is a bug when adding new security groups, the work-around was to change the default group [20:20:45] andrewbogott: you helped me setup a project, this is a new issue (related, but new) [20:20:51] ok [20:20:57] mutante: I cannot even edit default group [20:21:07] Huji: what are you trying to do? [20:21:21] then it sounds like you are lacking project adminship [20:21:23] mutante: I take it back, I can. But that's realy awkward to be able to edit new sec groups [20:21:34] mutante: I do have adminship, I think [20:22:00] andrewbogott: create a 'web' sec group, open port 80 to public, port 22 to 10.0.0.0/8 [20:22:15] pretty much following instructions from the Wikitech :) [20:22:44] Were you setting ‘source group’? [20:22:46] YuviPanda: logging out and in didn't help :( [20:23:09] andrewbogott: yes, to the gropu I created, shouldn't I? [20:23:23] no, ‘source group’ is… something else. [20:23:27] ahhh [20:23:34] well it didn't fell like it was something else [20:23:37] that's dumv [20:23:37] Bascically everything in a given source group can talk to everything else in the same group [20:23:44] oh okay [20:24:01] Also note that you can’t add new security groups to existing instances, so you’ll need to recreate your instance if you want to add membership in that group. [20:24:02] so now, how can I 'change' the security group of an instance? rebooting is an option. [20:24:09] That’s a ‘feature’ of nova, I don’t know why. [20:24:18] aaah. that is differnet from EC2 [20:24:34] Yeah, it seems broken to me. Maybe it’s fixed in future versions. [20:24:49] okay [20:24:53] two out of three resolved [20:25:04] the only thing that is left is for me to be able to SSH into my instance [20:25:16] now with amazon, when you create an instance you also create (or reuse) a key pair [20:25:19] but not with nova [20:25:25] hwo can I ssh into my instance then? [20:25:56] you need to register a key with wikitech. Then it’s automatically associated with all instances in projects that you’re a member of. [20:26:10] it will be in LDAP after you add it on wikitech ui [20:26:12] so my existing key that I use for everything is [20:26:15] shoudl work? [20:26:36] https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack [20:26:42] but now I am logged into Bastion using that key, how can I log into my instance from there? [20:26:49] I dont want to put my private key on Bastion [20:26:58] oh, if you can log into bastion then you’re in good shape. [20:27:03] One second, I will find you a doc link. [20:27:14] What OS are you on, locally? [20:27:37] Mac [20:27:45] ok, then this will work: https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_ProxyCommand_ssh_option_.28recommended.29 [20:27:57] Huji: with ProxyCommand we can avoid having to forward the agent or putting any private keys on bastion [20:28:23] mutante: hmm. I'm not sure if I can get that to work without upgrading my SSH on this mac [20:29:31] how old is your mac? [20:31:02] not too old, It is running the latest OS [20:31:04] Huji: it should be in it since like 5.4 [20:31:16] mutante: let me try to see which version I have [20:31:42] I'm pretty sure ProxyCommand has worked on OS X for quite awhile [20:32:20] it's actually the -W part [20:32:28] before you had to use netcat for the proxycommand [20:32:55] http://www.openssh.com/txt/release-5.4 [20:33:06] "Added a 'netcat mode' to ssh(1) .." [20:33:39] YuviPanda: you were correct; it did work! [20:33:48] I'm finally on my instnance's shell [20:34:13] now, is it safe to assume that once I add more people to the project, they can automatically log in using the same method as well? [20:34:26] automtically meaning: no further config needed. [20:35:20] Huji: nice! it should work when they are also project admins [20:35:34] only admins, got it mutante [20:35:40] well, ssh should work for everyone Huji [20:35:54] admins have ability to create new instances and add security groups and stuff [20:36:00] got it yvu [20:36:01] YuviPanda: [20:36:42] andrewbogott: forgive my forgetfulness, but what was your recommendation for setting up MW on the instance? [20:36:55] labs-vagrant is probably best. [20:38:12] setting up MW without labs-vagrant is crazy pants [20:38:53] bd808: not that I'm not already crazy pants, but I'll take your (and Andrew's) word and use vagrat [20:38:56] vagrant* [20:39:29] labs-vagrant is really just reusing the Puppet rules from the mediawiki-vagrant project on a labs instance [20:39:30] bd808: I'll try to get the vagrant package stuff done this wikimania [20:39:33] andrewbogott: so step 4 says "ssh to your instance, and wait " ... wait until when? [20:39:41] YuviPanda: ooh that would be cool [20:39:52] bd808: have you tested it with local installs of the deb? [20:39:56] and step 5 is "Run an initial labs-vagrant provision" but I don't know what a privison is [20:40:05] or how it is run! [20:40:14] bd808: hmm, actually, we can just add the origin repo maybe? [20:40:21] YuviPanda: some, yes. I have an instance I used to build it out [20:40:30] (PS: once you answer these, I'll update the Wikitech so they mean something to non-gurus!) [20:40:44] YuviPanda: I don't think their origin is a real apt repo :/ [20:40:46] Huji: I don’t know what instructions you’re following. But what you should do is “sudo puppet agent -tv” (or wait half an hour for that to happen on its own) [20:40:55] and then ‘labs-vagrant provision' [20:40:59] Just a bag of files in the cloud [20:41:15] Huji: I think that guide presumes that you already know how to use vagrant [20:41:27] andrewbogott: the guide is thinking too highly of me! [20:41:28] https://wikitech.wikimedia.org/wiki/Help:Labs-vagrant [20:41:45] YuviPanda: https://github.com/mitchellh/vagrant-installers/issues/12 [20:42:00] TL;DR "we don't care about deb users too much" [20:43:41] andrewbogott: I ran those commands in order and got a bunch of warnings from the latter (skipping a bunch of things due to unmet dependencies). Fingers crossed. [20:43:57] * Huji wishes there was a LABS 101 somwhere for people who think a puppet is a kind of doll [20:44:15] If the puppet run is throwing warnings then something bad is happening, it should be clean on a new instance [20:44:28] andrewbogott: that's what I thougth too [20:45:45] I'm running 'labs-vagrant provision' again [20:46:13] bd808: vagrant | 1:1.7.3 | trusty-wikimedia | amd64 [20:46:17] also according to bd808 just a second ago, it sounds like this might not work on jessie and you might need a trusty instance [20:47:04] yeah, I'm going to kill this instance and create a trusty instance instead [20:47:47] eh.. how are we ever going to upgrade to jessie [20:49:12] " fpm which eliminates the need to maintain source package. " .. no source? [20:50:30] bd808: done [20:51:24] andrewbogott, Huji: sadly mw-vagrant and labs-vagrant don't work on jessie yet [20:51:52] bd808: no worries, it took me less than a minute to kill this fresh instance and create a 14.04 one [20:52:06] we aren't using jessie on prod app servers and so haven't invested in making the the Puppet code work on both platforms [20:52:33] YuviPanda: {{done}} as in there is an apt package now? [20:52:53] bd808: yes [20:54:05] w00t [20:54:40] bd808: only on trusty tho [20:54:41] andrewbogott: good news, the puppet (am I using the right word) is populating itself! I see an apache default page so far [20:54:42] no jessie [20:54:53] bd808: technically.. shouldnt the newer version first be in labs so that it can be tested before prod uses it [20:55:03] YuviPanda: that's ok. our puppet code is trusty only for now [20:55:10] bd808: cool [20:55:38] mutante: sure, but the last time I asked there was no plan on the books at all to update from 14.04 to jessie for the MW servers [20:55:56] like we don't build hhvm for jessie even [20:57:34] *nod*.. and meanwhile we are converting stuff TO trusty https://phabricator.wikimedia.org/T87036 hrmm [20:59:25] asks on that tracking ticket if we can skip trusty, heh [21:02:31] bd808: re: the ticket, I can leave it up to you wether to close it now or wait for an actual sid backport. [21:03:18] YuviPanda: my test build was using that package, so its probably fine [21:03:48] YuviPanda: https://gerrit.wikimedia.org/r/#/c/193665/13/modules/vagrant/manifests/init.pp,unified [21:04:45] also I'm not pretty sure what I'm working on for one day of the hackathon :) [21:05:06] bd808: nice :) [21:05:34] okay, one last question guys: I enabled the "gadgets" role, but that extension is not enabled on my private wiki [21:05:44]