[00:08:30] Wrong is when the token doesn't match [00:09:00] I've never heard of bad token [00:11:31] i think bad token is if expired [00:25:16] There's no BAD_TOKEN constant [00:25:17] ala [00:25:18] const NEED_TOKEN = 12; [00:25:18] const WRONG_TOKEN = 13; [00:25:49] I'm wondering, I get a lot of "badtoken" sometime ago, where my login failed [00:26:07] Reedy: Maybe deprecated, but was presend in former mediawiki versions? [00:27:02] https://www.mediawiki.org/wiki/API:Errors_and_warnings#Standard_error_messages [00:27:07] badtoken Invalid token (did you remember to urlencode it?) [00:37:04] ah [02:08:30] (03CR) 10Legoktm: [C: 032] Send Community-Wishlist-Survey stuff to #wikimedia-commtech [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/279579 (owner: 10Niharika29) [02:09:06] (03Merged) 10jenkins-bot: Send Community-Wishlist-Survey stuff to #wikimedia-commtech [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/279579 (owner: 10Niharika29) [02:30:40] !log tools.wikibugs Updated channels.yaml to: fcffc4d11cddf0e11b5353bf50a9c36f4d989090 Send Community-Wishlist-Survey stuff to #wikimedia-commtech [02:30:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master [02:59:15] Cyberpower678: Pong. [03:00:36] I forgot why I pinged you. :p [03:03:26] Niharika, oh right. https://en.wikipedia.org/wiki/Wikipedia:Bots/Requests_for_approval/Cyberbot_II_5a [03:03:48] I think since this part of the bot is your code, that you are more qualified to comment. [03:05:09] Cyberpower678: Okay. Reading. [03:17:09] Cyberpower678: Done. [03:17:17] Now to debug the logger... [03:19:22] Niharika, Once I get this pushed through approval, is there anything else that needs to be done to Cyberbot II? [03:20:41] Cyberpower678: Spawning more bots? [03:21:00] I've got two running so far. [03:21:18] Cyberpower678: Do you have any idea why the encoding is messed up for page titles? [03:21:30] One is going across the 5 million articles, and the other across all the articles with dead tags on them [03:22:10] Cyberpower678: https://github.com/Niharika29/Deadlink_logger/blob/master/public_html/api/index.php#L8 This is the bit that should fix the encoding, but turns out its still messed up. [03:22:50] Cyberpower678: What about a few more bots which go through the wiki from different alphabetical points? [03:23:12] They're setup. [03:23:18] But not yet started [03:23:32] I have insufficient ram to do that. [03:24:06] I can't spawn 26 bots on a 4GB node with 16 other bots running on it. [03:24:24] Cyberpower678: How about 4-5 initially? [03:24:36] Well they would loop [03:24:50] Once they complete the run, they start over. [03:25:01] And they would only handle a specific alphabet [03:25:49] I see. So we can't have a single bot look after, say, A, B, C, D, E together. [03:26:53] yuvipanda, are you awake or are you in hibernation mode? :p [03:37:09] I can probably offload someof the bots into the regular queue instead of the Cyberbot one. [03:37:34] I can then probably spawn about 12-16 bots. [03:37:46] Niharika, ^ [03:38:22] Cyberpower678: That sounds good. That'll speed us up a lot. [03:38:48] Cyberpower678: How come yesterday it fixed almost 4x the links it did day before? [03:39:04] Because yesterday I started the second bot. :p [03:39:19] The one that's dedicated to working on only pages with tags on them. [03:39:59] Cyberpower678: Ah. We should definitely spawn a few more bots. [03:56:20] Cyberpower678: Could you name the second bot something different, so we can differentiate the records from those bots? [03:58:48] They're both running under Cyberbot II [03:59:45] Cyberpower678: Ah. So will all other bots then. No way to distinguish between edits made by different bots? Send in different names to the API? [04:02:35] Niharika, they're essentially the same bot. They're running on identical code. [04:02:45] They're just working in different places on Wikipedia. [04:03:17] They're essentially workers of the bot [04:03:30] Cyberpower678: Okay. [04:06:27] Cyberpower678: Are you spawning more bots? [04:07:07] Yes. I have A and B spawned [04:08:02] Cyberpower678: Okay. [04:08:18] Wait, something bugged up. [04:09:21] PHP Notice: Undefined variable: link in /data/project/cyberbot/bots/cyberbot-ii/IABot/Parser/parse.php on line 157 [04:09:33] link is critically important variable. [04:10:27] :( [04:19:09] Niharika, you got a respnse on the BRFA btw. [04:20:06] Cyberpower678: Oh, thanks for letting me know. [04:24:28] Niharika, okay I fixed the bug. It was minor [04:24:40] Cyberpower678: Okay. [04:24:49] Let's try spawning them again. [04:24:56] (Y) [04:32:06] Niharika, I spawned H as well [04:32:23] Cyberpower678: A,B and H? Why H? [04:32:37] A-H [04:32:48] Huh? [04:33:00] I have A-H spawned [04:33:38] Cyberpower678: 8 bots? [04:33:47] 10 actually [04:33:56] the other 2 are still running. [04:34:31] Cyberpower678: One of them is going through the tagged articles and the other one is...? [04:34:47] Going through the 5 million articles [04:35:20] Cyberpower678: The new 8 bots will do the same, right? [04:35:34] For just the first letter of the alphabet./ [04:35:54] So bot A is only handling articles that start with A. [04:36:06] Cyberpower678: But there will be redundancy between that bot and these bots. [04:36:29] Well yes, but how do I effectively program the bot to only pages that start with symbols. [04:37:59] I would remove the master bot if I could get it to do only symbols. [04:37:59] Cyberpower678: Leave those for now? Let these 8 bots complete a run each. Then we can move these 8 bots to work on the rest of the alphabets and when they've completed a run, we can let a bot run through the entire 5 million articles. [04:39:18] Niharika, once the initial run is complete, the DB will speed the runs up greatly. [04:40:41] Cyberpower678: How about if we feed it an array of symbols, and make it run once on articles beginning with each of those symbols? [04:41:25] Instead of looping back. [04:41:37] Same for other bots. Feed them an array of characters. [04:41:44] ? [04:42:19] This is a continuous bot. They start over once they finish. Otherwise linkrot won't continue to be addressed. [04:42:31] And I'm not sure what you mean by array of x [04:42:48] I don't think you can feed APPREFIX and array of characters. [04:43:34] Cyberpower678: Say, we feed bot X {'@', '$', '#'...} and then the bot works on fixing articles for each of these symbols, and then it loops back to the start of the array. [04:43:54] Cyberpower678: Obviously, we will only feed one character to apprefix at a time. [04:45:38] Urgh. [04:45:51] That would make for some painful programming. [05:06:31] PROBLEM - Puppet run on tools-docker-builder-03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [05:06:37] Niharika, I'm heading to bed. [05:06:46] o/ [05:41:33] RECOVERY - Puppet run on tools-docker-builder-03 is OK: OK: Less than 1.00% above the threshold [0.0] [05:58:20] 6Labs, 10Tool-Labs, 13Patch-For-Review, 3Scap3: Setup a proper deployment strategy for Kubernetes - https://phabricator.wikimedia.org/T129311#2154305 (10mmodell) >>! In T129311#2151640, @yuvipanda wrote: > Things #scap3 needs to have to make this possible: > > 1. Ability to setup a deployment server witho... [08:29:02] Hi! How do I change an entry in nanorc in tools.taxonbot [08:37:36] tools-bastion-05 is very slow and stucking [08:37:43] what's going on? [10:00:00] doctaxon, it seems fine for me. [10:03:15] fine again, ya [10:04:10] Lets just hope no-one does anything stupid, like compiling on the bastion :D [13:08:09] 6Labs, 10Analytics-Tech-community-metrics, 3DevRel-March-2016: Port 22 of korma.wmflabs.org not reachable - https://phabricator.wikimedia.org/T131046#2154538 (10Dereckson) [13:15:10] 6Labs, 10Analytics-Tech-community-metrics, 3DevRel-March-2016: External service can't login through SSH to korma.wmflabs.org - https://phabricator.wikimedia.org/T131046#2154542 (10Dereckson) [13:15:54] 6Labs, 10Analytics-Tech-community-metrics, 3DevRel-March-2016: External service can't login through SSH to korma.wmflabs.org - https://phabricator.wikimedia.org/T131046#2154444 (10Nemo_bis) Just in case: docs on how to access wmflabs.org instances are at https://wikitech.wikimedia.org/wiki/Help:Access [13:16:52] 6Labs, 10Analytics-Tech-community-metrics, 3DevRel-March-2016: External service can't login through SSH to korma.wmflabs.org - https://phabricator.wikimedia.org/T131046#2154545 (10Dereckson) The server korma.wmflabs.org listens publicly to 22 and accept incoming SSH connections. Could you try from the remot... [13:50:08] 6Labs: replace labs-ns0/1 with labs-ns2/3 - https://phabricator.wikimedia.org/T131052#2154579 (10Andrew) [14:10:25] wikitech admins: https://wikitech.wikimedia.org/wiki/User:Packersandmovers400 [14:10:28] our lovely spams [14:13:36] At least we don't get very much of it. [14:13:44] On wiki anyway. [14:16:29] 6Labs, 10Analytics-Tech-community-metrics, 3DevRel-March-2016: External service can't login through SSH to korma.wmflabs.org - https://phabricator.wikimedia.org/T131046#2154444 (10Krenair) ```alex@alex-laptop:~$ ssh korma.wmflabs.org Linux novaproxy-01 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u4 (201... [14:19:31] 6Labs, 13Patch-For-Review: replace labs-ns0/1 with labs-ns2/3 - https://phabricator.wikimedia.org/T131052#2154630 (10Andrew) Steps: [] Internal DNS change: https://gerrit.wikimedia.org/r/#/c/279946/ [] Update puppet to use new names (should be no-op w/respect to IP addresses) [] Update registrar with new ns0... [14:19:59] why have I been looged off from tools-bastion-05 this minute? (tools.taxonbot) [14:28:31] doctaxon, you must be having a really bad time with it. [14:31:59] doctaxon: are you running the tool itself on bastion-05? [14:32:31] yes [14:32:50] for testing only [14:33:45] how long was it running for? [14:33:49] but I am writing my scripts in nano, too. It's not good to logoff me from bastion-05 [14:34:28] doctaxon, you could use screen or tmux. [14:34:38] tmux? [14:34:45] there is a limit in limits.conf for runtime on a process due to all of teh complaints of people running aggressive things on bastion [14:34:57] so if it ran for x% of cpu time it would be terminated as an errant process [14:35:06] but that's only a guess without knowing more [14:35:19] but it's running about 15 seconds only [14:35:50] is it possible you have connection issues? I see above you were having sluggishness prior [14:36:13] my connections are good [14:37:03] and my script is not aggressive [14:37:21] I got logged off just now too, and I wasn't doing anything at all, just running bash. [14:37:39] yes, I tested in bash [14:41:47] The bastion just seems to be very unreliable of late. [14:41:51] Hm, I'm not able to recreate atm, but I'll launch of long running procs to see. doctaxon it would be better if you could develop on tools-dev.wmflabs.org and it's more pragmatic to not edit your code as the bot user itself [14:42:06] bastion-05 ist sucking now [14:42:14] stucking now [14:45:16] tools-dev is bastion-02 ? [14:45:42] that's where it points atm [14:46:24] what is better with bastion-02 than with bastion-05 [14:47:15] tools-dev is meant for dev so you don't affect users trying to launch and monitor jobs and so you don't get your process killed when someone is developing on -05 and we have to kill processes to allow users to launch and monitor jobs [14:47:19] it's polite and practical [14:47:22] That's the one where all the CPU intensive stuff that needs to be run interactivity, like compiling is supposed to be run isn't it? [14:48:03] a large part of the reason 05 sucks is that people are using for things we ask them not too [14:48:21] maybe that makes 02 suck then but there is a small portion of users doing heavy things and the rest would be unaffected [14:48:26] and easier to understand and accomodate [14:48:42] The other stuff is supposed to be run on the grid I think. [16:34:25] 6Labs, 10Tool-Labs, 10Continuous-Integration-Infrastructure, 7Blocked-on-RelEng, 13Patch-For-Review: debian-glue tries to fetch obsolete package - https://phabricator.wikimedia.org/T125999#2002458 (10mmodell) Can this be closed per @akosiaris's comment above? [17:07:01] PROBLEM - Puppet run on tools-worker-1010 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:17:00] RECOVERY - Puppet run on tools-worker-1010 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:39] Cyberpower678: There's a weird entry in the deadlink logging table for @earth [17:26:20] kaldari, what's wrong with it? [17:26:46] mainly, it doesn't look like Cyberbot ever edited that article [17:27:47] Note Revision ID 0 [17:27:52] but I guess it processed it maybe [17:27:52] also the revision is 0 [17:27:55] Which indicates a failed edit [17:28:32] ah [17:29:06] Cyberpower678: if it's a failed edit, it would probably be better not to send it to the logging API [17:29:25] Why not? [17:29:49] because, it's claiming that it fixed a link even though it didn't [17:30:22] It does draw attention to the edit that failed for some reason. [17:30:31] actually in this case, it's claiming it fixed 2 links [17:30:53] All failed edits carry revision ID 0. So it could be easily filtered out. [17:31:07] :/ [17:31:21] maybe it should send the entry to the API, but it should list them under "not fixed" instead of "fixed" [17:31:26] kaldari, you know I had this exact same discussion with Niharika :p [17:31:36] Oh, sorry :) [17:31:58] We could mark failed edits as red. [17:32:27] yes, we could create a new status of "fail" [17:32:32] Cyberpower678: Yeah, we talked about it. You could still add a 3-times retry or something. [17:32:33] The API could easily detect a revision ID of 0, and mark the edit failed [17:45:57] kaldari, Niharika: I would say leave them there and alter the query to filter them out. [17:46:02] Or mark them as red. [17:46:16] But it does give us an idea of articles the bot fails on. [17:46:25] yeah, we can do that [17:47:13] Cyberpower678: Why not retry? It was probably a one-off thing, an edit conflict or something. [17:48:23] Niharika, @earth is a repeated failure. [17:48:26] I think [17:49:10] Cyberpower678: Can you make the bot parse it once more so we can verify this? [17:49:32] Already did [17:49:33] Analyzing @earth (32543187)... [17:49:33] Rescued: 2; Tagged dead: 0; Archived: 0; Memory Used: 15.5 MB; Max System Memory Used: 114 MB [17:49:33] EDIT ERROR: Received a bad response from the API. [17:49:34] Response: [17:49:50] It literally got a blank response [17:49:50] y'all should consider switching to the commtech channel :) [17:49:55] 6Labs, 10Horizon, 13Patch-For-Review, 7Tracking: Increase horizon session length - https://phabricator.wikimedia.org/T130621#2155285 (10Andrew) [17:50:09] yuvipanda, sorry. :p [17:50:31] yuvipanda, BTW, I revived the Cyberbot project phab ticket [17:50:42] yuvipanda, is 64GB unreasonable to ask for? [17:51:05] I'm going to wait for andrew.bogott to chime in whenever he has time. I've no idea, and I think he's busy doing a DNS switchover now [17:51:33] yuvipanda, Can projects be upgraded, or would they have to be deleted and recreated? [17:51:33] chasemp would also know whenever he is back (he's the labs manager now) [17:52:04] Say can a project be upgraded from 8GB to 64GB of RAM? [17:52:14] Cyberpower678: instance or project? [17:52:20] project [17:52:31] for projects there are quota that can be increased [17:52:38] I know an instance would have to be recreated [17:52:59] valhallasw`cloud, cool. [17:53:09] what good would it do to increase RAM but not say cpu's along with it since instances sizes are fixed, unless this is a case of RAM exhaustion before disk and cpu [17:53:19] best to put up a task and outline [17:53:30] Then 8GB should be sufficient for the time being [17:54:01] chasemp, we are working on a global bots, and as we start to deploy on other wikis, we will need more RAM and CPUs. [17:54:19] chasemp: https://phabricator.wikimedia.org/T112881is the ticket with a lot of info already [17:57:40] if I'm not mistaken a large instance already has 8GB of RAM [17:57:52] Really? [17:58:02] re: https://phabricator.wikimedia.org/T112881#2153643 [17:58:05] yes I believe [17:58:24] When I requested a small project for xtools, we were given 32GB without question. [17:59:52] there are two level of abstraction here that are being overlayed. Projects have quotas that are deligated in fixed instance sizes. So a 100G of RAM for project does little good natively with a 16G max instance allocation. It's possible to adjust but not a normal request. [18:00:06] If you want a VM w/ 8GB of RAM then a normal project with a large instance should suffice [18:00:24] chasemp, well 8GB, will be enough to do bot work for one wiki., [18:00:32] So the instance size are sufficient. [18:00:43] We just need enough RAM to load all of the bots into. [18:01:00] chasemp, indeed [18:01:13] chasemp, and you can add CPUs and RAM later on? [18:01:24] If/when we need it? [18:01:25] matt_flaschen: Maybe you can take a look at https://gerrit.wikimedia.org/r/#/c/279423/ ? [18:02:58] Cyberpower678: to a project yes but as andrewbogott indicated here https://phabricator.wikimedia.org/T112881#1651309 really large asks like we are biting at the edges at here are very contentious resources wise [18:03:07] if you wanted say 2 projects worth of quota to do something sure [18:03:22] if you want 5+ ok now it's probably a case of doing something not sustainable [18:03:42] the best outline you can provide is welcome [18:04:47] chasemp, A small project will suffice. At least 8GB. [18:05:03] The resources have been greatly improved. [18:06:09] As I've outlined at the end of the thread. [18:06:38] Cyberpower678: does this ticket double as a request for the actual project creation 'cyberbot'? [18:06:53] Yes. [18:07:00] Wait.. what? [18:07:10] It is a request to have the cyberbot project created [18:07:27] Requesting a project size of at least 8GB [18:07:31] ah I thought there was a project creation tag sure [18:09:34] Cyberpower678: cyberbot project created, go ahead and create a lare instance w/ 8GB RAM, drop a note when we can clean up the exec nodes in Tools [18:11:39] So I get 50GB of RAM with 20CPUs? [18:18:18] kaldari, Niharika: Cyberbot project has been created. I have granted you access to it. [18:18:29] thanks [18:18:52] You both have access to the cyberbot.iabot service group. [18:19:03] Cyberpower678: Thanks. [18:19:38] I've never actually setup the internals of a project before, so I will have to play around with it. [18:24:23] Cyberpower678: please don't use service groups on non-tools projects. they'll probably go away at some point in the future. [18:24:38] use sudo rules if you want to grant people access in limited ways. [18:24:44] yuvipanda, what should I use instead? [18:24:55] okay [18:25:06] Now I really have to play around with the system. [18:33:05] Cyberpower678: If you want, I can help you [18:33:16] Luke081515, sure. [18:33:17] I got already some experience with projects [18:33:48] Let's start with getting into the project via the terminal using Putty. :p [18:35:24] 6Labs, 10Labs-Other-Projects, 10The-Wikipedia-Library: Create Cyberbot Project on Labs - https://phabricator.wikimedia.org/T112881#2155515 (10revi) [18:35:25] 6Labs, 7Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#2155514 (10revi) [18:36:21] 6Labs, 7Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#2155519 (10Cyberpower678) [18:36:23] 6Labs, 10Labs-Other-Projects, 10The-Wikipedia-Library: Create Cyberbot Project on Labs - https://phabricator.wikimedia.org/T112881#2155517 (10Cyberpower678) 5Open>3Resolved [18:38:10] yuvipanda, If I can't use service groups, do I SSH into the instances directly? [18:38:43] Cyberpower678: reading https://wikitech.wikimedia.org/wiki/Help:Instances and related pages will probably help [18:38:43] Cyberpower678: At first create an instance [18:38:48] How can I restrict access to certain instances from members of a project. [18:38:56] then ssh to bastion.wmflabs.org, and then ssh to them [18:41:24] Cyberpower678: if you don't want users to log into instances in your project you should probably not add them to the project. You can restrict their sudo access, though, but writing sudo policies that enumerate specific users. [18:41:32] yuvipanda: I think the incinga part of this page is really outdated? [18:43:06] cyberpower678@bastion-01:~$ ssh cyberbot-exec-01 [18:43:06] Permission denied (publickey). [18:43:16] Hey folks. I want to create a database that is readable by anyone. Do I just put a "_p" at the end of the name? [18:43:25] What went wrong. I have forwarding enabled. I think. [18:43:31] apparently not. [18:43:41] try ssh -v -v cyberbot-exec-01 to see what ssh is doing [18:43:44] halfak: yes [18:43:46] Luke081515, thanks for testing. If you check as root, does the globalblocking database exist? [18:44:08] Luke081515, did you test locally with VirtualBox, or on Labs? [18:44:11] Cyberpower678: are you on a windows box or mac, linux? [18:44:14] halfak: yup [18:44:22] matt_flaschen: I used an instance at labs [18:44:27] Luke081515: just deleted it [18:44:31] wait a moment, I will check the DB [18:44:35] yuvipanda: Ok, thanks [18:44:37] andrewbogott, windows using putty [18:44:57] Cyberpower678: ah, ok… I can't help much with access then, other than "I'm told you need to use pageant" [18:44:59] valhallasw`cloud, too much incomprehensible stuff. [18:45:03] Luke081515, I've never tested any of my MediaWiki-Vagrant stuff on Labs, so I don't know what if anything might need to be done differently. E.g. DB creation. [18:45:27] Luke081515, you should mention in code review/bugs/etc. if you're testing on Labs. [18:45:44] ok [18:45:59] andrewbogott, I have two instances. One should only be accessible by me and the other by the entire project members. [18:46:29] Cyberpower678: Protect your secrets with chmod [18:46:30] Cyberpower678: debug1: Next authentication method: publickey ... Trying private key: ... [18:46:47] that should have an entry referring to the key in your agent [18:48:13] matt_flaschen: I get: root@localhost:[globalblocking]> SHOW TABLES; -> Empty set (0.00 sec) [18:48:53] Interesting. So the DB exists, but not the table. [18:49:05] Luke081515, were there any provisioning errors? [18:49:36] Cyberpower678: specifically, an entry like 'debug1: Offering RSA public key: imported-openssh-key' where imported-openssh-key is the name pageant uses to refer to the key [18:49:43] Cyberpower678: I'm afraid I have to refer back to my earlier 'if you don't want users to log into instances in your project you should probably not add them to the project' [18:50:01] debug1: Next authentication method: publickey [18:50:02] debug1: Trying private key: /home/cyberpower678/.ssh/id_rsa [18:50:02] debug1: Trying private key: /home/cyberpower678/.ssh/id_dsa [18:50:02] debug1: Trying private key: /home/cyberpower678/.ssh/id_ecdsa [18:50:02] debug1: Trying private key: /home/cyberpower678/.ssh/id_ed25519 [18:50:22] right, so your agent forwarding is not working [18:50:34] It's enabled, in Putty [18:50:41] are you running an agent? [18:50:42] matt_flaschen: I get ==> default: Warning: Scope(Class[Role::Globalblocking]): Could not look up qualified variable '::mediawiki::dir'; class ::mediawiki has not been evaluated [18:51:03] valhallasw`cloud, not sure [18:51:26] I checked allow agent forwarding, inputed my private key [18:51:33] Luke081515, oh, okay. That is probably an easy fix. [18:51:37] Thanks [18:51:41] Cyberpower678: so you're not using an agent. [18:51:44] Can linux not handle encrypted putty private keys [18:51:57] I start the terminal connection [18:51:58] https://wikitech.wikimedia.org/wiki/Help:Putty#Connecting_to_your_instance [18:52:06] matt_flaschen: Ok, can you ping me, if you uploaded the next patch? Then I can try it again [18:52:09] Is horizon web proxy configuration up already? [18:52:11] no, juts because you have it selected in putty doesn't mean you're running /an agent/ [18:52:37] if it's in putty it will be used to set up /that/ connection, but it won't be available for any other connections [18:52:43] if you set up pageant, it will be [18:53:12] Luke081515, sure. [18:53:32] bleh, instead of wasting time on configuring an agent, installing msysgit and using standard ssh from its git-bash is so much easier [18:53:39] There it goes [18:53:49] valhallasw`cloud, thanks [18:54:44] valhallasw`cloud, how do I access my file directory in my project? [18:54:56] what file directory? [18:55:13] Doesn't each project have some sort of file system. [18:55:38] nvw [18:55:40] nvm [18:56:24] Cyberpower678: your instance has a filesystem. There is no shared filesystem between instances in the same project by default, and there is no shared filesystem between projects at all. [18:57:12] So the filesystem in instance cyberbot-exec-01 is different from cyberbot-exec-iabot-enwiki? [18:57:51] yes [18:57:55] yes entirely distinct, like prod you'll want some kind of deployment scenario [18:57:57] even if just scp [18:58:16] Is there a way to directly access the instance without sshing in from bastion? [18:59:06] no, but also, what do you have in mind? I can't think of anything else you would want [18:59:33] you can try to get the putty equivalent of proxyhost set up, but it's nontrivial to set up and debug: http://www.lshtm.ac.uk/its/staffservices/sshgateway/config_gateway_connection_windows_putty.html [18:59:44] I use SmartFTP as my graphical interface for transferring files around, as it easier for me to do that. [19:00:10] But I don't know how to route the SFTP connection into the instance if I can only access bastion at the surface. [19:01:16] you'll have to check the smartftp docs to see if they support a ProxyHost setup [19:01:31] Maybe I can open a tunnel to bastion so I can ssh directly into my instance with the application [19:01:55] if not, you can use port forwarding over ssh (localhost:2222 -> cyberbot-exec-...:22, then connect to localhost:2222) [19:02:10] Interesting...poking around horizon shows a few old VMs I thought I'd destroyed and had disappeared from wikitech already. [19:02:40] there is also https://wikitech.wikimedia.org/wiki/Help:Access_to_instances_with_PuTTY_and_WinSCP [19:03:06] Cyberpower678: https://wikitech.wikimedia.org/wiki/Help:Access_to_ToolLabs_instances_with_PuTTY_and_WinSCP#How_to_set_up_WinSCP_for_direct_access_to_your_Tool_Labs_account [19:03:08] I used to use winscp many years ago now and it's pretty solid [19:03:31] mutante: that's for toollabs, not for other projects [19:04:41] oh, ok, well i meant to say what everybody else said "WinSCP and proxy settings" [19:06:46] toollabs tool users can skip on the proxying, luckily :-) [19:08:00] 6Labs, 10Analytics-Tech-community-metrics, 3DevRel-March-2016: External service can't login through SSH to korma.wmflabs.org - https://phabricator.wikimedia.org/T131046#2155632 (10Lcanasdiaz) >>! In T131046#2154530, @Dereckson wrote: > @Lcanasdiaz As Korma is still operational in read only mode, Unbreak now!... [19:21:12] 6Labs, 10Analytics-Tech-community-metrics, 3DevRel-March-2016: External service can't login through SSH to korma.wmflabs.org - https://phabricator.wikimedia.org/T131046#2155663 (10Lcanasdiaz) 5Open>3Resolved a:3Lcanasdiaz It is working using the IP 208.80.155.168. Thank you guys! :) [19:37:45] 6Labs, 13Patch-For-Review, 5WMF-deploy-2016-03-22_(1.27.0-wmf.18), 5WMF-deploy-2016-03-29_(1.27.0-wmf.19): Switch to using Horizon/Designate for labs public dns - https://phabricator.wikimedia.org/T124184#2155676 (10Andrew) [19:37:47] 6Labs: Enforce true multi-tenancy for labs public DNS - https://phabricator.wikimedia.org/T130032#2155674 (10Andrew) 5Open>3Resolved This is as done as it's going to be. I've emailed the remaining holdouts to tell them that their domains are now owned by the 'wmflabsdotorg' project and they'll need to open... [19:39:26] 6Labs, 13Patch-For-Review, 5WMF-deploy-2016-03-22_(1.27.0-wmf.18), 5WMF-deploy-2016-03-29_(1.27.0-wmf.19): Switch to using Horizon/Designate for labs public dns - https://phabricator.wikimedia.org/T124184#1948403 (10Andrew) [19:39:28] 6Labs, 7Tracking: Make OpenStack Horizon useful for production labs - https://phabricator.wikimedia.org/T87279#2155680 (10Andrew) [19:39:30] 6Labs, 10Horizon, 13Patch-For-Review: Horizon dashboard for managing http proxies for labs instances - https://phabricator.wikimedia.org/T129245#2155677 (10Andrew) 5Open>3Resolved The dashboard works and is installed on production (although temporarily disabled during a dns migration). Alex, you're welc... [19:39:54] 6Labs: [Tracking] Create labtest cluster - https://phabricator.wikimedia.org/T120293#2155682 (10Andrew) [19:39:56] 6Labs, 7Tracking: Support instance manipulation, proxies, dns with Horizon (Quarterly goal tracking bug) - https://phabricator.wikimedia.org/T124181#2155681 (10Andrew) [19:40:22] 6Labs, 13Patch-For-Review, 5WMF-deploy-2016-03-22_(1.27.0-wmf.18), 5WMF-deploy-2016-03-29_(1.27.0-wmf.19): Switch to using Horizon/Designate for labs public dns - https://phabricator.wikimedia.org/T124184#2155685 (10Andrew) [19:40:24] 6Labs: Migrate DNS entries from ldap to designate - https://phabricator.wikimedia.org/T129272#2155683 (10Andrew) 5Open>3Resolved a:3Andrew [19:41:39] 6Labs, 13Patch-For-Review: replace labs-ns0/1 with labs-ns2/3 - https://phabricator.wikimedia.org/T131052#2155700 (10Andrew) [19:41:41] 6Labs, 13Patch-For-Review, 5WMF-deploy-2016-03-22_(1.27.0-wmf.18), 5WMF-deploy-2016-03-29_(1.27.0-wmf.19): Switch to using Horizon/Designate for labs public dns - https://phabricator.wikimedia.org/T124184#1948403 (10Andrew) [19:54:50] Some general question: Is there a limit of jobs I can submit in seconds? [19:54:56] (to the grid) [19:55:21] Luke081515: i think so [19:55:36] yuvipanda: Is 2 jobs in 2 seconds too much? [19:55:45] nope. [19:55:50] like forever or sort term? [19:56:01] because a job a second over a long haul is a bad idea I imagine [19:56:01] I'm using a little shell script to deploy my changes, but it starts only one job [19:57:06] chasemp: My script is doing the folowing: 1) git pull 2) delete my two running jobs 3) wait 10 seconds 4) start the first job 5) wait 1 second 6) start the second job [19:57:09] is that too much? [19:57:40] nah, assuming this is on some kind of user initiation for code deploy makes sense [19:58:36] every time I'm running this, and I got 2 running jobs, only the first is running after the script finishes [19:58:49] when I#m running it the second time, both jobs are submitted [20:07:03] PROBLEM - Host tools-worker-1011 is DOWN: PING CRITICAL - Packet loss = 100% [20:10:27] chasemp: Do you know, why only the first job is submited in this case? [20:10:48] I have no idea, I'm guessing an error is returned? [20:10:54] andrewbogott: How can I create a new proxy now, or when? [20:10:57] it's possibly a scheduling thing [20:11:00] you can look at job status [20:11:29] Luke081515: "I predict that the upstream changes will be fully settled down in around 48 hours, at which point I'll re-enable the proxy and DNS interfaces" [20:11:33] chasemp: No, there is just: (the first job is submited) -> (end of script) [20:11:41] ok [20:12:10] Luke081515: I couldn't really tell you...it seems like you aren't submitting it though you think you are in that case [20:12:40] that's strange, because if I'm running that script again, it works [20:14:00] chasemp: I got it. Seems like he needs more time to delete the first job, then the second [20:14:28] Luke081515: ah, neat plan btw makes sense [20:21:55] seems like tools bastion 05 is a bit instable? [20:22:08] WinSCP needs to reconnect 5 times to upload a change [20:22:20] normaly WinSCP does not need to reconnect [20:22:33] chasemp: ^ do you think this could be related to the cgroup stuff? [20:23:31] I can't of how, it's more likely related to ppl still abusing -05 but...try to -02 and see how that goes for fun [20:23:39] I would be curious if it worked fine [20:23:49] can't think of how even I mean [20:24:32] could be killing the process even if it isn't long running [20:25:11] there is a thing looking for that now, it's not cgroups it's limits.conf [20:25:23] and it only affects procs that have taken up equiv of 100% cpu for 360 minutes [20:25:44] all cgroups are doing atm is auditing stuff and making cpu sharing equal among users and capping mem usage by process [20:25:47] for users [20:25:55] none of which I can think of would do that [20:26:29] the thing not handled well atm at all is making sure one person doesn't eat all NFS BW there [20:26:41] and I'm not entirely sure how to do that for now [20:26:48] so it's an issue [20:26:58]