[00:03:18] Coren, ^ [00:03:59] carl-cbm, https://bugzilla.wikimedia.org/show_bug.cgi?id=47538#c4 [00:05:25] (I have no idea what you're even discussing, but I remember seeing that comment and it looks relevant) [00:23:47] Merlissimo: Are you sure you're not just not /seeing/ the replication because you're in a transaction? [00:24:17] carl-cbm: The exec nodes aren't submit hosts; they couldn't use qstat even if it were installed. [00:24:50] Coren: he's referring to http://ganglia.wmflabs.org/latest/graph_all_periods.php?c=tools&h=tools-login&v=3&m=s5&r=week&z=small&jr=&js=&st=1369957795&z=large [00:25:28] so it's not what *he* sees [00:25:29] Coren: i monitored replag from "outsite". you can also see my runs at 23:20 and 23:55 on http://ganglia.wmflabs.org/latest/graph_all_periods.php?c=tools&h=tools-login&v=1&m=s5&r=week&z=small&jr=&js=&st=1369959833&z=large [00:26:34] at 23:20 run there was a max replag of 219seconds [00:26:55] Hm. It's entirely possible that there is interaction between the way binlogs work and implicit table locks for consistency during a query. Given how quickly the replag goes back down, it's entirely possible the logs get only replayed afterwards. You'd have to ask someone with better understanding of mysql replication (aka Asher) for a definitive answer. [00:27:37] Coren: is there a query-killer running on the labs db? [00:27:49] it just a notice to keep an eye on this possible problem when load rises [00:27:56] * Coren nods. [00:28:03] my test query for importing enwiki which i will need for importing stuff to catgraph just got killed. [00:28:04] I'll certainly raise the issue with Asher. [00:28:12] JohannesK_WMDE: No, there isn't. [00:28:38] JohannesK_WMDE: Do you mean the local DB? [00:29:17] Coren: i mean enwiki.labsdb. [00:29:42] * Merlissimo needs to execute a lot of long running queries [00:30:26] * JohannesK_WMDE too :D [00:30:29] lol [00:34:35] I have two queries running since May 22nd on enwiki_p, and it hasn't caused any rise in replag. Merlissimo, could you post the query? [00:36:28] Hm. According to what doc I can find, this shouldn't be an issue with the current storage engine; there is no table locking by default. [00:38:23] $ echo 'SELECT /* SLOW_OK */ B.page_id, cl_from FROM categorylinks JOIN page AS B ON B.page_title = cl_to AND B.page_namespace = 14 AND B.page_id!=0 AND cl_from!=0;' | mysql --defaults-file=~/replica.my.cnf -h enwiki.labsdb enwiki_p > /tmp/enwiki-dump.dat [00:38:26] [...] [00:38:31] Killed [00:39:41] happened twice. got killed after 2 minutes or so. worked for dewiki though... [00:41:38] JohannesK_WMDE: ... I have no idea why that'd happen; there is no query killer running. [00:41:58] JohannesK_WMDE: But -login hasn't got a lot of memory. [00:42:24] Coren: i'm on -dev [00:42:48] and it should just pipe everything to /tmp. or is that on a ramdisk? [00:42:49] Does mysql format the output, when the query is fed on STDIN? [00:43:09] [876664.107588] Out of memory: Kill process 11649 (mysql) score 717 or sacrifice child [00:43:09] [876664.108112] Killed process 11649 (mysql) total-vm:3795148kB, anon-rss:1901880kB, file-rss:824kB [00:43:09] [876664.148043] mysql: page allocation failure: order:0, mode:0x201da [00:43:16] If it is formatted, it will load the whole result in memory to calculate the column widths. [00:43:21] ah, OOM [00:43:28] * Coren nodsnods. [00:43:41] You'd be better off submitting this to the grid. [00:43:57] i'd be better off not formatting it in any case? [00:44:12] JohannesK_WMDE: That's what I would try. [00:44:13] JohannesK_WMDE: That'd help as well, I'm sure. [00:45:01] if i turn on batch mode, will formatting be disabled? [00:45:53] --batch or --raw will disable formatting [00:46:48] k [00:46:52] Depends on whether you need escaping or not. [00:47:06] scfc_de: http://kpaste.net/31c3c25f0d [00:47:42] Merlissimo: Wow :-). [00:47:55] Holey sheets! [00:48:10] That's not a query, it's the Mother Query! [00:48:24] the query queen [00:48:52] do you want to see my long query codes, too? [00:49:22] JohannesK_WMDE: Just did some testing, and --batch doesn't seem to reduce max RSS when the query is fed on STDIN. [00:49:50] scfc_de: i tried --raw and got oom-killed again [00:51:35] JohannesK_WMDE: Could you try it with a short Perl/PHP/Python script? [00:54:57] JohannesK_WMDE: you should always add -wcAB to mysql arguments if you pipe that in [00:57:00] -w wating if connection limit is reached, -c add commons like your /* sl_ok */ -A speed up database access because of not hashing [01:02:32] and -B for batch. and --raw. still got oom-killed. will try a little python script tomorrow, after some sleep [01:10:49] night all! [01:15:13] Gute Nacht. [01:58:17] Merlissimo: where did that script come from [02:26:54] coren: where is the page listing all tool in tool labs ? [02:27:17] Oren_Bochman: https://tools.wmflabs.org/ [02:27:30] 10x [02:28:18] hmm orwell has a ui I guess it needs some code now :-) [02:28:26] JohannesK_WMDE: Need moar memories! :-) Try the grid, srsly. :-) [02:37:59] Coren: I changed my index.php but the changes are not appearing [02:38:40] Oren_Bochman: Err... there's no caching of any kind. You sure you're hitting the right tool? I see two orwells. :-) [02:38:52] yes [02:38:55] https://tools.wmflabs.org/orwell01/ [02:40:22] and the url is not reponding to ctrl+F5 or ctrl+r [02:43:44] hmm my bad [03:05:56] on typing : become ocounter [03:05:58] I get: sudo: sorry, a password is required to run sudo [03:07:00] what is that about [03:15:16] Oren_Bochman: why are you trying to use sudo? [03:15:34] I'm not I used become [03:15:53] the issue reolved when I logged out and logged into the tool labs [03:16:02] Oren_Bochman: are you part of the ocounter group? [03:16:08] yes [03:16:13] I just created it [03:16:24] but now become works [03:16:45] Oren_Bochman: if you create a project you need to log out/log in for your session to update your permissions [03:16:53] ok [03:17:07] took me a while to figure it out [03:17:17] its just a quirk with Unix file systems [03:18:10] permissions/user groups dont get updated immediately, just during login [03:20:40] thank Betacommand [03:20:52] btw were you in the AMS Hackathon ? [03:21:43] No [03:24:41] still you rock1 [03:45:39] Does anybody have an idea when I can expect Coren to be available again? [03:46:39] Cyberpower678: next week [03:47:00] Wait what? [03:47:05] He was on today? [03:47:11] yeah [03:47:28] So why next week? Is this a joke? [03:48:06] Cyberpower678: no [03:48:14] he is on vacation [03:48:30] Oh. [03:48:46] Could he at least quickly grant me access to archive? [03:49:03] that is Cyberbot, Peachy, and xtools [03:49:10] projects [04:32:47] !toolsdoc [04:32:55] wm-bot: ? [04:32:55] Hi legoktm, there is some error, I am a stupid bot and I am not intelligent enough to hold a conversation with you :-) [04:32:59] !tooldoc [04:32:59] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help [04:33:05] Warning: There is 1 user waiting for access to tools project: Heb (waiting 0 minutes) [04:46:34] Warning: There is 1 user waiting for access to tools project: Heb (waiting 13 minutes) [05:00:03] Warning: There is 1 user waiting for access to tools project: Heb (waiting 27 minutes) [05:13:24] Warning: There is 1 user waiting for access to tools project: Heb (waiting 40 minutes) [05:26:49] Warning: There is 1 user waiting for access to tools project: Heb (waiting 53 minutes) [05:40:19] Warning: There is 1 user waiting for access to tools project: Heb (waiting 67 minutes) [05:51:03] * Hazard-SJ says hello [05:52:12] Coren: Around, by chance? [05:53:18] he's on vacation, according to Betacommand [05:53:48] Warning: There is 1 user waiting for access to tools project: Heb (waiting 80 minutes) [05:54:10] hey [05:54:13] Thanks [05:54:16] in fact he is back [05:54:25] petan: You're actually around :O [05:54:46] !tr Heb [05:54:46] request page: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Heb?action=edit talk page: https://wikitech.wikimedia.org/wiki/User_talk:Heb?action=edit§ion=new&preload=Template:ToolsGranted link: https://wikitech.wikimedia.org/w/index.php?title=Special:NovaProject&action=addmember&projectname=tools [05:54:58] * Hazard-SJ never sees petan around nowadays :/ [05:55:15] I am around like every day :O [05:55:18] check logs [05:55:25] !logs [05:55:25] http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs/ [05:55:35] You're idle in here, I know [05:55:46] no I am talking in here [05:55:57] But anytime I pop in you're away ... well, usually :P [05:56:02] according to statistics I am most talkative at all :P [05:56:08] because I live in EU [05:56:25] lol [05:56:30] I just woke [05:56:47] And I should be getting ready for bed :P [05:57:06] But I'm somewhat annoyed :( [05:58:00] petan: Is it possible at all for me to use chown? [05:58:10] where [05:58:26] In local-hazard-bot [05:58:29] yes [05:58:32] there is take [05:58:53] take [05:58:59] :/ [05:59:02] * Hazard-SJ tries [05:59:05] that overtake it [05:59:29] petan: any recursive option? [05:59:46] I think it is recursive by default or not? [05:59:51] maybe try -r if not [06:01:37] It seems to be :) [06:12:37] brb 1h [06:12:45] * Hazard-SJ nods [06:19:28] zz_YuviPanda: ping [06:22:59] * Hazard-SJ gives up on this implementation [06:23:40] * Hazard-SJ says bye [07:00:34] here [08:11:56] Damianz I think you should tarball all the sql backups in your script and zip them [08:12:06] currently it produces huge data [08:12:14] that isn't effective :/ [10:54:57] !log tools petrb: killing process 3060 on -login (mahdiz 3060 1964 88 May30 ? 21:32:51 /bin/nano /tmp/crontab.Ht3bSO/crontab) it takes max cpu and doesn't seem to be attached [10:54:59] Logged the message, Master [12:03:39] Warning: There is 1 user waiting for access to tools project: Jimmyxu (waiting 0 minutes) [12:06:36] !tr Jimmyxu [12:06:36] request page: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Jimmyxu?action=edit talk page: https://wikitech.wikimedia.org/wiki/User_talk:Jimmyxu?action=edit§ion=new&preload=Template:ToolsGranted link: https://wikitech.wikimedia.org/w/index.php?title=Special:NovaProject&action=addmember&projectname=tools [12:07:29] petan: hey is there a page explaining the difference between 'bots' and 'tools'? [12:07:47] jimmyxu: there will be, there is no difference whatsoever [12:07:52] bots is a staging area of tools [12:08:04] that mean it's experimental version of tools project [12:08:09] well, it will be [12:08:15] at least, that is a plan [12:08:20] :P [12:08:28] petan: so technically I should work with my stuffs on bots and then move them to tools? [12:08:49] petan: or should I just stick to tools for now [12:08:54] you can actually work on tools right away. It's a testing environment for changes done on environment [12:09:05] stick to tools [12:09:08] okay [12:11:40] petan: we should really communicate 'use tools' to people who ask about bots vs tools [12:11:56] that is what I do? [12:12:42] right now there are so complicated systems running on "bots" project that can be hardly moved to tools project [12:12:43] petan: the way you described it is somewhat confusing: 'bots is a staging area of tools' etc [12:13:03] well I can't rename the project... [12:13:12] maybe in future once that is implemented to console [12:13:19] sure, but calling it a 'staging area of tools' does not help to reduce the confusion :-) [12:13:24] just stating 'ignore bots, use tools' does [12:13:37] ok but some people are running bots on "bots" [12:13:49] I can't just tell them "ignore bots" [12:13:56] there needs to be explanation why they should do that [12:14:18] except that those people are already in the labs world [12:15:23] basically: if you don't already know what bots and tools are for, you probably don't need to know anything other than 'use tools' [12:15:46] it's like less and more, or pip and easy_install :-) [12:27:15] !log tools petrb: removing project local-jimmyxu - per request on irc [12:27:16] Logged the message, Master [13:02:32] Warning: There is 1 user waiting for shell: Inkbug (waiting 0 minutes) [13:04:05] !log tools petrb: installing bashdb on tools and -dev [13:04:07] Logged the message, Master [13:09:46] The 'Rules' link in tools-login motd seems to be dead... [13:15:13] jimmyxu: the rules don't exist yet :-) [13:15:30] okay then... [13:16:06] Warning: There is 1 user waiting for shell: Inkbug (waiting 13 minutes) [13:29:40] Warning: There is 1 user waiting for shell: Inkbug (waiting 27 minutes) [13:39:07] legoktm: pong? [13:43:05] Warning: There is 1 user waiting for shell: Inkbug (waiting 40 minutes) [13:45:33] !rq Inkbug [13:45:33] https://wikitech.wikimedia.org/wiki/Shell_Request/Inkbug?action=edit https://wikitech.wikimedia.org/wiki/User_talk:Inkbug?action=edit§ion=new&preload=Template:ShellGranted https://wikitech.wikimedia.org/wiki/Special:UserRights/Inkbug [14:21:28] Hello, Labs! [14:25:28] hi [14:25:34] I have several things to you [14:25:49] hello Coren [14:25:51] Coren: did you read my email I sent you regarding +2 to labs/tools or whatever it's called? [14:26:02] also there is a bunch of stuff that should be merged in puppet [14:26:02] for some reason that sounded like "Hello Newman" in my head [14:26:16] Coren: there are also these 2 exec nodes... [14:26:20] we didn't finish since hackaton [14:26:38] petan: I have. It would make sense. I'll eventually figure out how. [14:26:48] petan: The two new nodes are on my plate for today. [14:26:52] ok [14:27:01] petan: Most of the stuff to merge in puppet I did yesterday. [14:27:06] ok [14:38:03] your branch is behin .... by 60 commits and can be fast... [14:38:08] how do I fix it [14:38:15] !gitexpert [14:38:17] ^^^ [14:38:58] git pull -ff [14:39:31] I tried git pull origin [14:40:06] hmm [14:40:12] git is slow :/ [14:40:20] neither works :( [14:47:15] ok I will do my favorite rm -rf which is the only command that actually works... [14:50:31] I am waiting for a day when git finally will become better than svn [14:50:51] * ConfusedPanda sends petan back to the day git was first released [14:51:00] it still suck [14:51:04] maybe it was better back then? [14:51:07] petan: are you on a branch off your own, or are you working on an existing branch? [14:51:17] I am on production branch [14:51:29] martijnHH btw just for smart guys like you I tarballed the fucked repo [14:51:31] before I deleted it [14:51:51] so if you want to spend your weekend fixing it and then convincing me it's actually easy to fix it, you can! [14:54:40] oh, cool! I want to get some more gitting experience. I'll trade you, I'll spend my weekend trying to do that and tell you what you could have done, you take care off bug 48859 for me ;) [14:55:04] !b 48859 [14:55:04] https://bugzilla.wikimedia.org/48859 [14:56:07] martijnHH I don't understand what you want in that bug [14:56:15] system wide software request? [14:56:32] if you mean to install anything to default labs image that is likely not going to happen [14:56:45] it is supposed to be as small as possible, it contains only software necessary to boot [14:57:47] ugh, I don't know what I'm talking about [14:58:26] I would like it available when I log in to shell. I don't know how to formulate that [14:58:32] to shell where? [14:58:44] labs contains many project with many virtual servers, each for different purpose [14:58:54] tools-login / tools-dev [14:58:59] so tools project! :D [14:59:01] here we go [14:59:03] changing the bug... [14:59:28] the problem to being new to something is that you don't know enough to properly formulate your request [14:59:36] at least, that goes for me [14:59:54] ok this can happen [15:00:05] you want that to live only on -dev or even on -login or somewhere else? [15:01:38] well, it's mainly a build tool, but it would be really nice if I could also use its capabilities in jobs [15:01:54] I'm don't really know what that would entail [15:03:35] !screenfix [15:03:35] script /dev/null [15:07:47] being able to DESCRIBE queries on labsdb would be really useful [15:09:24] martijnHH in that case you want that cluster-wide (on execution nodes as well) [15:09:40] that isn't that simple as long as it's not in a repository :/ [15:10:25] I can install this to -dev right now... [15:13:00] petan: AFAICT, -05 and -06 are now properly on queue. [15:13:07] !log tools petrb: installing /data/project/.system/deb/all/sbt.deb to -dev in order to test it [15:13:09] Logged the message, Master [15:13:16] JohannesK_WMDE: Need moar memories! :-) Try the grid, srsly. :-) [15:13:22] (in re yesterday) [15:14:06] JohannesK_WMDE: Known limit is mysql >=5.0; can't explain without having SELECT right on the underlying table, which we can't do. I'd be glad to run an explain for you though. [15:14:24] !log tools petrb: installing default-jre in order to satisfy its dependencies [15:14:25] Logged the message, Master [15:14:39] Coren: the import query works now that i used the normal _mysql bindings directly, without the added python bloat. but yes, i will need a vm with MOAR RAM for everything. but that was clear from the start [15:15:28] JohannesK_WMDE: Even if I give you a new instance, it'll still be gridengined for reliability. Only I'll make a special queue for you. :-) [15:15:50] JohannesK_WMDE: So that they can live together, and have lots of little baby processes. :-) [15:16:02] yay! :p [15:16:04] martijnHH it's on -dev [15:16:15] man sbt [15:16:17] meh [15:16:19] nvm [15:16:58] what does that actually mean Coren? "will still be gridengined" -- does that mean i will have gridengine available? yes, i would want that :) [15:17:01] Coren howcome we have no JRE anywhere on tools :P [15:17:45] Coren, hi. [15:17:59] petan: Watcha mean? java version "1.7.0_21" [15:17:59] OpenJDK Runtime Environment (IcedTea 2.3.9) (7u21-2.3.9-0ubuntu0.12.04.1) [15:18:12] mhm... there was none on -dev [15:18:13] Cyberpower678: Hoy hoy. [15:18:29] Do you think you could do something for me? :) [15:18:38] petan: Hm. Might be an oversight. [15:18:59] Coren: what was the reason we couldn't use these 2 exec nodes? [15:19:47] petan: I'm not entirely certain. I think I was just too distracted to turn them on right while at the Hackaton. I just added them normally to the queues and fired 'em up and it worked. :-) [15:19:59] Coren, Do you think you could do something for me? :) [15:20:01] ok :) [15:20:30] Cyberpower678: That what they pay me the big^H^H^Hmediu^Wreasonably-sized bucks for. [15:20:44] Cyberpower678 I think he could but you first need to tell him what... [15:21:05] Coren, you said you could let me access archive in the replication. [15:21:16] Do you think you could set that up? [15:21:33] Cyberpower678: That's not going to be instantaneous; it'll need some back-and-forth with legal. ~2 weeks, I'd expect. [15:21:58] Well so much for migrating X!'s edit counter and adminstats for now. [15:22:50] Cyberpower678: I need to set up a researcher-equivalent right with the same approval structure for Tool Labs first. [15:23:21] researcher has access to archive. [15:23:34] I know. That's why I need to set that up. :-) [15:24:25] Cool. [15:24:36] Coren, also give CU and Oversight. :D [15:24:47] * anomie updates http://tools.wmflabs.org/anomiebot/packages.html now that Coren and petan have the new exec nodes up [15:25:03] Cyberpower678: That's never going to happen. :-) [15:25:18] Then I'll just hack Wikipedia. [15:25:59] anomie can you tell me how it works? [15:26:05] anomie I would like to see the script which does it :> [15:26:10] seems handy [15:26:20] Coren, how do you connect to the DBs again? [15:26:49] Cyberpower678: Easy-sauce version: sql [15:26:58] Like: sql enwiki [15:27:03] petan: Sure. Code is in /data/project/anomiebot/pkgs. It just submits a job on each node that does dpkg --get-selections, then filters the results after they all complete. [15:27:13] Coren: what did you mean by "will still be gridengined" [15:27:29] JohannesK_WMDE: That the instance will be a grid engine exec host. [15:27:35] it seems that lot of stuff has changed in default images [15:27:43] there is no iptables for example [15:27:54] Coren, where's the replag table? [15:28:10] I am wondering howcome I installed ksh on exec-06 and not on 05 o.O anyway I think that ksh and csh should be default everywhere... [15:28:25] it's just a common shell [15:28:31] Cool. I can see the cu_log table. :p [15:29:30] and what does that imply exactly (or: why do you want that), Coren? [15:30:01] JohannesK_WMDE that the user processes (like bots) will run on it [15:30:40] Coren, you still there? [15:31:07] doesn't that kind of run counter to the possibility of permanently allocating CPU cores, Coren/petan? [15:31:12] JohannesK_WMDE: So that it remains managed through the same scheduler, and job status can be seen the same way as everything else. [15:31:36] JohannesK_WMDE: I can dedicate the host to a specific queue(s), for that tool. [15:31:41] Cyberpower678: Yes? [15:31:55] Coren, where's the replag table? [15:32:03] ok... [15:32:17] Cyberpower678: There isn't one yet. Nor is there a metainformation table. Also "yet". [15:32:32] Coren, ETA? [15:32:58] Cyberpower678: Metainformation soonish, next week I guess. replag is low priority; maybe the week after that. [15:33:33] So the edit counter will still need to wait a few weeks before it gets deployed on labs then. [15:34:27] o_O Why in blazes would an edit counter need a replag table? [15:34:28] !g 66264 [15:34:28] https://gerrit.wikimedia.org/r/#q,66264,n,z [15:34:43] Coren I think this needs a merge :o [15:34:57] Coren, the replag is informative when people look at the edit count of a user. [15:35:15] It's helped a lot for me too. [15:35:20] I was surprised that mysql-client wasn't there as all exec nodes have it [15:35:27] except for latest 2 [15:36:12] Cyberpower678: It's completely pointless in this setup. Why would you even /want/ to know that the editcounter is 5 or 20 seconds behind? Nobody needs that level of detail, it's certainly not worth stalling a tool to wait for it. [15:36:19] YuviPanda: i figured it out :D [15:36:34] Coren, not that specific. [15:36:51] When replag is high, that's when the message gets dispalyed. [15:37:00] oh, what was the deal, legoktm [15:37:06] Coren: It probably was more important on the toolserver where replag tended to get hours behind. [15:37:10] Like on toolserver when it was replaged for 48 hours that one time. [15:37:29] Coren: why do we have apache servers running on exec nodes? [15:37:56] petan: There shouldn't be. Some oddity, I expect, with image experimentation for a while. [15:38:06] I have no idea how well labs handles replag but I think it would still be nice to have the replag notice there when there ever is a time replication stops all of a sudden. [15:38:12] hmm, they are even in latest version so it's likely somewhere in puppet [15:38:15] Cyberpower678: More useful would be the timestamp of the last edit. [15:41:07] petan: -2'ed. Asking for mysql-client when we use mariadb-common == trouble. :-) [15:41:37] ok I was thinking it's aliased [15:46:23] !log tools petrb: replacing mysql with maria on exec nodes [15:46:24] Logged the message, Master [15:47:05] !log tools petrb: replacing mysql with maria on exec nodes [15:47:06] Logged the message, Master [15:47:10] damn [15:47:12] :D [15:47:21] bash history being counterproductive [15:48:48] at some point that sql tool should be probably everywhere as well [15:48:58] what if someone wanted to make a shell script which use it and submit to grid? :P [15:50:02] YuviPanda: actually i do need help with something. i was using the sh module, and when i did sh.svn.up() it failed, then I did svn = sh.Command('/usr/bin/svn') that failed too [15:50:13] it said it didnt exist or something when running w/ jsub [15:50:22] legoktm: PATH issue perhaps? [15:50:32] legoktm: does svn exist in the exec nodes? [15:50:50] i ssh'd to tools-exec-1 and svn worked there at /usr/bin/svn [15:51:14] ohhhh [15:51:16] not all of them have it [15:51:21] local-pywikibot@tools-exec-02:~$ svn [15:51:21] The program 'svn' is currently not installed. To run 'svn' please ask your administrator to install the package 'subversion' [15:51:26] * legoktm pokes petan ^ [15:51:34] can we get svn installed on all exec nodes? [15:51:52] legoktm yes, but what for? :P [15:51:54] just being curious [15:52:01] legoktm: hmm [15:52:02] legoktm: :D [15:52:59] petan: i wrote a script that automatically runs unittests after new pywikibot commits [15:53:03] ok [15:53:11] so it has to run svn up to update, then svn info to get the revision # [15:55:33] !log tools petrb: installed subversion to exec nodes 4 legoktm [15:55:35] Logged the message, Master [15:55:44] :D [15:55:46] ty [15:55:57] legoktm: :D [15:56:00] should work now [15:56:07] legoktm: also I like sh.svn('up') better, for some reason [15:56:52] petan: i also got it to send an message to #pywikipediabot via wm-bot :D [15:57:04] aha great :P [15:59:59] * legoktm grumbles about not enough memory [16:00:10] legoktm: did it get killed? [16:01:02] actually everything worked except sending the message via IRC> [16:01:22] sh.nc('bots-labs', '64834', _bg=True, _in="#pywikipediabot "+msg) [16:01:58] can somebody please help me to get a tool running on tool labs? [16:02:19] Pyfisch: That's what we're here for. What problem are you running into? [16:02:44] Coren: getting an internal server error. [16:02:58] Pyfisch: can you run it fine via shell? [16:03:08] if its python, try enabling cgitb [16:03:20] yes it is python [16:03:40] brb [16:03:58] i have enabled cgitb but I only get internal server error [16:04:14] Pyfisch: What is the name of your tool? [16:04:39] it is "wdrdr" [16:04:49] the file is in cgi-bin [16:05:16] and it is (not) accessiable over http://tools.wmflabs.org/wdrdr/cgi-bin/index.cgi?item=q1 [16:05:36] Pyfisch: The script needs to be owned by your tool, not you. :-) [16:05:55] Pyfisch: From your tool account, do: take /data/project/wdrdr/cgi-bin/index.cgi [16:06:21] Coren: how can I switch to the tool account? [16:06:35] Pyfisch: "become wdrdr" [16:06:41] Pyfisch, become [16:07:01] Pyfisch: You might want to "take ." once you did if you own every file there; gonna make things much simpler. [16:07:37] ok I have done this [16:09:30] Pyfisch: You're almost there. The script also needs to be executable. [16:09:46] Otherwise it won't... be... executed. :-) [16:10:03] Cheat sheet: chmod a+x your/script [16:11:25] Pyfisch: Look at the list at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Logs most 500 errors are explained by those simple errors. :-) [16:15:16] Coren: ok I checked the errors there, but it don't work [16:16:26] So I fixed the issue by throwing more memory at it. [16:16:29] Aha. I probably need to add this to the common errors FAQ: your file has returns in it (^M) which breaks many things -- in particular your shebang at the start. [16:17:29] Pyfisch: Unix only likes newlines at the end of lines, not carriage-returns followed by newlines. I can fix that one file for you simply enough, but you'll have to make sure you edit it with an editor that doesn't put CRs back in it. :-) [16:18:15] Pyfisch: Done. You can try it now. [16:18:41] Coren: he it works [16:19:07] Coren: if I edit it now with vim, will I break something with the newlines? [16:23:21] Pyfisch: Nope. [16:23:44] Pyfisch: Pro tip: If you edit a CR-NL file with vim, type ":set ff=unix" will automagically fix it. :-) [16:23:46] Coren: i wrote the graph to /tmp, and reading it in [16:23:51] is very slow for some reason [16:24:10] JohannesK_WMDE: I can't think of a reason why. /tmp is on a local filesystem. [16:24:42] yes, dd [16:25:02] Coren: added to my cheatset ;-) [16:25:15] the reading itself had OK throughput [16:27:04] but the graphcore-enwiki process was just sitting there with single-digit cpu usage for a while... it should use lots of cpu while importing (which it does now) [16:29:02] How odd. [16:32:05] does vmstat show usable values for swapin/out and io on the VMs? [16:33:30] or are those kind of dummy values and the real values are only available on the physical machines, Coren [16:34:49] JohannesK_WMDE: That distinction is really blurry in a setup like this; they are the VM's /share/ of the physical machine. [16:35:17] JohannesK_WMDE: Unless the physical machine is trashing (which should never happen because it doesn't overcommit) they are the same. [16:36:47] share? so the values should the amount of io on the physical machine, which is caused by this VM i guess [16:36:57] *should display [16:39:07] let's make everything openvz then you really get to see fun numbers! [16:39:08] * Damianz snort [16:39:44] Damianz: ? [16:41:55] Container based virtualization == processes are tagged, with no real isolation. Makes for fun numbers, esp as it's common practice to overcommit them... even more fun if you login to the host [17:13:07] the thing is, i can't see what the machine is doing Damianz - i'm reading a GB of data from disk and don't see any change in IO in vmstat [17:14:48] JohannesK_WMDE in that case the problem is not in disk storage, which machine is it Johaness? [17:15:12] tools-dev [17:16:00] in any case i should see *some* IO when i read a GB - or is the ram cache so large? [17:17:26] !log tools petrb: installed lsof to -dev [17:17:28] Logged the message, Master [17:18:52] ah, wait ... [17:19:28] the box is swapping like hell that is why [17:19:37] it actually slow because of memory [17:19:57] now it refreshes, and i see the IO [17:19:57] also it is waiting on cpu [17:20:15] yes, memory [17:20:19] it uses 2 threads both thread using 100% of cpu [17:20:33] i will try smaller wikis first then, until i can move to a larger vm [17:20:46] anyway, it did work in the end [17:21:42] and yes, the sorting is done in 2 threads, and they would use cpu [17:21:54] either run this with lower memory or on grid where we have 8gb boxes [17:22:20] yes, a separate vm is planned in the long run... [17:22:24] ok [17:23:03] i am not sure if even 8 gb will be enough if we import all the wikis (and i have been asked to do that by people) [17:23:41] but that is supposedly the good thing about labs: when we need more resources, new hardware will just be plugged in... or that's what i was told [17:24:10] !log tools petrb: removing old kernels from -dev because / is almost full [17:24:11] Logged the message, Master [17:24:35] yep, but we should try to conserve resources if possible [17:24:52] what are you trying to do? [17:25:31] i just imported the graph structure of enwiki into catgraph [17:25:37] http://tools.wmflabs.org/render-tests/catgraph/ [17:25:51] ah ok [17:26:08] I think daniel was telling me about this [17:26:18] we may be able to dedicate an instance to this [17:26:56] note that IO is an issue, though [17:27:10] if you are using local storage, it'll affect all other instances running on the same host [17:27:25] and for databases, it's the only way to really go about it [17:27:32] or is this all in-memory? [17:27:52] yeah, i talked to coren about it before, and he said dedicating an instance - or even a physical machine possibly - would be possible. and that we would be able to get cpu core(s) locked to a VM if needed. [17:28:01] it is in memory [17:28:12] well, physical machine is more work [17:28:16] but for reading the graph structure, i use a temporary file [17:28:38] locking cpus to an instance also isn't simple [17:29:02] I never said it's be *simple*, just that it'd be *possible*. :-) [17:29:12] CPU resources aren't a huge problem at this point, though [17:30:04] https://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&s=by+name&c=Virtualization%2520cluster%2520pmtpa&tab=m&vn= [17:30:20] we're at 20-50%, depending on which host [17:30:27] and new instances go on the less used hosts [17:31:32] locking a cpu core to an instance, how hard would that be / what would it entail? [17:32:48] I'd very much prefer to avoid that [17:33:01] there isn't likely to be much gain from it [17:33:05] JohannesK_WMDE: Honestly, I would want to see your code getting problems that need solving before we hard allocate resources. :-) [17:33:49] if this is totally in-memory I don't see much issue [17:34:12] memory's cheap [17:34:20] the biggest slow-down for instances at this point is disk io [17:34:44] the parsoid instances were able to cause 80% cpu utilization for 4 nodes [17:35:23] well ok -- if i have enough memory, i can skip the temp file, so updates will cause only some load on the sql server, and temp memory usage [17:35:24] larger instances get more cpus, btw [17:35:35] The dumps instances managed to saturate the network and disks :P [17:35:44] yeah.... [17:35:57] those… [17:35:58] heh [17:36:20] locking down a cpu core would be good to guarantee quick response. i'm not sure whether that will be needed or not [17:36:52] for large instances you'd need to lock down like 8 cpus [17:36:58] but a gcc which understands -march and -mtune properly would be nice [17:37:14] there's 24, thanks to hyperthreading, but that's very much eating up a host [17:37:21] oh, so you can't lock just 1 core? [17:37:32] effectively? no [17:37:45] hm [17:37:51] is this not multi-threaded? [17:38:19] You had one problem, found threads, now you have 50 problems, running at once [17:38:31] it uses 2 threads for sorting. queries are currently single-threaded, but they are so fast that this currently isn't an issue [17:39:06] so, ideally this would be a proof of concept for production, right? [17:39:48] is there any planned scaling mechanism for this? [17:39:54] well, i hope to get a separate mariadb instance running, with UDFs which talk to catgraph. then the mariadb instance would be connected to production using federatedx [17:39:56] we could dedicate more than one instance [17:40:27] https://ganglia.wikimedia.org/latest/?c=Virtualization%20cluster%20pmtpa&h=virt7.pmtpa.wmnet& [17:40:42] ugh. I need to shut down the log collectors [17:40:51] so, people can just talk to catgraph using sql - including joins - which makes it easy and efficient, but if there should go something wrong with catgraph, production dbs won't be affected [17:40:59] they're causing like 10% wait [17:41:19] someone in the channel here suggested this (scfc?), i think it's a great idea [17:41:24] * Ryan_Lane nods [17:41:54] only 30? [17:41:54] how would the catgraph specific part scale? [17:41:58] sharding? [17:42:10] * Damianz got his 3tb oracle databse server to a load of 800 yesterday [17:42:13] Damianz: yeah. it's probably not a huge issue right now [17:42:21] oh, it's not the load I'm worried about [17:42:24] it's the disk wait [17:42:39] wio is only a little high [17:42:43] yeah [17:42:52] probably not a massive issue yet [17:42:58] I want to get some log collectors on real hardware [17:43:08] used for both production and labs [17:43:22] That would be awesomeness [17:43:28] distributing it onto several vms would be possible. we could just start several graphserv processes, with each talking to some graphcores (i.e. wiki dbs), on several vms. then, there would be some mechanism to make clients be able to tell where they need to connect to for a specific graphcore instance. like dns - similar how it works with the "enwiki.labsdb" etc [17:43:41] JohannesK_WMDE: that sounds sane [17:44:01] Damianz: 800? o_o [17:44:08] I'm sure there's a wonderful story behind that? :P [17:44:12] We should just aim for labs to be the worlds largest openstack on ssd deployment and forget about disk lag [17:44:31] Damianz: I wouldn't mind having at least a couple of the nodes be ssd [17:44:36] FastLizard4: Blame oracle [17:44:47] Oh, I don't doubt it :P [17:44:51] and we could use scheduler hinting to say "give me an SSD instance" [17:44:56] Ryan_Lane: like i said, given enough memory i don't see much disk io happening with catgraph :) [17:45:13] Ryan_Lane: It would be awesome if we had something like netapp, where you get like 30tb of disk with 600gb of ssd cache on top per node [17:45:30] Damianz: well, we do have one, but it's actually faster using local storage [17:45:31] Well on the little nodes... never speced much past 100tb heh [17:45:50] and more stable [17:46:04] lol [17:46:07] with shared storage a single instance can cause performance problems for every other instance [17:46:25] of course if a node dies...... [17:46:29] :) [17:46:33] But you can live migrate magic :D [17:46:46] I want xen's capability to live migrate without shared storage [17:46:58] I kinda like the whole EBS idea, but then when you loose the disk it's just meh.. hey gluster [17:47:06] yep. exactly [17:47:21] I thought qemu supported node to node migrations without shared storage... maybe not openstack hmm [17:47:34] I'm not very keen on going back to shared storage [17:47:38] Damianz: it's there [17:47:41] Damianz: but it's unreliable [17:47:49] Oh it's one of *those* features [17:47:51] and the qemu folks say "don't use this" [17:47:51] Like resizing [17:48:00] actually, resizing works now! :) [17:48:04] oooh [17:48:19] Bet it doesn't work via OSM *raises feature ticket* [17:48:20] it has a shitload of possible issues, though [17:48:42] it can't do disk resizes with partitions [17:48:46] which causes an error condition [17:48:57] so, we need to hide the s1 instance sizes, I think [17:49:33] That kinda sucks - no mount the image, shrink the partition then shrink the image logic? [17:49:37] though I guess it's only the primary disk [17:49:42] the secondary is a full disk [17:50:04] I'd be happy to leave the disk static and just have more ram in a bunch of cases heh [17:50:19] I think that's what we have [17:50:20] ram to cpu sorta makes sense... 80% of the time... btu some apps are heavy one way [17:51:23] * Damianz really, really needs to find the option to disable the stupid left swipe thing in osx 10.8 as it makes chrome hard [17:51:30] ah [17:51:35] I already hide the s1 instances [17:51:44] the rest all have a 15GB root [17:51:50] (which is really closer to 13) [17:52:11] I had to increase that because of some security check that openstack now does [17:53:39] so, I should be able to add the resize action [17:53:47] Damianz: add a bug for me? [17:54:08] I tested it on 4 different instances [17:54:18] both shrink and increase [17:54:26] Just because you don't want to use the wanky bz interface :P [17:54:40] :) [17:54:44] I can enter it too :D [17:54:51] Duplicate bugs! [17:54:59] Though you've closed like 8 of my bugs recently =D [17:55:03] Only been there over a year [17:55:18] you mentioned adding in a bug, so I thought I'd let you go for it :) [17:55:35] hell, a year in wikimedia time isn't an eternity :) [17:55:49] ryan_lane, how do I add a region to a new folsom install? [17:55:58] I think my ldap extension patch was there a good 8 months :D [17:56:05] andrewbogott: oh, you mean the keystone stuff? [17:56:15] Damianz: yeah, I kind of ignore ldap authentication changes [17:56:19] I'm terrible at that [17:56:24] that extension scares me [17:56:34] It's clungy [17:56:41] well, I wrote it 7 years ago [17:56:42] Ryan_Lane: I mean whatever this is querying: $this->userNova->getRegions( 'compute' ) [17:56:48] andrewbogott: yeah [17:56:51] that's the one [17:56:56] I have a script for this :) [17:57:23] lemme find it [17:57:28] I'm looking at old nova docs that have 'nova zone-add' but I take it things are totally different now. [17:57:39] you actually use keystone [17:57:45] hmm [17:57:53] I can't search for an instance page by content anymore? [17:58:03] hm. why isn't virt1000 responding? [17:58:05] that's not good [17:58:27] yeah, it's dead [17:58:30] * Ryan_Lane goes to mgmt [17:58:49] yikes [17:59:03] oh [17:59:07] I bet its iptables [17:59:25] 49020 [18:00:59] btw, the replication tables, are they also mostly in RAM? i guess they do hit the disk right? [18:01:00] and if I put it against the right thing... it would be good [18:01:18] hmmmm [18:01:51] link is up. iptables is flushed [18:01:54] I doubt much of the data is buffered... indexes hopefully are [18:02:06] * Damianz thinks he might have just spammed like 10 people [18:03:50] andrewbogott: heh. the script is on virt1000 [18:04:22] oh [18:04:25] * andrewbogott is patient [18:04:26] there's one on virt0 too [18:04:29] Ryan_Lane: Since you can't search for an instance name and get the resource page anymore, how hard would it be to make Nova_Resource:tools-exec-06 map to Nova_Resource:I-00000774? [18:04:32] prod.sh [18:04:35] you'll need to modify it [18:04:37] ok [18:04:38] Since we're suppose to be going that way anyway... [18:04:38] sure. [18:04:50] Damianz: you can't do that anymore? [18:04:55] via the bot you mean? [18:05:15] If I search for bots-cb or bots-apache01 or tools-exe-06 on wikitech I don't get any result for the nova resource page [18:05:22] I use to get the instance page in like the 3rd hit [18:05:50] Maybe just a namespace thing since the merge or summut... [18:06:02] * Damianz goes and gets his phone so he can login and browse the table [18:11:37] Warning: There is 1 user waiting for shell: Tristessa de St Ange (waiting 0 minutes) [18:12:38] stupid computer [18:12:40] andrewbogott: regions are defined by keystone, and nova resources define their region via their config files [18:12:50] not sure what my last message sent was [18:13:14] Damianz: did you get my messages about how to search for stuff now? [18:14:50] nope [18:14:53] last was ' via the bot you mean?' [18:15:34] ah [18:15:44] Damianz: I removed the namespace from default search [18:15:49] ah [18:15:50] you can use "everything" [18:16:00] or do a custom search and include the namespace [18:16:07] So it's everything or an advanced serach :) [18:16:08] it was cluttering up the search [18:16:16] * Damianz shoots ux changes [18:16:19] yeah [18:16:29] I'm not a huge fan of mediawiki's search [18:17:12] Should make an extension to elasticsearch the whole thing [18:17:40] could use lucene [18:17:48] people want to test out solr somewhere, too :) [18:25:07] Warning: There is 1 user waiting for shell: Tristessa de St Ange (waiting 13 minutes) [18:25:44] does anyone have anything blocking on me? [18:26:50] Ryan_Lane, this script defines a bunch of *_URL things up top that aren't used… that's just left over from a former incarnation? [18:27:01] oh. lemme take a look [18:27:37] andrewbogott: which specifically? [18:28:09] they all look used to me [18:30:45] NOVA_INTERNAL_URL [18:31:19] I think it needs to be defined [18:31:44] Ah, read but a subscript. OK, in that case, where does $(tenant_id) come from? [18:31:48] *by [18:32:08] $(tenant_id) is a macro for keystone [18:32:21] oops, i just mailed roughly everybody i think [18:32:21] it's being injected directly [18:32:41] Ooooh, ok. I guess it's escaped [18:32:44] OK, thanks. [18:32:57] Hope you're feeling better, btw [18:33:19] I am, thanks :) [18:36:22] does tools handle ruby on rails? [18:36:41] or is it only php for "webtools"? [18:38:17] anything via cgi [18:38:22] Coren: ^^ [18:38:29] wsgi is going to be supported as well [18:38:41] Warning: There is 1 user waiting for shell: Tristessa de St Ange (waiting 27 minutes) [18:38:45] and I'd imagine other similar things [18:39:08] AzaToth: Anything that has Ubuntu support can be installed, basically. [18:39:23] AzaToth: Albeit I don't think I've had anyone ask for Ruby yet. [18:39:59] Coren: the tool I have on ts atm I want to rewrite in rails [18:40:07] Cyberpower678: and you were searching for Coren yesterday [18:41:06] Coren: if people don't think it is available, they might not think to ask about it and just ignore it [18:41:55] Version: 2.3.14-2 do the trick for you? [18:43:19] Jasper_Deng, ? [18:43:31] AzaToth: ^^ [18:44:11] Coren has assured me that he is busting his chops to create a researcher access level for the replication. [18:44:22] :p [18:44:37] So I can get my archive access. [18:45:12] Coren: I think most people doesn't use rails from the system, but uses http://rvm.io or similar [18:45:43] Coren: the issue is to have it be accessible from a good point [18:46:16] apache or nginx, passenger or whatever [18:46:17] I would really /really/ rather use the Distro-maintained version if at all possible. [18:46:45] it's going to be very difficult to use something other than the distro version [18:46:45] 2.3 is pretty old [18:46:55] Ryan_Lane: how so? [18:47:03] rvm is evil [18:47:10] and we'd need to backport the packages [18:47:15] ... [18:47:28] which has the possibility of affecting puppet (and other things we have running ruby) [18:47:38] those two sentences doesn't fit your first assertion [18:47:40] AzaToth: We have no installs from external repos. [18:48:02] gems is evil, pip/easy_install is evil, npm is evil, pear is evil, etc. [18:48:19] Ryan_Lane: well, offcourse rails apps will need gems [18:48:56] Ryan_Lane: please don't used "foo is evil" without asserting _why_ it's evil [18:49:22] because all of those things pull random versions of code from random places, often unsigned [18:49:40] * Damianz takes his virtualenv, pip installed package and hosts it on Ryan_Lane [18:50:01] none of them have security-stable releases [18:50:23] They also don't get free updates, which on one hand is bad and on the other yay don't break my app [18:50:28] all do security updates in master, which means if any gem has a specific version dependency the entire stack could have vulnerabilities [18:50:34] Ryan_Lane: evil to the point where labs will never* facilitate using that kind of dependency management (* for some reasonable value of never) [18:50:54] (that was meant as a question) [18:50:56] Ryan_Lane: I don't understand how that can be "evil" [18:50:59] well, we can't really have hard requirements. this is kind of a hard problem [18:51:03] If you want new ruby that bad, rvm it and use cgi :D [18:51:19] AzaToth: unpatchable vulnerabilities are evil [18:51:33] downloading and installing unsigned binaries is evil [18:51:38] It would be aweosme if the webservers could reverse proxy easily though... throw a virtualenv'd dir onto the grid, run gunicorn and proxy to the port from web would be cool [18:51:44] I think your understanding of the word "evil" is a bit wrong [18:51:46] heh [18:52:06] Warning: There is 1 user waiting for shell: Tristessa de St Ange (waiting 40 minutes) [18:52:32] it's shorter than "insecure and highly not preferred" [18:52:52] Ryan_Lane: it's a totally different meaning [18:52:53] alas, most ruby applications require like a billion gems, so i think we'll need to allow it [18:53:18] AzaToth: isn't evil in the eye of the beholder? ;) [18:53:32] Ryan_Lane: evil implies intent [18:54:10] rvm/pip/etc have an intent of operating in this way, even though they know it's relatively insecure [18:54:22] I consider that an evil on the world [18:54:23] I'd say allowing users to run their own code is pretty much the definition of 'insecure' ;) [18:54:33] martijnHH: thought about that as well [18:54:34] martijnHH: indeed [18:54:49] again, I don't think we can get around it. I just don't like it :) [18:55:02] labs has arbitrary code execution by design [18:55:07] yep [18:55:15] all the way to the root level [18:55:26] terrifying, right? [18:55:33] Ryan_Lane: I wonder whom generally is making most insecure code, a php programmer, or a rails programmer [18:55:46] Making? php, rails comes with them for free [18:55:50] hahaha [18:56:01] AzaToth: which php programmer, and which rails programmer? [18:56:02] heh [18:56:12] martijnHH: the stereotypical one [18:56:28] AzaToth: note that I have the same issues with pear [18:56:33] AzaToth: tl;dr you get support for signed packages. You can make a shared installation of Ruby if you want, with any update mechanism you wish (check out /shared) but it's user-supported then. If you break it, you get to keep both pieces. :-) [18:56:50] shared is likely not going to work well [18:57:17] chances of that breaking are pretty much guaranteed. [18:57:27] Coren: I don't think there needs to be a shared installation of rails/ruby, just a intrastructure to have it accessible on the web server [18:57:40] I'm never using shared installation [18:57:47] * Ryan_Lane nods [18:57:57] I think this is doable like wsgi [18:58:11] never used wsgi [18:58:30] wsgi is awesome [18:58:35] but so is gunicorn :D [18:58:43] smells python [18:59:00] wsgi is python, yes ;) [18:59:18] You could just Passenger up apache... but you'd have to tweak the configs for paths and stuff IIRC [18:59:25] reminds me of http://xkcd.com/927/ [18:59:48] I assume each tools isn't in it's own instance [19:00:14] Damianz: yeah, that would work as well [19:00:16] AzaToth: no, shared [19:00:22] we can add more as load dictates [19:00:34] Ryan_Lane: /shared serves two purposes: it's a location where developers can collaborate to maintain... well, shared stuff; and it will serve at regular intervals as an object lesson why it's important to have distro-maintained packaging and change management. :-) [19:01:01] Coren: yeah. I doubt many people are masochistic to use a shared rvm, though [19:01:05] Coren: You wouldn't believe how badly some people can miss the point of change control [19:01:15] *masochistic enough [19:01:24] it's different [19:01:27] security-stable things are awesome [19:01:47] people share a rvm is different from tools share a rvm [19:01:57] * Ryan_Lane nods [19:02:05] indeed [19:02:13] Ryan_Lane: Auto reboot without warning on unattended upgrades is not awesome though [19:02:16] tools sharing one makes a lot of sense [19:02:18] AzaToth: In more concrete terms, though, tell me what you /do/ need to do your thang. :-) [19:02:30] Damianz: I don't think we're doing that [19:02:35] it shouldn't, anyway [19:03:02] I've been pretty tempted to make that happen, though [19:03:16] sort of related: if a user requested package is not available through the ubuntu repos, does propviding it through a PPA help in any way shape or form? [19:03:27] You should put out and get landscape licenses donated :P [19:03:29] Coren, can you delete xtools? [19:04:30] martijnHH: Probably, because that makes it easy for me to dump in our local repo. [19:04:35] Coren: the only thing that is needed is a way to map http://tools.wmflabs.org/toolname via passenger or wsgi to the rails process [19:05:02] other than that I think it's up to the tool authors to use whatever mech they want for running the tool [19:05:27] Warning: There is 1 user waiting for shell: Tristessa de St Ange (waiting 53 minutes) [19:05:28] passenger runs the tool [19:05:37] normally I'd jsut run rack or such and reverse proxy form nginx [19:05:41] but that requires custom configs [19:05:55] Damianz: true [19:05:59] martijnHH: we avoid using PPAs [19:06:05] I've myself have only used passenger [19:06:09] martijnHH: instead we import the package into our custom repo [19:06:12] Damianz: Oh, that runs as a server? [19:06:42] martijnHH: PPAs can go away, which leaves you high and dry. we know this from (painful) experience [19:07:04] Coren, can you delete xtools? [19:07:04] ah. Coren had already answered that [19:07:05] Damianz: It's the same problem set as WSGI then. It's not that complicated, in theory, but I need to find a clean way to do it uniformly for all similar mechanisms. [19:07:34] Cyberpower678: Yes, I can. Have you backed up anything you'd want to keep? [19:07:45] Ryan_Lane : I was thinking of creating and maintaining the PPA myself [19:07:52] JackPotte did yes. [19:07:55] * Ryan_Lane nods [19:07:55] Coren: What, rack? It's technically middleware, the server bit is normally Unicorn or $pick_another [19:08:01] We are clean slating. [19:08:02] we'd still want to pull the packages into our repo [19:08:22] labs needs to follow ops policy and that policy says no ppas :) [19:08:25] Ryan_Lane: are you using reprepro? [19:08:30] AzaToth: yep [19:08:37] Damianz: Right, the only obstacle is to find some clean way for that bit of middleware to notify the webserver(s) where on the grid it ended up. [19:08:45] Coren: Same as running python, in tools you'd normally kick the server off as a continuious process on grid then map the port... but because things can move in "cloud" you normally track that [19:08:46] Ryan_Lane: I've used reprepro too much ツ [19:08:52] Ryan_Lane: ah, ok [19:08:53] Coren: yeah [19:08:54] martijnHH: that's a good way to let us easily pull in the packages, though [19:09:00] so, please, do maintain a ppa [19:09:00] But that should be easyish [19:09:09] We know where tools are running, just need to update configs [19:09:12] Via like magic and such [19:09:30] Coren: Totally install nginx with lua and write some lua to live query where a tool is running [19:09:32] martijnHH: What Ryan said. If you give me a PPA, I can pull the package and dump it into the labs repo. [19:09:33] Because that would be fucking a [19:09:36] Coren, We are clean slating. [19:10:23] Coren: you could add the ppa as a update for the reprepro repo [19:11:32] Damianz: Actually, I'm going to do it with apache rewrite rules and a rewritemap. :-) [19:11:33] Damianz: how does ports work? [19:11:44] petan, Coren seems to be a little busy. Do you think you can delete xtools? [19:11:49] Coren: That's just sadistic [19:11:55] I assume everyone uses port 3000 ヾ [19:12:06] Cyberpower678: I am, but how is this urgent? :-) [19:12:13] AzaToth: Ideally environment vars, where each tool is assigned one [19:12:19] ok [19:12:45] Coren, because we are all currently online and would like to get this moving before toolserver completely collapses. :) [19:12:54] aaand, offcourse you want to have access to a staging server as well ツ [19:12:57] Cyberpower678: /before/ you're too late [19:12:58] :P [19:13:14] hehe [19:13:25] Damianz, I don't get it. [19:13:26] Cyberpower678: Yeah, but deleting the tool can't possibly be a blocker unless you want to create it anew; and you might as well keep the one you have if that's what you intend. :-) [19:13:46] We want a clean slate. [19:13:58] * Coren chuckles. [19:14:03] rm -rf ~ [19:14:09] That project is completely broken. [19:14:10] rm -rf works really well, too. :-) [19:14:15] But yeah, gimme a sec [19:14:39] Cyberpower678 back [19:14:53] (Beware, there will be a period of instability while things get settled) [19:15:09] Coren: I would assume it's best to keep rails apps on it's own server separate from the "php dudes" ツ [19:15:31] Coren today I fixed the rmtool script [19:15:39] Coren it works for replica file as well [19:15:50] Coren if you want me to delete that project let me know :P [19:15:54] Cyberpower678 ^ [19:16:03] petan: Have fun. [19:16:41] petan, please. [19:16:49] Cyberpower678 do you really want to delete it? (it can be restored actually but it is rather complicated) [19:17:12] It's loaded full of broken stuff. [19:17:22] Better to just delete it all. [19:17:24] ok [19:17:45] !log tools deleting xtools project (requested by Cyberpower678) [19:17:47] Logged the message, Master [19:18:48] Warning: There is 1 user waiting for shell: Tristessa de St Ange (waiting 67 minutes) [19:19:23] Cyberpower678 you removed it from ldap already? [19:19:29] I mean the service group [19:19:37] normal users can't delete [19:19:39] only create [19:19:44] No I just recreate it. [19:19:49] Ryan_Lane aparently they can [19:19:54] o.O [19:19:59] But the original folder is still there. Why? [19:20:18] Cyberpower678 can you please confirm you /did/ delete the project using wikitech? [19:20:18] ah. right. service group members probably can [19:20:27] we might want to change that [19:20:37] Cyberpower678 I just need to make sure I dont break it :P [19:20:42] petan, not confirmed. I need to be a projectadmin. [19:21:02] but I cant find it :/ [19:21:06] petan, when a service group gets deleted, why does the folder stay? [19:21:30] Cyberpower678 because shell script rmtool needs to be executed in order to remove it, which is kind of too delicate for it to be automatic [19:21:31] :P [19:21:36] deleting stuff is dangerous [19:21:40] petan, delete it again. [19:21:50] I want everything to be gone. [19:21:57] Cyberpower678 hold on, I cant see a service group? who removed it? [19:22:23] petan, I just recreated it. It should be on the bottom now. [19:22:28] ah ok [19:22:35] that is problem [19:23:02] you need to delete it again, or rmtool will conflict with the tool which actually create a new account, which probably already happened [19:23:28] petan, I can't delete it. I'm not a projectadmin. [19:23:30] there is daemon called toolwatcher I dont know how it behaves when old folder exist while new group is created [19:23:35] o.O [19:23:50] ok... who deleted it before then? [19:23:55] Coren? [19:24:07] I will remove it this time and plz dont recreate it until I clear the current tool [19:24:11] * Damianz gets the popcorn [19:24:21] Damianz, why? [19:24:31] Amusing discussion [19:25:04] This is why I... suggested... that they should have kept the old one. :-) [19:25:50] Folder seems to be deleted now. [19:26:03] petan, clear to create? [19:26:06] :D [19:26:10] noez [19:26:12] wait [19:26:14] it is still running [19:26:16] waiting. [19:26:21] it backup all databases and stuff as well [19:26:35] not really necessary. [19:26:37] :p [19:26:42] it needs to clear the db user as well and crons and all that stuff [19:26:49] cool. [19:27:05] and webserver config :P [19:27:09] petan truly belives in 'move slowly and break things' [19:27:10] :D [19:27:16] :D [19:27:52] ok, there was no cron and the other configs will be reinstalled soon [19:28:07] Cyberpower678 you can create it now [19:29:10] petan, urg. Where's the folder? [19:29:33] hold on [19:31:10] it is there [19:31:21] petan: http://fr.wiktionary.org/w/index.php?title=d%C4%9Bkuji&diff=14704172&oldid=14615995 [19:31:45] petan, where's the .my.cnf file? [19:31:54] Cyberpower678: jackpotte@tools-login:~$ become xtools [19:31:54] sudo: sorry, a password is required to run sudo [19:31:54] jackpotte@tools-login:~$ [19:31:54] Cyberpower678 there is no such thing... [19:32:02] JackPotte logoff [19:32:04] try again :P [19:32:18] Warning: There are 2 users waiting for shell, displaying last 2: Tristessa de St Ange (waiting 80 minutes) Profoss (waiting 12 minutes) [19:32:24] JackPotte you need to be added to project by Cyberpower678 [19:32:45] petan, yes there is. [19:33:08] Cyberpower678 I mean now you get only 1 file by default [19:33:13] no .my.cnf automatically [19:33:25] My cyberbot and peachy project has it. How else does it connect to its personal database? [19:33:44] the replica.my.cnf should contain the password for local db as well [19:33:53] cool. [19:33:54] BUT apparently it doesnt let me investigate why [19:34:04] I think our rm process is missing some bits [19:34:24] AHA [19:34:35] I found a file it forget to delete :D [19:34:37] no worries [19:36:19] petan, I think you are mistaken. [19:36:25] lol that rmtool is getting huge [19:36:26] mysql is broken. [19:36:30] Cyberpower678 I know I know [19:36:32] it is being fixed [19:37:11] OK now it is broken :P [19:37:14] let me check it again [19:37:35] it's missing the .my.cnf file. [19:37:47] replica does not have the local access codes. [19:38:18] ok now I see what is wrong [19:38:44] part of tool-making scripts are on nfs which I cant access so i couldnt know it [19:38:48] fixing now [19:40:38] petan: the replica is working as well, thanks again [19:42:24] petan, ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) [19:42:39] local SQL is still busted. [19:43:03] Cyberpower678 fixed now [19:43:27] Cyberpower678 just do mysql --defaults-file=replica.cnf -h tools-db [19:43:39] I will fix sql tool very soon [19:44:09] Cyberpower678: thanks for being our guinea pig on tool deletion :) [19:44:28] Ryan_Lane, you're welcome. :p [19:44:48] Cyberpower678 everything should go now and I go have like 6 beers now [19:44:51] * work now [19:45:47] Warning: There are 2 users waiting for shell, displaying last 2: Tristessa de St Ange (waiting 94 minutes) Profoss (waiting 26 minutes) [19:46:11] addshore ^ [19:46:12] petan, xtools database is missing. [19:46:23] in tools-db [19:46:23] oh yes I am wondering how this works now [19:46:34] I think that new tools dont have it by default, Coren? [19:46:41] none of new tools have it [19:46:50] you can create new db like __dbname [19:46:50] Ok. [19:46:56] Right, no more default DB [19:47:01] user is that long shit :P [19:47:10] instead of cute name like xtools :P [19:47:27] db replica brought some disadvantages like ugly names [19:47:28] Most tools didn't use it, and couldn't create additional ones when they needed it. [19:47:43] petan: Unavoidable because of some mysql limitations on usernames. [19:47:51] ok [19:48:07] anyway, Cyberpower678 you know how to create dbś? [19:48:56] no [19:49:05] I haven't learned that yet. [19:49:11] you connect to db by doing something like sql local OR what I sent before [19:49:31] then you type create database p48358730291690573246813765835736425432__mycooldb [19:49:34] new db gives me a syntax error. [19:49:43] semicolon too [19:49:54] create database p436547573453048132054035834613405__blah; [19:50:20] how cute [19:50:21] :3 [19:50:26] :) [19:50:33] let me know if it didnt work [19:51:23] ERROR 1044 (42000): Access denied for user 'p50380g50570'@'%' to database 'xtools' [19:52:07] yes because there is no such database [19:52:43] I used create database xtools; [19:53:23] you need to prefix it with that p username [19:53:35] do cat replica.my.cnf [19:53:35] you will see it [19:53:35] What why? [19:53:35] it is ugly [19:53:42] That's just wrong. [19:53:42] you will see the username there [19:53:53] every db needs to be prefixed with username [19:54:05] why is it wrong [19:54:25] this ensures that every user can create as many unique names as they want [19:56:52] Cyberpower678: Like: create database p50380g50570__xtools; [19:57:52] Coren, already done. [19:57:56] Thanks [19:58:07] Now the migration can begin. [19:58:17] Coren have a look in -operations for a bit plz :P [19:59:20] Warning: There are 2 users waiting for shell, displaying last 2: Tristessa de St Ange (waiting 107 minutes) Profoss (waiting 39 minutes) [20:02:07] Cyberpower678 in a minute new sql will be installed so that you will be able to actually just type sql local to connect to local database without having to type these parameters [20:02:21] cool. [20:02:52] I think it is already on login [20:02:58] Cyberpower678 does sql local work now? [20:03:34] yes. [20:03:40] cool [20:11:22] New patchset: Petrb; "tool for removing of existing projects" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/62236 [20:11:22] New patchset: Petrb; "tarballing the produced folder so that we save some space" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/62238 [20:11:23] New patchset: Petrb; "fixed some bug" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/62239 [20:11:23] New patchset: Petrb; "inserted sql tool" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/66292 [20:11:23] New patchset: Petrb; "updated rmtool to latest version" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/66293 [20:11:43] um [20:11:50] that is not what i wanted [20:12:50] Warning: There are 2 users waiting for shell, displaying last 2: Tristessa de St Ange (waiting 121 minutes) Profoss (waiting 53 minutes) [20:13:05] Change abandoned: Petrb; "(no reason)" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/66293 [20:13:27] Change abandoned: Petrb; "(no reason)" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/66292 [20:13:37] Change abandoned: Petrb; "(no reason)" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/62239 [20:13:47] Change abandoned: Petrb; "(no reason)" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/62238 [20:17:14] New patchset: Petrb; "inserted rmtool to new folder" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/66294 [20:26:24] Warning: There are 3 users waiting for shell, displaying last 3: Tristessa de St Ange (waiting 134 minutes) Profoss (waiting 66 minutes) Manybubbles (waiting 2 minutes) [20:38:19] manybubbles: You are *happy camper*! We are *squirting* shell access for *parties*. [20:39:35] Coren: no *parties*, please. Those little "go" men always blew up my ships. [20:42:18] Warning: There is 1 user waiting for access to tools project: AzaToth (waiting 0 minutes) [20:42:47] why is that a "warning"? [20:44:52] AzaToth: It's to warn the lazy admins that they have work to do. Or else. :-) [20:45:12] hehe [20:46:04] AzaToth: I suppose it should only start crying "Warning" when the delay gets too long. I've added you, BTW [20:46:31] manybubbles: But the GO GO are so much *fun* for *dancing*! [20:46:32] Coren: Fatal! [20:47:51] http://api.openstack.org/api-ref.html#ext-os-used-limits [20:48:00] ^^ time to push in a change for that :) [20:48:30] Coren: I suppose I should play the game again so I can refresh my Orz knowledge. It has been too long. [20:48:57] *squirting* shell access was perfect, btw [20:49:06] manybubbles: http://sc2.sourceforge.net/ [20:49:54] manybubbles: I just sent that to a friend of mine yesterday. It is an important piece of cultural literacy, if you ask me. [20:50:02] Open source FTW. I recommend the highres and remix extension packs. :-) [20:50:16] Talking to yourself? :-) [20:50:30] i suppose so! [20:50:37] That game ranks in my top-5 of mostest best games of all times. [20:50:49] It has been years since I really used irc [20:51:07] yup - SC2 is certainly in my top 5 as well. [20:51:58] bah. stupid nova command can show quota usage, but only shows rate-limits [20:52:02] Along with Portal, UW2, U7, and Nethack. [20:52:43] it makes the right request, but only shows a portion of the data :( [20:53:17] Ryan_Lane: If only it were the /right/ portion. :-) [20:53:30] yep. I don't give much of a shit about rate limits [20:53:41] Has any progress been made with the freenode connection limit yet? [20:53:42] I do care about project quotas, thoguh [20:54:05] Only 2 out of 3 of my bots are able to connect it seems. [20:54:21] legoktm: I'm testing a solution as we speak. :-) [20:55:03] Coren: the sad thing is that means every project that wants irc connections needs a public ip :( [20:55:22] Coren: you are awesome :D [20:55:23] Coren: can you make the puppetization of this generic so that it can just be included via puppet? [20:55:31] Ryan_Lane: I think that once tool labs gets off the NAT, the standard I-line should suffice. [20:55:40] I don't think we can get off of NAT [20:55:54] I mean, off the generic NAT IP [20:56:05] ah. right [20:56:11] I mean for other projects than tools ;) [20:56:36] a short guide on how to make it work on wikitech would be nice as well [20:57:01] we need admin docs for stuff anyway [20:57:21] Coren, on toolserver there is a database that tells where you can find which wiki on what host. Is there something like that here? [20:57:32] Coren: seems someone forgot to update that site since 98 [20:57:44] AzaToth: What site? [20:57:48] It's very vital for several key programs. [20:57:52] Cyberpower678: Soon™ [20:58:08] Coren: sc2 [20:58:28] Coren, how soon? [20:58:31] :p [20:58:39] AzaToth: Last news on it dates from Wednesday, February 20 2013 [20:58:50] Cyberpower678: When It's Ready™ [20:58:54] Coren: I was referring to the layout :-P [20:59:01] I blody got headake from it [20:59:04] AzaToth: Well, it's a retro game. :-) [20:59:11] Very helpful. :/ [20:59:13] Coren: use ℠ [20:59:45] I've never actually seen ℠ used [21:00:01] Cyberpower678: It's on my to-do list, but I don't want to commit to a date unless I'm sure. [21:00:10] Coren, will it be laid out like: [21:00:15] +-----------------+------+------------+------------------------+-------+---------+-----------+--------------+--------------+---------------+--------+-------------+ [21:00:15] | dbname | lang | family | domain | size | is_meta | is_closed | is_multilang | is_sensitive | root_category | server | script_path | [21:00:15] +-----------------+------+------------+------------------------+-------+---------+-----------+--------------+--------------+---------------+--------+-------------+ [21:00:15] | aawikibooks_p | aa | wikibooks | NULL | 3 | 0 | 1 | 0 | 0 | NULL | 3 | /w/ | [21:00:32] Blah! Don't paste on-channel like this. [21:00:37] That didn't come out well. :/ [21:00:50] yea, spam on #wikimedia-operations instead :-P [21:00:51] Cyberpower678: But yeah, I'll model on the TS's to ease things. [21:01:06] cool. [21:01:21] Give me the host and database name as soon as it's ready. [21:01:32] Coren: so now that you've *squirted* me shell access, how do I use it? Right now I'm just going through tons of pages on the office wiki so I'm not sure when I'll have a chance to use it. [21:02:05] !tooldocs [21:02:06] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help [21:02:10] manybubbles: ^^ [21:04:16] awe, you have a copy and pastebot, how cute. thanks so much [21:10:07] Coren: is there a differences between "Tool" and "Tools"? [21:10:36] AzaToth: Not sure I get the context of your question? "tools" is the name of the project. [21:10:42] Tools™ [21:10:56] the help page says "Tool Labs is a special part of Labs: It consists of two projects named "Tools" and "Bots"" [21:11:13] is Nova Resource:Tools Tool or Tools:Tool? [21:11:23] erl Tool:Tools [21:12:02] if it's the former, then where is Bots? [21:12:08] bots.wmflabs.org [21:12:22] https://wikitech.wikimedia.org/wiki/Nova_Resource:Bots [21:12:22] I meant on the help page [21:12:30] I like the tools-experimental idea [21:12:33] if it's the latter, then there is Tool? [21:12:42] where* [21:12:46] Ryan_Lane: So do I. Now get to work on the "rename project" feature. :-) [21:13:02] AzaToth: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools ? [21:13:02] Warning: There is 1 user waiting for access to tools project: Tristessa de St Ange (waiting 0 minutes) [21:13:13] legoktm: is that Tool or Tools? [21:13:17] There is no "Tool" that I know of, AzaToth. There is only Zuul^H^H^H^H Tools. :-) [21:13:19] Tools. [21:13:26] I think renaming will never be an option [21:13:52] unless we put a *lot* of effort into it [21:13:57] Coren: So the "tool" on the help page means Tools? [21:14:15] Coren: is Bots a subproject of Tools then? [21:14:46] Conceptually. Strictly speaking, it's the predecessor, and an independent project. [21:14:47] Coren: it's the phrase "Tool Labs is a special part of Labs: It consists of two projects named "Tools" and "Bots"." that has my brain grind to a halt [21:15:40] tool labs is a concept [21:15:48] ... [21:15:54] * AzaToth hates concepts [21:15:56] tools and bots are two projects in labs [21:16:05] "Tool Labs" is the service. "tools" is the project used to implement it. "bots" is the community-run predecessor that is going to become "tools-experimental". If you're not confused yet, there is also "webtools" which is deprecated. :-) [21:16:06] they are always soo far away from reality [21:16:08] those two together are "Tool Labs" [21:16:29] Coren: heh [21:17:19] Coren: btw, when I did the signup before, it never added it to watchlist [21:20:31] "It" being? [21:23:21] Coren: the access form thingi [21:23:43] Oh, is "watchlist edited pages" on in your preferences? :-) [21:23:59] hmm [21:24:01] IIRC, it's off by default. [21:24:15] why? [21:26:31] Warning: There is 1 user waiting for shell: Chess64 (waiting 10 minutes) [21:26:32] Warning: There is 1 user waiting for access to tools project: Tristessa de St Ange (waiting 13 minutes) [21:26:36] Cyberpower678, Coren: I tried to find a way to generate that data on deployment-bastion (where you can use "mwscript eval.php --wiki=dbname" to easily get config vars), unfortunately it would have to be set up for all projects before it would work the way I wanted :( [21:27:37] Krenair: You mean stow config out of LocalSettings into the DB? [21:28:08] Krenair: Can't we get all that info from the API? [21:28:33] certain vars from it, yes Coren [21:28:50] yes, but this wouldn't require making loads of api calls legoktm [21:29:03] One API call per wiki isn't that much :P [21:29:09] Krenair: This is a worthwhile endeavor, and I approve of it. [21:29:53] I don't think you could do it in one call, legoktm [21:29:56] <^demon> Redoing config out of LocalSettings? [21:30:00] <^demon> Won't go in DB [21:30:04] <^demon> We'll use CDB files. [21:30:06] <^demon> Way faster. [21:30:11] <^demon> Long-term plan. [21:30:14] <^demon> There's a bug [21:30:25] ^demon: we're talking about the wiki database that the toolserver has and copying it to labs [21:30:29] IIRC it was echo $wgLanguageCode; echo $wgServer; echo SiteStats::pages(); and echo $wgScriptPath; [21:30:43] What about namespaces? [21:31:04] The dbname, domain, namespaceid and namespacetext data legoktm? [21:31:11] The identd, she is work. [21:31:38] Krenair: Yeah [21:31:41] Ryan_Lane: https://gerrit.wikimedia.org/r/#/c/66296/ [21:31:49] !b 48625 [21:31:49] https://bugzilla.wikimedia.org/48625 [21:31:52] !b 48626 [21:31:53] https://bugzilla.wikimedia.org/48626 [21:32:08] Ryan_Lane: Can you Verify it? Looks like Jenkins is stuck hours behind. [21:32:13] Coren: can you make ident a module instead? [21:32:21] and include the module [21:32:27] I also tried the same mwscript tricks with that, legoktm [21:32:30] Same issue of course [21:32:32] Hm [21:32:36] that way it can be more easily reused by other projects [21:32:50] Ryan_Lane: For just the one package{} ? I suppose. [21:33:16] for package + config of one that will likely be reused, yeah ;) [21:33:26] Heh. No worries. [21:33:44] deployment-bastion doesn't have all the wikis set up on it [21:34:36] looks like tim landscheidt and liangent tried similar stuff [21:35:00] actually I might try liangent's [21:40:01] Warning: There is 1 user waiting for shell: Chess64 (waiting 24 minutes) [21:40:02] Warning: There is 1 user waiting for access to tools project: Tristessa de St Ange (waiting 27 minutes) [21:41:28] Ryan_Lane: Made into a (trivial) patch. [21:46:44] I wonder what liangent's mediawiki-config/wmf-config/wgConf.php is... [21:48:34] Why does php's console hate the delete key so much!? [21:49:08] It insists that it's a tilde (~) [21:53:31] Warning: There is 1 user waiting for shell: Chess64 (waiting 37 minutes) [21:53:32] Warning: There is 1 user waiting for access to tools project: Tristessa de St Ange (waiting 40 minutes) [22:06:56] Warning: There is 1 user waiting for shell: Chess64 (waiting 50 minutes) [22:06:57] Warning: There is 1 user waiting for access to tools project: Tristessa de St Ange (waiting 54 minutes) [22:20:20] Warning: There is 1 user waiting for shell: Chess64 (waiting 64 minutes) [22:20:21] Warning: There is 1 user waiting for access to tools project: Tristessa de St Ange (waiting 67 minutes) [22:33:45] Warning: There is 1 user waiting for shell: Chess64 (waiting 77 minutes) [22:33:46] Warning: There is 1 user waiting for access to tools project: Tristessa de St Ange (waiting 80 minutes) [22:47:05] Warning: There is 1 user waiting for shell: Chess64 (waiting 91 minutes) [22:47:06] Warning: There is 1 user waiting for access to tools project: Tristessa de St Ange (waiting 94 minutes) [23:00:25] Warning: There is 1 user waiting for shell: Chess64 (waiting 104 minutes) [23:00:26] Warning: There is 1 user waiting for access to tools project: Tristessa de St Ange (waiting 107 minutes) [23:13:46] Warning: There is 1 user waiting for shell: Chess64 (waiting 117 minutes) [23:13:47] Warning: There is 1 user waiting for access to tools project: Tristessa de St Ange (waiting 120 minutes) [23:27:06] Warning: There is 1 user waiting for shell: Chess64 (waiting 131 minutes) [23:27:07] Warning: There is 1 user waiting for access to tools project: Tristessa de St Ange (waiting 134 minutes) [23:40:26] Warning: There is 1 user waiting for shell: Chess64 (waiting 144 minutes) [23:40:27] Warning: There is 1 user waiting for access to tools project: Tristessa de St Ange (waiting 147 minutes) [23:53:46] Warning: There is 1 user waiting for shell: Chess64 (waiting 157 minutes) [23:53:47] Warning: There is 1 user waiting for access to tools project: Tristessa de St Ange (waiting 160 minutes)