[02:12:44] RECOVERY Free ram is now: OK on deployment-sql deployment-sql output: OK: 20% free memory [02:16:45] hmmm [02:19:44] ? [02:33:06] Ryan_Lane: Hi [02:33:14] howdy [02:33:31] Ryan_Lane: I understood from a fellow cvn-staffer that there has been some talks about moving CVN services to Labs. [02:33:37] yeah [02:33:51] they'd like a shared account, but I'm against shared accounts [02:34:05] I've been sent a convo [02:34:06] another solution is now going to be a web console for the bots [02:34:19] which I prefer over shared accounts [02:34:30] There's some missing intel in that convo, misunderstandings and some wrong assumptions [02:34:35] oh? [02:35:28] So we've got 20 bots. Written in C#/Visual studio (not our choice, was originally the case and was maintained through the years). Compiled with xbuild on Linux and running with 'mono'. [02:35:39] brb [02:35:44] PROBLEM Free ram is now: WARNING on deployment-sql deployment-sql output: Warning: 19% free memory [02:35:51] 3 developers that would need shell access (including me), and about a dozen staffers that only need access to the web control panel [02:36:07] so this could be a project-group like testswarm, not a shared account. [02:36:29] we have a php control panel already which we could use (requires apache to be running on the same server as the bots) [02:36:55] home-brew script, currently hosted on toolserver as well. [02:37:58] * Firebolt lurk [02:38:04] Ryan is brb [02:40:44] RECOVERY Free ram is now: OK on deployment-sql deployment-sql output: OK: 24% free memory [02:41:02] on phone [02:41:11] ok, i'll type a little more :) [02:41:21] although, not sure whether it should be a project-group like 'testswarm' is. Since this is probably something for Tool Labs, not Devwiki Labs, right ? I understood ClueNet is already on Labs, I didn't' know Tool Labs was already on ? [02:55:21] I can't login at beta.wmflabs, I requested a password reminder but I'm not getting anything. [02:55:26] (checked spambox) [02:55:51] back [02:55:56] lemme read backscroll [02:56:49] Krinkle: mail isn't working in labs yet [02:56:56] we need to set up a new relay for it [02:57:06] in case someone decides to spam through labs [02:57:21] we wouldn't want labs getting all of our sites blacklisted [02:57:25] ok [02:57:37] my views on how labs will work are shifting over time [02:58:35] I think if people working on this are technically proficient enough to forward ssh agents, then let's set this up as a project, and run it like all the others [02:59:03] it could run in the bots project, too [02:59:11] majority (if not all) toolserver accounts know how to use ssh and sftp. [02:59:24] with agent forwarding? [02:59:28] no, not that. [02:59:34] right [02:59:43] for this, people would still connect through bastion [02:59:44] WHy is that required ? I don't have that setup either [03:00:04] how are you connecting to instances behind bastion, then? [03:00:29] @search proxy [03:00:29] No results found! :| [03:00:34] @search proxy- [03:00:34] No results found! :| [03:00:38] @search -proxy [03:00:39] No results found! :| [03:00:40] heh [03:00:44] until a few weeks ago I could used ssh -something [03:00:50] is that no longer supported (from bastion) [03:00:59] using a password? [03:01:04] I guess [03:01:11] yeah, that was a problem, passwords were never supposed to work [03:01:18] !access [03:01:18] https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [03:01:30] https://labsconsole.wikimedia.org/wiki/Access#Using_agent_forwarding [03:01:34] or... [03:01:43] https://labsconsole.wikimedia.org/wiki/Access#Using_ProxyCommand_ssh_option [03:01:59] the latter works without forwarding, but requires people to edit their config files [03:02:26] we don't have enough public IP addresses right now, so people can't directly connect to all instances [03:03:22] ok. cvn also has a php API though, so we do need a public subdomain [03:03:34] used in gadgets. to get data from bots [03:03:46] yeah, that's fine [03:03:55] okay :) [03:04:15] if we put this in the bots project, it can use the apache server there too [03:04:19] and it has a public IP [03:04:24] bots.wmflabs.org [03:04:32] hm.. shared access ? [03:04:41] not totally [03:04:56] each set of bots can get its own instance [03:05:14] ideally in the future we'll have this project work with no root access [03:05:33] and have another one, for development of the bots that still allows root [03:07:15] ok [03:07:38] Ryan_Lane: I see Ive been granted access on deployment-web, trying to run changePassword for my account. [03:07:48] krinkle@deployment-web:/var/www/commons_wiki/w/maintenance$ php changePassword.php --user [03:07:52] DB connection error: Unknown database 'commons_wiki' (deployment-sql) [03:07:56] heh [03:08:06] I have no idea how they set stuff up there [03:08:11] k [03:10:21] I'd really like some kind of API driven service for bots [03:10:52] where we can control where bots run, and which version of the bot software they run [03:11:28] so, we could do something like "Manage bots", and have an interface where you can add a new bot, remove a bot, move a bot to another instance, change the version that is running, restart the bot, etc [03:11:52] then we could have a service that sits on all of the instances, that can do these actions [03:12:46] manage it via a queue. bot-manager service on instanceA can stop a bot, add a message to the queue that it finished, and instanceB can start it. <- move operation [03:13:16] it could be managed per-project, and via a botadmin role [03:13:34] then it could just be another interface on labsconsole [03:14:01] I've got a `$ ps -f -u ` based hack on the Toolserver as control panel [03:14:31] but it's with a 1 minute delay because toolserver has separate webserver/botserver. so I write command to SQL and cronjob pop()'s the rows [03:14:50] yeah, using mysql as a queue works too [03:15:09] I'm thinking of adding a centralized queue for all projects and instances [03:15:25] so that everyone has access to a queuing servie [03:15:28] *service [03:15:30] http://i.imgur.com/NSwF4.png [03:16:05] it's a one-weekend hack from march-2010 [03:16:09] neat [03:16:17] yeah, that's a really good start [03:16:28] we have a *lot* more control in labs [03:16:33] so, we can make a pretty fancy service [03:16:35] log is the std_out file [03:17:00] is this all php? [03:17:03] yep [03:17:06] * Ryan_Lane nods [03:17:08] php and an .ini file [03:17:25] I do basically everything in python. heh [03:17:40] it avoids needing to install a web server [03:17:55] python can run a wsgi server via eventlet [03:17:56] for example: http://i.imgur.com/7fIBf.png [03:18:15] or, just launch as a daemon, and wait on a queue [03:18:31] yeah, watching a file [03:18:38] does that write messages to irc? [03:18:47] the control panel ? no [03:18:55] the irc bots started from there do [03:19:08] what's the last image? [03:19:12] ini file? [03:19:17] yes, a fragment of it [03:19:19] cool [03:19:27] yeah, this is exactly what I wanted [03:19:29] well, close anyway [03:19:38] but more distributed [03:19:40] statusdetectpath is used to get the PID and uptime from `ps` [03:19:48] I'd like to have a bunch of "bots" instances [03:20:00] a 1-minute cronjob on the botserver dumps output from `ps` in a txt file [03:20:02] where bots can run on any of the instances, and can be moved between them [03:20:04] so that php can read it :( [03:20:06] heh [03:20:18] that's rough [03:20:21] Ah, wrong. No it's not `ps` [03:20:22] it [03:20:29] it's `stat` [03:20:32] `qstat [03:20:45] a daemon with a queue would handle this quicker [03:20:47] Bots are stated through cronsub/SGE so that they're started when they go off for whatever reason [03:21:01] a daemon could also do health checks [03:21:05] so they are started on one of the toolserver servers that have resources available at time of start [03:21:14] ah. I see [03:21:35] toolservers's SGE/cronsub/qsub distributes them automatically at "random" [03:21:40] ah [03:21:53] it's something the toolserver admins wrote to do so? [03:22:34] All I know is that when I want to do a very slow query (e.g. something like SpecialWantedFiles's query on en-wiki), I write it in a .sql file and queue with qsub [03:22:34] I'm wondering how it determines which one is best to run stuff on [03:22:43] then it's executed within 10-20 seconds on one of the servers [03:22:47] * Ryan_Lane nods [03:22:49] interesting [03:23:46] and from crontab I have for example: [03:23:52] hm. it would really be nice to have projects for each bot group, where there's a centralized project for central services [03:23:53] 0,5,10,15,20,25,30,35,40,45,50,55 * * * * cronsub -l -s clogger $HOME/bots/clogger-start.sh [03:24:22] cronsub will then start it, but not if a prices by the name 'clogger' is runing already [03:24:27] proces* [03:24:40] (yeah the ugly 0,5,10 is thanks to Solaris) [03:24:45] * Ryan_Lane nods [03:25:00] that's an interesting, yet kind of clunky way to launch services [03:25:44] well, we drive with the tools we've been given - add the stress of vandal fighting and limited time/resources devoted to it from both ts-root and ts-users like me, and you get this. [03:25:51] yep [03:25:59] hey, it works. I'm not knocking it [03:26:04] :) [03:26:06] we don't have anything like this at all yet [03:26:53] oh, and not to forget the bi-monthly updates when everything breaks and we have to go search through maintenance logs to find out what was changed and have to fix everything :P [03:27:00] :D [03:27:06] well, that's less of a problem here [03:27:20] like this week mono was updated to 2.x unannounced. all cvnbots broke [03:27:24] heh [03:27:28] I heard about that [03:27:35] so that's why you spotted mono.old in the .ini file screenshot [03:27:39] we use LTS ubuntu releases [03:27:52] so, you get what you get for about 2-4 years [03:28:20] and you can upgrade on your own time [03:28:35] this is also a reason I'd like to have multiple bot instances [03:28:37] Ryan_Lane: So before I close up at 3am here. I was wondering to what extend WMF is in contact with WMDE. I mean it would seem like a waste if Toolserver would be, say, doing upgrading hardware upgrades, right ? [03:29:03] well, we have a ways to go to replace toolserver [03:29:12] I'm not even actively trying to do so [03:29:23] I'm just trying to set up cooler things, so that people naturally migrate [03:29:23] right [03:29:45] part of this is the community setting up what they want, as well [03:30:23] biggest things missing right now is probably: live wmf-db replica's, user databases (hm.. already exist), and automatic backups of user data. [03:30:27] I'd really recommend the toolserver people, at some point, starting to move things to labs themselves [03:30:33] meaning the admins [03:30:54] yeah, user databases and backups are incoming [03:30:59] kind of [03:31:11] the database replication is planned, but is likely further out [03:31:26] yeah, although I'm not sure if admins can do that. Right now ts stuff is good but not organized well. Everybody has it's own account and can do stuff. It's hard to migrate. [03:31:39] yeah [03:31:54] labs isn't really far enough along yet, anyway [03:32:01] still too many SPOF in the architecture, IMO [03:32:07] btw, you wanted to know resource usage of cvn-bots ? How can I measure that ? I'm in shell now [03:32:19] I dunno [03:32:29] I wanted to know how much storage space it would need [03:32:40] the rest I don't care as much about [03:32:47] storage is harder to expand than memory/CPU [03:32:53] ok. [03:33:09] hm.. looking for a recursive folder size measure command in unix.. [03:33:19] we have a *lot* of compute nodes slated for this [03:33:24] 16 of the cisco servers [03:34:15] I do remember that 2-3 years ago CVN was unofficially not allowed on TS due to bots being too resource intensive. So they were moved to Alex Zariv (WMF)'s private server. Bots have improved though and most are on Toolserver again. [03:34:33] * Ryan_Lane nods [03:34:45] we should probably have a team that tries to improve bot performance [03:35:49] thankfully, if we architect things correctly, we an isolate bots that perform terribly [03:36:11] by giving them their own instance [03:36:32] Toolserver also offers an svn repo per group-project. CVN is using that too. Any recommendations for migrating that ? Into gerrit ? Or a local git/svn server on the instance ? [03:36:40] gerrit, yeah [03:36:59] we can probably do a repo per openstack project [03:37:34] gerrit has support for LDAP groups, so we can define gerrit permissions on the repo based on the openstack project (which is an LDAP group) [03:38:08] I need to add gerrit control to OpenStackManager [03:38:18] so I can automatically create a repo on project creation [03:38:25] One good argument of labs vs toolserver is their stubborn love of Solaris. [03:38:34] fuck Solaris [03:38:38] :) [03:38:47] we use what the production cluster uses [03:38:53] which, right now, is Ubuntu Lucid [03:38:54] Well, Solaris has cool things. But Oracle is Teh Evulz. [03:39:12] we'll be adding Precise in when it's released [03:39:19] likely shortly before its released [03:39:40] looks like we've got 650M on /home/projects/cvn at Toolserver [03:39:49] oh. that's small [03:39:50] My production servers are still on lucid. Rock solid. [03:39:53] not including mysql [03:39:58] ah [03:40:03] but that's fairly small [03:40:10] yeah. that's not bad at all [03:40:14] we can handle that right now [03:40:37] we really need hardware SQL servers [03:40:53] maybe next week I'll concentrate on adding virt1 as a compute node, and adding the SQL servers [03:41:32] hype ron is working on a database as a service application [03:42:05] resources used: http://pastie.org/3192848 [03:42:08] until then, we'll have to come up with some simpler ways of managing the databases [03:42:17] (that's half actually, since we run bots on two servers) [03:42:23] * Ryan_Lane nods [03:42:35] that's not bad at all [03:42:43] so that times 2 [03:42:54] yeah [03:43:10] not sure if this means much to you as it doesn't say the server specs (or does it) [03:43:19] So, Ryan, any particular reason why the labs aren't v6? [03:43:23] about 400MB memory, total [03:43:34] Coren: because we haven't enabled it yet [03:43:44] Tsk. :-) [03:43:48] also, because I didn't start off with IPv6, now I need to figure out how to enable it [03:44:09] apparently, they expect you to create ipv4 network and ipv6 network together [03:44:10] If you need help there, I'm all yours. v6 is my battle horse. :-) [03:44:33] I already added the v4 network, so now I need to modify the database directly, likely [03:44:58] I really need to reach out to my openstack connections for more direct help :) [03:45:24] Krinkle: I'm not terribly worried about memory or CPU usage [03:45:31] ok [03:45:33] Krinkle: since your instances won't directly affect others [03:45:43] you are limited to whatever instance type is used [03:45:51] yay for virtualization [03:45:54] My cloak unhelpfully hides the fact that even my IRC is v6. :-) [03:45:56] so what's the limit ? [03:45:58] :P [03:46:01] Coren: heh [03:46:11] Krinkle: well, some instance types can be 32GB of ram [03:46:19] not that I ever see that being used [03:46:28] how much does the physical server have ? [03:46:32] 48 [03:46:37] the ciscos have more, I think [03:46:42] maybe 64 [03:46:53] *cloink* [03:46:58] that's.. incredible [03:47:04] did I mention we are going to have 16 of those? :) [03:47:08] hehe [03:47:15] plus a few more for SQL [03:47:35] cloink? [03:48:33] the throat sound, not sure how to write that in English :P [03:48:49] like swallowing in cartoons when something amazing is pointed out [03:49:02] am I making any sense ? [03:49:06] yes [03:49:10] yeah, we're going to have a lot of resources at our disposal. heh [03:49:40] now we just need to write the tools to manage them properly ;) [03:50:03] * Firebolt wishes he could use a server like that for minecraft [03:54:20] k, thanks Ryan_Lane , gonna go now. we'll talk later upcoming week. [03:54:31] ok. sounds good [03:54:32] * Ryan_Lane waves [04:17:40] * jeremyb pokes the wind [05:05:44] PROBLEM Free ram is now: WARNING on deployment-sql deployment-sql output: Warning: 19% free memory [05:38:54] PROBLEM Free ram is now: WARNING on bots-2 bots-2 output: Warning: 19% free memory [05:45:44] RECOVERY Free ram is now: OK on deployment-sql deployment-sql output: OK: 20% free memory [06:38:44] PROBLEM Free ram is now: WARNING on deployment-sql deployment-sql output: Warning: 19% free memory [08:33:06] Anyone here able and knowhow to run a maintenance script on beta-labs ? [08:33:14] umm [08:33:18] don't think so [08:33:19] email hasn't been set up and I apparnetly' can't login [08:33:23] need a pass reset on my account [08:33:41] yeah. email is not yet working in labs [08:33:47] we need a relay specific for labs [08:33:55] so we don't get everything blocked if someone spams from labs [08:34:06] you told me yesterday ;-) [08:34:12] (yesterday for me anyway0 [08:34:43] changePassword.php should work though [08:34:49] can't get it to work [08:35:47] orly ? Just got in. [08:35:51] nvm :) [08:39:33] I think it's wrapped with het deploy stuff [08:39:40] there's likely a wrapper script you need to run [08:40:48] yep, but couldn't find anything like that [08:41:04] the drive structure isnt' documented afaik [08:41:08] i.e. what is stored where [08:41:54] web root looks non-standard, I was looking for /srv/ but that didn't exist. Did find separate mediawiki checkouts in a folder somewhere, seemed to match the active wikis [08:42:21] that's either old and unused or labs isn't using SiteConf / shared source yet [08:51:48] hm [08:51:55] well, it shouldn't be in /srv anyway [08:51:58] should be under /usr [08:52:34] live and prototype use /srv/ ? [08:54:56] production does not [08:55:01] prototype did [08:55:09] production uses a directory under /usr [08:55:11] ok [08:55:22] so where would mw checkout and maintenance be etc. ? [08:56:13] under that directory [08:56:41] k,will look at it tonight, no ssh key where I am now [08:56:45] thx [08:56:47] it's here: /usr/local/apache/common [08:57:38] Ah, I've seen that on wmf logs etc. yeah, I remember :) [09:37:41] Ryan_Lane: hi [09:37:47] howdy [09:38:10] is it possible to run non console application in a lab [09:38:58] what do you mean? [09:39:10] you mean a graphical one? [09:39:14] yep [09:39:19] like an IDE [09:39:28] why would you run an IDE on labs? [09:39:31] or a JAVA GUI config [09:39:35] rather than on your local system? [09:39:40] to debug [09:40:37] I was thinking of writing a config util for search - but it would be gui based [09:40:52] and there is little point in doing that if we can't use it [09:41:38] we wouldn't use a GUI one anyway [09:41:45] unless it was completely web based [09:41:55] in which case you don't need a GUI [09:41:58] I thought as much [09:42:32] it's possible to have a GUI if you can forward X11 somehow [09:42:47] that's probably hard through a bastion host, though it may work with proxycommand [09:42:48] I saw some docs on that [09:43:01] but it's going to be incredibly laggy [09:43:14] GUI apps across the WAN usually suck [09:43:41] ok [09:44:12] sell so far I don't need it [09:44:19] so so far I don't need it [09:44:44] * Ryan_Lane nods [09:45:18] it should be possible to remotely connect to a debugger using a local IDE [09:45:57] there are about 4 diffrent ways of working with eclipse on a remote systems (an they all suck) [09:46:14] 4 that I know of [09:46:30] for connecting to a debugger? [09:48:04] more than just a debugger [09:48:21] also access to the file system and start stop stuff [09:53:26] there is a remote file system explorer [10:58:52] RECOVERY Free ram is now: OK on bots-2 bots-2 output: OK: 20% free memory [11:06:57] ACKNOWLEDGEMENT Disk Space is now: CRITICAL on puppet-lucid puppet-lucid output: DISK CRITICAL - free space: / 34 MB (2% inode=35%): [11:11:52] PROBLEM Free ram is now: WARNING on bots-2 bots-2 output: Warning: 19% free memory [11:46:52] RECOVERY Free ram is now: OK on bots-2 bots-2 output: OK: 20% free memory [12:15:57] ACKNOWLEDGEMENT Free ram is now: WARNING on bots-2 bots-2 output: Warning: 19% free memory [12:17:55] Oh, this is going to be funny and annoying [12:18:20] Bots-2 is hovering at ~20% free memory [12:51:46] !log bots installed libhtml-entities-numbered-perl [12:51:49] Logged the message, Master [12:51:54] !log bots installed liburi-perl [12:51:55] Logged the message, Master [12:52:34] !log bots installed libxml-perl [12:52:35] Logged the message, Master [12:53:50] !log bots installed libcarp-always-perl [12:53:51] Logged the message, Master [13:04:14] Johnduhart, Can you please help open the sa-wikisource beta page for the community? [13:10:19] I am getting a headache on bots-3 [13:11:53] Coren, how heavy is your bot on bots-3? [13:22:09] Coren, forget it, swapping bots did not solve my problem [13:23:32] Raama: you need to open a wiki? [13:23:42] beta wiki lab [13:23:45] ok [13:23:55] for sa.wikisource.org [13:24:52] how long will it take? [13:33:02] done [13:33:30] Raama: ^ [13:34:05] Beetstra, Coren I created a way for users of lab to configure nagios [13:34:19] unfortunatelly ldap doesn't work yet [13:34:28] so if you wanted I can create an account for you [13:39:06] ya [13:39:13] sure petan... [13:39:59] Anyways thanks for the help :) [13:42:14] :o [13:43:03] Can someone explain: [13:43:17] If I log in on en.wikipedia, and then switch to meta, then I am not logged in on meta [13:43:30] if I log in on meta.wikimedia, and then switch to en.wikipedia .. I am logged in .. [13:43:43] (I don't see the 'log in on all projects check-box anymore) [13:45:28] it should have log you in on all projects [13:45:43] Beetstra: are you talking about prod or beta? [13:45:54] the real thing [13:46:09] I think it should work for all sites [13:46:16] Yes, that is also what I expected [13:46:37] But I logged my bot in on en.wikipedia, and it did not save on meta .. and I did not have a single clue why [13:46:53] cookies? [13:47:13] no, both fully enabled [13:47:22] And I tried it from my laptop .. and I have the same effect [13:47:23] no idea [13:47:48] log in as bot on en.wikipedia, switch to meta - logged out. Log in on meta as Beetstra, switch to en, logged in as Beetstra, not as COIBot [13:47:49] it doesn't work really well [13:47:52] no, indeed [13:56:56] Oh now it gets even funnier .. [13:57:02] 1) log in on en.wikipedia.org as COIBot [13:57:08] 2) switch to meta - you are logged out [13:57:12] 3) log in as Beetstra [13:57:19] 4) switch to en.wikipedia, you are Beetstra ... [13:57:26] 5) switch back to meta, log out [13:57:35] 6) switch to en.wikipedia, and you are COIBot .. [13:58:40] * Beetstra calls an exorcist [14:31:27] Beetstra: Gentle on ram, reasonable on bandwidth, and will occasionally bring the CPU to its knees crying for momma. [14:33:43] OK .. one of my bots is a beast what that regards, but it could run next to yours [14:34:50] But the problem was not the perl install, it is something with my bots that I don't see [14:40:12] Beetstra: should I enable memory check on bots-2 [14:40:18] I disabled to avoid spam [14:40:34]