[06:22:14] @requests [06:22:14] There are no shell requests waiting [12:51:39] Coren: around? :) [12:54:07] oh well, ping me when you are? :D [13:00:18] addshore: Ping [13:00:23] :D [13:00:47] If I wanted a tool that isnt addbot i.e. it was just my tool, would I need a tool called addshore also? [13:01:42] Well, you could call it whatever you wanted. :-) [13:03:50] 'addshore' it is then ;p [13:04:54] And so it is. *poof* [13:05:03] You now have a tool alterego. : -) [13:05:45] mwahah ;p [13:09:50] [bz] (ASSIGNED - created by: m.p.roppelt, priority: Unprioritized - blocker) [Bug 45078] fcron not working on bots-4 - https://bugzilla.wikimedia.org/show_bug.cgi?id=45078 [13:12:58] [bz] (RESOLVED - created by: Peter Bena, priority: High - major) [Bug 42578] bots.wmflabs.org/~petrb/logs/ doesn't work - https://bugzilla.wikimedia.org/show_bug.cgi?id=42578 [13:57:08] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/Help was modified, changed by Addshore link https://www.mediawiki.org/w/index.php?diff=676400 edit summary: [+116] /* Web services */ note about user web directories [13:57:47] addshore: Wait, Thou art confused. [13:57:51] O_o [13:58:04] art I? [13:58:23] I created an addshore tool; it has 'local-addshore' as username, and *its* home has a public_html. [13:59:01] You shouldn't be using your home; in fact, if that worked, then I goofed and should make it /not/ work. :-) [13:59:02] yeee, :P i.e. shell users themselves do not have a webdirectory, only tools [13:59:27] Exactly. [14:00:06] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/Help was modified, changed by Addshore link https://www.mediawiki.org/w/index.php?diff=676402 edit summary: [-60] /* Web services */ fix? [14:00:31] thats better? :P [14:01:22] Well, now it's true at least. :-) [14:01:52] I was trying to put across that you could have something such as your username but only if you create a tool nammed after you ;p [14:02:30] also Coren as /data/project/addshore is owned by local-addshore should I ned be able to edit as addshore as addshore is a member of local-addshore? [14:03:09] You should be, but your current shell might not know if you were logged in before the group was added. [14:04:16] bingo :) [14:04:36] Coren: thought of having an FAQ page for when people start trying to move to labs onmass? [14:04:51] addshore: Clearly required. [14:04:53] rather than just documentation at /help? [14:05:16] addshore: Also: OMGTHERE'SNOTENOUGHHOURSAUUUGH! :-) [14:05:20] I may go back and look at all of your answers to my questions and write them all down somewhere ;p [14:05:32] addshore: That would be immensely cool. [14:06:00] infact I'll just look through the logs of this channel for everytime someone pings you ;p [14:11:30] !logs [14:11:30] logs http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs [14:21:57] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/Help was modified, changed by Addshore link https://www.mediawiki.org/w/index.php?diff=676410 edit summary: [+61] /* How */ note about where you can submit from [14:28:25] Warning: There are 1 users waiting for shell, displaying last 1: Sumurai8 (waiting 0 minutes) [14:41:56] Warning: There are 1 users waiting for shell, displaying last 1: Sumurai8 (waiting 13 minutes) [14:55:31] Warning: There are 1 users waiting for shell, displaying last 1: Sumurai8 (waiting 27 minutes) [15:09:01] Warning: There are 1 users waiting for shell, displaying last 1: Sumurai8 (waiting 40 minutes) [16:03:42] Ooo. Uplink upgrade. [16:14:44] [bz] (RESOLVED - created by: Antoine "hashar" Musso, priority: Normal - enhancement) [Bug 44041] [OPS] adapt varnish mobile puppet class on beta - https://bugzilla.wikimedia.org/show_bug.cgi?id=44041 [16:16:29] legoktm, I'm looking at your logbot changes now. I presume you've tested the new bot? [16:18:28] [bz] (RESOLVED - created by: Antoine "hashar" Musso, priority: Normal - normal) [Bug 38996] [OPS] apache2 need manual start - https://bugzilla.wikimedia.org/show_bug.cgi?id=38996 [16:20:31] andrewbogott: Can you give https://gerrit.wikimedia.org/r/#/c/59969/2 a good once-over? It's getting darn close to blocking for me as this will be where I'm going to stuff the NFS automounter config until it's done testing. [16:22:32] sure, looking... [16:29:13] Coren, does the webserver class actually work? It doesn't seem to install e.g. apache [16:30:09] andrewbogott: It's not meant to yet; right now it's just a skeleton with dependencies. Once the module is in, I'll be able to add configuration elements gradually to it rather than in one big opaque and impenetable lump. [16:30:16] ok. [16:31:16] This looks good, although I continue to be unclear on what should be a role vs. what should be a module. Most of what you're defining look like roles to me, but I don't think we have good policies about that. [16:31:44] I made a stab and clarifying such things here: https://wikitech.wikimedia.org/wiki/Puppet_usage#Roles but I'm not sure anyone else is interested in the discussion :( [16:31:53] It's all a bunch of roles, but it's a cohesive "all the roles for a tools lab" module. That's what Ryan felt was the ideal. [16:32:17] (So that it autoloads rather than pollute site.pp) [16:33:19] Yeah, I agree that it makes sense to put in a module, I just wish we had policies that encompased both things [17:17:30] Warning: There are 1 users waiting for shell, displaying last 1: Betacommand (waiting 0 minutes) [17:35:53] * Coren grumbles. [17:36:37] [13:30:40] andrewbogott: Mind giving the changeset a review? :-) [17:59:26] Coren, sorry to make your patch subject to a dumb naming-convention discussion… if no agreement about that emerges today then we can just merge it as it is and figure on renaming later. [18:02:38] andrewbogott: Ima comment on the talk page to put my 2c in [18:02:57] Coren: I sent an email to the ops list just now, might be a better place to comment at the moment. [18:03:01] Hm… are you on the ops list? [18:03:58] I am. [19:21:32] hmm, what determines which puppet classes are shown in configure instance? [19:23:26] I think you manually add those with "manage puppet classes" [19:24:19] aha, thanks! [19:45:45] Ryan_Lane: heya. could we get the quota limit for the analytics project group raised? we need to migrate an XL instance to upgrade to precise, but attempts to create a new XL fail. [19:50:32] Warning: There are 1 users waiting for shell, displaying last 1: EBernhardson (WMF) (waiting 0 minutes) [19:58:22] andrewbogott, Coren or Ryan_Lane, can you take a look at https://gerrit.wikimedia.org/r/60312 please? [19:59:19] MaxSem: There's a typo that breaks validation, but I'll take a look at the corrected one if you want. [20:01:08] argh [20:01:31] dschoon: I can't imagine which quota you'd be hitting [20:01:56] Ryan_Lane: maybe related to the kraken work ottomata was doing in that group? [20:02:47] did you guys actually get an error? [20:03:00] I don't see any quota error in the log [20:03:31] Coren, amended [20:03:45] dschoon: mind trying for me? [20:04:00] Ryan_Lane: sure, one sec. [20:04:07] Warning: There are 1 users waiting for shell, displaying last 1: EBernhardson (WMF) (waiting 13 minutes) [20:05:27] Ryan_Lane: failed again. [20:05:40] Cannot run any more instances of this type. [20:05:41] hm [20:05:42] weird [20:06:14] stupid apple-q [20:06:28] and stupid nova api for not saying *which* quota is exceeded [20:10:13] ok. you guys have 24 cores [20:10:42] 48128MB ram [20:11:12] 640G storage [20:11:33] allowed to have 60 cores. should be good there [20:11:39] 51200 ram [20:11:41] likely that [20:11:50] 1000G storage [20:11:52] also good there [20:11:54] let me up the ram [20:12:39] dschoon: why so many giant instances? [20:13:04] as i said, migrating kripke (an XL on lucid) to precise [20:13:10] * Ryan_Lane nods [20:13:11] ok [20:13:41] upped the ram quota [20:13:44] go for it [20:14:00] I need to add admin API support to OpenStackManager so that people can see their quotas [20:14:27] and so that we can modify it via the interface [20:14:54] thanks [20:15:01] <3 Ryan_Lane [20:15:07] yw [20:16:38] Ryan_Lane: still failed :/ [20:16:45] hm [20:17:07] there's only 11 instances... [20:17:37] Warning: There are 1 users waiting for shell, displaying last 1: EBernhardson (WMF) (waiting 27 minutes) [20:18:07] dschoon: try again [20:18:21] maybe it's trying to allocate more than I thought [20:18:48] Ryan_Lane: it *says* it succeeded. however, i have learned patience and skepticism. [20:18:50] :) [20:18:54] heh [20:18:56] i'll let you know if there's more trouble~ [20:18:57] ty! [20:19:01] well, if it succeeded it should work [20:19:11] awesome [20:19:12] there's only one open bug that causes "error" now [20:19:19] i remember the Old Days [20:19:22] yeah [20:19:37] was much worse before :D [20:19:49] I guess it's still possible to fail if the host it lands on doesn't have enough ram [20:19:59] but we have two hosts that are mostly empty [20:20:11] when that 206 multipart response would randomly corrupt random things during build [20:20:27] hahaha [20:20:28] yeah [20:20:28] that [20:20:51] also, now when the instance says "active" it should immediately be available to ssh into [20:21:01] and it should give you an echo notification [20:21:42] now that I have both Coren and andrewbogott it's possible for me to actually fix things! :) [20:21:43] sweet. [20:21:46] <3 [20:23:36] kirke? :) [20:23:56] you guys really love naming things similarly :D [20:24:20] kirche? [20:25:14] illiad, iliiad and iiliad sound like three /great/ host names to me. :-) [20:25:55] let's just name all of them after common commands [20:26:02] ssh, cp, rm, ls [20:26:34] Heh. Honestly, it'd be hard to beat "slashdot" [20:26:48] tell that to digg or reddit [20:26:49] :) [20:28:38] * Coren remembers, long ago, the great fun of trying to speak a slashdot url on the phone. yeah, slash dot dot org slash ... no, you whave to write out slash dot s-l-a-... yeah, dot org slash. No, an actual slash. no, no, still colon slash slash. Yeah, real slashes. [20:28:50] hashar, around? [20:28:58] MaxSem: conf call sorry [20:29:07] hp [20:29:07] *np [20:29:25] I need to figure out how to get access to the ee-dashboard.wmflabs.org instance. Who would be able to help me with that? [20:31:23] kaldari: you need to be a member of its project [20:31:36] I'd imagine it's in the editor-engagement project [20:31:42] kaldari: who manages that project? [20:31:56] !resource editor-engagement [20:31:56] https://labsconsole.wikimedia.org/wiki/Nova_Resource:editor-engagement [20:31:58] MaxSem: done. you have like 19 minutes :-D [20:32:12] kaldari: you are a project admin in that project :D [20:32:19] Ryan_Lane: it's not listed under the editor-engagement project strangely [20:32:25] no? [20:32:28] hm. [20:32:38] hashar, how do I deploy a new extension to labs (requires a DB change)? [20:32:43] unless it's an alias for something else [20:33:08] that's the public IP [20:33:17] kaldari: check "Manage addresses" [20:33:24] and see which instance it's associated with [20:33:56] I could probably put that info on the manage instances page, but it would be a bunch of relatively expensive queries [20:34:13] I guess I could memcache it [20:34:34] Ryan_Lane: Don't see it under manage addresses either [20:34:41] ah [20:34:47] that's because it's in analytics :D [20:34:51] dschoon: ^^ [20:35:10] wm-bot: help [20:35:10] Hi Ryan_Lane, there is some error, I am a stupid bot and I am not intelligent enough to hold a conversation with you :-) [20:35:14] -_- [20:35:17] Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.10.6.8 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot [20:35:21] @commands [20:35:21] Commands: there is too many commands to display on one line, see http://meta.wikimedia.org/wiki/wm-bot for a list of commands and help [20:35:21] ? [20:35:50] that actually might be OK, I'm a member of the analytics project (but not an admin for it) [20:35:58] @labs-resolve ee-dashboard.wmflabs.org [20:35:58] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID unless it was created less than 42 seconds ago [20:36:07] :( [20:36:33] petan: ^^ is it possible for it to do some lookups to resolve public hostnames to instances? [20:37:01] MaxSem: so we run update.php automatically via a jenkins job [20:37:14] I don't see ee-dashboards listed as an instance under the analytics project either :P [20:37:24] so I just need to enable the ext? cool, thanks [20:37:30] MaxSem: so if your extension supports update.php , the sql schema get upgraded once per hour. just need to enable it [20:37:53] MaxSem: alpha documentation https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/How_code_is_updated [20:37:58] MaxSem: still have to largely improve it [20:38:59] MaxSem: basically: make sure your ext is in mediawiki/extensions.git , enable it with one of the -labs.php file, on https://integration.wikimedia.org/ci/view/Beta/ look for the job named 'update-database' click on it, run build. That triggers update.php on all databases. [20:39:10] the ee-dashboards instance seems to be a bit mysterious [20:39:59] someone set it up for the EE dev teams to build their analytics dashboard on, but none of us have access to it :( [20:40:44] unfortunately we have to build two dashboards on it before the Echo deployment on Thursday [20:41:01] to en.wiki [20:46:42] kaldari: someone in analytics can give you sudo on the instance [20:47:37] OK, I'll try pinging someone in analytics then [20:49:09] Ryan_Lane: Hmm, can't find Dario. Are there any analytics people on 3 right now? [20:49:20] or on IRC? [20:49:28] that's why I pinged dschoon ^^ [20:49:31] !log deployment-prep Manually updating all mw extensions to make sure everything works fine. [20:49:34] Logged the message, Master [20:49:39] sorry [20:49:43] what was the question? [20:49:47] howdy [20:49:50] hihi [20:50:12] I need to get access to the ee-dashboard labs instance to set up some dashboards for the Echo project [20:50:21] that will be deploying to en.wiki on Thursday [20:50:37] hi kaldari [20:50:49] but so far I haven't even managed to figure out how to ssh to that instance [20:50:57] limn team to the rescue [20:50:57] yay [20:51:00] :) [20:51:09] maybe we should take this over to #wikimedia-analytics [20:51:14] sure [20:51:19] leave the nice labs people to their glusterfs stuff :) [20:54:50] Change on 12mediawiki a page Wikimedia Labs/Interface usability improvement project was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=676478 edit summary: [+242] [21:02:08] any GlusterFS firefighter could help with a stalled file please? volume is deployment-prep-project stalled file is /apache/common-local/php-master/extensions/.git/modules/AdManager/refs/remotes/origin/master [21:04:47] hashar: are you copying from gluster to a local fs? [21:05:04] Ryan_Lane: doing git submodule update --init [21:05:09] ah [21:05:10] so I don't think anything is copied [21:05:27] I thought you were going to move off of gluster? :) [21:05:41] or are you waiting for nfs? [21:06:34] I will get rid of any shared dir dependency [21:06:44] Coren proposed yesterday to use NFS instead of Gluster [21:06:59] but I guess that is just moving the problem :D [21:07:07] well, only kind of [21:07:10] I will get sync / scap and the rest instead [21:07:17] NFS is actually reliable [21:07:24] but yeah, it's better to replicate what's in production [21:07:29] it'll be faster, too [21:07:36] and make the apache instances to not use /data/project at all [21:07:43] nfs isn't great with lots of small files either [21:08:02] Ryan_Lane: That's actually a bit better with v4 [21:08:06] nor with instances restarting [21:08:17] hashar: it should be fine with instances restarting [21:08:20] that's what autofs is for [21:08:38] you guys were using fstab previously [21:08:47] I could have told you that was a bad idea ;) [21:09:21] (in fact, I did, which is why I guess you guys moved away from it :D ) [21:10:52] Ryan_Lane: can you clean up the /apache/common-local/php-master/extensions/.git/modules/AdManager/refs/remotes/origin/master file on deployment-prep-project please? [21:11:07] that is preventing git from updating the mediawiki extension on beta [21:11:17] there are most probably over files corrupted [21:11:42] hm [21:11:54] is it not cleanable from the client side? [21:12:20] (Input/output error) [21:12:20] :D [21:12:29] * Ryan_Lane grumbles [21:12:32] I hate this filesystem [21:15:20] hashar: try now [21:15:32] \O/ [21:15:58] might have others to fix [21:16:02] * Ryan_Lane nods [21:16:26] I'd like to test the security of the LaTeXML version of the Math extension. Especially I'd like to find out if people can hide somehow bad code inside the math tags. [21:16:33] :p [21:16:44] Ryan_Lane: /apache/common-local/php-master/extensions/ArticleFeedbackv5/.git/refs/remotes/origin/master: gfid differs on subvolume 1 [21:17:12] Ryan_Lane: maybe delete all .git/refs/remotes/origin/master ? [21:17:27] that's not the easiest thing in the world to do.... [21:17:31] Can I destroy something that is outside the project [21:17:41] Ryan_Lane: i will copy paste the errors so ;) [21:18:00] all of them in a single output would help [21:18:08] I have a script that can fix files [21:18:14] thanks to salt [21:21:25] git submodule foreach 'git checkout origin/master || /bin/true' [21:21:29] will grep the error log [21:24:10] andrewbogott: morebots has many things running atm; do you have a plan or something I should be aware of for tomorrow's maintenance? [21:25:41] Coren, those jobs will be automatically restarted afterwards, right? Or am I misunderstanding? [21:27:10] andrewbogott: They /should/ be, but I don't know how well they will cope with having an entire filesystem switch beneath them. If they can be killed and restarted without human intervention in the general case, then they should be able to work without intervention. [21:27:37] andrewbogott: I just wanted to make sure you were aware and would keep an eye on them as maintenance ends just to make sure. [21:27:46] Sounds like they will probably survive, then… it'll be obvious if they don't. [21:27:50] Yep, thanks for the reminder. [21:27:52] morebots can die for some period of time [21:27:58] it's no big deal [21:40:56] Ryan_Lane: and here are the broken file on deployment-prep-project http://paste.openstack.org/show/36513/ :D [21:41:11] hashar: <3 [21:41:23] you even made it so that I don't need to scrape through the text [21:42:49] grep -q '/apache/[^ ]*' ftw! [21:43:41] hashar: should be fixed [21:44:19] Ryan_Lane: thx. Rerunning the extension update :) [21:44:31] it looks to me like it split brained recently [21:44:34] which is a bad sign [21:44:57] [2013-04-22 21:44:43.043888] W [client3_1-fops.c:2385:client3_1_rename_cbk] 0-deployment-prep-project-client-1: remote operation failed: Cannot allocate memory [21:44:58] hehe [21:45:24] I guess that was when I shrunk the volume [21:47:24] [bz] (REOPENED - created by: Antoine "hashar" Musso, priority: Unprioritized - major) [Bug 47425] glusterFS could not allocate memory - https://bugzilla.wikimedia.org/show_bug.cgi?id=47425 [21:47:29] i have reopened https://bugzilla.wikimedia.org/show_bug.cgi?id=47425 [21:47:43] Ryan_Lane: beta fixed \O/ [21:47:53] great :) [21:50:26] though the cannot allocate memory error might break it again [21:51:37] well echo's working on wikitech Ryan_Lane ;) [21:51:43] heh [21:51:54] noticed I removed one of your rights, I'd imagine? :) [21:52:15] just had a conversation about rights in another channe [21:52:17] *channel [21:52:25] yes [21:52:26] admin on wikitech is problematic [21:52:29] hmm: (cur | prev) 19:09, 8 March 2013‎ Ryan Lane (Talk | contribs)‎ m . . (6 bytes) (0)‎ . . (Changed protection level for "Main Page" ([Edit=Block new and unregistered users] (indefinite) [Move=Block new and unregistered users] (indefinite))) [21:52:37] yet the tab is "view source" for me? [21:52:47] hm [21:52:48] weird [21:53:14] I wonder if there's some right I need to add in the config [21:53:36] maybe I don't have settings for autoconfirmed [21:53:53] mediawiki is such a pain in the ass to properly configure [22:10:36] I am out [22:10:40] *wave* [22:42:17] andrewbogott: I haven't since I didn't want to set up everything, but ori said he would try and test it today [22:51:10] legoktm: OK, let me know how it goes. [23:00:00] andrewbogott: heh. the mystery code in api was me adding the beginning of an API :) [23:00:29] Ryan_Lane: OK, hopefully I didn't lose anything of yours. [23:00:33] nope [23:00:34] didn't [23:03:58] hi, how to web-view the version 1.21.0rc4 ? [23:05:08] I want to know, whether https://gerrit.wikimedia.org/r/#/c/54986/ which is merhed into master is really part of 1.21.0rc4 [23:05:56] Hm… is 1.21.0rc4 a tag? [23:05:57] * andrewbogott looks [23:07:38] Hm, I don't see release tags in my repo. Sorry, I'm not sure how versions are marked. Probably best to ask in wikimedia-dev [23:08:03] ok [23:16:41] Warning: There are 1 users waiting for shell, displaying last 1: Skingry (waiting 0 minutes) [23:30:04] Warning: There are 1 users waiting for shell, displaying last 1: Skingry (waiting 13 minutes) [23:30:49] Coren|Food: ping [23:43:26] Warning: There are 1 users waiting for shell, displaying last 1: Skingry (waiting 26 minutes) [23:45:29] andrewbogott: check out https://nova-precise2.pmtpa.wmflabs/wiki/Special:NovaInstance [23:45:38] specifically, force refresh the page, then click "reboot" [23:45:42] on any instance [23:47:32] hm… which projects have instances? [23:47:39] testing [23:47:48] all in the error state, of course :) [23:48:42] oh, I guess i have to join the project [23:48:47] LD [23:48:49] err [23:48:49] :D [23:49:53] Ooh, fancy! instant gratification! [23:49:57] yep [23:50:00] Or, in this case, instant anti disappointment [23:50:03] now the case where it doesn't error [23:50:07] retry it [23:50:47] Is that going to get actual feedback about the reboot? [23:51:12] well, it switches the state to rebooting [23:51:15] [bz] (ASSIGNED - created by: Robert Hanke, priority: Normal - enhancement) [Bug 41023] Make a stats table for 85 W3C wikis - https://bugzilla.wikimedia.org/show_bug.cgi?id=41023 [23:51:17] I probably need to make that clearer [23:51:25] Oh, so it does. [23:51:29] hm, puppet still breaking the wikitech-test.wmflabs.org host I see [23:51:45] maybe I should highlight it for a short period of time then unhighlight it [23:52:02] Or you could display 'in progress' or 'rebooting' where you were displaying 'error' before. [23:53:26] hm. well, I'm already changing some text, I think bringing attention to it is probably the best option [23:53:54] I'd like the "error' badge to have a tooltip with the error message, too [23:56:52] Warning: There are 1 users waiting for shell, displaying last 1: Skingry (waiting 40 minutes)