[00:01:16] !ping [00:01:16] !pong [00:45:41] 3Wikimedia Labs / 3Infrastructure: Improve ganglia metrics for Labs infrastructure - 10https://bugzilla.wikimedia.org/40023#c1 (10Krinkle) 5NEW>3RESO/WOR This was done a while ago. Although it is currently down (see bug 46475). http://ganglia.wmflabs.org/ https://wikitech.wikimedia.org/wiki/Nova_Resourc... [00:46:41] 3Wikimedia Labs / 3Infrastructure: ganglia down / needs reinstall - 10https://bugzilla.wikimedia.org/63362#c10 (10Krinkle) Seems to be down still (or again). > There was an error collecting ganglia data (127.0.0.1:8654): fsockopen error: Connection refused [00:48:12] Coren: Could you try rebooting the ganglia aggregator? https://wikitech.wikimedia.org/wiki/Nova_Resource:Ganglia It seems that fixed it in May according to the SAL msg by Hashar [00:48:27] I'd also also to volunteer in helping out [00:48:50] I can try, certainly. [00:49:49] Krinkle: o/ [00:50:15] Krinkle: The problem is evident; the whole thing is horribly out of space. [00:51:10] Krinkle: Gimme a few, I can see if I can make it less bad. [00:51:31] Coren: :D [00:52:25] Yeah; that thing is configured to stuff all the ganglia data in /var; that's always going to bite. [00:54:09] * Coren attempts to locate its puppet config. [00:57:15] Krinkle: Do you happen to know where it lives? [00:57:29] I do not [00:57:30] Coren: It occured to me after yesterday's nfs issue, could someone from a VM cause the physical hard drive to thrash, in order to DoS? [00:57:34] (not yet) [00:58:29] a930913: Not likely; the disk bandwidth is considerably higher than the network bandwidth - though it'll certainly affect performance of other instances on the same server (because network) [01:00:40] Krinkle: afaict, that instance only has ganglia::web set; it looks like it was configured by hand (how evil) [01:02:06] !log ganglia set role::lvm::srv on aggregator so that it has actual space to use [01:02:11] Logged the message, Master [01:02:14] Coren: So find your way onto the right EC2 server, then thrash the hell out of it, to take down a site? [01:02:58] a930913: That'd take some (very noticable) doing in order to impact performance. [01:05:00] Coren: The idea would be to make it noticable, it's a DoS? [01:05:54] a930913: Noticable as in: easy to track down and squish. It's a freely accessible shared infrastructure; we are relying on AGF as with all projects. [01:06:25] Coren: AWS I mean. [01:06:35] Krinkle: Yeah, puppet is disabled on the box, and everything is configured manually. I can hotfix things, but that desperatly wants being puppetized. [01:06:55] One would hope ops/puppet classes were reusable for this [01:07:06] if not, we should probably try to do that instead of puppetising whatever was left here [01:07:22] Ah, on AWS? I don't know enough of the details of their infrastructure, but I expect it's also possible to consume a disproportionate part of resources there -- but they have a LOT of it and it'd probably be expensive to make a noticable dent [01:12:12] !log cvn Enable role role::labs::lvm::mnt on all cvn instances [01:12:13] Krinkle: Btw, relative to yesterday, it's tomorrow, if you get my drift. [01:12:16] Logged the message, Master [01:15:45] Krinkle: You really shouldn't be using /mnt if you have a choice -- that is only meant for transient removable media. The correct spot is /srv (and yes, /mnt was often used in the past or - even worse - /a in prod) [01:16:21] * Coren is actually configuring aggregator right now to use /srv [01:16:49] OK. /srv/ was often used for services on other linux servers I accessed in the past elsewhere [01:16:57] and old wmf used /srv/ as docroot even [01:17:08] -/srv/org/wikipedia/ [01:17:12] :D [01:17:19] That'd actually be quite correct usage. [01:17:20] :-) [01:17:37] Not if /srv is the diskroot [01:18:02] it'd be a bit weird to have org/ dangling in the root of that disk I guess [01:18:21] anyway, it wasn't a separate disk afaik [01:18:28] just a directory [01:18:46] there was and still is /srv/ssd/ on a few nodes though, which makes sense [01:19:00] Coren: Can I just swap mnt for srv? [01:19:04] nothing on it yet [01:20:35] If the class already applied you'll have to rejigger things a bit manually, but otherwise yes. [01:44:29] Krinkle: There are still issues with it - May 13 23:32:08 aggregator /usr/sbin/gmetad[18506]: data_thread() got no answer from any [nagios] datasource [01:44:45] And I need to go now; but at least the space issue is fixed. [01:44:49] k :) [01:51:17] !log cvn Disabled role::labs::lvm::mnt, and enabled role::labs::lvm::srv instead (separate changes) [01:51:19] Logged the message, Master [04:21:51] YuviPanda: The labs roles are empty [04:22:02] *labs-vagrant [04:24:12] !ping [04:24:12] !pong [04:24:16] !ping [04:24:16] !pong [06:03:37] prtksxna: I ran into similar problems. doing 'chmod a+x ~' as your user before running labs-vagrant provision might help [06:04:02] YuviPanda: Did the egg boil? [06:04:12] prtksxna: dunno, I'm trying on sugarfrosties [06:06:20] icy [06:07:28] prtksxna: http://sugarfrosties.wmflabs.org/ works [06:07:36] prtksxna: just needed the 'chmod a+x ~' [06:07:47] prtksxna: try that and then do sudo labs-vagrant provision? [06:08:10] YuviPanda: can't we just use sugarfrosties now? [06:08:31] prtksxna: no, since I'm doing some special setup there for the url redirection stuff [06:08:43] oh ok [06:15:47] Actually ZWS documentation in two large PDFs. Most of what's written on tswiki it from there. Its just that Google doesn't like official documentation these days (try searching for anything bash related). [06:28:45] prtksxna: apply this patch on /vagrant (https://gerrit.wikimedia.org/r/#/c/139069/) and make sure you have already done 'chmod a+x ~' [06:28:54] prtksxna: labs-vagrant list-roles should come back [06:31:51] YuviPanda: Nope [06:31:57] meh. [06:32:00] works for me [06:33:08] prtksxna: lol. actually, do 'cd /vagrant/mediawiki/lib' first and then do it after that patch? [06:33:11] i think that should work. [06:33:15] I'm an idiot, but that's k [06:34:16] YuviPanda: Sorry? [06:34:22] oh ok [06:35:50] YuviPanda: there was no lib/ in mediawiki/ but it worked in mediawiki/ [06:35:59] prtksxna: cool :) [06:36:13] prtksxna: yeah, so without that patch it'll always work if you are in /vagrant [06:36:21] prtksxna: so you can discard patch and run enable-role and list-roles from /vagrant [06:36:22] YuviPanda: oh [06:36:30] prtksxna: I'll figure out way to fix later, but this is temp-fix for now [06:36:32] YuviPanda: provisioning now, will then add hc… [06:36:39] prtksxna: ok [06:36:39] YuviPanda: Cool! Thanks! :D [06:36:42] prtksxna: :) [06:36:50] prtksxna: and remember to add proxy [06:36:55] added [06:36:58] prtksxna: cool [06:36:58] http://boiledegg.wmflabs.org/wiki/Main_Page [06:37:14] cool [06:37:38] YuviPanda: I was looking for the hovercards role :P [06:38:10] prtksxna: not peekcards? [06:38:30] huehuehue [06:41:57] YuviPanda: It all works! Thanks o/ [06:42:09] prtksxna: \o/ ya [06:42:09] YuviPanda: Free stay for you in Goa :P [06:42:14] prtksxna: woo! :) [06:57:59] 3Wikimedia Labs / 3deployment-prep (beta): HTML Tags are stripped from extension output - 10https://bugzilla.wikimedia.org/66516 (10physikerwelt) 3NEW p:3Unprio s:3normal a:3None Some HTML Tags are stripped from extension output on betalabs even though the "markerType" => 'nowiki' is used to prevent... [07:26:42] 3Wikimedia Labs / 3tools: Tool Labs: Provide anonymized view of the user_properties table - 10https://bugzilla.wikimedia.org/58196#c49 (10Silke Meyer (WMDE)) Summary of what was discussed during office hour (https://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20140611.txt, from [17:17:25] on): * This... [07:32:11] 3Wikimedia Labs / 3tools: Move wiki.toolserver.org to WMF - 10https://bugzilla.wikimedia.org/60220#c28 (10Silke Meyer (WMDE)) Summary of office hour: * We haven't heard from Reedy. * We questioned the importance of keeping the wiki again if no one has the resources to move it somewhere. * If no one does, we... [08:11:11] 3Wikimedia Labs / 3tools: Tool Labs: Provide anonymized view of the user_properties table - 10https://bugzilla.wikimedia.org/58196#c50 (10Nemo) (In reply to Silke Meyer (WMDE) from comment #49) > * As far as we know, "live" tools depended on #64115 (filtered view) which > has been resolved. But no live tools... [08:11:27] 3Wikimedia Labs / 3tools: Move wiki.toolserver.org to WMF - 10https://bugzilla.wikimedia.org/60220#c29 (10Nemo) (In reply to Silke Meyer (WMDE) from comment #28) > * If no one does, we'll keep a backup dump for later ressurrection. Full SQL dump? [08:24:59] 3Wikimedia Labs / 3tools: Move wiki.toolserver.org to WMF - 10https://bugzilla.wikimedia.org/60220#c30 (10Nemo) Ahem, not SQL. * Full database dump + images? [08:45:15] 3Wikimedia Labs / 3tools: Toolserver migration to Tools (tracking) - 10https://bugzilla.wikimedia.org/58788 (10Silke Meyer (WMDE)) [08:45:15] 3Wikimedia Labs / 3tools: Moving toolserver domain, mail and redirects - 10https://bugzilla.wikimedia.org/66113 (10Silke Meyer (WMDE)) [08:46:26] 3Wikimedia Labs / 3tools: Transfer domain toolserver.org to WMF - 10https://bugzilla.wikimedia.org/60864#c1 (10Silke Meyer (WMDE)) Manuel Schneider is expecting a notification once we are ready to transfer the domain. [13:45:12] 3Wikimedia Labs / 3deployment-prep (beta): Parsoid cache backend is a pmtpa IP address - 10https://bugzilla.wikimedia.org/66497#c1 (10Antoine "hashar" Musso) 5NEW>3RESO/INV I was confused yesterday. The addresses are fine. [15:43:16] !ping [15:43:16] !pong [16:26:42] 3Wikimedia Labs / 3(other): (Tracking) Database replication services - 10https://bugzilla.wikimedia.org/48930 (10Kunal Mehta (Legoktm)) [16:31:14] 3Wikimedia Labs: centralauth_p is missing tables - 10https://bugzilla.wikimedia.org/66533 (10Kunal Mehta (Legoktm)) 3NEW p:3Unprio s:3normal a:3None Tool labs: MariaDB [centralauth_p]> show tables; +---------------------------+ | Tables_in_centralauth_p | +---------------------------+ | global_group... [16:31:28] 3Wikimedia Labs / 3tools: Missing Toolserver features in Tools (tracking) - 10https://bugzilla.wikimedia.org/58791 (10Kunal Mehta (Legoktm)) [16:31:28] 3Wikimedia Labs / 3(other): (Tracking) Database replication services - 10https://bugzilla.wikimedia.org/48930 (10Kunal Mehta (Legoktm)) [19:14:41] 3Wikimedia Labs / 3tools: Tool Labs: Provide anonymized view of the user_properties table - 10https://bugzilla.wikimedia.org/58196#c51 (10Luis Villa (WMF Legal)) If the users are OK with the month-filtered view, and exclusion of small wikis, that seems like a workable compromise to me. [19:52:29] Can someone tell me what's going on here --> http://tools.wmflabs.org/mono/test/