[05:51:03] [bz] (8NEW - created by: 2Chris McMahon, priority: 4Unprioritized - 6normal) [Bug 50624] ULS "IME selector" gone from input boxes - https://bugzilla.wikimedia.org/show_bug.cgi?id=50624 [06:32:04] This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.10.8.18 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot [06:32:06] :O [06:32:11] @seen petan [06:32:11] Steiny: petan is in here, right now [06:44:14] Steiny: @notify petan [07:56:54] hello [08:01:12] hi, is anyone able to connect to its instances ? :D [08:04:09] hashar let me check [08:04:29] yes [08:04:52] hashar what doesn't work to you? [08:05:20] you may need to reboot the instances where ssh doesn't work since coren did some nfs hacking in past [08:05:29] I know beta had problems too [08:07:18] ahh [08:07:30] petan: thank you :-] going to reboot all of them I guess [08:07:40] I wish we had salt in labs :( [08:08:00] !log deployment-prep rebooting deployment-cache-upload04 [08:08:01] we do [08:08:04] Logged the message, Master [08:08:06] but only ops can access it [08:10:49] and Ganglia is no more reporting for deployment-prep doh [08:11:04] oh no [08:11:06] it is again [08:11:29] na it does not .. [08:13:15] zz_YuviPanda .. [08:13:29] these indian people are sleeping too much [08:13:38] :D [08:26:08] !log deployment-prep Set $wgLoadScript to points to bits instead of the wiki local docroot. {{gerrit|70322}} [08:26:11] Logged the message, Master [08:27:55] !log deployment-prep rebooting deployment-cache-text1 , maybe I can get ssh access this wa [08:27:56] y [08:27:58] Logged the message, Master [08:41:00] [bz] (8REOPENED - created by: 2Chris McMahon, priority: 4Unprioritized - 6major) [Bug 50623] Entering AFTv5 feedback causes error - https://bugzilla.wikimedia.org/show_bug.cgi?id=50623 [08:41:54] [bz] (8NEW - created by: 2Chris McMahon, priority: 4Unprioritized - 6major) [Bug 50622] Special:NewPagesFeed error - https://bugzilla.wikimedia.org/show_bug.cgi?id=50622 [08:42:59] [bz] (8ASSIGNED - created by: 2Niklas Laxström, priority: 4High - 6major) [Bug 48203] Purging does not work on deployment-prep / beta labs - https://bugzilla.wikimedia.org/show_bug.cgi?id=48203 [08:46:43] chrismcmahonafk: since you are still awake :-] [08:46:53] chrismcmahonafk: I got a huge fix for javascript [08:47:01] arch the notification above are from me hehe [10:14:25] [bz] (8ASSIGNED - created by: 2Antoine "hashar" Musso, priority: 4High - 6normal) [Bug 49470] rebuild the upload cache - https://bugzilla.wikimedia.org/show_bug.cgi?id=49470 [10:45:00] petan: sup [10:46:07] YuviPanda do you want to have a look at sources of http://tools.wmflabs.org/sulinfo [10:46:15] it run far slower than version on TS [10:46:26] I wanted to know if there is a way to optimize it a bit, maybe use memcache etc? [10:46:33] ah, sure! [10:46:41] I heard that as 'using redis etc' but sure! [10:47:07] :D [10:47:56] ok you are in project, so just doing become sulinfo get u there [10:48:18] it would be best to create a git repository for it [10:48:29] I created a simple repository in public_html but it has no remote [10:52:40] ok [10:52:46] what do you mean 'create'? [10:52:52] there's no repo already?! :O [11:02:25] no [11:02:28] there never was [11:04:24] so people just copied things around? [11:05:52] probably [11:06:01] I don't know I stolen this from TS :P [11:06:08] and modified for it to work [11:06:35] ah [11:06:40] who maintains it on TS? [11:06:51] so this is sortof third hand code :P [11:06:58] quentinv57 [11:07:09] maybe fourth [11:07:13] because he also stolen it [11:07:24] original version is by vvv [11:07:34] and he is in this channel :D [11:07:51] my idea is to totally rewrite it [11:08:06] what exactly does it do? [11:08:13] from 1 shitty script to multiple files, divided to many classes and being cute and nice [11:08:25] yeah, and use a non shitty language [11:08:30] like php? :D [11:08:32] LOL [11:08:37] php is still best for this kind of stuff [11:08:42] but I think I've to finish my current tool first (github -> gerrit thing) [11:09:23] seriously what is better than PHP for web stuff [11:09:26] python is crap [11:09:38] I can imagine using CGI like bugzilla [11:09:39] python as it is used by a lot of people on toolserver and co (CGI?) yeah [11:09:41] it is shitty [11:09:52] python is indeed very shitty :P [11:09:52] but use a proper microframework (ala flask) and it is awesome [11:09:59] CGI is what is shitty [11:10:00] who of you know what was the occupation of John Hartford was. Who don't know it? [11:10:11] LOL [11:10:19] and either way, one of the things I've learnt over the past months is to not to try to flame war on IRC, so I'll stay away :) [11:10:20] Pyfisch there is this wikipedia site you know [11:10:38] YuviPanda but with no flame wars internet is boring [11:10:48] petan: I asked you if you know it by heart not if it is on wikipedia ;-) [11:11:05] s/boring/productive/ [11:11:08] if it's on wikipedia I know I could know it at some point :3 [11:11:42] s/productive/boring, but productive [11:12:31] Pyfisch I have no idea who Hohn Hartford was [11:12:51] petan: ok ;-) [11:13:02] I also dont know :-) [11:13:14] John Cowan Hartford was an American folk, country and bluegrass composer and musician known for his mastery of the fiddle and banjo, as well as for his witty lyrics, unique vocal style, and extensive knowledge of Mississippi River [11:13:33] I can't think of a smile that fit this [11:13:37] :~P [11:13:48] no that's not mad enough [11:13:55] ;~0 [11:13:59] make that face now [11:16:07] I only wanted to know if I am the only person who don't know him [11:16:16] (I don't know him either) [11:51:03] YuviPanda do we have api that retrieve some basic one sentence information covering some article? [11:51:08] I don't think so :o [11:51:11] yeah, we do [11:51:14] really? [11:51:14] extracts [11:51:22] it's fairly new, as in around a year or so old [11:51:27] Coren: re :-] [11:51:29] should be a query prop [11:51:30] i think [11:51:37] ok [11:51:38] hashar: I'll be with you in 10? [11:51:56] Coren: sure, take your time! [11:52:16] petan: it was built for mobile, I think. [11:52:17] :P [11:52:26] petan: our API has so many things that it's hard to remember... [12:03:01] hashar: Moo! [12:03:21] Coren: -) [12:03:34] Coren: so for some reason I can't not ssh to some deployment-prep instances. [12:03:45] Which? [12:03:49] such as deployment-cache-upload04.pmtpa.wmflabs [12:04:02] I use bastion.wmflabS.org as a ssh proxy [12:04:25] upon connecting I do see the ssh banner "If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances" [12:04:42] but nothing else happen past debug1: Offering RSA public key: /home/hashar/.ssh/id_rsa [12:04:52] so I guess /home is unreachable on the server [12:05:11] I did try to reboot it via the web interface, not sure whether it worked [12:05:19] hashar: Gluster seems completely dead on that box at least. [12:05:35] and I thought /home was using NFS :-] [12:05:45] Yeah, that box was rebooted just a bit under 4h ago that I can see. [12:05:53] projectstorage.pmtpa.wmnet:/deployment-prep-home /home fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 0 0 [12:06:28] Didn't you generally switch to NFS in that project? [12:08:49] for /data/project [12:08:57] and only on instances that needs to access MediaWiki files [12:09:07] Ah. [12:09:11] (aka the deployment host and the application servers [12:09:49] oh and they have their /home on nfs as well now :-] [12:10:21] I should probably apply the labsnfs role on all instance of that project [12:10:39] ideally I would do that in base.pp but I am afraid it is not going to work properly :-] [12:12:55] hashar: Truth be told, I'd expect the odd mix of gluster and NFS depending on the instance within a single project is doing *wonders* for reliability. :-) [12:13:19] indeed [12:13:27] * hashar curses GlusterFS [12:16:30] ahh [12:21:21] * Coren doesn't quite get how gluster is broken /this/ time. [12:21:53] lets get rid of it on deployment-prep in favor of NFS :-) [12:22:05] it is not worth your time to investigate that piece of crap that never quite worked [12:22:25] That "should" be a simple as including the labnfs class and rebooting.\ [12:22:37] yeah working on a patch against base.pp to do that [12:23:21] Oy, gotta go for a bit. I'll be back 'round 9h (13h UTC) [12:23:57] Dogs demand walk. :-) [12:27:53] Coren, you're finally online. [12:28:45] And totally ignoring me. :p [12:29:33] [bz] (8NEW - created by: 2Liangent, priority: 4Unprioritized - 6normal) [Bug 50935] Install pastebinit - https://bugzilla.wikimedia.org/show_bug.cgi?id=50935 [12:30:20] Coren: whenever you come back, I got a base.pp hack to include role::labsnfs instead of glusterFS https://gerrit.wikimedia.org/r/72504 [13:56:33] hashar: O [13:57:17] Coren: yup ? :D [13:57:23] hashar: I'm a little hesitant to +2 a change to base.pp without a second opinion, but this looks fine to me. [13:57:42] yup you might want to ask random other people :) maybe mark [13:58:43] Yeah, mark is plenty random. :-) [14:00:06] Coren, you there? [14:00:30] Cyberpower678: Nope, clearly not. :-) [14:00:46] Okay. I'll ask later. [14:00:47] :p [14:01:08] Heh. [14:01:14] Coren, can you look at my crontab and my tasks running under local-cyberbot [14:01:21] and tell me what you think? [14:03:18] Cyberpower678: Yeah, remove that "* * * * * crontab $HOME/crontab" and it's all fine now. [14:03:29] Why? [14:03:41] That keeps my crontab up to date. [14:04:54] That also forces a reload of your crontab every minute. 90% of the crontab log activity is that line. I don't mean for your user, I mean globally. [14:05:27] Coren, you log the crontabs? [14:05:58] It's also completely silly -- why in blazes would you do something like this? Anything that updates your ~/crontab should just call crontab itself. [14:05:59] addshore does it too. [14:06:31] Cyberpower678: No, he doesn't. [14:06:48] Coren, he's the one who suggested it to me. [14:06:58] Cyberpower678: And if he did, I'd tell him too. [14:07:26] Cyberpower678: Well, perhaps he noticed how bad the suggestion was and stopped doing it long ago... or maybe he never did. He's certainly not doing it now. :-) [14:07:46] "Anything that updates your ~/crontab should just call crontab itself." Can you clsrify? [14:09:20] You're synchronizing, (forcibly and regardless of actual changes) your crontab against a file in the home. Whatever updates that latter file should just edit your read crontab instead (which cron notices), or at the very least call crontab only if it actually did change the file in your home. [14:09:40] s/read crontab/real crontab/ [14:10:06] Right now, you're just doing a very expensive noop every minute. :-) [14:10:21] what is the db name for wiktionary? [14:10:32] fale: Which language's? [14:10:36] Coren: it [14:11:25] fale: itwiktionary [14:11:30] Coren: thanks : [14:11:32] :) [14:14:21] I'm looking for ^d or Ryan to help me with a gerrit problem but since they aren't here, can someone else help? [14:14:43] rachel99: I might. What's up? [14:15:41] It looks like I have 2 accounts now, possibly. My original one with rachelqa99@gmail.com address, which I could not log into next week [14:16:24] And my old account rachelthomas_99@yahoo.com, which I can now log into, but doesn't have my commits/changes. [14:17:12] ^d did some kind of update last week to allow me to login, but it appears that he put the old address setting back, and it isn't corresponding to my account with the commits. [14:17:20] rachel99: Oy! Ttrying to merge account or sorting them out requires more gerrit-fu than I can muster with confidence. I think you want ^d for this, and not settle for me. :-) [14:18:11] Ok... where is he then? I've been sending him email since last Wed, but haven't heard back. [14:18:58] ... he's not OOO that I know of, but the muricans had a holiday on Thursday, and most took Friday off to have a long weekend. That might be what happened. [14:19:55] Yup, I'm one of them so i was off Friday too. However, I thought he might be in by now. Do you know when he would get in? [14:21:06] I don't know if he usually works on VA or CA time; if the latter, he'll probably be around in 40 mins. [14:21:45] ok, I'll try back.. if you do see him online, let him know that I am looking for him. [14:21:55] * Coren nods. [14:21:58] Will do. [14:22:05] Thank you. :) [14:22:15] addshore, what do you do? [14:24:22] the namespacename table is close to arrive, Coren? [14:25:30] fale: It's not on my priority list but, as I've explained previously, if someone makes a tool to set it up and keep it up to date before I get around to that I'll gladly deploy it to make it globally available. [14:26:57] Coren: roger :) [15:23:05] When I do a query like: select * from logging where log_type ='upload' and log_action = 'overwrite' limit 20; from tool labs its really slow [15:23:30] but it shouldn't be, since there should be an index on (log_type, log_action, log_timestamp) [15:23:40] at least default mw has one [15:27:16] bawolff: The index might be hidden by the view. Lemme check. [15:27:26] (logging is one of the more heavily redacted tables) [15:28:09] Coren: You mean I can't use my tools lab access to see all the oversight logs? [15:28:11] :( [15:28:34] There goes that plan to blackmail the wikimedia world [15:29:19] [bz] (8RESOLVED - created by: 2Chris McMahon, priority: 4Unprioritized - 6normal) [Bug 50624] ULS "IME selector" gone from input boxes - https://bugzilla.wikimedia.org/show_bug.cgi?id=50624 [15:29:25] bawolff: Hm. I see why indeed; I'd need to create a specialized view to make your query even vaguely efficient, but that one wouldn't have the rows for some classes of revdel entries at all instead of nulls in the column. [15:30:13] Though from what I see, that shouldn't be an issue in your case since you're actually querying with a where clause on log_action and wouldn't get rows with nulls back anyways. [15:30:14] Honestly, its not a big deal. The query I was running was a one off thing that I probably won't be running again [15:32:18] I'd still like to try. [15:32:30] You're not going to be the only one ever. :-) [15:32:35] Probably not [15:32:39] Where did you try this, enwiki? [15:32:45] commonswiki [15:33:17] Originally use case was I was investigated a bug report on commons, and needed to find a recently overwritten file. This seemed like the fastest method [15:34:15] part of the reason it was slow originally was also that I had spelled part of my where query wrong, so it was looking for log_action = 'reupload' not overwrite [15:35:53] bawolff: Try it now against the logging_logindex table? [15:37:22] still seems quite slow [15:39:16] bawolff: I'm looking an the explain, and it's only using the timestamp index, even on the original table. [15:41:18] on my local mediawiki install, it seems to use the (log_type,log_action,log_timestamp) index fine [15:41:51] Hm. Funnily enough, if you add 'order by log_timestamp desc' it then properly uses the index and gets blindingly fast. :-) [15:42:37] That's weird [15:42:45] * Coren stares at MariaDB's query optimizer. [15:43:04] select * from logging_logindex where log_type ='upload' and log_action = 'overwrite' order by log_timestamp desc limit 20; [15:43:09] 20 rows in set (0.02 sec) [15:43:48] Coren: but if you do a log_action that doesn't exist, that query is still super slow [15:44:41] bawolff: That's normal -- there is no index on just log_action so it can only exclude triples (type,action,timestamp) not on a single column. [15:45:08] I mean doing log_type='upload' and log_action='non-existent action' [15:45:15] An index on multiple columns does not have the semantics of the individual indices. [15:46:04] Specificly: select * from logging_logindex where log_type ='upload' and log_action = 'overwrited' order by log_timestamp desc limit 20; [15:46:16] should also be fast with that index, but isn't [15:47:44] No, actually, that's expected behaviour in mysql for the general case, afaik. It's not all that good at restricting the result set according to a multi-column index in most cases. [15:47:55] If there was also an index on log_action, that'd be instant. [15:48:17] Coren: Its instant on my local mediawiki db [15:49:09] Coren: https://dpaste.de/dmXWm/ [15:49:46] good morning, we are having gluster issues (ro) in the /data partition on parsoid-spof [15:49:48] I didn't say I didn't believe you, just that this is not the observable behaviour in production. :-) [15:50:38] gwicke: We have definitely a gluster issue: it's installed in many places. Or did you mean an extra, different issue? :-) I can go check. [15:51:45] That seems rather odd. If anything I'd expect production to have a better optimizer than my oldish local mysql install ;) [15:52:01] gwicke: What project is this? [15:52:10] Coren: visualeditor, vm parsoid-spof [15:52:23] Ryan_Lane: gluster issues again on /data/project ^^ [15:52:31] * Ryan_Lane grumbles [15:52:42] we could switch it to nfs.... [15:53:04] all the visualeditor vms would need to be switched too then [15:53:09] but likely more reliable [15:53:18] yeah. it's a puppet class [15:53:24] so it's easy enough [15:53:31] gwicke: I've been seeing gluster problems too in labs. [15:53:31] the data needs to be transferred [15:53:36] let me fix gluster right now [15:53:45] Ryan_Lane: thanks! [15:53:51] manybubbles: everyone always sees gluster issues. it's not terribly stable [15:53:51] Ryan_Lane: On it. [15:53:57] gwicke: Is it working now? [15:54:10] " gwicke: We have definitely a gluster issue: it's installed in many places." [15:54:29] looks promising so far.. [15:54:44] Coren: what did you do? [15:54:45] slow as ever, but not bombing yet [15:54:55] Ryan_Lane: fair enough. I've got mysql running against it now (because it is big enough) and it seems to be mostly working [15:54:57] Ryan_Lane: start force on the volume. [15:54:58] killed the daemons and did a force restart? [15:55:00] * Ryan_Lane nods [15:55:11] Ryan_Lane: No, the deamon was already dead [15:55:20] manybubbles: against gluster? on /data/project? [15:55:29] manybubbles: that's going to fail miserably :) [15:55:41] gluster doesn't support directIO [15:55:46] you're going to lose data [15:55:50] Ryan_Lane: In fact, funny thing: [15:55:50] Coren: works again, thanks! [15:55:57] I would move it to /mnt [15:55:59] (a) process not there. [15:56:09] (b) gluster volume start visualeditor-project [15:56:09] Volume visualeditor-project already started [15:56:17] Ryan_Lane: yeah.... if I had a large enough mnt I would just rebuild it all against /mnt [15:56:27] Coren: yeah. it's not checking for running processes there [15:56:43] Coren: it's checking to see if the volume is in the "running" state [15:56:44] (c) process still not there [15:56:44] (d) gluster volume start visualeditor-project force [15:56:44] Starting volume visualeditor-project has been unsuccessful [15:56:44] (e) process there and working. [15:56:44] :-) [15:57:02] manybubbles: what data are you putting into this sql database? [15:57:12] Ryan_Lane: enwiki [15:57:14] stupid gluster [15:57:18] manybubbles: … ? [15:57:26] manybubbles: why not use the replica? [15:57:26] Ryan_Lane: well, the restore of the articles. [15:57:42] Ryan_Lane: the what? [15:57:53] Ryan_Lane: I'd love to not have to screw around with this [15:57:58] the database replicas? from production? [15:58:41] Ryan_Lane: Speaking of, we need to coordinate with {Asher,Leslie} to do that IP subnet thang -- I want to get rid of that [bleep bleep] stupid NAt. [15:58:56] * Ryan_Lane nods [15:59:10] * Coren adds this to the etherpad for the techops meet [15:59:11] manybubbles: we replicate the databases from production to a set of labsdbs [15:59:15] manybubbles: like toolserver [15:59:22] Ryan_Lane: if there is documentation for how to point a local MW instance to it I'd love to use it [15:59:40] hm. I wonder if it's usable for a local mw [15:59:57] maybe it is, if you use a separate user table [16:00:44] Ryan_Lane: it'd save me a ton of time. [16:01:02] Ryan_Lane: You'd definitely need another auth, but mw otherwise works okay with readonly db [16:01:03] Coren: is that actually doable ^^ ? [16:01:07] cool [16:01:17] Except, of course, that it's readonly. :-) [16:03:45] https://bugzilla.wikimedia.org/show_bug.cgi?id=50923 <— heh [16:03:46] Well, actually, you don't even need the separate user table if you only want to read as anon. [16:03:51] Ryan_Lane and Coren: readonly would be mostly ok. I wouldn't really need to log in to test just about everything [16:04:05] !toolsdoc [16:04:06] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help [16:04:14] I guess it's not way less convenient to require confirmation for a reboot [16:04:24] Coren: yeah - I'm mostly going to be building a search index, searching, and reading [16:04:28] manybubbles: it's also redacted, so you'll have lots of info missing [16:04:37] (https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Production_replicas) [16:04:45] YuviPanda: less redacted then the dumps, probably. Thanks [16:04:46] no access to fulltext, for example. [16:04:54] YuviPanda: None that should be relevant; everything that could be gotten from the dumps is there. [16:05:08] Oh! Right! Actual text! [16:05:09] Coren: dumps have fulltext, does labs? [16:05:10] * Coren chuckles. [16:05:12] yeah [16:05:16] thaat :P [16:05:45] Yeah, for a search engine that's... not optimal. :-) [16:06:13] oh my - yeah, I need the text. [16:06:16] yeah, so I've no idea how much use labsdb will be for manybubbles. [16:06:17] heh [16:06:22] yeah. the replica won't help, then [16:06:25] however, gluster. [16:06:35] sounds like gluster is just a bad bad bad idea [16:06:41] yes. it is [16:06:56] if you make a large enough instance your /mnt will be large [16:07:08] manybubbles: You really need a local disk for this, not a networked FS [16:07:31] Coren: I suppose I can run it on my laptop [16:07:35] Ryan_Lane: Isn't, like, 160G the biggest? Does text /fit/ in that? [16:07:56] Coren: text should fit just about in that. it might be tight. [16:07:59] manybubbles: "local to the instance"; doesn't need to be spinning rust directly attached. :-) [16:08:00] no. and there's no reason anyone should import /all/ of rext anyway [16:08:07] *text [16:08:22] Ryan_Lane: True, he'd only need the latest revision. [16:08:26] (I think) [16:08:32] Ryan_Lane - yeah - just the latest revision [16:08:36] I don't want the whole table [16:08:46] that is crazy land - searching old revisions [16:09:21] * Coren could think of a few valuable use cases for being able to search old revisions, tbh, but it's not worth the trouble in the general case I expect. [16:09:38] it's dangerous eating that much local disk [16:09:45] yeah [16:10:04] is it necessary to import all of text? [16:10:29] Ryan_Lane: just all the current revisions. [16:10:37] which the dumps have [16:10:47] well, I'd say use NFS for this [16:11:00] Ryan_Lane: Do we already have an etherpad for today's meeting? [16:11:05] though that's going to be crappy for IO [16:11:36] Ryan_Lane: I'll give it a shot [16:12:52] Ryan_Lane: but it looks like I don't have the quota resources [16:13:07] manybubbles: that's gluster [16:13:19] you'd need to apply a puppet class for NFS [16:13:29] we'll eventually be switching everything [16:13:45] but right now we have both and you need to do something special to get NFS [16:14:16] Ryan_Lane: ah! I was trying to spin up a new instance with a big /mnt partition but it wanted too many cpus. [16:14:21] ah [16:14:34] Ryan_Lane: if I can switch /data/project/ to nfs I'd be happy with that [16:14:41] you meant for creating a larger instance [16:14:51] yeah [16:15:41] I certainly don't need 8 cpus but if it gets me the /mnt/ space I'll take 'em. [16:15:41] but I'll take whatever solution works. [16:17:42] Coren: what's tools-tyrant? [16:18:19] manybubbles: role::labsnfs::client <— if you add that class to your host, using "configure" it'll switch you to nfs [16:18:32] you'll need to add the class, force run puppet, then reboot the instance [16:18:58] the data isn't automatically transferred [16:19:16] is /mnt also on nfs? [16:19:21] no [16:19:29] /mnt is local to the virtual machine's host [16:19:36] cool [16:20:11] Ryan_Lane: uWSGI Emperor in tyrant mode. Since we already had tools-master, tools-slave, "tools-tyrant" was nicely thematic. :-) [16:20:14] heh [16:20:54] why not tools-emperor? :) [16:21:32] I liked the sound of tyrant better. :-) [16:21:43] heh [16:22:25] are there the latest dumps available? [16:24:41] fale: Look at /public/datasets/public. [16:24:47] scfc_de: thanks :) [16:32:23] Ryan_Lane: any chance you could try to sort the gerrit account for our OPW intern Rachel? I think Chad is on the road to SF right now. rachel99 is here in the channel right now if that's helpful. [16:32:45] what issues is she having? [16:32:47] Ryan_Lane: or would it make more sense for her to make an entirely new account in gerrit? [16:33:22] Ryan_Lane: email thread title "cannot assign user name" [16:34:44] isn't someone a backup for gerrit? [16:35:30] her shell account name is v-xxx? [16:35:44] Ryan_Lane: afaik yes [16:37:10] ryan_lane: I couldn't get login on Wed. so Chad did some update, which seemed to put my old email address back [16:37:11] I'm looking. of course if chad didn't get this working, it's unlikely I can [16:38:39] rachel99: does gerrit show an account id in your preferences? [16:39:50] it shows v-xxx [16:40:04] yes, but does it show an Account ID? [16:40:29] underneath "Registered" [16:40:35] in the profile section [16:41:38] Ryan_Lane: we're looking [16:42:09] 940 [16:42:27] Ryan_Lane: ^^ Account ID for Rachel [16:42:34] * Ryan_Lane nods [16:42:57] its set back to my old email address, so I can't see my commits/changes [16:44:54] what email address should be set, and what's set? [16:45:33] I changed to my new email rachelqa99 in Gerrit a few weeks ago, it appeared to work fine for several weeks. [16:45:59] I've read the emails [16:46:04] Then I couldn't log in on Wed. to gerrit. So I contacted Chad. He did some kind of update, not sure what. [16:46:41] So, my settings are back at rachelthomas_99@yahoo.com, my original email from several weeks ago AND I am not seeing my current commits. [16:46:45] I need to know the answer to the question I asked [16:47:09] what email address should be set, and what's set? [16:47:13] rachelqa99@gmail.com is what it should be, rachelthomas_99@yahoo.com is what it is now [16:47:18] ok [16:52:07] rachel99: log out of gerrit [16:52:46] I am logged out [16:59:06] rachel99: try logging in now [17:02:27] rachel99: ? [17:02:29] Ryan_Lane: error Cannot assign user name on login [17:03:14] rachel99: you are logging in with Rachel99, right? [17:03:15] Ryan_Lane: I may have had another window logged into Gerrit , my old account, while you were working. Not sure if that would have made a dfference [17:03:28] Ryan_Lane: yes with Rachel99 [17:03:58] ok. one sec [17:04:59] rachel99: try now [17:06:40] ok [17:08:07] I'm still getting same error message [17:11:59] hm [17:12:48] rachel99: try now [17:13:31] ok [17:14:45] same error: Cannot assign user name [17:15:09] hi Coren, wanted to touch base on the wikimetrics configuration in tools-login [17:15:13] i can't ssh into it atm [17:15:16] (just hangs) [17:16:50] rachel99: ah. I think I have it this time. try now [17:19:21] Great! It works. [17:19:49] However, it has my old email address, though all my commits are there. Should I just leave old email address? [17:21:53] Ryan_Lane: Should I stick with my old email address? Pls let me know if I can change it now, and have everything still work [17:22:15] what do you mean it has your old email address? [17:22:32] what email address do you actually want set? [17:22:41] Under settings, it has rachelthomas_99@yahoo.com [17:22:44] you can change the email address by usig wikitech [17:22:52] wikitech.wikimedia.org [17:23:01] That will change it for gerrrit? [17:23:02] go to your preferences, change your email address there [17:23:07] yep [17:23:11] it changes it in ldap [17:23:18] gerrit pulls its info from ldap [17:23:24] Ryan_Lane: thanks very much, we'll take it from there. much appreciated. [17:23:28] yw [17:23:35] thanks. [17:35:55] milimetric: It does? Lemme check. [17:36:20] yeah, still doing it to me [17:36:28] milimetric: I have no issue logging in, myself. What client do you use to connect? [17:36:37] just ssh from ubuntu [17:36:48] so i go ssh tools-login.pmtpa.wmflabs [17:36:52] eh [17:36:54] it's ok brb [17:37:55] milimetric: Ah, that'd be why. Try tools-login.wmflabs.org; it's not clear to me what the firewall rules to reach the private IP would be. [17:38:09] (At the very least, that'd isolate the problem) [17:59:51] hi guys, you can take a look here [17:59:51] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Mollywhite [18:00:44] Coren, ^ Ryan_Lane ^ [18:00:56] on it [18:01:13] done [18:01:42] Ryan_Lane, thank you [18:01:48] yw [18:13:49] OK Coren, ssh problem sorted [18:13:58] but now I see that Flask-Login is still the old version [18:14:07] you gotta wait for someone to merge your patch? [18:14:52] Coren usually +2's them himself :) [18:24:12] Coren: please update: https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/June [18:24:25] especially the metrics section for tools project [18:24:32] Ryan_Lane: kk. [18:24:36] I just added that line. it's good for us to show those stats :) [18:24:52] put in any other stats you'd want to show there [18:29:05] Some of the cooler stuff is hard to count. "We got bots running on more and more projects" is hard to quantify. [18:30:09] add Redis? :) [18:32:59] So, uh, are new project requests severely backlogged or are the old ones just contentious? [18:40:22] Is it against policy to ask for wiki user/passes in a tool? [18:41:06] Ryan_Lane: See above re: requests for new projects, maybe you're off somewhere else [18:41:28] a930913: You mean phishing? Pretty sure that's a little bad. [18:41:43] marktraceur: mostly contentious [18:41:59] Ah. [18:42:07] or have been rejected [18:42:10] a930913: it is, yeah. [18:42:14] a930913: wait for oauth :) [18:42:23] Ryan_Lane: I put one in for bawolff, I think I may drive one more to you, should I just ping you about those individually? [18:42:40] it's good to submit them [18:42:45] so that we can discuss them on wiki [18:42:56] marktraceur: multimedia is a new team? [18:43:06] "new" in that there are people actually in it now [18:43:12] *shrug* [18:43:14] heh [18:43:20] If there's an existing project we'd take that [18:43:51] And yeah, I'll have them submitted, I just need to know whether you should be pinged instead of just waiting [18:51:57] marktraceur: ah. yeah, pinged is good [18:52:07] in a meeting right now. I'll look at this in a bit [18:52:52] YuviPanda: Snuggle is asking for wiki login. [18:53:04] *nod* ta [18:53:04] a930913: they got clearance from legal, IIRC [18:53:13] marktraceur: oh, you're in the multimedia team now? [18:53:14] nice [18:53:21] YuviPanda: "in" [18:53:29] YuviPanda: I'm "in" fundraising still [18:53:38] you're in... fundraising? [18:53:38] I'm going to be "in" Multimedia next week [18:53:42] i thought you were in parsoid [18:53:49] I was in Parsoid before. [18:53:58] Life Is Complicated (TM) [18:54:04] of course [18:55:08] YuviPanda: How'd they pull that one off? :o [18:55:44] a930913: unsure, ask Ryan_Lane :) [18:56:04] pull what off? [18:56:49] Ryan_Lane: Apparently Snuggle has clearance to ask for wiki user/passes. [18:57:08] they actually don't [18:57:30] oh? [18:57:34] i remember conversation here about that exact thing... [18:59:37] yeah, and the result was that it wasn't going to be given an exception [18:59:58] and that launch would wait for oauth [19:08:46] So who's going to shut it down? :p [19:33:13] a930913: bleh. it's still running that way? [19:33:22] a930913: you're tyler, right? [19:34:22] ... something on labs is asking for U/P pairs? [19:34:27] yes [19:34:53] * Coren facepalms. [19:35:10] yeah, was brought up internally a while ago [19:50:01] hey Coren; question: how likely will it be to have mod_wsgi support for apache in the tools project by the end of this week? [19:50:26] drdee: hey! no mod_wsgi for us. Coren said that the code sucks, etc. [19:50:43] drdee: he's working on a uwsgi emperor / tyrant based server for python wsgi apps though [19:50:50] * YuviPanda goes to update bz [19:51:20] mod_wsgi is the default way in python world to deploy web apps, what does it mean that the code 'sucks'? [19:51:37] not really - nobody I know of uses mod_wsgi + apache. mostly it is uwsgi + nginx [19:51:40] s/deploy/run [19:51:42] (or gunicorn + nginx) [19:51:57] Coren might be able to fill you in on the details, I'm just repeating what he said :) [19:52:26] WSGI is the standard for deploying python apps, no? it shouldn't matter if it is mod_wsgi or uwsgi [19:53:54] drdee: By the end of this week is possible, but optimistic. More likely in the middle of next. [19:54:14] drdee: Unless it's a blocker on a project, then I can reshuffle priorities around. [19:54:17] Coren: can you also update https://bugzilla.wikimedia.org/show_bug.cgi?id=49058? [19:54:58] for milimetric, myself and the wikimetrics team, it's a hard dependency as we have to ship by july 24th [19:55:30] drdee: you can already run python stuff with WSGI CGI Wrapper. I've a few tools running already that way. [19:55:32] drdee: Development can proceed with a CGI wedge without affecting functionality in the meantime. [19:55:33] a930913, YuviPanda: Isn't Snuggle not one of a930913's projects and thus a Catch-22 of the different kind? [19:55:33] with flask it is fairly simple [19:55:53] k [19:55:55] Coren: so, yeah, snuggle doesn't run on labs [19:56:01] currently [19:56:08] so we can't necessarily control it [19:56:41] Ryan_Lane: It's still evil, but at least it's not /our/ evil. :-) [19:56:51] YuviPanda can you point me to one or two of your examples? [19:56:57] drdee: sure! moment [19:57:04] k [19:57:49] drdee: https://github.com/addshore/dumpscan/blob/master/dumpscan.py is the CGI/WSGI wrapper [19:57:56] ty [19:58:07] drdee: it uses virtualenv, but you can get rid of the first three lines if you don't want virtualenv [19:58:30] and this runs on nginx? [19:58:38] drdee: no, this runs on apache [19:58:41] k [19:58:46] drdee: so currently this runs on apache -> CGI [19:58:50] drdee: soon it'll be apache -> uwsgi [19:59:13] ty sir [19:59:36] drdee: the files need to be owned by the tool, and must have the execute bit set. [20:03:19] drdee: There is a per-request performance hit for this workaround, which is why WSGI is on my list, but that should hold you over until next week. :-) [20:03:58] Coren: what is the position of this on your todo list? [20:04:02] that's fine, the rollout is to a very small group initially [20:04:06] and what is the position of 'puppetize everything'? [20:04:36] drdee: any particular reason this is being deployed on toollabs instead of production? [20:05:34] YuviPanda: WSGI? It's sharing the top spot of "longer things" right now, vying for attention with the small-but-important stuff done in parallel. [20:06:14] so... the list has now sortof become a bag... [20:06:34] YuviPanda: "No battle plan survives contact with the enemy". :-) [20:06:43] :D [20:35:42] Coren, any word on the Flask-login upgrade? [20:36:04] scfc_de: petan ^ can you upgrade flask-login on webservers (apt?) [20:36:47] (I still think not using virtualenv is a bad idea) [20:37:21] YuviPanda: we *can't* use virtualenv as it's not allowed in production [20:37:41] I would of course use it otherwise [20:37:47] milimetric: ... wait, that didn't go through? [20:37:56] true true, I just think not using that in production itself is bad :( [20:37:56] no, it doesn't seem to have [20:38:03] milimetric: Darn, the package must be held at the old version. I'll force update. [20:38:22] milimetric: but we've been through this before, so :) [20:38:33] :) [20:38:43] yeah, I'll brb, restarting [20:42:17] Ah! Interesting. For some reason, the deitro version has higher priority. [20:44:25] distro* [20:44:52] milimetric: There. I forcibly installed the version without the cobwebs. :-) [20:44:58] thanks Coren [20:45:01] i'll give it a shot [20:45:33] Version: 0.2.4-1 [20:47:01] /usr/bin/python: cannot import name promise [20:47:07] It's a dependency of Celery's [20:47:30] so I think my approach to this sounds pretty sane: [20:47:40] 1. pip install the dependencies [20:47:48] 2. freeze those dependencies in a separate repo [20:48:03] 3. bless those dependencies and any updates through them via gerrit [20:48:11] 4. pip install from that repo in production [20:48:31] milimetric: agreed. [20:48:45] this seems to me better, safer, and incredibly easier than the other ways we try to jump through the npm/pip/mvn hoop [20:48:56] +1 [20:49:00] good afternoon :) [20:49:12] hello, mistress of the network equipment! [20:49:23] i was thinking the same thing... [20:49:32] so Coren, here's what I'm doing [20:49:34] hehe [20:49:42] python -m wikimetrics.run [20:49:51] network virtualization? SDN? [20:49:55] so, what's the problem/end goal ? [20:49:57] that would run the project if all the dependencies were OK [20:50:25] jeremyb: morebots is programmed to call her that [20:50:37] YuviPanda: hence why i was thinking it! [20:50:44] ah, of course ;) [20:53:17] milimetric: allright. So you got more missing dependencies. :-) As you saw, adding new stuff is simple enough if I have a list -- you might want to keep that list up to date though. :-) [20:54:49] no Coren, I think that was just a recursive dependency issue [20:55:03] my list didn't change, promise is a dependency of celery [20:55:26] Ah. Hm. My build process can't actually find those up. [20:55:44] yea, I'm not sure we should try... [20:55:45] It's a limitation of py2deb, I believe. [20:55:49] LeslieCarr: currently we have a number of IPs NAT'd from the client to represent each different database [20:55:50] I mean that's kind of re-inventing pip [20:56:03] we want to move the IPs to the servers and NAT on that side [20:56:16] wait, let me rephrase [20:56:18] milimetric: Which is the point given that we don't want pip in production. Once we enumerated the whole list, we've "automagically" got packages for the whole thing. [20:56:25] we want to port forward on the db side [20:57:01] so, we just need to have a range of IPs on the systems, with filtering rules relaxed for them [20:57:05] pip should be totally fine in production as long as it's pulling from our blessed gerrit repo [20:57:14] and putting stuff in gerrit with pip should be fine too [20:57:35] on the labs databases ? [20:57:53] I think I will more eloquently argue my position on the list whenever my managers tell me I'm not under the gun :) [20:58:00] LeslieCarr: yep [20:58:05] Coren: ^^ accurate? [20:58:09] LeslieCarr: Basically. The problem is that right now each shard is available through IP/port pair; whereas every tool presumes that an IP uniquely identifies a shard. [20:58:55] so do you need public ip's or would private ip's suffice ? [20:59:17] Private IP would do nicely; I'm using 192s with the ugly NAT hack ATM [20:59:31] eww [20:59:31] So, Coren, I think the solution right now has to be: access the labsdb slaves from a separate labs instance. [21:00:13] so, we can give the machines more ip's in the 10.64.37.0/24 range super easily [21:00:14] LeslieCarr: Chosen specifically so that they cannot clash with your real IPs. They never leave the box. [21:00:47] LeslieCarr: Ideally, giving them a /28 or so to future proof and have just one rule to ring them all. :-) [21:01:10] milimetric: That's actually going to become easier this afternoon. :-) [21:01:33] cool :) [21:01:47] I'm ok waiting a bit for that, it seems like a low effort path, right? [21:01:47] ok, so they should never leave the box and/or be externally reachable ? [21:02:08] because we could also do it such that each machine just has several (private) ip's [21:02:12] LeslieCarr: No, that was my hack -- you can do normal stuff for the "real" deal. [21:02:19] ok [21:02:21] cool [21:02:49] (I just wanted to make sure that I didn't pollute your network so I NAT'ed only from 192s and filtered anything I didn't nat) [21:03:10] :) [21:06:25] so labsdb1001-1003 each need 8 extra ip;s ? [21:08:57] Actually, a group of 16 that we can move around all three would be best; so that we can move shards from one to the other at need, and add. [21:12:29] ok, so i can allocate them and we can use puppet and the interface_ip command [21:12:41] * Coren nods. [21:12:48] like shown in nescio 's site.pp entry [21:13:52] ah, the bigger problem is, can we have separate ip entries for the row b and row c labs databases ? [21:14:23] because that turns it from an easy "machines can arp and we don't have to do anything on the network level" to a "have to actually pay attention to which machines have what ip's" [21:14:38] LeslieCarr: We /can/, having them contiguous would just have been simpler management. [21:14:56] Oh, but that'd mean we couldn't move an IP from one row to another, wouldn't it? [21:15:36] exactly [21:16:02] It would have been doubleplusgood if we could just point the IP wherever when we move databases around; but it's not a requirement and we can work around it through DNS. [21:17:06] I don't expect moving stuff around is going to be common. [21:19:18] Speaking of, Ryan_Lane, I've got a pretty little script that consumes *.dblist and creates a list of DNS RRs; my understanding is that we want to stuff them in labs LDAP? Where should that script run from? [21:19:30] virt0? [21:19:48] yeah, that's a good place for it [21:19:59] let me guess…. it's written in perl? :) [21:20:20] Ryan_Lane: No, actually, it's ugly python. I'll push it to gerrit later tonight so you can ridicule it. :-) [21:20:24] hahahaha [21:20:27] \o/ [21:20:48] Hey, I did promise. I've been reading up on python since AMS. [21:21:00] sweet [21:21:17] (And none of what I've been reading makes me hate it any less -- in fact, the general attitude of documentation makes me loathe reading more of it) [21:21:28] Just for the record (*ping* petan): [21:21:28] (23:15:44) Steinsplitter: @add #wikimedia-commons-admin [21:21:28] (done) [21:21:37] python documentation is pretty good overall [21:22:00] Oh, it's good in substance -- I didn't ever expect to have to read documentation that was /condescending/ however. :-) [21:22:01] have you accessed docs within an interactive shell? [21:22:11] :D [21:22:34] Coren: http://pastebin.com/NgADrz7R ? does that look good ? [21:22:41] have you looked at *mediawiki* documentation? :) [21:23:19] LeslieCarr: I wuw you! :-) [21:23:21] fyi that would be for 10.64.20.101-108 for row b and 10.64.37.101-108 for row c [21:23:23] :) [21:23:24] yay [21:24:54] YuviPanda: Heh. I find that RTFS is the only decent mw documentation. Except for the contents of LocalSettings, which /tends/ to be fairly well documented, it suxx0rz. [21:25:23] LeslieCarr: Also, your selection for CT's gift seems to have been well received. Yay! [21:25:24] [bz] (8RESOLVED - created by: 2Krinkle, priority: 4Normal - 6normal) [Bug 49373] [Regression] Labs: Logo should be transparent - https://bugzilla.wikimedia.org/show_bug.cgi?id=49373 [21:26:07] yay :) [21:26:22] omg the colors of wm-bot's notifications hurt [21:27:03] Coren: the dns change/reservation is checked in - if you want me to review any puppet manifests (or give help) when you know which ip you want where, i'd be more than happy to :) [21:27:07] there's colors? heh. my client doesn't display them :) [21:28:04] LeslieCarr: Will do. I'll probably have a changeset for you tomorrow, but I'll want Asher's hand in it too so that the DBs actually /listen/ on them IPs; that may turn out to be useful as well. :-) [21:28:50] Ryan_Lane: Or wait, did we decide to keep the DBs themselves as-is and to incoming portmapping through iptables? I know we discussed both, I don't remember which one we picked. :-) [21:29:02] either will work [21:29:04] cool :) [21:29:22] I think you should discuss changing the db config with binasher, if you want to go that route [21:29:31] I think port forwards is simplier [21:30:03] well, maybe not since we don't have a good way to handle it in puppet [21:30:20] Ryan_Lane: Actually, we really should have an iptables class for the general case. [21:31:49] Ryan_Lane: I'll check with Asher. If he prefers not fiddling with the listening ports, I'll make an iptable class so we can properly puppetize this. [21:32:51] s/class/module/ [21:33:23] LeslieCarr: Thanks again! [21:33:26] * Coren hungers! [21:34:35] nom [21:39:04] ther seems to be a bug, petan ar you here? [21:39:40] ok, i am heading out of this channel :) [21:41:05] (23:34:26) Steinsplitter: @RC+ commons Commons:Administrators' noticeboard/Blocks and protections [21:41:05] (23:34:27) wm-bot: Inserted new item to feed of changes [21:41:05] (23:34:34) Steinsplitter: @RC+ commons Commons:Administrators' noticeboard/Vandalism [21:41:05] (23:34:35) wm-bot: There is already this string in a list of watched items [21:41:08] bug^^ o_O? [21:41:15] There is already this string in a list of watched items o_O [21:42:47] [bz] (8ASSIGNED - created by: 2Yuvi Panda, priority: 4High - 6normal) [Bug 49058] Support WSGI for Running Python Scripts - https://bugzilla.wikimedia.org/show_bug.cgi?id=49058 [21:50:22] anyone know why i might not be able to ssh from bastion into my instance? i haven't changed any configs for entry, but for some reason, today i can't get in. ssh'ing from bastion gets stuck at offering the rya pub key [21:51:28] Is bad running a pywiki bot with a post - throttled lower than 5 from tool labs? [21:54:48] problem done: using "_", yeah! :) [21:58:47] i LOVE wm-bot. Amazing helpful bot :) [22:06:57] dan-nl, are you still having access problems? [22:07:44] andrewbogott: yes [22:08:06] dan-nl, what project and instance? [22:08:12] And, I take it you've had this working in the past, right? [22:08:29] andrewbogott: last line of the ssh says it's offering the rya pubkey … yes, it was working fine before using the -A option to forward [22:08:50] getting the instance info [22:09:24] andrewbogott: https://wikitech.wikimedia.org/wiki/Nova_Resource:I-000004b1 [22:11:26] dan-nl: OK, I'm looking. So far I can't log in either [22:11:41] ah, okay, i guess that's a good sign ... [22:13:15] sort of :) [22:13:52] Krenair: https://gerrit.wikimedia.org/r/#/c/68126/ <— fixed echo issues, I believe [22:14:15] andrewbogott: i tried rebooting it about an hour ago, but that didn't help ... [22:15:23] andrewbogott: testlabs-buildtest <— I'm going to use this instance as a resize guinea pig. ok? [22:15:49] Ryan_Lane: Sure, I haven't used it in quite a while. You can delete it when you're done. [22:16:01] ok. cool [22:16:32] dan-nl: Looks like that instance is pretty broken… I can't tell quite what's up. [22:17:19] Ryan_Lane, have a look at the console log for glam-gwtoolset. Network failure of some sort? [22:18:41] when was the last time this instance worked? [22:18:44] andrewbogott: okay, so does that mean i should delete and re-create? [22:19:09] dan-nl: don't do that yet [22:19:17] andrewbogott: I see ipv6 issues, but that's it [22:19:29] Ryan_Lane: maybe the last time i logged in was two weeks ago [22:19:39] oh, you're right. red herring [22:19:41] Ryan_Lane: the web service us fine [22:20:12] it pings [22:20:23] http://gwtoolset.wmflabs.org/index.php/GWToolset [22:23:30] I'm betting it's OOM'd in some way [22:23:33] trying a hard reboot [22:26:00] seems resize is still working fine [22:26:27] it won't work if we have differently sized / partitions between flavors, though, so we need to be careful [22:27:12] Having a weird issue. Locally (outside Labs): [22:27:13] curl 'http://toro.wmflabs.org/w/api.php?format=json&action=query&meta=siteinfo&siprop=namespaces%7Cnamespacealiases%7Cmagicwords%7Cfunctionhooks%7Cextensiontags%7Cgeneral%7Cinterwikimap%7Clanguages%7Cprotocols' [22:27:15] works fine. [22:27:29] However, I can't run it on toro (the same maching it's requesting from). [22:27:36] It gives "couldn't connect to host" [22:27:44] This is a blocker for Parsoid. [22:27:59] superm401: you need to use parsoid's internal hostname or IP [22:28:11] if you are inside of labs [22:28:24] which should be parsoid-spof.pmtpa.wmflabs [22:28:30] The external hostnames aren't routable at all inside Labs? [22:28:36] Ryan_Lane, not trying to connect to that parsoid. [22:28:42] This is a test wiki, so it has its own. [22:28:55] Unless that one works for toro's own wikitext. [22:29:01] then you need to use toro's internal [22:29:46] there's something with the NAT config that makes this not work [22:30:13] and it's apparently a normal issue in OpenStack deployments. mark briefly looked at the issue and wasn't able to determine the cause [22:30:46] superm401: I came around that on beta using iptables [22:31:17] Ryan_Lane: thanks for trying the hard reboot … just tried to ssh from bastion, but no luck yet [22:31:19] hashar, hmm, it would be great if it Just Worked throughout Labs. [22:31:21] superm401: while doing the curl locally, you could ask the command to proxy its request to localhost: curl -x localhost:80 http://public hostname of the instance you are on/foo [22:31:34] Unless there's a security reason of course. [22:31:49] hashar, the curl was just for testing. [22:31:55] Looks like I just need to use toro.pmtpa.wmflabs for the real Parsoid config. [22:31:57] Trying that now. [22:31:58] superm401: the nice way to solve it is to have DNS split horizon. Aka the DNS server would reply with a different IP address depending on the source address. [22:32:13] or you could use the internal hostname indeed :-] [22:33:13] That works. Thanks to you both. [22:35:13] dan-nl: hm [22:35:37] that makes no sense [22:35:53] dan-nl: did you modify the fstab on that instance? [22:36:06] nothing that i know of [22:36:18] ok. let me check it by mounting its disk [22:36:21] i mostly go into the www and run a git pull and that's usually all [22:37:23] nothing weird in the fstab [22:37:41] commenting out /mnt for now [22:38:33] ah [22:38:35] / is full [22:38:44] oh shit [22:38:46] worse than that [22:38:56] ah [22:39:39] are we uploading too much? [22:39:42] /var/lib/nova/instances is full on virt5 [22:42:05] seems people have been going crazy with /mnt [22:42:38] k, have no idea what that is ... [22:45:47] fuuuuuuccckkk [22:45:58] dan-nl: so, I know you're going to hate me [22:46:10] I just accidentally deleted the wrong instance, and it was yours.... [22:46:27] Ryan_Lane: oh boy … lol [22:46:44] Ryan_Lane: don't hate you for it … we all make mistakes [22:47:10] Ryan_Lane: the important thing is that it's not a mistake on a person ;) [22:47:33] Ryan_Lane: so i'll need to re-create the instance and re-set-up everything? [22:48:09] yeah :( [22:48:10] sorry [22:48:22] Ryan_Lane: k, is the server okay now? [22:49:29] Ryan_Lane: i'll run through the set-up process tomorrow [22:50:30] Ryan_Lane: thanks for at least checking and trying … i should have most of the set-up documented so it'll give me a chance to run through it again and double-check it [22:50:48] ok. cool. sorry about the screw up [22:51:03] Ryan_Lane: i understand ... [22:51:42] Ryan_Lane: And thus the case is strengtened for the "confirm deletion" step. :-) [22:52:12] Coren|Dinner: confirm deletion step is there [22:52:21] I didn't pay enough attention to what I was doing [22:52:28] confirm reboot is what's currently missing ;) [22:52:35] Oh, right. [22:52:52] Then we need to special case your user account so that it asks "Are you /really/ sure?" :-P [22:53:36] solr instances are eating a shitload of disk space [22:53:52] virt5 is out of disk space [22:54:01] Don't feel bad. I once dropped a production database thinking it was the staging one. Restoring from a week-old backup plus binlogs while a large team was wating to have their bugtracker back so that they could work during a beta crunch was was "fun". :-) [22:54:04] virt6 is at 91% [22:54:26] I'm going to cold migrate some instances [22:55:20] [bz] (8REOPENED - created by: 2MZMcBride, priority: 4Normal - 6normal) [Bug 36885] wmflabs.org does not resolve - https://bugzilla.wikimedia.org/show_bug.cgi?id=36885 [22:58:21] hm. and somehow virt5 just recovered 177G? [22:58:29] ah. I wonder if they are swapping [22:58:36] * Ryan_Lane grumbles [22:58:59] no way it's swapping that much though [22:59:04] The only confirmation that ever worked for me was cryptsetup's (?) "Are you sure? (Type uppercase yes):" :-). [23:00:03] bleh. I really don't want www.wmflabs.org being handled by some random instance [23:00:24] which is why I removed it the last time [23:18:50] chrismcmahon: Have you seen https://en.wikipedia.org/w/index.php?title=Wikipedia:Village_pump_%28technical%29&diff=563175054&oldid=563172242 ? Someone asking "How do I volunteer to test?". [23:19:24] thanks anomie|away looking now... [23:22:02] anomie|away: :) [23:35:54] scfc_de: You should see the confirmation dialog I have for firing up the NFS server. :-)