[00:16:11] <Ryan_Lane>	 bd808: yeah, probably
[00:16:58] <Ryan_Lane>	 bd808: looks like it's there to me
[00:17:13] <Ryan_Lane>	 hm. or not
[00:17:20] <bd808>	 Ryan_Lane: I just nuked the instance and started over without the nfs role. 
[00:17:32] <Ryan_Lane>	 there's no gluster there either, though
[00:17:34] <Ryan_Lane>	 which is weird
[00:18:27] <bd808>	 The project is brand new as of today so I guess there could be some weirdness from that
[00:19:49] <andrewbogott>	 bd808: what's the project name?
[00:20:06] <bd808>	 andrewbogott: wikimania-support
[00:20:40] <andrewbogott>	 bd808: OK, yeah, shared home dirs aren't turned on currently.  I'll enable that
[00:20:50] <andrewbogott>	 And, do you want shared project storage as well?
[00:21:16] <bd808>	 andrewbogott: Sure. That may make things easier to iterate on
[00:21:43] <andrewbogott>	 ok -- may take a few minutes for things to get set up.
[00:22:08] <bd808>	 andrewbogott: no worries. I'm about to call it a night and start again in the morning
[00:22:35] <bd808>	 8:30-18:30 is a full day for me ;)
[00:23:16] <andrewbogott>	 Or for anyone I would hope
[01:05:56] <Coren>	 andrewbogott: You must be new here.
[01:06:15] <andrewbogott>	 yep!  Still working, too
[01:07:16] <Coren>	 The point, she is well made.  :-)
[12:44:12] <grrrit-wm>	 (03PS1) 10Zfilipin: Ping #wikimedia-dev and #wikimedia-qa when mediawiki/selenium is updated [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/90537 
[13:21:58] <T13|needsCoffee>	 Anyone here that has AWB in front of them that can give another user a walk through of how to make a list of user talk pages transcluding a template? (Maybe Reedy?)
[13:30:05] <Pratyya>	 !helper
[13:30:18] <Pratyya>	 Hey guys can't I edit user talk pages with AWB? When I use What links here to [[Template:uw-vandalism1]] it brings 0. But if you see in WP it's more than 500. why is this problem? 
[13:32:03] <Yarl>	 hi, I need someone with access to monuments database
[13:39:20] <Pratyya>	 @helper
[13:39:20] <wm-bot>	 I am running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.20.2.0 my source code is licensed under GPL and located at https://github.com/benapetr/wikimedia-bot I will be very happy if you fix my bugs or implement new features
[13:39:29] <Pratyya>	 !help
[13:39:30] <wm-bot>	 !documentation for labs !wm-bot for bot
[13:39:40] <Pratyya>	 @help
[13:39:40] <wm-bot>	 I am running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.20.2.0 my source code is licensed under GPL and located at https://github.com/benapetr/wikimedia-bot I will be very happy if you fix my bugs or implement new features
[13:39:48] <Pratyya>	 !helper
[15:21:11] <csteipp>	 Hey hashar: I'm going to restart the apaches on beta in just a few minutes, after I update the apache configs to not redirect to https. 
[15:21:49] <csteipp>	 (just fyi)
[15:23:04] <Coren>	 csteipp: Speaking of, you guys, the test wikis really should have the Labs wiki disclaimer somewhere visible, *especially* if you de-https them.
[15:23:05] <hashar>	 csteipp: thanks :-]
[15:23:56] <Coren>	 https://wikitech.wikimedia.org/wiki/Wikitech:Labs_Terms_of_use#If_my_tools_include_a_beta_or_test_wiki...
[15:24:42] <Coren>	 Also, the account creation should sport the notice at https://wikitech.wikimedia.org/wiki/Wikitech:Labs_Terms_of_use#If_my_tools_have_account_creation... if it doesn't.  Does it?
[15:51:23] <bd808>	 Coren: I'm still having issues with role::labsnfs::client on instances in the wikimania-support project. `Unable to create and initialize directory '/home/bd808'` which I take to mean that the NFS mount isn't present.
[15:52:03] <bd808>	 scholarship-alpha.pmtpa.wmflabs is the current instance I have running with the problem
[15:52:06] <Coren>	 bd808: You're correct; there seems to be something wrong with the exports for new projects.
[15:52:38] <Coren>	 Hm.  Apparently, andrew's management script doesn't like its new home.  Give me a minute, it's probably something trivial.
[15:53:07] <bd808>	 Coren: Excellent. I'm happy as long as it's not my fault :)
[15:53:24] <Coren>	 Yeah, logs shows it keeps dying for some reason.
[15:54:15] <bd808>	 I've got a ton of code review to do this morning so I'll just check back on it later. Thanks for looking.
[16:03:51] <Coren>	 bd808: Ah, there was a permission problem that prevented the daemon from updating the list of exports.  fix't
[16:04:55] <Coren>	 bd808: Restarting autofs will have it see the filesystems (or, in a pinch, a reboot will also do it)
[16:05:13] <bd808>	 Coren: rebooting now since I couldn't ssh in
[16:16:27] <bd808>	 Coren: Same error after rebooting. Also spun up a new instance and it's not getting gluster now.
[16:16:56] <Coren>	 ... that's odd, I did test it.  What's the instance name?
[16:17:43] <bd808>	 scholarship-alpha is the one that seems to not have NFS. scholarship-bd808 is new instance that didn't get Gluster
[16:18:11] <bd808>	 Coren: gluster is there now?
[16:18:25] <bd808>	 Maybe I can't read today
[16:19:33] * Coren  looks into it

[16:20:01] <bd808>	 Trying to add nfs to scholarship-bd808 instance to see if that works
[16:21:51] <Cyberpower678>	 Coren, ding dong
[16:22:21] <Cyberpower678>	 Coren, ping pong
[16:23:10] <Cyberpower678>	 Coren, I swear you're here.
[16:23:22] <Cyberpower678>	 Or not.
[16:23:38] <Cyberpower678>	 petan, any clue on deleted edits.
[16:24:01] <petan>	 what u talk about
[16:24:20] <Cyberpower678>	 You know.  Same thing I always pester Coren about.
[16:24:25] <Coren>	 bd808: Don't yet - there is something funky going on at the network level I don't get -- for some reason your instances can't seem to talk to the NFS server.
[16:24:30] <Coren>	 Cyberpower678: Patience, grasshopper.
[16:24:40] <bd808>	 Coren: Too late :) Not a big deal
[16:24:40] * bd808  shoots some broken instances in the head

[16:25:00] <Cyberpower678>	 Coren, I am being patient.  But it's been 2 weeks since I last asked about it.
[16:25:11] <Cyberpower678>	 And mid-october has come.
[16:25:51] <Coren>	 And you are under the impression that the more often you ask, the faster the production schema change will come?
[16:26:26] <Cyberpower678>	 Coren, yes.  Yes I do. :p
[16:27:10] <Cyberpower678>	 Actually, I only ask to get updates on it's progress and since I haven't asked in 2 weeks, I decided to get an update.  I want to finally shut down the tool server version.
[16:27:27] <Coren>	 Well, given that even if it worked in principle I wouldn't be the right person to pester...  :-)
[16:28:57] <Cyberpower678>	 Where can I find the right person then?
[16:29:04] <Coren>	 bd808: Looking at the network now, trying to figure out what packets get lost.
[16:29:40] <bd808>	 Coren: Okey doke. Back to code review screens for me.
[17:02:05] <yurik_>	 hi, how long does it usually take for beta-labs to pick up settings update from the master?
[17:02:17] <YuviPanda|zzz>	 yurik_: theoretically, immediately
[17:03:52] <liangent>	 toollab died again?
[17:04:02] <liangent>	 Coren: petan ^
[17:04:38] <Coren>	 liangent: Minor stalls while I'm working on NFS; usually less than 90s but that last one lasted ~2m
[17:05:32] <Coren>	 Shouldn't affect anything but performance, but it's entirely possible that some web queries time out.
[17:05:47] <Coren>	 I should be done in <15m
[17:06:07] <liangent>	 Coren: well my mosh connection froze
[17:06:15] <liangent>	 .. and seems back now
[17:06:42] <Coren>	 Right; file systems reads would have been really slow for a few minutes.
[17:06:46] <petan>	 what
[17:06:49] <petan>	 liangent?
[17:07:52] <Coren>	 bd808: Actually, all your problems seem to be caused by the same strange issue and only your project seems affected.  I'm trying to find a workaround now.
[17:08:09] <liangent>	 petan: coren already answered that :)
[17:08:43] <liangent>	 I guess I shouldn't ping more than one person next time
[17:08:57] <liangent>	 and only when the first doesn't respond for some time
[17:09:37] <bd808>	 Coren: My project was created yesterday, so … that may explain why it's the only one effected by weirdness
[17:11:44] <Coren>	 bd808: I think I managed to work around it.  Try now?
[17:14:00] <Coren>	 (For some reason, your project homes were created oddly, then incorrectly exported and the NFS server insisted on exporting them wrong until I renamed the old one out of the way and created them anew)
[17:19:34] <bd808>	 Coren: \o/ I can log in and /home is nfs
[17:20:08] <Steinsplitter>	 tools dos not load. offline? o-O
[17:20:25] <Steinsplitter>	 no. only sloq
[17:20:27] <Steinsplitter>	 *w
[17:20:32] <yurik_>	 anyone to +2 labs settings https://gerrit.wikimedia.org/r/90570   please
[17:20:34] <yurik_>	 thx :)
[17:24:22] <yurik_>	 thanks Coren !  now i hope it goes into betalabs quickly enough :)
[17:42:00] <renoirb>	 Hey shepazu 
[17:42:09] <shepazu>	 hey, renoirb 
[17:42:35] <shepazu>	 I just asked this on #mediawiki, but this might be a better place
[17:42:36] <shepazu>	 does anyone have experience with using edge-side includes (ESI) with mediawiki?
[17:42:59] <shepazu>	 Ryan Lane suggested that we use them for a project on webplatform.org
[17:43:52] <shepazu>	 Ryan_Lane, speak of the devil
[17:44:01] <Ryan_Lane>	 :D
[17:44:06] <Ryan_Lane>	 howdy
[17:44:17] <shepazu>	 howdy
[17:44:30] <shepazu>	 looking at the compatibility tables extension again
[17:44:51] <shepazu>	 trying to reload in my brain the problem and solution sets for the caching of generated content
[17:45:25] <shepazu>	 Ryan_Lane, can you refresh my memory on what we need, and why ESI is the golden ticket?
[17:46:32] <shepazu>	 (edge-side includes)
[17:47:10] <Ryan_Lane>	 sure. the compat tables extension needs to be loaded in every single page, basically
[17:47:15] <Ryan_Lane>	 and it changed weekly or so
[17:47:20] <shepazu>	 right
[17:47:33] <Ryan_Lane>	 if you just embed it as a normal extension, you'll purge the entire cache once a week
[17:47:56] <shepazu>	 right
[17:48:30] <Ryan_Lane>	 rather than embedding the compat tables directly, you can make a special page that takes inputs and gives out the table
[17:49:08] <Ryan_Lane>	 the extension could then output ESI into the page contents, pointing at the special page (with the proper parameters)
[17:49:33] <Ryan_Lane>	 which would allow you to never purge the page cache, but the special pages would be purged instead
[17:49:45] <yurik_>	 Ryan_Lane: ESI is still dead in varnish, no idea why :)
[17:49:46] <renoirb>	 I was thinking of generating a static html document then including it
[17:49:58] <Ryan_Lane>	 yurik_: this is for webplatform.org ;)
[17:50:08] <yurik_>	 oh, nvm :)
[17:50:09] <Ryan_Lane>	 renoirb: including it how?
[17:50:11] <Ryan_Lane>	 iframes?
[17:51:16] <renoirb>	 instead of calling curl and working like it is in the extension. We fork with a condition to check if a generated version is young enough, if yes, read the file. Otherwise, generate a static file, then include it via simple php include.   
[17:51:36] <shepazu>	 Ryan_Lane, but how would the extension know to generate the page? it needs to read each target page to get the name of the feature that it's generating the table for, right? 
[17:51:38] <yurik_>	 does anyone know where all the betalabs server names are and how they are configured? Basically looking for where deployment-cache-mobile01 and other servers are described, which roles they have, etc
[17:51:40] <Ryan_Lane>	 it can't just be a php include
[17:51:41] <renoirb>	 Make sure the MW page has appropriate HTTP headers to NOT cancel caching with varnish
[17:51:49] <Ryan_Lane>	 you need the actual content being delivered to be static
[17:52:11] <renoirb>	 https://github.com/webplatform/mediawiki/blob/master/extensions/CompaTables/compatables.php
[17:52:22] <Ryan_Lane>	 shepazu: {{compattables:<params>}}
[17:52:27] <Ryan_Lane>	 that goes into the wiki page
[17:52:34] <shepazu>	 ok
[17:52:48] <Ryan_Lane>	 it generates ESI that says: load from /wiki/Special:CompatTables/params
[17:53:08] <shepazu>	 Ryan_Lane, you know of anyone who could help us make this extension?
[17:53:09] <Ryan_Lane>	 renoirb: the HTML that is sent to the client needs to be static
[17:53:19] <renoirb>	 https://github.com/webplatform/mediawiki/blob/master/extensions/CompaTables/compatables.php#L107 this could be between ob_start();, then sent to a file instead. Then do a check on the age of the file.
[17:53:24] <Ryan_Lane>	 shepazu: wikiworks or hallowalt
[17:53:46] <Ryan_Lane>	 renoirb: you're thinking of the problem from the wrong place
[17:54:01] <renoirb>	 sure Ryan_Lane, no client side generation here.
[17:54:27] <renoirb>	 but the user triggers the view, MW will make sure it is still valid, or regenerate it. so stale data is prevented and no cronjob to generate pages.
[17:54:27] <Ryan_Lane>	 including the compat tables HTML into the wiki HTML output caches all of it together in the varnish frontend cache
[17:54:45] <shepazu>	 Ryan_Lane, anyone else? ;)
[17:54:49] <Ryan_Lane>	 whenever you change the compat tables, the frontend cache is invalid
[17:54:58] <Ryan_Lane>	 shepazu: umm. one sec
[17:55:20] <renoirb>	 ok. Then, like you say makes more sense to me :)
[17:56:23] <renoirb>	 I have test.webplatform.org that can be used to work and it uses Varnish like the production
[17:57:43] <Ryan_Lane>	 either ESI or iframes is necessary here
[17:57:49] <Ryan_Lane>	 ESI is the better option
[17:58:29] <shepazu>	 Ryan_Lane, and we decided a clientside JS fetch was a bad idea, right?
[17:59:06] <Ryan_Lane>	 yep
[17:59:06] <renoirb>	 Ryan_Lane: and including the hook on save (or any more appropriate that can keep the generated version)
[17:59:20] <Ryan_Lane>	 renoirb: no need for a hook on save
[17:59:27] <Ryan_Lane>	 you'd have a parser hook
[17:59:36] <Ryan_Lane>	 and it would output ESI or iframes
[17:59:46] <Ryan_Lane>	 the special page would just read the compat tables output directly
[17:59:50] <shepazu>	 Ryan_Lane, you know anyone with direct experience with ESI and mediawiki extensions?
[17:59:59] <Ryan_Lane>	 I may have someone to help you, I'm asking now
[18:00:11] <renoirb>	 oh, a simple <es:include … />
[18:00:23] <renoirb>	 then make sure in some way that what gets called is fine
[18:00:48] <shepazu>	 Ryan_Lane, great
[18:01:05] <renoirb>	 thanks!
[18:01:08] <renoirb>	 brb
[19:42:52] <yurik_>	 would i ruin everything if i run  # puppet --disable    on  cache-mobile02  (betalabs)
[19:52:00] <hashar>	 csteipp: you are probably lunching, I have purged the https redirect for loginwiki in beta
[19:52:02] <hashar>	 csteipp: ( https://bugzilla.wikimedia.org/show_bug.cgi?id=55804 )
[19:52:15] <csteipp>	 hashar: THanks!
[19:52:25] <hashar>	 I get:
[19:52:26] <hashar>	 Location: http://login.wikimedia.beta.wmflabs.org/wiki/Main_Page
[19:52:49] <hashar>	 though that page redirects to https
[19:52:50] <hashar>	 grr
[19:53:30] <hashar>	 purged
[19:53:31] <hashar>	 as well
[19:53:53] <csteipp>	 Ah, that's better :)
[19:53:56] <hashar>	 ideally we would want to purge every http://login.wikimedia.org/* URL that are redirects
[19:54:03] <hashar>	 but I am not sure that is possible
[19:54:16] <yurik_>	 oh, hashar you are here! :)  after a number of very unsucessful attempts to deal with ESI, I came to sad conclusion - I will simply kill puppet on cache-mobile02, and modify varnish files by hand
[19:54:44] <yurik_>	 and later will restore my changes and re-enable puppet
[19:54:48] <yurik_>	 do you think this can work?
[19:54:59] <hashar>	 do you mean editing /etc/varnish/ files directly ? 
[19:55:03] <hashar>	 yeah you can do that
[19:55:06] <yurik_>	 yep
[19:55:19] <yurik_>	 would   puppet --disable   do the trick?
[19:55:22] <hashar>	 might not be easy to figure out later on how to do the puppet stuff
[19:55:33] <hashar>	 puppet runs from a cronjob I think
[19:55:39] <hashar>	 I am not sure how ops manage to disable it entirely
[19:55:55] <yurik_>	 --disable will put some sort of a lock
[19:55:59] <yurik_>	 hope it won't run :)
[19:56:06] <hashar>	 your call :-]
[19:56:25] <hashar>	 you can also get a virtual instance on your server
[19:56:32] <hashar>	 err
[19:56:35] <hashar>	 your machine/laptop
[19:56:50] <hashar>	 and copy the varnish conf there for local testing, will save you the ru -> pmtpa roundtrip latency
[19:56:55] <yurik_>	 hashar: i wish i could - it seems like ESI works just fine on my machine :)
[19:57:02] <hashar>	 great ;)
[19:57:10] <hashar>	 got to process some more mails ..
[19:57:10] <yurik_>	 but not in prod :)
[19:57:16] <yurik_>	 have fun 
[19:57:25] <yurik_>	 is anyone using that server?
[19:57:39] <Krenair>	 Ryan_Lane, what are all the modified files in the OSM repos on nova-precise2?
[19:58:19] <Krenair>	 I feel like renaming one of them to w-DONOTLEAVEUNCOMITTEDCHANGESHERE/
[19:59:03] <T13|needsCoffee>	 Where?
[19:59:23] <andrewbogott>	 Krenair: Those modified files are me trying incremental tests… 
[19:59:32] <andrewbogott>	 it should always be safe to reset --hard origin
[19:59:51] <andrewbogott>	 At least, my work flow involves editing files elsewhere and then rsyncing into the running dir
[20:00:01] <andrewbogott>	 And, apparently, I almost always forget to clean up my mess afterwards
[20:00:43] <Krenair>	 andrewbogott, this is /w/ right?
[20:00:48] <Krenair>	 not 2 or 3?
[20:00:53] <andrewbogott>	 yeah
[20:00:57] <andrewbogott>	 I mean, I'm erratic.
[20:01:01] <andrewbogott>	 But, yeah, that's where I was working last week
[20:01:19] <Krenair>	 I wonder who is responsible for the changes in w2 and w3...
[20:01:35] <andrewbogott>	 possibly also me...
[20:01:53] <andrewbogott>	 But, in any case, you can always commit changes, stash them in a branch, and then do what you need
[20:02:35] <andrewbogott>	 Krenair, want me to look?
[20:02:56] <Krenair>	 no it's fine. I just cleared the osm repo on /w/
[20:03:18] <andrewbogott>	 'k
[20:03:29] <andrewbogott>	 sorry about the mess, I forget that other people use that instance
[20:05:59] <yurik_>	 hashar: quick question - is caching-mobile02 the only varnish server in betalabs?
[20:06:13] <hashar>	 all caches are varnishes
[20:06:30] <hashar>	 a list it at https://wikitech.wikimedia.org/wiki/Special:NovaAddress
[20:06:34] <hashar>	 for deployment-prep
[20:06:55] <hashar>	 deployment-cache-bits03 deployment-cache-upload04 deployment-cache-text1 deployment-cache-mobile01	
[20:07:05] <hashar>	 those four caches are meant to use puppet from the production branch
[20:07:06] <yurik_>	 hashar: but do they all handle en.m.wikipedia.... traffic?
[20:07:10] <hashar>	 and must not use puppet master self
[20:07:25] <hashar>	 the en.m.wikipedia*  is directed to ward the mobile cache ie  deployment-cache-mobile01
[20:07:28] <hashar>	 that is made by DNS
[20:07:52] <hashar>	 *.m.wikipedia.beta.wmflabs.org   -->  deployment-cache-mobile01
[20:07:52] <hashar>	 *.beta.wmflabs.org --> deployment-cache-text1	
[20:07:58] <yurik_>	 hashar: hold on, but i am connecting to deployment-cache-mobile02
[20:08:03] <yurik_>	 is that not running?
[20:08:16] <yurik_>	 btw, when i connected to mobile01 i got tons of SSH warnings
[20:08:17] <hashar>	 that is why you need port forwarding remember ? :-]
[20:08:31] <hashar>	 deployment-cache-mobile02 is a standalone, no traffic  arrive there 
[20:08:35] <yurik_>	 gotcha
[20:08:39] <hashar>	 unless you use port forwarding from your host to there
[20:08:42] <yurik_>	 right
[20:08:48] <hashar>	 ssh -N -L  I.cant.remember...
[20:09:04] <yurik_>	 i will try to break mobile01- i tried port forwarding but i also kept getting nginx
[20:09:13] <hashar>	 ssh -N -L  1080:deployment-cache-mobile02.pmtpa.wmflabs:80   deployment-cache-mobile02.pmtpa.wmflabs
[20:09:14] <hashar>	 probably
[20:09:23] <hashar>	 DONT change mobile01
[20:09:27] <yurik_>	 :(
[20:09:28] <hashar>	 it is for staging puppet changes :-]
[20:09:37] <hashar>	 it must be running from production
[20:09:50] <yurik_>	 can i break it for just half an hour?
[20:09:51] <yurik_>	 :)
[20:10:00] <yurik_>	 and later do the puppet agent --enable
[20:10:12] <hashar>	 ohh hacking /etc ? 
[20:10:16] <yurik_>	 yep
[20:10:22] <hashar>	 I guess you could
[20:10:27] <yurik_>	 yei!
[20:10:41] <hashar>	 make sure to ping the mobile team
[20:10:45] <hashar>	 #wikimedia-mobile iirc
[20:10:48] <yurik_>	 i did
[20:10:50] <hashar>	 they are using beta for testing
[20:10:53] <yurik_>	 i am part of mobile, remember :)
[20:10:54] <hashar>	 so yeah
[20:10:56] <hashar>	 disable puppet
[20:11:03] <hashar>	 MAKE SURE YOU BRING IT BACK !! :-]
[20:11:05] <hashar>	 hack
[20:11:08] <hashar>	 fix! ;-]
[20:11:13] <hashar>	 yeah that would work
[20:11:22] <hashar>	 I should have thought about that last time we talked to each other :(
[20:11:31] <hashar>	 for a quick hack, that is probably easier
[20:11:32] <yurik_>	 should i run puppetd -tv afterwards?
[20:11:37] <yurik_>	 exactly :)
[20:11:58] <Ryan_Lane>	 Krenair: my workflow is similar to andrew's
[20:12:14] <Ryan_Lane>	 I push changes into gerrit, then do a cherry-pick locally
[20:12:36] <Ryan_Lane>	 so if there's ever anything there it's fine to wipe it out
[20:12:49] <Ryan_Lane>	 andrewbogott: btw, that's a lot easier than rsync ;)
[20:12:59] <yurik_>	 hashar: are you sure moblie01 is what it should be?
[20:13:01] <Ryan_Lane>	 you can use anon http to do the cherry-picj
[20:13:03] <yurik_>	 Warning: the RSA host key for 'deployment-staging-cache-mobile01' differs from the key for the IP address '10.4.1.68'
[20:13:03] <Ryan_Lane>	 *pick
[20:13:04] <yurik_>	 Offending key for IP in /home/yurik/.ssh/known_hosts:9
[20:13:06] <yurik_>	 Matching host key in /home/yurik/.ssh/known_hosts:15
[20:13:11] <Ryan_Lane>	 yurik_: remove the offending key
[20:13:17] <Ryan_Lane>	 someone deleted and recreated the instance
[20:13:22] <yurik_>	 gotcha, thx
[20:13:36] <Ryan_Lane>	 or actually, in this case it's just a reused IP
[20:13:41] <Ryan_Lane>	 we should really turn off IP checking
[20:13:47] <yurik_>	 but the strange thing is -  when i connect, i get  mediawiki-temp-test
[20:13:50] <Ryan_Lane>	 such a shitty feature
[20:14:03] <yurik_>	 and it doesn't seem to have varnish
[20:14:14] <Ryan_Lane>	 you ssh'd into the IP, or to the hostname?
[20:14:18] <yurik_>	 hostname
[20:14:20] <Ryan_Lane>	 hm
[20:14:31] <yurik_>	  ssh deployment-staging-cache-mobile01
[20:15:34] <Ryan_Lane>	 hm
[20:15:38] <Ryan_Lane>	 there's another entry with the same IP
[20:15:43] <yurik_>	 lovelly
[20:15:46] <yurik_>	 so where's mobile01?
[20:15:56] <Ryan_Lane>	 one sec
[20:16:06] <Ryan_Lane>	 that may have been from when we were testing the DNS changes
[20:16:15] <yurik_>	 LOLOL
[20:16:30] <Ryan_Lane>	 well, this says it's active
[20:17:01] <yurik_>	 so when i connect to http://en.m.wikipedia.beta.wmflabs.org/ - what servers do i actually hit?
[20:17:15] <Ryan_Lane>	 does this actually exist? deployment-staging-cache-mobile01
[20:17:41] <yurik_>	 no idea :)
[20:17:52] <yurik_>	 hashar said that's the one i should be breaking
[20:18:13] <Ryan_Lane>	 its IP is 10.4.1.82
[20:18:35] <Ryan_Lane>	 ah. there's a bad record for an older deleted deployment-staging-cache-mobile01
[20:18:41] <Krenair>	 Ryan_Lane, andrewbogott: Have either of you two touched the Echo repository in wikis/w/?
[20:18:52] <andrewbogott>	 not me
[20:18:56] <Ryan_Lane>	 Krenair: nope
[20:18:58] <yurik_>	 Ryan_Lane: i was able to connect to mobile01 via ip
[20:18:59] <Krenair>	 Looks like someone rscync'd a newer copy of Echo to it or something, rather than git pull...
[20:19:06] <Ryan_Lane>	 heh
[20:19:07] <yurik_>	 wonder how the DNS record persisted
[20:19:10] <Ryan_Lane>	 Krenair: wipe away
[20:19:14] <Ryan_Lane>	 yurik_: a bug, I'm sure
[20:19:19] <Ryan_Lane>	 I'm deleting the old one
[20:19:27] <yurik_>	 record or the instance?
[20:19:32] <Ryan_Lane>	 record
[20:20:04] <Ryan_Lane>	 in an hour it'll be fixed
[20:20:15] <yurik_>	 Ryan_Lane: do you know how the resolution happens for the http://en.m.wikipedia.beta.wmflabs.org/
[20:20:24] <yurik_>	 is there a way to tracert ?
[20:21:04] <yurik_>	 because m , being a mobile, would connect to a different mobile cache, right?
[20:21:45] <Ryan_Lane>	 go to Manage Addresses
[20:21:48] <Ryan_Lane>	 in wikitech interface
[20:21:59] <Ryan_Lane>	 you'll see which instance it's associated with
[20:22:50] <yurik_>	 Ryan_Lane: that's for admins only
[20:22:53] <hashar>	 yurik_: has the instance changed ? 
[20:23:03] <yurik_>	 could you add me to admins (i won't create instances :)
[20:23:14] <yurik_>	 hashar: i just disabled puppet two seconds ago
[20:23:16] <yurik_>	 nothing else
[20:23:18] <yurik_>	 (yet)(
[20:23:19] <hashar>	 10.4.1.82
[20:23:21] <Ryan_Lane>	 it should be viewable as a member
[20:23:33] <Ryan_Lane>	 yurik_: does the inferface tell you you don't have rights?
[20:24:08] <Ryan_Lane>	 hashar: is there any reason why schema-review is in the SHUTOFF state?
[20:24:28] <hashar>	 yurik_: and *.m.wikipedia.beta.wmflabs.org  is tied to deployment-cache-mobile01
[20:24:38] <hashar>	 Ryan_Lane: I have no idea what this instance is 
[20:24:41] <yurik_>	 Ryan_Lane: i only see mobile and mediawiki-api -- the other ones i see just the title, no content
[20:24:51] <yurik_>	 for the first two i am an admin
[20:25:05] <Ryan_Lane>	 hm. weird. it should display things
[20:25:17] <Ryan_Lane>	 please open a bug for that
[20:25:23] <Ryan_Lane>	 mediawiki extensions, openstackmanager
[20:27:49] <yurik_>	 grr, is there a way to have SSH be as resiliant to connection resets as IRC ???
[20:28:28] <Ryan_Lane>	 it should be?
[20:28:44] <yurik_>	 putty has been reseting connections non stop in russia :(
[20:37:09] <johang>	 so I applied for access to the tools project. but then I was added to group called shellusers. what does that mean?
[20:37:39] <johang>	 wmflabs is very confusing.
[20:38:31] <johang>	 I can log into bastion, but I suspect that's something else.
[20:40:05] <andrewbogott>	 johang, being in 'shellusers' just means you can log in to things.
[20:42:04] <andrewbogott>	 So, it's a necessary step.
[20:42:28] <johang>	 right. I guess I'm not in the tools project yet. because I can't login to tools-login
[20:42:57] <andrewbogott>	 what is your username on wikitech?
[20:43:38] <johang>	 johang
[20:44:36] <andrewbogott>	 try now?
[20:45:06] <johang>	 it works now. thanks.
[20:45:49] <johang>	 what's the point of bastion when there's a separate tools-bastion?
[20:46:15] <andrewbogott>	 johang:  tool-labs is just one small part of labs.
[20:46:21] <andrewbogott>	 bastion is for other bits of labs.
[20:46:44] <johang>	 I see.
[21:18:50] <DarTar>	 what's the usual replag for labsdb? We're currently seeing roughly 2 days of delay on enwiki
[21:21:09] <Ryan_Lane>	 it's usually not 2 days ;)
[21:21:11] <Ryan_Lane>	 Coren: ^^
[21:23:41] <DarTar>	 last user registration on enwiki.logging: 20131017134907
[21:24:58] <Ryan_Lane>	 it's generally seconds
[21:25:04] <Ryan_Lane>	 in the small number
[21:25:53] <^d>	 20131017 though? No registrations since > yesterday?
[21:28:30] <DarTar>	 for comparison, this is the same from the internal slave (db1047) 20131018211434
[21:29:06] <DarTar>	 which is basically NOW() :)
[22:34:16] <Coren>	 DarTar: I'll ask Sean to take a look at it.  I knew he had to do some magick for s3 some time ago.
[22:35:46] <Coren>	 DarTar: But yeah, last edit I see in revision on enwiki is >24h old
[22:35:53] <DarTar>	 thanks
[22:36:06] <Coren>	 dewiki is up to date, so it looks like it's just S1