[00:35:31] I can’t access tools.wmflabs.org (ping / http / https) from tools-login.wmflabs.org at the moment. [00:35:38] anyone else having this problem? [00:36:53] Coren: ^ [00:37:35] ireas: -eqiad ? [00:37:39] ireas: That's not so much a problem as a limitation of nova-network. You can't reach external IPs from within, but you /can/ use the interal IP [00:37:42] oh, that's the default [00:37:56] ireas: tools-webproxy, for that server. [00:38:29] (Yeah, it sucks. I wish we could have upgraded to neutron during migration, but it would have delayed it too much) [00:39:19] hmm, okay … did this change recently? it used to work in tampa, and I think it also worked after the eqiad migration [00:41:24] Okay, tools-webproxy works. So I’ll have to check if I’m on Labs or somewhere else in the world wide web … thanks for the quick reply! [00:42:00] It never worked in pmtpa [00:42:20] Stupid nat/nova issue [00:43:29] Damianz, I used that code for several weeks, and it did work ^^ [00:44:02] then something was very broken [00:44:10] though interesting [00:44:29] :D [00:44:39] https://meta.wikimedia.org/wiki/Meta:Requests_for_help_from_a_sysop_or_bureaucrat#Site-wide_links_to_https:.2F.2Ftools.wmflabs.org.2Fguc.2F [00:44:53] We had a host alias for tools.wmflabs.org in ptmpa. [00:45:10] https://en.wikipedia.org/w/index.php?title=Wikipedia:Village_pump_(technical)&diff=600186951&oldid=600184390 [00:45:21] is this advice correct? [00:47:24] * Coren reads said advice. [00:48:11] Coren: I only replied because I didn't expect anyone else to answer it... also here https://en.wikipedia.org/w/index.php?title=Wikipedia:Village_pump_(technical)&diff=600180348&oldid=600174944 [00:48:30] incognitus: It's pretty much correct. the trailing slash thing is my current config being overly by-the-book about URLs; but that's on my "this week" list to fix. [00:48:53] yeah, it worked before (apache?) [00:48:57] But yeah, on the subtance, restarting the webservice is what to do. [00:49:37] incognitus: Yeah, the previous config did an implicit redirect from foo to foo/ as appropriate; currently it doesn't try (and fails correctly, but unhelpfully) [00:57:03] My cron settings have not been carried over to the new Tool Labs [00:57:26] actually, I will look into the labs-l mailing list to see if I can find something there before asking here. [01:01:38] Coren: another question [01:05:29] uh, does anyone who was subscribed to this list know what happened to our cron jobs? The archive is not functional. [01:11:42] Magog_the_Ogre, I already looked for that information but could not find in on the list :/ [01:12:20] I would open up another bug, but I don't know it is one yet... [01:42:33] Magog_the_Ogre: Coren said that there was a problem in reinstating crontabs after migration and that he wanted to fix it. I'm not sure what's the status of that. [01:43:48] thanks scfc_de [01:49:07] * Coren is not really here, but suggests that people look in a file named ...DATA.crontab [02:03:00] Magog_the_Ogre: Generally safest to put crontabs in a text file, BTW. [02:03:11] yeah [02:03:15] Then you can use "crontab /path/to/crontab.txt" .. yeah. [02:03:17] I did so about a month ago [02:03:22] ohhhhhhhh [02:03:23] Put version control! [02:03:26] never did that before [02:03:29] Yeah. [02:03:30] I HAVE VERSION CONTROL [02:03:35] It's trivial to kill a crontab. [02:03:37] You go girl! [02:03:44] Put in * [02:03:54] President Putin. [02:04:01] but my Labs doesn't have write access to my repository [02:04:08] Yeah, shit seems fucked. [02:04:16] Or was yesterday when I poked. [02:04:22] I'm sure things are improving. [02:04:22] I have [02:04:23] no idea [02:04:26] what you're talking about [02:04:37] Putin, yesterday being poked? [02:04:57] Pretty much. [02:05:02] Don't worry about it. [02:05:05] ok [02:05:37] hey I like my dictators petty, murderous, and part of the mob too. [02:06:24] :-) [02:06:55] "crontab -r" versus "crontab -e" is one of the more evil parts of Unix. [02:07:33] I like talking about crontabs. [02:07:38] I should eat. [02:09:55] stop telling us to stuff beans up our nose (Google it) [02:34:20] one new English term per day from just reading Magog_the_Ogre [02:34:45] heh, the beans up your nose isn't an English-ism, it's a Wikipedia-ism :) [02:34:53] heh, ok [02:35:31] i'll have to catch up on world news by reading wikinews some time later i guess [02:35:39] oh don't do that [02:36:06] by paying attention to the news, I long ago realized that the Just World Fallacy is a beautiful fallacy to have, because you won't realize how fucked up the world is [02:36:56] ignorance is bliss [02:37:02] there's a new one :) [02:37:10] so, is Putin still alive? [02:37:16] or what did "poked" mean [02:37:31] ok, don't tell me [02:38:21] I haven't a clue what Gloria meant by poked [02:39:01] maybe someone poked her with a bobby pin? [02:39:06] >me keeps using complicated words [02:39:14] */me [02:39:33] please do [02:40:56] Magog_the_Ogre: you should do sound recordings for en.wikt, can always use more .ogg [02:41:16] that sounds like fun [02:41:22] Commons is severely pissing me off [02:41:38] so you will be the voice when people click the little sound icon next to a word [02:41:47] to the extent I'm ready to be the last sane member of the community to finally give up on it. [02:42:28] what's so insane there currently [02:42:37] wait, is that like watching the news? [02:44:35] I have very strong political opinions [02:45:22] and someone other than my favorite candidate is running the nation I live in. What's worse, I have very strong political opinions on foreign events, and IMO the world is going to hell in a handbasket [02:45:26] * Magog_the_Ogre nudges mutante|away [02:46:05] i meant Commons, heh [02:46:11] oh [02:46:30] well the place is the dumping grounds for all the editors who got banned at en.wp because the people there are sane [02:46:34] and then i stopped myself, thinking if you tell me, then it's like i'm paying attention to the new [02:46:41] and you just told me not to [02:46:49] haha [02:46:50] and they've come to form a plurality, so they can get people in trouble for doing things like *pointing out their copyvios* [02:47:53] this is why i like wiktionary [02:48:25] a small community, still a lot left to do, and since everything is templates, you don't have to argue about style either (unless you want to change templates, heh) [02:49:01] andrewbogott_afk: yes, and I have. [02:49:05] (depending on which page you mean) [02:52:35] Magog_the_Ogre: here's a fun project. grab a microphone, go to some place where you can expect people speaking multiple languages, then have them just read http://en.wiktionary.org/wiki/Category:Vulgarities_by_language into a mic and tell them how they really help a good cause by doing that:) [02:53:33] I had switched subjects when I said poked. [02:53:43] I poked [at Labs yesterday during the server maintenance]. [02:53:56] And it was pretty broken, of course. I think things have improved today. [02:53:59] Though parts may still be down. [02:54:40] ah, i see [03:39:41] can't login to new eqiad instance, Unable to create and initialize directory [03:40:19] it says it's creating my home, then that it fails, then shows me motd, then closes connection. it sounds like known issue [04:46:17] Krinkle: The note on the progress page says 'Do not break, actively used...' [04:46:21] Is that out of date? [04:46:29] No, I added it last weekend [04:47:01] andrewbogott: i couldn't migrate because of home dir issue on eqiad [04:47:27] me too [04:48:12] https://bugzilla.wikimedia.org/show_bug.cgi?id=62771 [04:50:10] confirmed [04:50:17] Krinkle: was cvn-app3 migrated from pmtpa or is it a newly-created instance? [04:50:33] the databases are still being moved ? [04:50:45] (I still don't see mine) [04:51:33] andrewbogott: i get it with newly created instances in eqiad [04:51:38] ok [04:51:44] while i can login fine into the old pmtpa instance [04:51:46] in same project [04:51:48] using same key [04:51:57] just changing the bastion in ssh config [04:52:12] the reason is home dir not being mounted [04:52:20] i can see my project storage on labstore1001 [04:52:27] incl. my key [04:53:23] the instance says "Unable to create and initialize directory" [04:53:29] just like in Krinkle's report [04:53:52] andrewbogott: I don't know about the ability to migrate entire intances. I created it fresh [04:54:13] same [04:55:51] Hm… this bug is squarely in Coren's area of influence. I've seen it before but have never done anything smarter than just reboot a bunch of times :( [04:56:15] reboot the instances or reboot labstore1001:) [04:56:28] i saw this [04:56:40] service manage-nfs-volumes restart [04:56:52] but unsure if that is exactly the one he did last time [04:57:54] tries rebooting instances [04:58:12] andrewbogott: Tell me more about the migration of entire instances? I don't plan on that but it's interesting to know. Also, more practically for me, is there migration for home or data storage of a project? [04:58:24] (assuming those aren't shared between eqiad and pmtpa) [04:58:37] Krinkle: It's something that basically only I can do. People that need instances migrated intact can open bugzilla bugs about it. [04:58:47] Um… instance migration, that is. [04:58:48] k, nvm then :) [04:58:52] hah [04:59:03] It usually works, but not so much if the instance is lucid or uses self-hosted puppet. [04:59:07] I mean, I dont' need it and there's no manual for me to get familiar with when helping others. [04:59:44] andrewbogott: I'll just recreate the instances. Good practice anyway to make sure everything is indeed properly put in storage and my instances shouldn't keep to much state anyway. [05:00:07] I've migrated (I think) all of the old gluster storage. It's in the equivalent directories in eqiad under 'glustercopy' [05:00:11] I'm interested in migrating the project store shared between instances though [05:00:23] nfs shared storage I haven't gotten to yet [05:00:36] so /home and /data/project/cvn basically [05:00:58] I assume that'll have to be initiated by a project admin to avoid it being copied at an arbitrary point in time. [05:01:27] It's being written to all the time, I won't need a copy until I'm ready to migrate the running processes. Otherwise they'll get out of date. [05:01:40] Um… ": I've migrated (I think) all of the old gluster storage. It's in the equivalent directories in eqiad under 'glustercopy'" [05:01:45] Is that not the storage you're talking about? [05:01:51] so I'm preparing the new instance first, and then when its good and ready, kill pids, copy data, start new ones on the eqiad instances. [05:02:00] You tell me [05:02:02] But, yeah, if you need a last-minute rsync after you set up the new instances, I can do that [05:02:21] it worked this time [05:02:28] dzahn@wikistats-willitblend:~$ [05:02:42] so.. "have you tried turning it off..." [05:02:48] for reals.. [05:16:17] andrewbogott - you say: "I've migrated (I think) all of the old gluster storage." <- does that include the databases, I still can't find the database that is being used by linkwatcher and coibot [05:16:54] Beetstra: if you're talking about tools databases, then that's pretty much unrelated. [05:17:02] OK [05:17:03] :-( [05:17:08] it is indeed on tools-db [05:24:22] Krinkle: you can see for yourself (if you're ever able to long in). Just look in /data/projects/glustercopy and see if what you need is already there [05:24:30] likewise /home/glustercopy [05:24:51] It won't because the data is changing every minute. (sqlite database among other things) [05:25:20] until the cvn bots are set up on cvn-app3 in eqiad (which I haven't been able to log into yet), the bots are still running in pmtpa and actively changing the dataset [05:26:18] as soon as all bots are running in cvn-app3 with empty data sets, I'll kill the dry-running ones in eqiad, kill the ones in pmtpa, ask for a data migration, and boot the eqiad ones as live instead of dryrun. [05:26:44] the migration of those bots will take a while, once I'm ready to flip the switch I'll need to do it with you standing by to migrate the data [05:27:23] ok, that sounds about right. [12:11:31] replag [12:11:34] !replag [12:11:37] @replag [12:11:37] Replication lag is approximately 00:00:00.6288190 [12:24:59] Almost done. [12:30:44] (03Abandoned) 10Hashar: grrrit: switch default channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/117839 (owner: 10AzaToth) [12:31:46] Coren: good morning! Thank you for the /mnt dependency fix :] [12:32:00] I am happy my crazy dependency hack ends up being acceptable \O/ [12:32:47] It's not hackish, really, require => Mount['foo'] is perfectly legitimate. The conditional is annoying, but temporary. [12:33:05] indeed :] [12:33:15] (There's a lot of "if $::site" crap we had to add for migration that we'll be able to clean up soon) [12:34:02] hopefully I will remember to clean it up later on [12:34:39] also someone asked to add the package joe (a text editor) on tools labs dev_environ : https://gerrit.wikimedia.org/r/#/c/118595/ [12:46:30] it never ends :-( [12:46:43] I could use a reboot of deployment-upload.pmtpa.wmflabs ( which is I-00000793.pmtpa.wmflabs ) [12:46:50] can't reboot it via wikitech [12:46:56] and can't ssh either [13:13:16] Coren: ping [13:13:29] Heyo [13:13:52] Coren: Have you saved the crontabs from the old tools login somewhere? [13:14:38] Yes, they're all still there, but it should have been copied over already. Did you do the finish-migration? [13:15:25] Coren: I didn't migrate per hand... but I ran finish-migration [13:15:37] nothing [13:15:49] Allright, that should have restored your crontab. What's your tool name? [13:16:42] Those crons were just running from my main account "hoo" [13:16:44] not a tool [13:18:13] Ah! [13:18:25] Those /weren't/ copied, because you shouldn't have been doing that. [13:19:17] I can copy them for you into a tool of your choosing. [13:19:19] Great... neat that *nobody* said that [13:19:42] copy them into the hoo tool... [13:19:45] Dude, it's rule number 1: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Rules [13:20:07] Sure thing. I'll put them in a file though so as to not mess with your current crontab. [13:20:29] There shouldn't be any. that project is for webhosting only [13:20:31] well, it was [13:21:30] hashar: stay strong, say no to more editors:) [13:22:04] hoo: I just copied it into tools.hoo's crontab (commented out) [13:22:13] hoo: By the way, you know you can do things like: [13:22:37] 0,30 18,19,20 * * * jsub xxxxx [13:22:55] To avoid having to repeat a line in the crontab for different hours? [13:22:58] Just FYI [13:23:00] Coren: Hi, I can't modify or rm in my ToolLabs project, because "/data/project/rxy/public_html" and others owner UID is wrong. Could you please fixing it? detail --> http://pastebin.com/Zet4ccAp [13:23:35] Coren: Yeah, I know... but it's just a lot of jobs (one for every WMF wiki basically)... that's why I grouped those into bash files and have these run every whatever or so [13:23:44] so each thing is only running once a day [13:24:25] rxy: {{fixed}} [13:24:41] Coren: Thank you! :) [13:26:46] Coren: could you please run chown -R tools.hoo:tools.hoo /data/project/hoo/ [13:28:35] hoo: {{done}} [13:28:42] thx [13:31:58] mutante: hey :-] [13:44:41] hashar: I just logged in to deployment-upload.pmtpa.wmflabs, does that mean all is well now? [13:44:54] sorry, how did i reach the pmtpa host? [13:45:31] fluff, could you be more specific? [13:45:46] can I ssh to the pmtpa host? [13:46:00] looking for the crontab for my user [13:46:35] fluff: Ah, I see. I don't know :) [13:46:42] andrewbogott: yup came back ~55 minutes [13:46:43] ago [13:46:59] andrewbogott: the wikitech reboot might have ended up working [13:47:08] ok [13:47:17] andrewbogott: while you are around can we clone that instance as is from pmtpa to eqiad albeit with a different IP address ? [13:47:35] hashar: Sure. Stay tuned... [13:47:40] thought i saw something like tools-login.pmtpa.wmflabs.org [13:47:44] deployment-upload is a huge hack created back in July 2013 to emulate the production media server we have been using before Swift [13:47:52] it is not in puppet of course :-] [13:48:18] hashar: migrating requres a shutdown of the tampa instance... [13:48:36] wow, you've been busy! Lots of eqiad instances :) [13:48:42] yeah :-] [13:48:55] ok, here goes... [13:49:07] would you have time now to copy paste the instance ? [13:49:22] I will need to keep the one in pmtpa though or upload will end up broken :D [13:49:41] but we can get face an outage of upload on beta for some time (not too long though) [13:52:42] it's copying now. After it finishes I can restart the tampa instance as well. [13:59:15] hoo: On eqiad, you should now be able to run sudo chown -R tools.$TOOL:tools.$TOOL /data/project/$TOOL as the tool account. [14:00:40] hoo: Corrections: For new tools; we still need to fix LDAP for the old ones. *argl* [14:02:25] mh... [14:02:42] The help page could probably benefit from a few collapse boxes. [14:04:06] fluff: I've just send an email to labs-l with information about crontabs et. al. Chances are, you only need to "finish-migration your_tool" to get your crontabs back. [14:04:14] hashar: Ok, you should now have two running deployment-uploads. Want to check my work? [14:04:21] andrewbogott: you are awesome [14:04:25] (But the entries will be commented out by default) [14:04:47] hashar: He is, but we have to few of him. :-) [14:04:50] andrewbogott: the pmtpa one rebooted properly and I am logged on the eqiad one. You are saving me a loooooot of time [14:05:21] for op in [coren,andrewboggot]: op.clone(name="random()") [14:05:28] ops.update(op) [14:05:34] hashar: Migrated instances throw some puppet errors due to an inability to mount some of Coren's fancy new shared volumes. I don't know offhand how to fix but it should be easy (if you care) [14:06:01] I would need /data/project I guess I can figure it out [14:06:12] that one works [14:06:20] it's just new stuff like /scratch [14:07:15] andrewbogott: I'm working on the "first mount too fast" issue right now; I'm going to try a couple of ideas to force serialization. [14:07:26] Coren: thank you! [14:07:33] Is it a race with manage-nfs-volumes? [14:09:10] Coren: yeah, i was looking for my users crontab, the tool's crontab are ok [14:11:26] andrewbogott: Basically. If the mount happens before manage-nfs-volumes updated the exports, the NFS server caches the lack of access for ages. [14:11:56] So, could be ameliorated (sp?) by reducing the refresh time... [14:12:04] fluff: User crontabs weren't copied over. Mostly because you shouldn't /have/ user crontabs at all in the first place. :-) [14:12:06] Well, that's not a proper fix of course :) [14:12:15] andrewbogott: and I need the eqiad upload instances to be configured in the puppet lvs service_ip . I am not sure what is the process to get that conf updated though since it might impact production (thought he lvs config hash is namespaced with $::realm it should be a noop). Pathc is https://gerrit.wikimedia.org/r/#/c/119478/ [14:14:10] * Coren goes afk for a little bit. [14:14:33] Coren: Yeah, I know, so it's lost? [14:14:33] hashar, I… cannot predict what that will do :) You could mark it as labs only. [14:16:01] fluff: I'll make you a copy after lunch. For use in a tool only. :-) [14:16:19] Coren: Alright, I'll move my files meanwhile [14:16:23] Coren: Thanks [14:17:20] andrewbogott: would attempt to get Faidon/Mark to merge it in :] [14:17:36] the eqiad upload instance indeed complains about /data/scratch [14:18:51] http://paste.openstack.org/show/73826/ Permission denied - /data/scratch :D [14:25:57] is pmtpa labs infrastructure being shutdown on March 31st ? [14:29:44] Moin Moin zusammen, ich bekomme bei http://tools.wmflabs.org immer "Internal error". [14:34:10] Crazy1880: meinst du wirklich die Startseite oder ein bestimmtes Tool> [14:35:32] Nein, nicht die Startseite, sorry. Tools, Koordinaten, Wikidata ToDo http://tools.wmflabs.org/wikidata-todo oder https://tools.wmflabs.org/geohack/geohack.php?pagename=Remlingen_(Niedersachsen)&language=de¶ms=52.116666666667_N_10.666666666667_E_region:DE-NI_type:city(1822) [14:36:22] Crazy1880: dir ist bekannt dass Tools gerade im Umzug ist zu einem neuen Rechenzentrum? [14:37:00] Hatte man an mich herangetragen, aber den konkreten Zeitplan konnte mir keiner nennen. [14:37:00] also je nach Tool, ist es noch nicht migriert oder der maintainer muss mal schauen ob alles gestartet ist [14:37:14] nun ja, seit Montag so richtig [14:37:42] Okai, dann weis ich Bescheid. Wollte da nur sicher gehen. Vielen Dank [14:37:46] hashar: Yes, or shortly thereafter. [14:38:02] Crazy1880: Weisst du wer dieses Tool normalerweise betreut? [14:38:46] Bei den Koordinaten leider nicht, weil es ja jede Sprache betrifft. Bei den anderen Tools ist es Magnus' [14:40:48] Crazy1880: sieht so aus als wenn da jemand nur den Webserver starten muss, ich frag mal [14:40:59] is it easy for you guys to just start the webserver on [14:41:02] http://tools.wmflabs.org/wikidata-todo [14:41:12] mutante: That tools is still copying over [14:41:22] according to a mail earlier today [14:41:24] Crazy1880: what Hoo said [14:41:36] Yes, thanks. [14:41:43] :) [14:44:10] Thanks and have a nice day. [14:47:08] !log glam released ip, 208.80.153.148, and domain name, gwtoolset.wmflabs.org, from i-00000962.pmtpa.wmflabs [14:47:54] Logged the message, Master [14:48:24] !log glam started i-000001af.eqiad.wmflabs [14:48:26] Logged the message, Master [14:51:44] !log glam created web proxy for i-000001af.eqiad.wmflabs. Successfully added gwtoolset entry for IP address 208.80.155.156. Successfully created new proxy gwtoolset.wmflabs.org for backend gwtoolset.eqiad.wmflabs:80. [14:51:46] Logged the message, Master [15:23:59] Anyone who intercept the new ssh key, could surely intercept or change the wiki page? [15:26:48] Yes, but the history should show who changed that. I'm going to protect the pages anyhow. [15:28:44] scfc_de: The main point about interception still stands. [15:29:03] If we're screwed, we're screwed. [15:29:52] a930913: Oh, you mean someone intercepts the user accessing the wiki page? Yes, then the user is screwed. [15:30:09] Well, no, https would require a valid certificate. [15:32:15] Coren, call? [15:34:43] hey tgr or bd808, if either of you have time this week. would you be able to review https://gerrit.wikimedia.org/r/#/c/119467/ ? [15:35:04] * bd808 hides from dan-nl :) [15:35:09] I'll take a look [15:35:21] :) [15:35:26] thanks! [15:41:14] anyone happen to know how to add a host entry to bastion so that it knows about eqiad.wmflabs? i tried ssh gwtoolset.eqiad.wmflabs, but the host is unknown. had to ssh to the instance ip in order to get there [15:44:28] dan-nl: I tried to contact you yesterday about this. [15:44:33] I take it you are not subscribed to labs-l? [15:46:18] bd808: good morning! I restarted logstash1.eqiad.wmflabs a few minutes ago, it is stalled still :-] Also did a puppet tweak to have udp2log on beta to send its log to the eqiad instance https://gerrit.wikimedia.org/r/119493 :] [15:46:32] bd808: and figured out the salt/puppetmaster you have setup on beta. It is nice! [15:47:20] hashar: It's stalled? Like not getting new log data recorded? Or something else? [15:47:46] bd808: na the instance itself is stalled. Waiting for it to reboot [15:48:07] * bd808 logged in [15:49:26] !log deployment-prep deleted local user l10nupdate on deployment-bastion. It is in ldap now. [15:49:27] hey andrewbogott, yes, thanks for posting to my user page … as far as i understand it ii have things configured properly now using the eqiad instance you created for us [15:49:29] Logged the message, Master [15:49:46] dan-nl: if you are not subscribed to labs-l, please do so right now [15:49:52] ssh logstash1.eqiad.wmflabs [15:49:52] channel 0: open failed: administratively prohibited: open failed [15:49:52] ssh_exchange_identification: Connection closed by remote host [15:49:52] :( [15:49:53] k, i will [15:50:18] !log deployment-prep fixed upd2log-mw daemon not starting on eqiad bastion ( /var/log/udp2log belonged to wrong UID/GID) [15:50:20] Logged the message, Master [15:50:21] dan-nl: Also, you'll want to rearrange this page to indicate that your project is 'migrated' rather than 'mothballed' https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration/Progress [15:50:40] dan-nl: You can't ssh to your instance because the firewall rule only permits access from tampa. I'll fix that right now. [15:52:01] bd808: I was using the wrong hostname :] [15:52:22] hashar: ha. That's happened to me more than once. [15:52:38] * bd808 sees the missing deployment- [15:53:39] hashar: I think it's all setup and ready for the udp2log feed. I was going to work on that this morning, but it looks like you are on it. [15:53:55] bd808: na just fixed udp2log-mw on the bastion :] [15:54:00] bd808: I havent looked at logstash. [15:54:27] dan-nl: OK, I've done some tune-up, the instance should be mostly OK now. [15:54:37] Ok. I'll see what that needs. I should be easy once you have logs showing up on the new bastion. [15:55:36] puppet change https://gerrit.wikimedia.org/r/119493 would point udp2log to the proper logstash instance :D [15:55:54] I harassed ops too much today, I think I have no more karma left to get that change merged in *grin* [15:56:31] Lets cherry pick it to the local puppetmaster and go from there :) [15:56:39] That's why we have it [15:56:48] ahh [15:56:55] I can start setting more of the hosts up to use it [15:57:00] now I understand what "Senior" means in your job title hehe [15:57:28] andrewbogott: thanks, added myself to the labs-l list and updated https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration/Progress [15:57:28] senior == go around obstacles :) [15:57:34] I kind of hate the manual setup but I am not sure how we could point to our puppetmaster by default [15:57:40] probably use realm.pp or base [15:57:45] dan-nl: thanks [15:58:38] bd808: also there is no traffic on the eqiad bastion cluster, so udp2log is probably doing nothing at all [15:59:50] I'll see if I can get something just to prove it works. [16:00:08] I'll start with manually switching the puppet/salt config and go from therem [16:00:11] *there [16:00:27] bd808: that changes is merged [16:00:32] re: logging [16:00:42] thanks mutante [16:04:25] -pipe 1 /usr/bin/log2udp -h logstash.pmtpa.wmflabs -p 8324 [16:04:25] +pipe 1 /usr/bin/log2udp -h deployment-logstash1.eqiad.wmflabs -p 8324 [16:04:25] \O/ [16:04:38] there is still deployment-fluoride.pmtpa.wmflabs will have to figure it out [16:06:54] andrewbogott: anything else i should look at or does all seem well? only thing i noticed that seemed like it might be an issue is that the image id is ubuntu-12.04-precise (deprecated 12-16-2013) [16:07:23] dan-nl: no need to worry about the image type -- that's just how we keep track of which is the very latest. [16:07:35] If you're happy with the web service you're getting then I think you're done. [16:07:51] cool, excellent. thanks for your patience and help! [16:07:56] dan-nl: if you had files in /home or /data/project that you care about… they are probably stashed in the 'glustercopy' subdir. [16:08:05] easy enough [16:08:38] no files were stored there so we should be all set [16:12:13] andrewbogott: sorry to bother … it looks like i can't ssh into the instance anymore [16:12:15] ssh -A gwtoolset.eqiad.wmflabs [16:12:15] ssh: Could not resolve hostname gwtoolset.eqiad.wmflabs: Name or service not known [16:12:46] and ssh 10.68.16.160 yields If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [16:12:47] Permission denied (publickey). [16:15:26] !log deployment-prep Applying role::logging::mediawiki::errors on deployment-fluoride.eqiad.wmflabs . It is not receiving anything yet though. [16:15:28] Logged the message, Master [16:18:07] andrewbogott: was able to ssh into the instance about an hour ago without issue ... [16:18:17] dan-nl, let me check [16:18:57] dan-nl: It… works for me? [16:19:19] From where are you running that ssh -A command? [16:20:06] in terminal [16:20:12] ssh dan-nl@bastion.wmflabs.org -A [16:20:19] your local system [16:20:24] you have proxycommand set up and such? [16:20:26] then from dan-nl@bastion1:~$ ssh gwtoolset.eqiad.wmflabs [16:20:32] oh, I see. [16:20:37] ok, let me try it that way [16:20:48] you only need 1 of those options, ProxyCommand ..OR .. use -A to forward agent.. Proxy is better [16:21:07] unfortunately i don't have the proxy command available [16:21:46] dan-nl: it works for me from tampa bastion as well :/ [16:22:20] with pmta i ran the above with the second ssh being ssh gwtoolset.pmta.wmflabs [16:22:25] hmm [16:22:51] can you ping it? [16:23:30] very odd, now it works [16:23:38] dan-nl, wait, just a second ago you said 'gwtoolset.pmta.wmflabs' which /definitely/ won't work [16:23:43] dan-nl: did it show you the MOTD but then disconnect you again? [16:23:57]