[00:09:33] Hi. [08:44:38] hi, after completely successfully the migration (migrate-tool + finish-migration) I found a ...DATA.olduser file in the new home, can I remove it? [09:04:38] invalid argument with chown on new labs instance? could this be nfs related? [09:08:30] this is probably related to Bug 61899 [10:48:45] @replag [10:48:46] Replication lag is approximately 00:00:00.9672830 [11:00:41] !tunnel [11:00:41] ssh -f user@bastion.wmflabs.org -L :server: -N Example for sftp "ssh chewbacca@bastion.wmflabs.org -L 6000:bots-1:22 -N" will open bots-1:22 as localhost:6000 [11:44:24] ok… petan, I have a bit of time before breakfast, let's sort out this proxy thing! Can you tell me what exact url is broken for you just now? [12:52:27] petan: sorry, internet hates me today. Back now (for more than 5 minutes I hope) [13:35:21] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Adding akosiaris' key at root's [labs/private] - 10https://gerrit.wikimedia.org/r/118286 (owner: 10Alexandros Kosiaris) [14:22:47] !ping [14:22:47] !pong [15:49:30] (03PS1) 10Gerrit Patch Uploader: Show patrolled changes in #wmt-ko [labs/tools/WMT] - 10https://gerrit.wikimedia.org/r/118549 [15:49:33] (03CR) 10Gerrit Patch Uploader: "This commit was uploaded using the Gerrit Patch Uploader [1]." [labs/tools/WMT] - 10https://gerrit.wikimedia.org/r/118549 (owner: 10Gerrit Patch Uploader) [15:52:07] andrewbogott: Back to life? [15:52:32] Coren: sort of. Visiting a friend in Seattle so only semi-working today and tomorrow. But on the ground at least. [15:52:45] Two 8-hour flights, the first 'had' wifi but it was broken, the second didn't have it at all :( [15:52:59] andrewbogott: No emergency then; ping me when you're back in the groove. [15:53:00] Coren: anything pressing amiss? [15:53:14] you are aware wikitech is down? [15:53:21] um… no? [15:53:21] eh, was [15:53:37] o_O Works great for me. [15:53:57] well, i'm not the only reporter [15:54:03] but anyways [15:54:27] mutante: doesn't look so good from here either :( [15:55:01] Huh. Odd. It's working fine, and reasonably snappy for me. [15:56:14] mutante: reported where/how? [15:56:52] andrewbogott: ^d on -ops [15:57:17] 08:56 < mutante> ^d: i know, i reported as ticket a while ago [15:57:19] <^d> Very much down for me. [15:57:22] eh, wrong paste [15:57:39] The server at wikitech.wikimedia.org is taking too long to respond. [15:59:43] ... what in blazes is going on? It's not only working fine, but fast for me. [16:00:00] mutante: tracepath to port 443? [16:00:48] Me too. [16:01:31] scfc_de: Me too broken, or me too it works? [16:02:25] (In fact, I don't remember Wikitech /ever/ being that fast, so I guess that's suspicious) [16:02:53] Coren, $ ping wikitech.wikimedia.org [16:02:57] PING wikitech.wikimedia.org (208.80.152.32): 56 data bytes [16:02:58] Request timeout for icmp_seq 0 [16:02:58] Request timeout for icmp_seq 1 [16:02:59] Coren: it is going to Tampa [16:03:07] weirdly I have an ssh session that's totally happy [16:03:07] paste about to come up, it's still running [16:03:13] it's supposed to be tampa, it's still virt0 [16:03:45] Too many hops: pmtu 1500 [16:03:45] Resume: pmtu 1500 [16:04:02] http://paste.debian.net/87533/ [16:04:09] marc@mordor:~$ ping wikitech.wikimedia.org [16:04:09] PING wikitech.wikimedia.org (208.80.152.32) 56(84) bytes of data. [16:04:09] 64 bytes from virt0.wikimedia.org (208.80.152.32): icmp_seq=1 ttl=55 time=57.1 ms [16:04:09] 64 bytes from virt0.wikimedia.org (208.80.152.32): icmp_seq=2 ttl=55 time=57.1 ms [16:04:33] wtfwri? [16:04:40] (wierd routing issue) [16:05:12] Coren: Works for me. [16:05:36] Ah, interesting. I get asymetric routing. [16:05:42] <^d> Yeah I can't ping either. [17:16:42] !ping [17:16:42] !pong [17:47:09] !ping [17:47:09] !pong [18:15:28] !ping [18:15:28] !pong [18:31:59] I don't know why but wm-bot is having network issues since migrated to eqiad [18:32:10] it never did disconnect so often as now [18:33:16] its instances are spread on multiple freenode servers and they all disconnect frequently on "ping timeout" despite it pings at least every 20 seconds [18:37:44] does eqiad not have its own freenode server yet? [18:38:40] SamB: It does; dickson.freenode.net [18:39:04] Though it's a "normal" server; the only eqiad-ness about it is that it is physically there. I has no other magic. [18:39:36] yes, but it still might be a good idea for wm-bot to prefer using that one? [18:51:27] SamB: for its primary instance maybe, but using multiple servers together sounds like a better idea to me [18:51:56] when one die lower number of channels is affected [18:52:34] !freenode is dickson.freenode.net [18:52:34] Key was added [19:29:54] !migration [19:29:54] https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration_Progress [19:41:45] That page just makes me think of http://www.youtube.com/watch?v=p-3e0EkvIEM [19:55:51] Hello, after migration my tool templatetiger disappear from the list: http://tools.wmflabs.org/ [19:56:53] How to bring it back. The tool itself works. [19:57:27] The list isn't going to appear correctly until migration end; you can look at the "correct" future list at http://tools-eqiad.wmflabs.org/ [19:57:44] Coren: tried 'qacct' on eqiad? I get '/var/lib/gridengine/default/common/accounting: No such file or directory' followed by 'no jobs running since startup', is there something I need to do? [19:58:07] Ok, thanks. I can life with it so long. [19:58:46] Nettrom: No, I probably just need to look into it; I don't think I've tested that yet. [19:59:57] Nettrom: Huh. Odd. [20:00:15] Coren: my thoughts exactly! :) [20:04:01] Bah! gridengine will only append to the file and not create it. [20:04:30] So yeah; it'll work, but only from this point on. [20:05:51] Coren: thanks for fixing that!