[07:59:31] so there are two alerts for cr1-eqisin [07:59:37] *eqsin [08:00:15] from https://wikitech.wikimedia.org/wiki/Network_design#/media/File:Wikimedia_network_overview.png it seems the Telia link from eqsin to codfw [08:00:52] and we have a scheduled maintenance [08:00:58] but the location says NY [08:04:00] there was a correspondent flap in OSPF with xe-5-1-2.cr1-codfw.wikimedia.org. [08:04:12] (now resolved) [08:05:27] ah I have now seen the email to ops-maintenance [08:05:39] it is indeed this link [08:09:52] I don't get why the BGP status hasn't recovered though [08:12:02] (External AS 1299): received NOTIFICATION code 6 (Cease) subcode 5 (Connection Rejected) [08:12:08] that might explain it :D [08:12:47] <_joe_> yeah :D [08:13:32] <_joe_> uhm we have a ripe alert on eqsin [08:13:38] <_joe_> I'd be tempted to depool it [08:14:07] <_joe_> nevermind they just recovered [08:19:10] I think that we should follow up with Telia and ask why the BGP session is still getting refused, but there might be something trivial that I don't see/do [08:20:02] RECOVERY - BGP status on cr1-eqsin is OK: BGP OK - up: 264, down: 2, shutdown: 0 [08:20:06] \o/ [08:20:31] Cc: XioNoX as FYI --^ [09:58:21] hallo all i plan to push https://gerrit.wikimedia.org/r/c/operations/puppet/+/510216/ in ~30 minutes. this changes adds a CI check to ensure all python files have a .py extention so the can be checked via CI. please shoult if you want me to hold off [10:02:09] <_joe_> I guess you tested it on the current state of the repo, so +1 [10:03:39] _joe_: last week i went through and updated most files i still have a few changes to rename and tidy the last few but otherwise should be good [10:04:33] what's happening with the erb files? [10:04:58] my change dosen't touch them right now will tackle that in a futre cr [10:05:09] 👍 [13:22:17] hey! I need some help powering on elastic2038. It was shutdown for memory replacement. Memory has been replaced and I need it back up to join its clusters [13:25:25] onimisionipe: hey! I'll help later on! [13:27:37] Ok. Thanks! [13:29:55] so my home connection also crapped out [13:30:23] nice timing [14:17:42] elukey: thanks! yeah for the traffic on tunnel link alerts I'd recommend to depool [14:29:23] but not right now I don't think! [16:05:34] jbond42: the maint-announce group should be fixed now! https://phabricator.wikimedia.org/T223388 turns out NONE of the available group types was selected.. somehow Google updated it so that the type was like "nothing" [16:06:06] selected "collab inbox" and "reset group" etc.. and it looks fixed. if you agree you can tell OIT we got it and they can close zendesk [16:16:15] jbond42: spoke too soon.. the button is back but apparently filters don't work anymore as before :( [16:16:25] thanks Google ..grrr [16:25:57] _joe_ ok if I roll restart nutcracker in codfw to pick up the new config? [16:27:01] mutante: great thanks [16:27:47] jbond42: except the filters stopped working and now i want to give that back to OIT .. i wonder if i can add messages to the zendesk ticket.. [16:27:55] seems like a legit google bug [16:28:14] yes good be [16:28:21] also wish they'd use a less closed system [16:28:52] well cant help there :) [16:28:56] now the status is: we can mark as "no action needed" but the "all unresolved" filter doesn't filter anything [16:29:07] jbond42: maybe with the ticket number they gave you? [16:31:37] "Collaborative Inbox: "Marked Complete" erratic behavior" .. erratic is the best [16:32:37] sorry meeting right now will catch up after [17:29:57] mutante: do you mean update te ticket i have with OIT or something elses? [17:30:26] jbond42: i mean let me know the ticket number so i can try to amend to it or if not possible refer to it in a new ticket [17:30:40] yes, the Zendesk one with OIT [17:30:48] techsupport@ [17:30:50] yes thats cool ticket i have is http://wmf.zendesk.com/hc/requests/18272 [17:30:58] ok, thanks! [17:31:22] ill see if i can add you not logged into zendesk before :) [17:35:57] mutante: you ar on the CC as well not sure if that allows you to update it but that is all i seem to be able to do [17:40:02] jbond42: i can login with Google and then see the ticket :) even just that is something nice. before i was thinking of just emails [17:44:35] wow, and yes, i can add comments to existing ticket from somebody else in the browser.. more than expected :) [17:51:13] i guess you can probably reply to the email as well then [18:14:59] hmm.. puppet-compiler says "(pending—Waiting for next available executor " but the one before that is finished [18:15:15] and in the past it would not say it like that even when it was actually waiting [18:21:46] looks like it is busy since hours on "puppet-compiler-test" on https://gerrit.wikimedia.org/r/c/operations/puppet/+/510249 .but that is already merged... trying to cancel that [18:23:51] or not.. just had to mention it and now things started to work [18:25:23] mutante: will https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/509926/ switch away from hhvm? [18:27:35] andrewbogott: dont know yet. looking at the compiler output https://puppet-compiler.wmflabs.org/compiler1001/16571/labweb1002.wikimedia.org/ .. ideally it would start the php-fpm process but it would be a separate step to change apache proxy_fcgi config [18:27:52] ok. I can depool one of the hosts and test if/when you want to try it [18:30:25] andrewbogott: ah, cool! yea, that would be useful but need to stare at it a bit more first [18:30:41] do you just type "depool" on the server for that? [18:34:22] mutante: yep! [18:34:27] and stop puppet on the other one :) [18:37:42] andrewbogott: ok, i think i'd like to do that later, but today. if you're ok with it i can just do it or i can first ping/warn [18:38:04] or i could schedule it ..whatever works [18:38:06] mutante: yep, just let me know. I'll be gone starting around 3:45 your time [18:38:14] alright! will do [20:35:16] hey bblack while you're here, what's the state of sitemaps for some wikis, if you know? (for use by google) [20:35:36] I remember last year (?) a few got done as a one-off to fix some google search cache issues [20:35:42] we have a new case, [20:36:01] https://phabricator.wikimedia.org/T223408 :-/ [21:27:51] hrmm.. phab1001 and phab1003 somehow have the same un-mapped v6 IP on their interface.. how did that happen [21:28:10] shouldn't that be MAC based [21:28:43] it's an additional one for phab vcs though.. probably from hiera [21:28:53] removing that from the non-prod host [21:31:19] mutante: could it be related to https://phabricator.wikimedia.org/T219803#5182817 (i didn;t have a chance to look into this today) [21:34:32] jbond42: hmm.. intersting, thanks. i think no.. but let me try to verify that [21:35:45] yes if your seeing it in ip addr probably not but i saw ipv6 and thoght i best shout :) [21:35:49] trying to "del" it from the interface it tells me "Cannot assign" .. well i dont want you to assign, i want you to delete :) [21:36:10] yep, thx. i would think facter gets it from the same place i see it [21:36:18] exactly [21:36:25] which box is it and which address [21:36:46] root@phab1003:~# ip -6 addr del 2620:0:861:ed1a::3:16/128 dev eno1 [21:36:47] * jbond42 reads back [21:37:35] oh.. wait.. nvm :) [21:37:37] its on the lo for some reason [21:37:41] yea, that :) [21:37:54] yea, removed.. and now running puppet [21:38:01] ack [21:38:46] "# phabricator's git backend uses a separate sshd with separate IPs" that's why it has these [21:38:57] they are defined in hiera [21:39:01] tha fact its on the loop back makes me think its part of lvs/pybal but i have not looked at that yet so a very uneducated guess [21:39:06] yea, it is [21:39:38] ok cool :D [21:39:59] "Warning: phabricator::vcs::listen_address is empty'" ok.. but will figure this out [21:40:32] that's my pending change i need to merge.. i think i'm good. thanks [21:59:40] and i emailed her may 8th, then again on 10th [21:59:41] and now today [21:59:46] bah, wrong window [21:59:56] (ignore my procurement complaints ;)