[07:26:46] 06Traffic: Deploy geoip lookup script on 2 hosts - https://phabricator.wikimedia.org/T393927#10858161 (10Fabfur) 05Open→03Resolved [07:29:27] 06Traffic: Deploy maxmind lookup configuration everywhere - https://phabricator.wikimedia.org/T395295 (10Fabfur) 03NEW [07:30:44] 06Traffic: Deploy maxmind lookup configuration everywhere - https://phabricator.wikimedia.org/T395295#10858196 (10Fabfur) [07:34:00] FIRING: AnycastHealthcheckerRestarted: anycast-healthchecker service restarted on durum7002:9100 - https://wikitech.wikimedia.org/wiki/Anycast#Anycast_healthchecker_not_running - https://grafana.wikimedia.org/d/dxbfeGDZk/anycast?orgId=1&var-protocol=BGP&var-site=magru&var-cluster=All&var-ip_version=All - https://alerts.wikimedia.org/?q=alertname%3DAnycastHealthcheckerRestarted [07:38:56] XioNoX: are you working on something that could related to the alert on durum7002? [07:39:00] RESOLVED: [2x] AnycastHealthcheckerRestarted: anycast-healthchecker service restarted on doh7002:9100 - https://wikitech.wikimedia.org/wiki/Anycast#Anycast_healthchecker_not_running - https://grafana.wikimedia.org/d/dxbfeGDZk/anycast?orgId=1&var-protocol=BGP&var-site=magru&var-cluster=All&var-ip_version=All - https://alerts.wikimedia.org/?q=alertname%3DAnycastHealthcheckerRestarted [07:39:10] :) [07:39:14] moritzm: ^ ? [07:39:35] fabfur: https://phabricator.wikimedia.org/T394263#10858200 [07:39:57] ack, tnx! [07:41:18] there'll be one more for ncredir7002 later as well [07:41:38] 👍 [07:45:59] 06Traffic, 06Experimentation Lab: SDS 2.4.4 Edge Uniques Production Cookie Deployment - https://phabricator.wikimedia.org/T391411#10858232 (10Vgutierrez) [07:57:33] fabfur: I need to do the same Ganeti change as above to bast7001, which will interrupt your ongoing SSH session(s) over it, can you switch to a different bastion or take a short break? [07:57:52] COFFEE TIME!!! [07:58:40] (I have several connections open, none of them is important, pls go ahead and don't worry) [07:58:49] ok, will do that now [07:59:53] isn't bast7001 awfully slow for you fabfur? [08:00:22] fabfur: bast7001 is back, you can use it again [08:00:59] vgutierrez: I use all dc bastions to connect to hosts in that dc [08:01:45] I do that for some and it looks like I should include bast7001 too [08:01:56] I'd save 7ms RTT apparently [08:02:36] why only for some? [08:03:33] bast7001 will soon be replaced, better hold on a few days with that one :-) [13:12:54] 06Traffic, 06DC-Ops, 10ops-codfw, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#10859334 (10Fabfur) Hi @Jhancock.wm, is there a rough timeline for this tasks so we can organize our work around it? [14:06:44] 06Traffic, 06DC-Ops, 10ops-codfw, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#10859545 (10Jhancock.wm) @Fabfur I'm planning on racking them tomorrow. I have a few servers ahead of it in the imaging queue but I'm hoping to have everything wrapped u... [15:19:36] 06Traffic, 06Experimentation Lab, 13Patch-For-Review: SDS 2.4.4 Edge Uniques Production Cookie Deployment - https://phabricator.wikimedia.org/T391411#10859833 (10Vgutierrez) [15:49:55] 10netops, 06Infrastructure-Foundations, 06SRE: Stage and configure new Juniper switches in codfw rows E/F - https://phabricator.wikimedia.org/T394021#10859945 (10Jhancock.wm) @cmooney got it set and confirmed it pings [15:50:41] 10netops, 06Infrastructure-Foundations, 06SRE: Stage and configure new Juniper switches in codfw rows E/F - https://phabricator.wikimedia.org/T394021#10859956 (10cmooney) >>! In T394021#10859945, @Jhancock.wm wrote: > @cmooney got it set and confirmed it pings awesome, thank you! [16:08:25] 06Traffic, 07Browser-Support-Apple-Safari, 07Browser-Support-Firefox, 07Browser-Support-Google-Chrome, 07User-notice: Discovery: Deprecation of TLS 1.2 - https://phabricator.wikimedia.org/T367821#10860123 (10Bugreporter) This only affect 3rd party tools like AWB, not Edge browser which is based on Chromi... [16:16:02] 06Traffic, 07Browser-Support-Apple-Safari, 07Browser-Support-Firefox, 07Browser-Support-Google-Chrome, 07User-notice: Discovery: Deprecation of TLS 1.2 - https://phabricator.wikimedia.org/T367821#10860193 (10BBlack) FYI from the edge TLS stats POV (the first graph in the description), if we ignore the `n... [18:00:33] 06Traffic, 06SRE, 13Patch-For-Review: Lower geodns TTLs for dyna.wm.org and upload.wm.org from 300s (5 min) to 180s (3 min) - https://phabricator.wikimedia.org/T394312#10860899 (10ssingh) [18:01:19] 06Traffic, 06SRE, 13Patch-For-Review: Lower geodns TTLs for dyna.wm.org and upload.wm.org from 300s (5 min) to 180s (3 min) - https://phabricator.wikimedia.org/T394312#10860906 (10ssingh) 05Open→03Resolved a:03ssingh [18:07:12] 06Traffic, 13Patch-For-Review: Import BingBot prefixes - https://phabricator.wikimedia.org/T395358#10860922 (10Aklapper) [20:04:52] 06Traffic, 06collaboration-services, 13Patch-For-Review, 10Release-Engineering-Team (Radar): Separate Gerrit https and ssh/git hostnames - https://phabricator.wikimedia.org/T394271#10861275 (10LSobanski) >>! In T394271#10850552, @thcipriani wrote: > Splitting traffic will mean acute, short-term pain (there... [21:14:40] 06Traffic, 06collaboration-services, 10Gerrit, 13Patch-For-Review, 10Release-Engineering-Team (Radar): Separate Gerrit https and ssh/git hostnames - https://phabricator.wikimedia.org/T394271#10861431 (10A_smart_kitten) [21:18:54] 06Traffic, 10DNS, 06serviceops, 06SRE, 13Patch-For-Review: Create redirect from tj.*.org to tg.*.org - https://phabricator.wikimedia.org/T393803#10861433 (10Scott_French) Ah, I realized when cross-checking https://gerrit.wikimedia.org/r/1148981 against DNS that the `tj` records have not been added yet th...