[05:54:01] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Review analytics-in4/6 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10elukey) From my tcpdumps it seems that no more https calls are made via ipv6 without going through the proxy. @ayounsi, we can proceed... [07:08:07] 10netops, 10Operations, 10monitoring, 10User-fgiunchedi: Update ACLs for newer graphite hosts - https://phabricator.wikimedia.org/T202846 (10fgiunchedi) p:05Triage>03Normal [08:07:45] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Review analytics-in4/6 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10elukey) [08:25:07] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache servers to stretch - https://phabricator.wikimedia.org/T200445 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ema on sarin.codfw.wmnet for hosts: ``` ['cp2019.codfw.wmnet', 'cp2017.codfw.wmnet'] ``` The log can be found in `/var/log/w... [08:53:20] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache servers to stretch - https://phabricator.wikimedia.org/T200445 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['cp2019.codfw.wmnet', 'cp2017.codfw.wmnet'] ``` and were **ALL** successful. [09:18:37] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache servers to stretch - https://phabricator.wikimedia.org/T200445 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ema on sarin.codfw.wmnet for hosts: ``` ['cp3034.esams.wmnet', 'cp2020.codfw.wmnet'] ``` The log can be found in `/var/log/w... [09:50:46] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache servers to stretch - https://phabricator.wikimedia.org/T200445 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['cp2020.codfw.wmnet', 'cp3034.esams.wmnet'] ``` and were **ALL** successful. [10:34:21] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache servers to stretch - https://phabricator.wikimedia.org/T200445 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ema on sarin.codfw.wmnet for hosts: ``` ['cp3035.esams.wmnet', 'cp2024.codfw.wmnet'] ``` The log can be found in `/var/log/w... [11:06:32] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache servers to stretch - https://phabricator.wikimedia.org/T200445 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['cp2024.codfw.wmnet', 'cp3035.esams.wmnet'] ``` and were **ALL** successful. [12:00:29] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache servers to stretch - https://phabricator.wikimedia.org/T200445 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts: ``` ['cp3037.esams.wmnet', 'cp3041.esams.wmnet'] ``` The log can be found in `/var/l... [12:33:17] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache servers to stretch - https://phabricator.wikimedia.org/T200445 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['cp3037.esams.wmnet', 'cp3041.esams.wmnet'] ``` and were **ALL** successful. [14:21:40] 10Traffic, 10Operations, 10Continuous-Integration-Config, 10Patch-For-Review: Add CI to all operations/software/varnish/* repositories and archive obsolete ones - https://phabricator.wikimedia.org/T180329 (10hashar) 05Open>03Resolved a:03ema CI has been configured by @ema via various tasks [14:48:31] 10Traffic, 10Analytics, 10Operations, 10Services (blocked): Add Accept header to webrequest logs - https://phabricator.wikimedia.org/T170606 (10Ottomata) Hm, either of these solutions is fine, but even if Accept isn't requested from others, it might be something fairly interesting to just include in the fu... [15:09:23] 10Wikimedia-Apache-configuration, 10Operations, 10Patch-For-Review: Redirect 2030.wikimedia.org to the new movement strategy portal - https://phabricator.wikimedia.org/T202498 (10nebber) Super cool, thanks a lot! (Can I like comments here, or thank people?) [15:30:22] 10Wikimedia-Apache-configuration, 10Operations, 10Patch-For-Review: Redirect 2030.wikimedia.org to the new movement strategy portal - https://phabricator.wikimedia.org/T202498 (10Dzahn) 05Open>03Resolved a:03Dzahn @nebber :) You can try "Award token" if you like. In general in Phabricator you can direc... [17:31:59] 10Traffic, 10Analytics, 10Operations, 10Services (blocked): Add Accept header to webrequest logs - https://phabricator.wikimedia.org/T170606 (10Ottomata) @Pchelolo we discussed this in standup today. If the data you need is small enough (can we filter on a URI?) and you only need a sample (say from a sing... [17:36:45] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Review analytics-in4/6 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10ayounsi) ```lang=diff [edit firewall family inet filter analytics-in4 term default then] - reject; + discard; [edit firewal... [17:39:59] 10Traffic, 10Analytics, 10Operations, 10Services (blocked): Add Accept header to webrequest logs - https://phabricator.wikimedia.org/T170606 (10Pchelolo) > If the data you need is small enough (can we filter on a URI?) and you only need a sample (say from a single cache host), AND if traffic folks don't mi... [17:50:23] 10Traffic, 10Operations, 10ops-eqsin: cp5001 unreachable since 2018-07-14 17:49:21 - https://phabricator.wikimedia.org/T199675 (10RobH) > Equinix Support, > > Please note this ticket is for both a site access/visitor ticket for two Unisys Engineers as well as a SmartHands Escort & Supervise ticket for those... [18:43:56] is there a doc about depooling a site? To add "downtime one of those alerts https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=traffic+drop" to the steps if any [18:45:31] 10Traffic, 10Operations, 10monitoring, 10Patch-For-Review: False alarms on varnish-http-requests 70% GET drop in 30 min alert - https://phabricator.wikimedia.org/T201630 (10ayounsi) 05Open>03Resolved All checks are green: https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=traffic+drop... [18:46:03] XioNoX: https://wikitech.wikimedia.org/wiki/Global_traffic_routing#Disabling_a_Site [18:46:34] thx! [19:34:11] 10netops, 10Operations, 10monitoring, 10User-fgiunchedi: Update ACLs for newer graphite hosts - https://phabricator.wikimedia.org/T202846 (10ayounsi) ```lang=diff [edit firewall family inet filter analytics-in4 term graphite from destination-address] + /* graphite1004 */ + 10.64.16.149/32;... [19:36:59] 10netops, 10Operations, 10monitoring, 10User-fgiunchedi: Update ACLs for newer graphite hosts - https://phabricator.wikimedia.org/T202846 (10ayounsi) 05Open>03Resolved Feel free to reopen or file a new task for the deletion part. [19:49:52] 10Traffic, 10netops, 10Operations, 10ops-ulsfo: ulsfo migration tracking - https://phabricator.wikimedia.org/T202433 (10RobH) @ayounsi: Can you review https://docs.google.com/spreadsheets/d/19f8XkjqQIKZ66uCY8vcEvqOdooTFxR8guLY6m_5yzXM/edit?usp=sharing and update the ulsfo transit/x-connections and/or confi... [20:30:46] 10Traffic, 10netops, 10Operations, 10ops-ulsfo: ulsfo migration tracking - https://phabricator.wikimedia.org/T202433 (10RobH) Ok, @ayounsi and I reviewed the xconnection listing on the google sheet. It is now accurate. Do we need to have new LoAs generated for this migration, or is it enough to simply pr... [20:55:07] 10Traffic, 10netops, 10Operations, 10ops-ulsfo: ulsfo migration tracking - https://phabricator.wikimedia.org/T202433 (10RobH) Also the most recent copy of the Unitedlayer invoice may help track down the links we dont pay for xconnects on, or that seem to terminate with UL directly rather than DR/Telx . [22:35:50] cache misc is merging into cache text, rgith? [22:36:22] i'm adding new backends (and eventually routes) for 3 new ganeti VMs for https://phabricator.wikimedia.org/T202011 [22:36:31] should I put the configs in misc.yaml, text.yaml, or both? [23:10:20] text [23:10:25] ottomata: ^ [23:10:36] it's already moved, we just haven't finished decomming the old stuff [23:19:59] ah ok great danke [23:21:37] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/455743/