[07:28:18] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache servers to stretch - https://phabricator.wikimedia.org/T200445 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts: ``` ['cp4032.ulsfo.wmnet', 'cp2005.codfw.wmnet'] ``` The log can be found in `/var/l... [07:28:31] moar upgrades!! [08:01:05] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache servers to stretch - https://phabricator.wikimedia.org/T200445 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['cp2005.codfw.wmnet', 'cp4032.ulsfo.wmnet'] ``` and were **ALL** successful. [08:01:13] \o/ [08:14:41] :D [08:15:04] **ALL** [08:21:54] :-P [09:03:16] 10Traffic, 10Operations, 10TechCom-RFC, 10Patch-For-Review, and 3 others: Harmonise the identification of requests across our stack - https://phabricator.wikimedia.org/T201409 (10mobrovac) >>! In T201409#4500541, @Joe wrote: > We also need internal requests to be traced, so I would assume we need all servi... [09:29:32] 10Traffic, 10netops, 10Operations, 10Patch-For-Review: eqiad row D switch upgrade - https://phabricator.wikimedia.org/T172459 (10Marostegui) [09:35:08] 10Traffic, 10Operations, 10Wikidata, 10wikiba.se, and 2 others: [Task] move wikiba.se webhosting to wikimedia misc-cluster - https://phabricator.wikimedia.org/T99531 (10Addshore) >>! In T99531#4513194, @abian wrote: >>>! In T99531#4411395, @abian wrote: >> wikiba.se is a bit unstable. Today it has been dow... [10:52:07] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache servers to stretch - https://phabricator.wikimedia.org/T200445 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ema on neodymium.eqiad.wmnet for hosts: ``` ['cp4031.ulsfo.wmnet', 'cp2026.codfw.wmnet'] ``` The log can be found in `/var/l... [11:24:30] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache servers to stretch - https://phabricator.wikimedia.org/T200445 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['cp2026.codfw.wmnet', 'cp4031.ulsfo.wmnet'] ``` and were **ALL** successful. [11:24:45] on a roll [11:31:37] ema: re:cumin aliases, will the misc hosts be moved to spare role soon? [11:32:40] volans: soonish I think, yes [11:32:54] ack, thx [12:41:58] bblack: ema hey, sorry to bother again. I'm still working on poolcounter to limit number of connections to ores.wm.o and the biggest problem is whitelisting internal IPs, I was thinking maintaining a list of internal IPs using puppet, which is good but for a short term solution for now, is it possible to depend on a request header? a header that is different when something is request behind varnish [12:43:46] Amir1: all internal IPs? [12:44:11] volans: yup if possible (I'm not sure) [12:44:45] Using a notion in header seems easier but also easier to work around [12:44:46] then see $production_networks in puppet (modules/network/manifests/constants.pp) [12:47:50] Thanks [15:01:12] 10Traffic, 10Operations, 10TechCom-RFC, 10Patch-For-Review, and 3 others: Harmonise the identification of requests across our stack - https://phabricator.wikimedia.org/T201409 (10Imarlier) >>! In T201409#4513970, @mobrovac wrote: > > If a service receives a request without a req id it means we have a hole... [15:18:21] 10Traffic, 10Operations, 10TechCom-RFC, 10Patch-For-Review, and 3 others: Harmonise the identification of requests across our stack - https://phabricator.wikimedia.org/T201409 (10Ottomata) > We also need internal requests to be traced, so I would assume we need all services to generate a request Id wheneve... [15:22:40] 10Traffic, 10Operations, 10TechCom-RFC, 10Patch-For-Review, and 3 others: Harmonise the identification of requests across our stack - https://phabricator.wikimedia.org/T201409 (10Pchelolo) > unique user agent (generally the name of the calling job or application). Might be worth exploring doing that as wel... [19:59:26] 10Traffic, 10Fundraising-Backlog, 10Operations, 10fundraising-tech-ops: SSL cert for links.email.wikimedia.org - https://phabricator.wikimedia.org/T188561 (10Jgreen) I reran the SSLLabs analyzer on links.e.uso.org today and it's still scored a B, looks like for several issue (still including weak DH). [23:05:57] 10Traffic, 10Operations, 10monitoring: False alarms on varnish-http-requests 70% GET drop in 30 min alert - https://phabricator.wikimedia.org/T201630 (10fgiunchedi) I believe this alert has fired a few times now and most were false positives, also it is not clear what's the actionable. I went ahead and "soft...