[07:39:50] 10netops, 10Operations: Faulty port cr2-eqord:xe-0/1/1 - https://phabricator.wikimedia.org/T252988 (10ayounsi) p:05Triage→03Low [07:40:18] 10netops, 10Operations, 10ops-eqord: eqord - ulsfo Telia link down - IC-313592 - https://phabricator.wikimedia.org/T221259 (10ayounsi) 05Open→03Resolved a:03ayounsi Physically moving the optic to a different port solved the issue. Opened T252988 to troubleshot that specific issue. [08:33:02] 10Traffic, 10Operations, 10observability: prometheus-trafficserver-exporter: InsecureRequestWarning - https://phabricator.wikimedia.org/T252993 (10ema) [08:33:10] 10Traffic, 10Operations, 10observability: prometheus-trafficserver-exporter: InsecureRequestWarning - https://phabricator.wikimedia.org/T252993 (10ema) p:05Triage→03Low [09:00:37] 10Traffic, 10Operations, 10observability: prometheus-trafficserver-exporter: InsecureRequestWarning - https://phabricator.wikimedia.org/T252993 (10Vgutierrez) in this case is pretty obvious, ats-tls is the only instance listening on TLS only, but yeah, +1 to provide unique SylogIdentifiers [09:51:32] 10Traffic, 10Operations, 10Patch-For-Review: Discarded VCL files stuck in auto/busy state cause high number of backend probe requests - https://phabricator.wikimedia.org/T236754 (10ema) What is happening is quite clear: due to some sort of race when reloading and/or discarding VCL, `vcl->busy` isn't decreme... [10:33:07] 10Traffic, 10Operations, 10Patch-For-Review: Discarded VCL files stuck in auto/busy state cause high number of backend probe requests - https://phabricator.wikimedia.org/T236754 (10ema) I have tried getting rid of `vcl-e7ac6c17-ad61-4947-afdb-835f4eee6caa` on cp3050 at ~09:12 by setting `busy` to 0. After a... [13:01:42] 10Acme-chief, 10cloud-services-team (Kanban): tools/toolsbeta: improve acme-chief integration - https://phabricator.wikimedia.org/T252762 (10aborrero) Sorry, I forgot what thing we agreed on? Anyway, I agree with everything that simplifies the setup :-) I'm kind of blocked by this. It would be interesting for... [13:35:16] 10Wikimedia-Apache-configuration, 10Documentation: https://www.wikimedia.org/api/ links don't work - https://phabricator.wikimedia.org/T203155 (10Aklapper) >>! In T203155#6105486, @DannyS712 wrote: > The content comes from https://meta.wikimedia.org/wiki/API_listing_template per {T107086} Thanks for finding t... [13:38:13] <_joe_> ema, vgutierrez I'm going to take over puppet on the caches soon to start consuming from kafka from purged [13:38:24] ack [13:38:25] <_joe_> it should just take a few minutes though [13:38:50] I'm still waiting on our lovely CI for a V:+2 on an ATS release [13:38:54] <_joe_> given purged doesn't restart with my change, I will just do forced puppet runs [13:38:54] so you have some time :) [13:39:03] <_joe_> vgutierrez: see you tomorrow? :P [13:39:17] probably.. I have 3 meetings in a row starting soon [13:39:29] Monday pleasures :) [14:25:15] 10Traffic, 10Operations, 10vm-requests: Create a Ganeti VM for Wikidough - https://phabricator.wikimedia.org/T253024 (10ssingh) [14:48:25] 10Traffic, 10Operations, 10serviceops, 10Patch-For-Review, and 2 others: Make CDN purges reliable - https://phabricator.wikimedia.org/T133821 (10Joe) Status update: purged is now consuming purges from restbase directly via kafka and not via multicast anymore. This should unblock the complete migration of c... [15:05:46] 10netops, 10Operations: RRDP status alert - https://phabricator.wikimedia.org/T245121 (10ayounsi) 05Open→03Resolved a:03ayounsi * Routinator upgraded in T252010. Which helped to remove the "dubious" targets. * Since this task has been opened, proxies have been moved to new hosts and performance has incre... [16:42:36] 10Traffic, 10Analytics, 10Analytics-Kanban, 10EventStreams, and 2 others: EventStreams drops the connection after 15 minutes, which makes it unreliable - https://phabricator.wikimedia.org/T242767 (10Ottomata) Hio, I see some upgrades happened, should they have fixed this? [17:23:30] 10Traffic, 10Analytics, 10Analytics-Kanban, 10EventStreams, and 2 others: EventStreams drops the connection after 15 minutes, which makes it unreliable - https://phabricator.wikimedia.org/T242767 (10Ottomata) Heh, just tried myself, I guess not; still got disconnected after 15 minutes. [19:45:19] 10netops, 10Operations, 10ops-eqiad: asw2-d1-eqiad:VCP failure - https://phabricator.wikimedia.org/T252797 (10wiki_willy) a:03Jclark-ctr