[08:44:31] <wikibugs>	 06Traffic, 06SRE, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Sustainability (Incident Followup): alerts should be triggered if druid fails to consume webrequest_sampled kafka topic - https://phabricator.wikimedia.org/T410019#11399416 (10Ge...
[10:09:44] <wikibugs>	 06Traffic, 10Hiddenparma, 13Patch-For-Review: Add ipblock-source objects and logic - https://phabricator.wikimedia.org/T402014#11399891 (10JMeybohm) I've added tree ipblock sources we haven't had in `fetch_external_clouds_vendors_nets` so far: - duckduckbot: Googlebot format - telegrambot, rssapi: Plaintext...
[10:22:09] <wikibugs>	 06Traffic, 06SRE, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Sustainability (Incident Followup): alerts should be triggered if druid fails to consume webrequest_sampled kafka topic - https://phabricator.wikimedia.org/T410019#11399911 (10Ge...
[12:40:43] <jinxer-wm>	 FIRING: HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=esams&var-instance=cp3070&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[12:45:43] <jinxer-wm>	 FIRING: [17x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages  - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[12:55:43] <jinxer-wm>	 RESOLVED: [17x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages  - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[14:27:42] <sukhe>	 hcaptcha deploy starting soon. puppet will be disabled, please don't enable it without asking me, thanks
[14:28:26] <wikibugs>	 06Traffic, 06SRE, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Sustainability (Incident Followup): alerts should be triggered if druid fails to consume webrequest_sampled kafka topic - https://phabricator.wikimedia.org/T410019#11400587 (10Ge...
[14:34:56] <wikibugs>	 06Traffic, 10observability, 06SRE, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), and 3 others: alerts should be triggered if druid fails to consume webrequest_sampled kafka topic - https://phabricator.wikimedia.org/T410019#11400611 (10Gehel)
[14:35:49] <wikibugs>	 06Traffic, 10observability, 06SRE, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), and 3 others: alerts should be triggered if druid fails to consume webrequest_sampled kafka topic - https://phabricator.wikimedia.org/T410019#11400632 (10Gehel) As webrequest is critical for operational support,...
[14:57:04] <sukhe>	 enabled again
[15:39:13] <wikibugs>	 06Traffic, 10observability, 06SRE, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), and 3 others: alerts should be triggered if druid fails to consume webrequest_sampled kafka topic - https://phabricator.wikimedia.org/T410019#11401023 (10Gehel) 05Open→03Resolved
[15:54:09] <wikibugs>	 10netops, 06Infrastructure-Foundations: rancid: message has lines too long for transport - https://phabricator.wikimedia.org/T410606#11401128 (10LSobanski) p:05Triage→03Low
[16:17:30] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Codfw row C/D servers need to boot/reimage in UEFI mode - https://phabricator.wikimedia.org/T410910 (10cmooney) 03NEW p:05Triage→03Medium
[16:17:41] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Codfw row C/D servers need to boot/reimage in UEFI mode - https://phabricator.wikimedia.org/T410910#11401242 (10cmooney)
[16:27:09] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Codfw row C/D servers need to boot/reimage in UEFI mode - https://phabricator.wikimedia.org/T410910#11401277 (10cmooney)
[17:05:43] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Codfw row C/D servers need to boot/reimage in UEFI mode - https://phabricator.wikimedia.org/T410910#11401603 (10cmooney)
[17:53:47] <wikibugs>	 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Measure request frequency of thumbnail sizes - https://phabricator.wikimedia.org/T410304#11401955 (10akosiaris) Turnilo for the Telegram Logo (first hit in what @Ladsgroup ) says: Google Proxy as the ISP, in an staggering 85% o...
[18:58:24] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11402451 (10RobH) Day 9 Update: * 9 hosts moved, 10 remain - 300 hosts total at start of migration * John worked with Ben directly to migrate the (8) Data Pla...
[22:32:12] <wikibugs>	 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Measure request frequency of thumbnail sizes - https://phabricator.wikimedia.org/T410304#11403354 (10Ladsgroup) ` spark-sql (default)> select uri_path, count(*) as hits from wmf.webrequest where webrequest_source='upload' and y...
[22:53:48] <wikibugs>	 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Measure request frequency of thumbnail sizes - https://phabricator.wikimedia.org/T410304#11403412 (10Ladsgroup) The query was wrong, the like should have an extra % at the end. Let me try again.
[22:55:24] <wikibugs>	 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Measure request frequency of thumbnail sizes - https://phabricator.wikimedia.org/T410304#11403413 (10Ladsgroup) ` spark-sql (default)> select uri_path, count(*) as hits from wmf.webrequest where webrequest_source='upload' and y...