[08:01:12] <wikibugs>	 10Data-Engineering (Q2 2024 October 1st - December 31th), 06DBA, 07Schema-change-in-production: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781#10342862 (10ABran-WMF) Will do, thanks for the info @bvibber !
[08:45:03] <jinxer-wm>	 FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_frontend ingested an unexpected number of records for a Kafka topic partition. ...
[08:45:04] <jinxer-wm>	 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_frontend&var-kafka_topic=webrequest_frontend_upload&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected
[09:43:24] <wikibugs>	 10Data-Engineering (Q2 2024 October 1st - December 31th), 06DBA, 07Schema-change-in-production: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781#10343043 (10ABran-WMF) @bvibber: logfile is in your home directoryon `deploy2002`: `/home/bvibber/mw-script.codfw.6497oh...
[09:55:08] <wikibugs>	 10Data-Engineering (Q2 2024 October 1st - December 31th), 06DBA, 07Schema-change-in-production: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781#10343058 (10ABran-WMF)
[10:23:31] <wikibugs>	 06Data-Engineering, 10ConfirmEdit (CAPTCHA extension), 06Data Products, 10MediaWiki-extensions-EventLogging, 10Metrics Platform: Send captcha API response data to event logging - https://phabricator.wikimedia.org/T379179#10343104 (10VirginiaPoundstone) @phuedx could be. I need a little more context. Let'...
[11:45:03] <jinxer-wm>	 FIRING: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_frontend ingested an unexpected number of records for a Kafka topic partition. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin  - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected
[12:11:04] <wikibugs>	 (03PS2) 10Btullis: Update Spark to version 3.5.3 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1093393 (https://phabricator.wikimedia.org/T338057)
[12:15:21] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Update Spark to version 3.5.3 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1093393 (https://phabricator.wikimedia.org/T338057) (owner: 10Btullis)
[13:10:03] <jinxer-wm>	 RESOLVED: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_frontend ingested an unexpected number of records for a Kafka topic partition. ...
[13:10:03] <jinxer-wm>	 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_frontend&var-kafka_topic=webrequest_frontend_upload&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected
[14:40:53] <wikibugs>	 10Data-Engineering (Q2 2024 October 1st - December 31th): load haproxykafka topics into HDFS via gobblin - https://phabricator.wikimedia.org/T377931#10344111 (10gmodena)
[14:42:49] <wikibugs>	 (03PS1) 10Gmodena: gobblin: webrequest_frontend: read from earliest offset [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1093926 (https://phabricator.wikimedia.org/T377931)
[14:50:58] <wikibugs>	 (03CR) 10Gmodena: [C:03+2] "+1 by Joseph in Slack." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1093926 (https://phabricator.wikimedia.org/T377931) (owner: 10Gmodena)
[14:51:01] <wikibugs>	 (03CR) 10Gmodena: [V:03+2 C:03+2] gobblin: webrequest_frontend: read from earliest offset [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1093926 (https://phabricator.wikimedia.org/T377931) (owner: 10Gmodena)
[15:31:40] <wikibugs>	 14Analytics-Radar, 06Product-Analytics: Investigate running Stan models on GPU - https://phabricator.wikimedia.org/T286493#10344405 (10mpopov) 05Declined→03Open I wanna give this a try.
[15:33:35] <wikibugs>	 06Data-Engineering, 06Data Products, 10MediaWiki-extensions-EventLogging, 10Metrics Platform, 07Technical-Debt: Migrate EventLogging to use DefaultEventSubmitter - https://phabricator.wikimedia.org/T375749#10344417 (10cjming) p:05Triage→03Low
[15:36:22] <wikibugs>	 14Analytics-Radar, 06Product-Analytics: Investigate running Stan models on GPU - https://phabricator.wikimedia.org/T286493#10344448 (10mpopov)
[15:36:26] <wikibugs>	 06Data-Engineering, 10Data Pipelines, 06Data Products, 10Dumps 2.0, 06Movement-Insights: Keep canonical_data.wikis updated - https://phabricator.wikimedia.org/T241741#10344450 (10cjming)
[15:44:35] <wikibugs>	 06Data-Engineering, 06Data Products, 10MediaWiki-extensions-EventLogging, 10MediaWiki-extensions-WikimediaEvents, and 3 others: Decide on how data platform wants to monitor bundle sizes - https://phabricator.wikimedia.org/T378772#10344508 (10cjming) p:05Triage→03Medium
[16:11:03] <jinxer-wm>	 FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_frontend ingested an unexpected number of records for a Kafka topic partition. ...
[16:11:03] <jinxer-wm>	 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_frontend&var-kafka_topic=webrequest_frontend_upload&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected
[16:16:03] <jinxer-wm>	 RESOLVED: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_frontend ingested an unexpected number of records for a Kafka topic partition. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin  - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected
[16:32:43] <wikibugs>	 06Data-Engineering, 10Event-Platform: Implement stream of HTML content on mw.page_change event - https://phabricator.wikimedia.org/T360794#10344796 (10leila) @ahoelzl (following up on our conversation on November 7th where I flagged the need for investing on HTML dumps to you) This task is the one that will be...
[16:33:03] <jinxer-wm>	 FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_frontend ingested an unexpected number of records for a Kafka topic partition. ...
[16:33:03] <jinxer-wm>	 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_frontend&var-kafka_topic=webrequest_frontend_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected
[16:42:02] <wikibugs>	 06Data-Engineering, 10Data-Platform-SRE (2024.11.09 - 2024.11.29): Design a suitable DAG deployment method - https://phabricator.wikimedia.org/T368033#10344885 (10brouberol) We had a very interesting talk with @Gehel @hashar @gmodena @dcausse and @JAllemandou this morning, and thanks to them, I think I found a...
[17:37:10] <fabfur>	 quick question: I need to apply these acls to the jumbo cluster: https://phabricator.wikimedia.org/T380373
[17:37:26] <fabfur>	 do I need to do that on all jumbo hosts or one is enough and gets replicated? 
[17:38:41] <fabfur>	 (I suppose yes, as they should be stored in zookeeper) 
[18:39:00] <wikibugs>	 (03PS1) 10Gmodena: Revert "gobblin: webrequest_frontend: read from earliest offset" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1093982
[18:39:24] <wikibugs>	 (03CR) 10Gmodena: [V:03+2 C:03+2] Revert "gobblin: webrequest_frontend: read from earliest offset" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1093982 (owner: 10Gmodena)
[19:07:15] <jinxer-wm>	 FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent
[20:03:03] <jinxer-wm>	 RESOLVED: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_frontend ingested an unexpected number of records for a Kafka topic partition. ...
[20:03:03] <jinxer-wm>	 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_frontend&var-kafka_topic=webrequest_frontend_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected
[20:19:21] <icinga-wm>	 PROBLEM - Webrequests Varnishkafka log producer on cp2038 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka
[20:19:57] <sukhe>	 looking
[20:21:21] <icinga-wm>	 RECOVERY - Webrequests Varnishkafka log producer on cp2038 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka
[20:47:15] <jinxer-wm>	 RESOLVED: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent
[20:49:46] <wikibugs>	 06Data-Engineering, 10Data Pipelines, 06Product-Analytics: Add TikTok's in-app browser to ua-parser library - https://phabricator.wikimedia.org/T325611#10346357 (10Cpetrillo) Following this as well. Do we still need Tiktok to weigh in on the UA edge cases for this to be unblocked?
[20:52:05] <wikibugs>	 06Data-Engineering, 10Data Pipelines, 06Product-Analytics: Add TikTok's in-app browser to ua-parser library - https://phabricator.wikimedia.org/T325611#10346369 (10Cpetrillo) Along with the above- is there a broader opportunity to update all known user agent strings and add new ones to ensure we are getting...