[02:28:18] FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [06:28:18] FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [07:11:40] 10Data-Engineering (Q2 2024 October 1st - December 31th): [SPIKE] Learn and document how to use Flink-CDC from MediaWiki MariaDB locally - https://phabricator.wikimedia.org/T373144#10385655 (10NoZeroDay) @Ottomata After some additional testing, the schema change evolution does indeed capture modification of colu... [09:27:32] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Discovery-Search, 10Data-Platform-SRE (2024.11.30 - 2024.12.20): Create and distribute a flink base image with flink 1.20.0 - https://phabricator.wikimedia.org/T377134#10385878 (10brouberol) a:03brouberol [09:41:49] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Discovery-Search, 10Data-Platform-SRE (2024.11.30 - 2024.12.20), 13Patch-For-Review: Create and distribute a flink base image with flink 1.20.0 - https://phabricator.wikimedia.org/T377134#10385897 (10brouberol) Hmm, seems like the build fails beca... [10:24:17] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Discovery-Search, 10Data-Platform-SRE (2024.11.30 - 2024.12.20), 13Patch-For-Review: Create and distribute a flink base image with flink 1.20.0 - https://phabricator.wikimedia.org/T377134#10385933 (10brouberol) 05Open→03Resolved ` == Step... [10:28:18] FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [12:24:00] 06Data-Engineering, 10Wmfdata-Python: Add type annotations to Wmfdata-Python and make the type-checking CI blocking - https://phabricator.wikimedia.org/T381656 (10nshahquinn-wmf) 03NEW [12:24:14] 06Data-Engineering, 10Wmfdata-Python: Add type annotations to Wmfdata-Python and make the type-checking CI blocking - https://phabricator.wikimedia.org/T381656#10386291 (10nshahquinn-wmf) p:05Triage→03Medium [12:28:35] 06Data-Engineering, 06Data-Platform, 06DBA, 07Schema-change-in-production: Change page.page_links_updated to fixed-length timestamp in wmf wikis - https://phabricator.wikimedia.org/T371742#10386305 (10Ladsgroup) [12:35:40] 06Data-Engineering, 10Wmfdata-Python: Fix lint issues in Wmfdata-Python and make the linting CI blocking - https://phabricator.wikimedia.org/T381657 (10nshahquinn-wmf) 03NEW p:05Triage→03Medium [12:38:10] 06Data-Engineering, 06Movement-Insights, 10Wikidata, 10Wikidata Analytics, and 2 others: Add linter and formatter to wmfdata-python (and link check) - https://phabricator.wikimedia.org/T348999#10386348 (10nshahquinn-wmf) @fkaelin good point that we should track the work needed to make the pipeline pass... [13:21:25] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10CirrusSearch, 03Discovery-Search (Current work), 13Patch-For-Review: Move oozie/util/swift/upload/ out of the refinery oozie folder - https://phabricator.wikimedia.org/T380343#10386452 (10Gehel) 05Open→03Resolved [13:24:20] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Traffic: Rollout haproxykafka on all hosts - https://phabricator.wikimedia.org/T378578#10386462 (10Fabfur) 05Open→03In progress [14:28:18] FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [15:28:16] 06Data-Engineering, 06Data-Platform, 10Dumps-Generation, 10Data-Platform-SRE (2024.11.30 - 2024.12.20): Hide autoblocks from the globalblocks table database dump - https://phabricator.wikimedia.org/T376726#10386758 (10xcollazo) > I'm more than happy to delete on the grounds of being tidy. I just don't like... [15:38:33] 06Data-Engineering, 06Data-Platform-SRE: Do performance testing of a big Hadoop Table hosted by Ceph - https://phabricator.wikimedia.org/T381416#10386786 (10BTullis) I'm all for this performance testing, but we haven't got //any// tables hosted by Ceph yet, so we can't run a similarly big query yet. Getting t... [16:52:34] 06Data-Engineering, 06Data-Platform, 10Dumps-Generation, 10Data-Platform-SRE (2024.11.30 - 2024.12.20): Hide autoblocks from the globalblocks table database dump - https://phabricator.wikimedia.org/T376726#10387100 (10BTullis) 05Open→03Resolved I have deleted the directories on both clouddumps serv... [16:54:54] 06Data-Engineering, 06Data-Platform, 10Dumps-Generation, 10Data-Platform-SRE (2024.11.30 - 2024.12.20): Hide autoblocks from the globalblocks table database dump - https://phabricator.wikimedia.org/T376726#10387111 (10Dreamy_Jazz) Hmm. I'm not getting a 404 even when I disable cache. {F57784797} [16:57:43] 06Data-Engineering, 06Data-Platform, 10Dumps-Generation, 10Data-Platform-SRE (2024.11.30 - 2024.12.20): Hide autoblocks from the globalblocks table database dump - https://phabricator.wikimedia.org/T376726#10387121 (10BTullis) >>! In T376726#10387111, @Dreamy_Jazz wrote: > Hmm. I'm not getting a 404 ev... [17:43:43] 06Data-Engineering, 06Data-Platform, 10Dumps-Generation, 10Data-Platform-SRE (2024.11.30 - 2024.12.20): Hide autoblocks from the globalblocks table database dump - https://phabricator.wikimedia.org/T376726#10387277 (10xcollazo) Maybe our rsync config attempts to honor last n dumps? [17:50:34] 06Data-Engineering, 06Data-Platform, 10Dumps-Generation, 10Data-Platform-SRE (2024.11.30 - 2024.12.20): Hide autoblocks from the globalblocks table database dump - https://phabricator.wikimedia.org/T376726#10387292 (10BTullis) 05Resolved→03Open >>! In T376726#10387277, @xcollazo wrote: > Maybe our rsyn... [18:28:18] FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [20:54:52] 06Data-Engineering, 06Research, 10Data-Platform-SRE (2024.11.30 - 2024.12.20), 03Discovery-Search (Current work): Low available space on Hadoop / HDFS - https://phabricator.wikimedia.org/T381707 (10Gehel) 03NEW [20:58:29] 06Data-Engineering, 06Research, 10Data-Platform-SRE (2024.11.30 - 2024.12.20), 03Discovery-Search (Current work): Low available space on Hadoop / HDFS - https://phabricator.wikimedia.org/T381707#10387906 (10Gehel) [20:58:59] 06Data-Engineering, 06Research, 10Data-Platform-SRE (2024.11.30 - 2024.12.20), 03Discovery-Search (Current work): Low available space on Hadoop / HDFS - https://phabricator.wikimedia.org/T381707#10387907 (10Gehel) [21:05:44] 06Data-Engineering, 06Research, 10Data-Platform-SRE (2024.11.30 - 2024.12.20), 03Discovery-Search (Current work): Low available space on Hadoop / HDFS - https://phabricator.wikimedia.org/T381707#10387913 (10Gehel) [21:45:52] 06Data-Engineering, 06Research, 10Data-Platform-SRE (2024.11.30 - 2024.12.20), 03Discovery-Search (Current work): Low available space on Hadoop / HDFS - https://phabricator.wikimedia.org/T381707#10387943 (10EBernhardson) For search-platform i reviewed the directories, we essentially have the following data... [22:28:18] FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent