[01:20:51] (03PS7) 10Reedy: README.md: Code isn't in Phabricator repo :) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/925029 [01:21:00] (03PS3) 10Reedy: package.json: Update homepage [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/925032 [01:21:15] (03PS3) 10Reedy: GraphPanel.vue: Move . into i18n messages [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/925027 (https://phabricator.wikimedia.org/T337032) [01:21:28] (03PS3) 10Reedy: GraphPanel.vue: Make "Filter and split this metric" translateable [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/925028 (https://phabricator.wikimedia.org/T337032) [01:21:31] (03PS3) 10Reedy: i18n: Bring in Month abbreviations [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/925030 (https://phabricator.wikimedia.org/T336815) [01:21:39] (03PS2) 10Reedy: Remove unused consts from *.config.js [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/925064 [01:21:43] (03PS4) 10Reedy: Update footer links [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/925031 (https://phabricator.wikimedia.org/T336765) [01:22:40] (03Abandoned) 10Reedy: package-lock.json: Update version to 2.10.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/925065 (owner: 10Reedy) [01:23:54] (03PS1) 10Snwachukwu: Add new coeditors_metrics hql file to support temp account changes. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1084282 (https://phabricator.wikimedia.org/T377352) [02:16:16] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data Pipelines, 06Data Products, 06Privacy Engineering, 07Epic: Add more languages to Wikipedia Clickstream - https://phabricator.wikimedia.org/T289532#10275463 (10Ahoelzl) a:03JAllemandou [02:24:31] 06Data-Engineering, 07Epic: Data Platform Data Incidents - https://phabricator.wikimedia.org/T378559 (10Ahoelzl) 03NEW [02:24:53] 06Data-Engineering, 07Epic: Data Platform Data Incidents - https://phabricator.wikimedia.org/T378559#10275492 (10Ahoelzl) [07:25:05] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06DBA, 07Schema-change-in-production: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781#10275645 (10ABran-WMF) 05Stalled→03In progress ack, resumed this instant [09:13:40] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06DBA, 07Schema-change-in-production: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781#10275932 (10ABran-WMF) >>! In T367781#10274534, @Ladsgroup wrote: > Without looking at the logs, db2155 is dump replica... [09:45:55] 06Data-Engineering, 06Data Products, 06Traffic: Prepare puppet configuration to send haproxy logs to haproxykafka socket - https://phabricator.wikimedia.org/T374473#10275985 (10Fabfur) 05Open→03Resolved [09:47:06] 06Data-Engineering, 06Data Products, 06Traffic, 13Patch-For-Review: New software: haproxykafka - https://phabricator.wikimedia.org/T370668#10275978 (10Fabfur) 05Open→03In progress p:05Triage→03High [09:53:55] 06Data-Engineering, 06Traffic: Rollout haproxykafka on all hosts - https://phabricator.wikimedia.org/T378578 (10Fabfur) 03NEW [10:27:55] (03CR) 10Joal: [C:03+1] "I have not reviewed the field list, I trust you on this 😊" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1084208 (https://phabricator.wikimedia.org/T364398) (owner: 10Snwachukwu) [10:33:10] (03CR) 10Joal: Add Query for cu_log table in sqoop. (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1082515 (https://phabricator.wikimedia.org/T364398) (owner: 10Snwachukwu) [10:33:26] (03CR) 10Joal: [V:03+2 C:03+2] "Merging for deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1084208 (https://phabricator.wikimedia.org/T364398) (owner: 10Snwachukwu) [10:42:14] Hi folks, I'm going for a heavy train-deploy today, sorry for the noise about manual ops :) [10:43:37] !log Pause webrequest-refinement airflow jobs (2) before starting deployment as table change needs to happen [10:43:40] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:44:51] (03CR) 10Joal: [V:03+2] Shift is_redirect_to_pageview upstream to webrequest [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1078733 (https://phabricator.wikimedia.org/T375527) (owner: 10Milimetric) [10:44:58] (03PS5) 10Mforns: Shift is_redirect_to_pageview upstream to webrequest [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1078733 (https://phabricator.wikimedia.org/T375527) (owner: 10Milimetric) [10:45:00] (03CR) 10Joal: [C:03+2] Shift is_redirect_to_pageview upstream to webrequest [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1078733 (https://phabricator.wikimedia.org/T375527) (owner: 10Milimetric) [10:45:03] (03CR) 10Joal: [V:03+2 C:03+2] Shift is_redirect_to_pageview upstream to webrequest [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1078733 (https://phabricator.wikimedia.org/T375527) (owner: 10Milimetric) [10:46:43] (03CR) 10Joal: [V:03+2 C:03+2] "Merging for deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1082543 (https://phabricator.wikimedia.org/T375527) (owner: 10Mforns) [10:47:02] (03PS3) 10Mforns: Modify the automated traffic detection pipeline to include redirects [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1082543 (https://phabricator.wikimedia.org/T375527) [10:47:04] (03CR) 10Joal: [V:03+2 C:03+2] Modify the automated traffic detection pipeline to include redirects [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1082543 (https://phabricator.wikimedia.org/T375527) (owner: 10Mforns) [11:16:03] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06DBA, 07Schema-change-in-production: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781#10276277 (10Ladsgroup) >>! In T367781#10275932, @ABran-WMF wrote: >>>! In T367781#10274534, @Ladsgroup wrote: >> Without... [11:33:50] !log Deploying refinery using scap [11:33:51] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:37:10] (03PS2) 10Gerrit maintenance bot: Add tcy.wiktionary to pageview allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1084071 (https://phabricator.wikimedia.org/T378463) [11:38:24] (03CR) 10Joal: [V:03+2 C:03+2] "Merging" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1084071 (https://phabricator.wikimedia.org/T378463) (owner: 10Gerrit maintenance bot) [11:42:37] (03PS2) 10Joal: Add tcy.wikisource to pageview allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1084074 (https://phabricator.wikimedia.org/T378471) (owner: 10Gerrit maintenance bot) [11:45:54] (03PS3) 10Joal: Add tcy.wikisource to pageview allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1084074 (https://phabricator.wikimedia.org/T378471) (owner: 10Gerrit maintenance bot) [11:46:23] (03CR) 10Joal: [V:03+2 C:03+2] "Merging" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1084074 (https://phabricator.wikimedia.org/T378471) (owner: 10Gerrit maintenance bot) [11:51:30] !log Alter wmf.webrequest table adding new is_redirect_to_pageview field - NOTE: needed to be done in 2 steps with Hive to have the field correctly inserted in the middle of the table [11:51:32] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:53:30] !log Create new table wmf_raw.mediawiki_private_cu_log to handle newly sqooped data [11:53:32] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:56:55] !log Alter tables wmf.webrequest_actor_metrics_hourly, wmf.webrequest_actor_metrics_rollup_hourly, wmf.webrequest_actor_label_hourly adding 2 new fields - NOTE: needed to be done in 2 steps with Hive to have the field correctly inserted in the middle of the tables [11:56:57] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:06:31] !log Alter wmf.webrequest table adding new is_redirect_to_pageview field to the test cluster [12:06:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:10:37] !log pause webrequest_actor jobs (3) to wait for webrequest success before trying new code [12:10:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:13:40] !log deploying refinery onto HDFS (prod + test) [12:13:42] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:21:08] !log unpause webrequest refine airflow jobs [12:21:09] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:31:33] (03CR) 10Dbrant: [C:03+1] "@Joal Mind looking and merging?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1083192 (https://phabricator.wikimedia.org/T377259) (owner: 10GOlson) [12:40:48] (03CR) 10Snwachukwu: "Thank you Joseph!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1084208 (https://phabricator.wikimedia.org/T364398) (owner: 10Snwachukwu) [12:41:15] (03CR) 10Joal: "Hey Dmitry, I'm not very knowledgeable on the event-sanitization. I'll leave it to @mforns@wikimedia.org or @dandreescu@wikimedia.org." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1083192 (https://phabricator.wikimedia.org/T377259) (owner: 10GOlson) [13:36:54] !log Unpause webrequest_actor jobs after successful webrequest refinement [13:36:56] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:57:01] !log Scap deploy analytics airflow dags [13:57:04] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:03:49] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data Pipelines, 06Data Products, 06Privacy Engineering, 07Epic: Add more languages to Wikipedia Clickstream - https://phabricator.wikimedia.org/T289532#10276952 (10JAllemandou) Code has been merged and deployed. I'll keep this task open until I... [14:09:06] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data-Platform, 06Traffic, 10Data Products (Data Products (Data Products Sprint 21 🪂)): NEW BUG REPORT - Issues in calculation logic for unique devices tables - https://phabricator.wikimedia.org/T375527#10276969 (10JAllemandou) The code has been de... [14:35:21] (03CR) 10Joal: Add Query for cu_log table in sqoop. (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1082515 (https://phabricator.wikimedia.org/T364398) (owner: 10Snwachukwu) [15:05:53] 10Quarry, 10superset.wmcloud.org, 10cloud-services-team (FY2024/2025-Q1-Q2): Replace Quarry with an installation of Superset - https://phabricator.wikimedia.org/T169452#10277359 (10joanna_borun) 05Open→03Declined [15:18:27] 06Data-Engineering, 06cloud-services-team, 10Data-Services, 06Privacy Engineering: Increased visibility in wiki-replicas for volunteers fighting vandals - https://phabricator.wikimedia.org/T284944#10277439 (10joanna_borun) p:05Triage→03High [15:48:48] 06Data-Engineering, 06Product-Analytics, 10Wmfdata-Python: Wmfdata-Python triggers a Pandas warning during mariadb.run and hive.run - https://phabricator.wikimedia.org/T324135#10277601 (10jsn.sherman) See https://github.com/wikimedia/wmfdata-python/pull/54/ [16:03:29] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data-Platform, 10Temporary accounts: Add MW table 'cu_log' to data lake - https://phabricator.wikimedia.org/T364398#10277694 (10Snwachukwu) As [[ https://wikimedia.slack.com/archives/C07U36RRHHA/p1730226062099069 | requested ]] by @jwang and @mpopo... [16:06:10] 06Data-Engineering, 10Data-Platform-SRE (2024.10.19 - 2024.11.08): MaxMind seems to be mapping the same IP to different countries - https://phabricator.wikimedia.org/T366369#10277753 (10brouberol) With the exception of a worker that seemingly can't be reached, the DB seem to be the exact same everywhere: ` bro... [16:13:01] (03CR) 10Shay Nowick: [C:03+1] "LGTM" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1083192 (https://phabricator.wikimedia.org/T377259) (owner: 10GOlson) [16:15:10] 06Data-Engineering, 10Data-Platform-SRE (2024.10.19 - 2024.11.08): MaxMind seems to be mapping the same IP to different countries - https://phabricator.wikimedia.org/T366369#10277856 (10brouberol) @fkaelin Are we still observing the same discrepancy on your end? [18:05:31] 06Data-Engineering, 06cloud-services-team, 10Data-Services, 06Privacy Engineering: Increased visibility in wiki-replicas for volunteers fighting vandals - https://phabricator.wikimedia.org/T284944#10278336 (10fnegri) > excluding all data related to abusefilter (e.g. abusefilter, abuse_filter, abuse_filter_... [18:50:30] (03CR) 10Mforns: Add new coeditors_metrics hql file to support temp account changes. (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1084282 (https://phabricator.wikimedia.org/T377352) (owner: 10Snwachukwu) [19:12:46] Hey, it looks like archiva1002 is nearly full - anything I can do to help with that? [19:13:24] Same with snapshot1012 [19:14:46] /var/cache/archiva/ has a significant-sized temporary file blob that looks ripe for removal [19:18:24] That temp dir is still in use by the archiva process so I won't touch anything [19:28:04] (03PS1) 10Jennifer Ebe: Add New Edit Hourly HQL for Temp Account Change [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1084873 (https://phabricator.wikimedia.org/T377767) [19:29:52] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data-Platform, 10Temporary accounts: Add MW table 'cu_log' to data lake - https://phabricator.wikimedia.org/T364398#10278727 (10mpopov) Thank you @Snwachukwu! [19:39:54] (03PS2) 10Jennifer Ebe: Add New Edit Hourly HQL for Temp Account Change [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1084873 (https://phabricator.wikimedia.org/T377767) [19:55:46] (03PS3) 10Jennifer Ebe: Add New Edit Hourly HQL for Temp Account Change [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1084873 (https://phabricator.wikimedia.org/T377767)