[00:33:26] RECOVERY - Check unit status of monitor_refine_eventlogging_legacy on an-launcher1002 is OK: OK: Status of the systemd unit monitor_refine_eventlogging_legacy https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [09:52:13] (03CR) 10Kosta Harlan: [C: 03+1] mediawiki/client/error: Add component field [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/700244 (owner: 10Gergő Tisza) [09:53:57] (03CR) 10Kosta Harlan: mediawiki/client/error: Add canonical special page name (031 comment) [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/700243 (https://phabricator.wikimedia.org/T266504) (owner: 10Gergő Tisza) [10:57:59] 10Analytics, 10Analytics-Data-Quality, 10Datasets-Webstatscollector: Add alarms for high volume of views to pages with replacement characters - https://phabricator.wikimedia.org/T117945 (10Nikerabbit) I believe Language-Team was tagged to provide input, which was given, so I am removing the tag. I also notic... [12:32:06] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: EventLogging background queue beforeunload event handler blocks Back-Forward cache - https://phabricator.wikimedia.org/T285220 (10Gilles) [12:40:35] hi all! [12:41:55] Hiya mforns [12:42:24] Hi the rest of the team! [12:42:39] When is Ben starting? Is it today? [12:48:37] Thanks to nuria for sharing the kafka streams otter book, I’ll paste it again in case anybody missed out :) https://www.gentlydownthe.stream/ [12:50:00] razzi: I think it's next monday, but might be wrong! [12:52:21] Re: stream book, I feel like I finally understand kafka partitions and consumer groups! [13:23:42] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade the Hadoop masters to Debian Buster - https://phabricator.wikimedia.org/T278423 (10razzi) Didnt' end up doing the failover last week since it was all hands; I think this can be done whenever, @elukey how do you feel about me doing the fail... [13:24:02] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review, 10User-razzi: Upgrade the Hadoop masters to Debian Buster - https://phabricator.wikimedia.org/T278423 (10razzi) [13:24:34] razzi: anytime, +1 [13:24:35] :) [13:28:21] razzi: mforns yep, next monday! [13:30:48] o/ [13:32:37] o/ ottomata ! [13:34:35] elukey: an-master1002 looks healthy, doing the failover to an-master1002, will leave it there for a couple days and monitor it [13:35:13] !log sudo -u hdfs /usr/bin/hdfs haadmin -failover an-master1001-eqiad-wmnet an-master1002-eqiad-wmnet [13:35:15] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:35:25] Failover to NameNode at an-master1002.eqiad.wmnet/10.64.21.110:8040 successful [13:35:25] good [13:40:32] remember to check metrics once in a while :) [13:45:10] hi ottomata, qq: how can I add a path to airflow's pythonpath (in an-test-coord)? [13:54:11] (03PS3) 10Fdans: Adapt wiki selector to allow more than one wiki [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/700098 (https://phabricator.wikimedia.org/T285050) [13:56:34] hm [13:56:57] mforns: what are you trying to do, for an airflow dag to run? [13:57:04] or for a pythonoperator to run? [13:57:28] sorry: for an airlfow dag to 'parse' [13:57:34] or for pythonoperator task to run? [13:58:45] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review, 10User-razzi: Upgrade the Hadoop masters to Debian Buster - https://phabricator.wikimedia.org/T278423 (10razzi) an-master1002 is active; assuming nothing goes wrong, we'll keep it active for a couple days so we're confident it's safe to reimage... [13:59:16] ottomata: I'm trying to import some code from a DAG [14:00:02] mforns: code you wrote? [14:00:06] or a dependency? [14:00:15] yes, I wrotw [14:00:21] *wrote [14:00:26] can you just put the code in the plugins directory? [14:00:30] and import it? [14:00:36] plugins should be on PYTHONPATH [14:00:59] I see, so if I add a directory there with a __init__.py file, that can be imported as a module? [14:01:03] this is how ra zzi did it: [14:01:04] https://phabricator.wikimedia.org/T284172#7143760 [14:01:19] not sure __init__.py ils even needed [14:01:20] but yest [14:01:56] ok, cool! thanks :] [14:17:05] (03PS2) 10Ottomata: mediawiki/api/request: Add performer field and bump to 1.0.0 [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/700253 (https://phabricator.wikimedia.org/T285113) (owner: 10Urbanecm) [14:17:38] (03CR) 10jerkins-bot: [V: 04-1] mediawiki/api/request: Add performer field and bump to 1.0.0 [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/700253 (https://phabricator.wikimedia.org/T285113) (owner: 10Urbanecm) [14:17:47] (03CR) 10Ottomata: "Cool!" [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/700253 (https://phabricator.wikimedia.org/T285113) (owner: 10Urbanecm) [14:18:54] (03PS3) 10Ottomata: mediawiki/api/request: Add performer field and bump to 1.0.0 [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/700253 (https://phabricator.wikimedia.org/T285113) (owner: 10Urbanecm) [14:19:32] (03CR) 10jerkins-bot: [V: 04-1] mediawiki/api/request: Add performer field and bump to 1.0.0 [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/700253 (https://phabricator.wikimedia.org/T285113) (owner: 10Urbanecm) [14:19:35] (03PS4) 10Ottomata: mediawiki/api/request: Add performer field and bump to 1.0.0 [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/700253 (https://phabricator.wikimedia.org/T285113) (owner: 10Urbanecm) [14:30:23] (03PS4) 10Fdans: Adapt wiki selector to allow more than one wiki [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/700098 (https://phabricator.wikimedia.org/T285050) [14:31:06] 10Analytics, 10Data-release, 10Privacy Engineering, 10Research, 10Privacy: Apache Beam go prototype code for DP evaluation - https://phabricator.wikimedia.org/T280385 (10Isaac) Thanks all for the discussion / details on the engineering side! Chiming in on some things that were discussed from a policy per... [14:35:55] razzi: i know, so funny [15:02:48] 10Analytics, 10Data-release, 10Privacy Engineering, 10Research, 10Privacy: Apache Beam go prototype code for DP evaluation - https://phabricator.wikimedia.org/T280385 (10gmodena) Hi, > Small update: The Apache Beam Go SDK is about to [[ https://lists.apache.org/thread.html/re3589c797d2dc6c5a9c1015683b4a... [15:03:06] a-team standup! [15:03:19] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Services: EventGate should use recent service-runner (^2.8.1) with Prometheus support - https://phabricator.wikimedia.org/T272714 (10Ottomata) a:03Ottomata [15:39:41] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: EventLogging background queue beforeunload event handler blocks Back-Forward cache - https://phabricator.wikimedia.org/T285220 (10DLynch) That instrumentation is still being collected -- my specific case to be supported was logging aborted edits, whi... [15:47:58] 10Analytics-Clusters: Upgrade Druid to 0.20.1 (latest upstream) - https://phabricator.wikimedia.org/T278056 (10elukey) a:05elukey→03None [15:48:05] 10Analytics-Kanban: Analytics Hardware for Fiscal Year 2020/2021 - https://phabricator.wikimedia.org/T255145 (10Ottomata) [15:48:07] 10Analytics-Clusters, 10Patch-For-Review: Move Superset and Turnilo to an-tool1010 - https://phabricator.wikimedia.org/T268219 (10Ottomata) 05Open→03Resolved [15:48:13] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Update Spicerack cookbooks to follow the new class API conventions - https://phabricator.wikimedia.org/T269925 (10elukey) a:05elukey→03None [15:48:55] 10Analytics-Clusters, 10Patch-For-Review: Decommisison the Hadoop backup cluster and add the worker nodes to the main Hadoop cluster - https://phabricator.wikimedia.org/T274795 (10Ottomata) 05Stalled→03Resolved a:03Ottomata [15:49:08] 10Analytics-Clusters, 10SRE: Backport kafkacat 1.6.0 from bullseye to buster-backports or buster-wikimedia - https://phabricator.wikimedia.org/T268936 (10Ottomata) 05Open→03Resolved [15:50:36] 10Analytics-Clusters: Re-create deployment-aqs cluster - https://phabricator.wikimedia.org/T272722 (10Ottomata) 05Open→03Resolved a:03Ottomata [15:50:45] 10Analytics-Clusters: Configure Yarn to be able to locate nodes with a GPU - https://phabricator.wikimedia.org/T264401 (10Ottomata) 05Stalled→03Resolved [15:50:49] 10Analytics, 10Machine-Learning-Team: Configure the Hadoop cluster to use the GPUs available on some workers - https://phabricator.wikimedia.org/T276791 (10Ottomata) [15:52:38] 10Analytics-Radar, 10EventStreams, 10MediaWiki-API, 10Patch-For-Review, 10User-Urbanecm: Add user field to mediawiki/api/request - https://phabricator.wikimedia.org/T285113 (10odimitrijevic) [15:54:15] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Expand Wikiselector to allow more than one wiki - https://phabricator.wikimedia.org/T285050 (10odimitrijevic) p:05Triage→03High [15:55:22] 10Analytics-Kanban: Analytics Hardware for Fiscal Year 2020/2021 - https://phabricator.wikimedia.org/T255145 (10Ottomata) [15:58:51] 10Analytics-Clusters: Refresh Druid nodes (druid100[1-3]) - https://phabricator.wikimedia.org/T255148 (10Ottomata) a:03razzi [15:59:08] 10Analytics-Clusters, 10User-razzi: Refresh Druid nodes (druid100[1-3]) - https://phabricator.wikimedia.org/T255148 (10Ottomata) [16:44:09] razzi, one question please: I'm trying to replicate your code in https://phabricator.wikimedia.org/T284172#7143760 [16:44:45] but when I try to instantiate the HadoopFileSystem class with `hdfs = HadoopFileSystem()` [16:45:05] I get an error: takes at least 1 positional argument (0 given) [16:45:24] 10Analytics-Radar, 10Growth-Team, 10Growth-Team-Filtering, 10Product-Analytics: Measurement for AfC improvement (April 2018) - https://phabricator.wikimedia.org/T192515 (10nettrom_WMF) 05Open→03Declined As obvious from the earlier discussion in this task, completing this is complicated. It's unclear at... [16:45:25] Did that code work for you with no arguments? [16:50:14] 10Analytics-Radar, 10Product-Analytics: Move page creation queries to Hive - https://phabricator.wikimedia.org/T205654 (10nettrom_WMF) 05Open→03Declined The dashboards that used this data are no longer in use, and there has been no feedback suggesting a need for the data. I see no need to keep this task ar... [17:14:45] mforns: I think I never ran that code, all I tested was that the imports worked [17:15:06] ok, ok, np figuring out what to do, thanks! [17:36:39] ottomata: I think I broke airflow's scheduler in test... [17:36:54] how can I restart it? [17:39:20] mforns: i thinkk you can do sudo -u analytics systemctl restart airflow-scheduler@analytics-test [17:39:53] mforns: as for HadoopFileSystem, the example below raz zi's works [17:40:06] ottomata: oh, ok! [17:40:08] v [17:40:09] hdfs = HadoopFileSystem.from_uri('hdfs://analytics-test-hadoop/') [17:40:32] aha, I tried analytics-hadoop-test cause I saw it in puppet, but no luck, thanks a lot! [17:53:55] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: EventLogging background queue beforeunload event handler blocks Back-Forward cache - https://phabricator.wikimedia.org/T285220 (10Gilles) I think that only registering the unload handler for browsers that don't support pagehide makes sense. Those old... [18:18:32] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: EventLogging background queue beforeunload event handler blocks Back-Forward cache - https://phabricator.wikimedia.org/T285220 (10Gilles) a:03Gilles [18:23:41] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: EventLogging background queue beforeunload event handler blocks Back-Forward cache - https://phabricator.wikimedia.org/T285220 (10DLynch) I think (?) it should be as simple as ` if ( !( 'onpagehide' in window) ) { window.addEventListener( 'unloa... [18:26:30] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: EventLogging background queue beforeunload event handler blocks Back-Forward cache - https://phabricator.wikimedia.org/T285220 (10DLynch) Granted, that'd fail to register it if someone has outright done `window.onpagehide = ...` somewhere. But I doub... [18:38:03] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: EventLogging background queue beforeunload event handler blocks Back-Forward cache - https://phabricator.wikimedia.org/T285220 (10Gilles) Indeed, it's that simple. However, @krinkle reminded me that our JS-enabled experience now excludes IE9-10: http... [18:43:58] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: EventLogging background queue beforeunload event handler blocks Back-Forward cache - https://phabricator.wikimedia.org/T285220 (10DLynch) Oh yeah, if we're already not even loading that code because of the sniffing, remove away! [18:56:30] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: EventLogging background queue beforeunload event handler blocks Back-Forward cache - https://phabricator.wikimedia.org/T285220 (10Gilles) You just need to make sure that any code you rely on that creates EventLogging events at the end of the page lif... [19:02:06] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: EventLogging background queue beforeunload event handler blocks Back-Forward cache - https://phabricator.wikimedia.org/T285220 (10DLynch) Since we can 100% count on the pagehide hook running, we can just remove that handler right now. The handler bo... [19:41:44] mforns: gentle poke: https://github.com/wikimedia/jsonschema-tools/pull/31 :) [22:58:13] 10Analytics-Radar, 10Readers-Web-Backlog, 10Performance-Team (Radar), 10Vue.js (Vue.js Search Experience (Vector modern)): Revise schema and performance dashboards for Vue.js search - https://phabricator.wikimedia.org/T250336 (10nray) 05Open→03Resolved [22:58:23] 10Analytics-Radar, 10Readers-Web-Backlog, 10Performance-Team (Radar), 10Vue.js (Vue.js Search Experience (Vector modern)): Revise schema and performance dashboards for Vue.js search - https://phabricator.wikimedia.org/T250336 (10nray) It's not clear to me whether this task is focused on real user monitorin... [23:20:04] 10Analytics-Radar, 10Readers-Web-Backlog, 10Performance-Team (Radar), 10Vue.js (Vue.js Search Experience (Vector modern)): Revise schema and performance dashboards for Vue.js search - https://phabricator.wikimedia.org/T250336 (10Jdlrobson) Thanks Nick!