[00:02:47] (03CR) 10Mholloway: "Yeah, sorry about that, I think we need to set this repo up (maybe primary too?) with a gate-and-submit job." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/661273 (https://phabricator.wikimedia.org/T263663) (owner: 10Eric Gardner) [00:24:00] PROBLEM - Check the last execution of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [00:34:36] RECOVERY - Check the last execution of produce_canary_events on an-launcher1002 is OK: OK: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [00:44:31] 10Analytics-Radar, 10Better Use Of Data, 10Product-Analytics, 10Product-Data-Infrastructure: prefUpdate schema contains multiple identical events for the same preference update - https://phabricator.wikimedia.org/T218835 (10Mholloway) I've been poking at this as time permits but haven't yet managed to trac... [06:28:41] goood morning [06:37:31] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: WDCM_Sqoop_Clients.R fails from stat1004 - https://phabricator.wikimedia.org/T274866 (10elukey) For the first question I'd say yes, for the second it really depends how your are creating tables in Hive, if you are using an external location... [06:45:13] 10Analytics: Inconsistent systemd default task max on hadoop workers - https://phabricator.wikimedia.org/T274860 (10elukey) After the roll reboot this is the status: ` elukey@cumin1001:~$ sudo cumin 'A:hadoop-worker' 'uname -r' 59 hosts will be targeted: an-worker[1078-1116].eqiad.wmnet,analytics[1058-1077].eqi... [07:04:39] !log reboot stat1004/stat1006/stat1007 for kernel upgrades [07:04:41] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:33:07] 10Analytics, 10Analytics-Kanban: Generalize the current Airflow puppet/scap code to deploy a dedicated Analytics instance - https://phabricator.wikimedia.org/T272973 (10elukey) Reporting some chats that we have been doing: 1) I followed up with upstream to see if there is a multi-tenancy solution different fr... [08:10:11] (03CR) 10WMDE-Fisch: [C: 03+1] "> Patch Set 12:" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/656901 (https://phabricator.wikimedia.org/T269986) (owner: 10WMDE-Fisch) [08:18:02] the new package for Presto has been built and the test cluster is upgraded [08:18:21] I can wait for a +1 from somebody in the team to upgrade the main presto cluster as well :) [08:21:00] 10Analytics, 10Patch-For-Review: Decide to move or not to PrestoSQL/Trino - https://phabricator.wikimedia.org/T266640 (10elukey) Test cluster upgraded with the new package, all good! Going to wait for the +1 from somebody in the team to proceed with the prod upgrade :) [08:21:23] 10Analytics, 10Patch-For-Review: Presto should warn or prevent users from querying without Hive partition predicates - https://phabricator.wikimedia.org/T273004 (10elukey) The test cluster is now fully functional :) [09:26:11] 10Analytics, 10ops-eqiad: an-worker1112 reports I/O errors for a disk - https://phabricator.wikimedia.org/T274981 (10elukey) [09:37:56] 10Analytics-Radar, 10WMDE-Templates-FocusArea, 10MW-1.36-notes (1.36.0-wmf.29; 2021-02-02), 10Patch-For-Review, and 2 others: Adjust edit count bucketing for CodeMirror - https://phabricator.wikimedia.org/T273471 (10awight) [09:38:15] 10Analytics-Radar, 10WMDE-Templates-FocusArea, 10MW-1.36-notes (1.36.0-wmf.29; 2021-02-02), 10Patch-For-Review, and 2 others: Adjust edit count bucketing for CodeMirror - https://phabricator.wikimedia.org/T273471 (10lilients_WMDE) [09:38:24] 10Analytics-Radar, 10WMDE-Templates-FocusArea, 10Patch-For-Review, 10WMDE-TechWish (Sprint-2021-02-03), 10WMDE-TechWish-Sprint-2021-02-17: Adjust edit count bucketing for TemplateWizard, segment all metrics - https://phabricator.wikimedia.org/T273475 (10awight) [09:45:56] 10Analytics-Radar, 10WMDE-Templates-FocusArea, 10MW-1.36-notes (1.36.0-wmf.30; 2021-02-09), 10WMDE-TechWish (Sprint-2021-02-03): Compensate for sampling - https://phabricator.wikimedia.org/T273454 (10awight) New recommendation after discussing with mforns is that we should write the compensated metrics to... [10:23:39] 10Analytics, 10SRE, 10ops-eqiad: an-worker1112 reports I/O errors for a disk - https://phabricator.wikimedia.org/T274981 (10MoritzMuehlenhoff) p:05Triage→03Medium [10:24:59] 10Analytics: Purge deprecated reportupdater outputs - https://phabricator.wikimedia.org/T274986 (10awight) [10:37:18] 10Analytics, 10WMDE-Templates-FocusArea, 10WMDE-TechWish-Sprint-2021-02-17: Backfill metrics for TemplateWizard and VisualEditor - https://phabricator.wikimedia.org/T274988 (10awight) [10:37:28] 10Analytics, 10WMDE-Templates-FocusArea, 10WMDE-TechWish-Sprint-2021-02-17: Backfill metrics for TemplateWizard and VisualEditor - https://phabricator.wikimedia.org/T274988 (10awight) [10:39:42] 10Analytics-Radar, 10WMDE-Templates-FocusArea, 10MW-1.36-notes (1.36.0-wmf.30; 2021-02-09), 10WMDE-TechWish (Sprint-2021-02-03), 10WMDE-TechWish-Sprint-2021-02-17: Compensate for sampling - https://phabricator.wikimedia.org/T273454 (10awight) [10:42:04] 10Analytics-Radar, 10WMDE-Templates-FocusArea, 10MW-1.36-notes (1.36.0-wmf.29; 2021-02-02), 10Patch-For-Review, and 2 others: Adjust edit count bucketing for CodeMirror - https://phabricator.wikimedia.org/T273471 (10awight) Putting points on this task again: I need to adjust one of our queries so that it s... [11:09:52] I have created https://gerrit.wikimedia.org/r/c/operations/puppet/+/664788, that should ease the job of reimaging workers [11:19:21] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, and 5 others: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10awight) [11:34:19] 10Analytics, 10Analytics-EventLogging, 10Community-Tech, 10Event-Platform, and 2 others: CodeMirrorUsage Event Platform Migration - https://phabricator.wikimedia.org/T275005 (10awight) [11:39:43] (03PS1) 10Awight: Add CodeMirrorUsage schema to analytics/legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664792 (https://phabricator.wikimedia.org/T275005) [11:39:51] going afk for lunch! [11:46:10] 10Analytics, 10Event-Platform, 10WMDE-TechWish: ReferencePreviewsBaseline Event Platform Migration - https://phabricator.wikimedia.org/T275007 (10awight) [11:47:38] (03PS1) 10Awight: Add ReferencePreviewsBaseline schema to to analytics/legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664795 (https://phabricator.wikimedia.org/T275007) [11:48:16] 10Analytics, 10Event-Platform, 10WMDE-TechWish: ReferencePreviewsCite Event Platform Migration - https://phabricator.wikimedia.org/T275008 (10awight) [11:48:29] (03PS1) 10Awight: Add ReferencePreviewsCite schema to to analytics/legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664796 (https://phabricator.wikimedia.org/T275008) [11:51:04] 10Analytics, 10Event-Platform, 10WMDE-TechWish: ReferencePreviewsPopups Event Platform Migration - https://phabricator.wikimedia.org/T275009 (10awight) [11:52:30] (03PS1) 10Awight: Add ReferencePreviewsPopups schema to to analytics/legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664798 (https://phabricator.wikimedia.org/T275009) [11:55:21] Minor feature request: the event schemas should validate examples as part of the test suite. [11:56:01] (03PS2) 10Awight: Add ReferencePreviewsBaseline schema to to analytics/legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664795 (https://phabricator.wikimedia.org/T275007) [11:59:38] 10Analytics, 10Event-Platform, 10WMDE-TechWish: TemplateDataApi Event Platform Migration - https://phabricator.wikimedia.org/T275011 (10awight) [12:00:43] (03PS1) 10Awight: Add TemplateDataApi schema to to analytics/legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664799 (https://phabricator.wikimedia.org/T275011) [12:01:37] 10Analytics, 10Event-Platform, 10WMDE-TechWish: TemplateDataEditor Event Platform Migration - https://phabricator.wikimedia.org/T275012 (10awight) [12:03:17] (03PS1) 10Awight: Add TemplateDataEditor schema to to analytics/legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664801 (https://phabricator.wikimedia.org/T275012) [12:04:29] 10Analytics, 10Event-Platform, 10WMDE-TechWish: TwoColConflictConflict Event Platform Migration - https://phabricator.wikimedia.org/T275013 (10awight) [12:08:28] (03PS1) 10Awight: Add TwoColConflictConflict schema to to analytics/legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664802 (https://phabricator.wikimedia.org/T275013) [12:09:18] 10Analytics, 10Event-Platform, 10WMDE-TechWish: TwoColConflictExit Event Platform Migration - https://phabricator.wikimedia.org/T275014 (10awight) [12:12:31] (03PS1) 10Awight: Add TwoColConflictExit schema to to analytics/legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664803 (https://phabricator.wikimedia.org/T275014) [12:13:27] 10Analytics, 10Event-Platform, 10WMDE-TechWish: VisualEditorTemplateDialogUse Event Platform Migration - https://phabricator.wikimedia.org/T275015 (10awight) [12:14:19] (03PS1) 10Awight: Add VisualEditorTemplateDialogUse schema to to analytics/legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664804 [12:18:39] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, and 5 others: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10awight) [12:18:46] sorry! [12:54:13] morning team [12:57:38] Hi team [13:55:47] 10Analytics: The most visited wiki in Uzbekistan on Feb 14th at 6am UTC is mediawiki.org - https://phabricator.wikimedia.org/T274823 (10JAllemandou) There seem to be a broader issue with related countries: https://pageviews.toolforge.org/siteviews/?platform=desktop&source=pageviews&agent=user&range=latest-20&sit... [13:58:34] 10Analytics, 10Product-Analytics: Big increase in traffic for projects except 'wikipedia' family since Feb 14th - https://phabricator.wikimedia.org/T274823 (10JAllemandou) [14:18:28] hello teammm [14:27:04] (03CR) 10DannyS712: Add ReferencePreviewsCite schema to to analytics/legacy (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664796 (https://phabricator.wikimedia.org/T275008) (owner: 10Awight) [14:27:27] (03CR) 10DannyS712: Add TwoColConflictConflict schema to to analytics/legacy (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664802 (https://phabricator.wikimedia.org/T275013) (owner: 10Awight) [14:27:36] (03CR) 10DannyS712: Add TwoColConflictExit schema to to analytics/legacy (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664803 (https://phabricator.wikimedia.org/T275014) (owner: 10Awight) [14:27:44] (03CR) 10DannyS712: Add VisualEditorTemplateDialogUse schema to to analytics/legacy (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664804 (owner: 10Awight) [14:27:56] (03CR) 10DannyS712: Add TemplateDataEditor schema to to analytics/legacy (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664801 (https://phabricator.wikimedia.org/T275012) (owner: 10Awight) [14:28:08] (03CR) 10DannyS712: Add TemplateDataApi schema to to analytics/legacy (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664799 (https://phabricator.wikimedia.org/T275011) (owner: 10Awight) [14:28:20] (03CR) 10DannyS712: Add ReferencePreviewsPopups schema to to analytics/legacy (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664798 (https://phabricator.wikimedia.org/T275009) (owner: 10Awight) [14:28:32] (03CR) 10DannyS712: Add ReferencePreviewsBaseline schema to to analytics/legacy (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664795 (https://phabricator.wikimedia.org/T275007) (owner: 10Awight) [14:29:17] (03PS2) 10Awight: Add ReferencePreviewsCite schema to analytics/legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664796 (https://phabricator.wikimedia.org/T275008) [14:54:43] hmm, [14:54:46] https://airbnb.io/projects/omniduct/ looks pretty cool [14:57:45] 10Analytics, 10GrowthExperiments, 10Growth-Team (Current Sprint), 10MW-1.36-notes (1.36.0-wmf.30; 2021-02-09): eventgate_validation_error for NewcomerTask, HomepageTask, and HomepageVisit schemas - https://phabricator.wikimedia.org/T273700 (10Ottomata) <3 [14:59:32] 10Analytics, 10Product-Infrastructure-Team-Backlog, 10Chinese-Sites, 10Pageviews-Anomaly: Top read is showing one page that had fake traffic in zhwiki - https://phabricator.wikimedia.org/T274605 (10MSantos) Thanks for the clarification @JAllemandou, we will be tracking it, please ping us in case wikifeeds... [14:59:53] mforns: FYI, awight is helping us with the migrations of his team's schemas! https://phabricator.wikimedia.org/T275005 [14:59:54] <3 [15:00:29] ottomata: he told me, awight rocks! [15:00:46] +1 [15:00:57] mforns rocks too btw [15:00:59] :) [15:01:09] O.o! [15:01:14] * mforns blushes [15:04:37] (03CR) 10Ottomata: Add ReferencePreviewsCite schema to analytics/legacy (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664796 (https://phabricator.wikimedia.org/T275008) (owner: 10Awight) [15:11:50] All y'all rock, as far as I can tell :) [15:16:28] 10Analytics-Radar, 10Event-Platform, 10MW-1.36-notes (1.36.0-wmf.9; 2020-09-15), 10Platform Team Workboards (Clinic Duty Team), 10Wikimedia-production-error: PHP Notice: Array to string conversion (from EventBus.php) - https://phabricator.wikimedia.org/T262462 (10Ottomata) 05Open→03Resolved a:03Otto... [15:29:05] 10Analytics-Radar, 10Event-Platform, 10MW-1.36-notes (1.36.0-wmf.9; 2020-09-15), 10Platform Team Workboards (Clinic Duty Team), 10Wikimedia-production-error: PHP Notice: Array to string conversion (from EventBus.php) - https://phabricator.wikimedia.org/T262462 (10Reedy) a:05Ottomata→03Pchelolo [15:32:49] klausman: <3 [15:33:01] elukey: it'd probably be really good for razzi to learn some about partman, can you walk him through your reuse patch sometime? [15:36:04] elukey: other q, we'll be on hadoop 2.10 for a while, right? [15:40:41] (03PS2) 10Awight: Add TwoColConflictConflict schema to analytics/legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664802 (https://phabricator.wikimedia.org/T275013) [15:41:10] (03PS2) 10Awight: Add TwoColConflictExit schema to analytics/legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664803 (https://phabricator.wikimedia.org/T275014) [15:41:37] (03PS2) 10Awight: Add VisualEditorTemplateDialogUse schema to analytics/legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664804 [15:41:56] (03PS2) 10Awight: Add TemplateDataEditor schema to analytics/legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664801 (https://phabricator.wikimedia.org/T275012) [15:42:12] (03PS2) 10Awight: Add TemplateDataApi schema to analytics/legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664799 (https://phabricator.wikimedia.org/T275011) [15:42:30] (03PS2) 10Awight: Add ReferencePreviewsPopups schema to analytics/legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664798 (https://phabricator.wikimedia.org/T275009) [15:42:44] (03PS3) 10Awight: Add ReferencePreviewsBaseline schema to analytics/legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664795 (https://phabricator.wikimedia.org/T275007) [15:44:01] ottomata: yes sure! For 2.10, I think possibly some months, but Bigtop 3.x is already in the making (with Hadoop 3.x etc..) [15:44:20] https://issues.apache.org/jira/browse/BIGTOP-3471 [15:44:49] there will also be oozie 5.2 :P [15:44:51] (03PS3) 10Awight: Add ReferencePreviewsCite schema to analytics/legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664796 (https://phabricator.wikimedia.org/T275008) [15:45:04] (03CR) 10Awight: Add ReferencePreviewsCite schema to analytics/legacy (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664796 (https://phabricator.wikimedia.org/T275008) (owner: 10Awight) [15:47:02] 10Analytics-Radar, 10Product-Analytics, 10wmfdata-python: Consider rewriting wmfdata-python to use omniduct - https://phabricator.wikimedia.org/T275038 (10nshahquinn-wmf) [15:47:10] (03PS3) 10Awight: Add VisualEditorTemplateDialogUse schema to analytics/legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/664804 [15:51:19] elukey: i'im considerng putting the hadoop 2.10 jars in the spark package manually, just to have it working as close as possible to how it worked in 2.6 [15:51:30] but, that means when we upgrade hadoop we'll have to deal with this again [15:52:02] let's plan to do spark 3 before we do the next hadoop upgrade [15:52:21] that way we can address the hadoop jar version probem then [15:52:23] rather than now [15:52:46] (I was going to make a hadoopless spark deb, but i think going with a hadoop 2.10 is less change) [15:53:23] ottomata: sure makes sense [15:54:36] ok [15:59:03] ottomata: I'm working on the oozie job for session_length, and have a q: the job uses event.mediawiki_client_session_tick as an input table, and uses the corresponding dataset definition. But partitions refined by Refine, might or might not be complete, right? [15:59:42] mforns: they most likely will be complete, but it is possible they aren't, e.g. if camus is paused for some reason [15:59:45] so, I'm adding a couple more hours to the requirements, so that the oozie workflow only runs a couple hours after the given partition has been refined for the first time [15:59:49] we wait delay 2 hours for this reason [16:00:07] ok, so 2 hours is enough, that was my question :] [16:00:21] thanks! [16:00:22] if say camus is paused for more than 2 hours [16:00:41] refine will refine the last import from camus (2 hours ago), which may partially be imported at that time [16:01:00] then, when camus is restarted, refine will see that that hour has new data, and re-refnie the whole lthing [16:01:12] at that point, the _REFINED file will have an updated timestamp [16:01:36] but, yeah, I don't oozie can know about the timestamp to know that i tmight have to re-run its workflow for that hour [16:02:58] ottomata: not related, but I keep forgetting to ask - would it be ok to add a little bit of scala code to remove _REFINE_FAILED flags when placing a _REFINED one, or is there a corner case that would need both? [16:03:13] right, i don't thnk therrer's a need to keep it [16:03:25] i mean, maybe its nice to know that a previous refine failed? but not really. [16:03:37] elukey: i think that makes sense, wouldlnt' be too hard to add [16:03:41] I can attempt to file a code review if you are ok! [16:03:44] sure! [16:03:46] ottomata: but if camus stops, no new data is going to be refined, right? so if the oozie coord requires a couple subsequent hours after the one to be processed, then we're fine, no? [16:04:52] elukey: probably can add that herer [16:04:53] https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/refine/RefineTarget.scala#L346-L359 [16:04:57] can add a [16:05:07] if failureFlagExists { remove it } block [16:05:44] ohhhh [16:05:49] like the oozie depends on those hours [16:05:54] yes mforns most likelyl that will work [16:05:58] unless...backfill [16:06:03] k :] [16:06:08] but if backfilll the oozie jobs will proably need to be manually rerun anyway [16:06:09] cool [16:06:23] ok, makes sense, thanks! [16:13:25] Good morning all! [16:14:27] !log rebalance kafka partitions for eqiad.mediawiki.api-request [16:14:28] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:14:40] hello! [16:18:31] 10Analytics, 10SRE: Augment NEL reports with GeoIP country code and network AS number - https://phabricator.wikimedia.org/T263496 (10CDanis) @Ottomata just one more question for you! >>! In T263496#6744142, @CDanis wrote: >>>! In T263496#6744057, @Ottomata wrote: >> The long term solution here is still not cl... [16:20:11] lexnasser: o/ I know that you will not believe it but I am bootstrapping the test cassandra cluster :D [16:33:02] 10Analytics-Clusters, 10Analytics-Kanban: Re-create deployment-aqs cluster - https://phabricator.wikimedia.org/T272722 (10elukey) ` elukey@aqs-test1001:~$ nodetool status Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens... [17:01:48] a-team standup or what [17:01:57] hold your hat, PA sync over [17:02:45] 10Analytics, 10SRE: Augment NEL reports with GeoIP country code and network AS number - https://phabricator.wikimedia.org/T263496 (10Ottomata) Let's do the former, I think doing the latter (using schemas to configure included data) is going to be the right solution after all. So, special case these headers ju... [17:06:19] (03CR) 10Awight: [C: 03+1] "PS 6: another simple mistake, "row" is now a reserved word." [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/656210 (https://phabricator.wikimedia.org/T273471) (owner: 10Awight) [17:08:20] 10Analytics-Radar, 10Better Use Of Data, 10Product-Analytics, 10Product-Data-Infrastructure: prefUpdate schema contains multiple identical events for the same preference update - https://phabricator.wikimedia.org/T218835 (10Mholloway) OK, after running a couple of queries for the past couple of days, it lo... [17:17:03] elukey: you coming to the meeting? [17:18:02] fdans: I have the ml weekly meeting in a bit, should I be there? [17:18:32] elukey: it's ok, we'll catch you up [17:18:37] fdans: thanks :) [17:36:28] 10Analytics-Clusters: Balance Kafka topic partitions on Kafka Jumbo to take advantage of the new brokers - https://phabricator.wikimedia.org/T255973 (10razzi) [17:43:55] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Update Spicerack cookbooks to follow the new class API conventions - https://phabricator.wikimedia.org/T269925 (10elukey) [17:44:44] !log rebalance kafka partitions for webrequest_upload partition 0 [17:44:46] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:53:25] (03PS1) 10Milimetric: Revert to escaping to not break header evolution [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/664880 [17:55:02] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Revert to escaping to not break header evolution [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/664880 (owner: 10Milimetric) [17:58:37] 10Analytics, 10Event-Platform, 10Services, 10Patch-For-Review, 10User-mobrovac: Investigate drop in revision_create events / apparent high rate of null edits - https://phabricator.wikimedia.org/T128838 (10bd808) >>! In T128838#2189391, @ori wrote: > Has anyone reached out to the operator of Cyberbot_I to... [17:58:55] 10Analytics, 10SRE: Augment NEL reports with GeoIP country code and network AS number - https://phabricator.wikimedia.org/T263496 (10Ottomata) (Sorry, just edited ^, somehow a very important 'not' did not make it through my typing fingers) [18:05:29] hey ottomata, do you have a 1 min to answer a newpyter related question? [18:07:45] dsaez: ya ask! slightly afk/lunching so may answer async [18:09:36] oh, no rushes. I'm trying to move to newpyter, but still don't get how to install packages there. I've created a new enviroment. Then test it with mwparserfromhell [18:10:15] dsaez: either conda install ... or pip install ... [18:10:28] https://wikitech.wikimedia.org/wiki/Analytics/Systems/Anaconda#Installing_packages_into_your_user_conda_environment [18:10:38] and with that package I get all the errors possible :D If I try to import (on pyspark ) get ModuleNotFoundError [18:10:48] if a try to install with pip I get already exist [18:10:48] ? [18:10:54] hm [18:11:03] in just a regular python notebook via newpyter? [18:11:04] and if y try to update !pip -U install [18:11:07] I get now persmision [18:11:18] pyspark kernel [18:11:44] PySpark - Yarn [18:13:15] * no permision [18:13:35] Read-only file system: 'WHEEL' [18:16:21] https://pasteboard.co/JOO6QHN.png [18:24:09] I am going afk folks, have a nice rest of the day :) [18:32:39] (03PS1) 10Mforns: [WIP] Add oozie job for session length computation [analytics/refinery] - 10https://gerrit.wikimedia.org/r/664885 (https://phabricator.wikimedia.org/T273116) [18:36:36] oh dsaez are you using the base anaconda readonly or did you make your own user conda env? [18:36:44] when you logged in and started your server? [18:36:58] I selected Create new enviroment [18:37:00] hm [18:37:36] dsaez: it works for me, maybe your conda env is borked somehow? [18:37:40] can you create a new one and try? [18:37:48] sure [18:38:14] go to hub control panel, stop my server, then launch server, and use drop down to make a new one [18:41:00] ottomata, exactly the same error. [18:41:24] let me see in another machine [18:43:03] that was stat1008 [18:49:21] ottomata, same behavior on stat1007 [18:49:39] weird [18:49:44] let me make anew env and try [18:51:07] OH [18:51:12] dsaez: you are doing pyspark yarn [18:51:12] hm [18:51:44] yes [18:52:33] my understanding was that newpyter allows to install packages in all workers ... [18:52:59] or should I go for python, pack the enviroment, and do the find.spark stuff _ [18:53:00] ? [18:53:51] fkaelin:yt? can you link your script? [18:54:21] dsaez: no it doesnt' allow 'installing' in workers, but the base anaconda is avail on all workers...although i'm not sure if it is in the pythonpath by default [18:55:06] got it [18:55:08] the prompt you get in jupyter is running a in yarn client mode, which means the spark driver process is on stat1008 [18:55:14] if you pip install, it'll just install to stat1008 [18:55:18] but yeah now i can repro [18:55:19] OH [18:55:22] ohoh [18:55:22] wait [18:55:35] dsaez: we are gonig to get rid of the custom spark kernels [18:55:42] can you try via wmfdata or findspark? [18:55:50] https://wikitech.wikimedia.org/wiki/User:Ottomata/Jupyter#PySpark_and_wmfdata [18:56:03] so, regular python notebook [18:56:05] import wmfdata [18:56:18] spark = wmfdata.spark.get_session() [18:57:18] the findspark I now. I'm reading about wmfdata [18:57:56] so, those docs aren't in the main space yet on wikitech because i'm waiting for a wmfdata release [18:58:00] before they will work as written [18:58:14] wmfdata is really just a thin wrapper around findspark [18:58:42] wmfdata brings the data to the master? I mean is like toPandas() ? [18:59:00] yes [18:59:01] dsaez if you need to python dependencies on the workers (ie as part of the spark job, not just in the notebook itself), you can do this https://www.irccloud.com/pastebin/dQVrXPSq/ [18:59:03] if you do wmfdata.run [18:59:03] yes [18:59:07] wmfdata.spark.run [18:59:08] it does that [19:00:19] I see, and both the findspark and the wmfdata, allows to deliver the env to the workers and have the anaconda base by default? [19:01:06] I saw an example with ELMO linked on the docs. This is similar to some of my use cases [19:01:28] dsaez: i think there is more work to be done around maknig sure that the base anaconda deps is used by default pyspark on the workers, but fabian's script will do just that [19:01:38] packing your own conda env iis optional, only if you need deps that aren't in the base anaconda [19:04:49] got it. Tomorrow I'm doing an onboarding with jupyter notebooks and pyspark for paragon and two interns. The recommendation will be to go directly with findspark instead of using the pyspark kernal, true? [19:06:14] yes, avoid the custom pyspark kernels [19:06:21] you can send them https://wikitech.wikimedia.org/wiki/User:Ottomata/Jupyter [19:06:50] with the caveat that the wmfdata code snippets are not 100% compatible with the version there yet [19:08:01] dsaez: sorry this isn't all finalized [19:08:06] priorities are hard! [19:08:10] got it, I think for interns that will be here just for 12 weeks, I go with the custom kernels that I know better, and with Pablo we will start working with the new env [19:08:21] ok counds good [19:08:47] ottomata, no rushes from side :D I love all these services, I used to do all this manually [19:32:55] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10MW-1.36-notes (1.36.0-wmf.32; 2021-02-23), 10Performance-Team (Radar): Convert WikimediaEvents to use ResourceLoader packageFiles - https://phabricator.wikimedia.org/T253634 (10Mholloway) 05Open→03Resolved [20:19:10] 10Analytics-Clusters: Balance Kafka topic partitions on Kafka Jumbo to take advantage of the new brokers - https://phabricator.wikimedia.org/T255973 (10razzi) Ok! Now that we're on to the final and highest traffic topics, webrequest_upload and webrequest_text, we're switching to migrating one partition at a time... [20:27:49] 10Analytics-Clusters: Balance Kafka topic partitions on Kafka Jumbo to take advantage of the new brokers - https://phabricator.wikimedia.org/T255973 (10Ottomata) Nice. [20:42:13] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: WDCM_Sqoop_Clients.R fails from stat1004 - https://phabricator.wikimedia.org/T274866 (10GoranSMilovanovic) @elukey Unfortunately... ` Executing command: USE goransm; DROP TABLE IF EXISTS wdcm_clients_wb_entity_usage; [WARN] Failed to creat... [21:25:52] 10Analytics, 10Product-Analytics: Big increase in traffic for projects except 'wikipedia' family since Feb 14th - https://phabricator.wikimedia.org/T274823 (10kzimmerman) a:03cchen @JAllemandou it looks like you checked the main dimensions to investigate; the other thing is that the jump only happens on desk... [21:30:20] razzi: i think https://gerrit.wikimedia.org/r/c/operations/debs/spark2/+/664922 does what we want [21:30:31] i went with manually copying in the hadoop 2.10.1 jars [21:30:38] probably th;e right thing to do is not to include them [21:30:45] but, am going for as few changes as possible atm [21:32:11] 10Analytics, 10Patch-For-Review: Repackage spark without hadoop, use provided hadoop jars - https://phabricator.wikimedia.org/T274384 (10Ottomata) Actually, at the moment I am pursuing option 3. 3. Use spark-2.4.4-bin-hadoop2.6.tgz but remove the Hadoop 2.6 jars and manually include the Hadoop 2.10.1 jars. I... [21:35:33] 10Analytics, 10Patch-For-Review: Repackage spark without hadoop, use provided hadoop jars - https://phabricator.wikimedia.org/T274384 (10Ottomata) The tricky part is, I don't know how to test this well beyond installing it! I have tested Refine using the spark assembly jar built this way, and it works just... [22:04:07] (03PS1) 10Milimetric: Remove disabled jobs from reportupdater [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/664937 [22:05:30] (03PS1) 10Milimetric: Fix syntax error [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/664938 [23:09:32] 10Analytics-Clusters, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install an-worker11[18-41] - https://phabricator.wikimedia.org/T260445 (10RobH)