[06:03:43] bonjour [06:50:07] Good morning [06:50:59] elukey: Hi :) [06:51:14] elukey: I haz questionz [06:51:51] elukey: Can you confirm that what was labsdb1021 is now clouddb1021, and that it is multi-instances? [06:53:53] (03CR) 10Joal: [C: 03+1] "Code looks good - Let's try on deployment cluster before going live :)" [analytics/aqs] - 10https://gerrit.wikimedia.org/r/675523 (https://phabricator.wikimedia.org/T278699) (owner: 10Hnowlan) [06:55:11] joal: I cannot confirm nor deny your statement [06:55:18] * elukey runs away [06:55:19] :D [06:55:23] :D [06:55:48] yes yes everything seems working as expected, all good from the instances/networking/etc.. point of view [06:56:10] ok - We need to merge and deploy Dan's patch [06:56:24] thanks for the confirmation elukey :) [06:56:49] one thing just to triple check - for sqoop we need "only" s1-s8 right? [06:56:53] not stuff like x1 [06:57:05] elukey: I'm not sure [06:58:28] probably not, x1 contains special tables but not wikis/projects iirc [06:58:32] so we should be good [06:58:43] elukey: I'm triple checking [07:00:20] +1 for merging/deploying Dan's patch [07:00:34] that IIRC requires a follow up in puppet [07:00:44] correct elukey [07:11:57] 10Analytics-Radar, 10SRE, 10Patch-For-Review, 10Services (watching), 10User-herron: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345] - https://phabricator.wikimedia.org/T225005 (10elukey) @herron ping :) Should we work on this in Q4? I can allocate some time to help, at... [07:13:05] * elukey bbiab! [07:22:28] (03CR) 10Joal: "Mostly ok :)" (035 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/673604 (https://phabricator.wikimedia.org/T212451) (owner: 10Ottomata) [07:32:08] (03CR) 10Joal: "Continued discussion/ideas." (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/674304 (owner: 10Ottomata) [07:42:59] (03CR) 10Joal: [C: 03+1] "For me it's ready. I'll deploy today and sync with Razzi about the puppet patch." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/666209 (https://phabricator.wikimedia.org/T274690) (owner: 10Milimetric) [07:47:19] 10Analytics-Radar, 10SRE, 10Patch-For-Review, 10Services (watching), 10User-herron: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345] - https://phabricator.wikimedia.org/T225005 (10elukey) Also FYI in T271136 Cas is going to add the IPv6 AAAA records for the codfw cluste... [07:49:10] (03CR) 10Phuedx: [C: 03+1] Add new analytics/pref_diff schema (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668529 (https://phabricator.wikimedia.org/T261842) (owner: 10Nray) [09:09:02] (03CR) 10Mforns: Add support for finding RefineTarget inputs from Hive (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/673604 (https://phabricator.wikimedia.org/T212451) (owner: 10Ottomata) [09:21:09] 10Analytics, 10Data-Services, 10Machine-Learning-Team, 10ORES, and 2 others: Generate dump of scored-revisions from 2018-2020 for English Wikipedia - https://phabricator.wikimedia.org/T277609 (10JAllemandou) Hi @Suriname0 , I have generated one-off files here: https://analytics.wikimedia.org/published/data... [09:30:57] 10Analytics, 10WMDE-Analytics-Engineering: wmde-toolkit-analyzer-build.service fails on stat1007 - https://phabricator.wikimedia.org/T278665 (10elukey) I have re-created the repo via puppet, initialized git lfs and checked out the jar, since for some reason there was only the git lfs placeholder. Now I get th... [09:34:00] another good news/bad news: copy of the large tables takes around 3 hours including checksumming. Import takes... a _long_ time - it's been over 12 hours and it's at 34% for a single instance. We can parallelise this though and do 4 instances at the same time but still [09:55:32] hnowlan: it seems overall good for our use case, the import time is the one that we care about since we need to stop all loading jobs right? [09:55:56] if it takes ~36 hours to load the sstables on the new cluster it is fine, we can then backfill [09:56:17] so it looks good from my point of view! [09:57:13] +1 to elukey point hnowlan [09:57:28] sweet [09:57:38] We can: stop import while copying, then restart it (service up all good) [09:58:09] then when the data is fully loaded on the new cluster, we backfill a few days of import be fore swapping [09:58:35] elukey, hnowlan --^ ok with the details of that approach? [09:59:29] sounds good to me [10:00:18] +1 [10:13:57] (03PS1) 10Joal: Update mediawiki-dumps-importer [analytics/refinery] - 10https://gerrit.wikimedia.org/r/675763 (https://phabricator.wikimedia.org/T278551) [10:23:35] * elukey lunch! [10:57:32] (03CR) 10Awight: "@mforns Seems like this patch might work in the wild. However, I verified that I'm still blocked on the interleaved parquet logging, it s" [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/667192 (https://phabricator.wikimedia.org/T193169) (owner: 10Awight) [11:07:50] joal: what does the procedure for stopping imports look like? Not looking to start it, just curious :) [11:08:09] hnowlan: we can either stop/suspend jobs in oozie [11:08:57] hnowlan: cassandra-laoding jobs are scheduled/managed by oozie [11:09:12] so stopping (suspending) imports means suspend the oozie jobs [11:14:01] ah, cool - are the jobs managed as code somewhere like puppet? [11:14:24] nope hnowlan, we manage them manually [11:14:50] hnowlan: job config is in code, but start/stop is manual [11:17:40] cool [11:17:43] thanks! [11:18:17] np hnowlan - let me know if you want links to code for instance [11:18:45] 10Analytics-Radar, 10Patch-For-Review: Reportupdater output can be corrupted by hive logging - https://phabricator.wikimedia.org/T275757 (10awight) We have several jobs which are currently blocked (and for over a month) because of this bug. The issue has been present since the beginning it seems, based on the... [11:18:51] hnowlan: I didn't send it on purpose (reading XML should be asked for, not offered :) [11:18:54] joal: that would be great thanks! just to get my head around it [11:18:56] hah! :D [11:18:58] 10Analytics-Radar, 10Patch-For-Review: Reportupdater output can be corrupted by hive logging - https://phabricator.wikimedia.org/T275757 (10awight) p:05Triage→03High [11:19:29] hnowlan: here is the stuff: https://github.com/wikimedia/analytics-refinery/tree/master/oozie/cassandra [11:19:50] hnowlan: I can drive you through it if you wish, it's a not-so-simple case [11:23:54] joal: I'll give this stuff a glance and then I might bug you for an explainer - I'm more curious than anything else [11:24:13] sure hnowlan :) [11:29:15] (03PS2) 10Joal: Update mediawiki-dumps-importer [analytics/refinery] - 10https://gerrit.wikimedia.org/r/675763 (https://phabricator.wikimedia.org/T278551) [11:29:37] (03CR) 10Joal: [V: 03+1] "Manually checked in dry-run mode, looks ok." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/675763 (https://phabricator.wikimedia.org/T278551) (owner: 10Joal) [11:29:43] !log upgrade hive client packages to 2.3.6-1 on an-launcher1002 (already applied to all stat100x) [11:29:46] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:30:00] !log ERRATA: upgrade to 2.3.6-2 [11:30:01] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:31:39] 10Quarry, 10Patch-For-Review, 10cloud-services-team (Kanban): Prepare Quarry for multiinstance wiki replicas - https://phabricator.wikimedia.org/T264254 (10UOzurumba) >>! In T264254#6942821, @Bstorm wrote: >>>! In T264254#6940914, @UOzurumba wrote: >> Hello @Bstorm, >> Please, I am having issues with Quarr... [11:40:53] 10Analytics: Produce a list of wiki projects ranked by number of eligible voters in Board elections - https://phabricator.wikimedia.org/T278815 (10Qgil) [11:44:48] 10Analytics-Radar, 10Patch-For-Review: Reportupdater output can be corrupted by hive logging - https://phabricator.wikimedia.org/T275757 (10elukey) Commented in the patch about https://github.com/apache/hive/blob/branc-2.3/common/src/main/resources/parquet-logging.properties#L60, that is probably more up to da... [11:49:13] 10Analytics: Produce a list of wiki projects ranked by number of eligible voters in Board elections - https://phabricator.wikimedia.org/T278815 (10JAllemandou) Building list is not complicated with the tools we have. The concern I have is about delay in data: our computation system get's updated every month. If... [11:54:33] 10Analytics-Radar, 10Patch-For-Review: Reportupdater output can be corrupted by hive logging - https://phabricator.wikimedia.org/T275757 (10awight) In theory, I thought I would be able to test the patch by creating a copy of the properties file on a stat machine and running: ` HADOOP_CLIENT_OPTS=-Djava.util.lo... [11:55:17] 10Analytics: Produce a list of wiki projects ranked by number of eligible voters in Board elections - https://phabricator.wikimedia.org/T278815 (10nshahquinn-wmf) > If creating this list is too difficult or expensive, a ranking of wiki projects by monthly active editors would be good enough as well. What matters... [12:03:05] 10Analytics, 10Data-Services, 10Machine-Learning-Team, 10ORES, and 2 others: Generate dump of scored-revisions from 2018-2020 for English Wikipedia - https://phabricator.wikimedia.org/T277609 (10Suriname0) Thanks @JAllemandou! I don't seem to have permission to access the newly-created file dumps. Please l... [12:06:07] 10Quarry, 10Patch-For-Review, 10cloud-services-team (Kanban): Prepare Quarry for multiinstance wiki replicas - https://phabricator.wikimedia.org/T264254 (10Majavah) >>! In T264254#6955602, @UOzurumba wrote: > > I have tried entering databases (Wikibase, MediaWiki) and I keep getting an error message. Kindly... [12:28:58] 10Analytics-Radar, 10Patch-For-Review: Reportupdater output can be corrupted by hive logging - https://phabricator.wikimedia.org/T275757 (10elukey) >>! In T275757#6956022, @elukey wrote: > I also deployed on an-launcher1002 the same patch for T276121, that caused some troubles when executing hive via systemd.... [12:30:15] 10Analytics-Clusters, 10Analytics-Kanban: AQS Cassandra storage: Investigate incorrect storage report on Grafana - https://phabricator.wikimedia.org/T278234 (10hnowlan) We have two options on this one - we could do a roll-restart to get a real-time view of our current usage and know that in future our real-tim... [12:30:45] !log restart reportupdater-codemirror on an-launcher1002 fro T275757 [12:30:48] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:30:50] T275757: Reportupdater output can be corrupted by hive logging - https://phabricator.wikimedia.org/T275757 [12:31:23] I'm gonna move the keyspaces listed in https://phabricator.wikimedia.org/T278231 to another directory for 3 days and them remove them if there are no objections [12:31:33] +1 [12:35:50] !log Depooling aqs1004 for another transfer of local_group_default_T_pageviews_per_article_flat [12:35:52] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:42:43] 10Analytics-Clusters, 10Analytics-Kanban: AQS Cassandra storage: Investigate incorrect storage report on Grafana - https://phabricator.wikimedia.org/T278234 (10elukey) I think this is a clear bug of Cassandra, let's focus on 3.11 and see how it goes. If we need these metrics for hw forecasting we'll just issue... [12:45:30] 10Analytics-Radar, 10Patch-For-Review: Reportupdater output can be corrupted by hive logging - https://phabricator.wikimedia.org/T275757 (10elukey) In any case let's fix the parquet logging too, what I'd need is a minimal script.hql able to reproduce this problem (with steps about how to do it etc..). [13:06:22] 10Analytics: Produce a list of wiki projects ranked by number of eligible voters in Board elections - https://phabricator.wikimedia.org/T278815 (10Qgil) @JAllemandou end of March would be totally fine but, actually, I think this spreadsheet that @nshahquinn-wmf links to is all we need! Give me a bit of time to c... [13:23:09] (03CR) 10Ottomata: [C: 03+1] Update mediawiki-dumps-importer [analytics/refinery] - 10https://gerrit.wikimedia.org/r/675763 (https://phabricator.wikimedia.org/T278551) (owner: 10Joal) [13:57:16] 10Analytics: Produce a list of wiki projects ranked by number of eligible voters in Board elections - https://phabricator.wikimedia.org/T278815 (10Qgil) Ok, after checking with my team, this list is good to get us going, and if this is all we can get, it's fine. However, if building the list based on the criteri... [13:57:44] 10Analytics-Radar, 10SRE, 10Patch-For-Review, 10Services (watching), 10User-herron: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345] - https://phabricator.wikimedia.org/T225005 (10herron) >>! In T225005#6954669, @elukey wrote: > Should we work on this in Q4? I can alloc... [14:01:08] 10Analytics: Produce a list of wiki projects ranked by number of eligible voters in Board elections - https://phabricator.wikimedia.org/T278815 (10JAllemandou) I'll wait for March data to flow in (probably April 3rd) and then will generate the data. About publishing the data, I know that some communities are le... [14:03:17] 10Analytics-Radar, 10SRE, 10Patch-For-Review, 10Services (watching), 10User-herron: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345] - https://phabricator.wikimedia.org/T225005 (10elukey) Nice! I have used Stevie's reuse-part partman script: ` kafka-jumbo100[1-9]) ech... [14:11:49] (03CR) 10Mforns: "> Patch Set 6:" [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/667192 (https://phabricator.wikimedia.org/T193169) (owner: 10Awight) [14:11:56] heya teammmmm [14:12:54] hola! [14:28:17] (03CR) 10Mholloway: [C: 03+1] Image recommendations table for android [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668244 (owner: 10Sharvaniharan) [14:43:29] (03PS2) 10Mforns: WikipediaPortal schema whitelist request [analytics/refinery] - 10https://gerrit.wikimedia.org/r/666223 (owner: 10Erin Yener) [14:43:39] (03CR) 10Mforns: WikipediaPortal schema whitelist request (036 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/666223 (owner: 10Erin Yener) [14:43:48] (03PS3) 10Mforns: WikipediaPortal schema whitelist request [analytics/refinery] - 10https://gerrit.wikimedia.org/r/666223 (owner: 10Erin Yener) [14:45:21] mforns: o/ do you have a min? I am a bit lost in debugging https://phabricator.wikimedia.org/T275757 [14:45:32] (03CR) 10Mforns: [V: 03+2 C: 03+2] "I just added the last adjustments that EYener et. al. discussed with Analytics." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/666223 (owner: 10Erin Yener) [14:45:45] elukey: sure! [14:45:58] is there a way to repro the problem? [14:46:15] elukey: looking [14:46:29] I have deployed a new version of hive that may have solved something, but I tried to re-run a timer on launcher and I see a lot of failures [14:46:36] like AttributeError: 'NoneType' object has no attribute 'strftime' [14:47:46] elukey: hmm... [14:47:59] elukey: wanna pair in da cave? [14:48:13] sure [14:48:16] ok [15:14:51] (03Abandoned) 10Mforns: Adjusting fields requested per privacy conversion earlier this week [analytics/refinery] - 10https://gerrit.wikimedia.org/r/673164 (owner: 10Erin Yener) [15:16:17] 10Analytics: Hive log4j logging is misconfigured - https://phabricator.wikimedia.org/T216294 (10elukey) Neil I think that this can be closed, what do you think? [15:17:22] 10Quarry, 10cloud-services-team (Kanban): Quarry should detect a dead worker and report something better than "running" forever - https://phabricator.wikimedia.org/T278583 (10Bstorm) >>! In T278583#6951002, @zhuyifei1999 wrote: > Hmm. Is the goal trying to find when a worker gets SIGKILL-ed? Celery does > inte... [15:22:44] (03PS4) 10Mforns: MobileWikiAppiOSFeed Whitelist Request [analytics/refinery] - 10https://gerrit.wikimedia.org/r/666227 (owner: 10Erin Yener) [15:23:25] (03CR) 10Mforns: MobileWikiAppiOSFeed Whitelist Request (035 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/666227 (owner: 10Erin Yener) [15:23:54] (03PS5) 10Mforns: MobileWikiAppiOSFeed Whitelist Request [analytics/refinery] - 10https://gerrit.wikimedia.org/r/666227 (owner: 10Erin Yener) [15:24:29] (03CR) 10Mforns: [V: 03+2 C: 03+2] "After our conversation with EYener et.al., this patch is looking ready to deploy. Merging!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/666227 (owner: 10Erin Yener) [15:24:50] (03Abandoned) 10Mforns: Resolving problemating fields our discussion earlier this week [analytics/refinery] - 10https://gerrit.wikimedia.org/r/673161 (owner: 10Erin Yener) [15:28:21] (03PS2) 10Mforns: MobileWikiAppFeed Whitelist Request [analytics/refinery] - 10https://gerrit.wikimedia.org/r/666229 (owner: 10Erin Yener) [15:29:10] (03CR) 10Mforns: MobileWikiAppFeed Whitelist Request (035 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/666229 (owner: 10Erin Yener) [15:29:18] (03PS3) 10Mforns: MobileWikiAppFeed Whitelist Request [analytics/refinery] - 10https://gerrit.wikimedia.org/r/666229 (owner: 10Erin Yener) [15:29:50] (03CR) 10Mforns: [V: 03+2 C: 03+2] "As per our conversation with EYener et.al. I think this is ready to deploy. Merging!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/666229 (owner: 10Erin Yener) [15:30:12] (03Abandoned) 10Mforns: Modified whitelist request for MobileWikiAppFeed [analytics/refinery] - 10https://gerrit.wikimedia.org/r/673157 (owner: 10Erin Yener) [15:41:22] 10Analytics: Cleanup cassandra keyspaces and host - https://phabricator.wikimedia.org/T278231 (10hnowlan) All of the above tables have been moved to `/srv/cassandra-{a,b}/test_tables_T278231`, I will remove these on Thursday. [16:12:44] 10Analytics, 10Data-Services, 10Machine-Learning-Team, 10ORES, and 2 others: Generate dump of scored-revisions from 2018-2020 for English Wikipedia - https://phabricator.wikimedia.org/T277609 (10JAllemandou) Arf - My bad - Let me try to fix that :) [16:18:59] 10Analytics, 10Data-Services, 10Machine-Learning-Team, 10ORES, and 2 others: Generate dump of scored-revisions from 2018-2020 for English Wikipedia - https://phabricator.wikimedia.org/T277609 (10JAllemandou) It should work now. [16:26:48] (03PS8) 10Ottomata: Add support for finding RefineTarget inputs from Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/673604 (https://phabricator.wikimedia.org/T212451) [16:26:52] (03CR) 10Ottomata: Add support for finding RefineTarget inputs from Hive (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/673604 (https://phabricator.wikimedia.org/T212451) (owner: 10Ottomata) [16:31:12] (03CR) 10jerkins-bot: [V: 04-1] Add support for finding RefineTarget inputs from Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/673604 (https://phabricator.wikimedia.org/T212451) (owner: 10Ottomata) [16:41:33] (03CR) 10Awight: "> Yes, agree that this patch is fine to deploy." [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/667192 (https://phabricator.wikimedia.org/T193169) (owner: 10Awight) [16:43:31] (03CR) 10Awight: "I see that I've put bad information into the commit message: the date cannot be written as {year}-{month}-{day} because the numbers won't " [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/667192 (https://phabricator.wikimedia.org/T193169) (owner: 10Awight) [16:52:11] 10Analytics-Clusters, 10Analytics-Kanban: AQS Cassandra storage: Investigate incorrect storage report on Grafana - https://phabricator.wikimedia.org/T278234 (10JAllemandou) Waiting for cassandra 3 is no problem :) [17:04:38] * elukey afk! [17:07:20] !log rebalance kafka partitions for webrequest_text partitions 5 and 6 [17:07:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:22:13] 10Analytics-Clusters, 10Analytics-Kanban: Balance Kafka topic partitions on Kafka Jumbo to take advantage of the new brokers - https://phabricator.wikimedia.org/T255973 (10razzi) [17:25:02] (03CR) 10Ottomata: Improve Refine failure report email (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/674304 (owner: 10Ottomata) [17:25:08] (03PS8) 10Ottomata: Improve Refine failure report email [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/674304 [17:25:49] (03PS9) 10Ottomata: Add support for finding RefineTarget inputs from Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/673604 (https://phabricator.wikimedia.org/T212451) [17:54:04] (03PS9) 10Ottomata: Improve Refine failure report email [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/674304 [17:54:55] joal: ok i think all patches ready to go if they look ok to you [18:06:03] will check ottomata [18:06:04] thanks :) [18:06:11] mforns: shall we deploy? [18:06:27] joal: doing your cr right now [18:06:32] Ah! [18:06:42] thank you for that mforns :) [18:29:54] (03CR) 10Mforns: Update mediawiki-dumps-importer (033 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/675763 (https://phabricator.wikimedia.org/T278551) (owner: 10Joal) [18:30:11] joal: sorry for taking long, I wanted to understand the change [18:30:24] (03CR) 10Joal: "I just realized about this :S sorry" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/673604 (https://phabricator.wikimedia.org/T212451) (owner: 10Ottomata) [18:30:24] and I had never seen that code IIRC [18:30:48] ottomata: I just realized there was a better way for the is_valid val [18:30:51] joal: but whyYyYy? [18:30:53] see my comment above [18:30:57] then I have to copy/paste all the params again [18:31:13] NOOOO! those parameters are inner to the class :) [18:31:29] how can you declare the constructor? [18:31:37] OH [18:31:40] i see [18:31:43] just call it [18:31:48] not override the constructor [18:31:49] just call it in the body [18:31:50] ok [18:32:04] indeed ottomata [18:32:17] dong... [18:32:18] doing*( [18:32:24] ottomata: you could even for the whole piece of code in class body, but eh, function looks better :) [18:32:27] thanks a lot ottomata [18:33:17] reading mforns - thanks for the review [18:33:22] np! [18:33:26] mforns: that code is quite old [18:34:07] joal: is a def ... Unit better instaed of Boolean then [18:34:10] and make it private? [18:35:19] (03PS10) 10Ottomata: Add support for finding RefineTarget inputs from Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/673604 (https://phabricator.wikimedia.org/T212451) [18:35:23] ottomata: I prefer the explicit call in constructor - but this is mere preference [18:35:37] (03CR) 10Ottomata: Add support for finding RefineTarget inputs from Hive (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/673604 (https://phabricator.wikimedia.org/T212451) (owner: 10Ottomata) [18:35:50] joal: yese explicit call still [18:35:53] see patch [18:36:33] great ottomata - I like that :) [18:36:36] thanks again [18:36:38] gr8 me too [18:37:32] (03PS11) 10Ottomata: Add support for finding RefineTarget inputs from Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/673604 (https://phabricator.wikimedia.org/T212451) [18:37:50] (03PS10) 10Ottomata: Improve Refine failure report email [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/674304 [18:41:20] (03CR) 10Joal: [V: 03+1] Update mediawiki-dumps-importer (033 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/675763 (https://phabricator.wikimedia.org/T278551) (owner: 10Joal) [18:43:20] (03PS3) 10Joal: Update mediawiki-dumps-importer [analytics/refinery] - 10https://gerrit.wikimedia.org/r/675763 (https://phabricator.wikimedia.org/T278551) [18:43:26] mforns: second round! --^ [18:43:28] :) [18:43:35] joal: lookin [18:45:15] (03CR) 10Mforns: [C: 03+2] "LGTM!" (033 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/675763 (https://phabricator.wikimedia.org/T278551) (owner: 10Joal) [18:47:02] thanks mforns :) [18:47:08] mforns: deploy? [18:47:15] joal: sure! [18:47:24] wanna bc? or chat? [18:47:36] mforns: batcave feels easier if ok for you [18:47:44] ok! [18:47:46] at least for the beginning sync [18:50:35] (03CR) 10Joal: [C: 03+2] "Merging for deploy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/673202 (https://phabricator.wikimedia.org/T277536) (owner: 10Joal) [18:51:45] (03CR) 10Mforns: [V: 03+2 C: 03+2] Update mediawiki-dumps-importer [analytics/refinery] - 10https://gerrit.wikimedia.org/r/675763 (https://phabricator.wikimedia.org/T278551) (owner: 10Joal) [18:56:16] (03PS6) 10Mforns: Update mysql resolver to work with cloud replicas [analytics/refinery] - 10https://gerrit.wikimedia.org/r/666209 (https://phabricator.wikimedia.org/T274690) (owner: 10Milimetric) [18:56:21] (03CR) 10Mforns: [V: 03+2 C: 03+2] Update mysql resolver to work with cloud replicas [analytics/refinery] - 10https://gerrit.wikimedia.org/r/666209 (https://phabricator.wikimedia.org/T274690) (owner: 10Milimetric) [18:57:26] (03Merged) 10jenkins-bot: Update WMF domain list with Cloud and toolforge [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/673202 (https://phabricator.wikimedia.org/T277536) (owner: 10Joal) [18:59:32] (03CR) 10Joal: [C: 03+2] "Merging for deploy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/670269 (https://phabricator.wikimedia.org/T273789) (owner: 10Ottomata) [19:06:46] (03Merged) 10jenkins-bot: Rename whitelist to allowlist for Refine sanitization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/670269 (https://phabricator.wikimedia.org/T273789) (owner: 10Ottomata) [19:16:59] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for deploy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/670321 (https://phabricator.wikimedia.org/T273789) (owner: 10Ottomata) [19:17:30] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for deploy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/673604 (https://phabricator.wikimedia.org/T212451) (owner: 10Ottomata) [19:18:04] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for deploy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/674304 (owner: 10Ottomata) [19:21:55] (03PS1) 10Joal: Bump changelog.md to v0.1.3 for deloy [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/675891 [19:22:51] (03CR) 10Mforns: [V: 03+2 C: 03+2] "LGTM!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/675891 (owner: 10Joal) [19:32:58] (03PS12) 10Razzi: Upgrade superset to 1.0.1 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/665130 (https://phabricator.wikimedia.org/T272390) [19:42:55] (03PS13) 10Razzi: Upgrade superset to 1.0.1 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/665130 (https://phabricator.wikimedia.org/T272390) [19:47:07] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.1.3 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/675894 [19:49:33] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/675894 (owner: 10Maven-release-user) [19:52:40] 10Analytics: Produce a list of wiki projects ranked by number of eligible voters in Board elections - https://phabricator.wikimedia.org/T278815 (10mforns) Hi! We tried this query to extract the rank of wikis per voter base: ` WITH base_data AS ( SELECT wiki_db, event_user_id, MAX(even... [19:57:23] !log Refinery-source released to archiva and new jars commited to refinery (v0.1.3) [19:57:26] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:57:34] !log Deploying refinery using scap [19:57:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:09:35] 10Analytics-Clusters, 10Analytics-Kanban: Configure the HDFS Namenodes to use the log4j rolling gzip appender - https://phabricator.wikimedia.org/T276906 (10Ottomata) @elukey if we puppetize the hadoop log4j file for this it will make {T265126} easier for @razzi. [20:14:09] 10Analytics-Clusters, 10Analytics-Kanban, 10observability, 10User-fgiunchedi: Setup Analytics team in VO/splunk oncall - https://phabricator.wikimedia.org/T273064 (10razzi) Ok thanks @fgiunchedi. I'll try adding alerting to the superset service for starters. [20:16:24] 10Analytics-Clusters, 10Product-Analytics: Can't re-run failed Oozie workflows in Hue/Hue-Next (as non-admin) - https://phabricator.wikimedia.org/T275212 (10razzi) @nshahquinn-wmf any luck with that setting? [20:19:19] !log Deploying refinery onto HDFS [20:19:21] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:23:28] 10Analytics-Radar, 10Patch-For-Review: Reportupdater output can be corrupted by hive logging - https://phabricator.wikimedia.org/T275757 (10awight) I'm strangely unable to produce any parquet logs at the moment, but will update here once I stumble across a good test case. Even a daily failing query like `code... [20:25:03] !log Kill-Restart data_quality_stats-hourly-bundle after deploy [20:25:05] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:27:35] Deployment done :) [20:27:41] Gone for tonight ! [20:28:06] thanks joal! laters [20:43:43] 10Analytics-Radar, 10Patch-For-Review: Reportupdater output can be corrupted by hive logging - https://phabricator.wikimedia.org/T275757 (10awight) Confirmed, ` hive -e 'set system:java.util.logging.config.file' system:java.util.logging.config.file=/usr/lib/hive/bin/../conf/parquet-logging.properties... [21:10:58] PROBLEM - Check unit status of refine_sanitize_eventlogging_analytics_immediate on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit refine_sanitize_eventlogging_analytics_immediate https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [21:26:09] ^ looking into the above, seeing java.lang.ClassNotFoundException: org.wikimedia.analytics.refinery.job.refine.EventLoggingSanitization [21:55:36] (03PS1) 10GoranSMilovanovic: Qurator Curious Facts - Minor [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/675916 [21:56:01] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] Qurator Curious Facts - Minor [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/675916 (owner: 10GoranSMilovanovic) [23:44:40] (03PS1) 10GoranSMilovanovic: WDCM ML [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/675927 [23:44:56] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] WDCM ML [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/675927 (owner: 10GoranSMilovanovic)