[02:03:10] 10Analytics, 10Analytics-Kanban, 10EventBus, 10serviceops, and 3 others: Set up LVS for eventgate-main on port 32192 - https://phabricator.wikimedia.org/T222899 (10Dzahn) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=lvs2003&service=PyBal+IPVS+diff+check https://icinga.wikimedia.org/... [04:24:54] PROBLEM - Check the last execution of monitor_refine_eventlogging_analytics on an-coord1001 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_eventlogging_analytics [06:17:57] morning! [06:23:18] refine eventlogging is really grumpy in these days :D [07:57:34] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 4 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10akosiaris) [07:57:40] 10Analytics, 10Analytics-Kanban, 10EventBus, 10serviceops, and 3 others: Set up LVS for eventgate-main on port 32192 - https://phabricator.wikimedia.org/T222899 (10akosiaris) 05Open→03Resolved >>! In T222899#5179002, @Dzahn wrote: > https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=lv... [08:54:13] RECOVERY - Check the last execution of monitor_refine_eventlogging_analytics on an-coord1001 is OK: OK: Status of the systemd unit monitor_refine_eventlogging_analytics [08:58:49] Good morning team - Bad week for regular schedule :S Naé is sick with otitis and stays home today [09:00:14] ouch :( [09:01:06] elukey: no big deal though - No strong fever, no strong pain (or at least she has not showed us), but still some rest is needed [09:01:45] Plus the fact that tomorrow Melissa can't be on time, plus teachers strike on Thursday - This makes a bizzare week for me in term of schedule - I apologize for that [09:02:32] joal: you're not saying "I am going a couple of days to Ibiza, see ya!" :D [09:02:38] family first! [09:03:34] You know me elukey - Family first, and team commitment :) [09:04:17] I think I'll need to be rather unproductive during the next days as well, I haven't started peer reviews :( [09:04:52] elukey: wow - I haven't started that either :S [09:06:21] oh noes, Friday is the deadline for the self review as well /o\ [09:13:21] heh [11:02:49] 10Analytics, 10Core Platform Team Backlog, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), 10Performance: > 2% of API wall time spent generating UUIDs - https://phabricator.wikimedia.org/T222966 (10Pchelolo) > What is this mediawiki.api-request data ultimately for and why does it require... [11:13:26] (03PS1) 10GoranSMilovanovic: initial [analytics/wmde/WD/WD_percentUsageDashboard] - 10https://gerrit.wikimedia.org/r/510132 [11:13:41] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] initial [analytics/wmde/WD/WD_percentUsageDashboard] - 10https://gerrit.wikimedia.org/r/510132 (owner: 10GoranSMilovanovic) [12:17:13] good morners! [12:18:12] o/ [13:04:00] hey all [13:04:38] elukey: I want your opinion on a Druid extension [13:04:58] sure! [13:06:07] ok, it's more involved than your basic extension: https://github.com/yahoo/maha/tree/master/druid-lookups [13:06:19] if you want we can chat in the cave? [13:06:31] 10Analytics, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), 10Core Platform Team Backlog (Next), 10Performance: > 2% of API wall time spent generating UUIDs - https://phabricator.wikimedia.org/T222966 (10Pchelolo) [13:07:14] so it provides basically a way to join values from Druid with values in a lookup stored in Mongo or RocksDB [13:08:05] Druid has this built-in, but it can only join to very small in-memory lists or still small off-heap cached lists [13:08:24] from my research, those lists can't get bigger than about 10 million, Druid docs say 10 thousand! [13:08:55] We need probably around 300 million entries at least, so I looked around and this seems to be the only solution if lookups are to work for us at all [13:10:02] I can describe what I've learned about the service, like the pieces it seems to need, and we can look over the code together to get an idea of operational load [13:10:22] like, I think this seems like a great addition and solves a lot of problems, but not if it would be a headache for you all to run [13:10:55] milimetric: gimme 10 mins and I'll be ready to chat [13:11:03] np, ping anytime [13:23:16] milimetric: ok I am ready [13:23:25] cool, omw [13:23:59] joal or others: if you want to hang out - druid lookups chat [13:24:06] (in the cave elukey) [13:25:22] 10Analytics, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), 10Core Platform Team Backlog (Next), 10Performance: > 2% of API wall time spent generating UUIDs - https://phabricator.wikimedia.org/T222966 (10Ottomata) > Could the WebRequest::getRequestId() suffice instead We also log this a... [13:29:07] 10Analytics, 10Analytics-Kanban, 10EventBus: Port usage of mediawiki_ApiAction to mediawiki_api_request - https://phabricator.wikimedia.org/T222267 (10Ottomata) @EvanProdromou whatcha think about a timeline on this? Turning off the ApiAction events is a goal for this quarter, and this task has to be resolve... [13:30:49] 10Analytics, 10Analytics-Kanban, 10Discovery, 10Discovery-Analysis, and 3 others: Port usage of mediawiki_CirrusSearchRequestSet to mediawiki_cirrussearch_request - https://phabricator.wikimedia.org/T222268 (10Ottomata) a:03EBernhardson Heya Erik! I know you said you would try to get to this after the H... [13:51:53] a team, fyi am looking into these eventlogging camus / monitor refin stuff [13:56:48] ottomata: o/ I didn't find anything obvious this morning, but then I got sidetracked by other stuff.. after re-running the monitor refine check everything seemed good [13:57:24] elukey: in the time ranges where things were affected [13:57:25] https://grafana.wikimedia.org/d/000000505/eventlogging?panelId=6&fullscreen&orgId=1&from=now-24h&to=now-5m was also interesting [13:57:33] i see a big jump in CitationUsagePageLoad events [13:58:00] ah yes it could be what the EL throghput graph shows [13:58:04] (brb) [14:44:09] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10MW-1.34-release, and 2 others: Remove deprecated EventLogging schema modules - https://phabricator.wikimedia.org/T221281 (10Milimetric) [14:44:23] 10Analytics, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), 10Core Platform Team Backlog (Next), 10Performance: > 2% of API wall time spent generating UUIDs - https://phabricator.wikimedia.org/T222966 (10Krinkle) >>! In T222966#5179973, @Pchelolo wrote: > We have established a rule tha... [14:59:59] 10Analytics, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), 10Core Platform Team Backlog (Next), 10Performance: > 2% of API wall time spent generating UUIDs - https://phabricator.wikimedia.org/T222966 (10Ottomata) If Mediawiki retries the HTTP request to produce the event, or EventGate'... [15:05:34] 10Analytics, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), 10Core Platform Team Backlog (Next), 10Performance: > 2% of API wall time spent generating UUIDs - https://phabricator.wikimedia.org/T222966 (10Ottomata) Q: Is there any reason constructing the event couldn't be deferred until... [15:07:47] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10MW-1.34-release, and 2 others: Remove deprecated EventLogging schema modules - https://phabricator.wikimedia.org/T221281 (10Milimetric) @Krinkle I found three tasks that I attached as subtasks, they're all resolved already. If you know of any ot... [15:21:48] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Multimedia, and 2 others: Remove deprecated using schema.* syntax from MultimediaViewer - https://phabricator.wikimedia.org/T223284 (10Milimetric) [15:21:52] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10MediaWiki-extensions-WikimediaEvents, and 2 others: Remove deprecated using schema.* syntax from WikimediaEvents - https://phabricator.wikimedia.org/T223285 (10Milimetric) [15:21:54] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10WikiEditor, and 2 others: Remove deprecated using schema.* syntax from WikiEditor - https://phabricator.wikimedia.org/T223286 (10Milimetric) [15:24:38] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Multimedia, and 2 others: Remove deprecated using schema.* syntax from MultimediaViewer - https://phabricator.wikimedia.org/T223284 (10Milimetric) [15:33:35] !log restart turnilo to clear deleted datasource [15:33:50] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:34:08] Hey milimetric - sorry was away when you pinged - May I help in any way? [15:34:34] joal: sorry Nae's not feeling well, I read up after pinging [15:34:46] joal: only if you want to hang out and chat about it, happy to [15:35:07] np - She's very tough, like nothing is bad in her behavior, but her eyes are swollen and she starts having fever :( [15:35:14] let's talk :) [15:36:53] omw cave [15:54:20] elukey: what's the status of the hadoop test cluster atm? [15:54:23] is it fully kerberized? [15:54:25] (03PS1) 10Mforns: Add new dimensions to edit_houly data set [analytics/refinery] - 10https://gerrit.wikimedia.org/r/510188 [15:54:29] can/should I use it to test https://hadoop.apache.org/docs/current/hadoop-openstack/index.html#Configuring stuff ? [15:54:32] oops sorry [15:54:35] meant to paste https://phabricator.wikimedia.org/T219544 [15:55:13] ottomata: not yet, me and Moritz haven't finalized the kerberos service [15:55:16] so it is still wip [15:55:21] you can use it if you want [15:56:40] ok awesome [16:00:41] 10Analytics, 10Product-Analytics: Add additional dimensions to edits_hourly in Turnilo and Superset - https://phabricator.wikimedia.org/T219323 (10mforns) The new datasource is available in [[ https://turnilo.wikimedia.org/#edits_hourly/3/N4IgbglgzgrghgGwgLzgFwgewHYgFwhLYCmAtAMYAWcATmiADQgYC2xyOx+IAomuQHoAqgBU... [16:07:24] 10Analytics, 10Product-Analytics: Add additional dimensions to edits_hourly in Turnilo and Superset - https://phabricator.wikimedia.org/T219323 (10MNeisler) [16:11:05] (03CR) 10Milimetric: "I think we should fix the other two problems described here: https://phabricator.wikimedia.org/T219112#5172950, from your last comment it " [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/509867 (https://phabricator.wikimedia.org/T219112) (owner: 10Fdans) [16:13:01] 10Analytics: Upgrade pandas in spark SWAP notebooks - https://phabricator.wikimedia.org/T222301 (10Ottomata) Oh @Groceryheist! It is possible to run distributed pyspark jobs with custom virtualenvs. I've never done it, but @EBernhardson knows how. I think he has some documentation, but I can't find it. [16:15:43] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban: Upgrade Spark to 2.4.2 - https://phabricator.wikimedia.org/T222253 (10Ottomata) [16:18:18] neilpquinn: please take a look at https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/510188/ to see addition of tags to turnilo [16:18:31] neilpquinn: (re: edit_hourly dataset) [16:20:13] 10Analytics, 10Analytics-SWAP: Upgrade R in SWAP notebooks to 3.4+ - https://phabricator.wikimedia.org/T222933 (10mpopov) By the way, there's already a task for this: T220542 In which I do point out that there's an official Debian Stretch repo from CRAN that we could get the latest version of R from: https://... [16:30:03] 10Analytics, 10EventBus, 10Operations, 10observability, and 4 others: Upgrade statsd_exporter to 0.9 - https://phabricator.wikimedia.org/T220709 (10fgiunchedi) All production has been updated ! Leaving open for now in case there's still upgrades to be done in k8s (cc @akosiaris ) [16:32:28] RoanKattouw: were you going to join the delayed job meeting (happening now) [17:30:59] 10Analytics, 10Analytics-SWAP: Upgrade R in SWAP notebooks to 3.4+ - https://phabricator.wikimedia.org/T222933 (10mpopov) @Groceryheist: https://meta.wikimedia.org/wiki/User:MPopov_(WMF)/Notes/RStan [17:40:43] * milimetric running out for a longer lunch [17:46:04] 10Analytics, 10Product-Analytics: Update R from 3.3.3 to 3.5.3 on stat and notebook machines - https://phabricator.wikimedia.org/T220542 (10mpopov) [18:05:04] * elukey off! [18:58:47] ottomata: yt? [18:59:02] nuria: ya [18:59:09] ottomata: i have backfilled data for 04-01 hour 20th in kafka [18:59:18] ottomata: but i assume i need to run refine [18:59:22] hmm [18:59:31] with am earlier dayte for it to be refined? [18:59:32] yes you probably do, refine only looks back so far [19:00:01] you should be able to do it like https://wikitech.wikimedia.org/wiki/Analytics/Systems/Refine#Rerunning_jobs [19:00:16] ottomata: will it override data currently there? [19:00:18] sudo -u hdfs /usr/local/bin/refine_eventlogging_analytics --since=XXXX --until=XXXX [19:00:21] ottomata: yes, right? [19:00:44] yes, but as long as the old data still exists in /wmf/raw [19:00:50] it should re-refien the full hour [19:00:57] with your new data too [19:01:27] ottomata: k, will do now [19:01:32] do you know if you have new data in /wmf/raw from camus? [19:01:40] btw, the --since and --until can take ISO-8601 dts [19:01:44] ottomata: ah no, let me look, it should be there [19:01:45] so you don't have to calcualte hours [19:02:02] /wmf/data/raw * [19:02:18] e.g. nuria what's a schema that was backfilled? [19:02:25] ottomata: all [19:02:42] so in [19:02:48] ottomata: as we had an outage and data did not made it to event_* partitions [19:02:58] so in [19:02:59] /mnt/hdfs/wmf/data/raw/eventlogging/eventlogging_VirtualPageView/hourly/2019/04/01/20 [19:03:06] there shoudl be new files with recent timestamps [19:03:09] and there are! [19:03:09] so yay! [19:03:22] so [19:03:31] if you rerun refine and tell it to look in the proper timerange with --since and --until [19:03:33] it should notice the new data [19:03:36] and re-refine those house [19:03:37] hours [19:03:45] I thikn you'll only have to override since and until [19:03:53] sudo -u hdfs /usr/local/bin/refine_eventlogging_analytics --since XXX --until XXX [19:04:03] this is on an-coord1001 [19:04:25] ottomata: ok, then, i am going to backfill all hours (i have two more and it takes a while) and i will run refine after [19:04:30] ottomata: superthnks [19:05:36] nuria: if you want to double check [19:05:39] you can run with a --limit [19:05:51] that will make the job runa little quicker and only do a few hours at once [19:06:00] then you can yarn-logs the thing (since it will run in yarn cluster) [19:06:02] and see what it refined [19:06:10] and check that there is new data there [19:06:18] i guess you could select count(*) on an hour before and after you refein [19:06:20] refine [19:07:45] ottomata: ya, i did the select counts already [19:08:02] coo [19:20:57] 10Analytics, 10Discovery, 10Operations, 10Research: Make hadoop cluster able to push to swift - https://phabricator.wikimedia.org/T219544 (10Ottomata) Found some better docs here: https://docs.openstack.org/sahara/latest/user/hadoop-swift.html So configs will go in core-site.xml. We can probably do this... [19:37:02] ottomata: k, all documented here: https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/Backfilling#Backfilling_a_kafka_event_%3Cschema%3E_topic [19:38:33] nice [19:40:00] edited a bit :) [19:40:10] some topics do have more than one partition [19:43:38] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Sanitization should be run a second time - https://phabricator.wikimedia.org/T212014 (10Nuria) 05Open→03Resolved [19:43:51] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Fix jobs after mediawiki-history refactor - https://phabricator.wikimedia.org/T222425 (10Nuria) 05Open→03Resolved [19:44:22] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: 15.wikipedia.org missclassified as a pageview, same for query.wikidata.org - https://phabricator.wikimedia.org/T222460 (10Nuria) 05Open→03Resolved [19:45:16] 10Analytics: Move reportupdater queries from limn-* repositories to reportupdater-queries - https://phabricator.wikimedia.org/T222739 (10Nuria) a:05Milimetric→03fdans [19:46:37] 10Analytics: Move reportupdater queries from limn-* repositories to reportupdater-queries - https://phabricator.wikimedia.org/T222739 (10Nuria) [19:47:09] 10Analytics: Move reportupdater queries from limn-* repositories to reportupdater-queries - https://phabricator.wikimedia.org/T222739 (10Nuria) Assigning to @fdans, if we can do away with limn-* repositories we are set to move reportupdater queries away from accessing eventlogging data in hadoop entirely. [19:48:03] 10Analytics, 10Product-Analytics, 10Epic, 10User-Elukey: Provide feature parity between the wiki replicas and the Analytics Data Lake - https://phabricator.wikimedia.org/T212172 (10Nuria) [19:48:07] 10Analytics, 10Analytics-Kanban: Provide edit tags in the Data Lake edit data - https://phabricator.wikimedia.org/T161149 (10Nuria) 05Open→03Resolved [19:49:27] 10Analytics, 10Analytics-Kanban: [Bug] Type mismatch for a few other schemas - https://phabricator.wikimedia.org/T216771 (10Nuria) 05Open→03Resolved [19:49:29] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Performance-Team (Radar): [Bug] Type mismatch between NavigationTiming EL schema and Hive table schema - https://phabricator.wikimedia.org/T214384 (10Nuria) [19:50:18] 10Analytics, 10Analytics-Kanban: Refine eventlogging pipeline should not refine data for domains that are not wikimedia's - https://phabricator.wikimedia.org/T219828 (10Nuria) a:03mforns [19:51:56] milimetric: is there anything remaining for the geowiki yearly job? [19:52:05] milimetric: for "edits" , not editors [19:52:49] 10Analytics, 10good first bug: Productionize and run 2018 job for Global Innovation Index from Hadoop Geowiki data - https://phabricator.wikimedia.org/T190535 (10fdans) a:05fdans→03Milimetric [19:53:04] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Mediawiki-History fixes before deploy - https://phabricator.wikimedia.org/T222141 (10Nuria) 05Open→03Resolved [19:53:08] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Mediawiki History Release - 2019-04 snapshot - https://phabricator.wikimedia.org/T221824 (10Nuria) [19:53:16] it’s just broken, it needs to be fixed. But it’s just an aggregate of the monthly one. Should probably just be done manually every year, I did it mostly ‘cause I was curious about yearly jobs [19:53:26] nuria: ^ [19:53:46] milimetric: broken, in what way? [19:54:27] nuria: I just messed up the input condition I think [19:55:03] milimetric: ok, I think we should fix i rather than leaving code like that no? [19:55:47] nuria: sure, I can do it now if you think it takes priority [19:56:26] milimetric: I think it can wait until you are done with your current task but let just not forget it. [19:57:17] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 4 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10Nuria) [19:57:20] 10Analytics, 10Analytics-Kanban, 10EventBus, 10serviceops, and 2 others: Use new eventgate chart release analytics for eventgate-analytics service. - https://phabricator.wikimedia.org/T222962 (10Nuria) 05Open→03Resolved [19:57:49] ok, makes sense nuria, I’ll do it next [20:11:47] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 4 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10Nuria) 05Open→03Resolved [20:11:50] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Core Platform Team Backlog (Watching / External), and 2 others: Modern Event Platform: Stream Intake Service: Migrate Mediawiki Eventbus events to EventGate - https://phabricator.wikimedia.org/T211248 (10Nuria) [20:21:10] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Operations, and 2 others: Upgrade python-kafka - https://phabricator.wikimedia.org/T221848 (10Gilles) I'd like to understand this bug better before rolling back the package for coal. It's not a big deal per se if coal is a little behind events. W... [20:29:20] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Clickstream dataset for Persian Wikipedia only includes external values - https://phabricator.wikimedia.org/T191964 (10Nuria) Thanks @JAllemandou and sorry @Ladsgroup that in totally missed on my tests that your patch fixed the problem. @JAllemandou can y... [20:31:55] joal: yt? [20:33:08] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Technical-Debt: Remove Zero support in analytics - https://phabricator.wikimedia.org/T213770 (10Nuria) 05Open→03Resolved [20:33:42] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: The three sqoop jobs that scoop mediawiki history should do in sequence - https://phabricator.wikimedia.org/T222378 (10Nuria) [20:33:54] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: The three sqoop jobs that scoop mediawiki history should do in sequence - https://phabricator.wikimedia.org/T222378 (10Nuria) 05Open→03Resolved [20:34:12] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Mandatory success_email_to parameter in mediawiki_history_check coordinator - https://phabricator.wikimedia.org/T222422 (10Nuria) 05Open→03Resolved [20:34:28] 10Analytics-Kanban, 10Product-Analytics: Address data quality issues in the mediawiki_history dataset - https://phabricator.wikimedia.org/T204953 (10Nuria) [20:34:31] 10Analytics-Kanban, 10Patch-For-Review: Coordinate work on minor changes for Edit Data Quality - https://phabricator.wikimedia.org/T213603 (10Nuria) 05Open→03Resolved [20:34:46] 10Analytics-Kanban: Meta-statistics on MediaWiki history reconstruction process - https://phabricator.wikimedia.org/T155507 (10Nuria) [20:34:49] 10Analytics, 10Analytics-Kanban: Add caused_by_user_text to mediawiki_page_history - https://phabricator.wikimedia.org/T167608 (10Nuria) 05Open→03Resolved [20:35:08] 10Analytics, 10Analytics-Kanban, 10Anti-Harassment, 10Product-Analytics: Add partial blocks to mediawiki history tables - https://phabricator.wikimedia.org/T211950 (10Nuria) [20:35:17] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Mediawiki History Release - 2019-04 snapshot - https://phabricator.wikimedia.org/T221824 (10Nuria) [20:35:20] 10Analytics, 10Product-Analytics, 10Epic, 10User-Elukey: Provide feature parity between the wiki replicas and the Analytics Data Lake - https://phabricator.wikimedia.org/T212172 (10Nuria) [20:35:23] 10Analytics, 10Analytics-Kanban, 10Anti-Harassment, 10Product-Analytics: Add partial blocks to mediawiki history tables - https://phabricator.wikimedia.org/T211950 (10Nuria) 05Open→03Resolved [20:35:49] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Mediawiki History Release - 2019-04 snapshot - https://phabricator.wikimedia.org/T221824 (10Nuria) [20:35:51] 10Analytics, 10Analytics-Wikistats: Feedback on hive table mediawiki_history by Erik Z - https://phabricator.wikimedia.org/T178591 (10Nuria) [20:35:53] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Add user_is_bot_by to MediaWiki history - https://phabricator.wikimedia.org/T219177 (10Nuria) 05Open→03Resolved [20:36:06] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Mediawiki History Release - 2019-04 snapshot - https://phabricator.wikimedia.org/T221824 (10Nuria) [20:36:09] 10Analytics-Kanban, 10Product-Analytics: Address data quality issues in the mediawiki_history dataset - https://phabricator.wikimedia.org/T204953 (10Nuria) [20:36:12] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Some registered users have null values for event_user_text and event_user_text_historical in mediawiki_history - https://phabricator.wikimedia.org/T218463 (10Nuria) 05Open→03Resolved [20:36:22] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Fundraising-Backlog, and 3 others: Fix EventLogging schemas that use array for items type - https://phabricator.wikimedia.org/T218617 (10Nuria) [20:37:54] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10Product-Analytics: mediawiki_history datasets have null user_text for IP edits - https://phabricator.wikimedia.org/T206883 (10Nuria) [20:40:52] 10Analytics, 10Analytics-Wikistats: Wikistats 2: New Pages split by editor type wrongly claims no anonymous users create pages - https://phabricator.wikimedia.org/T185342 (10Nuria) Fixed with 2019-04 snapshot, closing. Thanks for reporting. [20:41:27] hmmm nuria [20:41:30] when you backfilled [20:41:37] did you produce with kafka:// [20:41:41] or with confluent-kafka:// [20:43:18] 10Analytics, 10Analytics-Wikistats: Wikistats 2: New Pages split by editor type wrongly claims no anonymous users create pages - https://phabricator.wikimedia.org/T185342 (10Nuria) 05Open→03Resolved [20:43:26] ? [20:43:49] 10Analytics, 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics, 10Patch-For-Review: "Edit" equivalent of pageviews daily available to use in Turnilo and Superset - https://phabricator.wikimedia.org/T211173 (10Nuria) [20:43:51] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Set up edit_hourly data set in Hive - https://phabricator.wikimedia.org/T220092 (10Nuria) 05Open→03Resolved [20:44:50] 10Analytics-Kanban, 10Patch-For-Review: Fix mediawiki-history-checker after field rename - https://phabricator.wikimedia.org/T219484 (10Nuria) 05Open→03Resolved [20:45:40] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Mediawiki History Release - 2019-04 snapshot - https://phabricator.wikimedia.org/T221824 (10Nuria) Ping @JAllemandou the tasks not closed on 2019-04 snapshot should probably be moved to 2019-05 snapshot cc @fdans [20:45:57] 10Analytics-Kanban, 10Product-Analytics: Address data quality issues in the mediawiki_history dataset - https://phabricator.wikimedia.org/T204953 (10Nuria) [20:46:00] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10Product-Analytics: mediawiki_history datasets have null user_text for IP edits - https://phabricator.wikimedia.org/T206883 (10Nuria) 05Open→03Resolved [20:47:12] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review: Make Refine use JSONSchemas of event data to support Map types and proper types for integers vs decimals - https://phabricator.wikimedia.org/T215442 (10Nuria) 05Open→03Resolved [20:47:20] 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10EventBus, and 6 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Nuria) [20:47:34] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), and 2 others: Modern Event Platform: Schema Registry: Implementation - https://phabricator.wikimedia.org/T206789 (10Nuria) [20:47:36] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), and 3 others: Schema Registry HTTP Service - https://phabricator.wikimedia.org/T219552 (10Nuria) 05Open→03Resolved [20:50:03] ottomata: yt? [20:50:08] ya [20:50:14] ottomata: i am like super lost as to what is this: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/507326/ [20:50:24] nuria btw, did you see muy ping ^^^^^ [20:50:37] ottomata: no, sorry, just now? [20:50:40] ya a bit ago [20:50:47] Andrew Otto hmmm nuria [20:50:47] 16:41:31 when you backfilled [20:50:47] 16:41:38 did you produce with kafka:// [20:50:47] 16:41:42 or with confluent-kafka:// [20:51:00] sorry, or with kafka-confluent:// [20:51:19] nuria: re change, that was a nice little schema registry UI :) [20:52:13] ottomata: re:kafka [20:52:17] https://www.irccloud.com/pastebin/J83TjMaA/ [20:52:33] nuria, try [20:52:34] ssh -N -L 8190:schema.svc.eqiad.wmnet:8190 stat1007.eqiad.wmnet [20:52:35] then [20:52:38] http://localhost:8190/#!/ [20:52:44] OH rats no [20:52:46] that won't work [20:52:58] HMMM interesting nuria. [20:53:01] NOT MY BACKFILLIN! [20:53:10] OH yes it will [20:53:16] * nuria puffff [20:53:24] haha [20:53:25] sorry [20:53:27] we are having 2 convos at once [20:53:28] ottomata: what is the difference [20:53:35] ok first that change you linked [20:53:39] do that ssh tunnel [20:53:40] then [20:53:46] http://localhost:8190/#!/ [20:54:07] ottomata: OOOHHH [20:54:23] will work with multi repos too, if we just clone new ones in [20:54:34] anyway, that is probably not final form [20:54:49] but it was a super simple html file browser that was better than the built in one [20:54:51] that's all [20:54:54] ok next convo [20:54:55] EL backfill [20:55:00] the reason I ask [20:55:08] is that I am seeing April 1 timestamps in camus history files! [20:55:22] ottomata: ya, is that making camus alarms? [20:55:22] which caused the most recent CamusPartitionChecker alert for VirtualPageView [20:55:25] ya [20:55:26] but [20:55:29] afaict [20:55:34] kafka-confluent:// writer [20:55:39] doesn't set the event time. [20:55:48] so, i don't know how kafka+camus would have gotten the april timestamp [20:55:54] it should have just used broker receive time [20:56:07] kafka:// well set kafka message timestamp [20:56:10] ottomata: wait, it is the time at which event was produced [20:56:52] so [20:56:54] with kafka:// [20:56:55] https://github.com/wikimedia/eventlogging/blob/master/eventlogging/handlers.py#L259-L281 [20:57:10] but [20:57:14] kafka-confluent:// [20:57:14] https://github.com/wikimedia/eventlogging/blob/master/eventlogging/handlers.py#L462-L463 [20:57:20] doesn't set any timestamp [20:57:22] so [20:57:33] if the kafka produce request doesn't have a set timestsamp [20:57:42] the kafka broker will set the current receive time as the message timestamp [20:58:00] camus is saving two things to its history files when it reads from kafka: [20:58:11] the kafka message timestamp, and the kafka message offset [20:58:32] so, this makes total sense if you had used kafka:// [20:58:37] but since you used kafka-confluent:// [20:58:39] i'm not sure... [20:58:52] dunno how it would see april 1 at all.....OH [20:58:52] wait [20:59:01] camus doesn't know about kafka messages timestamps DUH [20:59:03] ottomata: there are 2 producers right? [20:59:04] it extracts them out of the message [20:59:06] content [20:59:15] because its a really old client, pre timestamp aware stuff [20:59:23] sorry i am confusing myself here [20:59:32] so that timestamp that is written to camus offset files IS from the message timestamp itself [20:59:37] which is why we see the april 1 time. [20:59:56] ya, ya i see what you mean [21:00:36] so this is a false alrm [21:00:37] emailing... [21:01:50] ottomata: i still do not understand why the alarm is triggered though [21:02:39] nuria just sent email, might explain [21:02:43] k [21:03:41] ottomata: got it [21:04:17] 10Analytics-Engineering: Can't log into Superset - https://phabricator.wikimedia.org/T223335 (10kaldari) [21:04:34] 10Analytics: Can't log into Superset - https://phabricator.wikimedia.org/T223335 (10kaldari) [21:06:45] 10Analytics: Can't log into Superset - https://phabricator.wikimedia.org/T223335 (10Nuria) Please try again, we have a patch upstream that shoudl resolve this issue, in the meantime I have added you as a user manually. [21:08:20] ottomata: it will take me couple more hours but i am going to backfill all data so we will get two more alarms [21:09:06] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Mediawiki History Release - 2019-04 snapshot - https://phabricator.wikimedia.org/T221824 (10Nuria) [21:09:11] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Update wikimedia-history revision data with deleted field (and find it a new name?) - https://phabricator.wikimedia.org/T178587 (10Nuria) 05Open→03Resolved [21:09:29] nuria: ya k should be fine [21:09:40] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Remove dead code from refinery/oozie folders - https://phabricator.wikimedia.org/T221460 (10Nuria) 05Open→03Resolved [21:10:07] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review: Make Refine use JSONSchemas of event data to support Map types and proper types for integers vs decimals - https://phabricator.wikimedia.org/T215442 (10Nuria) [21:10:12] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Fundraising-Backlog, and 3 others: Fix EventLogging schemas that use array for items type - https://phabricator.wikimedia.org/T218617 (10Nuria) 05Open→03Resolved [21:12:30] ottomata: in what repo are the schemas served by that registry? [21:14:02] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), and 3 others: Schema Registry HTTP Service - https://phabricator.wikimedia.org/T219552 (10Nuria) [21:14:08] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Operations, and 5 others: Create schema[12]00[12] (schema.svc.{eqiad,codfw}.wmnet) - https://phabricator.wikimedia.org/T219556 (10Nuria) 05Open→03Resolved [21:14:22] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade analytics cluster to Cloudera CDH 5.16.1 - https://phabricator.wikimedia.org/T218343 (10Nuria) 05Open→03Resolved [21:14:35] 10Analytics, 10Analytics-Kanban: Check logging table after next sqoop for log_user type - https://phabricator.wikimedia.org/T214437 (10Nuria) 05Open→03Resolved [21:15:25] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Generate edit totals by country by month/year - https://phabricator.wikimedia.org/T215655 (10Nuria) [21:16:09] nuria mediawiki/event-schemas [21:16:42] ottomata: k, same place than before then [21:16:47] ya [21:27:13] ottomata: will schema registry be available at an internal dns ? [21:27:20] yaya [21:27:32] refine is using it for the new eventgate events [21:29:35] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Multimedia, and 2 others: Remove deprecated using schema.* syntax from MultimediaViewer - https://phabricator.wikimedia.org/T223284 (10Milimetric) @Cparle, @matthiasmullie, @MarkTraceur, @egardner, or anyone on Multimedia, I started working on mi... [21:31:19] 10Analytics, 10Product-Analytics, 10Growth-Team (Current Sprint): Update ServerSideAccountCreation schema whitelist - https://phabricator.wikimedia.org/T222101 (10nettrom_WMF) @JAllemandou and @Ottomata : you're both listed as reviewers on https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/508626/ A cha... [21:32:16] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 3 others: Modern Event Platform: Stream Intake Service: Implementation - https://phabricator.wikimedia.org/T206785 (10Nuria) [21:32:18] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): EventGate should extract event time from events and produce to kafka with timestamp - https://phabricator.wikimedia.org/T219513 (10Nuria) 05Open→03Resolved [21:34:30] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 3 others: Modern Event Platform: Stream Intake Service (EventGate): Implementation - https://phabricator.wikimedia.org/T206785 (10Nuria) [21:35:34] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Modern Event Platform: Stream Intake Service - https://phabricator.wikimedia.org/T201068 (10Nuria) [21:35:41] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 3 others: Modern Event Platform: Stream Intake Service (EventGate): Implementation - https://phabricator.wikimedia.org/T206785 (10Nuria) 05Open→03Resolved [21:36:30] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 3 others: Modern Event Platform: Stream Intake Service (EventGate): Implementation - https://phabricator.wikimedia.org/T206785 (10Nuria) [21:36:32] 10Analytics, 10EventBus, 10Release Pipeline, 10serviceops, 10Services (watching): Modern Event Platform: Stream Intake Service: Documentation - https://phabricator.wikimedia.org/T219332 (10Nuria) 05Open→03Resolved [21:38:37] 10Analytics, 10Product-Analytics, 10Growth-Team (Current Sprint): Update ServerSideAccountCreation schema whitelist - https://phabricator.wikimedia.org/T222101 (10Ottomata) Added a few others as reviewers who would probably know better than Joal or I! :) [21:39:23] byeeee [21:45:49] (03CR) 10Nuria: Update ServerSideAccountCreation whitelist (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508626 (https://phabricator.wikimedia.org/T222101) (owner: 10Nettrom) [22:14:37] 10Analytics: Can't log into Superset - https://phabricator.wikimedia.org/T223335 (10kaldari) Thanks Nuria! I can log in now, but when I do I get a bunch of 401 errors and nothing loads besides the header (regardless of which tab I click on). Not sure if that's just me or how it is for everyone. [22:18:07] 10Analytics: Can't log into Superset - https://phabricator.wikimedia.org/T223335 (10Nuria) While we have an outage for superset for EU time tomorrow, it should work now. Can you open an incognito tag and try to log in again? [22:56:21] 10Analytics: Can't log into Superset - https://phabricator.wikimedia.org/T223335 (10kaldari) Tried logging in again from an incognito tab, but still just get 401s for all the actual content. And if I try to go directly to any of the content URLs like https://superset.wikimedia.org/superset/recent_activity/154/?l... [22:58:09] 10Analytics: Can't log into Superset - https://phabricator.wikimedia.org/T223335 (10Nuria) Ok, you must have some permits missing, will check tomorrow if you are in the right ldap-nda authorized group. [23:01:14] 10Analytics, 10Analytics-Kanban, 10Wikimedia-Incident: attempt to backfill eventlogging data from eventlogging-client-side topic into per schema topics - https://phabricator.wikimedia.org/T220421 (10Nuria) The "rectangular" spikes on graph are the backfill processes: https://grafana.wikimedia.org/d/00000050...