[06:04:40] good morning! [06:04:58] so the failures for data quality seem to be related to app checks [06:04:59] https://yarn.wikimedia.org/cluster/app/application_1583418280867_273663 [06:21:55] 10Analytics, 10Event-Platform, 10Inuka-Team (Kanban), 10KaiOS-Wikipedia-app (MVP): Capture and send back client-side errors - https://phabricator.wikimedia.org/T248615 (10hueitan) Thanks @jlinehan @Ottomata When I tried to send event to the page `https://intake-logging.wikimedia.org`, it currently return... [07:51:52] * elukey brb! [08:07:16] 10Analytics: Support language variations on Wikistats - https://phabricator.wikimedia.org/T251091 (10fdans) [08:23:01] (03CR) 10Fdans: Replace numeral with numbro and fix bytes formatting (032 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/585725 (https://phabricator.wikimedia.org/T199386) (owner: 10Fdans) [10:06:19] * elukey out to get groceries [10:55:08] \o/ I'm back! My ISP was locally down this morning [10:57:08] joal: bonjour! [10:57:30] elukey: I guess we should sync on this weekend alarms shouldn't we? [10:58:22] joal: sure! I have to admit that I didn't spend a ton of time on them :( [10:58:42] No worries, let's just sync ::) [10:59:43] joal: so the only weird thing was the node manager ending up with GC problems [10:59:51] but it is kinda "known" sadly [11:00:22] elukey: feels like a spark issue? Or possibly another job [11:00:32] I'll investigate [11:05:35] could be yes, the patter was high pressure on the old gen pool of the jvm [11:06:03] this morning there were some workers misbehaving but they were OS/host issues [11:06:11] fixed with some variation of the hammer [11:06:27] and then there is the data quality error [11:16:10] elukey: quick look at the hadoop dashboard tells me there have been heavy activity yesterday [11:16:36] yeah I had the same impression [11:18:29] elukey: Interestingly, usage patterns show that the cluster is used more during weekends :) [11:19:06] at least in the past weeks [11:24:36] I blame addshore [11:24:37] :D [11:24:44] :) [11:24:59] going to be afk for a bit to have a quick lunch joal, is it ok? [11:25:08] please elukey :) [11:25:23] o/ [11:49:32] Hahha, that could be partly me! (But not this weekend) ;) [12:13:56] lexnasser: Hello - please ping me when you're in - There are a lot of oozie coordinators under your user with the same name running currently - I assume you don't kill your test-coordinators :S [12:26:28] ah joal not sure if you saw https://gerrit.wikimedia.org/r/#/c/operations/software/druid_exporter/+/592261/ [12:26:46] did it during the last couple of days, side coding project [12:27:18] in theory after this the exporter should be super configurable, and support any metric that we want via json file [12:27:42] \o/ elukey for the win :) [12:28:23] :) [12:56:25] hello elukey ! want to do eventgate tls this morning? [12:57:17] ottomata: sure! [12:57:33] ok! there is a 9am meeting with search that I am goingi to see if they need me for, they might not! [12:58:32] ack, anytime :) [12:58:33] ok elukey gimme 10 mins to emailize and then lets do it [12:59:20] luca it is so cool that people are sending druid exporter PRs!!! [13:00:39] ottomata: yes! I was a bit ashamed of my code and I re-did it all, but the original author of the PR helped a ton [13:00:46] :) [13:00:55] so we ended up with https://gerrit.wikimedia.org/r/#/c/operations/software/druid_exporter/+/592261/ [13:01:12] if it works fine etc.. I'll also send an email to druid users to get more feedback [13:02:31] !log superset 0.36.0 deployed to an-tool1005 [13:02:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:04:25] 10Analytics, 10LDAP-Access-Requests, 10Operations, 10Product-Analytics (Kanban): LDAP access to the wf group for Antonino Hemmer (superset, turnilo, hue) - https://phabricator.wikimedia.org/T251123 (10ahemmer) [13:05:42] 10Analytics, 10LDAP-Access-Requests, 10Operations, 10Product-Analytics (Kanban): LDAP access to the wf group for Antonino Hemmer (superset, turnilo, hue) - https://phabricator.wikimedia.org/T251123 (10RhinosF1) [13:06:22] 10Analytics, 10Event-Platform, 10Inuka-Team (Kanban), 10KaiOS-Wikipedia-app (MVP): Capture and send back client-side errors - https://phabricator.wikimedia.org/T248615 (10Ottomata) I guess the task doesn't say this explicitly, but the full URL is 'https://intake-logging.wikimedia.org/v1/events?hasty=true'... [13:08:50] 10Analytics, 10LDAP-Access-Requests, 10Operations, 10Product-Analytics (Kanban): LDAP access to the wf group for Antonino Hemmer (superset, turnilo, hue) - https://phabricator.wikimedia.org/T251123 (10Dzahn) [13:13:30] ok elukey bc? [13:13:54] gimme 2 [13:14:07] 10Analytics, 10LDAP-Access-Requests, 10Operations, 10Product-Analytics (Kanban): LDAP access to the wmf group for Antonino Hemmer (superset, turnilo, hue) - https://phabricator.wikimedia.org/T251123 (10Aklapper) [13:14:57] k [13:15:10] 10Analytics, 10LDAP-Access-Requests, 10Operations, 10Product-Analytics (Kanban): LDAP access to the wmf group for Antonino Hemmer (superset, turnilo, hue) - https://phabricator.wikimedia.org/T251123 (10Aklapper) (For future reference, see https://phabricator.wikimedia.org/project/profile/1564/ for required... [13:43:41] 10Analytics, 10Analytics-Kanban, 10Research: Proposed adjustment to wmf.wikidata_item_page_link to better handle page moves - https://phabricator.wikimedia.org/T249773 (10Isaac) @Nuria understood -- my proposed change doesn't modify any underlying tables, just the query to produce the item_page_link table. B... [13:52:29] 10Analytics, 10Analytics-Kanban: Create anaconda .deb package with stacked conda user envs - https://phabricator.wikimedia.org/T251006 (10Ottomata) Ping @MoritzMuehlenhoff too. More context in {T224658} and [[ https://docs.google.com/document/d/1r-oqMXViWvQCqsYz0qzezZBWpip8LvkvCGF6GivFB_8 | Newpyter Design Do... [13:54:44] (03CR) 10Gilles: Retain broad context for CPU benchmark (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/591307 (owner: 10Gilles) [13:58:09] 10Analytics, 10Patch-For-Review: Enable TLS encryption from Eventgate Analytics to Kafka Jumbo - https://phabricator.wikimedia.org/T250149 (10Ottomata) We enabled Kafka TLS for eventgate-analytics today. We will do eventgate-main tomorrow. [14:03:24] (03CR) 10Gilles: Retain broad context for CPU benchmark (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/591307 (owner: 10Gilles) [14:29:53] 10Analytics: Idea: Add 'top X bigger than Y' sanitization method to EL-to-Druid - https://phabricator.wikimedia.org/T251145 (10JAllemandou) [14:34:42] DOOHHHH joal we never merged the refine transform function change! [14:34:48] so it didn't go out last week [14:34:48] doh [14:35:02] ottomata: no way ? [14:35:10] https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/586447 [14:35:44] :( [14:36:07] rebasing... [14:42:41] (03PS11) 10Ottomata: Unify Refine transform functions and add user agent parser transform [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/586447 (https://phabricator.wikimedia.org/T238230) [14:43:01] joal: can you +1 or +2 https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/586447 [14:43:02] ? [14:43:09] yessir on it [14:43:55] (03CR) 10Joal: [C: 03+2] "Merging for this week deploy (missed last week)" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/586447 (https://phabricator.wikimedia.org/T238230) (owner: 10Ottomata) [14:44:01] ty [14:49:32] (03Merged) 10jenkins-bot: Unify Refine transform functions and add user agent parser transform [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/586447 (https://phabricator.wikimedia.org/T238230) (owner: 10Ottomata) [14:55:15] (03CR) 10Nuria: [C: 03+2] Correct typo in mobile metric areas i18n [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/588408 (https://phabricator.wikimedia.org/T247725) (owner: 10Fdans) [14:57:48] (03CR) 10Nuria: [C: 03+2] Retain broad context for CPU benchmark [analytics/refinery] - 10https://gerrit.wikimedia.org/r/591307 (owner: 10Gilles) [14:57:52] (03CR) 10Nuria: [V: 03+2 C: 03+2] Retain broad context for CPU benchmark [analytics/refinery] - 10https://gerrit.wikimedia.org/r/591307 (owner: 10Gilles) [14:59:44] ping ottomata , mforns [15:04:59] AHHHHH [15:05:00] SORRY [15:25:23] joal: oops, didn't know about killing the oozie coordinators, do I just do that in hue? [15:25:37] lexnasser: you can :) [15:25:51] joal: great, I'll do that! [15:25:54] lexnasser: no big deal probably, but better not to keep them :) [15:39:13] joal: I think I killed all of them successfully, can you verify? [15:39:55] sure lexnasser [15:40:20] all good lexnasser - Thanks a lot :) [16:01:10] 10Analytics, 10Analytics-Kanban, 10Pageviews-API, 10Patch-For-Review: Pageviews missing for titles with emojis since April 23, 2019 - https://phabricator.wikimedia.org/T245468 (10lexnasser) 05Open→03Resolved Did some final verification of pageviews for characters above 0xFFFF, and looks like everything... [16:04:19] Was looking at some different pages for the pageview API, and came across this: https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/fr.wiktionary/all-access/user/Ꚋ/daily/2020012700/2020042600 . Is there a reason for the several-day gaps between some data entries? [16:06:52] 10Analytics, 10LDAP-Access-Requests, 10Operations, 10Product-Analytics (Kanban): LDAP access to the wmf group for Antonino Hemmer (superset, turnilo, hue) - https://phabricator.wikimedia.org/T251123 (10crusnov) @Dzahn is this complete then? [16:07:43] 10Analytics, 10LDAP-Access-Requests, 10Operations, 10Product-Analytics (Kanban): LDAP access to the wmf group for Antonino Hemmer (superset, turnilo, hue) - https://phabricator.wikimedia.org/T251123 (10Dzahn) @crusnov No, it's not complete. There were just 2 tickets for the same thing. But it still needs d... [16:41:10] 10Analytics, 10LDAP-Access-Requests, 10Operations: LDAP access to the wmf group for Antonino Hemmer (superset, turnilo, hue) - https://phabricator.wikimedia.org/T251123 (10nshahquinn-wmf) [16:47:11] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Newpyter - SWAP Juypter Rewrite - https://phabricator.wikimedia.org/T224658 (10kzimmerman) @Ottomata sorry, didn't see that you replied! Looping in @SNowick_WMF; please reach out to her and she'll be available to bounce ideas off of & can help coordinate r... [16:52:23] 10Analytics, 10Analytics-Kanban, 10Privacy Engineering, 10Privacy, and 3 others: Identify pending analyses needing access to data older than 90 days - https://phabricator.wikimedia.org/T250857 (10LGoto) p:05Triage→03Medium [16:54:26] joal (other than our time and availability) do you see any issues with enabling bots code this week? [16:54:54] nuria: None - we have data starting beginning of year [16:55:19] nuria: something I have thought of: enabling bots for AQS will be a bit more complicated [16:55:45] joal: we would not reload though, just enable pageview_hourly calculations to take marker into account from thsi week, ya? [16:56:11] nuria: no reload needed, just a swap of data (as data is available in my table) [16:56:14] joal: just updating this: https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/578373/ [16:56:41] joal: we need to add 1 more column to cassandra right? [16:57:17] nuria: there are multiple things: we need to update the API for aggregated to accept the 'automated' value [16:57:56] 10Analytics: deploy bots changes to AQS - https://phabricator.wikimedia.org/T251169 (10Nuria) [16:58:13] joal: ah yes, also update schema in cassandra [16:58:16] nuria: And we need to load the data for automated in per-article - The bot column existed but has never been used - maybe we could use it (with plenty comments)? [16:59:04] nuria: and then, I'd really like to relaod this data historicall in cassandra to provide full 2020 with automated if you're ok [16:59:12] joal: ah ya, handy [16:59:16] joal: let's use it [16:59:30] joal: i do not think we should reload [16:59:35] ok nuria - preping a CR for that later tonight [17:00:23] joal: i think we should mark traffic going forward but not backwards , so as not to change metrics that might have already been communicated [17:00:31] ok ack [17:02:42] 10Analytics: deploy bots changes to AQS - https://phabricator.wikimedia.org/T251169 (10Nuria) [17:03:09] 10Analytics: Wikistats UI offers splits with agent_type; spider, user and automated - https://phabricator.wikimedia.org/T251170 (10Nuria) [17:52:16] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10mforns) I like that approach! [18:19:28] 10Analytics: Statistics on a CN banner - https://phabricator.wikimedia.org/T251177 (10Urbanecm) [18:31:30] hm mforns yt? i need a brain bounce about camus and refine and stream config [18:36:14] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Nuria) +1, seems very straight forward [18:41:25] (03PS1) 10Ottomata: RefineTarget shouldRefine should consider both table whitelist and blacklist [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/592739 (https://phabricator.wikimedia.org/T238230) [18:57:11] ottomata: was having a snak, back now, wanna bc? [18:57:21] *snack [18:58:31] mforns: gimme a few mins to finish a thought i just had an idea [18:58:39] k [19:02:20] ok mforns bc [19:02:24] ok omw [19:45:07] Hi mforns :) [19:45:14] would be still working? [19:57:16] 10Analytics, 10Event-Platform, 10Core Platform Team Workboards (External Code Reviews): EventBus should make better use of DI - https://phabricator.wikimedia.org/T204295 (10Pchelolo) 05Declined→03Open I think now that we almost have a new hook system it's time to do it and followup with extending test co... [20:12:44] 10Analytics, 10Operations, 10decommission, 10ops-eqiad: Decommission analytics100[1,2] - https://phabricator.wikimedia.org/T205507 (10Papaul) [20:20:55] 10Analytics, 10Performance-Team (Radar), 10Vue.js: Revise schema and performance dashboards for Vue.js search - https://phabricator.wikimedia.org/T250336 (10Gilles) [20:27:25] 10Analytics, 10Analytics-Kanban: Clean up superset 'Databases' - https://phabricator.wikimedia.org/T250089 (10Nuria) 05Open→03Resolved [20:28:21] 10Analytics, 10Analytics-Kanban: Fix wikidata_item_page_link job - https://phabricator.wikimedia.org/T248228 (10Nuria) 05Open→03Resolved [20:28:45] 10Analytics, 10Analytics-Kanban: Move systemd timer from an-coord1001 to an-launcher1001 - https://phabricator.wikimedia.org/T249593 (10Nuria) 05Open→03Resolved [20:29:11] 10Analytics, 10Analytics-Kanban: Drastically reduce build time for languages - https://phabricator.wikimedia.org/T246778 (10Nuria) 05Open→03Invalid [20:29:28] 10Analytics, 10Inuka-Team, 10Product-Analytics: Set up preview counting for KaiOS app - https://phabricator.wikimedia.org/T244548 (10Nuria) [20:29:30] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Virtual pageviews should set access_method per schema definition - https://phabricator.wikimedia.org/T246309 (10Nuria) 05Open→03Resolved [20:29:40] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Virtual pageviews should set access_method per schema definition - https://phabricator.wikimedia.org/T246309 (10Nuria) [20:29:59] 10Analytics, 10Analytics-Kanban, 10Better Use Of Data, 10Product-Infrastructure-Team-Backlog, and 2 others: EventLogging MEP Upgrade Phase 1 - https://phabricator.wikimedia.org/T244521 (10Nuria) 05Open→03Resolved [20:30:02] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), and 2 others: EventLogging MEP Upgrade - https://phabricator.wikimedia.org/T238544 (10Nuria) [20:31:11] 10Analytics, 10Analytics-Kanban, 10Analytics-SWAP, 10Product-Analytics: pip not accessible in new SWAP virtual environments - https://phabricator.wikimedia.org/T247752 (10Nuria) @nshahquinn-wmf was this issue resolved? [20:31:48] 10Analytics, 10Analytics-Kanban: Problem with Matomo page overlay - https://phabricator.wikimedia.org/T246046 (10Nuria) 05Open→03Invalid [20:32:09] 10Analytics, 10Analytics-Kanban: "Month over month" i18n tag being mixed with locales - https://phabricator.wikimedia.org/T246750 (10Nuria) 05Open→03Resolved [20:32:39] 10Analytics, 10Analytics-Kanban, 10Research: Proposed adjustment to wmf.wikidata_item_page_link to better handle page moves - https://phabricator.wikimedia.org/T249773 (10Milimetric) Ok... been thinking, @Isaac / @MGerlach, I may have a simpler and more accurate approach. We could use the stream of page mov... [20:34:41] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops: Move netflow to TLS encryption/authentication via librdkafka - https://phabricator.wikimedia.org/T248980 (10Nuria) 05Open→03Resolved [20:35:14] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Create and use new schema repositories - https://phabricator.wikimedia.org/T240985 (10Nuria) 05Open→03Resolved [20:35:16] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Modern Event Platform: Schema Registry: Implementation - https://phabricator.wikimedia.org/T206789 (10Nuria) [20:38:11] 10Analytics, 10Analytics-Kanban: Test aqs_hourly job from Airflow testing instance - https://phabricator.wikimedia.org/T248328 (10Nuria) Thanks for writing docs cc @mforns [20:38:19] 10Analytics, 10Analytics-Kanban: Test aqs_hourly job from Airflow testing instance - https://phabricator.wikimedia.org/T248328 (10Nuria) 05Open→03Resolved [20:38:22] 10Analytics, 10Patch-For-Review: Spike: POC of refine with airflow - https://phabricator.wikimedia.org/T241246 (10Nuria) [20:48:16] (03PS3) 10Nuria: Correcting examples in README for data quality jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/591956 (https://phabricator.wikimedia.org/T249759) [20:49:27] (03PS4) 10Nuria: Correcting examples in README for data quality jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/591956 (https://phabricator.wikimedia.org/T249759) [20:50:25] (03CR) 10Nuria: Correcting examples in README for data quality jobs (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/591956 (https://phabricator.wikimedia.org/T249759) (owner: 10Nuria) [21:00:37] (03CR) 10Milimetric: [C: 03+2] Handle punctuation chars in paths for mediarequests per file [analytics/aqs] - 10https://gerrit.wikimedia.org/r/588396 (https://phabricator.wikimedia.org/T244373) (owner: 10Fdans) [21:01:30] (03Merged) 10jenkins-bot: Handle punctuation chars in paths for mediarequests per file [analytics/aqs] - 10https://gerrit.wikimedia.org/r/588396 (https://phabricator.wikimedia.org/T244373) (owner: 10Fdans) [21:02:02] (03CR) 10Milimetric: [C: 03+2] "love the tests and everything, and just for the record I resisted commenting that you should use a ternary operator in your encode functio" [analytics/aqs] - 10https://gerrit.wikimedia.org/r/588396 (https://phabricator.wikimedia.org/T244373) (owner: 10Fdans) [21:51:26] 10Analytics, 10Analytics-Kanban, 10Pageviews-API, 10Patch-For-Review: Pageviews missing for titles with emojis since April 23, 2019 - https://phabricator.wikimedia.org/T245468 (10MusikAnimal) Thank you, all! I just wanted to further compliment T245468#6046245 -- that was a fascinating read!