[00:38:45] 10Analytics, 10Analytics-Wikistats: Wikistats Bug - inaccurate lists for top editors - https://phabricator.wikimedia.org/T258233 (10Quiddity) [00:56:01] PROBLEM - Check the last execution of monitor_refine_eventlogging_analytics_failure_flags on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_eventlogging_analytics_failure_flags https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [00:56:39] PROBLEM - Check the last execution of monitor_refine_eventlogging_legacy_failure_flags on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_eventlogging_legacy_failure_flags https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [06:43:16] 10Analytics-Clusters, 10Discovery-Search, 10Operations, 10vm-requests: VM request for Analytics -> Elastic Search ML models update - https://phabricator.wikimedia.org/T258189 (10elukey) [06:50:56] 10Analytics-Clusters, 10Discovery-Search, 10Operations, 10vm-requests: VM request for Analytics -> Elastic Search ML models update - https://phabricator.wikimedia.org/T258189 (10elukey) @RKemper I am following https://wikitech.wikimedia.org/wiki/Ganeti#Create_a_VM, I'll list all the steps/details etc.. in... [07:00:42] 10Analytics-Clusters, 10Discovery-Search, 10Operations, 10vm-requests: VM request for Analytics -> Elastic Search ML models update - https://phabricator.wikimedia.org/T258189 (10elukey) ` elukey@ganeti1011:~$ sudo gnt-group list Group Nodes Instances AllocPolicy NDParams row_A 4 39 preferred o... [08:05:24] 10Analytics-Clusters, 10Discovery-Search, 10Operations, 10vm-requests, 10Patch-For-Review: VM request for Analytics -> Elastic Search ML models update - https://phabricator.wikimedia.org/T258189 (10elukey) ` elukey@cumin1001:~$ sudo cookbook sre.ganeti.makevm eqiad_D search-loader1001.eqiad.wmnet --vcpus... [08:33:27] 10Analytics-Clusters, 10Discovery-Search, 10Operations, 10vm-requests, 10Patch-For-Review: VM request for Analytics -> Elastic Search ML models update - https://phabricator.wikimedia.org/T258189 (10elukey) ` elukey@cumin1001:~$ sudo cookbook sre.ganeti.makevm codfw_D search-loader2001.codfw.wmnet --vcpus... [09:22:47] 10Analytics-Clusters, 10Discovery-Search, 10Operations, 10vm-requests, 10Patch-For-Review: VM request for Analytics -> Elastic Search ML models update - https://phabricator.wikimedia.org/T258189 (10elukey) VMs ready! [09:25:59] 10Analytics-Clusters, 10Discovery-Search, 10Operations, 10vm-requests, 10Patch-For-Review: VM request for Analytics -> Elastic Search ML models update - https://phabricator.wikimedia.org/T258189 (10elukey) 05Open→03Resolved a:03elukey [09:28:30] 10Analytics-Clusters, 10Discovery: Move - https://phabricator.wikimedia.org/T258245 (10elukey) [09:28:54] 10Analytics-Clusters, 10Discovery: Move mjolnir kafka daemon from ES to search-loader VMs - https://phabricator.wikimedia.org/T258245 (10elukey) [10:29:52] * elukey lunch! [11:27:58] 10Analytics, 10Analytics-Wikistats: Wikistats Bug - inaccurate lists for top editors - https://phabricator.wikimedia.org/T258233 (10JAllemandou) My understanding of the https://en.wikipedia.org/wiki/Wikipedia:List_of_Wikipedians_by_number_of_edits#1%E2%80%931000 page is that it provides top-editors from the be... [11:47:37] hey teammm [12:09:36] o/ [12:10:06] \o [12:25:28] (03CR) 10Joal: "A bunch of comment - sorry to have taken so long" (0316 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [12:27:26] "a couple of nits" [12:27:28] :D [12:27:46] elukey: you noticed I didn't dare [12:27:48] :D [12:34:12] !log deprecate pivot.wikimedia.org (to ease CAS work) [12:34:13] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:34:20] bye bye pivot [12:34:26] Ciaooooo :) [12:43:18] (03CR) 10Joal: [C: 03+1] "Great change - a lot cleaner IMO (minimal nits if you fell like it, but ready as is) :)" (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/613251 (https://phabricator.wikimedia.org/T255818) (owner: 10Ottomata) [12:44:46] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Patch-For-Review: Refine drops $schema field values - https://phabricator.wikimedia.org/T255818 (10JAllemandou) Knowing that we're moving EL to MEP, I think we're ok with the current situation :) [12:47:18] turnilo going to CAS on monday https://gerrit.wikimedia.org/r/c/operations/puppet/+/613626 [12:47:21] * elukey dances [12:47:32] \o/ [12:47:43] * joal bows to elukey for efficiency [12:49:01] joal: we need so say thank you to Moritz and John! [12:50:00] Thank ou moritzm and jbond42 :) [13:01:25] 10Analytics-Radar, 10Product-Analytics, 10Wikipedia-Android-App-Backlog, 10Wikipedia-iOS-App-Backlog, 10Epic: [EPIC] Count unique iOS & Android users precisely and in a privacy conscious manner that does not require opt in to send data - https://phabricator.wikimedia.org/T202664 (10JAllemandou) @nuria ha... [13:55:02] 10Analytics-Radar, 10Performance-Team: Invalid navigation timing events - https://phabricator.wikimedia.org/T254606 (10Milimetric) FYI: A ResourceTiming event just failed to refine encodedBodySize = 18446744073709552000. Chrome 84. I do think we need some general way of handling these errors, it seems these... [14:03:47] (03CR) 10Milimetric: Fix unique-devices per-project-family cassandra loading (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/612905 (https://phabricator.wikimedia.org/T258064) (owner: 10Joal) [14:08:39] (03CR) 10Milimetric: "Hi Connie, we're trying to get to this, but have to schedule it after we wrap up goals work for Q4 from last year. Adding myself so I don" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/607361 (https://phabricator.wikimedia.org/T256050) (owner: 10Conniecc1) [14:14:39] 10Analytics: de-duplicate archive records matching revision records in mediawiki_history - https://phabricator.wikimedia.org/T152546 (10Milimetric) This task took a wild ride through our board, not sure what happened and why I deprioritized it, but it seems like something to look into to ensure the quality of th... [14:17:51] 10Analytics-General-or-Unknown, 10Analytics-Radar, 10Product-Analytics, 10Wikimedia-Interwiki-links, and 2 others: there should be a comparison of clicks count on interlanguage links on different platforms - https://phabricator.wikimedia.org/T78351 (10Milimetric) 05Open→03Declined [14:19:08] 10Analytics: Easter Egg: wikistats classic style on wikistats 2.0 - https://phabricator.wikimedia.org/T177408 (10Milimetric) a:03Milimetric I definitely want to do this, sorry for being so late with it. [14:21:29] (03CR) 10Ottomata: Refine - Don't merge Hive schema by default when reading input data (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/613251 (https://phabricator.wikimedia.org/T255818) (owner: 10Ottomata) [14:21:45] (03PS6) 10Ottomata: Refine - Don't merge Hive schema by default when reading input data [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/613251 (https://phabricator.wikimedia.org/T255818) [14:22:20] (03CR) 10Ottomata: [C: 03+2] Overloaded methods to make working with default Refine related classes easier [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/607788 (owner: 10Ottomata) [14:22:44] (03CR) 10Ottomata: [C: 03+2] Overloaded methods to make working with default Refine related classes easier (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/607788 (owner: 10Ottomata) [14:27:00] 10Analytics-General-or-Unknown, 10Analytics-Radar, 10Product-Analytics, 10Wikimedia-Interwiki-links, and 2 others: there should be a comparison of clicks count on interlanguage links on different platforms - https://phabricator.wikimedia.org/T78351 (10Amire80) 05Declined→03Open It's necessary, even if... [14:27:08] 10Analytics-Clusters: Review an-coord1001's usage and failover plans - https://phabricator.wikimedia.org/T257412 (10elukey) I was able to add TLS support to the analytics-meta test instance on analytics1030, and I tested connections with/without TLS, all good. Hive and Oozie seem to work fine. Next step is to re... [14:27:29] (03Merged) 10jenkins-bot: Overloaded methods to make working with default Refine related classes easier [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/607788 (owner: 10Ottomata) [14:35:59] 10Analytics: Larger system text causes problems on mobile - https://phabricator.wikimedia.org/T258273 (10Milimetric) [14:38:35] 10Analytics-General-or-Unknown, 10Analytics-Radar, 10Product-Analytics, 10Wikimedia-Interwiki-links, and 2 others: there should be a comparison of clicks count on interlanguage links on different platforms - https://phabricator.wikimedia.org/T78351 (10Milimetric) Ok, right now this is not tagged Analytics... [14:52:32] (03CR) 10Joal: Fix unique-devices per-project-family cassandra loading (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/612905 (https://phabricator.wikimedia.org/T258064) (owner: 10Joal) [14:54:47] (03CR) 10Joal: [C: 03+1] "LGTM thanks for the changes and answers Andrew :)" (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/613251 (https://phabricator.wikimedia.org/T255818) (owner: 10Ottomata) [15:38:09] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10elukey) Today I added the user `repl` (with a custom password saved in the private repo) to mariadb on matomo1002 and an-coord1001, th... [15:48:46] why using TLS certificates in Java is so annoying? [15:49:01] * elukey rants on a Friday afternoon [15:51:38] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10elukey) Ok so on Monday I'll restart mariadb on an-coord1001 and matomo1002 to pick up TLS changes. With the repl user created, in the... [15:51:41] elukey: I'd generalize the question: Why is java so annoying! [15:52:04] sigh [15:52:35] on monday I should be able to enable TLS support for meta and matomo dbs, and hopefully enable replication to db1108 [15:53:05] 10Analytics-Radar, 10Better Use Of Data, 10Product-Analytics, 10Epic, and 2 others: Session Length Metric. Web implementation - https://phabricator.wikimedia.org/T248987 (10jlinehan) >>! In T248987#6309400, @Nuria wrote: > @jlinehan; some suggestions > > 1) let's start a new patch to avoid unnecessary con... [15:53:08] the next nice step would be to figure out how to force all daemons using an-coord1001 to use TLS if needed [15:53:13] like druid, superset, etc.. [15:53:46] that requires to point every daemon to the puppet CA certificate (to trust TLS certs properly) [15:53:48] elukey: you make things happen extremly fast! [15:53:49] BUT [15:53:55] for java based things, I need a truststore [15:54:07] with a password, even if the cert is public [15:54:09] sigh [15:54:14] :( [15:54:30] joal: I am sad that this work is taking so long, I would have loved to finish it last quarter :( [15:55:09] I also want to have a good procedure to failover an-coord1001, I am pretty sure it will happen when we'll need the most (like when sqoop starts :D) [15:55:13] hm - so long is not exactly the definition I'd give to make so moving pieces try to collaborate :) [15:55:27] of course elukey - that makes sense [16:05:22] (03CR) 10Ottomata: [C: 03+2] Refine - Don't merge Hive schema by default when reading input data [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/613251 (https://phabricator.wikimedia.org/T255818) (owner: 10Ottomata) [16:07:05] RECOVERY - Check the last execution of monitor_refine_eventlogging_analytics_failure_flags on an-launcher1002 is OK: OK: Status of the systemd unit monitor_refine_eventlogging_analytics_failure_flags https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [16:11:51] RECOVERY - Check the last execution of monitor_refine_eventlogging_legacy_failure_flags on an-launcher1002 is OK: OK: Status of the systemd unit monitor_refine_eventlogging_legacy_failure_flags https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [16:14:00] (03CR) 10Nuria: [C: 03+2] Fix mediawiki-history skewed join bug [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/609465 (https://phabricator.wikimedia.org/T255548) (owner: 10Joal) [16:15:50] * elukey off! [16:15:54] have a good weekend folks :) [16:19:11] (03Merged) 10jenkins-bot: Fix mediawiki-history skewed join bug [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/609465 (https://phabricator.wikimedia.org/T255548) (owner: 10Joal) [16:34:38] (03PS1) 10Joal: Update mediawiki-history-denormalize job jar version [analytics/refinery] - 10https://gerrit.wikimedia.org/r/613655 (https://phabricator.wikimedia.org/T255548) [16:45:55] Very interesting and fun: http://lacker.io/ai/2020/07/06/giving-gpt-3-a-turing-test.html [16:53:47] Gone for tonight - have a good weekend team [17:17:27] 10Analytics-Radar, 10Better Use Of Data, 10Product-Analytics, 10Epic, and 2 others: Session Length Metric. Web implementation - https://phabricator.wikimedia.org/T248987 (10Nuria) >This is helpful, but is probably better to post as a comment on the patch CR, no? Indeed! I added comment to gerrit patch. [17:22:41] 10Analytics-Radar, 10Product-Analytics, 10Wikipedia-Android-App-Backlog, 10Wikipedia-iOS-App-Backlog, 10Epic: [EPIC] Count unique iOS & Android users precisely and in a privacy conscious manner that does not require opt in to send data - https://phabricator.wikimedia.org/T202664 (10Nuria) >but this is no... [17:26:38] 10Analytics: explicitily exclude mobile-app traffic from unique devices per domain computation - https://phabricator.wikimedia.org/T258288 (10Nuria) [17:27:06] joal: see if this makes sense: https://phabricator.wikimedia.org/T258288 [17:54:28] 10Analytics, 10Product-Analytics, 10Structured Data Engineering, 10Structured-Data-Backlog (Current Work): Instrument MediaSearch results page - https://phabricator.wikimedia.org/T258183 (10nettrom_WMF) [17:55:45] milimetric or nuria: publicly reported geoeditors would not include anonymous editors right? [18:00:42] I remember we discussing this, but the code includes them, and actually aggregates them with the registered editors, which is not accurate, because an editor can edit as both registered and anonymous and should be counted still as just one. [18:07:10] Also, in the comments on top of the query, it says we'll report central projects like commons, but the code does not reflect that [18:07:35] is that sth we left for later, or should we include them from the beginning? [18:31:07] mforns: let me dig out teh specifications of this data [18:31:09] *the [18:31:37] nuria: here's the doc: https://docs.google.com/document/d/1D-v2vTtFt94xZ9HVSky7BKpzF4H2LZ2yC5H9KR_rgKI/edit [18:31:58] but doesn't say anything about anonymous [18:32:22] although that one is more about the api.. [18:33:29] mforns: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Edits/Geoeditors/Public [18:33:44] mforns: so, only wikipedias and [18:35:47] nuria: in here https://phabricator.wikimedia.org/T131280 there's also some discussion about anonymous, but I can not see a final decision [18:36:55] mforns: ok, remembering now [18:37:03] mforns: we report both: https://phabricator.wikimedia.org/T131280#5310317 [18:37:34] mforns: you are right that it might double count someone but that is not a concern, probably our public docs need to list that more explicitily [18:39:31] so we assume there will be very few editors that edit as both anonymous and registered at the same month, so we would count them twice, but that is expected, correct? [18:39:44] ok, will make that clear in the docs. [18:40:00] thankkkkss [19:15:02] mforns: k, thank you