[00:03:53] 10Analytics, 10Analytics-Wikistats: stats.wikimedia.org not available in Finnish - https://phabricator.wikimedia.org/T266974 (10Reedy) https://github.com/wikimedia/analytics-wikistats2/tree/master/src/i18n shows 37 json files, and Finnish is one of them, so it's presumably over the export threshold... But only... [03:14:35] 10Analytics, 10Analytics-Wikistats: Trends for editor types, and new editors in particular (in Wikistats 2.0) - https://phabricator.wikimedia.org/T186791 (10Tgr) This is real important and seems fairly easy to achieve. The graphs as they are now have limited usefulness for answering real-world questions (which... [03:16:06] 10Analytics, 10Analytics-Wikistats: Trends for editor types, and new editors in particular (in Wikistats 2.0) - https://phabricator.wikimedia.org/T186791 (10Tgr) (Granted, it could be done in a Toolforge tool just as easily, the API already provides all the necessary data.) [05:30:47] Hey, a weird question, do we store the port of IPs requests are coming from? We need it for https://phabricator.wikimedia.org/T265692 (security) [05:31:19] webrequest didn't have it, maybe wmf_raw.webrequest have it? [05:31:29] https://phabricator.wikimedia.org/T181368 <- This is public version of it [06:53:35] Hey Amir1, if you mean source client port, I don't think so [07:00:09] elukey: thanks. Do you think it's doable to add it? [07:00:24] It's useful for reporting extreme abuse to ISPs [07:00:29] (since they use NAT) [07:02:06] Amir1: if varnish exposes the value yes it is doable, but I'd need to talk with my team about it first.. I'll send an email to internal@ with you and the task in Cc [07:08:08] Amir1: done :) [07:08:17] hopefully you'll have news during the day [07:08:25] elukey: thanks. I assume it's not super straightforward given the scale [07:09:49] Amir1: in theory, if a value reaches the Varnish frontend level, it is a matter of filing a patch for puppet to make varnishkafka include it. After that, it will be visible in Kafka, and it should then be a matter of making it visible also to hive tables (again, should be an alter table etc..) [07:10:24] some work but not impossible, we already handled requests like this one.. the problem is always if the value belongs to the dataset, etc.. [07:10:39] yeah I understand [07:10:49] (also it is an integer added to every request's kafka json, not a ton of bytes more, it is tolerable) [07:11:51] Awesome! [07:11:55] Amir1: an example of prev request is https://phabricator.wikimedia.org/T170606 [07:12:16] and a more recent one is the TLS specs, we added them for the traffic team [07:13:20] Cool <3 [07:14:55] :) [07:19:34] RECOVERY - Check the last execution of performance-asoranking on stat1007 is OK: OK: Status of the systemd unit performance-asoranking https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [07:31:17] 10Analytics-Clusters, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: replace onboard NIC in kafka-jumbo100[1-6] - https://phabricator.wikimedia.org/T236327 (10elukey) `firmware-bnx2x` installed manually on kafka-jumbo1006, we can retry the switch anytime to see if it works. [07:44:01] 10Analytics-Radar, 10Revision-Slider, 10Two-Column-Edit-Conflict-Merge, 10WMDE-Technical-Wishes-Team, 10Documentation: Where should keys used for stats in an extension be documented? - https://phabricator.wikimedia.org/T185111 (10thiemowmde) [07:46:08] 10Analytics-Radar, 10Growth-Team, 10MediaWiki-Page-editing, 10Research ideas, and 3 others: statistics about edit conflicts according to page type - https://phabricator.wikimedia.org/T139019 (10thiemowmde) [07:58:46] Good morning [08:02:48] bonjour [08:03:27] How are you elukey ?g [08:07:07] good! and you? [08:08:04] Mwarf - No real bad news, but feeling overwhelemed by my environment today [08:08:24] yeah I can imagine, not great news from mine as well [08:08:45] yeah I have read that numbers for Italy are bad as well :( [08:09:15] we'll see later on in the day, some new measures should be announced IIUC [08:09:36] keep me posted elukey so that I can send some positive thoughts [08:11:05] :) [08:43:41] two hosts of the backup cluster have difficulties to reimage, I am working on them but it doesn't look good [08:44:09] we may need to use the new hosts (not yet racked) for the backup cluster [08:54:11] ack elukey - Dont sweat too much on it, it'll wait for new hosts [09:02:14] 10Analytics-Clusters, 10Patch-For-Review: Create a temporary hadoop backup cluster - https://phabricator.wikimedia.org/T260411 (10elukey) analytics1044 is in an endless PXE install loop, and it is not due to NIC-before-hdd (already checked), but because for some reason /dev/sda is not one the raid1 in the flex... [09:14:31] (03PS1) 10Joal: Add caching to maxmind readers in core package [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/638021 [09:20:26] (03CR) 10jerkins-bot: [V: 04-1] Add caching to maxmind readers in core package [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/638021 (owner: 10Joal) [09:42:17] joal: just to double check - the new aqs nodes will have 8x1.92TB SSDs, meanwhile now we have the same amount but 1.5T [09:42:46] ram and cores looks also very good [09:42:52] elukey: this means ~2Tb addition per host, right? [09:43:49] joal: a little bit less but if we round it up yes [09:43:58] sounds great [09:44:07] perfect [09:46:45] all right moved the task forward with dcops, hopefully we can get the new hw this quarter [09:47:06] great [09:47:40] * elukey bbiab [10:12:55] very interesting view on stat1005 [10:12:56] https://grafana.wikimedia.org/d/000000377/host-overview?viewPanel=4&orgId=1&refresh=5m&var-server=stat1005&var-datasource=thanos&var-cluster=analytics&from=now-30d&to=now [10:13:13] after the upgrade to 512G of ram there was only a big peak of usage [10:14:04] and a ton of cached memory was requested [10:14:23] yup I was going to add that - most usage on cached [10:18:19] 10Analytics-Clusters, 10Cassandra: Set up a testing environment for the AQS Cassandra 3 migration - https://phabricator.wikimedia.org/T257572 (10elukey) A lot of things changed, we are getting the 6 new nodes for AQS (due to hw refresh) sooner so our idea is the following: - install Debian buster on the new n... [10:49:11] going to take my lunch break earlier today because of errands, ttl! [10:49:40] (03PS2) 10Joal: Add caching to maxmind readers in core package [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/638021 [10:57:59] (03PS1) 10Joal: Upgrade maven configuration and plugins [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/638040 [11:09:12] Just noting that stat1007 will not spawn a jupyter kernel for me. I can workaround by using 1005, where it starts normally. I don't think I have any custom kernels configured on either machine. This doesn't block my work. [11:41:29] (03CR) 10Joal: "Putting all non-ops as this patch updates config" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/638040 (owner: 10Joal) [11:49:09] 10Analytics, 10Analytics-Wikistats: stats.wikimedia.org not available in Finnish - https://phabricator.wikimedia.org/T266974 (10Jnovikov) Ergo Finnish could be added in the near future? [12:05:06] (03PS1) 10Joal: Improve webrequest-refine query shuffle stage [analytics/refinery] - 10https://gerrit.wikimedia.org/r/638086 [12:07:35] 10Analytics, 10Analytics-Kanban: Improve webrequest-refine shuffle-sort - https://phabricator.wikimedia.org/T267008 (10JAllemandou) [12:07:47] 10Analytics, 10Analytics-Kanban: Improve webrequest-refine shuffle-sort - https://phabricator.wikimedia.org/T267008 (10JAllemandou) a:03JAllemandou [12:08:11] (03PS2) 10Joal: Improve webrequest-refine query shuffle stage [analytics/refinery] - 10https://gerrit.wikimedia.org/r/638086 (https://phabricator.wikimedia.org/T267008) [12:08:23] elukey: I think you're gonna like that --^ :) [12:10:51] 10Analytics, 10Analytics-Kanban: Add caching for maxmind functions used on cluster - https://phabricator.wikimedia.org/T267009 (10JAllemandou) [12:11:05] 10Analytics, 10Analytics-Kanban: Add caching for maxmind functions used on cluster - https://phabricator.wikimedia.org/T267009 (10JAllemandou) a:03JAllemandou [12:11:50] (03PS3) 10Joal: Add caching to maxmind readers in core package [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/638021 [12:12:44] (03PS4) 10Joal: Add caching to maxmind readers in core package [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/638021 (https://phabricator.wikimedia.org/T267009) [12:26:52] joal: indeed! nice! [12:27:05] awight: thanks a lot for the ping, lemme check [12:40:25] !log forced re-creation of base jupyterhub venvs on stat1007 [12:40:27] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:42:58] awight: if you have time, can you try https://wikitech.wikimedia.org/wiki/Analytics/Systems/Jupyter#Resetting_user_virtualenvs ? [13:09:37] morning team! [13:11:32] hola [13:11:39] Hi fdans [13:21:24] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10I18n: [[Wikimedia:Wikistats-metrics-top-mediarequests-name/jam]] translation issue - https://phabricator.wikimedia.org/T266669 (10fdans) Hi @Dentonius ! Thank you so much for contributing to the internationalization of Wikistats. I can't see a trans... [13:22:00] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Improve webrequest-refine shuffle-sort - https://phabricator.wikimedia.org/T267008 (10Hermann) [13:22:02] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Add caching for maxmind functions used on cluster - https://phabricator.wikimedia.org/T267009 (10Hermann) [13:23:02] elukey: when you have a minute can you take a look at this CR? https://gerrit.wikimedia.org/r/c/operations/puppet/+/634650 [13:23:03] 10Analytics, 10Analytics-Wikistats: stats.wikimedia.org not available in Finnish - https://phabricator.wikimedia.org/T266974 (10Hermann) [13:25:22] fdans: do you want me to merge? [13:26:25] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Add caching for maxmind functions used on cluster - https://phabricator.wikimedia.org/T267009 (10DannyS712) [13:26:32] elukey: if you think the pp file in there is correct, yes please :) [13:26:36] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Improve webrequest-refine shuffle-sort - https://phabricator.wikimedia.org/T267008 (10DannyS712) [13:27:31] 10Analytics, 10Analytics-Wikistats: stats.wikimedia.org not available in Finnish - https://phabricator.wikimedia.org/T266974 (10DannyS712) [13:31:24] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Improve webrequest-refine shuffle-sort - https://phabricator.wikimedia.org/T267008 (10elukey) [13:31:26] 10Analytics-Clusters: Review recurrent Hadoop worker disk saturation events - https://phabricator.wikimedia.org/T265487 (10elukey) [13:32:49] fdans: merged but I found a problem, namely /srv/dumps/xmldatadumps/public/other/pageview-complete [13:32:55] on disk it is /srv/dumps/xmldatadumps/public/other/pageview_complete [13:33:43] so puppet fails [13:34:21] 10Analytics, 10Analytics-Wikistats: stats.wikimedia.org not available in Finnish - https://phabricator.wikimedia.org/T266974 (10fdans) Hi @Jnovikov, thank you for your interest in Wikistats i18n. At 31% completion, Finnish is currently far below our soft threshold to include it in production. We feel that enab... [13:34:41] elukey: should I revert and resend or hotfix? [13:35:40] fdans: I created https://gerrit.wikimedia.org/r/638102, lemme know if it makes sense [13:36:06] elukey: damn luca, you're too quick :) [13:36:13] what creates the dir pageview_complete ? [13:36:45] ah ok a ton of data already there [13:36:46] okok [13:37:58] elukey: yea it's an rsync in the stats.pp file within dumps (IIRC) [13:38:25] fdans: now puppet works [13:39:05] fdans: can you verify if all works? [13:40:54] !log roll restart zookeeper ok an-conf* to pick up new openjdk upgrades [13:40:56] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:44:30] elukey: submitting correction patch, sorry [13:54:01] elukey: this should do it https://gerrit.wikimedia.org/r/c/operations/puppet/+/638105 [13:59:35] fdans: deployed, can you re-check? [14:02:52] elukey: is it possible for you to remove manually the rule that links index.html to pageview-complete_index.html ? [14:03:12] I think I should have done an ensure => absent before changing the route [14:03:26] what do you mean with the rule that links? [14:04:17] I can remove index.html [14:05:28] 10Analytics, 10Analytics-Wikistats: stats.wikimedia.org not available in Finnish - https://phabricator.wikimedia.org/T266974 (10Jnovikov) @fdans Thanks for information. I will translate Finnish to near 100 % in the near future 😄 [14:05:41] fdans: or is it the link in https://dumps.wikimedia.org/other/analytics/ ? [14:05:52] it may be cached, I tried now and I see readme.html [14:06:14] elukey: this url https://dumps.wikimedia.org/other/pageview_complete/ [14:06:19] should point directly to the files [14:06:22] not the readme [14:06:28] the readme should be in https://dumps.wikimedia.org/other/pageview_complete/readme.html [14:08:18] fdans: should work now [14:08:24] (I deleted manually index.html) [14:09:22] elukey: yessss all working as expected now [14:09:26] thank you so mcu [14:09:28] much [14:10:36] :) [14:11:40] I am currently a little blocked on the tests for the analytics-hive.eqiad.wmnet DNS record, since webrequest_load doesn't work in hadoop test.. I'd need refinery source to be deployed, do we have the plan to do it tomorrow? [14:17:17] elukey: tomorrow is going to be hard because it's dan's ops week and it's a WMF holiday [14:17:23] but maybe we could do it today milimetric ? [14:19:53] 10Analytics, 10Analytics-Wikistats: Add mutually exclusive (radio button-style) dimension values - https://phabricator.wikimedia.org/T267026 (10fdans) [14:20:36] that would be massive, even just building source [14:32:29] salve elukey, do you have a few minutes? [14:35:57] cdanis: in a meeting :( [14:36:04] ok np, it can wait! [14:51:43] (03CR) 10Ottomata: [C: 03+1] Upgrade maven configuration and plugins [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/638040 (owner: 10Joal) [15:11:28] cdanis: I am free now! [15:14:22] elukey: o/ [15:14:26] will https://gerrit.wikimedia.org/r/c/operations/puppet/+/638116/1/modules/profile/manifests/analytics/refinery/job/canary_events.pp do what I want? [15:14:28] i think so, right? [15:14:38] (not super familiar with systemd timer interval syntax) [15:15:24] checking [15:17:18] ottomata: I am not sure if it changes a lot but I have always seen a 00, like *-*-* *:00/15:00 [15:17:34] there is also another occurrence of this in puppet [15:18:27] oh ok [15:18:54] ah just the extra 0? i like that better too [15:18:54] ok [15:19:51] t [15:19:51] ty [15:29:01] np :) [15:38:40] elukey: sorry, just got out of a meeting myself [15:39:04] so I wanted to ask you and ottomata a few bigger-picture things about eventgate, and druid, and logstash [15:39:38] right now with Network Error reports we're doing a 5% sampling rate, sending to eventgate-logging-external, and recording in logstash [15:40:16] where I still haven't successfully built a dashboard I like, because honestly working with kibana is quite frustrating, and there's a few things that need to be fixed there as well (eg T266906) [15:40:16] T266906: update logging ES's template index to type the 'age' field as an integer - https://phabricator.wikimedia.org/T266906 [15:40:43] so I was thinking about other options, like if the realtime Druid thing that was built for wmf_netflow is reproducible [15:50:55] cdanis: so the realtime druid option is available, but we'd need to reason about the data to ship to druid, just to make sure that we'll not have a dimension with a huge cardinality etc... [15:51:41] the realtime netflow data is composed by two parts - a batch ingestion, and the "realtime" one from kafka (the former overrides the latter periodically) [15:52:51] this would be another ad-hoc setup though, I am not sure if we have something pre-baked for eventgate to use, let's way for ottomata's opinion :) [15:52:58] elukey: any idea how hive-site.xml gets into __spark_conf__.zip that is created and used when spark --deploy-mode=cluster? I'm having trouble with my local bigtop install and i can't see how prod (not bigtop, but should be similar) achieves similar [15:53:33] elukey: ack understood, we can hopefully get some estimates of the cardinality based on what's in logstash so far (or start shipping to Hive as well) [15:54:13] cdanis: elsaticsearch has a cardinality query :) not sure if it applies here [15:54:36] hopefully :) [16:05:50] ebernhardson: one way IIRC is to use --files to upload hive-site.xml to HDFS, and then reference it locally (rather than include it in the zip) [16:07:09] we also add it to hdfs, see profile::hive::site_hdfs [16:07:17] elukey: right, i could modify all my jobs to ship it explicitly, but somehow in prod it's being shipped auto-magically. Maybe it's comeing from hdfs somehow, will check [16:08:21] i tried adjusting spark defaults spark.yarn.dist.files to include it, and that works but then it fails if you use --files (as --files seems to overwrite spark.yarn.dist.files) [16:17:06] (03PS1) 10Milimetric: Update changelog.md for v0.0.138 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/638129 [16:17:25] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Update changelog.md for v0.0.138 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/638129 (owner: 10Milimetric) [16:21:48] ebernhardson: we do link hive-site.xml into /etc/spark/conf [16:22:33] interesting though, it has been a while since I thought about it, fell off my brain's LRU :D [16:22:50] yup, i've symlinked it in there too. I was slightly suspicious of symlink (the relevant spark code checks .isFile, which i'm not sure a symlink is), but in prod it's also a symlink so "should" work [16:23:14] i suppose i should just rebuild this so i can inject a debugger and step through it making this archive...it's juts such a pain [16:27:10] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Data: mw.user.generateRandomSessionId should return a UUID - https://phabricator.wikimedia.org/T266813 (10sdkim) p:05Triage→03Medium [16:28:03] (03PS3) 10Joal: Improve webrequest-refine query shuffle stage [analytics/refinery] - 10https://gerrit.wikimedia.org/r/638086 (https://phabricator.wikimedia.org/T267008) [16:28:14] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Data: mw.user.generateRandomSessionId should return a UUID - https://phabricator.wikimedia.org/T266813 (10sdkim) a:03Mholloway [16:28:45] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Data: mw.user.generateRandomSessionId should return a UUID - https://phabricator.wikimedia.org/T266813 (10jlinehan) See also https://phabricator.wikimedia.org/T240461 [16:28:49] (03CR) 10Joal: Improve webrequest-refine query shuffle stage (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/638086 (https://phabricator.wikimedia.org/T267008) (owner: 10Joal) [16:36:57] (03CR) 10Elukey: [C: 03+1] "I like it thanks!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/638086 (https://phabricator.wikimedia.org/T267008) (owner: 10Joal) [16:38:07] 10Analytics, 10Analytics-EventLogging, 10Product-Analytics, 10Product-Infrastructure-Data, 10Documentation: Document how ad blockers / tracking blockers interact with EventLogging - https://phabricator.wikimedia.org/T263503 (10jlinehan) [16:50:31] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.0.138 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/638136 [17:02:41] 10Analytics-Clusters, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: TBD) rack/setup/install an-worker10[18-41] - https://phabricator.wikimedia.org/T260445 (10Cmjohnson) [17:16:21] 10Analytics-Clusters, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: TBD) rack/setup/install an-worker10[18-41] - https://phabricator.wikimedia.org/T260445 (10Cmjohnson) @wiki_willy and @elukey I do not have enough 10G rack space to fit 24 2U servers, Currently, I have 17 2U spaces in 10G racks. This is a... [17:18:36] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Add refinery-source jars for v0.0.138 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/638136 (owner: 10Maven-release-user) [17:19:19] elukey: refinery-source v0.0.138 is built, and artifacts are added to refinery, but I'm not going to do a refinery deploy and sync them to HDFS unless you ask me to, just let me know [17:22:45] yep yep all good, thanks :) [17:29:42] ottomata: are you having lunch or not et? [17:34:25] Ok - I'm gonna go for diner and ping ou later ottomata :) [17:36:35] joal: ah am in sre meeting [17:36:46] Ah yes forgot about that - will ping later :) [17:36:49] k :) [17:55:12] 10Analytics-Radar, 10Product-Analytics: Add DesktopWebUIActionsTracking fields to the allowlist - https://phabricator.wikimedia.org/T263143 (10MNeisler) 05Open→03Resolved [18:19:49] elukey / ottomata: either of you available to help me prioritize my tasks? [18:21:20] razzi: sure gimme 5-10mins [18:21:35] Cool [18:22:17] mforns: /etc/refinery/network_region_config.yaml is now on an-launcher1002 :) [18:22:33] ottomata: yay \o/ [18:22:40] thanks! [18:23:14] razzi: actually maybe 15 mins...gonna grab a coffee brb :) [18:23:21] sounds good [18:29:58] 10Analytics, 10Analytics-Wikistats: pagecounts-ez uploads stopped after 9/24 - https://phabricator.wikimedia.org/T265378 (10S1magreene) 05Open→03Resolved [18:39:37] ok razzi bc! [18:47:58] awight: yt? [18:55:09] (razzi: sorry I am in an interview, didn't see the ping until now) [19:07:53] elukey: no worries, chatting with Andrew [19:08:21] super [19:09:31] * elukey afk! [19:11:16] (03PS6) 10Mforns: Add Refine transform function for Netflow data set [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/634328 (https://phabricator.wikimedia.org/T254332) [19:11:46] (03PS7) 10Mforns: Add Refine transform function for Netflow data set [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/634328 (https://phabricator.wikimedia.org/T254332) [19:21:27] Heya ottomata - would you have time now? [19:27:09] joal: yes! [19:27:10] 10Analytics, 10Analytics-Kanban, 10Event-Platform: eventgate-analytics-external occasionally seems to fail lookups of dynamic stream config from MW EventStreamConfig API - https://phabricator.wikimedia.org/T266573 (10Ottomata) [19:27:13] bc lets go [19:27:14] \o/ [19:27:15] sure [19:27:25] actually - in half a minute :) [19:29:41] (03PS8) 10Mforns: Add Refine transform function for Netflow data set [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/634328 (https://phabricator.wikimedia.org/T254332) [19:38:37] 10Analytics, 10Operations, 10SRE-Access-Requests: Requesting access to production shell groups for JAnstee - https://phabricator.wikimedia.org/T266249 (10JAnstee_WMF) We are working to get our Director onboarded to phabricator and will hopefully be able to add to the card soon for approval! [19:46:19] 10Analytics-Radar, 10Operations, 10SRE-Access-Requests: Nuria's volunteer account - https://phabricator.wikimedia.org/T266086 (10Dzahn) email address changed: https://gerrit.wikimedia.org/r/c/operations/puppet/+/636936 [20:00:06] 10Analytics, 10Product-Analytics: Streamline Superset signup and authentication - https://phabricator.wikimedia.org/T203132 (10nshahquinn-wmf) [20:07:20] (03CR) 10Mforns: [V: 03+2] "OK, tested this last version with all comments addressed, and looking good." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/634328 (https://phabricator.wikimedia.org/T254332) (owner: 10Mforns) [20:09:02] razzi: I just finished all changes requested to the netflow transform function, including your suggestion of adding tests. Please, feel free to look at the code again, or let me know if you'd like to do a live pair review :] [20:10:06] ottomata: do you see any problem in moving the netflow refined dataset to be part of the event database? instead of wmf? [20:12:27] ottomata: even after migrating netflow to event platform? [20:14:05] * razzi taking some time offline [20:18:52] 10Analytics-Clusters, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: TBD) rack/setup/install an-worker10[18-41] - https://phabricator.wikimedia.org/T260445 (10wiki_willy) [20:19:03] 10Analytics-Clusters, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: TBD) rack/setup/install an-worker10[18-41] - https://phabricator.wikimedia.org/T260445 (10wiki_willy) Thanks for the heads up @Cmjohnson . @elukey - do you have any servers on existing servers on 10g switches, that you might be able to d... [20:24:48] 10Analytics, 10Analytics-Wikistats, 10Inuka-Team, 10Language-strategy, and 2 others: Have a way to show the most popular pages per country - https://phabricator.wikimedia.org/T207171 (10mforns) @lexnasser Thanks for the analysis of Andorra's traffic! In this case, no article would make it to the ranking be... [20:41:20] hmmm [20:41:55] hmm mforns does netflow have an event schema? [20:42:02] it doesn't right? [20:42:06] ottomata: not yet that I know! [20:42:13] i guess we could make one :) [20:42:18] ottomata: https://phabricator.wikimedia.org/T248865 [20:42:22] hmmm [20:42:39] oh right. [20:43:03] I think we can do that whenever we need/want, but just thinking of https://phabricator.wikimedia.org/T231339 [20:43:16] i remember this [20:43:17] also [20:43:27] my comment here: https://phabricator.wikimedia.org/T248865#6011075 [20:44:00] ottomata: I was concerned that once ingestion is done through mep, then we could have difficulties having a separate job for netflow? Probably I'm wrong [20:44:14] right. this makes sense [20:44:14] ok [20:44:25] using eventgate is optional. [20:44:41] but for the Hive ingestion automation [20:44:43] the rest is not [20:45:03] riight luca is already working on this [20:45:04] ok ok [20:45:06] remembering [20:45:31] mforns: as of last month the stream/topic discovery for ingestion is done by stream config [20:45:34] so [20:45:37] ottomata: I see, but my question is: can we still have a separate refine job for netflow even if it is within the event database [20:45:38] ? [20:45:53] ah [20:45:54] yes [20:46:03] oh, ok! :] [20:46:05] we have different refine jobs that write to event eb now anyway [20:46:06] e.g. [20:46:07] evnetlogging legacy [20:46:12] aha, I understand [20:46:20] hmmm [20:46:31] so, then, I don't see any problem in moving netflow to event database no? [20:46:35] we might need a separate camus job...but there is already one, right? [20:46:39] this way we can apply sanitization to it [20:46:47] yeah i think that is fine [20:46:56] :] OK [20:47:04] can you do the Refine with jsonschema if we make an eventschema [20:47:06] ? [20:47:07] https://gerrit.wikimedia.org/r/c/schemas/event/secondary/+/608077/ [20:47:36] it can be a different Refine job, but using the same settings to look up the schema to create and evolve the hive table? [20:49:19] (03CR) 10Ottomata: [C: 03+1] Add Refine transform function for Netflow data set [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/634328 (https://phabricator.wikimedia.org/T254332) (owner: 10Mforns) [20:49:19] ottomata: sure! do you want me to work on this data set before we do the other mep things, like https://phabricator.wikimedia.org/T253069 ? [20:49:33] the netflow data? [20:49:38] yes [20:50:24] oh, heheh, hm, i think that is a separate task/goal altogetherr, maybe talk to luca since he's already been working on it? not sure. i don't have a big stake in netflow atm, so my personal preference is the mep dev env goal...but that's just my own [20:50:29] we need a prioritizer! [20:50:38] is francisco the prioritizer?! [20:50:44] hehe [20:51:32] I think moving netflow to mep is not adding a lot of value for now, except when it comes to unify all input streams, so I'm with you [20:52:28] ottomata: but what I do want to do now (as background task while doing mep), is to allow sanitization of netflow, which is out of compliance right now. [20:52:43] that's why I was asking [20:53:06] it involves backfilling, so it's good for background task [20:53:33] mforns: ok so by putting netflow into event, you get some automation there...i think that should be ok. it'd be nice to get the schema stuff working the same for an event.netflow table. hm. [20:54:02] i'd worry that by putting it in event now, the schema lineage will kinda change later...since now it is just inferred from the json data i guess? and is missing fields that would be included in the MEP schema? [20:54:14] it'd be similiar to the eventlogging legacy data in that case though. [20:54:15] hm [20:54:33] oh [20:55:08] mforns: i think it is probably ok. it might cause a small data integration headache if we put it in event db now but later switch to a different data integration method (event schema based) [20:55:23] oh... [20:55:31] buut also it might not :) [20:56:03] i think it would be pretty simliar to the eventlogging legacy migration. [20:56:21] so, a little bit of extra work to do, but not so bad, and we'd get sanitization now rather than later, so that is good. [20:56:29] ok [20:56:37] and also the users of netflow data are pretty limitied, so if we break it we can work with them :p [20:57:25] ok, ottomata thanks! [20:58:42] ottomata: will do break now, and be back later, if you want to give me mep tasks then, I'd start with that [21:00:21] ok cool ! [21:14:42] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10Ottomata) [21:15:20] !log evolved Hive table event.contenttranslationabusefilter to match migrated event platform schema - T259163 [21:15:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:15:23] T259163: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 [21:23:39] (03CR) 10Joal: "One little comment request - and ready to go :) Thanks @mforns :)" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/634328 (https://phabricator.wikimedia.org/T254332) (owner: 10Mforns) [21:27:13] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform: Exclude canary events from refined event Hive tables - https://phabricator.wikimedia.org/T267073 (10Ottomata) [21:27:23] 10Analytics, 10Analytics-EventLogging, 10Event-Platform: Exclude canary events from refined event Hive tables - https://phabricator.wikimedia.org/T267073 (10Ottomata) [21:30:24] (03PS9) 10Mforns: Add Refine transform function for Netflow data set [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/634328 (https://phabricator.wikimedia.org/T254332) [21:30:38] (03CR) 10Mforns: Add Refine transform function for Netflow data set (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/634328 (https://phabricator.wikimedia.org/T254332) (owner: 10Mforns) [21:34:05] hey ottomata I'm back [21:34:05] (03CR) 10Joal: [C: 03+2] "LGTM! Thanks mforns" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/634328 (https://phabricator.wikimedia.org/T254332) (owner: 10Mforns) [21:34:11] thanks joal! :] [21:34:17] np mforns :) [21:34:23] Gone for tonight team [21:34:25] \o [21:34:30] the patch looks a lot better now! [21:34:36] byeeee cya [21:34:59] Thanks for the patience mforns :) [21:35:15] thank you! [21:35:47] mforns: am here! [21:35:55] want to bc? [21:35:58] yep! [21:36:07] omw [21:40:50] (03Merged) 10jenkins-bot: Add Refine transform function for Netflow data set [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/634328 (https://phabricator.wikimedia.org/T254332) (owner: 10Mforns) [22:43:09] Hi, would it be easy to give me a rough number of "users on French Wikipedia per hour" [22:48:53] mutante: logged in? [22:48:57] mutante: or just readers? [22:49:33] nuria: ehm.. logged in, actually! [22:50:11] trying to find out roughly how many users were affected by an incident that was just the fr.wp mainpage though [22:50:28] so i would also have to say "using the main page" but that might make it too complicated [22:53:51] mutante: editors you mean then as users there might be millions but editors are afew [22:55:53] nuria: yea, i know it was just loading slow when logged in because users got cached results [22:56:30] mutante: TLDR; there is not precise data on that until mediawiki sends an even when users log in /log out [22:57:10] mutante: to estimate it you can see ghow many edits by distinct editors were done on that hour and that would give you some bounds, probably any hour would do, makes sense? [22:58:10] nuria: yes, it does. though there are probably also many who have an account and are logged by default even when most of the time they are just reading. thank you [23:00:33] mutante: the table in the cluster that has that info is editors_daily [23:02:56] mutante: we can get number of affected requests pretty easily, but ofc getting from there to any concept of # of users is harder [23:03:06] https://w.wiki/jrz [23:03:19] nuria: thanks! btw, in another matter, we finished the switch from "wmf" to "nda" for you. you should not have noticed a difference but let us know if anything is tied to "wmf only" unexpectedly [23:03:28] mutante: WILL do [23:03:38] mutante: SUPER thanks as always [23:04:10] mutante: let me try my credentials with these selects [23:07:07] mutante: here are some bounds [23:08:45] mutante: from editors_daily (not counting anonymous editors) there are about [23:09:02] 2500 distinct editors on any given day (looked at a couple days) [23:09:31] mutante: so that gives you apretty good upper bound of "users logged in frwiki doing stuff" [23:11:55] nuria: awesome! thank you :)