[00:13:43] 10Analytics, 10Contributors-Analysis, 10Product-Analytics: Streamline Superset signup and authentication - https://phabricator.wikimedia.org/T203132 (10Neil_P._Quinn_WMF) >>! In T203132#4544021, @Nuria wrote: >>Ask the Analytics team to create a new Superset account. > This should no longer be the case with... [00:14:10] 10Analytics, 10Contributors-Analysis, 10Product-Analytics: Streamline Superset signup and authentication - https://phabricator.wikimedia.org/T203132 (10Neil_P._Quinn_WMF) [00:18:02] 10Analytics, 10Contributors-Analysis, 10Product-Analytics: Streamline Superset signup and authentication - https://phabricator.wikimedia.org/T203132 (10Nuria) Ok, note that any of the steps above are needed to access any kind of tool (even our internal vpn) at the foundation, so while I agree it might not b... [00:26:40] (03PS1) 10Nuria: Updating dashiki dashboard in analytics.wikimedia.org [analytics/analytics.wikimedia.org] - 10https://gerrit.wikimedia.org/r/456310 [00:27:51] (03PS2) 10Nuria: Updating dashiki dashboard in analytics.wikimedia.org [analytics/analytics.wikimedia.org] - 10https://gerrit.wikimedia.org/r/456310 (https://phabricator.wikimedia.org/T202792) [00:28:31] (03CR) 10Nuria: [V: 032 C: 032] "Merging to deploy after testing changes" [analytics/analytics.wikimedia.org] - 10https://gerrit.wikimedia.org/r/456310 (https://phabricator.wikimedia.org/T202792) (owner: 10Nuria) [00:29:06] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Update superset (we have 0.20.6, 0.26.3 is available) - https://phabricator.wikimedia.org/T201430 (10Nuria) 05Open>03Resolved [00:29:42] 10Analytics, 10Analytics-Kanban: Reimage thorium to Debian Stretch - https://phabricator.wikimedia.org/T192641 (10Nuria) [00:29:49] 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Move internal sites hosted on thorium to ganeti instance(s) - https://phabricator.wikimedia.org/T202011 (10Nuria) 05Open>03Resolved [00:30:14] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Review analytics-in4/6 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10Nuria) 05Open>03Resolved [00:30:24] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Install pyArrow in Cluster - https://phabricator.wikimedia.org/T202812 (10Nuria) 05Open>03Resolved [00:30:44] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Upgrade librdkafka on eventlog1002 - https://phabricator.wikimedia.org/T200769 (10Nuria) 05Open>03Resolved [00:30:46] 10Analytics-Kanban, 10Patch-For-Review: Eventlogging's processors stopped working - https://phabricator.wikimedia.org/T200630 (10Nuria) [00:31:13] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Add app_install_id and other renamed fields to EL sanitization whitelist - https://phabricator.wikimedia.org/T200095 (10Nuria) [00:31:27] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Add app_install_id and other renamed fields to EL sanitization whitelist - https://phabricator.wikimedia.org/T200095 (10Nuria) 05Open>03Resolved [00:32:16] 10Analytics, 10Contributors-Analysis, 10Product-Analytics: Streamline Superset signup and authentication - https://phabricator.wikimedia.org/T203132 (10Neil_P._Quinn_WMF) I think the remaining avenues for improvement are: * Set up a process that doesn't require non-technical users to request LDAP themselves.... [02:58:04] 10Analytics, 10MediaWiki-extensions-JsonConfig, 10MW-1.32-release-notes (WMF-deploy-2018-08-28 (1.32.0-wmf.19)), 10Patch-For-Review, 10Wikimedia-log-errors: Config:Dashiki:* on Meta can't be opened - https://phabricator.wikimedia.org/T203029 (10Mholloway) >>! In T203029#4543634, @Jdforrester-WMF wrote: >... [06:00:02] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade Archiva (meitnerium) to Debian Stretch - https://phabricator.wikimedia.org/T192639 (10elukey) >>! In T192639#4543940, @Smalyshev wrote: > archiva.wikimedia.org works fine for me now. Thanks for the feedback! [06:02:29] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade Archiva (meitnerium) to Debian Stretch - https://phabricator.wikimedia.org/T192639 (10elukey) Next steps to finish the upgrade: * Ask people to review and merge https://gerrit.wikimedia.org/r/456156 (rsync listening IPv6) * Decomm... [07:11:52] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade Archiva (meitnerium) to Debian Stretch - https://phabricator.wikimedia.org/T192639 (10elukey) >>! In T192639#4544426, @elukey wrote: > Next steps to finish the upgrade: > * Verify that archiva1001 is correctly backed up via bacula... [07:35:39] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade Archiva (meitnerium) to Debian Stretch - https://phabricator.wikimedia.org/T192639 (10elukey) >>! In T192639#4544426, @elukey wrote: > Next steps to finish the upgrade: > * Update ops' pwstore with new admin passwords (plus remove... [07:41:13] hi, I'm looking for a better technique to debug a process running via yarn: today I add some print statements and login to yarn, copy/paste the stderr link, ssh to a stat matchine, curl this url (http://analytics1075.eqiad.wmnet:8042/node/containerlogs/....), decypher the html output [07:41:55] hm [07:41:58] Hi dcausse [07:42:02] hi joal! [07:42:39] some questions to be sure I understand correctly - You're willing to look at logs of a yarn container [07:43:02] yes [07:43:44] today it's a bit painful because the links on yarn.wikimedia.org sometimes points to analytics1075.eqiad.wmnet:8042 [07:44:04] so I have to ssh on a stat machine to access them [07:44:38] dcausse: I have no better solution than ssh-tunneling onto the given machione [07:44:42] ok [07:45:16] or, you can also actually ssh to the machine and look at logs locally, but I don't do it this way, prefering the browserd solution [07:45:49] I don't think I have access to analytics1xxx machines on stat1xxx ones [07:46:17] hm perhaps I should use a text browser on stat machine directly, instead of using yarn.wikimedia.org [07:50:49] joal: there's no place in hdfs where the overall logs are collected and stored after a container is done? Like /user/yarn/etc.. [07:51:08] not sure why but I recall inspecting something on hdfs when debugging in labs [07:51:15] but I am probably misremembering [07:55:08] dcausse: I didn't find any good http proxy rule to make yarn looks better in these cases :( [07:56:46] elukey: what about installing lynx or elinks on a stat machine? :) [07:57:32] is the analytics* http port available from there? [07:57:35] (never tested it) [07:58:19] elukey: yes I usually use curl but it's a pain :) [07:58:53] ah yes I can imagine [08:06:49] dcausse: patch coming [08:08:33] elukey: wow, thanks for looking into this! [08:12:48] dcausse: if you are on stat1004 lynx is there now [08:12:51] stat1005 in progres [08:12:59] *progress (puppet takes a long time there) [08:13:54] elukey: it works like a charm on stat1004! thanks!! [08:37:28] (03PS1) 10GoranSMilovanovic: Most popular entries [analytics/wmde/WiktionaryCognateDashboard] - 10https://gerrit.wikimedia.org/r/456349 [08:37:45] (03CR) 10GoranSMilovanovic: [V: 032 C: 032] Most popular entries [analytics/wmde/WiktionaryCognateDashboard] - 10https://gerrit.wikimedia.org/r/456349 (owner: 10GoranSMilovanovic) [09:03:27] joal: if you are ok I'd start the reboots [09:03:32] (hadoop) [09:48:47] (03PS1) 10GoranSMilovanovic: Implement Most Popular Tab [analytics/wmde/WiktionaryCognateDashboard] - 10https://gerrit.wikimedia.org/r/456363 [09:49:31] (03CR) 10GoranSMilovanovic: [V: 032 C: 032] Implement Most Popular Tab [analytics/wmde/WiktionaryCognateDashboard] - 10https://gerrit.wikimedia.org/r/456363 (owner: 10GoranSMilovanovic) [10:19:51] Wow - excuse me elukey, got IRC unaware for long ! [10:20:04] elukey: I'm asuming you've started (and maybe even finished) the reboots [10:20:45] nope I didn't! I wrote some python code for the python user checker, wanted to have your green light before proceeding (not really urgent, I'll start this afternoon) [10:21:22] np - good for me ;) [10:21:44] elukey: about dcausse issue for logs, I wonder if he knows he yarn logs command [10:28:05] joal: it might be another option yes, doesn't require sudo for his username [10:28:10] forgot about checking it [10:33:28] going out for lunch! ttl :) [10:33:35] later :) [10:33:43] after lunch I'll start the reboots [10:33:48] sounds good Q! [10:51:34] hi a team :D [10:52:03] why are some metircs such as editors on the main page of wikistats2 not available for "all projects" ? [10:52:58] addshore: cause we weren't using editor-text as base for computation, but editor-id - making cross-project deduplication impossible [10:53:12] :( [10:53:33] addshore: your question makes me realize that, from last deploy, since we have changed to use user-text and page-title, we can now do it :D [10:53:44] joal: that would be amazing xD [10:53:50] This is a low-hanging metric fruit I had not thought off :D [10:53:58] * addshore posts joal more cookies [10:54:28] addshore: I'll double check the metric, but this seem indeed veru feasible !! [10:54:48] addshore: do you know if there is a place where histoical version of thqat metric would be available? [10:55:08] * joal thanks addshore a lot for the cookies [10:55:36] I dont think I have ever seen that metric before, thats why im so curious :p [10:55:59] addshore: hm - Im pretty sure it has already been done [10:56:08] addshore: I'll look at wikistats-1 [11:21:00] joal: why is unique devices not available for all projects? [11:21:05] that can't have the same user id issue? [11:22:46] addshore: we don't compute them [11:22:51] oh :( [11:24:17] addshore: unique-devices computation is complicated (in term of correctness) and computation-power intensive - We have not been asked to do it for all-projects ever - So we have not even considered it [11:24:18] i wonder if https://www.comscore.com/Insights/Rankings/ saying 128 million is at all accurate [11:25:43] addshore: OHHHH ! [11:25:47] addshore: my mistake [11:26:03] addshore: The reason for which we don't have them is cause they require another cookie [11:26:48] addshore: We have uniques-per-domain, uniques-per-project-family, but we don't have them global [11:27:02] D: [11:27:26] addshore: The technique we use for unique-devices is Last-Access-Cookie - We have 2 of them - One per domain, one per project-family [11:27:42] hmm, how can I get to the per project numbers? [11:27:45] addshore: we could ask to have another one global for all WMF websites [11:28:05] IMO that's a valuable metric [11:28:20] addshore: I'm not opposed - ticket? [11:28:48] can do, I dont think the per project family numbers are available in wikistats2? at least i can't make it give them to me [11:29:19] addshore: project-families are not (yet) available in WKS2 - True [11:30:48] addshore: I can't recall if we have a dashboard for per-project-family somewhere [11:30:57] addshore: data is available in dumps format, of in hive [11:33:36] addshore: acutally, in druid !!! in turnilo [11:34:10] :D [11:35:45] addshore: is the comscore thing about daily? [11:36:15] or monthly? [11:36:31] joal whenever you have two minutes, can I ask you a spark question in the cave? [11:36:43] fdans: A la baticueva!! [11:36:55] la batcaveaux! [11:39:59] joal: monthlky [11:42:07] addshore: from turnilo, uniques-per-project-family-monthly, for US only, between 270 and 300 millions unique-devices per month since beginning of year for wikipedias [13:00:28] hello people, as FYI I am going to start the reboot of all the analytics hadoop worker nodes for kernel updates. I'll try to do small batches and drain them first from any job, but you might experience failures [13:01:14] +1 elukey :) [13:16:42] 10Analytics, 10Contributors-Analysis, 10Product-Analytics: Streamline Superset signup and authentication - https://phabricator.wikimedia.org/T203132 (10Ottomata) BTW, there may be some interest from the newish security team in SRE to work on an analytics VPN. We aren't sure, but they just did an audit of ou... [13:18:17] 10Analytics, 10Contributors-Analysis, 10Product-Analytics: Streamline Superset signup and authentication - https://phabricator.wikimedia.org/T203132 (10Ottomata) > Make the choice between "wiki" and "shell" usernames (some LDAP services want one, some want another) less confusing. I imagine it's out of scope... [13:42:23] so rebooted two hosts [13:42:27] one is not coming up [13:42:31] seems bad disk [13:42:35] lovely [13:44:21] whatcha rebooting? [13:45:47] all the hadoop cluster, kernel updates [13:46:26] we'll be rebooting everything eventually (CPU-related vuln affecting all servers) [13:46:56] moritzm: shhh let me convince that I have only 50 hosts to reboot :D [13:47:31] * moritzm didn't say anything [13:56:10] maybe fsck did the magic [13:56:16] but the disk is not really good [13:56:28] so if an1029 boots I'd open a task to swap the disk [13:56:36] (there were I/O errors reported) [13:58:59] ah [13:59:09] yall da best [14:01:43] joal: I left the thing loading during lunch but I'm pretty sure that's going nowhere [14:35:31] * elukey restarts nodes without worrying about journal nodes since we have 5 [14:44:01] 10Analytics, 10Analytics-Kanban: Reboot Analytics hosts for kernel security upgrades - https://phabricator.wikimedia.org/T203165 (10elukey) p:05Triage>03High [14:49:36] !log re-run webrequest-load-wf-text-2018-8-30-12, failed due to worker nodes reboots [14:49:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:55:06] | | | [14:55:08] | | | [14:55:11] __| | |__ [14:55:13] / \ [14:55:16] / ^^\ /^^ \ [14:55:18] | 0 0 | LOL HY [14:55:21] ( | | ) KICK ROLLOFFLE [14:55:24] | U || PORFA IS ON IGNORE [14:55:26] \ \____/ /+ JESUITX YOU FAT FUCKER [14:55:30] \ '--- ' / PLZ STOP ABUSE LOL STOP LOL [14:55:31] / |\_____/| [14:55:34] / | | [14:55:37] / ; } [14:55:39] ; '::. ;\/\ /\ { [14:55:42] |. ':. ;``"``\ [14:55:44] / '::'::' / ; [14:55:46] |':::' '::' / D H | [14:55:49] \ '::' _.-`; I E ; [14:55:51] ottomata: I am not an op in here, not sure what to do for --^ [14:55:51] /`-..--;` ; | K A | [14:55:54] ; ; ; ; ; | K R | [14:56:02] thank AlexZ [14:56:06] *thanks! [14:56:53] addshore: we cannot compute unique devices for all sites with current method ccjoal [14:57:13] addshore: because it will require for cookies to be read across domains and that is not possible [14:57:27] AlexZ: any advice about how to be able to +o in here for me? To be able to kick if you are not around [14:57:45] addshore: now, unique devices for ALL wikipedias is 95% of the number that that might be (cc joal) if not more [14:57:55] 4 elukey +ARfiorstv [modified 1y 42w 1d ago] [14:58:08] +1 nuria [14:58:22] addshore: yeah but if I try to +o I get perm denied [14:58:32] so it is probably me missing something trivial [14:58:33] elukey: through chanserv? [14:58:37] yeah [14:59:04] addshore: now, unique devices for ALL wikipedias is the number you are looking for more or less, the numbers for smaller projects are very small in comparation [14:59:36] nuria: yup, i guess looking at just wikipedia would be a pretty good comparison! [15:00:10] addshore: data available in turnilo for dailies so you TOO can know the numbers [15:01:00] ping ottomata elukey fdans [15:01:07] sorryyy [15:01:23] addshore: sorry I only tried with /mode #channelname +o nickname [15:02:19] eeee [15:02:25] elukey: I’ll look into it a bit later today. Have to hit the road before traffic starts out here [15:05:19] I could use some help setting up a new scheme for eventlogging, please. This is to support the EU copyright landing page project. The developer is using https://meta.wikimedia.org/wiki/Schema:EUCCStats and https://meta.wikimedia.org/wiki/Schema:EUCCVisit. They sent some events, but AFAICT the schema has not been created in the database - unless I'm looking in the wrong place, which is entirely possible. [15:07:51] addshore, AlexZ --^ [15:07:53] I am dumb [15:07:56] thanks for the help [15:09:52] elukey: op yourself again and invite Sigyn in here [15:10:12] will kill a large proportion of the spammers [15:13:52] addshore: is it /invite Sigyn in here while I am +o ? [15:14:46] there --^ [15:14:57] thanks addshore [15:18:42] [= [15:23:56] !log re-run webrequest-load-wf-upload-2018-8-30-13, failed due to hadoop worker reboots [15:23:57] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:24:21] Thanks elukey for the reboot and the unbotifier :) [15:26:07] CindyCicaleseWMF: https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/TestingOnBetaCluster explains how to test in beta schemas before they go to production, the country is already available server side (in hdfs storage , not mysql) so need to send it from client side [15:29:16] CindyCicaleseWMF: seems that all info on that schema is available server side, can you send link to site? [15:29:34] nuria: thanks for the link. Regarding the country, is there more information about how we would combine that information with the information in mysql when doing analysis of the data? [15:30:56] nuria: The demo site is at https://wmfeucc.demos.hallowelt.com/, but I'm not sure if they have the event logging code enabled there yet. [15:31:26] CindyCicaleseWMF: your data is also on hive [15:31:45] CindyCicaleseWMF: so no need to combine it, it is available ALL of it on hdfs plus some extra bits [15:31:51] CindyCicaleseWMF: such us country [15:32:19] CindyCicaleseWMF: country of origin cannot be sent from client side reliably in fresh pageviews anyways so iyt has to be calculated server side [15:33:39] CindyCicaleseWMF: all the info on schema will be already available on cluster if this site is fronted by varnish [15:34:13] CindyCicaleseWMF: you have language on url and country is added to request from request IP so ... do you really need to send that data at all? [15:34:14] nuria: The live site will be at fixcopyright.wikimedia.org. [15:35:45] 10Analytics, 10MediaWiki-extensions-JsonConfig, 10MW-1.32-release-notes (WMF-deploy-2018-08-28 (1.32.0-wmf.19)), 10Patch-For-Review, 10Wikimedia-log-errors: Config:Dashiki:* on Meta can't be opened - https://phabricator.wikimedia.org/T203029 (10Nuria) Sounds good @Mholloway , will close this task and ope... [15:38:57] 10Analytics: Deprecate reportcard: https://analytics.wikimedia.org/dashboards/reportcard/ - https://phabricator.wikimedia.org/T203128 (10fdans) p:05Triage>03Normal [15:39:46] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Create .deb package for Presto - https://phabricator.wikimedia.org/T203115 (10fdans) p:05Triage>03Normal [15:40:25] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Use the "private" and "closed" sitematrix keys instead of "INVALID" - https://phabricator.wikimedia.org/T203105 (10fdans) a:03fdans [15:40:33] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Use the "private" and "closed" sitematrix keys instead of "INVALID" - https://phabricator.wikimedia.org/T203105 (10fdans) p:05Triage>03Normal [15:42:05] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Use the "private" and "closed" sitematrix keys instead of "INVALID" - https://phabricator.wikimedia.org/T203105 (10fdans) [15:42:07] 10Analytics, 10Analytics-Wikistats: Wikistats should not show private wikis in drop downwiki selection menu - https://phabricator.wikimedia.org/T203103 (10fdans) [15:42:35] CindyCicaleseWMF: some hive eventlogging docs https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging#Hive [15:42:52] all refined eventlogging tables in hive have geocoded data [15:43:51] e.g. [15:43:52] select geocoded_data from Popups where month=8 and day=1 and hour=0 limit 10; [15:44:01] 10Analytics, 10MediaWiki-extensions-JsonConfig, 10MW-1.32-release-notes (WMF-deploy-2018-08-28 (1.32.0-wmf.19)), 10Patch-For-Review, 10Wikimedia-log-errors: Config:Dashiki:* on Meta can't be opened - https://phabricator.wikimedia.org/T203029 (10fdans) 05Open>03Resolved a:03fdans Closing in favor of... [15:44:22] {"city":"XXX","latitude":"XX.X","timezone":"America/New_York","country":"United States","longitude":"-XX.XX","continent":"North America","country_code":"US","subdivision":"XXX","postal_code":"XXX"} [15:45:23] OK, so it sounds like https://meta.wikimedia.org/wiki/Schema:EUCCVisit is unnecessary because the page visits will already be captured in hive and the url associated with the visit will have the language code. So we will be able to query hive for visits to the page and get the country/language information from there. Is that correct? [15:48:37] CindyCicaleseWMF: yes [15:48:47] CindyCicaleseWMF: for the data you have on your schema [15:49:00] CindyCicaleseWMF: that information will be availabe for 90 days in hive [15:51:31] OK. I will ask if 90 days is sufficient. So, for https://meta.wikimedia.org/wiki/Schema:EUCCStats, it isn't clear to me how we could correlate get the language/country from hive corresponding to the other fields in the schema if we remove language/country from that schema. [15:53:58] CindyCicaleseWMF: the schema data will be on hive together with a field (created by our code in hive) that has geocoded info for every record on that schema [15:54:47] ah, gotcha [15:55:26] ah you mean in pageviews not eventlogging [15:55:27] cool [15:55:30] ya better [15:57:35] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Create .deb package for Presto - https://phabricator.wikimedia.org/T203115 (10Ottomata) [15:59:34] !log rerun of pageview-druid-hourly-wf-2018-8-30-13, hadoop worker reboots in progress [15:59:35] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:03:09] So, to summarize, we only need the https://meta.wikimedia.org/wiki/Schema:EUCCStats schema. It will have the country and language removed. We will be able to query hive for page views of https://fixcopyright.wikimedia.org/wiki/Main_Page, which will have the country and, from the uselang parameter, the language. We will also be able to query the EUCCStats information from hive, which will be geocoded with [16:03:10] the country information (but how will we get the language?). The page view information will only be available for 90 days. What about the EUCCStats information? [16:03:32] 10Analytics-Kanban: Add endpoints to RESTBase for new WKS2 endpoints - https://phabricator.wikimedia.org/T203175 (10JAllemandou) [16:23:12] CindyCicaleseWMF: i would keep "language" in https://meta.wikimedia.org/wiki/Schema:EUCCStats even if it is on URL now i would remove https://meta.wikimedia.org/wiki/Schema:EUCCVisit completely [16:23:35] CindyCicaleseWMF: all info will be available by default for 90 days [16:32:34] 10Analytics: Wikistats: add functions you apply to all data in UI such as "accumulate" - https://phabricator.wikimedia.org/T203180 (10Nuria) [16:36:59] 10Analytics, 10Analytics-Wikistats: "Total Article Count" Wikistats metric (per project and overall) - https://phabricator.wikimedia.org/T198425 (10Nuria) a:03fdans [16:38:44] 10Analytics: Wikistats: add functions you apply to dimensional data such as "accumulate" - https://phabricator.wikimedia.org/T203180 (10Nuria) [16:38:46] 10Analytics, 10Analytics-Wikistats: "Total Article Count" Wikistats metric (per project and overall) - https://phabricator.wikimedia.org/T198425 (10Nuria) [16:38:48] 10Analytics: Wikistats: add functions you apply to dimensional data such as "accumulate" - https://phabricator.wikimedia.org/T203180 (10Nuria) [16:40:07] 10Analytics, 10Analytics-Wikistats: "Total Article Count" Wikistats metric (per project and overall) - https://phabricator.wikimedia.org/T198425 (10Nuria) Let's define a new metric: * Total article count * metric is defined on "all" time range * metric is bookmarkable * metric is defined per project and all-p... [16:43:15] 10Analytics, 10Analytics-Wikistats: "Total Article Count" Wikistats metric (per project and overall) - https://phabricator.wikimedia.org/T198425 (10Nuria) Second step might be to enrich "new pages" metric such we can calculate the accumulation via a functiuon defined in the UI such us "accumulate". See: http... [16:45:12] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install analytics-master100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T201939 (10Cmjohnson) [16:48:22] CindyCicaleseWMF: you probably alredy have permits to access hive just ssh into stats1005 or stats1004 and see if you can type hive [16:48:37] and get access to a hive prompt [16:48:48] CindyCicaleseWMF: still, before doing all this I would test in beta [16:49:45] ottomata: good news! analytics1042 rebooted and it went through PXE [16:50:04] I have access to stats1006, but when I try to log in to stats1004 or stats1005, it prompts me for my password [16:53:09] Cc joal --^ [16:53:58] =o [16:54:02] no data for CindyCicaleseWMF [16:55:43] looks like you are in researchers, but you may also have to be in analytics-privatedata-users ? [16:56:10] based on the 3 wmde requests for eventlogging data that we have put through in the past 2 weeks [16:57:10] thanks, addshore. I'm guessing that requires filing a Phab task? [16:57:15] yup [16:57:29] elukey: yeehaw [16:57:47] CindyCicaleseWMF: https://wikitech.wikimedia.org/wiki/Production_shell_access#Additional_permissions_for_existing_users [16:58:04] Riccardo is checking all the boot params, apparently there is a weird bug ongoing.. it must have been set when we reimaged in may [16:58:09] I checked and this is the first reboot [16:58:23] addshore: 😀 Thanks! I was just going to try to find that. [16:58:26] [= [17:03:13] addshore: do you think RESOURCE should be hive OR stats1004/stats1005 OR analytics-privatedata-users OR some other thing? [17:05:24] CindyCicaleseWMF: analytics-privatedata-users is fine, you can say EventLogging in Hive [17:05:32] or pageviews [17:05:45] thanks! [17:10:36] :) [17:17:15] OK, now looking at testing on the beta cluster. It requires access to deployment-eventlog05. Is that another Phab task? [17:25:44] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service - https://phabricator.wikimedia.org/T201063 (10Ottomata) [17:41:34] ottomata: so apparently some of our hosts have the boot order wrong, namely NIC before Hard disk [17:42:24] so we need to sweep all the hosts that we have and check (via mgmt) racadm get bios.BiosBootSettings.BootSeq [17:42:40] and if NIC is first, reboot in BIOS and change the boot order [17:42:46] otherwise it'll go in PXE [17:42:51] all the other ways are not ok [17:42:59] (at least for the moment, after a chat with SREs) [17:44:46] an1042 just came back as it was before the reboot, luckily [17:45:06] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 2 others: RFC: Modern Event Platform: Schema Registry - https://phabricator.wikimedia.org/T201643 (10Ottomata) [17:47:17] just checked and an100[1-3] have the correct setting [17:47:36] so I stopped at an1044 iirc, tomorrow I'll do all the checks and then reboot [17:47:43] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 2 others: RFC: Modern Event Platform: Schema Registry - https://phabricator.wikimedia.org/T201643 (10Ottomata) I've edited the RFC description to account for: - add more info about schema usage metadata vs schema registry - modify R... [17:48:27] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform: Scalable Event Intake - https://phabricator.wikimedia.org/T201068 (10Ottomata) [17:50:26] CindyCicaleseWMF: no, that is regular lab usage access for a lab instance #wikimedia-cloud can help , if you can ssh to labs hosts you can probably ssh there [17:50:48] all right after this nice adventure, I am logging off :) [17:51:12] please people be careful if rebooting any host :) [17:51:15] * elukey off! [17:53:16] CindyCicaleseWMF: do you have a lab user?, maybe I can give you ssh permits, not sure [17:53:56] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service - https://phabricator.wikimedia.org/T201063 (10Ottomata) > 1.) these don't necessarily correspond to the most important needs... [18:30:01] nuria: yes, I do have a labs account (cicalese). I just need to have access to deployment-eventlog05.eqiad.wmflabs [18:35:31] CindyCicaleseWMF: hello :) [18:35:44] what is your shell name? [18:35:56] hi :-) it is cicalese [18:36:17] and Wikitech? [18:37:43] ah shell is fine [18:38:35] so I just added you to the analytics labs project [18:39:07] and also to deployment-prep [18:39:56] Cool - thanks! I'm trying to follow the instructions at https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/TestingOnBetaCluster. [18:40:17] are you able to log in now to deployment-eventlog05 ? [18:40:49] Yes! Now I am :-) Thank you! [18:41:00] super :) [18:41:04] * elukey off again! [18:42:45] elukey: ok thanks for the update, let me know if ic an help [18:42:45] CindyCicaleseWMF: deployment-prep, right that is what it was thanks elukey i forget every time [18:44:02] CindyCicaleseWMF: testing in beta cluster requires you to have your code deployed to beta cluster, which you might have yet if your wiki domain is not enabled , it might be something like fixcopyright-beta.wikimedia.org [18:45:59] nuria: so it isn't possible to send events to beta from an external site? We just had the production site created, but I do not believe that a corresponding beta site was created. It wasn't requested. [18:46:41] CindyCicaleseWMF: you can send events to beta from anywhere https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/TestingOnBetaCluster#How_to_log_a_client-side_event_to_Beta_Cluster_directly [18:46:57] CindyCicaleseWMF: but that would not tell you whether events in your site are working [18:47:14] CindyCicaleseWMF: it will just tell you whether your event validates to schema if that makes sense [18:47:55] CindyCicaleseWMF: also let log in into labs machine to make sure it is all good [18:48:20] ok, so I can test the schema on beta to make sure it is sane. But any events sent from the external test site will go to the production eventlogging database. [18:48:59] CindyCicaleseWMF: if they are valid, yes [18:50:07] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service - https://phabricator.wikimedia.org/T201063 (10Ottomata) In yesterdays meeting for {T201643}, we decided to split out the Sche... [18:51:01] nuria: that curl command is impressive. How to create one from my schema without making an error so that I can be sure any errors are in the schema and not the curl command? [18:51:52] CindyCicaleseWMF: right, that is why i was saying that we use beta to test from sites in beta labs , we normally do not use curl [18:52:30] CindyCicaleseWMF: you can try cut and paste the events being sent from your site on labs [18:53:24] CindyCicaleseWMF: that would be easiest [18:54:14] nuria: ok, thanks. I'll give that a try. [18:56:20] CindyCicaleseWMF: curious, how are you developing? [18:56:24] mediawiki-vagrant? [18:57:43] ottomata: actually, I'm not doing the development. It is a contractor in Germany. They have their own test machine. The code is in gerrit and is undergoing security review before deploying it on a wikimedia wiki. [19:01:35] ah hm [19:03:31] Yeah, so I'm just trying to help them test their events. Not ideal. [19:16:45] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service - https://phabricator.wikimedia.org/T201063 (10Nuria) >Also, perhaps MW (or whatever) could do the phoning-home to get configu... [19:18:31] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service - https://phabricator.wikimedia.org/T201063 (10Ottomata) > the difference is that as envisioned on this system they are not be... [19:51:33] I have some questions about the https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/TestingOnBetaCluster page. [19:53:42] CindyCicaleseWMF: i can try to answer :) [19:53:53] Both of the files it mentions in /srv/log/eventlogging are empty. But, if I do "kafka-tools -b deployment-kafka-jumbo-2.deployment-prep.eqiad.wmflabs:9092 print_topics", I do see both of the schemas I created that the developer is using listed. Does that indicate that some of their tests got through? They are using https://en.wikipedia.beta.wmflabs.org/beacon/event/.... [19:54:12] I tried the curl command with the URL they generated, but I get no feedback. [19:55:38] I also tried the other kafka command listed on that page, but it just sits waiting, even if I try the curl command. [19:56:35] wow kafka-tools! [19:56:40] i dind't even know that was installed, crazy [19:56:48] probably not the best tool to use, buuut beside the point [19:57:06] I'm clearly grasping at straws here and don't know how everything fits together or what I'm supposed to be looking for. [19:57:13] :) [19:57:21] CindyCicaleseWMF: can you paste your curl command somewhere? [19:57:36] Sure. Here? [19:57:40] might be long here [19:57:43] pastebin or gist or something? [19:57:53] you can also paste the command you are using with kafka-tools [19:58:02] i usually use https://gist.github.com/ [19:58:37] CindyCicaleseWMF: and, yes, if those topics exist, it means that at least some events got through [19:58:39] https://pastebin.com/rJcVT9v0 [19:59:55] cool, yeah [19:59:56] And here's the kafka command: https://pastebin.com/h2LGcHbC [20:00:09] CindyCicaleseWMF: if you tail the processor logs you can see the validation error [20:00:18] tail -f tail -f /var/log/eventlogging/eventlogging-processor@client-side-0*.log [20:00:23] tail -f /var/log/eventlogging/eventlogging-processor@client-side-0*.log [20:00:24] then do your curl [20:00:49] 'country' is a required property [20:01:01] Ah, ok, so I removed two of the fields from the schema on meta. How do they get synced? [20:01:12] i think your curl most have the wrong revision in it [20:01:17] ah [20:01:25] 22revision%22%3A18340060 [20:01:32] revision 18340060 ? [20:02:15] Right! It should be 18341798 [20:04:01] I changed it and didn't see anything in the processor log . . . [20:05:22] but still nothing in the event log, either [20:06:25] CindyCicaleseWMF: i see it in kafka [20:06:30] kafka-tools -b deployment-kafka-jumbo-2.deployment-prep.eqiad.wmflabs:9092 consume_topic eventlogging_EUCCStats [20:06:32] woohoo! [20:06:34] at least, when i do it [20:06:35] :) [20:06:45] with that revision [20:07:10] yes, I did too! [20:07:36] hm there are no events in all-events .log [20:07:38] intersetring. [20:08:08] THERE THEY Go [20:08:11] whoa it was stuck [20:08:12] who knows why [20:08:19] beta ¯\_(ツ)_/¯ [20:08:26] lol - weird [20:09:36] OK, so I guess we can say that the schema is valid! [20:10:19] :D [20:10:25] Is the country information that nuria mentioned available on beta? Or only once I get access to hive? [20:10:31] hm, no we don't have hive in beta [20:10:40] there's a bunch of big cluster infrastructure and automation that do that in prod [20:10:58] but CindyCicaleseWMF...there's no reason you couldn't curl that against prod atm... [20:11:10] the event would be processed and validated in the same way in prod [20:11:17] but also be consumed into hive (about once an hour) [20:11:24] and geocoded along the way [20:12:01] cool - but I just requested hive access so won't get that for 3 days [20:12:35] but, I can check on that once I get access [20:13:04] so all events get geocoded or just page view events? [20:13:46] I think I just answered my own question, since she said to remove country from the event, so it must be geocoded. [20:17:25] Thank you for the help, ottomata! [20:19:32] ya the eventlogging events are geocoded [20:19:35] you are welcome glad its working! [20:19:50] sorry that process is so nasty, hopefully we'll get some of that fixed as part of the Modern Event Platform program over the next year [20:53:23] (03PS1) 10Joal: Update WKS2 endpoints to accept project families [analytics/aqs] - 10https://gerrit.wikimedia.org/r/456442 (https://phabricator.wikimedia.org/T188550) [20:54:26] (03CR) 10Joal: [V: 031] "Tested on existing data using druid-tunneling." [analytics/aqs] - 10https://gerrit.wikimedia.org/r/456442 (https://phabricator.wikimedia.org/T188550) (owner: 10Joal) [20:54:32] CindyCicaleseWMF: yes, all events get geocoded but it is for very few where that info is available beyond 90 days [20:54:43] Gone for tonight team - See you tomorrow :) [20:59:37] CindyCicaleseWMF: i am available if want to talk more [21:31:31] nuria: thanks! I've done as much as I can until it's morning for the devs in Germany, but it looks like we're in good shape now. I appreciate all of the help from your team, as always. [21:37:38] 10Analytics-Kanban, 10Patch-For-Review: Fix mediawiki-history-druid oozie job - https://phabricator.wikimedia.org/T201620 (10Nuria) [21:37:46] 10Analytics-Kanban, 10Patch-For-Review: Fix mediawiki-history-druid oozie job - https://phabricator.wikimedia.org/T201620 (10Nuria) 05Open>03Resolved [21:46:21] 10Analytics: Data Quality Alarms - https://phabricator.wikimedia.org/T198986 (10Nuria) [21:46:25] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: EventLogging in Hive data loss due to Camus and Kafka timestamp.type=CreateTime change - https://phabricator.wikimedia.org/T198906 (10Nuria) [21:46:31] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Wikimedia-Incident, 10cloud-services-team (Kanban): Alarms on throughput on camus imported data - https://phabricator.wikimedia.org/T198908 (10Nuria) 05Open>03Resolved [21:47:52] 10Analytics-Kanban, 10Outreach-Programs-Projects, 10Google-Summer-of-Code (2018): [Analytics] Improvements to Wikistats2 front-end - https://phabricator.wikimedia.org/T189210 (10Nuria) [21:47:54] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Improve Wikistats2 map zoom - https://phabricator.wikimedia.org/T198867 (10Nuria) 05Open>03Resolved [21:51:16] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: [Wikistats 2] Bug in time-range selector on detail page - https://phabricator.wikimedia.org/T200497 (10Nuria) 05Open>03Resolved [21:51:33] 10Analytics-Kanban: Update mediawiki-history-reduced datasource-name in druid to use underscore - https://phabricator.wikimedia.org/T202305 (10Nuria) 05Open>03Resolved [21:54:57] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, and 3 others: Redesign EventStreams for better multi-dc support - https://phabricator.wikimedia.org/T199433 (10Nuria) 05Open>03Resolved [21:55:10] 10Analytics-Kanban, 10Patch-For-Review: Deploy wikistats from master branch - https://phabricator.wikimedia.org/T203017 (10Nuria) 05Open>03Resolved [21:55:53] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: EventLogging sanitization - https://phabricator.wikimedia.org/T199898 (10Nuria) [21:56:00] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: [EL sanitization] Add ability to salt and hash to eventlogging sanitization in Hive - https://phabricator.wikimedia.org/T198426 (10Nuria) 05Open>03Resolved