[01:55:53] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Update wikimedia-history revision data with deleted field (and find it a new name?) - https://phabricator.wikimedia.org/T178587 (10Neil_P._Quinn_WMF) Thanks for getting back to me! 🙂 >>! In T178587#5017759, @Milimetric wrote: > Ok, here's what we think Ne... [05:13:39] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Update reportupdater to be able to query the new db cluster that will substitute 1002 - https://phabricator.wikimedia.org/T215289 (10chelsyx) > @chelsyx, Nuria told me you're working on a project named Toledo, that she thinks might provide similar stats th... [07:09:48] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Move AQS to nodejs 10 - https://phabricator.wikimedia.org/T210706 (10elukey) @Milimetric if the test went fine I'd say to proceed with production :) What I'd do is the following: - Merge https://gerrit.wikimedia.org/r/496110 (puppet) and run puppet on th... [10:38:18] (03PS1) 10GoranSMilovanovic: Sqoop re-factor [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/496140 [10:38:38] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] Sqoop re-factor [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/496140 (owner: 10GoranSMilovanovic) [10:40:45] (03PS1) 10GoranSMilovanovic: new Sqoop procedure [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/496141 [10:41:06] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] new Sqoop procedure [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/496141 (owner: 10GoranSMilovanovic) [10:46:51] * elukey afk for a bit [14:32:15] 10Analytics, 10Product-Analytics, 10Research, 10WMDE-Analytics-Engineering, and 2 others: Replace the current multisource analytics-store setup - https://phabricator.wikimedia.org/T172410 (10jcrespo) Can we resolve this already? I am guessing there may be many followups, but technically this has been done... [14:39:33] 10Analytics, 10Product-Analytics, 10Research, 10WMDE-Analytics-Engineering, and 2 others: Replace the current multisource analytics-store setup - https://phabricator.wikimedia.org/T172410 (10Marostegui) Fine by me [15:05:11] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Move AQS to nodejs 10 - https://phabricator.wikimedia.org/T210706 (10mforns) @elukey > we could do it one of these EU evenings? Sure! I'll be on vacation, though, starting this Fri 15th (included), and will be back on Mon 25th. [15:13:23] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Update reportupdater to be able to query the new db cluster that will substitute 1002 - https://phabricator.wikimedia.org/T215289 (10mforns) @chelsyx thanks for the clarification! @Amire80 it seems then, that we'll have to find other solutions... My only... [15:16:32] 10Analytics, 10Knowledge-Integrity, 10Research, 10Epic, 10Patch-For-Review: Citation Usage: run third round of data collection - https://phabricator.wikimedia.org/T213969 (10RyanSteinberg) Hi @leila. I don't think we ever collectively defined what an external link was in our schema. Using the `external`... [15:38:20] 10Analytics, 10Analytics-Data-Quality, 10Pageviews-Anomaly: Anomalous statistics results in eu.wikipedia siteviews - https://phabricator.wikimedia.org/T212879 (10MusikAnimal) [15:55:50] (03PS1) 10Milimetric: Update aqs to bad550b [analytics/aqs/deploy] - 10https://gerrit.wikimedia.org/r/496189 [16:01:12] a-Team: i am hosting an Irc meeting at this time today, will send e-scrum [16:01:19] k [16:23:07] (03CR) 10Ladsgroup: [C: 03+2] "Kicking jenkins" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/494190 (owner: 10Thiemo Kreuz (WMDE)) [16:31:59] ottomata: done with standup? [16:32:46] ottomata: now that we have api data flowing as json through event gate into kafka we will be able to refine it using our actual refine code , correct? [16:54:45] am I correct in assuming that https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Edits/Metrics will have 1 metric per day? [16:55:00] well, 1 metric, per wiki, per metric name, per day? [16:58:32] also, if there a way to figure out what the latest snapshot it? [16:58:33] *is [16:59:16] Hi addshore [16:59:43] addshore: hdfs dfs -ls /wmf/data/wmf/mediawiki/metrics [17:00:02] :D [17:00:07] thats one i really need to remember [17:00:24] addshore: as for the time granularity of the metrics, IIRC it's included in the metric name [17:00:26] I feel like whenever i mouse over "snapshot" on wiki it should tell me that [17:00:43] :) [17:01:14] im doing some crappy predictions about the growth of wikidata, and that table is super useful [17:01:26] :) [17:01:30] eg, we had 192,353,549 edits in 2017, lovely :D [17:01:41] In term of non-archived edits, wikidata has overgrown enwiki [17:01:48] I looked at that today [17:02:09] 819008874 edits for enwiki, 862862053 for wikidata [17:02:35] yupp [17:02:36] :P [17:02:42] its getting chubby [17:02:44] addshore: we should monitor this so that we party at 1G :) [17:03:14] It should be around december 2019 i guess [17:03:18] maybe a bit after [17:03:31] 2018 we have 208,944,716 edits [17:03:38] addshore: So much data human-created but machine-oriented drives me a in the middle of crazy-worry :) [17:13:05] addshore: also given the slope og growth and current number of edits, I actually think we'll get to 1G edits for wikidata around October ;)( [17:26:04] heya joal, one of yesterday's alerts has a row of true data loss in the results of the check_dataloss_false_positives.sparksql script, is this sth that can be fixed by rerunning the hour? [17:26:26] mforns: you can try, but I don't see why this would change :) [17:27:00] mforns: I'd rather try to check with elukey for instance if there had been any issue on varnish or something like that [17:28:12] yea makes sense [17:28:17] will wait for him [17:28:20] thx [17:29:38] mforns: what timeframe? text/upload? [17:37:14] milimetric: heya - would you have a minute? [17:39:29] * elukey off! [17:43:28] oops, lost him [17:47:18] (03PS1) 10Milimetric: Update aqs to bad550b [analytics/aqs/deploy] - 10https://gerrit.wikimedia.org/r/496214 [17:48:03] joal: hey, I've got 13 minutes [17:48:08] cave? [17:48:09] To the batcave ! [17:54:34] 10Analytics, 10EventBus, 10Operations, 10Core Platform Team (Modern Event Platform (TEC2)), and 3 others: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019. - https://phabricator.wikimedia.org/T217359 (10herron) [18:20:54] Hey. I'm investigating a bug that suggests quicksurveys do not display on mobile. @leila can you confirm one way or the other whether the previous survey you ran received responses from mobile? [18:21:16] isaacj: ^ [18:22:42] I'll check and get back on that shortly @jdlrobson [18:23:01] my concern is you've only displayed your last survey to desktop users [18:23:24] (desktop skin users) [18:24:23] luckily it was a pilot so if that's the case, good thing we tested :) [18:31:07] (03Abandoned) 10Milimetric: Update aqs to bad550b [analytics/aqs/deploy] - 10https://gerrit.wikimedia.org/r/496189 (owner: 10Milimetric) [18:35:41] @jdlrobson looking at the browser agents for people who saw the survey, Chrome tops out at 123K but Chrome Mobile is at 80K and Mobile Safari 78K so looks like was showing fine to mobile. [18:35:41] query: [18:35:41] select useragent.browser_family, count(distinct(event.surveysessiontoken)) as count FROM quicksurveyinitiation WHERE event.surveyCodeName = 'reader-demographics-en-pilot' AND event.eventname = "impression" and year = 2019 and month = 3 and (day = 4 or day = 5) GROUP BY useragent.browser_family ORDER BY count LIMIT 10000;) [18:36:11] cool. This matches what I'm seeing, but it looks like there is a problem in the code that means it won't show 100% of the time [18:36:26] so if you received less surveys from mobile than expected, this might be why [18:39:02] hmm...interesting - i guess we could compare those numbers against webrequest logs to see how skewed they are but they seem somewhat reasonable as far as desktop vs. mobile. for the responses too, we also see predominantly chrome mobile and mobile safari, so mobile users were definitely see the survey [18:40:29] i actually have been trying to figure out another potential bug, which is that about 10% of our survey responses don't seem to have any eventlogging associated with them. not sure what's going on with that though (no eventlogging so i have no good way to determine whether it's associated with a particular browser etc.) [18:41:58] isaacj: here's the bug i've discovered: https://phabricator.wikimedia.org/T218243 [18:42:31] maybe this relates to the 10% of survey responses without EventLogging [18:42:46] but it could also be doNotTrack [18:45:45] hmm...yeah. i'll follow that task. the doNotTrack should prevent the survey from being shown too though, right? so they wouldn't have been able to take the survey either [18:45:52] I don't think so [18:46:08] oh no you are right [18:46:10] ignore that theory :) [18:47:27] cool - i'm going to continue digging to see if my eventlogging is missing any major browsers in case it's browser-specific [19:02:34] hey hey a-team! [19:02:35] https://grafana.wikimedia.org/d/POYzU8rmz/eventgate-analytics?refresh=1m&orgId=1&from=1552501920268&to=1552503720269&var-dc=eqiad%20prometheus%2Fk8s&var-service=eventgate-analytics [19:02:47] 4K events / second right now! :D [19:03:07] O.o! [19:03:11] :D [19:03:12] ottomata: ooohhhhhh [19:03:15] The thing works [19:03:34] ottomata: did you see my question before about api events and persistence in hive? [19:03:50] scrolling! nuria was in meetings at the time scrolling! [19:04:25] nuria: yes [19:04:25] isaacj: to see whether surveys where displayed on mobile/desktop you can also look at the hostname [19:04:35] once we get the refinery schema stuff deployed [19:04:38] isaacj: it will be prefixed differently in either case [19:04:38] it needs sthat to do map types [19:04:41] milimetric: i get a 404 at https://hue.wikimedia.org/oozie/list_oozie_coordinator ? [19:04:49] planning on working on that aas soon as i get a chance [19:05:14] HaeB: FYI that is an interface to all our jobs , probably not the best place to find things easily [19:05:31] ottomata: ah i forgot, i knew there was something yes [19:05:35] ottomata: the map types [19:06:09] HaeB: sorry, https://hue.wikimedia.org/oozie/list_oozie_coordinators/ [19:06:14] (chopped off the s maybe) [19:06:46] EventGate is such a good name Andrew [19:07:11] nuria: its a better place to find jobs status than any other :) [19:07:25] thanks milimetric thanks to nuria for liking that one! [19:07:33] i dismissed it originally [19:07:48] milimetric: thanks! (added it to the notes) [19:08:05] thank you [19:08:41] nuria: yes, dan mentioned it as a general option to find the specific coordinator bearloga had been looking for [19:09:43] HaeB: sure, i mean "it is not the best place to look whether mediawiki history has completed" which was bearloga's original question [19:10:11] bearloga: i need to look into ticket but i think next snapshot shoudl send e-mail [19:11:31] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Update wikimedia-history revision data with deleted field (and find it a new name?) - https://phabricator.wikimedia.org/T178587 (10Milimetric) >>! In T178587#5019862, @Neil_P._Quinn_WMF wrote: > So overall, I really think we should go with: > * `page_is_de... [19:14:20] 10Analytics, 10Product-Analytics: Eventbus revisions are duplicated in event.mediawiki_revision_tags_change - https://phabricator.wikimedia.org/T218246 (10chelsyx) [19:16:33] btw this was already kind of documented on wikitech (https://wikitech.wikimedia.org/w/index.php?title=Analytics/Data_Lake/Edits/Mediawiki_history&diff=1761120&oldid=1761113 ) but it might be good to explain the email notification option there too [19:17:05] (bearloga ^) [19:20:10] nuria: good point -- didn't see webhost somehow when i first looked at the schema. 55% to en.m.wikipedia.org and almost all of the rest to en.wikipedia.org. mobile is overrepresented then in the actual responses (~66%) [19:21:36] notably, while chrome seems to be about 20% of traffic from that day, it's 34% of our survey initiations, so @jdlrobson is right in that it seems that mobile was being undersampled by some not in substantial amount [19:24:51] (03CR) 10Milimetric: [V: 03+2 C: 03+2] "merging to deploy" [analytics/aqs/deploy] - 10https://gerrit.wikimedia.org/r/496214 (owner: 10Milimetric) [20:33:52] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Move AQS to nodejs 10 - https://phabricator.wikimedia.org/T210706 (10Milimetric) Ok, everything looks good in the deployment-aqs cluster. Ready to deploy whenever yall want tomorrow. [20:36:54] 10Analytics, 10Core Platform Team Kanban, 10EventBus, 10Services (doing): EventBus extension should never log unserialized events - https://phabricator.wikimedia.org/T218254 (10Pchelolo) [20:47:51] 10Analytics, 10EventBus, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), 10Core Platform Team Kanban (Doing), 10Services (doing): EventBus extension should never log unserialized events - https://phabricator.wikimedia.org/T218254 (10mobrovac) [21:05:00] 10Analytics, 10EventBus, 10Core Platform Team Kanban (Doing), 10Services (doing): Decrease timeout for EventBus extension for analytics events - https://phabricator.wikimedia.org/T218260 (10Pchelolo) [21:05:53] 10Analytics, 10EventBus, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), 10Core Platform Team Kanban (Doing), and 2 others: EventBus extension should never log unserialized events - https://phabricator.wikimedia.org/T218254 (10Pchelolo) [21:06:01] 10Analytics, 10EventBus, 10Core Platform Team Kanban (Doing), 10Services (doing): Decrease timeout for EventBus extension for analytics events - https://phabricator.wikimedia.org/T218260 (10Pchelolo) [21:11:36] Gone for tonight team - See you tomorrow (AQS deploy I heard ?) [21:35:51] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Set up automated email to report completion of mediawiki_history snapshot and Druid loading - https://phabricator.wikimedia.org/T206894 (10Nuria) >Looking at the patch, it seems like it doesn't actually include our email address ( Th... [21:41:04] 10Analytics, 10EventBus, 10Operations, 10Prod-Kubernetes, 10serviceops: eventgate-analytics k8s pods occasionally can't produce to kafka - https://phabricator.wikimedia.org/T218268 (10Ottomata) [21:41:24] 10Analytics, 10EventBus, 10Operations, 10Prod-Kubernetes, 10serviceops: eventgate-analytics k8s pods occasionally can't produce to kafka - https://phabricator.wikimedia.org/T218268 (10Ottomata) @akosiaris let's try to figure this out tomorrow. :) [22:01:07] 10Analytics, 10EventBus, 10Core Platform Team Kanban (Doing), 10Services (doing): Decrease timeout for EventBus extension for analytics events - https://phabricator.wikimedia.org/T218260 (10mobrovac) It'd be awesome if we could target 1s, but let's perhaps start with a less-aggressive number, like 5 or 10... [22:05:07] (03CR) 10Mforns: [C: 04-1] "LRGTM overall!! I think I spotted one thing in the filtering of active editors (see inline comment). Will -1 just in case this is a bug, b" (036 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/494241 (https://phabricator.wikimedia.org/T187806) (owner: 10Fdans) [22:20:23] (03CR) 10Mforns: [C: 03+2] "LGTM! :]" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/485710 (https://phabricator.wikimedia.org/T213603) (owner: 10Joal) [22:24:42] (03Merged) 10jenkins-bot: Update delete/restore in mediawiki-history [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/485710 (https://phabricator.wikimedia.org/T213603) (owner: 10Joal) [22:29:35] (03Merged) 10jenkins-bot: Refactor mediawiki-history core data gathering [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/491494 (https://phabricator.wikimedia.org/T206883) (owner: 10Joal) [22:31:27] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Set up automated email to report completion of mediawiki_history snapshot and Druid loading - https://phabricator.wikimedia.org/T206894 (10Nuria) Ok, the workflow has the success e-mail: https://hue.wikimedia.org/oozie/list_oozie_coo... [22:36:08] mforns: still there? [22:36:15] nuria, yes [22:36:42] sup? [22:36:44] mforns: so i found out what was going on with the job that had to send data to product analysts, the e-mail was missspelled. [22:36:53] mforns: https://phabricator.wikimedia.org/T206894 [22:36:59] ooohhh [22:37:05] mforns: we can restart job , right? [22:38:00] mforns: i know how to kill jobs real well but I am not sure how to restart this one, do we kill the other one and just put -Demail=blah@.com? [22:38:44] nuria, I believe it will have a properties file no? [22:38:53] let me look for the code [22:39:33] mforns: no, in this case teh properties file doe snot list e-mail on purpose [22:39:41] mforns: it is an argument that you must pass [22:39:42] nuria, oh [22:39:51] mforns: this is to avoid accidental spamming [22:39:56] I see [22:40:01] rmrmber [22:40:11] *remember [22:43:08] mforns: so this coordinator needs to be restarted with the good -Demail value [22:43:20] nuria, yes, I believe that we can kill and then restart with the command specified in the properties file comments: oozie job -Duser=$USER -Dstart_time=2018-06-01T00:00Z -Dsuccess_email_to=YOUREMAIL@wikimedia.org -submit -config oozie/mediawiki/history/check_denormalize/coordinator.properties [22:43:49] I can try [22:44:20] mforns: it has to be hdfs right? [22:44:39] mforns: as that is the oozie directory you want to look under, [22:44:43] I guess so, wanna kill the job, and I relaunch it? [22:45:12] mforns: i will kill it and we can relaunch though command line cause i do not think there is a way to relaunch from UI right? [22:45:25] nuria, no, must be the command line [22:47:17] nuria, can you review the command? https://pastebin.com/J9p0FCWa [22:47:31] if you're ok, I can execute in an-coord1001 [22:47:39] mforns: let me KILL [22:48:28] mforns: ya, killed now [22:48:41] ok [22:48:50] !log killed oozie job 0131427-181112144035577-oozie-oozi-C to correct e-mail address [22:48:52] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:51:08] mforns: sounds good , just killed other job [22:51:26] nuria, I tried to launch it from stat1007 and have some problems with OOZIE_URL [22:54:45] mforns: batcave? [22:54:56] wait I think I got it [22:56:06] nuria, yes, seems to be running fine: https://hue.wikimedia.org/oozie/list_oozie_coordinator/0147256-181112144035577-oozie-oozi-C/ [22:56:24] mforns: what was the cmd? [22:56:43] I just added --oozie $OOZIE_URL [22:56:52] to the one in the pastebin [22:57:29] https://www.irccloud.com/pastebin/fUrp1Gha/ [22:57:50] mforns: i see, this has to be done from stats machines right not an-coord-1001 [22:58:05] yes [22:58:11] mforns: ok [22:58:32] I went first to an-coord, but then I saw I didn't have refinery there checked out, so went back to stat1007 [22:58:37] !log mediawiki-check denormalized restart ed 0147256-181112144035577-oozie-oozi-C [22:58:39] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:59:19] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Set up automated email to report completion of mediawiki_history snapshot and Druid loading - https://phabricator.wikimedia.org/T206894 (10Nuria) @mforns restarted job, this one shoudl have correct e-mail: https://hue.wikimedia.org/o...