[00:04:04] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service - https://phabricator.wikimedia.org/T201063 (10Neil_P._Quinn_WMF) >>! In T201063#4560280, @Tbayer wrote: > In any case, I have...
[00:20:25] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service - https://phabricator.wikimedia.org/T201063 (10Neil_P._Quinn_WMF) Actually, @Ottomata, I can think of additional use case: * /...
[00:33:16] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service - https://phabricator.wikimedia.org/T201063 (10Ottomata) Ya makes a lot of sense.  In addition to high level type documentatio...
[00:33:32] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service - https://phabricator.wikimedia.org/T201063 (10Ottomata)
[02:36:15] <wikibugs>	 (03PS13) 10Zhuyifei1999: Update dependencies [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/428140 (https://phabricator.wikimedia.org/T192731) (owner: 10Framawiki)
[02:36:17] <wikibugs>	 (03PS10) 10Zhuyifei1999: Port to Python3 [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/440007 (https://phabricator.wikimedia.org/T192698) (owner: 10Framawiki)
[02:36:19] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Port to Python3 [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/440007 (https://phabricator.wikimedia.org/T192698) (owner: 10Framawiki)
[02:38:00] <wikibugs>	 (03PS14) 10Zhuyifei1999: Update dependencies [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/428140 (https://phabricator.wikimedia.org/T192731) (owner: 10Framawiki)
[02:40:41] <wikibugs>	 (03PS11) 10Zhuyifei1999: Port to Python3 [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/440007 (https://phabricator.wikimedia.org/T192698) (owner: 10Framawiki)
[02:41:24] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Port to Python3 [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/440007 (https://phabricator.wikimedia.org/T192698) (owner: 10Framawiki)
[02:46:01] <wikibugs>	 (03CR) 10Zhuyifei1999: "I'm merging this patch into https://gerrit.wikimedia.org/r/#/c/analytics/quarry/web/+/440007/" [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/454079 (https://phabricator.wikimedia.org/T192698) (owner: 10Framawiki)
[02:46:03] <wikibugs>	 (03PS15) 10Zhuyifei1999: Update dependencies [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/428140 (https://phabricator.wikimedia.org/T192731) (owner: 10Framawiki)
[02:48:04] <wikibugs>	 (03PS16) 10Zhuyifei1999: Update dependencies [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/428140 (https://phabricator.wikimedia.org/T192731) (owner: 10Framawiki)
[02:55:28] <wikibugs>	 (03PS12) 10Zhuyifei1999: Port to Python3 [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/440007 (https://phabricator.wikimedia.org/T192698) (owner: 10Framawiki)
[02:58:41] <wikibugs>	 (03PS13) 10Zhuyifei1999: Port to Python3 [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/440007 (https://phabricator.wikimedia.org/T192698) (owner: 10Framawiki)
[02:59:23] <wikibugs>	 (03Abandoned) 10Zhuyifei1999: Update Vagrant files to py3 [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/454079 (https://phabricator.wikimedia.org/T192698) (owner: 10Framawiki)
[03:05:48] <wikibugs>	 (03PS17) 10Zhuyifei1999: Update dependencies [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/428140 (https://phabricator.wikimedia.org/T192731) (owner: 10Framawiki)
[03:06:57] <wikibugs>	 (03PS14) 10Zhuyifei1999: Port to Python3 [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/440007 (https://phabricator.wikimedia.org/T192698) (owner: 10Framawiki)
[03:30:14] <wikibugs>	 (03PS18) 10Zhuyifei1999: Update dependencies [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/428140 (https://phabricator.wikimedia.org/T192731) (owner: 10Framawiki)
[03:30:54] <wikibugs>	 (03PS15) 10Zhuyifei1999: Port to Python3 [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/440007 (https://phabricator.wikimedia.org/T192698) (owner: 10Framawiki)
[03:39:37] <wikibugs>	 10Analytics-Kanban: Make sampling by session more obvious in eventlogging module - https://phabricator.wikimedia.org/T203612 (10Nuria)
[04:04:53] <wikibugs>	 (03PS1) 10Zhuyifei1999: app.py: Load Redis for session from Connections [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/458351 (https://phabricator.wikimedia.org/T202588)
[04:16:02] <wikibugs>	 10Quarry, 10Patch-For-Review: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10zhuyifei1999) @Framawiki I think we can proceed with the migration (the test instance at https://quarry-dev.wmflabs.org/ shares NFS with the production instance so I'm too afraid to do any real testing ther...
[06:50:24] <elukey>	 morning!
[06:50:54] <elukey>	 joal: I forgot to tell you that yesterday I restarted oozie to pick up new smtp settings (localhost rather than smtp server directly)
[06:51:12] <elukey>	 that was requested by Infra Foundation to have some queueing etc..
[06:51:28] <elukey>	 it seems working fine but lemme know if anything looks weird
[07:17:50] <wikibugs>	 10Analytics, 10Analytics-Kanban: Replace the Analytics HDFS/Yarn masters (hardware refresh) - https://phabricator.wikimedia.org/T203635 (10elukey) p:05Triage>03Normal
[07:33:05] <joal>	 Hi elukey :)
[07:33:16] <joal>	 No problem, so far so good I'd say :)
[07:36:18] <joal>	 JEI
[07:41:08] <elukey>	 :D ??
[07:41:27] <joal>	 miss-window :)
[07:43:12] <elukey>	 I created https://phabricator.wikimedia.org/T203635 to track all of the thoughts about switching the hadoop masters
[07:43:22] <elukey>	 yesterday I had a chat with Andrew about doing it after the offsite
[07:43:26] <elukey>	 so we have time
[07:43:40] <joal>	 ack!
[07:43:46] <elukey>	 maybe the tuesday back to work? How does it sound?
[07:43:59] <elukey>	 Andrew will be afk that thu/fri
[07:44:16] <elukey>	 so I think the compromise between me and you being jetlagged and him being afk is good :D
[07:44:31] <elukey>	 we have two days of overlap
[07:44:38] <joal>	 :D
[07:44:46] <elukey>	 so if anything acts weirdly we can check all together
[07:48:15] <elukey>	 joal: that would be the 25th
[07:50:08] <wikibugs>	 10Analytics, 10Analytics-Kanban: Replace the Analytics HDFS/Yarn masters (hardware refresh) - https://phabricator.wikimedia.org/T203635 (10elukey)
[07:53:53] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 2 others: RFC: Modern Event Platform: Scalable Event Intake Service - https://phabricator.wikimedia.org/T201963 (10daniel) That's what I hoped, thanks for confirming!
[07:59:46] <elukey>	 joal: we are also close to get stat1007
[08:00:38] <elukey>	 so that eventually we'll free stat1005 and we'll start checking if that GPU can start crunching numbers rather than doing nothing :D
[08:46:31] <wikibugs>	 10Analytics, 10Analytics-Kanban: Replace the Analytics HDFS/Yarn masters (hardware refresh) - https://phabricator.wikimedia.org/T203635 (10elukey)
[09:18:29] <elukey>	 started a list of things to do in https://etherpad.wikimedia.org/p/analytics-swap-masters
[09:19:58] <elukey>	 the main issue though it that the druid clusters will likely not be happy about hadoop being down
[09:20:14] <elukey>	 and turnilo/superset will also not work
[09:20:23] <elukey>	 not a big deal, we'll have to announce this properly
[09:20:32] <elukey>	 but the procedure seems easy
[09:21:02] <elukey>	 in theory when everything is down (and puppet disabled), it is only a matter of configuring the new masters properly
[09:21:22] <elukey>	 copying /var/lib/hadoop/name from the old to the new masters
[09:21:26] <elukey>	 and then starting the daemons
[09:21:38] <elukey>	 puppet will take care of the rest when it will run on the workers
[10:18:54] <joal>	 a-team - With the workers home I actually need to spend time with them - I'm gonna take the day off, possibly work tonight when they're gonew
[10:19:17] <elukey>	 ack!
[10:19:22] * elukey will miss Joseph
[12:00:56] <wikibugs>	 10Analytics, 10Analytics-Kanban: Replace the Analytics HDFS/Yarn masters (hardware refresh) - https://phabricator.wikimedia.org/T203635 (10elukey)
[12:02:58] <wikibugs>	 10Analytics, 10Analytics-Kanban: Replace the Analytics HDFS/Yarn masters (hardware refresh) - https://phabricator.wikimedia.org/T203635 (10elukey) So I tested the procedure in labs, that worked fine with the only caveat that hadoop-hdfs-zkfc seems to store the names of the master nodes in zookeeper, under `/ha...
[12:09:48] <fdans>	 hellooooo elukey: whenever you have a moment do you wanna take a look at the geowiki numbers? it's just a confirmation :)
[12:12:21] <elukey>	 fdans: sure! is it ok in say 1h? I still need to have lunch, got carried out by hadoop :)
[12:12:50] <fdans>	 elukey: oh look at who is having lunch at 2pm!!!
[12:12:53] <fdans>	 VERY INTERESTING
[12:13:10] <fdans>	 elukey: have a nice lunch, see you in an hour :D
[12:13:11] <elukey>	 hahaha
[12:13:20] <elukey>	 THANK YOU
[12:13:22] <elukey>	 <3
[12:45:00] <fdans>	 elukey: you know what's up with the virtualpageview hourly job?
[12:54:50] <elukey>	 here I am
[12:55:19] <fdans>	 elukey: cave for 2min?
[12:58:10] <elukey>	 I am checking the job, do you want to discuss it or geoip?
[12:58:56] <elukey>	 so https://yarn.wikimedia.org/jobhistory/task/task_1531216937660_194961_m_000000 is the one failing
[12:59:25] <elukey>	 and surprise surprise
[12:59:28] <elukey>	 oozie_launcher_memory 256
[13:01:19] <fdans>	 elukey: yeah what i was suspecting :D
[13:01:24] <fdans>	 elukey: should I restart it?
[13:02:17] <elukey>	 yep, I have never touched a virtual page view coordinator, but I guess it is as simple as the webrequest one
[13:02:31] <fdans>	 elukey: restarting
[13:03:46] <fdans>	 !log restarted virtualpageview_hourly coordinator
[13:03:49] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:04:11] <fdans>	 cool, wanna do geowiki elukey ?
[13:04:59] <elukey>	 fdans: sure! 
[13:05:45] <fdans>	 elukey:  in tha cave!
[13:13:55] <elukey>	 ottomata: o/
[13:14:26] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Turn off old geowiki jobs - https://phabricator.wikimedia.org/T190059 (10fdans)
[13:15:19] <elukey>	 the procedure to swap the hadoop master nodes works
[13:15:23] <elukey>	 \o/
[13:16:37] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Turn off old geowiki jobs - https://phabricator.wikimedia.org/T190059 (10fdans) After extracting a sample ranging all wikipedias, through the whole timespan of geowiki from the static files in Thorium, me and @elukey verified that all values match in the a...
[13:17:01] <elukey>	 also I am about to reboot kafka-jumbo1001 :)
[13:18:31] <ottomata>	 o/ yeehaw elukey
[13:20:56] <wikibugs>	 10Analytics, 10Analytics-Kanban: Replace the Analytics HDFS/Yarn masters (hardware refresh) - https://phabricator.wikimedia.org/T203635 (10Ottomata) greaaaattttttttt
[13:22:55] <ottomata>	 nice procedure elukey, you da best
[13:23:53] <elukey>	 thanksss!! Still need to finish it, for example i think that druid might need some restart/care 
[13:24:00] <elukey>	 like the historical nodes
[13:30:07] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review: Eventlogging's processors stopped working - https://phabricator.wikimedia.org/T200630 (10elukey) a:05elukey>03None
[13:30:21] <ottomata>	 joal:  yt?
[13:31:32] <elukey>	 I think he is still afk, workers at home
[13:31:42] <elukey>	 "With the workers home I actually need to spend time with them - I'm gonna take the day off, possibly work tonight when they're gonew"
[13:35:10] <ottomata>	 ahh yar ok
[13:35:15] <ottomata>	 i have so many weird esoteric scala problems
[13:38:40] <chasemp>	 I read that as 'weird esoteric social problems' and I was about to come to your defense ottomata but scala yeah, ...can't argue that 
[13:38:53] * elukey hugs chasemp 
[13:42:21] <ottomata>	 haha
[13:45:26] <elukey>	 ottomata: kafka-jumbo1001 looks good, keep going with the reboots
[13:58:10] <wikibugs>	 10Analytics: Return to real time banner impressions in Druid - https://phabricator.wikimedia.org/T203669 (10Jseddon)
[13:58:58] <ottomata>	 +1
[14:48:14] <wikibugs>	 10Analytics, 10Analytics-Kanban: Reboot Analytics hosts for kernel security upgrades - https://phabricator.wikimedia.org/T203165 (10elukey)
[15:15:57] <wikibugs>	 10Analytics, 10Analytics-Kanban: Reboot Analytics hosts for kernel security upgrades - https://phabricator.wikimedia.org/T203165 (10elukey)
[15:16:09] <elukey>	 Kafka jumbo reboots completed
[15:16:15] <elukey>	 I also done kafka2001
[15:16:22] <elukey>	 and during the next days I'll do the rest
[15:16:55] <elukey>	 I am also planning tomorrow (early morning EU time) to drain the cluster for a bit to ease the master nodes + analytics1003's reboot
[15:20:54] <wikibugs>	 10Analytics-Kanban: Make sampling by session more obvious in eventlogging module - https://phabricator.wikimedia.org/T203612 (10Nuria) a:03Nuria
[15:21:27] <ottomata>	 coool
[15:41:03] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 2 others: RFC: Modern Event Platform: Schema Registry - https://phabricator.wikimedia.org/T201643 (10Ottomata) @daniel what's next for the RFC? How do I keep it moving?
[16:01:03] <nuria>	 a-team: holaaa standup
[16:01:25] <nuria>	 ping elukey milimetric 
[16:01:33] <nuria>	 ping joal
[16:02:13] <elukey>	 yep coming sorry
[16:02:20] <elukey>	 I was in another meeting
[16:21:32] <wikibugs>	 10Analytics: turnilo x axis improperly labeled - https://phabricator.wikimedia.org/T197276 (10atgo) Was just coming here to report this - this is a major impact to usefulness of Turnilo.
[16:23:43] <wikibugs>	 10Analytics: Turnilo has arrow (fullscreen?) element laid over it such that can't dig into dates along the x-axis - https://phabricator.wikimedia.org/T203691 (10atgo)
[16:29:45] <wikibugs>	 10Analytics: Turnilo has arrow (fullscreen?) element laid over it such that can't dig into dates along the x-axis - https://phabricator.wikimedia.org/T203691 (10Nuria) Never seen that before, what browser do you see this one, Chrome?
[16:33:36] <wikibugs>	 10Analytics: Turnilo has arrow (fullscreen?) element laid over it such that can't dig into dates along the x-axis - https://phabricator.wikimedia.org/T203691 (10atgo) Chrome Version 68.0.3440.106 (Official Build) (64-bit)  Ooh I just tried it in an incognito browser and it worked. Should I plan to use incognito...
[16:43:59] <wikibugs>	 10Analytics: Upgrade Hive to ≥1.3 or ≥2.1 - https://phabricator.wikimedia.org/T203498 (10fdans) p:05Triage>03Normal
[16:44:56] <wikibugs>	 10Analytics: Turnilo has arrow (fullscreen?) element laid over it such that can't dig into dates along the x-axis - https://phabricator.wikimedia.org/T203691 (10Nuria) Having never seen that before using same browser I think maybe a browser restart is in order? Let us know if you see it still after a restart
[16:45:13] <wikibugs>	 10Analytics: Turnilo has arrow (fullscreen?) element laid over it such that can't dig into dates along the x-axis - https://phabricator.wikimedia.org/T203691 (10fdans) p:05Triage>03Low
[16:45:39] <wikibugs>	 10Analytics: Return to real time banner impressions in Druid - https://phabricator.wikimedia.org/T203669 (10fdans) We're considering a new version of these jobs for Q3
[16:46:00] <wikibugs>	 10Analytics: Return to real time banner impressions in Druid - https://phabricator.wikimedia.org/T203669 (10fdans) p:05Triage>03Normal
[16:46:34] <wikibugs>	 10Analytics: Flip blacklist for MySQL eventlogging consumer to be a whilelist of allowed schemas - https://phabricator.wikimedia.org/T203596 (10fdans) p:05Triage>03High
[16:46:56] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: CentralNoticeImpression event schema too large for MySQL - https://phabricator.wikimedia.org/T203592 (10fdans) 05Open>03Resolved
[16:47:38] <wikibugs>	 10Analytics, 10Analytics-Dashiki, 10Analytics-Kanban, 10CX-analytics, 10Language-2018-July-September: Setup Config:Dashiki:CX2Translations as a public chart and update the Dashiki documentation accordingly - https://phabricator.wikimedia.org/T203516 (10fdans)
[16:50:04] <wikibugs>	 10Analytics: Update CDH to 6 - https://phabricator.wikimedia.org/T203693 (10fdans)
[16:50:28] <wikibugs>	 10Analytics: Upgrade Hive to ≥1.3 or ≥2.1 - https://phabricator.wikimedia.org/T203498 (10fdans)
[16:50:31] <wikibugs>	 10Analytics: Update CDH to 6 - https://phabricator.wikimedia.org/T203693 (10fdans)
[16:50:41] <wikibugs>	 10Analytics: Default hive table creation to parquet - needs hive 2.3.0 - https://phabricator.wikimedia.org/T168554 (10fdans)
[16:50:44] <wikibugs>	 10Analytics: Update CDH to 6 - https://phabricator.wikimedia.org/T203693 (10fdans)
[16:52:45] <wikibugs>	 10Analytics: Update CDH to 6 or alternatives - https://phabricator.wikimedia.org/T203693 (10fdans)
[16:54:56] <wikibugs>	 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: [Wikistats2] Broken down data with different time ranges for each line (or column set) breaks the chart - https://phabricator.wikimedia.org/T198630 (10fdans)
[16:58:17] <wikibugs>	 10Analytics, 10Analytics-Cluster, 10Discovery, 10Patch-For-Review: Migrate Mediawiki Monolog Kafka producer to Kafka Jumbo - https://phabricator.wikimedia.org/T188136 (10fdans)
[17:13:51] <fdans>	 sorry a-team my computer went completely crazytown
[17:35:40] <wikibugs>	 10Analytics: Turnilo has arrow (fullscreen?) element laid over it such that can't dig into dates along the x-axis - https://phabricator.wikimedia.org/T203691 (10atgo) 05Open>03Invalid Better with Chrome update and restart - closing for now. If it happens again I'll reopen. Thanks for the responsiveness.
[17:37:03] * elukey off!
[18:09:34] <wikibugs>	 10Analytics-Tech-community-metrics, 10Code-Health, 10Release-Engineering-Team (Kanban): Develop canonical/single record of origin, machine readable list of all repos deployed to WMF sites. - https://phabricator.wikimedia.org/T190891 (10Jrbranaa) @Aklapper, I've been spending a bit of time on this and one of...
[18:23:21] <joal>	 Heya ottomata - Scala time?
[18:26:54] <ottomata>	 heyaaa ya sure!
[19:01:36] <wikibugs>	 10Analytics, 10Pageviews-API, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs, 10iOS-app-v6.0.1-Walrus-On-A-Golf-Cart: Large increase on 404s from the Wikipedia IOS app - https://phabricator.wikimedia.org/T203688 (10bearND)
[19:11:11] <wikibugs>	 10Analytics, 10Pageviews-API, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs, 10iOS-app-v6.0.1-Walrus-On-A-Golf-Cart: Large increase on 404s from the Wikipedia IOS app - https://phabricator.wikimedia.org/T203688 (10JMinor) Hey, thanks for raising this. We're looking into it on the iOS client side, but a pre...
[19:11:15] <wikibugs>	 10Analytics, 10Pageviews-API, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs, 10iOS-app-v6.0.1-Walrus-On-A-Golf-Cart: Large increase on 404s from the Wikipedia IOS app - https://phabricator.wikimedia.org/T203688 (10Mhurd) I get the same response if I ask for data for `Cat` article with the same date for bot...
[20:13:57] <wikibugs>	 10Analytics, 10Pageviews-API, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs, 10iOS-app-v6.0.1-Walrus-On-A-Golf-Cart: Large increase on 404s from the Wikipedia IOS app - https://phabricator.wikimedia.org/T203688 (10bearND) @Mhurd The date in your request is too new. The daily results have not been  generate...
[20:26:03] <wikibugs>	 10Quarry, 10Patch-For-Review: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10zhuyifei1999) Scheduled for 7pm UTC next Wednesday
[22:15:40] <wikibugs>	 10Analytics, 10Pageviews-API, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs, 10iOS-app-v6.0.1-Walrus-On-A-Golf-Cart: Large increase on 404s from the Wikipedia IOS app - https://phabricator.wikimedia.org/T203688 (10JMinor) p:05Unbreak!>03High Given jcrespo comment: "These don't cause any issue in our in...