[00:03:57] <wikibugs>	 10Analytics, 10CirrusSearch, 10Discovery, 10EventBus, and 3 others: Exception thrown while running DataSender::sendData in cluster codfw: Data should be a Document, a Script or an array containing Documents and/or Scripts - https://phabricator.wikimedia.org/T191024#4099168 (10EBernhardson)
[00:04:13] <wikibugs>	 10Analytics, 10CirrusSearch, 10Discovery, 10EventBus, and 3 others: Exception thrown while running DataSender::sendData in cluster codfw: Data should be a Document, a Script or an array containing Documents and/or Scripts - https://phabricator.wikimedia.org/T191024#4091269 (10EBernhardson) a:03EBernhardson
[00:55:55] <wikibugs>	 10Analytics, 10Analytics-Dashiki: Paginate table-timeseries visualization - https://phabricator.wikimedia.org/T191270#4099283 (10Milimetric)
[01:15:50] <wikibugs>	 (03PS1) 10Milimetric: Fix the header while scrolling table-timeseries [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/423589 (https://phabricator.wikimedia.org/T189070)
[01:18:19] <wikibugs>	 10Analytics-Dashiki, 10Analytics-Kanban: Show most recent data in table-timeseries - https://phabricator.wikimedia.org/T191273#4099340 (10Milimetric)
[01:18:42] <wikibugs>	 (03PS1) 10Milimetric: Show most recent dates in table-timeseries [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/423590 (https://phabricator.wikimedia.org/T191273)
[01:19:20] <wikibugs>	 10Analytics-Dashiki, 10Analytics-Kanban, 10Patch-For-Review: Show most recent data in table-timeseries - https://phabricator.wikimedia.org/T191273#4099340 (10Milimetric) p:05Triage>03Normal
[01:20:19] <wikibugs>	 10Analytics-Dashiki, 10Analytics-Kanban, 10Patch-For-Review: Show most recent data in table-timeseries - https://phabricator.wikimedia.org/T191273#4099340 (10Milimetric) Deployed here: https://language-reportcard.wmflabs.org/interlanguage/
[01:20:44] <wikibugs>	 10Analytics, 10Analytics-Dashiki, 10Patch-For-Review: Make the header of table-timeseries tables fixed when vertically scrolling the table - https://phabricator.wikimedia.org/T189070#4030475 (10Milimetric) Deployed here: https://language-reportcard.wmflabs.org/interlanguage/
[01:22:14] <wikibugs>	 10Analytics-Dashiki, 10Analytics-Kanban, 10Patch-For-Review: Make the header of table-timeseries tables fixed when vertically scrolling the table - https://phabricator.wikimedia.org/T189070#4099363 (10Milimetric)
[01:22:29] <wikibugs>	 10Analytics-Dashiki, 10Analytics-Kanban, 10Patch-For-Review: Make the header of table-timeseries tables fixed when vertically scrolling the table - https://phabricator.wikimedia.org/T189070#4030475 (10Milimetric) p:05Triage>03Normal
[01:22:51] <wikibugs>	 (03CR) 10Milimetric: [V: 032 C: 032] Show most recent dates in table-timeseries [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/423590 (https://phabricator.wikimedia.org/T191273) (owner: 10Milimetric)
[01:23:01] <wikibugs>	 (03CR) 10Milimetric: [V: 032 C: 032] Fix the header while scrolling table-timeseries [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/423589 (https://phabricator.wikimedia.org/T189070) (owner: 10Milimetric)
[01:23:28] <wikibugs>	 10Analytics-Dashiki, 10Analytics-Kanban, 10Patch-For-Review: Make the header of table-timeseries tables fixed when vertically scrolling the table - https://phabricator.wikimedia.org/T189070#4099368 (10Milimetric) a:03Milimetric
[01:23:38] <wikibugs>	 10Analytics-Dashiki, 10Analytics-Kanban, 10Patch-For-Review: Show most recent data in table-timeseries - https://phabricator.wikimedia.org/T191273#4099370 (10Milimetric)
[01:38:10] <wikibugs>	 (03CR) 10Amitjoki: ">" (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423144 (https://phabricator.wikimedia.org/T182990) (owner: 10Amitjoki)
[01:50:53] <wikibugs>	 (03PS8) 10Amitjoki: Label map and top metrics with the month they belong to [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423144 (https://phabricator.wikimedia.org/T182990)
[03:59:53] <chelsyx>	 Hi! Can we access Spark via notebook1003? I can open a pyspark shell in the terminal on JupyterLab, but `SHOW DATABASES` returns 'default', instead of a list of databases on hadoop
[04:00:16] <chelsyx>	 Just wanna leave a message here when any of you have time to check. No rush! :)
[07:03:49] <elukey>	 morningggg
[07:04:15] <elukey>	 so if nobody opposes I'd roll restart the druid public zk cluster to apply prometheus monitoring
[07:04:23] <elukey>	 afaics no indexation is ongoing for mw history etc..
[07:04:43] <elukey>	 but as precautionary step I'll also roll restart overlord/middlemanagers
[07:32:55] <wikibugs>	 (03CR) 10Ladsgroup: [C: 032] "I have worked with this code." [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/423145 (https://phabricator.wikimedia.org/T191111) (owner: 10Jonas Kress (WMDE))
[07:33:01] <wikibugs>	 (03Merged) 10jenkins-bot: Add recent changes mobile edits [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/423145 (https://phabricator.wikimedia.org/T191111) (owner: 10Jonas Kress (WMDE))
[07:39:18] <wikibugs>	 (03PS1) 10Ladsgroup: Add recent changes mobile edits [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/423615 (https://phabricator.wikimedia.org/T191111)
[07:39:25] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add recent changes mobile edits [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/423615 (https://phabricator.wikimedia.org/T191111) (owner: 10Ladsgroup)
[07:39:30] <wikibugs>	 (03CR) 10Ladsgroup: [C: 032] Add recent changes mobile edits [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/423615 (https://phabricator.wikimedia.org/T191111) (owner: 10Ladsgroup)
[07:39:38] <wikibugs>	 (03Merged) 10jenkins-bot: Add recent changes mobile edits [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/423615 (https://phabricator.wikimedia.org/T191111) (owner: 10Ladsgroup)
[07:39:53] <wikibugs>	 (03CR) 10Ladsgroup: [C: 032] "recheck" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/423615 (https://phabricator.wikimedia.org/T191111) (owner: 10Ladsgroup)
[07:40:14] <wikibugs>	 10Analytics, 10EventBus, 10JobRunner-Service, 10MediaWiki-Database, and 2 others: Wikimedia\Rdbms\LoadBalancer::{closure}: found writes pending - https://phabricator.wikimedia.org/T191282#4099795 (10jcrespo)
[07:48:32] <wikibugs>	 (03CR) 10Addshore: Add recent changes mobile edits (031 comment) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/423145 (https://phabricator.wikimedia.org/T191111) (owner: 10Jonas Kress (WMDE))
[07:49:19] <elukey>	 and hdfs trash is now configurable via hiera!
[07:49:27] <elukey>	 will wait for standup to decide values
[07:49:39] <elukey>	 (it involves a hdfs namenode restart of course)
[07:51:11] <elukey>	 I am rolling restarting zk on druid100[456]
[08:31:04] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Add the prometheus jmx exporter to all the Zookeeper daemons - https://phabricator.wikimedia.org/T177460#4099924 (10elukey)
[08:31:23] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Add the prometheus jmx exporter to all the Zookeeper daemons - https://phabricator.wikimedia.org/T177460#3659754 (10elukey)
[10:24:30] * elukey lunch + errand, bb in ~2h
[11:08:09] <mforns>	 hi team :]
[11:08:14] <joal>	 Hey mforns :)
[11:08:28] <mforns>	 hieee
[11:19:37] <wikibugs>	 (03CR) 10Joal: [C: 031] "Looks good to me. Shall we merge that?" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/423064 (https://phabricator.wikimedia.org/T191022) (owner: 10Nuria)
[11:23:34] <wikibugs>	 10Analytics, 10EventBus, 10JobRunner-Service, 10MediaWiki-Database, and 2 others: Wikimedia\Rdbms\LoadBalancer::{closure}: found writes pending - https://phabricator.wikimedia.org/T191282#4100210 (10jcrespo)
[12:34:37] <wikibugs>	 10Analytics-Dashiki, 10Analytics-Kanban, 10Patch-For-Review: Make the header of table-timeseries tables fixed when vertically scrolling the table - https://phabricator.wikimedia.org/T189070#4100414 (10CCicalese_WMF) 05Open>03Resolved Nice! Thank you!
[12:45:28] <wikibugs>	 (03PS1) 10Joal: Add PageviewTagger to webrequest [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/423679
[12:48:08] <wikibugs>	 (03PS2) 10Joal: Split webrequest into smaller datasets [analytics/refinery] - 10https://gerrit.wikimedia.org/r/357814 (https://phabricator.wikimedia.org/T164020)
[13:39:04] <wikibugs>	 (03CR) 10Ottomata: "Hm, I'm still not sure about this 'split' name." (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/357814 (https://phabricator.wikimedia.org/T164020) (owner: 10Joal)
[13:40:09] <wikibugs>	 (03CR) 10Ottomata: [C: 031] Add PageviewTagger to webrequest [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/423679 (owner: 10Joal)
[13:48:03] <ottomata>	 !log re-enable job queue topic mirroring from main -> eqiad
[13:48:05] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:49:57] <wikibugs>	 10Analytics, 10Operations, 10Ops-Access-Requests, 10Research, and 2 others: Restricting access for a collaboration nearing completion - https://phabricator.wikimedia.org/T189341#4100549 (10herron) p:05Triage>03Normal a:05DarTar>03herron
[13:51:37] <ottomata>	 elukey:  woah, offsets were present for job topics, its trying to restart them from last week :o eeek...
[13:52:13] <elukey>	 uffff
[13:55:20] <wikibugs>	 10Analytics, 10EventBus, 10JobRunner-Service, 10MediaWiki-Database, and 2 others: Wikimedia\Rdbms\LoadBalancer::{closure}: found writes pending - https://phabricator.wikimedia.org/T191282#4100576 (10Pchelolo) Looking at the logs there're 2 sources of these:  1. First one with url `/rpc/RunSingleJob.php` is...
[13:55:54] <icinga-wm>	 PROBLEM - Kafka MirrorMaker main-eqiad_to_jumbo-eqiad dropped message count in last 30m on einsteinium is CRITICAL: CRITICAL - scalar(sum(increase(kafka_tools_MirrorMaker_MirrorMaker_numDroppedMessages{mirror_name=main-eqiad_to_jumbo-eqiad} [30m]))): 62389.54127268165 1000.0 https://grafana.wikimedia.org/dashboard/db/kafka-mirrormaker?var-datasource=eqiad%2520prometheus%252Fops&var-mirror_name=main-eqiad_to_jumbo-eqiad
[13:56:03] <icinga-wm>	 PROBLEM - Kafka MirrorMaker main-eqiad_to_jumbo-eqiad max lag in last 10 minutes on einsteinium is CRITICAL: CRITICAL - scalar(max(max_over_time(kafka_burrow_partition_lag{group=kafka-mirror-main-eqiad_to_jumbo-eqiad} [10m]))): 108507431.0 100000.0 https://grafana.wikimedia.org/dashboard/db/kafka-mirrormaker?var-datasource=eqiad%2520prometheus%252Fops&var-mirror_name=main-eqiad_to_jumbo-eqiad
[13:57:41] <ottomata>	 yar i can't reset offsets in this version of mirror maker
[13:57:42] <ottomata>	 GAHHH!
[13:58:15] <ottomata>	 siliencing, gonna be alerting for a while while i fix
[14:01:35] <wikibugs>	 10Analytics, 10EventBus, 10JobRunner-Service, 10MediaWiki-Database, and 2 others: Wikimedia\Rdbms\LoadBalancer::{closure}: found writes pending - https://phabricator.wikimedia.org/T191282#4100586 (10Pchelolo) Also, this first ever started happening with a lower rate on Mar 28 19:07 which correlates exactly...
[14:01:39] <wikibugs>	 10Analytics, 10EventBus, 10JobRunner-Service, 10MediaWiki-Database, and 2 others: Wikimedia\Rdbms\LoadBalancer::{closure}: found writes pending - https://phabricator.wikimedia.org/T191282#4100587 (10jcrespo) Sorry to add Services, but you would be able, as you just did, to do a better triage than I did.
[14:05:14] <wikibugs>	 (03CR) 10Mforns: [C: 04-1] "Thanks for the change Amit," (038 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423144 (https://phabricator.wikimedia.org/T182990) (owner: 10Amitjoki)
[14:08:06] <wikibugs>	 10Analytics, 10CirrusSearch, 10Discovery, 10EventBus, and 3 others: Exception thrown while running DataSender::sendData in cluster codfw: Data should be a Document, a Script or an array containing Documents and/or Scripts - https://phabricator.wikimedia.org/T191024#4100609 (10Pchelolo)
[14:08:26] <wikibugs>	 10Analytics, 10EventBus, 10JobRunner-Service, 10MediaWiki-Database, and 2 others: Wikimedia\Rdbms\LoadBalancer::{closure}: found writes pending - https://phabricator.wikimedia.org/T191282#4100611 (10mmodell) I think it's time to roll-back and abandon 1.31.0-wmf.27. I really don't see how 1.31.0-wmf.28 will...
[14:08:32] <wikibugs>	 10Analytics, 10Analytics-Wikistats, 10Easy, 10Patch-For-Review: [Wikistats2] The detail page for tops and maps metrics does not indicate time range - https://phabricator.wikimedia.org/T182990#4100612 (10mforns) Please, @Amitjoki, do assign yourself to tasks you're working on. Even if anyone can contribute...
[14:15:08] <wikibugs>	 10Analytics, 10EventBus, 10JobRunner-Service, 10MediaWiki-Database, and 2 others: Wikimedia\Rdbms\LoadBalancer::{closure}: found writes pending - https://phabricator.wikimedia.org/T191282#4100642 (10Pchelolo) Edited https://phabricator.wikimedia.org/T191282#4100576 to be more precise on the timings of the...
[14:22:19] <wikibugs>	 (03CR) 10Mforns: [V: 031 C: 031] "Code looks great to me." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423172 (https://phabricator.wikimedia.org/T191121) (owner: 10Sahil505)
[14:23:14] <icinga-wm>	 RECOVERY - Kafka MirrorMaker main-eqiad_to_jumbo-eqiad dropped message count in last 30m on einsteinium is OK: OK - scalar(sum(increase(kafka_tools_MirrorMaker_MirrorMaker_numDroppedMessages{mirror_name=main-eqiad_to_jumbo-eqiad} [30m]))) within thresholds https://grafana.wikimedia.org/dashboard/db/kafka-mirrormaker?var-datasource=eqiad%2520prometheus%252Fops&var-mirror_name=main-eqiad_to_jumbo-eqiad
[14:24:09] <wikibugs>	 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Page views and Country name table columns overlapping in the Page Views By Country metric on Dashboard - https://phabricator.wikimedia.org/T191121#4100700 (10mforns) I CR'd the code and looks good!  I think we should test on different browsers before me...
[14:33:22] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Adding ranks to the map tooltip - https://phabricator.wikimedia.org/T191141#4100708 (10mforns) @Amitjoki   > it would be a good addition for users who want to see how countries fare with respect to other countries. I think the "ranking" use case is thoroughly covered by the...
[14:44:47] <wikibugs>	 (03CR) 10Mforns: [V: 032 C: 031] "Code looks great :]" (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422438 (owner: 10Amitjoki)
[14:50:01] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Improve scoping of CSS - https://phabricator.wikimedia.org/T190915#4100748 (10mforns) @sahil505 I think that would be indeed a good sub-task for T189210! I others are not opposed, I'll make this a sub-task of the GSoC project. @fdans, @milimetric, @nuria?
[14:55:22] <wikibugs>	 (03CR) 10Amitjoki: ">" (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422438 (owner: 10Amitjoki)
[14:58:02] <joal>	 fighting with connection, trying to join 
[14:58:25] <wikibugs>	 (03PS4) 10Amitjoki: Limit pan in Wikistats2 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422438 (https://phabricator.wikimedia.org/T189195)
[15:00:03] <wikibugs>	 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Display of radio buttons in Wikistats 2 is somewhat confusing - https://phabricator.wikimedia.org/T183185#3846152 (10mforns) Thanks @Amitjoki for taking this task. I think that one of the major concerns with this switch-on-off breakdowns, is that the lo...
[15:01:02] <wikibugs>	 (03CR) 10Mforns: [C: 04-1] "I think this needs more changes, see the discussion in the phab task. Cheers!" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422407 (https://phabricator.wikimedia.org/T183185) (owner: 10Amitjoki)
[15:01:11] <elukey>	 a-team: standupppp
[15:01:17] <nuria_>	 ottomata, mforns ping
[15:01:25] <mforns>	 trying to join!
[15:02:30] <nuria_>	 ping ottomata 
[15:06:18] <wikibugs>	 10Analytics, 10Operations, 10Ops-Access-Requests, 10Research, and 3 others: Restricting access for a collaboration nearing completion - https://phabricator.wikimedia.org/T189341#4100822 (10herron) @DarTar @Ottomata could you please review patch 423711?  It appears these users are already members of `statis...
[15:08:47] <joal>	 I'm BACK !
[15:09:00] <ottomata>	 ACKKK
[15:09:49] <icinga-wm>	 RECOVERY - Kafka MirrorMaker main-eqiad_to_jumbo-eqiad max lag in last 10 minutes on einsteinium is OK: OK - scalar(max(max_over_time(kafka_burrow_partition_lag{group=kafka-mirror-main-eqiad_to_jumbo-eqiad} [10m]))) within thresholds https://grafana.wikimedia.org/dashboard/db/kafka-mirrormaker?var-datasource=eqiad%2520prometheus%252Fops&var-mirror_name=main-eqiad_to_jumbo-eqiad
[15:09:58] <wikibugs>	 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Display of radio buttons in Wikistats 2 is somewhat confusing - https://phabricator.wikimedia.org/T183185#4100826 (10Amitjoki) @mforns I guess the confusion stems from this "on-off" switch mechanism. Please see https://semantic-ui.com/modules/checkbox.h...
[15:18:16] <nuria_>	 joal: let's try appear in 
[15:18:20] <nuria_>	 https://appear.in/wmf-analytics
[15:18:26] <nuria_>	 cc elukey , mforns ottomata 
[15:19:18] <mforns>	 a-team, room is full :'(
[15:19:24] <mforns>	 only 4 people
[15:20:06] <mforns>	 allowed
[15:20:13] <nuria_>	 mforns: back to hangouts
[15:20:21] <joal>	 trying audio-only
[15:20:26] <joal>	 sorry folks :(
[15:20:50] <nuria_>	 ottomata: hangouts works for  you, right?
[15:25:52] <elukey>	 !log killed a jvm belonging to hdfs-balancer stuck from march 9th
[15:25:54] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[15:26:12] <elukey>	 !log manually run hdfs balancer on an1003 (tmux session)
[15:26:13] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[16:06:48] <wikibugs>	 (03PS3) 10Joal: Split webrequest into smaller datasets [analytics/refinery] - 10https://gerrit.wikimedia.org/r/357814 (https://phabricator.wikimedia.org/T164020)
[16:06:57] <wikibugs>	 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: CirrusSearchCheckerJob should have a title - https://phabricator.wikimedia.org/T190958#4101062 (10Pchelolo) I can see the patch was SWATed and reverted. Was something wrong with it?
[16:07:38] <ottomata>	 elukey:  have a few mins for a mirror maker brain bounce?
[16:07:59] <elukey>	 sure
[16:08:19] <wikibugs>	 (03CR) 10Joal: "@ottomata: From the list you suggested, I prefer split, but I have no strong opinion as of the name. If the team prefers something else, l" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/357814 (https://phabricator.wikimedia.org/T164020) (owner: 10Joal)
[16:10:33] <ottomata>	 in bc
[16:11:25] <wikibugs>	 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: CirrusSearchCheckerJob should have a title - https://phabricator.wikimedia.org/T190958#4101072 (10dcausse) No it was due to a scap failure (T191029) so I had to revert to keep tin in sync with what's deployed.
[16:11:52] <wikibugs>	 10Analytics, 10Code-Stewardship-Reviews, 10Operations, 10Tools, 10Wikimedia-IRC-RC-Server: IRC RecentChanges feed: code stewardship request - https://phabricator.wikimedia.org/T185319#4101074 (10Nuria) Analytics agrees to be stewards of this service once it is migrated to be on top of  akafka stream, cc...
[16:20:31] <wikibugs>	 10Analytics, 10Analytics-Wikistats, 10Easy, 10Patch-For-Review: [Wikistats2] The detail page for tops and maps metrics does not indicate time range - https://phabricator.wikimedia.org/T182990#4101131 (10sahil505) @mforns : No worries at all :-) I'll find other tasks to work on. Removing myself from 'Assign...
[16:20:49] <wikibugs>	 10Analytics, 10Analytics-Wikistats, 10Easy, 10Patch-For-Review: [Wikistats2] The detail page for tops and maps metrics does not indicate time range - https://phabricator.wikimedia.org/T182990#4101132 (10sahil505) a:05sahil505>03None
[16:24:27] <wikibugs>	 (03CR) 10Sahil505: "Thanks for the review Marcel :-) I'll test on different browsers and update here as soon as I can. Thanks." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423172 (https://phabricator.wikimedia.org/T191121) (owner: 10Sahil505)
[16:24:40] <joal>	 Going for diner - back in a bit
[16:45:41] <wikibugs>	 10Analytics, 10Analytics-Wikistats, 10Easy, 10Patch-For-Review: [Wikistats2] The detail page for tops and maps metrics does not indicate time range - https://phabricator.wikimedia.org/T182990#4101231 (10Amitjoki) @sahil505 and @mforns sorry! I'll make sure to let it be known if I am working on a task. Sorr...
[16:45:58] <wikibugs>	 10Analytics, 10Analytics-Wikistats, 10Easy, 10Patch-For-Review: [Wikistats2] The detail page for tops and maps metrics does not indicate time range - https://phabricator.wikimedia.org/T182990#4101232 (10Amitjoki) a:03Amitjoki
[16:46:48] <wikibugs>	 10Analytics, 10Code-Stewardship-Reviews, 10Operations, 10Tools, 10Wikimedia-IRC-RC-Server: IRC RecentChanges feed: code stewardship request - https://phabricator.wikimedia.org/T185319#4101234 (10greg) >>! In T185319#4101074, @Nuria wrote: > once it is migrated to be on top of  akafka stream  Great! But w...
[16:48:28] <wikibugs>	 10Analytics, 10Code-Stewardship-Reviews, 10Operations, 10Tools, 10Wikimedia-IRC-RC-Server: IRC RecentChanges feed: code stewardship request - https://phabricator.wikimedia.org/T185319#4101252 (10Nuria) One of the analytics engineers.
[16:56:18] <wikibugs>	 (03CR) 10Mforns: [V: 032 C: 032] "LGTM!" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422438 (https://phabricator.wikimedia.org/T189195) (owner: 10Amitjoki)
[16:58:47] <wikibugs>	 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: CirrusSearchCheckerJob should have a title - https://phabricator.wikimedia.org/T190958#4101310 (10debt)
[16:59:48] <wikibugs>	 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: CirrusSearchCheckerJob should have a title - https://phabricator.wikimedia.org/T190958#4089095 (10debt)
[17:05:58] <wikibugs>	 10Analytics, 10Code-Stewardship-Reviews, 10Operations, 10Tools, 10Wikimedia-IRC-RC-Server: IRC RecentChanges feed: code stewardship request - https://phabricator.wikimedia.org/T185319#4101328 (10greg) 05Open>03Resolved a:03Nuria Awesome!  https://www.mediawiki.org/w/index.php?title=Developers%2FMai...
[17:24:10] <ottomata>	 ohh elukey v relevant i just learned
[17:24:13] <ottomata>	 batch.size is in bytes
[17:24:15] <ottomata>	 not messages....
[17:26:03] <elukey>	 ah!
[17:29:22] <ottomata>	 assuming upwards of 400 msgs / sec and an upper bound of 5K per message
[17:29:46] <ottomata>	 since a single batch would only fit 3 or so messages
[17:29:54] <ottomata>	 buffer memroy is 32M
[17:30:06] <ottomata>	 with a 16K batch size, that's only 2048 batches in memory
[17:36:02] <wikibugs>	 (03CR) 10Nuria: [C: 032] Add wikidata tag to webrequest refine process [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/423064 (https://phabricator.wikimedia.org/T191022) (owner: 10Nuria)
[17:38:59] <wikibugs>	 (03CR) 10Mforns: [C: 032] "LGTM!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/415255 (https://phabricator.wikimedia.org/T155507) (owner: 10Joal)
[17:47:54] <wikibugs>	 (03PS9) 10Amitjoki: [WIP] Label map and top metrics with the month they belong to [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423144 (https://phabricator.wikimedia.org/T182990)
[17:48:17] <elukey>	 ottomata: if you are ok I'd go offline to commute home, if you want I can check later for code reviews!
[17:50:06] <wikibugs>	 (03CR) 10Amitjoki: "> Uploaded patch set 9." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423144 (https://phabricator.wikimedia.org/T182990) (owner: 10Amitjoki)
[17:50:48] <wikibugs>	 (03CR) 10Nuria: "I think we should throughly test this in several wikis for several tables to make sure it is feasible. I can see how different mysql types" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/423235 (owner: 10Joal)
[17:50:57] <ottomata>	 ok cool, elukey i'm likely going to just try some things, i'm reasoning a bit about batching ,etc. but i'm not totally sure it all makes sense
[17:51:07] <ottomata>	 elukey:  quick bb before you go?
[17:51:28] <elukey>	 ottomata: sure!
[17:52:20] <ottomata>	 k in bc
[17:52:39] <wikibugs>	 (03CR) 10Amitjoki: "> > Uploaded patch set 9." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423144 (https://phabricator.wikimedia.org/T182990) (owner: 10Amitjoki)
[17:55:55] <wikibugs>	 (03CR) 10Joal: [V: 031] "Will do a full sqoop test after beginning-of-month jobs finish, in order to test perf improvement in mediawiki-history computatation." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/423235 (owner: 10Joal)
[18:24:13] <wikibugs>	 (03CR) 10Mforns: [WIP] Label map and top metrics with the month they belong to (038 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423144 (https://phabricator.wikimedia.org/T182990) (owner: 10Amitjoki)
[18:35:15] * elukey off!
[18:43:22] <wikibugs>	 (03CR) 10Mforns: "Code looks great!" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423351 (https://phabricator.wikimedia.org/T188268) (owner: 10Sahil505)
[18:44:20] <wikibugs>	 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Goal, and 2 others: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus - https://phabricator.wikimedia.org/T190327#4070155 (10awight) Hi, it looks like these changes knocked out the ORES job.  We need some help restoring service...
[18:46:21] <ottomata>	 !log restart main -> jumbo MirrorMaker with  request.timeout.ms = 2 minutes
[18:46:22] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[18:49:02] <wikibugs>	 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Goal, and 2 others: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus - https://phabricator.wikimedia.org/T190327#4101750 (10awight) We're now thinking that the job might be running correctly, but the metric may have changed....
[18:56:00] <wikibugs>	 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Goal, and 2 others: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus - https://phabricator.wikimedia.org/T190327#4070155 (10Pchelolo) @awight oh I didn't know you've got your own metric for it. I've updated the metric, the job...
[18:56:02] <wikibugs>	 (03CR) 10Mforns: [C: 031] "Code LGTM! Just left a typo comment." (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/423235 (owner: 10Joal)
[18:59:34] <wikibugs>	 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Goal, and 2 others: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus - https://phabricator.wikimedia.org/T190327#4101822 (10awight) @Pchelolo Thank you!
[19:22:51] <ottomata>	 !log bouncing main -> jumbo MirrorMaker setting session.timeout.ms = 125000
[19:22:53] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[19:32:55] <ottomata>	 !log bouncing main -> jumbo MirrorMaker unsetting http://session.timeout.ms/, this has a restiction on the broker in 0.9 :(
[19:32:57] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[19:37:01] <wikibugs>	 (03CR) 10Nuria: [V: 031 C: 031] "Looks good, tests run clean." (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/423679 (owner: 10Joal)
[19:49:13] <nuria_>	 joal: yt?
[19:49:16] <joal>	 yup
[19:50:16] <joal>	 nuria_: anything I can help with?
[19:51:00] <nuria_>	 joal: the filters on superset geowiki dashboard are kaput
[19:51:52] <joal>	 hm
[19:51:55] <wikibugs>	 (03PS1) 10Sahil505: Corrected kebab case - key name mismatch of metrics in topic selector [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423772 (https://phabricator.wikimedia.org/T188268)
[19:51:57] <wikibugs>	 (03PS2) 10Joal: Add PageviewTagger to webrequest [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/423679 (https://phabricator.wikimedia.org/T191022)
[19:53:45] <wikibugs>	 (03Abandoned) 10Sahil505: Corrected kebab case - key name mismatch of metrics in topic selector [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423351 (https://phabricator.wikimedia.org/T188268) (owner: 10Sahil505)
[19:54:09] <joal>	 fixed nuria_ 
[19:54:23] <nuria_>	 joal:what was the problem?
[19:54:40] <joal>	 nuria_: even for filters, you need to specify the data period you look at
[19:54:50] <joal>	 nuria_: period was last 60 days
[19:54:57] <joal>	 And there is no data for last 60 days
[19:55:20] <joal>	 At monthly agregation level
[19:55:22] <nuria_>	 joal:  ah , cause it is monthly it broke the 2nd of march right?
[19:55:28] <joal>	 correct
[19:55:32] <joal>	 maybe even the 1st
[19:55:35] <nuria_>	 joal: ok, i see
[19:55:50] <joal>	 I updated the filter to get 90 days
[19:56:28] <joal>	 nuria_: We're gonna have the same problem with other tiles, or show old data
[19:56:43] <joal>	 For instance the map is data about 3month to 2month
[19:56:47] <nuria_>	 joal: also there is no way now in dashboard to select a month
[19:57:15] <nuria_>	 joal: nor to know what month of data you are looking at
[19:57:42] <nuria_>	 joal: now we have two points right? jan and february monthly agreggations correct?
[19:58:44] <joal>	 nuria_: updated the dashboard with time selectors
[19:59:39] <nuria_>	 joal: i will try to work on it some more so anyone that lands there knows what data is about 
[20:00:12] <joal>	 nuria_: One important thing to add is to always select month relative in dashboard time selector, since data is monthly
[20:00:28] <joal>	 But now you can get data for more than a single month at a time :)
[20:01:39] <nuria_>	 joal: but how does user know that is what is being displayed?
[20:01:55] <joal>	 nuria_: look at dates in filter
[20:02:58] <nuria_>	 joal: it is not intuitive enough i think, if you select "2 months ago" you will get no data
[20:03:15] <joal>	 nuria_: as of today no, data is not yet available
[20:03:28] <joal>	 nuria_: once the jobs will be done, data should be available
[20:03:44] <nuria_>	 joal: there is data  for jan and feb though so 2 months ago
[20:04:00] <joal>	 nuria_: 2 fields: from X to Y
[20:04:05] <nuria_>	 joal: makes total sense it would show data  for february no?
[20:05:51] <joal>	 nuria_: data for feb is there (using real dates work
[20:06:06] <joal>	 I think when using "month ago", it ounts current month as 1
[20:06:20] <joal>	 So 2 month ago is actually beginning of march
[20:06:43] <joal>	 Or more precisely: 2 month ago is the 3rd of feb
[20:06:54] <joal>	 And data is loaded with date of the 1st
[20:06:59] <joal>	 so no data for 2 month ago
[20:07:02] <joal>	 nuria_: --^
[20:07:25] <joal>	 Using fixed dates makes more sense
[20:07:53] <joal>	 I saved the dashboard with a version with fixed dates
[20:07:57] <wikibugs>	 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Fix Mirror Maker erratic behavior when replicating from main-eqiad to jumbo - https://phabricator.wikimedia.org/T189464#4102048 (10Ottomata) @elukey here are some things I've learned, mostly from From https://cwiki.a...
[20:08:24] <nuria_>	 joal:  right ,cause the monthy cuts make sense "computationally" but it is not going to make sense to an end user that does  not know how is data computed
[20:08:27] <wikibugs>	 (03CR) 10Sahil505: "@mforns - there might be commit issue with this one as well. Please let me know if there is. Something went wrong on my computer. Trying t" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423772 (https://phabricator.wikimedia.org/T188268) (owner: 10Sahil505)
[20:08:35] <joal>	 I hear that nuria_ 
[20:08:43] <joal>	 We should direct our users to use fixed dates
[20:08:47] <nuria_>	 joal: let's talk in batcave for a sec
[20:08:58] <joal>	 omw
[20:11:28] <ottomata>	 !log bouncing main -> jumbo mirrormaker to apply batch.size = 65536
[20:11:30] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[20:25:24] <wikibugs>	 10Analytics-Kanban: Vet new geo wiki data - https://phabricator.wikimedia.org/T191343#4102098 (10Nuria) p:05Triage>03High
[20:33:48] <wikibugs>	 (03CR) 10Nuria: [C: 032] Add PageviewTagger to webrequest [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/423679 (https://phabricator.wikimedia.org/T191022) (owner: 10Joal)
[20:39:38] <wikibugs>	 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Page views and Country name table columns overlapping in the Page Views By Country metric on Dashboard - https://phabricator.wikimedia.org/T191121#4102192 (10sahil505) @mforns It works fine on 1. Chrome on OSX & 2. Safari for now. I'll test the other 3...
[20:42:35] <wikibugs>	 10Analytics, 10New-Readers, 10Mobile, 10Readers-Web-Backlog (Tracking): Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#4102195 (10Nuria)
[21:05:09] <wikibugs>	 10Analytics, 10New-Readers, 10Mobile, 10Readers-Web-Backlog (Tracking): Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#3961955 (10Nuria) mmm, i think opera mini traffic is shifted all arround. See: {F16641812}   There are spikes on US traffic (wayy up) , South Africa traffic...
[21:14:25] <wikibugs>	 10Analytics, 10New-Readers, 10Mobile, 10Readers-Web-Backlog (Tracking): Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#4102311 (10Tbayer) Thanks @Nuria - this actually matches what had been discussed further above in the task. We already have an action item.
[21:16:08] <wikibugs>	 10Analytics, 10New-Readers, 10Mobile, 10Readers-Web-Backlog (Tracking): Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#4102316 (10Tbayer)
[21:23:21] <nuria_>	 HaeB: just read your comment but i do not understand, sorry, what is the action item here? https://phabricator.wikimedia.org/T187014
[21:23:59] <HaeB>	 nuria_: anne and dfoy reaching out to opera
[21:24:47] <nuria_>	 HaeB: to get what info?
[21:25:23] <HaeB>	 btw could you comment on this part https://phabricator.wikimedia.org/T187014#4095670 , regarding fixing the data in pivot etc?
[21:25:43] <HaeB>	 what info: see https://phabricator.wikimedia.org/T187014#4095670 too
[21:26:04] <HaeB>	 obviously we don't know yet what they can/will give us
[21:27:19] <nuria_>	 HaeB: I am not sure how would we fix it ... the ip address we are getting geo locates to us per maxmind
[21:28:25] <nuria_>	 HaeB: see us jump on my posting.
[21:28:27] <HaeB>	 yes, like i wrote there, it might become easiest if/when maxmind fixes this upstream (assuming that that's really the reason)
[21:29:10] <nuria_>	 HaeB:  if that is the reason, it very well could be x-forwarded fors for opera mini traffic
[21:29:13] <HaeB>	 ...but it is just a few easily queryable IP ranges, that could also be done by hand (not saying it's worth the effort)
[21:29:44] <HaeB>	 oh you mean something going wrong with our own XFF handling?
[21:30:20] <nuria_>	 or opera's
[21:30:26] <nuria_>	 opera mini is a proxy
[21:30:43] <nuria_>	 thus the pageview is initiated from their cdn 
[21:32:53] <nuria_>	 HaeB: it could be either
[21:32:56] <HaeB>	 yes of course. that's what is meant by "rerouted" earlier in the task (one could have called it "reproxied" too)
[21:33:19] <HaeB>	 of course opera mini is a proxy, i mean
[21:35:31] <nuria_>	 HaeB:  so several things: 1) could be a browser issue -> browser is sending opera mini address rather than clients 
[21:36:03] <HaeB>	 looks to sudden for a client-side software change  (these usually take a while to roll out)
[21:36:08] <nuria_>	 2)  varnish layer might need to read x-forwraded for, it might not be there or not being read (not sure) 
[21:36:21] <HaeB>	 *too* sudden
[21:36:29] <nuria_>	 3) maxmind cannot deal ONLY with adresses from opera all of a sudden
[21:36:53] <nuria_>	 without any further evidence i vote for issues being at opera layer
[21:36:58] <nuria_>	 juas
[21:38:47] <nuria_>	 HaeB: specially cause South africa seems to have now nigeria's traffic and i bet they have a cdn over there, something to check
[21:39:36] <HaeB>	 nuria_: agreed, it's most likely on their side ... that's why contacting them seems a reasonable next step
[21:42:59] <wikibugs>	 10Analytics, 10New-Readers, 10Operations, 10Traffic, and 2 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#4102458 (10Nuria)
[21:46:53] <nuria_>	 HaeB: ok, so you understand what i mean here with "there is no fixing  the data"
[21:47:45] <HaeB>	 not quite?
[21:48:36] <wikibugs>	 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Page views and Country name table columns overlapping in the Page Views By Country metric on Dashboard - https://phabricator.wikimedia.org/T191121#4102474 (10sahil505) @mforns : Just checked on other 3 browsers, it works perfectly. We should be good to...
[21:52:48] <wikibugs>	 10Analytics, 10New-Readers, 10Operations, 10Traffic, and 2 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#4102475 (10Nuria) >If we can get updates from them, can we repair the date/update things on our side retroactively? I Not likely as we would need the original re...
[21:55:33] <wikibugs>	 (03CR) 10Sahil505: "@mforns : it works for all the browsers listed above. Thanks." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423172 (https://phabricator.wikimedia.org/T191121) (owner: 10Sahil505)
[21:57:33] <wikibugs>	 (03Restored) 10Sahil505: Corrected kebab case - key name mismatch of metrics in topic selector [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423351 (https://phabricator.wikimedia.org/T188268) (owner: 10Sahil505)
[21:58:33] <wikibugs>	 (03PS2) 10Sahil505: Corrected kebab case - key name mismatch of metrics in topic selector [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423351 (https://phabricator.wikimedia.org/T188268)
[21:59:23] <wikibugs>	 (03Abandoned) 10Sahil505: Corrected kebab case - key name mismatch of metrics in topic selector [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423772 (https://phabricator.wikimedia.org/T188268) (owner: 10Sahil505)
[22:03:19] <nuria_>	 HaeB: there is no "upstream fix" if issue is on opera's side
[22:03:59] <nuria_>	 HaeB: meaning that opera will correct issue -if any- but we will not be able to correct the data, makes sense?
[22:04:22] <HaeB>	 upstream was referring to maxmind, not opera
[22:05:05] <HaeB>	 if opera has just switched around IP ranges, that could be something that maxmind can and should correct for
[22:06:59] <nuria_>	 HaeB: what i mean is that maxmind is fine 
[22:07:28] <nuria_>	 HaeB: maxmind is geolocating correctly _most likely_ the ips sent by opera
[22:07:44] <nuria_>	 HaeB: it is just the Ips are operas cdns 
[22:08:09] <nuria_>	 HaeB: need to triple check
[22:08:10] <wikibugs>	 (03Restored) 10Sahil505: Corrected kebab case - key name mismatch of metrics in topic selector [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423772 (https://phabricator.wikimedia.org/T188268) (owner: 10Sahil505)
[22:08:15] <nuria_>	 HaeB: a possible theory
[22:08:29] <nuria_>	 HaeB:  seems quite unlikey 
[22:08:39] <nuria_>	 HaeB: that maxmind will stop working for just 1 browser
[22:08:42] <nuria_>	 HaeB: right?
[22:09:38] <nuria_>	 HaeB:  let me check couple more things
[22:09:38] <HaeB>	 oh it's not about the browser per se... i think we agreed that it has most likely to do with their proxying, no?
[22:09:44] <wikibugs>	 (03Abandoned) 10Sahil505: Corrected kebab case - key name mismatch of metrics in topic selector [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423772 (https://phabricator.wikimedia.org/T188268) (owner: 10Sahil505)
[22:10:56] <wikibugs>	 (03CR) 10Sahil505: "Sorry for this back and forth. Was having some commit/merging issues. Was able to restore successfully. Consider this as the final one. Ch" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423351 (https://phabricator.wikimedia.org/T188268) (owner: 10Sahil505)
[22:15:25] <wikibugs>	 10Analytics, 10Patch-For-Review: Some metrics don't work in the topic selector - https://phabricator.wikimedia.org/T188268#4102587 (10sahil505) Sorry for all the above mess. Was facing some issues with merging/commits. Successfully restored.
[22:25:59] <wikibugs>	 10Analytics, 10New-Readers, 10Operations, 10Traffic, and 2 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#4102622 (10atgo) >>! In T187014#4102475, @Nuria wrote: >>If we can get updates from them, can we repair the date/update things on our side retroactively? I > Ano...
[22:27:03] <wikibugs>	 10Analytics, 10MobileFrontend, 10Readers-Web-Backlog, 10Performance-Team (Radar), 10Technical-Debt: Figure out XAnalytics stuff - https://phabricator.wikimedia.org/T190381#4102625 (10Jdlrobson) Wow this dates back to the days when Awjrichards  was working on the code introduced here: https://gerrit.wikim...
[22:34:34] <wikibugs>	 10Analytics, 10MobileFrontend, 10Readers-Web-Backlog, 10Performance-Team (Radar), 10Technical-Debt: Figure out XAnalytics stuff - https://phabricator.wikimedia.org/T190381#4102673 (10Jdlrobson) I'm not sure if it helps, but confusingly there is also some code in MobileFrontend, added a long time ago http...
[22:34:49] <wikibugs>	 10Analytics, 10MobileFrontend, 10Performance-Team (Radar), 10Readers-Web-Backlog (Tracking), 10Technical-Debt: Figure out XAnalytics stuff - https://phabricator.wikimedia.org/T190381#4102677 (10Jdlrobson)
[22:59:42] <wikibugs>	 10Analytics, 10New-Readers, 10Operations, 10Traffic, and 2 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#4102802 (10Nuria) @atgo I have checked data for january and march for Opera and i just see us receiving IP addresses of Opera's Proxy endpoints instead of Ip adr...