[04:15:03] elukey groceryheist: seems https://yarn.wikimedia.org/cluster/scheduler is working fine again [05:45:28] HaeB: thanks! Did it re-happen again after my restart on saturday by any chance? [05:46:42] no, not that i noticed (i also wasn't aware you had restarted it then) [05:46:57] super thanks [05:47:26] basically the yarn master was an-master1002, not 1001, and we have proxy rules in httpd to use only an-master1001 [05:47:36] this is why you guys were getting the redirect [05:47:53] it seems an issue with connection to zookeeper (that triggered the automatic failover) [05:48:03] i see - thanks again for fixing it! [05:48:10] but since those are the new master nodes, I want to keep a close eye on them :) [05:48:22] thanks for the ping! I'll try to add an alarm later on today [05:48:56] (/me phisiotherapy until 9:15 CEST) [06:19:56] elukey: I think that it working not long after you restarted [07:17:16] * elukey back [07:17:27] groceryheist: ack! Thanks for the feedback [07:17:42] morning elukey [07:17:56] bonjour! [07:18:38] joal: since we cannot start a monday without a good problem [07:18:38] the issue witj ZK connection from yarn master is weird, isn't it? [07:18:39] /dev/mapper/eventlog1002--vg-data 870G 780G 47G 95% /srv [07:18:47] Ah crap :( [07:18:50] hm [07:19:45] ah nice, /srv/log/eventlogging/archive seems gigantic [07:20:04] was checking that elukey [07:20:14] elukey: We could move some of that to HDFS? [07:22:38] elukey: I imagine the increase is due to the increase in EL traffic [07:22:51] in theory yes, but the problem will get back in a few days, I think that we'd need to keep less logs [07:23:20] we have maxage 30 [07:23:24] in logrotate [07:24:11] mmmm but some files are before the 8th [07:25:55] ah yes logrotate -d tells me [07:25:55] removing /srv/log/eventlogging/archive/client-side-events.log-20180906.gz [07:25:58] removing old log /srv/log/eventlogging/archive/client-side-events.log-20180906.gz [07:26:01] renaming /srv/log/eventlogging/client-side-events.log to /srv/log/eventlogging/archive/client-side-events.log-20181008 [07:26:04] removing old log /srv/log/eventlogging/archive/client-side-events.log-20180907.gz [07:26:15] so going to remove those now [07:27:55] elukey: How come logrotate doens't remove them ?? [07:29:11] it runs sometimes during the day [07:29:17] I just anticipated it [07:29:26] the -d option tells you what it would do [07:29:34] if it was running [07:31:09] I think that we should lower down retention to something like 20 days [07:31:48] but we can wait for andrew to be online [07:32:03] works for me elukey [07:32:05] there is no real hurry now, I'll keep checking and possibly remove some logs if needed [07:32:26] elukey: I double checked sqoop-cron issue - false alarm (we need to solve that, I think there is a ticket) [07:33:02] was it the retry false alarm problem? [07:33:22] elukey: However oozie jobs still not started - Due to namespace-map error - We should have an alarm on that, and also I should provide the puppet patch fixing the issue (code has been fixed last month, forgot puppet - sorr) [07:33:29] correct elukey [07:34:49] elukey: I recall why I didn't submit a puppet patch [07:35:05] !log Manually run download-project-namespace-map with proxy [07:35:07] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:36:33] ack [07:36:55] joal: what do you think about moving druid's db to an-coord1001? [07:37:07] elukey: +1 elukey [07:37:42] to be safe I think that we should stop all the indexations [07:38:05] elukey: Let's stop the hourly ones - we're safe on daily [07:38:18] yep [07:39:33] Ok, oozie-mediawiki-load started after manual import [07:41:12] super [07:41:48] elukey: I need help with puppet please :) [07:42:19] sure [07:43:05] elukey: I want to add a proxy variable (from hiera) in profile::analytics::refinery::job::project_namespace_map [07:44:18] sure [07:44:29] so you'd need to add the variable to the class first [07:45:02] something like [07:45:02] class profile::analytics::refinery::job::project_namespace_map( $proxy = hiera('profile::analytics::refinery::job::project_namespace_map::proxy', ''), [07:45:04] ) { [07:45:07] (can we have a default?) [07:45:22] and then add the hiera variable to the role analytics_cluster coordinator yaml file [07:46:47] elukey: I think having a default is super fine [07:47:29] I wondered about adding a variable in the class as you suggested - Other exampels I have seen in the folder use define I think [07:47:52] Ah, found one example (streams-check) [07:48:29] yes basically the example above no? [07:48:48] correct [07:59:58] joal: other than webrequest/pageview hourly what should we stop? [08:00:03] elukey: just submitted a patch :) [08:01:26] elukey: I think that's all elukey [08:02:54] couple of issues for the patch: 1) what happens if we get the default and -p is without parameter? 2) missing ) after the parameter :) [08:08:38] elukey: puppet-compiler still not happy :( [08:09:49] can I quickly fix it or do you prefer? [08:09:56] elukey: as you wish :) [08:10:09] elukey: problem with {} |? [08:10:48] elukey: and actually, I also inverter the test (should be a !- [08:10:52] != sorry [08:11:44] so the '' is outside the parentheses [08:12:00] and the ${} needs to be used only in strings [08:12:12] but I like the solution, I'd only swap '' with undef [08:12:16] and have something like [08:12:16] ok elukey [08:12:25] if $http_proxy { [08:12:27] etc..} [08:12:30] * joal needs to code more puppet to get used to syntax :S [08:13:00] other thing - you cannot reassign values to a variable [08:13:10] so I'd suggest to use a "else" branch [08:13:18] makes sense [08:19:20] \o/ elukey ! it validates :) [08:28:03] left two commens but it looks good! [08:28:31] about druid I did this - dumped the databases from an1003, loaded on an-coord1001 [08:28:37] and then counted the rows for each table [08:28:40] they are not the same [08:29:02] there are less on an1003, so I guess that we also need to stop everything that pokes with druid, like dropping things [08:35:15] even if the drop crons should only run once a day [08:35:30] was I so unlucky to get one of them running after the mysqldump? [08:38:25] joal: merged and deployed [08:39:08] elukey: Thanks for the merge/deploy [08:39:14] elukey: about druid [08:39:57] elukey: 0 7 15 * * [08:40:04] I think you indeed went unlucky [08:40:25] yeah I was thinking the same [08:40:33] re-trying :D [08:40:39] :S :D [08:48:34] no something is off, I keep getting different number of rows for druid segments [08:51:11] wow this is not nice :( [08:52:04] elukey: could be related to dropping-data rules? [08:54:41] no sure, there might be something that we don't see that intererferes :D [08:54:57] if this is the case, then it might be better to do the work once the database is down [08:55:07] * joal don't like that [08:55:22] elukey: let's try to see what is different, shall we? [08:57:21] can I access the an-coord1001 mysql DB? [08:58:03] I don't think so [08:58:15] because you'd need to be root [08:58:22] right [08:58:38] pairing in the cave elukey? Or maybe it;s not worth/ [08:58:47] I can check don't worry, I'll try to get a diff [09:00:55] I'll use the bc only as last resort so you can proceed with your work :) [09:02:08] because there is a good chance that I am not taking the db dump correctly :D [09:04:20] joal: ok I think I found the problem, I used table_schema to get the rows of each table but it is not correct [09:04:34] ?? [09:04:56] SELECT table_name, table_rows FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = 'druid' [09:05:06] this one seems to not return the correct values [09:05:25] if I do select count (*) on the tables on both coordinators I get the same numbers [09:05:31] Interesting !!!!! [09:05:37] yeah :( [09:05:57] elukey: some (not so) funny question for our beloved DBAs [09:12:00] those counters may not be super accurate by design (I think) [09:12:03] but I'll ask! [09:15:44] ok we are good! [09:17:04] \o/ @! [09:26:54] (sorry I had to answer to the door bell) [09:27:04] no problemo [09:28:48] running puppet on druid1001 [09:29:45] ok looks good, running on all nodes (will not restart daemons) [09:31:51] ah! [09:31:51] org.skife.jdbi.v2.exceptions.CallbackFailedException: io.druid.java.util.common.ISE: Database default character set is not UTF-8. [09:32:18] all right lemme fix it [09:37:04] ok druid1001 looks good, applying to the others [09:37:17] ( I am restarting overlord and coordinator on each node) [09:39:16] all right druid analytics swapped joal [09:40:55] \o/ ! Awesome elukey :) [09:41:08] if you want to check and confirm I'll proceed with public [09:41:35] elukey: let's reenable ingestion first please [09:41:56] sure [09:42:05] elukey: good for me from turnilo - Let's confirm ingestion, then +1 for public [09:42:20] done [09:43:48] elukey: monitoring indexation to check everything iso k [10:02:04] joal: 2018-10-08T09:54:31,238 INFO io.druid.indexing.overlord.TaskRunnerUtils: Task [index_hadoop_pageviews_hourly_2018-10-08T09:47:38.709Z] status changed to [SUCCESS]. [10:02:08] looks good [10:02:40] 2018-10-08T09:54:31,262 INFO io.druid.indexing.overlord.TaskRunnerUtils: Task [index_hadoop_pageviews_hourly_2018-10-08T09:47:38.709Z] status changed to [SUCCESS]. [10:02:53] (same one, middlemanager vs overlord) [10:05:30] I am ready to restart the druid_public_eqiad cluster but I'll wait for a green light [10:09:55] +1 elukey - successfull indexation and new available in turnilo - [10:26:13] !log swapped db settings from analytics1003 to an-coord1001 on both Druid clusters (restarted coordinators and overlords) [10:26:15] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:26:16] all good! [10:26:32] first step of the migration completed [10:26:41] tomorrow we'll do oozie and hive [11:21:14] Thinking of that elukey - couldn't we do the same thing for hive metastore? [11:25:08] joal: we could but shouldn't hive be down to be sure that nothing is changing the db? [11:25:14] (this is why I wanted to do it tomorrow) [11:25:27] makes sense elukey [11:25:39] this is my paranoia talking [11:26:00] elukey: you paranoid has saved us a few times already :) [11:40:55] joal: I have also did a lot of damages :P [11:43:50] elukey: I confirm the wikitext-importer works as expected - I let you do a review before merging? [11:44:17] joal: nono please go ahead, it would take me a bit of time to review, don't want to slow donw! [11:44:50] elukey: I did the import manually this month - I can wait a month before productionization - take your time :D [11:45:44] (03Abandoned) 10Joal: Manual importer of xml dumps to hdfs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/409960 (owner: 10Joal) [11:46:53] (03Abandoned) 10Joal: Add XmlConverter spark job [analytics/wikihadoop] - 10https://gerrit.wikimedia.org/r/361440 (https://phabricator.wikimedia.org/T186559) (owner: 10Joal) [11:48:06] (03PS6) 10Joal: Add MediawikiXMLDumpsConverter spark job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/463370 (https://phabricator.wikimedia.org/T202490) [11:49:00] (03PS7) 10Joal: Add MediawikiXMLDumpsConverter spark job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/463370 (https://phabricator.wikimedia.org/T202490) [11:49:18] (03CR) 10Joal: [C: 032] "Merging before next deploy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/463370 (https://phabricator.wikimedia.org/T202490) (owner: 10Joal) [11:52:27] (03PS3) 10Joal: Add mediawiki-history-wikitext oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/463548 (https://phabricator.wikimedia.org/T202490) [11:53:49] elukey: actually, before merging (puppetizing to be precise) the wikitext-importer, we'll need to provide a mount point on an-coord1001 to read xmldumpd folder (as on stat1005) [12:00:56] sure [12:07:13] joal: I am going to take today off, and I had to finish up some wikistats stuff on Friday so I didn’t get to push anything on the missing page Ids. I do want to keep looking at that tomorrow and push a patch though, if that’s ok with you [12:07:27] np milimetric :) [12:07:50] milimetric: today was supposed to be off for you, and we planned on discussing this tomorrow - all good for me :) [12:07:50] (03Merged) 10jenkins-bot: Add MediawikiXMLDumpsConverter spark job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/463370 (https://phabricator.wikimedia.org/T202490) (owner: 10Joal) [12:08:19] k, sweeet [12:27:39] (03PS1) 10Amire80: Change the label to the last day of the week [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/465152 (https://phabricator.wikimedia.org/T206456) [12:28:35] (03PS1) 10Fdans: Release 2.4.4 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/465154 [12:31:26] (03CR) 10Fdans: [V: 032 C: 032] Release 2.4.4 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/465154 (owner: 10Fdans) [12:54:24] 10Analytics, 10Operations, 10ops-eqiad: analytics1068 doesn't boot - https://phabricator.wikimedia.org/T203244 (10Cmjohnson) This is one of the issues we have with leasing.....Dell has it so Farnam is the owner not us. I think it's sorted now and attempting to get it resolved. [12:55:28] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Replace the Analytics Hadoop coordinator - Hive/Oozie/etc... (hardware refresh) - https://phabricator.wikimedia.org/T205509 (10elukey) This morning we successfully moved the Druid clusters to an-coord1001, tomorrow will do hive/oozie and t... [12:58:46] 10Analytics, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10wikidata-tech-focus: ApiAction log in data lake doesn't record Wikibase API actions - https://phabricator.wikimedia.org/T174474 (10Addshore) p:05Triage>03Lowest [13:45:23] heya teaaam [13:46:34] o/ [14:35:04] (03CR) 10Mforns: [C: 031] "Looks good to me too!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/463548 (https://phabricator.wikimedia.org/T202490) (owner: 10Joal) [14:42:36] (03CR) 10Mforns: [C: 032] Allow whole metric areas to be collapsed [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464800 (https://phabricator.wikimedia.org/T206311) (owner: 10Fdans) [14:43:15] (03CR) 10Mforns: [V: 031 C: 031] "LGTM! Sorry, gave a +2, but then saw Dan's comment. +1!" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464800 (https://phabricator.wikimedia.org/T206311) (owner: 10Fdans) [14:44:07] (03CR) 10Mforns: [V: 031 C: 031] "Ah! I think I couldn't delete the +2 before CI started submit jobs... sorry." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464800 (https://phabricator.wikimedia.org/T206311) (owner: 10Fdans) [14:44:27] mforns: nah it won't submit it don't worry [14:45:25] ok :] [15:00:00] (03CR) 10Nuria: [C: 04-1] "Please see my first comment: Let's talk about whether we need this work, I am of the opinion that we should load the data we have for fa" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/464800 (https://phabricator.wikimedia.org/T206311) (owner: 10Fdans) [15:09:32] I'm testing something for hue.wikimedia.org, apologies if there's any interruptions [15:36:59] 10Analytics, 10Analytics-Kanban: Parametize eventlogging to druid ingestion with a whitelist instead of a blacklist - https://phabricator.wikimedia.org/T206342 (10mforns) p:05Triage>03Normal [15:37:03] 10Analytics, 10Analytics-Wikistats, 10User-Elukey: Git push and pull don't complete - https://phabricator.wikimedia.org/T206331 (10elukey) p:05Triage>03Low a:05Ottomata>03elukey [15:39:01] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Make area metrics collapsible - https://phabricator.wikimedia.org/T206311 (10mforns) p:05Triage>03Low [15:39:50] 10Analytics, 10EventBus, 10Core Platform Team Kanban (Current), 10Services (doing): Revision visibility change event sets a wrong performer - https://phabricator.wikimedia.org/T206277 (10mforns) p:05Triage>03Low [15:42:47] 10Analytics, 10Analytics-Cluster, 10Contributors-Analysis, 10Product-Analytics: Hive join fails when using a HiveServer2 client - https://phabricator.wikimedia.org/T206279 (10mforns) @Neil_P._Quinn_WMF, can you please use hive client instead of beeline. We know it says it's deprecated in favor of beeline,... [15:43:34] 10Analytics, 10Analytics-Cluster, 10Contributors-Analysis, 10Product-Analytics: Hive join fails when using a HiveServer2 client - https://phabricator.wikimedia.org/T206279 (10mforns) 05Open>03declined I will decline this task, please reopen if your query continues to fail on the hive client. Thanks! [15:43:59] 10Analytics, 10Analytics-Kanban, 10Research: Create labeled dataset for bot identification - https://phabricator.wikimedia.org/T206267 (10mforns) [15:45:39] 10Analytics, 10Services (watching): Consider converting AQS to TypeScript - https://phabricator.wikimedia.org/T206269 (10mforns) Dan, we saw this task during Groskin' and we'll leave it for when you're here. Prepare to argue :] [16:01:19] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Add ability to bucketize integers as part of event ingestion - https://phabricator.wikimedia.org/T205641 (10mforns) [16:02:22] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Create reports in wikistats UI for "most prolific editors" (a.k.a "top contributors") - https://phabricator.wikimedia.org/T189882 (10mforns) [16:02:50] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Correct data-removal jobs for mediawiki tables (public and private) - https://phabricator.wikimedia.org/T198600 (10mforns) [16:03:20] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Wikistats 2.0: "aa.wikipedia.org" exists and has data available, but marked "Invalid" - https://phabricator.wikimedia.org/T187414 (10mforns) [16:04:20] 10Analytics, 10Analytics-Kanban: [Wikistats2] Bug in Top Viewed Articles since bookmark routing changes - https://phabricator.wikimedia.org/T198258 (10mforns) 05Open>03declined [16:04:58] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Add caching to wikistats 2 annotations request - https://phabricator.wikimedia.org/T205933 (10mforns) [16:05:28] 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Top metrics. Implement failsafe mechanism for when current month computations are not available - https://phabricator.wikimedia.org/T205915 (10mforns) [16:06:30] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Annotations need to use adjustedGraphData - https://phabricator.wikimedia.org/T206171 (10mforns) [16:06:54] 10Analytics, 10Analytics-Kanban, 10MW-1.32-notes (WMF-deploy-2018-10-16 (1.32.0-wmf.26)), 10Patch-For-Review: Improve Dashiki extension messaging - https://phabricator.wikimedia.org/T205644 (10mforns) [16:07:56] 10Analytics, 10Analytics-Kanban, 10Research, 10Patch-For-Review: Copy monthly XML files from public-dumps to HDFS - https://phabricator.wikimedia.org/T202489 (10mforns) [16:08:06] 10Analytics, 10Analytics-Kanban, 10Research, 10Patch-For-Review: Automate XML-to-parquet transformation for XML dumps (oozie job) - https://phabricator.wikimedia.org/T202490 (10mforns) [16:08:19] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Replace the Analytics Hadoop coordinator - Hive/Oozie/etc... (hardware refresh) - https://phabricator.wikimedia.org/T205509 (10mforns) [16:08:32] 10Analytics, 10Analytics-Kanban, 10MW-1.32-notes (WMF-deploy-2018-10-16 (1.32.0-wmf.26)), 10Patch-For-Review: Improve Dashiki extension messaging - https://phabricator.wikimedia.org/T205644 (10Nuria) Is this deployed, mmessage does not seem to have chnaged? https://meta.wikimedia.org/w/index.php?title=Conf... [16:08:52] 10Analytics, 10Analytics-Kanban, 10Page-Issue-Warnings, 10Product-Analytics, and 3 others: Ingest data from PageIssues EventLogging schema into Druid - https://phabricator.wikimedia.org/T202751 (10mforns) [16:09:12] 10Analytics, 10Analytics-Kanban, 10Page-Issue-Warnings, 10Product-Analytics, and 3 others: Ingest data from PageIssues EventLogging schema into Druid - https://phabricator.wikimedia.org/T202751 (10mforns) [16:09:21] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Ingest data into druid for readingDepth schema - https://phabricator.wikimedia.org/T205562 (10mforns) [16:09:33] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Make wikistats UI family aware: you should be able to select a family in drop down menu and request available metrics for it - https://phabricator.wikimedia.org/T205665 (10mforns) [16:11:01] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Make wikistats UI family aware: you should be able to select a family in drop down menu and request available metrics for it - https://phabricator.wikimedia.org/T205665 (10mforns) [16:13:11] 10Analytics: Provide edit tags in the Data Lake edit data - https://phabricator.wikimedia.org/T161149 (10mforns) p:05High>03Normal [16:13:34] 10Analytics: Provide edit tags in the Data Lake edit data - https://phabricator.wikimedia.org/T161149 (10mforns) p:05Normal>03Triage [16:15:22] 10Analytics, 10Analytics-Wikistats: Beta Release: Wikistats: support annotations in graphs - https://phabricator.wikimedia.org/T178015 (10mforns) [16:15:56] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Beta Release: Wikistats: support annotations in graphs - https://phabricator.wikimedia.org/T178015 (10mforns) [16:16:25] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Beta Release: Wikistats: support annotations in graphs - https://phabricator.wikimedia.org/T178015 (10mforns) p:05High>03Normal [16:17:45] 10Analytics, 10Analytics-Wikistats: roadmap of migration to Wikistats 2 - https://phabricator.wikimedia.org/T183180 (10mforns) p:05High>03Triage [16:17:48] 10Analytics, 10Analytics-Wikistats: roadmap of migration to Wikistats 2 - https://phabricator.wikimedia.org/T183180 (10mforns) [16:18:44] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Present Wikistats 2 charts for the period selected by the user. - https://phabricator.wikimedia.org/T183183 (10mforns) a:03Nuria [16:19:58] 10Analytics, 10Analytics-Wikistats: Changes to map projection in wikistats - https://phabricator.wikimedia.org/T188927 (10mforns) p:05High>03Normal [16:23:45] 10Analytics: Enable automatic ingestion from eventlogging into druid for some schemas - https://phabricator.wikimedia.org/T190855 (10mforns) This task refers to ingesting any number of schemas with just one job, that ideally reads schema registry / meta schema data. [16:23:51] 10Analytics: Enable automatic ingestion from eventlogging into druid for some schemas - https://phabricator.wikimedia.org/T190855 (10mforns) p:05High>03Normal [16:28:55] !log restart eventlogging on eventlog1002 for python security upgrades [16:28:57] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:30:21] 10Analytics, 10Analytics-Wikistats: Pixel ratio messed up on Windows Chrome - https://phabricator.wikimedia.org/T194428 (10mforns) p:05High>03Low [16:30:43] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Pixel ratio messed up on Windows Chrome - https://phabricator.wikimedia.org/T194428 (10mforns) [16:31:28] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Beta Release: Wikistats: support annotations in graphs - https://phabricator.wikimedia.org/T178015 (10mforns) [16:31:44] 10Analytics, 10Analytics-Wikistats: Organize annotations pages on meta by convention - https://phabricator.wikimedia.org/T194706 (10mforns) p:05High>03Low [16:41:45] 10Analytics, 10Analytics-Kanban, 10Contributors-Analysis, 10Product-Analytics: Decommision edit analysis dashboard - https://phabricator.wikimedia.org/T199340 (10mforns) a:05Milimetric>03None [16:41:55] 10Analytics, 10Analytics-Kanban, 10Contributors-Analysis, 10Product-Analytics: Decommision edit analysis dashboard - https://phabricator.wikimedia.org/T199340 (10mforns) p:05High>03Normal [16:43:09] 10Analytics, 10Analytics-Kanban: Table view of timely results in wikistats 2 should be ordered in time descending - https://phabricator.wikimedia.org/T199693 (10mforns) [16:44:13] 10Analytics: Scan npm dependencies for vulnerabilities - https://phabricator.wikimedia.org/T200717 (10mforns) p:05High>03Low [16:45:33] 10Analytics, 10Analytics-Wikistats, 10Operations, 10Traffic, 10Regression: [Regression] stats.wikipedia.org redirect no longer works ("Domain not served here") - https://phabricator.wikimedia.org/T126281 (10mforns) @BBlack ping, bumping this up [16:50:54] 10Analytics: MD5 checksums missing from pagecounts-all - https://phabricator.wikimedia.org/T73710 (10mforns) 05Open>03declined Please reopen if still an issue. [16:57:28] 10Analytics, 10Analytics-Wikistats: roadmap of migration to Wikistats 2 - https://phabricator.wikimedia.org/T183180 (10Nuria) >Most prolific contributors [2], and variations on that (bots, absentees, ip addresses), deal with individuals. These reports are already launched, please see examples for italian wiki... [17:03:16] 10Analytics, 10Analytics-Wikistats: roadmap of migration to Wikistats 2 - https://phabricator.wikimedia.org/T183180 (10Nuria) >it would be nice if people can also query by user name. We actually have gotten some pushback with stats such as these so we rolled back some API changes as of late. See: https://phabr... [17:03:31] 10Analytics, 10Analytics-Wikistats: roadmap of migration to Wikistats 2 - https://phabricator.wikimedia.org/T183180 (10Nuria) 05Open>03Resolved [17:03:34] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats 2.0 Remaining reports. - https://phabricator.wikimedia.org/T186121 (10Nuria) [17:18:11] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Present Wikistats 2 charts for the period selected by the user. - https://phabricator.wikimedia.org/T183183 (10Nuria) Closing as many bugs have been fixed on time selector since this one was filed, one month range is now correct: https://stats.wikimedia... [17:18:19] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Present Wikistats 2 charts for the period selected by the user. - https://phabricator.wikimedia.org/T183183 (10Nuria) 05Open>03Resolved [17:23:40] fdans: yt? [17:24:21] nuria: holaa [17:24:29] fdans: i think there is a bug on the top view: the url is carrying the time selection [17:25:02] fdans: like look at a metric (not top) and select a time range , say 1 month, now look at a top metric [17:25:19] fdans: the "time selection" remains in url but actually top data is only shown for 1 month: [17:25:27] fdans: see https://stats.wikimedia.org/v2/#/ar.wikipedia.org/contributing/top-editors/normal|table|2-Year|~total [17:26:00] fdans: let me know if that makes sense [17:27:01] nuria: yeah it’s carrying the state from vue, which seems to be out of sync from the url [17:28:07] fdans: teh time dimension shoudl be reset by top metric [17:28:23] fdans: as those are available just "monthly" [17:29:48] 10Analytics: Time dimension carried on url for top metrics - https://phabricator.wikimedia.org/T206479 (10Nuria) [17:29:52] fdans: https://phabricator.wikimedia.org/T206479 [17:30:48] fdans: until we fix this we cannot announce top metrics, it will be super confusing [17:35:48] nuria: I'm confused by the ticket [17:36:33] when selecting 1-month in a timeseries metric and switch to a top metric, the data requested is the latest month, but the url still says 1-month [17:37:45] nuria: I think this is an issue that is already reported [17:38:33] fdans: select 2 year range in any metric [17:38:44] fdans: if you move to atop metric it will say "2 years" on url [17:39:01] fdans: which is incorrect, as request is being done only on last months data [17:39:09] fdans: does this make more sense? [17:39:55] nuria: yeah, I pointed this out to milimetric a couple weeks ago, it's an issue we've had for a while, [17:40:33] fdans: we cannot announce the top metrics with this issue though [17:41:22] nuria: i mean we already have other 2 top metrics that live with this issue [17:41:33] 10Quarry: Store and show query execution time - https://phabricator.wikimedia.org/T206480 (10Framawiki) [17:43:31] fdans: that might be the case but that does not make it less important, "top editors" is meaningless if you do not know the time range for which it is reported, this is an "unbreak now" kind of bug for this metric [17:43:43] fdans: ditto for edited pages [17:43:55] nuria: but we're reporting the time range in the title of the metric [17:44:13] https://usercontent.irccloud-cdn.com/file/vvAcilkO/Screen%20Shot%202018-10-08%20at%206.44.00%20PM.png [17:44:35] fdans: but how can users know what is correct, the url or the title? [17:44:52] fdans: there is no way to know [17:45:35] elukey: we are still using analytics1003 right? [17:47:52] 10Quarry: Show query code revision history browser - https://phabricator.wikimedia.org/T206482 (10Framawiki) p:05Triage>03Low [17:48:07] 10Quarry: Show query code revision history - https://phabricator.wikimedia.org/T206482 (10Framawiki) [17:58:39] elukey: ahve we implemented the restrictions on firewall to consume from kafka? [17:58:43] *have [18:03:08] elukey: mmm.. answering my own question: no [18:09:06] nuria: sorry I was out running :) [18:09:24] we are not using anymore an1003 only for the Druid databases [18:09:29] the rest is still working [18:10:00] and in theory no, we don't have any restrictions for consuming in kafka atm.. any specific issue? [18:10:11] (brb in a few mins) [18:18:35] 10Analytics, 10Analytics-Cluster, 10Operations: Manage Hue via systemd unit - https://phabricator.wikimedia.org/T206484 (10MoritzMuehlenhoff) [18:19:40] (03PS1) 10Joal: Update DataFrameToHive for dynamic partitions [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/465202 (https://phabricator.wikimedia.org/T164020) [18:21:10] (03PS2) 10Joal: Update DataFrameToHive for dynamic partitions [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/465202 (https://phabricator.wikimedia.org/T164020) [18:23:03] (03PS3) 10Joal: Update DataFrameToHive for dynamic partitions [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/465202 (https://phabricator.wikimedia.org/T164020) [18:23:39] 10Analytics, 10Analytics-Cluster, 10Operations, 10User-Elukey: Manage Hue via systemd unit - https://phabricator.wikimedia.org/T206484 (10elukey) [19:02:40] 10Quarry: Store and show query execution time - https://phabricator.wikimedia.org/T206480 (10zhuyifei1999) [19:02:44] 10Quarry: Include query execution time - https://phabricator.wikimedia.org/T126888 (10zhuyifei1999) [19:09:44] 10Quarry: Mutiple columns with the same name will cause the result to not be shown - https://phabricator.wikimedia.org/T141233 (10zhuyifei1999) [19:09:48] 10Quarry: Quarry cannot store results with identical column names - https://phabricator.wikimedia.org/T170464 (10zhuyifei1999) [19:12:24] 10Quarry: Show query code revision history - https://phabricator.wikimedia.org/T206482 (10zhuyifei1999) See also {T137650} [19:26:41] 10Analytics-Kanban, 10User-Elukey: Upgrade Analytics infrastructure to Debian Stretch - https://phabricator.wikimedia.org/T192642 (10Nuria) [19:26:44] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Upgrade bohrium (piwik/matomo) to Debian Stretch - https://phabricator.wikimedia.org/T202962 (10Nuria) 05Open>03Resolved [19:27:11] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Add caching to wikistats 2 annotations request - https://phabricator.wikimedia.org/T205933 (10Nuria) 05Open>03Resolved [19:27:28] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Create report for "articles with most contributors" in Wikistats2 - https://phabricator.wikimedia.org/T204965 (10Nuria) [19:27:31] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Add ability to filter top metrics - https://phabricator.wikimedia.org/T205725 (10Nuria) 05Open>03Resolved [19:27:56] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Logrotate of refinery rotating on size rather than time - https://phabricator.wikimedia.org/T206020 (10Nuria) 05Open>03Resolved [19:28:13] 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Top metrics. Implement failsafe mechanism for when current month computations are not available - https://phabricator.wikimedia.org/T205915 (10Nuria) 05Open>03Resolved [19:28:27] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Add ability to bucketize integers as part of event ingestion - https://phabricator.wikimedia.org/T205641 (10Nuria) 05Open>03Resolved [19:28:30] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Ingest data into druid for readingDepth schema - https://phabricator.wikimedia.org/T205562 (10Nuria) [19:28:53] 10Analytics, 10Analytics-Kanban: Reboot Analytics hosts for kernel security upgrades - https://phabricator.wikimedia.org/T203165 (10Nuria) 05Open>03Resolved [19:29:11] 10Analytics-Kanban, 10Patch-For-Review: Update top-(editor/pages) endpoints in AQS to follow top-pageviews semantics - https://phabricator.wikimedia.org/T204707 (10Nuria) 05Open>03Resolved [19:29:31] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Move pages to date to "content" frrom "contributing" category on wikistats UI - https://phabricator.wikimedia.org/T205752 (10Nuria) 05Open>03Resolved [19:29:52] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats 2.0 Remaining reports. - https://phabricator.wikimedia.org/T186121 (10Nuria) [19:29:56] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Create reports in wikistats UI for "most prolific editors" (a.k.a "top contributors") - https://phabricator.wikimedia.org/T189882 (10Nuria) 05Open>03Resolved [19:31:20] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Annotations need to use adjustedGraphData - https://phabricator.wikimedia.org/T206171 (10Nuria) 05Open>03Resolved [19:49:54] 10Analytics: Provide edit tags in the Data Lake edit data - https://phabricator.wikimedia.org/T161149 (10Neil_P._Quinn_WMF) For the record, I actually think this is unblocked now! As I commented in T205940: >There is a refactor of the change_tag tables underway (T185355), but the new ct_tag_id columns and chang... [20:09:44] (03PS1) 10Joal: Add webrequest_subset_tags transform function [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/465206 (https://phabricator.wikimedia.org/T164020) [20:09:55] 10Analytics, 10Analytics-Cluster, 10Contributors-Analysis, 10Product-Analytics: Hive join fails when using a HiveServer2 client - https://phabricator.wikimedia.org/T206279 (10Neil_P._Quinn_WMF) 05declined>03Open >>! In T206279#4649749, @mforns wrote: > @Neil_P._Quinn_WMF, can you please use hive client... [20:28:13] 10Quarry: Include query execution time - https://phabricator.wikimedia.org/T126888 (10Framawiki) [20:29:02] 10Quarry: Include query execution time - https://phabricator.wikimedia.org/T126888 (10Framawiki) I don't seen any benefit of adding the pending time (only for monitoring purposes?), feel free to explain why it's needed [20:38:58] 10Analytics, 10Analytics-Cluster, 10Contributors-Analysis, 10Product-Analytics: Hive join fails when using a HiveServer2 client - https://phabricator.wikimedia.org/T206279 (10Neil_P._Quinn_WMF) >>! In T206279#4644644, @JAllemandou wrote: > Some more info on the issue as I understand it: Given the data size... [21:14:50] 10Analytics: Provide edit tags in the Data Lake edit data - https://phabricator.wikimedia.org/T161149 (10Nuria) I think you are talking about : https://phabricator.wikimedia.org/T205940 which is just adding tag tables to data lake and you are right that is not blocked, in order to have tag info per revision I th... [21:16:30] 10Analytics: Time dimension carried on url for top metrics - https://phabricator.wikimedia.org/T206479 (10Nuria) Also, look at a "top metric" from teh dashboard, despite data being shown for just 1 month the url would say, "2-years" [23:39:29] (03PS1) 10Nuria: [WIP] Time dimension should be reseted to "1-Month" for top metrics [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/465296 (https://phabricator.wikimedia.org/T206479) [23:40:13] 10Analytics, 10Patch-For-Review: Time dimension carried on url for top metrics - https://phabricator.wikimedia.org/T206479 (10Nuria) [23:55:56] (03PS2) 10Nuria: [WIP] Time dimension should be reseted to "1-Month" for top metrics [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/465296 (https://phabricator.wikimedia.org/T206479)