[01:57:34] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: Make webperf eventlogging consumers use eventlogging on Kafka - https://phabricator.wikimedia.org/T110903#4077954 (10Krinkle) [03:12:15] 10Quarry: Error: Too many results! - https://phabricator.wikimedia.org/T190608#4078018 (10Jdx) [03:15:43] 10Quarry: Error: Too many results! - https://phabricator.wikimedia.org/T190608#4078029 (10zhuyifei1999) p:05Triage>03High a:03zhuyifei1999 I though I reverted the change... [03:20:15] 10Quarry: Error: Too many results! - https://phabricator.wikimedia.org/T190608#4078032 (10zhuyifei1999) In T188564#4055351 the save limit was set to 10 mins. It's enough to cause an OOM, yet it's not enough to save 20 rows?! @Halfak ideas? [03:28:16] 10Quarry: Error: Too many results! - https://phabricator.wikimedia.org/T190608#4078050 (10zhuyifei1999) 05Open>03Resolved @Jdx https://quarry.wmflabs.org/query/21900 is killed by the restart. Please rerun. The real issue shall be tracked in T188564 [05:50:28] PROBLEM - Disk space on Hadoop worker on analytics1051 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/m 25 GB (0% inode=99%): /var/lib/hadoop/data/g 24 GB (0% inode=99%): /var/lib/hadoop/data/l 21 GB (0% inode=99%): /var/lib/hadoop/data/d 24 GB (0% inode=99%): /var/lib/hadoop/data/e 23 GB (0% inode=99%): /var/lib/hadoop/data/c 18 GB (0% inode=99%): /var/lib/hadoop/data/h 25 GB (0% inode=99%): /var/lib/hadoop/data/ [05:50:28] 99%): /var/lib/hadoop/data/j 16 GB (0% inode=99%): /var/lib/hadoop/data/b 24 GB (0% inode=99%): /var/lib/hadoop/data/f 22 GB (0% inode=99%): /var/lib/hadoop/data/i 24 GB (0% inode=99%) [06:08:08] PROBLEM - Disk space on Hadoop worker on analytics1046 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/l 20 GB (0% inode=99%): /var/lib/hadoop/data/k 26 GB (0% inode=99%): /var/lib/hadoop/data/f 25 GB (0% inode=99%): /var/lib/hadoop/data/c 16 GB (0% inode=99%): /var/lib/hadoop/data/j 21 GB (0% inode=99%): /var/lib/hadoop/data/g 19 GB (0% inode=99%): /var/lib/hadoop/data/d 19 GB (0% inode=99%): /var/lib/hadoop/data/ [06:08:08] 99%): /var/lib/hadoop/data/h 19 GB (0% inode=99%): /var/lib/hadoop/data/b 23 GB (0% inode=99%): /var/lib/hadoop/data/i 24 GB (0% inode=99%): /var/lib/hadoop/data/m 22 GB (0% inode=99%) [06:22:48] PROBLEM - Disk space on Hadoop worker on analytics1050 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/h 22 GB (0% inode=99%): /var/lib/hadoop/data/e 23 GB (0% inode=99%): /var/lib/hadoop/data/k 22 GB (0% inode=99%): /var/lib/hadoop/data/c 20 GB (0% inode=99%): /var/lib/hadoop/data/g 21 GB (0% inode=99%): /var/lib/hadoop/data/j 22 GB (0% inode=99%): /var/lib/hadoop/data/f 24 GB (0% inode=99%): /var/lib/hadoop/data/ [06:22:48] 99%): /var/lib/hadoop/data/d 19 GB (0% inode=99%): /var/lib/hadoop/data/m 18 GB (0% inode=99%): /var/lib/hadoop/data/i 23 GB (0% inode=99%): /var/lib/hadoop/data/b 16 GB (0% inode=99%) [06:27:48] PROBLEM - Disk space on Hadoop worker on analytics1050 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/h 22 GB (0% inode=99%): /var/lib/hadoop/data/e 23 GB (0% inode=99%): /var/lib/hadoop/data/k 21 GB (0% inode=99%): /var/lib/hadoop/data/c 19 GB (0% inode=99%): /var/lib/hadoop/data/g 21 GB (0% inode=99%): /var/lib/hadoop/data/j 22 GB (0% inode=99%): /var/lib/hadoop/data/f 25 GB (0% inode=99%): /var/lib/hadoop/data/ [06:27:48] 99%): /var/lib/hadoop/data/d 19 GB (0% inode=99%): /var/lib/hadoop/data/m 18 GB (0% inode=99%): /var/lib/hadoop/data/i 22 GB (0% inode=99%): /var/lib/hadoop/data/b 16 GB (0% inode=99%) [07:12:23] 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey, 10User-Joe: rack/setup/install conf1004-conf1006 - https://phabricator.wikimedia.org/T166081#4078152 (10Joe) [07:37:07] PROBLEM - YARN NodeManager JVM Heap usage on analytics1051 is CRITICAL: CRITICAL - analytics_hadoop_yarn_nodemanager is 3914407916 https://grafana.wikimedia.org/dashboard/db/analytics-hadoop?orgId=1&panelId=17&fullscreen [07:59:59] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Google-Summer-of-Code (2018): Proposal : [Analytics] Improvements to Wikistats2 front-end - https://phabricator.wikimedia.org/T189964#4078155 (10sahil505) [08:00:57] PROBLEM - Disk space on Hadoop worker on analytics1050 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/h 22 GB (0% inode=99%): /var/lib/hadoop/data/e 21 GB (0% inode=99%): /var/lib/hadoop/data/k 19 GB (0% inode=99%): /var/lib/hadoop/data/c 18 GB (0% inode=99%): /var/lib/hadoop/data/g 21 GB (0% inode=99%): /var/lib/hadoop/data/j 19 GB (0% inode=99%): /var/lib/hadoop/data/f 23 GB (0% inode=99%): /var/lib/hadoop/data/ [08:00:58] 99%): /var/lib/hadoop/data/d 17 GB (0% inode=99%): /var/lib/hadoop/data/m 17 GB (0% inode=99%): /var/lib/hadoop/data/i 20 GB (0% inode=99%): /var/lib/hadoop/data/b 16 GB (0% inode=99%) [08:03:41] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Google-Summer-of-Code (2018): Proposal : [Analytics] Improvements to Wikistats2 front-end - https://phabricator.wikimedia.org/T189964#4078157 (10sahil505) @mforns I have made the suggested changes. If there is anything else that can improve this pro... [08:12:48] RECOVERY - Disk space on Hadoop worker on analytics1051 is OK: DISK OK [08:12:58] RECOVERY - Disk space on Hadoop worker on analytics1050 is OK: DISK OK [08:13:03] !log kill failing query swamping the cluster (application_1520532368078_47226) [08:13:05] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:13:27] RECOVERY - Disk space on Hadoop worker on analytics1046 is OK: DISK OK [08:43:17] RECOVERY - YARN NodeManager JVM Heap usage on analytics1051 is OK: OK - analytics_hadoop_yarn_nodemanager is 3788976656 https://grafana.wikimedia.org/dashboard/db/analytics-hadoop?orgId=1&panelId=17&fullscreen [16:25:06] (03PS1) 10Amitjoki: Correct mixed case in Wikistats2 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421708 (https://phabricator.wikimedia.org/T190435) [16:27:09] 10Analytics, 10Analytics-Wikistats: Wikistat Beta: expand topic explorer by default - https://phabricator.wikimedia.org/T186335#4078471 (10Amitjoki) a:03Amitjoki [17:13:29] (03PS1) 10Amitjoki: Expand topic explorer by default as in T186335 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421711 [17:42:36] (03Abandoned) 10Amitjoki: Converted initial mixed cases to regular cases. Will do more. [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421605 (owner: 10Amitjoki) [18:42:15] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Review entries in https://github.com/Bitergia/mediawiki-repositories/ to exclude/include - https://phabricator.wikimedia.org/T187711#4078637 (10Aklapper) [18:42:26] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Review entries in https://github.com/Bitergia/mediawiki-repositories/ to exclude/include - https://phabricator.wikimedia.org/T187711#3983039 (10Aklapper) Using `grep -r "organization" wikimedia-affiliations.json | sort | uniq -c` to crea... [19:04:47] 10Analytics, 10Analytics-General-or-Unknown, 10Language-Team, 10Mobile-Apps, and 3 others: there should be a comparison of clicks count on interlanguage links on different platforms - https://phabricator.wikimedia.org/T78351#4078664 (10Amire80) [19:06:44] 10Analytics, 10Analytics-Wikimetrics, 10I18n: Add i18n support to Wikimetrics - https://phabricator.wikimedia.org/T60634#4078667 (10Amire80) [19:15:33] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Investigate detached duplicated accounts in DB with same username, same source, but different uuids - https://phabricator.wikimedia.org/T170093#4078678 (10Aklapper) 05stalled>03Open [19:49:31] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Investigate detached duplicated accounts in DB with same username, same source, but different uuids - https://phabricator.wikimedia.org/T170093#4078705 (10Aklapper) [19:49:33] 10Analytics-Tech-community-metrics, 10Developer-Relations (Apr-Jun-2018): Provide Hatstall to fix syncing / updating of identities on the wikimedia.biterg.io production instance - https://phabricator.wikimedia.org/T157898#4078706 (10Aklapper) [19:50:27] 10Analytics-Tech-community-metrics, 10Developer-Relations (Apr-Jun-2018): Provide Hatstall to fix syncing / updating of identities on the wikimedia.biterg.io production instance - https://phabricator.wikimedia.org/T157898#3124623 (10Aklapper) [19:50:53] 10Analytics-Tech-community-metrics, 10Developer-Relations (Apr-Jun-2018): Provide Hatstall to fix syncing / updating of identities on the wikimedia.biterg.io production instance - https://phabricator.wikimedia.org/T157898#3124637 (10Aklapper) [19:50:55] 10Analytics-Tech-community-metrics, 10Developer-Relations (Apr-Jun-2018): Advertise wikimedia.biterg.io more widely in the Wikimedia community - https://phabricator.wikimedia.org/T179820#4078714 (10Aklapper)