[00:28:09] 10Analytics, 06Research-and-Data: geowiki data for Global Innovation Index - https://phabricator.wikimedia.org/T131889#3182122 (10leila) Sure. Ping if you need help, Rafael. Otherwise, we will assume this is Done. [00:29:48] 10Analytics-Cluster, 06Analytics-Kanban, 06Operations, 13Patch-For-Review, 15User-Elukey: Update Zookeeper heap usage configuration and set alarms - https://phabricator.wikimedia.org/T157968#3182142 (10Dzahn) 05Resolved>03Open re-opening since Icinga has many alerts: https://icinga.wikimedia.org/cgi... [00:30:34] ACKNOWLEDGEMENT - Zookeeper node JVM Heap usage on conf1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [972.0] daniel_zahn https://phabricator.wikimedia.org/T157968 [00:30:35] ACKNOWLEDGEMENT - Zookeeper node JVM Heap usage on conf1002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [972.0] daniel_zahn https://phabricator.wikimedia.org/T157968 [00:30:36] ACKNOWLEDGEMENT - Zookeeper node JVM Heap usage on conf1003 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [972.0] daniel_zahn https://phabricator.wikimedia.org/T157968 [00:30:37] ACKNOWLEDGEMENT - Zookeeper node JVM Heap usage on druid1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [972.0] daniel_zahn https://phabricator.wikimedia.org/T157968 [00:30:37] ACKNOWLEDGEMENT - Zookeeper node JVM Heap usage on druid1002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [972.0] daniel_zahn https://phabricator.wikimedia.org/T157968 [00:30:38] ACKNOWLEDGEMENT - Zookeeper node JVM Heap usage on druid1003 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [972.0] daniel_zahn https://phabricator.wikimedia.org/T157968 [02:54:50] 06Analytics-Kanban, 10Analytics-Wikistats: Create and monitor Round2 consultation page - https://phabricator.wikimedia.org/T162155#3182315 (10Milimetric) List of users retrieved by combining the first round (https://meta.wikimedia.org/wiki/User:Milimetric_(WMF)/Wikistats) with this query: ``` select user_name... [03:16:47] (03PS1) 10Milimetric: Add pa.wikisource and wb.wikimedia to whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/348196 [03:17:38] (03CR) 10Milimetric: [V: 032 C: 032] Add pa.wikisource and wb.wikimedia to whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/348196 (owner: 10Milimetric) [03:46:09] (03CR) 10Nuria: "Thanks for doing this." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/348196 (owner: 10Milimetric) [03:58:05] (03CR) 10Nuria: Changes internal aqs api to accept a project or array of same (031 comment) [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/347305 (owner: 10Fdans) [08:36:04] elukey: Hi ! [08:36:14] elukey: any idea what the druid alerts were?n [08:36:55] joal: o/ - I think that icinga was in a weird state and the alarms are not super reliable [08:37:25] elukey: Ah ... Alarms that test the alarm system - I love the evaluation paradox :) [08:37:26] https://grafana.wikimedia.org/dashboard/db/zookeeper?refresh=5m&orgId=1&var-cluster=druid-eqiad&var-zookeeper_hosts=All&from=now-7d&to=now [08:37:53] elukey: This evaluation paradox was at the heart of my thesis ;) [08:38:16] the main issue is that I added xms everywhere, now the heap is more used [08:38:24] and the alarms are firing :D [08:38:27] :) [08:39:01] on the graph you sent me elukey, everything looks fine [08:39:23] for this particular use case though, we might think about raising the heap size to 2g [08:39:30] elukey: easy [08:39:53] 1g heaps are really small.. and maybe 2g could reduce a lot the young GC collections [08:40:05] elukey: it however surprises me that zookeeper uses that much heap [08:40:23] But yeah I agree, 1G is not that much [08:40:29] This is java after all [08:40:48] lol [08:40:57] * joal feels like trollday [08:52:40] joal: are you going to send a code review for Unexpected values in pageview for workflow ? [08:59:20] elukey: I think milimetric did that yesterday (https://gerrit.wikimedia.org/r/#/c/348196/) [09:02:18] nicezzz [11:12:14] (03PS1) 10Joal: [WIP] Upgrade scala to 2.11.8 and Spark to 2.1 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/348207 [11:15:32] (03PS2) 10Joal: [WIP] Upgrade scala to 2.11.8 and Spark to 2.1 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/348207 [11:18:06] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Upgrade scala to 2.11.8 and Spark to 2.1 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/348207 (owner: 10Joal) [11:18:35] Taking a break a-team [11:18:59] o/ [12:01:14] * elukey afk for a bit! [12:46:10] 10Analytics-Cluster, 06Analytics-Kanban, 06Operations, 13Patch-For-Review, 15User-Elukey: Update Zookeeper heap usage configuration and set alarms - https://phabricator.wikimedia.org/T157968#3182747 (10elukey) Multiple PEBKACs from my side: 1) I acked permanently the alarms in Icinga without realizing i... [12:50:17] hi team :] [12:51:32] mforns: o/ [12:51:41] hello elukey :] [13:00:02] if you guys want to laugh please check https://gerrit.wikimedia.org/r/348214 [13:00:27] multiple PEBKACs [13:11:13] :] good catch! [13:11:43] well it was Daniel telling me that monitors were not working :( [13:12:00] this is why those alarm fired yesterday [13:12:14] if the heap size is greater than 1Mb alarm [13:12:16] :/ [13:34:03] RECOVERY - Zookeeper node JVM Heap usage on conf1001 is OK: OK: Less than 60.00% above the threshold [921000000.0] [13:46:00] 10Analytics-Cluster, 06Analytics-Kanban, 06Operations, 13Patch-For-Review, 15User-Elukey: Update Zookeeper heap usage configuration and set alarms - https://phabricator.wikimedia.org/T157968#3182893 (10elukey) 05Open>03Resolved @Dzahn thanks a lot for the heads up, I should have fixed the issues. My... [13:49:55] Ah elukey, this actually breaks my troll about java this morning ;) [13:52:03] you can troll me if you want, I deserve it after this mess :D [13:52:11] huhu :) [13:52:35] My rule is: never troll your coworker on stuff you'd have done yourself [14:19:24] 06Analytics-Kanban: Security Upgrade for piwik - https://phabricator.wikimedia.org/T158322#3033379 (10elukey) As far as I can see the last version available for LTS 2.x is 2.17.1, and it will be required to upgrade the db schema. I have no idea if we can jump directly to 2.17.1 or if we need to upgrade all the i... [14:22:47] 06Analytics-Kanban: Piwik refactoring and puppetization - https://phabricator.wikimedia.org/T163000#3182952 (10elukey) [14:22:58] 06Analytics-Kanban: Piwik refactoring and puppetization - https://phabricator.wikimedia.org/T163000#3182965 (10elukey) [14:23:01] 10Analytics, 06Operations: sync bohrium and apt.wikimedia.org piwik versions - https://phabricator.wikimedia.org/T149993#3182966 (10elukey) [14:23:38] 06Analytics-Kanban: Security Upgrade for piwik - https://phabricator.wikimedia.org/T158322#3182969 (10elukey) [14:23:40] 10Analytics, 13Patch-For-Review, 15User-Elukey: Piwik puppet configuration refactoring and updates - https://phabricator.wikimedia.org/T159136#3182968 (10elukey) [14:23:42] 06Analytics-Kanban: Piwik refactoring and puppetization - https://phabricator.wikimedia.org/T163000#3182952 (10elukey) [14:23:58] 06Analytics-Kanban: Security Upgrade for piwik - https://phabricator.wikimedia.org/T158322#3033379 (10elukey) p:05Normal>03High [14:24:18] 10Analytics, 15User-Elukey: Piwik puppet configuration refactoring and updates - https://phabricator.wikimedia.org/T159136#3057772 (10elukey) p:05High>03Normal [14:52:25] (03PS1) 10Mforns: Add annotations to tabs layout [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/348227 (https://phabricator.wikimedia.org/T162482) [14:59:27] 06Analytics-Kanban: Verfify MaxMind is updated regularly - https://phabricator.wikimedia.org/T162616#3183055 (10Nuria) 05Open>03Resolved [14:59:43] 06Analytics-Kanban: Productionize Edit History Reconstruction and Extraction - https://phabricator.wikimedia.org/T152035#3183058 (10Nuria) [14:59:45] 06Analytics-Kanban, 13Patch-For-Review: Update sqoop job to add infra and version partition - https://phabricator.wikimedia.org/T160152#3183057 (10Nuria) 05Open>03Resolved [15:03:00] 06Analytics-Kanban: Security Upgrade for piwik - https://phabricator.wikimedia.org/T158322#3183061 (10elukey) a:03elukey [15:59:09] (03CR) 10Milimetric: Add annotations to tabs layout (032 comments) [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/348227 (https://phabricator.wikimedia.org/T162482) (owner: 10Mforns) [16:17:59] milimetric, yt? [16:30:49] (03CR) 10Mforns: Add annotations to tabs layout (032 comments) [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/348227 (https://phabricator.wikimedia.org/T162482) (owner: 10Mforns) [16:38:01] going offline people! Talk with you on Tuesday! [16:38:02] o/ [16:54:02] ciaoo [17:41:07] 10Analytics, 10ChangeProp, 10EventBus, 06Revision-Scoring-As-A-Service, and 3 others: Create generalized "precache" endpoint for ORES - https://phabricator.wikimedia.org/T148714#3183414 (10Halfak) [17:41:10] 10Analytics, 10ChangeProp, 10EventBus, 06Revision-Scoring-As-A-Service, and 3 others: Switch `/precache` to be a POST end point - https://phabricator.wikimedia.org/T162627#3183412 (10Halfak) 05Open>03Resolved [17:41:58] 10Analytics-Cluster, 06Analytics-Kanban, 06Operations, 13Patch-For-Review, 15User-Elukey: Update Zookeeper heap usage configuration and set alarms - https://phabricator.wikimedia.org/T157968#3183439 (10Dzahn) @elukey thank you for fixing :) They all look green now. I'll comment if i see them again. [20:37:18] (03PS5) 10Nuria: Changes internal aqs api to accept a project or array of same [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/347305 (owner: 10Fdans) [20:38:28] (03CR) 10Nuria: "Please @milimetric take a look and let me know if this is what you had in mind." [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/347305 (owner: 10Fdans)