[00:11:55] 10Scoring-platform-team, 10Discovery-Search: Consume ORES articletopic data from Kafka and store it in HDFS - https://phabricator.wikimedia.org/T240553 (10Tgr) As per above, this turned out to be a no-op. [00:17:20] 10MediaWiki-extensions-ORES, 10Scoring-platform-team, 10Discovery-Search, 10NewcomerTasks 1.1, 10Growth-Team (Current Sprint): Expose ORES drafttopic data in ElasticSearch via a custom CirrusSearch keyword - https://phabricator.wikimedia.org/T240559 (10Tgr) [00:56:33] 10Scoring-platform-team, 10Discovery-Search, 10Epic, 10Growth-Team (Current Sprint): [EPIC] Growth: Newcomer tasks 1.1.1 (ORES topics) - https://phabricator.wikimedia.org/T240517 (10Tgr) [01:45:24] 10ORES, 10Scoring-platform-team, 10Patch-For-Review: Review prometheus ORES rules for completeness - https://phabricator.wikimedia.org/T233448 (10colewhite) I see the value in a refactor/cleanup if what is currently being captured is not everything we need to (at least) recreate the current dashboards. I pr... [02:19:37] 10MediaWiki-extensions-ORES, 10Scoring-platform-team, 10Discovery-Search, 10NewcomerTasks 1.1, 10Growth-Team (Current Sprint): Expose ORES drafttopic data in ElasticSearch via a custom CirrusSearch keyword - https://phabricator.wikimedia.org/T240559 (10Tgr) > Open questions: > * what should be the exact... [02:42:10] 10Scoring-platform-team (Current), 10Discovery-Search (Current work), 10Growth-Team (Current Sprint): Newcomer tasks: ORES ontology mapping and score thresholds - https://phabricator.wikimedia.org/T244192 (10Tgr) >>! In T244192#5850209, @MMiller_WMF wrote: > What I came up with is in **[[ https://docs.google... [03:25:01] 10Scoring-platform-team, 10Discovery-Search: Consume ORES articletopic data from Kafka and store it in HDFS - https://phabricator.wikimedia.org/T240553 (10Tgr) 05Open→03Invalid [14:10:58] o/ [14:11:31] akosiaris, I think I have taken https://phabricator.wikimedia.org/T242705 as far as I can. [14:12:08] o/ [14:12:40] that strace looks interesting. [14:12:42] I'm not sure I made much progress, but I tried a lot of things. [14:12:43] 10Jade, 10Scoring-platform-team (Current), 10Documentation, 10Patch-For-Review: Jade local dev setup / README docs - https://phabricator.wikimedia.org/T244152 (10kevinbazira) @ACraze I've reviewed your patchset and the README looks good. Just wanted to confirm though, is `composer test` and `grunt` run in... [14:12:44] Oh good! [14:13:01] the child is trying to release about 400MB of memory or so [14:13:20] every munmap call is 32MB [14:13:49] and there are 14 of them [14:13:53] That tracks with the RES of each child process. [14:14:00] They report about 400MB or RES [14:21:13] It seems like the munmap calls don't appear right away. They only appear after the CPU and memory spikes. [14:21:47] Would you like a safe machine to test on? See ores-staging-01.eqiad.wmflabs. [14:22:04] It uses mostly identical puppet. [14:22:27] I 've depooled from traffic ores1001 [14:22:34] it's not serving any requests and I am testing there [14:22:54] interestingly enough I got 78 such calls on a freshly restarted worker [14:23:02] so more like 2.4G of memory were released... [14:23:12] weird [14:24:26] So, a celery worker will report about that much RES [14:24:47] Before we made changes to control what is loaded into memory for a uwsgi worker, it would load the same amount of RES [14:31:06] 10ORES, 10Scoring-platform-team, 10Patch-For-Review: Review prometheus ORES rules for completeness - https://phabricator.wikimedia.org/T233448 (10Halfak) OK that sounds good. What are the next steps for updating the dashboards? How do we map our current metrics onto Promethius-generated metrics. [16:23:52] akosiaris, any thoughts on our next steps for this uwsgi issue? I'm working on getting this all running on my laptop to see if a different version of uwsgi has this issue. [17:00:12] 10Jade, 10Growth-Team, 10MediaWiki-General, 10MediaWiki-extensions-FlaggedRevs, and 6 others: Hard deprecate Title::userCan - https://phabricator.wikimedia.org/T244923 (10DannyS712) [17:05:12] halfak, standup? [17:05:28] Oh crud. I'm double-booked. Forgot to say. I need to skip standup today [17:05:31] accraze, ^ [17:05:48] I'll async in here. [17:05:52] lol ok no worries! [17:06:28] Y: Mostly focused on uwsgi restart issue for ORES. No clear solution but tried a lot of things. All in the task. [17:06:58] T: A but more uwsgi restart. Trying to see if I can replicate on my laptop with a different version of uwsgi. [17:07:16] Otherwise, I have some media training (in that right now) and I'll continue with getting vagrant working. [18:33:09] Media training complete! [18:33:14] Grabbing lunch [19:48:10] 10Jade, 10Growth-Team, 10MediaWiki-General, 10MediaWiki-extensions-FlaggedRevs, and 7 others: Hard deprecate Title::userCan - https://phabricator.wikimedia.org/T244923 (10DannyS712) [19:51:26] uwsgi is finally alive on my laptop. What a slog. [19:51:39] I couldn't get any logs because I was running it under the wrong user :| [19:54:36] OK confirmed that it does the CPU/memory spike when shutting down [19:54:44] Trying some new things with the config... [19:55:23] Yeah. No dice. [19:55:36] OK. I did my best. Onto vagrant! [19:56:48] 10Jade, 10Growth-Team, 10MassMessage, 10MediaWiki-General, and 9 others: Hard deprecate Title::userCan - https://phabricator.wikimedia.org/T244923 (10DannyS712) [19:57:44] 10Jade, 10Growth-Team, 10MassMessage, 10MediaWiki-General, and 9 others: Hard deprecate Title::userCan - https://phabricator.wikimedia.org/T244923 (10DannyS712) [20:27:33] 10ORES, 10Scoring-platform-team, 10Patch-For-Review: Review prometheus ORES rules for completeness - https://phabricator.wikimedia.org/T233448 (10colewhite) I went ahead and updated [[ https://grafana.wikimedia.org/d/HIRrxQ6mk/ores?orgId=1 | this dashboard ]] and added the Prometheus version next to the Grap... [20:27:34] 10[1] 04https://meta.wikimedia.org/wiki/https://grafana.wikimedia.org/d/HIRrxQ6mk/ores%3ForgId%3D1 [20:47:55] halfak: I was trying of getting uwsgi out of the picture for some tests. exhonerate/incriminate it by say running it under gunicorn (another uwsgi server - pure python one). if under gunicorn the same behavior happens, we know it's not uwsgi to blame. otherwise it is to blame. [21:09:42] 10Jade, 10Growth-Team, 10MassMessage, 10MediaWiki-General, and 9 others: Hard deprecate Title::userCan - https://phabricator.wikimedia.org/T244923 (10DannyS712) p:05Triage→03Medium [21:22:32] 10Jade, 10Growth-Team, 10MassMessage, 10MediaWiki-General, and 9 others: Hard deprecate Title::userCan - https://phabricator.wikimedia.org/T244923 (10DannyS712) Woops, wrong task to link [21:28:41] akosiaris, that makes sense. Sorry was in meeting -- I hope you're enjoying your evening/night :) [21:29:06] Doing a reboot to register my owner key so I can use virtualbox :| [21:39:55] It works! [21:40:06] That was the most backwards weird thing I have done with Linux in a long time. [21:40:48] * halfak starts the download of the giant VM image [21:42:42] ...and I have a cryptic error [22:07:42] Uninstalling it all and trying again from scratch [22:12:22] 10Scoring-platform-team, 10Discovery-Search, 10Epic, 10Growth-Team (Current Sprint): [EPIC] Growth: Newcomer tasks 1.1.1 (ORES topics) - https://phabricator.wikimedia.org/T240517 (10Tgr) [22:21:49] Rebooting again [22:26:40] I think it might be working! [22:28:11] 1) I need to sign my kernel modules. That requires a couple of reboots [22:28:27] 2) I needed to enable direct memory virtualization in bios. Another reboot. [22:28:53] 3) An old virtualbox kernel models were hanging around messing things up so I had to manually remove them and re-sign the new modules. [22:29:06] And now I'm running puppet :) [22:35:26] I'm going to need to hit the road while I let puppet run. I'm really hoping I can pick up review work again tomorrow. [22:35:31] Take care, folks. [23:12:36] 10Scoring-platform-team (Current), 10Discovery-Search (Current work), 10Growth-Team (Current Sprint): Newcomer tasks: ORES ontology mapping and score thresholds - https://phabricator.wikimedia.org/T244192 (10MMiller_WMF) @Tgr -- I think they look fine. Thanks.