[00:59:25] 10ORES, 10Scoring-platform-team: ores.wmflabs.org - 503 icinga alerts - https://phabricator.wikimedia.org/T242819 (10Dzahn) [01:00:33] 10ORES, 10Scoring-platform-team: ores.wmflabs.org - 503 icinga alerts - https://phabricator.wikimedia.org/T242819 (10Dzahn) [01:03:06] 10ORES, 10Scoring-platform-team, 10Operations: ores.wmflabs.org - 503 icinga alerts - https://phabricator.wikimedia.org/T242819 (10Dzahn) [01:10:36] ^whoops forgot to set back [13:27:57] o/ kevinbazira [13:28:44] o/ halfak [13:29:00] I see you're running some fasttext ^_^ [13:30:12] I'm working on the pipeline too. Did you find my "w_labels" files? [13:30:31] Yep, I'm generating unsupervised skipgram 50 cell embeddings. [13:30:51] Yes I saw the "w_labels" files [13:32:30] We can use those to generate supervised embeddings -- which seem to perform better based on isaacj's work. [13:37:32] I'm working on a strategy for us to extract the vectors from the supervised model. [13:37:55] Alright, I'll start the process of generating the supervised embeddings soon as the enwiki unsupervised embeddings process completes. [13:39:17] OK sounds good. [13:39:37] I'll be able to share some code with you. I'm just putting some finishing touches on it. [13:39:48] Regretfully, my XML processor is behaving strangely. [13:40:00] My multiprocessing pool is hanging and I'm not sure why. [13:44:54] 10ORES, 10Scoring-platform-team, 10Operations: ores.wmflabs.org - 503 icinga alerts - https://phabricator.wikimedia.org/T242819 (10Halfak) Looks like celery has shut down on all of the workers. I'm looking into it now. I think we might be too close to the memory ceiling and an OOM is what's killing them.... [13:55:24] 10ORES, 10Scoring-platform-team, 10Operations: ores.wmflabs.org - 503 icinga alerts - https://phabricator.wikimedia.org/T242819 (10Halfak) Something really strange is going on. I cut our celery workers in half an we're still not able to actually start up celery because we get a MemoryError during the startu... [14:54:30] halfak: there's code for extracting embeddings etc. from fasttext in my code here if that's helpful: stat1005:/home/isaacj/topic_models/topic_modeling.py [14:54:41] thanks isaacj [16:38:50] 10ORES, 10Scoring-platform-team, 10Operations: ores.wmflabs.org - 503 icinga alerts - https://phabricator.wikimedia.org/T242819 (10Halfak) a:03Halfak [16:39:16] 10ORES, 10Scoring-platform-team, 10Operations: Ores celery OOM event in codfw - https://phabricator.wikimedia.org/T242705 (10Halfak) a:03Halfak [17:59:44] https://github.com/wikimedia/drafttopic/pull/44 [17:59:51] No more duplicate labels! [17:59:55] Was totally innocuous. [18:22:05] nice [18:30:36] <3! [18:30:41] Thanks for the merge [18:38:33] 10Scoring-platform-team, 10Discovery-Search, 10Growth-Team: Allow searching articles by ORES drafttopic - https://phabricator.wikimedia.org/T240517 (10MMiller_WMF) [19:06:50] lunch! [20:20:18] 10ORES, 10Scoring-platform-team, 10Operations: ores.wmflabs.org - 503 icinga alerts - https://phabricator.wikimedia.org/T242819 (10Dzahn) p:05Triage→03Normal [20:21:02] 10ORES, 10Scoring-platform-team (Current), 10Operations: Ores celery OOM event in codfw - https://phabricator.wikimedia.org/T242705 (10Halfak) [20:29:51] 10ORES, 10Scoring-platform-team, 10Operations: ores.wmflabs.org - 503 icinga alerts - https://phabricator.wikimedia.org/T242819 (10Halfak) Looks like the OOM error might have been old. Here's what I have now: ` $ sudo -u www-data ../venv/bin/python ores_celery.py /srv/ores/venv/lib/python3.5/site-package... [21:31:14] 10ORES, 10Scoring-platform-team (Current), 10Operations: ores.wmflabs.org - 503 icinga alerts - https://phabricator.wikimedia.org/T242819 (10Halfak) [21:36:10] 10Jade, 10Scoring-platform-team, 10Documentation: Write Jade documentation for secondary integrations - https://phabricator.wikimedia.org/T242008 (10Halfak) p:05Triage→03Low [21:36:37] 10Scoring-platform-team, 10WMF-Legal, 10Wikilabels: Add legal language to wikilabels - https://phabricator.wikimedia.org/T223313 (10Halfak) Ping. [21:45:02] 10Scoring-platform-team, 10drafttopic-modeling, 10revscoring, 10artificial-intelligence: Add wikidata features to topic models - https://phabricator.wikimedia.org/T241270 (10Halfak) p:05Triage→03Normal @dr0ptp4kt, trying to figure out how to prioritize this. We're wondering if there's a straightforwar... [22:32:23] ores.wmflabs.org is coming back online. [22:32:29] * halfak wipes brow [22:52:35] 10ORES, 10Scoring-platform-team (Current), 10Operations: ores.wmflabs.org - 503 icinga alerts - https://phabricator.wikimedia.org/T242819 (10Halfak) Turns out it was the statsd host. It changes from labsmon1001 to cloudmetrics1001. Now that I've done a new deployment with an updated config, we're back onli... [22:56:19] And, the ores paper is submitted [22:56:22] * halfak flexes. [23:53:51] 10ORES, 10Scoring-platform-team (Current), 10Operations: ores.wmflabs.org - 503 icinga alerts - https://phabricator.wikimedia.org/T242819 (10Dzahn) 05Open→03Resolved {F31513810, size=full}