[00:34:28] Don't let the scope get out of hand. We just need to make sure that our solution helps us come back to this later to look for trends. [00:34:48] Also this goddamn model has been building for 1.5 hours! [00:35:00] Gimme my CPU core back! [00:37:54] :D [00:38:08] I thought It takes a little bit time :D [00:52:30] halfak: just found out that we had a bug in puppet that prevented uwsgi from restarting after config changes (like worker counts!) were applied [00:52:45] just fixed it, but this meant that when you were tuning worker counts that might not have actually been reflected in actual workers [00:52:51] unless you restarted uwsgi manually [00:53:31] * YuviPanda heads off for a few hours [01:07:26] YuviPanda, was restarting uwsgi manually [01:07:42] I counted the uwsgi processes [01:07:58] Or rather, I looked at how many there were and decided that 3 pages of scrolling was about 256 [01:11:36] I haven't had a chance to pick it back up again. I hope to spend some time on it tomorrow afternoon CST [01:21:13] Still building the model... probably? [01:22:57] I mean, it's not done. It's just that I'm skeptical it is doing anything useful. [01:24:16] We multiplied the data by 5 and we went from ~10 minute build to a 3+ hour build [01:24:21] * halfak whistles [01:24:32] Might just let this run overnight [01:52:48] halfak: thanks :) [01:56:09] halfak: https://phabricator.wikimedia.org/T117425#1773790 [01:56:22] shuf rev_features.tsv > rev_features.shuffled.tsv [01:56:23] $ head -n 15000 rev_features.tsv > rev_features.train_set.tsv [01:56:45] the second line should be head -n 15000 rev_features.shuffled.tsv > rev_features.train_set.tsv [01:56:54] am I wrong? [14:04:46] OMG it finished! [14:06:39] Amir, .82 AUC [14:06:43] So no real change. [16:44:33] wiki-ai/wb-vandalism#63 (comment - f6b9b21 : amir): The build has errored. https://travis-ci.org/wiki-ai/wb-vandalism/builds/89682380 [17:25:32] * halfak sees aetilleys progress report :) [17:27:05] o/ YuviPanda, do you think you can help me strace what the heck is going on with the staging server? [17:27:12] ^ today [17:27:20] Not necessarily right now ^_^ [17:40:55] halfak: hopefully, but someoen broke glass in the house and didn't clean it up and guess who else is walking on one foot today >_> [17:41:04] so that has to be priority 1 unfortunately :( [19:18:33] Oh no! YuviPanda that's the worst! [19:19:11] I actually have been battling some glass shard in my wrist for about a week. They are so small that I can't get them with tweezers. [19:19:33] *shards -- like 3 or 4. [19:19:45] So the only option was just wait for my skin to grow out a bit. Eventually I'd slough them off. [19:20:15] In the meantime, every time I put my palm down on something, it feels like it is covered in spikes. [19:20:29] Luckily, the mouse & keyboard doesn't put enough force down to hurt. [19:20:37] Riding my bike was pretty interesting though. [19:51:58] o/ Amir1 [19:52:10] o/ halfak :) [19:52:17] Do you use "#REDIRECT" when making a redirect in fawiki or something localized? [19:52:37] we have localized one but #REDIRECT also works [19:52:55] Gotcha. Thanks. [19:53:10] * halfak needs to identify redirects cross-wiki for the article quality work. [19:54:08] Do you have an idea for how we might find that localization in javascript or site config? [20:00:00] I will find out [20:00:09] I'm pretty sure it's in API [20:00:18] because pywikibot finds it [20:00:24] let me digg in the code [20:00:29] Thanks [20:00:39] Oh! You can get it from the API sure. [20:00:44] I'm hoping to get it from the text :/ [20:01:17] Or you mean the localization [20:04:18] I meant localization [20:06:03] You can get them from list of magic words in API [20:06:19] Gotcha. Maybe in siteinfo [20:07:34] Got it! [20:07:35] https://fa.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=magicwords [21:16:07] halfak: a quick heads up. They already reached out to Lydia [21:16:17] I don't know what they told to her [21:16:19] *sigh* [21:16:25] but I forwarded our thread [21:16:44] Looks like this might spiral [21:17:49] Honestly, I would be happy to work with them. I do not want to agree that we'll not publish about our work. I do not want to give people credit for a codebase that they did not contribute to. But that does not mean that we can't work together. [21:21:11] Amir1, am I misremembering? Did we somehow agree to not publish about our work? [21:21:46] Maybe some discussion happened in the meeting I didn't attend about what features would be most effective? [21:21:57] * halfak is trying to figure out why they are so adamant. [21:32:32] Amir1, I don't see the email. Could you CC me on the discussion with Lydia? [21:40:06] o/ awight [21:41:07] could you give us an update on this task? https://phabricator.wikimedia.org/T112856 [21:41:18] I'd like to make sure I'm not blocking you :) [21:41:20] sorry halfakI was afk [21:41:33] halfak: [21:42:03] I forwarded out thread with nothing more to Lydia [21:42:05] *our [21:42:13] halfak: hey, not at all, it's just me being distracted. [21:42:50] I think we have an MVP already, it just needs testing [21:43:05] halfak: I know we can't not publish our code, it's rule of our work with WMF (and WMDE), I won't promise such thing [21:43:05] Pretty sure the maintenance jobs are everything we'll need for the first iteration. [21:43:11] awight, test cases or live testing? [21:43:42] halfak: Good point. I was imagining other people running it locally and reviewing, but it would benefit from unit tests. [21:44:26] awight, I think I can be a good ally for local review. Not sure about writing the tests in PHP. [21:44:38] Understandable :) [21:44:50] Yeah, sorry the "fun"draising season is nigh [21:45:14] So I guess the first step is to try to install it and I'll document whatever seems unclear to you. [21:45:15] halfak: ? [21:45:30] Amir1, to PM [21:45:40] sure :) [21:47:01] awight, understood. [21:48:01] halfak: I could also use some dialogue about the maintenance jobs... [21:48:19] awight, https://phabricator.wikimedia.org/T118039 [21:48:33] awight, happy to. Please direct me to the right spot. :) [21:49:26] We should probably arrange a security review and whatever else is needed before we can deploy anywhere on the main cluster. What's the release plan? testwiki only, or are we trying to get wiki communities to approve the new extension? beta feature? [21:49:51] beta feature [21:50:02] ok. I don't know what the steps are to deploy one of those [21:50:05] We have a security review of revscoring completed and I'm working on the feedback [21:50:13] of the extension as well? [21:50:18] aNegative [21:50:33] YuviPanda, said "There's a process for that" in a way that I stopped worrying about it. [21:51:06] :D [21:51:11] well it's taken care of, then :p [21:51:48] Suddenly I'm starting to worry about that again :S [21:53:40] hehe [21:54:19] It'll be a quick review, very little code that doesn't do much, but we don't have many people qualified to do that review. [21:56:31] awight, OK. I'll take on the task of figuring out what type of review we need for it. [21:56:41] halfak: Do you have a vagrant role yet? [21:56:44] I don't see one [21:56:44] BTW: Live test ORES extension: https://phabricator.wikimedia.org/T118039 [21:56:52] I don't think so. [21:56:55] yes! responding there now. [21:57:08] I could make a new role, not sure it makes sense, though [21:57:24] The nasty part about testing is that your revision IDs need to match things in the real world. [21:57:31] (if your config is pointed to the real server) [21:57:50] Otherwise, we need to install the entire toolchain in a dev environment so you're actually scoring things from your local wiki. [21:58:03] I just fast-forwarded my revision table, I'll note that in the task. [21:58:38] awight, can stand up a fake ORES. [21:58:42] That's OK with me. :) [21:59:13] Will make a great prediction. Last two digits of rev_id is vandal probability >:) [21:59:14] okeydokey, that makes testing even more fun. I'll assume you're doing so in my instructions [21:59:21] +1 [22:06:08] Eventlogging has a nice little demo server that comes with vagrant. I'll try to get one of those going for ORES. [22:08:37] I'm happy to help at a low level, I became very familiar with the vagrant puppet files, in the line of Fundraising duty [22:13:15] That'll be helpful. It's been a long time since I contributed to vagrant. [22:13:42] I'll see about making a test server really easy to stand up. I think it should be pretty trivial. [22:18:09] d'oh: two changes still need code review [22:21:05] awight, can you add those to our Review column. [22:21:40] Also, thanks for looking at this stuff with me. I know you're busy and I appreciate it. :) [22:23:46] I feel terrible that I had to drop more fun side things [22:24:12] (03PS3) 10Awight: Always return true from hooks; docstrings [extensions/ORES] - 10https://gerrit.wikimedia.org/r/247790 (https://phabricator.wikimedia.org/T112856) [22:24:27] Na. Please dpm [22:24:30] *don't [22:24:40] You're awesome and volunteering for this work. [22:24:44] :D [22:24:52] Next year ;) what's your budget look like... [22:25:34] halfak: Wanna suggest some language here, https://gerrit.wikimedia.org/r/#/c/247185/6/i18n/en.json,unified [22:29:34] "Draft saved" -- did it actually post my comment? [22:30:14] nah. Now you have to "review" to publish comments [22:30:38] Don't worry--pretty soon we'll migrate everyone to another totally fringe and unmaintainable CR system [22:30:54] yay! [22:31:43] (03CR) 10Halfak: Flag reverted risk rows using the recentChangesFlag (032 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/247185 (https://phabricator.wikimedia.org/T112856) (owner: 10Awight) [22:33:20] "ores-reverted-legend": "ORES predicts that this change may be damaging and should be reviewed" [22:33:23] ? [22:33:37] I think that sounds pretty good [22:34:54] (03PS7) 10Awight: Flag reverted risk rows using the recentChangesFlag [extensions/ORES] - 10https://gerrit.wikimedia.org/r/247185 (https://phabricator.wikimedia.org/T112856) [22:58:18] legoktm: where's the checklist for new extension deploys? [22:58:54] YuviPanda: https://www.mediawiki.org/wiki/Review_queue short https://www.mediawiki.org/wiki/Writing_an_extension_for_deployment long [22:59:11] right, ORES goes in the former I suppose [22:59:13] kindof [22:59:50] the checklist on [[Review queue]] is what's actually used [23:09:35] YuviPanda, will you have time to look at the weird queue issues with me today? ("No" is totally reasonable. It's 5PM here and I've opened a beer.) [23:10:12] * halfak is so incredibly confused by gerrit [23:18:00] I just had an idea about what could be causing the queue to not fill up on staging. [23:18:30] Maybe by maxing CPU, I'm blocking the code that precedes addition to the celery queue as well as the celery jobs. [23:18:45] So, the uwsgi queue gets full, [23:19:15] In order to test this, I need to "stess" the celery workers without "stress"ing the uwsgi process that happens after the initial queue [23:19:42] I think I'm going to hack the code to induce a sleep() in the celery processor. [23:19:57] Since we load ORES from a submodule, this should be pretty easy. [23:20:17] Yup... I finished typing that and it still makes sense. ONWARD [23:38:05] It works! [23:38:09] YuviPanda, :DDDDDD [23:38:11] WOOOOOO [23:38:36] Lesson learned. When trying to stress test celery, don't also stress test uwsgi or it will be have weirdly. [23:38:46] Now to figure out how many uwsgi processes we really need. [23:53:35] So... any time I restart uwsgi now, I get a connection dropped with the staging server [23:53:49] But it still seems to restart [23:57:16] OK... So I think I've figured out another problem. uwsgi will hang onto a request for a long time. We should have uwsgi kill a request that takes celery-timeout (15s) + 5 seconds = 20 seconds.