[10:05:50] hello ınternet [10:06:10] halfak I dont know if you are there or if you are about to fly (probably sleeping) [14:02:07] o/ [14:02:12] ToAruShiroiNeko, [14:02:28] hello [14:02:40] I do not know if you have the time but I have two uikies [14:02:48] tr wiki is near completin with its campaign [14:03:03] do you think you can release the worksets people are sleeping over [14:03:17] Cool! Would be great to have a new model for TR. [14:03:27] Yeah. We can. [14:03:32] and also [14:03:35] in my work set [14:03:42] one of them seems to be problematic [14:03:44] I have 49 done [14:03:48] 1 I cannot see [14:03:53] not sure if it is deleted [14:06:15] Does it show up with a deleted error? [14:06:20] Or can't find revision? [14:07:26] brb getting coffee [14:10:34] It shows nothing [14:10:37] as if I am clicking nothing [14:10:48] it is the only one I have not labelled [14:11:17] if you could look for things assigned to me on trwiki that I have not labelled you should see it. I am unsure what the uery would look like [14:11:34] its not too important but it could be a bug we missed [14:30:43] Maybe you could tell me the revid [14:30:57] Maybe you could look in the dev console for an error [14:38:01] ToAruShiroiNeko, ^ [14:38:24] Do you know your global user id? [14:39:28] I show 76 unlabeled tasks [14:41:03] 66 of them are assigned to someone. [14:43:13] OK. All unassigned. [14:49:37] https://meta.wikimedia.org/wiki/Talk:Wiki_labels/Work_log/2015-10-04 [15:03:46] o/ Amir [15:09:21] o/ halfak [15:09:27] hey [15:09:29] Sorry for not being around [15:10:08] Don't sweat it. I knew you had stuff going on. [15:10:14] How's the state of wb-vandalism? [15:11:08] I'm finishing off tests [15:11:21] wrote about 50 tests [15:12:08] OK. We'll want to get the data flowing for that soon so that we can get some baseline models together. [15:13:33] https://github.com/Ladsgroup/wb-vandalism/commits/master [15:14:44] https://github.com/Ladsgroup/wb-vandalism/blob/master/wb_vandalism/features_list/wikidata.py [15:14:45] :D [15:15:05] If you can fill out what you think would make for a good feature list now, I'll use that to dig into what you have and how it is tested. [15:15:19] I'm guessing it's going to be nearly everything in features? [15:16:34] sure thing [15:20:11] halfak: yes it would be everything and some stuff from revscoring (e.g. user-related features) [15:21:02] Cool. BTW, I posted stats for the fawiki wikilabels models [15:21:24] See here: https://en.wikipedia.org/wiki/Wikipedia_talk:Labels/Edit_quality#Complete.21 [15:21:40] 94 AUC for both damaging and goodfaith. [15:21:58] Oh wait. More like 95 for damaging :D [15:22:06] Awesome awesome! [15:22:12] great! [15:22:18] I've been working on getting those deployed. [15:22:21] When do we deploy this? [15:22:26] I'll *at least* be testing on staging today [15:22:28] :D [15:22:38] * halfak has been working hard to get these things up [15:22:42] Is there any kind of help that I can do? [15:22:55] Right now, no, but I'll have a series of pull requests to review soon. [15:23:08] I did a lot of self-merging yesterday because I was impatient :S [15:23:30] We're going to need to change our policy or set up a better review process. [15:23:40] Had some sitting for 2.5 weeks again. [15:24:06] I've got a meeting in 30 mins for a workshop -- after that I'll kick into gear. [15:24:11] So, 1.5 hours from now. [15:26:02] In the meantime, I'm rebuilding our models [15:26:07] So that we can report stats better. [15:26:29] :D [15:26:33] Check this out: http://ores-staging.wmflabs.org/scores/enwiki/reverted/ [15:26:40] Soon that'll report AUC stats [15:26:51] And the truth table of the training set. :) [15:26:58] okay [16:00:32] Check this out: http://ores-staging.wmflabs.org/scores/enwiki/wp10/ [16:00:34] * halfak runs to meeting [16:28:18] Ladsgroup/wb-vandalism#18 (master - f4c146d : amir): The build was fixed. https://travis-ci.org/Ladsgroup/wb-vandalism/builds/83568515 [16:30:18] halfak: ^ [16:30:22] :) [16:33:09] https://codecov.io/github/Ladsgroup/wb-vandalism [16:33:11] and this [16:33:42] Tonight you will have all of them, also probably revscoring tests will pass [16:34:00] I have to wait until a new patch comes [16:57:55] OK back. [16:58:41] enwiki 'reverted' AUC is up to .84 [16:58:49] That's a substantial improvement. [16:58:56] I've been working with the feature set. [16:59:44] ptwiki 'reverted' AUC is up to .88 [16:59:46] woot [17:02:42] \o/ [17:08:13] halfak: I'm here now if you want some deploy stuff [17:08:33] Just about to push the new models to staging. :) [17:08:39] halfak: awesome [17:08:53] is this including the metrics stuff and the symlink stuff? [17:09:04] This'll add metrics collection, unblock the ORES extension and double the number of models we have hosted. [17:09:14] :D [17:09:15] nice [17:10:21] adding new models means we'll have mode burden for the precaching system [17:10:46] I think we've been doing ok in terms of load [17:10:50] tools.wmflabs.org/nagf [17:11:03] This'll triple the load, I think [17:11:38] should probably still be ok [17:11:44] http://ores-staging.wmflabs.org/scores/enwiki/damaging/32243421/ [17:11:46] It works! [17:12:21] http://ores-staging.wmflabs.org/scores/enwiki/?models=damaging|reverted|goodfaith&revids=32243421 [17:12:22] can you ssh in and verify the submodules? [17:12:44] e.g. that our imports are coming from the submodules? [17:14:02] yeah [17:14:10] https://gist.github.com/halfak/7db5728056d47e54246e [17:14:14] yuvipanda, ^ [17:14:18] and that the submodules are checked out to appropriate buts [17:14:29] buts? [17:14:31] cooool [17:14:33] gah [17:14:35] bits [17:14:39] sha1s [17:15:28] Updated: https://gist.github.com/halfak/7db5728056d47e54246e [17:15:37] Our butts are solid [17:15:41] +1 [17:16:19] I'm just going to run the precached for a bit to make sure it goes well on staging. [17:17:37] Can now check timings in graphite :D [17:18:39] halfak: I see bar, foo and foobar? [17:18:53] ooooh [17:18:55] ofc [17:18:58] I see them on -staging [17:18:58] Under testing :D [17:19:00] ! [17:19:02] nice [17:19:03] Yup :) [17:19:25] I had a bit of a hickup, but it wouldn't be unfair to say that statsd worked as intended on the first try [17:19:32] Ot [17:19:43] yay [17:19:50] It's hard to test locally though, so there's a "logging" based metrics collector option. [17:19:55] ah nice [17:20:01] yeah that's doubly good I think [17:20:03] There's also a Null that works like /dev/null by default [17:20:12] testing statsd is always a PITA [17:21:13] Looks like we're doing pretty good. [17:21:44] yeah [17:22:28] CPU is a little high [17:22:35] But staging is just one machine [17:22:46] Doing both web and celery [17:23:38] do you want us to add more celeries before deploying? [17:23:46] Na. I think we're going to be fine. [17:24:04] So long as precaching doesn't take more than one whole host of 4 cores/16 workers, we're set. [17:24:27] Looks like we're at about 90% CPU [17:24:34] We were at 60 for a while [17:24:35] on -staging? [17:24:40] Yeah [17:24:45] I'm looking at user CPI [17:24:46] *CPU [17:25:27] yeah it's almost all CPU as well [17:25:29] err [17:25:31] celery [17:25:33] nice [17:25:50] OK. I think we're ready. [17:26:11] FWIW, my last deploys to staging had no issue once I fixed my statsd hiccup. [17:26:18] I'm 3 deploys past that. [17:26:31] nice [17:26:34] We'll need to do an update of requirements.txt, but I don't expect to compile anything. [17:26:58] So, I should start by running the update and then deploy? [17:27:02] my trust of pip has gone down a bit [17:27:04] but yes [17:27:09] let's do that than the other way around [17:27:16] pip + wheels might not be so bad. [17:27:21] That's another conversation. [17:28:02] indeed, but we should switch to debs soon [17:28:12] now that the quarter is over I should have more time [17:28:22] maybe this week, I hopefully think [17:28:25] Cool. What was crunching last quarter? [17:29:16] there were two labs goals and I spent the first half of the quarter driving one to completion as much as possible and then the second half driving the other [17:29:39] tools.wmflabs.org/nagf is now running on k8s (the second goal!) and labs had >99.5 uptime! [17:30:07] kubernits? [17:30:11] koobernets [17:30:38] Anyway, that's pretty awesome. [17:31:20] Ack! Out update virtualenv doesn't update git first! [17:31:26] yeah but I also got smashed the last few weeks since I also moved and had this big life style change.... [17:31:50] bah [17:32:00] halfak: you can do fab update_git to update git [17:32:23] cool [17:32:46] update_git only ran on the web hosts. [17:32:50] * halfak checks it out. [17:33:11] hmm [17:33:15] it has @roles('web') [17:33:24] needs to be @roles('web', 'worker') [17:35:46] yuvipanda, https://github.com/wiki-ai/ores-wikimedia-config/pull/37 [17:35:49] Quick check? [17:36:02] halfak I am stıll asigned it [17:36:06] I dont see the rev id [17:36:22] I am not familiar with the debug api [17:36:28] ToAruShiroiNeko, I just cleared all assignments that have no label. [17:36:28] halfak: merg [17:36:30] d [17:36:38] ToAruShiroiNeko, see javascript console in browser [17:36:44] thank yuvipanda [17:36:56] looking [17:37:10] Mind that its still blocking me from asking for new workset [17:37:34] looking at java [17:37:52] in page source? right [17:37:54] ? [17:37:54] ToAruShiroiNeko, it shouldn't be. [17:37:58] What is error [17:38:00] I need more details [17:38:12] I know but I get no errors [17:38:18] ToAruShiroiNeko, javascript. GOogle "developer console" [17:38:25] Refresh page and try again? [17:38:38] I tried ctrl+f5 [17:38:44] it just shows the prior work item [17:38:52] and de-labells [17:38:59] prior as in what I clicked before [17:39:06] Dunno what you' [17:39:08] re saying [17:39:15] I can screen share [17:39:40] * halfak waits for web nodes to restart [17:39:43] :D [17:39:52] did pip try to compile anything? [17:39:57] Nope [17:40:13] Pulled down my wheels though :D [17:40:28] nice [17:40:56] halfak_on_wheels working on developer console [17:41:01] kk [17:41:10] OMG RESTART FASTER [17:41:54] so I have a list of pre-caching diff for ##### [17:42:05] That's right [17:42:22] haha the restart is slow as fuck isn't it [17:42:27] They do not look like 50 items [17:42:29] OMG WHY [17:43:02] aaand we're down [17:43:06] ugh [17:43:59] halfak I am struggling to figure out the info [17:44:09] ToAruShiroiNeko, can't help now [17:44:12] yuvipanda, we're down [17:44:17] uh ok [17:44:18] Logs have 500 -- no error messgaes [17:44:26] looking [17:45:22] missing statd dependency [17:45:24] arg! [17:45:32] hot fixing [17:45:34] ok [17:47:21] * halfak restarts web [17:48:01] gooooooo [17:48:17] workers should be good [17:48:41] We're up [17:48:48] 4 minutes [17:48:51] Not too bad [17:48:59] Could be attributed to a network blip [17:49:21] sorry I didn't catch that in CR [17:49:37] No worries. I was self-merging a lot yesterday :/ [17:49:40] I'll fix that right now. [17:49:50] SO statsd is optional. Should be in ores-wikimedia-config. [17:50:04] Also, look at this beautiful thing: http://ores.wmflabs.org/scores/enwiki/ [17:50:15] Get yer stats! [17:52:13] yuvipanda, https://github.com/wiki-ai/ores-wikimedia-config/pull/38 [17:52:16] I think that'll do it. [17:52:34] * halfak starts the precacher back up. [17:52:55] Amir1, new models are live :D [17:53:09] See http://ores.wmflabs.org/scores/fawiki/ [17:55:27] halfak: congratulations! [17:55:38] :D [17:55:48] Thanks for being around to help today. [17:55:58] One take-away, I want this deploy to get *way* more stable. [17:56:09] At some point we'll stop developing revscoring/ores so aggressively. [17:56:23] we need to also figure out why the restarts are taking so lon [17:56:25] g [17:56:30] Yeah. Any ideas? [17:56:39] yeah I think it's waiting for requests to drain [17:56:47] so when you ask it to restart, it stops accepting new connection [17:56:55] and waits for current connections to finish [17:56:58] before restarting [17:57:01] across the 8 processes [17:57:13] this is so we don't actually actively kill any connections - 'graceful restart [17:57:15] ' [17:57:23] that seems to match observed behavior I think [17:58:00] That might not be bad then. [17:58:20] I can try shutting down the precached a little earlier next time. [17:58:27] mmm maybe yeah [17:58:36] Then again, restart takes super long on staging too. [17:58:50] is there a precached hitting staging too? [17:59:01] we can test this theory actually [17:59:12] is it just restarts that are slow or even starts? [17:59:18] if restarts are slow but starts aren't [17:59:24] then it's graceful restart [17:59:40] if start is also slow then it's just us loading to many things into memory too slowly and we can take a profiler to that [18:00:40] The precached does not hit staging. [18:00:49] Only for brief periods when I am testing. [18:01:06] ok then we should do the test and see how that fares [18:01:16] yuvipanda, could be related to memory loading. It does take a moment to boot up something that imports revscoring. [18:01:34] yeah totally could be [18:01:52] halfak: I'm going to test it on staging [18:02:09] hmm [18:02:13] the stop is taking forever [18:02:27] precached is not going [18:02:32] you can tell by the graphite [18:03:01] It takes 1s to load revscoring and read a model file. [18:04:06] Looks like the main issue is loading a language [18:05:34] We're not spending much time loading after the first language though. [18:05:35] https://gist.github.com/halfak/e10a81604f626a70426b [18:05:42] NLTK is probably putting some stuff in memory. [18:08:46] halfak: interesting. in staging 'stop' takes 1m30s and 'start' is subsecond [18:09:01] restart is still running [18:09:22] Weird that it takes so long to stop [18:09:33] Must be iterating through each worker. [18:09:42] *flask worker [18:10:22] yeah so restarts are also 1m30s [18:10:32] so it looks like 'stop' takes way more time than start [18:19:52] New models listed at https://meta.wikimedia.org/wiki/Objective_Revision_Evaluation_Service [18:22:31] halfak: yay nice :) [18:27:26] Looks like I messed up the keys in the models. [18:27:36] It thinks that "True" and "False" are strings. [18:27:50] I'll need to do another deploy quick once I update the models so that they know the label is boolean. [18:28:01] This is a minor fix. [18:28:38] ok [18:28:52] * halfak regenerates models. [18:28:57] I'll be going to staging first. [18:29:04] Anything you want me to try? [18:30:06] I guess you wouldn't need to run pip install [18:31:59] shouldn't [18:32:08] Just pulling the models from git and updating [18:32:16] btw, did you see large file storage? [18:32:24] It seems like that would be perfect for our models. [18:32:30] The wp10 one is 80MB [18:32:46] Goddamn random forrest [18:33:34] I think at 80 regular git is ok for us [18:33:52] we've to deploy to production from gerrit, remember. I don't know if that uspports LFS [18:34:05] Gotcha [18:36:12] * halfak goes to staging with the new models [18:36:35] * halfak waits for restart and dies of hunger [18:36:55] It's 1337-o-clock in MN [18:37:42] nice [18:37:57] I'm in an introductory IEG commitee call [18:38:16] OK. I'll keep the random pings to a min :) [18:38:40] hmm... well that didn' [18:38:42] t work [18:39:23] Oh. yes it did [18:39:42] OK. Going to prod. Should be easy-peasy. [18:40:26] halfak: ok! [18:41:06] We need to have a bot that reads the info off of ores and writes it to meta.wikimedia.org/wiki/ORES [18:41:18] * halfak is getting tired of copy-pasting [18:43:43] I've thought of writing a GitHub to Wiki sync' bot type thing [18:43:50] so you can do gadget deploy by pushing [18:43:51] to github [18:44:10] That'd be pretty cool. [18:44:19] I want to be able to execute something though. [18:44:26] Some way to template the JS. [18:54:03] And {{done}} [18:54:12] ^ yuvipanda -- all is good. [18:54:29] wooooo [18:54:31] nice [18:54:36] Amir1, [18:54:38] ^ [20:46:21] halfak sprry was in a meeting then a call [20:46:25] *sorry [20:47:01] No worries. [20:47:07] DId you see the ping re. UK? [20:47:23] I was afk since my last mesage [20:48:02] UK as in united kingdom or ukranian? [20:48:12] Can you remind me? [20:48:36] Ukrain [20:48:39] Check your pings [20:48:47] enwiki [20:49:02] wait... might be meta [20:49:16] oh talk pages you mean [20:50:17] https://meta.wikimedia.org/wiki/Talk:Wiki_labels#Edit_quality_campaign_for_ukwiki [20:50:40] yeah [20:50:46] I will work on that sometime this week [20:50:54] I am uite a mess these two days :( [20:51:08] Sorry about this. But you will have this sorted by the time you are back. [20:51:34] If you have the time I want to screen share what I see with you briefly as I do not know how to reproduce the problem I am observing [20:51:49] I realise you may be packing so it can wait [20:52:16] Screen shot plz [20:54:56] there is nothing to screen shot [20:54:58] thats the problem [20:55:08] I cannot demonstrate the absence of action [20:55:25] when you click on the work item nothing happens [20:55:32] aside from labells becoming unlabelled [20:55:34] Click on what button/ [20:55:40] ? [20:55:50] the boxes for work items [20:55:53] there are 50 of them [20:55:59] when I click one of them nothing happens [20:56:02] all other 49 works fine [20:56:10] But when you click the others, something happens? [20:56:21] yes [20:56:27] the diff is loaded etc [20:56:36] Anything interesting pop into your dev console when you click on that one? [20:56:36] but if I click on this one, the last item is displayed [20:56:47] nothing changes in dev [20:57:21] thats why I am unsure what to report [20:57:32] it prevets me from getting a new work item [20:57:39] *work set [20:57:56] I can save the labelling but I dont even know what I am scoring [20:58:11] Mark it unsure and save it. [20:58:14] I'll catch it in the DB [20:58:16] ok [20:58:30] saved [21:00:49] DId you label any after it? [21:01:05] a few in the new work set I asked [21:01:11] two [21:01:12] Don't do that :P [21:01:19] Sorry :( [21:01:34] revid: 15372268 [21:01:52] Looks like I was deleted. [21:02:15] https://tr.wikipedia.org/wiki/?oldid=15372268 ? [21:02:15] So it won't matter. [21:02:17] I see it [21:02:34] https://tr.wikipedia.org/wiki/?diff=15372268 [21:02:38] Well that's interesting [21:03:09] could it be oversighted? [21:03:27] https://tr.wikipedia.org/w/index.php?title=%C3%96zel:G%C3%BCnl%C3%BCk/delete&page=%C5%9Eablon:Kullan%C4%B1c%C4%B1_beyaz_liste [21:03:32] there is a delete history [21:03:57] I did not see a this page was deleted error [21:04:41]