[12:24:41] 06Revision-Scoring-As-A-Service, 10revscoring, 10rsaas-editquality: [Spike] Explore using PR-AUC to score when tuning - https://phabricator.wikimedia.org/T133698#2239792 (10Ladsgroup) https://meta.wikimedia.org/wiki/Research_talk:Revision_scoring_as_a_service/Work_log/2016-05-23 [13:19:04] p/ [13:20:49] halfak: o/ [13:32:16] halfak: I'm testing wikilabels' new json system [13:32:43] CORS + Post-only for write actions? [13:32:50] I suppose POST/DELTE [13:35:47] halfak: yup [13:35:48] :) [13:36:03] halfak: do you have some time to check this: https://phabricator.wikimedia.org/T135941 [13:36:53] Looking now. [13:37:01] I'd like to have someone more opsy than me comment. [13:37:09] This seems like a lot of code to send a ping. [13:37:52] Also, we have to copy the license. [13:37:59] of the code you re-used. [13:38:09] I think that GPL might break our MIT [13:42:21] halfak: let's check for license [13:42:54] halfak: I checked everything for coding, it seems it's not possible easier [13:43:14] I checked and tested every part [13:44:25] Gotcha. Seems not crazy then. [13:44:31] BRB [14:04:39] back! [14:06:01] I'm helping out with a workshop so I'll be in and out [14:06:02] halfak: welcome back [14:06:35] Amir1, so looking at this precached PR.... Will we have icinga or something else listening to these systemd pings? [14:07:30] nope [14:07:37] halfak: it would be too much [14:07:38] :D [14:07:56] kk. So what do we get? [14:08:01] systemd checks watchdog pings [14:08:23] And if the pings stop, it tries to restart? [14:08:24] and restart once it hasn't received the watchdog for a long time [14:08:28] Great [14:08:30] timeout [14:08:31] Any tests run? [14:09:34] yup on the systemd [14:10:13] So you showed that systemd can/will restart precaching with this change? [14:49:25] halfak: not that much [14:49:50] I will do this once we can get some changes there [14:50:11] Would be good to know if the service will be turned back on after you `kill -9` it. [14:50:13] Cool [15:05:00] halfak: we need some puppet changes but I'm waiting until the new patch on uwsgi is there and then I make other stuff [15:08:01] https://gerrit.wikimedia.org/r/#/q/owner:%22Alexandros+Kosiaris+%253Cakosiaris%2540wikimedia.org%253E%22,n,z [15:41:21] halfak: around? [15:42:37] 10Revision-Scoring-As-A-Service-Backlog: [Spike] Explore issues with enwiki wp10 model - https://phabricator.wikimedia.org/T136005#2318741 (10Halfak) [15:44:27] 10Revision-Scoring-As-A-Service-Backlog: [Spike] Could we build a "list article" quality model - https://phabricator.wikimedia.org/T136006#2318758 (10Halfak) [15:44:45] o/ [15:44:54] Sorry was typing up some tasks as fast as I could [15:45:06] Amir1, ^ [15:45:24] don't worry halfak, If you want [15:45:35] you can test my PR in labels-experiment [15:46:19] I did another commit and pushed it but it is not there [15:46:24] super strange [15:47:45] Amir1, were you able to kill the process and have systemd restart it? [15:48:16] I'm talking about the CORS PR [15:48:21] https://github.com/wiki-ai/wikilabels/pull/123/commits [15:50:58] I tested it, it needs a commit, I did it and pushed it but it's not there [15:51:03] Weird [15:51:11] lol @ no_country_for_jsonp [15:53:48] It seems it's there (I cloned it in another folder and it was there) [15:53:55] but not in GUI [15:54:24] halfak: I think this needs manual merge [15:54:29] Can I do it? [15:54:51] 458ee73689cfe63f639448b5157f423e366464f7 [15:54:55] hash of the missing commit [15:55:08] I still only see one commit. [15:55:48] https://github.com/wiki-ai/wikilabels/commit/458ee73689cfe63f639448b5157f423e366464f7 [15:55:49] Looks like you are a commit behind master. https://github.com/wiki-ai/wikilabels/tree/no_country_for_jsonp [15:56:01] that's l10n change [15:56:05] nothing important [15:56:28] halfak: ^ that is the commit, you can see it's in the correct branch [15:56:55] Yeah... very weird. [15:57:18] Either way, I don't see why a manual merge is necessary. You should be able to rebase. [15:57:31] I'd try that first -- pushing a rebase commit. [15:57:41] *rebase commit sequence [15:57:49] yeah [15:58:02] let's see if that works [15:59:24] halfak: rebased and pushed [15:59:29] nothing changed [15:59:31] :| [16:01:08] Amir1, pushing to the right repo? [16:01:16] Pushing to the right branch? [16:01:25] halfak: yeah, doubled checked everything several times [16:13:19] 10Revision-Scoring-As-A-Service-Backlog: [Document] Recent performance improvements for revscoring - https://phabricator.wikimedia.org/T136012#2318910 (10Halfak) [16:19:25] 10Revision-Scoring-As-A-Service-Backlog: [Spike] Explore issues with enwiki wp10 model - https://phabricator.wikimedia.org/T136005#2318956 (10Halfak) @Ladsgroup suggested that we look into using parsoid output. [16:40:41] 06Revision-Scoring-As-A-Service, 10ORES: [spike] Find out if we can still get health check warnings after lb rebalance - https://phabricator.wikimedia.org/T134782#2319058 (10Halfak) From within labs, you can ping a web node directly. E.g. `http://ores-web-03:8080` This can only work from within labs. [16:42:53] 06Revision-Scoring-As-A-Service, 10ORES: [spike] Find out if we can still get health check warnings after lb rebalance - https://phabricator.wikimedia.org/T134782#2319068 (10Halfak) You can find the hosts of the web nodes in hiera https://wikitech.wikimedia.org/wiki/Hiera:Ores [16:55:04] 10Revision-Scoring-As-A-Service-Backlog, 07Documentation: Recent performance improvements for revscoring - https://phabricator.wikimedia.org/T136012#2319133 (10Danny_B) [17:01:23] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: [research] Why is the japanese 'reverted' model so bad? - https://phabricator.wikimedia.org/T133405#2319193 (10Halfak) @Ladsgroup, Can you please put these on a wiki and use [[:ja:Special:Diff/...]] to make them linked nicely? [17:01:23] 10[1] 04https://meta.wikimedia.org/wiki/:ja:Special:Diff/..%2E [17:01:37] BRB dog ball time [17:02:31] halfak: it's fixed now: https://github.com/wiki-ai/wikilabels/pull/123 [17:02:35] stupid github [17:03:29] Did we just need to wait? [17:05:17] halfak: no, it shows the commit is there now [17:05:41] the thing is since github was down today, if we encounter such issues, we just need to wait [17:05:55] but it's very likely that won't happen again [17:06:06] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 10wikilabels, 07Documentation: Document maintenance tasks (restart something, deploy new versions, revert, etc...) - https://phabricator.wikimedia.org/T106271#2319204 (10Danny_B) [17:06:10] 10Revision-Scoring-As-A-Service-Backlog, 07Documentation: Document interesting potential uses of revscores - https://phabricator.wikimedia.org/T102339#2319206 (10Danny_B) [17:07:52] halfak: if you're talking about merging, It needs to be done manually, it seems their services has issues right now [17:08:43] we can wait thought and it might fixed [17:08:54] *though [17:09:13] 10Revision-Scoring-As-A-Service-Backlog, 07Documentation: Models in progress - https://phabricator.wikimedia.org/T132438#2319225 (10Danny_B) [17:21:17] halfak: I need to go, be back in a few minutes [17:22:37] halfak: mergable now: https://github.com/wiki-ai/wikilabels/pull/123 [17:34:39] looking [17:46:09] back [17:46:12] afk for dinner [17:46:13] :) [17:46:27] and some time with the boyfriend [17:51:56] Have good food fun time :) [18:29:49] back [20:33:34] 06Revision-Scoring-As-A-Service, 06Research-and-Data, 10Research-management, 06WMF-NDA-Requests: NDA for Amir Sarabadani - https://phabricator.wikimedia.org/T134651#2321058 (10Halfak) @Aklapper, ping. If there's something Amir or I can do to get this done, please let me know. [20:37:20] halfak: ^ thanks :) [20:37:27] halfak: https://etherpad.wikimedia.org/p/ores_weekly_update [20:37:37] \o/ totally forgot about the update [20:37:42] AKA the Amir show :) [20:39:07] :D [20:39:21] soon you will be on board again and it'll be the halfak show :) [20:39:28] PROBLEM - ORES home page on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:39:39] PROBLEM - ORES worker on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:40:03] \o/ [20:40:03] Oh shiut [20:40:09] don't worry [20:40:19] I heard there'll be down times [20:40:25] (see labs-l) [20:40:30] kk [20:41:45] halfak: just confirmed it again in #wikimedia-labs [20:53:51] Still down :( [20:54:00] 06Revision-Scoring-As-A-Service, 10wikilabels: Enable CORS for wikilabels - https://phabricator.wikimedia.org/T135377#2321074 (10Ladsgroup) https://github.com/wiki-ai/wikilabels/pull/123 [20:54:40] halfak: :( Can you review this: https://etherpad.wikimedia.org/p/ores_weekly_update [20:54:43] I want to post it [20:54:54] * halfak clicks [20:56:09] Confirmed! Post away! [20:56:11] Amir1, ^ [20:56:20] awesome [20:56:24] thanks halfak :) [20:59:53] halfak: done :) [21:00:04] awesome. [21:03:00] RECOVERY - ORES home page on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 418 bytes in 1.167 second response time [21:03:08] \o/ [21:03:19] RECOVERY - ORES worker on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 454 bytes in 0.026 second response time [21:03:36] yay [21:04:01] on the other hand, I updated the PR on ores about the watchdog, same issue again [21:05:10] Arg. Was just going to look at that. [21:05:13] Let's wait a few mins. [21:06:17] halfak: https://github.com/wiki-ai/ores/commit/cb69b44c17847949f8ca837c157496a3decae722 [21:06:21] that's the commit [21:07:43] halfak: I jumped to conclusion too fast, I just pushed my commit [21:09:08] https://github.com/wiki-ai/ores/pull/143/files [21:09:14] https://github.com/wiki-ai/ores/pull/143#issuecomment-221096963 [21:10:51] halfak: I was actually wanted to ask you, where can I test it? I tested the systemd part fully [21:11:00] but not the branch [21:14:28] Amir1, do we have an experimental VM for ores like with wikilabels? [21:14:29] let's stick with the precaching instance we have laready [21:14:40] nope [21:14:50] we do have one that you setup for sabya [21:15:11] we have another one that alex uses [21:17:34] Hmm... that's where I'd want to test it out. You could try your dev machine too, I guess. [21:18:08] Oh, I was wondering if when restarting the service, would systemd also kill a zombie that is left over? [21:21:39] halfak: I'm trying to work on the preaching instance [21:22:00] https://www.freedesktop.org/software/systemd/man/systemd.service.html [21:22:05] wrt your question [21:22:13] depends on the settings [21:22:34] I want to set restart=on-failure [21:32:40] We need to make sure to kill the old process before starting a new one. [21:33:51] halfak: the thing about watchdog is that if it's frozen and times out, the systemd kills it with a signal [21:34:09] Great [21:34:18] we can bring it back online by saying: if it was killed by this signal, start again [21:34:25] Hmm... How do we simulate a "freeze" [21:34:48] I guess we could pause the process. [21:35:28] one way to do that is actually running on no wiki. is it possible? [21:38:46] aand the instance is totally down [21:38:57] Labs derp? [21:39:43] I don't think so, it's was DNS. This is differnet [21:39:54] Yeah... [21:40:00] the self-hosted puppetmaster hasn't been set properly [21:40:16] gives deprecation warning all the time [21:40:30] I want to delete this instance and build a new one [21:40:38] specially with a proper name [21:44:03] halfak: is it okay? [21:44:19] I think so. I was never using it. [21:44:29] We have precached running on ores-web-03, right? [21:44:44] yup [21:50:06] viva ores-experiment [22:15:20] halfak: a note about using self-hosted puppetmaster. Use it only in disposable instances. I think it has issues with jessie [22:15:31] (plus, update is really hard) [22:51:47] * halfak has not played with self-hosted puppet master yet