[01:31:00] Amir1: hello [03:20:13] Zppix: hey, I just woke up [03:20:16] what's up [03:20:52] nothing much [03:21:25] halfak and myself are trying to get change prop for labs --> prod so i emailed security team to see if its possible [03:25:57] Amir1: [03:34:26] sorry, I missing the context (and sorry, was afk for coffee) [03:34:32] Zppix: [03:35:58] we wat change prop for ores [03:36:02] for labs [03:36:49] oh I see, I don't think it will be ready any time soon as the infra is not there for labs, they need kafka, they need event bus, etc. [03:41:13] Amir1: hence is why im talking top ops, cloud, and security [04:11:35] okay [05:45:59] (03PS1) 10Ladsgroup: Rework highlighting frontend to make it work everywhere [extensions/ORES] - 10https://gerrit.wikimedia.org/r/358311 (https://phabricator.wikimedia.org/T155930) [05:47:12] (03Abandoned) 10Ladsgroup: Add javascript highlighting to Special:Contributions as well [extensions/ORES] - 10https://gerrit.wikimedia.org/r/333497 (owner: 10Ladsgroup) [05:47:18] (03Abandoned) 10Ladsgroup: Enhance ORES support in enhanced changes list [extensions/ORES] - 10https://gerrit.wikimedia.org/r/333493 (https://phabricator.wikimedia.org/T155903) (owner: 10Ladsgroup) [05:47:47] (03CR) 10jerkins-bot: [V: 04-1] Rework highlighting frontend to make it work everywhere [extensions/ORES] - 10https://gerrit.wikimedia.org/r/358311 (https://phabricator.wikimedia.org/T155930) (owner: 10Ladsgroup) [05:49:25] (03PS2) 10Ladsgroup: Rework highlighting frontend to make it work everywhere [extensions/ORES] - 10https://gerrit.wikimedia.org/r/358311 (https://phabricator.wikimedia.org/T155930) [05:50:53] (03CR) 10jerkins-bot: [V: 04-1] Rework highlighting frontend to make it work everywhere [extensions/ORES] - 10https://gerrit.wikimedia.org/r/358311 (https://phabricator.wikimedia.org/T155930) (owner: 10Ladsgroup) [05:56:13] (03PS3) 10Ladsgroup: Rework highlighting frontend to make it work everywhere [extensions/ORES] - 10https://gerrit.wikimedia.org/r/358311 (https://phabricator.wikimedia.org/T155930) [14:07:50] o/ Saw some messages after I left. [14:08:24] I wasn't trying to advocate for changeprop from labs --> prod. But rather changeprop from prod --> labs. [14:08:44] Right now, you can't get changeprop for a labs service in any sort of easy way. [14:40:12] o/ paladox. I talked to ottomata a bit about eventbus/changeprop in labs. Want to join us in #wikimedia-analytics? [14:40:20] Ok [14:59:50] paladox, other than icinga, what other wiki stuff are you working on? [15:00:11] halfak i work on making polygerrit great (and gerrit) :) [15:00:25] polygerrit is gerrit but it's gerrit's new ui built with pure html + js :) [15:01:15] halfak im also testing wmf next upgrade to gerrit [15:01:21] which is 2.14.1 :) [15:04:51] Gotcha. seems like you're into dev support systems :) [15:05:33] yep :) [15:13:21] halfak: o/ [15:13:28] I have just woke up from nap [15:13:31] o/ glorian_wd [15:13:42] I have seen your comments on the PR, I am working on it now [15:14:33] I need to think more about the issue with your image lookups in commons. [15:22:58] o/ [15:23:12] halfak: we got more tamil help :) [15:25:36] Zppix, we did? [15:26:32] yes [15:26:45] halfak: i got an email asking what we needed [15:27:25] it goes to show if you need help just send an email and you will get it :) [15:27:53] That's the same person who hasn't been working on it for almost a month :P [15:28:10] halfak: no its a different [15:28:20] Kaartic [15:28:44] ill forward you the email whats your address? [15:30:22] halfak: i emailed security they said talk to cloud services and ops [15:30:31] Oh! Great. [15:30:48] Zppix, we don't want to do changeprop from labs --> prod [15:30:52] halfak: do you want me to handle that or you [15:30:53] Instead, labs --> labs. [15:31:07] oh well we still need to talk to cloud [15:31:11] wait [15:31:13] no we dont [15:31:18] thats easy asf [15:31:36] we got fabric don't we ? [15:33:58] I use fabric to deploy ORES in labs but we should use scap [15:34:16] do we currently have scap? [15:34:19] halfak: concerning to the problem in my commons image lookup, if I correctly understand what you're saying, there's already a function in revscoring which able to trigger MW API right? [15:35:04] Now my job is try to trigger the Commons API using that function [15:38:57] Right but I think we should avoid interacting directly with commons [15:39:38] halfak: we should spend some time to deploy ores [15:39:48] I will probably do it tomorrow if that's okay for you [15:39:53] talking to cloud about getting scap setup halfak [15:40:05] Amir1, sounds good. [15:40:16] nice Zppix :) [15:41:06] * Zppix halfak: https://wikitech.wikimedia.org/wiki/User:BryanDavis/Scap3_in_a_Labs_project [15:41:20] weird [15:41:28] i didnt type /me [15:41:51] * Zppix [15:44:11] @ping [15:44:11] Zppix: pong [15:57:58] halfak: then, could we use the Wikidata API that you've attached in the comment? For example : https://www.wikidata.org/w/api.php?action=query&prop=pageimages&revids=477165750&piprop=original [15:58:30] Note that the example that I gave here, uses the revision ID instead of title (item ID) [16:08:43] halfak: gotta buy some food shortly. Another question related to my PR before I go, I think instead of using Datasource, I should use Feature for all my functions right? [16:09:03] I'll be back in around 2 hr [16:09:04] hrs* [16:10:22] can someone tell me what the hell kanban is [16:10:46] Zppix: Kanban in SW development? [16:11:23] It's one of Agile SW dev methods. See https://en.wikipedia.org/wiki/Kanban_(development) [16:11:30] oh [16:11:53] thanks glorian_wd [16:25:43] hey halfak why do we do mit license? [16:53:35] glorian_wd, add your patchset for getting suggested properties from the api to https://phabricator.wikimedia.org/T158430 [16:53:42] It will help us track [16:56:36] akosiaris, I heard that sb2005 is down [16:56:39] Did you know about this? [16:56:44] *scb [16:57:36] halfak: tracked in https://phabricator.wikimedia.org/T167638 [16:57:58] looks like the motherboard is having issues to me [16:58:29] akosiaris, thanks. [16:58:32] Posting to the AI list. [16:58:57] that a box has issues ? [16:59:19] what aftermath does this have ? [16:59:19] halfak: did you see the link i sent about scap? [16:59:56] a single box failing shouldn't matter much, or actually warrant informing multiple people, hence me asking [17:00:42] akosiaris, I want to hear if people are running into issues that we're not noticing. It's a nice "FYI, we're at slightly reduced capacity, but you *shouldn't* see issues" [17:00:49] shouldn't != won't [17:00:52] ah [17:00:58] ok that makes sense [17:01:16] it does have a risk of false positives reports coming in ofc [17:01:16] no big deal. I think it makes our users feel nice to get a bit of an FYI from time to time :) [17:01:19] halfak: i agree with akosiaris if theres issues someone will say something regardless no? [17:01:31] but I can't quantify that [17:01:34] Zppix, we've had major issues that went unreported in the past :\ [17:01:35] it's just a gut feeling [17:01:47] halfak: arent we using eqiad atm not codfw? [17:01:54] We're using both. [17:02:03] yeah, it's one of the active/active services [17:02:25] depending on your location you might end up in codfw and not eqiad [17:02:42] codfw is where again? [17:02:49] dallas [17:03:12] dfw = https://www.dfwairport.com/ [17:03:32] halfak: crossposted that msg [17:05:32] Zppix, did not see scap link [17:05:43] Zppix, we use MIT because it's a common license that is very permissive. [17:06:00] Regretfully, switching to apache 2 at this point would be very difficult for most of our projects. [17:06:24] https://github.com/wiki-ai/revscoring/graphs/contributors [17:06:27] That one could work. [17:06:33] hhal https://wikitech.wikimedia.org/wiki/User:BryanDavis/Scap3_in_a_Labs_project [17:06:43] Maybe this one too https://github.com/wiki-ai/ores/graphs/contributors [17:06:56] halfak: are you saying you want me to switch those repos to apache? [17:07:06] Zppix, so we'd manage our own scap deployment server [17:07:11] That seems silly. [17:07:25] Wouldn't a shared scap server be useful for all of labs. [17:07:28] halfak: its the only way cloud doesnt run one [17:07:37] i asked that [17:07:40] Zppix, we can't just change the license legally. [17:07:52] i guess thats true [17:08:01] Zppix, I think we (you) should run a shared one for all of cloud :) [17:08:14] halfak: i can request a labs project and run a shared one [17:08:18] OK now I need ot run away and eat lunch or I don't get to eat [17:08:19] \o/ [17:08:22] Cool :) [17:08:29] halfak: buy me something too [17:17:29] akosiaris: replied to you on t167602 [17:20:31] Zppix: I doubt you are going to get much useful info from icinga RAM checks but feel free to implement it, I 'd like to be proven wrong. Thanks for the answer btw. From that I gather that T167602 is not related to production but labs, which is really useful information to me [17:20:33] T167602: Create memory checks for instances - https://phabricator.wikimedia.org/T167602 [17:21:36] no problem [17:27:10] wiki-ai/revscoring#1034 (deutsch-badwords - ef74dec : Devin/Zppix): The build passed. https://travis-ci.org/wiki-ai/revscoring/builds/242108893 [18:08:20] how does one add a test halfak [18:08:36] see the "tests/" folder in that directory [18:13:44] halfak: the shared scap project was sdenied by labs [18:20:04] halfak hmm, we should answer https://phabricator.wikimedia.org/T162629#3187911 though, i geneually think the ui is a feature. [18:24:28] Zppix, am talking to them now. Some releng people like the idea. [18:57:44] paladox, hard for me to comment there :/ [18:57:52] ok [18:57:58] Who are the people who primarily maintain icinga in prod? [18:58:23] Uh, i think akosiaris [19:01:05] I've worked with mutante when I needed icinga issues fixed in the past. [19:01:24] It might be a good idea to talk specifically to those folks and see how they think about an icinga upgrade. [19:01:24] ok [19:01:38] halfak i speaked to mutante. [19:14:08] I promised to talk to codezee, but it looks like he's gone. [19:16:29] why do you want prod to upgrade for yor prod instances [19:31:36] Zppix, asking me? I'm confused. [19:31:54] yes [19:32:15] halfak: also did you see https://phabricator.wikimedia.org/T167612 [19:32:54] Zppix, what's the scheme you're working toward? [19:33:03] ?? [19:33:12] what should the names be? [19:33:30] drop the research prefixes and name the same as github [19:34:21] Gotcha. Let's list out all the repos you want and the names you propose. [19:34:31] I'll review and suggest changes and then let's move forward [19:35:09] ok [19:36:17] done [19:44:47] Zppix, made some updates https://phabricator.wikimedia.org/T167612 [19:45:37] ok [19:49:58] Let's get Amir1 to confirm before moving forward [19:50:07] I suspect he might have opinions. [19:53:23] sure [19:55:43] halfak: have you heard of miraheze [19:55:51] Nope [19:56:20] they are a wikifarm im devoloping a proxy-bot that blocks open proxies for [19:56:26] https://github.com/Pix1234/Proxy-bot [20:06:59] halfak: I attached the link to my patch on the mentioned phab card [20:07:21] glorian_wd, "the link" [20:08:12] I've just read your chat again carefully. So, I guess should do the other way around huh? attach the phab card on the patch right? [20:12:11] glorian_wd, you'll need to forgive me. I don't remember what you are talking about. [20:12:39] halfak: you said "glorian_wd, add your patchset for getting suggested properties from the api to https://phabricator.wikimedia.org/T158430" [20:12:53] or maybe I did that correctly? [20:12:54] lol [20:13:22] I was switching context between #wikidata (PropertySuggester) and here, so I may misread your chat. [20:14:31] Ahh yeah. Good. [20:15:08] halfak: Ok. BTW, notice that Daniel said in #wikidata, PropertySuggester used to be able to suggest properties for items that don't have classifying properties [20:19:53] glorian_wd, sorry working on other stuff. Do you want me to read a chat or is that unnecessary? [20:20:23] halfak: not that unnecessary, just want to tell you that we're a bit late because the API now has been tuned to work only on classifying properties :) [20:20:43] Ok, now I need to work on your feedback in the PR [20:20:59] not necessary* [20:21:26] which pr is this [20:21:53] Zppix: My PR on the wikiclass [20:27:02] i reviewed waiting for halfak to sign off [20:27:15] Zppix, there are serious problems with that PR [20:27:25] i didnt get any errors [20:27:51] Zppix, oh did you run a feature extraction with the new feature set? Train a model with it? [20:27:55] :P [20:28:21] o/ codezee [20:28:23] See my notes that have not even been addressed. [20:28:42] oh now i can get errors took it a bit... [20:30:06] weird these errors werent related to this pr? halfak [20:30:18] Zppix: I am now trying to address halfak's comment lol [20:30:42] o/ Zppix halfak [20:32:10] codezee: please take a seat halfak will be with you shortly xD [20:32:34] sure... [20:34:48] sorry was having chat lag :P [20:37:46] :/ [20:39:00] lol [20:40:05] halfak: In your comments on the PR, you were basically complaining the usage of Datasources [20:40:08] Am I right? [20:40:22] Yes. That won't work. [20:40:22] So, I think I should've used Feature rather than Datasources right? [20:40:31] All "features" need to be based on Feature [20:40:48] Datasource is a precursor to Feature that is less constrained. [20:41:01] Also, the calls to the API that you have are problematic. [20:41:07] So we need to think through that. [20:41:26] halfak: just want to confirm, you are talking about 'Feature' in /revscoring/features correct? [20:41:47] glorian_wd, yes. [20:42:15] glorian_wd: see https://github.com/wiki-ai/revscoring/blob/master/ipython/feature_engineering.ipynb for the fine difference b/w datasources and features [20:42:20] halfak: Cool! Concerning to API, could we use wikidata API just like what you mentioned there? [20:42:26] For example, https://www.wikidata.org/w/api.php?action=query&prop=pageimages&revids=477165750&piprop=original [20:42:39] it uses revision ID instead of item ID that you suggested in the comment [20:42:39] glorian_wd, see extractor for how we do API calls. [20:42:43] codezee: *checking* [20:42:51] I'm worried about conflating CPU and IO [20:42:59] So we might need to get clever. [20:43:53] halfak: what? CPU and IO? are those parables?lol [20:45:26] Though I haven't checked it, from what you're saying, it's possible to use Wikidata API in the revscoring to check the image [20:46:22] Yes. See revscoring/extractors [20:47:15] I think we're not going to be able to define the extractor for images there though. [20:47:31] Because what are "images" anyway. [20:47:41] We might still need to use your solution. [20:56:22] halfak: "We might still need to use your solution. " => but perhaps with Wikidata API instead of Commons API right? [20:56:41] Right. [20:57:04] But I think we'll end up needing to request data about each image individually -- not at the page level like the link I gave. [20:57:37] One problem that I notice with Commons API, if the image is moved and as a result, the API does not identify the image, it won't return any result. In Wikidata API, even when the image is already moved, it still return the width and height value. Probably Wikidata API has some sort of cache to do that [20:58:12] redirect? [21:00:58] halfak: hmm not sure. Check out Trump's revision: https://www.wikidata.org/w/index.php?&oldid=491554358. If you try to check the image in this revision in Commons API, it will return no result because the image was moved [21:01:26] In the latest revision of this item, the image is already changed with another image [21:01:51] halfak: maybe what you meant was, Wikidata API identify the latest image instead of the image in the old revision [21:02:20] glorian_wd, should we fail to generate a score when an image has been removed? [21:02:59] halfak: using the current Commons API, yes. Check out: https://commons.wikimedia.org/w/api.php?action=query&titles=Image:Donald_Trump_official_portrait.jpg&prop=imageinfo&iiprop=dimensions&iimetadataversion=latest [21:03:12] It does not give the width and height information. [21:04:40] However if the same revision is used in the Wikidata API, it gives the width and height information. See https://www.wikidata.org/w/api.php?action=query&prop=pageimages&revids=491554358&piprop=original [21:05:50] Interesting [21:06:46] glorian_wd, looks like it's just returning the current [21:06:49] Not the historical [21:06:50] halfak: [21:06:52] yeah [21:06:53] We need the historical [21:06:56] I was typing that [21:07:07] it seems it takes the current revision image width and height [21:08:09] halfak: then if we need historical image information, we need to use Commons API [21:08:32] My current solution is, if there is no result, just return 0 [21:09:03] glorian_wd, but then the historical score would change [21:09:11] Sometimes it would be zero and othertimes it would be positibe. [21:09:15] *positive [21:10:08] halfak: but this is Commons API. If the image is already moved, then it won't exist anymore? [21:10:24] how come the historical score could change? [21:10:35] I imagine the score could change if we use Wikidata API [21:10:45] Because in one case, there was an image and in another case, there was not. [21:10:59] For the same revision! [21:11:17] Hmm, is it possible someone add an image to old revision? [21:11:24] or change image in the old revision? [21:11:34] no [21:11:47] But you can delete the referenced image. [21:12:52] Ah I guess I get what you mean. So, maybe in one day, the image still there, we will get some score. Then in the next day, the image is gone, and in this case, we shouldn't get any score [21:14:30] I'd suggest caching the referenced image to save performance [21:15:49] glorian_wd, right. This is a weird problem. [21:16:03] For revisions, we just error if a critical bit of information has been deleted. [21:16:17] E.g. the previous revision was suppressed -- now we don't know what the diff was. [21:16:31] The user has been suppressed -- now we don't know who saved it. [21:17:02] or api is depressed and it cannot talk to it :P [21:18:01] halfak: I'm considering to remove the feature at all. After all, it seems not really predictive [21:18:18] glorian_wd, it isn't? [21:18:25] It certainly is a pain in the ass :) [21:18:36] I agree. Forget it :) [21:18:37] And, another reason, image quality is another big topic. Actually, you can't just measure image quality with image megapixels [21:18:56] Set it aflame and flush it down the toilet [21:19:14] So, even though we manage to get some way to calculate the image megapixel with the correct revision, essentially the way we measure image quality is not completely right [21:19:24] dont clog the toilet ill tell maintaince to make you fix it :P [21:20:13] halfak: if you see the guideline here: https://commons.wikimedia.org/wiki/Commons:Image_guidelines#Quality_and_featured_photographic_images, there is other image quality criteria such like JPEG compression, noise, color, etc. [21:21:15] To take everything in account, we need extra image processing techniques. I think that's just not worth to spend a lot of effort on 1 feature that might not be very helpful [21:21:41] LOL. I was laughed when you said "Forget it and the toilet" [21:22:02] :) [21:22:04] maybe we should keep the code but dont ever run the code incase commons decides they want it [21:22:15] na. Kill the code. I don't want to see it. [21:22:21] Not that complex to rewrite later. [21:22:26] Zppix: It just a code to calculate Megapixels [21:22:30] right, it is not that complex [21:22:37] So far, I've figure out like 5 ways to get dimensions from the API :) [21:22:41] oh then have a match [21:22:47] and a flush [21:23:00] And let us never speak of what happened here again ;) [21:23:01] and a bomb? [21:23:14] OMG A TERRORIST [21:23:18] ;) [21:23:41] allu akbar? xD [21:23:43] halfak: keep those 5 ways. Maybe Commons will ask them one day :D [21:24:12] Zppix: don't say that. Your message will be cached by NSA by saying that :P [21:24:18] https://www.youtube.com/watch?v=pofUsd9hEi8 [21:24:32] im already on every watchlist probably glorian_wd before i was born [21:25:28] halfak: next project a bot that codes whatever we ask [21:25:31] Woops. that's not the right video [21:25:47] https://www.youtube.com/watch?v=PCN31uK45GE [21:25:49] That one [21:25:56] .... [21:27:08] halfak: I'd be curious what's gonna happen if someone says "bomb" during the flight :D [21:30:37] 9/11 [21:31:15] 911* [21:32:23] glorian_wd, I wouldn't try that. [21:33:11] halfak: left some performance info on task, to me the bottleneck looks to be efficient WSD [21:33:40] WSD? [21:33:53] halfak: Word sense disambiguation [21:34:07] halfak: a very raw idea - maybe we can get away by only scoring adjectives and Nouns, since most sentiment is in those( assuming POS-tagging should be fast) [21:36:07] codezee, I had thing for you. One sex. [21:36:09] *sec [21:36:11] damn itr [21:36:18] HR dept. I swear I didn't mean to [21:36:20] :D [21:36:59] halfak: im reporting to hr for sexual harrassment xD [21:37:04] PROBLEM - mem-check on gerrit-test.git.eqiad.wmflabs is WARNING: Warning: 16% free memory [21:37:15] umm ignore [21:37:21] paladox: wtf you doing [21:37:42] Zppix accidentially added a line to the script [21:37:53] what line [21:38:24] Zppix the line that forwards the notifications to the log [21:38:41] ok [21:42:17] Just got back from brushing my teeth and laughed so hard because halfak's typo :P [21:43:26] Maybe they shouldn't put the letter "X" very close to the letter "C" in the US keyboard :D [21:43:52] codezee, https://gist.github.com/halfak/045c016f7ab3d55d7b76e30dec199514 [21:44:09] "content" strips infobox and other templates and markup [21:46:30] halfak: I'm already using that in the PR [21:46:45] Oh my bad :) [21:48:26] halfak: text is clean, I'll do some research around faster ways to get polarity score [21:48:55] codezee, cool. [21:49:11] Seems like there should be a fast dict lookup for this. [21:50:09] I guess, I'm thinking if people have also done scoring without sense disambiguation, so *looking* [21:51:52] (03CR) 10Gergő Tisza: Rework highlighting frontend to make it work everywhere (032 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/358311 (https://phabricator.wikimedia.org/T155930) (owner: 10Ladsgroup) [22:13:04] * halfak leaves for dinner. Gave a good one folks! [22:13:52] glorian_wd: on Turkish Airlines 3 years ago, a passenger asked flight attendant to "open air conditioner or we will explode". All plane searched for bomb, all passengers moved to other plane after xray check, that passenger interrogated by police. [22:13:52] http://www.milliyet.com.tr/klimalari-acin-patladik-sozu-ucakta-istanbul-yerelhaber-145454/ [22:14:10] * paladox it's 23:13pm here :) [22:14:25] * glorian_wd paladox it's 00:14 AM here :) [22:14:36] 1hour later then me :) [22:14:38] * paladox is on bst [22:14:49] paladox: I guess you're based in the UK [22:14:54] yep [22:15:16] to be precise 1hour away from london [22:15:23] mavrikant: lol. Thanks for the information :D