[00:36:34] thanks amir for the merge [01:26:37] Amir1: do you need the fix ssl branch? [01:26:39] on wikilabels [01:27:17] Zppix: Not needed there, wikilabels doesn't have swagger [01:27:29] so yes? [01:27:34] no* [01:31:12] Amir1: huwiki wants bad words system ran [01:32:01] Zppix: oh, please don't delete branches here and there [01:32:25] ? [01:33:04] if its not needed i dont see the point in having it just sit around [01:33:21] I was thinking that you meant this one: https://gerrit.wikimedia.org/r/#/c/357331/ [01:33:30] and doing it for wikilabels [01:34:03] oh lol ill restore if you want [01:35:12] Zppix: yeah, please [01:36:51] done Amir1 [01:41:00] wikilabels should think of changing from mit, apache license or GNU is better [02:52:26] want me just to merge it since all it is a coc Amir1 [02:52:43] forgot to make a pr [02:52:45] wait a sec [02:52:52] kk [02:53:09] Zppix: https://github.com/wiki-ai/wikilabels/pull/182 [02:53:10] Thanks [02:53:28] I'm making another one for revscoring too [02:53:49] no access for revscoring [02:54:02] okay [02:54:03] :( [02:54:24] its a .md im sure no-one will complain about a self-merge [02:54:38] Let me ask Aaron to give you proper rights [02:54:44] yeah, that's a good idea [02:58:00] wiki-ai/revscoring#995 (coc - d6aaaed : Amir Sarabadani): The build passed. https://travis-ci.org/wiki-ai/revscoring/builds/239811728 [02:58:03] i need to do this for my repos too [02:58:58] Right now, only mediawiki core has it and people are building a script to do it for all extensions too: T165540 [02:58:59] T165540: Add CODE_OF_CONDUCT.md to Wikimedia projects - https://phabricator.wikimedia.org/T165540 [02:59:02] https://gerrit.wikimedia.org/r/#/c/355875/ [02:59:19] i dont run an extension i run 2 bots [02:59:21] I did this first, so we would have one unified text everywhere [03:00:37] have you guys thought of using wm-bot for github notifications? [03:01:33] It would be good but we didn't think about it. [03:02:02] Its not hard to setup [03:03:00] Amir1: are you an offical wmf employee or a volunteer? [03:03:38] That's complex, I'm contractor for WMDE and my main project is Wikidata but I work for ores in 20% of my time [03:04:16] WMDE contracts alot ive noticed [03:05:08] For software engineering, that's only two persons AFAIK [03:05:18] me and someone else [03:05:32] i think aude is wmde and contracted [03:05:44] that's the "someone elsE" [03:06:46] I'm unaflliated from any wmf chapter im just a volunteer i started on enwiki and then found about development and now here i am [03:07:33] I've been a volunteer developer since 2010, I got hired in 2015 [03:07:47] still lots of my time is volunteer time [03:07:55] cool [03:08:04] (And have been editing Wikipedia since 2006) [03:08:11] I'm ooold [03:08:20] i started editing in 2015 [03:08:34] I remember editing by bot when there wasn't an API [03:08:40] started development early ?2016? [03:08:51] no api... just kill me [03:09:29] :))) [03:11:48] Amir1: we need an failsafe for deleted edits on wikilabels maybe instead of error messages you just auto skip over that edit? [03:13:08] Yeah, We need to something like that but when people see it's auto jumping in their workset they might be surprised [03:13:25] these kind of stuff need a UX designer [03:14:09] Wikilabels needs lots of UX love (this error message is a problem in that part too) [03:14:31] maybe ask collab team for a ux help:P [03:15:27] believe me, we did. I did at least ten times [03:15:35] they are pretty much busy already [03:16:25] maybe we need to be apart of the next hackathon see if that could help? [03:21:05] Amir1: https://phabricator.wikimedia.org/T167061 [03:21:22] Zppix: yeah, probably [03:28:42] maybe we could add like small text on labels.wmflabs saying Want to contribute? and link to the proper areas [03:29:05] I mean AI is smaller team than others [14:23:09] o/ [14:23:44] "AI is smaller team than others" :) Yeah... we operate under half the time of a fulltime WMF staffer (me). [14:23:57] We have no engineers or designers, but a few other teams help out. [14:24:30] halfak rocks! [14:24:32] :P [14:24:42] My title is principal research scientist, but I do science around technologies so it works that do a lot of engineering myself :) [14:24:46] o/ glorian_wd [14:24:57] Saw you make some changes to your PR. I hope to take a look at that soon. [14:25:52] halfak: yeah. Basically, I am trying to work on the stuffs that already obvious for changes given to the feedback that you gave so far [14:26:16] Right now, I am working on breaking the large 'external_sources_ratio' method into smaller pieces [14:26:28] Will be finished in the next couple of hours [14:26:34] :) [14:35:05] cool [14:45:06] halfak: good morning [14:45:14] o/ Zppix [14:46:13] So I and amir added a COC to wikilabels and revscoring last night [14:46:25] Saw that. Looks good. [14:47:02] (I need to do this for my repos still :P) [14:48:23] anything new? since ive been away [14:50:02] Nope. Looks like you were online later than I was and I just got on now and started working on emails. [14:50:07] Meeting in 10 minutes :/ [14:50:34] Yep I was and okay seems like this week is meeting week :P [14:52:42] Yeah. Most every week for me these days. [14:52:43] I' [14:52:50] m resisting transitions into management [14:53:04] But there's meeting tendrils that keep trying to drag me away from productive work. [14:53:32] halfak: but your a perfect match for mangament [14:56:31] lol [14:56:54] In my not so humble opinion, I have too much practical expertise to do easily into that dark night [14:57:03] *go easily [14:59:51] Lol [15:03:55] halfak: im going thru backlog workboard seeing if i can find anything [15:09:39] Zppix, will get back to you shortly if you don't find anything [15:10:06] Okay have fun in your meeting halfak [15:48:29] Amir1: are you around? [16:00:36] Zppix: I think Amir1 is also in the meeting. Today at around this time, is the Scrum meeting for the SW dev team in WMDE :P [16:00:51] Its ok i found what i needed [16:24:09] halfak: I have modified my PR by following all of your feedback except the one concerning to move some part of the code to wikiclass. [16:26:47] because if I correctly understand you, you want to help me moving the code to wikiclass [17:06:17] akosiaris, https://phabricator.wikimedia.org/T167149 FYI [17:06:28] Looks like we can't use a unix socket for some very stupid reason [17:06:37] Either that or let's deploy a monkey patch in prod. [17:06:41] I'd rather not do that. [17:07:03] * halfak curses and kicks the dirt [17:07:04] I [17:07:22] 'm very annoyed to have been working on that so long to get stuck on a dumb error that hasn't been addressed for months! [17:16:01] halfak: I was unable to find anything that looks easy for me to do. [17:16:56] Gotcha. Good timing. I have a "break" now. [17:17:05] * halfak looks [17:18:23] halfak: me and you both know wmf staff dont get breaks [17:32:22] halfak: Icould set up t147163 using a icinga instance myself and paladox run on wmflabs [17:35:27] Zppix, I'd be interested in you looking into this. [17:36:03] halfak: its not that hard for me to do infact i could do it right now [17:36:32] I can give you access to our icinga2 ui if your curious [17:37:05] Awesome. [17:37:21] Maybe if you need me to look at something in review ^_^ [17:37:39] Ill get you an UI acct set up and ill collab with paladox with the channel set up [18:12:01] halfak: we hit a little hiccup regarding firewalls but we will try to work around it [18:16:19] halfak: labs instances would just go in horizon for the hiera config [18:35:48] Zppix, can work on the labs instances hiera. Same line? [18:36:03] yes [18:41:01] Zppix, I don't suppose you have an example of this line in another hiera, do you? [18:41:29] halfak: Idont paladox ^ [18:41:38] halfak yep i do [18:41:45] I want to make sure I get it exactly right :) [18:41:54] See https://wikitech.wikimedia.org/wiki/Hiera:Git [18:42:13] It's this line [18:42:14] "profile::base::nrpe_allowed_hosts": 127.0.0.1,10.68.23.211 [18:42:16] the ip is gerrit-mysql (running the icinga2 :)) [18:43:14] https://wikitech.wikimedia.org/w/index.php?title=Hiera:Ores&diff=prev&oldid=1761351 [18:43:17] Should be done [18:43:22] yep :) [18:43:32] try running puppet to see if it applys it please :) [18:44:42] we're setting up notifications for this channel asap [18:46:12] * halfak puppets [18:46:39] is there any other memebers of the team whom will need ui acces halfak ? [18:46:45] * paladox finds syntax for adding notifications for just the hosts your interested in :) [18:47:21] Zppix Amir1 will [18:47:37] halfak: okay will do [18:47:42] puppet was a success. [18:47:53] halfak did it change /etc/nagios/nrpe.cfg [18:47:54] ? [18:48:04] halfak: sup? [18:48:14] /etc/nagios/nrpe_local.cfg changed [18:48:31] Amir1, was just saying you'll want access to the icinga thing that Zppix is setting up for disk space monitoring [18:48:47] oh, great [18:48:51] thanks folks [18:49:07] np [18:49:16] halfak thanks :). do you see server_address=10.68.23.211 in there? [18:50:45] halfak: Amir1 gerrit-icinga.wmflabs.org login is same as wikitech halfak i wasnt aware we used ldap for our instance so the login info i gave you is invalid [18:51:54] halfak could you open port 5666 please? (its nagios port that it runs on to be able to communicate with hosts) :) [18:52:35] should be able to do it through horizion then Access and Security and which ever profile is applie to your instance. Then add port 5666 [18:52:36] :) [18:54:22] ah and you have to do service nrpe restart [18:54:53] paladox, in horizon? [18:54:58] Yep [18:55:03] Ahh yeah. You said that. [18:55:04] Derp [18:57:07] halfak in /etc/nagios/nrpe_local.cfg do you see something similar to https://phabricator.wikimedia.org/P5551 [18:58:30] yup [18:58:53] ok [18:59:20] try sudo service nagios-nrpe-server restart. [18:59:30] Still enabling 5666 [18:59:33] got to love security features make everything take 200 times longer [18:59:34] Since it should not be failing with CHECK_NRPE: Error - Could not complete SSL handshake. [18:59:40] Can't batch edit security group assignments :( [18:59:59] I am starting to think the port is fine at the moment [19:00:14] Since the error it is giving me sounds nothing like a port failure. [19:00:20] https://geekpeek.net/could-not-complete-ssl-handshake/ [19:01:09] paladox, which machine are you trying? [19:01:14] *host [19:01:21] deployment-ores-redis-01.deployment-prep.eqiad.wmflabs [19:01:29] Oh! wrong labs :) [19:01:33] oh [19:01:34] But we can do that one too [19:01:49] which instance are you on? [19:01:52] ores-web-05.ores.eqiad.wmflabs [19:01:57] We have a full on labs deployment [19:02:06] we're done both [19:02:10] oding both [19:02:12] omfg [19:02:17] i cant type [19:02:19] Zppix you didnt add ores-web-05 [19:02:20] lol [19:02:34] im doing all the hosts on that wikitech page you gave me halfak [19:05:24] yay [19:05:26] it works [19:05:29] halfak ^^ [19:05:37] for 05 [19:05:37] OK - load average: 0.46, 0.23, 0.09 [19:06:14] halfak now for the check_disk command [19:06:15] NRPE: Command 'check_disk' not defined [19:06:16] i will need you to add a file [19:06:17] * paladox find it in puppet [19:07:34] oh wait [19:07:37] you already have the file [19:07:51] need to add a line to nrpe.cfg [19:09:08] halfak in /etc/nagios/nrpe.cfg (scroll near to bottom where it shows this line command[] then add command[check_disk]=/usr/lib/nagios/plugins/check_disk -w 5% -c 2% -l -e --exclude-type=tracefs [19:09:23] then save the file and do sudo service nagios-nrpe-server restart [19:13:53] * halfak is working on cfg file now [19:14:11] paladox, we have, ~12 hosts. [19:14:18] You want me to do this in every host or just pick one? [19:14:31] Zppix, those hosts are from prod [19:14:36] We're looking at labs now, yes? [19:14:40] You can do it in all the hosts you want. [19:14:47] halfak: yes [19:14:51] kk [19:15:15] halfak i have a puppet repo for icinga2 https://github.com/paladox/icinga2-puppet-wikimedia [19:17:55] https://wikitech.wikimedia.org/wiki/ORES/Instances#The_labs_setup [19:18:05] Im on the openstack now [19:18:15] paladox, seems like puppet would be the right way to operate if we want this to apply to all hosts. [19:19:07] halfak oh, that one is only for the icinga2 host [19:19:36] that will make it so that i can create a host on github and then puppet is applied on the icinga2 main host which then adds it to the webui. [19:19:42] and starts monotoring the host [19:21:14] or you be a rebel and use the webui [19:23:06] I'm confused. [19:23:15] Do we need to modify a config on every host or no? [19:23:29] BTW, just fleshed out host names in https://wikitech.wikimedia.org/wiki/ORES/Instances#The_labs_setup [19:23:33] on any host you want monitored [19:23:37] halfak yes for the icinga host to reach those hosts [19:23:38] :) [19:23:50] OK so puppet then. How do we puppet in this config value? [19:24:35] uh not sure. [19:24:46] where will you do it with puppet? [19:24:55] wikimedia puppet repo? or https://github.com/paladox/icinga2-puppet-wikimedia [19:25:31] hm... I suppose this is labs specific. [19:25:45] yep. [19:33:34] So probably belongs in hiera somehow. I need someone with better puppet-fu than me. [19:34:14] halfak is this in the ores project? [19:34:31] you added the line there https://wikitech.wikimedia.org/wiki/Hiera:Ores so all instances should be able to be reached in that porject now [19:34:31] :) [19:34:48] Yeah. Right. Just wondering about the last edit you asked me to do. [19:34:53] [14:09:08] 19<paladox> halfak in /etc/nagios/nrpe.cfg (scroll near to bottom where it shows this line command[] then add command[check_disk]=/usr/lib/nagios/plugins/check_disk -w 5% -c 2% -l -e --exclude-type=tracefs [19:35:03] ah [19:35:04] yep [19:35:08] in /etc/nagios/nrpe.cfg [19:35:11] you find the line [19:35:15] command [19:35:19] which is near the bottom [19:35:21] Somehow, we should do that in hiera [19:35:30] yeh. [19:35:33] So it applies to all hosts [19:35:37] yep [19:38:18] doing that should fix [19:38:23] NRPE: Command 'check_disk' not defined [20:00:10] test [20:00:32] CUSTOM - Host ores-05 is UP: PING OK - Packet loss = 0%, RTA = 0.81 ms halfak [20:00:42] there we go. It works :) [20:01:49] paladox, do you know how we get the check_disk in the config from hiera or should I ping some puppet masters? [20:02:44] halfak probaly need to speak to a puppet master. But i think that would require a change in the puppet repo to be able to support that. [20:02:53] Editing the file should be ok. [20:07:59] CUSTOM - Host jenkins-slave-01.git.eqiad.wmflabs is UP: PING OK - Packet loss = 0%, RTA = 0.32 ms make sure ai doesnt get irrevelent notifications [20:08:07] err paladox [20:09:48] im goign to make wikilabels code look pretty is that okay? halfak [20:10:46] paladox: jenkins-slave still gets sent to both channels [20:17:22] Zppix, like, how? [20:17:26] pep8? [20:18:48] if we do pep8 then nevermind (im still learning about pep8) [20:20:04] Yeah. pep8. Conventions FTW [20:20:24] I have strong opinions that disagree with pep8 in some circumstances, but I'd rather be consistent [20:20:28] Conventions that i mainly use is Zppix [20:21:04] if it works its fine the way it is thats how i see it :P [20:23:45] Zppix oh [20:24:37] Boo. Conformity. [20:24:41] ;) [20:24:46] paladox, we have 'command[check_disk_space]=/usr/lib/nagios/plugins/check_disk -w 5% -c 2% -l -e --exclude-type=tracefs' -- that's not what you asked for, but will it work? [20:25:05] we use it without _space [20:25:09] just remove _space [20:26:08] Regretfully, there's a whole puppet config strategy that requires _space [20:26:16] S P A C E ~~ [20:26:33] halfak heh. [20:26:35] paladox: if thats true we need to modify our end ? [20:26:38] remove _space [20:27:02] space is overrated [20:27:37] it will fail to find that [20:27:38] anyways [20:27:47] needs to be [20:27:47] check_disk [20:27:50] halfak ^^ [20:28:43] Looks like the other name is baked into puppet. Sure we can't change it on the other side? [20:29:38] halfak maybe [20:29:43] yeh [20:29:58] paladox: we need to change our config then [20:30:25] yeh [20:30:28] im doing that [20:31:02] halfak: Amir1 was wondering if you could give me access to revscoring last night [20:31:39] Nettrom: connection problems? [20:32:14] Zppix, done [20:32:44] got it [20:35:52] halfak done with https://github.com/paladox/icinga2-puppet-wikimedia/commit/b92ad140fd1b761d256b2022d12a00c65b847acb and https://github.com/paladox/icinga2-puppet-wikimedia/commit/9d1db241f622fa59e16c68d50e0bd2e8ec4971d6 [20:36:13] woo [20:36:14] it works [20:36:16] / 12.48 GiB 18.58 GiB 17.65 GiB 18.20 GiB [20:36:17] /run 80.00 MiB 791.00 MiB 751.00 MiB 775.00 MiB [20:36:18] /sys/fs/cgroup 0.00 B 1978.00 MiB 1879.00 MiB 1938.00 MiB [20:36:19] /run/lock 0.00 B 5.00 MiB 4.00 MiB 4.00 MiB [20:36:20] /dev/shm 0.00 B 1978.00 MiB 1879.00 MiB 1938.00 MiB [20:36:21] /dev 0.00 B 10.00 MiB 9.00 MiB 9.00 MiB [20:36:22] \o. [20:36:22] halfak ^^ [20:36:26] \o/ [20:36:29] :D [20:36:30] Nice [20:36:39] :) [20:36:48] I am going to add notifications for services for ores [20:36:59] will ack the memory service (will fail as the file does not exist) [20:37:33] halfak: If you ever want any addition you and Amir1 are able to do so thru that github repo or thru the director tab on the webui (which is easier) [20:37:40] or you can ask me or paladox [20:38:11] cool :) [20:39:57] once notifications are set up and testing on them are done ill close the task [20:40:52] Zppix, we need it in prod more than in labs :) [20:41:05] oh yeah i forgot about that [20:41:20] prod will have to wait we need a firewall expection rule or somethung [20:41:57] Zppix, can't use the prod facilities? [20:42:03] we should bug -operations and ask about it [20:42:19] halfak: prod is seprate from wmflabs via firewall or such [20:42:41] if you go to operations and ask they may be able to allow us to conenct to your ores prod instances [20:42:45] Right. There must be other ways that disk usage is tracked in prod [20:43:44] I know I've gotten notifications about disk usage issues for analytics prod servers before. [20:44:19] its thru icinga, but its th e prod version which is old and not to mention a pain to get it confiugred without waiting for someone with access [20:44:53] whereas our instance is latest version and we can configure instantly [20:44:58] Meh. Should be OK. If we're working between 1400 and 1800 UTC, akosiaris will be online to give us a hand with it. [20:45:22] Even if we get an exception, it'll probably need to get approved by akosiaris [20:45:37] all it will need is probably a sign off by security [20:46:55] ill look into it further see if theres established process if so ill look into going thru it [20:47:53] but lets ensure labs will be reliable so that way if theres a problem we're not fixing 20 more hosts [20:48:04] PROBLEM - check disk_space on ores-05 is CRITICAL: NRPE: Command 'check_disk' not defined [20:48:13] halfak there we go ^^ [20:48:53] RECOVERY - check disk_space on ores-05 is OK: DISK OK [20:49:04] nice [20:49:35] i have check load on hosts as well halfak so if a server is overloaded you will know [20:50:20] Cool. I trust your judgement WRT starting thresholds. We can see how they hold. [20:51:30] halfak: we use the default i believe [20:52:03] it can be adjusted if you wish, right now besides us your the only other team using our instance atm [20:55:06] What project are you using the instance for? [20:56:52] halfak prod is already covered by operations maint of icinga [20:57:01] they use the really old version though [20:57:05] halfak: git [20:57:05] spoke with mutante [20:57:23] paladox ores-webhost needs its check-disk fixed as well as new hosts i just added [20:57:29] Ahh so the scb* hosts have disk monitoring already? [20:57:36] yep i think so [20:57:49] but not by us sadly [20:59:28] They use a really old version Zppix [20:59:33] icinga2 is really good :) [20:59:44] I know [21:01:31] Is the goal to get monitoring working nicely for all of labs/cloud? [21:02:10] If a team or people want us to monitor their stuff we will [21:03:37] halfak, i did request that here https://phabricator.wikimedia.org/T163213 [21:03:48] but they are waiting on very good reasons on why to do it. [21:03:55] Zppix: I’ve got no idea why my laptop decided to reconnect like that, sorry :( [21:04:04] paladox canyou fix the errors theres still some hosts that need their check_disk command fixed [21:04:14] oh [21:04:15] ok [21:04:27] Nettrom: check your wifi settings look for off on sleep or something like that [21:05:01] Zppix: ah, that might be it, I think my Mac can go online to check for updates, that might trigger Colloquy to reconncet as well… thanks [21:05:24] np [21:08:23] \o/ [21:08:31] * halfak watched Amir1 merge his stuff [21:08:44] I just woke up [21:09:01] good morning (in Pacific ocean's timezone) [21:09:12] o/ [21:09:51] Pacific ocean has a lot of timezones O.O [21:10:43] halfak: is there any update on https://phabricator.wikimedia.org/T164671 ? [21:11:59] Hi mavrikant. Not yet regretfully. [21:12:07] Maybe I can convince Zppix to look into that :D [21:12:26] let me look [21:12:27] Or maybe Nettrom would be interested. [21:12:30] :D [21:13:12] Zppix, we'd need to implement an extractor for trwiki. E.g. https://github.com/wiki-ai/wikiclass/tree/master/wikiclass/extractors [21:13:19] I can barely type in english :P and im native to english speaking so maybe not me [21:14:01] but ill keep it in mind incase no-one else touches it [21:14:05] Ha. Me too. [21:14:18] I've started to get good at working around wikis in languages that I don't speak :) [21:14:26] Was just browsing frwiki no problem :) [21:14:48] i know enough speak to convey i dont speak spanish and to please speak english :P [21:14:49] I speak like 3 words of french I learned from children's shows [21:15:05] i know enough french to say hello, and whatever the hell fere jacques means [21:15:19] lol [21:15:41] i know enough german to get punched in the face [21:16:12] Oh! Want to help with our German badwords list? [21:16:14] ;) [21:16:18] I know lots of German swear words as when I'm in the office, people swear a lot (Mostly Daniel) when things are not working [21:16:25] You problem already have what i know halfak [21:16:34] https://github.com/wiki-ai/revscoring/blob/master/revscoring/languages/german.py [21:16:37] kk [21:16:37] I hope to have more diverse colleagues to learn more swear words [21:16:47] Amir1: i would be fired for offesnive lang for how many times i swear [21:17:16] :P [21:17:24] Someone's going to do a blind analysis of github commits one day and I'm going to win the record for the most curses committed in the most languages. [21:18:11] halfak: dont forget to do that in markdown lang as well :P [21:18:28] Amir1: i stole your coc.md for my repos fyi [21:18:32] halfak: let me get back to you on that once reading group is over :) [21:18:48] halfak: https://github.com/Mavrikant/wikiclass/blob/master/wikiclass/extractors/trwiki.py [21:18:53] this may work. [21:19:07] Zppix: I stole those alreadyfrom mediawiki core :D [21:19:16] Amir1: didnt you do the core? [21:19:19] Nettrom, cool [21:19:30]