[10:00:50] <Amir1>	 o/
[10:58:48] <Amir1>	 I'll be afk to run some errands, will be back soon
[13:36:15] <Amir1>	 I'm back
[13:57:50] <halfak>	 o/
[14:03:18] <halfak>	 Hey Amir1.  Ready to meet? 
[15:46:50] <Amir1>	 awight: do you have some minutes to review: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ORES/+/462779 
[15:46:56] <Amir1>	 halfak: awight: https://github.com/wikimedia/ores/pull/268
[15:47:51] <halfak>	 Why not use python-logstash?
[15:48:01] <Amir1>	 and https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ORES/+/461815
[15:48:30] <wikibugs>	 (03CR) 10Awight: [C: 032] "Seems elegant." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/462779 (https://phabricator.wikimedia.org/T204961) (owner: 10Ladsgroup)
[15:48:52] <Amir1>	 halfak: I have it written somewhere, It doesn't support python3 (properly, it works but pretty buggy), it's just four files, it hasn't been touched in the past two years (unmaintained) 
[15:49:15] <halfak>	 I wonder if we could upstream your work :) 
[15:49:26] <halfak>	 Otherwise, looks good to me.  
[15:49:30] * halfak continues to review
[15:50:51] <halfak>	 So the idea is that we reference this formatter in the deployment configuration's logging_config.yaml?
[15:50:55] <halfak>	 Amir1, ^ 
[15:51:01] <Amir1>	 yup
[15:51:11] <Amir1>	 I already did in my localhost
[15:51:59] <Amir1>	 and it recieves the messages and shows them into local logstash, but in beta it's not working, my plan for rest of today is to find out why and have it properly tested in beta
[15:53:42] <halfak>	 I'm not sure about grabbing all fields from the record object with a black list.  That seems sketchy.  Why not use a whitelist strategy instead?
[15:53:44] <halfak>	 Amir1, ^ 
[15:55:15] <Amir1>	 Celery might want to send different type of logs than uwsgi then might have different keys, the blacklist seems more fitting in this case
[15:55:42] <Amir1>	 the reason for blacklist is that if it does let it pass through, it would cause fatals
[15:55:45] <wikibugs>	 (03Merged) 10jenkins-bot: Catch and gracefully handle when service is not responding properly to ApiHooksHandler [extensions/ORES] - 10https://gerrit.wikimedia.org/r/462779 (https://phabricator.wikimedia.org/T204961) (owner: 10Ladsgroup)
[15:56:03] <Amir1>	 otherwise, it  should be logstash's responsibility to parse to json 
[15:56:07] <Amir1>	 *the json
[15:57:54] <Amir1>	 I might be a little bit late
[15:57:56] <wikibugs>	 (03CR) 10jenkins-bot: Catch and gracefully handle when service is not responding properly to ApiHooksHandler [extensions/ORES] - 10https://gerrit.wikimedia.org/r/462779 (https://phabricator.wikimedia.org/T204961) (owner: 10Ladsgroup)
[16:14:39] <icinga2-wm>	 RECOVERY - check load on ORES-redis02.experimental is OK: OK - load average: 0.00, 0.00, 0.00
[16:14:54] <icinga2-wm>	 RECOVERY - check users on ORES-redis02.experimental is OK: USERS OK - 0 users currently logged in
[16:15:23] <icinga2-wm>	 RECOVERY - ssh on ORES-redis02.experimental is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u4 (protocol 2.0)
[16:19:44] <icinga2-wm>	 PROBLEM - ssh on ORES-redis02.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:21:31] <icinga2-wm>	 PROBLEM - check load on ORES-redis02.experimental is UNKNOWN: <Timeout exceeded.><Terminated by signal 9 (Killed).>
[16:21:46] <icinga2-wm>	 PROBLEM - check users on ORES-redis02.experimental is UNKNOWN: <Timeout exceeded.><Terminated by signal 9 (Killed).>
[16:25:30] <icinga2-wm>	 RECOVERY - ssh on ORES-web01.Experimental is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u4 (protocol 2.0)
[16:25:39] <icinga2-wm>	 RECOVERY - check users on ORES-web01.Experimental is OK: USERS OK - 0 users currently logged in
[16:25:41] <icinga2-wm>	 RECOVERY - check disk on ORES-web01.Experimental is OK: DISK OK
[16:28:49] <icinga2-wm>	 RECOVERY - ssh on ORES-web02.Experimental is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u4 (protocol 2.0)
[16:29:00] <icinga2-wm>	 RECOVERY - check users on ORES-worker02.experimental is OK: USERS OK - 0 users currently logged in
[16:29:02] <icinga2-wm>	 RECOVERY - check load on ORES-worker01.experimental is OK: OK - load average: 0.04, 0.04, 0.00
[16:29:11] <icinga2-wm>	 RECOVERY - ssh on ORES-worker01.experimental is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u4 (protocol 2.0)
[16:29:54] <icinga2-wm>	 PROBLEM - ssh on ORES-web01.Experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:30:13] <Zppix>	 Ugh
[16:32:30] <icinga2-wm>	 PROBLEM - check users on ORES-web01.Experimental is UNKNOWN: <Timeout exceeded.><Terminated by signal 9 (Killed).>
[16:32:32] <icinga2-wm>	 PROBLEM - check disk on ORES-web01.Experimental is UNKNOWN: <Timeout exceeded.><Terminated by signal 9 (Killed).>
[16:33:14] <icinga2-wm>	 PROBLEM - ssh on ORES-web02.Experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:33:36] <icinga2-wm>	 PROBLEM - ssh on ORES-worker01.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:35:47] <icinga2-wm>	 PROBLEM - check users on ORES-worker02.experimental is UNKNOWN: <Timeout exceeded.><Terminated by signal 9 (Killed).>
[16:35:56] <icinga2-wm>	 PROBLEM - check load on ORES-worker01.experimental is UNKNOWN: <Timeout exceeded.><Terminated by signal 9 (Killed).>
[16:50:41] <halfak>	 hoo, anything you want to chat about right now? 
[16:50:57] <halfak>	 maybe you could share what you have and/or talk through what you're getting stuck on. 
[16:52:48] <hoo>	 I'll be back in a bit (dinner time)
[16:53:06] <wikibugs>	 10Scoring-platform-team (Current), 10JADE: Review CSCW workshop paper for JADE - https://phabricator.wikimedia.org/T205892 (10awight)
[16:54:06] <wikibugs>	 10Scoring-platform-team (Current), 10JADE: Review CSCW workshop paper for JADE - https://phabricator.wikimedia.org/T205892 (10awight)
[16:55:00] <halfak>	 kk
[16:55:03] <halfak>	 Lunch for me then too
[17:01:14] <wikibugs>	 (03Abandoned) 10Awight: [WIP] abstractions to clean up tests [extensions/JADE] - 10https://gerrit.wikimedia.org/r/463174 (owner: 10Awight)
[17:04:14] <wikibugs>	 (03PS2) 10Awight: Add another namespace alias for the "Judgement_talk" spelling [extensions/JADE] - 10https://gerrit.wikimedia.org/r/463412 (https://phabricator.wikimedia.org/T205548)
[17:06:00] <Amir1>	 halfak|Lunch: fyi https://phabricator.wikimedia.org/T205849
[17:07:47] <halfak|Lunch>	 Nice.  Seems like a good ally
[17:08:50] <wikibugs>	 (03CR) 10Ladsgroup: [C: 032] Add another namespace alias for the "Judgement_talk" spelling [extensions/JADE] - 10https://gerrit.wikimedia.org/r/463412 (https://phabricator.wikimedia.org/T205548) (owner: 10Awight)
[17:11:41] <wikibugs>	 (03CR) 10Awight: "Looking over https://www.mediawiki.org/wiki/Manual:Job_queue/For_developers and JobRunner.php, I don't see anything that will retry our jo" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/461815 (https://phabricator.wikimedia.org/T204753) (owner: 10Ladsgroup)
[17:12:53] <wikibugs>	 (03Merged) 10jenkins-bot: Add another namespace alias for the "Judgement_talk" spelling [extensions/JADE] - 10https://gerrit.wikimedia.org/r/463412 (https://phabricator.wikimedia.org/T205548) (owner: 10Awight)
[17:15:03] <wikibugs>	 (03CR) 10jenkins-bot: Add another namespace alias for the "Judgement_talk" spelling [extensions/JADE] - 10https://gerrit.wikimedia.org/r/463412 (https://phabricator.wikimedia.org/T205548) (owner: 10Awight)
[17:18:13] <wikibugs>	 (03CR) 10Ladsgroup: "When the job returns false, it means the job needs to be retried. That's what Petr told me but I can search for the logic." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/461815 (https://phabricator.wikimedia.org/T204753) (owner: 10Ladsgroup)
[17:36:52] <wikibugs>	 (03CR) 10Awight: "Okay, I think I see the caller logic now.  JobRunner won't ack() a job if it returns false, and $job->allowRetries() is true (the default)" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/461815 (https://phabricator.wikimedia.org/T204753) (owner: 10Ladsgroup)
[17:42:35] <Amir1>	 halfak|Lunch: https://github.com/wikimedia/ores/pull/268/files this is done, do you think we can merge it? Also we can turn it into a library later if we think it's a good fit
[17:42:44] <Amir1>	 right now, it's too small IMO
[17:43:18] <halfak|Lunch>	 Those vars should really be at the class level because they don't change for each instance, right?
[17:44:24] <halfak|Lunch>	 IMO, re-usability is the scope of a library -- not size.  But I don't want to block on that
[17:44:26] <halfak|Lunch>	 Amir1, ^ 
[17:44:52] <Amir1>	 halfak|Lunch: I turned it to a class variable as you said
[17:47:08] <halfak|Lunch>	 It's a member variable now. 
[17:47:28] <halfak|Lunch>	 class variables appear in cope above the def __init__():
[17:49:01] <halfak>	 e.g. https://github.com/wikimedia/revscoring/blob/master/revscoring/scoring/models/sklearn.py#L27
[17:49:25] <Amir1>	 oh okay
[17:49:46] <icinga2-wm>	 PROBLEM - ssh on ORES-redis02.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:51:36] <icinga2-wm>	 PROBLEM - check load on ORES-redis02.experimental is UNKNOWN: <Timeout exceeded.><Terminated by signal 9 (Killed).>
[17:51:51] <icinga2-wm>	 PROBLEM - check users on ORES-redis02.experimental is UNKNOWN: <Timeout exceeded.><Terminated by signal 9 (Killed).>
[17:59:56] <icinga2-wm>	 PROBLEM - ssh on ORES-web01.Experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:00:28] <Amir1>	 halfak: done
[18:02:31] <icinga2-wm>	 PROBLEM - check users on ORES-web01.Experimental is UNKNOWN: <Timeout exceeded.><Terminated by signal 9 (Killed).>
[18:02:36] <icinga2-wm>	 PROBLEM - check disk on ORES-web01.Experimental is UNKNOWN: <Timeout exceeded.><Terminated by signal 9 (Killed).>
[18:02:58] <wikibugs>	 (03CR) 10Ladsgroup: "There is a $maxTries config in JobQueue, I think it's 30." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/461815 (https://phabricator.wikimedia.org/T204753) (owner: 10Ladsgroup)
[18:02:59] <halfak>	 I don't know if you can reference class vars just like member vars.  I'll be watching the to see if tests pass. 
[18:03:16] <icinga2-wm>	 PROBLEM - ssh on ORES-web02.Experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:03:41] <icinga2-wm>	 PROBLEM - ssh on ORES-worker01.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:05:51] <icinga2-wm>	 PROBLEM - check users on ORES-worker02.experimental is UNKNOWN: <Timeout exceeded.><Terminated by signal 9 (Killed).>
[18:05:56] <icinga2-wm>	 PROBLEM - check load on ORES-worker01.experimental is UNKNOWN: <Timeout exceeded.><Terminated by signal 9 (Killed).>
[18:19:15] <icinga2-wm>	 ACKNOWLEDGEMENT - ssh on ORES-redis02.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds paladox ack
[18:19:16] <icinga2-wm>	 ACKNOWLEDGEMENT - ssh on ORES-web02.Experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds paladox ack
[18:19:16] <icinga2-wm>	 ACKNOWLEDGEMENT - ssh on ORES-web01.Experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds paladox ack
[18:19:16] <icinga2-wm>	 ACKNOWLEDGEMENT - ssh on ORES-worker01.experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds paladox ack
[18:19:43] <icinga2-wm>	 ACKNOWLEDGEMENT - check disk on ORES-web01.Experimental is UNKNOWN: <Timeout exceeded.><Terminated by signal 9 (Killed).> paladox ack
[18:19:44] <icinga2-wm>	 ACKNOWLEDGEMENT - check load on ORES-worker01.experimental is UNKNOWN: <Timeout exceeded.><Terminated by signal 9 (Killed).> paladox ack
[18:19:44] <icinga2-wm>	 ACKNOWLEDGEMENT - check users on ORES-web01.Experimental is UNKNOWN: <Timeout exceeded.><Terminated by signal 9 (Killed).> paladox ack
[18:19:44] <icinga2-wm>	 ACKNOWLEDGEMENT - check users on ORES-worker02.experimental is UNKNOWN: <Timeout exceeded.><Terminated by signal 9 (Killed).> paladox ack
[18:20:30] <icinga2-wm>	 ACKNOWLEDGEMENT - check load on ORES-redis02.experimental is UNKNOWN: <Timeout exceeded.><Terminated by signal 9 (Killed).> paladox ackACKNOWLEDGEMENT - check users on ORES-redis02.experimental is UNKNOWN: <Timeout exceeded.><Terminated by signal 9 (Killed).> paladox ack
[18:22:13] <awight>	 Reference the variable like Klass.varname
[18:27:16] <awight>	 If the tests pass, it's because we shadowed the class variable with a local variable
[18:27:18] <wikibugs>	 10Scoring-platform-team, 10ORES: ORES command line service sometimes hangs - https://phabricator.wikimedia.org/T205909 (10Hall1467)
[18:27:19] <awight>	 Amir1: halfak: ^
[18:29:27] <Amir1>	 I see
[18:29:28] <Amir1>	 thanks
[18:36:58] <icinga2-wm>	 PROBLEM - ssh on ORES-web02.Experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:55:16] <wikibugs>	 10Scoring-platform-team (Current), 10DBA, 10JADE, 10Operations, and 3 others: Write our anticipated "phase two" schemas and submit for review - https://phabricator.wikimedia.org/T202596 (10awight) @Marostegui I'm not sure if this helps, but I'll try to better illustrate my question using a real-world examp...
[19:25:38] <Amir1>	 halfak: Do you think we can drop ores-misc-01?
[19:25:44] <Amir1>	 Is anyone using it?
[19:26:12] <Amir1>	 the ores-staging project doesn't have any more memory so I can't make a dedicated node to test logstash there
[19:26:31] <halfak>	 Amir1, use ores-misc-01
[19:26:49] <Amir1>	 mwhahahaha
[19:26:55] <Amir1>	 mw = mediawiki
[19:28:00] <notconfusing>	 Hey, sorry for missing the meeting this morning. One question I had is if there are any standard ways of announcing new/experiment Wikilabels campaigns.
[19:31:51] <halfak>	 mimic this: https://en.wikipedia.org/wiki/Wikipedia:Labels/Edit_quality
[19:31:55] <halfak>	 notconfusing, ^ 
[19:32:04] <notconfusing>	 thanks
[19:48:47] <awight>	 notconfusing: We missed the opportunity to call you *Mr.* confusing
[20:01:36] <wikibugs>	 10Scoring-platform-team (Current), 10JADE: Review CSCW workshop paper for JADE - https://phabricator.wikimedia.org/T205892 (10awight) [I've received some comments from halfak]
[20:07:00] <icinga2-wm>	 PROBLEM - ssh on ORES-web02.Experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:16:26] <Amir1>	 I'm trying to install logstash, elastic and kibana on ores-misc-01
[20:16:31] <Amir1>	 it's soooooo fun...
[20:17:48] <awight>	 Amir1: That sounds like pay band 6 duties O_O
[20:19:19] <wikibugs>	 10Scoring-platform-team, 10ORES: ORES command line service sometimes hangs - https://phabricator.wikimedia.org/T205909 (10Halfak) I talked to @Hall1467 about this issue and advised him to update his ORES utility and try again.  We'll see how that goes.
[20:22:19] <notconfusing>	 awight: yeah, i'm upgraded. actually its interesting, its not like a birthday where having a different age number doesn't feel any different, this is more like, "ok i have a whole new set of priorities in life"
[20:24:18] <awight>	 LOL can't wait to hear about it
[20:25:08] <awight>	 Kids was a nice upgrade too, cos suddenly staying out drinking with friends became a non-thing.
[20:25:27] <awight>	 &, nicely formulated.
[20:52:25] <wikibugs>	 10Scoring-platform-team (Current), 10editquality-modeling, 10revscoring, 10artificial-intelligence: Create a newcomerquality meta-model for revscoring - https://phabricator.wikimedia.org/T205926 (10notconfusing)
[21:11:37] <awight>	 harej: I'm toying with a core patch, (fun. https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/463822/ ) and then will revise [[mw:JADE]]
[21:11:38] <AsimovBot>	 10[8] 10https://www.mediawiki.org/wiki/JADE
[21:12:09] <Amir1>	 I'm really done for the day, see you on Wednesday! I'm around tomorrow in case you need me for five minutes or everything is on fire  but will be working on wikidata stuff
[21:16:50] <paladox>	 Amir1 mw-vagrant allows you to setup logstash / elasticsearch and kibanan
[21:16:55] <paladox>	 *kibana with ease
[21:27:50] <wikibugs>	 10Scoring-platform-team, 10ORES: ORES command line service sometimes hangs - https://phabricator.wikimedia.org/T205909 (10Hall1467) I updated the ORES utility to 1.2.0 and rerun. It ran for a little bit and then hung for 20 minutes. I then hit control-c and received the following traceback:  ```     conn.conne...
[21:37:03] <icinga2-wm>	 PROBLEM - ssh on ORES-web02.Experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:53:03] <awight>	 Amir1: o/ enjoy your day "off"
[21:53:18] <awight>	 harej: k looking mw:JADE
[21:53:52] <harej>	 halfak: I just saw. We hired another Minneapolis person!
[21:57:04] <harej>	 "My research focuses on how we can bring greater transparency and accountability to algorithms, particularly in support of causes like knowledge equity"
[21:57:10] <harej>	 Aaaand now it's my job to talk to him :P
[21:59:17] <halfak>	 \o/  Forgot about Isaac starting today
[22:05:39] <halfak>	 Alright.  I'm gonna head out.  I think I'll check out mw:JADE in the AM.  Don't be blocked on me. 
[22:08:35] <hoo>	 halfak: How would you recommend I talk to the API endpoint? I figured I need to define two Datasource instances here… but not entirely sure how to propagate them
[22:10:20] <halfak>	 hoo, don't sweat that for now.  For right now, just grab the output of the endpoint and build up the dependency flow. 
[22:10:26] <halfak>	 Using Datasource and Feature. 
[22:10:47] <halfak>	 Once you have that nailed down, let's take a careful look at revscoring.extractor.api.Extractor ;) 
[22:11:29] <hoo>	 Ok, I'll use plain mwapi for now
[22:11:53] <halfak>	 That works.  Or just copy-paste from your browser :D 
[22:13:42] <halfak>	 hoo, don't sweat it now, but when we get there, we'll need to make something like this: https://github.com/wikimedia/revscoring/blob/master/revscoring/extractors/api/datasources.py
[22:14:00] <halfak>	 ItemPropertySuggestionDoc
[22:14:01] <halfak>	 or whatever
[22:15:13] <wikibugs>	 10Scoring-platform-team, 10ORES: ORES command line service sometimes hangs - https://phabricator.wikimedia.org/T205909 (10Halfak) @Ladsgroup, I think this might be related to poolcounter.  Can you remind me of the constraints that will result in 429 responses?  I plan to take a pass through the code and do a s...
[22:15:34] <halfak>	 OK actually leaving now. 
[22:15:38] <halfak>	 Have a good one, folks!
[22:20:09] <wikibugs>	 10Scoring-platform-team (Current), 10Bad-Words-Detection-System, 10revscoring, 10Patch-For-Review, and 2 others: Add language support for galician - https://phabricator.wikimedia.org/T201142 (10Halfak) Merged!
[22:20:45] <halAFK>	 sshhh wikibugs. 
[22:50:34] <awight>	 harej: My core patch is being helpfully, repeatedly kicked in an alleyway, but I'm still hoping to get to mw:JADE...
[22:52:20] <harej>	 I'm like 0 for 3 on core patches so I definitely relate.
[22:54:39] <awight>	 :)
[22:55:52] <awight>	 It's also hard to know when log spam and other glitches were introduced by my work
[22:55:57] <harej>	 As it turns out, they don't like very much when you change the method signature of something in the parser.
[22:56:31] <harej>	 Thus breaking every extension that comes to rely on that method
[22:57:16] <harej>	 If I remember correctly, I was trying to get the page title be part of the data bundle that's handed off to the template expander so that templates could be aware of the pages they are on.
[23:00:33] <awight>	 hehe very nice.  I can imagine that being a hit
[23:07:06] <icinga2-wm>	 PROBLEM - ssh on ORES-web02.Experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:40:52] <wikibugs>	 10Scoring-platform-team (Current), 10Community-Tech, 10JADE, 10WMF-Communications: Blog about JADE - https://phabricator.wikimedia.org/T183200 (10Niharika) @awight Is this dependent on #community-tech?
[23:43:09] <wikibugs>	 10Scoring-platform-team (Current), 10Community-Tech, 10JADE, 10WMF-Communications: Blog about JADE - https://phabricator.wikimedia.org/T183200 (10Harej) I think that was in reference to @DannyH being pinged above. It is my understanding that Aaron worked with Danny to address the issue in the blog post.
[23:50:21] <wikibugs>	 10Scoring-platform-team, 10Growth-Team, 10MediaWiki-Recent-changes, 10MediaWiki-extensions-ORES, and 2 others: ORES and Enhanced Recent Changes: Background color gets stuck after patrolling - https://phabricator.wikimedia.org/T171953 (10Catrope)