[00:04:37] <halfak>	 Fun story!  Most of our models are nor compatible with revscoring 2.1.0
[00:04:44] <halfak>	 woo.  And I can't roll back this mess. 
[00:04:48] <halfak>	 Why did this work on staging? 
[00:04:49] <halfak>	 hmmm
[00:05:12] <halfak>	 It doesn't work on staging.  Double "hmmm"
[00:05:26] <halfak>	 Oh well.  experimental is experimental.  I'm sure a little downtime won't start anyone on fire. 
[00:05:48] <halfak>	 I'm going to get a new revscoring version that is robust to this issue and that should solve things. 
[00:11:47] <halfak>	 OK that's not going to work nicely.  
[00:11:59] <halfak>	 Looks like I'm rebuilding all of the models.  That probably a good idea anyway. 
[04:06:11] <icinga2-wm>	 DOWNTIMEEND - ping4 on ORES-redis01.experimental is UNKNOWN: check_ping: Invalid hostname/address - ores-redis-01.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4 zppix WIP
[04:06:11] <icinga2-wm>	 DOWNTIMEEND - ssh on ORES-redis01.experimental is UNKNOWN: Usage:check_ssh [-4 zppix WIP
[04:06:11] <icinga2-wm>	 DOWNTIMEEND - check users on ORES-redis01.experimental is WARNING: Could not resolve hostname ores-redis-01.ores.eqiad.wmflabs: Name or service not known zppix WIP
[04:06:11] <icinga2-wm>	 DOWNTIMEEND - check disk on ORES-redis01.experimental is WARNING: Could not resolve hostname ores-redis-01.ores.eqiad.wmflabs: Name or service not known zppix WIP
[04:06:11] <icinga2-wm>	 DOWNTIMEEND - Host ORES-redis01.experimental is DOWN: check_ping: Invalid hostname/address - ores-redis-01.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4 zppix WIP
[04:06:13] <icinga2-wm>	 DOWNTIMEEND - puppet on ORES-redis01.experimental is WARNING: Could not resolve hostname ores-redis-01.ores.eqiad.wmflabs: Name or service not known zppix WIPDOWNTIMEEND - check load on ORES-redis01.experimental is WARNING: Could not resolve hostname ores-redis-01.ores.eqiad.wmflabs: Name or service not known zppix WIP
[04:42:30] <icinga2-wm>	 PROBLEM - Host ORES-redis01.experimental is DOWN: check_ping: Invalid hostname/address - ores-redis-01.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[05:22:30] <icinga2-wm>	 PROBLEM - Host ORES-redis01.experimental is DOWN: check_ping: Invalid hostname/address - ores-redis-01.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[06:02:34] <icinga2-wm>	 PROBLEM - Host ORES-redis01.experimental is DOWN: check_ping: Invalid hostname/address - ores-redis-01.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[06:25:25] <icinga-wm>	 PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:26:05] <icinga-wm>	 PROBLEM - ORES web node labs ores-web-03 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:32:11] <icinga2-wm>	 DOWNTIMEEND - check disk on ORES-web03.experimental is WARNING: Could not resolve hostname ores-web-03.ores.eqiad.wmflabs: Name or service not known zppix https://gerrit.wikimedia.org/r/404584
[06:32:11] <icinga2-wm>	 DOWNTIMEEND - Host ORES-web03.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-03.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4 zppix https://gerrit.wikimedia.org/r/404584
[06:32:11] <icinga2-wm>	 DOWNTIMEEND - check load on ORES-web03.experimental is WARNING: Could not resolve hostname ores-web-03.ores.eqiad.wmflabs: Name or service not known zppix https://gerrit.wikimedia.org/r/404584
[06:32:11] <icinga2-wm>	 DOWNTIMEEND - ssh on ORES-web03.experimental is UNKNOWN: Usage:check_ssh [-4 zppix https://gerrit.wikimedia.org/r/404584
[06:32:12] <icinga2-wm>	 DOWNTIMEEND - ping4 on ORES-web03.experimental is UNKNOWN: check_ping: Invalid hostname/address - ores-web-03.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4 zppix https://gerrit.wikimedia.org/r/404584
[06:32:13] <icinga2-wm>	 DOWNTIMEEND - check users on ORES-web03.experimental is WARNING: Could not resolve hostname ores-web-03.ores.eqiad.wmflabs: Name or service not known zppix https://gerrit.wikimedia.org/r/404584
[06:32:14] <icinga2-wm>	 DOWNTIMEEND - puppet on ORES-web03.experimental is WARNING: Could not resolve hostname ores-web-03.ores.eqiad.wmflabs: Name or service not known zppix https://gerrit.wikimedia.org/r/404584
[06:42:39] <icinga2-wm>	 PROBLEM - Host ORES-redis01.experimental is DOWN: check_ping: Invalid hostname/address - ores-redis-01.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[07:00:13] <travis-ci>	 wiki-ai/wikiclass#40 (master - 54349b6 : Phantom42): The build has errored. https://travis-ci.org/wiki-ai/wikiclass/builds/329692446
[07:05:04] <icinga2-wm>	 PROBLEM - Host ORES-web03.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-03.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[07:22:44] <icinga2-wm>	 PROBLEM - Host ORES-redis01.experimental is DOWN: check_ping: Invalid hostname/address - ores-redis-01.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[07:45:04] <icinga2-wm>	 PROBLEM - Host ORES-web03.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-03.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[08:02:44] <icinga2-wm>	 PROBLEM - Host ORES-redis01.experimental is DOWN: check_ping: Invalid hostname/address - ores-redis-01.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[08:25:09] <icinga2-wm>	 PROBLEM - Host ORES-web03.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-03.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[10:50:10] <wikibugs>	 10Scoring-platform-team (Current), 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Investigate code generation for model makefile maintenance - https://phabricator.wikimedia.org/T168455#3905713 (10Ladsgroup) a:03Ladsgroup
[11:19:13] <travis-ci>	 wiki-ai/editquality#44 (templating - 37850de : Adam Roses Wight): The build failed. https://travis-ci.org/wiki-ai/editquality/builds/329823931
[13:21:22] <Zppix>	 Morning adam ( awight )
[13:21:49] <awight>	 Zppix: How’s the home front?
[13:22:04] <Zppix>	 Lol good
[14:35:17] <wikibugs>	 (03PS1) 10Awight: [WIP] Take advantage of MaintenanceBaseTestCase [extensions/ORES] - 10https://gerrit.wikimedia.org/r/404677
[14:36:55] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [WIP] Take advantage of MaintenanceBaseTestCase [extensions/ORES] - 10https://gerrit.wikimedia.org/r/404677 (owner: 10Awight)
[14:55:21] <halfak>	 o/
[14:59:33] <awight>	 halloo
[15:25:15] <travis-ci>	 wiki-ai/editquality#47 (templating - 2968fdd : Amir Sarabadani): The build was fixed. https://travis-ci.org/wiki-ai/editquality/builds/329919380
[15:51:02] <wikibugs>	 10Scoring-platform-team, 10ORES, 10cloud-services-team: dpkg ailing on ores-misc-01.ores-staging.eqiad.wmflabs - https://phabricator.wikimedia.org/T184494#3906299 (10Andrew) 05Open>03Resolved a:03Andrew Either someone fixed this or it settled down on its own.
[15:58:51] <wikibugs>	 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10ORES, 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Special:RecentChanges broken on Jenkins slaves - https://phabricator.wikimedia.org/T184938#3906336 (10zeljkofilipin) Special:RecentChanges is broken on a clean mediawiki-vagrant VM wit...
[16:00:59] <wikibugs>	 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10ORES, 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Special:RecentChanges broken on Jenkins slaves - https://phabricator.wikimedia.org/T184938#3901120 (10awight) I think E:ORES should handle the corner case more gracefully.  Anyway, thi...
[16:08:48] <wikibugs>	 (03PS6) 10Zfilipin: WIP Create Selenium UI tests for ORES damaging and good faith filters. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/402863 (https://phabricator.wikimedia.org/T184451) (owner: 10Etonkovidova)
[16:10:27] <wikibugs>	 (03CR) 10Zfilipin: "Patch set #6 creates a page (via API)." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/402863 (https://phabricator.wikimedia.org/T184451) (owner: 10Etonkovidova)
[16:10:35] <wikibugs>	 (03CR) 10Zfilipin: "check experimental" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/402863 (https://phabricator.wikimedia.org/T184451) (owner: 10Etonkovidova)
[16:11:08] <Amir1>	 awight: halfak what do you think if I remove enwiki config file and work with some smaller wikis and do the migration as much as possible until I do another round of automation improvement?
[16:11:36] <awight>	 +1 or rename it?
[16:11:58] <Amir1>	 yeah, that also work :D
[16:12:02] <Amir1>	 I need to go
[16:12:04] <awight>	 I think getting the WIP to a point where it can be merged is a big step...
[16:12:05] <Amir1>	 be back soon
[16:17:57] <wikibugs>	 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10ORES, 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Special:RecentChanges broken on Jenkins slaves - https://phabricator.wikimedia.org/T184938#3906386 (10zeljkofilipin) Creating a page (via API) did not fix the problem ([[ https://gerri...
[17:01:12] <awight>	 Amir1: halfak|Network_d: I forgot to announce that the crazy starts today.  I have to take off at 18:00 UTC and probably won’t be back until tomorrow.
[17:07:14] <Amir1>	 Okay :)
[17:14:33] <Amir1>	 halfak: when do you have some time to review the automation patch? :D
[17:15:25] <halfak>	 Amir1, sure!
[17:15:33] <halfak>	 What do you mean by "automation"?
[17:15:40] <Amir1>	 I update the pr in  a sec
[17:16:53] <halfak>	 I'll need you for review of the model PRs I'll be sending in. 
[17:17:00] <halfak>	 There'll be minor makefile changes :\
[17:18:31] <Amir1>	 sure, one thing: Do you think I should remove arwiki make file from the manual as the automated is practically the same or we should keep it until we are done with all? 
[17:19:23] <Amir1>	 halfak: I tend to lean towards removing because it makes the make file a little bit smaller and also we don't end up diverging (in case someone changes the manual file)
[17:19:32] <awight>	 +1 ^
[17:19:42] <Amir1>	 awight: I thought you left :D
[17:19:53] <awight>	 I also think that the normal makefile should call the automated one
[17:19:58] <awight>	 hehe not for another 40 minutes
[17:21:12] <Amir1>	 hmm, yeah, awight: What is your suggested command to add to the Makefile?
[17:21:49] <awight>	 It’s usually “make -f $(SUBDIR)/Makefile"
[17:22:20] <awight>	 oh hey
[17:22:20] <awight>	 https://www.gnu.org/software/make/manual/html_node/Recursion.html
[17:22:49] <awight>	 eh nvm without the subdir, we do want make -f
[17:23:40] <halfak>	 " remove arwiki make file from the manual" 
[17:23:44] <halfak>	 I'm not sure I understand that
[17:24:24] <awight>	 The idea is that, once the arwiki rules are correct in the code-generated file  (=== the existing manually-written rules), remove from the manual Makefile
[17:24:36] <halfak>	 awight, can we "template.py $(SUBDIR)/Makefile.template | make -read-stdin" or something like that?
[17:25:14] <halfak>	 That way we don't need to actually write out the file, but we can if we want. 
[17:25:39] <awight>	 IMO it’s better to have the rules written out in a real file
[17:25:49] <awight>	 So we can diagnose if there are problems, etc.
[17:25:55] <halfak>	 awight, generally, the file itself is ephemeral 
[17:26:05] <halfak>	 But it is nice that we *can* write out to the file. 
[17:26:09] <halfak>	 I agree. 
[17:26:30] <Amir1>	 Updated the PR, ready for merge :D
[17:26:44] <Amir1>	 so I can move on to another wiki and automate even more
[17:26:45] <Amir1>	 \
[17:26:45] <awight>	 It’s a murky world with CG.  This isn’t like a compiled binary, this is an itermediate file which is used to compile further things.
[17:26:49] <Amir1>	 \o/
[17:27:28] <awight>	 halfak: What reasons do you have in mind for wanting the Makefile.out to be ephemeral?
[17:27:32] <halfak>	 awight, I could see keeping the file if the file itself is part of a dependency chain. 
[17:27:54] <halfak>	 E.g. if we change enwiki/template.lol then it would automatically rebuild enwiki/Makefile. 
[17:28:00] <awight>	 Just for example, when using autoconf the standard is to keep the resulting ./configure etc. committed to the repo
[17:28:23] <awight>	 That way, a person picking up your repo to use locally doesn’t have to install the meta-dependendencies
[17:28:58] <halfak>	 awight, aere you available to review Amir1's PR?
[17:28:58] <awight>	 s/den//
[17:29:13] <halfak>	 lol 
[17:29:18] <awight>	 halfak: I did—but want to give it another look?
[17:29:29] <awight>	 This is a lot of my code, so a 3rd set of eyes would be nice.
[17:29:43] <halfak>	 Na.  I'll let y'all focus on that.  If I don't get ores.wmflabs.org ready soon, we're gonna be sad pandas. 
[17:29:58] <halfak>	 Hmm... Maybe that's a good point. 
[17:30:34] <awight>	 Anything I can do for wmflabs, or is it just watching progress bars on stat1005?
[17:35:53] <halfak>	 progress bars for now and then reviewing PRs with lotsa models. 
[17:36:04] <halfak>	 Also, I'm dealing with some reboots :\
[17:39:38] <halfak>	 awight, I think I want to try to get through this work before I review given the constraints and how data processing often has surprise delays. 
[17:39:56] <halfak>	 I'll side with you and Amir1 re. templating Makefiles and deal with the consequences :) 
[17:40:16] <awight>	 halfak: No problem, I think that’ll work :)
[17:43:05] <awight>	 Amir1: I was wrong about make -f, see my CR comment
[17:43:12] <awight>	 “include” will do exactly what we want, though.
[17:45:12] <Amir1>	 I was looking for include
[17:45:25] <Amir1>	 cool, I fix one small thing and submit it again
[17:45:55] <awight>	 Amir1: Are we okay with “cg.py”, or should that be wrapped as a utility script?
[17:46:19] <Amir1>	 yeah, the only thing is that in that case we will be doing a lot in one go
[17:46:26] <Amir1>	 that's the thing I'm trying to avoid
[17:46:33] <awight>	 kk that’s fine with me
[17:49:42] <Amir1>	 awight: submitted
[17:50:54] <awight>	 Amir1: Have you tried it locally?
[17:51:12] <awight>	 I think the high-level rules will conflict, and should be comment out in the template.
[17:51:15] <Amir1>	 Both differ and make file (the differ before removing)
[17:51:23] <awight>	 e.g. models:
[17:52:13] <halfak>	 IT WORKS
[17:52:18] <halfak>	 Well, preliminarily. 
[17:52:26] <halfak>	 Looks like most of the models missed the reboot :) 
[17:52:52] <Amir1>	 hmm, okay
[17:53:45] <awight>	 Amir1: Interesting!  the makefile doesn’t fail like I thought
[17:54:10] <awight>	 I think the rule defined later wins.
[17:54:26] <awight>	 Luckily, the main makefile has models <- arwiki_models ...
[17:54:49] <Amir1>	 awight: submitted some changes
[17:54:54] <Amir1>	 please take a look
[17:54:58] <Amir1>	 works locally
[17:57:30] <awight>	 Amir1: Would you prefer that I squash-merge?
[17:57:56] <Amir1>	 yeah
[17:58:54] <Amir1>	 Thank yooouuu
[17:59:07] <Amir1>	 Let's march forward with some other wikis :P
[17:59:15] <awight>	 likewise!  Awesome to see a fun project take off.
[17:59:50] <awight>	 back in a few hours, if I’m lucky
[18:02:03] <halfak>	 https://github.com/wiki-ai/draftquality/pull/19
[18:03:36] * halfak starts training the last models for editquality. 
[18:10:11] <halfak>	 heading to lunch.  Back in a bit. 
[18:55:21] <Amir1>	 halfak|Lunch: Tell me when you're back
[18:57:18] <Zppix>	 ATTENTION: ICINGA 2 HOSTS WILL BE GOING DOWN IN 3 MINUTES
[19:02:35] <travis-ci>	 wiki-ai/editquality#56 (more_templating - fc97824 : Amir Sarabadani): The build passed. https://travis-ci.org/wiki-ai/editquality/builds/330017573
[19:02:58] <Amir1>	 halfak|Lunch: when you're back, please review this ^
[19:05:29] * halfak|Lunch opens
[19:07:13] <halfak>	 Amir1, Makefile.automated is the generated file?
[19:07:20] <Amir1>	 halfak: yup
[19:07:38] <halfak>	 cool
[19:08:04] <halfak>	 Amir1, what's Makefile.j2?
[19:08:09] <halfak>	 oh nevermiund
[19:09:47] <codezee>	 halfak: from a licensing standpoint if i put some graphs on commons that need to go in paper, is it ok?
[19:09:59] <Amir1>	 halfak: that's the crazy template
[19:10:06] <halfak>	 codezee, yes.  Don't sweat that. 
[19:10:23] <halfak>	 If we get called out, we might need to put some licensing info in the paper. 
[19:10:31] <halfak>	 But since you generated the plots, you can dual license. 
[19:20:24] <Amir1>	 I'm going home, will work a little from there
[19:20:28] <Amir1>	 see you
[19:21:10] <travis-ci>	 wiki-ai/editquality#58 (2.1_rebuild - 486a71a : Aaron Halfaker): The build passed. https://travis-ci.org/wiki-ai/editquality/builds/330024263
[19:22:07] <halfak>	 https://github.com/wiki-ai/editquality/pull/116
[19:22:17] <halfak>	 Safe travels, Amir1
[19:22:36] <codezee>	 halfak: https://commons.wikimedia.org/wiki/File:Drafttopic_roc_auc.jpg and https://commons.wikimedia.org/wiki/File:Drafttopic_pr_auc.jpg
[19:22:45] <codezee>	 i'm not sure but the PR_AUC curve looks weird
[19:24:27] <halfak>	 Looks OK to me.  What do you think looks weird codezee?
[19:24:59] <codezee>	 STEM.Technology stands out with a sharp dip then levels out opposite to others
[19:30:37] <codezee>	 halfak: ok, i'm done with graphs, had to learn some R to get myself round to it, i shall now be able to share with you  the final draft by tomorrow or day after
[19:33:45] <halfak>	 codezee, great!  I'll be aiming to get you writeup around the new page review process this weekend.  I think that'll put us in a really good place. 
[20:10:43] <halfak>	 codezee, just did a little bit of related work digging for talking about page review and found a great paper that I totally forgot about 
[20:10:53] <halfak>	 (and I'm one of the authors on the paper -- lol)
[20:11:02] <halfak>	 https://www-users.cs.umn.edu/~halfaker/publications/Accept_Decline_Postpone/schneider14accept.pdf
[20:29:46] <Amir1>	 halfak: I'm back, I explained in the PR :D
[20:34:05] <codezee>	 halfak: oh, :D thats interesting i'll go through it
[20:34:47] <codezee>	 as a matter of fact i had gone through some of your papers while looking for the reference on Keilana effect...and found some useful others too
[20:36:40] <halfak>	 :DDD
[20:42:06] <travis-ci>	 wiki-ai/wikiclass#41 (2.1_rebuild - 9aa8fde : Aaron Halfaker): The build passed. https://travis-ci.org/wiki-ai/wikiclass/builds/330058901
[20:47:54] <halfak>	 Amir1, https://github.com/wiki-ai/wikiclass/pull/60
[20:50:01] <halfak>	 awight, back? 
[20:50:08] <awight>	 Hi!
[20:50:20] <halfak>	 I was just about to merge https://github.com/wiki-ai/editquality/pull/115 
[20:50:22] <awight>	 Yep, but only for 1 nap = c. 1.5 hr
[20:50:27] <halfak>	 Want to have a look? 
[20:50:34] <awight>	 holy cow
[21:06:39] <halfak>	 Nice!  I'll get staging up. 
[21:10:05] <travis-ci>	 wiki-ai/wikiclass#43 (master - f8bdf38 : Amir Sarabadani): The build passed. https://travis-ci.org/wiki-ai/wikiclass/builds/330071000
[21:15:33] <awight>	 halfak: +1 on that +2 if you wish
[21:16:07] <awight>	 Amir1: halfak: IMO we should be consistent about removing manual rules as soon as the generated rules match
[21:16:27] <halfak>	 I agree. 
[21:16:56] <Amir1>	 awight: They only thing is I want to give reviewer chance to compare 
[21:17:11] <Amir1>	 because they are not 100% the same but the changes are minor
[21:17:34] <Amir1>	 some are very minor like "=" instead of " " in arguments or order of them
[21:20:10] <halfak>	 Hmm.. The comparison is helpful.  Maybe you could instead post a diff on the PR
[21:20:32] <Amir1>	 good idea 
[21:21:35] <awight>	 Amir1: Another really pervy thing we could do is, replace the manual Makefile rules with the auto generated ones.  But that’s an equally annoying amount of footwork.
[21:22:31] <Amir1>	 I think my cat tries to find Narnia
[21:23:06] <awight>	 wardrobe?
[21:23:10] <paladox>	 zuul v3 supports github
[21:23:32] <paladox>	 i think zuul v3 will be tested at some point (far down the road).
[21:23:34] <Amir1>	 yup
[21:24:00] <paladox>	 zuul v2 looks like it's discontinued and zuul v3 drops support for jenkins but it still uses the plugin so bascially it will still work (lol).
[21:24:50] <Amir1>	 I will make the diff tomorrow, It's in my work laptop and cloning that beast into my home laptop will take really long time
[21:29:18] <awight>	 Amir1: I’ll try to make the diff, mostly to see if I can.
[21:29:31] <awight>	 looks like the syntax is just “python differ.py bnwiki”
[21:29:48] <halfak>	 All looks good on ores-staging.  Going for the full cloud cluster
[21:29:50] <Amir1>	 awight: it's python differ "Bengali Wikipedia"
[21:30:07] <awight>	 ooh fancy k
[21:30:12] <Amir1>	 To use the header name 
[21:30:29] <Amir1>	 but let's make people think it parsing it :D
[21:31:52] <awight>	 lmao
[21:34:25] <wikibugs>	 (03CR) 10Ladsgroup: [C: 032] Tests for PopulateDatabase [extensions/ORES] - 10https://gerrit.wikimedia.org/r/403870 (https://phabricator.wikimedia.org/T184140) (owner: 10Awight)
[21:35:03] <awight>	 Amir1: Watch out, if we hit 100% test coverage then we’re obligated to keep the needle pinned in perpetuity.
[21:35:49] <halfak>	 Confirmed.  I'll make it a goal/commandment. 
[21:36:08] <Amir1>	 :)))
[21:36:13] <awight>	 With totalitarian CI enforcement
[21:36:18] <wikibugs>	 (03Merged) 10jenkins-bot: Tests for PopulateDatabase [extensions/ORES] - 10https://gerrit.wikimedia.org/r/403870 (https://phabricator.wikimedia.org/T184140) (owner: 10Awight)
[21:36:23] <Amir1>	 Thou shall not write CI tests
[21:36:25] <halfak>	 Benevolent Totalitarianism
[21:36:28] <Amir1>	 sounds very mighty
[21:38:10] <halfak>	 https://ores.wmflabs.org/v3/scores/enwiki/32456784/wp10?features
[21:38:11] <halfak>	 WORKS
[21:38:12] <halfak>	 DONE
[21:38:14] <halfak>	 WOO
[21:39:04] <wikibugs>	 10Scoring-platform-team (Current), 10ORES: Convert CloudVPS instances to stretch. - https://phabricator.wikimedia.org/T184296#3907553 (10Halfak) Done! https://ores.wmflabs.org/v3/scores/enwiki/32456784/wp10?features
[21:39:26] <wikibugs>	 10Scoring-platform-team (Current), 10ORES: Back up ores-misc-01 to ores-staging-01 - https://phabricator.wikimedia.org/T184765#3894658 (10Halfak) a:03Sumit
[21:39:32] <halfak>	 Time to kill ores-misc-01
[21:39:40] <halfak>	 Last minute concerns?
[21:39:43] <wikibugs>	 (03CR) 10jenkins-bot: Tests for PopulateDatabase [extensions/ORES] - 10https://gerrit.wikimedia.org/r/403870 (https://phabricator.wikimedia.org/T184140) (owner: 10Awight)
[21:39:56] <awight>	 https://en.wikipedia.org/wiki/Tears_in_rain_monologue
[21:39:56] <halfak>	 I said I'd kill it at 1700 UTC in the task but I'm just getting to it now. 
[21:40:18] <halfak>	 awight, how apt
[21:40:24] <awight>	 {{done}}
[21:40:24] <AsimovBot>	 How efficient, awight!
[21:40:38] * awight basks in newfound goodness
[21:41:37] <halfak>	 5
[21:41:39] <halfak>	 4
[21:41:41] <halfak>	 3
[21:41:43] <halfak>	 2
[21:41:45] <halfak>	 1
[21:41:47] <halfak>	 0.5
[21:41:49] <halfak>	 0.25
[21:41:55] <codezee>	 0.125
[21:42:00] <awight>	 Ship’s self-destruct sequence aborted.
[21:42:09] <awight>	 DIV_ZERO_ERROR
[21:42:33] <halfak>	 :) 
[21:42:53] <codezee>	 halfak: btw the word2vec binary is in a google drive link and not CLI friendly I remember last time I had to upload it from my system which I had downloaded back in university...
[21:43:05] <codezee>	 any chance we can upload it somewhere with a public access?
[21:43:10] <halfak>	 codezee, we could re-host it. 
[21:43:14] <codezee>	 for these cases of moving around
[21:43:35] <halfak>	 Yeah.  Make a task and show me what exactly we should re-host and I'll get it online. 
[21:43:42] <codezee>	 ok
[21:43:56] <halfak>	 Long live ores-misc-01 (again)
[21:45:48] * halfak waits for ores-misc-01 to accept his public key
[21:46:27] <wikibugs>	 10Scoring-platform-team (Current): Host Google-News-word2vec.bin publicly - https://phabricator.wikimedia.org/T185147#3907561 (10Sumit)
[21:47:36] <wikibugs>	 10Scoring-platform-team (Current): Host Google-News-word2vec.bin publicly - https://phabricator.wikimedia.org/T185147#3907572 (10Sumit) The binary *was* on ores-misc-01 which is now nuked. I'll upload it to ores-staging-01 from my system again from where it can be put somewhere public.
[21:48:16] <halfak>	 lol woops
[21:48:19] <halfak>	 I did a count down
[21:50:05] <codezee>	 \me schedules to again setup drafttopic and wordvectors on ores-staging-01 :/
[21:50:17] <awight>	 I’ve found with kids that counting up is infinitely more effective.
[21:50:20] <awight>	 No idea why.
[21:50:36] <codezee>	 *infinitely* being the key... :P
[21:50:45] <halfak>	 lol
[21:50:47] <awight>	 Countably infinite
[21:50:55] <awight>	 and i *will*
[21:50:57] <Zppix>	 Is redis on wmflabs gone for life?
[21:51:06] <halfak>	 1, 2, 3, ..., I N F I N I T Y 
[21:51:24] <halfak>	 Zppix, nope.  Still there
[21:51:29] <halfak>	 I'll have a server list soon. 
[21:51:32] <Zppix>	 Ok
[21:53:40] <wikibugs>	 10Scoring-platform-team (Current), 10ORES: Update docs, monitoring, etc. for new labs servers - https://phabricator.wikimedia.org/T185148#3907580 (10Halfak)
[21:53:40] <halfak>	 https://phabricator.wikimedia.org/T185148
[21:53:52] <halfak>	 I'll work on wikitech if you work on icinga, Zppix 
[21:56:10] <wikibugs>	 10Scoring-platform-team (Current), 10ORES: Update docs, monitoring, etc. for new labs servers - https://phabricator.wikimedia.org/T185148#3907593 (10Halfak) https://wikitech.wikimedia.org/w/index.php?title=ORES/Instances&diff=1780854&oldid=1778638
[21:56:17] <wikibugs>	 10Scoring-platform-team (Current), 10ORES: Update docs, monitoring, etc. for new labs servers - https://phabricator.wikimedia.org/T185148#3907594 (10Halfak)
[21:58:06] <Zppix>	 halfak: yeah
[21:58:38] <Zppix>	 paladox: perhaps we need to find a way for easier server addition for here?
[22:07:36] <wikibugs>	 10Scoring-platform-team (Current), 10ORES: Update docs, monitoring, etc. for new labs servers - https://phabricator.wikimedia.org/T185148#3907646 (10Halfak)
[22:07:46] <wikibugs>	 10Scoring-platform-team (Current), 10ORES: Update docs, monitoring, etc. for new labs servers - https://phabricator.wikimedia.org/T185148#3907580 (10Halfak) Just waiting on icinga now.
[22:08:02] <awight>	 gtg
[22:08:41] <wikibugs>	 10Scoring-platform-team (Current), 10ORES: Update docs, monitoring, etc. for new labs servers - https://phabricator.wikimedia.org/T185148#3907580 (10Zppix) {{Working}}
[22:08:42] <AsimovBot>	 10[1] 10https://meta.wikimedia.org/wiki/Template:Working
[22:09:09] <Zppix>	 halfak: well i would do it if they didnt kill wikitech
[22:10:00] <halfak>	 Who killed wikitech?
[22:10:06] <halfak>	 o/ awight|afk 
[22:10:35] <Zppix>	 halfak: andrew
[22:33:36] <wikibugs>	 (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/404857 (owner: 10L10n-bot)
[22:39:08] <wikibugs>	 10Scoring-platform-team (Current), 10ORES: Convert ores-misc-01 to stretch - https://phabricator.wikimedia.org/T184766#3894669 (10Halfak) This is now done.
[23:26:33] <icinga2-wm>	 DOWNTIMEEND - check load on ORES-web05.experimental is WARNING: Could not resolve hostname ores-web-05.ores.eqiad.wmflabs: Name or service not known zppix https://gerrit.wikimedia.org/r/404584
[23:26:33] <icinga2-wm>	 DOWNTIMEEND - ssh on ORES-web05.experimental is UNKNOWN: Usage:check_ssh [-4 zppix https://gerrit.wikimedia.org/r/404584
[23:26:34] <icinga2-wm>	 DOWNTIMEEND - check disk on ORES-web05.experimental is WARNING: Could not resolve hostname ores-web-05.ores.eqiad.wmflabs: Name or service not known zppix https://gerrit.wikimedia.org/r/404584
[23:26:34] <icinga2-wm>	 DOWNTIMEEND - puppet on ORES-web05.experimental is WARNING: Could not resolve hostname ores-web-05.ores.eqiad.wmflabs: Name or service not known zppix https://gerrit.wikimedia.org/r/404584
[23:26:35] <icinga2-wm>	 DOWNTIMEEND - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4 zppix https://gerrit.wikimedia.org/r/404584
[23:26:36] <icinga2-wm>	 DOWNTIMEEND - ping4 on ORES-web05.experimental is UNKNOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4 zppix https://gerrit.wikimedia.org/r/404584
[23:26:36] <icinga2-wm>	 DOWNTIMEEND - check users on ORES-web05.experimental is WARNING: Could not resolve hostname ores-web-05.ores.eqiad.wmflabs: Name or service not known zppix https://gerrit.wikimedia.org/r/404584
[23:46:16] <wikibugs>	 (03PS1) 10Halfak: Bumps models and requirements for conversion to Debian Stretch. [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/404886
[23:46:32] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[23:47:02] <wikibugs>	 (03PS2) 10Halfak: Bumps models and requirements for conversion to Debian Stretch. [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/404886 (https://phabricator.wikimedia.org/T182799)
[23:47:21] <wikibugs>	 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Make sure ORES is compatible with stretch - https://phabricator.wikimedia.org/T182799#3908155 (10Halfak)
[23:49:17] <Zppix>	 Well the patch is merger
[23:49:24] <Zppix>	 I dont have ssh access to deploy
[23:49:33] <Zppix>	 And i swore we had puppet auto deploying
[23:49:41] <Zppix>	 So paladox where is you