[13:58:51] <halfak>	 o/ schana 
[13:59:00] <halfak>	 GOing to be a little late to meeting.  Need to reboot quickly. 
[13:59:04] <schana>	 no worries
[14:25:30] <wikibugs>	 06Revision-Scoring-As-A-Service, 10Wikilabels: Load full campaign for WikiEd article topic - https://phabricator.wikimedia.org/T138626#2405739 (10Halfak) This is now done. :)
[14:30:54] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES, 06Operations, 10Traffic, 07HTTPS: https://ores.wikimedia.org redirects me to HTTP when I don't include a trailing slash - https://phabricator.wikimedia.org/T138682#2407209 (10Halfak) a:03Ladsgroup
[14:43:02] <Amir1>	 halfak: o/
[14:51:06] <halfak>	 o/ Amir1 
[14:51:19] <halfak>	 Are you back from Wikimania?
[14:53:11] <Amir1>	 halfak: yeah, visiting my parents though 
[14:53:21] <Amir1>	 so not 100% but 70%
[14:53:39] <halfak>	 kk 
[14:53:48] <halfak>	 We had a monster of a week!
[14:53:57] <Amir1>	 https://etherpad.wikimedia.org/p/revscoring
[14:53:58] <halfak>	 schana, I forgot that we had make-up sync today 
[14:54:11] <Amir1>	 my to-do is tripled now
[14:54:26] <schana>	 yeah, halfak, I'll be there
[14:54:36] <halfak>	 (when I said "see you on IRC", I should have said "see you in an hour")
[14:54:37] <halfak>	 :) 
[14:54:53] <schana>	 I figured as much :D
[16:02:01] <Amir1>	 halfak: ping
[16:57:12] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: ORES prod-deploy fix and deployment - https://phabricator.wikimedia.org/T138863#2412371 (10Halfak)
[16:58:27] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Announce deployment of reverted model to nowiki - https://phabricator.wikimedia.org/T138625#2405698 (10Halfak)
[16:59:01] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Announce deployment of reverted model to svwiki community - https://phabricator.wikimedia.org/T138624#2405685 (10Halfak) @Johan, could you help us with this announcement?
[16:59:28] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Announce deployment of wp10 models to ruwiki community - https://phabricator.wikimedia.org/T138623#2405672 (10Halfak) @Putnik, could you help us with this announcement?
[16:59:59] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Announce deployment of wp10 models to ruwiki community - https://phabricator.wikimedia.org/T138623#2412394 (10Halfak) See https://ores.wmflabs.org/v2/scores/ruwiki/wp10/?model_info for model info and endpoint.
[17:02:13] <wikibugs>	 06Revision-Scoring-As-A-Service, 06Design-Research, 10MediaWiki-extensions-ORES: Heuristic review of ORES review tool - https://phabricator.wikimedia.org/T138249#2412398 (10Halfak)
[17:10:47] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: Convert ores-wikimedia-config to ores-wmflabs-deploy - https://phabricator.wikimedia.org/T138253#2412427 (10Halfak)
[17:12:52] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: Convert ores-wikimedia-config to ores-wmflabs-deploy - https://phabricator.wikimedia.org/T138253#2412430 (10Halfak) This is done and deployed.  See https://github.com/wiki-ai/ores-wmflabs-deploy/pull/64
[17:15:58] <schana>	 goodnight all
[17:16:21] <halfak>	 o/ schana 
[17:16:32] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 05Wikimania-Hackathon-2016: A training session on supporting ORES in more languages in Wikimania Hackathon - https://phabricator.wikimedia.org/T134628#2412438 (10Ladsgroup) https://www.youtube.com/watch?v=WnM5Ys2-Cb8
[17:16:47] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES, 05Wikimania-Hackathon-2016: A training session on supporting ORES in more languages in Wikimania Hackathon - https://phabricator.wikimedia.org/T134628#2412439 (10Ladsgroup)
[17:31:46] <halfak>	 o/ sabya 
[17:31:58] <sabya>	 o/ halfak 
[17:32:18] <sabya>	 talking after a long time :)
[17:32:39] <halfak>	 Yeah.  Was just thinking that I should reach out and see how you are doin. 
[17:34:09] <sabya>	 for sometime I started contributing to https://phabricator.wikimedia.org/project/view/1391/
[17:34:45] <halfak>	 Oh!  Gotcha. 
[17:36:09] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: Convert ores-wikimedia-config to ores-wmflabs-deploy - https://phabricator.wikimedia.org/T138253#2412479 (10Halfak) a:03Halfak
[17:38:11] <sabya>	 I'll  get back to the HashingVectrorizer and conclude it.
[17:40:00] <halfak>	 That would be great.  It's not a huge rush, but we really value your work :) 
[17:40:42] <sabya>	 halfak: thanks :) 
[17:44:11] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog: Report on projects covered at HPI hackathon - https://phabricator.wikimedia.org/T137802#2412492 (10Ladsgroup) @Lydia_Pintscher, hey is there something like that?
[18:06:38] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog: Report on projects covered at HPI hackathon - https://phabricator.wikimedia.org/T137802#2412545 (10Lydia_Pintscher) http://blog.wikimedia.de/2016/06/15/wikidata-beim-hackathon-hackhpi-in-potsdam/ We invited them to the office. Still working on the logistics.
[19:13:43] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES, 10revscoring: Score multiple models with the same cached dependencies - https://phabricator.wikimedia.org/T134606#2412676 (10Halfak)
[20:08:23] <halfak>	 Amir1, oh!  I forgot to mention my migration plan for using ores.applications.* rather than ores_*.py
[20:09:05] <Amir1>	 I saw and I merged some your patches for labs
[20:09:12] <halfak>	 So I replaced the ones in https://github.com/wiki-ai/ores-wmflabs-deploy/blob/master/ores_celery.py and https://github.com/wiki-ai/ores-wmflabs-deploy/blob/master/ores_wsgi.py
[20:09:23] <halfak>	 So now in labs, we can switch without interruption
[20:09:31] <Amir1>	 It would be great to do it for prod
[20:11:11] <halfak>	 Working on it now :) 
[20:13:00] <Amir1>	 https://en.wikipedia.org/wiki/Atat%C3%BCrk_Airport_attack
[20:13:08] <Amir1>	 I was there seven times
[20:14:00] <Amir1>	 I'm worried about Wikimedians going through Istanbul
[20:33:36] <halfak>	 Amir1, https://gerrit.wikimedia.org/r/#/c/296456/
[20:33:41] * halfak clicks on wiki link
[20:34:22] <halfak>	 Scary stuff. 
[20:35:43] <Amir1>	 {{merged}}
[20:35:44] <AsimovBot>	 10[1] 04https://meta.wikimedia.org/wiki/Template:merged
[20:36:00] <halfak>	 Amir1, do you have time to walk me through a deploy?
[20:36:35] <Amir1>	 halfak: yeah, first we should deploy it in beta
[20:36:53] <Amir1>	 do you have access for beta cluster?
[20:37:18] * halfak connects to deployment-tin
[20:38:02] <halfak>	 Am in "/srv/deployment/ores/deploy" 
[20:38:09] <Amir1>	 nice
[20:38:12] <Amir1>	 git pull
[20:38:16] <halfak>	 So I need to pull and then "git submodule update --init", right?
[20:38:34] <Amir1>	 probably a "git pull" is enough 
[20:38:37] <halfak>	 "please commit your changes" re. ores-web-labs and ores-worker-labs
[20:39:00] <Amir1>	 oh, it seems we made some changes without commiting
[20:39:02] <Amir1>	 git diff?
[20:39:20] <halfak>	 -deployment-ores-web.deployment-prep.eqiad.wmflabs
[20:39:20] <halfak>	 +deployment-sca03.deployment-prep.eqiad.wmflabs
[20:39:49] <halfak>	 Some dirty changes to submodule/ores and submodule/wheels too
[20:40:01] <Amir1>	 yeah, that was for test
[20:40:31] <Amir1>	 do "git reset --hard origin/master"
[20:40:37] <halfak>	 OK will do
[20:41:56] <Amir1>	 It seems that submodules are dirty too
[20:42:11] <halfak>	 Yeah.  Just fixed 'em
[20:42:15] <halfak>	 Seems we're good to go
[20:42:28] <Amir1>	 nice
[20:42:46] <Amir1>	 wheels is not okay
[20:42:49] <Amir1>	 git status
[20:44:44] <halfak>	 woah
[20:45:11] <Amir1>	 git reset --hard origin/master is magic
[20:45:20] <halfak>	 yes it is :) 
[20:45:50] <halfak>	 OK looks like I get a key error. 
[20:45:52] <halfak>	 Pasting
[20:46:17] <halfak>	 http://pastebin.ca/3652633
[20:47:00] <Amir1>	 it seems you don't have access 
[20:47:08] <Amir1>	 as a deployer
[20:47:13] <Amir1>	 let me ask to fix
[20:47:20] <halfak>	 OK.  Is this within the beta project?
[20:51:32] <Amir1>	 halfak: yeah
[20:51:50] <Amir1>	 one thing is that try to login with your account in sca03
[20:51:55] <Amir1>	 deployment-sca03
[20:51:58] <Amir1>	 it might solve issues
[20:52:02] <halfak>	 OK.  Maybe we should have you continue the deploy in the meantime.  You could just give me a play-by-play. 
[20:52:08] * halfak tries
[20:52:54] <halfak>	 Yup.  Failed. 
[20:53:04] <halfak>	 Maybe I need to use ssh-agent or something?
[20:53:36] <halfak>	 Yeah.  Sure enough I can't ssh there from deployment-tin
[20:53:38] <halfak>	 So...
[20:53:39] <halfak>	 hmm
[20:53:49] * halfak has not had luck with ssh-agent
[20:53:58] <Amir1>	 it's super complicated 
[20:54:12] <Amir1>	 and I have issues all the time
[20:56:01] <halfak>	 Hmm... I start up my ssh-agent
[20:56:05] <halfak>	 I ssh-add my key
[20:56:10] <halfak>	 And then it still doesn't work. 
[20:59:45] * yuvipanda pops up
[21:01:28] <Amir1>	 halfak: keyholder?
[21:01:47] <yuvipanda>	 ah, this isn't an ssh access labs issue but a scap3 issue
[21:01:49] * yuvipanda disappears
[21:02:13] <Amir1>	 https://wikitech.wikimedia.org/wiki/Keyholder
[21:10:48] <Amir1>	 hey
[21:10:52] <Amir1>	 what happened?
[21:11:09] <halfak>	 I added "ForwardAgent yes" to my .ssh/config
[21:11:25] <halfak>	 Now waiting on 0%
[21:11:49] <halfak>	 oooh.  Am config_deploy and then promote and restart_service
[21:12:09] <halfak>	 Done!
[21:12:32] <halfak>	 http://ores-beta.wmflabs.org/v2/scores/ruwiki/wp10/23456789
[21:13:48] <halfak>	 OK.  Now to log in releng.  "!log deploying ores 6979a98"
[21:13:54] <halfak>	 Does that look right to you, Amir1?
[21:14:10] <Amir1>	 yeah
[21:14:19] <halfak>	 Not ores-beta?
[21:14:37] <Amir1>	 we don't have ores-beta project
[21:14:46] <Amir1>	 so in releng
[21:15:10] <Amir1>	 halfak: first, don't do it in your machine that you connect to prod, according to L3 
[21:15:26] <halfak>	 kk
[21:15:31] <Amir1>	 secondly let's test if it's working properly :)
[21:15:57] <halfak>	 Have hit all of the routes
[21:16:02] <halfak>	 *new routes
[21:16:16] <halfak>	 http://ores-beta.wmflabs.org/v2/scores/ruwiki/wp10/23456789?features
[21:16:17] <halfak>	 E.g.
[21:16:26] <Amir1>	 nice
[21:16:29] <Amir1>	 one thing
[21:16:36] <halfak>	 checked /ui/ and /v2/
[21:16:38] <Amir1>	 ores-beta.wmflabs.org
[21:16:50] <Amir1>	 in view source it still loads from wikimedia ui in labs
[21:17:05] <Amir1>	 we should fix them, or you fixed and it's not working
[21:17:09] <halfak>	 Oh!  The homepage still does
[21:17:20] <halfak>	 I fixed /ui, /v1 and /v2
[21:17:21] <Amir1>	 the GUI is okay?
[21:17:31] <Amir1>	 okay nice
[21:17:38] <halfak>	 Sorry.  Forgot about index. 
[21:17:43] <halfak>	 Can fix that now if you like
[21:18:00] <Amir1>	 do you want to fix the home and have a quick deploy before prod?
[21:18:03] <Amir1>	 it would be easy 
[21:18:30] <halfak>	 Sure. 
[21:18:33] <halfak>	 Working on it now
[21:18:35] <Amir1>	 I promise to merge everything for you :D
[21:18:48] <Amir1>	 review
[21:18:52] <halfak>	 :P 
[21:18:57] <Amir1>	 I should have said review :D
[21:19:29] <Amir1>	 http://ores-beta.wmflabs.org/wikimedia-ui-static/MW/elements.html
[21:19:30] <Amir1>	 nice
[21:19:37] <halfak>	 \o/
[21:19:46] <halfak>	 Was a bit of a pain.  Happy to have that all together :) 
[21:19:54] <halfak>	 And we can use the same code in other projects now too :) 
[21:21:00] <halfak>	 https://github.com/wiki-ai/ores/pull/153
[21:21:01] <halfak>	 Amir1, ^ 
[21:21:38] <Amir1>	 okay, now the patch in gerrit 
[21:21:39] <Amir1>	 :D
[21:23:00] <halfak>	 Hmm... mirror is slow?
[21:23:55] <Amir1>	 it might be
[21:24:13] <Amir1>	 in the mean time let's enjoy yuvipanda in the wikimania: https://commons.wikimedia.org/wiki/File:YuviPanda-IMG_3002.jpg
[21:24:33] <halfak>	 Awesome!
[21:25:11] <Amir1>	 and existence of this category: https://commons.wikimedia.org/wiki/Category:Asaf_Bartov_with_a_tie
[21:25:34] <halfak>	 lol
[21:25:52] <halfak>	 This is one of my go-tos
[21:25:53] <halfak>	 https://upload.wikimedia.org/wikipedia/commons/4/4d/Heringsschwarm.gif
[21:27:41] <halfak>	 Amir1, https://gerrit.wikimedia.org/r/#/c/296465/
[21:28:08] <Amir1>	 merged too
[21:30:29] <halfak>	 OK.  HERE WE GO!
[21:31:23] <halfak>	 gogogogoogo
[21:31:31] <halfak>	 fetch...
[21:32:00] <Amir1>	 :D
[21:33:07] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: ORES prod-deploy fix and deployment - https://phabricator.wikimedia.org/T138863#2413030 (10Halfak) https://gerrit.wikimedia.org/r/#/c/296456/ -- Fix repo  https://github.com/wiki-ai/ores/pull/153 -- Fix local asset issue on homepage
[21:34:22] <Amir1>	 halfak: tell me when the deployment is done
[21:34:33] <halfak>	 Done on beta now
[21:34:48] <Amir1>	 nice
[21:34:54] <Amir1>	 everything looks okay
[21:34:57] <Amir1>	 let's go on prod
[21:35:00] <Amir1>	 are you ready?
[21:35:06] <halfak>	 I think so, yeah
[21:35:16] <Amir1>	 awesome
[21:35:23] <Amir1>	 tin.eqiad.wmnet
[21:35:42] <Amir1>	 then go to /srv/deployment/ores/deploy
[21:35:49] <Amir1>	 git pull and all things
[21:36:35] <Amir1>	 then edit scap/ores and scap/ores-worker and remove all lines except scb2001.codfw.wmnet
[21:37:16] <halfak>	 Keep scb2002.codfw.wmnet?
[21:38:50] <Amir1>	 yup
[21:39:19] <halfak>	 OK
[21:40:18] <halfak>	 In the instructions, I see "remove all nodes except scb2001.codfw.wmnet"
[21:40:34] <halfak>	 Should I change that to include scb2002?
[21:40:34] <Amir1>	 yeah
[21:40:44] <Amir1>	 one of codfw is enough 
[21:40:52] <halfak>	 Oh.  So do not keep scb2002
[21:40:53] <Amir1>	 doesn't matter which one
[21:41:09] <Amir1>	 yup, only one
[21:42:16] <halfak>	 Deploying to scb2001.codfw
[21:42:43] <Amir1>	 nice
[21:42:57] <Amir1>	 log it in #wikimedia-operations 
[21:44:08] <halfak>	 Hmmm curl isn't working
[21:44:10] <halfak>	 just hangs
[21:44:54] <Amir1>	 let me check
[21:45:08] <halfak>	 I guess the workers are down
[21:46:03] <halfak>	 celery is not running
[21:47:36] <Amir1>	 halfak: curl 0.0.0.0:8081 is working for me in codfw
[21:47:48] <halfak>	 Amir1, try to make it score
[21:48:02] <halfak>	 curl 0.0.0.0:8081/v2/scores/testwiki/reverted/34567
[21:48:14] <Amir1>	 halfak: I don't think that would work
[21:48:20] <halfak>	 Well
[21:48:22] <halfak>	 It shoudl work
[21:48:24] <Amir1>	 because celery goes to redis
[21:48:29] <halfak>	 Indeed. 
[21:48:34] <Amir1>	 redis sends it to other nodes
[21:48:36] <halfak>	 So, we ought not expect it to work?
[21:48:44] <Amir1>	 I'm not sure actually
[21:48:53] <Amir1>	 but that's my "feeling"
[21:49:04] <halfak>	 Yeah... if there's no redis, then the celery workers should fail to start
[21:49:20] <halfak>	 Seems like our deployment should "fail" if the workers can't start up. 
[21:49:42] <Amir1>	 since https://ores-beta.wmflabs.org/scores/wikidatawiki/reverted/7654345 works fine I guess it's okay
[21:49:46] <Amir1>	 https://ores.wikimedia.org/v2/scores/enwiki/?models=damaging&revids=724030089
[21:49:56] <Amir1>	 also this which means the whole system is up
[21:50:22] <halfak>	 OK.  Yeah.  Otherwise seems OK
[21:50:50] <Amir1>	 we should have a system to test that but my guess is that beta is the best place for now
[21:50:56] <halfak>	 Makes sense. 
[21:51:00] <halfak>	 Ready for prod then. 
[21:51:04] <Amir1>	 given that everything is identical 
[21:51:07] <halfak>	 So just revert ores/ores-worker?
[21:51:19] <Amir1>	 you can do  git reset
[21:51:28] <Amir1>	 "git reset --hard origin/master
[21:51:58] <halfak>	 OK>  Here we go!
[21:52:04] <halfak>	 fetch
[21:54:23] <halfak>	 promote and restart
[21:55:09] <Amir1>	 I'm monitoring
[21:55:22] <Amir1>	 service looks okay from the outside
[21:56:07] * halfak watches 50% of requests find the ruwiki/wp10 model
[21:56:31] <halfak>	 Deployed!
[21:56:36] <Amir1>	 nice!
[21:57:00] <Amir1>	 it's up
[21:57:03] <Amir1>	 yay
[21:57:09] <halfak>	 All seems to be golden
[21:57:12] <halfak>	 \o/ 
[21:57:13] <halfak>	 Success
[21:57:16] <halfak>	 Time to log
[21:57:35] <halfak>	 So "!log deploying ores beec291" in -operations?
[21:57:59] <Amir1>	 yeah
[21:58:25] <Amir1>	 I think it's better to do it beforehand
[21:59:01] <Amir1>	 one thing halfak, before deployment keep the hash of current revision so you would know which revision you need to rollback to
[21:59:02] <halfak>	 Oh.  Sure.  Will do next time
[21:59:16] <Amir1>	 in case of a messy deploy
[21:59:35] <halfak>	 Yeah.  I saw that in the Ores/Deploy instructions and had it. 
[21:59:57] <halfak>	 So I think we're at victory!
[22:00:02] <halfak>	 Thanks for the help 
[22:00:29] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: ORES prod-deploy fix and deployment - https://phabricator.wikimedia.org/T138863#2412371 (10Halfak) Now deployed!
[22:00:39] <Amir1>	 thank you for deploying
[22:00:59] <halfak>	 Now we should be ready to do the puppet changes for applications, but I want to do some more testing on that first
[22:01:13] <halfak>	 I've got to go run to the vet to get some meds for the ferrets
[22:01:21] <halfak>	 I'll be AFK for ~ an hour
[22:01:58] <Amir1>	 halfak: I'm writing the weekly update
[22:02:07] <Amir1>	 can you take a look once it's done?
[22:02:12] <Amir1>	 even in telegram
[22:02:19] <halfak>	 Awesome.  I can promise to be back in an hour for it. 
[22:02:26] <halfak>	 Maybe even sooner via my mobile phone. 
[22:02:26] <Amir1>	 okay
[22:02:31] <halfak>	 Ping me on telegram when you are done. 
[22:02:33] <halfak>	 o/
[22:04:06] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: Monitor worker nodes in icigna - https://phabricator.wikimedia.org/T138882#2413086 (10Ladsgroup)
[22:04:29] <Amir1>	 okay sure
[22:04:30] <Amir1>	 o/
[22:14:00] <wikibugs>	 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Train `reverted` model for cswiki - https://phabricator.wikimedia.org/T138885#2413141 (10Ladsgroup)
[22:17:23] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES, 06Operations, 10Traffic, 07HTTPS: https://ores.wikimedia.org redirects me to HTTP when I don't include a trailing slash - https://phabricator.wikimedia.org/T138682#2413170 (10Ladsgroup) ``` amir@amir-Lenovo-ideapad-300-15IBR:~/workspace/numerical methods$ curl "ht...
[22:51:20] <Amir1>	 halfak: sent
[22:52:04] <Amir1>	 SPAMS COMING
[22:52:34] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: Monitor worker nodes in icigna - https://phabricator.wikimedia.org/T138882#2413245 (10Ladsgroup) 05Open>03Resolved
[22:52:36] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: ORES prod-deploy fix and deployment - https://phabricator.wikimedia.org/T138863#2413246 (10Ladsgroup) 05Open>03Resolved
[22:52:38] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES, 06Operations, 10Traffic, 07HTTPS: https://ores.wikimedia.org redirects me to HTTP when I don't include a trailing slash - https://phabricator.wikimedia.org/T138682#2413248 (10Ladsgroup) 05Open>03Resolved
[22:52:40] <wikibugs>	 06Revision-Scoring-As-A-Service, 10Wikilabels: Load full campaign for WikiEd article topic - https://phabricator.wikimedia.org/T138626#2413247 (10Ladsgroup) 05Open>03Resolved
[22:52:42] <wikibugs>	 06Revision-Scoring-As-A-Service, 10Wikilabels, 05Wikimania-Hackathon-2016: Cannot read property 'badrevids' of undefined - https://phabricator.wikimedia.org/T138576#2413249 (10Ladsgroup) 05Open>03Resolved
[22:52:45] <wikibugs>	 06Revision-Scoring-As-A-Service, 10Beta-Cluster-Infrastructure, 10ORES, 13Patch-For-Review, 05Wikimania-Hackathon-2016: Resurrect ores-beta with production roles - https://phabricator.wikimedia.org/T138445#2413250 (10Ladsgroup) 05Open>03Resolved
[22:52:47] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES, 10bwds, 10revscoring: Create documentation for BWDS review process - https://phabricator.wikimedia.org/T138419#2413251 (10Ladsgroup) 05Open>03Resolved
[22:52:49] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: Convert ores-wikimedia-config to ores-wmflabs-deploy - https://phabricator.wikimedia.org/T138253#2413253 (10Ladsgroup) 05Open>03Resolved
[22:52:52] <wikibugs>	 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10ORES, 05Wikimania-Hackathon-2016: Present Wikidata AI work at Wikimania - https://phabricator.wikimedia.org/T138351#2413252 (10Ladsgroup) 05Open>03Resolved
[22:52:55] <wikibugs>	 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10ORES: Announce ORES prod (technical lists) - https://phabricator.wikimedia.org/T138252#2413254 (10Ladsgroup) 05Open>03Resolved
[22:52:58] <wikibugs>	 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10ORES: [Epic] Complete outstanding tasks for ORES extension deployments - https://phabricator.wikimedia.org/T138251#2413256 (10Ladsgroup)
[22:53:00] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: Document safe method to deploy ORES in prod - https://phabricator.wikimedia.org/T138234#2413255 (10Ladsgroup) 05Open>03Resolved
[22:53:03] <wikibugs>	 06Revision-Scoring-As-A-Service, 10EventBus, 06Services, 15User-mobrovac, 05Wikimania-Hackathon-2016: [Discuss] Switch from precached utility to Change Propagation - https://phabricator.wikimedia.org/T138090#2413257 (10Ladsgroup) 05Open>03Resolved
[22:53:06] <wikibugs>	 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10ORES: [Epic] Complete outstanding tasks for ORES extension deployments - https://phabricator.wikimedia.org/T138251#2394761 (10Ladsgroup)
[22:53:08] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: Add 'nonbot_edit' event to precached - https://phabricator.wikimedia.org/T137967#2413258 (10Ladsgroup) 05Open>03Resolved
[22:53:10] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: New ORES deployment - https://phabricator.wikimedia.org/T137828#2413260 (10Ladsgroup) 05Open>03Resolved
[22:53:12] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES, 05Wikimania-Hackathon-2016: ORES /etc/ores does not get populated on worker nodes - https://phabricator.wikimedia.org/T137813#2413261 (10Ladsgroup) 05Open>03Resolved
[22:53:22] <wikibugs>	 06Revision-Scoring-As-A-Service: Develop training session material for Wikimania Hackathon - https://phabricator.wikimedia.org/T134785#2413267 (10Ladsgroup) 05Open>03Resolved
[22:53:23] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES, 05Wikimania-Hackathon-2016: A training session on supporting ORES in more languages in Wikimania Hackathon - https://phabricator.wikimedia.org/T134628#2413268 (10Ladsgroup) 05Open>03Resolved
[22:53:25] <wikibugs>	 06Revision-Scoring-As-A-Service, 10Wikilabels, 05Wikimania-Hackathon-2016: Edit quality campaign for Swedish Wikipedia - https://phabricator.wikimedia.org/T131451#2413271 (10Ladsgroup) 05Open>03Resolved
[22:53:27] <wikibugs>	 06Revision-Scoring-As-A-Service, 10bwds, 10revscoring, 05Wikimania-Hackathon-2016: Language assets for Czech - https://phabricator.wikimedia.org/T131637#2413270 (10Ladsgroup) 05Open>03Resolved
[23:43:37] <wikibugs>	 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10ORES: [Epic] Complete outstanding tasks for ORES extension deployments - https://phabricator.wikimedia.org/T138251#2394761 (10Ladsgroup) 4 is done 2 more to go!
[23:46:32] * legoktm huggles Amir1
[23:46:54] <legoktm>	 nothing better than my inbox being spammed with a bunch of bugs being resolved <3
[23:47:21] <Amir1>	 legoktm: :D thanks :)
[23:47:25] <Amir1>	 hugs
[23:47:37] <Amir1>	 It was great to find such issues 
[23:47:47] <Amir1>	 which made by you :)