[01:47:36] * yuvipanda waves at people [07:27:35] akosiaris: hey, I'm in Istanbul ariport now waiting for the next flight. In the mean time I might be able to deploy ores-config into labs and prod so you can merge this patch: https://gerrit.wikimedia.org/r/#/c/292516/ [07:27:46] does it sound like a plan? [07:35:05] went to staging, worked just fine [08:05:13] 06Revision-Scoring-As-A-Service, 10ORES: Fix race condition between scap and puppet - https://phabricator.wikimedia.org/T137524#2370794 (10Ladsgroup) [08:07:53] 06Revision-Scoring-As-A-Service, 10ORES: Fix race condition between scap and puppet - https://phabricator.wikimedia.org/T137524#2370808 (10Ladsgroup) [08:07:55] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES, 10ORES: Announce the first deployment of the ORES extension (fawiki) - https://phabricator.wikimedia.org/T137436#2370807 (10Ladsgroup) [08:57:19] 10Revision-Scoring-As-A-Service-Backlog, 05Wikimania-Hackathon-2016: A training session on supporting ORES in more languages in Wikimania Hackathon - https://phabricator.wikimedia.org/T134628#2370954 (10Elitre) [16:07:26] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES: Deploy ORES extension in fawiki - https://phabricator.wikimedia.org/T130211#2372074 (10Halfak) [16:08:18] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES, 10ORES: Announce the first deployment of the ORES extension (fawiki) - https://phabricator.wikimedia.org/T137436#2372078 (10Halfak) [16:18:58] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Document ORES scap3 deploy process on wikitech - https://phabricator.wikimedia.org/T137570#2372131 (10Halfak) [16:20:10] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Document ORES scap3 deploy process on wikitech - https://phabricator.wikimedia.org/T137570#2372145 (10Ladsgroup) a:03Ladsgroup [16:26:45] 06Revision-Scoring-As-A-Service, 10ORES: Document ORES scap3 deploy process on wikitech - https://phabricator.wikimedia.org/T137570#2372154 (10Ladsgroup) [16:27:34] 06Revision-Scoring-As-A-Service, 10DBA, 10MediaWiki-extensions-ORES: Review ORES extension for data leaks - https://phabricator.wikimedia.org/T137567#2372170 (10Ladsgroup) [16:28:44] 06Revision-Scoring-As-A-Service, 10DBA, 10MediaWiki-extensions-ORES: Review ORES extension for data leaks - https://phabricator.wikimedia.org/T137567#2372053 (10Ladsgroup) Oh I need to note, these schema changes are being invoked using [[https://github.com/wikimedia/mediawiki-extensions-ORES/blob/master/incl... [16:40:48] 06Revision-Scoring-As-A-Service, 10DBA, 10MediaWiki-extensions-ORES: Review ORES extension for data leaks - https://phabricator.wikimedia.org/T137567#2372228 (10jcrespo) What is the privacy requirements of these score tables? Are they all public? Is their privacy dependent on the revision they refer to? What... [16:55:19] 06Revision-Scoring-As-A-Service, 10DBA, 10MediaWiki-extensions-ORES: Review ORES extension for data leaks - https://phabricator.wikimedia.org/T137567#2372283 (10Ladsgroup) All of these data came from a public endpoint (the ORES service). So no private dependencies. We do not support private wikis (yet) becau... [17:05:02] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES: Deploy ORES extension in fawiki - https://phabricator.wikimedia.org/T130211#2372328 (10jcrespo) [17:10:03] 06Revision-Scoring-As-A-Service, 10DBA, 10MediaWiki-extensions-ORES: Review ORES extension for data leaks - https://phabricator.wikimedia.org/T137567#2372370 (10Ladsgroup) Sure, Thanks :) [20:05:35] Well... Just noticed some ORES downtime. [20:05:45] Looks like we aren't connecting to redis appropriately [20:06:35] is the ORES service in production now? [20:06:54] Yes. but labs is still the primary endpoint until we finish testing [20:07:37] mkay. I saw bugs about the MW extension deployment moving forward so I got excited ^.^ [20:08:19] Yeah. it will to fawiki soon :) [20:14:51] OK looks like some puppet changes have made the labs installation broken for 6+ hours. [20:14:53] Geeeze [20:14:58] akosiaris, around? [20:39:20] 06Revision-Scoring-As-A-Service: Find out why recent 503 did not result in paging for ORES - https://phabricator.wikimedia.org/T137592#2372838 (10Halfak) [20:39:58] 06Revision-Scoring-As-A-Service: Find out why recent 503 did not result in paging for ORES - https://phabricator.wikimedia.org/T137592#2372851 (10Halfak) See https://wikitech.wikimedia.org/wiki/Incident_documentation/20160610-ORES#Actionables [20:40:17] OK back online and I filed a stub of documentation [20:40:20] https://wikitech.wikimedia.org/wiki/Incident_documentation/20160610-ORES [20:40:23] Now I need to run [21:02:47] halfak: around? [21:03:53] Sort of [21:03:56] Amir1, ^ [21:04:00] What's up? [21:04:09] check telegram [21:13:00] halfak: the way to fix it properly is 1- remove those puppet config from role::worker [21:13:10] 2- move the config parts to ores::base [21:13:52] 3- add a dir definition for /etc/ores [21:14:11] Sounds good [21:14:16] It's super strange that it works in prod but gives us error in labs [21:14:24] (I mean the third part) [21:14:37] halfak: tomorrow in the venue I will finish this [21:14:44] Sounds good to me [21:16:07] I even have this crazy idea, the labs setup is unstable, move some important tools to prod until I fix this [21:16:17] and then bring them back [21:16:34] and after finishing tests we again bring them to prd [21:16:36] *prod [21:17:33] but I can't think clearly atm [21:18:15] I don't think that is a good idea. Labs is stable. We just need to stop kicking it over. [21:18:36] "Labs is stable" words have never been spoken before [21:18:53] We need freedom in "prod" right now to deploy things without serious worry. [21:19:50] hmm, We don't make any changes in puppet until I fix it [21:19:57] +1 [21:20:21] I'm going to get in the car for travel in 10 mins [21:20:37] because it's working because it's badly broken, if someone tries to fix some of it, the whole thing fails [21:20:48] take care halfak [21:21:13] I'm super sorry for what happened. I was just trying to make things more reobust [21:21:16] *robust [21:21:31] Try not to let it get to you. :) o/ [21:21:39] I can walk naked in streets of Berlin and people calling "shame" [21:21:56] (if you have seen GoT, you would know what I mean)