[07:13:24] akosiaris: hey, good morning from WMDE office :) I made a documentation page for deploying ores. Please checkout to see if everything is correct (or if you want to add anything) https://wikitech.wikimedia.org/wiki/Ores/Deployment [07:13:26] thanks :) [07:13:53] 06Revision-Scoring-As-A-Service, 10ORES: Document ORES scap3 deploy process on wikitech - https://phabricator.wikimedia.org/T137570#2372131 (10Ladsgroup) https://wikitech.wikimedia.org/wiki/Ores/Deployment [07:37:01] 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review: Create grafana dashboard for ores.wikimedia.org - https://phabricator.wikimedia.org/T137367#2375338 (10Ladsgroup) I polished it a little, looks better now :) [09:01:59] 06Revision-Scoring-As-A-Service, 10ORES: Fix race condition between scap and puppet - https://phabricator.wikimedia.org/T137524#2375442 (10akosiaris) 05Open>03Resolved a:03akosiaris That was resolved with https://gerrit.wikimedia.org/r/#/c/292516/ [09:02:01] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES, 10ORES: Announce the first deployment of the ORES extension (fawiki) - https://phabricator.wikimedia.org/T137436#2375445 (10akosiaris) [09:47:00] 06Revision-Scoring-As-A-Service: Find out why recent 503 did not result in alerting for ORES - https://phabricator.wikimedia.org/T137592#2375530 (10akosiaris) [16:10:29] Hi, I'm just dropping by to high five Amir1 and team! [16:11:08] also--looks like you need a full scap, maybe? The ORES legend in fawiki/Special:RecentChanges is showing untranslated strings. [16:11:08] awight: high five, still a lot to do [16:11:21] maybe [16:11:53] awight: still a lot needs to be done [16:12:23] I was kind of following the task, and I don't understand how you overcame: > It is not technically possible to deploy this extension before ORES is in production, as production cannot talk to *.wmflabs.org domains. [16:12:52] awight: ores is in prod now [16:12:57] it's about a week now [16:12:59] oh, fantastic! [16:13:12] we finished some testing [16:13:24] ok I'll poke my nose back out. I'm impressed... [16:15:28] thanks, Ops helped a lot :) [16:15:34] mostly akosiaris :) [16:18:12] Ops is like a sleeping dragon, if they're on your side then all the treasure is yours. If not, there be trouble. [16:19:08] :D [16:44:33] akosiaris I do know it'll slow to a crawl without precached. CPU pegs. [16:47:54] yuvipanda: it does however raise the question of why calculating all the recent changes when some of them might not requested soon. And there is the issue of the mediawiki extension anyway submitting requests to ORES [16:49:10] akosiaris: fawiki is very small one in number of requests (considering that it only sends non-bots edits) [16:49:31] so, even double calculating it is bearable [16:49:52] akosiaris the mw extension is jobqueue based so has some latency, and also it isn't calculated for some. there was a ticket about this [16:50:33] https://phabricator.wikimedia.org/T137370 [16:50:45] I actually talked about it with @Halfak, 1- The extension hit the service for only damaging model and not other models which means no precaching for them 2- number of wikis that have the extension enabled stay too low comparing to all wikis the service supports until for at least several months. So disabling the precaching is not an option for now. [16:50:50] akosiaris ^ [16:51:02] yeah I 've seen that [16:51:13] marko is probably on to something with changeprop though [16:51:28] I agree, but I don't think we should block deployment on that [16:51:36] since the problem is more 'where do we put it' [16:51:41] and it should still run once only ideally [16:51:50] so I'd say order is deploy and then move to changeprop [16:51:54] rather than other way around [16:52:28] not sold on that yet [16:52:41] https://grafana.wikimedia.org/dashboard/db/ores [16:52:52] useful dashboard if you need it ^ [17:10:55] 06Revision-Scoring-As-A-Service, 05WMDE-Tech-Communication-Mentoring-And-Events, 03WMDE-Tech-Communication-Q2-2016: Collection of topics for HPI hackathon - https://phabricator.wikimedia.org/T127047#2376714 (10johl) I would like to close this and all blocking tasks, now that the hackathon is over. Anyone aga... [17:31:23] halfak Amir1 as an update, our labs graphite box is a bit saturated, and I'm going to stop sending metrics to it from most instances except those in whitelisted projects (probably tools, ores, deployment-prep). We're now ordering a new box with ssds to replace it [17:31:41] sorry about that [17:31:56] also graphite.wmflabs.org doesn't work but https://graphite.wmflabs.org works fine [17:36:35] yuvipanda: halfak is ooo today [17:36:53] ah ok [17:36:58] it's okay [17:37:19] yuvipanda: the new graphite dashboard for ores in labs is actually grafana.wikimedia.org/dashboard/db/ores-labs [17:37:29] the ores is the one for prod [17:37:30] yeah [17:37:35] it seems to be working, right? [17:41:07] yuvipanda: the prod is working like a charm [17:41:18] the labs always struggles [17:41:31] yeah [17:56:02] Amir1 so labs graphite is struggling but not too bad right? not completely unusable? [17:57:33] yuvipanda: sometime unusable, sometimes partially, actually that was the reason we didn't know about a major down time in our worker nodes [17:57:57] oh I see [18:17:35] yuvipanda: oh, we are actually deploying ores extension in fawiki which at first requests score of lots of edit. That might be the reason behind the CPU issue [18:20:31] from where? prod? [18:23:44] yuvipanda: yeah, prod [18:24:23] cool :D [18:24:26] prod should be fine [18:55:13] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 13Patch-For-Review: ORES extension tables should be able to be populated using maintenance scripts - https://phabricator.wikimedia.org/T123795#2377170 (10Jdforrester-WMF) This was a blocker to this morning's production release, and in the end @thc... [18:58:18] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 13Patch-For-Review: ORES extension tables should be able to be populated using maintenance scripts - https://phabricator.wikimedia.org/T123795#2377181 (10Ladsgroup) Dear @Jdforrester-WMF this task is not making the tables, it's about populating it... [20:10:41] (03PS1) 10Ladsgroup: Change link to beta features. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/294114 [20:12:01] (03CR) 10Jforrester: [C: 032] "Thank you!" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/294114 (owner: 10Ladsgroup) [20:12:13] (03PS1) 10Jforrester: Change link to beta features. [extensions/ORES] (wmf/1.28.0-wmf.5) - 10https://gerrit.wikimedia.org/r/294115 [20:14:15] (03Merged) 10jenkins-bot: Change link to beta features. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/294114 (owner: 10Ladsgroup) [20:20:57] 06Revision-Scoring-As-A-Service, 05WMDE-Tech-Communication-Mentoring-And-Events, 03WMDE-Tech-Communication-Q2-2016: Collection of topics for HPI hackathon - https://phabricator.wikimedia.org/T127047#2377409 (10Lydia_Pintscher) 05Open>03Resolved The open tasks are still relevant also for other hackathons... [22:55:51] (03CR) 10Dereckson: [C: 032] "SWAT" [extensions/ORES] (wmf/1.28.0-wmf.5) - 10https://gerrit.wikimedia.org/r/294115 (owner: 10Jforrester) [22:57:02] (03Merged) 10jenkins-bot: Change link to beta features. [extensions/ORES] (wmf/1.28.0-wmf.5) - 10https://gerrit.wikimedia.org/r/294115 (owner: 10Jforrester) [23:21:08] Amir1 still around? :) [23:21:36] yuvipanda: yeah, but I'm about to pass out, just waiting for the SWAT to deploy it in fawiki [23:22:02] if it can keep me busy for several minutes, I would love to see [23:22:31] :D [23:22:45] Amir1 indeed, nice moment for you etc :) Enjoy and pass out later :) [23:22:54] say hi to the others in berlin too :D [23:23:32] sure, will you come into Wikimania yuvipanda ? [23:23:46] :D [23:23:51] yup! [23:24:06] unfortunately italy has no doner kaba [23:24:08] b [23:24:13] nice [23:24:28] Do you want me to bring anything fro Berlin? [23:24:39] doner kababs, but I don't think they'll survive the trip :D [23:24:42] i think the Kabab would be spolied by then [23:24:48] *spoiled [23:24:51] yeah [23:24:52] yeah :( [23:24:53] unfortunately [23:25:21] I love beers here [23:25:50] yeah :D [23:25:52] I stopped drinking mostly [23:27:08] :D [23:29:35] yuvipanda: we had anlaytics off-site yesterday, more people will be joining us soon :) [23:29:41] yeah! [23:29:44] I was talking to madhu [23:30:20] yeah, she was here :)