[08:46:33] PROBLEM - puppet on ORES-redis02.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:49:02] PROBLEM - puppet on ORES-worker02.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:50:34] PROBLEM - puppet on ORES-web02.Experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:58:02] PROBLEM - puppet on ORES-web01.Experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:01:11] PROBLEM - puppet on ORES-worker01.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:12:33] RECOVERY - puppet on ORES-redis02.experimental is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:15:03] RECOVERY - puppet on ORES-worker02.experimental is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:16:30] RECOVERY - puppet on ORES-web02.Experimental is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [09:24:01] RECOVERY - puppet on ORES-web01.Experimental is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [09:27:10] RECOVERY - puppet on ORES-worker01.experimental is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:20:34] 10ORES, 10Scoring-platform-team, 10Patch-For-Review: Refactor ORES puppet for Kubernetes - https://phabricator.wikimedia.org/T182332 (10Ladsgroup) What I don't understand is there won't be any puppet inside containers so none of these modules will be applied, does this really needed? @akosiaris Should we cal... [11:51:54] 10ORES, 10Scoring-platform-team, 10Patch-For-Review: Refactor ORES puppet for Kubernetes - https://phabricator.wikimedia.org/T182332 (10akosiaris) >>! In T182332#4971501, @Ladsgroup wrote: > What I don't understand is there won't be any puppet inside containers so none of these modules will be applied, does... [12:04:36] 10ORES, 10Scoring-platform-team, 10Operations: [Epic] Deploy ORES in kubernetes cluster - https://phabricator.wikimedia.org/T182331 (10Ladsgroup) [12:04:41] 10ORES, 10Scoring-platform-team, 10Patch-For-Review: Refactor ORES puppet for Kubernetes - https://phabricator.wikimedia.org/T182332 (10Ladsgroup) 05Open→03Invalid Let's do it. [12:08:23] afk for dinner, will be back soon [12:08:26] *lunch [13:01:43] halfak / Amir1 - I have fixed the gerrit replica for you [13:01:53] (draftquality) [13:05:24] 10ORES, 10Scoring-platform-team (Current), 10Release-Engineering-Team, 10draftquality-modeling, and 2 others: Gerrit mirror from phab broken for source/draftquality - https://phabricator.wikimedia.org/T216616 (10MarcoAurelio) @mmodell @Halfak - I re-tried with the K18 credential and updated the repo here.... [13:08:45] hauskatze: thanks! [13:09:01] np, happy to help [13:09:10] still deserves some investigation [13:43:03] halfak: hey whats status on T216616 ? [13:43:03] T216616: Gerrit mirror from phab broken for source/draftquality - https://phabricator.wikimedia.org/T216616 [13:55:06] Zppix: we fixed it I think [13:55:22] hauskatze: sweet [13:56:27] Thank you hauskatze! [13:56:45] I am not sure if it is the definitive fix though [13:56:59] thus feel free to close the task or leave it open [13:57:25] phabricator is a bit of a brat sometimes, needs a slap from time to time to be taught :P [13:59:02] hauskatze: isnt that phabricator abuse :P [13:59:24] nah [13:59:47] hauskatze: hey if its broken then its on wmf servers :P [14:00:42] Not sure. 20after4 said it was a credentials issue. Maybe he fixed something but forgot to tell us. This morning the update worked. [14:01:09] hauskatze: who knows this is phab we are talking about [14:01:27] apparently it was not phab but gerrit [14:01:33] which was refusing something [14:01:51] anyway, I don't have access to the whole logs so I'd be speculating [14:03:30] (03PS1) 10Halfak: Updates to draftquality submodule. [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/491963 (https://phabricator.wikimedia.org/T215406) [14:03:48] Amir1, https://gerrit.wikimedia.org/r/#/c/mediawiki/services/ores/deploy/+/491963 [14:04:50] Looks like I could pull the changes. I think we'll know for sure when we make a new commit to draftquality and see if it propagates. [14:05:32] I could always improve the README to see. [14:05:35] (03CR) 10Ladsgroup: [V: 03+2 C: 03+2] Updates to draftquality submodule. [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/491963 (https://phabricator.wikimedia.org/T215406) (owner: 10Halfak) [14:05:56] halfak: Yes, but I was about to make the patch [14:06:06] ? [14:06:10] Which patch? [14:06:11] we are stepping on each other's toes [14:06:20] the one you made [14:06:37] Ahh. Yeah sorry. I just figured I'd pick that up since it finished up the config updates. Sorry about that. [14:07:28] on which I was about to make the patch too in yesterday [14:10:36] * halfak assigned the task to himself a few days ago. [14:11:02] I had a patch in progress when I ran into the missing trwiki last week. [14:11:27] Sorry for the mixup, but I had hoped taking on the task would communicate that. [14:11:50] It was about rebuilding not deploying [14:12:26] anyway, it's on beta now [14:16:27] 10ORES, 10Scoring-platform-team (Current), 10Release-Engineering-Team, 10draftquality-modeling, and 3 others: Gerrit mirror from phab broken for source/draftquality - https://phabricator.wikimedia.org/T216616 (10Halfak) I made https://github.com/wikimedia/draftquality/commit/264071ce5312e67561b597d31ada86a... [14:22:04] halfak: https://gerrit.wikimedia.org/r/plugins/gitiles/scoring/ores/draftquality/+/264071ce5312e67561b597d31ada86a1b3487747 <-- was this as expected? :) [14:57:58] 10ORES, 10Scoring-platform-team, 10Analytics, 10Dumps-Generation, and 3 others: Decide whether we will include raw features - https://phabricator.wikimedia.org/T211069 (10Milimetric) This increase in data sounds fine, and the proposed example path looks fine too. Hundreds of subfolders are only annoying w... [15:21:31] yay! Looks like the commit made it through! [15:22:44] 10ORES, 10Scoring-platform-team (Current), 10Release-Engineering-Team, 10draftquality-modeling, and 4 others: Gerrit mirror from phab broken for source/draftquality - https://phabricator.wikimedia.org/T216616 (10Halfak) 05Open→03Resolved a:03MarcoAurelio https://gerrit.wikimedia.org/r/plugins/gitiles... [15:23:03] halfak: the commit finally commited itself to make it all the way through :P [15:23:10] :P [15:24:17] Amir1, could you review/merge https://github.com/wikimedia/ores-wmflabs-deploy/pull/104 ? [15:25:59] halfak: Done [15:26:35] I'll need to be away for the 1800UTC deployment window. I've got an ice dam based leak in my house that I'll need to address during my "lunch". [15:26:47] Maybe we can tagteam the next one. [15:26:49] :) [15:27:24] halfak: i dont recommend having ice being apart of the house [15:28:04] Agreed. :( [17:02:27] I need to run an errand really quickly, I'll be back [18:09:35] back now [18:16:13] (03PS1) 10Ladsgroup: Change default task serializer of celery from pickle to json [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/492013 (https://phabricator.wikimedia.org/T206333) [18:21:19] (03CR) 10Ladsgroup: [V: 03+2 C: 03+2] "Going beta" [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/492013 (https://phabricator.wikimedia.org/T206333) (owner: 10Ladsgroup) [19:12:35] Ice broken. Salt laid down. And I didn't even fall off of the roof! [19:12:37] Success. [19:12:42] o/ Amir1 [19:12:46] How did the deploy go? [19:12:51] smooth [19:13:02] I even deployed the config change [19:13:10] now tasks are in json [19:13:15] no error so far [19:17:18] SPAM IS COMING [19:17:21] 10MediaWiki-extensions-ORES, 10Scoring-platform-team (Current), 10ORES-Support-Checklist, 10MW-1.32-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), and 2 others: Change mentions of wp10 to articlequality in products - https://phabricator.wikimedia.org/T203080 (10Ladsgroup) 05Open→03Resolved [19:17:32] 10Scoring-platform-team (Current), 10Operations, 10User-Ladsgroup: Spec out migrating ORES to kubernetes - https://phabricator.wikimedia.org/T210109 (10Ladsgroup) 05Open→03Resolved [19:17:44] 10Scoring-platform-team, 10Edit-Review-Improvements-Integrated-Filters, 10Growth-Team, 10Patch-For-Review: Enable ORES RCFilters for German Wikipedia (dewiki) - https://phabricator.wikimedia.org/T215354 (10Ladsgroup) [19:17:55] 10Scoring-platform-team (Current), 10Edit-Review-Improvements-Integrated-Filters, 10editquality-modeling, 10Growth-Team (Current Sprint), and 2 others: Enable ORES filters on RC for Italian Wikipedia - https://phabricator.wikimedia.org/T211032 (10Ladsgroup) [19:17:57] 10ORES, 10Scoring-platform-team (Current), 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Audit deployments of ORES quality models and new filters - https://phabricator.wikimedia.org/T215353 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup [19:18:13] 10Scoring-platform-team (Current), 10revscoring, 10User-Ladsgroup, 10artificial-intelligence: Remove python 3.4 from revscoring travis config - https://phabricator.wikimedia.org/T215279 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup [19:18:17] 10ORES, 10Scoring-platform-team (Current), 10editquality-modeling, 10artificial-intelligence: feature.len() has already been added to {wikibase.revision.diff}. Could be overwritten? - https://phabricator.wikimedia.org/T214979 (10Ladsgroup) 05Open→03... [19:18:27] 10MediaWiki-extensions-ORES, 10Scoring-platform-team (Current), 10MW-1.33-notes (1.33.0-wmf.17; 2019-02-12), 10Patch-For-Review, 10User-Ladsgroup: ORES extension is broken on master - https://phabricator.wikimedia.org/T215719 (10Ladsgroup) 05Open→03Resolved [19:18:33] 10Scoring-platform-team (Current), 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Implement better defaults for autolabel utlity - https://phabricator.wikimedia.org/T215671 (10Ladsgroup) 05Open→03Resolved [19:18:37] 10ORES, 10Scoring-platform-team (Current), 10Patch-For-Review, 10User-Ladsgroup: Rebuild all models with revscoring 2.3.3 - https://phabricator.wikimedia.org/T215406 (10Ladsgroup) [19:18:39] 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Improve ORES test coverage - https://phabricator.wikimedia.org/T212090 (10Ladsgroup) 05Open→03Resolved [19:18:43] 10Scoring-platform-team (Current), 10articlequality-modeling, 10draftquality-modeling, 10editquality-modeling, 10artificial-intelligence: Create features using "words to watch" - https://phabricator.wikimedia.org/T195778 (10Ladsgroup) 05Open→03Resolved [19:18:46] 10MediaWiki-extensions-ORES, 10Scoring-platform-team: Deploy ORES Review Tool in kowiki - https://phabricator.wikimedia.org/T161628 (10Ladsgroup) [19:18:48] 10Scoring-platform-team (Current), 10editquality-modeling, 10Epic, 10artificial-intelligence: [Epic] Edit quality models (damaging/goodfaith) - https://phabricator.wikimedia.org/T130213 (10Ladsgroup) [19:18:51] 10MediaWiki-extensions-ORES, 10Scoring-platform-team (Current), 10MW-1.33-notes (1.33.0-wmf.14; 2019-01-22), 10Patch-For-Review, 10User-Ladsgroup: Implement JS ORES client in mw-ORES extension - https://phabricator.wikimedia.org/T201691 (10Ladsgroup) 05Open→03Resolved [19:18:53] 10Scoring-platform-team (Current), 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Train/test damaging and goodfaith models for kowiki - https://phabricator.wikimedia.org/T161629 (10Ladsgroup) 05Open→03Resolved [19:18:58] 10Scoring-platform-team (Current), 10editquality-modeling, 10Patch-For-Review, 10Spanish-Sites, and 2 others: Train/test damaging & goodfaith models for Spanish Wikiquote (eswikiquote) - https://phabricator.wikimedia.org/T209784 (10Ladsgroup) 05Open→03Resolved [19:19:00] 10ORES, 10Scoring-platform-team: Implement sentinel for ORES production Redis - https://phabricator.wikimedia.org/T122676 (10Ladsgroup) [19:19:02] 10ORES, 10Scoring-platform-team (Current): ORES command line service sometimes hangs - https://phabricator.wikimedia.org/T205909 (10Ladsgroup) 05Open→03Resolved [19:19:04] 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Add support for redis-sentinel in score cache - https://phabricator.wikimedia.org/T210579 (10Ladsgroup) 05Open→03Resolved [19:19:06] 10Scoring-platform-team (Current), 10User-Ladsgroup, 10User-Zppix: Wiki-ai Travis-CI Image upgrade - https://phabricator.wikimedia.org/T183214 (10Ladsgroup) 05Open→03Resolved [19:19:08] 10ORES, 10Scoring-platform-team (Current), 10Patch-For-Review, 10User-Ladsgroup: Change default serializer of celery from pickle to json - https://phabricator.wikimedia.org/T206333 (10Ladsgroup) [19:19:10] 10ORES, 10Scoring-platform-team (Current), 10Patch-For-Review, 10User-Ladsgroup: Rebuild all models with revscoring 2.3.3 - https://phabricator.wikimedia.org/T215406 (10Ladsgroup) 05Open→03Resolved [19:19:12] 10Scoring-platform-team (Current), 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Deploy edit quality models for itwiki - https://phabricator.wikimedia.org/T130286 (10Ladsgroup) [19:19:14] 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Update deprecated cv_train usage - https://phabricator.wikimedia.org/T192412 (10Ladsgroup) 05Open→03Resolved [19:19:16] 10ORES, 10Scoring-platform-team (Current), 10Epic: Migrate Scoring services to Debian Stretch - https://phabricator.wikimedia.org/T186238 (10Ladsgroup) [19:19:18] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Complete itwiki edit quality campaign - https://phabricator.wikimedia.org/T130265 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup [19:19:20] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10User-revi, 10artificial-intelligence: Complete editquality campaign for Korean Wikipedia - https://phabricator.wikimedia.org/T161627 (10Ladsgroup) 05Open→03Resolved [19:19:22] 10Scoring-platform-team (Current), 10editquality-modeling, 10Epic, 10artificial-intelligence: [Epic] Edit quality models (damaging/goodfaith) - https://phabricator.wikimedia.org/T130213 (10Ladsgroup) [19:19:24] 10ORES, 10Scoring-platform-team (Current), 10Continuous-Integration-Config, 10User-Ladsgroup: Migrate ORES CI to Stretch - https://phabricator.wikimedia.org/T186239 (10Ladsgroup) 05Open→03Resolved [19:19:27] 10Scoring-platform-team (Current), 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Deploy edit quality models for itwiki - https://phabricator.wikimedia.org/T130286 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup [19:19:29] 10Jade, 10Scoring-platform-team (Current), 10MW-1.33-notes (1.33.0-wmf.14; 2019-01-22), 10Patch-For-Review: Make Jade UI elements translatable - https://phabricator.wikimedia.org/T211346 (10Ladsgroup) 05Open→03Resolved [19:19:33] 10Jade, 10Scoring-platform-team (Current), 10Operations, 10TechCom, and 4 others: Deploy JADE extension to production - https://phabricator.wikimedia.org/T183381 (10Ladsgroup) [19:33:47] Thanks for the update Amir1! I'm glad it went smoothly :) [19:34:12] There's one last thing I need to do, will do it asap [19:37:36] what's that? [19:38:41] halfak: anything particularly pressing you want to talk about in 1:1? if not i'm going to take this time to eat lunch and start migrating toward the office [19:39:23] harej, yeah. Was hoping we could continue discussing AI product and maybe go over the Topic Embeddings proposal. [19:39:32] But we could reschedule. [19:39:37] oh! those are good things [19:39:45] give me a minute and i'll hop on the line [19:39:46] * halfak just joined the meeting and continued working. [19:39:48] kk [19:39:58] Feel free to grab some food first. :) [19:40:48] (03PS1) 10Ladsgroup: Drop accepting pickle altogether [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/492033 (https://phabricator.wikimedia.org/T206333) [19:44:37] (03CR) 10Ladsgroup: [V: 03+2 C: 03+2] Drop accepting pickle altogether [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/492033 (https://phabricator.wikimedia.org/T206333) (owner: 10Ladsgroup) [20:08:00] 10ORES, 10Scoring-platform-team (Current), 10Patch-For-Review, 10User-Ladsgroup: Upgrade celery to 4.1.0 for ORES - https://phabricator.wikimedia.org/T178441 (10Ladsgroup) [20:08:05] 10ORES, 10Scoring-platform-team (Current), 10Patch-For-Review, 10User-Ladsgroup: Change default serializer of celery from pickle to json - https://phabricator.wikimedia.org/T206333 (10Ladsgroup) 05Open→03Resolved It's done \o/ [20:16:30] 10Scoring-platform-team (Current), 10Wikilabels, 10User-Ladsgroup: wikilabels is failing on master - https://phabricator.wikimedia.org/T216762 (10Ladsgroup) [20:20:16] okay I go eat something, will be back soon [21:06:11] back [21:25:17] halfak: are you aware of the Redis relicensing? Is this something we should be concerned about? My understanding is that the license change is mostly meant to target SaaS resellers like Amazon but I’m not a lawyer and evidently it was bad enough to provoke a fork [21:26:52] I didn't. [21:27:03] What's the license changing to? [21:30:47] harej, ^ [21:31:14] They’re making up a new one that doesn’t allow you to commercially resell their service [21:34:34] 10ORES, 10Scoring-platform-team: Implement sentinel for ORES production Redis - https://phabricator.wikimedia.org/T122676 (10Ladsgroup) Hey @Joe and @akosiaris. What do you think of the next steps here? Can we move forward or we need to discuss more our options? If you need more hands to help on anything, just... [21:37:18] Amir1, any other notes on https://github.com/wikimedia/editquality/pull/181 ? [21:37:30] harej, oh that might be OK for us. [21:37:56] https://redislabs.com/blog/redis-license-bsd-will-remain-bsd/ [21:39:38] Oh, they’re only relicensing some of their more specialized products. Not a bad business model probably. [21:42:50] halfak: I [21:42:56] *I'll check tomorrow [21:43:05] Gotcha. OK. [21:45:16] is it a blocker for the other work? [21:45:25] hewiki new labelling campaign? [21:48:41] yeah. [21:49:00] Unless we can figure out a way to mix-and-match sampling strategies in our current Make generator. [22:31:03] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/492142 (owner: 10L10n-bot)