[00:19:35] 10Scoring-platform-team, 10Data-Services, 10Wikilabels, 10Patch-For-Review, 10cloud-services-team (Kanban): clouddb1002 low on space -- move wikilabelsdb - https://phabricator.wikimedia.org/T224062 (10Bstorm) It should be safe to swap it over at any time now. The only quirk is that it would be read-only... [11:44:25] 10Scoring-platform-team, 10Data-Services, 10Wikilabels, 10Patch-For-Review, 10cloud-services-team (Kanban): clouddb1002 low on space -- move wikilabelsdb - https://phabricator.wikimedia.org/T224062 (10Halfak) Hi @Bstorm, I just got back from the Wikimedia Hackathon and I'm catching up on other things. I... [12:00:11] o/ [12:00:24] * halfak is catching up on work post-hackathon [12:00:29] Reviewing a paper new. [12:00:31] *now [12:47:57] hi [12:48:04] I'm getting up early to fly to DC [12:48:34] I'm fitting models for causal inference [12:49:01] no change in time-to-revert, and an increase in the inequality of whoes doing reverts [12:49:18] probably an increase in the number of reverts, though one of the models is not converging [12:49:49] yesterday started migrating to the stats machine [12:50:07] since debugging model convergence is one of the things that is really hard in Jupyter for some reason [12:52:09] about to call a lyft [13:15:19] PROBLEM - puppet on ORES-redis02.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:40:19] 10ORES, 10Scoring-platform-team, 10translatewiki.net, 10Security: New ORES model relies on translatewiki.net API, which is not hosted on WMF production - https://phabricator.wikimedia.org/T213131 (10Maintenance_bot) [13:44:11] RECOVERY - puppet on ORES-redis02.experimental is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:02:48] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @alaa_wmde & @nuria - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:52:47] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @alaa_wmde & @nuria - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:10:39] 10Scoring-platform-team, 10Data-Services, 10Wikilabels, 10cloud-services-team (Kanban): clouddb1002 low on space -- move wikilabelsdb - https://phabricator.wikimedia.org/T224062 (10Maintenance_bot) [15:19:48] 10Jade, 10Scoring-platform-team (Current): Rename "JADE" extension to "Jade" - https://phabricator.wikimedia.org/T211046 (10Maintenance_bot) [15:24:39] 10Scoring-platform-team, 10Edit-Review-Improvements-Integrated-Filters, 10Growth-Team, 10Patch-For-Review: Enable ORES RCFilters for German Wikipedia (dewiki) - https://phabricator.wikimedia.org/T215354 (10MMiller_WMF) @Ladsgroup -- I wanted to follow up on my question above. Since dewiki previously banne... [15:45:52] 10ORES, 10Scoring-platform-team, 10Operations, 10Release Pipeline, 10Release-Engineering-Team (Backlog): Execution of the deployment pipeline should be configurable via .pipeline/config.yaml - https://phabricator.wikimedia.org/T210267 (10Maintenance_bot) [15:47:30] 10Scoring-platform-team, 10Edit-Review-Improvements-Integrated-Filters, 10Growth-Team, 10Patch-For-Review: Enable ORES RCFilters for German Wikipedia (dewiki) - https://phabricator.wikimedia.org/T215354 (10Halfak) For clarity, I don't think ORES was ever "banned", but we did receive resistance when we firs... [17:07:51] 10Scoring-platform-team, 10Data-Services, 10Wikilabels, 10cloud-services-team (Kanban): clouddb1002 low on space -- move wikilabelsdb - https://phabricator.wikimedia.org/T224062 (10Bstorm) @Halfak if you are cool with deploy-on-friday, I'll do it! :) The replica is replicating now, so it should be pretty... [17:09:13] 10Scoring-platform-team, 10Data-Services, 10Wikilabels, 10cloud-services-team (Kanban): clouddb1002 low on space -- move wikilabelsdb - https://phabricator.wikimedia.org/T224062 (10Bstorm) The new server, FYI is clouddb-wikilabels-01.clouddb-services.eqiad.wmflabs, so you can validate that you can make a r... [17:11:51] 10Scoring-platform-team, 10Cloud-VPS, 10Wikilabels, 10Epic, and 2 others: Replace labsdb100[4567] with instances on cloudvirt1019 and cloudvirt1020 - https://phabricator.wikimedia.org/T193264 (10Bstorm) 05Open→03Resolved This is done! [17:13:32] 10Scoring-platform-team, 10Wikilabels, 10cloud-services-team (Kanban): Move wikilabels DB to it's own instance with replica - https://phabricator.wikimedia.org/T221673 (10Bstorm) [17:13:35] 10Scoring-platform-team, 10Data-Services, 10Wikilabels, 10cloud-services-team (Kanban): clouddb1002 low on space -- move wikilabelsdb - https://phabricator.wikimedia.org/T224062 (10Bstorm) [18:02:49] 10Scoring-platform-team (Current), 10Wikilabels: Migrate Wikilabels to new DB server - https://phabricator.wikimedia.org/T224163 (10Halfak) [18:03:58] 10Scoring-platform-team (Current), 10Wikilabels: Migrate Wikilabels to new DB server - https://phabricator.wikimedia.org/T224163 (10Bstorm) Ideally, there should be no config change needed if you are using the wikilabels.db.svc.eqiad.wmflabs address now. [18:05:27] 10Scoring-platform-team (Current), 10Wikilabels: Migrate Wikilabels to new DB server - https://phabricator.wikimedia.org/T224163 (10Halfak) Yup. We've got that in the config. At the very least, we'll need to do a restart to reconnect to the new DB based on the DNS change. But we'll also probably want a litt... [18:06:18] 10Scoring-platform-team (Current), 10Wikilabels: Migrate Wikilabels to new DB server - https://phabricator.wikimedia.org/T224163 (10Halfak) Would 1800 UTC on Friday, May 24th work for you, @Bstorm? [18:12:22] 10Scoring-platform-team (Current), 10Wikilabels: Migrate Wikilabels to new DB server - https://phabricator.wikimedia.org/T224163 (10Bstorm) Yup! I'm trying other avenues to keep toolsdb from running out of space on the replica as well. If it goes nuclear for some reason before then, I'll try to reach you bef... [18:20:17] 10ORES, 10Scoring-platform-team (Current), 10Wikimedia-Hackathon-2019 (Newcomer friendly): Generate quality score average and add to WikidataQuality gadget - https://phabricator.wikimedia.org/T223845 (10Halfak) [18:20:19] 10ORES, 10Scoring-platform-team (Current), 10articlequality-modeling, 10artificial-intelligence: Add article quality model for cswiki - https://phabricator.wikimedia.org/T223385 (10Halfak) [18:20:22] 10ORES, 10Scoring-platform-team, 10revscoring, 10artificial-intelligence, 10good first bug: Add support for a new language in ORES - https://phabricator.wikimedia.org/T223384 (10Halfak) [18:20:24] 10ORES, 10Scoring-platform-team, 10good first bug: Improve Czech Language assets - https://phabricator.wikimedia.org/T223383 (10Halfak) [18:20:27] 10ORES, 10Scoring-platform-team (Current), 10Epic, 10Wikimedia-Hackathon-2019 (Newcomer friendly): Improvements to ORES localization and support - https://phabricator.wikimedia.org/T223382 (10Halfak) [18:20:54] 10Scoring-platform-team, 10revscoring, 10Chinese-Sites, 10artificial-intelligence: Chinese language utilities - https://phabricator.wikimedia.org/T109366 (10Halfak) [18:21:04] 10ORES, 10Scoring-platform-team (Current), 10Epic, 10Wikimedia-Hackathon-2019 (Newcomer friendly): Improvements to ORES localization and support - https://phabricator.wikimedia.org/T223382 (10Halfak) [18:21:41] 10ORES, 10Scoring-platform-team (Current), 10Wikimedia-Hackathon-2019 (Newcomer friendly): Improve dutch language assets - https://phabricator.wikimedia.org/T224168 (10Halfak) [18:21:44] 10ORES, 10Scoring-platform-team (Current), 10Wikimedia-Hackathon-2019 (Newcomer friendly): Improve dutch language assets - https://phabricator.wikimedia.org/T224168 (10Halfak) https://etherpad.wikimedia.org/p/dutch_badwords [18:22:48] 10ORES, 10Scoring-platform-team (Current), 10Wikimedia-Hackathon-2019 (Newcomer friendly): Improve dutch language assets - https://phabricator.wikimedia.org/T224168 (10Halfak) https://github.com/wikimedia/revscoring/pull/439 [18:23:42] 10Scoring-platform-team, 10revscoring, 10Chinese-Sites, 10artificial-intelligence: Chinese language utilities - https://phabricator.wikimedia.org/T109366 (10Halfak) I worked with @zhuyifei1999 to develop https://etherpad.wikimedia.org/p/chinese_word_lists and then implemented it in https://github.com/wikim... [18:24:55] 10Scoring-platform-team, 10revscoring, 10Chinese-Sites, 10artificial-intelligence: Chinese language utilities - https://phabricator.wikimedia.org/T109366 (10Halfak) I trained some damaging and goodfaith models. They are performing... OK. We're getting in the upper 80s for ROC-AUC. I would expect a solid... [19:34:46] wikimedia/editquality#547 (zhwiki - 9ffa170 : Aaron Halfaker): The build passed. https://travis-ci.org/wikimedia/editquality/builds/535932547 [19:38:30] 10Scoring-platform-team (Current), 10VPS-project-icinga2, 10User-Zppix: Ensure ORES experimental hosts nrpe_local.cfg allows icinga2’s ip - https://phabricator.wikimedia.org/T223578 (10Halfak)