[00:29:51] 10DBA, 10JADE, 10Operations, 10Scoring-platform-team, 10TechCom-RFC: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) [05:17:56] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging, 10Patch-For-Review: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) [05:23:38] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging, 10Patch-For-Review: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) [05:29:14] So, what would you like me to do during the failover? All the monitoring? [05:30:12] I think I will do the script running and the deploy [05:30:39] but I will need you when it fails [05:30:46] it won't! :) [05:30:47] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging, 10Patch-For-Review: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) [05:30:53] I will take care of the monitoring then [05:31:01] You want me to go thru the pre- failover steps now? [05:32:36] did you saw what was the issue with pt-heartbeat? [05:32:42] yeah :( [05:32:47] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging, 10Patch-For-Review: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) [05:32:48] kinda what you expected [05:33:16] I think after now I will change so that I try to parse each parameter individually [05:33:31] as the pt-heartbeat may change in the future [05:34:03] What I didn't get at the time (I read the task yesterday) is, what were the older vs new parameters [05:34:29] I documented that somewhere [05:34:32] let me search it [05:34:45] Thanks :) [05:34:47] on our checklist [05:34:51] have a look [05:35:00] let's seeee [05:35:28] you can start with it, actually, if you want [05:35:46] The failover checklist on the etherpad? [05:35:50] yes [05:36:37] Ah right, I had to refresh, I was going crazy XDD [05:36:47] I see it [05:36:49] it happens if you left it there for a day [05:37:01] or hibernated, etc. [05:37:18] Yeah but I wasn't getting the banner of "reconnect" that you get sometimes [05:38:46] you will find on my .mysql_history on neodimium what I use to fix the grants if you need it, starting on set sq_log_bin=0 [05:41:11] I think for replication, on es1017 looks fine [05:41:23] es1014 will need correction once reimaged [05:41:50] agree? [05:41:56] checking now [05:42:37] and pt-heartbeat is right because I corrected it yesterday [05:42:57] (it used to be wrong, and it is still wrong on all not restarted masters) [05:43:13] but please double confirm both [05:43:22] pt-show-grants h=es1017.eqiad.wmnet,F=/root/.my.cnf | grep repl [05:43:35] ps aux | grep perl [05:45:11] es1017 looks good for repl@10.64.% but not for 10.% (which we don't use) [05:45:35] yes, we should nuke that an others [05:45:48] there are several account like that [05:46:07] do you want to do it now or later? [05:46:22] let's do that later [05:46:25] 'repl'@'10.%' and 'repl'@'208.80.152.%' [05:46:29] should be gone [05:46:41] (they are inactive already) [05:46:55] I will mark it as post-task [05:47:10] cool [05:47:54] written down [05:47:54] heartbeat on es1014 running with the new version [05:48:16] yes, I killed it yesterday [05:48:27] I would do the same on all masters, tough [05:48:36] (but now rigth now) [05:48:38] yeo [05:49:29] i don't see gtid in use [05:49:35] I just killed it [05:49:36] but maybe you disabled now [05:49:51] there is technically 2 replicas to move [05:50:00] eqiad AND codfw [05:50:15] I will let you do that [05:50:40] I will monitor tendril and errors [05:51:06] it is done now [05:51:43] I can see it [05:51:46] and saw no errors [05:52:21] I guess your trick is that this kind of move doesn't really need to wait 5 seconds? [05:53:02] Yeah, that is what I have seen from my experience [05:55:01] I can see puppet disabled on both hosts [05:55:07] yep [05:55:09] and patch merged [05:55:28] lets merge both [05:57:06] tell me when happy, and I will go to -operations [05:57:31] change merged [05:57:33] let's go to operations [06:09:14] 10DBA, 10Patch-For-Review: switchover es1014 to es1017 - https://phabricator.wikimedia.org/T197073 (10Marostegui) Congratulations @jcrespo for doing the first automated failover with the new script! The errors lasted: First error at 06:01:27 and last error 06:02:05 Very impressive! :) [06:38:23] 10DBA, 10Operations: Investigate slow servermon updating queries on db1016 - https://phabricator.wikimedia.org/T165674 (10jcrespo) I caught one slow query while doing other monitoring, putting it here for extra debugging: ``` db1063 22457446 puppet dbproxy1001 puppet 31s UPDATE hosts SET environment = 'product... [06:39:13] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging, 10Patch-For-Review: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) [07:19:49] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging, 10Patch-For-Review: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) [07:20:39] 10DBA, 10Operations-Software-Development: Improve database master switchover script - https://phabricator.wikimedia.org/T200306 (10jcrespo) [07:20:40] 10DBA, 10Operations-Software-Development: Improve database master switchover script - https://phabricator.wikimedia.org/T200306 (10jcrespo) p:05Triage>03Low [07:22:43] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging, 10Patch-For-Review: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) [07:27:36] 10DBA, 10Operations-Software-Development: Improve database master switchover script - https://phabricator.wikimedia.org/T200306 (10jcrespo) [07:28:26] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging, 10Patch-For-Review: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) [07:29:53] 10DBA, 10Operations-Software-Development: Improve database master switchover script - https://phabricator.wikimedia.org/T200306 (10Marostegui) [07:31:53] 10DBA, 10Patch-For-Review: switchover es1014 to es1017 - https://phabricator.wikimedia.org/T197073 (10jcrespo) 05Open>03Resolved a:03jcrespo This is now done, and while there are things pending to do related to es1014 maintenance, the main task T183585 is unblocked from all SPOF db hosts. [07:32:44] how do you feel about https://gerrit.wikimedia.org/r/447754 ? should I wait? [07:33:08] It has been more than 1.30h now since the failover [07:33:11] I think we are good [07:33:24] Also you will not fomart /srv/ so the data will still be there [07:40:19] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging, 10Patch-For-Review: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) [07:44:49] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging, 10Patch-For-Review: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) [07:45:15] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging, 10Patch-For-Review: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) @Ladsgroup All the affected wikis on s3 are fixed. You may run the script on s3 now. [08:03:22] marostegui: Thank you! [08:05:53] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging, 10Patch-For-Review: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) 05Open>03Resolved @Ladsgroup s4 is also fixed. That means all the reported wikis on the above list are now fixed! [08:06:06] 10DBA, 10Data-Services, 10MediaWiki-Change-tagging, 10Patch-For-Review: Recent duplicate entries on change_tag on sanitarium hosts - https://phabricator.wikimedia.org/T200061 (10Marostegui) [08:07:30] \o/ [08:07:48] \o\ |o| /o/ [08:20:26] 10DBA, 10Operations: Investigate slow servermon updating queries on db1016 - https://phabricator.wikimedia.org/T165674 (10Volans) FYI servermon will probably be decommissioned soon, see T198939. [09:46:43] 10DBA, 10MediaWiki-extensions-EducationProgram: Archive, publish and delete EducationProgram data - https://phabricator.wikimedia.org/T200316 (10Reedy) 05Open>03stalled [09:47:20] 10DBA, 10MediaWiki-extensions-EducationProgram: Archive, publish and delete EducationProgram data - https://phabricator.wikimedia.org/T200316 (10Reedy) [09:50:57] 10DBA, 10MediaWiki-extensions-EducationProgram: Archive, publish and delete EducationProgram data - https://phabricator.wikimedia.org/T200316 (10Marostegui) I am going to merge this table with: T174802 [09:51:10] 10DBA: Drop education program (ep_*) tables on wikis where it is not enabled - https://phabricator.wikimedia.org/T174802 (10Marostegui) [09:52:12] 10DBA: Drop education program (ep_*) tables on wikis where it is not enabled - https://phabricator.wikimedia.org/T174802 (10Marostegui) [09:52:14] 10DBA, 10MediaWiki-extensions-EducationProgram: Archive, publish and delete EducationProgram data - https://phabricator.wikimedia.org/T200316 (10Marostegui) [09:52:39] 10DBA, 10MediaWiki-extensions-EducationProgram: Archive, publish and delete EducationProgram data - https://phabricator.wikimedia.org/T200316 (10jcrespo) We could generate dumps (I guess anybody could, according to T125618#4449826 but we DBAs not know the workflow, if one, to publish dumps. I guess in any cas... [09:54:07] 10DBA, 10Data-Services: Drop education program (ep_*) tables on wikis where it is not enabled - https://phabricator.wikimedia.org/T174802 (10Marostegui) [09:55:09] 10DBA, 10Data-Services, 10Dumps-Generation: Drop education program (ep_*) tables on wikis where it is not enabled - https://phabricator.wikimedia.org/T174802 (10jcrespo) @ArielGlenn What is the process for one time dumps, if there is one? Does security team need to be involved? We can dump easily, but what... [10:01:19] 10Blocked-on-schema-change, 10DBA, 10Wikidata, 10Patch-For-Review, 10Schema-change: Drop eu_touched in production - https://phabricator.wikimedia.org/T144010 (10Marostegui) This table fails to get altered when there is high concurrency (I have had to stop replication on s4 slaves to be able to alter it),... [10:01:33] 10Blocked-on-schema-change, 10DBA, 10Wikidata, 10Patch-For-Review, 10Schema-change: Drop eu_touched in production - https://phabricator.wikimedia.org/T144010 (10Marostegui) [10:02:49] 10DBA, 10Operations, 10Epic: DB meta task for next DC failover issues - https://phabricator.wikimedia.org/T189107 (10Marostegui) [10:02:52] 10Blocked-on-schema-change, 10DBA, 10Wikidata, 10Patch-For-Review, 10Schema-change: Drop eu_touched in production - https://phabricator.wikimedia.org/T144010 (10Marostegui) [10:04:46] 10DBA, 10Patch-For-Review, 10Schema-change: Convert UNIQUE INDEX to PK in Production - https://phabricator.wikimedia.org/T199368 (10Marostegui) [10:12:51] Re: high concurrency- note that the master usually has lower concurrency on tables than the replicas [10:13:16] it is up to you, but don't assume because the replicas failed master will [10:13:42] e.g. page failed on enwiki main and api replicas, but it was easy on the master [10:14:12] 10DBA, 10Wikidata, 10Wikidata-Campsite: Rename tmp1 index on wb_terms databases to something more meaningful - https://phabricator.wikimedia.org/T197854 (10Marostegui) @Ladsgroup what do you think about this task? Should we merge it with T85414? I still think the best thing we can do for now is: T197854#4311753 [10:14:44] jynus: they were failing with duplicate key (which didn't exist really) so I think it was a matter of high writes coming in and maybe race condition when the alter was finishing and doing the final rename [10:15:03] oh [10:15:32] That is my theory seeing that: it was failing with different keys every time and once replication was stopped, it would work straightaway [10:17:12] 10DBA: Failover DB masters in row D - https://phabricator.wikimedia.org/T186188 (10Marostegui) Let's plan for the next DC failover to do this or at least move a couple of them. [10:17:38] 10DBA, 10Wikidata: Provision a separate DB shard for wbc_entity_usage - https://phabricator.wikimedia.org/T176277 (10Marostegui) @hoo is this still something to be done? [10:17:40] yes, it makes sense [10:17:55] I was just sharing my experiende that one time [10:18:12] Sure sure [10:18:28] I wasn't too explicit with what exactly failed and the environment conditions :) [10:18:48] I will expand my comment on the ticket [10:18:49] :) [10:19:17] no need [10:19:21] I understood you now [10:19:39] Yeah, but better to have it there so others can understand better, I am copy&pasting what I just wrote here :) [10:19:42] Not much work! [10:29:33] 10DBA, 10Collaboration-Team-Triage, 10Growth-Team, 10MediaWiki-extensions-PageCuration, 10Schema-change: Drop ptrl_comment in production - https://phabricator.wikimedia.org/T157762 (10Marostegui) I have renamed the column on testwiki and test2wiki on db2036 to make sure nothing breaks: ``` +------------+... [10:31:01] 10DBA, 10Collaboration-Team-Triage, 10Growth-Team, 10MediaWiki-extensions-PageCuration, 10Schema-change: Drop ptrl_comment in production - https://phabricator.wikimedia.org/T157762 (10Marostegui) Same on db2055 (enwiki): ``` root@db2055.codfw.wmnet[enwiki]> show create table pagetriage_log\G ************... [10:32:22] I want to run compare.py on es3, all hosts [10:32:37] on all records written since yesterday [10:32:55] For the tests we did? [10:33:10] test yesterday, failover today, reimage etc, [10:33:14] sure :) [10:33:24] note replication cannot break if records are missing [10:33:30] yeah [10:33:39] so I want to be 100% nothing was lost [10:34:01] 1 day of edits, at 1000-1500 edits per minute is not a lot of rows [10:34:25] 2M records, should be fast [10:34:50] then I wonder if I should reimage es1015 or wait a bit and focus on the goal [10:35:45] 10DBA, 10Data-Services, 10Dumps-Generation: Drop education program (ep_*) tables on wikis where it is not enabled - https://phabricator.wikimedia.org/T174802 (10ArielGlenn) We've never had requests for specific tables like these; the ep tables aren't dumped as part of the regular dumps either. If you want t... [10:36:16] jynus: If I were you, I would do it and then "forget" about it till the DC switch [10:36:23] When we can reimage es1011 [10:40:01] 10DBA, 10Epic: [META ticket] Automation for our DBs tracking task - https://phabricator.wikimedia.org/T156461 (10Marostegui) [10:40:03] 10DBA, 10Operations-Software-Development: Improve database master switchover script - https://phabricator.wikimedia.org/T200306 (10Marostegui) [10:40:40] 10DBA, 10Epic: [META ticket] Automation for our DBs tracking task - https://phabricator.wikimedia.org/T156461 (10Marostegui) [10:40:42] 10DBA: For switchovers: A way to check if slaves are up to date - https://phabricator.wikimedia.org/T156465 (10Marostegui) 05Open>03Resolved a:03jcrespo This is now done with the first version of the script described at T199224 and the tracking task for future improvements here T200306 [10:43:47] 10DBA: For switchovers: A way to check if slaves are up to date - https://phabricator.wikimedia.org/T156465 (10jcrespo) To add more context, this is implemented on the library WMFReplication.py, with the replica.is_caught_up_to_master(master) method. Will document all things on wikitech soon as part of that impr... [10:59:46] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Truncate SHA-1 indexes - https://phabricator.wikimedia.org/T51190 (10Marostegui) [10:59:48] 10DBA, 10Patch-For-Review, 10Schema-change: Convert UNIQUE INDEX to PK in Production - https://phabricator.wikimedia.org/T199368 (10Marostegui) [11:00:05] 10Blocked-on-schema-change, 10DBA, 10Wikidata, 10Patch-For-Review, 10Schema-change: Drop eu_touched in production - https://phabricator.wikimedia.org/T144010 (10Marostegui) [11:06:11] 10DBA, 10Data-Services, 10Dumps-Generation: Drop education program (ep_*) tables on wikis where it is not enabled - https://phabricator.wikimedia.org/T174802 (10Reedy) >>! In T174802#4449895, @jcrespo wrote: > Does security team need to be involved? I don't think so. As long as we NULL out the course token... [11:07:43] 10DBA, 10Data-Services, 10Dumps-Generation: Drop education program (ep_*) tables on wikis where it is not enabled - https://phabricator.wikimedia.org/T174802 (10Reedy) [11:09:10] 10DBA, 10Data-Services, 10Dumps-Generation: Drop education program (ep_*) tables on wikis where it is not enabled - https://phabricator.wikimedia.org/T174802 (10jcrespo) > As long as we NULL out the course token in the same way that it is on the labs replicas Then this is not as trivial- it seems the best w... [13:19:09] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Truncate SHA-1 indexes - https://phabricator.wikimedia.org/T51190 (10Marostegui) s7 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1002 [] db1125 [] db1101 [] db1098 [] db1094 [] db1090 [] db1086 [] db1079 [] db1062 [13:19:23] 10DBA, 10Patch-For-Review, 10Schema-change: Convert UNIQUE INDEX to PK in Production - https://phabricator.wikimedia.org/T199368 (10Marostegui) s7 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1002 [] db1125 [] db1101 [] db1098 [] db1094 [] db1090 [] db1086 [] db1079 [] db1062 [13:19:25] 10Blocked-on-schema-change, 10DBA, 10Wikidata, 10Patch-For-Review, 10Schema-change: Drop eu_touched in production - https://phabricator.wikimedia.org/T144010 (10Marostegui) s7 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1002 [] db1125 [] db1101 [] db1098 [] db1094 [] db1090 [] db... [14:36:39] 10DBA, 10Patch-For-Review: Productionize old/temporary eqiad sanitariums - https://phabricator.wikimedia.org/T196376 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['db1120.eqiad.wmnet'] ``` The log can be found in `/var/log/wmf-auto-reimage/20... [14:54:09] 10DBA, 10Patch-For-Review: Productionize old/temporary eqiad sanitariums - https://phabricator.wikimedia.org/T196376 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db1120.eqiad.wmnet'] ``` and were **ALL** successful. [15:04:00] 10Blocked-on-schema-change, 10DBA, 10Wikidata, 10Patch-For-Review, 10Schema-change: Drop eu_touched in production - https://phabricator.wikimedia.org/T144010 (10GoranSMilovanovic) @Marostegui Hi, could you please confirm the following: no other changes in the `wbc_entity_usage` schema will take place exc... [15:05:19] 10Blocked-on-schema-change, 10DBA, 10Wikidata, 10Patch-For-Review, 10Schema-change: Drop eu_touched in production - https://phabricator.wikimedia.org/T144010 (10Marostegui) There is no other schema change related to wbc_entity_usage on-going apart from this, yes :-) [15:07:23] 10Blocked-on-schema-change, 10DBA, 10Wikidata, 10Patch-For-Review, 10Schema-change: Drop eu_touched in production - https://phabricator.wikimedia.org/T144010 (10GoranSMilovanovic) @Marostegui Thank you very much! Can you estimate **when** would the new schema become operational? [15:09:04] 10Blocked-on-schema-change, 10DBA, 10Wikidata, 10Patch-For-Review, 10Schema-change: Drop eu_touched in production - https://phabricator.wikimedia.org/T144010 (10Marostegui) >>! In T144010#4450637, @GoranSMilovanovic wrote: > @Marostegui Thank you very much! Can you estimate **when** would the new schema... [15:51:06] 10DBA, 10Wikidata, 10Wikidata-Campsite: Rename tmp1 index on wb_terms databases to something more meaningful - https://phabricator.wikimedia.org/T197854 (10Ladsgroup) I think this is a subtask of the main one. This would make sense. [16:59:48] marostegui: https://tendril.wikimedia.org/report/slow_queries?host=%5Edb&user=wikiuser&schema=wik&qmode=eq&query=change_tag&hours=72 [17:00:12] The change for removing the first query is deployed on enwiki, commonswiki, wikidatawiki, and dewiki [17:00:26] basically the load time got dropped from 1 minute to one second now [17:04:04] 10DBA, 10MediaWiki-Change-tagging, 10Schema-change, 10User-Ladsgroup: change_tag table needs redesign - https://phabricator.wikimedia.org/T164167 (10Ladsgroup) 05Open>03Resolved a:03Ladsgroup This is being done as part of T185355 [18:07:26] 10DBA, 10JADE, 10Operations, 10Scoring-platform-team, 10TechCom-RFC: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Mooeypoo) I just want to make a point about the specific term "Judgment" -- it has a big potential of setting the t... [18:11:10] 10DBA, 10JADE, 10Operations, 10Scoring-platform-team, 10TechCom-RFC: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) @Mooeypoo Very interesting, thanks for flagging this! I think the term may have originally come about due... [18:40:34] Amir1: Not sure what you want me to see on that URL :-) [18:42:16] marostegui: the top slowest query will be gone, these are historical (I just deployed the change on enwiki, commonswiki, dewiki, wikidata) [18:44:57] Ah ok ok :) [18:45:02] I wasn't sure about what to look at [18:45:42] Glad to see that change_tag will stop being a problem [18:45:43] Well "a problem" [18:45:43] :) [18:47:00] There are so many things I still need to fix about that table :/ [18:48:45] Yeah, but it is great to see progress on it [18:51:22] \o/ [18:58:07] 10DBA, 10JADE, 10Operations, 10Scoring-platform-team, 10TechCom-RFC: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Harej) We did recently just rename the namespace from `Jade` to `Judgment` since `Judgment` is better semantics (na... [19:00:09] 10DBA, 10JADE, 10Operations, 10Scoring-platform-team, 10TechCom-RFC: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) Before this gets out of hand, let's please discuss the name in a subtask, it's important that the deeper, s... [19:00:43] 10DBA, 10JADE, 10Operations, 10Scoring-platform-team, 10TechCom-RFC: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Halfak) {T200365} [20:33:32] 10DBA, 10JADE, 10Operations, 10Scoring-platform-team, 10TechCom-RFC: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Krinkle) >>! In T196547#4446016, @awight wrote: > Here are the notes from our meeting, plus some more discussion af... [20:36:53] 10DBA, 10JADE, 10Operations, 10Scoring-platform-team, 10TechCom-RFC: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Milimetric) Just discussed this in TechCom, saw the minutes of the meeting with @mark and @jcrespo. Would you like... [20:41:49] 10DBA, 10JADE, 10Operations, 10Scoring-platform-team, 10TechCom-RFC: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) @Milimetric Great, I'd love to have an IRC meeting any time that's convenient, and happy to also discuss ra...