[01:58:29] 10DBA, 06Labs, 10Tool-Labs, 13Patch-For-Review: Provisioning MySQL replica users fails on tool labs - https://phabricator.wikimedia.org/T151014#2804751 (10yuvipanda) [05:10:23] 10DBA, 10MediaWiki-API, 10MediaWiki-Database, 07Performance: ApiQueryExtLinksUsage::run query has crazy limit - https://phabricator.wikimedia.org/T59176#2804914 (10Anomie) Status: Proposal in T59176#603751, looking for DBA approval of the plan before writing up the schema change and such. ---- In an atte... [07:11:22] 10DBA, 06Operations, 10ops-codfw: db2035: RAID disk about to fail - https://phabricator.wikimedia.org/T150511#2805005 (10Marostegui) Hey @Papaul That disk looks good now, but there is another one in predictive failure: ``` logicaldrive 1 (3.3 TB, RAID 1+0, OK) physicaldrive 1I:1:1 (port 1I:box... [08:25:12] labsdb1001 and .3 crashed again last night, I will investigate in a bit to see if I can find a common reason. A quick glance reveals massive queries with: ROWS_READ: 14632696908 [08:25:16] I will check further in a bit [08:26:41] ERROR 1062 (23000) at line 1: Duplicate entry '53260621' for key 'PRIMARY' [08:27:04] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: db2049 overheated and restarted - https://phabricator.wikimedia.org/T150876#2805084 (10Marostegui) I have started MySQL and let it recover, as there was no errors. I have started replication. Even though the burning tests were fine, I have depooled the... [08:27:18] labs servers? :( [08:28:05] half of s4 production page, including the master [08:28:41] what? [08:29:09] what page? [08:29:33] the page table [08:30:10] you scared me. I understood: "half of production s4 servers paged for that error" [08:30:25] well, they didn't paged it [08:30:37] but that doesn't make it less serious [08:31:01] I am not getting in, you are saying that half s4 servers have repl broken? [08:31:06] I am not seeing that anywhere [08:31:09] no [08:31:25] I think they have internal brokenness [08:31:58] can you give me a hostname? [08:32:39] I am giving you a full report in a second [08:32:45] thanks [08:47:25] 10DBA, 10MediaWiki-API, 10MediaWiki-Database, 07Performance: ApiQueryExtLinksUsage::run query has crazy limit - https://phabricator.wikimedia.org/T59176#2805110 (10jcrespo) Anomie you say "a local MySQL instance", but we do not use mysql on production. Can you test it on the latest 10.0 MariaDB version. Th... [08:59:36] 10DBA: duplicate key problems on s4 - https://phabricator.wikimedia.org/T151029#2805124 (10jcrespo) [08:59:49] ^marostegui [09:00:01] checking [09:01:45] :( [09:02:13] Any idea why is that happening? As in: any idea where could that come from? [09:02:59] 10DBA: duplicate key problems on s4 - https://phabricator.wikimedia.org/T151029#2805144 (10jcrespo) ``` grep ERROR alter.log ERROR 1062 (23000) at line 1: Duplicate entry '14-TEP70_locomotives_of_Ukraine' for key 'name_title' ERROR 1062 (23000) at line 1: Duplicate entry '53249753' for key 'PRIMARY' ERROR 1062... [09:04:00] well, we depool one servers and we do surgery [09:04:51] Yes, I know. I meant if that could ring a bell like: Oh I remember that alter table when we did bla bla bla, it could be because of that [09:05:31] well, remember that query you did that had the same issue? [09:05:39] this is more of the same [09:06:06] old format, the table hasn't been recreated, export and reimport [09:06:21] is it a big table? [09:06:23] 10DBA, 06Operations, 10ops-codfw: Degraded RAID on db2035 - https://phabricator.wikimedia.org/T150973#2805147 (10Volans) [09:06:31] medium [09:07:28] although one error is in dbstore2001.codfw.wmnet [09:08:08] but the table technically wasn't recreated either [09:09:14] exactly [09:09:19] it is 10GB [09:09:21] I was going to say that it was copied but not recreated [09:09:29] so it makes sense to be broken there too if the original server is broken [09:09:41] the alter goes fast because it mostly fits into the bufferpool [09:10:59] SELECT * FROM page WHERE page_id=53249753; -> Empty set (0.00 sec) [09:11:32] 10DBA, 06Operations, 10ops-codfw: Degraded RAID on db2035 - https://phabricator.wikimedia.org/T150973#2805150 (10Marostegui) That disk was in predictive failure and was changed by @Papaul yesterday, so that is why it is probably marked as failed yesterday. Now it looks good (even though there is another one... [09:12:08] the plot thickens [09:12:31] because it is complained about a duplicate key on a key that doesn't exist [09:12:37] *complaning [09:13:03] How is that possible? :| [09:13:03] o wait, wrong server [09:13:05] ah [09:13:06] XD [09:14:48] it is interesting, because it is complaining about rows that are high in number [09:15:09] which could mean the problem is with the alter algorithm [09:15:25] because I assume the AUTO_INC was never chagned, rigtht? [09:15:30] which could be the explanation for that [09:15:32] but it is rare [09:16:20] no, alter backfills changes after doing the alter [09:16:42] which I assume could be badly programmed [09:17:45] I've got the same problem on db1058, the row doesn't exist [09:19:37] That is not normal unless you've played with the AUTO_INC in the wrong way [09:19:44] I mean, I cannot even understand how is that possible [09:21:10] 10DBA, 13Patch-For-Review: db2034: investigate its crash and reimage - https://phabricator.wikimedia.org/T149553#2805156 (10Marostegui) I am doing a transfer from db2070 to db2034 which would normally break db2034 after a while. We will see what happens now. [09:24:00] if page is like that, imagine revision [09:25:06] Would that arise in an alter table engine=innodb? [09:25:14] like if we just recreate the table..would it complain? [09:26:21] engine=innodb, force [09:26:43] the original changed to a noop essentially some time ago [09:28:36] No, what I was thinking about is that I compressed s4 revision table (and s1) and it didn't complain, so maybe that table isn't that bad :) [09:28:46] 10DBA: Compress InnoDB on S5 (wikidatawiki, dewiki) - https://phabricator.wikimedia.org/T150747#2805220 (10Marostegui) wikidatawiki finished 788G -> 313G dewiki finished. From 365G to 144G So S5 went from: `1.1T to 457G` [09:29:01] 10DBA: Compress InnoDB on S5 (wikidatawiki, dewiki) - https://phabricator.wikimedia.org/T150747#2805221 (10Marostegui) 05Open>03Resolved [09:29:03] 10DBA: Meta ticket: Deploy InnoDB compression where possible - https://phabricator.wikimedia.org/T150438#2805222 (10Marostegui) [09:29:23] marostegui, which would reinforce that the problem is a race condition on the alter [09:29:53] exactly [09:30:03] how to prove that is the hard thing though [09:30:36] I think the table is small enough to export and reimport [09:30:45] it should not take more than 30 minutes [09:31:23] 10DBA, 13Patch-For-Review: Import S5 to dbstore2001 and dbstore2002 + compression - https://phabricator.wikimedia.org/T150518#2805226 (10Marostegui) This is all done. Both dbstore2001 and dbstore2002 are running now dewiki and wikidatawiki (s5) and compressed See results here: T150747#2805220 [09:31:26] I import 4GB tables in 2 minutes [09:31:40] 10DBA, 13Patch-For-Review: Import S5 to dbstore2001 and dbstore2002 + compression - https://phabricator.wikimedia.org/T150518#2805228 (10Marostegui) 05Open>03Resolved [09:32:08] jynus: you are planning on a mysqldump + insert it? [09:32:22] no [09:32:26] load data [09:32:38] to a txt file I can examine [09:33:08] ah right [09:33:19] even compare between servers [09:43:46] the interesting part is that it didn't happen on other shards, only here [09:44:44] I will take over T151029 and start debugging if you are ok with that [09:44:44] T151029: duplicate key problems on s4 - https://phabricator.wikimedia.org/T151029 [09:44:54] sure [09:45:05] However, it makes sense that it only happens in one shard, no? [09:45:09] maybe you can stop touching the dbstores [09:45:15] yes, I have stopped now [09:45:16] so I can debug there too [09:45:19] I am investigating labs [09:45:22] yes, dbstores are all for you [09:45:23] :) [09:45:28] yes, I was going to suggest that [09:46:45] Yeah, I noticed that I didn't leave a select now() when lkogging to a file, so it will make things a bit harder to correlate the crashes [09:46:59] I have left another watchdog now with the now() anyways [09:47:06] It looks like the servers only crashes at night [09:47:14] So I will check tomorrow morning too to see what we got [09:51:39] 10DBA, 07Schema-change, 07Tracking: Schema changes for Wikimedia wikis (tracking) - https://phabricator.wikimedia.org/T51188#2805264 (10jcrespo) [09:51:48] 07Blocked-on-schema-change, 10DBA, 10Wikimedia-Site-requests, 06Wikisource, and 2 others: Schema change for page content language - https://phabricator.wikimedia.org/T69223#2805262 (10jcrespo) 05Open>03stalled [09:52:42] 10DBA: duplicate key problems on s4 - https://phabricator.wikimedia.org/T151029#2805266 (10jcrespo) p:05Triage>03High [09:53:37] 10DBA: duplicate key problems on s4 - https://phabricator.wikimedia.org/T151029#2805124 (10jcrespo) [09:53:41] 07Blocked-on-schema-change, 10DBA, 10Wikimedia-Site-requests, 06Wikisource, and 2 others: Schema change for page content language - https://phabricator.wikimedia.org/T69223#2284358 (10jcrespo) [09:54:41] 10DBA: duplicate key problems on s4 - https://phabricator.wikimedia.org/T151029#2805271 (10jcrespo) a:03jcrespo [10:09:23] marostegui, I am going to retry the alter table with a different algorithm on dbstore2001, ok, marostegui? [10:09:33] jynus: go ahead [10:10:16] I am going to downtime s4 lag there [10:31:13] 10DBA, 10Phabricator: db1048 has started lagging - https://phabricator.wikimedia.org/T151039#2805396 (10jcrespo) [10:35:28] 10DBA, 10Phabricator: db1048 has started lagging - https://phabricator.wikimedia.org/T151039#2805442 (10jcrespo) a:03jcrespo [10:35:38] 10DBA, 10Phabricator: db1048 has started lagging - https://phabricator.wikimedia.org/T151039#2805396 (10jcrespo) ``` Device Present ================ Virtual Drives : 1 Degraded : 0 Offline : 0 Physical Devices : 14 Disks : 12 Critical Di... [10:36:56] 10DBA, 10Phabricator: db1048 has started lagging - https://phabricator.wikimedia.org/T151039#2805396 (10Marostegui) ``` root@db1048:~# megacli -PDList -aALL | grep -e '^$' -e Slot -e Count Slot Number: 0 Media Error Count: 17 Other Error Count: 0 Predictive Failure Count: 32 Slot Number: 2 Media Error Cou... [10:37:29] 10DBA, 10Phabricator: db1048 has started lagging - https://phabricator.wikimedia.org/T151039#2805451 (10jcrespo) Yeah, both with SMART errors. [10:38:52] where is that guide where I can see the physical arragement of disks? [10:39:26] https://wikitech.wikimedia.org/wiki/MegaCli ? [10:39:31] yes, thanks [10:41:04] they are on different spans, at least [10:41:08] \o/ [10:41:22] thank godness [10:42:38] 10DBA: db1048 has started lagging - https://phabricator.wikimedia.org/T151039#2805490 (10jcrespo) a:05jcrespo>03None This is 99% sure a disk issue. [10:44:06] 10DBA: Media errors on db1048 are creating lag - https://phabricator.wikimedia.org/T151039#2805494 (10jcrespo) [10:45:41] 10DBA: Media errors on db1048 are creating lag - https://phabricator.wikimedia.org/T151039#2805396 (10jcrespo) [10:54:22] 10DBA, 10Edit-Review-Improvements-RC-Page, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), 13Patch-For-Review: Implement functionality for RC page 'Experience level' filters - https://phabricator.wikimedia.org/T149637#2805529 (10Pginer-WMF) >>! In T149637#2804020, @jmatazzoni wrote: > - Once t... [11:10:06] 10DBA: Media errors on db1048 are creating lag - https://phabricator.wikimedia.org/T151039#2805577 (10jcrespo) BTW, this is the best graph to see this issues (assuming the load has not changed): https://grafana.wikimedia.org/dashboard/db/mysql?var-dc=eqiad%20prometheus%2Fops&var-server=db1048&from=1479078536607&... [11:12:59] 10DBA, 10MediaWiki-JobQueue, 10MediaWiki-extensions-Renameuser: Fix use of DB schema so RenameUser is trivial - https://phabricator.wikimedia.org/T33863#2805583 (10MarcoAurelio) [11:17:54] 10DBA, 10Edit-Review-Improvements-RC-Page, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), 13Patch-For-Review: Implement functionality for RC page 'Experience level' filters - https://phabricator.wikimedia.org/T149637#2805589 (10jcrespo) >>! In T149637#2799749, @SBisson wrote: > Pinging @jcrespo... [11:22:14] 10DBA, 10Cognate, 10Wikidata, 15User-Addshore, 03WMDE-QWERTY-Team-Board: Cognate DB review - https://phabricator.wikimedia.org/T148988#2805597 (10Addshore) Final patch modifying the schema can now be found at https://gerrit.wikimedia.org/r/#/c/320743/ Ready for review. [11:27:30] 10DBA, 10Edit-Review-Improvements-RC-Page, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), 13Patch-For-Review: Implement functionality for RC page 'Experience level' filters - https://phabricator.wikimedia.org/T149637#2805622 (10jcrespo) > If so, could you help us find a better strategy? Use `... [11:36:43] 10DBA, 10Edit-Review-Improvements-RC-Page, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), 13Patch-For-Review: Implement functionality for RC page 'Experience level' filters - https://phabricator.wikimedia.org/T149637#2805673 (10Pginer-WMF) >>! In T149637#2804020, @jmatazzoni wrote: > I also wro... [11:42:11] 10DBA, 13Patch-For-Review: db2034: investigate its crash and reimage - https://phabricator.wikimedia.org/T149553#2805705 (10jcrespo) When/if this fixes, we will want to copy it from db2042 for partitioning reasons, but only when db2042 itself gets its disks fixed. [11:44:10] 10DBA, 13Patch-For-Review: db2034: investigate its crash and reimage - https://phabricator.wikimedia.org/T149553#2805706 (10jcrespo) In fact, I believe this crashed again? [11:52:48] 10DBA: duplicate key problems on s4 - https://phabricator.wikimedia.org/T151029#2805748 (10jcrespo) With ALGORITH=COPY, it worked without problems: ``` MariaDB [commonswiki]> SET SESSION innodb_lock_wait_timeout=1; SET SESSION lock_wait_timeout=60; set session sql_log_bin=0; alter table page ADD COLUMN page_lan... [11:57:29] 10DBA, 13Patch-For-Review: db2034: investigate its crash and reimage - https://phabricator.wikimedia.org/T149553#2805754 (10Marostegui) Hey @Papaul The server crashed again - twice. :-) ``` iLO Standard 2.50 at Sep 23 2016 Server Name: Server Power: Off iLO Standard 2.50 at Sep 23 2016 Server Name: Serv... [12:16:41] 10DBA, 06Labs, 10Tool-Labs, 13Patch-For-Review: Provisioning MySQL replica users fails on tool labs - https://phabricator.wikimedia.org/T151014#2804720 (10Marostegui) users created in labsdb1001 and labsdb1003 for both, labstore1004 and labstore1005 ``` root@neodymium:~# host labstore1004; host labstore10... [12:30:54] 10DBA: duplicate key problems on s4 - https://phabricator.wikimedia.org/T151029#2805814 (10Marostegui) Are you going to file a bug report to MariaDB? [12:32:18] 10DBA: duplicate key problems on s4 - https://phabricator.wikimedia.org/T151029#2805815 (10jcrespo) No, I cannot replicate, and as long as the alter gets reverted, I am not even sure this is a bug. [12:32:58] 10DBA: duplicate key problems on s4 - https://phabricator.wikimedia.org/T151029#2805816 (10jcrespo) Although it would be nice to correlate with version numbers. [13:51:53] 10DBA, 06Operations, 10ops-codfw: Degraded RAID on db2035 - https://phabricator.wikimedia.org/T150973#2805997 (10Marostegui) 05Open>03Invalid We are following the latest events of this box here: T150511#2805005 [14:52:46] 10DBA, 10Datasets-General-or-Unknown, 06Release-Engineering-Team, 13Patch-For-Review, and 2 others: Automatize the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459#2806107 (10hashar) [15:00:56] 10DBA, 10Edit-Review-Improvements-RC-Page, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), 13Patch-For-Review: Implement functionality for RC page 'Experience level' filters - https://phabricator.wikimedia.org/T149637#2806165 (10SBisson) >>! In T149637#2805622, @jcrespo wrote: >> If so, could y... [15:11:05] 10DBA, 10Edit-Review-Improvements-RC-Page, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), 13Patch-For-Review: Implement functionality for RC page 'Experience level' filters - https://phabricator.wikimedia.org/T149637#2806197 (10Halfak) For what it's worth, I've requested that an index be made o... [15:56:28] 10DBA, 06Operations, 10ops-eqiad: labsdb1009 boot issues (power supply and controller?) - https://phabricator.wikimedia.org/T150211#2806307 (10Cmjohnson) Replaced the PSU, return shipment tracking is 1ZW0948Y9081215654 [16:12:05] 10DBA, 06Operations, 10ops-eqiad: labsdb1009 boot issues (power supply and controller?) - https://phabricator.wikimedia.org/T150211#2806370 (10jcrespo) Sadly, it still doesn't allow to boot from the disk device, and when going to to the hp raid configuration utility it says: ``` error: no such device: HPEZC... [16:24:04] 10DBA, 10MediaWiki-API, 10MediaWiki-Database, 07Performance: ApiQueryExtLinksUsage::run query has crazy limit - https://phabricator.wikimedia.org/T59176#2806419 (10Anomie) Apologies for speaking inexactly, my local "MySQL" instance actually is MariaDB. More specifically, it's from [[https://packages.debian... [16:29:09] 10DBA, 10MediaWiki-API, 10MediaWiki-Database, 07Performance: ApiQueryExtLinksUsage::run query has crazy limit - https://phabricator.wikimedia.org/T59176#2806429 (10jcrespo) Oh, no, that is great- then I have no further comments, let's go with it. the problem with externallinks.ibd size is not the indexes,... [16:37:12] 10DBA, 10MediaWiki-API, 10MediaWiki-Database, 07Performance: ApiQueryExtLinksUsage::run query has crazy limit - https://phabricator.wikimedia.org/T59176#603671 (10Anomie) a:03Anomie [17:07:44] 10DBA, 13Patch-For-Review: db2034: investigate its crash and reimage - https://phabricator.wikimedia.org/T149553#2806554 (10Papaul) @Marostegui Thannks for contact HP again and update you. [17:22:41] 07Blocked-on-schema-change, 10DBA, 10Wikimedia-Site-requests, 06Wikisource, and 2 others: Schema change for page content language - https://phabricator.wikimedia.org/T69223#2806589 (10jcrespo) This is the list of changes that failed to apply and failure was preferred over causing production issue (plus the... [17:34:48] 10DBA, 06Operations, 10ops-eqiad: labsdb1009 boot issues (power supply and controller?) - https://phabricator.wikimedia.org/T150211#2778014 (10fgiunchedi) I was confused by that message too @jcrespo, though it is sufficient to wait for the underlying linux to fully boot. You'll be dropped into `hpssacli` aft... [17:48:53] 10DBA, 06Operations, 10ops-eqiad: labsdb1009 boot issues (power supply and controller?) - https://phabricator.wikimedia.org/T150211#2806644 (10jcrespo) Thanks, @fgiunchedi, but I have not advanced much: ``` => controller slot=0 rescan => controller slot=1 pd all show status Error: The specifie... [17:59:06] 10DBA, 06Operations, 10ops-eqiad: labsdb1009 boot issues (power supply and controller?) - https://phabricator.wikimedia.org/T150211#2806649 (10jcrespo) Compare with the equivalent, well-working, labsdb1010: ``` => controller slot=1 pd all show status physicaldrive 1I:1:1 (port 1I:box 1:bay 1, 1600.3 GB... [18:17:19] 10DBA, 06Labs, 10Tool-Labs, 13Patch-For-Review: Provisioning MySQL replica users fails on tool labs - https://phabricator.wikimedia.org/T151014#2806678 (10yuvipanda) Thanks a lot @Marostegui! I think we should keep the labstore1001 / 2 ones enabled for now, in the unlikely event we have to roll back our mi... [18:32:03] marostegui: whenever you have a second dude, could you possibly look at what the grants are for labsdbadmin on lasdb*? It's teh user that is used to create creds throughout tools and the mechanism is fairly terrible. I'm wondering if it can drop users and update passwords [18:33:17] chasemp, it should be able to [18:34:00] ok I tried to drop a test account to no avail [18:34:11] what does it say? [18:34:13] not sure if me, I'll do a bit of digging and make a task [18:34:41] oh, yes [18:34:48] it has create user with grant option [18:34:53] but no other grant [18:34:56] let me fix that [18:35:09] the feedback now is: pymysql.err.InternalError: (1396, "Operation DROP USER failed for 'cbench'@'%'") [18:35:17] where cbench is just a test tool I own [18:35:50] create a task anyway, even ig it is a one line [18:35:54] kk [18:35:55] I will know that it is [18:35:59] *what [18:36:36] actually [18:36:56] "CREATE USER Enable use of CREATE USER, DROP USER, RENAME USER, and REVOKE ALL PRIVILEGES" [18:38:12] 10DBA, 06Labs: labsdbadmin needs to be able to drop users and update passwords - https://phabricator.wikimedia.org/T151076#2806773 (10chasemp) [18:38:17] I'm not sure what that means exactly [18:38:26] 10DBA, 06Labs: labsdbadmin needs to be able to drop users and update passwords - https://phabricator.wikimedia.org/T151076#2806785 (10chasemp) p:05Triage>03Normal [18:38:28] https://phabricator.wikimedia.org/T151076 [18:38:34] that there are no extra privileges for drop user [18:38:43] if you have create user, you can drop user too [18:38:51] with no extra grants needed [18:39:24] huh, I was attemping w/ a pre-existing delete-dbuser script [18:39:33] I'll have to dig further to see where that's messed up [18:39:35] do you want me to try and drop that user? [18:39:44] just to confirm everthing is ok? [18:39:58] as labsdbadmin? yes please [18:40:37] well, that user doesn't exist on labsdb1001 [18:40:47] so maybe it was that [18:41:37] same for labsdb1005 [18:41:45] labsdbadmin [18:41:46] ? [18:41:56] no, cbench doesn't exist at all [18:41:58] ah [18:42:05] so I cannot drop it [18:42:18] my test account is a victim of hte problem I'm looking to solve by testing w/ the test account [18:42:21] that's pretty meta [18:42:24] ok thanks jynus [18:42:45] I believe labsdbadmin can drop users already [18:42:56] ok cool [18:43:10] reopen the ticket even if it is for "I need db help" [18:43:20] will do [18:44:34] 10DBA, 06Labs: labsdbadmin needs to be able to drop users and update passwords - https://phabricator.wikimedia.org/T151076#2806809 (10jcrespo) 05Open>03Resolved a:03jcrespo I believe labsdbadmin can drop users already ok cool reopen the ticket even if it is for "I need db help"... [20:30:54] 10DBA, 06Operations, 10ops-codfw: db2035: RAID disk about to fail - https://phabricator.wikimedia.org/T150511#2807112 (10Papaul) Dear Mr Papaul Tshibamba, Thank you for contacting Hewlett Packard Enterprise for your service request. This email confirms your request for service and the details are below. Yo... [20:33:27] 10DBA, 13Patch-For-Review: db2034: investigate its crash and reimage - https://phabricator.wikimedia.org/T149553#2807115 (10Papaul) I talked with HP , they will send a replacement main board. Hello Papaul, As discussed, We would try replacing the system board on the server and monitor it. I will be setting... [22:48:18] 10DBA, 10Edit-Review-Improvements-RC-Page, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), 13Patch-For-Review: Implement functionality for RC page 'Experience level' filters - https://phabricator.wikimedia.org/T149637#2807388 (10jmatazzoni) @Pginer-WMF writes: > We may want to also capture the g...