[05:54:14] 10DBA, 06Operations, 10ops-codfw: db2058: Predictive RAID failure - https://phabricator.wikimedia.org/T165498#3272571 (10Marostegui) And the disk finally failed: T165629 ``` physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 600 GB, Failed) ``` Let's follow up on the failed task. [05:54:56] 10DBA, 06Operations, 10ops-codfw: db2058: Predictive RAID failure - https://phabricator.wikimedia.org/T165498#3272573 (10Marostegui) 05Open>03Invalid [05:55:57] 10DBA, 06Operations, 10ops-codfw: Degraded RAID on db2058 - https://phabricator.wikimedia.org/T165629#3272144 (10Marostegui) p:05Triage>03Normal a:03Papaul Please @Papaul proceed and change this disk when you have time. Thanks! [06:00:58] 10DBA, 13Patch-For-Review: Unify revision table on s2 - https://phabricator.wikimedia.org/T162611#3272580 (10Marostegui) db1069 is done: ``` root@neodymium:/home/marostegui# for i in `cat /home/marostegui/T162611`; do echo $i; mysql --skip-ssl -hdb1069 -P3312 $i -e "show create table revision\G";done bgwiki **... [06:01:56] 10DBA, 13Patch-For-Review: Unify revision table on s2 - https://phabricator.wikimedia.org/T162611#3272581 (10Marostegui) db2017 (codfw master) is done: ``` root@neodymium:/home/marostegui# for i in `cat /home/marostegui/T162611`; do echo $i; mysql --skip-ssl -hdb2017.codfw.wmnet $i -e "show create table revisi... [06:28:22] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3272592 (10Marostegui) `ptwiki` on codfw is now done after executing it on the master (dbstore2... [06:32:22] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3272593 (10Marostegui) `ptwiki` db1069 done and replicated downstream to the labs hosts (which... [06:44:08] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3272611 (10Marostegui) `ptwiki` is done on all hosts but (dbstore2001 and labs hosts, which wil... [06:45:36] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3272612 (10Marostegui) [07:03:23] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3272616 (10Marostegui) `plwiki` on codfw is now done after executing it on the master (dbstore2... [07:12:16] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3272633 (10Marostegui) `plwiki` db1069 done and replicated downstream to the labs hosts (which... [07:21:15] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3272686 (10Marostegui) `plwiki` is done on all hosts but (dbstore2001 and labs hosts, which wil... [07:21:24] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3272687 (10Marostegui) [07:56:57] db1069 alter is not ongoing, right? [07:57:09] nope, it is over [07:57:22] there is an alter ongoing on labsdb1001 [07:57:29] ok [07:57:34] so that must be it [08:15:19] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3272767 (10Marostegui) `enwiki` on codfw is now done after executing it on the master (dbstore2... [08:16:49] Can I run an alter on db1069? It will take around 1-2 minutes [08:17:39] yes. I am not touching it [08:17:48] ok! Thanks [08:24:21] there is heavy qps on s6, I think someone is parsing all pages again [08:24:42] last time it was enwiki no? [08:26:42] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, 15User-Ladsgroup: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3272781 (10Marostegui) db1069 and db1095 done and replicated downstream to the labs hosts: ```... [08:46:03] I am going to reimage db2051 [08:46:09] ok [08:49:47] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 4 others: Concerns about ores_classification table size on enwiki - https://phabricator.wikimedia.org/T159753#3272822 (10Marostegui) db1089 has been optimized: ``` root@db1089:/srv/sqldata/enwiki# ls -lh ores_class... [08:50:09] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 2 others: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3272823 (10Marostegui) `enwiki` db1089 is done: ``` root@neodymium:/home/marostegui/git/software/dbto... [08:54:16] 10DBA: Run pt-table-checksum on s7 - https://phabricator.wikimedia.org/T163190#3272829 (10Marostegui) centralauth checksum is done. There are tiny differences on: ``` Differences on db2047 centralauth.localuser Differences on db2054 centralauth.localuser Differences on db2061 centralauth.localuser Difference... [08:54:25] 10DBA: Run pt-table-checksum on s7 - https://phabricator.wikimedia.org/T163190#3272833 (10Marostegui) [09:00:05] 10DBA, 13Patch-For-Review: Unify revision table on s2 - https://phabricator.wikimedia.org/T162611#3272849 (10Marostegui) db1090 is done: ``` root@neodymium:/home/marostegui/git/software/dbtools# for i in `cat /home/marostegui/T162611`; do echo $i; mysql --skip-ssl -hdb1090 $i -e "show create table revision\G";... [09:16:27] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 4 others: Concerns about ores_classification table size on enwiki - https://phabricator.wikimedia.org/T159753#3272864 (10Marostegui) db1083 has been optimized: ``` root@db1083:/srv/sqldata/enwiki# ls -lh ores_class... [09:17:43] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 2 others: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3272866 (10Marostegui) `enwiki` db1083 is done: ``` root@neodymium:/home/marostegui/git/software/dbto... [09:41:46] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 4 others: Concerns about ores_classification table size on enwiki - https://phabricator.wikimedia.org/T159753#3272895 (10Marostegui) db1080 has been optimized: ``` root@db1080:/srv/sqldata/enwiki# ls -lh ores_class... [09:42:01] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 2 others: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3272896 (10Marostegui) `enwiki` db1080 is done: ``` root@neodymium:/home/marostegui/git/software/dbto... [09:50:33] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 2 others: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3272908 (10Marostegui) `enwiki` db1073 is done: ``` root@neodymium:/home/marostegui/git/software/dbto... [09:52:54] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 4 others: Concerns about ores_classification table size on enwiki - https://phabricator.wikimedia.org/T159753#3272911 (10Marostegui) db1073 has been optimized: ``` root@db1073:/srv/sqldata/enwiki# ls -lh ores_class... [11:33:18] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 2 others: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3273110 (10Marostegui) `enwiki` db1072 is done: ``` root@neodymium:/home/marostegui/git/software/dbto... [11:36:16] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 4 others: Concerns about ores_classification table size on enwiki - https://phabricator.wikimedia.org/T159753#3273125 (10Marostegui) db1072 has been optimized: ``` root@db1072:/srv/sqldata/enwiki# ls -lh ores_class... [12:33:06] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 13Patch-For-Review, and 2 others: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3273221 (10hoo) [12:38:14] 10DBA, 13Patch-For-Review: Unify revision table on s2 - https://phabricator.wikimedia.org/T162611#3273229 (10Marostegui) db1076 is done: ``` root@neodymium:/home/marostegui/git/software/dbtools# for i in `cat /home/marostegui/T162611`; do echo $i; mysql --skip-ssl -hdb1076 $i -e "show create table revision\G";... [12:53:06] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 2 others: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3273272 (10Marostegui) `enwiki` db1066 is done: ``` root@neodymium:/home/marostegui/git/software/dbto... [12:53:46] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 4 others: Concerns about ores_classification table size on enwiki - https://phabricator.wikimedia.org/T159753#3273274 (10Marostegui) db1066 has been optimized: ``` root@db1066:/srv/sqldata/enwiki# ls -lh ores_class... [12:56:27] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 2 others: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3273283 (10Marostegui) `enwiki` db1065 is done: ``` root@neodymium:/home/marostegui/git/software/dbto... [12:58:05] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 4 others: Concerns about ores_classification table size on enwiki - https://phabricator.wikimedia.org/T159753#3273288 (10Marostegui) db1065 has been optimized: ``` root@db1065:/srv/sqldata/enwiki# ls -lh ores_class... [13:00:25] 10DBA, 10Monitoring, 07Documentation, 13Patch-For-Review, 07Puppet: Document performance optimization of servermon and/or puppet reporting tools - https://phabricator.wikimedia.org/T164604#3273290 (10akosiaris) The patch above does document everything in the servermon.rb reporter (which is the applicatio... [13:03:35] 10DBA, 10Monitoring, 07Documentation, 13Patch-For-Review, 07Puppet: Document performance optimization of servermon and/or puppet reporting tools - https://phabricator.wikimedia.org/T164604#3273297 (10jcrespo) Let's close this as the scope is for me done, and let's open a new ticket with lower priority wi... [13:14:08] 10DBA, 06Operations: Investigate slow servermon updating queries on db1016 - https://phabricator.wikimedia.org/T165674#3273365 (10akosiaris) p:05Triage>03Lowest [13:15:00] 10DBA, 10Monitoring, 07Documentation, 13Patch-For-Review, 07Puppet: Document performance optimization of servermon and/or puppet reporting tools - https://phabricator.wikimedia.org/T164604#3239514 (10akosiaris) 05Open>03Resolved a:03akosiaris Agreed. Relevant stuff copied over to T165674 (marked lo... [13:15:48] 10DBA, 06Operations: Investigate slow servermon updating queries on db1016 - https://phabricator.wikimedia.org/T165674#3273350 (10akosiaris) Note that this is occuring seldomly and not causing any issues whatsoever. It's mostly out of personal interest that we are looking into this, hence the very low priority. [13:20:37] 10DBA, 10Monitoring, 07Documentation, 13Patch-For-Review, 07Puppet: Document performance optimization of servermon and/or puppet reporting tools - https://phabricator.wikimedia.org/T164604#3273391 (10jcrespo) The "separate task" I usually suggest in these cases has a double reason- it makes clear the sco... [13:42:14] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 4 others: Concerns about ores_classification table size on enwiki - https://phabricator.wikimedia.org/T159753#3273413 (10Marostegui) db1055 has been optimized: ``` root@db1055:/srv/sqldata/enwiki# ls -lh ores_class... [13:42:17] 07Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 06Scoring-platform-team, and 2 others: Deploy uniqueness constraints on ores_classification table - https://phabricator.wikimedia.org/T164530#3273414 (10Marostegui) `enwiki` db1055 is done: ``` root@neodymium:/home/marostegui/git/software/dbto... [14:58:31] 10DBA: Run pt-table-checksum on s7 - https://phabricator.wikimedia.org/T163190#3273503 (10Marostegui) centralauth is done: There are differences on the `archive` table on the hosts core hosts. And `dbstore1002` has differences on: ``` archive geo_tags logging user ``` [14:58:49] 10DBA: Run pt-table-checksum on s7 - https://phabricator.wikimedia.org/T163190#3273504 (10Marostegui) [15:18:00] 10DBA, 06Operations, 10ops-codfw: Degraded RAID on db2058 - https://phabricator.wikimedia.org/T165629#3273518 (10Papaul) Dear Mr Papaul Tshibamba, Thank you for contacting Hewlett Packard Enterprise for your service request. This email confirms your request for service and the details are below. Your reque... [15:18:26] 10DBA, 06Operations, 10ops-codfw: Degraded RAID on db2058 - https://phabricator.wikimedia.org/T165629#3273519 (10Marostegui) Thanks! [15:29:24] 10DBA, 13Patch-For-Review: Unify revision table on s2 - https://phabricator.wikimedia.org/T162611#3273532 (10Marostegui) db1074 is done: ``` root@neodymium:/home/marostegui/git/software/dbtools# for i in `cat /home/marostegui/T162611`; do echo $i; mysql --skip-ssl -hdb1074 $i -e "show create table revision\G";... [15:50:06] 10DBA, 103d, 06Multimedia, 13Patch-For-Review: Have search recognise STL files as a new kind of media file ('type:3d' or whatever) - https://phabricator.wikimedia.org/T157348#3273555 (10Jdforrester-WMF) Dear wonderful #DBA s, could one of you briefly look at https://gerrit.wikimedia.org/r/#/c/336454/ (espe... [16:02:51] 10DBA, 103d, 06Multimedia, 13Patch-For-Review: Have search recognise STL files as a new kind of media file ('type:3d' or whatever) - https://phabricator.wikimedia.org/T157348#3002239 (10jcrespo) I would have to test that I am right at the CR to provide a more accurate ETA. It will not be 100% trivial in an... [17:20:37] 10DBA, 06Operations, 10Pybal, 07Availability: Create a backend check for pybal to monitor the MySQL protocol being up - https://phabricator.wikimedia.org/T165677#3273775 (10jcrespo)