[02:20:15] 10DBA, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences, 10Patch-For-Review, 10Schema-change: DBA review for GlobalPreferences schema - https://phabricator.wikimedia.org/T184666#4025804 (10MaxSem) > Maybe some preferences that are now the default can be detected and cleaned up? We have `cleanu... [02:20:35] 10DBA, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences, 10Patch-For-Review, 10Schema-change: DBA review for GlobalPreferences schema - https://phabricator.wikimedia.org/T184666#3891821 (10kaldari) >So this has several concerns, mostly related to "dead rows"- basically data on the database that... [02:23:12] 10DBA, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences, 10Patch-For-Review, 10Schema-change: DBA review for GlobalPreferences schema - https://phabricator.wikimedia.org/T184666#4025813 (10MaxSem) >>* Deleted global preferences will also be kept, in case they are reenabled > This should probabl... [06:58:11] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1068 - https://phabricator.wikimedia.org/T188187#4026142 (10Marostegui) Almost there: ``` root@db1068:~# megacli -PDRbld -ShowProg -PhysDrv [32:9] -aALL Rebuild Progress on Device at Enclosure 32, Slot 9 Completed 97% in 920 Minutes. ``` [07:28:44] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1068 - https://phabricator.wikimedia.org/T188187#4026164 (10Marostegui) 05Open>03Resolved ``` ˜/icinga-wm 8:26> RECOVERY - MegaRAID on db1068 is OK: OK: optimal, 1 logical, 2 physical, WriteBack policy ``` ``` root@db1068:~# megacli -LDPDInfo -aAll A... [07:34:34] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 eqiad machines - https://phabricator.wikimedia.org/T183469#4026166 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['db2037.codfw.wmnet'] ``` Th... [08:02:59] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 eqiad machines - https://phabricator.wikimedia.org/T183469#4026189 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db2037.codfw.wmnet'] ``` and were **ALL** successful. [11:00:59] 10DBA, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences, 10Patch-For-Review, 10Schema-change: DBA review for GlobalPreferences schema - https://phabricator.wikimedia.org/T184666#4026455 (10jcrespo) > While I agree that figuring out how to optimize preference storage is something we need to do,... [12:55:58] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 eqiad machines - https://phabricator.wikimedia.org/T183469#4026799 (10Marostegui) I would like to failover m5 to get rid of db1009 and make db1073 the master. This is what I had in mind 1- merge: https://gerri... [12:56:15] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Epic: Make wb_terms table fancy - https://phabricator.wikimedia.org/T188992#4026732 (10Ladsgroup) [12:59:04] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Epic: Make wb_terms table fancy - https://phabricator.wikimedia.org/T188992#4026810 (10Lucas_Werkmeister_WMDE) > Normalize term_text column Should we normalize the text on its own or the tuple of text and language code? I feel like treating t... [13:06:16] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Epic: Make wb_terms table fancy - https://phabricator.wikimedia.org/T188992#4026839 (10Ladsgroup) It actually makes a big difference as there are lots of cases when someone adds label for all languages at the same time, for humans. [[https://w... [13:07:47] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Epic: Make wb_terms table fancy - https://phabricator.wikimedia.org/T188992#4026841 (10Lucas_Werkmeister_WMDE) Ah, good point, I didn’t think about labels :) [13:15:06] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 eqiad machines - https://phabricator.wikimedia.org/T183469#4026866 (10jcrespo) Please note m5-master is not pointing to dbproxy1005. Ideally it should (after the failover happens), but with T188589 ongoing, I d... [13:17:47] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 eqiad machines - https://phabricator.wikimedia.org/T183469#4026868 (10Marostegui) >>! In T183469#4026866, @jcrespo wrote: > Please note m5-master is not pointing to dbproxy1005. Ideally it should (after the fai... [13:18:43] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 eqiad machines - https://phabricator.wikimedia.org/T183469#4026873 (10jcrespo) > Yeah, that is why I thought about completely stop MySQL on db1009 apart from the read_only=ON Depending on the implementation th... [13:19:44] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 eqiad machines - https://phabricator.wikimedia.org/T183469#4026875 (10Marostegui) >>! In T183469#4026873, @jcrespo wrote: >> Yeah, that is why I thought about completely stop MySQL on db1009 apart from the read... [13:25:54] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 eqiad machines - https://phabricator.wikimedia.org/T183469#4026912 (10Marostegui) To reflect the latest comments: merge: https://gerrit.wikimedia.org/r/#/c/416650/ move db2037 under db1073 merge: https://gerri... [13:32:39] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 eqiad machines - https://phabricator.wikimedia.org/T183469#4026941 (10jcrespo) s/grab binlog position for db1073 just in case/grab it from both in case we want to repoint the original master "just in case"/ Af... [13:35:12] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 eqiad machines - https://phabricator.wikimedia.org/T183469#4026946 (10Marostegui) >>! In T183469#4026941, @jcrespo wrote: > s/grab binlog position for db1073 just in case/grab it from both in case we want to re... [14:04:14] 10DBA, 10Operations, 10cloud-services-team: Failover m5 master from db1009 to db1073 - https://phabricator.wikimedia.org/T189005#4027038 (10Marostegui) p:05Triage>03Normal [16:07:06] Amir1: around? [16:09:10] oh, I was wrong, ignore my ping- sorry [16:18:06] 10DBA, 10MediaWiki-Database, 10Wikidata: IndexPager::buildQueryInfo (contributions page unfiltered) query needs tuning - https://phabricator.wikimedia.org/T189026#4027715 (10jcrespo) [16:19:33] 10DBA, 10MediaWiki-Database, 10Wikidata: IndexPager::buildQueryInfo (contributions page unfiltered) query needs tuning - https://phabricator.wikimedia.org/T189026#4027729 (10jcrespo) Adding Anomie hoping he will not get mad at me 0:-) but he was involved with Springle last time. [16:29:36] 10DBA, 10MediaWiki-Database, 10Wikidata: IndexPager::buildQueryInfo (contributions page unfiltered) query needs tuning - https://phabricator.wikimedia.org/T189026#4027748 (10jcrespo) [16:34:34] 10DBA, 10Patch-For-Review, 10Wikimedia-Incident: Investigate why query killer didn't kill 1-hour long queries - https://phabricator.wikimedia.org/T188505#4027753 (10jcrespo) 05Open>03Resolved @Lucas_Werkmeister_WMDE I am going to resolve this ticket once it has been deployed to all of s8 (wikidata databa... [16:36:36] 10DBA, 10Operations, 10cloud-services-team: Failover m5 master from db1009 to db1073 - https://phabricator.wikimedia.org/T189005#4027759 (10chasemp) a:03Andrew @andrew volunteered to help steer this change :) [16:57:02] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: dbstore1001 crashed: Multibit ECC errors were detected on the RAID controller. - https://phabricator.wikimedia.org/T186596#4027828 (10jcrespo) a:05jcrespo>03None [17:00:35] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: dbstore1001 crashed: Multibit ECC errors were detected on the RAID controller. - https://phabricator.wikimedia.org/T186596#4027837 (10jcrespo) p:05High>03Normal We have copied away everything we needed to keep- we are blocked on DC ops to do the ful... [17:07:45] 10DBA, 10MediaWiki-Database, 10Wikidata: IndexPager::buildQueryInfo (contributions page unfiltered) query needs tuning - https://phabricator.wikimedia.org/T189026#4027856 (10Anomie) ``` wikiadmin@db1101(wikidatawiki)>explain SELECT rev_id,rev_page,rev_text_id,rev_timestamp,rev_minor_edit,rev_deleted,rev_len... [17:17:47] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Patch-For-Review: IndexPager::buildQueryInfo (contributions page unfiltered) query needs tuning - https://phabricator.wikimedia.org/T189026#4027906 (10jcrespo) I am all for deploying fast, seeing how queries react, reevaluate if needed, rather than being stationar... [17:38:46] jynus, marostegui: ok to reboot dbmonitor or is it a bad time? [17:39:49] can you do dbmonitor2001, then 1001 ? [17:40:12] sure, can do [17:40:16] or is it for a host restart? [17:40:31] reboot of dbmonitor2001 and dbmonitor1001 for spectre [17:40:38] cool [17:40:53] starting in a few minute, will log in operations [17:40:59] thanks [17:48:39] both rebooted, https://tendril.wikimedia.org/ working fine [17:50:13] thanks [17:51:49] 10DBA, 10Operations, 10Patch-For-Review: Create less overhead on bacula jobs when dumping production databases - https://phabricator.wikimedia.org/T162789#3174710 (10jcrespo) a:03jcrespo This is happening at the same time than T184696 [18:04:48] 10DBA, 10Patch-For-Review, 10Wikimedia-Incident: Investigate why query killer didn't kill 1-hour long queries - https://phabricator.wikimedia.org/T188505#4028147 (10Lucas_Werkmeister_WMDE) @jcrespo okay, thanks a lot for your help! So far everything seems okay to me (no crazy server load spikes that I can se... [18:15:02] 10DBA, 10Reading List Service, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review, 10Reading-Infrastructure-Team-Backlog (Kanban): Update duplicate handling in reading lists API - https://phabricator.wikimedia.org/T184680#4028181 (10jcrespo) Moving to blocked until we are... [18:21:38] 10DBA, 10MediaWiki-Database: Investigation: Using log_search to query for logged actions against IPs in a given range - https://phabricator.wikimedia.org/T187584#4028198 (10jcrespo) I wonder if MySQL is the right storage for these, as there seems to be a need to search instantly for large ranges, and our searc... [18:30:14] 10DBA, 10Data-Services: Make Dispenser's principle_links table accessible in new Wiki replica cluster - https://phabricator.wikimedia.org/T180636#4028234 (10jcrespo) @Dispenser Can you point to the source code of what you use for generating such tables - I am mostly interested on the storage and data manipulat... [19:17:45] 10DBA, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences, 10Patch-For-Review, 10Schema-change: DBA review for GlobalPreferences schema - https://phabricator.wikimedia.org/T184666#4028731 (10kaldari) The impact of this feature on the user_properties table should be less than introducing a single... [19:21:55] 10DBA, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences, 10Patch-For-Review, 10Schema-change: DBA review for GlobalPreferences schema - https://phabricator.wikimedia.org/T184666#4028755 (10jcrespo) @mark T184666#4021193