[07:23:26] 10DBA: hitcounter and _counter tables are on the cluster but were deleted/unsused? - https://phabricator.wikimedia.org/T132837#2711972 (10Marostegui) Tables dropped from S5 (dewiki, wikidatawiki) [07:27:55] 10DBA: hitcounter and _counter tables are on the cluster but were deleted/unsused? - https://phabricator.wikimedia.org/T132837#2711981 (10Marostegui) I have dropped the tables in S6 master (db1050) wikis: `frwiki jawiki ruwiki` ``` root@neodymium:/home/marostegui/git/software/dbtools# for i in frwiki jawiki ruw... [08:42:50] 10DBA, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2712068 (10hashar) @mmodell seems the way search now works is the source of lot of confusion... [08:59:44] 10DBA, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2712108 (10mmodell) [09:14:19] 10DBA, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2712139 (10hashar) From a conversation with Mukunda, D413 should address it and is deployed... [11:41:31] 10DBA, 13Patch-For-Review: Reimage dbstore2001 as jessie - https://phabricator.wikimedia.org/T146261#2712493 (10jcrespo) Allow me to suggest keeping the original tar as I think that migration is possible, but the opposite is not. For testing, s3 is a bad candidate, as just mysql_upgrade may take 30-minutes/1-h... [11:43:00] 10DBA, 13Patch-For-Review: Reimage dbstore2001 as jessie - https://phabricator.wikimedia.org/T146261#2712511 (10jcrespo) Adding @chasemp so he is in the loop because even if the ticket is not nominatively related to labs, we are testing the same procedure we are about to apply for the new labsdbs there. [11:49:19] 10DBA, 10CirrusSearch, 06Discovery, 06Discovery-Search: CirrusSearch SQL query for locating pages for reindex performs poorly - https://phabricator.wikimedia.org/T147957#2712534 (10jcrespo) [12:20:02] 10DBA, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2712578 (10Paladox) I found an ssh app on the iPhone so I used that. I have installed this h... [12:48:10] 10DBA, 13Patch-For-Review: Multiple Puppet class make MySQL load /etc/my.cnf twice - https://phabricator.wikimedia.org/T133780#2712629 (10jcrespo) 05Open>03Resolved a:03Volans Considered fixed unless someone complains of the opposite. [12:48:49] jynus: ack :) [12:48:58] 10DBA, 10Dumps-Generation, 10MediaWiki-Database, 10Utilities-mwdumper: Wikipedia requires a patch to load its data from the dumps with mwdumper - https://phabricator.wikimedia.org/T147148#2712633 (10jcrespo) p:05Triage>03Low Low because it is not yet verified or clear. [12:49:41] volans, I am going to try to clean up some tickets [12:50:14] if it is "mostly done" I will resolve it- it doesn't make sense to keep things open when 99% of the work is done, and the rest is not important. [12:50:33] I am lately having troubles searching for tickets [12:50:34] fair enough [12:50:54] jynus: Are you using "+" :p [12:51:13] * jynus eyerolls [12:51:21] XD [12:52:36] lol [12:53:12] 10DBA, 10MediaWiki-General-or-Unknown, 06Operations, 13Patch-For-Review, 05WMF-deploy-2016-10-11_(1.28.0-wmf.22): img_metadata queries for PDF files saturates s4 slaves - https://phabricator.wikimedia.org/T147296#2712641 (10jcrespo) As this is being worked at mediawiki level, I am going to move us into m... [12:53:43] I think I am going to generate some spam, but I can no longer see important tickets [12:53:52] on the pending queue [12:53:54] so many of them [12:54:32] Tomorrow we can try to organize that in our hangouts and see how we can handle them better/faster [12:54:58] I still struggle to identify which of those are important/Easy/doable/high priorty :( [12:57:58] yes [12:58:02] I was starting early [12:58:09] with the clear resolved ones [12:58:12] sure [12:58:16] or wontfix [12:58:17] that will help :) [12:58:47] I think there must be some "it is almost finished, but keeping ticket open" [12:58:54] and now resolved [12:58:58] like the first above [13:13:11] 10DBA: hitcounter and _counter tables are on the cluster but were deleted/unsused? - https://phabricator.wikimedia.org/T132837#2712669 (10Marostegui) S6 (ruwiki,jawiki,frwiki) got the tables removed. [14:40:21] 10DBA, 06Operations, 10ops-eqiad: Physically move db1053 to a different rack - https://phabricator.wikimedia.org/T147774#2712933 (10Marostegui) Server downtimed MySQL stopped Server powered off [14:44:11] 10DBA, 13Patch-For-Review: Reimage dbstore2001 as jessie - https://phabricator.wikimedia.org/T146261#2712950 (10Marostegui) dbstore2001 is now working and replicating fine using GTID. I have installed `10.0.27-1` and found something which I guess isn't expected? ``` root@dbstore2001:/opt/wmf-mariadb10# /etc/i... [14:47:12] 10DBA, 13Patch-For-Review: Unify commonswiki.revision - https://phabricator.wikimedia.org/T147305#2712953 (10Marostegui) db1068 is now done: ``` MariaDB PRODUCTION s4 localhost commonswiki > show create table revision\G *************************** 1. row *************************** Table: revision Crea... [15:15:42] jynus: we have libdbd-mysql-perl installed on all db servers, where is gets pulled in via percona-toolkit and mha4mysql-node, should be harmless to update (for current security update) or should I deploy this on some canary hosts first? [15:16:04] it may be a dependency for mysql, too [15:16:12] (it has been added to the new packages) [15:16:16] but a passive one [15:16:30] like, scripts that are rarely run, so just go on [15:16:40] ok, will do [15:16:50] try to not do 100% at the same time [15:16:58] but a couple of passes is ok [15:17:07] sure, I'll deploy these in stages [15:17:24] I doublechecked with the mysql package in Debian; it doesn't depend/recommend it either [15:17:35] oh, sorry [15:18:56] it is libdbi-perl [15:19:06] which may be in common for jessie [15:19:20] but common, client and server are all togheder on our packages [15:19:23] but same thing applies [15:19:43] not a hard dependency, it is only for certain maintenance scripts [15:19:48] 10DBA, 06Operations, 10ops-eqiad, 13Patch-For-Review: Physically move db1053 to a different rack - https://phabricator.wikimedia.org/T147774#2713063 (10Cmjohnson) 05Open>03Resolved a:03Cmjohnson @Marostegui db1053 has been moved to A2 DNS updated Switch Cfg updated Racktables updated [15:21:25] 10DBA, 13Patch-For-Review: Reimage dbstore2001 as jessie - https://phabricator.wikimedia.org/T146261#2713067 (10jcrespo) That is not a bug, that is intended. But I am having issues here: https://gerrit.wikimedia.org/r/315228 [15:32:36] 10DBA, 06Operations, 13Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#2713121 (10Cmjohnson) [15:55:49] 10DBA, 06Operations, 10ops-eqiad, 13Patch-For-Review: Physically move db1053 to a different rack - https://phabricator.wikimedia.org/T147774#2713220 (10Marostegui) /etc/network/interfaces changed to reflect the new IP. All good - thanks Chris. [16:00:57] I can see the lag going down on s1: https://grafana-admin.wikimedia.org/dashboard/db/mysql-aggregated [16:01:25] (for the rest of the poeple, on an under-maintenance host) [16:16:57] 10DBA, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2713364 (10mmodell) Thanks @paladox! I'll reindex. [17:21:28] 10DBA, 10CirrusSearch, 06Discovery, 06Discovery-Search: CirrusSearch SQL query for locating pages for reindex performs poorly - https://phabricator.wikimedia.org/T147957#2713641 (10Smalyshev) [17:59:16] 10DBA, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2713758 (10Paladox) Your welcome, I guess we reindex twice now :) [18:00:44] 10DBA, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2713762 (10Paladox) We should deftly deploy this If this improves things more. [18:13:45] 10DBA, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2713790 (10Paladox) @mmodell are we running reindexing on iridium, since It looks like it wor... [18:14:43] 10DBA, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2713792 (10mmodell) @paladox: reindexing already happened a while ago. [18:15:24] 10DBA, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2713794 (10Paladox) Oh, so we doint need to reindex with your change? [18:17:23] 10DBA, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2713829 (10mmodell) Right. This just changes the query to always include + before every word.... [18:19:13] 10DBA, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2713850 (10Paladox) Ok, thankyou. [19:46:43] 10DBA, 06Operations, 13Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#2714212 (10jcrespo) [19:46:46] 10DBA, 06Operations: db1034 decommission - https://phabricator.wikimedia.org/T139280#2714209 (10jcrespo) 05Open>03Resolved a:03jcrespo Will create a separate task when we stop using it, it is currently on full production. [19:49:47] 10DBA, 06Operations: Decommission db1035 - https://phabricator.wikimedia.org/T148078#2714228 (10jcrespo) [19:52:05] 10DBA, 06Operations: Decommission db1015, db1035 and db1044 - https://phabricator.wikimedia.org/T148078#2714243 (10jcrespo) [22:52:18] 10DBA, 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2714880 (10mmodell)