[06:26:58] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3743536 (10Marostegui) [06:35:57] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3743543 (10Marostegui) 05Open>03Resolved The only two pending hosts to optimize are db1052 (s1 master) and db1068 (s4 master). The tables there ar... [06:37:06] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3743545 (10Marostegui) [06:40:59] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3743546 (10Marostegui) Going to start this for the wikis where it is enabled: https://github.com/wikimedia/oper... [06:46:27] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3743549 (10Marostegui) [06:48:43] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3743551 (10Marostegui) a:03Marostegui [07:56:53] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3743571 (10Marostegui) I have altered: s1,s2,s3,s6 and s7 in codfw (with replication) s5 is pending on codfw as... [09:18:58] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3743754 (10Marostegui) s7 eqiad is also done. [09:19:14] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3743755 (10Marostegui) [10:06:25] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3743817 (10Marostegui) s6 eqiad is done. [10:06:35] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3743818 (10Marostegui) [11:05:46] 10DBA, 10Analytics, 10Operations, 10User-Elukey: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3743903 (10elukey) Just sent a summary of what's happening to engineering@ and analytics@, new deadlines: ``` - November 13th: the analytics-slave CNAME move... [11:15:05] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3743942 (10Marostegui) s2 eqiad is done [11:15:24] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3743943 (10Marostegui) [11:40:24] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3744047 (10Marostegui) [11:40:33] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3490037 (10Marostegui) s3 eqiad is done [11:44:17] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3744054 (10Marostegui) s1 eqiad is done [11:44:26] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3744055 (10Marostegui) [12:23:06] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3744152 (10Marostegui) Until the other big alter is done in codfw s5, I am altering slaves in s5 eqiad. Once co... [12:26:48] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1059 - https://phabricator.wikimedia.org/T179727#3744155 (10Marostegui) @Cmjohnson if you have some time today, could we the failed disk swapped? Thank you! [12:32:31] 10DBA, 10Cloud-Services: archive/archive_userindex is not filled in eswiki_p - https://phabricator.wikimedia.org/T133251#3744163 (10Marostegui) 05Open>03Resolved a:03Marostegui This works with the new replicas, please switch to use them (https://phabricator.wikimedia.org/phame/post/view/70/new_wiki_repli... [12:34:52] cumin [13:09:03] marostegui: I ran a check for apt sources in production which are not managed by puppet and there's three db hosts among them: [13:09:06] https://phabricator.wikimedia.org/P6286 [13:10:27] ah, had a look, those are actually commented out and carry the note "disabled on upgrade to precise" :-) [13:11:55] so I'll remove them [15:55:28] 10Blocked-on-schema-change, 10MediaWiki-extensions-ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Review and deploy schema change on dropping oresc_rev_predicted_model index - https://phabricator.wikimedia.org/T180045#3744862 (10Ladsgroup) [15:55:41] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Review and deploy schema change on dropping oresc_rev_predicted_model index - https://phabricator.wikimedia.org/T180045#3744862 (10Ladsgroup) a:05Ladsgroup>03None [17:01:26] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1059 - https://phabricator.wikimedia.org/T179727#3745078 (10Cmjohnson) Disk has been swapped [17:04:15] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Review and deploy schema change on dropping oresc_rev_predicted_model index - https://phabricator.wikimedia.org/T180045#3745087 (10Marostegui) [17:05:14] marostegui: I forgot to mention that this patch just got merged, it's not deployed yet. Is it a blocker? [17:05:38] Amir1: not really, we normally wait till it is merged [17:06:35] it will get deployed next week I assume? [17:06:39] yes [17:06:51] oh, thanks for clarification [17:06:53] sure, I wasn't planning to start the alters this week, as tomorrow is a public holiday here in madrid :) [17:07:39] cool, Have fun! [17:07:50] thanks! :) [17:19:25] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1059 - https://phabricator.wikimedia.org/T179727#3745134 (10Marostegui) It failed again, was this a brand new disk, @Cmjohnson? ``` root@db1059:~# megacli -pdlist -a0 Adapter #0 Enclosure Device ID: 32 Slot Number: 0 Drive's position: DiskGroup: 0, Span:... [17:23:04] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3745141 (10Marostegui) Only pending: s5 primary master (db1063) s5 codfw (will be done with replication once t... [17:45:11] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3745180 (10Marostegui) [17:55:47] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3745203 (10Marostegui) s5 codfw is done [18:49:12] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad: Decommission db1033 and db1028 - https://phabricator.wikimedia.org/T174076#3745566 (10Cmjohnson) [18:49:43] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad: Decommission db1035 - https://phabricator.wikimedia.org/T176931#3745571 (10Cmjohnson) [18:50:10] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: decommission db1036 - https://phabricator.wikimedia.org/T176311#3745574 (10Cmjohnson) [18:52:07] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1037 - https://phabricator.wikimedia.org/T174902#3745579 (10Cmjohnson) [18:52:27] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1038 - https://phabricator.wikimedia.org/T177911#3745582 (10Cmjohnson) [20:36:10] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad: Decommission db1033 and db1028 - https://phabricator.wikimedia.org/T174076#3745843 (10Cmjohnson) [20:36:34] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad: Decommission db1035 - https://phabricator.wikimedia.org/T176931#3745845 (10Cmjohnson) [20:36:51] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: decommission db1036 - https://phabricator.wikimedia.org/T176311#3745846 (10Cmjohnson) [20:37:08] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1037 - https://phabricator.wikimedia.org/T174902#3745847 (10Cmjohnson) [20:37:29] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1038 - https://phabricator.wikimedia.org/T177911#3745850 (10Cmjohnson) [20:38:02] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1041 - https://phabricator.wikimedia.org/T173915#3745851 (10Cmjohnson) [20:43:38] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1059 - https://phabricator.wikimedia.org/T179727#3745857 (10Cmjohnson) 05Open>03Resolved Looks like the rebuild is complete and all disks are back online root@db1059:~# megacli -PDList -aALL |grep "Firmware state" Firmware state: Online, Spun Up Firmw... [20:53:56] 10DBA, 10Operations, 10Phabricator, 10ops-eqiad: Decommission db1048 (was Move m3 slave to db1059) - https://phabricator.wikimedia.org/T175679#3745890 (10Cmjohnson) @jcrespo @marostegui is it safe to finish off db1048? [20:58:02] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1015 - https://phabricator.wikimedia.org/T173570#3745891 (10Cmjohnson) @marostegui during my decom checks I found db1015 in this file. Should a replacement be identified? modules/admin/files/enforce-users-groups.sh [21:46:46] 10DBA, 10Analytics, 10Operations, 10User-Elukey: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#2987618 (10Tgr) Does that mean it's not going to be possible to JOIN EventLogging tables with MediaWiki tables in the future? I'm not working with user-facing... [21:52:35] 10DBA, 10Analytics, 10Operations, 10User-Elukey: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3746008 (10Nuria) @Tgr What types of selects were you doing? We think the best place to do this type of joining is hadoop, we are working into refining EL d...