[06:36:52] 10DBA, 10Analytics, 10Analytics-Kanban, 10User-Elukey: Review dbstore1002's non-wiki databases and decide which ones needs to be migrated to the new multi instance setup - https://phabricator.wikimedia.org/T212487 (10Marostegui) Thanks @greg - I have done the mysqldump for it. @elukey I have left it at: `c... [06:39:12] 10DBA, 10Analytics, 10Analytics-Kanban, 10User-Elukey: Review dbstore1002's non-wiki databases and decide which ones needs to be migrated to the new multi instance setup - https://phabricator.wikimedia.org/T212487 (10Marostegui) [06:44:34] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Marostegui) [06:53:55] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Marostegui) [07:11:57] 10DBA, 10Patch-For-Review: Drop tag_summary table - https://phabricator.wikimedia.org/T212255 (10Marostegui) [07:17:17] 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui) [08:02:36] 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10elukey) [08:49:59] db1068 down to 4.001 errors [08:50:07] yeah [08:50:08] almost done! [09:58:45] which databases are you working now? [09:59:02] dbstore1003, db2089:3316 [09:59:15] mmm, I can see some extra down [09:59:24] just finished with db1113:3315 and db1113:3316 [09:59:42] ah, ok, so it was that [09:59:44] thanks [09:59:51] those should be up now [09:59:53] I just repooled [10:00:07] what about db1118, I would like to do some backup testing [10:00:13] all for you [10:00:16] can that be dropped? [10:00:24] totally [10:00:27] ok [10:00:29] thanks [10:00:35] you can do anything you like with it [10:00:35] didn't want to step on you [10:00:49] I may do that and maybe some codfw restarts [10:01:11] sure [10:01:35] I will probably take db1090:3317 too, that ok? [10:02:28] Actually no [10:02:31] yeah, I will not touch eqiad [10:02:31] I won't take that one [10:02:43] will do that on monday instead [10:02:47] that is why I was trying to sync with you [10:03:40] :) [10:20:59] 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui) [10:22:49] I will take db2087:3316 [10:28:30] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Marostegui) What's the plan for this? Is this T212346#4845733 ready? [10:31:28] 10DBA, 10Data-Services, 10Datasets-General-or-Unknown, 10User-notice: Archive and drop education program (ep_*) tables on all wikis - https://phabricator.wikimedia.org/T174802 (10Marostegui) [10:33:57] 10DBA, 10Data-Services, 10Datasets-General-or-Unknown, 10User-notice: Archive and drop education program (ep_*) tables on all wikis - https://phabricator.wikimedia.org/T174802 (10Marostegui) [10:47:29] 10DBA, 10Operations, 10Patch-For-Review: correctable memory errors db1068 (commons primary master database) - https://phabricator.wikimedia.org/T213664 (10jcrespo) 05Open→03Resolved a:03jcrespo The errors are mostly gone, resolving and keeping an eye on it in case it happens again. [11:21:58] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10Patch-For-Review, 10User-Ladsgroup: Drop change_tag.ct_tag column in production - https://phabricator.wikimedia.org/T210713 (10Marostegui) [11:28:57] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Zoranzoki21) >>! In T212346#4891848, @Marostegui wrote: > What's the plan for this? Is this T212346#4845733 ready? It is live on wmf wikis, this should be done, we have consensus. [11:29:21] 10DBA, 10Commons, 10Growth-Team, 10MediaWiki-Change-tagging, and 3 others: Allow filtering based on tag on Special:NewFiles - https://phabricator.wikimedia.org/T124214 (10Marostegui) I wonder if this is still relevant after all the work that has been done with `change_tag` {T185355} [11:29:42] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Marostegui) Do you want to give it a go on Monday? [11:30:03] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Zoranzoki21) >>! In T212346#4891970, @Marostegui wrote: > Do you want to give it a go on Monday? It is ok for me. [11:30:38] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Marostegui) Cool, comment on this task before you are ready to start :) [11:32:01] doing a mysql_install_db takes over 2 minutes on a large server [11:32:23] still faster than an mysql_upgrade on s3 :p [11:32:31] ha, you got me there [11:37:37] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Zoranzoki21) >>! In T212346#4891972, @Marostegui wrote: > Cool, comment on this task before you are ready to start :) Ok, until Monday I will try to delete all pages which have less than 5000 revi... [12:07:58] 10DBA, 10Commons, 10Growth-Team, 10MediaWiki-Change-tagging, and 3 others: Allow filtering based on tag on Special:NewFiles - https://phabricator.wikimedia.org/T124214 (10Ladsgroup) Yeah, it might be possible. It needs some updates to reflect the normalization and updates that happened to the table and als... [12:59:38] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Marostegui) If possible comment here before starting so we can confirm it is ok to proceed. Just in case there is something going on at that moment (ie: incident, an unexpected maintenance etc) Th... [13:56:13] 10DBA, 10Serbian-Sites: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Zoranzoki21) >>! In T212346#4892118, @Marostegui wrote: > If possible comment here before starting so we can confirm it is ok to proceed. > Just in case there is something going on at that moment (... [14:11:49] hey guys [14:11:55] I think i'm going to have the JDs posted as is [14:11:59] any final thoughts/comments? [14:12:10] let me check [14:12:22] same status as we left it yesterday? [14:12:29] pretty much [14:13:12] I am ok with that then [14:14:26] yeah, send it [14:14:50] cool :) [16:03:11] 10DBA, 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review: swap a2-eqiad PDU with on-site spare - https://phabricator.wikimedia.org/T213748 (10faidon) >>! In T213748#4890195, @RobH wrote: > Synced up with Chris via IRC: > > All systems were able to come back up within a2 without incident. The s... [16:05:18] 10DBA, 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review: swap a2-eqiad PDU with on-site spare - https://phabricator.wikimedia.org/T213748 (10RobH) 05Resolved→03Open a:05RobH→03Cmjohnson >>! In T213748#4892549, @faidon wrote: >>>! In T213748#4890195, @RobH wrote: >> Synced up with Chris... [16:12:15] 10DBA, 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review: swap a2-eqiad PDU with on-site spare - https://phabricator.wikimedia.org/T213748 (10Cmjohnson) Correct, it was only the fuse [18:38:15] 10DBA, 10Jade, 10Operations, 10TechCom-RFC, and 3 others: Introduce a new namespace for collaborative judgements about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) >>! In T200297#4887041, @Krinkle wrote: > 3. The feature proposes to store arbitrary text (specifically, wikitext) insi...