[07:27:08] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [07:27:23] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [08:30:25] 10DBA, 10AbuseFilter: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Marostegui) [08:50:13] 10DBA, 10AbuseFilter: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Marostegui) [09:56:04] 10DBA, 10AbuseFilter: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Marostegui) s2 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [] db1129 [] db1125 [] db1122 [] db1105 [] db1103 [x] db1095 [] db1090 [] db1076 [] db1074 []... [09:56:26] 10DBA, 10AbuseFilter: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Marostegui) [10:05:28] 10DBA, 10AbuseFilter: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Marostegui) [10:11:49] 10DBA, 10Operations, 10ops-codfw: pc2010 possibly broken memory - https://phabricator.wikimedia.org/T227552 (10Marostegui) 05Resolved→03Open a:05Marostegui→03Papaul Looks like this happened again and mysql crashed: @Papaul could this be the memory slot? Should we swap the DIMM with another existing D... [10:12:49] 10DBA, 10Analytics, 10Analytics-EventLogging, 10Operations: Decommission m4 proxies (dbproxy1004 and dbproxy1008) - https://phabricator.wikimedia.org/T228768 (10Marostegui) a:03Marostegui Great - thanks. I will get them decommissioned [10:15:47] 10DBA, 10Analytics, 10Analytics-EventLogging, 10Operations: Decommission dbproxy1004 and dbproxy1009 - https://phabricator.wikimedia.org/T228768 (10Marostegui) [10:16:01] 10DBA, 10Analytics, 10Analytics-EventLogging, 10Operations, 10decommission: Decommission dbproxy1004 and dbproxy1009 - https://phabricator.wikimedia.org/T228768 (10Marostegui) [10:23:36] marostegui: o/ [10:23:52] RE: db1073 m5-master [10:24:28] in case of server power loss, data corruption or other disasters, do we have backups and/or a standby replacement server or whatever? [10:26:31] yes, we have a replica [10:26:45] a dns name change would be required and a commit to mediawiki [10:27:10] a commit for mediawiki for wikitech, right? [10:27:18] Yeah, replacing db1073 with db1133 [10:27:27] As that will be the new host I want to promote soon (to decommission db1073) [10:27:36] https://noc.wikimedia.org/conf/highlight.php?file=db-eqiad.php [10:27:39] look for db1133 :) [10:27:55] although I don't think there will be PDU work today after what happened yesterday [10:28:11] ok [10:28:27] anyway we are ready [10:28:33] great [10:29:48] if db1073 goes AWOL for whatever reason, we just switch the DNS for m5-master, ok? [10:37:20] ideally also change replication so the other replica and codfw replicate from the new master [10:37:26] and change mediawiki [10:38:00] ok makes sense. We will stay alert once (if) the operations begins [10:38:31] I will probably be around but I doubt it will happen today [10:39:27] 👍 [12:21:25] 10DBA, 10Analytics, 10Analytics-EventLogging, 10Operations, 10decommission: Decommission dbproxy1004 and dbproxy1009 - https://phabricator.wikimedia.org/T228768 (10Marostegui) [12:21:53] 10DBA, 10Analytics, 10Analytics-EventLogging, 10Operations, 10decommission: Decommission dbproxy1004 and dbproxy1009 - https://phabricator.wikimedia.org/T228768 (10Marostegui) I have stopped haproxy on both hosts, and will leave it like that for 24h, just to be fully sure nothing uses it. [14:33:58] 10DBA, 10AbuseFilter: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Marostegui) [14:41:22] 10DBA, 10AbuseFilter: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Marostegui) [14:44:06] 10DBA, 10AbuseFilter: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Marostegui) s5 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1003 [] db1130 [] db1124 [x] db1113 [] db1110 [x] db1102 [] db1100 [] db1097 [] db1096 [] db1082 [... [14:53:46] 10DBA, 10AbuseFilter: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Marostegui) [16:40:00] 10DBA, 10Operations, 10ops-codfw, 10Goal, 10Patch-For-Review: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Papaul) [16:40:50] 10DBA, 10Operations, 10ops-codfw, 10Goal, 10Patch-For-Review: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Papaul) [16:47:54] 10DBA, 10Operations, 10ops-codfw, 10Goal, 10Patch-For-Review: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Papaul) [16:56:55] 10DBA, 10Operations, 10ops-codfw, 10Goal, 10Patch-For-Review: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Papaul) [17:07:32] 10DBA, 10Operations, 10ops-codfw, 10Goal, 10Patch-For-Review: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Papaul) [17:20:38] 10DBA, 10Operations, 10ops-codfw, 10Goal, 10Patch-For-Review: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Papaul) [18:00:41] 10DBA, 10Operations, 10ops-eqiad: Upgrade db1100 firmware and BIOS - https://phabricator.wikimedia.org/T228732 (10Cmjohnson) @Marostegui This can be done any day...Let's plan 8/6 @1000EDT /1400UTC [19:10:23] 10DBA, 10AbuseFilter, 10Patch-For-Review, 10Schema-change: afl_filter should be split in afl_filter_id and afl_global - https://phabricator.wikimedia.org/T220791 (10Krinkle) [20:25:26] 10DBA, 10Operations, 10ops-eqiad: Upgrade db1100 firmware and BIOS - https://phabricator.wikimedia.org/T228732 (10wiki_willy) a:03Cmjohnson [21:30:33] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Papaul) [21:34:36] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Papaul)