[00:46:17] 10DBA, 10Cognate, 10ContentTranslation, 10Growth-Team, and 9 others: Restart x1 database master (db1103) - https://phabricator.wikimedia.org/T281212 (10Johan) Making sure I read the task correctly: Only English Wikipedia and Growth features (trivially) will be affected. Echo, CX and a few other things wi... [04:06:25] 10DBA, 10Patch-For-Review: Switchover s1 from db1083 to db1163 - https://phabricator.wikimedia.org/T278214 (10Marostegui) [04:07:46] 10DBA, 10Patch-For-Review: Switchover s1 from db1083 to db1163 - https://phabricator.wikimedia.org/T278214 (10Marostegui) [04:23:56] 10DBA, 10Patch-For-Review: Switchover s1 from db1083 to db1163 - https://phabricator.wikimedia.org/T278214 (10Marostegui) [04:24:57] 10DBA, 10Patch-For-Review: Switchover s1 from db1083 to db1163 - https://phabricator.wikimedia.org/T278214 (10Marostegui) [04:25:09] 10DBA, 10Patch-For-Review: Switchover s1 from db1083 to db1163 - https://phabricator.wikimedia.org/T278214 (10Marostegui) All pre-steps are done [04:25:59] 10DBA, 10Cognate, 10ContentTranslation, 10Growth-Team, and 9 others: Restart x1 database master (db1103) - https://phabricator.wikimedia.org/T281212 (10Marostegui) @Johan all wikipedias will be affected, as x1 holds echo for all the projects. [04:28:06] 10DBA, 10SRE, 10ops-codfw: codfw: Relocate servers in 10G racks - https://phabricator.wikimedia.org/T281135 (10Marostegui) I will get db2096 ready for you. [04:48:30] 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10brennen) I tried to track down a rationale for current Gerrit backup settings and the answer might be "no one still in RelEng remembers". Looks like @d... [05:02:59] 10DBA, 10Patch-For-Review: Switchover s1 from db1083 to db1163 - https://phabricator.wikimedia.org/T278214 (10Marostegui) [05:03:44] 10DBA, 10Patch-For-Review: Switchover s1 from db1083 to db1163 - https://phabricator.wikimedia.org/T278214 (10Marostegui) [05:05:02] 10DBA, 10Patch-For-Review: Switchover s1 from db1083 to db1163 - https://phabricator.wikimedia.org/T278214 (10Marostegui) [05:07:20] 10DBA, 10Patch-For-Review: Switchover s1 from db1083 to db1163 - https://phabricator.wikimedia.org/T278214 (10Marostegui) [05:09:57] 10DBA, 10Patch-For-Review: Switchover s1 from db1083 to db1163 - https://phabricator.wikimedia.org/T278214 (10Marostegui) The switchover was done. RO starts: 05:00:41 RO stops: 05:01:38 Total RO time: 57 seconds [05:10:18] 10DBA, 10Patch-For-Review: Switchover s1 from db1083 to db1163 - https://phabricator.wikimedia.org/T278214 (10Marostegui) [05:10:42] 10DBA: Switchover s1 from db1083 to db1163 - https://phabricator.wikimedia.org/T278214 (10Marostegui) [05:11:15] 10Blocked-on-schema-change, 10DBA: Schema change for renaming new_name_timestamp to rc_new_name_timestamp in recentchanges - https://phabricator.wikimedia.org/T276292 (10Marostegui) Old s1 master (db1083) done [05:11:25] 10Blocked-on-schema-change, 10DBA: Schema change for renaming new_name_timestamp to rc_new_name_timestamp in recentchanges - https://phabricator.wikimedia.org/T276292 (10Marostegui) [05:14:11] 10DBA, 10Orchestrator, 10Patch-For-Review, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) >>! In T266483#7033854, @Marostegui wrote: > s1 is fully done only pending the master (T278214) done [05:14:35] 10DBA, 10Orchestrator, 10Patch-For-Review, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) [05:17:04] 10DBA: Switchover s1 from db1083 to db1163 - https://phabricator.wikimedia.org/T278214 (10Marostegui) All the switchover steps are done. I will continue with the follow up items once 24h have passed to ensure the new master is working fine. [05:29:54] 10DBA, 10Orchestrator: Cleanup heartbeat.heartbeat on all production instances - https://phabricator.wikimedia.org/T268336 (10Marostegui) [05:30:02] 10DBA, 10Orchestrator: Cleanup heartbeat.heartbeat on all production instances - https://phabricator.wikimedia.org/T268336 (10Marostegui) s1 fully clean now [05:30:19] s1 is now in orch: https://orchestrator.wikimedia.org/web/cluster/db1163.eqiad.wmnet:3306 [05:30:28] https://orchestrator.wikimedia.org/web/cluster/alias/s1 [05:32:22] 10Blocked-on-schema-change, 10DBA: Schema change for renaming new_name_timestamp to rc_new_name_timestamp in recentchanges - https://phabricator.wikimedia.org/T276292 (10Marostegui) [05:37:26] 10Blocked-on-schema-change, 10DBA: Schema change for renaming new_name_timestamp to rc_new_name_timestamp in recentchanges - https://phabricator.wikimedia.org/T276292 (10Marostegui) All done [05:37:36] marostegui: if you have time today to supervise, we're ready to do the real mailman3 install/db setup [05:37:46] 10Blocked-on-schema-change, 10DBA: Schema change for renaming new_name_timestamp to rc_new_name_timestamp in recentchanges - https://phabricator.wikimedia.org/T276292 (10Marostegui) 05Open→03Resolved [05:37:47] legoktm: sure, when do you want to do it? [05:39:04] marostegui: in ~20 min at 6 UTC if that works? [05:39:13] that works yep [05:46:07] 10DBA, 10DiscussionTools, 10OWC2020, 10Editing-team (FY2020-21 Kanban Board), 10Patch-For-Review: DBA review: conversation subscriptions - https://phabricator.wikimedia.org/T263817 (10Marostegui) >>! In T263817#7039894, @matmarex wrote: > Thank you for taking a look! Replies below: > > >>>! In T263817#... [05:49:13] 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10jcrespo) > We'd like to aim for once daily. I'm not sure I am comfortable to provide a worse coverage than before for backups. A lot can be commited i... [05:52:37] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [05:53:11] 10DBA, 10Data-Persistence-Backup, 10Patch-For-Review: Upgrade all sanitarium masters to 10.4 and Buster - https://phabricator.wikimedia.org/T280492 (10Marostegui) [05:53:21] 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10jcrespo) CCing @akosiaris for rationale for hourly backups (see recent comments) as committer of the original schedule, in case he can provide extra ba... [05:54:19] 10DBA, 10Data-Persistence-Backup, 10Patch-For-Review: Upgrade all sanitarium masters to 10.4 and Buster - https://phabricator.wikimedia.org/T280492 (10Marostegui) [06:01:17] 10DBA, 10SRE, 10ops-codfw: codfw: Relocate servers in 10G racks - https://phabricator.wikimedia.org/T281135 (10Marostegui) @Papaul db2096 is now off, so you can proceed as needed. [06:01:25] 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10jcrespo) Another option would be to have an intermediate step - e.g. an hourly rsync-able tar or filesystem snapshot (whatever is possible) to an inter... [06:06:05] 10Data-Persistence-Backup, 10Patch-For-Review: xtrabackup --prepare hits open_files_limit on buster - https://phabricator.wikimedia.org/T281094 (10jcrespo) 2 successful backups since deploy, everything looking good. [06:11:08] 10Data-Persistence-Backup, 10Patch-For-Review: xtrabackup --prepare hits open_files_limit on buster - https://phabricator.wikimedia.org/T281094 (10jcrespo) Not generating a 0.5 package for stretch as they are scheduled to disappear soon and not affected by the bug.{F34430902} [06:11:19] ^FYI [06:12:07] +1 [06:12:46] 10DBA: Switchover s1 from db1083 to db1163 - https://phabricator.wikimedia.org/T278214 (10Marostegui) [06:32:59] 10Blocked-on-schema-change, 10DBA: Schema change to make rc_id unsigned and rc_timestamp BINARY - https://phabricator.wikimedia.org/T276150 (10Marostegui) db1083 (old s1 master) done [06:33:18] 10Blocked-on-schema-change, 10DBA: Schema change to make rc_id unsigned and rc_timestamp BINARY - https://phabricator.wikimedia.org/T276150 (10Marostegui) [06:34:07] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) a:03Marostegui [06:37:20] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) @Ladsgroup can you confirm whether the second ALTER is meant to be for the `image` table and `img_timestamp` column? If it is for `recentcha... [06:43:54] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Marostegui) a:03Marostegui [06:44:09] 10Blocked-on-schema-change, 10DBA: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486 (10Marostegui) a:03Marostegui [06:55:52] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Ladsgroup) Yes. you're right. Copy pasta error [06:56:08] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Ladsgroup) [06:56:29] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) \o/ [06:57:44] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) [06:58:04] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Marostegui) [06:58:33] 10Blocked-on-schema-change, 10DBA: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486 (10Marostegui) [06:59:43] 10Blocked-on-schema-change, 10DBA: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486 (10Marostegui) Will alter a host in s6 codfw and eqiad and leave it till next week before going for all of them [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbsto... [06:59:55] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) Will alter a host in s6 codfw and eqiad and leave it till next week before going for all of them [] labsdb1011 [] labsdb1010 [] labsdb1009... [06:59:57] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Marostegui) Will alter a host in s6 codfw and eqiad and leave it till next week before going for all of them [] labsdb1011 [] labsdb1010 []... [07:04:04] 10Blocked-on-schema-change, 10DBA: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486 (10Marostegui) [07:04:15] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) [07:04:38] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Marostegui) [07:13:06] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) [07:45:53] 10Blocked-on-schema-change, 10DBA: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486 (10Marostegui) @Ladsgroup this looks good? ` # mysql.py -hdb2089:3316 frwiki -e "show create table user_newtalk\G" *************************** 1. row *************... [07:46:15] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) @Ladsgroup this looks good?: ` # mysql.py -hdb2089:3316 frwiki -e "show create table image\G" *************************** 1. row ***********... [07:46:39] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Marostegui) @Ladsgroup this looks good? ` # mysql.py -hdb2089:3316 frwiki -e "show create table watchlist\G" *************************** 1.... [08:30:18] 10DBA, 10SRE, 10ops-codfw: codfw: Relocate servers in 10G racks - https://phabricator.wikimedia.org/T281135 (10fgiunchedi) [09:02:01] 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10akosiaris) >>! In T274463#7040650, @jcrespo wrote: > CCing @akosiaris for rationale for hourly backups (see recent comments) as committer of the origin... [09:03:34] marostegui: Can I clean commons again (with smaller batches)? [10:27:13] 10DBA, 10Cognate, 10ContentTranslation, 10Growth-Team, and 9 others: Restart x1 database master (db1103) - https://phabricator.wikimedia.org/T281212 (10Johan) @Marostegui Thanks! And notifications for this minute will be lost, not just delayed? [11:40:43] Amir1: sorry, I was out taking a break [11:40:44] sure [11:40:45] go for it [11:41:36] 10DBA, 10Cognate, 10ContentTranslation, 10Growth-Team, and 9 others: Restart x1 database master (db1103) - https://phabricator.wikimedia.org/T281212 (10Marostegui) I am not fully sure about that, as I don't know whether they get re-tried or not. [11:41:55] Thanks! [11:41:57] All good [11:42:14] I will get ready to restart mysql on the master [11:42:53] xD [11:42:59] I'll get ready to hide [11:51:27] 10DBA: Switchover s1 from db1083 to db1163 - https://phabricator.wikimedia.org/T278214 (10Marostegui) [11:52:47] one of my favorite outages happened outside of Wikimedia was that a rogue process in db took a lot of memory for a short period and then did. OOM got into action, looked what process is taking the most memory to kill and killed mysql [11:52:52] freaking genious [11:53:03] yeah, that's a classic [11:53:19] There are ways to prevent that with oom scores and such, but I have never got them to work properly [12:03:08] Amir1: can you confirm the table I posted at T266486 T268392 T273360 look good? [12:03:09] T268392: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 [12:03:09] T273360: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 [12:03:09] T266486: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486 [12:03:14] So I know I am good to go next week [12:04:17] sure, I need to double check. I'll do it in half an hour [12:04:18] sorry [12:05:39] no worries [12:05:52] I thought you were going on holidays soon [12:05:59] So better to ask before you go :) [12:06:12] I want to advance as much as I can with those schema changes that are not online, so we can get them done with the DC switch [12:06:41] oh I'm ON holiday atm [12:06:58] that's why I'm doing volunteer stuff like mailman [12:07:23] Hahahaha [12:07:36] Mailman on holidays, what can possibly go wrong [12:07:39] my options for holidays are limited due to current circumstances [12:07:51] Amir1: it was mailman or *links tables no? [12:08:03] exactly [12:08:13] that's the next one on my list :((( [12:08:22] https://www.mediawiki.org/wiki/User:Ladsgroup/Volunteer_stuff#Future? [12:08:52] Clean up FlaggedRevs extension -> jeeeeez [12:09:20] lol. We already dropped +5k lines from it [12:11:37] it's not linked there T277883 [12:11:37] T277883: Drop all low-use and unused features of FlaggedRevs to make it more maintainable - https://phabricator.wikimedia.org/T277883 [12:12:16] Buf I am so glad I am not you [12:12:35] Only going around the flaggedrev tables make me nervous [12:13:12] oh we are going to drop a lot of its schema once I'm done with the code mwhaha [12:13:17] you'll have so much fun [12:15:39] hahaha [12:15:43] I am happy when I need to drop things [12:19:24] speaking of dropping things, did you check watchlist in wikidatawiki? [12:19:37] I think I'm done with wikidatawiki [12:19:40] Ah no, let me do that [12:20:16] Thanks! [12:26:01] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Ladsgroup) WRT this patch it looks okay while `wl_user` shouldn't have a default (at least comparing with https://github.com/wikimedia/media... [12:27:15] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Marostegui) Thanks, removing a default is an online DDL so it can be done in a different task and it will be easier than changing a data type [12:28:20] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Ladsgroup) It seems it's a drift back from 2007. I will create a separate ticket for that later (it'll just show up in the drift report). [12:29:56] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Ladsgroup) That field looks okay. Didn't check the whole table in depth (if any issues, it's a separate from this one and will show up in drift reports) [12:31:42] btw we migrated user table to abstract schema last weekend, now only revision and searchindex left (searchindex doesn't exist in our production) [12:31:51] oh nice congrats! [12:33:30] 10Blocked-on-schema-change, 10DBA: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486 (10Ladsgroup) Looks good to me! [12:45:51] 10DBA, 10SRE, 10ops-codfw: codfw: Relocate servers in 10G racks - https://phabricator.wikimedia.org/T281135 (10Papaul) @Marostegui @jcrespo thanks [13:00:16] 10DBA, 10Orchestrator, 10Patch-For-Review, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) [13:00:18] 10DBA, 10Data-Persistence-Backup, 10Patch-For-Review: Upgrade all sanitarium masters to 10.4 and Buster - https://phabricator.wikimedia.org/T280492 (10Marostegui) [13:01:53] 10DBA, 10Orchestrator, 10Patch-For-Review, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) s6 is fully done apart from the master (db1131) [13:02:10] 10DBA, 10Orchestrator, 10Patch-For-Review, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) [14:11:33] Amir1: [14:11:39] -rw-rw---- 1 mysql mysql 11G Apr 28 12:22 watchlist.ibd [14:11:43] -rw-rw---- 1 mysql mysql 2.8G Apr 28 14:11 watchlist.ibd [14:11:48] YES [14:12:03] Hope that gives you some space [14:12:10] Hard to know though how much was due to the clean up, as God knows when was that table last optimized [14:12:25] I think wikidata is rather healthy in general [14:12:38] it's mostly the cleanup given the number of rows [14:12:54] it used to be 108M now it's around 30-40M [14:13:09] niiiice [14:14:45] so commons will go from ~180M rows to ~160M. It's not that much, I let you decide if you want to optimize it or not [14:15:09] at worst, it just keeps it from growing even further for a while [14:15:24] for enwiki it was around 5% not worth probably [14:15:32] Yeah, 5% isn't that much [14:15:41] Probably the same for commons, it is not that much I think [14:15:51] We can always at least optimize it on the masters once eqiad is standby [14:15:55] to get that out of the way [14:16:10] did you know if the feature was disabled be default, on config or by users (adding new items)? [14:16:16] *by [14:17:05] so the feature was on by default (as it should be) but now it's disabled for bots [14:17:15] so it won't grow back (at least in wikidata) [14:17:23] it made no sense to have the bots adding stuff to their watchlist [14:17:24] cool [14:17:49] marostegui: one bot alone was responsible for one third of watchlist table in fawiki [14:17:54] :-/ [14:19:10] Amir1: If that's in 1.36 and a script, I might add that to my list of migration tasks [14:21:23] RhinosF1: so the feature is in but cleanup is manual (and I go ask around people to approve deletions) and then delete it manually [14:21:39] ah ok [14:21:45] wrote a bash script for it, which is suboptimal but I don't have time to make it a proper maint script [14:22:02] link? [14:22:21] T258098 [14:22:22] T258098: Purge unused watchlist rows - https://phabricator.wikimedia.org/T258098 [14:22:47] T258108 [14:22:48] T258108: Ignore auto-watchlist preferences for bots - https://phabricator.wikimedia.org/T258108 [14:23:23] the script is pretty basic [14:23:27] https://www.irccloud.com/pastebin/8epy2nxY/ [14:24:31] ah ok [14:32:00] 10DBA, 10Cognate, 10ContentTranslation, 10Growth-Team, and 9 others: Restart x1 database master (db1103) - https://phabricator.wikimedia.org/T281212 (10Trizek-WMF) IIRC, notifications work on a one-time attempt. I'll ask around. This read-only is a big one, affecting multiple wikis and cross-wiki services... [14:33:01] ours aren't bots on meta at least [14:33:09] 2nd worst is a retired user [14:33:16] 10DBA, 10Cognate, 10ContentTranslation, 10Growth-Team, and 9 others: Restart x1 database master (db1103) - https://phabricator.wikimedia.org/T281212 (10Marostegui) From what I can see on the last time we did this, it was just announce on tech news: https://phabricator.wikimedia.org/T273758#6832250 [14:33:26] 10DBA, 10SRE, 10ops-codfw: codfw: Relocate servers in 10G racks - https://phabricator.wikimedia.org/T281135 (10Papaul) [14:33:48] 10DBA, 10Cognate, 10ContentTranslation, 10Growth-Team, and 9 others: Restart x1 database master (db1103) - https://phabricator.wikimedia.org/T281212 (10Marostegui) @Trizek-WMF if you want me to create a task for that I can do that too, whatever works best for you. [14:47:46] 10DBA, 10Cognate, 10ContentTranslation, 10Growth-Team, and 9 others: Restart x1 database master (db1103) - https://phabricator.wikimedia.org/T281212 (10Trizek-WMF) Last time, [[ https://phabricator.wikimedia.org/T273762#6815905 | we setup a banner ]]. :) [14:48:24] 10DBA, 10Cognate, 10ContentTranslation, 10Growth-Team, and 9 others: Restart x1 database master (db1103) - https://phabricator.wikimedia.org/T281212 (10Marostegui) @Trizek-WMF this is what we listed as impact last time: T273762 [14:49:48] 10DBA, 10SRE, 10ops-codfw: codfw: Relocate servers in 10G racks - https://phabricator.wikimedia.org/T281135 (10Papaul) [14:56:18] 10DBA, 10SRE, 10ops-codfw: codfw: Relocate servers in 10G racks - https://phabricator.wikimedia.org/T281135 (10Papaul) [14:57:29] 10DBA, 10SRE, 10ops-codfw: codfw: Relocate servers in 10G racks - https://phabricator.wikimedia.org/T281135 (10Papaul) [15:04:42] 10DBA, 10SRE, 10ops-codfw: codfw: Relocate servers in 10G racks - https://phabricator.wikimedia.org/T281135 (10Eevans) Almost too late to the party here, but it's fine to relocate sessionstore2001. Will it be put under maintenance in Icinga? [15:31:39] 10DBA, 10SRE, 10ops-codfw: codfw: Relocate servers in 10G racks - https://phabricator.wikimedia.org/T281135 (10Papaul) [15:32:14] 10DBA, 10SRE, 10ops-codfw: codfw: Relocate servers in 10G racks - https://phabricator.wikimedia.org/T281135 (10Papaul) [17:14:09] 10DBA, 10SRE, 10ops-codfw: codfw: Relocate servers in 10G racks - https://phabricator.wikimedia.org/T281135 (10Papaul) [20:16:31] 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10brennen) Thanks for the detailed responses. To try and answer some questions: > Do the timestamp changes after each export, even if no file content c... [20:52:50] 10DBA, 10Data-Persistence-Backup, 10SRE, 10Wikimedia-Mailing-lists: Create production databases for mailman3 - https://phabricator.wikimedia.org/T278614 (10Legoktm) >>! In T278614#7025271, @jcrespo wrote: > Backups have been enabled and access seems correct. I saw the dbs are right now empty, but please pi... [22:37:50] 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10brennen) Ok, so having re-read this entire thread and discussed a bunch with @thcipriani, @Sfigor, & @strofimovsky, I'll go back to what @jcrespo said...