[05:35:36] 10Blocked-on-schema-change, 10DBA: Schema change for renaming new_name_timestamp to rc_new_name_timestamp in recentchanges - https://phabricator.wikimedia.org/T276292 (10Marostegui) p:05Triage→03Medium [05:47:02] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) [05:49:53] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) [05:51:20] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) s2 progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [x] db1171 [x] db1170 [] db1162 [] db1155 [x] db1146 [] db1129 [] db1125 [] db1122 [] db1105 [... [05:51:32] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) [05:54:20] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) [05:55:00] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) [05:56:10] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) [05:57:27] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) s4 progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [] db1160 [] db1155 [x] db1150 [] db1149 [] db1148 [] db1147 [] db1146 [x] db1145 [] db1144 []... [06:07:57] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) [06:09:54] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) [06:11:44] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) s8 progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1005 [x] db1172 [] db1154 [x] db1126 [] db1124 [x] db1116 [x] db1114 [x] db1111 [] db1109 [] db1104... [06:14:46] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) [06:15:29] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) [06:17:29] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) s7 progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1003 [x] db1174 [x] db1170 [] db1155 [x] db1136 [x] db1127 [] db1125 [x] db1116 [x] db1101 [x] db10... [06:18:54] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) [06:19:38] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) [06:22:24] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) s1 progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1003 [x] db1169 [x] db1164 [x] db1163 [] db1154 [x] db1140 [x] db1139 [x] db1135 [x] db1134 [x] db11... [06:24:46] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) [06:32:36] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) [06:36:10] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) s3 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [] db1175 [x] db1171 [] db1166 [] db1157 [] db1154 [] db1124 [] db1123 [] db1112 [] clou... [06:43:29] If someone uses dbctl today, please note that !log isn't being sent to IRC: https://phabricator.wikimedia.org/T276299 but the change does work [07:53:20] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) [08:57:32] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [08:57:55] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) db1164 is now slowly automatically being pooled in s1 (running 10.4.18) [08:58:40] 10DBA, 10decommission-hardware: decommission db1084.eqiad.wmnet - https://phabricator.wikimedia.org/T276302 (10Marostegui) [08:59:02] 10DBA, 10decommission-hardware: decommission db1084.eqiad.wmnet - https://phabricator.wikimedia.org/T276302 (10Marostegui) Not ready yet, let's give its replacement (db1164) a few days of serving traffic. [08:59:36] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [08:59:38] 10DBA, 10decommission-hardware: decommission db1084.eqiad.wmnet - https://phabricator.wikimedia.org/T276302 (10Marostegui) [08:59:47] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [09:51:58] FYI I've marked db1162 as failed in netbox (was active, but T275309 and was showing up in the puppetdb report) [09:51:59] T275309: db1162 crashed - https://phabricator.wikimedia.org/T275309 [09:52:10] thanks [09:58:19] 10DBA, 10Analytics-Clusters, 10Patch-For-Review: Convert labsdb1012 from multi-source to multi-instance - https://phabricator.wikimedia.org/T269211 (10elukey) @razzi some suggestions for the host rename plan: - instead of merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/661529, let's first use `... [10:09:09] 10DBA, 10Analytics-Clusters, 10Patch-For-Review: Convert labsdb1012 from multi-source to multi-instance - https://phabricator.wikimedia.org/T269211 (10Marostegui) We could also do the transfer with the insetup role most likey (unless we have some specific FW rules that are not in `insetup` and are on its fin... [10:11:43] restarted prometheus exporter on db2102 [10:13:18] thanks, I upgraded to 10.4.18 yesterday and daemon may have filed without me noticing [10:13:24] *failed [10:13:26] no worries! [10:30:01] * volans never disappointed by mariadb: [10:30:01] https://jira.mariadb.org/browse/MDEV-9930?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel [10:32:54] that'd be an interesting feature indeed [10:47:21] 10Blocked-on-schema-change, 10DBA: Drop default of oldimage.oi_timestamp - https://phabricator.wikimedia.org/T272511 (10Marostegui) The last pending host is db1123 (s3 master) which I will do tomorrow morning as it will take 15h to complete. [10:55:19] 10Blocked-on-schema-change, 10DBA: Schema change to make rc_id unsigned - https://phabricator.wikimedia.org/T276150 (10Marostegui) Note: This requires a master failover for each section as we are changing the data type and that cannot be done online I have altered db2089:3316 and will leave it running for a f... [10:55:32] 10Blocked-on-schema-change, 10DBA: Drop default of rc_timestamp - https://phabricator.wikimedia.org/T276156 (10Marostegui) I have altered db2089:3316 and will leave it running for a few days before going for a host in eqiad. @Ladsgroup does this look good? ` # mysql.py -hdb2089:3316 frwiki -e "show create tab... [10:56:41] 10DBA, 10Patch-For-Review: Productionize db21[45-52] and db11[76-84] - https://phabricator.wikimedia.org/T275633 (10Marostegui) a:03Marostegui [10:59:42] 10Data-Persistence-Backup, 10SRE, 10SRE-swift-storage, 10Traffic, and 2 others: Depool codfw swift cluster - https://phabricator.wikimedia.org/T267338 (10jcrespo) [11:00:20] 10Data-Persistence-Backup, 10SRE, 10SRE-swift-storage, 10Traffic, and 2 others: Depool codfw swift cluster - https://phabricator.wikimedia.org/T267338 (10jcrespo) [11:31:03] Hi All i wrote a quick wiki with the instructions i have used to get a database up and running in the cloud environment. i uspect its full of mistakes so input welcome however keep in mind but this is only for cloud to get something up and running quickly so doesn't need to be perfect :). https://wikitech.wikimedia.org/wiki/User:Jbond/maridb [11:32:48] what's that software call maridb? [11:33:04] :-) [11:33:53] :) fixed https://wikitech.wikimedia.org/wiki/User:Jbond/Mariadb [11:35:52] in the past, the default role "role::mariadb" used to do a mosly automatic setup (with non-wmf packages,etc) [11:39:38] jynus: could be, i think i tried that when i installed the db on idp and couldn;t get things to work. however i think that could well have been because i had not run `mysql_install_db ` to initials things. ill give that a shot when i have a spare 10 minutes [11:40:12] I am relatively sure it doesn't work now [11:40:20] oh ok :) [11:40:40] specially with wmf packages, those are thought for "never break things/do things automatically" [11:40:58] that is why we recommend the regular debian ones for full automatic db creation, etc. [11:41:22] for you you probably want the "difficult" ones as I am guessing you use them for testing [11:41:48] but others just want a random db for their bot, so no need to make things complicated [11:41:55] yes its for testing so far my use cases are very simple so creating a db and useres manually is no issue [11:42:23] I think stevie was working on automatic integration testing with db deployer [11:42:45] maybe that could help, too [11:44:12] yes possibly either way right now im all good so not making a request here. this is more like "this is what im doing, shout if its really bad and nug me if there is a better way" ;) [11:44:51] oh, it is useful, I am complaining that we could make it easier, but needs more work [11:45:04] yes sure :) [12:20:36] hello data-pesistence, I've a riddle for you... do you have any idea what might have caused this RAM usage on cumin1001 in the last months? [12:20:41] https://grafana.wikimedia.org/d/000000377/host-overview?viewPanel=4&refresh=5m&orgId=1&var-server=cumin1001&var-datasource=thanos&var-cluster=management&from=now-6M&to=now [12:21:01] (the host was rebooted 8d ago, so that explains the rightmost side of it) [12:23:26] volans: without checking, both backups and mysql client are not "services", we just run commands from time to time [12:25:01] yeah I thought maybe some stuck command in a screen/tmux, the other option is a massive memory leak in something [12:25:09] even the largest possible memory hog- comparing tables (which should not use client memory) should release memory after finishing [12:25:10] I'll keep an eye on it in the next weeks [12:26:58] largest memory usage ATM is prometheus-node-exporter [12:28:09] maybe bacula backups? [12:30:29] we'll see in few weeks [13:13:35] 10Data-Persistence-Backup, 10SRE, 10SRE-swift-storage, 10Epic, 10Goal: WMF media storage must be adequately backed up in a remote location - https://phabricator.wikimedia.org/T262668 (10fgiunchedi) [13:14:05] 10Data-Persistence-Backup, 10SRE, 10SRE-swift-storage, 10Traffic, and 2 others: Depool codfw swift cluster - https://phabricator.wikimedia.org/T267338 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi This has happened! We're back to swift active/active, tentatively resolving [14:46:57] 10DBA, 10Patch-For-Review: Productionize db21[45-52] and db11[76-84] - https://phabricator.wikimedia.org/T275633 (10Marostegui) [14:47:07] 10DBA, 10Patch-For-Review: Productionize db21[45-52] and db11[76-84] - https://phabricator.wikimedia.org/T275633 (10Marostegui) Destination for codfw slaves added [16:25:33] 10DBA, 10SRE, 10ops-eqiad: db1162 crashed - https://phabricator.wikimedia.org/T275309 (10Cmjohnson) This has been moved to this coming Friday at 10am local time (1500UTC) [19:22:57] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Cmjohnson) [19:23:37] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Cmjohnson) [19:24:33] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Cmjohnson) [19:31:09] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Cmjohnson) [19:32:03] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Cmjohnson) [19:32:35] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Cmjohnson) [21:29:50] 10Blocked-on-schema-change, 10DBA: Schema change to make rc_id unsigned - https://phabricator.wikimedia.org/T276150 (10Ladsgroup) YES. Thank you!