[06:05:57] 10DBA, 10Patch-For-Review: Test upgrading sanitarium hosts to Buster + 10.4 - https://phabricator.wikimedia.org/T268742 (10Marostegui) Compression finished on s2. [06:12:42] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) [06:31:43] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) s7 eqiad [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1003 [] db1136 [] db1127 [] db1125 [x] db1116 [x] db1101 [x] db1098 [x] db1094 [x]... [07:01:41] 10DBA, 10Patch-For-Review: Test upgrading sanitarium hosts to Buster + 10.4 - https://phabricator.wikimedia.org/T268742 (10Marostegui) On-going transfer from db1121 to db1155:3314 s4 sanitarium positions {P13705} [07:12:14] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) [07:53:04] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) [08:19:02] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) s8 eqiad [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1005 [] db1154 [] db1126 [] db1124 [x] db1116 [] db1114 [] db1111 [] db1109 [] db1... [08:19:22] good morning [08:20:03] Morning [08:22:17] jynus!!!!! [08:22:38] hi, I had some issues with my nick, I had to regain it [08:22:45] what happened? [08:22:48] maybe someone used it while I was out? [08:25:39] But what happened? [08:27:04] "jynus :Nick/channel is temporarily unavailable" [08:27:26] And what did you do to recover it? [08:28:55] just command regain, I have it registered, but not sure how long until it gets deregistered on freenode if you don't use it [08:29:09] I think it takes quite a lot [08:32:13] there has been updates at https://jira.mariadb.org/browse/MDEV-13115 [08:32:44] and https://jira.mariadb.org/browse/MDEV-24439 [08:34:24] Yep, I am following the last one [09:03:30] jynus: you probably have hundreds of mails, not urgent but with some deadlines, if you can check: https://phabricator.wikimedia.org/T271540#6731598 and https://phabricator.wikimedia.org/T270877#6712934 [09:03:53] thanks [09:11:31] 10DBA: db2078 m1 mysqld process crashed - https://phabricator.wikimedia.org/T270877 (10jcrespo) a:03jcrespo [09:12:29] Re:db2078 any thoughts regarding 10.4 stability? (feel free to tell me "too long to type") [09:12:58] yes, I will create the invite once it is confirmed with alex [09:13:17] jynus: I think it was just a "normal" crash, nothing that suggests the similar pattern with any of the other crashes [09:31:05] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) [09:56:16] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) [10:17:18] 10DBA: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10akosiaris) >>! In T271540#6735233, @Marostegui wrote: > Excellent, thanks. I will double check with @akosiaris to see if he can be around in case etherpad requires some action. Fine by me. [10:22:22] 10DBA: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10Marostegui) Thank you both, calendar invite sent. [10:23:46] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) s1 eqiad [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1003 [] db1154 [x] db1140 [x] db1139 [] db1135 [] db1134 [x] db1133 [] db1124 [] db... [10:24:16] 10DBA, 10Patch-For-Review: Test upgrading sanitarium hosts to Buster + 10.4 - https://phabricator.wikimedia.org/T268742 (10Marostegui) s4 on db1155:3314 is being sanitized [10:32:13] 10Data-Persistence-Backup, 10SRE-tools: Some Data Persistence Backup clusters apparently do not support IPv6 - https://phabricator.wikimedia.org/T271148 (10jcrespo) > If you could take some time to put any information you have about supporting IPv6 on these clusters, specific plans for doing so, or if it will... [12:05:10] 10DBA, 10Wikimedia-Etherpad, 10User-notice: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10Marostegui) [12:10:25] 10DBA, 10Wikimedia-Etherpad, 10User-notice: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10Marostegui) Pinging the service owners for this upcoming maintenance: @jbond @ayounsi @MoritzMuehlenhoff @Trizek-WMF just to let you know that etherpad might be read-only of a br... [12:12:45] 10DBA, 10Wikimedia-Etherpad, 10User-notice: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10jbond) fine by me, thanks [12:17:13] 10DBA, 10Wikimedia-Etherpad, 10User-notice: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10MoritzMuehlenhoff) >>! In T271540#6735878, @Marostegui wrote: > Pinging the service owners for this upcoming maintenance: @jbond @ayounsi @MoritzMuehlenhoff That's fine, this on... [13:05:47] 10DBA, 10Wikimedia-Etherpad, 10User-notice: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10Johan) Not familiar with all the databases – is Etherpad the only user-facing thing being affected here? There'll be no other potential disruptions? [13:07:24] 10DBA, 10Wikimedia-Etherpad, 10User-notice: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10Marostegui) >>! In T271540#6735954, @Johan wrote: > Not familiar with all the databases – is Etherpad the only user-facing thing being affected here? There'll be no other potentia... [13:12:42] 10DBA, 10Wikimedia-Etherpad, 10User-notice: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10Marostegui) a:03Marostegui [13:14:28] 10DBA, 10Patch-For-Review: Test upgrading sanitarium hosts to Buster + 10.4 - https://phabricator.wikimedia.org/T268742 (10Marostegui) db1155:3314 is now replicating. So far no InnoDB errors. Triggers seem to be working fine Going to start compression there. Check private data came back clean. [13:15:48] 10DBA, 10Patch-For-Review: Test upgrading sanitarium hosts to Buster + 10.4 - https://phabricator.wikimedia.org/T268742 (10Marostegui) [13:18:28] 10DBA, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10LSobanski) @Jclark-ctr @Cmjohnson, what's a realistic ETA for completing the work on these servers? It would help us plan the next steps for this quarter. [13:22:23] 10DBA, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10Jclark-ctr) @lsobanski I will be working on these this week. As long as nothing comes up urgent it should not be that long [13:31:36] 10DBA, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10LSobanski) Thanks! [14:12:10] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) [14:24:18] 10DBA, 10Wikimedia-Etherpad, 10User-notice: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10ayounsi) Fine for LibreNMS, just don't do it during a major network outage :) [14:33:23] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) [14:34:19] 10DBA, 10Wikimedia-Etherpad: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10Johan) [14:37:02] 10DBA, 10Wikimedia-Etherpad: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10Johan) OK, thanks for the heads-up – I don't think a couple of minutes of Ehterpad read-only is enough of a problem to be included in Tech News, but appreciate the tag. Rather remove it ten times... [14:37:22] 10DBA, 10Wikimedia-Etherpad: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10Marostegui) Thanks :) [14:52:13] 10Blocked-on-schema-change, 10DBA: Schema change for renaming user_properties_property index - https://phabricator.wikimedia.org/T270187 (10Marostegui) s3 eqiad [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [] db1154 [] db1124 [] db1123 [] db1112 [x] db1095 [] db1078 [] db1075 [] clo... [15:34:01] 10DBA, 10Wikimedia-Etherpad: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10Trizek-WMF) Is there a way to add a notice on Etherpad like we do on Mediawiki during read-onlys? Do we have data about the average number of edit/usages of this service? [15:35:22] 10DBA, 10Wikimedia-Etherpad: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10Marostegui) >>! In T271540#6736300, @Trizek-WMF wrote: > Is there a way to add a notice on Etherpad like we do on Mediawiki during read-onlys? No as far as I know > > Do we have data about the... [15:35:36] 10DBA, 10SRE, 10ops-codfw: db2140 crashed due to HW memory errors - https://phabricator.wikimedia.org/T271084 (10Papaul) The DIMM is on site, I will replace it tomorrow once onsite. [15:38:48] 10DBA, 10Wikimedia-Etherpad: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10jcrespo) > Is there a way to add a notice on Etherpad like we do on Mediawiki during read-onlys? @Trizek-WMF If I may suggest a way, I think in the past a simple email to wikitech-l list was enou... [15:58:31] marostegui: heads-up, to save you from doing work too early: I'm going to create multiple DBs now :). [15:59:09] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for niawiki - https://phabricator.wikimedia.org/T270414 (10Urbanecm) >>! In T270414#6705698, @LSobanski wrote: > Thanks, let us know when the database is created, so we can sanitize it. DB was created. [16:00:20] Urbanecm: sounds good, I will take care of them during the week [16:00:24] thanks! [16:00:52] okay, thanks marostegui :) [16:14:57] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for niawiktionary - https://phabricator.wikimedia.org/T270410 (10Urbanecm) >>! In T270410#6705702, @LSobanski wrote: > Thanks, let us know when the database is created, so we can sanitize it. That happened few moments ago. [16:15:31] 10DBA, 10Data-Services: Prepare and check storage layer for diqwiktionary - https://phabricator.wikimedia.org/T270276 (10Urbanecm) >>! In T270276#6698249, @LSobanski wrote: > Thanks, let us know when the database is created, so we can sanitize it. That happened a while ago. [16:29:28] 10DBA, 10Wikimedia-Etherpad: Upgrade and restart m1 master (db1080) - https://phabricator.wikimedia.org/T271540 (10Trizek-WMF) I forgot about the wikitech-l email. I will send one tomorrow. [16:51:00] 10DBA, 10Data-Services: Prepare and check storage layer for bclwiktionary - https://phabricator.wikimedia.org/T270280 (10Urbanecm) >>! In T270280#6698245, @LSobanski wrote: > Thanks, let us know when the database is created, so we can sanitize it. Wiki was created. [18:31:56] 10DBA, 10MediaWiki-Page-derived-data, 10Platform Engineering, 10TechCom-RFC, and 2 others: RFC: Normalize MediaWiki link tables - https://phabricator.wikimedia.org/T222224 (10Ladsgroup) I just want to say that with the recent changes with revision table (MCR work). Now the biggest table of enwiki is pageli... [18:43:20] 10DBA, 10wikitech.wikimedia.org: wikitech database has almost all of its varbinary fields wrong - https://phabricator.wikimedia.org/T269348 (10Ladsgroup) Ping? :D [18:50:20] 10DBA, 10wikitech.wikimedia.org: wikitech database has almost all of its varbinary fields wrong - https://phabricator.wikimedia.org/T269348 (10Marostegui) I would suggest we try first on the labstestwiki - @andrew? [21:33:11] 10DBA, 10MediaWiki-Page-derived-data, 10Platform Engineering, 10TechCom-RFC, and 2 others: RFC: Normalize MediaWiki link tables - https://phabricator.wikimedia.org/T222224 (10daniel) >>! In T222224#6737148, @Ladsgroup wrote: > I just want to say that with the recent changes with revision table (MCR work).... [22:21:58] 10DBA, 10Data-Services, 10Patch-For-Review, 10User-Kormat, 10cloud-services-team (Kanban): Parametrize wmf-pt-kill so it can connect to different sockets - https://phabricator.wikimedia.org/T260511 (10Bstorm) That patch was safe on the old servers (no change). On the multi-instance I see the error: `Acce... [22:27:06] 10DBA, 10Data-Services, 10Patch-For-Review, 10User-Kormat, 10cloud-services-team (Kanban): Parametrize wmf-pt-kill so it can connect to different sockets - https://phabricator.wikimedia.org/T260511 (10Bstorm) Yep, the user is not created. Creating the grant using the info in modules/role/templates/mariad... [22:42:32] 10DBA, 10Data-Services, 10Patch-For-Review, 10User-Kormat, 10cloud-services-team (Kanban): Parametrize wmf-pt-kill so it can connect to different sockets - https://phabricator.wikimedia.org/T260511 (10Bstorm) I believe all replicas pass puppet now (after creating that grant). @Marostegui if you can check...