[01:15:32] 10DBA, 10Patch-For-Review: Upgrade m2 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T257540 (10Adamant.pwn) Hi there! I wanted to analyse some categorylinks data (namely, collect //all// categories which are reachable from given category), but it's hierarchical and traversing it usually means... [05:08:24] 10DBA, 10Patch-For-Review: Upgrade m2 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T257540 (10Marostegui) @Adamant.pwn I assume you need wikireplicas service for that and not m2? If that is the case, we are not yet sure when wikireplicas will be upgraded, but it will take a few months still f... [06:22:21] 10DBA: page_restrictions indexes have been majestically drifting from code - https://phabricator.wikimedia.org/T256682 (10Marostegui) s6 eqiad progress [x] labsdb1012 [x] labsdb1011 [x] labsdb1010 [x] labsdb1009 [x] dbstore1005 [x] db1140 [x] db1139 [x] db1131 [x] db1125 [x] db1113 [x] db1098 [x] db1096 [x] db1... [06:22:38] 10DBA: page_restrictions indexes have been majestically drifting from code - https://phabricator.wikimedia.org/T256682 (10Marostegui) [06:51:46] 10DBA, 10Patch-For-Review: Upgrade m2 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T257540 (10Marostegui) [06:58:36] 10DBA, 10Patch-For-Review: Upgrade m2 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T257540 (10Marostegui) @jcrespo @akosiaris @Volans @dpifke #recommendation-api #research I would like to failover the master Tuesday 4th Aug at 08:00 AM UTC This means around 1 minute of read only time. [06:58:52] 10DBA, 10Recommendation-API, 10Research, 10Patch-For-Review: Upgrade m2 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T257540 (10Marostegui) [07:00:31] 10DBA: page_restrictions indexes have been majestically drifting from code - https://phabricator.wikimedia.org/T256682 (10Marostegui) [07:06:43] 10DBA, 10Recommendation-API, 10Research, 10Patch-For-Review: Upgrade m2 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T257540 (10Volans) @Marostegui: green light from me for Debmonitor, usually no action is needed as it reconnects automatically but I can be around just in case. [07:11:07] 10DBA, 10OTRS, 10Recommendation-API, 10Research, 10Patch-For-Review: Upgrade m2 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T257540 (10akosiaris) >>! In T257540#6336427, @Marostegui wrote: > @jcrespo @akosiaris @Volans @dpifke #recommendation-api #research I would like to failover the... [07:51:45] 10DBA: page_restrictions indexes have been majestically drifting from code - https://phabricator.wikimedia.org/T256682 (10Marostegui) s5 eqiad progress [x] labsdb1012 [x] labsdb1011 [x] labsdb1010 [x] labsdb1009 [x] dbstore1003 [x] db1145 [x] db1144 [x] db1130 [x] db1124 [x] db1113 [x] db1110 [x] db1100 [x] db1... [07:52:20] 10DBA: page_restrictions indexes have been majestically drifting from code - https://phabricator.wikimedia.org/T256682 (10Marostegui) [08:13:06] 10DBA, 10Operations, 10User-Kormat, 10User-jbond: Refactor mariadb puppet code - https://phabricator.wikimedia.org/T256972 (10Kormat) [08:13:51] 10DBA, 10Operations, 10User-Kormat: Remove unused parameters from profile::mariadb::monitor::prometheus - https://phabricator.wikimedia.org/T256879 (10Kormat) 05Open→03Resolved a:03Kormat Closing this, refactoring of the mysql exporter puppet code will be covered by the parent task. [08:13:54] 10DBA, 10Operations, 10User-Kormat, 10User-jbond: Refactor mariadb puppet code - https://phabricator.wikimedia.org/T256972 (10Kormat) [08:18:39] Can I get a sanity check on https://gerrit.wikimedia.org/r/c/operations/dns/+/616341? [08:21:43] 10DBA: page_restrictions indexes have been majestically drifting from code - https://phabricator.wikimedia.org/T256682 (10Marostegui) [08:24:43] looking [08:24:49] ta [08:26:22] jynus: o/ - if you have time this week we could set up some time to test bacula backups for db1108, it is now replicating from matomo1002 and an-coord1001 [08:35:03] ok [09:11:35] 10DBA: page_restrictions indexes have been majestically drifting from code - https://phabricator.wikimedia.org/T256682 (10Marostegui) s2 eqiad progress [x] labsdb1012 [x] labsdb1011 [x] labsdb1010 [x] labsdb1009 [x] dbstore1004 [x] db1146 [x] db1129 [x] db1125 [x] db1122 [x] db1105 [x] db1095 [x] db1090 [x] db... [09:12:00] 10DBA: page_restrictions indexes have been majestically drifting from code - https://phabricator.wikimedia.org/T256682 (10Marostegui) [09:14:05] 10DBA: page_restrictions indexes have been majestically drifting from code - https://phabricator.wikimedia.org/T256682 (10Marostegui) [09:21:48] 10DBA: page_restrictions indexes have been majestically drifting from code - https://phabricator.wikimedia.org/T256682 (10Marostegui) s4 eqiad progress [x] labsdb1012 [x] labsdb1011 [x] labsdb1010 [x] labsdb1009 [x] dbstore1004 [x] db1149 [x] db1148 [x] db1147 [x] db1146 [x] db1145 [x] db1144 [x] db1143 [x] db1... [09:22:08] 10DBA: page_restrictions indexes have been majestically drifting from code - https://phabricator.wikimedia.org/T256682 (10Marostegui) [10:02:18] jynus: meeting? [11:45:04] jynus kormat I have filled the sre doc with stuff, remember to add your stuff! [11:45:09] (nothing bolded for now I think) [11:58:22] marostegui: hey, just checking, did you manage to confirm with your team T257943 should be on s5? [11:58:23] T257943: Create Wikipedia Kotava - https://phabricator.wikimedia.org/T257943 [11:58:50] (I have talked with Amir1, he seems to be confortable with using a live wiki for creation) [11:58:57] Urbanecm: yeah, we are fine with it on s5, but I want to get an answer from Reedy or Amir1 on the email thread, about how confident we are of not using a dummy wiki [11:59:06] Urbanecm: ah, ok, I didn't know that [11:59:22] Urbanecm: I guess we'd need to change documentation then, no? [11:59:36] no problem, thanks! Could you please update the ticket with the info, please? [11:59:52] sure [11:59:57] you've got it handy? [12:00:05] T257943 [12:00:09] thank you [12:00:17] let me answer on the email thread [12:00:20] sure [12:01:33] I have commented on that and on the database specific one [12:01:33] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for avkwiki - https://phabricator.wikimedia.org/T258077 (10Marostegui) This wiki will be created on s5 [12:02:00] cool, thanks [12:13:41] 10DBA: page_restrictions indexes have been majestically drifting from code - https://phabricator.wikimedia.org/T256682 (10Marostegui) [12:24:09] 10DBA: page_restrictions indexes have been majestically drifting from code - https://phabricator.wikimedia.org/T256682 (10Marostegui) s7 eqiad progress [x] labsdb1012 [x] labsdb1011 [x] labsdb1010 [x] labsdb1009 [x] dbstore1003 [x] db1136 [x] db1127 [x] db1125 [x] db1116 [x] db1101 [x] db1098 [x] db1094 [x] db1... [12:26:22] 10DBA: page_restrictions indexes have been majestically drifting from code - https://phabricator.wikimedia.org/T256682 (10Marostegui) [12:44:24] marostegui: once you get some time, I'd appreciate if you could reivew the db-*.php config at https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/612897 [12:44:37] sure [12:44:39] let me check [12:45:58] thanks marostegui ! [13:35:33] 10DBA, 10Operations, 10User-Kormat: Refactor tendril+zarcillo roles/profiles - https://phabricator.wikimedia.org/T258566 (10Kormat) 05Open→03Resolved Refactoring success \o/ [13:35:38] 10DBA, 10Operations, 10Epic, 10User-Kormat: Use zarcillo as an authoritative inventory of db instances/roles - https://phabricator.wikimedia.org/T257814 (10Kormat) [13:44:08] 10DBA, 10Operations, 10User-Kormat, 10User-jbond: Standardize/centralize mapping from section to mariadb port/socket and prom-mysql-exporter port - https://phabricator.wikimedia.org/T257033 (10herron) p:05Triage→03Medium [13:46:09] 10DBA, 10User-Kormat: Create testing environment for db automation - https://phabricator.wikimedia.org/T256602 (10Kormat) p:05Triage→03Medium [13:46:24] 10DBA: Research changes on prometheus-mysqld-exporter after buster/mariadb upgrade - https://phabricator.wikimedia.org/T252761 (10Kormat) p:05Triage→03Medium [13:52:23] 10DBA, 10CheckUser, 10Growth-Team, 10Thanks, 10User-DannyS712: Monitor the growth of CheckUser tables thanks to the addition of Thanks data - https://phabricator.wikimedia.org/T257223 (10Huji) Oh well. When we get to the similar task that has to do with the addition of the login logs to the CU logs, I wi... [13:54:40] 10DBA, 10CheckUser, 10Growth-Team, 10Thanks, 10User-DannyS712: Monitor the growth of CheckUser tables thanks to the addition of Thanks data - https://phabricator.wikimedia.org/T257223 (10DannyS712) Just wanted to say thanks to everyone for the hard work [14:13:04] so something I don't have clear [14:13:27] is should I put OKR milestones on working updates or not? [14:39:41] not sure what you mean [14:40:43] for example the quotes for dbprov hosts [14:41:10] that should be only on betterworks? [14:41:42] I guess so? [14:41:50] Like, it is something important for the overall okr, no? [14:42:00] but not on SRE updates? [14:42:45] I don't know :( [14:42:54] Probably mark knows [14:58:32] 10DBA: Remove commonswiki.templatelinks partitions - https://phabricator.wikimedia.org/T258956 (10Marostegui) [14:58:43] 10DBA: Remove commonswiki.templatelinks partitions - https://phabricator.wikimedia.org/T258956 (10Marostegui) p:05Triage→03Medium [15:11:53] 10DBA, 10Operations: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092 - https://phabricator.wikimedia.org/T216240 (10Marostegui) [16:21:23] was there a database alert or did I imagine it? [16:22:40] maybe it was just db1124 ? [16:23:36] it was a resolve from victorops, about db1079 [16:23:41] ah, ok [16:34:33] 10DBA, 10CheckUser, 10Growth-Team, 10Thanks, 10User-DannyS712: Monitor the growth of CheckUser tables thanks to the addition of Thanks data - https://phabricator.wikimedia.org/T257223 (10Huji) [16:36:14] 10DBA, 10CheckUser, 10Growth-Team, 10Thanks, 10User-DannyS712: Monitor the growth of CheckUser tables thanks to the addition of Thanks data - https://phabricator.wikimedia.org/T257223 (10Huji) @Marostegui assuming that numbers you pull tomorrow are not drastically different than the last 3 weeks, we coul... [17:19:48] 10DBA, 10OTRS, 10Recommendation-API, 10Research, 10Patch-For-Review: Upgrade m2 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T257540 (10dpifke) No objection here. [18:11:53] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for lijwikisource - https://phabricator.wikimedia.org/T258389 (10Bstorm) a:03Bstorm [18:28:16] 10DBA, 10Operations: db1082 crashed - https://phabricator.wikimedia.org/T258336 (10Marostegui) [19:51:10] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for lijwikisource - https://phabricator.wikimedia.org/T258389 (10Bstorm) So far, I've run the index and view scripts on all four replicas, however, we have a bug that has popped up in the DNS script that is going to block... [19:57:26] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for arywiki - https://phabricator.wikimedia.org/T257725 (10Bstorm) [19:58:15] 10DBA, 10Data-Services, 10User-Ladsgroup, 10cloud-services-team (Kanban): Prepare and check storage layer for shnwiktionary - https://phabricator.wikimedia.org/T256010 (10Bstorm) [20:34:39] 10DBA, 10Gerrit, 10Patch-For-Review: Make sure both `reviewdb-test` (used forgerrit upgrade testing) and `reviewdb` (formerly production) databases get torn down - https://phabricator.wikimedia.org/T255715 (10Dzahn) If it hasn't already happened you can remove the grant for the "gerritro" user on m2 as well.... [23:20:41] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 2020-09-14) rack/setup/install dbprov2003.codfw.wmnet - https://phabricator.wikimedia.org/T258749 (10Papaul)