[06:53:15] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) The following hosts have been serving fine during the weekend, no crashes, no InnoDB errors. clouddb1013:3311 clouddb1013:3313 clouddb1015:3316 clouddb1017:3311 cl... [06:59:57] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) [07:00:03] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Marostegui) [07:06:40] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission es1012.eqiad.wmnet - https://phabricator.wikimedia.org/T268101 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: `es1012.eqiad.wmnet` - es1012.eqiad.wmnet (**PASS**) - Downtimed host on Icinga... [07:07:01] volans elukey thanks - I have tried the decom and the only error I got this time was: https://phabricator.wikimedia.org/P13359 but it finished with exit_code=0 anyways [07:08:03] 10DBA, 10decommission-hardware, 10Patch-For-Review: decommission es1012.eqiad.wmnet - https://phabricator.wikimedia.org/T268101 (10Marostegui) a:05Marostegui→03wiki_willy [07:08:31] marostegui: hola! Yes that is the temp fix that Riccardo put in place, it happened also to me that after some retries it goes through and finishes the rest correctly [07:08:43] \ø/ [07:09:03] the netbox api is a little weird IIUC [07:12:25] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Marostegui) s5 progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1003 [x] db1150 [] db1145 [] db1144 [] db1130 [] db1124 [x] db1113 [x] db1110 [] db1100 [x]... [07:12:28] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) s5 progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1003 [x] db1150 [] db1145 [] db1144 [] db1130 [] db1124 [x] db1113 [x] db1110 [] db1100 [x] db... [07:15:58] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) [07:16:13] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Marostegui) [07:19:04] 10Blocked-on-schema-change, 10DBA: Schema change for timestamp fields of jobs table - https://phabricator.wikimedia.org/T268391 (10Marostegui) p:05Triage→03Medium [07:19:32] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Marostegui) p:05Triage→03Medium [07:26:07] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) [07:30:11] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) On-going transfers: db1125:3314 -> clouddb1017 db1125:3314 -> clouddb1019 [07:33:00] 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Deploy labsdbuser and views to new clouddb hosts - https://phabricator.wikimedia.org/T268312 (10Marostegui) >>! In T268312#6637752, @bd808 wrote: >>>! In T268312#6637396, @Bstorm wrote: >> @Marostegui Random question: where does ce... [07:41:28] 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Deploy labsdbuser and views to new clouddb hosts - https://phabricator.wikimedia.org/T268312 (10Marostegui) @Bstorm can you test the script against clouddb1013:3311? On that host I have: - Created the labsdbuser role - Granted all... [07:45:03] 10DBA, 10Orchestrator, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) [08:14:09] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) [08:14:13] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Marostegui) [08:17:32] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Marostegui) s2 progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [] db1146 [] db1129 [] db1125 [] db1122 [] db1105 [x] db1095 [x] db1090 [] db1076 [] db... [08:17:35] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) s2 progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [] db1146 [] db1129 [] db1125 [] db1122 [] db1105 [x] db1095 [x] db1090 [] db1076 [] db1074 [08:17:55] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) [08:18:06] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Marostegui) [08:23:36] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) [08:23:41] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Marostegui) [08:25:34] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Marostegui) [08:25:46] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) [08:29:20] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) [08:29:24] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Marostegui) [08:32:59] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Marostegui) s4 progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [x] db1150 [] db1149 [] db1148 [] db1147 [x] db1146 [x] db1145 [x] db1144 [] db1143 []... [08:33:02] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) s4 progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [x] db1150 [] db1149 [] db1148 [] db1147 [x] db1146 [x] db1145 [x] db1144 [] db1143 [] db... [08:33:23] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) [08:33:36] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Marostegui) [08:38:14] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10Marostegui) @Cmjohnson es1011, es1012 and es1014 are now ready for you to unrack. They are on different racks (and they are also 2U) so we can use those three rack... [08:39:08] 10DBA, 10decommission-hardware: decommission es1013.eqiad.wmnet - https://phabricator.wikimedia.org/T268436 (10Marostegui) [09:30:30] marostegui: yeah, what elukey said [09:39:58] jynus: the time manuel suggested was an hour earlier - 16:00 -> 17:00 [09:40:14] oh [09:40:17] let me change it [09:40:36] wait, 16 utc or cest, I think I used utc [09:40:40] *cet [09:40:54] and he had onfire before [09:42:03] i would assume CET, as we're all in CET. he declined the onfire meeting [09:42:09] ok [09:48:58] yeah if we can do 4pm-5pm cet [09:49:02] I would appreciate it [10:00:33] I saw no updates on db1139, I am going to start it [10:46:39] 10DBA, 10Anti-Harassment, 10CheckUser, 10Stewards-and-global-tools, and 3 others: Incomplete i18n for log entries in CheckUser - https://phabricator.wikimedia.org/T41013 (10MarcoAurelio) [10:53:06] heh [10:53:14] I actually made it on time back, but let's not move the meeting around more times :) [11:04:41] 10DBA, 10Data-Services: Prepare and check storage layer for skrwiki - https://phabricator.wikimedia.org/T268412 (10LSobanski) Thanks, let us know when the database is created, so we can sanitize it. [11:09:58] 10DBA, 10Data-Services: Prepare and check storage layer for skrwiki - https://phabricator.wikimedia.org/T268412 (10LSobanski) https://phabricator.wikimedia.org/T265321 was resolved last week so the ipblocks_restrictions alter should no longer be necessary. [11:15:33] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) [11:19:13] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) [11:25:51] 10DBA, 10Data-Services: Prepare and check storage layer for skrwiki - https://phabricator.wikimedia.org/T268412 (10LSobanski) p:05Triage→03Medium [11:59:09] 10DBA, 10mariadb-optimizer-bug: Investigate possible optimizer regression on 10.4.17 with DELETE statements - https://phabricator.wikimedia.org/T268457 (10Marostegui) [12:10:39] 10DBA, 10mariadb-optimizer-bug: Investigate possible optimizer regression on 10.4.17 with DELETE statements - https://phabricator.wikimedia.org/T268457 (10Marostegui) Reported to mariadb: https://jira.mariadb.org/browse/MDEV-24266 [12:11:44] 10DBA, 10mariadb-optimizer-bug: Investigate possible optimizer regression on 10.4.17 with DELETE statements - https://phabricator.wikimedia.org/T268457 (10Marostegui) p:05Triage→03High Let's not upgrade more hosts until we are sure on what's going on here. I want to upgrade another pc1 host (maybe pc2007)... [12:23:48] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) >>! In T267090#6639561, @Marostegui wrote: > On-going transfers: > > db1125:3314 -> clouddb1017 > db1125:3314 -> clouddb1019 These two crashed after being starte... [12:24:58] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) [12:59:36] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) [12:59:41] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Marostegui) [13:00:57] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Marostegui) [13:00:57] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) [13:04:23] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Marostegui) s7 progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1003 [] db1136 [] db1127 [] db1125 [x] db1116 [x] db1101 [x] db1098 [] db1094 [] db1090 [] d... [13:04:25] 10Blocked-on-schema-change, 10DBA: Drop default of ip_changes.ipc_rev_timestamp - https://phabricator.wikimedia.org/T267399 (10Marostegui) [13:04:27] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) s7 progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1003 [] db1136 [] db1127 [] db1125 [x] db1116 [x] db1101 [x] db1098 [] db1094 [] db1090 [] db1... [13:04:31] 10DBA, 10Operations, 10Security, 10User-Kormat: Reboot es1024 (es5 master) - https://phabricator.wikimedia.org/T268469 (10Kormat) [13:04:33] 10Blocked-on-schema-change, 10DBA: Drop default of protected_titles.pt_expiry - https://phabricator.wikimedia.org/T267335 (10Marostegui) [13:04:37] 10DBA, 10Operations, 10Security, 10User-Kormat: Reboot es1024 (es5 master) - https://phabricator.wikimedia.org/T268469 (10Kormat) [13:05:12] 10DBA, 10Operations, 10Security, 10User-Kormat: Reboot es1024 (es5 master) - https://phabricator.wikimedia.org/T268469 (10Kormat) [13:20:51] 10DBA, 10Operations, 10Patch-For-Review, 10Security, 10User-Kormat: Reboot es1024 (es5 master) - https://phabricator.wikimedia.org/T268469 (10Kormat) [13:21:42] 10DBA, 10Operations, 10Patch-For-Review, 10Security, 10User-Kormat: Reboot es1024 (es5 master) - https://phabricator.wikimedia.org/T268469 (10Kormat) [15:47:43] 10DBA, 10Operations, 10Patch-For-Review, 10Security, 10User-Kormat: Reboot es1024 (es5 master) - https://phabricator.wikimedia.org/T268469 (10Kormat) [15:51:17] 10DBA, 10Operations, 10Patch-For-Review: Cleanup heartbeat.heartbeat on all production instances - https://phabricator.wikimedia.org/T268336 (10Kormat) Cleaning up the heartbeat tables in prod is a bit tricky, as there's a lot of cruft, and a mix of STATEMENT vs ROW replication. My suggestion is that we upda... [16:54:01] 10DBA: New database request: sockpuppet - https://phabricator.wikimedia.org/T268505 (10hnowlan) [17:32:00] 10DBA: New database request: sockpuppet - https://phabricator.wikimedia.org/T268505 (10akosiaris) [17:41:48] 10DBA: New database request: sockpuppet - https://phabricator.wikimedia.org/T268505 (10WDoranWMF) [17:50:47] 10DBA: New database request: sockpuppet - https://phabricator.wikimedia.org/T268505 (10jcrespo) Apologies if this has already been discussed but from the initial description, there are 3 things that are not entirely clear from the thescription. I have added the DBAs as they will be the best people to help you re... [18:22:27] 10DBA: New database request: sockpuppet - https://phabricator.wikimedia.org/T268505 (10hnowlan) >>! In T268505#6641734, @jcrespo wrote: > > * Could you describe more in detail about the kind of load this will receive? While it seems to be light on reads (few queries) from the description, it looks that there wi... [21:13:46] Did I tell you that now 75% of core tables have been migrated to abstract schema? [21:13:58] Thanks to Ammar [23:17:56] We should probably create an extension (tarball/wmf prod) hit list too [23:19:01] though, having said that, I think I created tasks already [23:19:08] Amir1: Don't forget, you did numerous of them too ;) [23:20:26] ^^ [23:20:31] Wikibase is already done [23:20:49] you did lots of extensions too. Abusefilter would be a nice one IMO [23:21:12] have everything already separated: https://github.com/wikimedia/mediawiki-extensions-AbuseFilter/blob/master/abusefilter.tables.sqlite.sql [23:28:56] Wasn't the AbuseFilter one blocked on something... [23:29:06] Or was that PG support blocked on abstract schema? [23:30:19] https://phabricator.wikimedia.org/T261911 [23:30:27] only OATHAuth left for tarball [23:30:59] prod has a few more [23:31:00] https://phabricator.wikimedia.org/T261912 [23:33:30] I might create a load more tasks to get it on various teams radars [23:33:59] Amir1: there's still at least 2 Wikibase related extensions to migrate still [23:34:24] oooh [23:34:27] I forgot [23:42:11] 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Deploy labsdbuser and views to new clouddb hosts - https://phabricator.wikimedia.org/T268312 (10Bstorm) I'm aiming to set up https://gerrit.wikimedia.org/r/c/operations/puppet/+/642503 to be a sorta noop on the existing labsdb* th...