[04:39:38] 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for banwiki - https://phabricator.wikimedia.org/T234770 (10Marostegui) Let us know when the database is created so we can sanitize it on labs hosts [05:08:43] 10DBA, 10conftool, 10Performance-Team (Radar): #dbctl: manage 'externalLoads' data - https://phabricator.wikimedia.org/T229686 (10Marostegui) >>! In T229686#5547838, @CDanis wrote: > @Marostegui please give me some input on the UI -- what would you like dbctl to call the section that will represent "everythi... [05:21:12] 10DBA: Remove ar_comment from sanitarium triggers - https://phabricator.wikimedia.org/T234704 (10Marostegui) [05:44:07] 10DBA: Remove ar_comment from sanitarium triggers - https://phabricator.wikimedia.org/T234704 (10Marostegui) [06:39:51] 10DBA, 10conftool, 10Performance-Team (Radar): #dbctl: manage 'externalLoads' data - https://phabricator.wikimedia.org/T229686 (10Volans) @CDanis my 2 cents are with Manuel to use `es[123]` on the dbctl side and have the mapping `es->cluster` in `mediawiki-config` code as it is right now (with comments in th... [06:43:38] 10DBA, 10conftool, 10Performance-Team (Radar): #dbctl: manage 'externalLoads' data - https://phabricator.wikimedia.org/T229686 (10Marostegui) >>! In T229686#5550742, @Volans wrote: > @CDanis my 2 cents are with Manuel to use `es[123]` on the dbctl side and have the mapping `es->cluster` in `mediawiki-config`... [06:47:45] 10DBA, 10conftool, 10Performance-Team (Radar): #dbctl: manage 'externalLoads' data - https://phabricator.wikimedia.org/T229686 (10jcrespo) es1: currently read only clusters, all <24 es2 es3: currently rw clusters containing cluster24 and 25 respectively (but distribution may change in the future), they will... [07:12:36] 10DBA: Remove grants for the old dbproxy hosts from the misc databases - https://phabricator.wikimedia.org/T231280 (10Marostegui) The following grants for dbproxy1006 IP's have been removed from m1 databases: ` # host dbproxy1006 dbproxy1006.eqiad.wmnet has address 10.64.16.159 root@db1135.eqiad.wmnet[mysql]>... [07:12:45] 10DBA: Remove grants for the old dbproxy hosts from the misc databases - https://phabricator.wikimedia.org/T231280 (10Marostegui) [07:13:28] 10DBA: Decommission dbproxy1006.eqiad.wmnet - https://phabricator.wikimedia.org/T233207 (10Marostegui) [07:23:36] 10DBA: Decommission dbproxy1006.eqiad.wmnet - https://phabricator.wikimedia.org/T233207 (10Marostegui) [08:18:28] hi labsdb1010/1011 had issues over the weekend [08:18:44] arturo: I updated the task yesterday night [08:19:09] oh [08:20:10] thanks! [08:20:24] yw [09:14:52] 10DBA, 10Operations: Switchover s1 primary database master db1067 -> db1083 - 14th Nov 05:00 - 05:30 UTC - https://phabricator.wikimedia.org/T234800 (10Marostegui) [09:15:01] 10DBA, 10Operations: Switchover s1 primary database master db1067 -> db1083 - 14th Nov 05:00 - 05:30 UTC - https://phabricator.wikimedia.org/T234800 (10Marostegui) p:05Triage→03Normal [09:27:56] akosiaris: puppet finally run cleanly on backup* \o/ [09:28:02] *ran [09:28:06] \o/ [09:28:12] small question [09:28:23] are we confident enough on running it on old hosts as is? [09:28:32] puppet is disabled there at the moment [09:28:43] or do we need extra tweaks on service, etc? [09:30:30] jynus: hey, is it okay if I turn on all of wikidata to write_both? the contention will go up slightly [09:31:11] Amir1: sorry, disconnected with wikidata state, would ask manuel [09:31:22] Amir1: if needed, can that be turned off? [09:31:29] Sure [09:31:31] marostegui: yup [09:31:34] jynus: we can try with heze and see if everything is ok [09:31:38] ok [09:31:49] let me know if things go mayhem [09:32:04] akosiaris: doing that [09:32:07] Amir1: I am going to go into a meeting in a bit and then lunch, can we do it later or tomorrow morning? [09:32:42] it will happen in 1 pm CEST [09:33:15] is that set in stone? [09:33:28] That's SWAT window, I can pick another time [09:33:49] I would prefer to do it later than that, I will be getting out from a meeting at that time and going for lunch [09:34:13] akosiaris: a refresh (done automatically) worked ok [09:34:20] should I do a full restart? [09:34:53] Amir1: there is nothing scheduled at 3pm, maybe we can do it at that time? [09:34:58] actually a refresh may have done that: "heze bacula-sd[3535]: Starting Bacula Storage daemon...: bacula-sd" [09:35:12] marostegui: sure thign [09:35:15] *thing [09:35:16] Amir1: thanks :) [09:36:01] akosiaris: will leave it working for some time, monitor logs, will wait for helium [09:38:03] There is some more worring stuff on logs, however: blk_update_request: I/O error, dev sdb, sector 4104 [09:39:00] jynus: IIRC bacula-sd is not refreshable, just restartable [09:39:17] yeah, that is what I saw [09:39:33] and yes, ps concurs on that as well [09:40:04] still, not the first time I have seen "It didn't work after reboot" if you know what I mean [09:40:43] I am just not sure we have lots of time left to be super- careful :-D [09:41:49] heh, I don't think there is too much rush [09:42:01] oh, I was referring to the disks complaining [09:42:03] I mean we can suffer from some probably impending hard disk related doom [09:42:30] I will, as I said, leave puppet disabled on helium for now [09:42:47] while I finish bacula setup on the new ones [09:52:05] 10DBA, 10Operations: Decommission db1061-db1073 - https://phabricator.wikimedia.org/T217396 (10Marostegui) [13:00:56] 10DBA: Change PK and remove partitions from the logging table - https://phabricator.wikimedia.org/T233625 (10Marostegui) [13:16:19] 10DBA: Change PK and remove partitions from the logging table - https://phabricator.wikimedia.org/T233625 (10Marostegui) [13:19:43] marostegui: Let me know when you're ready to deploy this [13:19:53] Amir1: go for it! [13:37:10] 10DBA, 10Analytics: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10elukey) [13:44:28] 10DBA, 10Analytics: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10Marostegui) I am fine with this plan. I assume this service will be still owned and maintained by Analytics, right? (Of course we can help with the setup and all that as we normally do). What I w... [13:50:14] 10DBA, 10Analytics: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10elukey) >>! In T234826#5552110, @Marostegui wrote: > I am fine with this plan. I assume this service will be still owned and maintained by Analytics, right? (Of course we can help with the setup... [13:51:46] 10DBA, 10Analytics: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10Marostegui) >>! In T234826#5552125, @elukey wrote: >>>! In T234826#5552110, @Marostegui wrote: >> I am fine with this plan. I assume this service will be still owned and maintained by Analytics,... [13:55:02] 10DBA, 10Analytics: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10elukey) >>! In T234826#5552128, @Marostegui wrote: >>>> Important note about the log database: the plan is to take a full snapshot of the db and archive it in HDFS before starting any procedure.... [14:01:58] 10DBA, 10Analytics: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10Marostegui) Sure, I just wanted to make sure expectations for the users will be handled beforehand :) [14:12:32] 10Blocked-on-schema-change, 10DBA: Schema change to rename user_newtalk indexes - https://phabricator.wikimedia.org/T234066 (10Marostegui) [14:12:43] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) [15:02:19] andrewbogott: I think we'd need to plan that with a bit more time to be honest, it is 5pm now, we have the meeting in 1h and I have been working since 6:30 AM :-) so I rather not break things this late for me :) [15:02:34] IF I remember, that was the database you decided to run on your own? [15:02:52] andrewbogott: We'd probably need to plan a bit, the database, review all the tables that'd need conversion etc [15:04:02] marostegui: that's fair. We'll see how to get untangled from the state we're in now [15:04:57] andrewbogott: As suggested earlier, maybe you just want to get the current blocker fixed by hacking latin1 and then we can revisit all the charset conversion as I am sure this will be an issue on the next upgrade too [15:05:13] yep, working on that [15:05:29] cool [15:06:38] feel free to create a new task and get me added there so we can discuss further steps for this [15:07:09] marostegui: could you please create the table again by hand? [15:07:18] sure [15:07:20] we will hack the migartion script to don't try to create it at all [15:07:30] created [15:07:39] you can just add CREATE TABLE IF NOT EXISTS to the script [15:07:45] or as you said, just by pass it [15:09:00] the thing is that I'm not sure where to introduce that `IF NOT EXISTS` [15:09:11] BTW our last try didn't work [15:09:25] CREATE TABLE IF NOT EXISTS `qos_dscp_marking_rules` blablabla [15:09:53] I mean, the script is a python script, not a SQL script [15:10:09] (a complex python script, for more context) [15:10:19] Ah sorry - I thought you meant the mysql syntax :) [15:14:09] ok, one things [15:14:17] we are blocked in the middle of the migration [15:16:38] * arturo does this make sense? https://phabricator.wikimedia.org/P9249#55506 [15:17:01] from a mysql point of view, it does:) [15:17:36] ok, and the create tables by hand OR try to insert the CHARSET= thing in that cmd as well [15:18:00] not sure if I understand that last sentence [15:18:14] You mean what is preferred? [15:27:25] 10DBA, 10Analytics: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10Milimetric) p:05Triage→03High [15:27:33] 10DBA, 10Analytics: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10Milimetric) p:05High→03Normal [15:29:33] 10DBA: Change PK and remove partitions from the logging table - https://phabricator.wikimedia.org/T233625 (10Marostegui) [15:33:27] apparently this did the trick [15:33:33] `neutron-db-manage --database-connection 'mysql+pymysql://neutron:@m5-master.eqiad.wmnet/neutron?charset=latin1' upgrade head ` [15:33:43] uff [15:34:06] yes, gross. We will make a follow up task to get the encoding in order [15:34:06] you have to do what you have to do [15:34:13] we seem to be unstuck for now, we'll open a ticket about patching up that database later [15:34:32] which is what I actually suggested, don't get me wrong [15:36:27] 10DBA, 10Cloud-VPS, 10cloud-services-team (Kanban): CloudVPS: m5-master databases for openstack may require re-enconding - https://phabricator.wikimedia.org/T234830 (10aborrero) [15:36:37] also, https://phabricator.wikimedia.org/P9249#55506 doesn't seem right as it seems like a patch for the postgress connector [15:36:44] 10DBA, 10Cloud-VPS, 10cloud-services-team (Kanban): CloudVPS: m5-master databases for openstack may require re-enconding - https://phabricator.wikimedia.org/T234830 (10aborrero) p:05Triage→03High [15:36:51] I just created T234830 [15:36:52] T234830: CloudVPS: m5-master databases for openstack may require re-enconding - https://phabricator.wikimedia.org/T234830