[05:05:00] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) >>! In T238966#6118001, @B... [05:05:42] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) a:03Marostegui [05:07:37] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) @daniel I just realised th... [05:10:03] excellent labsdb1011 is already importing T248086_wb_terms [05:10:11] by monday I think we'll be able to start replication [05:12:42] 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['pc1010.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/20200508... [05:15:53] 10DBA: tl_from index on templatelinks is lingering in production - https://phabricator.wikimedia.org/T252126 (10Marostegui) a:03Marostegui Interesting, this only exists on `jawiki` and `ruwiki` but not on `frwiki` [05:18:50] 10DBA: tl_from index on templatelinks is lingering in production - https://phabricator.wikimedia.org/T252126 (10Marostegui) 05Open→03Resolved Fixed: ` root@cumin1001:/home/marostegui# for i in frwiki jawiki ruwiki; do echo $i; mysql.py -hdb1096:3316 $i -e "show create table templatelinks\G" | grep UNIQUE;don... [05:18:56] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Sustainability (Incident Prevention), 10WorkType-NewFunctionality: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Marostegui) [05:21:34] 10DBA, 10cloud-services-team (Kanban): Reimage labsdb1011 to Buster and 10.4 - https://phabricator.wikimedia.org/T249188 (10Marostegui) Update after 7 days. The host is now running the import of `wikidatawiki`, hopefully by Monday we'll be able to start replication. [05:31:03] 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['pc1010.eqiad.wmnet'] ` and were **ALL** successful. [05:46:09] what's the first thing we should do after it finishes [05:46:19] we could create a copy, but how? [05:46:43] if we shutdown, it may have issues due to different mysql format [05:46:57] if we run mysql_upgrade, we may break things [05:47:21] if we run xtrabackup, we could also crash it [05:53:18] 10DBA: Automate the detection of netcat listen port in transfer.py - https://phabricator.wikimedia.org/T252171 (10jcrespo) p:05Triage→03Medium [06:01:53] 10DBA: transfer.py fails to run 2 commands - https://phabricator.wikimedia.org/T252175 (10jcrespo) The error you are getting is probably because you are using ubuntu, and I think that uses its own userpace solution "ufw" (uncomplicated firewall). We at Wikimedia support only Debian and we use iptables. Another p... [06:09:34] 10DBA: transfer.py fails to run 2 commands - https://phabricator.wikimedia.org/T252175 (10jcrespo) The other "error" you are getting, is not an error, cumin reports failure if the command returns other than 0, but some of the commands will actually return different than 0, and that is ok. In particular, ` bash... [06:11:23] 10DBA: Automate the detection of netcat listen port in transfer.py - https://phabricator.wikimedia.org/T252171 (10jcrespo) Remember to send a patch, even if not finished, sooner than later- it is easier to talk about patches than empty code :-D. [06:48:39] 10DBA: Refactor transfer.py - https://phabricator.wikimedia.org/T252172 (10jcrespo) I think a couple of ideas to start about this: * Move open_firewall and close_firewall to its own Firewall class with open and close methods * Move start_slave and stop_slave to its own MariaDB class with start_replication and s... [07:09:37] jynus: what is the first we should do after what finishes? [07:09:48] ah, labsdb1011 [07:10:15] jynus: probably maybe store the content (binary) somewhere else via transfer [07:10:20] do we have space somewhere? [07:15:38] 10DBA: Upgrade parsercache to Buster and 10.4 - https://phabricator.wikimedia.org/T252182 (10Marostegui) [07:16:38] 10DBA: Upgrade parsercache to Buster and 10.4 - https://phabricator.wikimedia.org/T252182 (10Marostegui) p:05Triage→03Medium a:03Kormat Assigning to @kormat as I think it can be a good exercise as it involves MW deployments. [07:32:44] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10ArielGlenn) >>! In T238966#6118136, @M... [07:40:19] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) >>! In T238966#6118219, @A... [07:52:50] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10daniel) >>! In T238966#6118138, @Maros... [07:53:43] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10daniel) >>! In T238966#6118136, @Maros... [07:56:49] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10daniel) >>! In T238966#6117973, @Bstor... [08:07:10] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) >>! In T238966#6118228, @d... [08:14:29] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10ArielGlenn) When these new fields rev_... [08:17:36] jynus: where could we place the content of labsdb1011 once it has finished importing (a binary copy I mean) [08:17:38] ? [08:18:02] anyware, really [08:18:10] it could even live on labsdb10011 if compressed [08:18:44] otherwise it should fit on dbprov or backup hosts [08:18:57] jynus: I don't mean the logical dump, the binary files, those will be around 7TB [08:19:03] the problems is it is in another network, so some things may not work [08:19:15] yeah, but compressed should take half of that [08:19:34] yeah, but after import is done, we'll be using 7 out of 9 [08:19:38] so I rather get them somewhere else [08:20:02] there're around 2TB free on the pv though, I could expand it [08:20:09] so backup1002 if I would be able to set it up (it is blocked) [08:20:09] but I would prefer to store it outside of it [08:21:03] we don't really have lots of options [08:22:01] backup1001 if it is temporary [08:22:20] there is 19TB under /srv/production [08:22:36] yeah, it will be temporary [08:22:57] but you will need to open network holes [08:23:32] yeah, probably [08:23:37] I will do a test [08:33:54] 0 bytes correctly transferred from labsdb1011.eqiad.wmnet to backup1001.eqiad.wmnet [08:33:55] worked [08:34:57] 0 bytes? [08:34:59] :-D [08:35:11] hehe it was just a 0 lines file :) [08:35:14] maybe try at least 1 byte [08:35:43] 773356 bytes correctly transferred from labsdb1011.eqiad.wmnet to backup1001.eqiad.wmnet [08:35:47] ok :-D [08:35:52] then yeah [08:41:03] https://vitess.io/blog/2020-04-29-announcing-vitess-6/ [08:55:01] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10daniel) >>! In T238966#6118241, @Ariel... [09:00:02] 10DBA, 10Operations, 10Patch-For-Review: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases - https://phabricator.wikimedia.org/T162070 (10Dzahn) I have sent email to all administrators of the projects using it and told them I created a new role simplelamp2 to replace sim... [09:03:38] marostegui: well, i've figured out the exact on-disk blocks that contain the lv. that's not a bad start for the insanity plan :) [09:05:38] that's promising! [09:05:54] I wonder if it is the same on hosts with same disks [09:06:37] root@d-i-test:/tmp# dmsetup table [09:06:37] tank-data: 0 18776064 linear 254:6 2048 [09:07:00] it doesn't really matter - this command gives us all the info we need [09:07:29] go to device 254:6, index in 2038*512, and that's the start of the lv [09:07:33] yeah, it is just curiosity - disks with the same size might have equal blocks [09:07:38] and it's 18776064*512B long [09:07:42] aye [09:28:46] 10DBA, 10Growth-Team, 10MediaWiki-Recent-changes, 10Schema-change: recentchanges table indexes: tmp1, tmp2 and tmp3 - https://phabricator.wikimedia.org/T206103 (10Marostegui) So, I haven't found any more queries that regress without those two indexes being removed. The only query I have found that uses `tm... [09:30:10] 10DBA, 10Growth-Team, 10MediaWiki-Recent-changes, 10Schema-change: recentchanges table indexes: tmp1, tmp2 and tmp3 - https://phabricator.wikimedia.org/T206103 (10Marostegui) @Ladsgroup can you run your script against this table to get an overview of how messy this is in prod? [09:46:54] with this deployment, I think we should be able to close T237927, too https://gerrit.wikimedia.org/r/c/operations/puppet/+/595145 [09:46:54] T237927: Add replication lag (and other checks) to misc all hosts - https://phabricator.wikimedia.org/T237927 [09:47:45] oh nice [09:48:04] You running a PPC? [09:48:10] on it [09:48:14] <3 [09:52:18] the only worry would really be if nagios failed, but I think that was solved during the other deployment [09:52:35] yeah, I checked those hosts [09:53:24] arg, I gave the wrong gerrit number [09:53:30] haha [09:53:40] I hate when that happens or when I give the wrong FQDN [09:53:46] like db1XXX.codfw [09:53:51] or db2XXX.eqiad [09:55:20] I am going at a ticket/day [09:55:42] if I keep this rythm, we would solve all our tickets in 500 days [09:55:58] if we don't create more, of course [09:56:04] shall I put phab in read_only? [09:56:09] https://puppet-compiler.wmflabs.org/compiler1003/22417/ [09:56:12] +1 [09:56:25] any person against? [09:57:10] +1ed [09:57:32] I think it is safe to do quite quickly, revert on any issue [09:57:39] yep [09:57:40] unlike the core deployment yesterday [09:58:34] once monitoring is on all hosts we can thing of a proper abastracted mariadb_monitoring class [09:58:44] yep [09:59:14] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10ArielGlenn) One more question: are the... [10:00:56] were you around when we used to have 4 puppet mysql modules? [10:01:12] yes [10:01:27] AND it was on a different tree, so all deployment required 2 commits? [10:02:27] ah no, that no [10:02:39] I think joined when there was still that but not for mysql [10:02:41] he, so much fun [10:10:29] db2093 is alerting [10:10:31] that is tendril host [10:10:34] I didn't check that [10:10:36] Fixing it [10:11:03] fixed [10:11:06] why did it page? [10:11:11] no idea [10:11:35] did it really page? [10:11:40] it paged me [10:11:46] yeah, me too [10:12:01] I think it may have only paged us [10:12:14] could be [10:12:18] that's good [10:12:23] good yes [10:12:25] but strange [10:12:25] at least we didn't bother people [10:12:27] yeah [10:12:48] maybe group dba means page to dbas? [10:13:12] I think that will all go away after we start using victorops [10:14:09] I thought dba meant "email to dba" [10:14:26] define contactgroup { [10:14:26] contactgroup_name dba [10:14:26] members hoo,jcrespo,volans,marostegui [10:14:26] } [10:16:01] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10daniel) >>! In T238966#6118366, @Ariel... [10:17:02] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10daniel) [10:17:48] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10daniel) [10:19:02] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10daniel) [10:37:23] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) >>! In T238966#6118422, @d... [10:52:09] 10DBA: Refactor transfer.py - https://phabricator.wikimedia.org/T252172 (10Aklapper) @Privacybatm: Where / in which repository is that file located? Context welcome in task descriptions. :) [10:52:43] lol @ Marostegui comment [10:52:52] ? [10:52:53] clearly the systems is not clear enough [10:53:08] well, https://puppet-compiler.wmflabs.org/compiler1003/22420/db1097.eqiad.wmnet/index.html shows admins [10:53:11] and admins is IRC, no? [10:53:24] that is the part that was unclear [10:53:29] and I don't blame you [10:53:44] is_critical => true adds sms automatically [10:53:54] or admins, sms actually [10:54:09] technically, the contact is not needed [10:54:12] automatically but not showing on the puppet compuler? [10:54:15] compiler [10:54:26] depends on the host [10:54:39] note those are the defaults [10:54:44] and may be overriden [10:54:51] if is_critical = false [10:55:04] just to be clear, not discussing how critical that is [10:55:13] yeah, but if you check that compiler output, we are adding admins [10:55:16] and not sms [10:55:17] just saying the whole thing is very confusing [10:55:19] for some services [10:55:23] it is very very confusing [10:55:26] he [10:55:33] so I wasn't laughing at you [10:55:33] I just want to make sure we are not stopping paging for certain things [10:55:36] just at the situation [10:55:49] this shouldn't as complicated [10:55:56] but it is [10:55:59] :-( [10:56:22] in any case, note those are defaults [10:56:34] they will be most likely overriden on call [10:56:49] 🤦 [10:57:12] hoo: https://gerrit.wikimedia.org/r/c/operations/puppet/+/595149/ should fix it forever [10:57:39] just to be clear, not saying the patch is complete or right, it is a first version [10:57:57] we may even want to keep dba for things that could make sense [10:58:05] My point is...I want to make sure we keep SMS alerting for processes [10:58:17] yeah, is_critical => true will do that [10:58:23] no matter the contact [10:58:33] the contact only matters if is_critical => false [10:58:35] ok for me then [10:58:50] it makes not much sense I 100% agree [10:59:13] that is why I was laughing in a "resignation" way :-( [10:59:47] the question is- is there a reason to page to dbas only? [11:00:15] no, of course not [11:00:22] either everyone or no one [11:00:23] a situation where we need to wake up Marius? :-D [11:00:34] or riccardo [11:00:44] then taht should do [11:00:55] also we should check if they should be critical or not [11:00:56] I am fine with waking up riccardo [11:00:59] I saw some mistakes [11:01:07] marostegui: yo know what they say, right? [11:01:13] :) [11:01:30] jynus.com/better-call-volans.jpg [11:02:25] I saw some mistakes, though [11:02:45] "MariaDB disk space #page" is enabled on codfw, as well as mysql processes [11:02:59] not sure that should be for non primary mediawiki? [11:03:16] no, definitely not [11:03:33] that is a separate patch, I think [11:03:39] yep [11:51:19] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) [11:55:24] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) @daniel I have refactored... [12:06:40] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10daniel) >>! In T238966#6118547, @Maros... [12:11:37] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) Thank you - once the chang... [12:12:34] marostegui: so... bad news re: the insane idea earlier [12:12:39] it works [12:12:42] hahaha [12:13:00] kormat: And how easy is to maintain? :) [12:13:20] mm. also bad news: pretty easy, i think. [12:13:51] kormat: let's comment on the task with the details and all that so we can take a deep look and discuss there? I presume we'll need to involve our infra foundations folks [12:14:15] i think a new task for this, but yeah [12:14:19] https://phabricator.wikimedia.org/P11173 is what i have so far [12:14:36] it needs some error checking etc [12:14:58] but basically everything only depends on the formats that dmsetup and sfdisk use, which are unlikely to change [12:15:07] anyway, i'll make a task :) [12:15:18] the good thing is that along the way, we'll be able to get rid of lvm :) [12:15:22] even for existing hosts [12:15:38] how many tasks do we have now for this? XD [12:17:39] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Reedy) >>! In T238966#6118588, @Marost... [12:20:24] marostegui: there is T251768 and T252027 [12:20:25] T252027: debian-installer: partman doesn't allow lvm LVs to be reused when reimaging - https://phabricator.wikimedia.org/T252027 [12:20:25] T251768: Make partman/custom/no-srv-format.cfg work - https://phabricator.wikimedia.org/T251768 [12:20:44] Ah ok [12:20:47] this feels to me like a third thing, which ties into them both [12:20:51] I thought we had more (for some reason) [12:20:55] sure [12:21:02] if we do this, then no-srv-format.cfg can be made to work, sanely [12:21:05] ok :) [12:21:19] I guess this is a subtask of the no-srv-format one [12:21:25] that works [12:21:31] good [12:25:23] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10daniel) >>! In T238966#6118597, @Reedy... [12:26:16] 10DBA: In-place switching from LVM to normal partition - https://phabricator.wikimedia.org/T252195 (10Kormat) [12:26:33] 10DBA: In-place conversion from LVM to normal partition - https://phabricator.wikimedia.org/T252195 (10Kormat) [12:28:19] 10DBA: In-place conversion from LVM to normal partition - https://phabricator.wikimedia.org/T252195 (10Marostegui) [12:28:27] ^ I have reformatted a bit the task description [12:28:59] 10DBA: In-place conversion from LVM to normal partition - https://phabricator.wikimedia.org/T252195 (10Kormat) [14:00:38] I have the review the other misc sections, but this would be the idea: https://gerrit.wikimedia.org/r/c/operations/puppet/+/595153/2/modules/role/manifests/mariadb/misc.pp [14:03:20] aside from the pending work, I won't deploy any of this today [14:28:04] 10DBA: Refactor transfer.py - https://phabricator.wikimedia.org/T252172 (10Privacybatm) [14:32:52] 10DBA: Refactor transfer.py - https://phabricator.wikimedia.org/T252172 (10Privacybatm) [14:37:47] 10DBA: Refactor transfer.py - https://phabricator.wikimedia.org/T252172 (10Privacybatm) >>! In T252172#6118468, @Aklapper wrote: > @Privacybatm: Where / in which repository is that file located? Context welcome in task descriptions. :) @Aklapper Thank you for pointing that out. I will keep that in mind in the f... [14:40:59] 10DBA: Refactor transfer.py - https://phabricator.wikimedia.org/T252172 (10Privacybatm) [14:42:40] 10DBA, 10Growth-Team, 10MediaWiki-Recent-changes, 10Schema-change: recentchanges table indexes: tmp1, tmp2 and tmp3 - https://phabricator.wikimedia.org/T206103 (10Ladsgroup) >>! In T206103#6118337, @Marostegui wrote: > @Ladsgroup can you run your script against this table to get an overview of how messy th... [14:53:34] 10DBA: Automate the detection of netcat listen port in transfer.py - https://phabricator.wikimedia.org/T252171 (10Privacybatm) [14:53:36] 10DBA: Refactor transfer.py - https://phabricator.wikimedia.org/T252172 (10Privacybatm) [14:53:44] 10DBA: transfer.py fails to run 2 commands - https://phabricator.wikimedia.org/T252175 (10Privacybatm) [14:56:46] 10DBA: database query timeouts since April 26th 2020 - https://phabricator.wikimedia.org/T252209 (10doctaxon) [14:58:24] 10DBA, 10Data-Services: database query timeouts since April 26th 2020 - https://phabricator.wikimedia.org/T252209 (10jcrespo) [15:00:27] 10DBA, 10Data-Services: database query timeouts since April 26th 2020 - https://phabricator.wikimedia.org/T252209 (10Marostegui) We are running with less capacity due to issues being discussed at T249188 and the query killer has been set to kill queries that take more than 60 minutes to avoid generating even m... [15:09:20] 10DBA, 10Data-Services: database query timeouts since April 26th 2020 - https://phabricator.wikimedia.org/T252209 (10doctaxon) @Marostegui: my client user is u4802 Is it possible, to exempt this client, that my queries will not longer fail any more? [15:14:00] 10DBA, 10Data-Services: database query timeouts since April 26th 2020 - https://phabricator.wikimedia.org/T252209 (10Marostegui) >>! In T252209#6118981, @doctaxon wrote: > @Marostegui: my client user is u4802 > > Is it possible, to exempt this client, that my queries will not longer fail any more? No, I am s... [15:21:16] 10DBA, 10Data-Services: database query timeouts since April 26th 2020 - https://phabricator.wikimedia.org/T252209 (10doctaxon) @Marostegui: Thank you for your explanation! [15:26:34] 10DBA, 10Data-Services: database query timeouts since April 26th 2020 - https://phabricator.wikimedia.org/T252209 (10jcrespo) For longer term, you should know that we are in constant conversation with cloud team on the need to increase available resources for wikireplicas. That is how things will be able to ru... [15:44:29] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Pchelolo) > So this means that beta wi... [15:50:03] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) >>! In T238966#6119110, @P... [16:00:32] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Bstorm) @daniel I've got this definiti... [16:46:20] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: TBD) rack/setup/install backup1002 + array - https://phabricator.wikimedia.org/T250816 (10Cmjohnson) @jynus I moved the DAC cable to the correct network port now. You should be good to go [17:01:45] 10DBA, 10CheckUser, 10Growth-Team, 10Thanks, 10User-DannyS712: Thanks activities should be logged by CheckUser - https://phabricator.wikimedia.org/T252226 (10DannyS712) a:03DannyS712 Tagging #dba for feedback regarding whether its okay to be expanding the `cu_changes` table [17:03:01] 10DBA, 10CheckUser, 10Growth-Team, 10Thanks, 10User-DannyS712: Thanks activities should be logged by CheckUser - https://phabricator.wikimedia.org/T252226 (10Huji) [17:03:44] 10DBA, 10CheckUser, 10Growth-Team, 10Thanks, 10User-DannyS712: Thanks activities should be logged by CheckUser - https://phabricator.wikimedia.org/T252226 (10Huji) >>! In T252226#6119429, @DannyS712 wrote: > Tagging #dba for feedback regarding whether its okay to be expanding the `cu_changes` table See... [17:05:23] 10DBA, 10CheckUser, 10Growth-Team, 10Thanks, 10User-DannyS712: Thanks activities should be logged by CheckUser - https://phabricator.wikimedia.org/T252226 (10DannyS712) >>! In T252226#6119436, @Huji wrote: >>>! In T252226#6119429, @DannyS712 wrote: >> Tagging #dba for feedback regarding whether its okay... [17:09:39] 10DBA, 10CheckUser, 10Growth-Team, 10Thanks, 10User-DannyS712: Thanks activities should be logged by CheckUser - https://phabricator.wikimedia.org/T252226 (10Huji) >>! In T252226#6119438, @DannyS712 wrote: >>>! In T252226#6119436, @Huji wrote: >>>>! In T252226#6119429, @DannyS712 wrote: >>> Tagging #dba... [17:15:10] 10DBA, 10CheckUser, 10Growth-Team, 10Thanks, 10User-DannyS712: Thanks activities should be logged by CheckUser - https://phabricator.wikimedia.org/T252226 (10ST47) In T247540, Reedy indicated that the table has about 60,000,000 rows on wikidata and 10,000,000 on enwiki. [17:32:07] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 31st May) rack/setup/install db213[6-9] and db2140 - https://phabricator.wikimedia.org/T251639 (10Papaul) [19:53:59] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10DannyS712) [20:12:21] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Schema-change: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Jdforrester-WMF) [20:36:55] 10DBA, 10CheckUser, 10Growth-Team, 10Thanks, 10User-DannyS712: Thanks activities should be logged by CheckUser - https://phabricator.wikimedia.org/T252226 (10Huji) [21:01:58] 10DBA: transfer.py fails to run 2 commands - https://phabricator.wikimedia.org/T252175 (10Privacybatm) Thank you for the clarification. I have >>! In T252175#6118162, @jcrespo wrote: > The error you are getting is probably because you are using ubuntu, and I think that uses its own userpace solution "ufw" (unc...