[05:06:21] 10DBA, 10Patch-For-Review: Decommission db1054 - https://phabricator.wikimedia.org/T197063#4277886 (10Marostegui) [05:22:28] 10DBA, 10Patch-For-Review: Productionize old/temporary eqiad sanitariums - https://phabricator.wikimedia.org/T196376#4314072 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['db1095.eqiad.wmnet'] ``` The log can be found in `/var/log/wmf-auto-re... [05:43:30] 10DBA, 10Patch-For-Review: Productionize old/temporary eqiad sanitariums - https://phabricator.wikimedia.org/T196376#4314089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db1095.eqiad.wmnet'] ``` and were **ALL** successful. [05:43:46] 10DBA, 10Patch-For-Review: Productionize old/temporary eqiad sanitariums - https://phabricator.wikimedia.org/T196376#4314090 (10Marostegui) [05:43:52] 10DBA, 10Patch-For-Review: Clean up sanitarium_multisource related code - https://phabricator.wikimedia.org/T196527#4314091 (10Marostegui) 05stalled>03Resolved a:03Marostegui db1095 is now a spare and I have not found any more traces of sanitarium code, so considering this resolved. [05:43:55] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4314094 (10Marostegui) [05:59:17] 10DBA, 10Patch-For-Review: Decommission db1054 - https://phabricator.wikimedia.org/T197063#4314105 (10Marostegui) [06:00:32] 10DBA, 10Operations, 10decommission, 10ops-eqiad, 10Patch-For-Review: Decommission db1054 - https://phabricator.wikimedia.org/T197063#4314111 (10Marostegui) a:05Marostegui>03Cmjohnson db1054 is now ready to be handed over to DCOps for its decommissioning [06:03:24] 10DBA, 10Patch-For-Review: Decommission db1051-db1060 (DBA tracking) - https://phabricator.wikimedia.org/T186320#4314112 (10Marostegui) db1054 is now handed over to DCOps for decommissioning. The only pending host is db1052 (s1 primary master) which is scheduled to be failed over the 18th July (T197069) [06:15:16] 10DBA, 10MediaWiki-API, 10Patch-For-Review, 10Performance: plwiki API request is excessively slow when including a badrevid - https://phabricator.wikimedia.org/T140302#2459623 (10Marostegui) Just FYI, with all the work done to normalize revision's table indexes this T140302#2459710 is no longer happening:... [06:32:20] 10DBA, 10Patch-For-Review: Productionize old/temporary eqiad sanitariums - https://phabricator.wikimedia.org/T196376#4314157 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['db1102.eqiad.wmnet'] ``` The log can be found in `/var/log/wmf-auto-re... [06:49:52] 10DBA, 10Patch-For-Review: Productionize old/temporary eqiad sanitariums - https://phabricator.wikimedia.org/T196376#4314167 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db1102.eqiad.wmnet'] ``` and were **ALL** successful. [07:37:45] 10DBA, 10Patch-For-Review: Productionize old/temporary eqiad sanitariums - https://phabricator.wikimedia.org/T196376#4314251 (10Marostegui) [07:45:09] 10DBA, 10Operations, 10ops-eqiad: Physically move es1017 from D to C row - https://phabricator.wikimedia.org/T197072#4314287 (10Marostegui) @Cmjohnson do you have an estimate date more or less when we can do this movement? Asking just to organize ourselves (within DBA) [07:48:38] 10DBA: switchover es1014 to es1017 - https://phabricator.wikimedia.org/T197073#4314293 (10Marostegui) p:05Triage>03Normal [07:54:35] 10Blocked-on-schema-change, 10Wikibase-Quality, 10Wikibase-Quality-Constraints, 10Wikidata, 10Schema-change: Deploy schema change for adding numeric primary key to wbqc_constraints table - https://phabricator.wikimedia.org/T189101#4314315 (10Marostegui) Just an update: we might go for a DC failover durin... [08:01:35] 10DBA, 10Wikimedia-Site-requests: Server-side deletion of User:LorenzoMilano/sandbox - https://phabricator.wikimedia.org/T198156#4314330 (10revi) Hi DBA, adding #dba per our last interaction (around April). I think ajr didn’t bother to attempt deletion because it has +40000 revisions; do you want me to try del... [08:03:21] 10DBA, 10Wikimedia-Site-requests: Server-side deletion of User:LorenzoMilano/sandbox - https://phabricator.wikimedia.org/T198156#4313878 (10Marostegui) I guess you'd need to delete it in small batches [08:06:49] 10DBA, 10Wikimedia-Site-requests: Server-side deletion of User:LorenzoMilano/sandbox - https://phabricator.wikimedia.org/T198156#4314358 (10revi) One page with 40000 revisions - I've never heard of the way to delete one page in small batches. Is it using sort of API? [08:13:30] 10DBA, 10Wikimedia-Site-requests: Server-side deletion of User:LorenzoMilano/sandbox - https://phabricator.wikimedia.org/T198156#4314388 (10Marostegui) >>! In T198156#4314358, @revi wrote: > One page with 40000 revisions - I've never heard of the way to delete one page in small batches. Is it using sort of API... [08:27:12] 10DBA, 10Wikimedia-Site-requests: Server-side deletion of User:LorenzoMilano/sandbox - https://phabricator.wikimedia.org/T198156#4314413 (10revi) Hmm, OK, then I think it's fair to attempt to delete it via ?action=delete, because I don't know, and you don't know it either. I guess it will fail, but I think it'... [08:27:57] 10DBA, 10Wikimedia-Site-requests: Server-side deletion of User:LorenzoMilano/sandbox - https://phabricator.wikimedia.org/T198156#4314415 (10jcrespo) Just try, we will see what happens. [08:29:04] 10DBA, 10Wikimedia-Site-requests: Server-side deletion of User:LorenzoMilano/sandbox - https://phabricator.wikimedia.org/T198156#4314416 (10revi) Deleting... [08:29:31] 10DBA, 10Wikimedia-Site-requests: Server-side deletion of User:LorenzoMilano/sandbox - https://phabricator.wikimedia.org/T198156#4314417 (10revi) ```To avoid creating high replication lag, this transaction was aborted because the write duration (3.9338247776031) exceeded the 3 second limit. If you are changing... [08:31:08] 10DBA, 10Wikimedia-Site-requests: Server-side deletion of User:LorenzoMilano/sandbox - https://phabricator.wikimedia.org/T198156#4314421 (10jcrespo) As predicted :-) but it is doing what it is supposed to do, instead of crashing it for all users. Batching is the key here, but it may need code and functionality... [08:40:34] 10DBA, 10Wikimedia-Site-requests: Server-side deletion of User:LorenzoMilano/sandbox - https://phabricator.wikimedia.org/T198156#4314428 (10revi) AFAIK such tool (batching for deletion or sorta) does not exist, and everyone prefers shiny new tools over the (unseen) maintenance works, so I dunno if such tool wi... [08:48:13] jynus: wrt that enwiki bigdelete, can I try to do so via API? I think we've deleted other 40k stuff in the past. [08:49:35] if with api means doing the same call- it will fail, too; however, if it means a script does things differently, sure [08:50:22] 10DBA, 10Wikimedia-Site-requests: Server-side deletion of User:LorenzoMilano/sandbox - https://phabricator.wikimedia.org/T198156#4313878 (10MarcoAurelio) Let me try something. [08:59:40] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), and 2 others: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459#4314461 (10jcrespo) I ment... [09:04:09] 10DBA, 10Wikimedia-Site-requests: Server-side deletion of User:LorenzoMilano/sandbox - https://phabricator.wikimedia.org/T198156#4314464 (10MarcoAurelio) Nope. API requests also raising red flags here. You may wish to use [[ https://phabricator.wikimedia.org/source/mediawiki/browse/master/maintenance/deleteBa... [09:11:46] 10DBA, 10Wikimedia-Site-requests: Server-side deletion of User:LorenzoMilano/sandbox - https://phabricator.wikimedia.org/T198156#4314498 (10jcrespo) @MarcoAurelio that deletes several pages in batches, what we need is a way to delete revisions for a single page in batches. I am not sure there is really existin... [09:13:59] 10DBA, 10Wikimedia-Site-requests: Server-side deletion of User:LorenzoMilano/sandbox - https://phabricator.wikimedia.org/T198156#4313878 (10MarcoAurelio) @jcrespo Unless operations knows a way to do that, deleting the page at once is the only option for now even if it creates lag. I've created T198176 to reque... [09:15:59] 10DBA, 10Wikimedia-Site-requests: Server-side deletion of User:LorenzoMilano/sandbox - https://phabricator.wikimedia.org/T198156#4314554 (10jcrespo) > deleting the page at once is the only option That is not really an option- as you saw, it fails to work. I don't think we have an option with existing code. [09:20:07] 10DBA, 10Wikimedia-Site-requests: Server-side deletion of User:LorenzoMilano/sandbox - https://phabricator.wikimedia.org/T198156#4314564 (10MarcoAurelio) Maybe not via the wiki interface or API, but using deleteBatch for that single page will delete it. It'll create lag though, and I understand that DBAs hate... [09:22:11] 10DBA, 10Wikimedia-Site-requests: Server-side deletion of User:LorenzoMilano/sandbox - https://phabricator.wikimedia.org/T198156#4314572 (10jcrespo) deleteBatch will fail too, sorry, it just runs doDeleteArticle, which is the same thing you run on api or browser. [09:24:15] jynus: and https://phabricator.wikimedia.org/source/mediawiki/browse/master/maintenance/runBatchedQuery.php ? [09:24:52] Hauskatze: we literally need new code [09:24:55] I think you can issue a DELETE statement there but I assume it won't create a log entry? [09:25:05] sure, I can delete rows [09:25:15] you sure do :) [09:25:19] but delete rows <> delete article [09:25:39] it does other stuff- purges, remove pages references, links, cache [09:25:42] maybe something can help in the /maintenance or mediawikimaintenance thing [09:26:02] did you see that task I created about the batch deleting? [09:26:08] so if the problem was "please delete 40000 rows" I can do that [09:26:18] I don't have the knowledge to do the other stuff :-( [09:27:17] I think the rows get moved to another table [09:27:25] fields change [09:27:27] etc. [09:27:48] well, delete just archives the revisions in another table yep [09:28:12] not just that, it cannot be done manually, or at least I cannot [09:28:29] maybe Reedy or legoktm knows? [09:29:38] you do not want to break other functionalities, we cannot just edit the tables manually- it is not that easy [10:51:59] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), and 2 others: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459#4314828 (10Ladsgroup) Rega... [10:52:51] Hopefully the task I created will be fixed sometime so this won't be a problem no more [10:52:58] I lack the knowledge to do so [12:14:43] <_joe_> jynus, marostegui can you take a look at JADE, specifically to https://phabricator.wikimedia.org/T196547 ? [12:27:11] <_joe_> I would like you to think if it's sustainable (1 page / revision) [12:38:14] it is not [14:32:44] 10DBA, 10Operations, 10ops-eqiad: Physically move es1017 from D to C row - https://phabricator.wikimedia.org/T197072#4315664 (10Cmjohnson) @marostegui next week sometime is the best I can do right now. [15:00:53] 10DBA, 10Operations, 10ops-eqiad: Physically move es1017 from D to C row - https://phabricator.wikimedia.org/T197072#4315764 (10Marostegui) Sounds good! Just let us know with a day in advance so we can organize it :) Thanks! [16:44:09] 10DBA, 10Operations, 10decommission, 10ops-eqiad: Decommission db1029 and db1031 - https://phabricator.wikimedia.org/T184054#4316274 (10Cmjohnson) [16:44:26] 10DBA, 10Operations, 10decommission, 10Goal: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#4316279 (10Cmjohnson) [16:44:28] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 eqiad machines - https://phabricator.wikimedia.org/T183469#4316278 (10Cmjohnson) [16:50:51] 10DBA, 10Operations, 10monitoring, 10Patch-For-Review: Followup for TLS MariaDB server roll-out - https://phabricator.wikimedia.org/T157702#4316326 (10jcrespo) [16:51:35] 10DBA, 10Operations, 10monitoring, 10Patch-For-Review: Followup for TLS MariaDB server roll-out - https://phabricator.wikimedia.org/T157702#3013769 (10jcrespo) CC @vgutierrez pending work regarding TLS rollout FYI [17:36:50] 10DBA, 10Operations, 10decommission, 10Goal: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#4316559 (10Marostegui) 05Open>03Resolved All the hosts are now totally decommissioned. So this is all done! Thanks everyone for getting all these hosts decommissioned!!