[01:09:33] 07Blocked-on-schema-change, 10DBA, 10Wikidata, 13Patch-For-Review, 03Wikidata-Sprint: Deploy schema change for adding term_full_entity_id column to wb_terms table - https://phabricator.wikimedia.org/T162539#3250296 (10aude) @jcrespo no hurry on our side. (was just more curious how this is going and seems... [05:57:49] 10DBA, 10MediaWiki-Database, 10MediaWiki-Recent-changes: Special:RecentChanges can show recently created page as red link - https://phabricator.wikimedia.org/T129399#2104435 (10Marostegui) So checking the rc slaves for enwiki (db1051 and db1055) there was no lag so probably this is not due to lag on the rece... [06:03:21] 10DBA: Run pt-table-checksum on s7 - https://phabricator.wikimedia.org/T163190#3250592 (10Marostegui) fawiki is done: no differences found [06:04:23] 10DBA: Run pt-table-checksum on s7 - https://phabricator.wikimedia.org/T163190#3250593 (10Marostegui) [06:09:48] 07Blocked-on-schema-change, 10DBA, 10Wikidata, 13Patch-For-Review, 03Wikidata-Sprint: Deploy schema change for adding term_full_entity_id column to wb_terms table - https://phabricator.wikimedia.org/T162539#3250601 (10Marostegui) codfw master, db2023 is done: ``` root@neodymium:~# mysql --skip-ssl -hdb20... [06:10:23] 10DBA, 10Wikidata, 13Patch-For-Review, 07Schema-change: Drop the useless wb_terms keys "wb_terms_entity_type" and "wb_terms_type" on "wb_terms" table - https://phabricator.wikimedia.org/T163548#3250604 (10Marostegui) codfw master, db2023 is done: ``` root@neodymium:~# mysql --skip-ssl -hdb2023.codfw.wmnet... [06:18:38] 10DBA: Run pt-table-checksum on s7 - https://phabricator.wikimedia.org/T163190#3250613 (10Marostegui) >>! In T163190#3250608, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-operations), href=https://tools.wmflabs.org/sal/log/AVvw_2-HQMK9DA-FLN-e} [2017-05-10T06:15:23Z] Deploy a... [06:47:03] Going to reset replication s2: db1054: stop slave; reset slave all; [06:57:40] marostegui, jynus here, please reset replication s2: db1054 [06:58:24] 10DBA: Run pt-table-checksum on s7 - https://phabricator.wikimedia.org/T163190#3250637 (10Marostegui) The checksum on frwiktionary cannot be done yet as all the hosts have the wrong revision table PK (rev_page,rev_id - like if they were rc slaves) and the slave lags. So I will skip this database for now until w... [06:58:40] 10DBA: Run pt-table-checksum on s7 - https://phabricator.wikimedia.org/T163190#3250638 (10Marostegui) [07:14:47] jynus: thanks :) [07:21:52] 10DBA, 10Tool-Labs-tools-Other: Tired of APIError: readonly - https://phabricator.wikimedia.org/T164191#3225024 (10Framawiki) For your jstart problem, you can expressly set PWB path by using `python /shared/pywikipedia/core/pwb.py yourscriptname.py` as command. It's what I use for my bot. {T154011} will norma... [07:22:06] 10DBA, 10Tool-Labs-tools-Other: Tired of APIError: readonly - https://phabricator.wikimedia.org/T164191#3250682 (10Framawiki) [08:46:42] 10DBA, 06Operations: Increase timeout for mariadb replication check - https://phabricator.wikimedia.org/T163303#3250856 (10Marostegui) p:05Triage>03Normal [08:53:57] I've dedided to wait for PK alters until next week [08:54:13] because each one may take 1d-1week [08:54:26] and I do not want to create trouble while I am off [08:56:08] sure, makes total sense :) [08:56:25] I am going to go for 30 minutes, will be back in a bit [09:42:04] 10DBA, 13Patch-For-Review: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#3250959 (10jcrespo) [10:34:35] 07Blocked-on-schema-change, 10DBA, 13Patch-For-Review: Apply change_tag and tag_summary primary key schema change to Wikimedia wikis - https://phabricator.wikimedia.org/T147166#3251051 (10Marostegui) s2 in codfw is done: ``` root@neodymium:/home/marostegui/git/software/dbtools# for i in `cat /home/marostegui... [10:35:14] 07Blocked-on-schema-change, 10DBA, 13Patch-For-Review: Apply change_tag and tag_summary primary key schema change to Wikimedia wikis - https://phabricator.wikimedia.org/T147166#3251052 (10Marostegui) [10:36:02] 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 4 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3251055 (10Marostegui) s2 in codfw is done: ``` root@neodymium:/home/marostegui/git/software/dbtools# for i in `cat... [10:36:10] 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 4 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3251056 (10Marostegui) [10:40:08] Going to reset s7 replication codfw > eqiad, so db1062: stop slave; reset slave all; [10:41:27] db1062 looks good [10:41:39] maybe we should do it on all 8 shards [10:41:51] and leave it like that for some weeks [10:42:04] Only s7, s1 and s3 pending now [10:57:27] I think I am going to start to get ready to decommission db1023 the old old master from s6 [10:58:09] did we copy it elsewhere? [10:58:45] I have not finished checking s6 [10:59:17] can you wait for that? [11:07:31] Ah [11:07:32] sure sure [11:07:35] I thought you finished :) [11:07:43] No rush [11:07:47] well, I always intended to [11:07:51] I just saw it there when checking eqiad file [11:07:58] And I thought it was over [11:08:00] no worries! [11:08:02] No pressure [11:08:04] but something more important happens [11:08:38] yeah, 0 pressure [11:33:37] transfering db1056 while having lunch [11:34:14] hello people, I am writing a python script for the task to have a custom purge of EL old data (T163337) [11:34:15] T163337: Job queue corruption after codfw switch over (Queue worth, duplicate runs) - https://phabricator.wikimedia.org/T163337 [11:34:19] nope [11:34:35] T156933 [11:34:35] T156933: Improve purging for analytics-slave data on Eventlogging - https://phabricator.wikimedia.org/T156933 [11:34:39] better :) [11:35:45] I am wondering if this is something good from your point of view and if you have any suggestion about what could be a good place to run it [11:35:50] (maybe periodically in cron) [11:36:21] eventlogging_sync.sh runs on dbstore1002 and db1047 but it might be not the best place for the new script [11:36:45] we are completely neutral- data itself is something that we do not handle [11:36:56] of course, less rows == better everthing [11:37:48] sure sure, this is just to have a chat about the task, your ideas are important :) [11:37:58] regarding where- I would assume localy to the databases [11:38:15] it should work togheder with the custom replication setup [11:38:47] so one doesn't do and the other undo- think things are purged on the slaves, but taken from the master again [11:39:04] I do not mean they should be the same script [11:39:15] I only mean that the logic should be compatible [11:39:21] yep got it this is a good point [11:39:31] it is not as easy [11:39:40] because writes could also block each other [11:39:40] I'd be tempted to purge the master first and then the slaves, maybe from a maintenance host [11:40:00] that would also make sense, if you need coordination [11:40:00] * elukey takes notes [11:40:28] there are many ways to implement that [11:41:07] other things is do small transactions [11:41:19] so not just DELETE where timestamp < X [11:41:26] because that could be a lots of rows [11:41:35] the main requirement, that complicates the whole script, is that some attributes carrying json structures (like the User Agent) shouldn't be NULLed, but some of their fields needs to be kep [11:41:39] then there is the issue with legacy suff that cannot be deleted [11:41:39] *ket [11:41:43] ok today I can't write [11:41:43] yes [11:41:45] that is [11:41:57] that is when I just bailed out [11:42:12] wise man [11:42:13] it is a complex operation there that cannot be implemented with database logic/triggers [11:42:13] :D [11:42:19] no, I mean [11:42:31] deleting everthing older than X [11:42:34] is easy [11:42:40] but then requirements strated to creep up [11:42:45] yep yep got your point [11:42:50] and I said, this is not a database issue anymore [11:43:00] it is an application issue [11:43:18] and probably should be done on eventlogging itself [11:44:29] for inspiration on how to do good purges [11:44:45] check recentchanges or parsercache way of doing things [11:44:54] they have maintenance script to do that [11:45:19] they use batching, etc. [11:45:46] will note everything in the task, thanks! [11:46:25] so, my point is- ask anything in terms "what will be faster for the db? what whill create less issues for the db?" and I will be able to help you [11:47:25] if you ask me "how to do this complex thing than nobody else does"- I won't be able to help [11:48:36] got it, I needed some feedback like the ones that you gave me, thanks :) [11:49:24] if possible, I would try to run it locally [11:49:47] but you may need coordionation- in which case you may need to run it from a maintenance host [11:50:00] but that could make things more complex [11:50:01] any issue with requirements like having the python-mysql-connector on the db hosts in case? [11:50:12] no, absolutely not [11:50:14] although [11:50:19] we use pymysql [11:50:31] because it has better ssl compatibility [11:50:43] so I would recommend that- but it is not a huge deal [11:51:11] both labs and us use that as the preferred python connector -so I am recommending it to avoid you issues [11:51:46] you may even want to use our WMFMariaDB.py abstraction :-) [11:52:33] even better! I just need something to connect, nothing more [11:52:46] that is actually the point of that library [11:53:10] it will avoid account and password handling for you [11:54:07] if you want feedback, upload to gerrit anything that you may have, the earlier (even if it doesn't work) the better [11:58:03] elukey, https://gerrit.wikimedia.org/r/#/c/338809/4/modules/role/files/mariadb/WMFMariaDB.py [11:59:48] wonderful thanks! [12:00:07] it is not deployed on dbs, though- I have yet to package it [12:00:33] but you an use it to simplify your code or get inspired by it [12:02:42] I was about to start looking at this part for the script, so good timing :) [12:03:46] the script is silly, bit is just a trivial abstraction of pymysql, but it allows avoiding to handle accounts and authentication differences between servers [12:05:06] making this as easy as mysql=WMFMariaDB(host=host); print(mysql.execute('SELECT 1')) [12:09:59] \o/ [12:15:24] I need to document it appropiately, that is why I am delaying deployment [12:43:15] 10DBA, 10Wikidata, 07Performance, 15User-Daniel: DispatchChanges: Avoid long-lasting connections to the master DB - https://phabricator.wikimedia.org/T151681#3251333 (10Ladsgroup) For the record in history. [[https://logstash.wikimedia.org/app/kibana#/dashboard/49196f70-357c-11e7-8b40-49f487863170?_g=(refr... [12:53:09] 07Blocked-on-schema-change, 10DBA, 13Patch-For-Review: Apply change_tag and tag_summary primary key schema change to Wikimedia wikis - https://phabricator.wikimedia.org/T147166#3251339 (10Marostegui) s7 in codfw is done: ``` root@neodymium:~# for i in `cat /home/marostegui/s2_dbs`; do echo $i; mysql --skip-s... [12:54:46] 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 4 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3251340 (10Marostegui) s7 in codfw is done: ``` root@neodymium:~# for i in `cat /home/marostegui/s2_dbs`; do echo $... [12:55:02] 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 4 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3251343 (10Marostegui) [12:55:12] 07Blocked-on-schema-change, 10DBA, 13Patch-For-Review: Apply change_tag and tag_summary primary key schema change to Wikimedia wikis - https://phabricator.wikimedia.org/T147166#3251344 (10Marostegui) [13:09:43] Going to reset replication on s1 codfw > eqiad: db1052: stop slave ; reset slave all; [13:13:14] 10DBA, 10Wikidata, 07Performance, 15User-Daniel: DispatchChanges: Avoid long-lasting connections to the master DB - https://phabricator.wikimedia.org/T151681#3251386 (10jcrespo) So do you think this had something to do with reports like T123867 T164191? This is highly surprising- I was expecting low to no... [13:46:29] 10DBA, 10Wikidata, 07Performance, 15User-Daniel: DispatchChanges: Avoid long-lasting connections to the master DB - https://phabricator.wikimedia.org/T151681#3251549 (10Ladsgroup) Yes, Strangely I removed all selectors in logstash and added them back one by one and made some changes. it worked properly. [... [14:23:30] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: pdu phase inbalances: ps1-a3-codfw, ps1-c6-codfw, & ps1-d6-codfw - https://phabricator.wikimedia.org/T163339#3251689 (10Papaul) @RobH let me know when you want to start working on this. Next week works for me. [14:26:47] I have wasted even more time doing: https://phabricator.wikimedia.org/P5395#29087 [14:28:05] Ah nice, so now we can alert based on SSL if we wanted too! :) [14:30:23] 10DBA: Run pt-table-checksum on s7 - https://phabricator.wikimedia.org/T163190#3251706 (10Marostegui) hewiki is done and only found differences on dbstore1002: ``` Differences on dbstore1002 TABLE CHUNK CNT_DIFF CRC_DIFF CHUNK_INDEX LOWER_BOUNDARY UPPER_BOUNDARY hewiki.geo_tags 1 0 1 PRIMARY 127401 3506921 hewik... [14:30:39] 10DBA: Run pt-table-checksum on s7 - https://phabricator.wikimedia.org/T163190#3251707 (10Marostegui) [14:31:49] 10DBA, 10Monitoring, 06Operations: Create a check/calendar alert for MariaDB TLS certs - https://phabricator.wikimedia.org/T152427#3251708 (10jcrespo) I have created the foundations for this: https://phabricator.wikimedia.org/P5395#29087 [14:39:45] 10DBA, 06Community-Tech, 10MediaWiki-User-management: Do test queries for range contributions to gauge performance of using different tables - https://phabricator.wikimedia.org/T156318#3251737 (10MusikAnimal) @jcrespo Are you able to create that new table now, by chance? Here's the SQL that I think will do w... [14:56:55] 10DBA, 06Community-Tech, 10MediaWiki-User-management: Do test queries for range contributions to gauge performance of using different tables - https://phabricator.wikimedia.org/T156318#3251819 (10jcrespo) yes, I can do this now. [15:06:43] even if you think it was 100% chance, I had db1056 movement in mind for the reimage- I just wasn't sure it was that one [15:10:17] haha [15:10:26] I was surprised that you wanted to reinstall that host just today [15:10:29] And now all makes sense :p [15:10:57] well, it it finally didn't happen, at least it was something I could do in 1 day [15:11:04] unlike most of everthing else [15:11:20] I plan to leave you alone with all the alerts for 2 days [15:11:36] is there something we have pending to discuss? [15:12:22] Do you want me to finish db1056 for you? [15:12:28] Nothing from my side to discuss no [15:12:32] no, I can do that [15:12:41] assuming it comes back early [15:12:55] at most, maybe pooling tomorrow, etc, at your will [15:13:10] I am planning to logoff after the DC meeting (6.30pm) if there is something you don't finish, just send me an email and I can do it for you :) [15:13:15] ie: db1056 :) [15:13:23] ok, mostly that^ [15:13:36] where is the backup? dbstore1001? [15:13:39] don't get to crazy [15:13:41] yes [15:13:41] db1056 backu I mean [15:14:00] I plan to recover, it is the pooling that cannot control fully and will take time [15:14:07] Sure, I will take care of that [15:14:27] don't get to crazy with stuff this week [15:14:42] it would be nice to have one weekend without pages 0:-) [15:15:53] haha [15:15:54] yeah [15:15:58] no worries :) [15:16:06] I am quite tired still [15:16:30] I was planning to slow down a bit this week, but I have failed on that :p [15:16:34] too much stuff to do [15:24:53] are you seeing this: https://logstash.wikimedia.org/goto/a38878e35dea41427b5c803f540c8f15 ? [15:29:51] yeah, I saw you mentioned it earlier [15:29:56] is it logtash not working? :| [15:30:01] see ops- [15:31:54] are you sure I can leave you on your own for 2 days :-) [15:32:01] :( [15:32:13] too many changes tasks at the same time [15:32:15] (I need to tease you a bit becaue of this) [15:32:16] and too many changes [15:32:19] :-D [15:32:27] :-P [15:32:35] !!! [15:32:43] perfect time to deploy all the things [15:32:49] trick [15:32:58] time [15:33:01] open: https://noc.wikimedia.org/conf/highlight.php?file=db-eqiad.php [15:33:18] it will return the config from terbium or tin [15:33:25] (cannot remember which of the 2) [15:33:36] so you know for sure what is deployed and what is not [15:33:52] I am know what happened, I pushed db1097, and i mixed it with db1067 [15:33:55] great [15:33:59] also, after deploying, kibana open [15:34:07] I am going to fix it by creating an alias with git fetch and rebase for my tin user [15:34:15] so it doesn't happen again [15:36:19] Another fix I am going to apply is to slowdown from 120 mph to 100 mph [15:36:24] yep [15:36:31] mistakes will happen [15:36:37] I made millions of these [15:36:47] the important is not to avoid them [15:37:02] but to slow down and check afterwards and detect them [15:37:17] tin would have probably sent an alert in a few minutes [15:37:40] that is exactly what I said to those stressing us out yesterday [15:37:53] you want things done well, not fast! [15:38:24] yeah indeed [15:38:36] Specially those tricky changes where you add new columns and things like that [15:38:42] my 2 tricks, the web page, and doing "git status" all the time [15:38:51] if that helps [15:39:42] yeah, that will help a lot [15:39:52] what I normally do is [15:39:53] +2 [15:39:57] ssh to deployment [15:40:02] and leave git fetch and rebase ready on the CLI [15:40:05] so it is the first thing I see [15:40:11] when I get the merged alert on -ops [15:40:22] I connect, I do "git status" (has anyone f*ed before me?) I fetch, I do git status (how many things am I going to roll forward?, has anyone merged but not deployed) rebase git stattus + git log (am I deploying the right code?) [15:40:47] the other thing I watch for is db-eqiad and db-codfw [15:41:00] so I just take my time [15:41:17] the load, btw, was 0 [15:41:25] I know [15:41:35] but the logs go crazy- so make sure you know you created 0 errors [15:41:43] hehe [15:41:47] yeah, but not nice still [15:41:49] I consider the amount of logs a bug [15:42:50] hopefully, I didn't stress you more, I hope you know me enought that I can joke with you [15:42:57] about this [15:43:15] 10DBA: Drop titlekey table from all databases - https://phabricator.wikimedia.org/T164949#3252028 (10demon) [15:43:45] 10DBA, 07Epic, 07Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2429727 (10demon) [15:44:38] Titlekey is *huge*, actually [15:44:43] That'd be a nice one to drop [15:44:57] jynus: haha yeah, I know you now! [15:44:59] I think you've got almost 1 entry per entry in `page` [15:45:43] jynus: don't know if you saw this: https://gerrit.wikimedia.org/r/#/c/348930/ did that some weeks ago, just take a look and let me know (of course not today, not even next week, just have it on your radar!) [15:45:55] 10DBA: Drop titlekey table from all databases - https://phabricator.wikimedia.org/T164949#3252028 (10jcrespo) Thank you for ticket. For any chance you won't have a way to know on which wikis it was deployed to, even if it is as vague as "all.dblist" 2 years ago? [15:46:01] RainbowSprinkles: yeah, the titkey is actually indeed!! [15:46:08] actually big [15:47:02] 10DBA: Drop titlekey table from all databases - https://phabricator.wikimedia.org/T164949#3252070 (10demon) all.dblist as of about 2 years ago sounds right, to be honest. The more accurate answer is "all wikis that weren't born with Cirrus already as default" [15:47:10] lol [15:47:18] Nearly 5G on enwiki [15:47:18] I said 2 years ago at random [15:47:26] Probably about 3, but close enough [15:47:29] It's going to be /most/ wikis [15:47:50] that seems like an easier target that others [15:48:12] other drops are not as clear or are somtimes present, sometimes not [15:48:42] Yeah, it's been completely obsolete since Cirrus became default everywhere and I undeployed TitleKey [15:48:43] spacee is not as problematic as the huge amount of objects on s3 [15:48:58] 07Blocked-on-schema-change, 10DBA, 07Schema-change: Drop titlekey table from all databases - https://phabricator.wikimedia.org/T164949#3252091 (10Reedy) [15:48:58] And, if we hypothetically ever wanted it again, you can rebuild the data. [15:49:06] 1 table deleted == 800 less objects per server [15:49:11] xddddddd [15:49:33] Reedy, we do not use Blocked-on-schema-change [15:49:52] but it's now so much effort to remove it!! [15:49:54] as documented on https://wikitech.wikimedia.org/wiki/Schema_changes#What_is_not_a_schema_change [15:49:55] oneone11!1! [15:50:06] the reasoning [15:50:12] 10DBA, 07Schema-change: Drop titlekey table from all databases - https://phabricator.wikimedia.org/T164949#3252028 (10Reedy) [15:50:15] is that drops normally do not block anyone [15:50:32] so that helps up prioritize blocking other people [15:50:54] I am just saying so you understand the reason, not just purely pedantic [15:51:04] to make the important changes faster [15:52:06] 10DBA, 07Epic, 07Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3252102 (10demon) I'm curious if dropping unused **core** tables (interwiki, updatelog, etc) is a good idea actually. They can be safely truncated to save... [15:52:19] Reedy: General thought for you mostly ^ [15:52:53] Hence some comments of... [15:52:54] "Not sure if it can be dropped, but safe to truncate if the space is needed" [15:52:54] :P [15:53:04] Well they can safely be dropped [15:53:19] 10DBA, 07Epic, 07Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3252115 (10jcrespo) No, we should definitely NOT drop core tables- we should just truncate them to avoid private data leaking. but we need at least somewhe... [15:53:22] see^ [15:53:27] It's a matter of do we want to cuz future addWiki calls will create them [15:53:44] the idea is to keep all wikis in sync [15:53:55] I'll remove them from the table then [15:54:01] no no [15:54:04] keep them [15:54:11] Mark them as not removeable [15:54:23] but comment the idea that they sood not be dropped, but should be truncated [15:54:25] Keeps the "audit trail" [15:54:29] yep [15:54:34] agreeing on that [15:54:38] And save someone else down the line going "why is this empty table still here?" [15:54:48] +100 to that [15:55:19] I guess an empty, unused table is a pretty minimal maintenance burden [15:55:27] yeah [15:55:43] it is much more burde them existing on some wikis on not on anothers [15:55:56] and the plan is to go back to mediwiki core as much as possible [15:56:05] K [15:56:39] so, to clarfiy, it is not my intention to delete as many tables as possible [15:57:03] it is to drop those that are orphan and a garbage [15:57:15] mostly undeployed ones [15:57:27] * RainbowSprinkles nods [15:57:29] either from extensions, or dropped from mediawiki core [15:57:36] those are the ones that cause issues [15:57:44] thanks for helping , both of you! [15:58:01] this is real archeology [15:58:27] it will take some time, but now we are dropping faster than we are creating, so we are progressing :-) [15:59:44] 10DBA, 07Epic, 07Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3252145 (10demon) [15:59:55] I just put them all as NO, just truncate instead [16:08:39] marostegui: Speaking of easy, the smw_* tables on wikitech is just 2 databases :) [16:10:49] heh [16:10:58] Should probably backup them first, just in case [16:10:58] 10DBA, 10RESTBase, 10Reading List Service, 06Reading-Infrastructure-Team-Backlog (Kanban), and 2 others: Investigate requirements for MySQL access in RESTBase - https://phabricator.wikimedia.org/T164805#3252207 (10Fjalapeno) a:03Tgr [16:11:01] But shouldn't be any lost data in it [16:27:08] 10DBA, 06Operations, 10ops-eqiad: Decommission db1024 - https://phabricator.wikimedia.org/T164702#3252301 (10Cmjohnson) I am ready whenever you are to decom and remove the rack [17:28:45] 10DBA, 07Epic, 07Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3252544 (10Dinoguy1000) [17:57:59] 10DBA, 06Operations: remove mira wikitech grants - https://phabricator.wikimedia.org/T164968#3252704 (10RobH) [18:25:55] 10DBA, 10Wikidata, 07Performance, 15User-Daniel: DispatchChanges: Avoid long-lasting connections to the master DB - https://phabricator.wikimedia.org/T151681#3252828 (10hoo) With the current state, we still have the same amount of connections to the master DBs, but we don't use `GET_LOCK` etc. on them anym... [18:28:18] 10DBA, 10Wikidata, 07Performance, 15User-Daniel: DispatchChanges: Avoid long-lasting connections to the master DB - https://phabricator.wikimedia.org/T151681#3252834 (10jcrespo) > With the current state, we still have the same amount of connections to the master DBs, but we don't use GET_LOCK etc. on them... [18:45:56] 10DBA, 10CirrusSearch, 06Discovery, 06Discovery-Search, 07Performance: archive table needs index starting with timestamp - https://phabricator.wikimedia.org/T164975#3252901 (10Smalyshev) [18:46:48] 07Blocked-on-schema-change, 10DBA, 07Schema-change: Drop titlekey table from all wmf databases - https://phabricator.wikimedia.org/T164949#3252915 (10Umherirrender) [18:48:22] 10DBA, 07Schema-change: Drop titlekey table from all wmf databases - https://phabricator.wikimedia.org/T164949#3252918 (10demon) Removing blocked tag, nothing is blocked on this. [18:49:11] 07Blocked-on-schema-change, 10DBA, 06Labs, 10wikitech.wikimedia.org: Drop Semantic Database tables from wikitech wikis - https://phabricator.wikimedia.org/T164887#3252920 (10Umherirrender) [19:09:19] 10DBA, 10CirrusSearch, 06Discovery, 06Discovery-Search, and 2 others: archive table needs index starting with timestamp - https://phabricator.wikimedia.org/T164975#3252993 (10Umherirrender) [20:10:40] 10DBA, 06Labs, 10wikitech.wikimedia.org, 07Schema-change: Drop Semantic Database tables from wikitech wikis - https://phabricator.wikimedia.org/T164887#3253267 (10Reedy) [20:58:09] 10DBA, 06Operations: dbtree: don't return 200 on error pages - https://phabricator.wikimedia.org/T163143#3253407 (10MoritzMuehlenhoff) p:05Triage>03Normal [21:28:25] 10DBA, 06Operations, 06Performance-Team, 10Traffic: Cache invalidations coming from the JobQueue are causing slowdown on masters and lag on several wikis, and impact on varnish - https://phabricator.wikimedia.org/T164173#3253496 (10aaron) The job run rate and type run rate graphs seem uninteresting in that... [21:29:29] 10DBA, 10CirrusSearch, 06Discovery, 06Discovery-Search, and 3 others: archive table needs index starting with timestamp - https://phabricator.wikimedia.org/T164975#3253505 (10jcrespo) Adding platform because they know the risks of overindexing. [21:34:05] 10DBA, 06Operations, 06Performance-Team, 10Traffic: Cache invalidations coming from the JobQueue are causing slowdown on masters and lag on several wikis, and impact on varnish - https://phabricator.wikimedia.org/T164173#3253516 (10jcrespo) I think this was a one-time user doing multiple purges, we can clo... [22:02:10] 10DBA, 10CirrusSearch, 06Discovery, 06Discovery-Search, and 3 others: archive table needs index starting with timestamp - https://phabricator.wikimedia.org/T164975#3252901 (10Anomie) The danger here, as far as I can tell, would be planner confusion over whether to use `name_title_timestamp` or this new ind... [22:55:02] 10DBA, 10CirrusSearch, 06Discovery, 06Discovery-Search, and 3 others: archive table needs index starting with timestamp - https://phabricator.wikimedia.org/T164975#3253705 (10Smalyshev) I think the way we're using them now (either search by namespace/title prefix, or scan by timestamp) if we add timestamp...