[05:47:31] 10DBA, 10Patch-For-Review: dbstore2001: s5 thread isn't able to catch up with the master - https://phabricator.wikimedia.org/T168354#3464488 (10Marostegui) 05Open>03Resolved So, the scope of this task (s5) was fixed a couple of weeks ago. However, there is a performance problem with this host, which will o... [05:49:15] 10DBA, 10Patch-For-Review: Finish dbstore2002 migration to multi-instance - https://phabricator.wikimedia.org/T171321#3464491 (10Marostegui) Once all the possible shards that fit in this host are done, we better copy them to dbstore2001 as part of: T168409 - see this for context: T168354#3464488 [05:57:21] 10DBA, 10Operations, 10Performance-Team, 10Availability (Multiple-active-datacenters): Perform testing for TLS effect on connection rate - https://phabricator.wikimedia.org/T171071#3453087 (10Marostegui) Hey, I would be nice to do a test with MariaDB 10.0 and 10.1 if possible, to see if there are any reg... [06:30:55] 10DBA, 10Epic: Meta ticket: Migrate multi-source database hosts to multi-instance - https://phabricator.wikimedia.org/T159423#3464554 (10jcrespo) [06:30:57] 10DBA, 10Patch-For-Review: Setup dbstore2002 with 2 new mysql instances from production and enable GTID - https://phabricator.wikimedia.org/T169510#3464549 (10jcrespo) 05Open>03Resolved GTID was enabled last week- there is some merit to ask if we should share the same server and domain id, but for now it w... [06:32:17] 10DBA, 10Patch-For-Review: Finish dbstore2002 migration to multi-instance - https://phabricator.wikimedia.org/T171321#3460914 (10jcrespo) [06:38:55] 10DBA: Truncate table "job" on all wmf sites - https://phabricator.wikimedia.org/T169377#3464572 (10Marostegui) 05Open>03Resolved a:03Marostegui The current situation is the ideal situation :-) : s1 - table is empty s2 - table is empty s3 - table is empty - and exists on all the wikis on s3 dblist. s4 - t... [06:38:59] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3464575 (10Marostegui) [06:48:03] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3464591 (10jcrespo) [07:03:31] 10DBA: Delete _image_new.ibd table if exists somewhere - https://phabricator.wikimedia.org/T171429#3464614 (10Marostegui) [07:04:06] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2547780 (10Marostegui) [07:07:17] 10DBA, 10MediaWiki-extensions-ClickTracking, 10Operations: Drop the tables old_growth, hitcounter, click_tracking, click_tracking_user_properties from enwiki, maybe other schemas - https://phabricator.wikimedia.org/T115982#3464633 (10Marostegui) [07:09:01] 10DBA, 10MediaWiki-extensions-ClickTracking, 10Operations: Drop the tables old_growth, hitcounter, click_tracking, click_tracking_user_properties from enwiki, maybe other schemas - https://phabricator.wikimedia.org/T115982#1737574 (10Marostegui) old_growth status: ``` Only exists on s1 ``` [07:09:21] 10DBA, 10MediaWiki-extensions-ClickTracking, 10Operations: Drop the tables old_growth, hitcounter, click_tracking, click_tracking_user_properties from enwiki, maybe other schemas - https://phabricator.wikimedia.org/T115982#3464641 (10Marostegui) [07:12:35] 10DBA, 10MediaWiki-extensions-ClickTracking, 10Operations: Drop the tables old_growth, hitcounter, click_tracking, click_tracking_user_properties from enwiki, maybe other schemas - https://phabricator.wikimedia.org/T115982#3464658 (10Marostegui) click_tracking status: ``` Exists on: s1, s2, s3, s4, s5, s6, s... [07:13:07] 10DBA, 10MediaWiki-extensions-ClickTracking, 10Operations: Drop the tables old_growth, hitcounter, click_tracking, click_tracking_user_properties from enwiki, maybe other schemas - https://phabricator.wikimedia.org/T115982#3464659 (10Marostegui) [07:15:18] 10DBA, 10MediaWiki-extensions-ClickTracking, 10Operations: Drop the tables old_growth, hitcounter, click_tracking, click_tracking_user_properties from enwiki, maybe other schemas - https://phabricator.wikimedia.org/T115982#3464660 (10Marostegui) click_tracking_user_properties status: ``` Exists on: s1, s2, s... [07:15:33] 10DBA, 10MediaWiki-extensions-ClickTracking, 10Operations: Drop the tables old_growth, hitcounter, click_tracking, click_tracking_user_properties from enwiki, maybe other schemas - https://phabricator.wikimedia.org/T115982#3464661 (10Marostegui) [07:19:35] 10DBA, 10MediaWiki-extensions-ClickTracking, 10Operations: Drop the tables old_growth, hitcounter, click_tracking, click_tracking_user_properties from enwiki, maybe other schemas - https://phabricator.wikimedia.org/T115982#3464672 (10Marostegui) I have renamed old_growth table on db1089 and will leave it lik... [08:02:17] 10DBA: Drop pre_ tables - https://phabricator.wikimedia.org/T118859#3464792 (10Marostegui) I have backuped the existing tables at dbstore1001: ``` root@dbstore1001:/srv/tmp/T118859# pwd /srv/tmp/T118859 root@dbstore1001:/srv/tmp/T118859# ls -lh total 997M -rw-r--r-- 1 root root 463M Jul 24 07:45 s2.tar.gz -rw-r-... [08:16:23] 10DBA, 10Patch-For-Review: Defragment s4: db1091, db1084, db1081, d1059 and probably the rest - https://phabricator.wikimedia.org/T161088#3464822 (10Marostegui) Servers that need to be migrated to file per table: db1059 db1084 db1091 db1097 ``` root@neodymium:/home/marostegui/git/software/dbtools# cumin 'db... [08:17:24] 10DBA, 10Patch-For-Review: Migrate some s4 hosts to file per table - https://phabricator.wikimedia.org/T161088#3464828 (10Marostegui) p:05Triage>03Normal [08:27:04] marostegui: just FYI you could also use a cumin's selector like: 'R:class = mariadb::heartbeat and R:class%shard = s4 and *.eqiad.wmnet' ;) (Re: ^^^) [08:28:18] volans: <3 [08:28:29] I basically started with two hosts only, and then found out I had to do all of them XDD [08:28:35] so I just added them to the regex hehe [08:28:44] eheheh :) [08:29:49] * marostegui definitely thinks that volans has a highlight for when cumin is mentioned on irc [08:30:54] * volans neither confirm nor deny (Glomar response) [08:31:35] XDDDDDD [08:45:26] 10DBA: Delete _image_new.ibd table if exists somewhere - https://phabricator.wikimedia.org/T171429#3464937 (10jcrespo) Maybe it was ongoing at the time and not failed. [11:12:31] 10DBA, 10Wikidata: Migrate wb_terms to using prefixed entity IDs instead of numeric IDs - https://phabricator.wikimedia.org/T114903#3465472 (10daniel) [11:15:26] 10DBA, 10Wikidata: Migrate wb_terms to using prefixed entity IDs instead of numeric IDs - https://phabricator.wikimedia.org/T114903#3465484 (10daniel) [11:15:30] 10Blocked-on-schema-change, 10DBA, 10Wikidata, 10Patch-For-Review, 10Wikidata-Sprint: Deploy schema change for adding term_full_entity_id column to wb_terms table - https://phabricator.wikimedia.org/T162539#3465482 (10daniel) [11:15:36] 10DBA, 10Wikidata: Migrate wb_terms to using prefixed entity IDs instead of numeric IDs - https://phabricator.wikimedia.org/T114903#1709272 (10daniel) [11:15:54] 10DBA, 10Wikidata: Migrate wb_terms to using prefixed entity IDs instead of numeric IDs - https://phabricator.wikimedia.org/T114903#1709272 (10daniel) [11:16:08] 10DBA, 10Wikidata: Migrate wb_terms to using prefixed entity IDs instead of numeric IDs - https://phabricator.wikimedia.org/T114903#1709272 (10daniel) [11:16:24] 10DBA, 10Wikidata: Migrate wb_terms to using prefixed entity IDs instead of numeric IDs - https://phabricator.wikimedia.org/T114903#1709272 (10daniel) [11:18:35] 10DBA, 10Wikidata: Migrate wb_terms to using prefixed entity IDs instead of numeric IDs - https://phabricator.wikimedia.org/T114903#3465506 (10daniel) [11:21:33] 10DBA, 10Wikidata: Populate term_full_entity_id on www.wikidata.org - https://phabricator.wikimedia.org/T171460#3465512 (10daniel) [11:23:13] 10DBA, 10Wikidata: Populate term_full_entity_id on test.wikidata.org - https://phabricator.wikimedia.org/T171461#3465526 (10daniel) [11:23:59] 10DBA, 10Wikidata: Populate term_full_entity_id on test.wikidata.org - https://phabricator.wikimedia.org/T171461#3465526 (10daniel) [11:24:15] 10DBA, 10Wikidata: Populate term_full_entity_id on www.wikidata.org - https://phabricator.wikimedia.org/T171460#3465512 (10daniel) [11:24:34] 10DBA, 10Wikidata: Migrate wb_terms to using prefixed entity IDs instead of numeric IDs - https://phabricator.wikimedia.org/T114903#3465544 (10daniel) [12:20:53] 10DBA, 10Wikidata: Populate term_full_entity_id on www.wikidata.org - https://phabricator.wikimedia.org/T171460#3465634 (10Marostegui) Hi, Thanks for the heads up. I assume this script has the proper throttling measures: ie, wait for replication? I would suggest you agreed on a running window for your scrip... [12:21:00] 10DBA, 10Wikidata: Populate term_full_entity_id on www.wikidata.org - https://phabricator.wikimedia.org/T171460#3465636 (10Marostegui) p:05Triage>03Normal [12:22:34] 10DBA, 10Wikidata: Populate term_full_entity_id on test.wikidata.org - https://phabricator.wikimedia.org/T171461#3465640 (10Marostegui) p:05Triage>03Normal Hi, My answer here is pretty much the one I gave at: T171460#3465634 [13:07:40] 10DBA, 10Wikidata: Populate term_full_entity_id on www.wikidata.org - https://phabricator.wikimedia.org/T171460#3465806 (10daniel) >>! In T171460#3465634, @Marostegui wrote: > I assume this script has the proper throttling measures: ie, wait for replication? It does - if anything, it waits for replication mor... [13:08:37] 10DBA, 10Wikidata: Populate term_full_entity_id on test.wikidata.org - https://phabricator.wikimedia.org/T171461#3465813 (10daniel) >>! In T171461#3465640, @Marostegui wrote: > Hi, > > My answer here is pretty much the one I gave at: T171460#3465634 The idea was to run this on the test site soon, and get so... [13:08:40] 10DBA, 10Wikidata: Populate term_full_entity_id on www.wikidata.org - https://phabricator.wikimedia.org/T171460#3465814 (10Marostegui) Awesome!! :-) Thank you [13:09:54] 10DBA, 10Wikidata: Populate term_full_entity_id on test.wikidata.org - https://phabricator.wikimedia.org/T171461#3465817 (10aude) i can maybe do this later today / this evening (US time) [13:11:19] 10DBA, 10Wikidata: Populate term_full_entity_id on test.wikidata.org - https://phabricator.wikimedia.org/T171461#3465827 (10Marostegui) >>! In T171461#3465813, @daniel wrote: >>>! In T171461#3465640, @Marostegui wrote: >> Hi, >> >> My answer here is pretty much the one I gave at: T171460#3465634 > > The idea... [13:15:14] 10DBA, 10Wikidata: Populate term_full_entity_id on test.wikidata.org - https://phabricator.wikimedia.org/T171461#3465846 (10Ladsgroup) Per RelEng policy, maintenance scripts that take more than one hour to finish must be reserved via a deployment window beforehand. My suggestion is to start running it and if i... [13:23:37] 10DBA, 10Wikidata: Populate term_full_entity_id on test.wikidata.org - https://phabricator.wikimedia.org/T171461#3465868 (10daniel) If it takes an hour for test.wikidata.org, I guess we know that we have to improve the script before we can run it on the live site. [13:52:52] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of Carrotkit → 胡蘿蔔: supervision needed - https://phabricator.wikimedia.org/T171474#3466021 (10Wong128hk) [13:53:42] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of Carrotkit → 胡蘿蔔: supervision needed - https://phabricator.wikimedia.org/T171474#3466035 (10Wong128hk) [13:55:39] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of Carrotkit → 胡蘿蔔: supervision needed - https://phabricator.wikimedia.org/T171474#3466021 (10Marostegui) When do you want to attempt to rename this user? I am happy to monitor the DBs for you while this happens. Sometimes we have to ease some dur... [13:58:27] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of Carrotkit → 胡蘿蔔: supervision needed - https://phabricator.wikimedia.org/T171474#3466047 (10Marostegui) p:05Triage>03Normal [13:59:50] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of Carrotkit → 胡蘿蔔: supervision needed - https://phabricator.wikimedia.org/T171474#3466050 (10Wong128hk) It is ready for now. Thx. [14:06:04] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of Carrotkit → 胡蘿蔔: supervision needed - https://phabricator.wikimedia.org/T171474#3466106 (10Marostegui) >>! In T171474#3466050, @Wong128hk wrote: > It is ready for now. Thx. There is a deployment going on right now - so let's wait a few minutes... [14:12:09] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of Carrotkit → 胡蘿蔔: supervision needed - https://phabricator.wikimedia.org/T171474#3466116 (10Marostegui) You can now go ahead. Once you start the process, please paste here the meta url so I can follow the progress per wiki Thanks [14:14:27] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of Carrotkit → 胡蘿蔔: supervision needed - https://phabricator.wikimedia.org/T171474#3466146 (10Wong128hk) Thx a lot. The rename is started. https://meta.wikimedia.org/wiki/Special:GlobalRenameProgress/%E8%83%A1%E8%98%BF%E8%94%94 [14:15:28] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of Carrotkit → 胡蘿蔔: supervision needed - https://phabricator.wikimedia.org/T171474#3466156 (10Marostegui) >>! In T171474#3466146, @Wong128hk wrote: > Thx a lot. The rename is started. > > https://meta.wikimedia.org/wiki/Special:GlobalRenameProgre... [14:45:25] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of Carrotkit → 胡蘿蔔: supervision needed - https://phabricator.wikimedia.org/T171474#3466244 (10Marostegui) This is now done, not big delays happened on the slowest slaves (10-20 seconds tops). Feel free to close this task if you think it is done fr... [14:46:50] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of Carrotkit → 胡蘿蔔: supervision needed - https://phabricator.wikimedia.org/T171474#3466247 (10Wong128hk) 05Open>03Resolved a:03Wong128hk Rename was completed. Thank you for supervision. [14:57:40] 10DBA, 10Cloud-Services, 10Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3466268 (10Marostegui) s2 has been imported on labsdb1009, labsdb1010 and labsdb1011. Views have been created so these wikis are now fully availa... [14:57:48] 10DBA, 10Cloud-Services, 10Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3466269 (10Marostegui) [14:58:39] 10DBA, 10Cloud-Services, 10Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3325353 (10Marostegui) a:03Marostegui I will start with s7, the last pending shard, on Monday as I am off from Tuesday. [15:11:25] 10DBA, 103d, 10Multimedia, 10MW-1.30-release-notes (WMF-deploy-2017-06-27_(1.30.0-wmf.7)), 10Patch-For-Review: Have search recognise STL files as a new kind of media file ('type:3d' or whatever) - https://phabricator.wikimedia.org/T157348#3466322 (10dr0ptp4kt) 05Open>03Resolved a:03dr0ptp4kt This h... [15:32:15] 10DBA, 10Commons, 10MediaWiki-File-management, 10Multimedia, 10PostgreSQL: Exception: Wikimedia\Rdbms\DBUnexpectedError when attempting to move within File: namespace - https://phabricator.wikimedia.org/T171390#3466422 (10dr0ptp4kt) p:05Triage>03Normal Updating tags, setting to Normal. At first glanc... [16:04:39] 10DBA, 10Analytics-EventLogging, 10Analytics-Kanban, 10Contributors-Analysis, and 2 others: Add index to mediawiki_page_create_1 table - https://phabricator.wikimedia.org/T170990#3466593 (10fdans) [16:05:11] 10DBA, 10Analytics-EventLogging, 10Analytics-Kanban, 10Contributors-Analysis, and 2 others: Add index to mediawiki_page_create_1 table - https://phabricator.wikimedia.org/T170990#3450014 (10fdans) a:03Ottomata [19:28:29] 10DBA, 10Mail, 10Operations, 10Patch-For-Review: Setup database for dmarc service - https://phabricator.wikimedia.org/T170158#3467519 (10herron) The mysql client IP addresses are: diadem 208.80.153.17 dysprosium 208.80.154.24 [23:39:51] 10DBA, 103d, 10Multimedia, 10MW-1.30-release-notes (WMF-deploy-2017-06-27_(1.30.0-wmf.7)), 10Patch-For-Review: Have search recognise STL files as a new kind of media file ('type:3d' or whatever) - https://phabricator.wikimedia.org/T157348#3468502 (10Krinkle) 05Resolved>03Open Re-opening until "the bu... [23:40:28] 10DBA, 103d, 10Multimedia, 10MW-1.30-release, and 3 others: Have search recognise STL files as a new kind of media file ('type:3d' or whatever) - https://phabricator.wikimedia.org/T157348#3468505 (10Krinkle)