[08:33:55] 10DBA, 10Data-Services, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3736701 (10Marostegui) [08:33:57] 10DBA, 10Data-Services: Database replica drift on web and analytics clusters - https://phabricator.wikimedia.org/T179767#3736698 (10Marostegui) 05Open>03Resolved a:05jcrespo>03Marostegui I have fixed this, thanks again for reporting it. [08:35:19] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Consider dropping the "wb_items_per_site.wb_ips_site_page" index - https://phabricator.wikimedia.org/T179793#3736250 (10Marostegui) Makes sense to do it at the same time with T114904 as we might need to depool slaves for that task. [08:36:43] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Consider dropping the "wb_items_per_site.wb_ips_site_page" index - https://phabricator.wikimedia.org/T179793#3736708 (10Marostegui) Should be done together with T179106 as that task requires depooling servers most likely. [08:57:17] 10DBA: Database error on es.wikipedia.org in function "Revision::insertOn": "Duplicate entry for key 'PRIMARY'" - https://phabricator.wikimedia.org/T48047#3736725 (10Marostegui) 05Open>03Resolved Going to close this as resolved, if someone feels it should not be closed, feel free to reopen! Thanks! [09:10:36] 10DBA, 10Data-Services, 10Patch-For-Review: Enable innodb-large-prefix on tools.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T179614#3736731 (10Marostegui) 05Open>03Resolved a:03Marostegui I have enabled it on both hosts and merged the config change. Remember that it will only work with DY... [09:26:58] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3736765 (10Marostegui) [09:43:14] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3736778 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db1103.eqiad.wmnet'] ``` and were **ALL** successful. [09:43:52] ^ haha only took 3 days :-) [09:44:34] WUT? [09:44:47] nah, don't worry [09:45:07] it reimaged fine, but the first puppet run was broken because of: https://gerrit.wikimedia.org/r/#/c/388427/ which moritzm just solved [09:45:51] ah ok, got it [09:46:11] 09:43:08 | db1103.eqiad.wmnet | Still waiting for Puppet after 4330.0 minutes [09:46:14] 09:43:09 | db1103.eqiad.wmnet | Puppet run checked [09:46:16] 09:43:09 | db1103.eqiad.wmnet | Reimage completed [09:46:19] 09:43:09 | db1103.eqiad.wmnet | REIMAGE END | retcode=0 [09:46:21] haha :) [09:50:52] did you alter the timeout in the code? is much lower IIRC [09:51:40] nope, i didn't change anything, just added —no-downtime for that host [09:53:02] ahahahah [09:53:06] my fault [09:53:39] there is a continue before checking the timeout [09:53:44] I'll send a fix [09:55:31] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3736795 (10Marostegui) [10:03:11] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3736812 (10Marostegui) [10:03:40] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3714503 (10Marostegui) db1103 is now done, I would like to pool it in eqiad soon, so we can start seeing how they work with load. [10:20:32] 10DBA, 10Data-Services, 10Patch-For-Review: Enable innodb-large-prefix on tools.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T179614#3736836 (10Marostegui) 05Resolved>03Open I have disabled it because I am seeing some performance issues, and I want to see if they are related (they look relate... [10:52:20] 10DBA, 10Data-Services, 10Patch-For-Review: Enable innodb-large-prefix on tools.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T179614#3736904 (10Marostegui) It matches with a spike on INSERTs for another user's db, so I want to try to isolate both things. [11:16:13] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3736953 (10Marostegui) [11:25:39] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3736980 (10Marostegui) db2051, codfw s4 master is done: ``` root@neodymium:/home/marostegui/T1... [11:25:50] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3736982 (10Marostegui) [12:29:41] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3737110 (10Marostegui) [13:32:18] 10DBA, 10Data-Services, 10Patch-For-Review: Enable innodb-large-prefix on tools.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T179614#3737275 (10Marostegui) I have had it enabled for a couple of hours without any issues, but before merging I am going to give it a few more hours so we can make sure... [14:22:53] 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3737393 (10Marostegui) [16:11:58] 10DBA, 10Data-Services, 10Patch-For-Review: Enable innodb-large-prefix on tools.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T179614#3737597 (10Marostegui) 05Open>03Resolved Closing this as resolved as it has been enabled for hours without issues. [20:17:20] 10DBA, 10Operations, 10Support-and-Safety, 10Patch-For-Review, 10Wiki-Setup (Create): Create elections committee private wiki - https://phabricator.wikimedia.org/T174370#3738541 (10Reedy) 05Open>03Resolved Wiki is created, but have not created any accounts for anyone as requested. I had created one f... [20:40:03] 10DBA, 10Cloud-Services, 10User-Jayprakash12345: Prepare and check storage layer for hif.wiktionary - https://phabricator.wikimedia.org/T173647#3738629 (10Reedy) Wiki has been created [20:42:41] 10DBA, 10Operations, 10Support-and-Safety, 10Patch-For-Review, 10Wiki-Setup (Create): Create elections committee private wiki - https://phabricator.wikimedia.org/T174370#3738636 (10Marostegui) Just to confirm: the filtering works fine, the wiki isn't on labs. ``` mysql:root@localhost [(none)]> select @@h... [20:45:34] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3738666 (10Marostegui) s6 codfw is done: ``` root@neodymium:/home/marostegui/git/software/dbto... [20:45:40] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3738670 (10Marostegui) [20:56:26] 10DBA, 10Cloud-Services, 10User-Jayprakash12345: Prepare and check storage layer for hif.wiktionary - https://phabricator.wikimedia.org/T173647#3738732 (10Marostegui) I will sanitize this tomorrow and once done will hand it over #cloud-services-team [21:58:48] 10DBA, 10Cloud-Services, 10User-Jayprakash12345: Prepare and check storage layer for hif.wiktionary - https://phabricator.wikimedia.org/T173647#3738935 (10Reedy) [21:58:56] 10DBA, 10Cloud-Services, 10User-Jayprakash12345: Prepare and check storage layer for hif.wiktionary - https://phabricator.wikimedia.org/T173647#3535715 (10Reedy) [22:33:15] 10DBA, 10Operations, 10Support-and-Safety, 10Patch-For-Review, 10Wiki-Setup (Create): Create elections committee private wiki - https://phabricator.wikimedia.org/T174370#3739045 (10jrbs) >>! In T174370#3738541, @Reedy wrote: > Wiki is created, but have not created any accounts for anyone as requested. I... [22:36:02] 10DBA, 10Operations, 10Support-and-Safety, 10Patch-For-Review, 10Wiki-Setup (Create): Create elections committee private wiki - https://phabricator.wikimedia.org/T174370#3739050 (10KTC) Thanks a lot everyone that have helped get this sorted. [22:38:34] 10DBA, 10Analytics, 10Data-Services, 10Research, 10cloud-services-team (Kanban): Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3739078 (10bd808)