[05:38:38] 10Blocked-on-schema-change, 10DBA: Extend echo_unread_wikis.euw_wiki - https://phabricator.wikimedia.org/T255174 (10Marostegui) [05:40:09] 10Blocked-on-schema-change, 10DBA: Extend echo_unread_wikis.euw_wiki - https://phabricator.wikimedia.org/T255174 (10Marostegui) As we use RBR on x1, this needs to be performed directly on the master - it cannot be done slave by slave first as otherwise the slaves breaks replication when an insert arrives: `... [05:43:19] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Configure WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Marostegui) +1 to exclude bots logins data from this. @Huji I guess there'll be new read queries for this feat... [06:06:39] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['dbproxy1020.eqiad.wmnet'] ` The log can be found in `/var/log/wm... [06:26:17] 10DBA: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbproxy1020.eqiad.wmnet'] ` and were **ALL** successful. [06:28:54] 10DBA: Make checksum parallel to the data transfer in transferpy package - https://phabricator.wikimedia.org/T254979 (10Privacybatm) I have incorporated the parallel md5sum in the code. But not working as expected! test1: I get the correct checksum when I run send and receive commands in the terminal, but runni... [07:35:23] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['dbproxy1016.eqiad.wmnet'] ` The log can be found in `/var/log/wm... [07:56:20] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbproxy1016.eqiad.wmnet'] ` and were **ALL** successful. [08:03:24] I am going to switchover the active m3 (phabricator) dbproxy https://gerrit.wikimedia.org/r/605531 [08:03:29] This should be transparent [08:03:48] dbproxy1016 runs Buster, so it will be the first dbproxy with acting as primary proxy [08:17:04] marostegui, jynus: I'd start with the cumin2001 reimage in a bit, any current blockers wrt databases/backups? [08:17:24] moritzm: not from my side [08:19:32] there is 2 backups running, but we can kill them [08:22:42] jynus: depends on the ETA to complete, I can also wait a bit for them to complete? [08:23:15] 1 is about to finish in 1 minute [08:23:21] we can wait for that [08:23:36] let's kill the other, I want to see if they complete or not with cumin dead [08:23:54] not sure if cumin will kill the job or it will continu successfully [08:24:23] ack, sounds good [08:25:09] but you can get ready to start the reimage [08:26:46] ack, I'm completing something else, will start in ~ 5 mins [08:26:58] if that job is done by then :-) [08:27:25] it finished already [08:27:50] end_date: 2020-06-15 08:26:22 [08:28:53] ack, thanks [08:32:42] jynus: I'm starting now [08:36:50] backup is still running, so no issue [08:40:51] ack, great [08:42:23] BTW, backups were supposed to run during night to prevent interactions, but they have started to take longer and longer [08:42:48] we'll buy more resources for them next fiscal [08:43:35] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [08:43:54] also you can move cumin2001 to staged if you want to prevent: https://netbox.wikimedia.org/extras/reports/puppetdb.PhysicalHosts/ [08:43:56] jynus: isn't easier to buy more night time? :-P [08:44:46] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) 05Open→03Resolved All hosts have been productionized. The hosts that are still pending to be decommissioned (dbproxy1003, dbproxy1008) will be handled in sepa... [08:44:48] 10DBA, 10Operations: rack/setup/install dbproxy101[2-7].eqiad.wmnet - https://phabricator.wikimedia.org/T196690 (10Marostegui) [08:44:51] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install dbproxy200[1-4] - https://phabricator.wikimedia.org/T223492 (10Marostegui) [08:49:56] 10DBA, 10Core Platform Team: text table still has old_* fields and indexes on some hosts - https://phabricator.wikimedia.org/T250066 (10Marostegui) [08:50:31] 10DBA, 10Core Platform Team: text table still has old_* fields and indexes on some hosts - https://phabricator.wikimedia.org/T250066 (10Marostegui) [08:51:06] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) 05Open→03Stalled Stalling u... [09:18:07] 10DBA, 10Core Platform Team: text table still has old_* fields and indexes on some hosts - https://phabricator.wikimedia.org/T250066 (10Marostegui) [09:29:46] 10DBA, 10Operations, 10decommission-hardware, 10ops-eqiad: decommission dbproxy1008.eqiad.wmnet - https://phabricator.wikimedia.org/T255406 (10Marostegui) [09:30:21] 10DBA, 10Operations, 10decommission-hardware, 10ops-eqiad: decommission dbproxy1008.eqiad.wmnet - https://phabricator.wikimedia.org/T255406 (10Marostegui) dbproxy1008 is no longer the active m3-master per https://gerrit.wikimedia.org/r/#/c/operations/dns/+/605531/ Let's give it a few days before starting i... [09:31:00] 10DBA, 10Operations, 10decommission-hardware, 10ops-eqiad: decommission dbproxy1008.eqiad.wmnet - https://phabricator.wikimedia.org/T255406 (10Marostegui) [09:31:02] 10DBA: Remove grants for the old dbproxy hosts from the misc databases - https://phabricator.wikimedia.org/T231280 (10Marostegui) [09:31:23] 10DBA: Remove grants for the old dbproxy hosts from the misc databases - https://phabricator.wikimedia.org/T231280 (10Marostegui) [09:31:52] 10DBA, 10Operations, 10decommission-hardware, 10ops-eqiad: decommission dbproxy1008.eqiad.wmnet - https://phabricator.wikimedia.org/T255406 (10Marostegui) [09:32:52] 10DBA, 10Core Platform Team: text table still has old_* fields and indexes on some hosts - https://phabricator.wikimedia.org/T250066 (10Marostegui) s5 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1003 [] db1130 [] db1124 [] db1113 [] db1110 [] db1102 [] db1100 [] db1096 []... [09:33:35] 10DBA, 10Core Platform Team: text table still has old_* fields and indexes on some hosts - https://phabricator.wikimedia.org/T250066 (10Marostegui) [09:42:53] 10DBA: Upgrade dbproxyXXXX to Buster - https://phabricator.wikimedia.org/T255408 (10Marostegui) [09:44:20] 10DBA: Upgrade dbproxyXXXX to Buster - https://phabricator.wikimedia.org/T255408 (10Marostegui) p:05Triage→03Medium [09:47:10] 10DBA, 10Operations, 10SRE-tools: Add native mysql module to spicerack - https://phabricator.wikimedia.org/T255409 (10Kormat) [09:49:45] 10DBA, 10Patch-For-Review: Upgrade dbproxyXXXX to Buster - https://phabricator.wikimedia.org/T255408 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['dbproxy2004.codfw.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202006150949_maro... [09:50:09] jynus, volans: https://phabricator.wikimedia.org/T255409 [09:50:21] thx! [10:10:33] 10DBA, 10Operations, 10SRE-tools, 10Patch-For-Review: Add native mysql module to spicerack - https://phabricator.wikimedia.org/T255409 (10jbond) p:05Triage→03Medium [10:12:59] 10DBA: Upgrade dbproxyXXXX to Buster - https://phabricator.wikimedia.org/T255408 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbproxy2004.codfw.wmnet'] ` and were **ALL** successful. [11:31:27] marostegui, jynus: cumin2001 is back up and all basic cumin/spicerack things are working by now, if you want to test anything DB or backup related on the new buster setup, it should be good to test now [11:32:12] I will test it now [11:34:28] I have to remember to save my bash_history for cumin1001, because I barely can work without ctrl-r [11:35:47] jynus: good idea XD [11:36:10] moritzm: thanks you, backup is running, will take a few hours to complete, but running fine so far [11:36:25] I expect no issues given only cumin calls and cron is done there [11:36:28] same here, I'm repeatedly surprised of the things this jmm did in the past :-) [11:36:33] ack, sounds good! [11:38:00] I will also test the new mariadb library, which should now be compatible with mysql 8 [11:39:01] mysql.py --version ~>/usr/local/sbin/mysql.py Ver 15.1 Distrib 10.4.13-MariaDB, for Linux (x86_64) using readline 5.2 [11:41:25] marostegui: you sould also test dbctl there once [11:41:40] will do [11:43:04] 10DBA: Upgrade dbproxyXXXX to Buster - https://phabricator.wikimedia.org/T255408 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['dbproxy2003.codfw.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202006151142_marostegui_218282.log`. [11:46:37] 10Blocked-on-schema-change, 10DBA, 10MachineVision, 10SDC-Statements (Machine-vision-depicts), and 2 others: Review & apply schema changes for T250748 - https://phabricator.wikimedia.org/T255003 (10matthiasmullie) Yeah the group by & the mvl_review = 0 clause were both significant issues for this join. He... [11:47:43] 10Blocked-on-schema-change, 10DBA, 10MachineVision, 10SDC-Statements (Machine-vision-depicts), and 2 others: Review & apply schema changes for T250748 - https://phabricator.wikimedia.org/T255003 (10matthiasmullie) If above query is fine, the schema migration can continue & I'll go update the other patch to... [11:54:44] 10Blocked-on-schema-change, 10DBA, 10MachineVision, 10SDC-Statements (Machine-vision-depicts), and 2 others: Review & apply schema changes for T250748 - https://phabricator.wikimedia.org/T255003 (10Marostegui) This query looks much better. 0.05 and it no longer scans the whole table [11:55:17] 10Blocked-on-schema-change, 10DBA, 10MachineVision, 10SDC-Statements (Machine-vision-depicts), and 2 others: Review & apply schema changes for T250748 - https://phabricator.wikimedia.org/T255003 (10Marostegui) a:03Marostegui [12:02:36] 10DBA: Upgrade dbproxyXXXX to Buster - https://phabricator.wikimedia.org/T255408 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbproxy2003.codfw.wmnet'] ` Of which those **FAILED**: ` ['dbproxy2003.codfw.wmnet'] ` [12:15:35] 10DBA: Upgrade dbproxyXXXX to Buster - https://phabricator.wikimedia.org/T255408 (10Marostegui) [12:57:22] 10DBA: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10Marostegui) [13:25:54] 10DBA: Upgrade dbproxyXXXX to Buster - https://phabricator.wikimedia.org/T255408 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['dbproxy2002.codfw.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202006151325_marostegui_32418.log`. [13:49:11] 10DBA: Upgrade dbproxyXXXX to Buster - https://phabricator.wikimedia.org/T255408 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbproxy2002.codfw.wmnet'] ` and were **ALL** successful. [13:53:48] 10DBA: Upgrade dbproxyXXXX to Buster - https://phabricator.wikimedia.org/T255408 (10Marostegui) [13:55:43] 10DBA: Compress enwiki InnoDB tables - https://phabricator.wikimedia.org/T254462 (10Marostegui) [14:01:24] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Configure WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji) >>! In T253802#6222616, @Marostegui wrote: > +1 to exclude bots logins data from this. > @Huji I guess t... [14:08:40] moritzm: are you planning to migrate cumin1001 anytime soon? [14:10:23] tentatively next week Monday [14:10:30] ah ok [14:10:45] some bits are still tested/fixed [14:11:00] Sure, that tentative date works to organize the long running tasks [14:11:02] thank you [14:11:22] I'm also planning to add a MOTD banner to cumin1001 to point people to use cumin2001 for now [14:11:45] sounds good [14:14:16] cumin2001: [14:11:59]: INFO - Backup finished correctly [14:15:24] jynus: ack, thx [14:15:48] I am guessing new things could come up, but it would be non-blockers/lesser things [14:19:52] 10DBA, 10Core Platform Team: text table still has old_* fields and indexes on some hosts - https://phabricator.wikimedia.org/T250066 (10Marostegui) [14:30:52] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Configure WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Reedy) Yeah, queries will stay the same. It will just be against a larger dataset due to the increased number... [14:32:07] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Configure WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Marostegui) Excellent - thanks for the clarification. Let's not add bot logins and let's enable this slowly to... [14:32:57] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Configure WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji) [14:35:02] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Configure WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji) [14:35:17] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Configure WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji) [14:40:13] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Configure WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji) [14:47:41] 10Blocked-on-schema-change, 10DBA, 10MachineVision, 10SDC-Statements (Machine-vision-depicts), and 2 others: Review & apply schema changes for T250748 - https://phabricator.wikimedia.org/T255003 (10Marostegui) s4 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [] db1... [14:54:21] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Configure WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji) >>! In T253802#6224206, @Marostegui wrote: > Excellent - thanks for the clarification. > Let's not add b... [15:04:25] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Configure WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Marostegui) [15:46:49] 10DBA, 10Core Platform Team: text table still has old_* fields and indexes on some hosts - https://phabricator.wikimedia.org/T250066 (10Marostegui) [16:26:49] 10DBA: Setup a global admin account that can only read/have limited privileges to databases for safer debugging - https://phabricator.wikimedia.org/T254756 (10jcrespo) >>! In T254756#6202249, @Volans wrote: >>>! In T254756#6202036, @jcrespo wrote: >> * Disadvantages > > ** At least for the Spicerack integration... [16:32:15] 10DBA, 10Core Platform Team: text table still has old_* fields and indexes on some hosts - https://phabricator.wikimedia.org/T250066 (10Marostegui) s2 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [] db1146 [] db1129 [] db1125 [] db1122 [] db1105 [] db1095 [] db1090 []... [16:49:02] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Configure WMF wikis to log login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji) [17:10:24] 10DBA, 10Operations, 10SRE-tools, 10Patch-For-Review: Add native mysql module to spicerack - https://phabricator.wikimedia.org/T255409 (10jcrespo) a:03Kormat [20:38:07] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10daniel) >>! In T238966#6205906, @Marostegui... [20:40:48] 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10daniel) >>! In T238966#6192600, @Marostegui...