[03:11:12] 10DBA, 06Labs, 10Tool-Labs: Credentials in replica.my.cnf gives Access Denied - https://phabricator.wikimedia.org/T154933#2932929 (10scfc) [03:13:34] 10DBA, 06Labs, 10Tool-Labs: Credentials in replica.my.cnf gives Access Denied - https://phabricator.wikimedia.org/T154933#2928837 (10scfc) (`user` was `('s51488',)` in `/data/project/danmicholobot/replica.my.cnf`; I fixed that to `s51488`, but it does not change the access problem.) [03:14:49] 10DBA, 06Labs, 10Tool-Labs: Credentials in replica.my.cnf gives Access Denied - https://phabricator.wikimedia.org/T154933#2932936 (10scfc) [03:52:11] 10DBA, 06Labs, 10Tool-Labs: Credentials in replica.my.cnf gives Access Denied - https://phabricator.wikimedia.org/T154933#2928837 (10Marostegui) The grants are there... ``` [root@labsdb1001 03:48 /root] # pt-show-grants | grep s51488 -- Grants for 's51488'@'%' GRANT SELECT, SHOW VIEW ON `%\_p`.* TO... [13:59:59] 10DBA, 06Labs, 10Tool-Labs: Spatial database for tool-labs - https://phabricator.wikimedia.org/T154497#2933264 (10akosiaris) @Tobias1984. Sorry I completely forgot about this. I 've created your account. Contact me privately on IRC (akosiaris) or via email so I can provide the credentials to you. [14:25:03] 10DBA, 10Wikimedia-Site-requests: special:Categories fatals - 1.29.0-wmf.7 - https://phabricator.wikimedia.org/T155091#2933306 (10TTO) ``` MariaDB [igwiki_p]> select cat_title,cat_pages from category where cat_title like 'Article_nke_ntakiri'; +---------------------+-----------+ | cat_title | cat_pag... [14:28:48] 10DBA, 10Wikimedia-Site-requests, 07Wikimedia-log-errors: special:Categories fatals - 1.29.0-wmf.7 - https://phabricator.wikimedia.org/T155091#2933310 (10TTO) [16:24:38] 10DBA, 10Wikimedia-Site-requests, 07Wikimedia-log-errors: special:pages fatals - 1.29.0-wmf.7 - https://phabricator.wikimedia.org/T155091#2933416 (10Matanya) [16:24:56] 10DBA, 10Wikimedia-Site-requests, 07Wikimedia-log-errors: special pages fatals - 1.29.0-wmf.7 - https://phabricator.wikimedia.org/T155091#2933244 (10Matanya) [17:19:56] 10DBA: Meta DBA ticket for the DC switchover - https://phabricator.wikimedia.org/T155099#2933574 (10Marostegui) [17:26:35] 10DBA: Meta DBA ticket for the DC switchover - https://phabricator.wikimedia.org/T155099#2933590 (10Marostegui) [17:27:17] 10DBA, 07Epic: Meta DBA ticket for the DC switchover - https://phabricator.wikimedia.org/T155099#2933574 (10Marostegui) p:05Triage>03Normal [17:36:01] 10DBA, 06Labs, 10Tool-Labs: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2933601 (10scfc) [17:36:47] 10DBA, 06Labs, 10Tool-Labs: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2928837 (10scfc) [17:42:14] 10DBA: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#2933609 (10Marostegui) [17:42:16] 10DBA, 07Epic: Meta DBA ticket for the DC switchover - https://phabricator.wikimedia.org/T155099#2933606 (10Marostegui) [17:45:33] 10DBA, 07Epic: Meta DBA ticket for the DC switchover - https://phabricator.wikimedia.org/T155099#2933612 (10Marostegui) [17:48:26] 10DBA, 06Labs, 10Tool-Labs: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2933613 (10scfc) Some more and command to list them: ``` scfc@tools-bastion-03:~$ sudo find -H /home /data/project -mindepth 2 -maxdepth 2 -type f -name replica.my.cnf -... [17:50:21] 10DBA, 07Epic: Meta DBA ticket for the DC switchover - https://phabricator.wikimedia.org/T155099#2933626 (10Marostegui) [17:50:23] 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#2933627 (10Marostegui) [17:54:14] 10DBA, 07Epic: Meta DBA ticket for the DC switchover - https://phabricator.wikimedia.org/T155099#2933629 (10Marostegui) [17:54:16] 07Blocked-on-schema-change, 10DBA: *_minor_mime are varbinary(32) on WMF sites, out of sync with varbinary(100) in MW core - https://phabricator.wikimedia.org/T73563#2933630 (10Marostegui) [18:02:32] 10DBA, 07Epic: Review capacity on codfw - https://phabricator.wikimedia.org/T155102#2933658 (10Marostegui) [18:03:07] 10DBA: Review capacity on codfw - https://phabricator.wikimedia.org/T155102#2933658 (10Marostegui) [18:12:51] 10DBA, 06Labs, 10Tool-Labs: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2933700 (10Marostegui) Do you happen to know to which of the labsdb servers are you connecting to? [18:26:49] 10DBA, 06Labs, 10Tool-Labs: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2933752 (10scfc) That's probably `10.64.4.11`: ``` scfc@tools-bastion-03:~$ host enwiki.labsdb enwiki.labsdb has address 10.64.4.11 scfc@tools-bastion-03:~$ ``` [18:29:44] 10DBA, 06Labs, 10Tool-Labs: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2928837 (10chasemp) Known broken AFAIK: tools.jackbot tools.danmicholobot tools.hasteurbot tools.reportsbot tools.enbbsb tools.botomatik tools.hgztools tools.xxn-tools t... [18:29:59] marostegui: https://phabricator.wikimedia.org/T154933#2933765 [18:30:10] s52526 and s51984 [18:30:38] 10DBA, 06Labs, 10Tool-Labs: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2933779 (10Marostegui) >>! In T154933#2933752, @scfc wrote: > That's probably `10.64.4.11`: > > ``` > scfc@tools-bastion-03:~$ host enwiki.labsdb > enwiki.labsdb has add... [18:34:29] 10DBA, 06Labs, 10Tool-Labs: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2933781 (10chasemp) ```root@labstore1004:~# nano /tmp/bad-tool-labsdb root@labstore1004:~# for tool in `cat /tmp/bad-tool-labsdb`; do /usr/local/sbin/maintain-dbusers del... [18:36:27] 10DBA, 06Labs, 10Tool-Labs: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2933783 (10Marostegui) @scfc can you try those ones that Chase recreated and see if they work? [19:25:15] 10DBA, 06Labs, 10Tool-Labs: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2933950 (10scfc) The credentials for the 13 tools now work: ``` scfc@tools-bastion-03:~$ for tool in alex botomatik boty danmicholobot enbbsb hasteurbot hgztools jackbot... [19:44:26] 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#2933975 (10Addshore) We ( @jcrespo @Bmueller @WMDE-leszek @Addshore ) had a quick discussion about this at #wikide... [21:09:00] 10DBA, 06Labs, 10Tool-Labs: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2934226 (10Marostegui) The user `danmichaelo` isn't in the list that @chasemp iterated over the first time to test if that would fix the issue. Looks like recreating that... [22:49:03] 10DBA, 07Epic: Meta DBA ticket for the DC switchover - https://phabricator.wikimedia.org/T155099#2934494 (10Marostegui) [23:38:52] 10DBA, 10MediaWiki-Special-pages, 10Wikimedia-Site-requests, 07Wikimedia-log-errors: special pages fatals - 1.29.0-wmf.7 - https://phabricator.wikimedia.org/T155091#2934956 (10Paladox)