[01:54:42] 10DBA, 10MediaWiki-API: prop=revisions with rvdir=newer gives internal_api_error_DBQueryError for page with many version - https://phabricator.wikimedia.org/T108968#2999970 (10Anomie) [07:03:13] 10DBA, 13Patch-For-Review: Remove partitions from metawiki.pagelinks in s7 - https://phabricator.wikimedia.org/T153300#3000426 (10Marostegui) db1039: ``` root@PRODUCTION s7[metawiki]> show create table pagelinks\G *************************** 1. row *************************** Table: pagelinks Create Ta... [07:46:12] 10DBA, 10MediaWiki-API: prop=revisions with rvdir=newer gives internal_api_error_DBQueryError for page with many version - https://phabricator.wikimedia.org/T108968#1536152 (10Marostegui) Since we have altered revision table on dewiki (T148967#2853843 and T148967#2868434) the query plan has changed a bit: ```... [07:47:37] 10DBA, 10Analytics: Json_extract available on analytics-store.eqiad.wmnet - https://phabricator.wikimedia.org/T156681#3000516 (10Marostegui) 05Open>03Invalid I will close this for now as we cannot support this on our current MariaDB version. [09:36:08] 10DBA, 06Labs, 10Tool-Labs: labsdb1001 and labsdb1003 short on available space - https://phabricator.wikimedia.org/T132431#3000896 (10marcmiquel) [09:36:11] 10DBA, 06Labs, 10Tool-Labs: u3532__ (=marcmiquel) table using 64G on labsdb1001 - https://phabricator.wikimedia.org/T133322#3000893 (10marcmiquel) 05Open>03Resolved a:03marcmiquel Ok! Thanks. Marc [12:59:26] nice presentation on MyRocks at FOSDEM [13:01:39] it is certainly something to keep in mind for our external storage [13:02:05] it was very inmature last time I checked, only recently was included in percona/mariadb [13:02:15] but very promising [13:07:18] 10DBA, 10Phabricator, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#3001526 (10mmodell) [13:21:44] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#3001567 (10Aklapper) @Paladox: Please do not add random projects to tasks. [14:22:52] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: db2060 not accessible - https://phabricator.wikimedia.org/T156161#3001821 (10Marostegui) @Papaul db2060 is now off so you can proceed whenever you want. Thanks! [14:25:28] 10DBA, 06Labs, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3001845 (10Marostegui) I have hit this: T151607#2826415 when importing the tables on labsdb1011. I am mysqldumping these three tables (which are not reall... [14:37:20] 10DBA, 06Labs, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#2889412 (10jcrespo) __wmf_checksums can just be deleted, specially on labs. [16:25:33] 10DBA, 13Patch-For-Review: Remove partitions from metawiki.pagelinks in s7 - https://phabricator.wikimedia.org/T153300#3002323 (10Marostegui) db1028: ``` root@neodymium:/home/marostegui/git# mysql --skip-ssl -hdb1028.eqiad.wmnet metawiki -e "show create table pagelinks\G" *************************** 1. row ***... [16:31:25] 10DBA, 10MediaWiki-API: prop=revisions with rvdir=newer gives internal_api_error_DBQueryError for page with many version - https://phabricator.wikimedia.org/T108968#3002341 (10Anomie) While it seems to be fixed for `rev_page = '7165976'`, db1070 and db1071 (the current replicas for the 'api' group on s5) still... [17:05:24] 10DBA, 10Analytics, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3002434 (10Ottomata) Ok, talked with Analytics folks about this. These boxes host lots more than just EventLogging data, and we won't be able to ween people off of MySQL for... [17:27:25] 10DBA, 10Gerrit, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#3002514 (10demon) [17:29:14] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: labsdb1005 (mysql) maintenance for reimage - https://phabricator.wikimedia.org/T157358#3002516 (10jcrespo) [17:29:53] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3002535 (10jcrespo) [17:31:34] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: labsdb1005 (mysql) maintenance for reimage - https://phabricator.wikimedia.org/T157358#3002565 (10jcrespo) [17:32:02] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: labsdb1005 (mysql) maintenance for reimage - https://phabricator.wikimedia.org/T157358#3002516 (10jcrespo) [17:32:19] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#3002567 (10jcrespo) 05stalled>03Open [17:37:47] 10DBA, 10Gerrit, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#3002584 (10Paladox) @Aklapper gerrit is maintained by releng. Which the task i added is not random. [17:58:08] 10DBA, 10MediaWiki-Change-tagging, 06Operations: db1072 change_tag schema and dataset is not consistent - https://phabricator.wikimedia.org/T156166#3002669 (10Marostegui) [17:58:11] 10DBA, 06Operations, 13Patch-For-Review: Reimage and clone db1072 - https://phabricator.wikimedia.org/T156226#3002667 (10Marostegui) 05Open>03Resolved The BBU looks good now and has been charging and now fully charged: ``` root@db1072:~# megacli -AdpBbuCmd -GetBbuStatus -a0 | grep -e '^isSOHGood' -e '^C... [18:01:57] 10DBA, 10Gerrit, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#3002686 (10demon) >>! In T145885#3001567, @Aklapper wrote: > @Paladox: Please do not add random projects to tasks... [19:56:51] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: db2060 not accessible - https://phabricator.wikimedia.org/T156161#3003162 (10jcrespo) I have restarted MySQL on db2060, as this is probably done now. Waiting for repl. to catch up, and for Manuel to resolve if he thinks it is ok. [21:31:48] 10DBA, 06Operations, 13Patch-For-Review: Restart pending mysql hosts with old TLS cert - https://phabricator.wikimedia.org/T152188#2841012 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by jynus on neodymium.eqiad.wmnet for hosts: ``` ['db2050.codfw.wmnet', 'db2045.codfw.wmnet'] ``` The log can... [22:02:19] 10DBA, 06Operations, 13Patch-For-Review: Restart pending mysql hosts with old TLS cert - https://phabricator.wikimedia.org/T152188#3003497 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db2050.codfw.wmnet', 'db2045.codfw.wmnet'] ``` and were **ALL** successful. [22:40:18] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3003555 (10Dzahn)