[05:22:17] 10DBA: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [05:38:44] 10DBA: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [05:55:08] 10DBA, 10Goal: Switchover codfw primary database masters to new hosts - https://phabricator.wikimedia.org/T230106 (10Marostegui) [08:09:55] 10DBA, 10Goal, 10Patch-For-Review: Switchover codfw primary database masters to new hosts - https://phabricator.wikimedia.org/T230106 (10Marostegui) [08:10:24] 10DBA, 10Goal, 10Patch-For-Review: Switchover codfw primary database masters to new hosts - https://phabricator.wikimedia.org/T230106 (10Marostegui) s6 codfw master swapped from db2046 to db2129 [09:20:02] 10DBA: Remove grants for the old dbproxy hosts from the misc databases - https://phabricator.wikimedia.org/T231280 (10Marostegui) [09:20:13] 10DBA: Remove grants for the old dbproxy hosts from the misc databases - https://phabricator.wikimedia.org/T231280 (10Marostegui) p:05Triage→03Normal [09:24:24] 10DBA: Remove grants for the old dbproxy hosts from the misc databases - https://phabricator.wikimedia.org/T231280 (10Marostegui) m4: Grants removed on db1107 and db1108 for dbproxy1004 and dbproxy1009 ` dbproxy1004.eqiad.wmnet has address 10.64.16.148 dbproxy1009.eqiad.wmnet has address 10.64.32.158 ` The use... [09:24:42] 10DBA: Remove grants for the old dbproxy hosts from the misc databases - https://phabricator.wikimedia.org/T231280 (10Marostegui) [10:02:01] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) dbproxy1016 is ready to replace dbproxy1003 on m3 ` # mysql --skip-ssl -hdbproxy1016 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB conne... [10:02:06] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [10:20:25] 10DBA, 10Operations: Remove sarin and neodymium GRANTs from all the databases - https://phabricator.wikimedia.org/T229796 (10Marostegui) a:03Marostegui I am going to start removing `sarin` grants first [10:44:51] 10DBA, 10Operations: Remove sarin and neodymium GRANTs from all the databases - https://phabricator.wikimedia.org/T229796 (10Marostegui) `sarin` grants have been removed everywhere. [10:45:17] 10DBA, 10Operations: Remove sarin and neodymium GRANTs from all the databases - https://phabricator.wikimedia.org/T229796 (10Marostegui) [11:20:22] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [11:24:51] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) dbproxy1019 is ready to replace dbproxy1011 ` root@dbproxy1019:~# echo "show stat" | socat /run/haproxy/haproxy.sock stdio # pxname,svname,qcur,qmax,scur,smax,sli... [12:30:42] 10DBA, 10MediaWiki-File-management, 10MW-1.34-notes (1.34.0-wmf.20; 2019-08-27), 10Performance-Team (Radar): Drop filejournal table from WMF - https://phabricator.wikimedia.org/T51195 (10Marostegui) Table renamed on db1089 which is a host with read traffic. Will leave it there for a couple of days whilst s... [12:30:57] 10DBA, 10MediaWiki-File-management, 10MW-1.34-notes (1.34.0-wmf.20; 2019-08-27), 10Performance-Team (Radar): Drop filejournal table from WMF - https://phabricator.wikimedia.org/T51195 (10Marostegui) [12:31:38] 10DBA, 10MediaWiki-File-management, 10MW-1.34-notes (1.34.0-wmf.20; 2019-08-27), 10Performance-Team (Radar): Drop filejournal table from WMF - https://phabricator.wikimedia.org/T51195 (10Marostegui) [18:43:22] 10DBA, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata.org, and 2 others: [Story] Monitor size of some Wikidata database tables - https://phabricator.wikimedia.org/T68025 (10Addshore) I just hit +2 on the patch. It runs weekly, which is "every sunday at 01 hours". So let's check back next week! [18:45:23] 10DBA, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata.org, and 2 others: [Story] Monitor size of some Wikidata database tables - https://phabricator.wikimedia.org/T68025 (10ArielGlenn) I think Reedy was away and didn't see my pings. Anyways, thanks for moving forward on this, and we'll see how it lo... [19:12:54] 10DBA, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata.org, 10Story: [Story] Monitor size of some Wikidata database tables - https://phabricator.wikimedia.org/T68025 (10Addshore) Looks like the data has landed. https://graphite.wikimedia.org/render/?width=586&height=308&target=daily.dbtables.*.slot... [19:50:30] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1063 - https://phabricator.wikimedia.org/T231199 (10Cmjohnson) @Marostegui Replaced the disk with one of the few remaining used spares. I did notice 2 more disks are starting to fail....you may want to speed up the decom process.