[00:22:59] 10DBA: Okay to use `user_properties` table for high frequency/volume options? - https://phabricator.wikimedia.org/T231044 (10nray) @Marostegui Thank you for your response! I should add that for this particular ticket, we are using `localStorage` instead of the `user_properties` table to mark when the user has se... [05:03:12] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1063 - https://phabricator.wikimedia.org/T231199 (10Marostegui) Thanks! ` root@db1063:~# megacli -LDPDInfo -aAll Adapter #0 Number of Virtual Disks: 1 Virtual Drive: 0 (Target Id: 0) Name : RAID Level : Primary-1, Secondary-0, RAI... [05:03:20] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1063 - https://phabricator.wikimedia.org/T231199 (10Marostegui) 05Open→03Resolved [05:04:16] 10DBA: db1115 (tendril) paged twice in 24h due to OOM - https://phabricator.wikimedia.org/T231165 (10Marostegui) 05Open→03Resolved a:03Marostegui I am going to close this, as there is not much to do on this ticket itself, but it should be followed up on {T231185} [05:04:18] 10DBA, 10Operations: Disable/remove unused features on Tendril - https://phabricator.wikimedia.org/T231185 (10Marostegui) [05:15:13] 10DBA, 10OTRS, 10Operations: Switchover m1 primary master: db1063 to db1135 - https://phabricator.wikimedia.org/T231403 (10Marostegui) [05:15:30] 10DBA, 10OTRS, 10Operations: Switchover m1 primary master: db1063 to db1135 - https://phabricator.wikimedia.org/T231403 (10Marostegui) p:05Triage→03Normal [05:15:59] 10DBA, 10OTRS, 10Operations: Switchover m1 primary master: db1063 to db1135 - https://phabricator.wikimedia.org/T231403 (10Marostegui) @akosiaris @Dzahn @jcrespo @ayounsi let me know if that proposed day and time would work for you. Thanks! [05:34:13] 10DBA, 10OTRS, 10Operations: Switchover m1 primary master: db1063 to db1135 - https://phabricator.wikimedia.org/T231403 (10ayounsi) 👍 [05:49:40] 10DBA, 10MediaWiki-Cache, 10Performance-Team (Radar), 10User-Marostegui: Replace parsercache keys to something more meaningful on db-XXXX.php - https://phabricator.wikimedia.org/T210725 (10Marostegui) pc1009 and pc2009 have now been optimized. As all the active parsercache have been cleaned up and optimize... [06:18:49] 10DBA, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata.org, 10Story: [Story] Monitor size of some Wikidata database tables - https://phabricator.wikimedia.org/T68025 (10ArielGlenn) When I look at that image it looks pretty empty, am I missing something? [06:21:31] 10DBA, 10Operations: Remove sarin and neodymium GRANTs from all the databases - https://phabricator.wikimedia.org/T229796 (10Marostegui) [06:21:53] 10DBA, 10Operations: Remove sarin and neodymium GRANTs from all the databases - https://phabricator.wikimedia.org/T229796 (10Marostegui) 05Open→03Resolved `neodymium` grants have been removed everywhere [06:25:05] 10DBA, 10Operations: Decommission db2053.codfw.wmnet - https://phabricator.wikimedia.org/T231407 (10Marostegui) [06:25:17] 10DBA, 10Operations: Decommission db2053.codfw.wmnet - https://phabricator.wikimedia.org/T231407 (10Marostegui) p:05Triage→03Normal [06:28:40] 10DBA, 10Operations, 10Patch-For-Review: Decommission db2053.codfw.wmnet - https://phabricator.wikimedia.org/T231407 (10Marostegui) [06:28:59] 10DBA, 10Operations: Predictive failures on disk S.M.A.R.T. status - https://phabricator.wikimedia.org/T208323 (10Marostegui) [06:32:35] 10DBA, 10Operations, 10Patch-For-Review: Decommission db2053.codfw.wmnet - https://phabricator.wikimedia.org/T231407 (10Marostegui) [07:54:32] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` dbproxy1018.eqiad.wmnet ` The log can be found in `/var/log/wmf-au... [08:11:58] 10DBA: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbproxy1018.eqiad.wmnet'] ` and were **ALL** successful. [08:16:46] 10DBA: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [08:22:08] 10DBA: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) dbproxy1018 is now monitoring the hosts: ` root@dbproxy1018:~# echo "show stat" | socat /run/haproxy/haproxy.sock stdio # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp... [08:40:01] why are all the backup sources showing that the last tendril contact was 2h ago? [08:41:17] I know why [08:41:26] The user specified as a definer ('root'@'10.64.32.20') does not exist [08:41:36] That's because I removed neodymium grants [08:41:40] Interesting... [08:41:46] I will get that fixed [08:44:06] db1095:3312 and 3313 fixed [08:46:26] all the other hosts also fixed [08:46:57] * volans curious on what has to do neodymium with backups [08:47:22] it is not neodymium, it is the tendril events [08:48:07] neodymium was used at the time, and hence it has the definer root@neodymium's ip [08:48:19] And I removed 'root'@'neodymium_ip' [08:48:23] (the user) [08:48:35] so the event wasn't being executed as the user didn't exist anymore [08:52:49] ahhhh got it [10:37:20] 10DBA, 10OTRS, 10Operations: Switchover m1 primary master: db1063 to db1135 - https://phabricator.wikimedia.org/T231403 (10Dzahn) Fine with me, speaking for RT and racktables. Note that RT is separate from OTRS which is more critical. [10:38:31] 10DBA, 10OTRS, 10Operations: Switchover m1 primary master: db1063 to db1135 - https://phabricator.wikimedia.org/T231403 (10Marostegui) >>! In T231403#5445157, @Dzahn wrote: > Fine with me, speaking for RT and racktables. Note that RT is separate from OTRS which is more critical. Correct! My bad, sorry! Amen... [10:38:42] 10DBA, 10Operations: Switchover m1 primary master: db1063 to db1135 - https://phabricator.wikimedia.org/T231403 (10Marostegui) [11:11:00] 10DBA: Okay to use `user_properties` table for high frequency/volume options? - https://phabricator.wikimedia.org/T231044 (10Marostegui) >>! In T231044#5444281, @nray wrote: > @Marostegui Thank you for your response! I should add that for this particular ticket, we are using `localStorage` instead of the `user_p... [11:40:33] 10DBA, 10Operations, 10wikitech.wikimedia.org, 10Patch-For-Review, 10cloud-services-team (Kanban): Switchover m5 primary master: db1073 to db1133: Tuesday 3rd Sept at 13:00 UTC - https://phabricator.wikimedia.org/T229657 (10Marostegui) [12:13:06] 10DBA: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [12:16:10] 10DBA, 10Operations, 10serviceops, 10Goal: Strengthen backup infrastructure and support - https://phabricator.wikimedia.org/T229209 (10Jclark-ctr) [13:19:02] 10DBA, 10conftool: set min_replicas on database sections in dbctl - https://phabricator.wikimedia.org/T231019 (10Marostegui) s6 done - set it to 3: ` root@cumin1001:~# dbctl -s eqiad section s6 get { "s6": { "master": "db1061", "min_replicas": 3, "readonly": false, "ro_reas... [13:19:30] 10DBA, 10conftool: set min_replicas on database sections in dbctl - https://phabricator.wikimedia.org/T231019 (10Marostegui) [13:21:26] 10DBA, 10conftool: set min_replicas on database sections in dbctl - https://phabricator.wikimedia.org/T231019 (10Marostegui) s8 done - set it to 4: ` root@cumin1001:~# dbctl -s codfw section s8 get { "s8": { "master": "db2079", "min_replicas": 4, "readonly": false, "ro_reaso... [13:21:52] 10DBA, 10conftool: set min_replicas on database sections in dbctl - https://phabricator.wikimedia.org/T231019 (10Marostegui) [14:02:11] 10DBA, 10Operations, 10wikitech.wikimedia.org, 10Patch-For-Review, 10cloud-services-team (Kanban): Switchover m5 primary master: db1073 to db1133: Tuesday 3rd Sept at 13:00 UTC - https://phabricator.wikimedia.org/T229657 (10Marostegui) [14:02:54] 10DBA, 10Operations, 10wikitech.wikimedia.org, 10Patch-For-Review, 10cloud-services-team (Kanban): Switchover m5 primary master: db1073 to db1133: Tuesday 3rd Sept at 13:00 UTC - https://phabricator.wikimedia.org/T229657 (10Marostegui) [14:03:24] 10DBA, 10Operations, 10wikitech.wikimedia.org, 10Patch-For-Review, 10cloud-services-team (Kanban): Switchover m5 primary master: db1073 to db1133: Tuesday 3rd Sept at 13:00 UTC - https://phabricator.wikimedia.org/T229657 (10Marostegui) [14:53:05] marostegui: i would like to make a backup (snapshot) of the database 'racktables' on m1-master.eqiad out of caution before i upgrade the app and server it runs on, what is the best way, do i run mysqldump myself or better ask DBA [14:53:37] mutante: we do have backups, but if you want a fresh one, I can do that for you [14:53:55] mutante: As a general rule, always use a replica, not a master :) [14:54:46] marostegui: ok! thanks, so that sounds like the backups are only a few days old? [14:54:59] in this case we should not be editing it anymore afaict [14:56:12] mutante: the last logical backup from m1 is from 27th, however, given that rt is super small, I have taken a backup only of the rt database just now [14:56:39] marostegui: rt and racktables are different ones [14:56:45] mutante: sorry [14:56:46] racktables [14:56:54] marostegui: cool, thank you! :) [14:57:14] mutante: for now it is at db1117:/srv/tmp.m1, where do you want it to go? [14:57:31] i am moving it off of jessie then and it's the test to move a misc backend to https [14:58:33] mutante: sure, up to you, I will leave the backup there for now, you can move it elsewhere if you want [14:58:47] mutante: Please let me know what you do with it later, or delete it if you move it somewhere else [14:59:41] marostegui: hmm.. sometimes i miss a "work host" for this kind of thing. i will move it to the misc host itself [14:59:57] mutante: maybe cumin1001? [15:00:01] the backup is just 3GB [15:00:08] sorry, 21M [15:00:08] XDDD [15:00:12] So maybe you can move it there [15:00:13] haha, miscweb1001.eqiad.wmnet:/home/dzahn/ [15:00:16] ok [15:00:35] sure, place it wherever you need :) [15:00:50] I am going offline [15:00:51] Bye! [15:00:57] thanks, bye bye [15:05:41] deleted from /srv, moved to my home on the same server [15:17:02] ok.the new software version wants to update the DB and create a TRIGGER and for that it wants SUPER and the app user does not have that as it shouldnt, will have to continue that tomorrow or later [15:22:46] 10DBA, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata.org, 10Story: [Story] Monitor size of some Wikidata database tables - https://phabricator.wikimedia.org/T68025 (10Addshore) There is only a single point right now (per metric) and it might have dropped off the time range by now! But it is there... [18:58:20] 10DBA: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10ayounsi) [19:28:21] 10DBA: Okay to use `user_properties` table for high frequency/volume options? - https://phabricator.wikimedia.org/T231044 (10nray) > I am sorry, but I don't really know what localStorage refer to, can you provide more details? [[ https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage | localStorag... [19:53:44] 10DBA, 10Performance-Team (Radar): Okay to use `user_properties` table for high frequency/volume options? - https://phabricator.wikimedia.org/T231044 (10Krinkle)