[09:13:47] tendril is back up [09:13:51] the data is on db2093 already [09:13:57] but replication is breaking there [09:14:12] maybe the data was corrupted during the transfer? :( [09:14:26] at least the tendril.processlist is the one failing so far [09:17:21] or maybe stuff was written to the db but not to the binlog? [09:18:35] I am seeing a DELETE on db1115 and that row doesn't exist on db2093 at all [09:29:24] given that it is not a super important table, I wonder if we should put an ignore on that one and maybe change the binlog to statement? or attempt the transfer again...:( [09:32:58] did you disable event on start? [09:33:24] no :( [09:33:34] I forgot about events [09:33:39] there you go [09:33:43] pffff [09:36:51] I will transfer it again tomorrow morning then and disable events on both hosts at start :( [10:17:38] it only requires it on the slave, although we should check if events disable the binary log [10:18:28] you mean if we disable events if it disables writing to the binlog? [10:22:25] I've found in the past events disabling the binary log before writing [10:22:46] Interesting... [10:22:49] we should grep sql_log_bin on tendril source code [10:23:06] but I think most of the problems come from having double-monitoring [10:23:15] do you have an example? [10:23:23] of what? [10:23:31] of replication error [10:23:51] I can check it myself [10:23:58] no, i have one [10:24:26] I have a delete on the binlog and that row doesn't exist on db2093 [10:24:28] give me a sec [10:24:53] yeah, I want to know table and id, "deleted by timestamp?" [10:25:06] select * from processlist where server_id=1232 and id=939631603 [10:25:07] so we can confirm they are the events and not something else [10:25:28] but what was the delete? [10:25:37] grabbing it, give me a sec :) [10:26:09] my thesis it is on a table being purged/dropping partitions [10:26:16] check our etherpad [10:26:18] there is another possiblity [10:26:19] for the DELETE [10:26:34] I guess that event removed the row upon start [10:26:41] and when the binlog arrived, it was already gone [10:26:43] which is tokudb + aria + read commited being incompatible with binary log, which would be less solvable [10:27:23] we need to know where that came from [10:27:52] I was checking some events [10:28:17] and the xxxx_activity event has a delete from processlist where serer_id = xxx [10:28:21] so might be that event [10:29:22] yep, I agree [10:30:02] I swear was going to ping you with a "remember to add event_scheduler=0" [10:30:23] but then I though "manuel is going to be mad because I am constantly pinging about things he knows" [10:30:47] and the only time it was needed, I don't do it :-D [10:36:45] hahahah [10:37:03] Always ping me! [10:37:04] Always! [10:37:07] better be safe [10:37:12] well, I know now [10:37:26] As we (at least me) are not used to have events, it is easy to forget :() [10:39:31] events are evil and shouldn't be used [10:39:43] +1305819389062462 [10:39:50] I never used them before [10:39:55] Yeah, me neither [10:40:25] did you have to discover them when you were reverse engineering tendril? :) [11:25:26] 10DBA, 10Operations, 10ops-eqiad: Disk #5 (count starts at #0) of db1111 has corrupted sectors - https://phabricator.wikimedia.org/T187526#4000783 (10MoritzMuehlenhoff) p:05Triage>03Normal [12:14:39] 10DBA, 10Patch-For-Review: Setup tendril database monitoring on 2 new hosts, one on eqiad and one on codfw - https://phabricator.wikimedia.org/T184704#4000953 (10Marostegui) [12:15:14] 10DBA, 10Patch-For-Review: Setup tendril database monitoring on 2 new hosts, one on eqiad and one on codfw - https://phabricator.wikimedia.org/T184704#3892870 (10Marostegui) db1011's content is now placed at `db1113:/srv/db1011_backup` [12:19:45] Happy Monday dear DBAs [12:21:12] I'm going to enable a feature on some s7 and s5 (dewiki) wikis that make wbc_entity_usage grow (for sure) but it won't explode as we made lots of sanity checks also it helps with job queue [12:21:22] is it okay [12:21:27] ok [12:21:38] ok thanks for the heads up [12:21:55] Thanks [12:25:58] 10DBA, 10Operations, 10ops-codfw: db2048: RAID with predictive failure - https://phabricator.wikimedia.org/T187983#4000988 (10jcrespo) 05Resolved>03Open This failed again, I guess because using a bad disk: Predictive Failure: 1I:1:1 [12:26:30] 10DBA, 10Operations, 10ops-codfw: db2048: RAID with predictive failure - https://phabricator.wikimedia.org/T187983#4000990 (10Marostegui) a:05Marostegui>03Papaul [12:27:27] 10DBA: Decommission db1011 - https://phabricator.wikimedia.org/T184703#4000991 (10Marostegui) db1011 was failed over to db1115 and moreover db1011's content has been copied over to `db1113:/srv/db1011_backup` I would still wait till the end of the week to start the decommissioning process of db1011 [13:41:01] dear DBAs, sorry to bother again, one of the things Wikidata team wants to work is to clean up wb_terms and normalize several columns there. This probably reduces average row size from 143 to around 100 which probably free up lots of storage for you but at the same time you need to do some schema changes on that table (the table is 1.4B rows now). Is it okay for you? [13:45:13] marostegui: jynus ^ [13:51:05] I was actually thinking the same last week [13:51:14] some types [13:51:17] could be enums [13:51:25] or foreign key references [13:51:32] and should not be strings [13:51:45] it is ok not to optimize when the table was small [13:52:06] but it is reaching a size in which we'll get a lot of space back [13:52:31] think 300GB - 50% reduction multiplied by 20 servers is a lot [13:52:46] plus less space, more things fit in memory and means more speed [13:55:46] Sure thing, in mediawiki, dev never go with enum because any simple change means schema change so small int that are saved somewhere like config would be fine I think or we go with the newly introduced names table: https://gerrit.wikimedia.org/r/#/c/404445/23/includes/Storage/NameTableStore.php,unified [14:01:51] those are inplementations details that are more or less to you- from the point of view of the db, we will just change strings into ids of some kind, which I am ok with it [14:02:47] technically, NameTableStore is just a normalization of strings without a referece table, but it is the same concept [14:04:30] I would like to apply normalization to more tables, but some were denormalized on purpouse as an optimization, rather than a lack of one [14:09:13] btw [14:09:21] i just pushed the eqiad DB refresh for approval [14:09:29] would 10G cards be helpful, e.g. during reimaging? [14:10:10] in general we do not buy those for dbs [14:10:37] if there is a policy for everything to be 10G, we will, but not helpful in general [14:10:52] it will be required for backups/automatic resting hosts [14:10:54] why would we have a policy for everyting 10G? [14:11:00] not nearly everything needs 10G :) [14:11:02] *if there was [14:11:06] and I know databases don't need it much [14:11:15] but maybe it's occasionally useful, that's why I'm asking :) [14:11:38] we can consider it in the future [14:11:44] not really at the moment [14:11:47] alright [14:12:39] it takes more time right now to compress and decompress data, etc. [14:13:41] but even if transfers were 10 times faster, we wouldn't transfer them so fast [14:14:13] first step is being able to serve multiple databases from the same host within 2 years :-) [14:14:28] as that would be the only moment in which provisioning is critical [14:14:34] total failure [14:15:03] the suggestion is fine, but not right now [14:16:08] https://grafana.wikimedia.org/dashboard/db/mysql-aggregated?panelId=2&fullscreen&orgId=1&from=now-7d&to=now [14:16:35] we do 150MB with all eqiad hosts aggregated, although I know you were suggesting for things like provisioning [14:17:24] definietly a good suggestion to reevaluate in the future [15:48:09] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1068 - https://phabricator.wikimedia.org/T188187#4001627 (10Marostegui) Thanks Chris: ``` root@db1068:~# megacli -PDRbld -ShowProg -PhysDrv [32:2] -aALL Rebuild Progress on Device at Enclosure 32, Slot 2 Completed 1% in 1 Minutes. ``` Once this is finish... [16:16:50] 10DBA, 10Operations, 10ops-eqiad: Disk #5 (count starts at #0) of db1111 has corrupted sectors - https://phabricator.wikimedia.org/T187526#4001826 (10Cmjohnson) The ssd was replaced, @marostegui please confirm and resolve after rebuild Return shipping informaitn USPS 9202 3946 5301 2438 0714 10 FEDEX 961191... [16:20:12] 10DBA, 10Operations, 10ops-eqiad: Disk #5 (count starts at #0) of db1111 has corrupted sectors - https://phabricator.wikimedia.org/T187526#4001835 (10Marostegui) @Cmjohnson looks like storage crashed and the FS became read-only. We are investigating why... [16:24:23] 10DBA, 10Operations, 10ops-eqiad: Disk #5 (count starts at #0) of db1111 has corrupted sectors - https://phabricator.wikimedia.org/T187526#4001842 (10Marostegui) This is all we have from the HW logs: ``` /admin1/system1/logs1/log1-> show record3 properties CreationTimestamp = 20180226161220.000000-360... [16:31:11] I am going to reboot db1111 [16:31:17] there is not much we can do it anyways now [16:31:22] The raid is offline [16:31:35] sure [16:31:35] Probably it is totally dead after reboot anyways [16:57:12] 10DBA, 10Data-Services, 10Toolforge, 10Tracking: Certain tools users create multiple long running queries that take all memory from labsdb hosts, slowing it down and potentially crashing (tracking) - https://phabricator.wikimedia.org/T119601#4002057 (10jcrespo) [16:57:37] 10DBA, 10Data-Services, 10Toolforge, 10Tracking: Certain tools users create multiple long running queries that take all memory from labsdb hosts, slowing it down and potentially crashing (tracking) - https://phabricator.wikimedia.org/T119601#1830725 (10jcrespo) [17:01:51] 10DBA, 10Operations, 10ops-eqiad: Disk #5 (count starts at #0) of db1111 has corrupted sectors - https://phabricator.wikimedia.org/T187526#4002101 (10Marostegui) Looks like the new disk has not been added automatically to the RAID. I have been digging around the PERC menu, but it is terribly slow from here,... [17:21:29] Amir1: you around? [17:22:15] if he doesn't answer, I say we revert [17:22:27] even if it is not that, it is bad enough [17:22:33] to try to revert [17:22:35] Yeah, agreed [17:23:01] and it is not only that host, all replicas have that increase [17:23:11] yeah, I was checking db1104 too [17:23:22] do you have a ticket number? [17:23:33] https://phabricator.wikimedia.org/T184812 [17:40:34] 10DBA, 10Operations, 10ops-codfw: db2048: RAID with predictive failure - https://phabricator.wikimedia.org/T187983#4002381 (10Papaul) a:05Papaul>03Marostegui Disk placement complete. [17:41:17] 10Blocked-on-schema-change, 10Reading List Service, 10Reading-Infrastructure-Team-Backlog (Kanban): Deploy ReadingLists schema change for efficient count(*) handling - https://phabricator.wikimedia.org/T188048#4002385 (10Tgr) @jcrespo do you have an estimate for when this might happen? [17:58:04] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1068 - https://phabricator.wikimedia.org/T188187#4002471 (10Marostegui) The rebuilt failed for this disk, I guess this disk was not in a good state: ``` PD: 0 Information Enclosure Device ID: 32 Slot Number: 2 Drive's position: DiskGroup: 0, Span: 1, Arm:... [18:19:35] marostegui: I'm around now, was afk for food [18:20:15] how can I help, I saw the UBN task just got back to the office, Lucas told me about it. Is there anything else making problems? [19:18:24] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team: Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4002933 (10Anomie) [19:18:45] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018): Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4002933 (10Anomie) [19:19:42] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018): Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4002933 (10Anomie) [22:32:24] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018): Schema change for refactored actor storage - https://phabricator.wikimedia.org/T188299#4002933 (10Anomie)