[01:25:06] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3505002 (10Bawolff) [03:15:40] 10DBA, 10Cloud-Services: Prepare and check storage layer for hi.wikiversity - https://phabricator.wikimedia.org/T171829#3505019 (10Jayprakash12345) @Marostegui Sir, How could be long this task. [04:48:15] 10DBA, 10Cloud-Services: Prepare and check storage layer for hi.wikiversity - https://phabricator.wikimedia.org/T171829#3505044 (10Marostegui) a:05Marostegui>03None Hi, Triggers are working fine and anonymizing the data correctly (I have checked my user-data after registering). So the DBA side is done he... [04:55:51] 10DBA, 10Cloud-Services: Prepare and check storage layer for wikimania2018wiki - https://phabricator.wikimedia.org/T155041#3505051 (10Marostegui) 05Open>03Resolved labsdb1011 is now ready too. [05:22:52] 10DBA, 10Data-Services: Prepare and check storage layer for hi.wikiversity - https://phabricator.wikimedia.org/T171829#3505063 (10bd808) [05:24:03] 10DBA, 10Data-Services, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3505067 (10bd808) [05:25:31] 10DBA, 10Data-Services: LabsDB infrastructure pending work - https://phabricator.wikimedia.org/T153058#3505073 (10bd808) [06:20:17] 10DBA, 10Operations, 10ops-eqiad: db1016 m1 master: Possibly faulty BBU - https://phabricator.wikimedia.org/T166344#3505078 (10Marostegui) The BBU is failing again, so we should try to give m1 master failover some priority amongst the other misc services. [06:23:20] 10DBA, 10cloud-services-team: Compress InnoDB on db1102 - https://phabricator.wikimedia.org/T172169#3505080 (10Marostegui) [06:23:38] 10DBA, 10cloud-services-team: Compress InnoDB on db1102 - https://phabricator.wikimedia.org/T172169#3488463 (10Marostegui) Three shards compressed ``` root@db1102:~# df -hT /srv/ Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/tank-data xfs 3.6T 1.6T 2.1T 44% /srv ``` [06:23:46] 10DBA, 10Data-Services: LabsDB infrastructure pending work - https://phabricator.wikimedia.org/T153058#3505083 (10Marostegui) [06:23:49] 10DBA, 10cloud-services-team: Compress InnoDB on db1102 - https://phabricator.wikimedia.org/T172169#3505082 (10Marostegui) 05Open>03Resolved [06:29:51] 10DBA, 10Operations, 10ops-eqiad: db1016 m1 master: Possibly faulty BBU - https://phabricator.wikimedia.org/T166344#3505090 (10Marostegui) After forcing the relearn, this recovered: ``` ˜/icinga-wm 8:29> RECOVERY - MegaRAID on db1016 is OK: OK: optimal, 1 logical, 2 physical, WriteBack policy ``` [07:13:45] 10DBA, 10Analytics, 10Contributors-Analysis, 10Chinese-Sites, 10Patch-For-Review: Data Lake edit data missing for many wikis - https://phabricator.wikimedia.org/T165233#3505101 (10Marostegui) Thanks for confirming @Milimetric - I have now fixed all the wikis listed at: T165233#3498662. Please try again a... [07:57:07] 10DBA, 10MediaWiki-Database, 10MediaWiki-Documentation, 10Documentation, and 3 others: Bump MediaWiki's minimum supported MySQL Version to 5.5.8 - https://phabricator.wikimedia.org/T161232#3505173 (10jcrespo) Note the "Supports all MySQL 5.1+ functionality", which probably means it misses prepared statemen... [08:11:54] 10DBA, 10Operations, 10ops-eqiad: db1016 m1 master: Possibly faulty BBU - https://phabricator.wikimedia.org/T166344#3505175 (10Marostegui) And again: `˜/icinga-wm 10:09> PROBLEM - MegaRAID on db1016 is CRITICAL: CRITICAL: 1 LD(s) must have write cache policy WriteBack, currently using: WriteThrough` [08:12:06] 10DBA, 10MediaWiki-Database, 10MediaWiki-Logging, 10Performance, 10Schema-change: Logging needs an index to optimize searching by log_title - https://phabricator.wikimedia.org/T68961#3505176 (10jcrespo) I don't think this is going to work log_namespace, log_title should probably be used instead, even if... [08:13:20] 10DBA, 10Operations, 10ops-eqiad: db1016 m1 master: Possibly faulty BBU - https://phabricator.wikimedia.org/T166344#3293244 (10jcrespo) Maybe we can setup m1 on db1069? [08:14:25] you mean re-using db1069 as m1 master ocne it is freed up? [08:16:49] yes [08:17:10] good idea, decent hardware, relatively new, under warranty.. [08:18:27] 10DBA, 10Operations, 10ops-eqiad: db1016 m1 master: Possibly faulty BBU - https://phabricator.wikimedia.org/T166344#3505184 (10Marostegui) >>! In T166344#3505178, @jcrespo wrote: > Maybe we can setup m1 on db1069? I like that idea, I'll try to work on: T166546 soon as I am about to finish with: T153743 [08:19:17] 10DBA, 10Data-Services: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127#3505187 (10jcrespo) [08:20:42] 10DBA, 10Data-Services: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127#731912 (10jcrespo) Nothing to do with MariaDB 10, I assume at some point was a blocker. This is probably a multi-month maintenance such as T132416. [09:05:58] I just now finished reading the close to hundred ticket comments I had pending [09:06:09] (over just a weekend) [09:07:24] happy monday! [09:36:33] 10DBA: Productionize 11 new eqiad database servers - https://phabricator.wikimedia.org/T172679#3505388 (10jcrespo) [09:36:41] 10DBA: Productionize 11 new eqiad database servers - https://phabricator.wikimedia.org/T172679#3505404 (10jcrespo) p:05Triage>03Normal [09:36:53] check^ [09:37:54] I've created it to track usage of servers, like T170662 [09:37:55] T170662: Productionize 22 new codfw database servers - https://phabricator.wikimedia.org/T170662 [09:38:28] also to pool a replacement on db1037 which may help the watchlist issue [09:39:02] db1037? [09:40:56] https://phabricator.wikimedia.org/T171027 [09:41:10] I bumped my internal priority for that [09:41:32] Ah - I had never seen that ticket before! [09:41:32] haha [09:41:48] I will copy data from db1050 and partition those tables [09:42:46] https://gerrit.wikimedia.org/r/370447 [09:43:09] great! [09:43:10] even if later we pool another server with that role, it will take less time on a new one [09:43:40] and another server to decomm, db1037 \o/ [09:44:25] If it is still slow, as I think it will, I can point at the new hardware [09:49:12] 10DBA, 10Patch-For-Review: Finish dbstore2002 migration to multi-instance - https://phabricator.wikimedia.org/T171321#3505430 (10Marostegui) [09:50:30] eh, -1 ? [09:50:55] +process_count => 5, ? [09:50:59] ah [09:51:01] yes [10:46:27] 10DBA, 10Data-Services, 10Security-Team, 10WMF-Legal, and 5 others: Make wbqc_constraints table available on Quarry et al. - https://phabricator.wikimedia.org/T170927#3505517 (10Marostegui) labsdb1011 has now puppet enabled so it can be treated normally for this task For the new labs infra be aware of: T17... [11:21:37] 10DBA, 10Operations, 10ops-eqiad: db1016 m1 master: Possibly faulty BBU - https://phabricator.wikimedia.org/T166344#3505584 (10Marostegui) ``` ˜/icinga-wm 12:19> RECOVERY - MegaRAID on db1016 is OK: OK: optimal, 1 logical, 2 physical, WriteBack policy ``` [11:22:44] 10DBA, 10Patch-For-Review: Finish dbstore2002 migration to multi-instance - https://phabricator.wikimedia.org/T171321#3505586 (10Marostegui) As we spoke, probably it is a good idea to leave dbstore2002 with the current 5 shards, so it has room to grow and we'd not have to revisit it in the next 6 months :-) ``... [11:32:58] 10DBA, 10Cloud-Services, 10Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3505589 (10Marostegui) [11:37:24] 10DBA, 10Data-Services: LabsDB infrastructure pending work - https://phabricator.wikimedia.org/T153058#3505595 (10Marostegui) [11:37:28] 10DBA, 10Cloud-Services, 10Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3505593 (10Marostegui) 05Open>03Resolved Hi, So the last shard pending to be imported (s7) is now on the new labs hosts, that means that th... [11:41:17] 10DBA, 10Analytics, 10Contributors-Analysis, 10Chinese-Sites, 10Patch-For-Review: Data Lake edit data missing for many wikis - https://phabricator.wikimedia.org/T165233#3505597 (10Marostegui) @Milicevic01 @JAllemandou @Nuria we have finished importing all the production shards into the new labs infra: T1... [11:44:13] 10DBA: Migrate dbstore2001 to multi instance - https://phabricator.wikimedia.org/T168409#3505613 (10Marostegui) Now that we are considering dbstore2002 done with its 5 shards (T171321) the idea would be: - Delete all the content from dbstore2001 - Reimage as strecth (it runs jessie) - Move its puppet role from... [11:49:41] 10DBA: Point labsdb1001 and labsdb1003 to db1095 and db1102 - https://phabricator.wikimedia.org/T166546#3505619 (10Marostegui) [11:50:51] 10DBA, 10Data-Services: LabsDB infrastructure pending work - https://phabricator.wikimedia.org/T153058#3505624 (10jcrespo) [11:50:53] 10DBA: Point labsdb1001 and labsdb1003 to db1095 and db1102 - https://phabricator.wikimedia.org/T166546#3505623 (10jcrespo) [11:51:24] 10DBA: Point labsdb1001 and labsdb1003 to db1095 and db1102 - https://phabricator.wikimedia.org/T166546#3505625 (10Marostegui) [11:51:31] 10DBA, 10Data-Services: LabsDB infrastructure pending work - https://phabricator.wikimedia.org/T153058#2868122 (10jcrespo) [12:16:24] 10DBA, 10Epic: Meta ticket: Migrate multi-source database hosts to multi-instance - https://phabricator.wikimedia.org/T159423#3505669 (10Marostegui) [12:16:26] 10DBA, 10Patch-For-Review: Finish dbstore2002 migration to multi-instance - https://phabricator.wikimedia.org/T171321#3505668 (10Marostegui) 05Open>03Resolved [13:13:14] I am not sure db1050 will boot back up [13:13:22] why? [13:13:30] it didn't the first time [13:13:42] ah, you rebooted it [13:13:45] I thought you meant mysql [13:13:51] yes, the server [13:13:57] it seems it finally did [13:14:27] Ah, I was going to suggest to ask chris to leave it totally disconnected so it gets rid of all the flea power and then try again [13:39:21] 10DBA, 10MediaWiki-Database, 10MediaWiki-Logging, 10Performance, 10Schema-change: Logging needs an index to optimize searching by log_title - https://phabricator.wikimedia.org/T68961#3505796 (10Huji) [13:40:20] 10DBA, 10MediaWiki-Database, 10MediaWiki-Logging, 10Performance, 10Schema-change: Logging needs an index to optimize searching by log_title - https://phabricator.wikimedia.org/T68961#723244 (10Huji) >>! In T68961#3505176, @jcrespo wrote: > ... log_namespace, log_title should probably be used instead, eve... [13:45:57] 10DBA, 10MediaWiki-Database, 10MediaWiki-Logging, 10Performance, 10Schema-change: Logging needs an index to optimize searching by log_title - https://phabricator.wikimedia.org/T68961#3505813 (10jcrespo) @Huji, but we already have that index :-) : https://phabricator.wikimedia.org/source/mediawiki/browse... [13:50:24] 10DBA, 10MediaWiki-Database, 10MediaWiki-Logging, 10Performance, 10Schema-change: Logging needs an index to optimize searching by log_title - https://phabricator.wikimedia.org/T68961#3505828 (10Huji) @Jcrespi: but that includes `log_timestamp` a field that in our current use cases (see related Tasks abov... [13:50:42] I am "Jcrespi" ! [13:50:58] Hello jcrespi [14:04:13] 10DBA, 10MediaWiki-Database, 10MediaWiki-Logging, 10Performance, 10Schema-change: Logging needs an index to optimize searching by log_title - https://phabricator.wikimedia.org/T68961#3505855 (10jcrespo) Left-most column index featur on MySQL/MariaDB allows to use an index on (A, B, C) for filters on {A},... [14:15:13] 10DBA, 10MediaWiki-Database, 10MediaWiki-Logging, 10Performance, 10Schema-change: Logging needs an index to optimize searching by log_title - https://phabricator.wikimedia.org/T68961#3505907 (10jcrespo) For example, we could search the last 100 logs for a page, and then search for a delete: ``` SELECT... [14:47:33] we forgot about es2013 [14:47:38] or at least I did [14:47:44] No, I didn't :) [14:47:56] was there followup from support? [14:48:17] I was planning to ask papaul in the meeting if he ever contacted support (not sure he did) [14:52:55] marostegui: I was trying to agre with you on https://phabricator.wikimedia.org/T172693 :D not sure how it seemed jsut fyi [14:53:14] haha I know! I was just confirming it doesn't :) [14:53:34] * marostegui needs to work out his English! [14:54:22] actually, that is indeed a translation issue [14:54:49] haha could be [14:54:55] what did you understand jynus? [14:55:01] we do not like the double negactive much, and unlike french we do not have a "si" equivalent [14:55:28] Going to modify the comment to make it clearer :) [14:55:28] oh, I understood you perfectly, but I understand also why a non-Spanish speaker would be confused [14:56:09] only triple negatives from here on out gents [14:56:13] So it is chasemp's problem!! [14:56:16] :-) [14:56:17] chasemp: it is our fault "spaniards" - but to deal with us, just don't start a question with a negative [14:56:31] unless you want the opposite answer :-) [14:56:54] this sounds like the beginnings of a nigerian prince scam that could work [14:57:13] You think reselling maintain-views.yaml could give us benefit? [14:57:54] I have 4000Billion records available to send it to you, for that you only need to create 4000 views [14:58:09] pls send script [14:58:11] I lol'd [14:58:15] xdddddddddddd [14:59:27] labsdbs are actually more like a pyramid scheme, we just involve more and more people maintaining them [14:59:54] except nobody on top gets rich [15:05:13] I'm waiting for my labsdb stock options to mature [15:05:40] then it's private island and mojitos time [15:53:52] 10DBA, 10Data-Services: Design a method for keeping user-created tables in sync across labsDBs - https://phabricator.wikimedia.org/T156869#3506376 (10bd808) [15:59:05] 10DBA, 10Cloud-Services, 10Cloud-VPS, 10Epic, 10Tracking: Labs databases rearchitecture (tracking) - https://phabricator.wikimedia.org/T140788#3506405 (10jcrespo) [15:59:58] 10DBA, 10Data-Services: LabsDB infrastructure pending work - https://phabricator.wikimedia.org/T153058#3506409 (10jcrespo) [16:48:27] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Operations, 10Performance-Team, and 5 others: Cache invalidations coming from the JobQueue are causing lag on several wikis - https://phabricator.wikimedia.org/T164173#3506604 (10mmodell) [16:48:37] jynus: Did you manage to dig up that list of errors relating to the two other unique -> PRIMARY index updates? [16:48:50] I proposed a dependant patch that fixes it all in MW as far as I can see [16:48:58] let me see [16:49:08] they will be gone on the logs [16:49:21] but maybe they were copied on the ticket [16:49:50] we did not an exahstive check- just left one server and saw if it gave at least 1 error [16:49:58] *didn't do [17:17:11] Reedy: this is the problem https://phabricator.wikimedia.org/T17441#3172790 [17:18:48] I would literally trust your patch is enough- reverting code is "easy" and the errors went unnoticed for weeks, so if there is some extra missing, it will be very low impact [17:19:47] the only thing I am not convinced is the workflow for non-wmf hosts [17:20:27] code and patch in this case is strongly dependent- while normally we can do alters in advance. [17:21:16] we have solved that on wmf by duplicating indexes temporarilly, but I wonder if we should send patches that, if alter is applied befor or after code deploy, things will break [17:21:57] maybe it is worth deploying things in 2 separate patches, even if it means running an alter twice, as we did exactly that? [17:22:51] as that is a mediawiki decision, I will leave that to you or other hackers (if I was a third party DBA, I would prefer the double-patch) [18:35:18] 10DBA, 10Analytics, 10Contributors-Analysis, 10Chinese-Sites, 10Patch-For-Review: Data Lake edit data missing for many wikis - https://phabricator.wikimedia.org/T165233#3507361 (10Neil_P._Quinn_WMF) >>! In T165233#3503803, @Milimetric wrote: > No rush but just a heads up to @Neil_P._Quinn_WMF, the 2017-0... [19:41:20] 10DBA, 10Analytics, 10Contributors-Analysis, 10Chinese-Sites, 10Patch-For-Review: Data Lake edit data missing for many wikis - https://phabricator.wikimedia.org/T165233#3507677 (10Milimetric) I think I worked out the bugs, should be ready soon unless something else goes wrong. [23:19:32] 10DBA, 10Patch-For-Review: Productionize 11 new eqiad database servers - https://phabricator.wikimedia.org/T172679#3505388 (10Luke081515) > Setup the eventual s8 for which wikis would that be, and is there a task for that / for the decision?