[01:14:58] 10DBA, 10MediaWiki-Database, 10MediaWiki-Logging, 10Performance, 10Schema-change: Logging needs an index to optimize searching by log_title - https://phabricator.wikimedia.org/T68961#3511030 (10Huji) I actually like that idea. I have seen that code elsewhere (have to think to remember where), whereby a q... [05:11:08] 10DBA, 10Patch-For-Review: Point labsdb1001 and labsdb1003 to db1095 and db1102 - https://phabricator.wikimedia.org/T166546#3511169 (10jcrespo) [05:53:06] 10DBA, 10Patch-For-Review: Point labsdb1001 and labsdb1003 to db1095 and db1102 - https://phabricator.wikimedia.org/T166546#3511223 (10jcrespo) [06:20:44] 10DBA, 10Patch-For-Review: Point labsdb1001 and labsdb1003 to db1095 and db1102 - https://phabricator.wikimedia.org/T166546#3511247 (10jcrespo) [07:38:07] 10DBA, 10Patch-For-Review: Point labsdb1001 and labsdb1003 to db1095 and db1102 - https://phabricator.wikimedia.org/T166546#3511290 (10jcrespo) [07:39:58] 10DBA, 10Data-Services: LabsDB infrastructure pending work - https://phabricator.wikimedia.org/T153058#3511295 (10jcrespo) [07:40:00] 10DBA: Point labsdb1001 and labsdb1003 to db1095 and db1102 - https://phabricator.wikimedia.org/T166546#3299593 (10jcrespo) 05Open>03Resolved Replication no longer going through db1069- keeping it alive and replicating for a while to detect problems and in case a revert is needed. [08:10:28] 10DBA, 10Patch-For-Review: Migrate dbstore2001 to multi instance - https://phabricator.wikimedia.org/T168409#3511330 (10jcrespo) a:03jcrespo [08:47:46] 10DBA, 10Patch-For-Review: Migrate dbstore2001 to multi instance - https://phabricator.wikimedia.org/T168409#3363748 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by jynus on neodymium.eqiad.wmnet for hosts: ``` ['dbstore2001.codfw.wmnet'] ``` The log can be found in `/var/log/wmf-auto-reimage/20... [09:10:40] 10DBA, 10Patch-For-Review: Migrate dbstore2001 to multi instance - https://phabricator.wikimedia.org/T168409#3511435 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['dbstore2001.codfw.wmnet'] ``` and were **ALL** successful. [09:12:00] 10DBA, 10Patch-For-Review: Point labsdb1001 and labsdb1003 to db1095 and db1102 - https://phabricator.wikimedia.org/T166546#3511437 (10Marostegui) Nice!!! Finally db1069 is going to be unused!!! [09:49:03] 10DBA, 10Patch-For-Review: Migrate dbstore2001 to multi instance - https://phabricator.wikimedia.org/T168409#3511504 (10Marostegui) I guess for: https://github.com/wikimedia/puppet/blob/ae13ed05960e223316c1ca169f39422b13db5624/modules/role/manifests/mariadb/dbstore_multiinstance.pp#L51 and all the stuff below... [10:05:56] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3511528 (10jcrespo) 10.0.32 is out- I will create a package soon. @MarkTraceur as you can see, this finally needs s4 read-only time, is that something you ca... [10:06:59] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3511529 (10Marostegui) >>! In T168661#3511528, @jcrespo wrote: > 10.0.32 is out- I will create a package soon. @MarkTraceur as you can see, this finally need... [10:41:10] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: es2013 faulty BBU - https://phabricator.wikimedia.org/T172265#3511599 (10Marostegui) @Papaul did you get in contact with Dell's support? Thanks! [13:44:49] 10DBA, 10Patch-For-Review: Productionize 22 new codfw database servers - https://phabricator.wikimedia.org/T170662#3512123 (10Marostegui) [13:46:13] 10DBA, 10Patch-For-Review: Productionize 22 new codfw database servers - https://phabricator.wikimedia.org/T170662#3438592 (10Marostegui) [13:52:28] 10DBA, 10Patch-For-Review: Migrate dbstore2001 to multi instance - https://phabricator.wikimedia.org/T168409#3512149 (10jcrespo) I am finishing adding s1 and s2 today, will continue adding the other 3 tomorrow. [14:11:07] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3512218 (10MarkTraceur) @jcrespo I can help justify it, sure! We're releasing the #3d extension soon on test wikis, then pushing to Commons shortly thereaft... [15:03:00] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: es2013 faulty BBU - https://phabricator.wikimedia.org/T172265#3512537 (10Papaul) @Marostegui wrote "For the record, after manually forcing the re-learn we got it back to healthy - let's see how long until it fails again:" this was your last comment so... [15:04:30] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: es2013 faulty BBU - https://phabricator.wikimedia.org/T172265#3512541 (10Marostegui) >>! In T172265#3512537, @Papaul wrote: > @Marostegui wrote "For the record, after manually forcing the re-learn we got it back to healthy - let's see how long until it... [15:05:09] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: es2013 faulty BBU - https://phabricator.wikimedia.org/T172265#3512542 (10Papaul) p:05Triage>03Normal [17:32:31] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: es2013 faulty BBU - https://phabricator.wikimedia.org/T172265#3513349 (10Papaul) @Marostegui since th BBU is on the raid controller, we will have to replace the whole controller . I will receive the new controller tomorrow. Please see below for case n... [17:36:23] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: es2013 faulty BBU - https://phabricator.wikimedia.org/T172265#3513360 (10Marostegui) @Papaul Thanks a lot! That was fast! :-) If you receive it tomorrow we can try to replace it next week if you like (as with Wikimania going on, my schedule is a bit err... [17:44:50] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: es2013 faulty BBU - https://phabricator.wikimedia.org/T172265#3492537 (10RobH) Please note that the old controllers configuration needs to be exported and imported into the new controller, or the system will need to be reimaged. [17:48:48] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: es2013 faulty BBU - https://phabricator.wikimedia.org/T172265#3513429 (10Marostegui) Thanks for pointing that out @RobH! @Papaul do you need us to shutdown the host for you to export the config? As a side note, given that codfw is passive, if data is lo... [18:00:23] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: es2013 faulty BBU - https://phabricator.wikimedia.org/T172265#3513472 (10RobH) So, I'm working off recollection here, and @papaul will have to confirm if it is how he would do this. With the hardware raid controller, it stores the config on the disks.... [18:07:16] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: es2013 faulty BBU - https://phabricator.wikimedia.org/T172265#3513483 (10Marostegui) Thanks Rob! Let me know if I can help in anyway (the host is depooled by the way). So we can shut it down when needed, we just need to stop MySQL first. [18:27:40] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: es2013 faulty BBU - https://phabricator.wikimedia.org/T172265#3513527 (10Papaul) Once in the RAID controller BIOS we should have an option to import foreign config under controller 0 F2/ foreign config/import @Marostegui next week works for me. [18:31:58] 10DBA: duplicate key problems - https://phabricator.wikimedia.org/T151029#3513553 (10Marostegui) I have fixed duplicate entries on s4 on the following hosts by importing+exporting the tables, after that compression went thru without any issues: db2073 db2065 For the tables: ``` linter page watchlist ``` [18:34:44] 10DBA: duplicate key problems - https://phabricator.wikimedia.org/T151029#3513572 (10Marostegui) I have fixed duplicate entries on db2075 for: dewiki.watchlist wikidatawiki.wb_items_per_site [19:32:57] 10DBA, 10Patch-For-Review: Productionize 11 new eqiad database servers - https://phabricator.wikimedia.org/T172679#3513858 (10Luke081515) @jcrespo Thank you very much for the detailed answer :) [19:44:41] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: es2013 faulty BBU - https://phabricator.wikimedia.org/T172265#3513915 (10RobH) >>! In T172265#3513483, @Marostegui wrote: > Thanks Rob! Let me know if I can help in anyway (the host is depooled by the way). So we can shut it down when needed, we just ne... [19:49:00] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, and 2 others: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3513930 (10hoo) The functionality needed for this has been merged into master now. Once it's deployed (which should ha... [19:49:54] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: es2013 faulty BBU - https://phabricator.wikimedia.org/T172265#3513934 (10Marostegui) >>! In T172265#3513915, @RobH wrote: >>>! In T172265#3513483, @Marostegui wrote: >> Thanks Rob! Let me know if I can help in anyway (the host is depooled by the way). S... [20:55:31] 10DBA: Drop m3 from dbstore servers - https://phabricator.wikimedia.org/T156758#3514245 (10Marostegui) I have removed replication from 'm3' thread on dbstore1002 and these are the values: ``` root@DBSTORE[(none)]> select @@hostname; +-------------+ | @@hostname | +-------------+ | dbstore1002 | +-------------+... [20:55:49] 10DBA: Drop m3 from dbstore servers - https://phabricator.wikimedia.org/T156758#3514246 (10Marostegui) a:03Marostegui [23:49:41] 10DBA, 10Patch-For-Review: duplicate key problems - https://phabricator.wikimedia.org/T151029#3514732 (10Marostegui) Fixed duplicate entries on db2045 for: ``` dewiki.watchlist wikidatawiki.wb_items_per_site ```