[00:14:41] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2974787 (10Paladox) @Marostegui and @jcrespo and @demon and @dzahn i found a new lib we can use it's https://mariadb.com/kb/en/ma... [00:15:17] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2974788 (10Paladox) Oh, wait, mysql connector supports this too. [00:27:16] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2974849 (10Paladox) Nope doesn't work on mysql's version of the connector. Works on mariadb's one though. [07:16:42] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2975376 (10Marostegui) Wow, so awesome to wake up and see so much progress has been done on this ti... [07:23:57] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2975381 (10Marostegui) >>! In T145885#2974787, @Paladox wrote: > @Marostegui and @jcrespo and @demon and @dzahn i found a new lib... [08:11:00] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2975422 (10Paladox) >>! In T145885#2975381, @Marostegui wrote: >>>! In T145885#2974787, @Paladox wrote: >> @Marostegui and @jcres... [08:14:55] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2975428 (10Paladox) >>! In T156373#2975376, @Marostegui wrote: > Wow, so awesome to wake up and see... [08:23:09] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2975448 (10Marostegui) MariaDB reopened the bug to fix it in 10.0 \o/ https://jira.mariadb.org/bro... [08:24:55] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2975451 (10Paladox) @Marostegui I doint think they know about the bug @Jcrespo filled. [08:30:02] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2975471 (10Marostegui) >>! In T156373#2975451, @Paladox wrote: > @Marostegui I doint think they kno... [08:31:35] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2975472 (10Marostegui) Looks like they fixed it and will be shipped with the 10.0.30: ``` Resoluti... [08:35:16] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2975487 (10Paladox) @Marostegui this is the fix for 10.0 https://github.com/MariaDB/server/commit/7... [08:41:33] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2975516 (10Marostegui) Yes, let's wait for Jaime to build the package with that to see how it works... [09:02:14] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2975551 (10Paladox) Ok :) [09:04:04] 10DBA: Meta ticket: Deploy InnoDB compression where possible - https://phabricator.wikimedia.org/T150438#2975562 (10Marostegui) s2 results on dbstore2001: It went from `1.1T` to `763G` [09:04:43] 10DBA, 13Patch-For-Review: Import S2,S6,S7,m3 and x1 to dbstore2001 and dbstore2002 - https://phabricator.wikimedia.org/T151552#2975563 (10Marostegui) s2 was compressed on dbstore2001 (1.1T -> 763G) [09:26:27] 10DBA, 06Labs, 10Labs-Infrastructure, 10Tool-Labs, 10Wikimedia-Developer-Summit (2017): Labsdbs for WMF tools and contributors: get more data, faster - https://phabricator.wikimedia.org/T149624#2758290 (10Legoktm) I have moved the notes to https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/La... [09:53:47] 10DBA, 07Epic: [META ticket] Automation for our DBs tracking task - https://phabricator.wikimedia.org/T156461#2975675 (10Marostegui) [09:54:04] 10DBA: Framework to transfer files over the LAN - https://phabricator.wikimedia.org/T156462#2975688 (10Marostegui) [09:54:35] 10DBA, 07Epic: [META ticket] Automation for our DBs tracking task - https://phabricator.wikimedia.org/T156461#2975675 (10Marostegui) [09:54:37] 10DBA: Framework to transfer files over the LAN - https://phabricator.wikimedia.org/T156462#2975688 (10Marostegui) [10:11:32] 10DBA: For switchovers: A way to check if slaves are up to date - https://phabricator.wikimedia.org/T156465#2975756 (10Marostegui) [12:41:44] 10DBA, 06Operations, 10ops-codfw: Change rack for servers in s1 - https://phabricator.wikimedia.org/T156478#2976188 (10Marostegui) [12:42:17] 10DBA, 06Operations, 10ops-codfw: Change rack for servers in s1 - https://phabricator.wikimedia.org/T156478#2976205 (10Marostegui) [13:19:37] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2976245 (10jcrespo) On the DBA side of things, I am ok with that proposal, asking for @demon opinion, or anyone at gerrit app lay... [13:22:28] so the good news is that phabricator backups are working perfectly [13:23:14] they are rolling forward, which will confirm no problem happens, then stop before the conflictive alter [13:23:26] when that happens, I will stop and snapshot/backup it [13:23:47] try once more- confirm it still happens- which is what we believe [13:24:13] nice! [13:24:16] and maybe then [13:24:17] :-) [13:24:27] compile mariadb 29 with the patch [13:24:35] a 29-2 [13:24:51] and hopefully it will work fine :) [13:25:14] and start to write release notes for our packages [13:26:28] haha yeah will be useful if we ever need to look back [13:27:03] independently of the whole issue [13:27:08] I wanted to use the backups [13:27:16] we need to get better and faster at them [14:43:15] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2976466 (10demon) >>! In T145885#2976245, @jcrespo wrote: > On the DBA side of things, I am ok with that proposal, asking for @de... [15:08:54] 10DBA, 10Wikidata, 07Performance, 15User-Daniel, and 2 others: Build an environment to test change dispatching using Redis-based locking - https://phabricator.wikimedia.org/T155190#2976544 (10Ladsgroup) http://redis-dispatching-client.wmflabs.org/wiki/Main_Page and http://redis-dispatching-repo.wmflabs.org... [15:09:21] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2976550 (10Paladox) >>! In T145885#2976466, @demon wrote: >>>! In T145885#2976245, @jcrespo wrote: >> On the DBA side of things,... [15:20:34] 10DBA, 06Operations, 10ops-codfw: Change rack for servers in s1 - https://phabricator.wikimedia.org/T156478#2976587 (10Papaul) @Marostegui yes it is doable. [15:32:16] 10DBA, 06Operations, 10ops-codfw: Change rack for servers in s1 - https://phabricator.wikimedia.org/T156478#2976660 (10Marostegui) Thanks, let's do it next week! [16:05:27] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2976762 (10Paladox) +1 :) [16:11:30] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2976777 (10Paladox) yep the mariadb plugin fixes it. sudo mysql -p Enter password: Welcome to the MariaDB monitor. Commands en... [16:11:54] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2976778 (10Paladox) that just proofs that the db is not actually utf8mb4. So the fix in jdbc works :) [16:17:31] 10DBA, 06Operations: Move db1073 to B3 - https://phabricator.wikimedia.org/T156126#2976786 (10Marostegui) db1072 has been moved out (T156226) to another rack. So right now (well, once db1072 is recloned) we will have 1 of the API servers out from D1. I still think we should move db1073 out (right now we have... [16:19:54] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#2976789 (10faidon) Ping! Early February is now a week away. [16:21:02] 10DBA, 06Operations, 13Patch-For-Review: Reimage and clone db1072 - https://phabricator.wikimedia.org/T156226#2976790 (10Marostegui) db1072 has been moved to B2 DNS updated db-eqiad,codfw files updated mysql and replication started finely. tendril updated Pending: reimage and reclone Thanks @Cmjohnson [16:57:52] 10DBA, 06Operations: Switchover s1 master db1057 -> db1052 - https://phabricator.wikimedia.org/T156008#2976902 (10faidon) [17:18:19] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2976990 (10jcrespo) MariaDB's regresion test does work on the new package: https://github.com/Maria... [17:19:13] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2976994 (10jcrespo) [17:23:51] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2977001 (10Paladox) :) [17:28:39] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2977015 (10mmodell) Awesome, this is really impressive work everyone. Thanks for helping @paladox! [17:29:18] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2977017 (10Paladox) Your welcome :) [17:30:25] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2977032 (10mmodell) And of course mad props and thanks to @jcrespo, @Marostegui, and @epriestley [17:39:07] indeed [17:39:15] this bug being fixed so quickly is just incredibl [17:39:20] incredible even :) [17:40:12] well, it crashed all servers minus one [17:40:24] and by pure chance it didn't crash all of them [17:40:35] it was close to UBN [19:45:37] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2977533 (10Marostegui) >>! In T156373#2976990, @jcrespo wrote: > MariaDB's regresion test does work... [20:42:11] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2977727 (10jcrespo) The production test seemed ok- 10.0.29-2 not crashing on db1048 (not sure if it... [21:34:29] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2977856 (10Marostegui) >>! In T156373#2977727, @jcrespo wrote: > The production test seemed ok- 10.... [23:38:42] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2978846 (10Paladox) :)