[07:38:02] 10DBA, 13Patch-For-Review: db2034: investigate its crash and reimage - https://phabricator.wikimedia.org/T149553#2915945 (10Marostegui) The server is still responding fine, which is great news. This is the state of the alter ``` Stage: 1 of 2 'copy to tmp table' 74.1% of stage done ``` [07:44:43] 10DBA: Defragment db1015 - https://phabricator.wikimedia.org/T153739#2915948 (10Marostegui) Compressing top biggest tables on this host to get some extra space. [07:57:34] 10DBA: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#2915953 (10Marostegui) What about starting to work on enwiki.revision on codfw? If we are going to do a DC switchover at some point, we can start with codfw clean and while eqiad is on sby m... [08:35:46] 10DBA: Remove partitions from enwiktionary.templatelinks in s2 - https://phabricator.wikimedia.org/T154097#2915969 (10Marostegui) Thanks for your answer and from the clue about codfw cloning, that might explain why codfw has partitions!! Very nice catch! Maybe db1054 was serving recentchanges at some point befor... [09:34:53] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916063 (10jcrespo) > we will not be able to use jdbc for this as it is not supported, so we will need to set it server side. Again, this is not a... [09:41:37] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2644059 (10Joe) I might be very unpopular with this opinion, but: I think emojis in code reviews should be avoided. Plain old written text is the w... [09:54:35] 10DBA: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038 - https://phabricator.wikimedia.org/T154485#2916083 (10Marostegui) Checking the latest version of percona toolkit (v2.2.20 released 6th Dec 2016) it contains Jaime's patch (actually since 2.2.19): * Fixed bug 1485195: pt-table-che... [10:42:57] 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1048 - https://phabricator.wikimedia.org/T152411#2916144 (10Marostegui) @Cmjohnson for this ticket, let's go one by one. Change one, we will let the raid rebuild and then change the other one. Thanks! [10:50:27] 10DBA, 06Labs, 10Tool-Labs: Spatial database for tool-labs - https://phabricator.wikimedia.org/T154497#2916154 (10akosiaris) >>! In T154497#2913411, @scfc wrote: > There is a PostgreSQL database that replicates data from OSM and is used by some tools; currently accounts for that database are managed manually... [13:37:52] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916378 (10Aklapper) (Same feelings here as Joe: Task could be declined.) [13:40:08] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916379 (10Paladox) But utf8 is not full unicode. It's better to just fix it then to leave it. Yes we shouldn't bug dba but this is a db thing. [14:05:29] 10DBA: Remove partitions from enwiktionary.templatelinks in s2 - https://phabricator.wikimedia.org/T154097#2916429 (10Anomie) The only change to `wmf-config/db-eqiad.php` in git touching db1054 before the "depool for codfw cloning" was when it was first added in {d67da49efc}. It may be that it was itself cloned... [14:08:56] 10DBA: Remove partitions from enwiktionary.templatelinks in s2 - https://phabricator.wikimedia.org/T154097#2916430 (10Marostegui) >>! In T154097#2916429, @Anomie wrote: > The only change to `wmf-config/db-eqiad.php` in git touching db1054 before the "depool for codfw cloning" was when it was first added in {d67d... [14:59:34] 10DBA, 10MediaWiki-Database, 07Performance: revision page_user_timestamp index problematic on large wikis - https://phabricator.wikimedia.org/T102532#2916523 (10Anomie) As far as usage, the API targets this index for ApiQueryContributors (although it only uses the prefix `rev_page,rev_user`) and for ApiQuery... [15:08:03] 10DBA, 10MediaWiki-Database, 07Performance: revision page_user_timestamp index problematic on large wikis - https://phabricator.wikimedia.org/T102532#1366835 (10Marostegui) Thanks again @Anomie! I wouldn't like to experiment with enwiki and for: T132416#2915953 I would just leave it there. Maybe we can see t... [15:51:40] 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1048 - https://phabricator.wikimedia.org/T152411#2916581 (10Marostegui) Chris has replaced 32:0 disk (which is part of the SPAN #0) It is rebuilding now: ``` root@db1048:~# megacli -PDRbld -ShowProg -PhysDrv [32:0] -aALL Rebuild Progress on Device at Enc... [15:54:16] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916586 (10demon) What @paladox said. Yes, this was filed in context of emojis, but really it's about all extended unicode that mysql/maria's "utf8"... [15:58:29] 10DBA, 10MediaWiki-Database, 07Performance: revision page_user_timestamp index problematic on large wikis - https://phabricator.wikimedia.org/T102532#2916607 (10Anomie) For experimenting with the lack of the index you could also take the queries and add `IGNORE INDEX (page_user_timestamp)`, I believe. For A... [16:19:57] 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1033 - https://phabricator.wikimedia.org/T152214#2916705 (10Cmjohnson) 05Open>03Resolved a:03Cmjohnson Disk 7 was replaced: cmjohnson@db1033:~$ sudo megacli -PDList -aALL |grep "Firmware state" Firmware state: Online, Spun Up Firmware state: Online,... [16:21:01] 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1048 - https://phabricator.wikimedia.org/T152411#2916710 (10Marostegui) Span #0 is now rebuilt. ``` root@db1048:~# megacli -PDRbld -ShowProg -PhysDrv [32:0] -aALL Device(Encl-32 Slot-0) is not in rebuild process Device Present... [16:21:53] 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1033 - https://phabricator.wikimedia.org/T152214#2916715 (10Marostegui) Thanks!! [16:22:22] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916718 (10Paladox) Oh, is that mysql binary or is it utf8 binary? [16:23:48] 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1048 - https://phabricator.wikimedia.org/T152411#2916721 (10Marostegui) 32:2 has been replaced and it is getting rebuilt ``` root@db1048:~# megacli -PDRbld -ShowProg -PhysDrv [32:2] -aALL Rebuild Progress on Device at Enclosure 32, Slot 2 Completed 2% in... [16:41:44] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916768 (10Dzahn) >>! In T145885#2916065, @Joe wrote: > I think emojis in code reviews should be avoided. > I honestly don't see why this should ev... [16:47:58] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916773 (10jcrespo) > this is a db thing To prove this is **not** a DB server issue, but a client code/driver bug: ``` $ mysql -h m2-master review... [16:48:43] ^ :-) [16:51:30] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916775 (10demon) >>! In T145885#2916773, @jcrespo wrote: >> this is a db thing > > To prove this is **not** a DB server issue, but a client code/d... [16:55:22] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916800 (10demon) >>! In T145885#2915101, @Paladox wrote: > Apparently jdbc does not support utf8mb4 > > https://www.google.co.uk/#q=fatal:+++cause... [16:55:55] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916801 (10jcrespo) > altering existing tables/data Altering tables and is not a problem, changing global configuration that affects other database... [17:04:54] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916841 (10Paladox) >>! In T145885#2916800, @demon wrote: >>>! In T145885#2915101, @Paladox wrote: >> Apparently jdbc does not support utf8mb4 >> >... [17:07:41] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916844 (10demon) >>! In T145885#2916841, @Paladox wrote: > oh so doing characterEncoding=utf8mb4 works? Because when i tried that it failed to sta... [17:08:32] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916846 (10Paladox) Oh, continuing to use utf-8 works but pasting an emoji still fails with 500 error. [17:11:13] 10DBA: Unknown cause is creating lag on db1048 under write load (but not on the other m3 slaves) - https://phabricator.wikimedia.org/T151039#2916855 (10Marostegui) [17:11:15] 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1048 - https://phabricator.wikimedia.org/T152411#2916852 (10Marostegui) 05Open>03Resolved a:03Cmjohnson All good now - thanks Chris!! ``` root@db1048:~# megacli -PDRbld -ShowProg -PhysDrv [32:2] -aALL Device(Encl-32 Slot-2) is not in rebuild proces... [17:14:34] 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1053 - https://phabricator.wikimedia.org/T151465#2916861 (10Cmjohnson) [17:15:24] 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1053 - https://phabricator.wikimedia.org/T151465#2818166 (10Cmjohnson) we do not have any spare disks at this time. We will need to order a few spare. [17:18:08] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2916878 (10Dzahn) @paladox What was the solution that resulted in it _not_ showing a 500 error but just a ? instead. Afair there was one? That would... [17:52:46] 10DBA, 10MediaWiki-Database, 07Performance: revision page_user_timestamp index problematic on large wikis - https://phabricator.wikimedia.org/T102532#2916969 (10Anomie) I see https://gerrit.wikimedia.org/r/#/c/329340/ would add a use of the index in HistoryAction, similar to the use in ApiQueryRevisions. [17:53:24] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#1936600 (10faidon) Early January is here and the 15th is coming up fast ­­-- @yuvipanda rightfully mentioned above that this will need a (presumably a... [17:55:47] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#2916980 (10yuvipanda) @jcrespo How does Jan 25 / 26 work for you? [18:03:49] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#2917046 (10jcrespo) >>! In T123731#2916980, @yuvipanda wrote: > @jcrespo How does Jan 25 / 26 work for you? +1. let's meet at some point to organize... [18:28:11] 10DBA, 10MediaWiki-Database, 07Performance: revision page_user_timestamp index problematic on large wikis - https://phabricator.wikimedia.org/T102532#2917168 (10jcrespo) @Anomie To clarify on the state of the ticket. This is one of the "we found this open ticket", which certainly is relevant to the work bein... [18:56:46] 10DBA, 10MediaWiki-Database, 07Performance: revision page_user_timestamp index problematic on large wikis - https://phabricator.wikimedia.org/T102532#2917254 (10Anomie) Thanks, I wasn't worried. Sharing my knowledge and experience is why I'm commenting here, we can't make an informed decision unless someone... [19:08:35] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2917289 (10Paladox) @Dzahn i if we do just &connectionCollation=utf8mb4_unicode_ci that should at least fix the 500 errors but emojis won't show. [19:29:14] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2917389 (10Paladox) @Dzahn and @demon setting ?useUnicode=true should stop the 500 errors but emojis won't show for now (db conversion required too) :) [19:30:16] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2917395 (10Paladox) https://bugs.mysql.com/bug.php?id=57694 [21:12:52] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2917759 (10Aklapper) >>! In T145885#2916586, @demon wrote: > this was filed in context of emojis, but really it's about all extended unicode that my... [21:47:01] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2917868 (10Dzahn)