[07:15:54] 10DBA, 06Operations, 10ops-codfw: db2035 reset - https://phabricator.wikimedia.org/T154189#2912029 (10Marostegui) 05Open>03Resolved a:03akosiaris The server looks fine now, so I will close this task for now as looks like it was Papaul by mistake. Thanks for taking care of this Alex! [07:35:20] 10DBA, 13Patch-For-Review: db2034: investigate its crash and reimage - https://phabricator.wikimedia.org/T149553#2912041 (10Marostegui) I have started the transfer now. [07:45:19] 10DBA: Defragment db1044 - https://phabricator.wikimedia.org/T153826#2912060 (10Marostegui) ``` marostegui@db1044:~$ df -hT /srv/ Filesystem Type Size Used Avail Use% Mounted on /dev/sda6 xfs 1.7T 1.5T 147G 92% /srv ``` Going to run a full optimize over this host to avoid it paging during the su... [07:53:03] 10DBA: Defragment db1044 - https://phabricator.wikimedia.org/T153826#2912084 (10Marostegui) Going to start with this (not with cebwiki/templatelinks yet, as there is not currently enough disk space to alter that table): ``` root@db1044:/srv/sqldata# find -type f -exec du -Sh {} + | sort -rh | head -n 5 207G ./c... [08:00:18] 10DBA: Defragment db1015 - https://phabricator.wikimedia.org/T153739#2912089 (10Marostegui) ``` root@db1015:/srv/sqldata# df -hT /srv/ Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/tank-data xfs 1.6T 1.5T 147G 91% /srv root@db1015:/srv/sqldata# find -type f -exec du -Sh {} + | sor... [08:06:13] 10DBA: Defragment db1038 - https://phabricator.wikimedia.org/T154465#2912092 (10Marostegui) [08:19:57] 10DBA: Defragment db1038 - https://phabricator.wikimedia.org/T154465#2912114 (10Marostegui) I have started the script to optimize revision, templatelinks and pagelinks on this host. [09:16:57] 10DBA, 13Patch-For-Review: db2034: investigate its crash and reimage - https://phabricator.wikimedia.org/T149553#2912153 (10Marostegui) First transfer finished correctly! Going for the second one to confirm it wasn't just coincidence (as it has happened in the past) [10:35:22] 10DBA, 13Patch-For-Review: Import S2,S6,S7,m3 and x1 to dbstore2001 and dbstore2002 - https://phabricator.wikimedia.org/T151552#2912254 (10Marostegui) Comparing the timing of both servers: dbstore2001 ``` stop all slaves - 8 seconds stop mysql - 9 minutes start mysql (and be able to get in the prompt: 10 min... [10:53:32] I am going to put db1035 under maintenance [10:53:52] ok, give me a sec [10:54:17] ok, you can go ahead now [10:58:24] 10DBA, 13Patch-For-Review: db2034: investigate its crash and reimage - https://phabricator.wikimedia.org/T149553#2912314 (10Marostegui) Second attempt was also good and the server is alive! Will try a third one and after that if it goes good too, I will start a massive alter that will take some days to see if... [11:14:01] 10DBA, 06Operations, 13Patch-For-Review: Throttle mysql backups on dbstore1001 in order to not saturate the node - https://phabricator.wikimedia.org/T134977#2912319 (10jcrespo) 05Open>03Resolved This seems solved now, no alerts are generated and all backups are created as expected. [12:01:37] 10DBA: Defragment db1038 - https://phabricator.wikimedia.org/T154465#2912374 (10jcrespo) a:05Marostegui>03jcrespo [12:53:45] 10DBA, 13Patch-For-Review: db2034: investigate its crash and reimage - https://phabricator.wikimedia.org/T149553#2912487 (10Marostegui) And the third one also went fine! [14:24:26] 10DBA, 13Patch-For-Review: db2034: investigate its crash and reimage - https://phabricator.wikimedia.org/T149553#2912725 (10Marostegui) The server is now replicating and I am running the following alter: ``` ALTER TABLE enwiki.revision row_format = compressed, DROP PRIMARY KEY, DROP INDEX rev_id, ADD PRIMARY... [14:45:12] 10DBA: Defragment db1044 - https://phabricator.wikimedia.org/T153826#2912782 (10Marostegui) I am compressing now the biggest tables on db1044 so we can get space. Reminder, this server is currently acting as master for db1095 (sanitarium2) [15:14:07] 10DBA: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038 - https://phabricator.wikimedia.org/T154485#2912830 (10Marostegui) [15:14:26] 10DBA: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038 - https://phabricator.wikimedia.org/T154485#2912844 (10Marostegui) p:05Triage>03Normal [15:45:45] 10DBA, 06Operations, 10ops-codfw: db2060 crashed (RAID controller) - https://phabricator.wikimedia.org/T154031#2912918 (10Marostegui) I have been talking to Papaul and he's kindly agreed to upgrade its BIOS on Thursday, so we will reboot and upgrade it. [15:48:22] 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1053 - https://phabricator.wikimedia.org/T151465#2912927 (10Marostegui) @Cmjohnson can we get this disk replaced? Thanks! [16:12:52] 10DBA: Delete prefstats tables - https://phabricator.wikimedia.org/T154490#2912996 (10Neil_P._Quinn_WMF) [16:13:54] 10DBA, 07Epic, 07Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2913014 (10Neil_P._Quinn_WMF) [16:13:56] 10DBA: Delete prefstats tables - https://phabricator.wikimedia.org/T154490#2913013 (10Neil_P._Quinn_WMF) [16:15:09] 10DBA: Delete prefstats tables - https://phabricator.wikimedia.org/T154490#2912996 (10Marostegui) Thanks for letting us know! We will take care of them and we will also backup them before. They do exist on production: ``` root@db2034:/srv/sqldata/enwiki# ls -lh prefstats.ibd -rw-rw---- 1 mysql mysql 300M Mar 12... [16:15:40] 10DBA: Delete prefstats tables - https://phabricator.wikimedia.org/T154490#2913019 (10Marostegui) p:05Triage>03Normal [16:17:43] 10DBA, 07Epic, 07Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#2913026 (10demon) [16:18:03] 10DBA: Delete prefstats tables - https://phabricator.wikimedia.org/T154490#2913030 (10Neil_P._Quinn_WMF) [16:25:12] I just noticed that: https://phabricator.wikimedia.org/T154097 the partitions are only on codfw, so our earlier conversation about the partitions being there for a reason is much clearer now [16:25:28] They aren't needed if they are not in eqiad and no one has said anything, don't you think? [16:29:30] or [16:29:50] they are needed and we haven't had the time to apply them [16:29:58] that is also true [16:30:11] I am asking anomie anyways :) [16:30:19] 1054 has them [16:30:29] yep, but it is the rc service [16:30:50] ah no, it is api [16:31:25] the problem is that he will not have a definitive answer [16:31:42] no, but at least maybe he can point us in the right direction [16:31:47] 10DBA: Remove partitions from enwiktionary.templatelinks in s2 - https://phabricator.wikimedia.org/T154097#2913076 (10Marostegui) @Anomie we were wondering if you'd have some thoughts about these partitions on this particular wiki. We were considering deleting them because they look like a mistake, and they are... [16:54:12] 10DBA: Defragment db1044 - https://phabricator.wikimedia.org/T153826#2913155 (10Marostegui) I am leaving a few of the biggest tables on the host getting compressed over the night. The server is depooled anyways (I silenced it anyways on Icinga just in case). ``` root@db1044:/srv/sqldata# find -type f -exec du -... [16:56:26] 10DBA, 10Edit-Review-Improvements-RC-Page, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), 05MW-1.29-release-notes, and 2 others: Implement functionality for RC page 'Experience level' filters - https://phabricator.wikimedia.org/T149637#2913183 (10jmatazzoni) [17:01:08] 10DBA, 10Edit-Review-Improvements-RC-Page, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), 05MW-1.29-release-notes, and 2 others: Implement functionality for RC page 'Experience level' filters - https://phabricator.wikimedia.org/T149637#2913208 (10jmatazzoni) Thanks for the recommendation @Etonk... [17:21:07] 10DBA, 10Edit-Review-Improvements-RC-Page, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), 05MW-1.29-release-notes, and 2 others: Implement functionality for RC page 'Experience level' filters - https://phabricator.wikimedia.org/T149637#2913276 (10SBisson) >>! In T149637#2913208, @jmatazzoni wro... [17:22:27] 10DBA, 10Edit-Review-Improvements-RC-Page, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), 05MW-1.29-release-notes, and 2 others: Implement functionality for RC page 'Experience level' filters - https://phabricator.wikimedia.org/T149637#2913278 (10jmatazzoni) 05Open>03Resolved [17:27:30] 10DBA: Remove partitions from enwiktionary.templatelinks in s2 - https://phabricator.wikimedia.org/T154097#2913294 (10Anomie) As far as I know, MediaWiki doesn't specify partitions at all, so it would be something a DBA did at some point. Note the origin of the partitioning probably predates our current DBAs. I... [17:27:48] 10DBA, 06Labs, 10Tool-Labs: Spatial database for tool-labs - https://phabricator.wikimedia.org/T154497#2913296 (10Tobias1984) [17:46:14] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2913402 (10Paladox) [17:47:24] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2644059 (10Paladox) [17:50:37] 10DBA, 06Labs, 10Tool-Labs: Spatial database for tool-labs - https://phabricator.wikimedia.org/T154497#2913411 (10scfc) There is a PostgreSQL database that replicates data from OSM and is used by some tools; currently accounts for that database are managed manually in `role::osm::master`, mainly by @akosiari... [17:59:04] 10DBA, 06Labs, 10Tool-Labs: Spatial database for tool-labs - https://phabricator.wikimedia.org/T154497#2913296 (10bd808) We have an [[https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Elasticsearch|Elasticsearch cluster in Tool Labs]]. Could you achieve the same end result by using the [[https://www.elastic... [18:12:35] 10DBA, 06Labs, 10Tool-Labs: Spatial database for tool-labs - https://phabricator.wikimedia.org/T154497#2913296 (10EBernhardson) Can this be accomplished with the geo keywords available on commons for search? https://www.mediawiki.org/wiki/Help:CirrusSearch#Geo_Search https://commons.wikimedia.org/w/index.ph... [18:17:41] 10DBA, 10Gerrit, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Gerrit: Schedule downtime for T154205 (To do with data loss) - https://phabricator.wikimedia.org/T154327#2913541 (10demon) Gerrit's built against latest stable-2.13 and uploaded for review. Need to pick a (very lengthy, low-tr... [19:03:14] 10DBA, 10Gerrit, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Gerrit: Schedule downtime for T154205 (To do with data loss) - https://phabricator.wikimedia.org/T154327#2913752 (10Paladox) @ostriches that could be done tonight as most of wikimedia is back but not everyone so it would be a... [19:03:48] 10DBA, 10Gerrit, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Gerrit: Schedule downtime for T154205 (To do with data loss) - https://phabricator.wikimedia.org/T154327#2913754 (10demon) I'm not rushing into it **tonight**, it will be scheduled and announced. [19:05:13] 10DBA, 10Gerrit, 06Operations, 06Release-Engineering-Team, 13Patch-For-Review: Gerrit: Schedule downtime for T154205 (To do with data loss) - https://phabricator.wikimedia.org/T154327#2913761 (10Paladox) Ok [22:23:22] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2914781 (10Paladox) [22:23:33] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2644059 (10Paladox) [23:21:11] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2915066 (10Paladox) >>! In T145885#2897016, @jcrespo wrote: >> ahaha it works now. >> >> you have to set >> >> character-set-client-handshake = FA... [23:27:30] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2915101 (10Paladox) Apparently jdbc does not support utf8mb4 https://www.google.co.uk/#q=fatal:+++caused+by+java.sql.SQLException:+Unsupported+char... [23:29:39] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2915102 (10Paladox) https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-charsets.html [23:32:56] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting an emoji - https://phabricator.wikimedia.org/T145885#2915107 (10Paladox) >>! In T145885#2897016, @jcrespo wrote: >> ahaha it works now. >> >> you have to set >> >> character-set-client-handshake = FA...