[07:11:14] 10DBA: db1026 (s5) needs some compression - https://phabricator.wikimedia.org/T154929#2951613 (10Marostegui) Both, wikidatawiki and dewiki have been compressed on db1026 and nice we have some nice room: ``` root@db1026:~# df -hT /srv/ Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/tank... [07:11:31] 10DBA: db1026 (s5) needs some compression - https://phabricator.wikimedia.org/T154929#2951614 (10Marostegui) 05Open>03Resolved [08:19:23] 10DBA, 13Patch-For-Review: Fix dbstore2001 and dbstore2002 - https://phabricator.wikimedia.org/T130128#2951866 (10Marostegui) >>! In T130128#2948559, @Marostegui wrote: > For the record: I have restarted dbstore2001 mysql to manually apply the variables to make innodb the default storage engine as well as incr... [08:47:21] 10DBA, 13Patch-For-Review: Deploy gtid_domain_id flag in our mysql hosts - https://phabricator.wikimedia.org/T149418#2951891 (10Marostegui) phabricator hosts (m3) got the flag enabled yesterday. I have subimited a change to enable it on the eventlogging host, so we can get all the misc shards done. It doesn't... [09:30:31] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2951958 (10Marostegui) Hi, I have been doing some tests with the gerritdb to sum up all the stuff that have been discussed here... [09:54:54] 10DBA, 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Create a cronjob/check to run check_private_data data script and report back - https://phabricator.wikimedia.org/T153680#2952014 (10Marostegui) I would like to get this deployed by Monday so we can watch its behaviour during the week - worst case sc... [10:15:00] 10DBA: Defragment db1015 - https://phabricator.wikimedia.org/T153739#2952048 (10Marostegui) p:05Triage>03Normal [10:19:36] 10DBA, 10Wikidata: Repeated reports of wikidatawiki (s5) API going read only - https://phabricator.wikimedia.org/T123867#1939840 (10Ladsgroup) I still get read-only mode when editing via API several times per week [10:36:37] 10DBA, 06Operations, 10ops-codfw: Degraded RAID on db2011 - https://phabricator.wikimedia.org/T153740#2952106 (10Marostegui) Hey @Papaul can disk be replaced someday this week or early next week? Thank you! [11:07:29] 10DBA, 07Schema-change, 07Tracking: Schema changes for Wikimedia wikis (tracking) - https://phabricator.wikimedia.org/T51188#2952265 (10Marostegui) [11:07:31] 10DBA, 07Schema-change: Dropping hitcounter table on wmf databases - https://phabricator.wikimedia.org/T86340#2952262 (10Marostegui) 05Open>03Resolved a:03Marostegui This was done: https://phabricator.wikimedia.org/T132837 [12:25:59] 10DBA: Review capacity on codfw - https://phabricator.wikimedia.org/T155102#2952416 (10Marostegui) Taking a look simply at the mediawiki config files, these are the differences we have in the 7 shards: s1 ``` API eqiad: 3x160GB codfw: 2x160GB General traffic: eqiad: 3x512GB codfw: 4x160GB (2 of them serving... [13:25:05] 10DBA: Defragment db1038 - https://phabricator.wikimedia.org/T154465#2952632 (10Marostegui) I am now compressing the massive templatelinks tables on cebwiki [14:36:36] 10DBA, 13Patch-For-Review: Wikidatawiki revision table needs unification - https://phabricator.wikimedia.org/T150644#2952758 (10Marostegui) Only db1049 (the master) is pending. I will run it on Monday, it is a non blocking operation so it should be fine to run it early in the morning. [14:45:59] marostegui: https://etherpad.wikimedia.org/p/toolsdb-upgrade as announcement email for tools db upgrade [14:47:59] checking [14:48:24] looks good to me [14:48:40] marostegui: ok! I might not be available that day, so I'll check with labs folks before confirming [14:49:18] yuvipanda: ok - I have no idea about what's going on (I don't think jaime and myself discussed it) but I am sure Jaime is aware, I will ping him either tomorrow or monday to make sure he's aware [14:49:39] marostegui: ok! should we wait for him to be available and confirm before sending announcement? [14:49:47] marostegui: also labsdb1008 is actually a postgres host, not mysql [14:50:01] yuvipanda: he is not in today, and I am unsure about tomorrow. Monday he'll be [14:50:21] my main concern is if 6 hours is realistic (might be either too much or too little) [14:50:30] I think it is ok though, should be enough! [14:50:39] marostegui: yeah, I'd always rather it be more than less [14:50:45] marostegui: so this is for three hosts, two mysql and one postgres [14:50:57] the two mysql hosts are really just one tho - one is just a slave for DR [14:51:05] can they all be down at the same time? [14:51:26] marostegui: yes [14:51:33] marostegui: as long as we don't lose data it's fine :) [14:52:02] then I guess it should be fine, we start moving data around at the same time for all of them (as I said, it might take 1.5h+1.5h to move it somewhere and then back) [14:52:21] marostegui: right. ok! [14:56:24] 07Blocked-on-schema-change, 10DBA, 06Collaboration-Team-Triage, 10Flow, and 2 others: Add primary keys to remaining Flow tables - https://phabricator.wikimedia.org/T149819#2952811 (10Marostegui) I think by next week we can start deploying this change. As the tables are rather small, it is probably safer to... [14:58:06] 07Blocked-on-schema-change, 10DBA, 06Collaboration-Team-Triage, 10Flow, and 2 others: Add primary keys to remaining Flow tables - https://phabricator.wikimedia.org/T149819#2764089 (10Trizek-WMF) What would be the possible impact for users? [15:00:09] 07Blocked-on-schema-change, 10DBA, 06Collaboration-Team-Triage, 10Flow, and 2 others: Add primary keys to remaining Flow tables - https://phabricator.wikimedia.org/T149819#2952819 (10Marostegui) >>! In T149819#2952815, @Trizek-WMF wrote: > What would be the possible impact for users? The table will be blo... [15:03:02] 07Blocked-on-schema-change, 10DBA, 06Collaboration-Team-Triage, 10Flow, and 2 others: Add primary keys to remaining Flow tables - https://phabricator.wikimedia.org/T149819#2952823 (10Trizek-WMF) Thanks @Marostegui. How can I explain that? "Some Flow databases are going to be changed, that may cause some s... [15:12:56] 07Blocked-on-schema-change, 10DBA, 06Collaboration-Team-Triage, 10Flow, and 2 others: Add primary keys to remaining Flow tables - https://phabricator.wikimedia.org/T149819#2952881 (10Marostegui) I just did a test to give you more detailed data about how long the change will take and the impact. On the smal... [15:25:09] 10DBA, 10MediaWiki-General-or-Unknown, 10ORES, 10Revision-Scoring-As-A-Service-Backlog, 15User-Ladsgroup: Fatal exception of type "DBQueryError" on sorting ORES contributions - https://phabricator.wikimedia.org/T155500#2952895 (10Halfak) a:03Ladsgroup [16:03:03] 10DBA, 10Monitoring, 06Operations: Create a check/calendar alert for MariaDB TLS certs - https://phabricator.wikimedia.org/T152427#2953089 (10Dzahn) Hi, i can take a shot at this. Did it for other certs before. where are the certs located please. I looked in files/ssl/ in puppet repo. Where do they get insta... [16:09:08] 10DBA, 10Monitoring, 06Operations: Create a check/calendar alert for MariaDB TLS certs - https://phabricator.wikimedia.org/T152427#2953095 (10Marostegui) Hey @Dzahn help is welcomed!! They get installed here: ``` /etc/mysql/ssl ``` [16:20:31] 10DBA, 10Monitoring, 06Operations: Create a check/calendar alert for MariaDB TLS certs - https://phabricator.wikimedia.org/T152427#2953129 (10jcrespo) @Dzahn, ideally, the check should be done connecting to the servers. The files could be there, but not loaded into memory after a restart, and files are not l... [16:23:21] 10DBA, 10Monitoring, 06Operations: Create a check/calendar alert for MariaDB TLS certs - https://phabricator.wikimedia.org/T152427#2953136 (10faidon) @Jcrespo is correct, files on disk aren't the right way to monitor this. `check_ssl` should work for this use case, has been explicitly been made to work with... [16:31:41] 07Blocked-on-schema-change, 10DBA, 06Collaboration-Team-Triage, 10Flow, and 3 others: Add primary keys to remaining Flow tables - https://phabricator.wikimedia.org/T149819#2953178 (10Trizek-WMF) >>! In T149819#2952881, @Marostegui wrote: > So the impact will be almost 0, so you might want to mention that a... [16:33:45] 07Blocked-on-schema-change, 10DBA, 06Collaboration-Team-Triage, 10Flow, and 3 others: Add primary keys to remaining Flow tables - https://phabricator.wikimedia.org/T149819#2953190 (10Marostegui) >>! In T149819#2953178, @Trizek-WMF wrote: > When do you plan to perform that change? Not sure yet, probably ea... [16:38:38] 07Blocked-on-schema-change, 10DBA, 06Collaboration-Team-Triage, 10Flow, and 3 others: Add primary keys to remaining Flow tables - https://phabricator.wikimedia.org/T149819#2953192 (10Trizek-WMF) >>! In T149819#2953190, @Marostegui wrote: > Not sure yet, probably early next week. I can update the ticket on... [16:39:31] 07Blocked-on-schema-change, 10DBA, 06Collaboration-Team-Triage, 10Flow, and 3 others: Add primary keys to remaining Flow tables - https://phabricator.wikimedia.org/T149819#2953194 (10Marostegui) >>! In T149819#2953192, @Trizek-WMF wrote: >>>! In T149819#2953190, @Marostegui wrote: >> Not sure yet, probably... [17:41:20] marostegui: still around? [17:53:36] hey jynus [17:53:38] are you working today?! [17:53:59] is it an emergency? [17:54:46] jynus: no :) so you aren't here! [17:54:49] I can't seee youuuu [17:54:59] :) [17:55:00] send an email, please [18:37:43] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#2953719 (10yuvipanda) Update: Since I'll be travelling on the 25th, I'm going to push this out to early February instead. I'll ping @jcrespo when he's... [18:42:55] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2953741 (10demon) >>! In T145885#2951958, @Marostegui wrote: > That will convert ALL tables - ie: tables with charset binary to u... [18:44:56] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#2953763 (10jcrespo) +1, let's meet before to clarify impact. [18:46:03] jynus: go and vacation! [18:47:17] this is me landing, I just need to start softly, that is why I did not want any meeting or anything today [18:48:57] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2953775 (10Paladox) >>! In T145885#2951958, @Marostegui wrote: > Hi, > > I have been doing some tests with the gerritdb to sum... [18:50:41] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2953779 (10demon) >>! In T145885#2953775, @Paladox wrote: > Oh, sorry didn't see your reply until now, thanks for also testing th... [18:51:48] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2953780 (10Paladox) Oh, we can ignore converting the tables that were binary as well the connection is already utf8 and we haven'... [19:09:46] 10DBA, 10Gerrit, 06Operations, 06Release-Engineering-Team: Gerrit: Schedule downtime to migrate db to utf8mb4 - https://phabricator.wikimedia.org/T155764#2953844 (10Paladox) [19:10:09] 10DBA, 10Gerrit, 06Operations, 06Release-Engineering-Team: Gerrit: Schedule downtime to migrate db to utf8mb4 - https://phabricator.wikimedia.org/T155764#2953859 (10Paladox) [19:10:15] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2894688 (10Paladox) [19:18:40] 10DBA, 10Gerrit, 06Operations, 06Release-Engineering-Team: Gerrit: Schedule downtime to migrate db to utf8mb4 - https://phabricator.wikimedia.org/T155764#2953926 (10Paladox) This patch https://gerrit.wikimedia.org/r/#/c/330455/ will need merging before taking gerrit offline. Will need to follow sql migrat... [19:42:00] 10DBA, 06Operations, 10Wikimedia-General-or-Unknown: Spurious completely empty `image` table row on commonswiki - https://phabricator.wikimedia.org/T155769#2953992 (10matmarex) [20:05:37] 10DBA, 10Gerrit, 06Operations, 13Patch-For-Review, 07Upstream: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2954071 (10Paladox) (i've created this patch against mysql connector, though we won't be using it, just putting it here for other...