[00:47:38] 10DBA, 06Operations, 13Patch-For-Review: Set up TLS for MariaDB replication - https://phabricator.wikimedia.org/T111654#2988603 (10jcrespo) 36 pending hosts: ``` db1030.eqiad.wmnet: NULL db1045.eqiad.wmnet: NULL db1020.eqiad.wmnet: NULL db1001.eqiad.wmnet: NULL db1039.eqiad.wmnet: NULL db1026.eqiad.wmnet: NU... [06:47:21] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2989006 (10Marostegui) db2012 caught up nicely so I believe this ticket can be closed. We can disc... [06:48:22] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team, 07Upstream: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time - https://phabricator.wikimedia.org/T156373#2989008 (10Marostegui) 05Open>03Resolved a:03jcrespo [06:50:32] 10DBA, 10Analytics-EventLogging, 06Analytics-Kanban, 13Patch-For-Review: Add autoincrement id to EventLogging MySQL tables. {oryx} - https://phabricator.wikimedia.org/T125135#2989010 (10Marostegui) >>! In T125135#2987577, @Ottomata wrote: >> This is key, is that somehow doable from the application side? It... [07:08:42] 10DBA, 10Analytics, 06Operations: Improve eventlogging replication procedure - https://phabricator.wikimedia.org/T124307#2989026 (10Marostegui) >>! In T124307#2987587, @Ottomata wrote: > @Marostegui ok! So the T125135 auto-increment thing is a very small piece of this larger issue. > > Let's see if we can... [07:16:22] 10DBA, 06Labs: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#2989039 (10Marostegui) I would like to start importing s4 soon, so for that we need a host to be its master. I originally thought about db1063 and db1067, but maybe that is not... [07:33:46] 10DBA, 06Operations: Move db1073 to B3 - https://phabricator.wikimedia.org/T156126#2989059 (10Marostegui) Let's not do this until db1072 issues are fixed (T156226#2984963) [08:27:00] I will let you know when I see jynus around here [08:27:00] @notify jynus [08:45:44] 07Blocked-on-schema-change, 06Collaboration-Team-Triage, 10Notifications, 13Patch-For-Review, 07Schema-change: Add primary key to echo_notification table - https://phabricator.wikimedia.org/T136428#2334714 (10Marostegui) I have been taking a look at these tables. They appear to exist on: - s3 (not in all... [09:40:23] hello people, I am trying to access https://phabricator.wikimedia.org/T155755 but I can see "#2013: Lost connection to MySQL server at 'reading initial communication packet', system error: 0" [09:40:39] hey [09:40:50] wow [09:42:52] elukey see operations [10:05:18] 10DBA, 10Phabricator: Phabricator master crashed - https://phabricator.wikimedia.org/T156905#2989313 (10Marostegui) [10:06:46] 10DBA, 10Phabricator: Phabricator master crashed - https://phabricator.wikimedia.org/T156905#2989326 (10Marostegui) [10:08:17] 10DBA, 10Phabricator: Phabricator master crashed - https://phabricator.wikimedia.org/T156905#2989313 (10Marostegui) ``` check_ferm invoked oom-killer: gfp_mask=0x2420848, order=0, oom_score_adj=0 check_ferm cpuset=/ mems_allowed=0-1 CPU: 9 PID: 29001 Comm: check_ferm Tainted: G I 4.4.0-2-amd64 #1... [10:13:15] 10DBA, 10Phabricator: Phabricator master crashed - https://phabricator.wikimedia.org/T156905#2989313 (10Volans) Looks like there was some heavy load on the server in the ~20 minutes before the OOM: https://grafana.wikimedia.org/dashboard/file/server-board.json?var-server=db1043&var-network=eth0&from=148594039... [10:19:13] 10DBA, 10Phabricator: Phabricator master crashed - https://phabricator.wikimedia.org/T156905#2989362 (10Marostegui) There are some really slow queries running at the time: https://tendril.wikimedia.org/report/slow_queries?host=%5Edb1043&user=&schema=&qmode=eq&query=&hours=1 The replicat (db1048 also crashed a... [10:31:52] 10DBA, 10Phabricator: Phabricator master crashed - https://phabricator.wikimedia.org/T156905#2989371 (10Marostegui) This is the last insert we got before the crash (of both, master and slave) ``` INSERT INTO `search_documentfield` (phid, phidType, field, auxPHID, corpus, stemmedCorpus) VALUES ('PHID... [10:33:03] 10DBA, 10Phabricator: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989373 (10Marostegui) p:05Triage>03High [10:34:19] 10DBA, 10Phabricator: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989313 (10Marostegui) I have tried to reproduce that insert on the following versions on my lab: - 10.0.23 - 10.0.28 And they didn't crash (just checking if has something to do with: T156373) [10:57:49] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989313 (10Marostegui) [11:12:05] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989405 (10Marostegui) >>! In T156905#2989343, @Volans wrote: > Looks like there was some heavy load on the server in the ~20 minutes before the OOM: > > https:/... [11:29:51] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989430 (10Paladox) @Marostegui hi, it looks like the server is running version Server version: 10.0.23-MariaDB-log Should we update it to 10.0.29 the package th... [11:38:43] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989448 (10Paladox) @Marostegui could it b a memory leak? could this https://github.com/MariaDB/server/commit/b7dc830 be the fix? [11:44:56] 10DBA, 13Patch-For-Review: Remove partitions from metawiki.pagelinks in s7 - https://phabricator.wikimedia.org/T153300#2989465 (10Marostegui) db2061: ``` root@neodymium:/home/marostegui/git# mysql --skip-ssl -hdb2061.codfw.wmnet metawiki -e "show create table pagelinks\G" *************************** 1. row **... [11:46:20] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989466 (10jcrespo) > However, db1048 (the slave) also crashed and that one is only supposed to have the replication thread running right? No, at around 09:39:31... [11:47:21] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989467 (10Marostegui) >>! In T156905#2989430, @Paladox wrote: > @Marostegui hi, it looks like the server is running version Server version: 10.0.23-MariaDB-log >... [11:51:48] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989474 (10Paladox) does this mean that phabricator needs improvements to it's query? [12:52:50] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989591 (10Marostegui) >>! In T156905#2989474, @Paladox wrote: > does this mean that phabricator needs improvements to it's query? I believe so, or at least on t... [13:24:58] I fixed the phab backups [13:25:10] I had deleted an account when securing it [13:25:35] I have put it back, and running backup now [13:57:36] 10DBA, 10Analytics, 06Operations: Improve eventlogging replication procedure - https://phabricator.wikimedia.org/T124307#2989663 (10Ottomata) @mforns, can you comment about large DELETES? Do they happen often? How large are they when it happens? Would LOAD DATA actually help replication? [14:14:24] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989711 (10epriestley) I'm not sure why the query is slow or requires a significant amount of memory. The structure of the query specifically tries to avoid this,... [14:16:56] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989717 (10epriestley) One vague possibility is that you may have a few documents which are extremely large: for example, perhaps someone wrote a 1GB comment on a... [14:22:13] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989733 (10jcrespo) [14:30:32] 10DBA, 10Analytics, 06Operations: Improve eventlogging replication procedure - https://phabricator.wikimedia.org/T124307#2989770 (10Marostegui) >>! In T124307#2989663, @Ottomata wrote: > > @Marostegui , Would LOAD DATA actually help replication? If you need to do massive data imports into the DB, it will h... [14:39:14] 10DBA, 10Analytics, 06Operations: Improve eventlogging replication procedure - https://phabricator.wikimedia.org/T124307#2989845 (10Ottomata) EventLogging is a stream of data. We can do batching because the data is consumed from Kafka, and then inserted into MySQL via a python MySQL client. So we could con... [14:56:10] 10DBA, 10Analytics, 06Operations: Improve eventlogging replication procedure - https://phabricator.wikimedia.org/T124307#2989873 (10Marostegui) LOAD DATA is a lot faster to bulk lots of data in the DB, there is a lot less overhead in parsing SQL statements and all the processes around that parsing. This is... [15:18:53] 10DBA, 06Operations, 13Patch-For-Review: Reimage and clone db1072 - https://phabricator.wikimedia.org/T156226#2989913 (10Cmjohnson) Requested a new BBU from Dell..will update task with shipping information. Confirmed: Request 943118173 was successfully submitted. [15:22:16] 10DBA, 10Monitoring, 06Operations: Create a check/calendar alert for MariaDB TLS certs - https://phabricator.wikimedia.org/T152427#2989923 (10jcrespo) MySQL, when compiled with openssl support, provides very easy way to check the time: ``` | Ssl_server_not_after | Jun 29 21:52:32 2020 GMT | Ssl_se... [15:26:13] marostegui: jynus no hurry on this or anything, only wondering what your thoughts on on teh remaining shards on new labsdb's. Somewhat out of curiosity for cutting a few things over seemlessly (tm!) and somewhat as I have had a few ppl ask [15:26:37] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989959 (10Marostegui) Hey @epriestley - thanks for the fast response! >>! In T156905#2989711, @epriestley wrote: > I'm not sure why the query is slow or requi... [15:27:02] chasemp: look at this: https://phabricator.wikimedia.org/T153743#2989039 :-) [15:28:01] marostegui: ah thanks, subscribed :) that task will be my response to folks from here on out [15:37:21] for some reason dbstore1001 is getting more and more frequently "CHECK_NRPE: Socket timeout after 10 seconds." [15:41:02] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989990 (10Marostegui) >>! In T156905#2989959, @Marostegui wrote: > > ``` > | 50776 | root | localhost | phabricator_search | Query | 605 |... [15:42:41] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2989991 (10epriestley) Hrrm. For comparison, a similar inner query on `secure.phabricator.com` (searching for "phabricator" instead of "affecting translatewiki.ne... [15:45:35] mariadb is horrible for EXPLAIN subqueries because they are executed [15:45:57] :( [15:45:58] 07Blocked-on-schema-change, 06Collaboration-Team-Triage, 10Notifications, 13Patch-For-Review, 07Schema-change: Add primary key to echo_notification table - https://phabricator.wikimedia.org/T136428#2989993 (10Catrope) @Marostegui Yes, they all need to be altered. I believe the aawik* ones and the kowiki... [15:48:39] 10DBA, 10Analytics, 06Operations: Improve eventlogging replication procedure - https://phabricator.wikimedia.org/T124307#1952524 (10Nuria) @ottomata: we do not delete data from eventlogging (other than the purging that it should happen after 90 days) the system just inserts batches of records. [15:52:00] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990013 (10Paladox) What if we dropped the phabricator_search table then re created and then do the reindexing for mysql, would that work? [15:53:48] 10DBA, 10Analytics, 06Operations: Improve eventlogging replication procedure - https://phabricator.wikimedia.org/T124307#2990018 (10jcrespo) > purging that it should happen after 90 days How do you implement purging? That surely must run deletes or some kind of updates? [15:56:34] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990019 (10Marostegui) >>! In T156905#2989991, @epriestley wrote: > Hrrm. For comparison, a similar inner query on `secure.phabricator.com` (searching for "phabri... [15:58:55] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990020 (10epriestley) Here's another possible formulation of the query based on Googling "FULLTEXT initialization", although I haven't yet found a real descripti... [16:02:25] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990021 (10jcrespo) Maybe tuning `innodb_ft_cache_size` and `innodb_ft_total_cache_size` is needed. Little to no tuning was done after converting the table from M... [16:03:58] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990030 (10epriestley) Ah! That seems pretty broken -- the same simple query takes `0.03s` on `secure.phabricator.com`, so yours is ~22,000x slower for ~10x more... [16:06:26] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990034 (10jcrespo) > Is that something you're comfortable trying? Yes, that makes lot of sense, too. [16:07:39] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990035 (10epriestley) Tuning `innodb_ft_*` parameters may also be fruitful, but I don't have any direct experience with it to provide guidance. [16:08:30] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990036 (10Marostegui) >>! In T156905#2990030, @epriestley wrote: > Ah! That seems pretty broken -- the same simple query takes `0.03s` on `secure.phabricator.com... [16:24:54] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990100 (10epriestley) If we can't get the trivial `SELECT * WHERE MATCH(...)` case performing quickly, I believe the engine isn't going to be usable no matter ho... [16:42:52] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990129 (10Paladox) We have elastic search as an experimental option. You can enable it through the pref by going to Developer Settings and under the elastic sear... [16:43:41] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990131 (10Marostegui) Good news: ``` root@MISC m3[phabricator_search]> optimize table search_documentfield; Stage: 1 of 2 'copy to tmp table' 0.023% of stage d... [16:48:19] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990173 (10Marostegui) And the big query finished in `5 rows in set (1 min 59.41 sec)`: ``` root@MISC m3[phabricator_search]> SELECT documentPHID, MAX(fieldScore... [16:49:08] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990174 (10Paladox) but our only problem with elasticsearch is you can't reindex both indexes. So elasticsearch index is out of date. we need someway to be able... [16:49:14] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990175 (10Nemo_bis) >>! In T156905#2989343, @Volans wrote: > Looks like there was some heavy load on the server in the ~20 minutes before the OOM: > > https://g... [16:49:33] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990176 (10jcrespo) Let's still try to tune the innodb parameters. 2 minutes is also too much for a simple search. [16:51:57] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990178 (10Paladox) >>! In T156905#2990175, @Nemo_bis wrote: >>>! In T156905#2989343, @Volans wrote: >> Looks like there was some heavy load on the server in the... [16:54:52] db2037 is taking 15 minutes to flush its buffer to disk [16:54:56] and still ongoing [17:04:26] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990225 (10Marostegui) >>! In T156905#2990176, @jcrespo wrote: > Let's still try to tune the innodb parameters. 2 minutes is also too much for a simple search. A... [17:10:48] 10DBA, 06Analytics-Kanban, 13Patch-For-Review: Set up auto-purging after 90 days {tick} - https://phabricator.wikimedia.org/T108850#2990284 (10Nuria) [17:11:21] 10DBA, 10Analytics, 13Patch-For-Review: Set up auto-purging after 90 days {tick} - https://phabricator.wikimedia.org/T108850#1532166 (10Nuria) [17:18:08] 10DBA, 10Analytics, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#2990353 (10Ottomata) [17:19:42] 10DBA, 10Analytics, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#2987618 (10Ottomata) @Marostegui, @jcrespo, we talked about this today. What is your timeline for replacing these boxes? We want to try to ween people off of EventLogging My... [17:22:22] 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Update page_assessments_projects schema for subprojects in production - https://phabricator.wikimedia.org/T156305#2990365 (10kaldari) [17:27:28] 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Update page_assessments_projects schema for subprojects in production - https://phabricator.wikimedia.org/T156305#2990391 (10kaldari) p:05Triage>03High Marking high priority since this is needed for T156856 and... [17:41:57] 10DBA, 10Analytics, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#2990444 (10jcrespo) > What is your timeline for replacing these boxes? The constraint, more than the decommission, is the budget for replacements. I do not know what is the d... [17:44:47] 10DBA, 10Analytics, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#2990448 (10jcrespo) > What is your timeline for replacing these boxes? BTW, I forgot to answer literally your question, the deadline for replacement is January 2014 (not a ty... [17:54:42] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990460 (10Paladox) >>! In T156905#2989959, @Marostegui wrote: > Hey @epriestley - thanks for the fast response! > > > >>>! In T156905#2989711, @epriestley wro... [18:08:18] 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Update page_assessments_projects schema for subprojects in production - https://phabricator.wikimedia.org/T156305#2990525 (10Niharika) Pinging @Marostegui in case he has time to look into this. [18:13:59] 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Update page_assessments_projects schema for subprojects in production - https://phabricator.wikimedia.org/T156305#2990531 (10jcrespo) @kaldari, when I ask when on the "form", is to speed up the implementation. I do... [19:27:48] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990866 (10Marostegui) This has happened again and crashed the master and the slave. I have run the ALTER table to optimize the table on the master while we were... [19:30:07] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990885 (10Paladox) p:05High>03Unbreak! Due to it taking down the db + forcing us to switch to elastic search. I am upping the priority to unbreak. [19:30:13] 10DBA, 06Labs, 10Tool-Labs: labsdb1001 and labsdb1003 short on available space - https://phabricator.wikimedia.org/T132431#2990891 (10Matthewrbowker) [19:30:17] 10DBA, 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Xtools: `s51187__xtools_tmp` database using 272G on labsdb1001 - https://phabricator.wikimedia.org/T133321#2990888 (10Matthewrbowker) 05Open>03Resolved I am going to call this task resolved as of now. There are currently 95 tables, and only 6 are older tha... [19:31:06] 10DBA, 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Xtools: `s51187__xtools_tmp` database using 272G on labsdb1001 - https://phabricator.wikimedia.org/T133321#2990893 (10jcrespo) Thank you, again! [19:32:43] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2990912 (10greg) p:05Unbreak!>03High Issue has been mitigated (we're using the ES backend for everyone now), lowering priority but keeping open for any follow... [20:38:05] 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Update page_assessments_projects schema for subprojects in production - https://phabricator.wikimedia.org/T156305#2991205 (10jcrespo) a:05jcrespo>03kaldari [20:40:38] 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Update page_assessments_projects schema for subprojects in production - https://phabricator.wikimedia.org/T156305#2991235 (10kaldari) [20:42:21] 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Update page_assessments_projects schema for subprojects in production - https://phabricator.wikimedia.org/T156305#2991250 (10jcrespo) a:05kaldari>03jcrespo Thanks, doing now. [20:43:58] 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Update page_assessments_projects schema for subprojects in production - https://phabricator.wikimedia.org/T156305#2991257 (10jcrespo) For the record, I am not trying to be an ass (which I may be in any case), but w... [20:44:07] 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Update page_assessments_projects schema for subprojects in production - https://phabricator.wikimedia.org/T156305#2991259 (10kaldari) @jcrespo: Thanks for the info. I've updated the description to explicitly list t... [22:21:23] 07Blocked-on-schema-change, 10DBA, 06Community-Tech, 10MediaWiki-extensions-PageAssessments: Update page_assessments_projects schema for subprojects in production - https://phabricator.wikimedia.org/T156305#2991552 (10jcrespo) a:05jcrespo>03kaldari Can you verify testwiki on all hosts? This was done in... [23:48:02] 10DBA, 06Operations, 10Phabricator, 06Release-Engineering-Team: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2991826 (10jcrespo) removing m3 from dbstore2001: db1043-bin.001457:753455796 [23:50:22] 10DBA, 06Operations, 13Patch-For-Review: Set up TLS for MariaDB replication - https://phabricator.wikimedia.org/T111654#2991827 (10jcrespo) ``` sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'Puppet' 130 ```