[05:04:03] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1067 - https://phabricator.wikimedia.org/T206500 (10Marostegui) p:05Triage>03High a:03Cmjohnson @Cmjohnson can we replace this as soon as possible? This is enwiki primary master
[05:08:02] <marostegui>	 banyek|away: so what happened with the semisync check in the end?
[07:30:23] <banyek>	 good morning
[07:30:43] <banyek>	 What happened: I'll starting with that now
[07:30:50] <marostegui>	 great
[07:31:21] <marostegui>	 Let's also try to get the schema change plan first draft ready during the week, so we can start discussing it too 
[07:31:40] <banyek>	 ok
[07:32:04] <marostegui>	 as I expect it to take long until we can actually start deploying it ;)
[07:43:28] <banyek>	 I am checking this semi-sync replication thing, and I have to ask was that set up in the past at all?
[07:43:35] <banyek>	 I mean a lot's a servers missing it
[07:44:00] <jynus>	 I mean a lot's a servers missing it- that is on purpose- see puppet
[07:44:14] <banyek>	 ok, cool
[07:44:34] <banyek>	 I am just trying to check it with the 'slaves' on tendril
[07:44:46] <banyek>	 I have to read (later) about this
[07:45:07] <banyek>	 I am using this: `SHOW VARIABLES LIKE 'rpl_semi_sync_%_enabled`
[07:45:34] <banyek>	 and the funny part that there are servers where the result is nothing - that's ok those servers are old like dbstore1002
[07:45:44] <banyek>	 the most of the servers have both (master|slave)
[07:45:48] <banyek>	 which is also ok
[07:46:07] <banyek>	 but there are servers where only the one of those variables exist
[07:46:33] <banyek>	 ```db1124
[07:46:33] <banyek>	 rpl_semi_sync_master_enabled	OFF
[07:46:33] <banyek>	 db1119
[07:46:33] <banyek>	 rpl_semi_sync_slave_enabled	ON
[07:46:33] <banyek>	 db1118
[07:46:33] <banyek>	 rpl_semi_sync_master_enabled	OFF
[07:46:33] <banyek>	 rpl_semi_sync_slave_enabled	ON```
[07:46:39] <banyek>	 like those ^ 
[07:47:40] <jynus>	 you are comparing apples and oranges
[07:47:47] <jynus>	 db1118 is not a core host
[07:48:07] <marostegui>	 and db1124 is sanitarium
[07:48:17] <jynus>	 so neither
[07:48:57] <marostegui>	 you probably want to us cumin aliases to narrow things down
[07:49:12] <banyek>	 yeah, probably
[07:49:32] <banyek>	 but now my problem was just this why the output there wasn't like
[07:49:38] <banyek>	 ```db1124
[07:49:38] <banyek>	 rpl_semi_sync_master_enabled	OFF
[07:49:38] <banyek>	 rpl_semi_sync_slave_enabled.    OFF
[07:49:38] <banyek>	 db1119
[07:49:38] <banyek>	 rpl_semi_sync_master_enabled	OFF
[07:49:38] <banyek>	 rpl_semi_sync_slave_enabled		ON
[07:49:38] <banyek>	 db1118
[07:49:39] <banyek>	 rpl_semi_sync_master_enabled	OFF
[07:49:39] <banyek>	 rpl_semi_sync_slave_enabled		ON
[07:49:40] <banyek>	 db1114
[07:49:40] <banyek>	 rpl_semi_sync_master_enabled	OFF
[07:49:41] <banyek>	 rpl_semi_sync_slave_enabled		ON
[07:50:05] <marostegui>	 check the default values for those roles in puppet
[07:50:11] <banyek>	 (actually I am playing with the ./software/dbtools/section 's output
[07:52:59] <marostegui>	 yeah, but that includes all the hosts in a section, doesn't matter if they are sanitariums or labs or tests host
[07:54:40] <banyek>	 ```or i in 080 083 089 106 114 119; do /usr/local/sbin/mysql.py -BN -h db1$i -e "SHOW VARIABLES LIKE 'rpl_semi_sync_slave_enabled'"; done```
[07:54:46] <banyek>	 *for
[07:54:49] <banyek>	 that works
[07:56:40] <jynus>	 "that includes all the hosts in a section" more like, that includes a full replica set, independently of the role
[07:56:50] <jynus>	 it was done thinking on schema changes
[07:57:09] <jynus>	 I can add "groups" of servers if that is helpful
[07:57:21] <banyek>	 it could be
[07:57:46] <jynus>	 for example, s1 will include labsdb1009/10/11
[07:57:53] <marostegui>	 that is exactly what I said ;)
[07:58:08] <jynus>	 which should have a different configuration
[07:58:14] <marostegui>	 All hosts == all replicas
[07:59:02] <jynus>	 so s1 is a name of a core section, but also the replica set of all servers replicating enwiki, even those outside of core
[08:00:03] <marostegui>	 We are talking about the same thing anyways, it is a wording thing
[08:09:14] <banyek>	 I am done from s1 to s8 all semi-sync slaves are good there
[08:11:11] <marostegui>	 great!
[08:11:47] <marostegui>	 changing subject, what's the plan with the logrotate for pt-kill?
[08:13:27] <banyek>	 1, I'll 'ack' the string 'logrotate' in the puppet repo
[08:13:44] <banyek>	 2, I'll 'find . | grep logrotate' in the puppet repo
[08:14:02] <banyek>	 probably we have custom set up logrotate
[08:14:35] <banyek>	 then I'll take my vm to check if a logrotate works with wm-pt-kill (reloads the daemon, things like that)
[08:15:27] <banyek>	 then I start implementing it based on a prevoiusly created logrotate config (maybe that will be the moment when I'll check if it is not better if I package the logrotate.d/wmf-pt-kill script next to the debian package
[08:15:32] <banyek>	 3, ??????
[08:15:34] <banyek>	 4, profit
[08:15:58] <banyek>	 something like that marostegui: ^
[08:16:03] <jynus>	 create a ticket for that if you start workin on that
[08:16:10] <marostegui>	 yeah
[08:16:12] <banyek>	 ok I'll do it
[08:16:15] <jynus>	 so you can track work and share things
[08:16:16] <marostegui>	 and close the other one
[08:16:28] <banyek>	 that's why I didn't closed the old one yet - not until the new ticket is ready
[08:16:35] <jynus>	 what is the status of pt-kill puppetization
[08:16:47] <jynus>	 is it does, except for polish like that?
[08:16:50] <jynus>	 *done
[08:16:59] <banyek>	 yes
[08:17:08] <jynus>	 cool, thank you
[08:17:27] <banyek>	 it was really fun to build it actually, so np :)
[08:17:43] <jynus>	 we may do at a latter time a productization for other servers
[08:17:46] <jynus>	 but nothing planned
[08:19:53] <banyek>	 what's the supposed outcome of the non-
[08:20:11] <banyek>	 non-'s' sections with the semi-sync? 
[08:21:00] <marostegui>	 banyek: if you will work on the background on the pt-kill, can you coordinate with papaul for: https://phabricator.wikimedia.org/T205257#4637549 ? We'd need to new if that BBU is usable or what's the deal with it
[08:21:19] <marostegui>	 banyek: Take a look and let me know if you have doubts about the plan
[08:21:24] <jynus>	 banyek: check puppet- as far as I know, no other servers other than core use semisync
[08:32:51] <wikibugs>	 10DBA, 10User-Banyek: BBU problems dbstore2002 - https://phabricator.wikimedia.org/T205257 (10Banyek) a:03Banyek @Papaul or you can coordinate with me, I'll be here all week
[08:33:55] <marostegui>	 banyek: you got what I wanted to do?
[09:02:18] <banyek>	 marostegui: to put the spare BBU in to the db2042 and see what happens, will it work, or will not
[09:02:38] <marostegui>	 yep, that's it
[09:02:46] <marostegui>	 have you checked the role for db2042?
[09:02:47] <banyek>	 I spent the last 30 minutes on the UPS site, it seems I won't get my work computer today
[09:03:00] <banyek>	 the address is wrong, but I don't have the InfoNotice to change it
[09:03:19] <banyek>	 so we wait until the globe turns a bit
[09:03:20] <banyek>	 :)
[09:03:22] <marostegui>	 I guess they will leave it on a UPS picking point?
[09:05:03] <banyek>	 i a not sure if there's any
[09:05:17] <banyek>	 *am
[09:06:13] <banyek>	 I registered, and if I get that code I can rewrite the address so no worries - until I don't have some adapter for the monitor I can't use it anyway
[09:06:26] <marostegui>	 yeah
[09:06:27] <marostegui>	 too bad
[09:07:45] <marostegui>	 so, be aware of db2042's role
[09:17:56] <banyek>	 I don't see it in the config anywere
[09:18:00] <banyek>	 *anywhere
[09:19:30] <marostegui>	 Do some research about it
[09:19:47] <marostegui>	 As you will be powering it down, you need to understand what are the implications and possible impact
[09:21:05] <banyek>	 the host itself says 'm3' (pahbricator)
[09:21:35] <banyek>	 and as I see db2078 replicates from it
[09:21:58] <banyek>	 right, it has an active replica
[09:22:31] <marostegui>	 is it active? can it be powered down any time?
[09:23:42] <banyek>	 hmm... reading puppet I'd say 'it can be powered down after failing back to eqiad, and after backups are complete'
[09:24:02] <marostegui>	 failing what back to eqiad?
[09:24:50] <marostegui>	 backups: correct, you need to wait for the backups, or run them manually if you power it down while they are supposed to run
[09:26:34] <banyek>	 I mean dc switchover
[09:26:44] <marostegui>	 is db2042 active?
[09:28:32] <banyek>	 well it acts as a master for db2078's m3 instance
[09:29:09] <marostegui>	 Yeah, but m3 has also stuff in eqiad, no?
[09:29:17] <marostegui>	 Which is the active master for m3?
[09:29:48] <banyek>	 but as I see db1117 has phabricator queries
[09:29:54] <banyek>	 yep
[09:30:12] <banyek>	 db1117 is the active one, so we only need to care about backups
[09:30:18] <banyek>	 what does db2078 then?
[09:30:30] <marostegui>	 db1117 is active? for what?
[09:32:37] <marostegui>	 how could you check which is the active master for a misc service?
[09:33:16] <banyek>	 I now checking the dbproxy hiera's and there it seems like a it's a master secondary
[09:33:32] <marostegui>	 so which is the active master for m3?
[09:34:05] <banyek>	 db1063 based on the proxy config, but I am not sure b/c it is not shows that way on tendril
[09:34:20] <marostegui>	 db1063? for m3?
[09:34:36] <marostegui>	 check again ;)
[09:36:47] <banyek>	 https://www.irccloud.com/pastebin/LU6odS5Z/
[09:36:56] <banyek>	 BUT I think the port number is the key
[09:37:07] <marostegui>	 why are you checking dbproxy1001?
[09:38:10] <marostegui>	 How did you end up checking dbproxy1001 is more the question
[09:38:22] <banyek>	 it was the first I checked from the output of 'ack' ing db1117 - but now I see that I have to find the correct port number for the instance of m3 on 1117 and check that proxy 
[09:38:25] <banyek>	 where it appears
[09:38:41] <marostegui>	 So, let's rewind a bit
[09:38:47] <banyek>	 stop
[09:38:50] <banyek>	 hammertime
[09:38:54] <banyek>	 give me a sec
[09:39:04] <marostegui>	 Do you know how to check which is the active master for a misc service?
[09:39:21] <banyek>	 I said wait a bit: I am on trail
[09:40:08] <marostegui>	 What I am trying to do us is understand how you are reasoning to help you achieve it by yourself instead of me telling you exactly how to do it
[09:40:38] <banyek>	 SO
[09:42:27] <banyek>	 after checking that the m3 instance of db1117 (/run/mysqld/mysqld.m3.sock) bound to port 3323 I checked the db1117:3323 in puppet. It seems that the dbproxy1003 and dbproxy1008 handles it, and their config says that the primary is 1072. I am pretty sure btw that the '3' in 'dbproxy1003' represents m3
[09:42:56] <marostegui>	 correct
[09:43:01] <marostegui>	 so why checking dbproxy1001?
[09:44:54] <banyek>	 b/c first I didn't remembered that the 'M' is behind proxies, and that was the first element in the list which came up for searching the string db1117 in puppet config - puppet is the source of truth, and if I don't know exactly what I am looking for, I have to check file-by-file first
[09:45:22] <marostegui>	 Do you know the CNAMEs m1-master, m2-master, m3-master etc?
[09:45:49] <marostegui>	 root@neodymium:~# host m3-master
[09:45:49] <marostegui>	 m3-master.eqiad.wmnet is an alias for dbproxy1003.eqiad.wmnet.
[09:46:11] <marostegui>	 So we are trying to determine if db2042 is the active master or which host is the active master
[09:46:14] <marostegui>	 for m3
[09:46:16] <marostegui>	 right?
[09:47:35] <banyek>	 now as you ask them I remember them, but it wasn't in my head
[09:47:54] <banyek>	 right
[09:47:57] <marostegui>	 What does tendril say for m3?
[09:48:01] <marostegui>	 Did you check the tree?
[09:49:57] <banyek>	 I checked but the tree is not shows the proxy part
[09:50:12] <banyek>	 or does?
[09:50:20] <marostegui>	 so, according to tendril which is the master for m3?
[09:50:47] <banyek>	 db1072
[09:50:55] <marostegui>	 does that match what the proxy says?
[09:51:12] <banyek>	 yes
[09:51:31] <marostegui>	 then db1072 is the master for m3?
[09:51:35] <banyek>	 but the secondary master is db1117:3323 according to proxy
[09:51:47] <marostegui>	 that's ok, no? as we are not touching it
[09:52:30] <banyek>	 that's ok in this case for us
[09:52:56] <marostegui>	 so, what's the only thing you need to be aware when deciding when to poweroff db2042 then?
[09:53:48] <banyek>	 does the backup running or not (and when it ran last time) I guess. But I am still not sure the role of db2078 then
[09:54:01] <banyek>	 which is replicating from db2042
[09:54:09] <banyek>	 for god knows why
[09:54:16] <banyek>	 (actually puppet probably knows)
[09:54:19] <marostegui>	 why do you think it replicates from db2042?
[09:55:33] <banyek>	 because on tendril it looks like
[09:55:46] <banyek>	 and I see that there's a host replicating from db2042
[09:56:01] <marostegui>	 yeah, but what's the purpose you think?
[09:56:07] <banyek>	 with server_id 180367445
[09:56:33] <marostegui>	 yes, db2078 replicates from db2042, but why do you think it does?
[09:56:35] <banyek>	 if I didn't know that the backups are running from here I'd guessing backups
[09:56:53] <marostegui>	 right, backups doesn't run there, so why do we have it?
[09:57:48] <banyek>	 some kind of archive, maybe?
[09:58:13] <marostegui>	 What if db2042 dies while you are doing the BBU operation?
[10:00:12] <banyek>	 setting up db2078's m3 instance as a replica of db1072.eqiad.wmnet and move the backup's there?
[10:00:32] <marostegui>	 yep, so why do we have db2078 there then? :)
[10:00:38] <banyek>	 as well as the db proxies
[10:00:50] <banyek>	 as a 'safety net' host
[10:00:56] <marostegui>	 There you go
[10:01:00] <banyek>	 I which I have a better word for it
[10:01:00] <marostegui>	 If eqiad dies completly
[10:01:11] <marostegui>	 You want to have redundancy in codfw, no? :-)
[10:01:13] <banyek>	 *wish
[10:01:17] <banyek>	 yep
[10:01:27] <marostegui>	 So that is the idea of having db2078 there as a slave
[10:01:31] <marostegui>	 To replicate what we have in eqiad
[10:02:05] <marostegui>	 Wasn't this long excercise better than just me telling you that db2042 is not active and it is only used for backups?
[10:02:26] <banyek>	 yea
[10:02:41] <banyek>	 it was
[10:02:53] <marostegui>	 So now, you need all the information you need to coordinate with papaul and to keep in mind what to do if db2042 never comes back
[10:03:18] <marostegui>	 And if you still have doubts, ask
[10:03:30] <banyek>	 ok
[10:04:04] <banyek>	 I guess I don't really have to worry about the binlog file/pos, because of the gtid
[10:04:17] <marostegui>	 Do you have this repo cloned? https://gerrit.wikimedia.org/r/#/admin/projects/operations/dns
[10:04:23] <marostegui>	 it can be useful to look for things too
[10:05:01] <banyek>	 now I have it
[10:05:09] <marostegui>	 If I were you, I would stop replication on db2042, grab binlog coordinates in case you need them for db2078
[10:05:36] <marostegui>	 Before shutting it down, I mean
[10:12:06] <banyek>	 👍
[10:36:35] <wikibugs>	 10DBA, 10User-Banyek: Solve logrotating on wmf-pt-kill - https://phabricator.wikimedia.org/T206521 (10Banyek)
[10:37:54] <wikibugs>	 10DBA, 10Data-Services, 10Goal, 10Patch-For-Review, 10cloud-services-team (FY2017-18): Migrate all users to new Wiki Replica cluster and decommission old hardware - https://phabricator.wikimedia.org/T142807 (10Banyek)
[10:37:57] <wikibugs>	 10DBA, 10Data-Services, 10Patch-For-Review, 10User-Banyek: Re-institute query killer for the analytics WikiReplica - https://phabricator.wikimedia.org/T183983 (10Banyek) 05Open>03Resolved
[10:39:50] <banyek>	 I am going to have some lunch
[10:42:24] <volans>	 quick question, is it ok to run the warmup caches script in eqiad right now or there is something going on that you prefer not to?
[10:43:00] <volans>	 marostegui, jynus ^^^
[10:43:01] <marostegui>	 volans: it is ok
[10:43:09] <volans>	 perfect, thanks!
[10:43:24] <marostegui>	 thank you!
[10:45:11] <jynus>	 we were waiting for you to actually do it!
[10:45:18] <volans>	 :)
[11:45:44] <wikibugs>	 10DBA, 10User-Banyek: Solve logrotating on wmf-pt-kill - https://phabricator.wikimedia.org/T206521 (10Banyek) p:05Triage>03Normal
[13:22:04] <jynus>	 I have completed the maintenance steps before and after switch on etherpad
[13:23:19] <marostegui>	 for the wikis move?
[13:24:12] <jynus>	 completed as in, filled in the steps and correcting them
[13:24:32] <jynus>	 lines 19-25
[13:24:45] <marostegui>	 checking
[13:24:48] <marostegui>	 banyek: ^
[13:26:27] <marostegui>	 jynus: shouldn't we do line 25 before 24?
[13:26:41] <jynus>	 sure, I added that
[13:26:54] <jynus>	 it wasn't before
[13:26:57] <marostegui>	 Ah sorry :)
[13:28:18] <marostegui>	 mmm line 22, that patch is no longer scheduled at least I cannot see it on deployments
[13:29:00] <marostegui>	 should we deploy that ourselves after the failover?
[13:29:25] <marostegui>	 or before, I don't mind
[13:36:26] <marostegui>	 jynus: do you want me to create a calendar event to grab db1070's position right before the switch so we don't forget?
[13:37:00] <jynus>	 no worries, we we'll see it on the binlogs
[13:37:37] <marostegui>	 yeah, I know, I just thought running show master status\G would just be easier than digging thru the binlogs
[14:25:26] <marostegui>	 Guys, I have copied our etherpad stuff to the SRE one
[14:25:38] <jynus>	 thanks
[14:44:03] <banyek>	 heads up: I finish the day today, and go for the kids. See you tomorrow, but I'll only show up between 10:30 and 11:00. (If the roof is on fire you can reach me.)
[14:45:33] <banyek>	 see you
[15:04:19] <wikibugs>	 10DBA, 10MediaWiki-Watchlist, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10Wikimedia-production-error: Deleting large watchlist takes > 4 seconds causing rollback due to write time limit - https://phabricator.wikimedia.org/T171898 (10kostajh) > That seems like a bug not a feature: a batched DELE...
[15:22:15] <wikibugs>	 10DBA: Research options for producing binary backups (lvm snapshots, cold backups, mariabackup) - https://phabricator.wikimedia.org/T206204 (10jcrespo) Testing mariabackup, got it compiler so far for buster: ``` # /opt/wmf-mariadb103/bin/mariabackup --help /opt/wmf-mariadb103/bin/mariabackup based on MariaDB ser...
[15:25:00] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: rack/setup/install db2096 (x1 codfw expansion host) - https://phabricator.wikimedia.org/T206191 (10Papaul)
[15:25:26] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: rack/setup/install db2096 (x1 codfw expansion host) - https://phabricator.wikimedia.org/T206191 (10Papaul)
[15:41:01] <wikibugs>	 10DBA: Research options for producing binary backups (lvm snapshots, cold backups, mariabackup) - https://phabricator.wikimedia.org/T206204 (10jcrespo) Basic test (no data) work as xtrabackup, but being linked with mariadb server, it doesn't crash:   ```lines=10 jynus@sangai:/srv/tmp$ sudo /opt/wmf-mariadb103/bi...
[15:45:47] <wikibugs>	 10DBA: Research options for producing binary backups (lvm snapshots, cold backups, mariabackup) - https://phabricator.wikimedia.org/T206204 (10jcrespo) Basic streaming works, too:   ```lines=10 root@sangai:/srv/tmp# /opt/wmf-mariadb103/bin/mariabackup --backup --user=root --stream=xbstream | pigz -c | pv > backu...
[16:17:51] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: rack/setup/install db2096 (x1 codfw expansion host) - https://phabricator.wikimedia.org/T206191 (10Papaul)
[16:29:54] <wikibugs>	 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install db2096 (x1 codfw expansion host) - https://phabricator.wikimedia.org/T206191 (10Papaul)
[16:39:39] <wikibugs>	 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install db2096 (x1 codfw expansion host) - https://phabricator.wikimedia.org/T206191 (10Papaul) Server is racked in B4  switch port information : asw-b4-codfw ge-4/0/0 IP address: 10.192.16.34
[16:45:22] <wikibugs>	 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install db2096 (x1 codfw expansion host) - https://phabricator.wikimedia.org/T206191 (10Papaul) ``` papaul@asw-b-codfw> show interfaces ge-4/0/0 descriptions  Interface       Admin Link Description ge-4/0/0        up    up   db2096 ``` ``` in...
[16:47:52] <wikibugs>	 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install db2096 (x1 codfw expansion host) - https://phabricator.wikimedia.org/T206191 (10Papaul)
[17:26:16] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1067 - https://phabricator.wikimedia.org/T206500 (10Cmjohnson) @Marostegui disk swapped
[17:32:14] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Banyek: db1092 crashed - BBU broken - https://phabricator.wikimedia.org/T205514 (10Cmjohnson) The battery was sent to our old office address in San Francisco, they are shipping a new battery...because it's a battery it has to go ground and will...
[17:38:50] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1073 - https://phabricator.wikimedia.org/T206254 (10Cmjohnson) it is a new disk...trying it again
[17:39:15] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1072 - https://phabricator.wikimedia.org/T206313 (10Cmjohnson) new disk...trying it again
[18:01:14] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1072 - https://phabricator.wikimedia.org/T206313 (10Marostegui)
[18:15:52] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1067 - https://phabricator.wikimedia.org/T206500 (10Marostegui) Failed: ``` PD: 1 Information Enclosure Device ID: 32 Slot Number: 7 Drive's position: DiskGroup: 0, Span: 1, Arm: 1 Enclosure position: 1 Device Id: 7 WWN: 5000C50070CACB6C Sequence Number: 2...
[18:30:00] <wikibugs>	 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install db2096 (x1 codfw expansion host) - https://phabricator.wikimedia.org/T206191 (10Papaul)
[22:28:14] <wikibugs>	 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install db2096 (x1 codfw expansion host) - https://phabricator.wikimedia.org/T206191 (10Papaul)
[22:33:00] <wikibugs>	 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install db2096 (x1 codfw expansion host) - https://phabricator.wikimedia.org/T206191 (10Papaul) a:05Papaul>03Marostegui @Marostegui All yours.
[22:41:45] <wikibugs>	 10DBA, 10JADE, 10Operations, 10TechCom-RFC, 10Scoring-platform-team (Current): Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) The fully implemented secondary schema is ready for #techcom review: https://gerrit.wikimedia.org...