[04:46:28] <wikibugs>	 10DBA: Compress enwiki InnoDB tables - https://phabricator.wikimedia.org/T254462 (10Marostegui)
[05:38:04] <wikibugs>	 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin2001.codfw.wmnet for hosts: ` ['db1118.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/20200622...
[05:46:19] <wikibugs>	 10DBA, 10Operations, 10decommission-hardware, 10ops-eqiad: decommission dbproxy1008.eqiad.wmnet - https://phabricator.wikimedia.org/T255406 (10Marostegui)
[05:55:34] <wikibugs>	 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1118.eqiad.wmnet'] `  and were **ALL** successful.
[06:11:57] <wikibugs>	 10DBA, 10Analytics: Upgrade analytics dbstore databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254870 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin2001.codfw.wmnet for hosts: ` ['dbstore1005.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto...
[06:30:34] <wikibugs>	 10DBA, 10Analytics: Upgrade analytics dbstore databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254870 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbstore1005.eqiad.wmnet'] `  and were **ALL** successful.
[07:01:00] <wikibugs>	 10DBA, 10Analytics, 10Patch-For-Review: Upgrade analytics dbstore databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254870 (10Marostegui)
[07:01:40] <wikibugs>	 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10Marostegui)
[07:01:40] <wikibugs>	 10DBA, 10Analytics, 10Patch-For-Review: Upgrade analytics dbstore databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254870 (10Marostegui) 05Open→03Resolved a:03Marostegui All done!
[07:19:23] <wikibugs>	 10DBA: Upgrade m1 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254556 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin2001.codfw.wmnet for hosts: ` ['db1117.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202006220719_marostegui_6426.log`.
[07:36:55] <wikibugs>	 10DBA: Upgrade m1 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254556 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1117.eqiad.wmnet'] `  and were **ALL** successful.
[07:43:11] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Degraded RAID on pc2007 - https://phabricator.wikimedia.org/T255904 (10Kormat) a:03Papaul
[07:43:39] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Degraded RAID on pc2007 - https://phabricator.wikimedia.org/T255904 (10Kormat) Hi @Papaul, can we get this disk replaced please? It should still be under warranty with dell.
[07:43:53] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Degraded RAID on pc2007 - https://phabricator.wikimedia.org/T255904 (10Kormat) idrac logs are here: https://phabricator.wikimedia.org/P11619
[07:47:56] <wikibugs>	 10DBA, 10Patch-For-Review: Upgrade m1 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254556 (10Marostegui)
[07:52:41] <wikibugs>	 10DBA, 10Patch-For-Review: Upgrade m1 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254556 (10Marostegui) @jcrespo @akosiaris @ayounsi  I would like to switchover the master to the new master that runs Buster and MariaDB 10.4.  m1 holds:  bacula librenms (which in previous switchovers has sho...
[07:55:01] <wikibugs>	 10DBA, 10Patch-For-Review: Upgrade m1 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254556 (10akosiaris) >>! In T254556#6243442, @Marostegui wrote:  > @jcrespo @akosiaris - I would like to do this maybe Thursday 25th at 08:00 AM UTC. Is that ok? Let me know if you prefer any other day/time  S...
[07:56:47] <wikibugs>	 10DBA, 10Patch-For-Review: Upgrade m1 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254556 (10Marostegui) Thank you, I have sent a calendar invite to you and to @jcrespo
[08:19:15] <wikibugs>	 10DBA, 10Patch-For-Review: Upgrade m1 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254556 (10jcrespo) ok.
[08:30:57] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: db1088 crashed - https://phabricator.wikimedia.org/T255927 (10Kormat)
[08:32:50] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: db1088 crashed - https://phabricator.wikimedia.org/T255927 (10Kormat) DCOps: The BBU on this machine has failed. Do you have a spare BBU in the DC, or if not, can we please order a replacement? Cheers.  ` /system1/log1/record7   Targets   Properties     number=7     severity...
[08:36:45] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: db1088 crashed - https://phabricator.wikimedia.org/T255927 (10Kormat) Mysql is started and catching up on replication. Once that's completed we'll perform a data consistency check.
[09:08:48] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Degraded RAID on pc2007 - https://phabricator.wikimedia.org/T255904 (10Marostegui) p:05Triage→03Medium
[09:43:28] <wikibugs>	 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin2001.codfw.wmnet for hosts: ` ['db1094.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/20200622...
[10:04:48] <wikibugs>	 10DBA, 10Epic: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1094.eqiad.wmnet'] `  and were **ALL** successful.
[10:19:56] <icinga-wm>	 PROBLEM - 5-minute average replication lag is over 2s on db1149 is CRITICAL: bad_data: parse error at char 91: unclosed left parenthesis https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1149&var-port=9104&var-dc=eqiad+prometheus/ops
[10:19:56] <icinga-wm>	 PROBLEM - 5-minute average replication lag is over 2s on db1106 is CRITICAL: bad_data: parse error at char 91: unclosed left parenthesis https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1106&var-port=9104&var-dc=eqiad+prometheus/ops
[10:19:56] <icinga-wm>	 PROBLEM - 5-minute average replication lag is over 2s on db1123 is CRITICAL: bad_data: parse error at char 91: unclosed left parenthesis https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1123&var-port=9104&var-dc=eqiad+prometheus/ops
[10:19:56] <icinga-wm>	 PROBLEM - 5-minute average replication lag is over 2s on db2076 is CRITICAL: bad_data: parse error at char 91: unclosed left parenthesis https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2076&var-port=9104&var-dc=codfw+prometheus/ops
[10:20:42] <marostegui>	 ^we are testing this
[10:23:48] <icinga-wm>	 PROBLEM - 5-minute average replication lag is over 2s on db1137 is CRITICAL: bad_data: parse error at char 91: unclosed left parenthesis https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1137&var-port=9104&var-dc=eqiad+prometheus/ops
[10:23:50] <icinga-wm>	 PROBLEM - 5-minute average replication lag is over 2s on db2111 is CRITICAL: bad_data: parse error at char 91: unclosed left parenthesis https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2111&var-port=9104&var-dc=codfw+prometheus/ops
[10:23:52] <icinga-wm>	 PROBLEM - 5-minute average replication lag is over 2s on es1014 is CRITICAL: bad_data: parse error at char 91: unclosed left parenthesis https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=es1014&var-port=9104&var-dc=eqiad+prometheus/ops
[10:23:54] <icinga-wm>	 PROBLEM - 5-minute average replication lag is over 2s on es2025 is CRITICAL: bad_data: parse error at char 91: unclosed left parenthesis https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=es2025&var-port=9104&var-dc=codfw+prometheus/ops
[10:24:08] <icinga-wm>	 PROBLEM - 5-minute average replication lag is over 2s on es2019 is CRITICAL: bad_data: parse error at char 91: unclosed left parenthesis https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=es2019&var-port=9104&var-dc=codfw+prometheus/ops
[10:26:13] <icinga-wm>	 PROBLEM - 5-minute average replication lag is over 2s on db1075 is CRITICAL: bad_data: parse error at char 91: unclosed left parenthesis https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1075&var-port=9104&var-dc=eqiad+prometheus/ops
[10:26:13] <icinga-wm>	 PROBLEM - 5-minute average replication lag is over 2s on db1082 is CRITICAL: bad_data: parse error at char 91: unclosed left parenthesis https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1082&var-port=9104&var-dc=eqiad+prometheus/ops
[10:26:13] <icinga-wm>	 PROBLEM - 5-minute average replication lag is over 2s on db1096 is CRITICAL: bad_data: parse error at char 92: unclosed left parenthesis https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1096&var-port=13316&var-dc=eqiad+prometheus/ops
[10:26:13] <icinga-wm>	 PROBLEM - 5-minute average replication lag is over 2s on db1101 is CRITICAL: bad_data: parse error at char 92: unclosed left parenthesis https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1101&var-port=13318&var-dc=eqiad+prometheus/ops
[10:26:13] <icinga-wm>	 PROBLEM - 5-minute average replication lag is over 2s on db1124 is CRITICAL: bad_data: parse error at char 92: unclosed left parenthesis https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1124&var-port=13318&var-dc=eqiad+prometheus/ops
[10:26:23] <icinga-wm>	 PROBLEM - 5-minute average replication lag is over 2s on db2107 is CRITICAL: bad_data: parse error at char 91: unclosed left parenthesis https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2107&var-port=9104&var-dc=codfw+prometheus/ops
[10:26:30] <icinga-wm>	 PROBLEM - 5-minute average replication lag is over 2s on db1093 is CRITICAL: bad_data: parse error at char 91: unclosed left parenthesis https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1093&var-port=9104&var-dc=eqiad+prometheus/ops
[10:26:30] <icinga-wm>	 PROBLEM - 5-minute average replication lag is over 2s on db1143 is CRITICAL: bad_data: parse error at char 91: unclosed left parenthesis https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1143&var-port=9104&var-dc=eqiad+prometheus/ops
[10:26:30] <icinga-wm>	 PROBLEM - 5-minute average replication lag is over 2s on db2120 is CRITICAL: bad_data: parse error at char 91: unclosed left parenthesis https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2120&var-port=9104&var-dc=codfw+prometheus/ops
[10:26:32] <icinga-wm>	 PROBLEM - 5-minute average replication lag is over 2s on es1016 is CRITICAL: bad_data: parse error at char 91: unclosed left parenthesis https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=es1016&var-port=9104&var-dc=eqiad+prometheus/ops
[11:03:08] <kormat>	 and this concludes our channel bandwidth test for the week.
[11:04:29] <volans>	 lol
[11:05:24] <paravoid>	 :D
[12:09:48] <wikibugs>	 10DBA, 10Cloud-Services, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) I deployed this schema change o...
[13:29:21] <wikibugs>	 10DBA: Use logging package instead of print statements in transferpy package - https://phabricator.wikimedia.org/T255999 (10Privacybatm)
[14:05:52] <jynus>	 remember when I said that core_multiinstance were used on backup sources?
[14:05:58] <jynus>	 it is not true
[14:06:08] <jynus>	 we use dbstore_multiinstance for those already
[14:06:52] <jynus>	 kormat: were you able to spot the parsing error in the end?
[14:07:08] <kormat>	 yeah. a simple missing `)`
[14:07:11] <jynus>	 cool
[14:07:23] <jynus>	 so dbstore and core I think no longer have a meaning
[14:07:32] <jynus>	 they are only there for historical naming
[14:08:17] <jynus>	 I won't change them, but something like mariadb::mw and mariadb::mw::backup_source may be better?
[14:08:28] <jynus>	 after we refactor?
[14:08:53] <kormat>	 ah hah
[14:08:59] <kormat>	 yeah makes sense
[14:09:13] <jynus>	 not sure, feel free to provide better suggestions
[14:09:48] <jynus>	 should also reconsider not having core and core_multiinstance?
[14:10:05] <jynus>	 everything is multiinstance, but 1 instance is allowed?
[14:10:38] <kormat>	 the port handling gets a bit weird i think
[14:10:45] <jynus>	 yeah
[14:11:03] <jynus>	 but if instances were not handled on puppet...
[14:11:16] <jynus>	 but we were closer to a mysql as a service model...
[14:11:34] <jynus>	 one can dream
[14:17:49] <jynus>	 one question, did x1 got upgraded fully to 10.4?
[14:18:16] <marostegui>	 no
[14:18:17] <wikibugs>	 10DBA, 10Patch-For-Review: Use logging package instead of print statements in transferpy package - https://phabricator.wikimedia.org/T255999 (10Privacybatm) I don't think we will be able to incorporate data transfer progress information. The RemoteExecution works in such a way that it will send the full data a...
[14:18:22] <marostegui>	 waiting on what to do with the backupsources
[14:18:30] <jynus>	 I was about to ask about taht
[14:18:49] <marostegui>	 https://phabricator.wikimedia.org/T254871#6209872
[14:18:55] <jynus>	 should we move things around? upgrade already in bulk?
[14:19:14] <jynus>	 as usual, we have questions but not answer :-DDD
[14:19:41] <jynus>	 let me see because I think we have some duplication
[14:19:45] <jynus>	 that was about to answer that
[14:19:50] <marostegui>	 my suggestion was to upgrade many hosts in sections where db1095 lives
[14:19:53] <marostegui>	 so it is worth upgrading it
[14:19:57] <marostegui>	 (and the codfw one)
[14:21:16] <jynus>	 here is the status
[14:21:38] <jynus>	 we have db1139 with 10.1 and s1 and s6
[14:22:01] <jynus>	 and db1140 with 10.4 and s1 and s6
[14:22:07] <jynus>	 both are working fine
[14:22:18] <jynus>	 I also requested one more host per dc
[14:22:41] <jynus>	 we can play with that
[14:22:57] <jynus>	 the issue is the --prepare, which has to happen on a version higer or equal
[14:23:15] <marostegui>	 jynus: keep in mind that you'd still have this: https://phabricator.wikimedia.org/T252512#6210863
[14:23:16] <jynus>	 although we have backup[12]002
[14:23:27] <marostegui>	 so that's another host we have to play with for a few days
[14:23:34] <marostegui>	 (as db1102 will be relocated somewhere else)
[14:23:39] <jynus>	 so that is some extra room
[14:23:49] <marostegui>	 and I still owe you a host for https://phabricator.wikimedia.org/T253217
[14:23:57] <jynus>	 I didn't see that comment, I may have been away
[14:24:40] <jynus>	 should we wait until the extra dbprov and source hosts are bought or should we work with what we have?
[14:24:40] <marostegui>	 no worries at all
[14:24:44] <jynus>	 can we wait?
[14:25:17] <marostegui>	 so db1102 needs to be replaced by that host, that host will be a backup source anyways
[14:25:17] <jynus>	 should we wait until more upgrades happen overal?
[14:25:21] <marostegui>	 so you can provision it anytime
[14:25:24] <jynus>	 yes
[14:25:39] <jynus>	 but that would only give us 4 extra sections duplicated
[14:25:45] <jynus>	 not 9
[14:25:51] <marostegui>	 You lost me
[14:26:02] <jynus>	 we have s1 and s6 on both 10.4 and 10.1
[14:26:07] <jynus>	 we can have 2 extra ones
[14:26:16] <jynus>	 e.g. x1 + other
[14:26:27] <marostegui>	 what does db1095 holds at the moment?
[14:26:30] <marostegui>	 x1 and?
[14:26:42] <jynus>	 s2 and s3
[14:26:57] <marostegui>	 right, so x1 is fully upgraded (apart from the masters)
[14:27:04] <jynus>	 oh
[14:27:07] <jynus>	 including source?
[14:27:09] <marostegui>	 should I upgrade more hosts in s2 and s3?
[14:27:13] <marostegui>	 no no, backup sources not
[14:27:15] <jynus>	 ah
[14:27:29] <jynus>	 so I can move s2 and s3 to the new host
[14:27:37] <jynus>	 and got for full x1 on 10.4
[14:27:37] <marostegui>	 the question I asked earlier was: is it worth upgrading db1095 already given that lots of s2, s3 and x1 hosts are running 10.4 already?
[14:27:52] <marostegui>	 or should I keep upgrading till it is worth upgrading dbprov hosts too?
[14:28:06] <jynus>	 and do a master failover (x1)?
[14:28:10] <marostegui>	 yep
[14:28:17] <jynus>	 or do you plan to do another first?
[14:28:27] <marostegui>	 I am going to do m1, es5 and I wanted to do x1 too
[14:28:37] <jynus>	 > "should I keep upgrading till it is worth upgrading dbprov hosts too?"---> don't know
[14:28:46] <jynus>	 we can play with backup1002 for some time
[14:28:54] <jynus>	 but not for a long time
[14:28:58] <marostegui>	 right now the following roles cannot be upgraded: primary masters, sanitarium masters, sanitarium and labs
[14:29:03] <marostegui>	 as we are blocked on labs running 10.1
[14:29:10] <jynus>	 ah, that is another blocker
[14:29:18] <marostegui>	 and of course candidate masters cannot be upgraded
[14:29:44] <jynus>	 I think labs should be "solved" first
[14:29:57] <jynus>	 I think it makes no sense to discuss sources and others until that is fixed
[14:30:12] <jynus>	 well, x1
[14:30:15] <marostegui>	 but not upgrading the sources makes impossible to upgrade x1
[14:30:16] <jynus>	 and misc
[14:30:26] <jynus>	 but that is easy to solve
[14:30:40] <jynus>	 I move s2 and s3 elsewhere
[14:30:44] <marostegui>	 m1 is almost fully upgraded (only pending the master)
[14:30:49] <jynus>	 and backup to backup1002
[14:30:54] <jynus>	 then we do a full upgrade
[14:31:11] <jynus>	 we can do that now
[14:31:17] <marostegui>	 db1117 is also upgraded
[14:32:01] <jynus>	 so let's do that
[14:32:04] <jynus>	 upgrade m1
[14:32:08] <jynus>	 then x1
[14:32:14] <jynus>	 then blocked on labsdb
[14:32:19] <jynus>	 for the most part
[14:32:25] <marostegui>	 no, upgade m1, then es5 and then x1
[14:32:29] <jynus>	 ok
[14:32:35] <marostegui>	 so you have more days
[14:32:39] <jynus>	 anything except x*
[14:32:41] <marostegui>	 to play around with moving the instances and all that
[14:32:42] <jynus>	 s*
[14:33:00] <jynus>	 we haven't tested swichover on 10.4 on production
[14:33:17] <marostegui>	 I tested it with the testing hosts
[14:33:24] <jynus>	 and no issue?
[14:33:35] <marostegui>	 nope
[14:33:49] <jynus>	 ok, give me that extra host and I will prepare x1
[14:34:00] <marostegui>	 sure, that host is waiting for you
[14:34:02] <jynus>	 then we will rethink the others when we have more hw
[14:34:11] <jynus>	 I had totally missed that
[14:34:21] <marostegui>	 keep in mind that db1145 does need to go at least to s4
[14:34:26] <marostegui>	 as it was purchased for SDC
[14:34:38] <marostegui>	 so ideally it needs to replace whatever db1102 has
[14:34:50] <jynus>	 now I got lost
[14:35:01] <jynus>	 I though the new one were for s4
[14:35:05] <marostegui>	 right, so db1145 is a new host bought for SDC
[14:35:07] <jynus>	 not the old ones
[14:35:09] <jynus>	 oh
[14:35:17] <jynus>	 what about the ones being replaced?
[14:35:31] <marostegui>	 db1102 is the one being replaced by db1145
[14:35:31] <jynus>	 can I get one instead?
[14:35:42] <marostegui>	 yes, db1102 is the one we have to give you back too
[14:35:45] <marostegui>	 so you can keep it
[14:35:57] <jynus>	 ok
[14:36:08] <marostegui>	 let me update this task: https://phabricator.wikimedia.org/T253217
[14:36:09] <jynus>	 so that is the same, we just move the asignment
[14:36:14] <jynus>	 of the sections
[14:36:16] <jynus>	 the usage is the same
[14:36:17] <marostegui>	 yes, but db1102 has smaller disks
[14:36:25] <marostegui>	 that is why I was saying that db1145 must go to s4
[14:36:30] <marostegui>	 (and wherever else)
[14:36:34] <jynus>	 ok, gotcha
[14:36:36] <jynus>	 we can do that
[14:36:47] <wikibugs>	 10DBA: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10Marostegui)
[14:36:49] <jynus>	 db1102 s4 -> db1145
[14:37:21] <marostegui>	 so the easier thing would be to mimic db1145 with whatever db1102 has and then use db1102 for whatever you want
[14:37:37] <jynus>	 yes
[14:38:27] <jynus>	 then I upgrade db1102
[14:38:34] <jynus>	 and put x1 there
[14:38:53] <jynus>	 and backup snapshots of it to backup1002
[14:39:00] <jynus>	 what about codfw?
[14:39:17] <jynus>	 we don't touch it yet?
[14:39:20] <marostegui>	 same problem :)
[14:39:40] <marostegui>	 x1 codfw is all done except the backup source
[14:39:43] <jynus>	 but we will have a host?
[14:40:07] <marostegui>	 an extra one? no, we don't
[14:40:20] <marostegui>	 we didn't have to buy as many there as those were kinda new and had bigger disks
[14:40:50] <marostegui>	 if you need one, I can probably give you one from https://phabricator.wikimedia.org/T253217
[14:41:19] <marostegui>	 but I rather not and maybe use db2102 for that
[14:41:23] <marostegui>	 (core test)
[14:41:57] <marostegui>	 or that was the backup testing host?
[14:42:39] <jynus>	 db2102 is the codfw backup testing host
[14:43:19] <jynus>	 db2101 MariaDB Replica Lag: x1
[14:43:24] <jynus>	 but x1 is already on its own
[14:43:29] <jynus>	 the backup source
[14:43:30] <marostegui>	 ah true
[14:43:31] <marostegui>	 yes
[14:43:36] <jynus>	 we we can just upgrade it
[14:43:39] <jynus>	 *so
[14:43:48] <marostegui>	 excellent 
[14:43:56] <marostegui>	 you take care of it?
[14:43:58] <jynus>	 the only palace we are blocked
[14:44:01] <marostegui>	 or you want me to?
[14:44:04] <jynus>	 I can do
[14:44:08] <marostegui>	 <3
[14:44:10] <jynus>	 is space for snapshots
[14:44:24] <jynus>	 we have backup1002 and backup2002
[14:44:35] <jynus>	 but we really would need those extra dbprovs
[14:44:43] <jynus>	 it is not ok long term
[14:45:07] <marostegui>	 for which Q of next FY have you scheduled them
[14:45:08] <marostegui>	 ?
[14:45:14] <jynus>	 Q1
[14:45:24] <marostegui>	 sweet
[14:45:35] <jynus>	 they would be on 10.4 from the start
[14:45:47] <jynus>	 and we would have 1 extra server room we would need
[14:45:57] <marostegui>	 good good
[14:46:13] <jynus>	 could we land by then the labs db unblocking, you think?
[14:46:18] <jynus>	 I know doesn't depend on you...
[14:46:33] <marostegui>	 in q1?
[14:46:36] <jynus>	 yes
[14:46:41] <marostegui>	 I don't think so
[14:46:58] <marostegui>	 I mean, from our side, setting up the multi-instance hosts might take a Q
[14:47:03] <jynus>	 so we would be blocked by that, but at least we would not use backup1002
[14:47:08] <marostegui>	 but from wmcs side it might take a little while
[14:47:31] <jynus>	 but that would put us beyond Q2 for a full upgrade
[14:47:39] <jynus>	 well, more than that
[14:47:46] <marostegui>	 Yep
[14:47:49] <jynus>	 to start working on it on Q2
[14:48:04] <marostegui>	 I would be surprised if in Q1 we have deprecated the old labsdb hosts
[14:48:26] <marostegui>	 Again, from our side I think it is "fast"
[14:48:36] <jynus>	 should we test 10.4 -> 10.1 replication?
[14:48:44] <jynus>	 outside of labsdb?
[14:49:01] <jynus>	 because I predict that to be a huge pain
[14:49:06] <marostegui>	 we could, but I would still feel a bit scared 
[14:49:18] <jynus>	 I mean 
[14:49:24] <marostegui>	 especially cause we have replication filters and triggers involved, and if they fail that's a big thing
[14:49:27] <jynus>	 once the alernative is working
[14:49:33] <marostegui>	 ah
[14:49:40] <marostegui>	 once the "new" labsdb are up?
[14:49:55] <jynus>	 because otherwise we would be blocked on that forever
[14:50:25] <marostegui>	 yes, this blocks all the final upgrades
[14:50:28] <jynus>	 e.g. do you see the current servers going away in less than a year?
[14:50:30] <marostegui>	 and it has been communicated 
[14:50:36] <marostegui>	 jynus: Yes
[14:50:40] <jynus>	 not replacing them
[14:50:44] <jynus>	 I mean removing the current ones
[14:50:46] <marostegui>	 We are not replacing them
[14:50:54] <marostegui>	 yeah, I hope so
[14:50:58] <jynus>	 buf
[14:51:03] <jynus>	 should we bet on it :-D
[14:51:07] <jynus>	 ?
[14:51:27] <jynus>	 anyway
[14:51:28] <marostegui>	 In 1 year maybe they won't be even able to replicate :)
[14:51:41] <jynus>	 I would test the replication just in case on a test host
[14:51:48] <jynus>	 not now
[14:51:52] <jynus>	 but at some point
[14:51:57] <marostegui>	 sure
[14:52:03] <jynus>	 ok
[14:52:04] <marostegui>	 I doubt it will work
[14:52:12] <jynus>	 so let's go over the plan
[14:52:17] <marostegui>	 especially with RBR, triggers and all that :)
[14:52:25] <jynus>	 please review so we are all ok and makes sense
[14:53:14] <jynus>	 1) I upgrade db2101 (x1) to 10.4
[14:53:24] <jynus>	 and send snapshots to backup2002
[14:53:46] <jynus>	 (I will keep dumps on the same dbprov)
[14:54:52] <jynus>	 2) I move db1102 (s4) to db1145
[14:55:17] <jynus>	 do I move s5, which is also on db1102 to it or somewhere else?
[14:55:19] <marostegui>	 would you also move the other section db1102 holds?
[14:55:23] <jynus>	 he he
[14:55:33] <marostegui>	 I would move it to db1145 just to make things easier
[14:55:37] <jynus>	 ok
[14:55:39] <marostegui>	 and then use db1102 wherever you like
[14:55:48] <jynus>	 2) I move db1102 (s4, s5) to db1145
[14:56:24] <jynus>	 db1145 will be on stretch
[14:56:31] <marostegui>	 sounds good
[14:56:44] <jynus>	 3) I put x1 on db1102 (buster)
[14:57:13] <jynus>	 (I then can remove it from db1095)
[14:57:33] <jynus>	 4) I backup x1 from db1102 to backup1002
[14:57:46] <jynus>	 something else?
[14:57:57] <marostegui>	 I think that's all yeah
[14:58:02] <jynus>	 on Q1
[14:58:10] <jynus>	 we will get dbprov[12]003
[14:58:25] <jynus>	 and will be on buster directly and return snapshots from buster to it
[14:58:42] <jynus>	 plus we will get an extra backup source also on buster for extra flexibility
[14:58:57] <jynus>	 and at that poing backups will not be a blocker
[14:59:15] <jynus>	 *pint
[14:59:23] <jynus>	 I will copy that as an answer to the ticket
[14:59:31] <jynus>	 and will get to it
[14:59:36] <marostegui>	 cool!
[14:59:38] <marostegui>	 thank you
[15:02:20] <wikibugs>	 10DBA: Upgrade x1 databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254871 (10jcrespo) This is the plan after a conversation on IRC: ` <jynus> 1) I upgrade db2101 (x1) to 10.4 <jynus> and send snapshots to backup2002 <jynus> 2) I move db1102 (s4, s5) to db1145 (stretch) <jynus> (I will ke...
[15:03:04] <wikibugs>	 10DBA: Productionize db114[1-9] - https://phabricator.wikimedia.org/T252512 (10jcrespo) Sorry, I hadn't seen this question, I will take care of this following discussion at T254871#6244800
[15:03:12] <jynus>	 sorry I didn't see that
[15:03:18] <marostegui>	 no worries
[15:03:30] <jynus>	 if you ping me while i am offline and see that I don't responde
[15:03:50] <jynus>	 it is likely it got buried below a pile of other mails/ticket updates
[15:04:00] <marostegui>	 haha don't worry
[15:04:04] <jynus>	 *I don't responf after I come back
[15:04:19] <marostegui>	 I know the feeling! :)
[15:04:31] <marostegui>	 I wasn't in a rush, so I didn't ping you after it
[15:05:31] <jynus>	 one last thing
[15:05:56] <jynus>	 oh, I see the confusion
[15:06:06] <jynus>	 db1102 is not backup testing it is backour source
[15:06:32] <jynus>	 what about backup testing, is that back (I don't need it now)?
[15:07:22] <marostegui>	 let me update it
[15:07:23] <wikibugs>	 10DBA: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10jcrespo)
[15:07:28] <jynus>	 I did that one
[15:07:32] <marostegui>	 ah
[15:07:32] <wikibugs>	 10DBA: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10Marostegui)
[15:07:32] <marostegui>	 hehe
[15:07:41] <wikibugs>	 10DBA, 10Data-Services, 10User-Ladsgroup, 10cloud-services-team (Kanban): Prepare and check storage layer for shnwiktionary - https://phabricator.wikimedia.org/T256010 (10Nintendofan885)
[15:07:44] <jynus>	 but I am asking about one host that got pooled into s8
[15:07:51] <jynus>	 it was on section test-s1
[15:08:02] <marostegui>	 jynus: yep, let's take one of those from the list
[15:08:07] <marostegui>	 not yet sure which one
[15:08:16] <jynus>	 ok, so it will be one of those, right?
[15:08:20] <marostegui>	 kormat: want to handle that new wiki creation?  ^
[15:08:22] <marostegui>	 jynus: yep
[15:08:31] <jynus>	 ok, not need for now, but just good to know
[15:09:02] <kormat>	 marostegui: yeah sure. it would beat dealing with puppet/pcc.
[15:09:43] <wikibugs>	 10DBA: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10jcrespo)
[15:09:50] <jynus>	 I've put db1081 (back to backup testing test-s1?)
[15:09:57] <jynus>	 but won't touch anyone for now
[15:10:02] <jynus>	 but I may need one at some point
[15:10:30] <jynus>	 to improve/test the recovery
[15:11:08] <marostegui>	 sounds good
[15:11:27] <marostegui>	 I will be able to give you db1084 before
[15:11:44] <marostegui>	 or actually now that I think about it
[15:11:49] <marostegui>	 On thursday I will give you db1135
[15:11:50] <marostegui>	 :)
[15:11:52] <jynus>	 up to you
[15:11:57] <marostegui>	 as that one will go away from m1
[15:11:57] <jynus>	 db1135?
[15:11:59] <marostegui>	 I will ping you
[15:12:08] <jynus>	 is that "mine?"
[15:12:13] <marostegui>	 it will be :)
[15:12:19] <jynus>	 or just mean it will be free
[15:12:19] <marostegui>	 don't know
[15:12:26] <jynus>	 because I just need one :-D
[15:12:33] <marostegui>	 Yeah, either db1084 or db1135
[15:12:41] <jynus>	 ok, will put in on db1084
[15:12:44] <marostegui>	 but yeah, there will be one
[15:12:44] <marostegui>	 sure
[15:12:46] <jynus>	 I chose the first one thinking
[15:12:54] <jynus>	 it will be the worst one
[15:12:59] <jynus>	 as I don't need performance
[15:13:03] <jynus>	 let me change it
[15:13:55] <marostegui>	 I just checked, you can take db1084 now if you want
[15:14:01] <wikibugs>	 10DBA: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10jcrespo)
[15:14:21] <jynus>	 I've marked it, will ping you when I repuppetize it
[15:14:41] <wikibugs>	 10DBA: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10Marostegui)
[15:14:43] <marostegui>	 cool
[15:14:53] <marostegui>	 you'd need to depool it
[15:15:07] <marostegui>	 as it is serving traffic at the moment
[15:24:12] <wikibugs>	 10DBA: Upgrade x1 databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254871 (10jcrespo) a:03jcrespo
[15:27:06] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: db1088 crashed - https://phabricator.wikimedia.org/T255927 (10wiki_willy) a:03Jclark-ctr @Jclark-ctr - I think there are some bbu's leftover from the last time you requested some spares to be ordered, but let me know if not.   Thanks, Willy
[15:29:42] <marostegui>	 kormat: thanks! so far it doesn't require any action as we are waiting for them to create the DB, but if you can let them know to ping us once it is created, move it to the blocked column etc that'd be nice - thank you!
[15:30:09] <marostegui>	 kormat: we also need to know if it is a public one or not
[15:30:17] <marostegui>	 the name indicates so, but let's ask
[15:30:57] <wikibugs>	 10DBA: Compress enwiki InnoDB tables - https://phabricator.wikimedia.org/T254462 (10Marostegui)
[15:33:28] <wikibugs>	 10DBA, 10Data-Services, 10User-Ladsgroup, 10cloud-services-team (Kanban): Prepare and check storage layer for shnwiktionary - https://phabricator.wikimedia.org/T256010 (10Kormat) Hey. Let me know when the DB has been created. Also, can you confirm this will be a public wiki?
[15:33:41] <kormat>	 marostegui: done
[15:33:46] <marostegui>	 thank you! :)
[15:33:52] <kormat>	 you're almost welcome
[15:34:08] <marostegui>	 I think I should logoff! it's been enough for todya!
[15:34:10] <marostegui>	 byeee
[15:34:31] <kormat>	 hehe o7
[15:36:33] <wikibugs>	 10DBA, 10Data-Services, 10User-Ladsgroup, 10cloud-services-team (Kanban): Prepare and check storage layer for shnwiktionary - https://phabricator.wikimedia.org/T256010 (10RhinosF1) >>! In T256010#6244973, @Kormat wrote: > Hey. Let me know when the DB has been created.  The wiki was created today >Also, can...
[15:53:44] <wikibugs>	 10DBA, 10Data-Services, 10User-Ladsgroup, 10cloud-services-team (Kanban): Prepare and check storage layer for shnwiktionary - https://phabricator.wikimedia.org/T256010 (10Kormat) Sanitization is in place, complete private data check running now.
[18:03:47] <volans>	 if you agree I'd say to start Wed. directly with the new system so that when DCOps wakes up they can start using it already and I can help if there is any issue and Cas will takeover helping for the second part of the day
[18:04:00] <volans>	 ops, wrong window, sorry