[06:27:06] <wikibugs>	 10DBA, 10cloud-services-team (Kanban): DB planning: include a writeable (?) misc DB cluster in codfw for WMCS - https://phabricator.wikimedia.org/T218570 (10Marostegui) 05Open→03Declined Thanks for the heads up @bd808 - that sounds good to me :-) Will m5 content be eventually migrated to those new hosts?...
[06:31:19] <wikibugs>	 10DBA: decommission dbproxy1002.eqiad.wmnet - https://phabricator.wikimedia.org/T245384 (10Marostegui)
[06:32:02] <wikibugs>	 10DBA: decommission dbproxy1007.eqiad.wmnet - https://phabricator.wikimedia.org/T245385 (10Marostegui)
[06:34:23] <wikibugs>	 10DBA: Remove grants for the old dbproxy hosts from the misc databases - https://phabricator.wikimedia.org/T231280 (10Marostegui)
[06:40:43] <wikibugs>	 10DBA: Remove grants for the old dbproxy hosts from the misc databases - https://phabricator.wikimedia.org/T231280 (10Marostegui)
[06:40:48] <wikibugs>	 10DBA, 10Patch-For-Review: decommission dbproxy1007.eqiad.wmnet - https://phabricator.wikimedia.org/T245385 (10Marostegui)
[06:40:49] <wikibugs>	 10DBA, 10Patch-For-Review: decommission dbproxy1002.eqiad.wmnet - https://phabricator.wikimedia.org/T245384 (10Marostegui)
[06:41:31] <wikibugs>	 10DBA, 10Patch-For-Review: decommission dbproxy1002.eqiad.wmnet - https://phabricator.wikimedia.org/T245384 (10Marostegui)
[06:42:05] <wikibugs>	 10DBA, 10Patch-For-Review: decommission dbproxy1007.eqiad.wmnet - https://phabricator.wikimedia.org/T245385 (10Marostegui)
[07:03:46] <wikibugs>	 10DBA: Test MariaDB 10.4 in production - https://phabricator.wikimedia.org/T242702 (10Marostegui) Pooled db1107 with weight 100 on main and weight 10 on API for the first time: {P10419}
[08:18:54] <wikibugs>	 10DBA: Compress table watchlist_expiry - https://phabricator.wikimedia.org/T245358 (10Mooeypoo) Will do, @Marostegui ; FYI, I'm also tracking the creation of the table through this ticket: {T244631}.   @Jdforrester-WMF has it on his list for after wmf.19 has stable'y passed through to production.
[08:20:07] <wikibugs>	 10DBA: Compress table watchlist_expiry - https://phabricator.wikimedia.org/T245358 (10Marostegui) Subscribed! Thanks :)
[08:24:16] <wikibugs>	 10DBA: Compress table watchlist_expiry - https://phabricator.wikimedia.org/T245358 (10Marostegui)
[09:29:38] <wikibugs>	 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui)
[09:30:04] <wikibugs>	 10DBA, 10cloud-services-team (Kanban): DB planning: include a writeable (?) misc DB cluster in codfw for WMCS - https://phabricator.wikimedia.org/T218570 (10aborrero) >>! In T218570#5888668, @Marostegui wrote: > Will m5 content be eventually migrated to those new hosts? >   I think on our side we are referring...
[09:31:54] <wikibugs>	 10DBA, 10cloud-services-team (Kanban): DB planning: include a writeable (?) misc DB cluster in codfw for WMCS - https://phabricator.wikimedia.org/T218570 (10Marostegui) Thanks for the answer. This is what we currently have in m5: ` +---------------------+ | Database            | +---------------------+ | desig...
[09:47:57] <wikibugs>	 10DBA: Remove grants for the old dbproxy hosts from the misc databases - https://phabricator.wikimedia.org/T231280 (10Marostegui)
[09:59:48] <wikibugs>	 10DBA, 10cloud-services-team (Kanban): DB planning: include a writeable (?) misc DB cluster in codfw for WMCS - https://phabricator.wikimedia.org/T218570 (10aborrero) Ok, I think we can safely assume we are talking only about openstack databases: designate, glance, keystone, neutron, nova*.
[10:07:27] <wikibugs>	 10DBA, 10cloud-services-team (Kanban): DB planning: include a writeable (?) misc DB cluster in codfw for WMCS - https://phabricator.wikimedia.org/T218570 (10Marostegui) Thanks for the clarification
[10:10:50] <wikibugs>	 10DBA, 10Parsing-Team: testreduce_vd database in m5 still in use? - https://phabricator.wikimedia.org/T245408 (10Marostegui)
[10:11:05] <wikibugs>	 10DBA, 10Parsing-Team: testreduce_vd database in m5 still in use? - https://phabricator.wikimedia.org/T245408 (10Marostegui) p:05Triage→03Medium
[10:15:33] <jynus>	 I may have to reconstruct s3 backup source from logical backup or another host
[10:16:29] <marostegui>	 I saw it failed, yeah, I figured that's the copy from db1095 to db1140?
[10:17:04] <jynus>	 I don't think the recovery failed
[10:17:25] <jynus>	 it may have been bad already, that used to fail a lot
[10:17:34] <marostegui>	 yes, could be :(
[10:18:01] <jynus>	 will recover and then run some consistency checks
[14:37:08] <jynus>	 https://phabricator.wikimedia.org/P10423
[14:38:28] <jynus>	 MariaDB 10.5.1, 10.4.12, 10.3.22 and 10.2.31, 10.1.44 and 5.5.67 now available!
[14:41:04] <marostegui>	 I compiled 10.4.12 a couple of days ago when it was released: https://gerrit.wikimedia.org/r/#/c/operations/software/+/568978/
[14:41:34] <marostegui>	 s/days/weeks
[14:42:56] <jynus>	 I was only pointing out that there is 6 supported major versions
[14:44:20] <marostegui>	 ah!
[14:44:41] <marostegui>	 I guess 10.1 and 5.5 are ending in sept both of them?
[14:44:53] <jynus>	 I haven't checked
[14:45:17] <marostegui>	 actually 5.5 ends 11th March 2020
[15:18:50] <jynus>	 Self note to check:[WARNING] - Row bigger than statement_size for fawikisource.image, sawikisource.image
[15:26:39] <wikibugs>	 10DBA, 10Parsing-Team: testreduce_vd database in m5 still in use? - https://phabricator.wikimedia.org/T245408 (10ssastry) We will use it again once we start running regular visual diff runs between Parsoid output and core PHP parser output (sometime in the next few months). If we change anything in the infrast...
[15:29:33] <jynus>	 heads up for https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=db1140&var-datasource=eqiad%20prometheus%2Fops&var-cluster=mysql&fullscreen&panelId=6&from=1581942562993&to=1581964162994
[15:29:51] <jynus>	 I don't think that will affect you, but so it doesn't suprise you on aggregated metrics
[15:48:12] <marostegui>	 ah, thanks :)
[15:48:31] <marostegui>	 we used to see lots of those warnings for image tables
[15:56:32] <wikibugs>	 10DBA, 10Parsing-Team: testreduce_vd database in m5 still in use? - https://phabricator.wikimedia.org/T245408 (10Marostegui) Excellent! Thank you! As a follow up, maybe not super related, but I also saw:  ` root@db1133:/srv/sqldata/testreduce_0715# ls -lShr total 520G -rw-rw---- 1 mysql mysql   54 May 23  2018...
[17:15:43] <wikibugs>	 10DBA, 10Operations, 10Wikimedia-Etherpad: Upgrade and restart m1 master (db1135) - https://phabricator.wikimedia.org/T244238 (10Trizek-WMF) >>! In T244238#5876674, @Marostegui wrote: > @Trizek-WMF so we are going to do this maintenance Thursday 20th at 09:00 AM UTC, can you post it on Technews?  I'm just co...
[17:18:21] <wikibugs>	 10DBA, 10Operations, 10Wikimedia-Etherpad: Upgrade and restart m1 master (db1135) - https://phabricator.wikimedia.org/T244238 (10Marostegui) >>! In T244238#5890721, @Trizek-WMF wrote: >>>! In T244238#5876674, @Marostegui wrote: >> @Trizek-WMF so we are going to do this maintenance Thursday 20th at 09:00 AM U...
[17:19:23] <wikibugs>	 10DBA, 10Operations, 10Wikimedia-Etherpad: Upgrade and restart m1 master (db1135) - https://phabricator.wikimedia.org/T244238 (10Trizek-WMF) We all run at high speeds. Don't worry though, I don't think that missing Tech News will be a blocker. :)
[22:58:42] <wikibugs>	 10DBA, 10Parsing-Team: testreduce_vd database in m5 still in use? - https://phabricator.wikimedia.org/T245408 (10ssastry) Do I have permissions to delete rows from this database? But ,perfstats, stats, results are all keyed on commit hashes in the commits table. The commit table has one commit hash per test ru...