[01:31:33] <wikibugs>	 10DBA, 10Operations, 10Privacy Engineering, 10Traffic, and 4 others: dbtree loads third party resources (from google.com/jsapi) - https://phabricator.wikimedia.org/T96499 (10Krinkle)
[07:27:09] <wikibugs>	 10DBA, 10OTRS, 10Operations, 10Recommendation-API, 10Research: Upgrade and restart m2 primary database master (db1132) - https://phabricator.wikimedia.org/T246098 (10Marostegui) Thanks @leila! @akosiaris does Tuesday 17th at 09:00 AM UTC work?
[07:28:05] <wikibugs>	 10DBA, 10OTRS, 10Operations, 10Recommendation-API, 10Research: Upgrade and restart m2 primary database master (db1132) - https://phabricator.wikimedia.org/T246098 (10akosiaris) >>! In T246098#5955323, @Marostegui wrote: > Thanks @leila! > @akosiaris does Tuesday 17th at 09:00 AM UTC work?  Fine by me.
[07:36:14] <wikibugs>	 10DBA, 10OTRS, 10Operations, 10Recommendation-API, 10Research: Upgrade and restart m2 primary database master (db1132) - https://phabricator.wikimedia.org/T246098 (10Marostegui) Excellent - going to send calendar invite and block that time on the deployment page.
[07:37:01] <wikibugs>	 10DBA, 10OTRS, 10Operations, 10Recommendation-API, 10Research: Upgrade and restart m2 primary database master (db1132) - https://phabricator.wikimedia.org/T246098 (10Marostegui)
[08:29:04] <jynus>	 backup1001 will warn about a couple of empty backups
[08:29:38] <jynus>	 will run them manually when logical backups finish
[08:30:19] <marostegui>	 Cool
[08:57:51] <jynus>	 should we announce the maintenance?
[08:57:59] <marostegui>	 yeah
[08:58:03] <marostegui>	 I will do in a sec
[08:58:06] <marostegui>	 I was getting my env ready
[08:58:46] <jynus>	 there was a fetchblob error just now
[08:59:00] <marostegui>	 it was me
[08:59:03] <jynus>	 ok
[08:59:12] <marostegui>	 testing the non existing es5 for now
[09:31:36] <jynus>	 some monolog fatals on mwdebug1001
[09:31:52] <marostegui>	 mmm related?
[09:32:05] <jynus>	 probably not, probably deployment logging related
[09:32:29] <jynus>	 it is mwdebug only anyway
[09:32:59] <jynus>	 higher offeder is set-time-limit, but same rate as before deploy
[09:33:04] <jynus>	 *offender
[09:33:30] <marostegui>	 yeah
[09:33:31] <jynus>	 so unless there is something weird on recentchanges, I would call it done
[09:33:38] <marostegui>	 Yep
[09:33:39] <marostegui>	 Going to close it
[09:34:00] <marostegui>	 let's evaluate in a few hours if we want to go ahead today and set es3 as RO
[09:34:27] <jynus>	 let me update zarcillo's roles
[09:34:46] <marostegui>	 thanks
[09:34:47] <jynus>	 or at least review them
[09:35:30] <jynus>	 yep, needs to add es5 masters
[09:42:20] <wikibugs>	 10DBA, 10Core Platform Team Workboards (Clinic Duty Team), 10Goal, 10Patch-For-Review: Enable es4 and es5 as writable new external store sections and set es2 and es3 as read only - https://phabricator.wikimedia.org/T246072 (10Marostegui) es5 has been enabled correctly. We will either disable es3 and set it...
[09:42:51] <wikibugs>	 10DBA, 10Core Platform Team Workboards (Clinic Duty Team), 10Goal, 10Patch-For-Review: Enable es4 and es5 as writable new external store sections and set es2 and es3 as read only - https://phabricator.wikimedia.org/T246072 (10Marostegui)
[09:45:43] <jynus>	 one last thing before break
[09:45:57] <jynus>	 I see 3 hosts on collection failures
[09:46:09] <jynus>	 db1114 (mysql 8, known)
[09:46:24] <jynus>	 db1111 and db1078, those expected?
[09:47:19] <marostegui>	 db1111 is kinda expected as it is the most loaded on s8
[09:47:36] <jynus>	 not about load
[09:47:45] <marostegui>	 oh sorry, collection
[09:47:48] <jynus>	 prometheus failures- however,I can see their graphs
[09:47:48] <marostegui>	 I read connection
[09:47:51] <jynus>	 sorry
[09:47:56] <jynus>	 prometheus collection
[09:48:03] <marostegui>	 those run 10.4
[09:48:04] <jynus>	 so not sure what that means?
[09:48:05] <marostegui>	 so yes
[09:48:11] <marostegui>	 expected as some of them aren't available on 10.4
[09:48:12] <jynus>	 but other have 10.4
[09:48:25] <jynus>	 and don't error out?
[09:48:26] <marostegui>	 https://phabricator.wikimedia.org/T244696
[09:48:48] <jynus>	 let me show the panel so you can tell
[09:49:08] <jynus>	 https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&fullscreen&panelId=4&from=1583812144231&to=1583833744231&var-dc=eqiad%20prometheus%2Fops&var-group=core&var-shard=All&var-role=All
[09:49:33] <marostegui>	 that's strange yeah
[09:49:39] <marostegui>	 I would expect the others 10.4 to also show up there
[09:49:44] <jynus>	 I thought there was other 10.4 hosts
[09:49:49] <jynus>	 ah, so it was not only me
[09:49:49] <marostegui>	 yeah, there are lots of others
[09:49:52] <marostegui>	 almost 1 per section
[09:49:57] <jynus>	 so why only those?
[09:50:02] <jynus>	 (rethorical question)
[09:50:21] <jynus>	 just pointing out and worth researching- could be a prometheus only issue
[09:50:41] <jynus>	 (not worried about metrics not being available)
[09:50:56] <jynus>	 were those the last ones installed?
[09:51:05] <jynus>	 or maybe the only ones that are not multiinstance?
[09:51:22] <marostegui>	 no, those were actually one of the first ones
[09:51:31] <jynus>	 interesting
[09:51:41] <marostegui>	 we havehttps://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&fullscreen&panelId=4&from=now-2d&to=now&edit&var-dc=codfw%20prometheus%2Fops&var-group=core&var-shard=All&var-role=All
[09:51:43] <jynus>	 will take a break and may poke them a bit
[09:51:44] <marostegui>	 in codfw we have more
[09:51:51] <marostegui>	 so something with 10.4
[09:51:54] <jynus>	 just to be clear- prometheus seems to be working there normally
[09:51:54] <marostegui>	 let's see what I can find
[09:52:12] <jynus>	 see the mysql stats working normally
[09:52:24] <marostegui>	 yeah
[09:53:01] <jynus>	 vs db1114 which I didn't setup propery and doesn't even appear
[09:55:03] <jynus>	 BTW, db1111 appear on test group
[09:55:05] <marostegui>	 so all of them having errors could be realted to the missing metrics indeed
[09:55:08] <jynus>	 needs to be moved to core
[09:55:09] <marostegui>	 no?
[09:55:30] <jynus>	 but aren't there missing metrics on the other 10.4 hosts?
[09:55:39] <marostegui>	 yes
[09:55:45] <marostegui>	 but all codfw show errors
[09:55:52] <jynus>	 but they do not show as "collection errors"?
[09:56:11] <marostegui>	 https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&fullscreen&panelId=4&from=now-2d&to=now&edit&var-dc=codfw%20prometheus%2Fops&var-group=core&var-shard=All&var-role=All
[09:56:11] <marostegui>	 they do
[09:56:34] <jynus>	 ah, I see
[09:56:42] <jynus>	 so it only shows the first ones!ª
[09:56:42] <marostegui>	 for some reason, the multi instance on eqiad aren't showing up there
[09:57:00] <jynus>	 ok, so no worry for now, right?
[09:57:03] <marostegui>	 root@db1111:~# curl localhost:9104/metrics | grep scrape | grep -v "#"
[09:57:04] <marostegui>	   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[09:57:04] <marostegui>	                                  Dload  Upload   Total   Spent    Left  Speed
[09:57:04] <marostegui>	 100  189k  100  189k    0     0  11.5M      0 --:--:-- --:--:-- --:--:-- 11.5M
[09:57:04] <marostegui>	 mysql_exporter_last_scrape_error 1
[09:57:08] <marostegui>	 it doesn't change
[09:57:10] <jynus>	 yep
[09:57:10] <marostegui>	 from what I can see
[09:57:19] <jynus>	 we will research that at a later time
[09:57:27] <marostegui>	 yeah
[09:57:36] <jynus>	 I thought there was something else weird
[09:59:26] <jynus>	 I ran "update instances set `group` = 'core' where name = 'db1111' limit 1;"
[09:59:40] <jynus>	 db1112 still on test-s4, for now?
[10:00:02] <jynus>	 and I promise to have a better edit interface soon(TM)
[10:00:44] <marostegui>	 no, db1112 is in s3 now
[10:00:49] <jynus>	 ah
[10:01:04] <jynus>	 ok, what about db1077?
[10:01:17] <marostegui>	 test-s4 yeah
[10:01:31] <marostegui>	 db1077 got broken BBU and we changed it with db1112
[10:01:35] <jynus>	 will update db1112
[10:01:37] <marostegui>	 I am guilty for not updating zarcillo
[10:01:38] <marostegui>	 sorry :(
[10:01:46] <jynus>	 not your fault, it is not easy preciselly
[10:04:39] <jynus>	 let me know if you see something else weird: https://phabricator.wikimedia.org/P10674
[10:05:11] <jynus>	 s10 may be missing, I guess
[10:05:57] <marostegui>	 yeah, s10 is db1133
[10:06:00] <marostegui>	 which is still m5
[10:06:30] <jynus>	 ok, not sure how to represent something being both s10 AND m5 :-D
[10:06:42] <marostegui>	 nah, I think just leave it in m5
[10:06:45] <jynus>	 ok
[10:08:14] <jynus>	 will run snapshot bacula transfers and will take a break
[10:08:48] <marostegui>	 cool!
[10:08:54] <marostegui>	 I am checking the exporter thing
[10:15:57] <jynus>	 db1111 started going green?
[10:16:50] <marostegui>	 I have restarted the exporter to debug 
[10:17:01] <marostegui>	 I think I have found something
[10:17:02] <marostegui>	 let me see
[10:17:31] <jynus>	 db1078 is red still, but db1111 started to "go green"
[10:19:08] <marostegui>	 I think it is the same case as the bug you saw with the password string
[10:19:18] <marostegui>	 I have created a /root/.my.cnf and the errors started to get gone
[10:19:26] <marostegui>	 from what I can see on the --debug
[10:19:37] <marostegui>	 let me confirm
[10:19:44] <jynus>	 created? but it should already be there, doesn't it?
[10:20:06] <marostegui>	 there was no .my.cnf on 7-root
[10:20:21] <marostegui>	 I am also seeing some errors with these lines: https://github.com/prometheus/mysqld_exporter/blob/master/mysqld_exporter.go#L142 and https://github.com/prometheus/mysqld_exporter/blob/master/mysqld_exporter.go#L119
[10:20:26] <marostegui>	 so it does look kinda related
[10:20:28] <jynus>	 there should not be .my.cnf on /root
[10:20:35] <jynus>	 on any host
[10:20:46] <jynus>	 I think you are trying to run it manually, which you shouldn't
[10:20:49] <jynus>	 as root
[10:20:57] <marostegui>	 yeah, but that cleaned up the errors
[10:21:07] <marostegui>	 I am seeing those errors on those lines
[10:21:08] <jynus>	 but prometheus shouldn't be run as root
[10:21:14] <marostegui>	 yeah, it was just for debugging
[10:21:15] <jynus>	 only started through systemd
[10:22:04] <jynus>	 still, for debugging, "sudo -u prometheus"
[10:22:36] <jynus>	 if it fails, then there may be extra grants neede by the user
[10:23:15] <jynus>	 on the new prometheus exporter for buster
[10:32:15] <marostegui>	 Mar 10 10:31:54 db1111 prometheus-mysqld-exporter[17089]: time="2020-03-10T10:31:54Z" level=debug msg="collect query: []" source="mysqld_exporter.go:142"
[10:35:46] <marostegui>	 Interesting, I just changed the ARGs for the systecmt unit to run it as normally but with --log-level=debug and now it is getting green too (.my.cnf is removed)
[10:38:01] <marostegui>	 And to make it even more interesting, just restarting the exporter (without touching ANYTHING) made db1078 to start decreasing its errors 
[10:38:10] <marostegui>	 I am thinking about this combination
[10:38:21] <jynus>	 something's weird there
[10:38:28] <marostegui>	 host restarts to buster, the exporter starts running, we start mysql and then we do the mysql_upgrade
[10:38:43] <marostegui>	 So maybe it reads from information_schema when the tables are still not updated to 10.4 schemas
[10:39:23] <jynus>	 maybe create a subticket about this for further research?
[10:39:32] <marostegui>	 yeah
[10:39:36] <marostegui>	 let me confirm it
[10:39:43] <marostegui>	 because db1078 is now getting healthy
[10:39:56] <marostegui>	 let me try on another recently upgraded host 
[10:40:02] <marostegui>	 just restarting the exporter
[10:40:06] <jynus>	 but what did you do, just restart?
[10:40:15] <marostegui>	 just restart the exporter
[10:40:21] <jynus>	 umh
[10:40:30] <marostegui>	 going to do that on https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&fullscreen&panelId=4&from=now-5m&to=now&var-dc=codfw%20prometheus%2Fops&var-group=core&var-shard=All&var-role=All
[10:40:32] <marostegui>	 let's see
[10:40:56] <marostegui>	 it could be because the exporter starts running before we update the internal tables to 10.4 structure
[10:45:16] <marostegui>	 I think it is confirmed: https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&fullscreen&panelId=4&from=now-5m&to=now&var-dc=codfw%20prometheus%2Fops&var-group=core&var-shard=All&var-role=All all the hosts but db2125 have been restarted and they are decreasing errors
[10:45:25] <marostegui>	 restarted as in mysqld exporter restarted just now
[11:05:55] <wikibugs>	 10DBA, 10OTRS, 10Operations, 10Recommendation-API, 10Research: Upgrade and restart m2 primary database master (db1132) - https://phabricator.wikimedia.org/T246098 (10Marostegui) Mail sent to wikitech-l: https://lists.wikimedia.org/pipermail/wikitech-l/2020-March/093175.html Deployments calendar window ad...
[11:24:36] <wikibugs>	 10DBA: mysql_exporter_last_scrape_error flag on prometheus-mysqld-exporter increases after 10.4 upgrade - https://phabricator.wikimedia.org/T247290 (10Marostegui)
[11:25:30] <wikibugs>	 10DBA: mysql_exporter_last_scrape_error flag on prometheus-mysqld-exporter increases after 10.4 upgrade - https://phabricator.wikimedia.org/T247290 (10Marostegui) p:05Triage→03Medium I still have to install some hosts with buster and 10.4 so I am going to confirm if that's the issue, by making sure to stop t...
[11:31:30] <wikibugs>	 10DBA: mysql_exporter_last_scrape_error flag on prometheus-mysqld-exporter increases after 10.4 upgrade - https://phabricator.wikimedia.org/T247290 (10Marostegui) The last host pending to restart from the above dashboard: ` root@db2125:~# curl -s localhost:9104/metrics | grep scrape | grep -v "#" mysql_exporter_...
[11:34:50] <jynus>	 akosiaris: production. LastWritten: 2019-12-12 04:40:20, VolRetention: 7,776,000, Expires in: 61,891
[11:34:56] <jynus>	 ^that is expected
[11:36:27] <jynus>	 but the oldest on Databases is from 2020-02-05
[11:37:14] <jynus>	 with expiresIn: 4,809,863
[11:37:31] <jynus>	 so maybe I just missed an update (only updated production)
[11:37:39] <jynus>	 we'll see how it behaves now
[11:38:12] <jynus>	 I have no alerts based on retention- I may add that soon
[11:39:25] <jynus>	 it is not recycling indded, a new volume just been created
[12:00:05] <volans>	 marostegui: count me in for the db1132 scheduled restart for the debmonitor part, although there should be nothing to do there
[13:43:27] <marostegui>	 thanks volans
[13:43:44] <volans>	 is there a calendar invite I can accept by any chance?
[13:43:54] <volans>	 just to make sure I don't forget
[13:45:26] <marostegui>	 Ah sure
[13:45:29] <marostegui>	 I can send it to you
[13:47:43] <volans>	 thx! <3
[13:56:35] <wikibugs>	 10DBA: Install 1 buster+10.4 host per section - https://phabricator.wikimedia.org/T246604 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db2121.codfw.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202003101355_marostegui_227323.log`.
[14:23:02] <wikibugs>	 10DBA: Install 1 buster+10.4 host per section - https://phabricator.wikimedia.org/T246604 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db2121.codfw.wmnet'] `  and were **ALL** successful.
[14:28:37] <wikibugs>	 10DBA: mysql_exporter_last_scrape_error flag on prometheus-mysqld-exporter increases after 10.4 upgrade - https://phabricator.wikimedia.org/T247290 (10Marostegui) I have done the workflow in a different way, which is essentially, starting mysql and running mysql_upgrade BEFORE starting the exporter and: ` root@d...
[14:38:16] <jynus>	 so, aside from documentation, do you see any puppet patch fixing that^
[14:38:38] <jynus>	 because we could add a dependency on systemd, but that would start mysqld
[14:38:48] <jynus>	 or we could add an if on puppet
[14:38:53] <marostegui>	 Yeah, or just wait for mysqld to be running
[14:38:55] <jynus>	 but it may be worse than doing nothing
[14:38:56] <marostegui>	 before starting
[14:38:59] <marostegui>	 exactly
[14:39:13] <jynus>	 I will let you come up with a recommendation
[14:39:15] <marostegui>	 I think it is not worth the effort, we can just issue a restart or something
[14:39:16] <marostegui>	 yeah
[14:39:29] <marostegui>	 I think I am going to add a page on wikitech with migration things
[14:39:30] <marostegui>	 like that
[14:39:39] <marostegui>	 or like that we have to remove stuff from grafana that no longer exists
[14:39:57] <marostegui>	 or bugs we are waiting for upstream to fix before we can enable things back (like the optimizer flag)
[14:43:46] <marostegui>	 jynus: I am thinking about setting es3 to RO tomorrow
[14:43:57] <jynus>	 as you wish
[14:43:58] <marostegui>	 Not today
[14:44:27] <marostegui>	 I am not sure
[14:44:47] <jynus>	 he he, I will help you whatever you decide :-D
[14:46:23] <cdanis>	 marostegui: \o/ re: es5, btw
[14:47:05] <marostegui>	 yaaay!
[15:10:47] <jynus>	 check_bacula.py  dbprov2001.codfw.wmnet-Monthly-1st-Sun-Databases-mysql-srv-backups-snapshots-latest ~> 2020-03-10 10:14:44: type: F, status: T, bytes: 1,590,029,087,872
[15:15:49] <jynus>	 https://grafana.wikimedia.org/d/413r2vbWk/bacula?orgId=1&var-dc=eqiad%20prometheus%2Fops&var-job=dbprov2001.codfw.wmnet-Monthly-1st-Sun-Databases-mysql-srv-backups-snapshots-latest&from=1583836961285&to=1583853319711
[15:16:06] <jynus>	 around 3TB in total size with both dbprovs
[15:29:25] <jynus>	 I've added the column `standalone` to `sections` table, so those can be set on the database instead of being hardcoded on the script
[15:30:01] <jynus>	 select name FROM sections where standalone=1; ~> es1, es2, tendril
[15:30:25] <jynus>	 will send a patch to the prometheus config generator now
[15:31:03] <jynus>	 which will make https://gerrit.wikimedia.org/r/c/operations/puppet/+/576655 unnecesary
[15:38:02] <marostegui>	 oooh thats useful yeah
[15:40:11] <jynus>	 this is not super necesary, but it will help understanding what information is there on tendril, that we need
[15:42:22] <marostegui>	 yeah, definitely
[15:42:39] <marostegui>	 And as soon as we start using it for more source of truth kind of thing, we will notice and change more things
[15:56:44] <jynus>	 I will put es1, es2, tendril, m4 and staging as "standalone" hosts (no replication)
[15:57:05] <marostegui>	 es3 tomorrow yeah
[15:57:09] <jynus>	 yep
[15:57:14] <marostegui>	 m4 is no longer, if you want to actually delete it
[15:57:33] <jynus>	 well, there is still db1108 around
[15:57:36] <marostegui>	 yeah
[15:57:42] <marostegui>	 but I think the dns is also gone
[15:58:05] <jynus>	 not a big deal just to make sure I have everything tendril has
[15:58:11] <marostegui>	 yeah
[15:58:20] <jynus>	 can be deleted later
[16:16:58] <jynus>	 will test it later to make sure it is a noop: https://gerrit.wikimedia.org/r/c/operations/puppet/+/578547
[16:19:35] <marostegui>	 for now https://wikitech.wikimedia.org/wiki/MariaDB#Stretch_+_10.1_-%3E_Buster_+_10.4_known_issues
[16:20:27] <jynus>	 cool, thanks!
[16:20:41] <jynus>	 link it from the task
[16:21:12] <marostegui>	 yep :)
[16:25:59] <wikibugs>	 10DBA: mysql_exporter_last_scrape_error flag on prometheus-mysqld-exporter increases after 10.4 upgrade - https://phabricator.wikimedia.org/T247290 (10Marostegui) 05Open→03Resolved a:03Marostegui Added to the list of known issues: https://wikitech.wikimedia.org/wiki/MariaDB#Stretch_+_10.1_-%3E_Buster_+_10....
[16:26:01] <wikibugs>	 10DBA: Test MariaDB 10.4 in production - https://phabricator.wikimedia.org/T242702 (10Marostegui)
[16:27:18] <wikibugs>	 10DBA: mysql_exporter_last_scrape_error flag on prometheus-mysqld-exporter increases after 10.4 upgrade - https://phabricator.wikimedia.org/T247290 (10Marostegui)
[17:03:44] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: backup2001.mgmt interface down - https://phabricator.wikimedia.org/T247324 (10jcrespo)
[17:43:13] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: backup2001.mgmt interface down - https://phabricator.wikimedia.org/T247324 (10Papaul) 05Open→03Resolved a:03Papaul Was cleaning up some old cables and accidentally disconnected  the mgmt cable. All back up now
[17:44:46] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: backup2001.mgmt interface down - https://phabricator.wikimedia.org/T247324 (10jcrespo) We love when it is causes are simple as this! (vs a complex to debug issue). Do not worry at all! Thanks!
[17:49:40] <jynus>	 Habemus snapshots on bacula!
[19:26:26] <wikibugs>	 10DBA, 10Cleanup, 10Data-Services, 10cloud-services-team (Kanban): Drop DB tables for now-deleted fixcopyrightwiki from production - https://phabricator.wikimedia.org/T246055 (10Bstorm) @Marostegui `maintain-views` can clean up views individually, but it won't drop the DB in any case.  May as well just dro...