[06:03:26] <wikibugs>	 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10Marostegui) db1087 has all its tables compressed but wb_terms. I am going to wait for labs to catch up a bit before moving them under codfw's sanitarium before starting the compression on that table, which I guess it will take aro...
[06:14:18] <wikibugs>	 10DBA, 10Parsing-Team: testreduce_vd database in m5 still in use? - https://phabricator.wikimedia.org/T245408 (10Marostegui) >>! In T245408#5895216, @ssastry wrote: >>>! In T245408#5891847, @Marostegui wrote: >>` >> | GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, ALTER, CREATE TEMPORARY TABLES ON `testre...
[06:30:25] <wikibugs>	 10DBA: Compress table watchlist_expiry - https://phabricator.wikimedia.org/T245358 (10Marostegui)
[06:49:06] <wikibugs>	 10DBA: Possibly disable optimizer flag: rowid_filter on 10.4 - https://phabricator.wikimedia.org/T245489 (10Marostegui) >>! In T245489#5893628, @Anomie wrote: > Yeah, in this case it makes little observable difference whether it fetches 501 rows via `el_index_60`, or fetches 553 rows via `el_index` and filesorts...
[07:04:18] <wikibugs>	 10DBA: Clean up wikiadmin2 user from core hosts - https://phabricator.wikimedia.org/T243512 (10Marostegui)
[08:23:23] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['es1020.eqiad.wmnet']...
[08:33:07] <wikibugs>	 10DBA: Clean up wikiadmin2 user from core hosts - https://phabricator.wikimedia.org/T243512 (10Marostegui)
[08:42:37] <wikibugs>	 10DBA: Clean up wikiadmin2 user from core hosts - https://phabricator.wikimedia.org/T243512 (10Marostegui)
[08:42:49] <wikibugs>	 10DBA: Clean up wikiadmin2 user from core hosts - https://phabricator.wikimedia.org/T243512 (10Marostegui) 05Open→03Resolved This is all done
[08:42:51] <wikibugs>	 10DBA: Productionize es1020-es1025, es2020-es2025 - https://phabricator.wikimedia.org/T243052 (10Marostegui)
[08:53:09] <wikibugs>	 10DBA: Remove grants for the old dbproxy hosts from the misc databases - https://phabricator.wikimedia.org/T231280 (10Marostegui)
[08:53:21] <wikibugs>	 10DBA: decommission dbproxy1007.eqiad.wmnet - https://phabricator.wikimedia.org/T245385 (10Marostegui)
[08:54:15] <wikibugs>	 10DBA: decommission dbproxy1007.eqiad.wmnet - https://phabricator.wikimedia.org/T245385 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: `dbproxy1007.eqiad.wmnet` -  dbproxy1007.eqiad.wmnet (**PASS**)   - Downtimed host on Icinga   - Downtimed management interfa...
[09:03:43] <jynus>	 Feb 19 09:03:25 db1140 systemd[1]: [/lib/systemd/system/mariadb@.service:77] Unknown lvalue 'RuntimeDirectoryPreserve' in section 'Service'
[09:03:44] <jynus>	 Feb 19 09:03:25 db1140 systemd[1]: [/lib/systemd/system/mariadb@.service:77] Unknown lvalue 'RuntimeDirectoryPreserve' in section 'Service'
[09:03:54] <marostegui>	 woot?
[09:04:12] <jynus>	 I warned about this some time ago
[09:04:14] <marostegui>	 so that flag is gone in buster or what?
[09:04:30] <jynus>	 no, that option is supposed to be on the systemd on buster only
[09:04:55] <marostegui>	 Ah, yes, db1140 is stretch
[09:04:56] <jynus>	 as systemd on stretch is not supported, so dir is wiped out on stop
[09:05:06] <marostegui>	 but it is the first time we've seen that no?
[09:05:12] <jynus>	 no
[09:05:29] <marostegui>	 I have been stopping multiinstance hosts a lot lately, and didn't see that
[09:05:42] <jynus>	 if you stop all at the same time, it wouldn't affect you
[09:05:55] <jynus>	 or they have a package not affected
[09:06:08] <marostegui>	 which package is db1140 running?
[09:06:16] <marostegui>	 10.1.43?
[09:06:22] <jynus>	 the latest we have, yes
[09:06:37] <marostegui>	 yeah, the ones I stopped didn't have that one I reckon
[09:06:42] <jynus>	 we need to upgrade to a -1
[09:06:59] <jynus>	 and we can do a hot upgrade, just substituting the systemd unit
[09:07:07] <marostegui>	 yeah, looks like it is indeed needed
[09:07:15] <marostegui>	 not a nice feature hehe
[09:09:41] <jynus>	 also, for some reason, s3 socket doesn't come up
[09:09:56] <jynus>	 but s2 and x1 does
[09:10:18] <jynus>	 the server does, but the socket doesn't
[09:10:27] <marostegui>	 no errors on logs?
[09:10:40] <jynus>	 the server comes up normally
[09:10:56] <jynus>	 although I started it the last
[09:12:04] <marostegui>	 I am taking a look too
[09:13:23] <marostegui>	 how is that possible? :-/
[09:13:54] <jynus>	 systemd
[09:14:22] <jynus>	 that option is not supported, so I thought it would only fail, but it is doing weird stuff
[09:14:49] <marostegui>	 yeah, I wonder how mysql was able to start without complaining about the socket
[09:14:58] <marostegui>	 maybe it created it during start and then once it was up, it got deleted
[09:15:11] <jynus>	 because it was created unix-like then deleted something something
[09:19:24] <jynus>	 I am going to rebuild a -1
[09:19:31] <marostegui>	 +1
[09:23:09] <jynus>	 I will also build, but not upload 10.1.44
[09:23:19] <marostegui>	 sounds good yeah
[09:23:20] <jynus>	 could you check the state of mariadb-client on buster?
[09:23:34] <marostegui>	 what do you need?
[09:23:36] <jynus>	 I think it was a blocker for cumin buster upgrade
[09:23:47] <jynus>	 just if there is a buster mariadb client we like
[09:23:50] <jynus>	 on repo
[09:23:52] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['es1020.eqiad.wmnet'] `  Of which those **FAILED**: ` ['es1020.eqiad.wmnet'...
[09:23:58] <marostegui>	 I built it and installed it on db1107
[09:24:07] <jynus>	 the client package?
[09:24:15] <marostegui>	 yeah
[09:24:37] <jynus>	 ok, so not a blocker for when buster is on cumin, right?
[09:24:46] <marostegui>	 nope, I haven't seen anything weird
[09:25:06] <marostegui>	 (I have been actually using it from db1107 precisely to test it) rather than using cumin in most cases
[09:25:20] <jynus>	 if you could find the ticket discussing that and just saying that, it would be great
[09:25:30] <marostegui>	 sure, let me check it
[09:26:29] <marostegui>	 done
[09:27:06] <jynus>	 thanks
[09:27:35] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Marostegui) >>! In T241359#5896178, @ops-monitoring-bot wrote: > Completed auto-reimage of hosts: > ` > ['es1020.eqiad.wmnet'] > `...
[09:42:25] <jynus>	 while I finish, could also ask you how many hosts are on the latest package version?
[09:42:56] <jynus>	 and/or if we could depool one multiinstance for testing the fix?
[09:44:34] <marostegui>	 sure, you can depool db1096:3315, db1096:3316
[09:44:37] <marostegui>	 do you want me to depool it?
[09:44:50] <jynus>	 not yet
[09:44:53] <marostegui>	 (most of the hosts are running 10.1.43)
[09:45:21] <jynus>	 but if there is any blocker to do it (e.g. ongoing maintenance) try to make it possible
[09:45:39] <marostegui>	 nope, that host can be depooled anytime
[09:46:03] <jynus>	 also let's test it first on non-critical hosts
[09:46:10] <marostegui>	 then let's go for codfw
[09:46:25] <marostegui>	 db2089 
[09:50:22] <jynus>	 it us uploaded now, could you help me with that one ^
[09:50:31] <marostegui>	 sure, depooling db2089
[09:50:34] <marostegui>	 and downtiming
[09:51:45] <marostegui>	 you can now proceed
[09:51:46] <jynus>	 what order should we do, first test the issue is solved, then test updating the systemd works withour a restart?
[09:51:55] <marostegui>	 yeah, that sounds good
[09:52:10] <jynus>	 then I prefer to do it on db1140 first
[09:52:18] <jynus>	 then on db2089
[09:52:19] <marostegui>	 up to you, db2089 is fully ready
[09:53:03] <jynus>	 see -ops
[09:53:12] <marostegui>	 sure
[09:54:00] <jynus>	 it will be package upgrade only (no server) so no mysqld_upgrade necessary FYI
[09:54:17] <marostegui>	 oki
[09:54:52] <jynus>	 dpkg: error: dpkg status database is locked by another process
[09:55:10] <marostegui>	 puppet running I guess?
[09:55:21] <jynus>	 you think a fluke?
[09:55:27] <jynus>	 it worked on next run
[09:55:49] <marostegui>	 Feb 19 09:45:58 db1140 puppet-agent[21372]: Applied catalog in 8.72 seconds
[09:55:51] <marostegui>	 so not that
[09:56:04] <jynus>	 could be debmonitor
[09:56:26] <jynus>	 we will see next time
[09:56:38] <volans>	 which host?
[09:56:41] <jynus>	 will stop daemons
[09:56:47] <jynus>	 db1140
[09:57:17] <marostegui>	 ok
[09:58:08] <jynus>	 pgrep ing mysqld as we cannot trust systemd
[09:58:20] <jynus>	 and starting them again, 1 by one
[09:58:24] <marostegui>	 ok
[09:58:49] <jynus>	 I think there may be a bit of a race condition
[09:59:02] <jynus>	 depending how much time between starts, etc
[09:59:15] <marostegui>	 maybe do it all at once with &&?
[09:59:28] <jynus>	 well, I am mostly worried about the stop :-D
[09:59:38] <jynus>	 which is what deletes the working directory
[09:59:51] <jynus>	 because directory preseve not available
[10:00:08] <marostegui>	 hehe yeah
[10:00:25] <jynus>	 things should be now normal
[10:00:27] <volans>	 debmonitor is triggered only by apt hooks and by a daily crontab that on db1140 runs at 17:51
[10:00:30] <volans>	 so should be unrelated
[10:00:42] <jynus>	 will wait for icinga to confirm
[10:02:01] <jynus>	 yeah, it is ok, there is lag but that is because I am running with skip-slave-start
[10:02:03] <jynus>	 now the stop
[10:02:06] <marostegui>	 cool
[10:03:20] <jynus>	 stops looking ok as expected
[10:05:18] <jynus>	 I think the issue could present only if the process is killed
[10:05:28] <jynus>	 which I technically did for s3
[10:05:37] <jynus>	 but still, not something we want
[10:06:22] <jynus>	 let me test the upgrade on db2089 to see if we will need a -3
[10:07:34] <marostegui>	 right
[10:07:47] <jynus>	 12 packages can be upgraded
[10:07:54] <jynus>	 should I do full or just wmf?
[10:08:02] <marostegui>	 nah, do everything I would say
[10:08:07] <jynus>	 it is mostly perl
[10:09:01] <jynus>	 no errors this time
[10:09:16] <jynus>	 so maybe we were doing a few apt manually?
[10:09:45] <jynus>	 post-inst only calls update-alternatives, does nothing with apt
[10:10:10] <jynus>	 what I don't remember is if I do systemd-reload
[10:10:16] <jynus>	 let me check
[10:11:21] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['es1020.eqiad.wmnet']...
[10:11:40] <jynus>	 yeah, it does /bin/systemctl daemon-reload
[10:13:04] <jynus>	 on paper this is fixed
[10:13:46] <marostegui>	 want to try db2089?
[10:13:52] <jynus>	 but if I stop the instances I am not sure we would be able to replicate it
[10:13:52] <marostegui>	 sorry
[10:13:54] <marostegui>	 you did already
[10:14:01] <jynus>	 no, didn't restart yet
[10:14:06] <marostegui>	 ah ok!
[10:14:36] <jynus>	 as in, would a systemd-reload be enough to forget the RunTimeDir?
[10:14:48] <marostegui>	 let's see!
[10:14:50] <jynus>	 or is it tied to the process
[10:15:22] <jynus>	 but if I do a normal shutdown, I am sure it would work in both cases, only a kill/systemd failed may have triggerd it
[10:15:49] <jynus>	 I will at least stop one instance
[10:15:55] <marostegui>	 sure
[10:17:10] <jynus>	 see -ops
[10:17:18] <marostegui>	 ok
[10:18:12] <jynus>	 yeah, no issue as expected
[10:18:48] <jynus>	 I belive the only thing left is to install -2 over all .43
[10:19:03] <marostegui>	 cool
[10:19:10] <marostegui>	 we can do that slowly indeed
[10:19:22] <marostegui>	 although it is mostly needed on multiinstance, no?
[10:19:33] <jynus>	 yeah, the others don't matter
[10:19:41] <marostegui>	 sweet, that can be done easily
[10:19:44] <jynus>	 because even if the dir is removed
[10:19:53] <jynus>	 no harm done
[10:19:56] <marostegui>	 yep
[10:20:08] <jynus>	 I guess it could fail on start?
[10:20:56] <jynus>	 but I think it is created on puppet run and on start
[10:21:06] <marostegui>	 yeah, I was going to say that puppet would take care of that
[10:21:12] <jynus>	 it also wouldn't have affected mediawiki
[10:21:23] <jynus>	 beacuse the servers would be up
[10:21:28] <jynus>	 but all monitoring would fail
[10:21:35] <jynus>	 as it is done based on socket
[10:22:06] <jynus>	 I will create the .44 package
[10:22:25] <marostegui>	 thanks
[10:23:01] <jynus>	 will upload it to install1002, but only to my home
[10:23:35] <marostegui>	 ok
[10:33:34] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['es1020.eqiad.wmnet'] `  and were **ALL** successful.
[10:35:32] <jynus>	 I also checked no more errors on new package d/system/mariadb@.service:77] Unknown lvalue 'RuntimeDirectoryPreserve'
[10:35:44] <marostegui>	 \o/
[10:35:47] <marostegui>	 good job
[10:35:57] <marostegui>	 can I repool db2089?
[10:37:18] <jynus>	 if caught up, yes
[10:37:44] <marostegui>	 yep it is
[10:39:09] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Marostegui)
[10:40:10] <jynus>	 I might upload the 10.1.44 client, though
[10:40:36] <marostegui>	 yeah, that's cool 
[10:41:01] <jynus>	 I left the server on my home
[10:41:05] <jynus>	 for manual install
[10:41:19] <jynus>	 but will not install anyware (10.1.44)
[10:43:12] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Marostegui) es1020 installed correctly: RAID10, 256k strip size, BBU and Cache policy right disk space, memory and cpus
[10:43:55] <jynus>	 this is me right now: https://i.imgur.com/T80xXuA.gifv
[10:45:09] <jynus>	 I need to fic the package, to fix s3 source, to take a backup, to improve the backup system
[10:45:55] <marostegui>	 hhahahahahahaha
[10:45:57] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['es1021.eqiad.wmnet',...
[10:46:00] <marostegui>	 I love that gif
[10:46:46] <jynus>	 it is from the malcom-in-the-middle-breaking-bad cinamatic universe
[10:46:52] <marostegui>	 yeah, I love malcom
[10:54:29] <jynus>	 yay for es102* hosts :-D
[10:59:39] <jynus>	 that's some fast logging
[10:59:54] <marostegui>	 looks like they might have failed
[10:59:58] <marostegui>	 I am seeing some cumin errors
[11:00:06] <jynus>	 3 reinstalls or just 1?
[11:03:05] <marostegui>	 looks only related to downtime
[11:03:41] <jynus>	 did you end up going for buster?
[11:03:45] <marostegui>	 nop
[11:04:03] <marostegui>	 don't want to do expirements with ES hosts and 10.4 :)
[11:04:04] <jynus>	 as I saw some installer issues being fixed, so I thought it was a new process
[11:04:26] <jynus>	 maybe it was different hosts
[11:04:47] <marostegui>	 those fixes were for strecht and all the 1G/10G stuff
[11:04:52] <jynus>	 I see
[11:05:35] <jynus>	 to be fair (although I don't want to overload you) it would be nice to test the buster installer before installing stretch over it
[11:05:54] <marostegui>	 I will see what I can do
[11:06:06] <jynus>	 yeah, we can test it with newer hardware next Q
[11:06:10] <jynus>	 no rush
[11:07:08] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['es1022.eqiad.wmnet', 'es1023.eqiad.wmnet', 'es1021.eqiad.wmnet'] `  and we...
[11:07:13] <marostegui>	 buster installer was tested with db1107 already though
[11:07:16] <jynus>	 also once we have faste recovery systems
[11:07:28] <jynus>	 we would have more margin for errors
[11:12:59] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Marostegui) >>! In T241359#5896474, @ops-monitoring-bot wrote: > Completed auto-reimage of hosts: > ` > ['es1022.eqiad.wmnet', 'es1...
[11:14:20] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Marostegui)
[11:17:43] <jynus>	 codfw prepare is now also failing
[11:18:10] <jynus>	 I wonder if I am targeting this wrong, the issue is ulimit on xtrabackup
[11:18:32] <jynus>	 and not related to mysql at all, and it just went over the 65K limit this month
[11:20:17] <jynus>	 nah, it just has 9 right now
[11:26:14] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Marostegui)
[11:28:36] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['es1024.eqiad.wmnet', 'es1025.eqiad.wmnet']...
[12:28:54] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['es1024.eqiad.wmnet'] `  Of which those **FAILED**: ` ['es1024.eqiad.wmnet'] `
[12:45:54] <jynus>	 db1140:s3 logical backup finishing, running snapshot now, will see how it goes and run compare later
[13:28:45] <jynus>	 lots of commonswiki errors
[13:28:57] <jynus>	 since 13:21
[13:29:32] <jynus>	 db1084
[13:31:41] <jynus>	 ^depooled it
[13:35:18] <wikibugs>	 10DBA, 10Operations: db1084 reboot causing commonswiki connection errors (crash?) - https://phabricator.wikimedia.org/T245621 (10jcrespo)
[13:52:24] <marostegui>	 :(
[13:52:55] <jynus>	 I was about to leave for lunch, either you research or I will later, things are stable
[13:53:00] <marostegui>	 I will take over
[13:53:06] <marostegui>	 Thank you for triagging it
[13:53:29] <jynus>	 the assigning it to you was mostly for noticing purposes
[13:53:30] <marostegui>	 Looks like BBU dioed
[13:53:50] <jynus>	 when I want your attention, I think that is a good way :-D
[13:53:56] <marostegui>	 definitely
[13:54:40] <jynus>	 if it has to be recovered, testing an s4 snapshot would be nice
[13:54:49] <jynus>	 afk
[13:55:05] <wikibugs>	 10DBA, 10Operations: db1084 reboot causing commonswiki connection errors (crash?) - https://phabricator.wikimedia.org/T245621 (10Marostegui) Looks like BBU died: `    Battery/Capacitor Count: 0 `  `  /system1/log1/record15   Targets   Properties     number=15     severity=Caution     date=02/19/2020     time=1...
[13:56:42] <wikibugs>	 10DBA, 10Operations: db1084 reboot causing commonswiki connection errors (crash?) - https://phabricator.wikimedia.org/T245621 (10Marostegui) @wiki_willy do we have spare HP BBUs in eqiad?
[13:57:17] <wikibugs>	 10DBA, 10Operations: db1084 crashed due to BBU failure - https://phabricator.wikimedia.org/T245621 (10Marostegui)
[13:57:33] <wikibugs>	 10DBA, 10Operations: db1084 crashed due to BBU failure - https://phabricator.wikimedia.org/T245621 (10Marostegui) p:05Triage→03Medium
[13:59:57] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review: db1084 crashed due to BBU failure - https://phabricator.wikimedia.org/T245621 (10Marostegui)
[14:02:21] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['es1024.eqiad.wmnet'] ` The log can be foun...
[14:04:05] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Marostegui) es1025: RAID10, 256k strip size, BBU and Cache policy right disk space, memory and cpus looking good.
[14:04:41] <cmjohnson1>	 marostegui thanks for finishing the es servers
[14:04:45] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Marostegui)
[14:05:03] <marostegui>	 cmjohnson1: es1024 doesn't have link I think, it cannot PXE :(
[14:05:07] <marostegui>	 The rest are good
[14:05:57] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Marostegui) @Cmjohnson can you double check es1024's link? It cannot PXE boot: ` Booting from PXE Device 1: Integrated NIC 1 Port 1 Partition 1 PXE: No m...
[14:06:29] <cmjohnson1>	 can you get out of the console for es1024 please
[14:06:35] <marostegui>	 sure
[14:06:48] <marostegui>	 cmjohnson1: done, all yours
[14:07:00] <cmjohnson1>	 thx
[14:20:48] <cmjohnson1>	 marostegui:  it's installing now I will ping you once it's finished.  Boot setting was wrong in bios. 
[14:20:59] <marostegui>	 ah cool!
[14:21:00] <marostegui>	 thank you
[14:21:21] <marostegui>	 cmjohnson1: do you happen to know if we have spare HP BBUs in eqiad?
[14:22:48] <cmjohnson1>	 I do not have any that I know of, there was talk that we were going to be ordering spare but I'm not sure the result of the talk.   
[14:23:05] <marostegui>	 Cool, will talk to Willy then! Thank you!
[14:34:55] <cmjohnson1>	 marostegui the install is done
[14:35:05] <marostegui>	 yay!
[14:35:18] <marostegui>	 cmjohnson1: thanks - I will do some checks and close the task
[14:38:47] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['es1024.eqiad.wmnet'] `  and were **ALL** successful.
[14:39:26] <wikibugs>	 10DBA, 10Operations: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Cmjohnson)
[14:41:32] <wikibugs>	 10DBA, 10Operations: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Marostegui) es1024: RAID10, 256k strip size, BBU and Cache policy right disk space, memory and cpus looking good.
[14:41:45] <wikibugs>	 10DBA, 10Operations: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Marostegui)
[14:41:56] <wikibugs>	 10DBA: Productionize es1020-es1025, es2020-es2025 - https://phabricator.wikimedia.org/T243052 (10Marostegui)
[14:41:59] <wikibugs>	 10DBA, 10Operations: (Needed by 31st January) eqiad: rack/setup/install  es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Marostegui) 05Open→03Resolved All hosts have been installed successfully. Thanks!
[14:42:12] <wikibugs>	 10DBA: Productionize es1020-es1025, es2020-es2025 - https://phabricator.wikimedia.org/T243052 (10Marostegui) 05Stalled→03Open
[14:42:14] <wikibugs>	 10DBA, 10Epic, 10Goal: Setup es4 and es5 replica sets for new read-write external store service - https://phabricator.wikimedia.org/T226704 (10Marostegui)
[14:52:21] <wikibugs>	 10DBA: Productionize es1020-es1025, es2020-es2025 - https://phabricator.wikimedia.org/T243052 (10Marostegui)
[15:27:57] <wikibugs>	 10DBA: Possibly disable optimizer flag: rowid_filter on 10.4 - https://phabricator.wikimedia.org/T245489 (10Anomie) The execution itself doesn't seem to be reading too many rows, ` wikiadmin@10.64.0.214(enwiki)> SELECT /* ApiQueryExtLinksUsage::run */ el_index_60, el_id, page_id, page_namespace, page_title, el_t...
[15:30:40] <wikibugs>	 10DBA: Possibly disable optimizer flag: rowid_filter on 10.4 - https://phabricator.wikimedia.org/T245489 (10Marostegui) >>! In T245489#5897481, @Anomie wrote: > > That suggests whatever is being slow is in the logic for doing the filtering rather than in the row fetching itself.  Yeah, that's why I believe that...
[15:32:06] <jynus>	 2020-02-19 13:04:47 139698789949888 [ERROR] InnoDB: Table csbwiktionary/watchlist_expiry in the InnoDB data dictionary has tablespace id 348750, but tablespace with that id or name does not exist
[15:32:55] <jynus>	 I guess it could be DDL + backup?
[15:33:10] <marostegui>	 what do you mean DDL?
[15:33:31] <jynus>	 that table maybe being created recently
[15:33:46] <marostegui>	 interesting
[15:33:51] <marostegui>	 that could be the on-going compressing I am doing
[15:33:54] <marostegui>	 for that table in s3
[15:34:05] <jynus>	 but not for source mysqls
[15:34:30] <marostegui>	 I am doing it on the master with replication (it is an empty table)
[15:34:35] <jynus>	 ah
[15:34:43] <jynus>	 well, is it still ongoing?
[15:35:04] <marostegui>	 yeah
[15:35:09] <jynus>	 ETA?
[15:35:25] <marostegui>	 end of day
[15:35:35] <jynus>	 I think if I stop replication, it shouldn't affect me
[15:35:47] <jynus>	 I will try like that
[15:36:00] <jynus>	 in any case, that should be a non-fatal error
[15:36:01] <marostegui>	 yeah, it won't :)
[15:36:17] <jynus>	 I am guessing each alerter takes <1 second or so?
[15:36:32] <jynus>	 so I can just stop slave and it will stop being applied
[15:36:48] <jynus>	 (for that host)
[15:38:35] <jynus>	 it is complaining about that, but, however, on a next prepare it compltes ok
[15:38:45] <jynus>	 vs. being stuck for 3+ hours
[15:38:56] <jynus>	 so that I think is "expected"
[15:39:34] <marostegui>	 each alerter?
[15:39:47] <marostegui>	 ah alter
[15:39:48] <marostegui>	 yeah
[15:39:49] <jynus>	 yeah, it sees movements on the transaction logs
[15:39:49] <marostegui>	 they are fast
[15:39:53] <marostegui>	 but I have a big sleep
[15:40:03] <jynus>	 so it gives an error that it will be ignored
[15:40:06] <marostegui>	 because otherwise s3 suffers (because of all the many files..)
[15:40:11] <jynus>	 because it is a table it doesn't know about at the time
[15:40:26] <jynus>	 yeah, your operations are not an issue, just to be clear
[15:40:36] <jynus>	 I am just trying to understand what xtrabackup is doing
[15:40:43] <jynus>	 and I think it is ok it errors out
[15:40:49] <jynus>	 and expected
[15:41:00] <jynus>	 not the root cause of prepare stalling
[15:41:18] <jynus>	 as on the next run, it finished in 10 seconds, with the same "errors"
[15:41:29] <jynus>	 but with an ok output
[15:41:42] <marostegui>	 Maybe it is comparing the original frm with the new one? (compressed)?
[15:42:00] <jynus>	 yeah, it is innodb, but same idea
[15:42:09] <jynus>	 on aleter the tablespace id will be reconstructed
[15:42:25] <jynus>	 so it cannot apply changes to it, so it warns about it
[15:42:39] <jynus>	 but that is ok, those will be applied on replication
[15:43:03] <jynus>	 but let me stop replication to remove that variable
[15:43:08] <marostegui>	 sure thing
[15:43:16] <jynus>	 and potentially understand the real reason for the stalls
[15:43:42] <jynus>	 it is very weird both s3 are failing at around the same time
[15:44:02] <marostegui>	 yeah, it is definitely the compression
[15:44:03] <jynus>	 so I am still thinking something like "number of objects grown over ulimit limit"
[15:44:15] <jynus>	 beacuse new table, or something like that
[15:44:29] <jynus>	 something that makes s3 special
[15:44:35] <jynus>	 vs the other sections
[15:44:58] <marostegui>	 open files or descriptors?
[15:45:00] <marostegui>	 maybe?
[15:45:31] <jynus>	 I thought that,but while stalled, I only saw 9 files open
[15:45:50] <jynus>	 some temporary ones and the logs, it wasn't scanning .ibds
[15:46:00] <jynus>	 but something along those lines is the theory
[15:46:03] <jynus>	 maybe os
[15:46:09] <jynus>	 or maybe mysql config
[15:46:14] <marostegui>	 maybe we should try to strace the process
[15:46:20] <marostegui>	 and see if we see something
[15:46:32] <jynus>	 e.g. open_file_limit for xtrabackup
[15:46:53] <marostegui>	 I guess it will just use the OS default?
[15:47:11] <jynus>	 mysql's default, as we only tune it for a regular mysql
[15:47:18] <jynus>	 but again, those are guesses
[15:47:39] <jynus>	 whenever I run it manually, I cannot reproduce it and backups works ok :-/
[15:48:45] <marostegui>	 root@db1140:~# cat /proc/27041/limits | grep files
[15:48:46] <marostegui>	 Max open files            200001               200001               files
[15:48:51] <marostegui>	 I really hope it is not opening more than 200k XD
[15:49:47] <jynus>	 yeah, but that is the mysql process
[15:49:54] <jynus>	 which is what I meant before
[15:50:04] <jynus>	 xtrabackup is actually ok
[15:50:15] <jynus>	 it is the prepare on dbprov* that might have issues
[15:50:19] <jynus>	 (just a guess)
[15:50:30] <jynus>	 the instance that is started up to run --prepare
[15:51:27] <jynus>	 since they re-arched xtrabackup I stopped looking at its internals
[15:51:54] <marostegui>	 I think strace might help
[15:52:03] <jynus>	 another option is that the upgrade to 10.1.43 was relatively recent
[15:52:10] <jynus>	 could have changed something
[15:52:48] <jynus>	 marostegui: yes if I can get to strace one where it stalls :-D
[15:52:51] <marostegui>	 https://mariadb.com/kb/en/mariadb-10143-changelog/
[15:53:36] <jynus>	 the main issue is not that it fails, it is that it sometimes works, sometimes stalls
[15:54:21] <jynus>	 sorry, don't want to distract you
[15:54:28] <jynus>	 please ignore me on this ranting
[15:55:19] <marostegui>	 maybe we can just strace by default until we get it to fail again
[15:55:27] <marostegui>	 and the see if there's something interesting there
[15:55:52] <jynus>	 also enable verbose logging
[17:00:07] <wikibugs>	 10DBA, 10Operations: db1084 crashed due to BBU failure - https://phabricator.wikimedia.org/T245621 (10wiki_willy) @Marostegui - we have a few spare BBUs in the process of being shipped onsite, one of them for T244958, which should be arriving early next week.  You can just shoot open a dc-ops task with us, and...
[17:47:21] <jynus>	 checking stats, I think thing my serial data check is taking significant resources: https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&var-dc=eqiad%20prometheus%2Fops&var-group=core&var-shard=s3&var-role=All&from=1582123613684&to=1582145213685&fullscreen&panelId=8
[17:47:31] <jynus>	 *is NOT taking
[18:19:51] <wikibugs>	 10DBA, 10Expiring-Watchlist-Items, 10Community-Tech (Kanban-Q3-2019-20), 10Core Platform Team Workboards (Clinic Duty Team), 10MW-1.35-notes (1.35.0-wmf.19; 2020-02-11): Create required table for new Watchlist Expiry feature - https://phabricator.wikimedia.org/T240094 (10Mooeypoo) 05Open→03Resolved T...
[19:03:18] <Amir1>	 marostegui: Hey, I have some question about s8, how much read should be considered as unhealthy? I know it's too general. Let me give you some context: We are gradually increasing the read on the new term store, when it was suddenly moved, it brought down everything but there are so many moving parts (like several performance improvements we recently deployed) that we can't say we need to do more or not. I want to gradaully increase 
[19:03:18] <Amir1>	 it
[19:03:45] <Amir1>	 but stop and do more once we see some red flags in s8
[19:12:28] <Amir1>	 there's one rather low hanging fruit left to improve but we get to it if needed
[20:31:50] <wikibugs>	 10DBA, 10Data-Services, 10MediaWiki-General, 10Security, 10Security Related: Make (redacted) log_search table available on Wiki Replicas - https://phabricator.wikimedia.org/T85756 (10bd808)