[07:18:13] <marostegui>	 volans: I am trying the prepare cookbook, and I am getting: https://phabricator.wikimedia.org/P74236
[07:20:16] <marostegui>	 And for what is worth, I am able to connect just fine from cumin1002 CLI
[07:29:12] <marostegui>	 Same from cumin2002
[07:36:47] <marostegui>	 The credentials at ./etc/spicerack/cookbooks/sre.switchdc.databases.yaml look good
[08:23:11] <_joe_>	 marostegui: did you look at the cert?
[08:23:59] <_joe_>	 i suspect somehow our cert bundle doesn't get read by python? but it's strange
[08:24:02] <marostegui>	 _joe_: I was trying with test-s4 and that one looks like works fine (there were some issues, but not related to the connection, and that was also expected)
[08:24:40] <marostegui>	 _joe_: We've not changed the cert since the last switchover, but I wasn't present so I don't know if this was something that also happened and was workarounded 
[08:27:01] <_joe_>	 marostegui: what's that server role?
[08:27:15] <marostegui>	 _joe_: The databases affected?
[08:27:21] <_joe_>	 yes
[08:27:36] <marostegui>	 mariadb::core
[08:27:44] <marostegui>	 They are all the same
[08:28:43] <_joe_>	 the cert has been generated last april
[08:28:59] <marostegui>	 so yes, no change since the last switch
[08:30:45] <_joe_>	 where is the config of the db server located?
[08:30:58] <marostegui>	 In /etc/my.cnf
[08:46:25] <_joe_>	 marostegui: sorry, do you know if it was connecting to port 3306 or 3307?
[08:46:56] <_joe_>	 nevermind, I can connect to both
[08:47:01] <_joe_>	 🤔
[08:47:14] <marostegui>	 :-/
[08:47:39] <marostegui>	 I can try with a different section to see if it is something specific for that host, given that test-s4 worked
[08:49:35] <_joe_>	 marostegui: I found the problem
[08:49:42] <marostegui>	 what is it?
[08:49:46] <_joe_>	 it's in spicerack, newer pymysql version
[08:50:11] <_joe_>	 conn = pymysql.Connection(host='db2196.codfw.wmnet', db='test', ssl={"ca": "/etc/ssl/certs/Puppet_Internal_CA.pem"}, port=3307) gives the error you get
[08:50:33] <_joe_>	 conn = pymysql.Connection(host='db2196.codfw.wmnet', db='test', ssl={"ca": "/etc/ssl/certs/Puppet_Internal_CA.pem"}, port=3307) does not
[08:50:37] <_joe_>	 err sorry
[08:50:52] <_joe_>	 conn = pymysql.Connection(host='db2196.codfw.wmnet', db='test', ssl_ca= "/etc/ssl/certs/Puppet_Internal_CA.pem", port=3307) does not
[08:51:46] <_joe_>	 but this means that any cookbooks for dbs would fail...
[08:52:15] <marostegui>	 It took me a while to find the difference
[08:52:25] <marostegui>	 between both lines
[08:54:21] <volans>	 hello
[08:54:29] <volans>	 what did I do? :D
[08:54:29] <marostegui>	 ciao volans
[08:55:11] <marostegui>	 _joe_: I wonder how test-s4 worked then
[08:55:18] <marostegui>	 They do have a different role
[08:55:21] <marostegui>	 so maybe that's why
[08:55:45] <volans>	 we set "ssl": {"ca": PUPPET_CA_PATH},
[08:55:50] <volans>	 in pymysql's config
[08:55:56] <_joe_>	 volans: yes that doesn't work anymore
[08:56:28] <volans>	 cumin hosts hvae bullseye pymysql (1.0.2-2)
[08:56:44] <volans>	 that's from 2021...
[08:57:18] <_joe_>	 volans: yes and pydoc seems to indicate it's still supported
[08:57:37] <volans>	 I wonder if with the refactor of the classes I did a mistake, checking both code and git history, give me a minute to understand 
[08:58:31] <_joe_>	 volans: conn = pymysql.Connection(host='db2196.codfw.wmnet', db='test', ssl={"ssl_ca": "/etc/ssl/certs/Puppet_Internal_CA.pem"}, port=3307) works
[08:59:14] <volans>	 ok so we can change ca with ssl_ca, but I need to understand if we can just change it or we need to do it conditionally only in some cases
[08:59:16] <_joe_>	 volans: the manual says: 
[08:59:19] <_joe_>	 :param ssl:
[08:59:19] <_joe_>	  |      A dict of arguments similar to mysql_ssl_set()'s parameters.
[09:01:06] <_joe_>	 which... shouldn't need the ssl_ prefix...
[09:01:53] <volans>	 marostegui: by any chance was the mysql/mariadb client updated on the cumin hosts?
[09:02:10] <marostegui>	 volans: I haven't done so 
[09:02:19] <_joe_>	 ahhh no volans mine is a red herring I think
[09:02:27] <marostegui>	 I am not sure whether moritzm did or they simply got shipped via updates
[09:02:42] <volans>	 I see wmf-mariadb105-client
[09:03:09] <marostegui>	 Yeah, we never install our packaged version there, as the client is good enough as it is shipped
[09:03:41] <volans>	 that's on the cumin hots
[09:03:45] <marostegui>	 yep
[09:03:56] <marostegui>	 I don't think it could be the client, as test-s4 worked fine
[09:04:01] <volans>	 k
[09:04:02] <marostegui>	 With the cookbook
[09:04:02] <_joe_>	 volans: sorry, mine was a red herring
[09:04:17] <volans>	 ok, ignoring your backlog then :)
[09:04:22] <_joe_>	 basically if you use the ssl_ca= syntax, you also need to declare ssl_verify_cert=True
[09:04:30] <marostegui>	 Note that test-s4 hosts have a different role: core_test
[09:04:31] <_joe_>	 🤦
[09:05:00] <_joe_>	 marostegui: I have the problem verifying the cert even from pure python cli
[09:05:06] <_joe_>	 just doing pymysql.connect
[09:05:15] <marostegui>	 that's so weird then, cause there was no connection error there
[09:05:15] <_joe_>	 so the problem seems to be indeed with the server cert
[09:05:25] <_joe_>	 oh it's not a connection error
[09:05:35] <_joe_>	 the client can't verify the identity of the server cert
[09:05:42] <_joe_>	 somehow
[09:05:48] <marostegui>	 _joe_: Yeah, but why does it work with those hosts, which run same version and same OS
[09:06:03] <_joe_>	 which hosts?
[09:06:11] <marostegui>	 test-s4 hosts, so db1176 and db2230
[09:06:59] <_joe_>	 ok I think I found the problem
[09:07:24] <_joe_>	 looks like the server has a cert that is signed with a ca that is in /etc/ssl/certs/wmf-ca-certificates.crt but not the puppet CA
[09:07:29] <moritzm>	 no, nothing has been upgraded related to mariadb in the recent weeks
[09:10:24] <_joe_>	 ok yeah connecting to the server directly
[09:10:36] <_joe_>	 $ openssl s_client -starttls mysql -connect  db2196.codfw.wmnet:3306
[09:10:49] <_joe_>	 i:CN = Puppet CA: palladium.eqiad.wmnet
[09:10:57] <_joe_>	 v:NotBefore: Feb 26 16:12:28 2024 GMT
[09:11:47] <marostegui>	 palladium? wasn't that decommissioned years ago?
[09:11:48] <_joe_>	 now
[09:11:52] <_joe_>	 $ openssl x509 -text -in /etc/mysql/ssl/cert.pem
[09:12:00] <_joe_>	 Not Before: Apr 22 10:33:31 2024 GMT
[09:12:20] <_joe_>	 CN=Wikimedia_Internal_Root_CA
[09:12:44] <_joe_>	 so the server is definitely running with a cert that's not the one on disk
[09:12:51] <volans>	 it never got reloaded?
[09:12:54] <_joe_>	 marostegui: that's the old CA
[09:13:11] <marostegui>	 marostegui@db2196:~$ uptime
[09:13:12] <marostegui>	  09:13:06 up 383 days
[09:13:33] <_joe_>	 yeah
[09:13:43] <_joe_>	 it needs to be restarted to pick up the new PKI
[09:14:00] <volans>	 didn;t mysql got the reload ssl stuff since a certain version?
[09:14:01] <_joe_>	 I see db2230 has the correct cert signed by the new puppet CA
[09:14:03] <marostegui>	 I can do a switchover now easily, or we can go ahead and try with also s6
[09:14:18] <volans>	 without the need of a restart
[09:14:20] <_joe_>	 marostegui: sorry but is test-s4 in production?
[09:14:26] <_joe_>	 volans: no idea
[09:14:27] <marostegui>	 _joe_: "production"
[09:14:38] <_joe_>	 marostegui: I mean do you need a switchover?
[09:14:53] <_joe_>	 in any case, I'll leave it to you, we need to reload the certs, basically
[09:14:56] <_joe_>	 then you should be fine
[09:16:07] <marostegui>	 volans: we have https://mariadb.com/kb/en/flush/#flush-ssl
[09:16:10] <marostegui>	 but I've never used it
[09:16:43] <marostegui>	 _joe_: I need a switchover if I have to stop mariadb on x1 master
[09:17:02] <marostegui>	 Let me try to run the cookbook on s6 then and see if it works as expected
[09:17:05] <_joe_>	 the alternative is
[09:17:27] <volans>	 we can also hotpatch spicerack
[09:17:34] <volans>	 if that helps
[09:17:38] <_joe_>	 we change the PUPPET_CA_PATH in spicerack to use the more inclusive path
[09:17:53] <volans>	 and then fix the thing properly
[09:18:06] <marostegui>	 The switchover shoulnd't take too long though, so as you guys prefer
[09:18:09] <_joe_>	 which is /etc/ssl/certs/wmf-ca-certificates.srt
[09:18:11] <marostegui>	 For now, ok to run this on test-s6?
[09:18:14] <marostegui>	 Sorry, s6
[09:18:15] <_joe_>	 yes
[09:18:23] <volans>	 will it have the same issue?
[09:18:27] <_joe_>	 no
[09:18:29] <marostegui>	 volans: it shouldn't
[09:18:33] <_joe_>	 it's just one server afaict
[09:18:43] <marostegui>	 Oh, wait
[09:18:43] <volans>	 try and let's see
[09:18:48] <marostegui>	 root@db2214:~# w
[09:18:48] <marostegui>	  09:18:38 up 345 days,
[09:18:49] <marostegui>	 maybe? XD
[09:18:55] <volans>	 because if you changed the CA everywhere
[09:19:05] <volans>	 it might not be the only one not reloaded
[09:19:28] <marostegui>	 | ssl_ca        | /etc/ssl/certs/wmf-ca-certificates.crt |
[09:19:31] <volans>	 let me try to connect to all masters and check
[09:28:18] <volans>	 marostegui: I get ssl failures for db2165, db2179, db2192, db2196 and db2204 among all the masters of core sections across both DCs
[09:28:32] <volans>	 sorry db2192 is ok, ignore it, bad paste
[09:29:08] <marostegui>	 volans: Also in eqiad?
[09:29:16] <marostegui>	 I am surprised if you'd get a failure for db1173 for instance
[09:29:26] <marostegui>	 As that was rebooted like a couple of weeks ago
[09:29:26] <volans>	 no it worked
[09:29:30] <marostegui>	 ah,ok
[09:29:43] <volans>	 i can check also the replicas if you want
[09:29:47] <marostegui>	 eqiad ones should be a lot better as they've been rebooted lately
[09:29:47] <volans>	 to get a better picture
[09:29:50] <marostegui>	 volans: No, it is fine
[09:29:54] <volans>	 yeah eqiad is all good
[09:29:59] <marostegui>	 es6 and es7 should also be fine
[09:30:04] <marostegui>	 for codfw as well
[09:30:15] <marostegui>	 So with all those masters, maybe it is easier then to patch the code
[09:30:18] <volans>	 I check all CORE_SECTIONS
[09:30:23] <volans>	 so ('s6', 's5', 's2', 's7', 's3', 's8', 's4', 's1', 'x1', 'es6', 'es7')
[09:30:39] <marostegui>	 And we'll get them fixed as soon as codfw is depooled (they need to get rebooted for kernel and mariadb upgrade)
[09:31:09] <marostegui>	 volans: Right, so es hosts are fine, so I am tempted to run the cookbook for es6 and es7 to confirm it is all good
[09:31:38] <volans>	 as you want, in the meanwhile I'll check if a single value for the CA works on both set of hosts
[09:31:46] <volans>	 or we need to hardcode some logic to pick the right one
[09:31:55] <marostegui>	 volans: Thanks, it should be just temporary to bypass this situation
[09:31:57] <marostegui>	 I will run for es6
[09:32:00] <volans>	 k
[09:32:14] <marostegui>	 I can disable writes on es6 to be fully sure that we don't break anything
[09:32:17] <marostegui>	 I will do that
[09:34:03] <marostegui>	 I will push https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1128351/
[09:34:33] <volans>	 sure, extra safety
[09:35:31] <marostegui>	 deploying
[09:41:02] <volans>	 marostegui: confirmed with "/etc/ssl/certs/wmf-ca-certificates.crt" I can connect to all, so I'll just hotpatch spicerack for you to use that one
[09:41:11] <marostegui>	 volans: thank you!
[09:41:36] <marostegui>	 scap still running
[09:45:02] <volans>	 hotpatch applied, retested, I can connect to all masters
[09:45:11] <marostegui>	 nice!
[09:45:37] <marostegui>	 volans: Let's try with es6 as writes will be disabled, and if it all goes, I will go for x1 again then
[09:45:49] <volans>	 SGTM
[10:01:08] <marostegui>	 [11:00:54]  <+logmsgbot> !log marostegui@deploy2002 Finished scap sync-world: Backport for [[gerrit:1128351|db-production.php: Disable writes on es6 (T388626)]] (duration: 23m 25s)
[10:01:09] <stashbot>	 T388626: Prepare databases circular replication for the DC switchover - https://phabricator.wikimedia.org/T388626
[10:01:10] <marostegui>	 going for es6
[10:01:58] <volans>	 wow it took its time
[10:01:59] <volans>	 (scap)
[10:02:45] <marostegui>	 worked like a charm the script
[10:02:48] <marostegui>	 https://orchestrator.wikimedia.org/web/cluster/alias/es6
[10:03:14] <volans>	 glad it did :)
[10:04:42] <marostegui>	 Going to revert es6 writes then
[10:04:47] <marostegui>	 And yeah, scap took quite long
[10:06:13] <marostegui>	 volans: es7 was done finely too, going for x1, the problematic one!
[10:06:21] <volans>	 finger crossed
[10:07:11] <marostegui>	 volans: all good! 
[10:07:17] <marostegui>	 _joe_ volans thanks a lot for the help!
[10:07:51] <volans>	 <3 sorry for the troubkle, I'm sending a fix to spicerack, we can probably replace everywhere the puppet path with the bundle cert, but I need to test it first
[10:07:59] <jynus>	 So what is/was the one line summary regarding the certs?
[10:08:35] <_joe_>	 jynus: cert and ca were changed, mysql not restarted/ssl not reloaded
[10:08:53] <volans>	 but in general, using /etc/ssl/certs/wmf-ca-certificates.crt is safer as it includes both the new PKI CA and the old puppet CA
[10:09:21] <_joe_>	 volans: indeed
[10:11:11] <jynus>	 volans: I think also then the TLS error was not caught because only recently mysql proto was used for connecting to mysql (before it was ssh + local socket)
[10:11:43] <jynus>	 in any case the checks did the right thing, which is abort the operation :-D
[10:19:46] <jynus>	 wasn't there use to be a test mode without actual action, or I am confusing it with something else?
[10:22:10] <marostegui>	 volans: We will probably have the same issue with sre.switchdc.databases.finalize ?
[10:22:59] <volans>	 with the hotpatch it should work anyway
[10:23:17] <volans>	 I'm preparing the fix but I'm not sure if we will deploy this week before the switchdc to not alter what they have already tested
[10:23:25] <volans>	 there are other changes in master pending release
[10:23:43] <marostegui>	 volans: We will run the finalize on thursday, so it is fine to leave it like this till thursday
[10:23:45] <volans>	 I plan to have the fix merged by today so that if we release you have the fix, if we don't release you have the hotpatch
[10:24:00] <volans>	 in both cases you should be covered
[10:24:08] <marostegui>	 es6 back to being writable 10:21:58 Finished scap sync-world: Backport for [[gerrit:1128356|Revert "db-production.php: Disable writes on es6"]] (duration: 14m 41s)
[10:37:28] <jynus>	 could someone test a regular ssh connection to dbprov2005. Just checking it is not only me and icinga?
[10:37:40] <jynus>	 (that I cannot connect)
[10:38:16] <jynus>	 but I see the host up and responding from the DRAC
[10:39:03] <jynus>	 ah, no need, it says so: "The Integrated NIC 1 Port 1 network link is down."
[10:39:22] <volans>	 both cumin and from me locally hangs
[10:39:38] <jynus>	 yeah, network is down, host is "fine"
[10:40:17] <jynus>	 I will try a greceful reboot to make sure it is not a one 1 thing then file a task for faulty cable, card or switch port/config
[10:40:24] <jynus>	 *one time
[10:40:28] <jynus>	 thanks, volans
[10:41:07] <jynus>	 the fact that I saw a disk getting evicted and united around the same time does look suspicious
[10:41:34] <jynus>	 ignore also that part, it is from 1 year ago
[10:58:11] <jynus>	 I filed T389052
[10:58:11] <stashbot>	 T389052: hw troubleshooting: network link loss of dbprov2005 - https://phabricator.wikimedia.org/T389052
[11:22:37] <Emperor>	 Amir1: I see some SVG changes going past from you - are you aware of https://en.wikipedia.org/wiki/Wikipedia:SVG_help#Rendering_issue ?
[11:23:11] <Emperor>	 I've checked the relevant thumbnail containers and they're working so it's not a repeat of T383053 (which is where it was reported to me)
[11:23:12] <stashbot>	 T383053: Container dbs for wikipedia-commons-local-thumb.f8 AWOL in codfw due to corruption - https://phabricator.wikimedia.org/T383053
[11:24:27] <Amir1>	 Emperor: funnily enough, this is one day before deployment of my change so it can't be caused by it (I just deployed the svg change)
[11:24:52] <Amir1>	 until an hour ago, all svg files were bypassing thumbnail steps
[11:25:33] <Emperor>	 'cos trying to visit the thumbs of e.g. https://en.wikipedia.org/wiki/File:%E0%A6%8F%E0%A6%B6%E0%A7%80%E0%A6%AF%E0%A6%BC_%E0%A6%AE%E0%A6%BE%E0%A6%B8_%E0%A6%B0%E0%A7%8C%E0%A6%AA%E0%A7%8D%E0%A6%AF%E0%A6%AA%E0%A6%A6%E0%A6%95.svg is returning me an internal server error
[11:26:21] <Emperor>	 swift list --prefix c/c6/এশীয়_মাস_রৌপ্যপদক.svg wikipedia-commons-local-thumb.c6 finds me small thumbs (25,38,50,60,106,150,159) but nothing larger 
[11:27:01] <Amir1>	 that's definitely not related to my change then, I do stuff in mw only
[11:27:11] <Emperor>	 Hm, maybe thumbor is sad then
[11:27:26] <Amir1>	 I was about to say, maybe a thumbor issue or an encoding issue?
[11:27:31] <Amir1>	 it's mojibake for me
[11:27:46] <Amir1>	 https://usercontent.irccloud-cdn.com/file/7vMU2CqG/image.png
[11:28:21] <Amir1>	 I tested on these svg files and both seem to work fine (and follow steps now) https://test.wikipedia.org/wiki/Inactive_Bot_Statistics
[11:28:51] <Emperor>	 inkscape opens the .svg for me OK
[11:29:37] <Emperor>	 Amir1: you sure that's not a font issue? filename works OK for me in my browser (but my terminal lacks the necessary font)
[11:29:55] <Amir1>	 yeah it's probably a font issue
[11:29:58] <Amir1>	 but meh :D
[11:30:10] <Amir1>	 It wouldn't crack the top 200 list of issues I need to handle
[11:31:01] <Emperor>	 I'll make a ticket and tag thumbor. Is there some wikimarkup to make the little "reported as Phab: XXX" on village pump?
[11:31:16] <Amir1>	 yeah, I think Template:Tracked
[11:31:41] <Amir1>	 https://en.wikipedia.org/wiki/Template:Tracked
[11:34:30] <Emperor>	 TY
[11:35:44] <Emperor>	 hnowlan: you OK to eyeball T389060 please?
[11:35:45] <stashbot>	 T389060: Thumbnail failures on some SVGs - https://phabricator.wikimedia.org/T389060
[11:41:41] <Emperor>	 amusingly, whilst my terminal lacks the relevant font, it does C&P without loss, so that ticket makes it look like my terminal can do all the characters OK
[18:22:30] <hnowlan>	 Emperor: sorry, I am OOO today - looks like something a bit deeper than just thumbor. requests aren't even hitting thumbor