[05:56:09] The MariaDB Foundation is pleased to announce the availability [05:56:10] of MariaDB 10.5.0, the first alpha release in the new [05:56:10] development series, MariaDB 10.5. [05:57:19] 10DBA, 10Performance-Team, 10conftool: #dbctl: manage 'externalLoads' data - https://phabricator.wikimedia.org/T229686 (10Marostegui) Thank you! :-) [05:57:41] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) [06:10:39] 10DBA: Decommission db2070.codfw.wmnet - https://phabricator.wikimedia.org/T239684 (10Marostegui) [08:58:32] 10DBA, 10Data-Services: Reimport wikidatawiki.{pagelinks,page} on labsdb1010 - https://phabricator.wikimedia.org/T238399 (10Marostegui) `wikidatawiki.page` has been reimported from labsdb1012. Views for that table have been recreated as well. [08:58:40] 10DBA, 10Data-Services: Reimport wikidatawiki.{pagelinks,page} on labsdb1010 - https://phabricator.wikimedia.org/T238399 (10Marostegui) [10:16:19] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: preform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10jbond) [10:17:21] marostegui: the CA change has been merged now are you able to test some of the mariadb systems? [10:17:30] yep [10:17:35] going to do that [10:17:49] "preform" [10:17:50] cheers i also created https://phabricator.wikimedia.org/T239791 to track progress [10:18:36] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10jbond) p:05Triage→03Normal [10:18:45] thats one of my common ones like does and dose [10:34:41] marostegui: hey, let me know when you're around [10:34:49] Amir1: will do [10:34:55] Thanks! [10:45:20] 10DBA, 10Operations, 10Patch-For-Review, 10Puppet, 10User-jbond: Extend Puppet CA Expiry date - https://phabricator.wikimedia.org/T236277 (10Marostegui) Checks performed: On a standalone slave (db2071) with no action on its master: - Run puppet - Stop slave ; start slave; - Stop MySQL, start MySQL Repl... [10:45:22] jbond42 jynus ^ [10:46:16] great thanks marostegui [10:47:29] which cert is using, the new one? [10:48:55] yes [10:49:00] did you also try enabling ssl-verify-server-cert? [10:49:09] No, I didn't try that [10:49:25] so not required, just curious if that would break it [10:50:12] (like in Mythbusters, first you try that it works, then you try to break it :-D) [10:50:33] XDDD [10:58:04] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [11:01:45] 10DBA: Switchover s8 primary database master db1109 -> db1104 - Date TBD - https://phabricator.wikimedia.org/T239238 (10Marostegui) [11:01:47] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [11:14:29] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10jcrespo) I wonder if some of these could be done on reimage, if/when there is one planned anyway. [11:17:43] this may actually interest you: https://gerrit.wikimedia.org/r/c/operations/puppet/+/554485 [11:19:22] ooooh nice!! [11:19:26] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) Probably, some of them will probably be covered with the reimage to buster I would say. [12:01:40] 10DBA, 10Operations, 10Patch-For-Review, 10Puppet, 10User-jbond: Extend Puppet CA Expiry date - https://phabricator.wikimedia.org/T236277 (10jbond) The new certificate has been distributed ` % sudo cumin 'A:all' 'openssl x509 -in $(sudo puppet config print localcacert 2>/dev/null) -noout -enddate ' ====... [12:03:48] 10DBA, 10Operations, 10Patch-For-Review, 10Puppet, 10User-jbond: Extend Puppet CA Expiry date - https://phabricator.wikimedia.org/T236277 (10jbond) get the same results from the following `sudo cumin 'A:all' 'openssl x509 -in /etc/ssl/certs/Puppet_Internal_CA.pem -noout -enddate '` [12:50:35] marostegui: jynus hey, if you're around and have time: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/554496 [13:17:56] Amir1: checking [13:18:22] we might need to revert it later today [13:18:31] but that's so far what I have got [13:18:51] I have merged it [13:19:54] marostegui: thanks. Can you run puppet agent on mwmaint1002? [13:20:12] sure thing [13:21:38] Notice: /Stage[main]/Profile::Mediawiki::Maintenance::Wikidata/Cron[wikidata-rebuildItemTerms]/ensure: removed [13:28:38] Thanks [13:48:27] 10DBA: Automate DB upgrades - https://phabricator.wikimedia.org/T239814 (10Marostegui) [13:49:05] 10DBA: Automate DB upgrades - https://phabricator.wikimedia.org/T239814 (10Marostegui) p:05Triage→03Normal [13:59:09] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [14:02:47] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [16:58:35] 10DBA, 10Operations, 10Patch-For-Review, 10Puppet, 10User-jbond: Extend Puppet CA Expiry date - https://phabricator.wikimedia.org/T236277 (10jbond) [17:15:25] 10DBA, 10Dumps-Generation, 10MW-1.35-notes (1.35.0-wmf.10; 2019-12-10): Some mw snapshot hosts are accessing main db servers - https://phabricator.wikimedia.org/T143870 (10jcrespo) p:05Normal→03Unbreak! [17:22:36] 10DBA, 10Dumps-Generation, 10MW-1.35-notes (1.35.0-wmf.10; 2019-12-10): Some mw snapshot hosts are accessing main db servers - https://phabricator.wikimedia.org/T143870 (10jcrespo) p:05Unbreak!→03High [18:04:36] 10DBA, 10Operations, 10Patch-For-Review, 10Puppet, 10User-jbond: Document all uses of the puppetCA certificate - https://phabricator.wikimedia.org/T237259 (10jbond) [18:12:17] For when you have time: https://gerrit.wikimedia.org/r/c/operations/puppet/+/554583 [18:35:54] 10DBA, 10TechCom-RFC: MediaWiki database policy and/or guidelines (2019) - https://phabricator.wikimedia.org/T220056 (10Milimetric) Moving to last call, @Krinkle or @Nikerabbit to incorporate comments from T220056#5644705. [19:04:34] 10DBA, 10Dumps-Generation, 10MW-1.35-notes (1.35.0-wmf.10; 2019-12-10): Some mw snapshot hosts are accessing main db servers - https://phabricator.wikimedia.org/T143870 (10jcrespo) I am seeing db1118 serving dumps. This is a high-throughput main-traffic enwiki replica. I thought at first this was the cause o... [19:09:05] 10DBA, 10Dumps-Generation, 10MW-1.35-notes (1.35.0-wmf.10; 2019-12-10): Some mw snapshot hosts are accessing main db servers - https://phabricator.wikimedia.org/T143870 (10Marostegui) For what is worth - confirming that db1118 is not configured on vslow/dumps: ` root@cumin1001:~# dbctl instance db1118 get {... [19:10:03] 10DBA, 10Dumps-Generation, 10MW-1.35-notes (1.35.0-wmf.10; 2019-12-10): Some mw snapshot hosts are accessing main db servers - https://phabricator.wikimedia.org/T143870 (10ArielGlenn) Do either of you have any of the queries run? And which host(s) were they from? If this is new and different from the issue p... [19:17:48] https://mariadb.org/mariadb-10-5-0-now-available/ [19:42:11] 10DBA, 10Dumps-Generation, 10MW-1.35-notes (1.35.0-wmf.10; 2019-12-10): Some mw snapshot hosts are accessing main db servers - https://phabricator.wikimedia.org/T143870 (10Ladsgroup) Can this be causing spikes desribed in {T229407} as well? Some graphs to look at: - [[https://grafana.wikimedia.org/d/00000010... [19:46:29] 10DBA, 10Phabricator, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Development services): Switch phabricator production to codfw - https://phabricator.wikimedia.org/T164810 (10mmodell) [20:28:02] 10DBA, 10Dumps-Generation, 10MW-1.35-notes (1.35.0-wmf.10; 2019-12-10): Some mw snapshot hosts are accessing main db servers - https://phabricator.wikimedia.org/T143870 (10ArielGlenn) We don't try to render anything. Not for wikidata entity dumps nor the xml dumps. So I don't have any good ideas about that. [23:27:05] 10DBA, 10Wikimedia-production-error: MediaWiki: "host db1062 is unreachable" (Connection refused) - https://phabricator.wikimedia.org/T239874 (10Krinkle) [23:27:57] 10DBA, 10Wikimedia-production-error: MediaWiki: "host db1062 is unreachable" (Connection refused) - https://phabricator.wikimedia.org/T239874 (10Krinkle) Ping {T239188} >>! In T239188#5708209, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-operations), href=https://tools.wmflabs.org/sal/...