[08:41:47] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Kormat) [08:51:21] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Kormat) [08:54:17] 10DBA, 10Operations, 10User-Kormat: orchestrator: Get packages into WMF apt - https://phabricator.wikimedia.org/T266023 (10MoritzMuehlenhoff) > @MoritzMuehlenhoff: Is it acceptable to download the pre-build .debs, and upload them into our apt repo? We do have packages we source from external repos, but they... [08:56:27] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Kormat) [09:01:57] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Kormat) [09:05:02] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Kormat) [09:08:16] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Kormat) [09:10:25] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Kormat) [09:12:28] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Kormat) [09:12:55] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Kormat) Replication re-enabling tracked here: https://phabricator.wikimedia.org/P13049 [09:18:23] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Marostegui) Thanks Stevie for working on that! We have double checked independently and replication is running ,without GTID everywhere within core on both directions: ` db1081... [10:12:31] marostegui: FYI https://security-tracker.debian.org/tracker/CVE-2020-15180 [10:13:16] \o/ [10:13:42] we are not in 10.1.45 [10:13:43] i don't see any details anywhere what the security issue was, but.. [10:13:56] It is probably the galera thing I think [10:13:57] Let me check [10:14:55] yeah [10:14:57] the galera issue [10:15:03] how can you tell? [10:15:18] it was also the same bug that they fixed on 10.4.15 (they released it just for that) [10:15:37] We should probably notify WMCS for their galera installation, not sure if they use 10.1 or 10.4 or what [10:15:44] kormat: https://jira.mariadb.org/browse/MDEV-23884 [10:16:08] kormat: do you want me to notify them or you do it? [10:16:15] ah. i saw that. but the lack of details isn't very useful [10:16:40] marostegui: i couldn't possibly take the pleasure away from you; be my guest ;) [10:16:44] haha [10:17:03] kormat: yeah, the Debian page is moderately useful on its own [10:21:20] created a task for them [10:35:16] see my thougts on check tables: https://phabricator.wikimedia.org/T265866#6571046 [10:37:12] I will also work on blocker T266125, but will wait until we can do maintenance again [10:37:13] T266125: Drop table profiling from WMF wiki mariadb servers - https://phabricator.wikimedia.org/T266125 [10:37:38] if we way "stop maintenance" droping tables seems like a bad idea [10:37:41] *say [10:38:43] so I will finish the first manual check then consult you on how to go about long-term planning [11:00:24] I have commented on the task [11:00:37] Going to create to temporary indexes on recentchanges on db1077 (testing host) [11:00:38] thanks, I think we are in agreement [11:00:55] I need to provide some info to mariadb about an open bug with the optimizer [11:01:07] cool, that is why testing host are for [11:01:16] *what [11:02:30] kormat maybe working on db2093? [11:03:30] CRIT: read_only: False, expected True, although could be monitoring need changing [11:04:11] that actually needs changing, as db2093 will need to be writable once we deploy orchestrator [11:04:21] makes sense [11:04:50] That's a Tendril DB, right? [11:04:56] yes [11:05:41] acked the alert for now [11:07:09] sobanski: if I understood correctly, I think manuel or stevie are working on repurposing or adding orchestrator on production on what was the passive/standby tendril host [11:07:32] correct [11:07:34] I really need to know more about your goals, please tell me on next meeting [11:07:50] so I am more aware, I am a bit disconnected after vacations [11:09:12] There is not much to share at this point, other than we are planning to use db2093 for the backend T266003, a vm for the central node (T265982) and the debian packages provided by orchestrator T266023 [11:09:13] T266023: orchestrator: Get packages into WMF apt - https://phabricator.wikimedia.org/T266023 [11:09:13] T266003: orchestrator: Select backend database solution - https://phabricator.wikimedia.org/T266003 [11:09:13] T265982: eqiad: New ganeti instance for orchestrator installation - https://phabricator.wikimedia.org/T265982 [11:09:33] We have a brainstorming doc, but I don't think it is worth sharing yet, as it is all very early stages [11:10:25] oh, don't need the details, I just wanted to know, that of the pile of things we should be doing, which ones were left for later [11:10:38] so I don't bother you about it :-D [11:11:22] The above tasks have some more subtasks and the outcome is that hopefully we'll be able to monitor one section from production, we are aiming for pc1, cause if replication breaks we don't care, if data breaks in codfw we don't care etc [11:11:34] that's cool [11:11:56] I will like to play with it on production [11:12:01] when it is setup [11:12:24] play as in "see how it works" [11:20:02] Thanks jira mariadb: The entered text is too long. It exceeds the allowed limit of 100000 characters. [11:25:47] ha ha [11:26:02] doesn't it have an attachment feature maybe? [11:26:12] I was looking for it, but it is gone [11:26:21] I remember I have used it before, but I wasn't finding it [11:26:32] I don't like that editor either, I like mw or phabricator, or even many of the rst-based ones, but I hate jira [11:26:34] Just pasted it on phabricator paste and sent the link [11:26:56] It is terrible [11:27:24] were we going to migrate to jira? [11:27:27] However we need to be careful, sobanski likes jira a lot [11:27:30] no, it was gitlab [11:27:56] jynus: but only to replace gerrit, not phab [11:28:03] Blasphemy! [11:29:05] * sobanski never said anything positive about Jira [11:29:30] XDDD [11:33:00] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Marostegui) The following tables will be checked across all the wikis: ` revision rev_id text old_id user user_id change_tag ct_id actor actor_id ipblocks ipb_id comment comment_id user... [11:33:18] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Marostegui) [11:42:28] 10DBA, 10Data-Persistence: Enable replication eqiad -> codfw and other checks - https://phabricator.wikimedia.org/T261914 (10Marostegui) [13:14:12] 10DBA, 10Patch-For-Review, 10Sustainability (Incident Followup): Automatically compare a few tables per section between hosts and DC - https://phabricator.wikimedia.org/T207253 (10Marostegui) [17:28:10] 10DBA, 10Data-Services, 10User-bd808, 10cloud-services-team (Kanban): Prepare and check storage layer for smnwiki - https://phabricator.wikimedia.org/T264900 (10bd808) 05Open→03Resolved a:03bd808 ` $ ssh cumin1001.eqiad.wmnet $ sudo /usr/local/bin/secure-cookbook wmcs.wikireplicas.add_wiki --task-id... [19:02:27] 10DBA: Productionize es20[26-34] and es10[26-34] - https://phabricator.wikimedia.org/T261717 (10Papaul)