[06:20:49] 10DBA, 10Community-Tech, 10Expiring-Watchlist-Items: Watchlist Expiry: Release plan [rough schedule] - https://phabricator.wikimedia.org/T261005 (10Marostegui) >>! In T261005#6583259, @ifried wrote: > @Marostegui Thank you for your help so far in the release process! Now that it has been a few weeks, we woul... [07:08:12] db1077 is labeled as test-s4 but it replicates enwiki, should I change it to test-s1? [07:12:03] 10DBA: Monitor the growth of CheckUser tables at large wikis - https://phabricator.wikimedia.org/T265344 (10Marostegui) [07:12:33] jynus: +1 [07:13:40] https://gerrit.wikimedia.org/r/c/operations/puppet/+/636818 [07:13:48] I will update it on zarcillo, too [07:14:28] 10DBA: Monitor the growth of CheckUser tables at large wikis - https://phabricator.wikimedia.org/T265344 (10Marostegui) There was a jump on eswiki, which grew 30MB in a week, but that's not too worrying I think. Even if it did grow like that it would be 1.5G a year. Let's do one more monitoring week before closi... [07:15:30] 10DBA: Monitor the growth of CheckUser tables at large wikis - https://phabricator.wikimedia.org/T265344 (10Marostegui) [07:16:15] 10DBA: Monitor the growth of CheckUser tables at large wikis - https://phabricator.wikimedia.org/T265344 (10Marostegui) [07:17:29] one important thing to note is that on sections called differently than its replica set (test-s1 vs s1) heartbeat can fail, and the replication check fails back to using show slave status [07:18:14] in an ideal world we would have 2 configs, the "replica set" and the "section" [07:18:50] Not a big deal for that host I think. Most likely I will get it to replicate from pc1 in a few days to do some testing anyways [07:19:02] For orchestrator testing [07:19:33] yeah, I wanted to mention as a decision about how the check works, which could impact in the future other hosts [07:22:30] any suggestion on how to "archive" this patch: https://gerrit.wikimedia.org/r/c/operations/software/wmfbackups/+/636007 [07:22:37] this is useless from a code perspective [07:22:50] 10DBA, 10Growth-Structured-Tasks, 10Growth-Team (Current Sprint): Add a link engineering: Determine format for accessing and storing link recommendations - https://phabricator.wikimedia.org/T261411 (10Marostegui) @kostajh thanks for the heads up. This makes me wonder if the statement about the data not being... [07:22:51] but I would like to keep it around for discussion purposes [07:23:24] should I paste it on the doc, commit it somewhere else? [07:23:34] jynus: maybe creating a task out of it to discuss a way to download all the images from a wiki? [07:24:01] yes, task exists, but where should I put it, on a comment? [07:24:49] or maybe a link to it, which I guess will remain alive even if you abandon that patch? [07:25:31] thanks, I will do both [08:06:37] is it too early to think about orchestaror db backups? [08:06:49] I think so yes [08:07:06] ok, ping me when you want to add those [08:07:09] 10DBA, 10Growth-Structured-Tasks, 10Growth-Team (Current Sprint): Add a link engineering: Determine format for accessing and storing link recommendations - https://phabricator.wikimedia.org/T261411 (10kostajh) >>! In T261411#6583831, @Marostegui wrote: > @kostajh thanks for the heads up. > This makes me wond... [08:07:10] wilco [08:22:00] 10DBA: Auto detect DC on orchestrator UI - https://phabricator.wikimedia.org/T266635 (10Marostegui) [08:22:18] 10DBA: Auto detect DC on orchestrator UI - https://phabricator.wikimedia.org/T266635 (10Marostegui) p:05Triage→03Medium [08:24:48] 10DBA, 10Data-Persistence-Backup: Orchestrator db logical backups - https://phabricator.wikimedia.org/T266636 (10Marostegui) [08:25:06] 10DBA, 10Data-Persistence-Backup: Orchestrator db logical backups - https://phabricator.wikimedia.org/T266636 (10Marostegui) 05Open→03Stalled p:05Triage→03Medium [08:27:10] jynus: db2093 is already backed up, so i think we're covered by default [08:27:35] you mean manually? [08:27:39] no? [08:28:07] `modules/profile/templates/mariadb/backup_config/dbprov2003.cnf.erb` [08:28:41] but that covers zarcillo only [08:28:57] I think, let me check [08:29:21] i mean, if it is, i'm wondering how `m5` indicates what databases to backup for db2078 [08:29:32] through grants [08:29:44] oh, huh [08:31:12] we can add orchestrator quite easilly, though [08:31:34] doesn't hurt and we can recheck later [08:31:38] GRANT SELECT, LOCK TABLES, SHOW VIEW, EVENT, TRIGGER ON `zarcillo`.* TO `dump`@`10.192.48.161` [08:31:41] yep, you're right [08:32:05] let me do this and we can rething backups later on [08:32:12] won't take me long [08:34:45] is it only 1 db "orchestrator"? [08:35:00] yes [08:36:56] snapshots backup everything, but we didn't do full backups on tendril because it was a ping [08:37:02] *pain [08:37:41] will run now a tendril backup to check [08:43:00] ah, the backups of "orchestator" failed [08:43:07] I should have learn spelling at scheel [08:43:15] :) [08:43:16] or basic typing [08:43:24] I guess I should unstall https://phabricator.wikimedia.org/T266636 ? [08:43:44] it is ok [08:43:57] I just wanted to do early stuff so I wasn't a blocker [08:44:05] we can re-review after you are happy with the setup [08:44:16] to make sure the backups work [08:45:38] I will update there, but can be stalled for now [08:49:08] kormat: dbprov2003:/srv/backups/dumps/latest/dump.zarcillo.2020-10-28--08-48-19 should be fine now [08:50:28] sweet [08:50:44] 10DBA, 10Data-Persistence-Backup: Orchestrator db logical backups - https://phabricator.wikimedia.org/T266636 (10jcrespo) I have added it to the logical backup process by adding the right grants to the existing dump user/process to the new database, but let's revisit once people working on the setup are happy... [08:50:58] 10DBA, 10Operations, 10User-Kormat: orchestrator: Get packages into WMF apt - https://phabricator.wikimedia.org/T266023 (10Marostegui) 05Open→03Resolved a:03Kormat Going to close this as resolved as the packages are uploaded. Thank you Stevie! Per T266023#6570807, medium-term we should take a look at c... [08:51:00] 10DBA, 10Operations, 10Patch-For-Review, 10User-Kormat: orchestrator: Puppetize - https://phabricator.wikimedia.org/T265990 (10Marostegui) [08:52:32] In my defense, the process was documented at: https://wikitech.wikimedia.org/wiki/MariaDB/Backups#Adding_a_new_dump [08:52:55] but I concede that the page is not in the best state, even if technically contains all info [08:54:16] I need to separate design decisions from day-to-day operations [08:56:13] sobanski: any objections to us requesting an `orchestrator` phabricator tag? it would make it easier to track things. [08:57:28] None whatsoever. [08:58:06] Does it require review/approval? [08:58:23] i think it requires a phab admin. i don't seem to have the privileges at least [08:58:32] currently looking to see if there's a doc saying how to request such a thing [08:58:44] kormat: I requested one "recently" let me check [08:58:46] in addition of the task, consider organizing a hierarchy if makes sense [08:58:47] last time i assigned it directly to aklapper, who responded saying "done, but please don't do that :)" [08:59:02] for example, I am sure T266636 could have a goal parent [08:59:02] T266636: Orchestrator db logical backups - https://phabricator.wikimedia.org/T266636 [08:59:23] kormat: https://phabricator.wikimedia.org/T233589 [08:59:26] kormat: there is a process on mediawiki, I think [09:00:42] kormat: https://www.mediawiki.org/wiki/Phabricator/Project_management#Types_of_Projects [09:01:46] I should have permissions to create tags, let me take a look to see if Andre will be cross with me if I go ahead and do it [09:02:22] "Component" ones (blue) should be ok to create directly, according to doc [09:13:46] That's what I thought [09:14:22] I'll go ahead and create it then [09:16:26] Done: https://phabricator.wikimedia.org/project/profile/5061/ [09:16:56] 10DBA, 10Orchestrator, 10Patch-For-Review: Populating orchestrator metadata on a per-server basis - https://phabricator.wikimedia.org/T266485 (10Marostegui) [09:17:16] 10DBA, 10Operations, 10Orchestrator, 10User-Kormat: orchestrator: integrate promotion rules into puppet - https://phabricator.wikimedia.org/T266002 (10Marostegui) [09:17:25] 10DBA, 10Operations, 10Orchestrator, 10Patch-For-Review, 10User-Kormat: orchestrator: Puppetize - https://phabricator.wikimedia.org/T265990 (10Marostegui) [09:17:28] sobanski: \o/ thanks [09:17:38] 10DBA, 10Data-Persistence, 10Orchestrator, 10Patch-For-Review, 10User-Kormat: orchestrator: Select backend database solution - https://phabricator.wikimedia.org/T266003 (10Marostegui) [09:32:02] 10DBA, 10Orchestrator, 10Patch-For-Review, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) [09:32:45] 10DBA, 10Operations, 10Orchestrator, 10User-Kormat: Integrate orchestrator with !log - https://phabricator.wikimedia.org/T266452 (10Marostegui) [09:33:03] 10DBA, 10Data-Persistence-Backup, 10Orchestrator: Orchestrator db logical backups - https://phabricator.wikimedia.org/T266636 (10Marostegui) [09:33:49] 10DBA, 10Orchestrator: Auto detect DC on orchestrator UI - https://phabricator.wikimedia.org/T266635 (10Marostegui) [09:33:56] 10DBA, 10Orchestrator: Orchestrator: Create basic documentation - https://phabricator.wikimedia.org/T266428 (10Marostegui) [09:34:06] 10DBA, 10Operations, 10Orchestrator, 10User-Kormat: orchestrator: Add service monitoring - https://phabricator.wikimedia.org/T266338 (10Marostegui) [09:34:15] 10DBA, 10Operations, 10Orchestrator, 10CAS-SSO, 10User-Kormat: orchestrator: Support SSO - https://phabricator.wikimedia.org/T266106 (10Marostegui) [11:14:20] 10DBA, 10Operations, 10Orchestrator, 10User-Kormat: Explore orchestrator hooks to integrate them with !log, irc alerts and emails - https://phabricator.wikimedia.org/T266452 (10Marostegui) [11:15:03] 10DBA, 10Operations, 10Orchestrator, 10User-Kormat: Explore orchestrator hooks to integrate them with !log, irc alerts and emails - https://phabricator.wikimedia.org/T266452 (10Marostegui) [11:15:12] 10DBA, 10Operations, 10Orchestrator, 10User-Kormat: Explore orchestrator hooks to integrate them with !log, irc alerts and emails - https://phabricator.wikimedia.org/T266452 (10Marostegui) Along with !log we should include sending an email/irc alerts on some of the most important cases like: PostUnsuccessf... [11:16:39] ^ more important than those would be the non-topology parts of switchover (although not sure if you plan on a switchover script to call orchestrator or the other way around) [11:18:41] 10DBA, 10Operations, 10Orchestrator, 10User-Kormat: Explore orchestrator hooks to integrate them with !log, irc alerts and emails - https://phabricator.wikimedia.org/T266452 (10Peachey88) [11:21:15] 10DBA: querycache qc_type and qc_title have different nullabality on s1 only - https://phabricator.wikimedia.org/T265349 (10Marostegui) 05Stalled→03Open [11:21:20] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Sustainability (Incident Followup), 10WorkType-NewFunctionality: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Marostegui) [11:21:38] 10Blocked-on-schema-change, 10DBA: Schema change to drop three indexes from wb_changes - https://phabricator.wikimedia.org/T264109 (10Marostegui) 05Stalled→03Open [11:35:45] 10DBA, 10Operations, 10Orchestrator: Run orchestrator as non-root - https://phabricator.wikimedia.org/T266656 (10Marostegui) [11:36:17] moritzm: ^ looks like repackaging orchestrator is going to come sooner than expected. ;) [11:36:53] 10DBA, 10Operations, 10Orchestrator: Run orchestrator as non-root - https://phabricator.wikimedia.org/T266656 (10Marostegui) p:05Triage→03Medium [11:38:14] hehe :-) [11:38:24] 10DBA, 10Operations, 10Orchestrator: Run orchestrator as non-root - https://phabricator.wikimedia.org/T266656 (10Marostegui) [11:38:27] 10DBA, 10Operations, 10User-Kormat: orchestrator: Get packages into WMF apt - https://phabricator.wikimedia.org/T266023 (10Marostegui) [11:39:14] 10DBA, 10Orchestrator: Support running orchestrator with sqlite backend - https://phabricator.wikimedia.org/T266657 (10Kormat) [11:39:28] 10DBA, 10Orchestrator: Support running orchestrator with sqlite backend - https://phabricator.wikimedia.org/T266657 (10Kormat) 05Open→03Stalled p:05Triage→03Medium [11:55:19] 10DBA, 10Operations, 10Orchestrator: Run orchestrator as non-root - https://phabricator.wikimedia.org/T266656 (10MoritzMuehlenhoff) If this is solely about the need to bind to a privileged port, ` sudo setcap 'cap_net_bind_service=+ep' $ORCHESTRATORBINARY ` might also simply work out? [12:44:11] 10DBA, 10Operations, 10Orchestrator: Run orchestrator as non-root - https://phabricator.wikimedia.org/T266656 (10Kormat) There's no reason it needs a privileged port. It will be behind a reverse proxy anyway. The package doesn't create a user/group, so that's the first thing to fix. [12:48:47] Note user management can be handled both on puppet or on package, whetever is preferred [12:50:13] I myself have sometime doubs if I should handle some basic setup on puppet or on package post-inst [12:50:20] for other packages [12:52:57] there was a spike of 10 seconds of lag 10 minutes ago on db1106 [13:00:48] 10DBA: Disconnect codfw -> eqiad replication - https://phabricator.wikimedia.org/T266663 (10Marostegui) [13:01:10] 10DBA: Disconnect codfw -> eqiad replication - https://phabricator.wikimedia.org/T266663 (10Marostegui) p:05Triage→03Medium Not before Thursday 29th Oct 2020 [14:04:46] 10DBA, 10Orchestrator, 10Patch-For-Review, 10User-Kormat: orchestrator: Select backend database solution - https://phabricator.wikimedia.org/T266003 (10LSobanski) [14:04:59] 10DBA, 10Operations, 10User-Kormat: Clean up role::mariadb::ferm and profile::mariadb::ferm - https://phabricator.wikimedia.org/T265901 (10LSobanski) [14:05:24] 10DBA, 10Operations, 10Release-Engineering-Team-TODO, 10Continuous-Integration-Config, 10User-Kormat: Create integration test env for wmfmariadbpy - https://phabricator.wikimedia.org/T265266 (10LSobanski) [14:05:46] 10DBA: transfer.py fails when copying data between es hosts - https://phabricator.wikimedia.org/T262388 (10LSobanski) [14:06:24] 10DBA: Remove muswiki and mhwiktionary from s3 - https://phabricator.wikimedia.org/T260112 (10LSobanski) [14:24:40] 10DBA, 10Orchestrator: Orchestrator: Create basic documentation - https://phabricator.wikimedia.org/T266428 (10Kormat) https://wikitech.wikimedia.org/wiki/Orchestrator created, just needs content now ;) [15:19:42] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-08-31) rack/setup/install es10[26-34].eqiad.wmnet - https://phabricator.wikimedia.org/T260370 (10Cmjohnson) [15:20:43] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-08-31) rack/setup/install es10[26-34].eqiad.wmnet - https://phabricator.wikimedia.org/T260370 (10Cmjohnson) a:05Cmjohnson→03RobH @robh These still need the raid setup, you mentioned you could do that. If not please let me know and I will take c... [16:27:44] 10DBA, 10Operations, 10Orchestrator, 10User-Kormat: Explore orchestrator hooks to integrate them with dbctl, !log, irc alerts and emails - https://phabricator.wikimedia.org/T266452 (10Marostegui) [16:27:59] 10DBA, 10Orchestrator: Support running orchestrator with sqlite backend - https://phabricator.wikimedia.org/T266657 (10Marostegui) [16:28:01] 10DBA, 10Operations, 10Orchestrator: Run orchestrator as non-root - https://phabricator.wikimedia.org/T266656 (10Marostegui) [17:41:44] 10DBA, 10Community-Tech, 10Expiring-Watchlist-Items: Watchlist Expiry: Release plan [rough schedule] - https://phabricator.wikimedia.org/T261005 (10ifried) @Marostegui Thanks for your response, and it's great news that we have general approval to now release to all wikis! One note about the calendar dates:... [22:21:23] 10DBA, 10Operations, 10Datacenter-Switchover: When switching DCs, update pc hosts in tendril - https://phabricator.wikimedia.org/T266723 (10RLazarus) [22:23:36] 10DBA, 10Operations, 10Datacenter-Switchover: When switching DCs, update pc hosts in tendril - https://phabricator.wikimedia.org/T266723 (10RLazarus) p:05Triage→03Medium [22:23:56] 10DBA, 10Operations, 10Datacenter-Switchover: When switching DCs, update pc hosts in tendril - https://phabricator.wikimedia.org/T266723 (10RLazarus)