[00:00:22] 10serviceops, 10Anti-Harassment, 10IP Info, 10SRE, 10Patch-For-Review: Update MaxMind GeoIP2 license key and product IDs for application servers - https://phabricator.wikimedia.org/T288844 (10Dzahn) further update: identified the relevant files in the private repo that hold the userId and license key for... [00:23:13] 10serviceops, 10Anti-Harassment, 10IP Info, 10SRE, 10Patch-For-Review: Update MaxMind GeoIP2 license key and product IDs for application servers - https://phabricator.wikimedia.org/T288844 (10Dzahn) @phuedx Hi, I would like to compare the UserId, LicenseKey and ProductIds between what I see in production... [00:30:57] 10serviceops, 10Anti-Harassment, 10IP Info, 10SRE, 10Patch-For-Review: Update MaxMind GeoIP2 license key and product IDs for application servers - https://phabricator.wikimedia.org/T288844 (10Dzahn) a:05wkandek→03Dzahn [00:34:33] 10serviceops, 10Anti-Harassment, 10IP Info, 10SRE, 10Patch-For-Review: Update MaxMind GeoIP2 license key and product IDs for application servers - https://phabricator.wikimedia.org/T288844 (10Dzahn) 05Open→03In progress [00:35:08] 10serviceops, 10Anti-Harassment, 10IP Info, 10SRE, 10Patch-For-Review: Update MaxMind GeoIP2 license key and product IDs for application servers - https://phabricator.wikimedia.org/T288844 (10Dzahn) p:05Medium→03High [00:46:47] 10serviceops, 10Anti-Harassment, 10IP Info, 10SRE, 10Patch-For-Review: Update MaxMind GeoIP2 license key and product IDs for application servers - https://phabricator.wikimedia.org/T288844 (10Dzahn) There is a mechanism that first downloads the database files centrally to the puppetmaster so that appser... [04:47:28] 10serviceops, 10DBA, 10Toolhub, 10database-backups, 10Patch-For-Review: Setup production database for Toolhub - https://phabricator.wikimedia.org/T271480 (10Marostegui) >>! In T271480#7375578, @bd808 wrote: >>>! In T271480#7354348, @Marostegui wrote: >> Thanks for the update, if you need something from u... [08:16:06] 10serviceops, 10Analytics, 10Platform Engineering, 10Wikibase change dispatching scripts to jobs: Better observability/visualization for MediaWiki jobs - https://phabricator.wikimedia.org/T291620 (10Michael) [08:26:53] 10serviceops, 10Analytics, 10Platform Engineering, 10Wikibase change dispatching scripts to jobs: Better observability/visualization for MediaWiki jobs - https://phabricator.wikimedia.org/T291620 (10Michael) >>! In T291620#7375141, @Ottomata wrote: > Data Eng (analytics) is in the process of [[ https://pha... [08:33:31] 10serviceops: Migrate WMF Production from PHP 7.2 to PHP 7.4 - https://phabricator.wikimedia.org/T271736 (10fgiunchedi) [08:33:36] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Radar): Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10fgiunchedi) 05Resolved→03Open I don't think this is resolved, see T275752 for jobrunner on buster slowness in upload [08:53:15] 10serviceops, 10Analytics, 10Platform Engineering, 10Wikibase change dispatching scripts to jobs: Better observability/visualization for MediaWiki jobs - https://phabricator.wikimedia.org/T291620 (10Joe) This task mixes quite a few things. I'll start by answering your questions to the best of my knowledge.... [08:54:11] 10serviceops, 10Analytics, 10Platform Engineering, 10Wikibase change dispatching scripts to jobs: Better observability/visualization for MediaWiki jobs - https://phabricator.wikimedia.org/T291620 (10Joe) I also want to underline that the problem statement is slightly misleading: while it's true observabili... [09:52:44] 10serviceops, 10Icinga, 10SRE, 10SRE Observability, 10observability: incident 20170323-wikibase did not trigger Icinga paging - https://phabricator.wikimedia.org/T161528 (10Marostegui) p:05High→03Medium What should we do with this task? (I don't think this is high anymore) [09:53:15] 10serviceops, 10Analytics, 10Platform Engineering, 10Wikibase change dispatching scripts to jobs: Better observability/visualization for MediaWiki jobs - https://phabricator.wikimedia.org/T291620 (10Michael) >>! In T291620#7376077, @Joe wrote: > This task mixes quite a few things. Yes, it is a wishlist o... [09:55:40] 10serviceops, 10SRE: Put rdb20[09|10] into service - https://phabricator.wikimedia.org/T281225 (10Marostegui) @akosiaris was this completed? [10:01:42] 10serviceops, 10Analytics, 10Platform Engineering, 10Wikibase change dispatching scripts to jobs: Better observability/visualization for MediaWiki jobs - https://phabricator.wikimedia.org/T291620 (10Michael) [10:19:13] 10serviceops, 10Analytics, 10Platform Engineering, 10Wikibase change dispatching scripts to jobs: Better observability/visualization for MediaWiki jobs - https://phabricator.wikimedia.org/T291620 (10Joe) >>! In T291620#7376163, @Michael wrote: >> [...] I do agree that tracing the jobs trees and execution t... [10:22:29] 10serviceops, 10Analytics, 10Platform Engineering, 10Wikibase change dispatching scripts to jobs: Better observability/visualization for MediaWiki jobs - https://phabricator.wikimedia.org/T291620 (10Joe) >>! In T291620#7376163, @Michael wrote: >>> *What were the (debug- or even trace-level?) logs of each j... [12:05:17] 10serviceops, 10SRE: Put rdb20[09|10] into service - https://phabricator.wikimedia.org/T281225 (10akosiaris) 05Open→03Resolved a:03akosiaris >>! In T281225#7376168, @Marostegui wrote: > @akosiaris was this completed? Looks like it. Resolving. [12:39:21] 10serviceops, 10MW-on-K8s, 10Performance-Team, 10SRE, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10akosiaris) >>! In T290536#7371552, @jijiki wrote: >>>! In T290536#7364817, @Joe wrote: >> I have some alternative ideas. Specifically, right now we have... [14:25:42] 10serviceops, 10Analytics, 10Platform Engineering, 10Wikibase change dispatching scripts to jobs: Better observability/visualization for MediaWiki jobs - https://phabricator.wikimedia.org/T291620 (10Ladsgroup) Let me add some points here: - While I agree jobs have better o11y than the current dispatching... [14:32:45] 10serviceops, 10Maps, 10Patch-For-Review, 10User-jijiki: Deploy tegola-vector-tiles to kubernetes - https://phabricator.wikimedia.org/T283159 (10jijiki) [14:40:59] 10serviceops, 10DBA, 10Toolhub, 10database-backups, 10Patch-For-Review: Setup production database for Toolhub - https://phabricator.wikimedia.org/T271480 (10bd808) 05Open→03Resolved >>! In T271480#7375854, @Marostegui wrote: > Let me know if this fixes the issue. The deployment worked this time, so... [14:41:56] 10serviceops, 10DBA, 10Toolhub, 10database-backups, 10Patch-For-Review: Setup production database for Toolhub - https://phabricator.wikimedia.org/T271480 (10Marostegui) \o/ glad to hear! You are welcome! [14:54:21] TIL how to tail logs across multiple pods with multiple containers: `kubectl logs -f --since=10m --all-containers=true --max-log-requests=10 -lapp=toolhub` [14:55:27] The --max-log-requests bit in there is because for this particular query the default of max 5 containers was exceeded. [15:39:26] Is there an equivalent to staging.svc.eqiad.wmnet for accessing ports exposed by the eqiad k8s cluster? [15:43:16] Using kubernetes1001.eqiad.wmnet / 10.64.0.121 seems to work [15:45:26] legoktm: Toolhub is alive in the eqiad cluster at https://10.64.0.121:4011/. Ready for the steps to expose it via LVS I think. [16:13:11] bd808: awesome!! I'll rebase / double-check the existing patches and then coordinate with another SRE to get it out on Monday [16:15:28] thanks. maybe this will see use before the quarter ends after all :) [16:26:31] 10serviceops, 10Anti-Harassment, 10IP Info, 10SRE, 10Patch-For-Review: Update MaxMind GeoIP2 license key and product IDs for application servers - https://phabricator.wikimedia.org/T288844 (10phuedx) >>! In T288844#7375732, @Dzahn wrote: > Could you share the new license information with me in a secure w... [16:40:31] 10serviceops, 10Anti-Harassment, 10IP Info, 10SRE, 10Patch-For-Review: Update MaxMind GeoIP2 license key and product IDs for application servers - https://phabricator.wikimedia.org/T288844 (10phuedx) cc'ing @dom_walden and @imaigwilo, the QTEs that work with Anti-Harassment Tools. Are the MaxMind databa... [16:55:25] 10serviceops, 10Anti-Harassment, 10IP Info, 10SRE, 10Patch-For-Review: Update MaxMind GeoIP2 license key and product IDs for application servers - https://phabricator.wikimedia.org/T288844 (10Dzahn) >>! In T288844#7376870, @phuedx wrote: > If you have a GPG key, then we could exchange public keys and I c... [17:03:30] 10serviceops, 10Anti-Harassment, 10IP Info, 10SRE, 10Patch-For-Review: Update MaxMind GeoIP2 license key and product IDs for application servers - https://phabricator.wikimedia.org/T288844 (10Dzahn) >>! In T288844#7376887, @phuedx wrote: > Are the MaxMind databases deployed to the Beta Cluster or will th... [18:03:47] 10serviceops, 10Analytics, 10Platform Engineering, 10Wikibase change dispatching scripts to jobs: Better observability/visualization for MediaWiki jobs - https://phabricator.wikimedia.org/T291620 (10Ottomata) > having job parameters in hadoop would be extremely useful, we keep all user requests (up to 90 d... [18:05:21] 10serviceops, 10Citoid: zotero paging / serving 5xxes after CPU spikes - https://phabricator.wikimedia.org/T291707 (10Legoktm) [18:07:35] 10serviceops, 10Citoid, 10Patch-For-Review: zotero paging / serving 5xxes after CPU spikes - https://phabricator.wikimedia.org/T291707 (10Legoktm) [18:11:51] 10serviceops, 10Citoid: zotero paging / serving 5xxes after CPU spikes - https://phabricator.wikimedia.org/T291707 (10Legoktm) Open questions: * Is there a better way to fix this / identify the problematic requests? * Is throwing more resources at zotero an acceptable long-term solution or just short-term? * H... [19:35:42] 10serviceops, 10Analytics, 10Platform Engineering, 10Wikibase change dispatching scripts to jobs: Better observability/visualization for MediaWiki jobs - https://phabricator.wikimedia.org/T291620 (10Ladsgroup) If I can do beeline in stat1005 and look at the data, I don't care about the rest. There is a ge... [23:43:07] 10serviceops, 10Anti-Harassment, 10IP Info, 10SRE, 10Patch-For-Review: Update MaxMind GeoIP2 license key and product IDs for application servers - https://phabricator.wikimedia.org/T288844 (10Dzahn) I received the new license info from @phuedx , encrypted with GPG. I decrypted it and added it to the pri...