[00:19:46] 10serviceops, 10DC-Ops, 10Platform Engineering, 10SRE, 10Patch-For-Review: Rename wtp* servers to parse* (Parsoid PHP servers) - https://phabricator.wikimedia.org/T245888 (10Dzahn) 05Open→03Declined [00:19:49] 10serviceops, 10SRE, 10Parsoid (Tracking): Upgrade Parsoid servers to buster - https://phabricator.wikimedia.org/T268524 (10Dzahn) [01:13:00] 10serviceops, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Replace production deployment servers and update them to Buster - https://phabricator.wikimedia.org/T265963 (10Dzahn) 05Open→03Resolved [01:13:04] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [05:56:10] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE: legoktm can't build CI docker images without using root because he's no longer in contint-admins - https://phabricator.wikimedia.org/T275731 (10Joe) >>! In T275731#6871524, @hashar wrote: > I would rather have cherry picked people that knows about... [06:45:07] 10serviceops, 10DNS, 10SRE, 10Traffic, and 3 others: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10JMeybohm) p:05Triage→03Medium [07:04:28] 10serviceops: wmf-utils has an outdated script to update known hosts files - https://phabricator.wikimedia.org/T275806 (10Aklapper) [08:09:25] 10serviceops, 10MW-on-K8s, 10SRE, 10observability: Keep calculating latencies for MediaWiki requests that happen k8s - https://phabricator.wikimedia.org/T276095 (10fgiunchedi) My two cents: keeping mtail (or similar, in the envoy case) processing on-host would be ideal I think: it'll be simpler to scale (i... [08:50:25] jayme: are you around and have time to help me parse some error messages I'm getting when deploying a new version of linkrecommendation service? [08:50:42] kostajh: sure [08:51:44] where would I need to look? deploy1002 or kibana? [08:53:07] Sending you a DM [09:53:42] 10serviceops, 10MW-on-K8s, 10SRE: Create a basic helm chart to test MediaWiki on kubernetes - https://phabricator.wikimedia.org/T265327 (10Joe) a:03Joe [09:54:28] 10serviceops, 10MW-on-K8s: Create MediaWiki httpd base image - https://phabricator.wikimedia.org/T276097 (10Joe) [09:54:31] 10serviceops, 10MW-on-K8s, 10SRE: Create a basic helm chart to test MediaWiki on kubernetes - https://phabricator.wikimedia.org/T265327 (10Joe) [09:55:16] ooohhhh [10:18:08] so joe if i was nuts enough to want to run a local minikube based on those helm charts with mediawiki containers... is that doable? [10:51:37] <_joe_> have you seen the work dancy has done? https://github.com/dancysoft/mw-k8s-dev/ [10:51:47] <_joe_> this is the first attempt to do so [10:53:01] <_joe_> what I think we will do now for production use is to create a basic chart that can work in production, and add additional stuff to a) run the database, etcd, memcached, etc locally (also in mediawiki-config), [10:53:10] <_joe_> at a later stage [10:53:52] <_joe_> so that ideally you can get to a situation where you can just mount your mediawiki core + extensions local directory into the containers and use it [10:54:38] <_joe_> I think that's the intended goal, to run mediawiki on kubernetes in production but also allow people to do like-in-production development [10:55:18] <_joe_> the biggest hurdle for that is actually mediawiki-config, which is not exactly conforming to a 12-factor app approach to configuration. [11:42:50] lol I suppose it is not [11:53:04] hey akosiaris - I'd like to apply some calico rules for the linkrecommendation service but I see a few pending changes in the config (adding 10.192.75.0/24 and removing rules for urldownloader2002). Are those safe to deploy? [12:45:37] hnowlan: hmmm not sure, let me double check [12:49:25] 10serviceops, 10Add-Link, 10Growth-Team (Current Sprint): Use envoy for making GET requests to lang.wikipedia.org/api.php - https://phabricator.wikimedia.org/T276217 (10kostajh) [12:52:01] hnowlan: yes, those are fine. It's removal of logstash and 10.192.75.0/24 (the new kubestage pods), it's fine to apply [12:52:10] I should have done that in fact, lemme do that [12:52:48] btw you probably want to add the linkrecommendation stuff in values.yaml networkpolicy stuff in helmfile.d/services stanza too [12:53:05] I think that the file you changed is not going to be around in ~3 week from now [13:02:03] I added an entry to helmfile.d/services/api-gateway/values-production.yaml [13:02:13] the file in admin is going away in favour of the admin_ng one yeah? [13:08:26] * apergos peeks in [13:15:49] 10serviceops, 10Add-Link, 10Growth-Team (Current Sprint): Use Envoy for making GET requests to lang.wikipedia.org/api.php - https://phabricator.wikimedia.org/T276217 (10kostajh) [13:47:42] hnowlan: Ah indeed, I just noticed. Sorry about that. Yes admin/ is going away overall. Hopefully no blockers will arise and it will be a done deal by EOQ [13:48:32] We will probably have to improve a bit on that structure to remove some duplication, especially across services, but otherwise the backing technology will be based on this from then on. [14:21:58] neat [14:40:51] ok, I'll bite; what is 'load-datasets'? [14:41:26] 10serviceops, 10DNS, 10SRE, 10Traffic, and 3 others: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10wkandek) https://gerrit.wikimedia.org/r/c/operations/puppet/+/667731 [15:00:33] apergos: A cronjob that fetches stuff from analytics.wikimedia.org, puts it in a mysql db so that linkrecommendation.discovery.wmnet can return recommendation of links based on the input [15:00:57] ok! thanks for that [15:01:20] I'm a back-seat not-even passenger for today's linkrecommendation updates [16:15:01] apergos: T266826 has (some of) the gory details [16:15:23] * apergos takes a look, why not [16:17:10] aaaand I don't have the brain cells to process most of it today [16:17:15] leaving the tab open though [16:52:03] 10serviceops, 10MW-on-K8s, 10Release Pipeline: Run stress tests on docker images infrastructure - https://phabricator.wikimedia.org/T264209 (10Legoktm) [17:14:16] 10serviceops, 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10Release-Engineering-Team-TODO: Apache on doc1001 does not see updated PHP files for hours/days after deployment - https://phabricator.wikimedia.org/T275468 (10thcipriani) [17:15:18] 10serviceops, 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10Release-Engineering-Team-TODO: Apache on doc1001 does not see updated PHP files for hours/days after deployment - https://phabricator.wikimedia.org/T275468 (10thcipriani) p:05Triage→03Medium [19:12:07] 10serviceops, 10MW-on-K8s, 10Release Pipeline: Run stress tests on docker images infrastructure - https://phabricator.wikimedia.org/T264209 (10Legoktm) >>! In T264209#6549716, @akosiaris wrote: > That being said, with compression on the client before the push taking also a significant time as well (per peopl... [20:10:52] 10serviceops, 10Add-Link, 10Growth-Team, 10Prod-Kubernetes: linkrecommendation: Can't connect to MySQL server on 'm2-master.eqiad.wmnet' - https://phabricator.wikimedia.org/T276268 (10kostajh) [20:12:08] 10serviceops, 10Add-Link, 10Growth-Team, 10Prod-Kubernetes: linkrecommendation: Can't connect to MySQL server on 'm2-master.eqiad.wmnet' - https://phabricator.wikimedia.org/T276268 (10kostajh) [21:51:14] 10serviceops, 10SRE: Upgrade docker-registry servers to Debian Buster - https://phabricator.wikimedia.org/T272550 (10Legoktm) a:03Legoktm [21:52:34] 10serviceops, 10MediaWiki-Containers, 10SRE, 10Patch-For-Review: Homepage for https://docker-registry.wikimedia.org - https://phabricator.wikimedia.org/T179696 (10Legoktm) 05Open→03Resolved [22:44:32] 10serviceops, 10SRE, 10Patch-For-Review: move mwmaint2002 into production, replace mwmaint2001 - https://phabricator.wikimedia.org/T275905 (10Dzahn) [23:43:06] 10serviceops, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Replace production deployment servers and update them to Buster - https://phabricator.wikimedia.org/T265963 (10Dzahn) [23:43:25] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn)