[07:06:39] 06serviceops, 06Commons, 10MediaWiki-File-management, 07Wikimedia-production-error: MediaWiki periodic job cleanup-upload-stash failed - https://phabricator.wikimedia.org/T412325#11450239 (10A_smart_kitten) Hello #serviceops :) Please could an SRE grab the stack trace and error time for this when you get a... [08:33:45] 06serviceops, 10MW-on-K8s, 06SRE: Pushing to the docker registry fails with 500 Internal Server Error - https://phabricator.wikimedia.org/T412265#11450416 (10jcrespo) [09:34:47] 06serviceops, 10MW-on-K8s, 06SRE: Pushing to the docker registry fails with 500 Internal Server Error - https://phabricator.wikimedia.org/T412265#11450550 (10Aklapper) For the records I had a similar issue running the train last month in T408272#11369913; however simply trying to run `scap train` again worke... [10:32:07] 06serviceops, 10Wikifunctions, 06Abstract Wikipedia team (26Q2 (Oct–Dec)), 07Essential-Work: WF memcached service is dc-local but used for dc-global content - https://phabricator.wikimedia.org/T411807#11450796 (10akosiaris) FYI, I 've continued posting updates in T405461 [12:42:56] 06serviceops, 13Patch-For-Review: Move EXCLUDED_SERVICES attribute from sre.discovery.datacentre to service catalog - https://phabricator.wikimedia.org/T412211#11451331 (10Blake) The change which adds the attribute and copies the extant services over (minus ones which no longer exist) has been submitted and me... [12:50:42] 06serviceops, 10MW-on-K8s, 06SRE: Pushing to the docker registry fails with 500 Internal Server Error - https://phabricator.wikimedia.org/T412265#11451401 (10Urbanecm_WMF) p:05Unbreak!→03High Decreasing to //High//, as several deployments completed since. I'll try again. Leaving open in case #serviceops... [20:46:35] 06serviceops, 10MW-on-K8s, 06SRE: Pushing to the docker registry fails with 500 Internal Server Error - https://phabricator.wikimedia.org/T412265#11452914 (10Urbanecm_WMF) Okay, I tried deploying the two patches (https://gerrit.wikimedia.org/r/1217573, https://gerrit.wikimedia.org/r/1217574) again. Same erro... [21:25:44] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11453052 (10VRiley-WMF) a:03VRiley-WMF [21:26:55] 06serviceops, 06collaboration-services, 10MW-on-K8s, 06SRE, 13Patch-For-Review: Use encrypted rsync for releases - https://phabricator.wikimedia.org/T289858#11453061 (10Dzahn) [23:52:10] 06serviceops, 06collaboration-services, 10MW-on-K8s, 06SRE: Use encrypted rsync for releases - https://phabricator.wikimedia.org/T289858#11453352 (10Dzahn) a:03Dzahn