[11:29:30] I'm having some issues with jobs running in jenkins (ex. https://integration.wikimedia.org/ci/job/tox-docker/17087/console), and I'm trying to debug it, but I can't see the config or anything, anyone from releng can help? (or that knows the setup xd) [14:12:26] what config? [14:12:45] there's a few failures in https://integration.wikimedia.org/ci/job/tox-docker/17087/console [14:12:52] 11:26:12 > worker = transport.Transport.new(self._config, target) [14:12:52] 11:26:12 E TypeError: 'NonCallableMagicMock' object is not callable [21:42:04] effie: rzl : in case you were wondering, yes Aaron and I effectively rehashed and thus possibly invalidated about 60% of memc keys at once last night. No, we didn't think it would be that much in terms of quantity. Yes, we will schedule it with ServiceOps going forward. (Revert was standing by, we saw no impact on appserver CPU or latencies, and relatively little impact on memc bandwidth which became my main concern after we realised the [21:42:04] magnitude, socket errors etc remained at 0, so we kept it). It was mostly small keys just a lot of them quantity-wise. [21:42:57] It was part of T252564, which goes by namespace, the last namespace to go after we already did all the big and small wikis last year was the "global" namespace (e.g. WANCache::makeGlobalKey, mostly for shared/cross-wiki stuff). [21:42:58] T252564: Let WANObjectCache store "sister keys" on the same backend as the main value key - https://phabricator.wikimedia.org/T252564 [21:43:46] * Krinkle created a Grafana annotation for easy reference in graphs [21:44:37] https://grafana.wikimedia.org/d/000000316/memcache?orgId=1&from=1612481991558&to=1612560971711 [21:45:03] Krinkle: out of curiosity did you record it in SAL at the time? [21:45:20] ah I guess not if you thought it wasn't impactful at the time [21:45:28] okay :) [21:45:33] Indeed, not beyond the standard scap sync-file entry [21:45:36] Krinkle: ack, appreciate the heads up [21:45:44] 👍 [21:45:51] thanks for letting us know [21:46:55] we don't have much SRE coverage that time of night so I'm glad it went smoothly :) agree it would be best to coordinate next time around [22:24:54] rzl: cdanis: also closing this one - https://phabricator.wikimedia.org/T239983#6807901