[00:50:22] PROBLEM - Free space - all mounts on deployment-kafka-jumbo-1 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka-jumbo-1.diskspace.root.byte_percentfree (<55.56%) [01:21:54] twentyafterfour https://phabricator.wikimedia.org/source/mediawiki/manage/basics/ says this: "Pull of 'mediawiki' failed: Command failed with error #128! COMMAND git fetch origin '+refs/*:refs/*' --prune STDOUT (empty) STDERR error: insufficient permission for adding an object to repository database objects fatal: failed to write object fatal: unpack-objects failed" [01:45:21] PROBLEM - Free space - all mounts on deployment-kafka-jumbo-1 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka-jumbo-1.diskspace.root.byte_percentfree (<44.44%) [02:16:37] 10Gerrit, 10Documentation: Gitiles does not display 
 blocks in repository descriptions - https://phabricator.wikimedia.org/T193468#4170493 (10Krinkle) Aside from whether we could safely and quickly replicate all references and internal branches and set up a new Gitiles server...  Whitelisting `
` seem...
[02:40:23] 	 PROBLEM - Free space - all mounts on deployment-kafka-jumbo-1 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka-jumbo-1.diskspace.root.byte_percentfree (<44.44%)
[03:17:44] 	 PROBLEM - Free space - all mounts on deployment-kafka-jumbo-2 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka-jumbo-2.diskspace.root.byte_percentfree (<50.00%)
[03:30:35] 	 10Deployments, 10Release-Engineering-Team (Watching / External), 10Operations, 10HHVM, and 3 others: Translation cache exhaustion caused by changes to PHP code in file scope - https://phabricator.wikimedia.org/T103886#4309643 (10Krinkle) @MoritzMuehlenhoff The alternative is to post-pone the switching of M...
[04:47:19] 	 https://www.bleepingcomputer.com/news/security/17-backdoored-docker-images-removed-from-docker-hub/
[10:27:44] 	 RECOVERY - Free space - all mounts on deployment-kafka-jumbo-2 is OK: OK: All targets OK
[10:50:19] 	 RECOVERY - Free space - all mounts on deployment-kafka-jumbo-1 is OK: OK: All targets OK
[11:01:59] 	 People were just randomly using docker images they didn't vet?
[14:52:03] 	 legoktm, you're surprised? :)
[16:44:31] 	 Krenair: just a bit
[16:44:40] 	 I'm not
[17:41:26] 	 10Deployments, 10Release-Engineering-Team (Watching / External), 10Operations, 10HHVM, and 3 others: Translation cache exhaustion caused by changes to PHP code in file scope - https://phabricator.wikimedia.org/T103886#4310002 (10MoritzMuehlenhoff) @Krinkle Ok, that piece of context was missing. Makes sense...
[17:43:57] 	 legoktm: vetting is also non-trivial. Same gotcha as npm, which is that.. the linked "git repo" may not match what is actually in the registry. Afaik docker registry is pushed from local and not enforced to match the public git repo. Unlike packagist which pulls server-side and should always be in sync as such, or at least harder to fool.
[17:44:10] 	 Like, there are/were various bad npm packages with links to github repos that are fine.
[17:45:01] 	 you'd have to inspect the whole file system recursively after pulling or something like that.
[17:45:49] 	 actually I don't know about docker, but al least for npm that's the way to vet. Maybe there's an easier way to verify with docker based on checksums and locally re-creating it based on the dockerfile in git.
[17:55:41] 	 Krinkle: with docker it's incredibly easy to rebuild from source...I would have thought people would have done that internally
[17:55:52] 	 Instead of pulling down arbitrary binary blobs
[17:56:07] 	 There are some neat docker-diff scripts floating around
[17:56:34] 	 legoktm: Right, so that means not using the registry or the standard pull syntax, but instead copying the dockerfile into your own infra somewhere and hosting your own registry and pulling from there?
[17:56:52] 	 or you mean to do it once locally without registry to verify?
[17:57:15] 	 I suppose that would work as long as the steps are deterministic 
[17:58:05] 	 but people love to download and execute arbitrary code
[17:58:11] 	 indeed
[17:58:18] 	 it's practically an addiction
[17:58:35] 	 javascript works this way
[17:58:48] 	 and have you seen the places that tell you to curl | sudo sh ?
[18:01:39] 	 Yeah
[18:01:52] 	 Ran it just a few minutes ago to install Homebrew on a mac mini colo.
[18:02:28] 	 https://brew.sh/
[20:37:11] 	 PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<11.11%)
[20:51:19] 	 PROBLEM - Free space - all mounts on deployment-kafka-jumbo-1 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka-jumbo-1.diskspace.root.byte_percentfree (<22.22%)
[21:10:50] 	 Krinkle: either I guess, rebuilding locally and using your own registry seems ideal. A service to rebuild images locally and verify that they match what's been pushed would be great, but I don't think most docker images are reproducible
[21:12:10] 	 PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<33.33%)
[21:42:02] 	 legoktm: Right, that's what I thought too.
[21:51:21] 	 PROBLEM - Free space - all mounts on deployment-kafka-jumbo-1 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka-jumbo-1.diskspace.root.byte_percentfree (<11.11%)
[22:48:42] 	 PROBLEM - Free space - all mounts on deployment-kafka-jumbo-2 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka-jumbo-2.diskspace.root.byte_percentfree (<10.00%)