[07:21:24] o/ [07:31:30] o/ [07:42:23] huh, I'm seeing lots of these in the error log for the new proxy vms: https://phabricator.wikimedia.org/P75678 [07:58:58] doesn't ring any bell [08:03:27] at least I've just found out we ship https://github.com/openresty/lua-resty-redis embedded in puppet instead of using the debian package [08:03:33] and the version embedded in puppet is very old [08:04:28] (the copy in puppet being added by Yuvi in 2013 and then never touched since :-) [08:12:28] yeah, using the updated & debian packaged version of that library works much better [08:14:41] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1140125 [08:17:11] +1d [08:17:54] * taavi still running PCC [08:41:45] i think we could start provisioning AAAA records for new proxies. that'd be https://gerrit.wikimedia.org/r/1088338 [08:48:08] LGTM. [08:58:11] and it works! [08:58:11] toolsbeta-static.wmcloud.org has IPv6 address 2a02:ec80:a000:1::1d [08:58:18] arturo: thank you for all the reviews! [08:58:27] np [09:52:14] I'm trying to understand how the alert in T383723 is labeled with "team:wmcs", the definition in operations/alerts contains "team: sre" but I guess there's a remapping somewhere [09:52:14] T383723: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723 [10:00:02] dhinus: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/common/profile/prometheus/ops.yaml#37 [10:00:12] taavi: thanks! [10:43:13] nice read: https://idiallo.com/blog/zipbomb-protection [15:55:45] fyi T393024, I don't think I can investigate until after the hackathon [15:55:46] T393024: lua entry thread aborted: runtime error: /etc/nginx/lua/domainproxy.lua:32: bad request - https://phabricator.wikimedia.org/T393024 [17:52:32] fyi gerrit is having an outage https://phabricator.wikimedia.org/T393034 [21:10:00] Fyi. Gerrit restored