[10:20:34] 10Traffic, 10Operations: cp3037 is currently unreachable - https://phabricator.wikimedia.org/T222041 (10Vgutierrez) Requested a power drain via remote hands: > Powered the server cp3037 for at least 15 seconds. After I plugged the power cable back in I was not be able to turn them manually on cp3037 mgmt inte... [10:21:41] it looks like cp3037 reached cache heaven... where hit rate is always 100% :_( [10:21:50] RIP [10:21:56] :( [11:15:03] 10Traffic, 10Varnish, 10Continuous-Integration-Infrastructure, 10Operations, 10Patch-For-Review: Make CI run Varnish VCL tests - https://phabricator.wikimedia.org/T128188 (10ema) VTC tests can now be run from dev workstations against PCC: ` ema@ariel:~/wmf/operations-puppet$ cd modules/varnish/files/tes... [13:50:06] 10Traffic, 10Operations, 10Thumbor: SwiftMedia URL rewrite returns some 404s with wrong Content-Length - https://phabricator.wikimedia.org/T222071 (10ema) [13:50:13] 10Traffic, 10Operations, 10Thumbor: SwiftMedia URL rewrite returns some 404s with wrong Content-Length - https://phabricator.wikimedia.org/T222071 (10ema) p:05Triage→03Normal [13:56:09] gilles: hey, does ./utils/pcc work on your machine? That's what run.py uses to get the puppet catalog [13:57:01] so basically `./utils/pcc 495643 cp1078.eqiad.wmnet` needs to do the right thing [13:57:35] ah but it seems that the build hosts are in pain [13:57:44] yeah, that's probably what's happening [13:57:47] [ 2019-04-29T13:56:26 ] INFO: Creating directories under /srv/jenkins-workspace/puppet-compiler [13:57:49] I figured it was something like that [13:57:58] error: copy-fd: write returned: No space left on device [13:58:02] [ 2019-04-29T13:57:27 ] INFO: Creating directories under /srv/jenkins-workspace/puppet-compiler [13:58:03] error: copy-fd: write returned: No space left on device [13:58:03] fatal: failed to copy file to '/srv/jenkins-workspace/puppet-compiler/16157/production/src/.git/objects/pack/pack-98408806b04fe7d669426d6ce40e0d2c3be9d1c1.pack': No space left on device [13:58:04] fatal: The remote end hung up unexpectedly [13:58:06] [ 2019-04-29T13:57:27 ] CRITICAL: Build run failed: [Errno 2] No such file or directory: '/srv/jenkins-workspace/puppet-compiler/16157/production/src' [13:58:08] Traceback (most recent call last): [13:58:10] File "build/bdist.linux-x86_64/egg/puppet_compiler/cli.py", line 44, in main [13:58:12] success = c.run() [13:58:14] File "build/bdist.linux-x86_64/egg/puppet_compiler/controller.py", line 116, in run [13:58:16] self.m.prepare() [13:58:18] File "build/bdist.linux-x86_64/egg/puppet_compiler/prepare.py", line 54, in prepare [13:58:20] self._prepare_dir(self.prod_dir) [13:58:22] File "build/bdist.linux-x86_64/egg/puppet_compiler/prepare.py", line 86, in _prepare_dir [13:58:24] with pushd(src): [13:58:26] File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ [13:58:28] return self.gen.next() [13:58:30] File "build/bdist.linux-x86_64/egg/puppet_compiler/prepare.py", line 16, in pushd [13:58:32] os.chdir(dirname) [13:58:34] OSError: [Errno 2] No such file or directory: '/srv/jenkins-workspace/puppet-compiler/16157/production/src' [13:58:36] Build step 'Execute shell' marked build as failure [13:58:38] Finished: FAILURE [13:58:40] FAIL [13:58:42] I'll try again later [13:59:41] sounds good, let me know how it goes! [14:03:01] filed https://phabricator.wikimedia.org/T222072 [14:10:57] 10Traffic, 10Operations, 10Performance-Team, 10Thumbor: SwiftMedia URL rewrite returns some 404s with wrong Content-Length - https://phabricator.wikimedia.org/T222071 (10Gilles) a:03Gilles [15:52:36] 10Traffic, 10Operations, 10Core Platform Team Backlog (Next), 10Services (next): Have Varnish set the `X-Request-Id` header for incoming external requests - https://phabricator.wikimedia.org/T221976 (10ema) p:05Triage→03Normal [16:15:45] 10Domains, 10Traffic: Register wiki(m|p)edia.ro - https://phabricator.wikimedia.org/T222080 (10Strainu) [16:17:38] 10Traffic, 10Operations, 10decommission, 10ops-codfw: Decommission acamar and achernar - https://phabricator.wikimedia.org/T198286 (10Papaul) ` papaul@asw-a-codfw# run show interfaces ge-5/0/13 descriptions Interface Admin Link Description ge-5/0/13 down down DISABLED papaul@asw-b-codfw#... [16:19:41] 10Traffic, 10Operations, 10decommission, 10ops-codfw: Decommission acamar and achernar - https://phabricator.wikimedia.org/T198286 (10Papaul) [17:09:00] 10netops, 10Operations, 10fundraising-tech-ops, 10Patch-For-Review: Revoke production prometheus fundraising access - https://phabricator.wikimedia.org/T217355 (10cwdent) commit 5e6951f6082627a1f61f38d27532a48197eada4f Author: Casey Dentinger Date: Mon Apr 29 17:03:... [17:09:30] 10netops, 10Operations, 10fundraising-tech-ops, 10Patch-For-Review: Revoke production prometheus fundraising access - https://phabricator.wikimedia.org/T217355 (10cwdent) commit 173b34cd01bdc0aa5998af347406d9775755656f (HEAD -> master, origin/master, origin/HEAD) Author: Casey Dentinger 10netops, 10Operations, 10fundraising-tech-ops, 10Patch-For-Review: Revoke production prometheus fundraising access - https://phabricator.wikimedia.org/T217355 (10cwdent) [17:38:54] 10Traffic, 10DNS, 10Operations, 10fundraising-tech-ops, 10Patch-For-Review: remove IBM/Silverpop 1024-bit domain key - https://phabricator.wikimedia.org/T214525 (10Jgreen) 05Open→03Resolved a:03Jgreen this is deployed [18:44:40] 10Traffic, 10Operations, 10Security-Team: scan external ranges with current Nessus rulesets - https://phabricator.wikimedia.org/T222097 (10chasemp) [19:01:17] 10Domains, 10Traffic, 10Operations: Register wiki(m|p)edia.ro - https://phabricator.wikimedia.org/T222080 (10Dzahn) Hi @Strainu please contact @CRoslof directly in this matter. Thank you! [19:11:00] 10netops, 10Operations, 10fundraising-tech-ops, 10Patch-For-Review: Network setup for frmon2001.frack.codfw.wmnet - https://phabricator.wikimedia.org/T221475 (10cwdent) @papaul it looks like you wired this up in T196557 but I have tried the hw addresses reported by racadm and am not seeing dhcp packets, co... [19:36:04] 10netops, 10Operations, 10fundraising-tech-ops, 10Patch-For-Review: Network setup for frmon2001.frack.codfw.wmnet - https://phabricator.wikimedia.org/T221475 (10Dzahn) a:05cwdent→03Papaul [20:23:01] 10Traffic, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: Decommission acamar and achernar - https://phabricator.wikimedia.org/T198286 (10Dzahn) [20:35:49] 10Traffic, 10Operations, 10Performance-Team: Send peering requests to AS with the worst TTFB - https://phabricator.wikimedia.org/T219486 (10kchapman) a:03Gilles [20:50:04] 10netops, 10Operations, 10fundraising-tech-ops, 10Patch-For-Review: Network setup for frmon2001.frack.codfw.wmnet - https://phabricator.wikimedia.org/T221475 (10Papaul) a:05Papaul→03cwdent @cwdent the switch ports were not setup. You should be good now. ` papaul@fasw-c-codfw# run show interfaces ge-[0... [20:55:26] 10Traffic, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: Decommission acamar and achernar - https://phabricator.wikimedia.org/T198286 (10Papaul) [20:55:42] 10Traffic, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: Decommission acamar and achernar - https://phabricator.wikimedia.org/T198286 (10Papaul) 05Open→03Resolved Complete