[00:10:51] (03PS2) 10Krinkle: mediawiki-core-npm: Use default entry point instead of tests/frontend [integration/config] - 10https://gerrit.wikimedia.org/r/183635 [00:11:03] (03PS3) 10Krinkle: mediawiki-core-npm: Use default entry point instead of tests/frontend [integration/config] - 10https://gerrit.wikimedia.org/r/183635 [00:37:04] 3MediaWiki-Core-Team, Release-Engineering, operations: Update servers in scap rsync proxy pool - https://phabricator.wikimedia.org/T1342#974944 (10Reedy) https://gerrit.wikimedia.org/r/#/c/184817/ [00:45:15] (03CR) 10Krinkle: [C: 032] "Pushed to Jenkins." [integration/config] - 10https://gerrit.wikimedia.org/r/183635 (owner: 10Krinkle) [00:53:55] (03Merged) 10jenkins-bot: mediawiki-core-npm: Use default entry point instead of tests/frontend [integration/config] - 10https://gerrit.wikimedia.org/r/183635 (owner: 10Krinkle) [00:57:57] 3Continuous-Integration: Migrate Jenkins slaves from Ubuntu Trusty to Debian Jessie - https://phabricator.wikimedia.org/T86728#975006 (10Krinkle) 3NEW [00:58:26] 3Continuous-Integration: Migrate Jenkins slaves from Ubuntu Trusty to Debian Jessie - https://phabricator.wikimedia.org/T86728#975006 (10Krinkle) [01:05:56] (03PS1) 10Krinkle: macro: Remove unused 'npm-in' builder [integration/config] - 10https://gerrit.wikimedia.org/r/184826 [01:54:37] Project beta-scap-eqiad build #38113: FAILURE in 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38113/ [02:04:02] !log integration-slave1006 IOError: Lock for file '/mnt/jenkins-workspace/workspace/mediawiki-extensions-hhvm/src/extensions/WikiGrok/.git/config' did already exist, delete '/mnt/jenkins-workspace/workspace/mediawiki-extensions-hhvm/src/extensions/WikiGrok/.git/config.lock' in case the lock is illegal [02:04:06] Logged the message, Master [02:05:13] !log There is some kind of race / conflict with the mediawiki-extensions-hhvm; I cleaned up the same error for a different extension yesterday [02:05:15] Logged the message, Master [02:05:25] Project beta-scap-eqiad build #38114: STILL FAILING in 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38114/ [02:07:41] !sal [02:07:41] https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:11:29] 3Continuous-Integration: Git clone corruption by mediawiki-extensions-hhvm job on integration-slave1006 - https://phabricator.wikimedia.org/T86730#975087 (10bd808) 3NEW [02:15:30] Yippee, build fixed! [02:15:30] Project beta-scap-eqiad build #38115: FIXED in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38115/ [03:54:06] Yippee, build fixed! [03:54:07] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #477: FIXED in 43 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/477/ [04:24:15] greg-g: Updated: https://wikitech.wikimedia.org/wiki/Deployments#Week_of_Jan_12th [04:24:38] However, let me know if we can move it earlier if someone is available to assist in that timezone. [04:37:30] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<33.33%) [04:37:40] Yippee, build fixed! [04:37:41] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #251: FIXED in 43 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/251/ [05:01:38] 3Phabricator: Prevent private information being leaked via Herald notifications - https://phabricator.wikimedia.org/T493#975222 (10mmodell) [05:02:08] 3Phabricator: Prevent private information being leaked via Herald notifications - https://phabricator.wikimedia.org/T493#5825 (10mmodell) This will be fixed when the next phabricator release goes live. [05:07:37] 3Phabricator: Show correct external IP addresses (XFF support) in activity log - https://phabricator.wikimedia.org/T840#975226 (10mmodell) @qgil: It was abandoned because @chasemp wasn't comfortable with it for some reason. He wanted this handled separate from the redirection code (although that's where it belon... [05:16:23] (03PS1) 10Ori.livneh: Ensure the repository configuration lock is released [integration/zuul] - 10https://gerrit.wikimedia.org/r/184838 [05:16:50] (03PS2) 10Ori.livneh: Ensure the repository configuration lock is released [integration/zuul] - 10https://gerrit.wikimedia.org/r/184838 [05:17:36] Krinkle|detached: ^ [05:19:48] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #454: FAILURE in 23 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/454/ [05:20:02] 3Phabricator: Show correct external IP addresses (XFF support) in activity log - https://phabricator.wikimedia.org/T840#975231 (10mmodell) Re-submitted this change: https://gerrit.wikimedia.org/r/#/c/184837/ [05:20:23] (03PS3) 10Ori.livneh: Ensure the repository configuration lock is released [integration/zuul] - 10https://gerrit.wikimedia.org/r/184838 (https://phabricator.wikimedia.org/T86734) [05:20:37] (03CR) 10Ori.livneh: "(This is completely untested, by the way.)" [integration/zuul] - 10https://gerrit.wikimedia.org/r/184838 (https://phabricator.wikimedia.org/T86734) (owner: 10Ori.livneh) [05:21:52] 3Phabricator: OAuth login refers to mediawiki.org:/ - https://phabricator.wikimedia.org/T623#975232 (10mmodell) @qgil: This is unfortunately somewhat difficult to fix. Since it's a very minor cosmetic detail I don't expect to get around to fixing this any time soon. [06:03:31] 3Phabricator: Show correct external IP addresses (XFF support) in activity log - https://phabricator.wikimedia.org/T840#975265 (10chasemp) Ah, yes we waited on this but only for post migration so this should be fine. [06:37:31] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [06:50:51] Yippee, build fixed! [06:50:51] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #405: FIXED in 26 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/405/ [08:15:04] Project beta-scap-eqiad build #38150: FAILURE in 46 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38150/ [08:25:21] Project beta-scap-eqiad build #38151: STILL FAILING in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38151/ [08:35:27] Yippee, build fixed! [08:35:27] Project beta-scap-eqiad build #38152: FIXED in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38152/ [08:46:45] 3operations, Beta-Cluster: File upload area resorts to 0777 permissions to for uploaded content - https://phabricator.wikimedia.org/T75206#975349 (10yuvipanda) Why is this needed again? Just having the user on the labstore machines seems to fix the permissions issues. [08:51:12] 3operations, Beta-Cluster: Renumber apache user/group to uid=48 - https://phabricator.wikimedia.org/T78076#975352 (10yuvipanda) Why is this needed again? T76086 seems to have fixed T75206. And as @ori said, we should be agnostic about the numeric values unless there's a very good reason. [08:54:33] 3Phabricator: "MediaWiki Userpage: Unknown" on Phabricator user profiles is tacky - https://phabricator.wikimedia.org/T903#975357 (10Qgil) Note that this is also the case for LDAP: LDAP User Unknown However, if a user doesn't introduce an IRC handle, the IRC field is not shown. I find more elegant the o... [08:58:58] 3Phabricator: "Blocked By" list sort criteria should be documented - https://phabricator.wikimedia.org/T75702#975363 (10Qgil) 5Open>3Resolved a:3Qgil Documented: https://www.mediawiki.org/w/index.php?title=Phabricator%2FHelp&diff=1354429&oldid=1350288 [09:03:21] 3Code-Review: Allow cloning of Phabricator hosted git repositories - https://phabricator.wikimedia.org/T128#975369 (10Qgil) [09:05:28] 3Beta-Cluster: Remove all ::beta roles in puppet - https://phabricator.wikimedia.org/T86644#975374 (10yuvipanda) [09:05:30] 3Beta-Cluster: Kill role::logstash::beta - https://phabricator.wikimedia.org/T86642#975372 (10yuvipanda) 5Open>3Resolved a:3yuvipanda [09:56:17] (03PS1) 10Hashar: check pipelines should only have lint jobs [integration/config] - 10https://gerrit.wikimedia.org/r/184862 [09:58:31] (03PS1) 10Hashar: Remove sphinx-doc jobs [integration/config] - 10https://gerrit.wikimedia.org/r/184863 [09:58:57] (03CR) 10Hashar: "tox and other test jobs being in check pipeline is an issue. I have cleared the Zuul configuration and added a test with https://gerrit.wi" [integration/config] - 10https://gerrit.wikimedia.org/r/184778 (owner: 10Hashar) [09:59:48] (03PS2) 10Hashar: check pipelines should only have lint jobs [integration/config] - 10https://gerrit.wikimedia.org/r/184862 [09:59:50] (03PS2) 10Hashar: Remove sphinx-doc jobs [integration/config] - 10https://gerrit.wikimedia.org/r/184863 [10:00:07] legoktm: ^^^ :-D [10:00:18] legoktm: I am getting rid of the tox / tests jobs from the Zuul check pipelines [10:01:41] (03CR) 10Hashar: [C: 032] check pipelines should only have lint jobs [integration/config] - 10https://gerrit.wikimedia.org/r/184862 (owner: 10Hashar) [10:02:57] (03PS3) 10Hashar: Remove sphinx-doc jobs [integration/config] - 10https://gerrit.wikimedia.org/r/184863 [10:02:59] (03Merged) 10jenkins-bot: check pipelines should only have lint jobs [integration/config] - 10https://gerrit.wikimedia.org/r/184862 (owner: 10Hashar) [10:04:10] hashar: did you see my zuul patch? [10:04:18] ori: not yet [10:04:38] https://gerrit.wikimedia.org/r/#/c/184838/ [10:06:02] my english is crap [10:06:13] what is the diff between: "did you see..." and "have you seen" ? [10:08:18] 'did you see' suggests that there was some event in the past and either you saw it or you didn't [10:08:30] 'have you seen' suggests there still exists opportunity to see something now [10:08:41] 'have you seen' would have been more appropriate of me to ask [10:08:44] the difference is quite subtle [10:09:25] ahhh [10:10:03] ori: thanks for the explanation. I am sure I will eventually forget about it but at least for a moment I feel like I have learned something. [10:10:41] oh that is actually a Zuul patch great [10:11:59] 3Continuous-Integration: Zuul-cloner failing to acquire lock sometimes ("IOError: Lock for file .git/config did already exist, lock is illegal") - https://phabricator.wikimedia.org/T86734#975471 (10hashar) [10:14:00] ori: do you have an account on OpenStack Gerrit http://review.openstack.org ? [10:14:16] ori: I get all patches proposed directly to upstream, that runs their test and get review from them [10:14:25] then eventually cherry pick in our fork [10:14:50] their repo is ssh://review.openstack.org:29418/openstack-infra/zuul [10:15:28] (03CR) 10Hashar: [C: 032] "I have deleted the jobs in Jenkins" [integration/config] - 10https://gerrit.wikimedia.org/r/184863 (owner: 10Hashar) [10:18:14] (03CR) 10Hashar: "Poor GitPython :-)" [integration/zuul] - 10https://gerrit.wikimedia.org/r/184838 (https://phabricator.wikimedia.org/T86734) (owner: 10Ori.livneh) [10:18:39] ori: and yeah your patch looks legit :] [10:21:31] cool. if you like, i can try to submit it upstream sometime tomorrow (i think i have an account there, i submitted some git-review patches in the past) [10:21:43] but if you want to do it yourself that's fine too [10:22:23] (03Merged) 10jenkins-bot: Remove sphinx-doc jobs [integration/config] - 10https://gerrit.wikimedia.org/r/184863 (owner: 10Hashar) [10:23:57] ori: I would prefer you send it, I don't mind following up with them to get it merged though [10:24:14] ori: that is for legal reason. I am not sure I can legally push code written by other on behalf of WMF [10:24:25] ah, ok [10:24:35] I am not a lawyer though :] [10:27:25] hashar: https://review.openstack.org/#/c/147101/ [10:28:27] * ori sleeps [10:28:45] (03CR) 10Ori.livneh: "Done: https://review.openstack.org/#/c/147101/" [integration/zuul] - 10https://gerrit.wikimedia.org/r/184838 (https://phabricator.wikimedia.org/T86734) (owner: 10Ori.livneh) [10:31:25] ori: sweet dreams! [10:40:36] 3Release-Engineering, Continuous-Integration: MWDS 2015 - State of CI, what we did in 2014 - https://phabricator.wikimedia.org/T86750#975528 (10hashar) 3NEW [10:42:25] 3Release-Engineering, Continuous-Integration: MWDS 2015 - State of CI, what we want to do in 2015 - https://phabricator.wikimedia.org/T86752#975540 (10hashar) 3NEW [10:51:47] (03PS3) 10Hashar: Activate flake8 on Python 3 for pywikibot [integration/config] - 10https://gerrit.wikimedia.org/r/182067 (owner: 10XZise) [11:01:42] (03CR) 10Hashar: [C: 032] "Awesome! I have deployed the Jenkins jobs, the Zuul layout is exactly what was needed." [integration/config] - 10https://gerrit.wikimedia.org/r/182067 (owner: 10XZise) [11:08:25] (03Merged) 10jenkins-bot: Activate flake8 on Python 3 for pywikibot [integration/config] - 10https://gerrit.wikimedia.org/r/182067 (owner: 10XZise) [12:26:12] (03PS12) 10Adrian Lang: Make mwext-WikibaseJavaScriptApi-qunit voting [integration/config] - 10https://gerrit.wikimedia.org/r/180418 [12:27:09] (03PS13) 10Adrian Lang: Make mwext-WikibaseJavaScriptApi-qunit voting [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (https://phabricator.wikimedia.org/T86176) [12:27:31] (03PS14) 10Adrian Lang: Make mwext-WikibaseJavaScriptApi-qunit voting [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (https://phabricator.wikimedia.org/T86176) [13:35:32] 3Phabricator: Prevent private information being leaked via Herald notifications - https://phabricator.wikimedia.org/T493#975777 (10chasemp) >>! In T493#975222, @mmodell wrote: > This will be fixed when the next phabricator release goes live. I don't know man, I think we probably can't enable herald for now. At... [14:06:49] Yippee, build fixed! [14:06:50] Project browsertests-Wikidata-PerformanceTests-linux-firefox-sauce build #119: FIXED in 48 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-PerformanceTests-linux-firefox-sauce/119/ [14:11:52] (03Restored) 10Hashar: Code review by whitelisted users should triggers tests [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/127259 (https://bugzilla.wikimedia.org/62429) (owner: 10Hashar) [14:12:34] (03PS1) 10Hashar: Code review by whitelisted users should triggers tests [integration/config] - 10https://gerrit.wikimedia.org/r/184886 (https://phabricator.wikimedia.org/T64429) [14:13:00] (03Abandoned) 10Hashar: Code review by whitelisted users should triggers tests [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/127259 (https://bugzilla.wikimedia.org/62429) (owner: 10Hashar) [14:13:50] (03CR) 10Hashar: [C: 04-1] "I have no idea how approval works, would need to craft a test for it." [integration/config] - 10https://gerrit.wikimedia.org/r/184886 (https://phabricator.wikimedia.org/T64429) (owner: 10Hashar) [14:26:16] (03CR) 10Hashar: "One path needs to be adjusted :-)" (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (https://phabricator.wikimedia.org/T86176) (owner: 10Adrian Lang) [14:29:20] (03PS15) 10Adrian Lang: Make mwext-WikibaseJavaScriptApi-qunit voting [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (https://phabricator.wikimedia.org/T86176) [14:30:14] (03CR) 10Adrian Lang: Make mwext-WikibaseJavaScriptApi-qunit voting (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (https://phabricator.wikimedia.org/T86176) (owner: 10Adrian Lang) [14:30:27] (03PS2) 10Hashar: Code review by whitelisted users should triggers tests [integration/config] - 10https://gerrit.wikimedia.org/r/184886 (https://phabricator.wikimedia.org/T64429) [14:30:48] I was following https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Case_1d:_new_extension [14:31:19] (and found that ContentTranslation is part of submodule, which is not deployed yet, is this because Beta deployment?) [14:31:25] Nikerabbit: ^ [14:34:41] Project beta-scap-eqiad build #38190: FAILURE in 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38190/ [14:39:40] (03CR) 10Hashar: Make mwext-WikibaseJavaScriptApi-qunit voting (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (https://phabricator.wikimedia.org/T86176) (owner: 10Adrian Lang) [14:39:55] (03PS16) 10Hashar: Make mwext-WikibaseJavaScriptApi-qunit voting [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (https://phabricator.wikimedia.org/T86176) (owner: 10Adrian Lang) [14:44:44] Project beta-scap-eqiad build #38191: STILL FAILING in 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38191/ [14:46:21] (03CR) 10Hashar: "Job refreshed :)" [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (https://phabricator.wikimedia.org/T86176) (owner: 10Adrian Lang) [14:51:04] 3Phabricator: Next Phabricator upgrade on 2015-01-14 - https://phabricator.wikimedia.org/T78243#975880 (10chasemp) p:5Low>3High [14:55:02] Yippee, build fixed! [14:55:02] Project beta-scap-eqiad build #38192: FIXED in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38192/ [14:55:30] 3Phabricator: Next Phabricator upgrade on 2015-01-14 - https://phabricator.wikimedia.org/T78243#975896 (10chasemp) [14:56:06] !log Cherry-pick https://gerrit.wikimedia.org/r/#/c/173336/11/ to Beta Labs [14:56:12] Logged the message, Master [14:56:43] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:10:17] I want to cherry pick https://gerrit.wikimedia.org/r/#/c/184714/ - says couldn't cherry-pick merge commit. [15:10:20] Anyone? [15:13:19] kart_: it has already been merged [15:13:36] oh, you want to cp it onto another branch? [15:16:43] kart_: I think the issue is the cherry pick fails: [15:16:46] (py27)valhallasw@maeglin:ContentTranslation$ git cherry-pick 8545e432f15732cfb3158d6629d99d0ae46a47a9 33e813d 0 [15:16:46] error: could not apply 8545e43... Publishing options: Update version to highest version automatically [15:17:02] that's for cherry-picking to remotes/origin/wmf/1.25wmf14 [15:21:40] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [15:30:36] 3Phabricator: Next Phabricator upgrade on 2015-01-14 - https://phabricator.wikimedia.org/T78243#975956 (10chasemp) I closed the maint window, I think things are ok (did brief resanity testing that the securityevent listener is...listening). Search has to reindex which will take awhile. Currently at 19% [15:31:29] 3Phabricator: Next Phabricator upgrade on 2015-01-14 - https://phabricator.wikimedia.org/T78243#975957 (10chasemp) fwiw indexable docs: ```Indexing 17 object of type CONP. Indexing 86144 object of type TASK. Indexing 5 object of type CDTL. Indexing 999 object of type PROJ. Indexing 371602 object of type CMIT. I... [15:36:37] 3Phabricator, Phabricator.org: "Access Denied" for commit emails sent by diffusion - https://phabricator.wikimedia.org/T78154#975966 (10chasemp) [15:37:17] valhallasw`cloud: any more clues? [15:37:34] 3Phabricator, Phabricator.org: "Access Denied" for commit emails sent by diffusion - https://phabricator.wikimedia.org/T78154#837052 (10chasemp) 5Open>3Resolved a:3chasemp I am breaking apart the perms and emails issues. The perms one if fixed, the email one is upstream still. [15:38:14] 3Phabricator, Phabricator.org: "Access Denied" for commit emails sent by diffusion - https://phabricator.wikimedia.org/T78154#975985 (10chasemp) note: {T86769} [15:38:17] kart_: I [15:38:20] valhallasw`cloud: If I want to cherry-pick master which is 'merge' commit. Don't know how to do that :) [15:38:32] kart_: I'm not sure what clue you're looking for [15:38:35] kart_: ?? [15:39:01] valhallasw`cloud: need to update extension to master [15:39:32] kart_: I have no clue what you mean. You mean you're using the extension and want to update it? [15:39:49] 3Phabricator, Phabricator.org: "Access Denied" for commit emails sent by diffusion - https://phabricator.wikimedia.org/T78154#975987 (10chasemp) [15:39:51] 3Phabricator: Access denied when trying to view Diffusion commits - https://phabricator.wikimedia.org/T85047#975988 (10chasemp) [15:42:13] valhallasw`cloud: yes [15:42:20] kart_: git pull, then [15:45:45] valhallasw`cloud: nope. This: https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Case_1c:_extension_update [15:47:23] kart_: okay, so you want to update branch wmf/1.25wmf14 to include that specific change you linked? Yes, then you need to cherry-pick, but you need to do it manually, because there's a merge conflict. [15:47:57] valhallasw`cloud: same if I want master, right? [15:48:18] kart_: no. [15:48:28] kart_: please explain more clearly what you're trying to accomplish. [15:48:44] I mean I want to update ContentTranslation to master in wmf/1.25wmf14 [15:48:45] are we talking about a WMF deployment here? a local wiki that has to be updated? a branch that has to be updated? [15:49:01] valhallasw`cloud: WMF deployment [15:51:00] kart_: ok... [15:51:37] kart_: so you cherry-pick the *changes* (not the merges!) that need to be included in the 1.25wmf14 [15:53:34] so in this case git cherry-pick 8545e432f15732cfb3158d6629d99d0ae46a47a9 [15:54:01] ok. As you said, there is conflict :) [15:56:59] (03CR) 10Adrian Lang: [C: 04-1] "Retriggered https://gerrit.wikimedia.org/r/#/c/183445/. It still fails, because testextension also needs the dependency (obviously)." [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (https://phabricator.wikimedia.org/T86176) (owner: 10Adrian Lang) [16:02:09] 3Phabricator: Next Phabricator upgrade on YYYY-MM-DD - https://phabricator.wikimedia.org/T86772#976049 (10Qgil) 3NEW [16:05:03] 3Phabricator: Next Phabricator upgrade on YYYY-MM-DD - https://phabricator.wikimedia.org/T86772#976071 (10Qgil) [16:06:48] 3Phabricator: Access denied when trying to view Diffusion commits - https://phabricator.wikimedia.org/T85047#976078 (10chasemp) [16:06:52] 3Phabricator, operations: The options of the Security dropdown in Phabricator need to be clear and documented - https://phabricator.wikimedia.org/T76564#976080 (10chasemp) [16:06:53] 3Phabricator, Phabricator.org: "Access Denied" for commit emails sent by diffusion - https://phabricator.wikimedia.org/T78154#976079 (10chasemp) [16:06:55] 3Phabricator, Phabricator.org: Phabricator search does not search substrings - https://phabricator.wikimedia.org/T679#976083 (10chasemp) [16:06:57] 3Phabricator: Prevent private information being leaked via Herald notifications - https://phabricator.wikimedia.org/T493#976081 (10chasemp) [16:06:58] 3Phabricator, Phabricator.org: Can't quote old Phabricator comments - https://phabricator.wikimedia.org/T85982#976084 (10chasemp) [16:07:04] 3Phabricator, Phabricator.org: Can't search an exact phrase in Phabricator - https://phabricator.wikimedia.org/T75743#976087 (10chasemp) [16:07:12] 3Phabricator: Next Phabricator upgrade on 2015-01-14 - https://phabricator.wikimedia.org/T78243#976075 (10chasemp) 5Open>3Resolved >>! In T78243#975957, @chasemp wrote: > fwiw indexable docs: > > ```Indexing 17 object of type CONP. > Indexing 86144 object of type TASK. > Indexing 5 object of type CDTL. > In... [16:07:44] 3Phabricator: Phabricator upgrade on 2015-01-14 - https://phabricator.wikimedia.org/T78243#976102 (10chasemp) [16:08:41] 3operations, Beta-Cluster: Renumber apache user/group to uid=48 - https://phabricator.wikimedia.org/T78076#976105 (10bd808) >>! In T78076#975352, @yuvipanda wrote: > Why is this needed again? T76086 seems to have fixed T75206. And as @ori said, we should be agnostic about the numeric values unless there's a very... [16:09:19] bd808: I’m confused. Isn’t NFS setting perms by username instead of uid? [16:09:25] at least that’s how I understood Coren’s message [16:09:36] and that’s how it seems to work for my projects (Quarry, which uses NFS for storage) [16:10:01] on the server, yes. On the hosts no it uses the local uid/gid mappings [16:10:19] A process has a uid not a username [16:10:52] valhallasw`cloud: done this, https://gerrit.wikimedia.org/r/#/c/184902/ - hope that it is correct :) [16:11:27] git-review always look for Change-Id as last line, while in conflict it won't be the case. [16:11:36] kart_: I think so, but there's a test failure. [16:11:52] YuviPanda: if you do an `ls -l` on all the hosts they need to agree that the uid/gid of the inode matches the uid/gid of the process in order to allow access via the file permissions [16:12:04] hmm [16:12:24] kart_: so you probably need to cherry-pick some older commits as well [16:12:30] valhallasw`cloud: gah. Who made rubocop tests voting? [16:12:39] bd808: I’m now curious how my Quarry set up works at all :) [16:12:42] kart_: hashar_, probably :-p [16:13:07] valhallasw`cloud: why older commit? We just need it on this commit. [16:13:11] (latest) [16:13:16] YuviPanda: ? Quarry runs a the same ldap user on all grid nodes does it not? [16:13:56] bd808: hmm, I don’t actually remember. [16:14:00] but yeah, can’t do that with apache [16:14:35] If quarry is a tool it runs as the tool user which comes from ldap and is thus the same on all hosts [16:15:18] the problem with the "apache" user in beta is that it is a host local user rather than an ldap user [16:15:21] Reedy: able to help in ContentTranslation deployment? :) [16:15:28] Are we doing it today? [16:15:44] Reedy: yes. Confirmed with greg. [16:16:06] Reedy: so I cherry picked CX to https://gerrit.wikimedia.org/r/#/c/184902/ [16:16:12] see if that's correct. [16:16:21] Excuse my brevity (etc) [16:16:34] bd808: no, quarry is its own project. [16:16:39] valhallasw`cloud: you've been very helpful. Thank you! [16:16:56] bd808: I’ll investigate more after food [16:17:22] kart_: I guess we can ignore the thing jenkins is complaining about? [16:17:56] Reedy: yep [16:18:23] hashar_: can you make Rubocop tests non-voting :D (no hurry) [16:18:33] YuviPanda: The thing that bit the beta project is that on the older Precise hosts the apache user was created via a deb package that set the uid at 48 and on the trusty hosts that package wasn't used so Puppet got a basically random uid assigned to the user. [16:24:18] kart_: which version of CX are you cherry-picking? [16:24:31] (I mean the master branch, not wmf/1.25wmf14 .) [16:24:59] 8545e432f15732cfb3158d6629d99d0ae46a47a9 - from CX [16:25:18] am I doing wrong? [16:25:40] https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Case_1b:_extension_changes - followed this [16:25:43] Reedy: ^ [16:25:58] Are you doing what wrong? [16:26:30] Reedy: cherry picking process :) [16:27:13] I'm slightly confused [16:27:16] What do you think you're doing wrong? [16:27:44] Reedy: Just followed https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Case_1b:_extension_changes to update ContentTranslation. [16:27:57] Right [16:28:00] 3operations, Code-Review, Wikimedia-Git-or-Gerrit: Chrome warns about insecure certificate on gerrit.wikimedia.org - https://phabricator.wikimedia.org/T76562#976157 (10mark) >>! In T76562#964609, @RobH wrote: > So this won't work behind misc-web-lb until after the gerrit dependency on using sshd on the same inte... [16:28:07] Ok. Thanks! [16:28:45] Do you only need to get a couple of commits? [16:28:49] https://github.com/wikimedia/mediawiki-extensions-ContentTranslation/branches [16:28:59] that says wmf14 is 47 commits behind master [16:31:51] ah [16:32:06] Are you wanting to get wmf14 the same as master? [16:32:10] Reedy: we need to update CX to master [16:32:12] yes [16:32:18] Right, in which case, it's a bit special [16:32:30] ^d: Delete branch, then rebranch, right? [16:33:56] <^d> You can do that, yeah [16:34:08] Certainly seems easier than 47 cherry picks :P [16:34:34] Blah to me. [16:34:46] <^d> Reedy: I suppose unless you're a cherry farmer. [16:34:53] kart_: I can sort that out [16:35:01] Except I don't currently have access to my usual dev checkout [16:35:05] * Reedy clones CT somewhere else [16:36:39] kart_: what about wmf15? [16:36:52] Wait... [16:37:03] wmf14 is going dead in a couple of hours in this deploy window [16:37:03] have we branched wmf15? [16:37:14] thought so. [16:37:17] oh [16:37:18] no [16:37:22] I'm tired [16:37:24] 1.24 != 1.25 [16:37:37] yeah, wmf14 is right [16:37:41] * Reedy deletes 1.25wmf14 [16:38:25] 3Continuous-Integration: Git clone corruption by mediawiki-extensions-hhvm job on integration-slave1006 - https://phabricator.wikimedia.org/T86730#976230 (10greg) p:5Triage>3Unbreak! @cmcmahon / @mmodell / @dduvall / @hashar / whoever is awake: This is blocking merges during a SWAT deploy right now. [16:39:11] 3operations, Code-Review, Wikimedia-Git-or-Gerrit: Chrome warns about insecure certificate on gerrit.wikimedia.org - https://phabricator.wikimedia.org/T76562#976232 (10Glaisher) [16:40:15] twentyafterfour: chrismcmahon: you two are online, so, you get IRC pings too ^^ :) [16:40:34] ok, kart_ https://git.wikimedia.org/log/mediawiki%2Fextensions%2FContentTranslation/refs%2Fheads%2Fwmf%2F1.25wmf14 [16:41:33] Reedy: nice [16:42:00] greg-g: should I update Deployment page? [16:42:06] So we just need to update the submodule reference in core now so we can deploy it [16:42:11] kart_: yeah, that'd help, I'm doing a few things at once right now :) [16:42:16] thank you :) [16:46:00] bd808: rm -rf'ing the WikiGrok checkout on that slave will "Just Work" and be recreated on next run without any other intervention? [16:46:19] * greg-g has the command ready to hit enter [16:46:54] timo and antoine aren't here to confirm with [16:47:16] though I have a 1:1 with antoine in 14 minutes, I want to talk about other things :) [16:47:17] Receiving objects: 100% (539617/539617), 290.26 MiB | 12.24 MiB/s, done. [16:47:19] core clone is big [16:47:21] Reedy: https://gerrit.wikimedia.org/r/#/c/181546/ - is the mw-config change [16:47:29] Reedy: yeah. [16:47:41] ^d: wonder if it's time for a core repack [16:47:58] greg-g: yes. that was how I fixed it the last time [16:48:23] permission denied [16:48:24] sudo? [16:48:24] <^d> Reedy: Running gerrit gc on all repos [16:48:47] and/or that :) [16:48:51] oh, reedy isn't voiced, I should fix that [16:49:02] greg-g: oh yeah sudo -u ... something. Do an ls -l to see who owns that stuff [16:49:19] jenkins-deploy wikidev [16:49:30] jenkins-deploy [16:49:41] ^d: you're doing that now? Or we should? Shall I file a ticket? [16:50:00] <^d> I am [16:50:04] <^d> It'll be done in an hour or so [16:50:09] ace, thanks [16:50:34] damnit [16:50:35] gjg is not allowed to run sudo on integration-slave1006. This incident will be reported. [16:50:57] greg-g: 2 weeks in and you're already on the naughty list [16:50:57] <^d> awwww, greg-g's in trouble! [16:51:04] :( :( [16:51:10] coal for me next year [16:51:33] do you need someone from ops to stare sternly at greg-g? [16:51:36] * YuviPanda stares sternly at greg-g [16:51:40] gj YuviPanda [16:51:49] ty Reedy [16:52:06] can someone with sudo: [16:52:13] on integration1006: cd /mnt/jenkins-workspace/workspace/mediawiki-extensions-hhvm/src/extensions [16:52:17] sudo -u jenkins-deploy rm -rf WikiGrok/ [16:52:20] greg-g: I don’t have sudo but let me give you sudo [16:52:25] ty [16:53:17] YuviPanda: I assume it'll be for all of the integration slaves in labs? [16:53:26] greg-g: oh, you probably already have sudo [16:53:27] greg-g: tru [16:53:28] try [16:53:29] sudo su [16:53:29] first [16:53:34] and then the command you tried before [16:54:33] YuviPanda: same [16:54:41] reported incident :( [16:54:49] kart_: https://gerrit.wikimedia.org/r/184908 is the bump commit [16:54:54] greg-g: hmm. looking. [16:54:57] wikitech is being terribly slow [16:55:15] greg-g: ooo, found it [16:55:15] moment [16:55:58] 3Continuous-Integration: Make sure everyone in RelEng has sudo on integration slaves in labs - https://phabricator.wikimedia.org/T86779#976298 (10greg) 3NEW [16:56:03] (you have 5 minutes) [16:56:19] Reedy: looking [16:56:41] greg-g: I am in if you wanna start [16:56:42] kart_, greg-g, Reedy - I am around if help is needed. [16:56:53] 11:52 <+ greg-g> on integration1006: cd /mnt/jenkins-workspace/workspace/mediawiki-extensions-hhvm/src/extensions [16:56:56] 11:52 <+ greg-g> sudo -u jenkins-deploy rm -rf WikiGrok/ [16:57:12] aharoni: cool [16:57:16] greg-g: try now. [16:57:17] kart_: what time are we supposed to be starting? [16:57:28] YuviPanda: works, thankyou! [16:57:38] (And I plan to stay around for a few more hours.) [16:57:41] greg-g: yw. [16:57:48] Reedy: +2 hours from now. [16:57:53] I will also be around if help is needed. [16:58:03] Ah, so part of the usual deploy window? [16:58:30] not having to run scap extra times is always nice [16:58:36] !log rm -rf'd the Wikigrok checkout in integration-slave1006:/mnt/jenkins-workspace/workspace/mediawiki-extensions-hhvm/src/extensions to (hopefully) fix https://phabricator.wikimedia.org/T86730 [16:58:39] Logged the message, Master [16:58:59] Reedy: we can do it whenever you're fine. Before or after :) [16:58:59] bah [16:59:04] 3Continuous-Integration: Git clone corruption by mediawiki-extensions-hhvm job on integration-slave1006 - https://phabricator.wikimedia.org/T86730#976307 (10greg) I rm -rf'd the WikiGroke checkout just now, let's see.... [16:59:14] ori has a patch for it [16:59:17] oh, hashar is back [16:59:33] greg-g: The reason I opened that bug instead of just fixing it last night was to get hashar's attention about the recurring problem. :) [16:59:38] hashar: I rm -rf'd / on all the integration slaves, hope that's ok! :P [17:00:09] 3Continuous-Integration: Git clone corruption by mediawiki-extensions-hhvm job on integration-slave1006 - https://phabricator.wikimedia.org/T86730#976313 (10hashar) That is a dupe of {T86734} , ori proposed a patch to Zuul to release the lock properly. [17:00:13] bd808: yeah, recurring issue still stands, but I didn't want to block merges [17:00:33] I am not sure why it started to occurs [17:00:51] maybe some jenkins jobs are interrupted in the middle of a zuul cloner operation which would leave the git lock behind [17:01:56] will attempt to cherry pick it later tonight [17:04:02] 3Continuous-Integration: Zuul-cloner failing to acquire lock sometimes ("IOError: Lock for file .git/config did already exist, lock is illegal") - https://phabricator.wikimedia.org/T86734#976333 (10hashar) [17:04:59] Reedy: as you wish/free - I and others from Language team are around. [17:07:57] I need to work out about going home. Preferably, I'd not have to run scap twice ;) [17:08:08] hashar: I work now! even with video! [17:19:13] Reedy: sure. Let me know. [17:29:35] YuviPanda: btw, what'd you do to give me sudo? [17:29:46] (so I can do it for others in my team when/if needed) [17:30:03] ("YOU get sudo! You get sudo! EVERYONE GETS SUDO!") [17:30:19] greg-g: sure! go to https://wikitech.wikimedia.org/wiki/Special:NovaSudoer, select ‘integration’ project, ‘modify’ for the default rule, and check more people :) [17:30:23] (but you've gotta be able to login first) [17:30:31] greg-g: you can also check with hashar and have it default to ‘all members’ and so everyone in the project will have sudo [17:30:31] Reedy: :P [17:30:38] * greg-g nods [17:30:44] sudo make me a sandwich [17:31:04] sudo meeting-attend -u Reedy [17:31:27] (1:1 right now :) ) [17:31:29] yeah [17:31:35] (03PS1) 10Hashar: WIP zuul tests WIP [integration/config] - 10https://gerrit.wikimedia.org/r/184920 [17:32:07] (03CR) 10Hashar: [C: 04-2] "Saving my local workspace externally :-D" [integration/config] - 10https://gerrit.wikimedia.org/r/184920 (owner: 10Hashar) [17:32:41] I guess all members of the RelengTeam should have sudo on integration project [17:32:48] anyway, I am off [17:53:28] Woo, yeah, project 1000! https://phabricator.wikimedia.org/project/view/1000/ [17:53:29] ;-) [17:55:14] James_F: you should re-use it as 'next deployment' tag ;-) [17:55:28] valhallasw`cloud: No, these are permanent tags for deployments. [17:55:42] valhallasw`cloud: See https://www.mediawiki.org/wiki/VisualEditor/changelog [17:55:57] 3: LocalisationUpdate broken since 2014-12-16 - https://phabricator.wikimedia.org/T85790#976537 (10Reedy) So we're back to ``` 02:16:13 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'php-1.25wmf13', '--include', 'php-1.25wmf13/cache', '--include', 'php-1.25wmf13/cache/l10n', '--... [17:57:25] 3: LocalisationUpdate broken since 2014-12-16 - https://phabricator.wikimedia.org/T85790#976551 (10bd808) >>! In T85790#976537, @Reedy wrote: > So we're back to > > ``` > 02:16:13 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'php-1.25wmf13', '--include', 'php-1.25wmf13/cache',... [17:58:19] 3: LocalisationUpdate broken since 2014-12-16 - https://phabricator.wikimedia.org/T85790#976553 (10Reedy) [17:58:21] 3: LocalisationUpdate is not updating messages - https://phabricator.wikimedia.org/T76061#976554 (10Reedy) [18:04:43] 3Phabricator: Phabricator upgrade on 2015-01-14 - https://phabricator.wikimedia.org/T78243#976586 (10Aklapper) [18:05:45] Yippee, build fixed! [18:05:45] Project browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs build #386: FIXED in 43 sec: https://integration.wikimedia.org/ci/job/browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs/386/ [18:21:46] oh, deployment-systems is a tag not a project? [18:23:09] Reedy: are we doing CX deployment before mw-train or after it? [18:23:13] 3Continuous-Integration: Zuul-cloner failing to acquire lock sometimes ("IOError: Lock for file .git/config did already exist, lock is illegal") - https://phabricator.wikimedia.org/T86734#976651 (10Krinkle) 5duplicate>3Open p:5Triage>3Normal a:3ori [18:24:43] 3Continuous-Integration: Zuul-cloner failing to acquire lock sometimes ("IOError: Lock for file .git/config did already exist, lock is illegal") - https://phabricator.wikimedia.org/T86734#975131 (10Krinkle) Manually ssh-ing into a machine and fixing a single workspace broken by this bug is not a resolution. The... [18:26:33] 3Phabricator: Next Phabricator upgrade on YYYY-MM-DD - https://phabricator.wikimedia.org/T86772#976685 (10Qgil) [18:26:35] Krinkle: https://review.openstack.org/#/c/147101/ [18:27:07] Krinkle: i didn't know that they target python 2.6 compat, though -- I think 2.6 doesn't have try / finally (but you can emulate it with try / except / else IIRC -- it's been a while!) [18:27:46] Wow, yeah [18:28:06] 3Phabricator: Next Phabricator upgrade on YYYY-MM-DD - https://phabricator.wikimedia.org/T86772#976049 (10Qgil) [18:28:19] actually, it looks like 2.6 has finally [18:28:52] i'm not sure the test failure is related, in that case [18:40:47] greg-g: from SoS, I'm going to highlight this for us: https://phabricator.wikimedia.org/T86374 [18:42:01] * greg-g nods [18:42:11] antoine knows, help appreciated [18:43:14] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #252: FAILURE in 36 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/252/ [18:48:19] ori: #openstack-infra here on Freenode, also. If you wanna hit 'em up. [19:03:03] Reedy: which one to go first? I think deployment announce bot should stay here too? [19:03:23] kart_: we need to scap to go with your updates to wmf14 [19:03:28] Which I'll be doing as part of the train [19:04:53] kart_: also, I guess we need the wikis to be on wmf14 before. So it makes sense to get out of the way first [19:05:42] kart_: more than happy if you want to do the config changes for your stuff [19:07:51] Reedy: if you've time and help me for config changes, that will be nice. First time for me :) [19:08:00] yeah, should be fine :) [19:09:43] Am I going mad, or did the position of the "Subscribe" option just go down a bunch in the Task view? [19:10:01] (it's working guys!) [19:10:17] James_F: you're not paranoid, we really are after you [19:10:25] :-P [19:14:00] James_F: There was a phab version update earlier today [19:14:39] bd808: Hmm. That might be it. Though why they'd push a frequent-use feature down is… I was going to say "beyond me", but it isn't. :-) [19:14:53] :) engineers [19:15:00] Bingo. [19:15:00] they just like to change stuff [19:15:12] It's like day traders. [19:15:18] They love change. [19:26:39] 3Phabricator: Cannot view Bugzilla migrated private/security tasks I am author/reporter of - https://phabricator.wikimedia.org/T75781#976892 (10chasemp) 5Open>3Resolved I programmatically went through every imported issue from the bugzilla security product and adapted the policies to include the original aut... [19:29:57] (03PS3) 10Hashar: Code review by whitelisted users should triggers tests [integration/config] - 10https://gerrit.wikimedia.org/r/184886 (https://phabricator.wikimedia.org/T64429) [19:29:59] (03PS1) 10Hashar: zuul: test 'recheck' behavior [integration/config] - 10https://gerrit.wikimedia.org/r/184967 [19:30:01] (03PS1) 10Hashar: zuul: test check/test behavior [integration/config] - 10https://gerrit.wikimedia.org/r/184968 [19:30:39] (03Abandoned) 10Hashar: WIP zuul tests WIP [integration/config] - 10https://gerrit.wikimedia.org/r/184920 (owner: 10Hashar) [19:34:09] (03CR) 10Hashar: "I am not sure whom else to ask for review :D" [integration/config] - 10https://gerrit.wikimedia.org/r/184967 (owner: 10Hashar) [19:34:19] 3Phabricator: Fix search in Wikimedia Phabricator - https://phabricator.wikimedia.org/T75854#976923 (10Aklapper) [19:34:44] 3Phabricator: Lots of unrelated results when searching for specific string - https://phabricator.wikimedia.org/T86805#976918 (10Aklapper) [19:39:41] (03CR) 10Hashar: "PS3 is ready for review. I have added some explanation to the inline diff." (035 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/184886 (https://phabricator.wikimedia.org/T64429) (owner: 10Hashar) [19:39:59] 3Continuous-Integration: Zuul: run 'test' jobs on jenkins when trusted user votes +1 and only 'check' jobs was ran - https://phabricator.wikimedia.org/T64429#976963 (10hashar) a:3hashar [19:56:40] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:57:50] Project beta-scap-eqiad build #38220: FAILURE in 3 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38220/ [19:58:58] Hmm [19:59:03] I can't create a paste in phab now [19:59:10] Request: POST http://phabricator.wikimedia.org/paste/create/, from 10.64.0.171 via cp1043 cp1043 ([10.64.0.171]:80), Varnish XID 1241111695 [19:59:10] Forwarded for: 2001:8b0:fafa:8023:74d4:b11a:5cef:ec4a, 10.64.0.171 [19:59:10] Error: 503, Service Unavailable at Wed, 14 Jan 2015 19:59:04 GMT [20:00:26] specific paste [20:01:42] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:02:22] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [20:04:08] 3: scap barfs if an extension has more than one remote - https://phabricator.wikimedia.org/T86811#977054 (10Reedy) p:5Triage>3Low [20:04:13] bd808|LUNCH: ^^ [20:04:21] sync-proxies: 0% (ok: 0; fail: 0; left: 6) [20:04:23] (03PS2) 10Krinkle: macro: Remove unused 'npm-in' builder [integration/config] - 10https://gerrit.wikimedia.org/r/184826 [20:04:28] (03CR) 10Krinkle: [C: 032] macro: Remove unused 'npm-in' builder [integration/config] - 10https://gerrit.wikimedia.org/r/184826 (owner: 10Krinkle) [20:04:51] (03PS4) 10Krinkle: Ensure the repository configuration lock is released [integration/zuul] - 10https://gerrit.wikimedia.org/r/184838 (https://phabricator.wikimedia.org/T86734) (owner: 10Ori.livneh) [20:07:30] 20:03:20 Started sync-proxies [20:07:30] sync-proxies: 100% (ok: 6; fail: 0; left: 0) [20:07:30] 20:05:27 Finished sync-proxies (duration: 02m 07s) [20:07:33] That looks promising [20:07:51] Yippee, build fixed! [20:07:51] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #530: FIXED in 56 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/530/ [20:08:31] Project beta-scap-eqiad build #38221: STILL FAILING in 4 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38221/ [20:10:43] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:10:50] (03CR) 10jenkins-bot: [V: 04-1] macro: Remove unused 'npm-in' builder [integration/config] - 10https://gerrit.wikimedia.org/r/184826 (owner: 10Krinkle) [20:12:43] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:13:25] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:14:23] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [20:14:49] (03PS5) 10Hashar: Ensure the repository configuration lock is released [integration/zuul] - 10https://gerrit.wikimedia.org/r/184838 (owner: 10Ori.livneh) [20:16:38] 3Continuous-Integration: Git clone corruption by mediawiki-extensions-hhvm job on integration-slave1006 - https://phabricator.wikimedia.org/T86730#977072 (10hashar) Ori proposed a patch upstream https://review.openstack.org/#/c/147101/1/ which properly release the lock after the config has been written. I have... [20:21:11] bd808: we might be on track for a 20 minute scap [20:21:20] sweet [20:21:38] That scap git-info bug is annoying [20:21:50] * kart_ is watching [20:22:23] Reedy: did the one scap proxy get reactivated yet? [20:22:24] * bd808 looks at it closer [20:22:30] should i check? [20:22:38] mutante: no, I don't think so [20:22:41] but I added more [20:22:52] ah, i saw that change, it got merged? [20:22:55] cool [20:23:06] yup, I got joe to do it earler [20:23:26] great, then i won't worry about it [20:29:58] Yippee, build fixed! [20:29:59] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #455: FIXED in 22 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/455/ [20:30:52] kart_: we should be nearly ready to go [20:30:57] 3Continuous-Integration: Git clone corruption by mediawiki-extensions-hhvm job on integration-slave1006 - https://phabricator.wikimedia.org/T86730#977163 (10hashar) p:5Unbreak!>3Normal I have deployed Ori patch on all labs slaves as well as the two production slaves. Should hopefully fix the issue. Lets kee... [20:31:00] I've not reviewed your config change yet, so I'll do that now [20:31:47] Nice :) [20:33:34] Parse error: syntax error, unexpected T_STRING in /srv/mediawiki-staging/wmf-config/interwiki.cdb on line 3029 [20:33:37] stupid php [20:34:21] Project beta-scap-eqiad build #38222: STILL FAILING in 20 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38222/ [20:36:42] !log Zuul applied Ori patch to fix a git lock contention in Zuul-cloner {{bug|T86730}} . Tagged wmf-deploy-20150114-1 [20:36:45] Logged the message, Master [20:37:04] !log Restarting Zuul [20:37:08] Logged the message, Master [20:37:22] kart_: couple of minor issues I saw [20:38:02] Reedy: sure. Can I fix right now? [20:38:09] Yup, that's fine :) [20:38:16] kart_: have the db tables been created in production? [20:38:48] Yes [20:38:59] Sean did that. [20:41:59] greg-g: Zuul cloner git lock should be fixed now [20:42:03] Project beta-scap-eqiad build #38223: STILL FAILING in 6 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38223/ [20:42:05] though previous lock would still be around [20:43:33] Reedy: updated [20:44:04] 3Continuous-Integration: Git clone corruption by mediawiki-extensions-hhvm job on integration-slave1006 - https://phabricator.wikimedia.org/T86730#977242 (10hashar) [20:44:31] 3Continuous-Integration: [fixed, pending upstream merge] Git clone corruption by mediawiki-extensions-hhvm job on integration-slave1006 - https://phabricator.wikimedia.org/T86730#977244 (10hashar) [20:44:32] hashar: yay [20:44:45] gotta clean up the lock file :/ [20:45:59] kart_: LGTM. Ready to do it? :) [20:46:11] Reedy: yeah [20:46:39] Reedy: consider me as total newbie in this unchartered territory :) [20:46:44] That's fine [20:46:48] Are you logged onto tin? [20:47:17] now [20:47:19] in [20:47:51] I just +2'd your commit, so jenkins will merge it [20:48:03] kart_: Want me to tell you what to do, or are you following wikitech? :) [20:48:30] Reedy: here, it will be good. I scated seeing wmf1.20 stuffs. [20:48:43] scared [20:48:55] haha, that's just the version numbers not being updated :) [20:48:58] I'll keep log and frame it :) [20:49:09] Yeah. [20:49:24] Oh damn it [20:49:27] I just noticed an issue [20:49:59] Project beta-scap-eqiad build #38224: STILL FAILING in 5 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38224/ [20:50:21] Reedy: is it in config? [20:50:26] Not that config [20:50:37] ContentTranslation isn't in the normal extension-list, so it's not going to be included in the localisation cache [20:51:03] So we need to scap again [20:51:15] :( [20:51:28] bahhhhh [20:51:47] but it's fast now! [20:52:02] and it will only rebuild l10n for one branch right? [20:52:04] Yeah [20:52:07] Oh, no [20:52:08] 2 [20:52:12] Reedy: How does one add an extension to the normal extension-list? [20:52:44] aharoni: kart_ https://gerrit.wikimedia.org/r/184994 [20:53:09] let me run scap quickly to fix this [20:53:35] Reedy: why removal from labs? [20:53:43] same question from me [20:53:46] Because labs loads the production one [20:53:54] the -labs one is for anything that's "just" on labs [20:53:59] oh here too. Good to know :) [20:55:29] Project beta-scap-eqiad build #38225: STILL FAILING in 5 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38225/ [20:55:53] OH NOES [20:58:34] (03CR) 10Hashar: "I have pushed it in our repo:" [integration/zuul] - 10https://gerrit.wikimedia.org/r/184838 (owner: 10Ori.livneh) [21:00:19] 3Phabricator, Phabricator.org: "Access Denied" for commit emails sent by diffusion - https://phabricator.wikimedia.org/T78154#977290 (10Nemo_bis) https://www.google.it/search?q="download+raw+diff"+site:phabricator.wikimedia.org currently has 33k results for me. Zero on yahoo.com. [21:01:19] Project beta-scap-eqiad build #38226: STILL FAILING in 4 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38226/ [21:02:12] bd808: 20:55:33 Updating LocalisationCache for 1.25wmf14 using 4 thread(s) [21:02:17] Can we use more threads? :P [21:02:27] it took 4 minutes and 8 seconds! [21:02:36] it picks based on the number of cores in the box [21:02:51] so get a bigger replacement for tin :) [21:03:07] hmm [21:03:10] tin has 6? [21:03:13] (or one with ssd which I think would make a bigger difference) [21:03:32] I think it does N-2 with a min of 2 [21:03:42] HT? [21:03:51] might help [21:04:01] 3operations, Code-Review, Wikimedia-Git-or-Gerrit: Chrome warns about insecure certificate on gerrit.wikimedia.org - https://phabricator.wikimedia.org/T76562#977293 (10RobH) Well, first Faidon advised this wouldn't work, and that I shouldn't waste time on it. I wanted to confirm why it wouldn't work, since I im... [21:04:50] hmm, ht is enabled already it seems [21:05:18] wikitech down by any chance? [21:05:40] (it is me, working) [21:07:34] Reedy: Is this one I need to do? https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment#In_your_own_repo_via_gerrit [21:08:29] Project beta-scap-eqiad build #38227: STILL FAILING in 4 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38227/ [21:09:19] Reedy: twentyafterfour we've had a big string of failed scaps in beta cluster: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/ [21:09:48] 21:06:19 Permission denied (publickey). [21:09:53] muther [21:10:41] 21:06:19 sudo -u mwdeploy -n -- /srv/deployment/scap/scap/bin/scap-rebuild-cdbs on deployment-rsync01.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-rsync01.eqiad.wmflabs,10.68.17.66' (ECDSA) to the list of known hosts. [21:11:49] that's the same issue we blamed on dns [21:17:03] "Permanently added 'deployment-rsync01.eqiad.wmflabs,10.68.17.66' (ECDSA) to the list of known hosts." is not the error, that's just noise on stderr. The real error is "Permission denied (publickey)." [21:17:45] something changed how the ssh keys propagate and/or removed the hacks in the private puppet repo that setup the keys [21:19:34] Project beta-scap-eqiad build #38228: STILL FAILING in 5 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38228/ [21:19:49] that sounds really familiar [21:20:32] https://gerrit.wikimedia.org/r/#/q/owner:%22Faidon+Liambotis+%253Cfaidon%2540wikimedia.org%253E%22++status:merged+message:key,n,z [21:20:41] see those changes from Jan 8 [21:20:42] YuviPanda: ya'll went on a local hack removal spree recently, right? did you do anything that might affect key management? [21:20:49] re: ssh host keys [21:21:09] eek [21:21:11] wtf [21:21:12] i suspect it is: [21:21:16] ssh: purge unmanaged ssh_known_hosts keys [21:21:28] can we please test things on beta cluster when we do shit like that/ [21:22:03] i am just guessing because they happened recently, https://gerrit.wikimedia.org/r/#/c/183531/ [21:22:09] do we know since when it fails? [21:22:17] if it worked after Jan 8, ignore me [21:22:38] mutante: right, I'll rescend comment for now :) [21:23:14] this shows the failing solidly jsut starting today: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/ [21:24:56] ok.. then it must be something else [21:25:30] kart_: Ok, now we're ready [21:26:05] Reedy: so I'm in tin and in /srv/mediawiki-staging [21:26:16] git pull [21:29:44] [steady, go?] [21:32:45] Project beta-scap-eqiad build #38229: STILL FAILING in 8 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38229/ [21:39:15] 3operations, Code-Review, Wikimedia-Git-or-Gerrit: Chrome warns about insecure certificate on gerrit.wikimedia.org - https://phabricator.wikimedia.org/T76562#977352 (10RobH) Ok, I was wrong about some of the above, Chad volunteered a bit more time so we could properly detail this. Gerrit runs and listens to 808... [21:40:02] Project beta-scap-eqiad build #38230: STILL FAILING in 6 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38230/ [21:43:28] 9:55:25 rsync: failed to connect to deployment-bastion.eqiad.wmflabs (10.68.16.58): Connection timed out (110) [21:43:30] 19:55:25 rsync error: error in socket IO (code 10) at clientserver.c(122) [Receiver=3.0.9] [21:43:41] that's the first failure I could find [21:44:15] it's connecting back to bastion from deployment-rsync01 [21:49:31] Project beta-scap-eqiad build #38231: STILL FAILING in 5 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38231/ [21:50:22] 3operations, Code-Review, Wikimedia-Git-or-Gerrit: Chrome warns about insecure certificate on gerrit.wikimedia.org - https://phabricator.wikimedia.org/T76562#977384 (10Dzahn) for the related issue that we want to make Gerrit listen on 22, and not on 29418, which we'd like anyways, besides the cert issue being in... [21:51:53] 3operations, Code-Review, Wikimedia-Git-or-Gerrit: Chrome warns about insecure certificate on gerrit.wikimedia.org - https://phabricator.wikimedia.org/T76562#977392 (10Dzahn) the suggestion to use an iptables rule to forward 22 to 29418 came from Faidon recently on IRC [22:00:16] there does seem to be a problem with the dns resolver configuration [22:00:55] twentyafterfour@deployment-rsync01:~$ host deployment-rsync01 [22:00:57] deployment-rsync01.eqiad.wmflabs has address 10.68.17.66 [22:00:59] Host deployment-rsync01.eqiad.wmflabs not found: 2(SERVFAIL) [22:01:01] Host deployment-rsync01.eqiad.wmflabs not found: 2(SERVFAIL) [22:07:19] Yippee, build fixed! [22:07:19] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #400: FIXED in 15 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/400/ [22:08:55] 3Phabricator: Duplicate notification received by phabricator for gerrit merged commits - https://phabricator.wikimedia.org/T78242#977431 (10Nemo_bis) It's a duplicate because gerrit already notifies me of my changes. [22:12:00] Project browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce build #424: FAILURE in 56 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce/424/ [22:18:40] Project beta-scap-eqiad build #38232: STILL FAILING in 22 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38232/ [22:23:01] twentyafterfour: if you're stuck (and with 10 minutes before our 1:1) file a bug with info and let's get Reedy to take a look when he's done deploying everything for everyone [22:23:57] greg-g: I'm just poking at it [22:24:10] getting to the bottom of what's happening [22:25:29] * greg-g nods [22:25:44] twentyafterfour: if you're in the middle, we can postpone our 1:1, too [22:26:13] 3Code-Review: Align Wikimedia and Phabricator code review processes - https://phabricator.wikimedia.org/T167#977491 (10bd808) [22:26:15] well how important is this beta scap job? I can continue after our 1:1 [22:26:34] I was just trying to figure out if it's really dns issue or not [22:26:47] I think dns is borked but not badly enough to be the cause [22:28:05] Project beta-scap-eqiad build #38233: STILL FAILING in 6 min 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38233/ [22:29:15] "rsync: failed to connect to deployment-bastion.eqiad.wmflabs (10.68.16.58): Connection timed out (110)" that doesn't look like dns [22:29:17] twentyafterfour: it should be fixed sooner than later [22:29:19] 3Phabricator: Mingle migration script - https://phabricator.wikimedia.org/T822#977507 (10Nemo_bis) What happens of the closed cards in mingle if they're not migrated? Is there a long-term preservation strategy for the mingle instances? [22:29:28] considering how long it's been dead for [22:29:40] bd808: yeah it's not dns [22:29:50] can't actually connect from rsync01 to bastion [22:30:03] new ferm firewall rules [22:30:05] I bet [22:30:27] there was an email on the ops list that ferm is everywhere now [22:30:41] ferm adds a default drop firewall rule [22:31:00] so every service has to declare a ferm rule to open ports [22:31:19] oh geez [22:31:26] even within virtual environments? [22:31:39] yup [22:31:49] It is host level firewalls [22:32:12] so mutante broke it :) [22:32:47] greg-g: meeting? [22:34:07] twentyafterfour: cool [22:34:10] ok so we have to add firewall rules in puppet? is this documented somewhere? [22:34:37] twentyafterfour: Grep operations/puppet for "ferm::service" [22:34:54] I bet we are just missing some for rsync that prod has [22:35:46] modules/beta/manifests/scap/master.pp and modules/beta/manifests/scap/rsync_slave.pp need them I woudl guess [22:35:58] Project beta-scap-eqiad build #38234: STILL FAILING in 6 min 49 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38234/ [22:36:46] just something like "ferm::service { 'rsync': proto => 'tcp', port => '873', }" should work [22:37:58] back [22:38:06] are those instance using the udp2log role? [22:38:24] deploment-bastion does, yes [22:38:27] the "ferm is everywhere" mail referred to a long process that has been going on [22:38:36] the recent change just happened to be the last user of the old rules [22:38:43] so, there were 2 steps to that [22:39:03] one change switched it over to ferm but still allowed everything not UDP [22:39:30] not a huge deal, we just need to open up the rsync port on deployment-bastion [22:39:36] and then there was https://gerrit.wikimedia.org/r/#/c/184695/ [22:40:05] twentyafterfour: So the ferm rule is probably only needed in modules/beta/manifests/scap/master.pp [22:40:16] yes, indeed, exactly what you said above , should go into a role class [22:42:20] 3Code-Review: Align Wikimedia and Phabricator code review processes - https://phabricator.wikimedia.org/T167#977552 (10Rastus_Vernon) [22:42:32] 3Phabricator: Remove usage of fontawesome-webfont.woff to save 1 second load time - https://phabricator.wikimedia.org/T86845#977553 (10Nemo_bis) 3NEW [22:42:48] 3Release-Engineering, Continuous-Integration: 2015 MediaWiki Developer Summit - State of continuous integration (CI), what we did in 2014 - https://phabricator.wikimedia.org/T86750#977560 (10MZMcBride) [22:43:24] 3Release-Engineering, Continuous-Integration: 2015 MediaWiki Developer Summit - State of continuous integration (CI), what we want to do in 2015 - https://phabricator.wikimedia.org/T86752#977563 (10MZMcBride) [22:44:22] so what you need , connections to 873 from ? [22:44:52] in prod it probably uses srange ALL_NETWORKS or INTERNAL or so [22:53:50] 3: scap barfs if an extension has more than one remote - https://phabricator.wikimedia.org/T86811#977596 (10bd808) I have a patch locally that "might" fix this, but I guess the first question I'd ask before complicating the function in scap is if we really want to allow multiple remotes for repos on tin. This se... [22:54:35] which bastion do you use to get to deployment-bastion [22:54:50] whatever the labs bastion is [22:54:55] shouldnt bastion mean it has a public IP [22:55:07] deployment-bastion is really "deployment-tin" badly named [22:55:34] 3: scap barfs if an extension has more than one remote - https://phabricator.wikimedia.org/T86811#977614 (10Reedy) >>! In T86811#977596, @bd808 wrote: > I have a patch locally that "might" fix this, but I guess the first question I'd ask before complicating the function in scap is if we really want to allow mult... [22:55:42] alright [22:56:25] Project beta-scap-eqiad build #38235: STILL FAILING in 19 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38235/ [22:56:26] renames it to "tin.wmflabs.org" :) [22:56:38] perfect [22:57:35] uses bastion-restricted [22:58:34] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<55.56%) [22:58:55] what.. that too now [23:00:11] 3Phabricator: Compress phabricator assets (favicon, CSS, JS etc.) - https://phabricator.wikimedia.org/T86849#977633 (10Nemo_bis) 3NEW [23:03:17] Project beta-scap-eqiad build #38236: STILL FAILING in 5 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38236/ [23:08:15] bd808: twentyafterfour https://gerrit.wikimedia.org/r/#/c/185085/ [23:09:06] LGTM can you cherry-pick and see if it fixes the scap jenkins job? [23:10:03] Project beta-scap-eqiad build #38237: STILL FAILING in 5 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38237/ [23:10:54] i dont know how that works [23:16:08] 3Phabricator: Remove usage of fontawesome-webfont.woff to save 1 second load time - https://phabricator.wikimedia.org/T86845#977672 (10Qgil) Are you asking to remove all the icons provided by fontawesone to save that second? Just curious: is this extra second for the first load of a page within a session or mo... [23:17:06] mutante: https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/How_code_is_updated#Cherry-picking_a_patch_from_gerrit [23:21:59] 3Phabricator: Duplicate notification received by phabricator for gerrit merged commits - https://phabricator.wikimedia.org/T78242#977696 (10Qgil) Well, then you can just adjust your notifications at https://phabricator.wikimedia.org/settings/panel/emailpreferences/ I think this is an Invalid task. Phabricator... [23:22:15] !log cherry-picked I1e5f9f7bcbbe6c4 on deployment-bastion [23:22:19] Logged the message, Master [23:22:20] bd808: did it [23:22:42] i saw the ferm change [23:23:01] now "scap jenkins job"?? [23:23:19] It is actually running right now -- https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/38238/console [23:23:41] alright [23:23:43] ACCEPT tcp -- 10.68.16.0/21 anywhere tcp dpt:rsync [23:31:24] * bd808 twiddles his thumbs while scap runs in beta [23:32:04] 23:30:01 Finished mw-update-l10n (duration: 14m 58s) [23:32:16] 23:30:16 Started sync-proxies ... [23:32:46] 23:32:32 Finished sync-proxies (duration: 02m 16s) [23:32:55] mutante: ^ I think it's going to work [23:34:02] that sounds great, i kept watching the jenkins job, but still running [23:35:20] twentyafterfour: ^ good news, hopefully [23:35:24] thanks mutante and bd808 [23:36:46] i checked how long it took in the past, and it was over 1h [23:37:13] oh, no, completely wrong.. wut.. [23:37:18] 1m ? [23:37:43] the time varies pretty wildly mostly based on the l10n update time [23:37:51] ok [23:37:55] which was long for this run (15m) [23:38:12] 23:32:32 Finished sync-proxies (duration: 02m 16s) [23:38:21] 23:36:47 Finished sync-apaches (duration: 04m 14s) [23:38:27] it's going to work [23:38:33] :) [23:38:37] those were the steps that were failing [23:38:53] just merge on prod master already? [23:39:07] sounds good to me [23:39:28] (03CR) 10Krinkle: [C: 032] "HTTP 503?" [integration/config] - 10https://gerrit.wikimedia.org/r/184826 (owner: 10Krinkle) [23:40:36] Yippee, build fixed! [23:40:36] Project beta-scap-eqiad build #38238: FIXED in 26 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38238/ [23:40:42] yay [23:40:55] any steps needed on deployment-salt after merge on prod? [23:41:18] greg-g: ^ see the bot [23:41:20] Yippee! [23:41:33] thanks much mutante [23:41:33] mutante: nope. it will automerge and clean itself up as long as the patch matches what was merged [23:41:45] bd808: perfect [23:41:46] greg-g: yw [23:43:23] Project beta-scap-eqiad build #38239: FAILURE in 1 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38239/ [23:43:37] eh, what ?:p [23:44:35] what's wrong with that next build now.. sigh [23:44:44] and so quick to fail [23:45:23] Yippee, build fixed! [23:45:24] Project beta-scap-eqiad build #38240: FIXED in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/38240/ [23:46:03] wmf-insecte: mixed messages :) [23:46:03] mutante you may not issue bot commands in this chat! [23:46:31] PROBLEM - Free space - all mounts on deployment-cache-upload02 is CRITICAL: CRITICAL: deployment-prep.deployment-cache-upload02.diskspace._srv_vdb.byte_percentfree.value (<100.00%) [23:46:42] (03Merged) 10jenkins-bot: macro: Remove unused 'npm-in' builder [integration/config] - 10https://gerrit.wikimedia.org/r/184826 (owner: 10Krinkle) [23:46:43] heh