[00:36:51] 6Release-Engineering-Team, 15User-greg: Outline roles and responsibilities for SWAT members - https://phabricator.wikimedia.org/T129905#2128195 (10greg) p:5High>3Normal Lowering priority. I am not going to have time to do this this week. The Annual Plan work is taking up all of my time and we're a little b... [01:31:39] (03PS2) 10Legoktm: Set MEDIAWIKI_ENVIRONMENT environment variable for MultimediaViewer browsertests job [integration/config] - 10https://gerrit.wikimedia.org/r/276469 (https://phabricator.wikimedia.org/T129472) (owner: 10Zfilipin) [01:32:32] WARNING:jenkins_jobs.registry:You have a macro ('doxygen') defined for 'builder' component type that is masking an inbuilt definition [01:32:32] WARNING:jenkins_jobs.registry:You have a macro ('tox') defined for 'builder' component type that is masking an inbuilt definition [01:39:43] (03CR) 10Legoktm: "I was going to deploy after seeing IRC scrollback this except it touches 44 jobs and I'm not that comfortable deploying that large of a ch" [integration/config] - 10https://gerrit.wikimedia.org/r/276469 (https://phabricator.wikimedia.org/T129472) (owner: 10Zfilipin) [02:48:56] PROBLEM - Free space - all mounts on deployment-jobrunner01 is CRITICAL: CRITICAL: deployment-prep.deployment-jobrunner01.diskspace.root.byte_percentfree (<40.00%) [03:10:00] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #914: 04FAILURE in 19 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/914/ [04:24:01] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #753: 04FAILURE in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/753/ [04:43:17] RECOVERY - Puppet run on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [05:27:18] Yippee, build fixed! [05:27:18] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #729: 09FIXED in 25 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/729/ [06:19:35] Since the codfw switchover was postponed, and the deployment freeze is lifted according to Mark's email, what's happening to wmf.18? Are things marked with WMF-deploy-2016-03-29_(1.27.0-wmf.18) in Phabricator still going to be deployed on March 29, or next week? [08:20:55] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #893: 04FAILURE in 54 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/893/ [08:49:21] 6Release-Engineering-Team: MW-1.27.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T128872#2128613 (10Legoktm) [08:55:42] 10Beta-Cluster-Infrastructure: deployment-jobrunner01/Free space - all mounts is CRITICAL - https://phabricator.wikimedia.org/T130179#2128617 (10hashar) [09:01:15] 10Beta-Cluster-Infrastructure: deployment-jobrunner01/Free space - all mounts is CRITICAL - https://phabricator.wikimedia.org/T130179#2128629 (10hashar) ``` du -d 1 -m /var/log/*|sort -rn|head -n5 1472 /var/log/hhvm 1297 /var/log/apache2 1021 /var/log/mediawiki 690 /var/log/account 425 /var/log/atop ``` Cleaned... [09:04:14] !log Upgrading hhvm and related extensions on jobrunner01 T130179 [09:04:15] T130179: deployment-jobrunner01/Free space - all mounts is CRITICAL - https://phabricator.wikimedia.org/T130179 [09:04:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:34:51] !log deployment-jobrunner01 deleted /var/log/apache/*.gz T130179 [09:34:52] T130179: deployment-jobrunner01/Free space - all mounts is CRITICAL - https://phabricator.wikimedia.org/T130179 [09:34:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:41:01] RECOVERY - Free space - all mounts on deployment-jobrunner01 is OK: OK: All targets OK [09:54:24] zeljkof: on gallium /var/lib/jenkins/jobs/browsertests-Wikidata-WikidataTests-linux-firefox-sauce/builds/5*/junitResult.xml [10:06:01] 10Beta-Cluster-Infrastructure: beta cluster 'labswiki' not referenced in all-labs.dblist causing jobrunner to error out - https://phabricator.wikimedia.org/T130184#2128791 (10hashar) [10:09:10] 10Beta-Cluster-Infrastructure: beta cluster 'labswiki' not referenced in all-labs.dblist causing jobrunner to error out - https://phabricator.wikimedia.org/T130184#2128811 (10hashar) The `labswiki` on beta cluster has been renamed to `deploymentwiki` by 352a57e1b18146fa945e0ace91045d2db750dae5 for {T72108}. Tha... [10:29:44] 10Beta-Cluster-Infrastructure: beta cluster 'labswiki' not referenced in all-labs.dblist causing jobrunner to error out - https://phabricator.wikimedia.org/T130184#2128850 (10hashar) [10:29:46] 10Beta-Cluster-Infrastructure, 10MediaWiki-JobRunner: Beta cluster job queue is quite crowed (~200 k jobs) - https://phabricator.wikimedia.org/T128357#2128849 (10hashar) [10:29:48] 10Beta-Cluster-Infrastructure: beta cluster 'labswiki' not referenced in all-labs.dblist causing jobrunner to error out - https://phabricator.wikimedia.org/T130184#2128791 (10hashar) On deployment-redis01: ``` > KEYS labswiki* 1) "labswiki:jobqueue:htmlCacheUpdate:h-data" 2) "labswiki:jobqueue:htmlCacheUpdate:h-... [10:29:58] 10Beta-Cluster-Infrastructure: deployment-jobrunner01/Free space - all mounts is CRITICAL - https://phabricator.wikimedia.org/T130179#2128854 (10hashar) [10:30:00] 10Beta-Cluster-Infrastructure: beta cluster 'labswiki' not referenced in all-labs.dblist causing jobrunner to error out - https://phabricator.wikimedia.org/T130184#2128852 (10hashar) 5Open>3Resolved a:3hashar [10:30:03] 10Beta-Cluster-Infrastructure, 10MediaWiki-JobRunner: Beta cluster job queue is quite crowed (~200 k jobs) - https://phabricator.wikimedia.org/T128357#2071532 (10hashar) [10:30:36] 5Gerrit-Migration, 10Differential: Phund Phacility to align Differential with Wikimedia workflow - https://phabricator.wikimedia.org/T130094#2128856 (10Aklapper) 5Open>3declined The listed subtasks are already tracked in #Gerrit-Migration and this task provides a specific solution ("Fund Phacility") instea... [10:30:46] 10Beta-Cluster-Infrastructure, 10MediaWiki-JobRunner: Beta cluster job queue is quite crowed (~200 k jobs) - https://phabricator.wikimedia.org/T128357#2071532 (10hashar) The jobrunner01 was kept busy due to errand labswiki jobs (the wiki has been renamed to deploymentwiki a while ago). At least that one is fix... [10:32:30] 10Beta-Cluster-Infrastructure: deployment-jobrunner01/Free space - all mounts is CRITICAL - https://phabricator.wikimedia.org/T130179#2128864 (10hashar) **Summary** The hhvm error.log is not rotated, I trimmed it The Apache vhost for the jobrunner RPC was filling with errors as well as /var/log/mediawiki/jobru... [10:42:42] 10Beta-Cluster-Infrastructure, 10MediaWiki-JobRunner: Beta cluster job queue is unmonitored / potentially not running all jobs - https://phabricator.wikimedia.org/T128357#2128892 (10hashar) [10:44:10] 10Beta-Cluster-Infrastructure, 10MediaWiki-JobRunner: Beta cluster job queue is unmonitored / potentially not running all jobs - https://phabricator.wikimedia.org/T128357#2071532 (10hashar) I originally filled the task stating the beta cluster job queue had 200k jobs. I made a mistake and was looking at the pr... [10:51:36] 10Beta-Cluster-Infrastructure: MediaWiki maintenance tasks from puppet not running on beta cluster - https://phabricator.wikimedia.org/T129095#2128911 (10hashar) [10:51:38] 10Beta-Cluster-Infrastructure: Run mediawiki::maintenance scripts? - https://phabricator.wikimedia.org/T125976#2128912 (10hashar) [10:52:29] 10Beta-Cluster-Infrastructure: Run mediawiki::maintenance scripts? - https://phabricator.wikimedia.org/T125976#2001990 (10hashar) [10:52:38] 10Beta-Cluster-Infrastructure: Run mediawiki::maintenance scripts? - https://phabricator.wikimedia.org/T125976#2001990 (10hashar) >>! In T129095#2115348, @Krenair wrote: > Unfortunately I think some of these crons try to do things like use all.dblist instead of all-labs.dblist, use s*.dblist, etc. > I wonder if... [10:53:06] 10Beta-Cluster-Infrastructure, 10MediaWiki-JobRunner: Beta cluster job queue is unmonitored / potentially not running all jobs - https://phabricator.wikimedia.org/T128357#2128919 (10hashar) Asked Giuseppe about it. In production the job queue statistics are generated by a maintenance script on terbium. It runs... [10:53:13] 10Beta-Cluster-Infrastructure, 10MediaWiki-JobRunner: Beta cluster job queue is unmonitored / potentially not running all jobs - https://phabricator.wikimedia.org/T128357#2128921 (10hashar) [10:53:15] 10Beta-Cluster-Infrastructure: Run mediawiki::maintenance scripts? - https://phabricator.wikimedia.org/T125976#2001990 (10hashar) [10:53:53] (03CR) 10Zfilipin: "I can deploy this myself, I am just waiting for somebody to +2 it. :)" [integration/config] - 10https://gerrit.wikimedia.org/r/276469 (https://phabricator.wikimedia.org/T129472) (owner: 10Zfilipin) [10:55:04] 5Gerrit-Migration, 10Differential: Define an equivalent to Gerrit's +-1 +-2 for code review evaluation - https://phabricator.wikimedia.org/T138#2128925 (10Aklapper) >>! In T138#2125419, @Ricordisamoa wrote: > How about switching to Differential's system "just because"? That does not sound like a good idea/"re... [10:56:59] 10Beta-Cluster-Infrastructure, 10MediaWiki-JobRunner: Beta cluster job queue is unmonitored / potentially not running all jobs - https://phabricator.wikimedia.org/T128357#2128929 (10hashar) [10:57:47] 10Beta-Cluster-Infrastructure, 10MediaWiki-JobRunner: Beta cluster job queue is unmonitored / potentially not running all jobs - https://phabricator.wikimedia.org/T128357#2071532 (10hashar) 5Open>3stalled p:5Triage>3Low CPU is no more a concern: {F3648860 size=full} Stalled until job queue metrics are... [11:02:31] (03CR) 10Hashar: [C: 032] Set MEDIAWIKI_ENVIRONMENT environment variable for MultimediaViewer browsertests job [integration/config] - 10https://gerrit.wikimedia.org/r/276469 (https://phabricator.wikimedia.org/T129472) (owner: 10Zfilipin) [11:02:38] zeljkof: https://gerrit.wikimedia.org/r/#/c/276469/ go go go :-} [11:03:50] hashar: deploying! 🚢 [11:04:17] 🍿 [11:04:45] U+1F37F [11:04:49] (03Merged) 10jenkins-bot: Set MEDIAWIKI_ENVIRONMENT environment variable for MultimediaViewer browsertests job [integration/config] - 10https://gerrit.wikimedia.org/r/276469 (https://phabricator.wikimedia.org/T129472) (owner: 10Zfilipin) [11:07:02] 10Beta-Cluster-Infrastructure, 7Easy, 7Mobile: MobileFrontEnd on Beta Cluster should display a different logo so that it is clearly not a production site - https://phabricator.wikimedia.org/T115078#2128943 (10hashar) [11:07:06] (03CR) 10Zfilipin: "All browsertests* jobs are refreshed." [integration/config] - 10https://gerrit.wikimedia.org/r/276469 (https://phabricator.wikimedia.org/T129472) (owner: 10Zfilipin) [11:07:15] \O/ [12:11:40] 10Gitblit-Deprecate, 6Release-Engineering-Team, 10Diffusion, 7WorkType-NewFunctionality: Use Diffusion as canonical location for browsing code repos (not gitblit) - https://phabricator.wikimedia.org/T752#2129134 (10hashar) [12:11:42] 5Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#2129131 (10hashar) 5Resolved>3Open https://www.mediawiki.org/wiki/Template:Git_file still invokes the the manually maintained mapping [[ https://www.mediawiki.org/wiki/Module:Callsig... [12:13:27] 10Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review, 7WorkType-NewFunctionality: redirect gerrit repo paths to diffusion callsigns - https://phabricator.wikimedia.org/T110607#2129136 (10hashar) Since rPHEX0beee3a5fe01 , the repo name -> call sign mapping is in Phabricator. We should then get rid of the ma... [12:15:39] 5Continuous-Integration-Scaling, 6DC-Ops, 6Operations, 10ops-eqiad: Reclaim SSD from labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T116936#2129237 (10Aklapper) [13:24:47] hashar: hey, scap3 doesn't support virtualenv (in order to deploy python codes) do you have suggestion on how to do that? [13:25:47] I couldn't find docs on how to register arbitrary bash commands [13:26:34] Amir1: I have no idea. Poke a task in phabricator for it ? :D [13:26:57] the proper persons to ask would be thcipriani / twentyafterfour / marxarelli I believe [13:27:26] I am off, have paper work sorry [13:27:48] I want to double check before I make redundant phab cards [13:27:54] have fun [13:27:58] thanks [13:35:03] Amir1: file a task, I don't believe it's a dupe (I just did a quick search) [13:35:31] thanks greg-g [13:37:45] 3Scap3: Scap3 should support virtualenv for deployment of python packages - https://phabricator.wikimedia.org/T130205#2129486 (10Ladsgroup) [13:43:43] 10Beta-Cluster-Infrastructure: Run mediawiki::maintenance scripts? - https://phabricator.wikimedia.org/T125976#2129502 (10greg) p:5Triage>3Normal [13:45:38] 10Beta-Cluster-Infrastructure, 6Labs, 7Tracking: Beta Cluster <-> WMF Labs policy compliance (tracking) - https://phabricator.wikimedia.org/T114615#2129523 (10greg) [13:45:40] 10Beta-Cluster-Infrastructure, 7Easy, 7Mobile: MobileFrontEnd on Beta Cluster should display a different logo so that it is clearly not a production site - https://phabricator.wikimedia.org/T115078#2129521 (10greg) 5Open>3Resolved Thanks @Jdlrobson ! [13:54:41] RECOVERY - Host cache-rsync is UP: PING OK - Packet loss = 0%, RTA = 1.07 ms [13:59:31] 5Continuous-Integration-Scaling, 10Pywikibot-i18n, 7WorkType-NewFunctionality: Migrate pywikibot/i18n npm job to Nodepool - https://phabricator.wikimedia.org/T130207#2129590 (10hashar) [14:03:18] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 7WorkType-NewFunctionality: Jobs should checkout source in a subdirectory of the workspace to avoid polluting it with CI related files - https://phabricator.wikimedia.org/T130208#2129672 (10hashar) [14:04:22] 3Scap3: Scap3 should support virtualenv for deployment of python packages - https://phabricator.wikimedia.org/T130205#2129486 (10mobrovac) The general approach to deploying services in WMF prod is to have a repo with the source code as well as its dependencies present. In other words, building dependencies on pr... [14:13:53] (03PS1) 10Hashar: castor: avoid polluting ZUUL variables [integration/config] - 10https://gerrit.wikimedia.org/r/278000 [14:15:41] 3Scap3: Scap3 should support virtualenv for deployment of python packages - https://phabricator.wikimedia.org/T130205#2129700 (10Ladsgroup) Thank you for your detailed comment. We have two repos one is the [[https://github.com/wiki-ai/ores|ores]] itself and the other one is [[https://github.com/wiki-ai/ores-wik... [14:19:21] (03PS1) 10Hashar: castor: share cache between ext and skins [integration/config] - 10https://gerrit.wikimedia.org/r/278001 [14:19:50] Amir1: thanks :) [14:20:09] np :) [14:20:26] I would appreciate if you chime in [14:21:08] Amir1: I will not. I am busy with a lot of other things, but surely rest of #releng will chime in [14:21:08] 3Scap3: Scap3 should support virtualenv for deployment of python packages - https://phabricator.wikimedia.org/T130205#2129734 (10Halfak) Hi @mobrovac. We've been working with #operations on this from day 1 to make sure that we have a good strategy for deploying in prod. We're breaking new ground here with this... [14:21:26] cool [14:21:50] Amir1: but yeah in short what marko said. We would want something like ores/deploy.git that would have the virtualenv to deploy [14:22:07] hashar, we've decided against that. [14:22:12] wikidata for example relies on bunch of libraries extensions all around the place [14:22:32] so they get a daily job building all the dependencies and pilling up all of that in a repo which is then what is being deployed [14:22:35] Yuvipanda is off on vacation now, but he confirmed that he thought that managing a giant virtualenv with all dependencies installed would be a pain. [14:22:44] whl files are pre-built. [14:22:52] Already compiled. [14:22:58] yup wheel might do [14:23:05] (I love wheels myself) [14:23:12] 5Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#2129741 (10demon) Then someone can copy over all the callsigns :) [14:23:15] I'm just beginning to appreciate them :) [14:23:38] pip / virtualenv automatically build wheel for your locally and keep them in cache [14:23:41] hashar: we store our wheels in research/ores/wheels.git [14:23:50] I am abusing that feature for CI and collect wheels to speed up the jobs [14:24:26] :D [14:24:46] Amir1: halfak: anyway the whole point is on the deploynt server we dont run pip install :p [14:25:11] hashar, "deployment server"? [14:25:43] pip install if talk to outside is dangerous but using a pre-built wheel can't do any harm [14:27:07] and we are not even installing system-wide packages [14:27:45] 3Scap3: Scap3 should support virtualenv for deployment of python packages - https://phabricator.wikimedia.org/T130205#2129748 (10hashar) I am interested in deploying a venv as well. My use case is the #zuul server. Building the deb package and fulfilling python modules dependencies have been a pain. I could use... [14:27:52] Yeah. I'm imagining that our production servers are firewalled and will only be able to talk to wikimedia domains. [14:28:05] So that we can send get requests to the public APIs [14:28:43] (03CR) 10Hashar: [C: 032] castor: avoid polluting ZUUL variables [integration/config] - 10https://gerrit.wikimedia.org/r/278000 (owner: 10Hashar) [14:28:45] (03CR) 10Hashar: [C: 032] castor: share cache between ext and skins [integration/config] - 10https://gerrit.wikimedia.org/r/278001 (owner: 10Hashar) [14:29:33] halfak: yup on tin, we can't forge the virtualenv and fetch the dependencies from pypi [14:29:43] hashar, no problem. [14:29:49] All dependencies are in our wheels repo [14:30:04] So "pip install" might as well say "pip extract_my_wheels" [14:30:13] potentially :-} [14:30:25] or have the wheels repo as a submodule [14:30:32] or whatever solution [14:30:47] Yup. We end up pulling it in as a submodule [14:30:55] I guess you get the hardest part covered: having the dependencies hosted in a Wikimedia git repo (wheels.git) [14:31:04] rest is about orchestrating that in scap. Should be doable [14:31:16] (03Merged) 10jenkins-bot: castor: avoid polluting ZUUL variables [integration/config] - 10https://gerrit.wikimedia.org/r/278000 (owner: 10Hashar) [14:31:17] PROBLEM - Host cache-rsync is DOWN: CRITICAL - Host Unreachable (10.68.23.165) [14:31:18] (03Merged) 10jenkins-bot: castor: share cache between ext and skins [integration/config] - 10https://gerrit.wikimedia.org/r/278001 (owner: 10Hashar) [14:31:20] I will let the scap3 expert follow up on the task ;-} [14:32:35] kk [14:32:42] Thanks for your thoughts hashar [14:32:54] the good thing is that i have the exact same use case [14:32:55] :D [14:33:12] Would be nice to make all our lives easier :D [14:33:14] i.e. deploy a python based soft that has a bunch of requirements unmatched by Precise/Trusty/Jessie [14:33:40] so I am definitely going to back your task, though I am not going to lead the effort [14:38:04] http://vim.wikia.com/wiki/Macros are awesome [14:40:47] Makes sense. Will keep pushing. [14:50:19] (03PS1) 10Hashar: zuul: introduce 'npm-legacy' template [integration/config] - 10https://gerrit.wikimedia.org/r/278003 (https://phabricator.wikimedia.org/T119143) [14:50:22] 3Scap3: Scap3 should support virtualenv for deployment of python packages - https://phabricator.wikimedia.org/T130205#2129486 (10mmodell) I don't see why you would need specialized support for this, you just do like @hashar said and push the virtual env. into a git repo and deploy that with scap3. [14:52:15] (03PS1) 10Hashar: Mass switch repositories npm job to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/278004 (https://phabricator.wikimedia.org/T119143) [14:53:36] 3Scap3: Scap3 should support virtualenv for deployment of python packages - https://phabricator.wikimedia.org/T130205#2129771 (10mmodell) Scap already has the ability to execute arbitrary commands after each stage of the deploy process using check commands. It should be possible to initialize the environment fro... [14:55:42] (03PS1) 10Hashar: Zuul: rm 'npm-node-4.3' project template [integration/config] - 10https://gerrit.wikimedia.org/r/278005 (https://phabricator.wikimedia.org/T119143) [14:56:20] (03CR) 10Jforrester: zuul: introduce 'npm-legacy' template (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/278003 (https://phabricator.wikimedia.org/T119143) (owner: 10Hashar) [14:57:37] (03CR) 10Hashar: zuul: introduce 'npm-legacy' template (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/278003 (https://phabricator.wikimedia.org/T119143) (owner: 10Hashar) [14:58:23] (03CR) 10Hashar: [C: 032] zuul: introduce 'npm-legacy' template [integration/config] - 10https://gerrit.wikimedia.org/r/278003 (https://phabricator.wikimedia.org/T119143) (owner: 10Hashar) [14:59:11] Amir1: on other news, have you fixed ORES red background not showing on beta cluster Recentchange ? [14:59:23] Amir1: I have seen some activity on the task but unsubscribed to it ;( [14:59:27] hashar: yes [14:59:28] :) [14:59:29] (03Merged) 10jenkins-bot: zuul: introduce 'npm-legacy' template [integration/config] - 10https://gerrit.wikimedia.org/r/278003 (https://phabricator.wikimedia.org/T119143) (owner: 10Hashar) [14:59:38] It was something else [14:59:47] the new pages were patrolled edits [15:00:02] and patrolled edits are not highlighted per design [15:00:08] OH [15:00:11] so broken by desing [15:00:13] 3Scap3: Scap3 should support virtualenv for deployment of python packages - https://phabricator.wikimedia.org/T130205#2129486 (10thcipriani) >>! In T130205#2129771, @mmodell wrote: > Scap already has the ability to execute arbitrary commands after each stage of the deploy process using check commands. It should... [15:00:30] but the bug causing them to be added when hidenondamaging = 1 [15:00:51] so I just added that [15:02:01] great! [15:03:00] (03CR) 10Hashar: [C: 04-2] "Gotta verify which repositories are actually migrated." [integration/config] - 10https://gerrit.wikimedia.org/r/278004 (https://phabricator.wikimedia.org/T119143) (owner: 10Hashar) [15:03:26] 3Scap3: Scap3 should support virtualenv for deployment of python packages - https://phabricator.wikimedia.org/T130205#2129861 (10Ladsgroup) let me give it a try. One thing: it's probably better to rename "check" to "commands" or something else. + change the docs. I got the impression that these are only used to... [15:05:02] 3Scap3: Scap3 should support virtualenv for deployment of python packages - https://phabricator.wikimedia.org/T130205#2129863 (10thcipriani) >>! In T130205#2129861, @Ladsgroup wrote: > One thing: it's probably better to rename "check" to "commands" or something else. + change the docs. I got the impression that... [15:18:59] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce build #367: 04FAILURE in 58 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce/367/ [15:38:13] scap3 people, how I can restart two service groups https://github.com/wiki-ai/ores-wikimedia-config/blob/scap/scap/scap.cfg#L7 [15:38:31] I have three service groups [15:38:59] I dont think that is supported [15:39:08] a workaround would be to have a service wrapping the three services [15:39:19] proper solution would be to support multiple services [15:39:55] Amir1: doc being at https://doc.wikimedia.org/mw-tools-scap/scap3/quickstart/setup.html#service-restarts-and-checks [15:40:10] thanks [15:40:49] I saw in the source that we can't do it but I thought maybe there is another way [15:44:32] Amir1: not really another way at this point. We were talking about whether that's something we want to implement (_n_ service restarts) or if it's something we want to leave in the realm of "checks" [15:44:37] 3Scap3: Scap3 should support virtualenv for deployment of python packages - https://phabricator.wikimedia.org/T130205#2129982 (10Ladsgroup) I put it in [[https://github.com/wiki-ai/ores-wikimedia-config/tree/scap/scap|there]]. The whole thing is very untested. Also it would be good if we had several service name... [15:45:36] thcipriani: I thought it would be great if we support it the same way scap3 support several dsh_targets [15:46:29] Amir1: how do you mean? On this collection of targets restart service X on this collection restart service Y? [15:46:44] thcipriani: Also is it okay to put several commands like that? https://github.com/wiki-ai/ores-wikimedia-config/blob/scap/scap/checks.yaml [15:47:11] thcipriani: no, canary_service_name: foo [15:47:27] service_name: bar (for default service group) [15:49:06] Amir1: multiple commands with && is probably the best for now. There's no way to guarantee check order. In theory, it should run from top-to-bottom, but we aren't ensuring that is the case. [15:49:39] okay [15:50:24] what about the second issue I talked (multiple service group restarts) [15:51:28] Amir1: can you file a task for that one? I feel like that's something we should implement, or address in some way. [15:51:42] cool [15:51:43] sure [15:51:50] hasharMeeting: Hi ive removed jsonlint here https://gerrit.wikimedia.org/r/#/c/277619/ [15:55:42] (03CR) 10Paladox: [C: 04-1] "@Hashar we could do what @legoktm did and run the test on all repos." [integration/config] - 10https://gerrit.wikimedia.org/r/278004 (https://phabricator.wikimedia.org/T119143) (owner: 10Hashar) [15:57:49] (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/277619 (owner: 10Paladox) [15:58:17] (03CR) 10jenkins-bot: [V: 04-1] Remove jsonlint infavour of using npm [integration/config] - 10https://gerrit.wikimedia.org/r/277619 (owner: 10Paladox) [15:59:08] PROBLEM - SSH on deployment-conftool is CRITICAL: Server answer [16:21:08] 10Continuous-Integration-Config, 10Dumps-Generation, 6Operations, 13Patch-For-Review, 7WorkType-Maintenance: operations/dumps repo should pass flake8 - https://phabricator.wikimedia.org/T114249#2130115 (10ArielGlenn) 5declined>3Open [16:21:32] 3Scap3: Scap3 should support virtualenv for deployment of python packages - https://phabricator.wikimedia.org/T130205#2130117 (10hashar) >>! In T130205#2129982, @Ladsgroup wrote: > I put it in [[https://github.com/wiki-ai/ores-wikimedia-config/tree/scap/scap|there]]. The whole thing is very untested. Also it wou... [16:24:23] &names [16:24:37] oh, sry, wrong button [16:39:30] Luke081515: I'm sure you are the first person ever to paste something stray into irc ;) [16:40:40] bd808: I didn't paste something, but my PC switched from german to english keyboard layout, so I typed something, and instead of a / the PC set a & ;) [16:41:57] * bd808 has a hard enough time with an en-us qwerty layout [16:43:48] 3Scap3: Scap3 should support virtualenv for deployment of python packages - https://phabricator.wikimedia.org/T130205#2130207 (10mobrovac) >>! In T130205#2129734, @Halfak wrote: > Hi @mobrovac. We've been working with #operations on this from day 1 to make sure that we have a good strategy for deploying in prod... [16:45:14] bd808: I got qwertz, but windows siwtcheds between these every time, when I'm pressing shift+alt, and since I'm using for example shift+alt+s at wiki lots of time, sometimes the keyboard switches :-/ [17:07:44] (03PS2) 10Smalyshev: Wikidata Query Service does not need npm anymore, since GUI now has separate repo. [integration/config] - 10https://gerrit.wikimedia.org/r/276667 [17:38:02] 6Release-Engineering-Team: MW-1.27.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T130227#2130361 (10greg) [17:38:23] 6Release-Engineering-Team: MW-1.27.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T129691#2112838 (10greg) [17:58:40] ostriches: Hey, I fixed everything and testing showed it's working except the part for git_fat [17:59:17] https://github.com/wiki-ai/ores-wikimedia-config/blob/scap/scap/ [18:03:22] what happened to deployment-bastion.eqiad.wmflabs ? [18:03:28] does it have a new name now? [18:03:50] (trying to follow https://wikitech.wikimedia.org/wiki/OCG#Deploying_the_latest_version_of_OCG ) [18:08:54] cscott: It should be there... [18:09:30] Erm... [18:09:31] ssh_exchange_identification: Connection closed by remote host [18:09:31] oh, it seems to have been renamed to deployment-tin: https://gerrit.wikimedia.org/r/270343 [18:09:54] I thought -tin was a bogus thing. [18:09:54] meh [18:15:12] ostriches: do you still don't deploy anything next week (I heared, that the failover test was postponed to april)? [18:15:21] (wikitech is down for now, so I can't check :P) [18:15:23] We will. Shit. [18:15:31] I was supposed to send an e-mail about that yesterday. [18:15:39] *WILL DO THE SECOND I'M OUT OF THIS MTG* [18:16:50] ostriches: no worries, if you want, I can send it, too :P [18:20:11] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer: NeedCaptcha (MediawikiApi::CreateAccountError) in MultimediaViewer browser tests - https://phabricator.wikimedia.org/T129472#2130526 (10Jdlrobson) [18:22:57] FlorianSW: https://lists.wikimedia.org/pipermail/wikitech-l/2016-March/085066.html [18:23:33] ostriches: great, thanks :) Btw.: I updated the roadmap (on mw.org) already [18:23:59] thx [18:24:05] and as far as I can see greg-g already did the same for the deployments calendar :) [18:24:10] np [18:24:13] :) [18:29:31] Amir1: Looking good so far... question tho. What's flower_dsh_targets and worker_dsh_targets? I don't think config knows what those are. [18:30:47] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #629: 04FAILURE in 46 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/629/ [18:31:47] !log updated OCG to version c1a8232594fe846bd2374efd8f7c20d7e97ac449 [18:31:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:35:29] ostriches: https://doc.wikimedia.org/mw-tools-scap/scap3/quickstart/setup.html [18:35:37] I added two service group [18:36:00] Oh yeah it does do that [18:36:04] * ostriches forgot [18:36:07] one is flower service group, after that it's worker and then default (web service group) [18:36:34] it works except it chokes when tries to fetch ores-config with submodules [18:36:56] as excpected [18:41:03] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer: NeedCaptcha (MediawikiApi::CreateAccountError) in MultimediaViewer browser tests - https://phabricator.wikimedia.org/T129472#2106692 (10Jdlrobson) Problem still exists :/ [18:53:34] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #632: 04FAILURE in 33 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/632/ [19:19:16] 10scap, 10Analytics-EventLogging, 10Scap3 (Scap3-Adoption-Phase1): Move EventLogging service to scap3 - https://phabricator.wikimedia.org/T118772#2130822 (10Nuria) [19:22:00] hashar: Hi [19:24:30] 3Scap3: Scap3 should support virtualenv for deployment of python packages - https://phabricator.wikimedia.org/T130205#2129486 (10Joe) @mobrovac the point of using wheels vs standard pip is exactly not needing to build anything on the production hosts; it's akin to using tar or dpkg, in some way. You get a binar... [19:24:38] hashar: Could you merge https://gerrit.wikimedia.org/r/#/c/264333/ so i can test quickly and if it fails i will revert unless you would want me to create a new builder and a template to test with and then if it passes move it to the builders we want it to work with. [19:24:49] I want to see if it fixes our issues with composer. [19:25:13] hashar: Could you also merge https://gerrit.wikimedia.org/r/#/c/274302/ and https://gerrit.wikimedia.org/r/#/c/274296/ please. [19:25:15] paladox: that one is in my backlog of stuff to work on sorry :( [19:25:22] hashar: Oh sorry [19:25:40] hashar: I see your migrating the mediawiki extensions to npm 4.3 :) [19:26:17] hashar: On the day that we decide to merge the patch that does that should we do before that add more resources to jessie to make sure everything runs smothly. [19:39:30] (03PS1) 10Hashar: Archive integration-zuul-layoutdiff layouts [integration/config] - 10https://gerrit.wikimedia.org/r/278066 [19:42:33] (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/278004 (https://phabricator.wikimedia.org/T119143) (owner: 10Hashar) [19:43:43] paladox: yeah that is a good point [19:43:50] I should get the pool size bumped a bit [19:45:08] hashar: Yeh, would we be taking some of the resources aimed at npm and divert them to npm-node-4.3 or can you create a new set of resources [19:46:37] hashar: We can mass test them like legoktm did when we migrated php53 to php55. [19:49:44] (03PS2) 10Hashar: Archive integration-zuul-layoutdiff layouts [integration/config] - 10https://gerrit.wikimedia.org/r/278066 [19:49:59] (03CR) 10Hashar: [C: 032] "publisher: -> publishers:" [integration/config] - 10https://gerrit.wikimedia.org/r/278066 (owner: 10Hashar) [19:52:09] (03Merged) 10jenkins-bot: Archive integration-zuul-layoutdiff layouts [integration/config] - 10https://gerrit.wikimedia.org/r/278066 (owner: 10Hashar) [19:52:54] paladox: I will just migrate I guess [19:53:09] hashar: Hi could npm be updated to 2.x since on https://integration.wikimedia.org/ci/job/mediawiki-core-npm-node-4.3/312/console its running npm 1.4.21 whereas according to https://nodejs.org/en/download/releases/ for 4.3 release it should be 2.13.12 [19:53:21] 2.14.12 [19:53:24] there is a task for that somewhere [19:53:36] about packaging a more recent npm version [19:53:40] hashar: Oh [19:53:46] hashar: Ok [19:53:54] https://phabricator.wikimedia.org/T124474 [19:54:13] Thanks [19:56:42] hashar: By migrate do you know when this will happen [19:59:18] (03CR) 10Hashar: "> @Hashar we could do what @legoktm did and run the test on all repos." [integration/config] - 10https://gerrit.wikimedia.org/r/278004 (https://phabricator.wikimedia.org/T119143) (owner: 10Hashar) [19:59:35] paladox: this evening hopefully :—} [19:59:45] hashar: :) [20:00:30] hashar: Looks like you can update npm at https://integration.wikimedia.org/ci/job/npm-node-4.3/682/console [20:00:35] 19:59:42 npm WARN package.json @ No repository field. [20:00:35] 19:59:50 npm@3.8.1 node_modules/npm [20:01:42] ohhh [20:02:16] Meaning that way people wanting the newer version could probly get it. But i doint know if you think that is a risk by doing that. [20:02:21] hashar ^^ [20:02:54] I guess packages can mention they have a min requirement for npm and then self update [20:03:22] It just lets them install see https://integration.wikimedia.org/ci/job/npm-node-4.3/683/console but it still uses the old version [20:07:11] 6Release-Engineering-Team: MW-1.27.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T129691#2131088 (10Krinkle) [20:07:22] 6Release-Engineering-Team: MW-1.27.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T129691#2112838 (10Krinkle) Adding T121793 as blocker since it was only reverted in wmf.17, not in master. [20:07:46] 6Release-Engineering-Team: MW-1.27.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T128872#2131097 (10greg) [20:08:18] 6Release-Engineering-Team: MW-1.27.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T128872#2088867 (10greg) No longer blocked by the "Not logged in" issue, the change was reverted from wmf.17 and wmf.17 is being deployed to all wikis now. [20:13:46] Krinkle: thanks for the task management re 1.27 and wmf.18 [20:16:32] paladox: I found a way to trigger the npm-node-4.3 job ;-} [20:16:44] hashar: oh :) [20:17:09] hashar: Ive managed to install npm 3 but doint know how to link to it so grunt will run it. [20:18:03] hashar: How would you mass test them. [20:19:59] hashar: Would the way im installing npm 3 compromise the secuirty on your servers if it would i will stop. [20:20:10] https://integration.wikimedia.org/ci/job/npm-node-4.3/689/ [20:25:58] hashar: Ive figured it :) [20:25:59] https://integration.wikimedia.org/ci/job/npm-node-4.3/692/console [20:33:12] paladox: the instances are deleted on job completion [20:33:23] paladox: so that has no impact on the next build [20:33:43] hashar: No but it lets a user on a per job basis choose to install it. [20:33:53] https://integration.wikimedia.org/ci/job/npm-node-4.3/698/console [20:35:10] hashar: would doing this [20:35:10] "test": "npm install npm && node node_modules/npm/cli.js -v && node node_modules/npm/cli.js run-script grunt", [20:35:11] "grunt": "grunt test" [20:35:18] lets grunt use the new npm version. [20:35:41] Its a hack but it allow users to install the npm version they want. [20:39:37] Project browsertests-Wikidata-PerformanceTests-linux-firefox build #140: 04FAILURE in 36 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-PerformanceTests-linux-firefox/140/ [20:41:04] 6Release-Engineering-Team, 6Operations, 6Performance-Team, 7Availability, and 2 others: Dig through logs from 15 Mar 2016 read-only test and file bugs - https://phabricator.wikimedia.org/T129973#2131383 (10Krinkle) a:3aaron [20:44:33] hashar: Looks like they brought out a 4.4.0 lts release. [20:44:34] https://nodejs.org/en/ [20:45:29] paladox: in https://gerrit.wikimedia.org/r/#/c/278075/12/package.json,cm [20:45:36] have you tried to add 'npm' to require-dev ? [20:45:42] err [20:45:44] I mean "devDependencies" [20:45:54] hashar: Nope but i will now [20:45:56] or maybe even as a dependency :} [20:46:47] hashar: Do you mean [20:47:01] "devDependencies": { [20:47:01] "npm": "~3" } [20:47:05] "dependencies": { "npm": ~3.0.0 } [20:47:08] or something like that yeah [20:47:08] Thanks [20:47:14] not sure what will happen [20:47:19] should try both and see what happens [20:47:41] hashar: Thanks and thanks for suggesting that [20:47:45] Im uploading it now. [20:48:03] (03CR) 10Hashar: "From a very basic parsing of the Zuul layout file, that causes 311 extensions to switch to Nodepool! Need more capacity" [integration/config] - 10https://gerrit.wikimedia.org/r/278004 (https://phabricator.wikimedia.org/T119143) (owner: 10Hashar) [20:49:05] hashar: It works [20:49:07] https://integration.wikimedia.org/ci/job/npm-node-4.3/703/console [20:49:13] npm 3.0.0 before the npm test [20:49:14] :) [20:50:17] 6Release-Engineering-Team: MW-1.27.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T128218#2131524 (10Luke081515) [20:51:33] paladox: can you have a short entry on https://www.mediawiki.org/wiki/Continuous_integration/Entry_points#JavaScript please? [20:51:53] maybe a section == recent npm version == [20:51:59] hashar: Ok [20:52:17] hashar: Where should i put it in that section [20:52:32] and https://phabricator.wikimedia.org/T124474 :-) [20:52:59] maybe after the 'JavaScript documentation' section [20:53:24] hashar: Ok i will also add you as an reference since your the run who thought of doing it that way. Thanks very much [20:53:25] Ok [20:54:12] We could also do it on the one on trusty [20:54:20] Meaning a work around for oojs [20:54:25] Just need to figure out nodejs [20:56:39] hashar: Something like https://www.mediawiki.org/wiki/Continuous_integration/Entry_points#Recent_npm_version [20:59:30] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 10OOjs, 13Patch-For-Review, 7WorkType-NewFunctionality: Provide the Chrome binary on the nodepool servers so OOjs, VE core can use it in testing - https://phabricator.wikimedia.org/T128090#2131692 (10Krinkle) [20:59:32] 5Continuous-Integration-Scaling, 13Patch-For-Review, 7WorkType-NewFunctionality: Migrate javascript npm CI jobs to Nodepool - https://phabricator.wikimedia.org/T119143#2131691 (10Krinkle) [21:01:25] (03PS2) 10Krinkle: Zuul: rm 'npm-node-4.3' project template [integration/config] - 10https://gerrit.wikimedia.org/r/278005 (https://phabricator.wikimedia.org/T119143) (owner: 10Hashar) [21:02:54] hashar: We can try https://packagist.org/packages/mouf/nodejs-installer for oojs ui [21:05:48] paladox: What problem are you trying to solve? [21:06:34] Installing Node.js or npm as part of the job does not make sense [21:06:47] Krinkle installing nodejs updated version for trusty in oojsui [21:07:08] That is not a problem. What problem are you trying to solve? [21:07:35] Krinkle oh no problem them. Im just trying to get nodejs to install locally like we did with npm. [21:07:44] installing 'npm' within the job doesn't work. It will not be used by the outer job. Is the npm version not working? We can just upgrade it [21:07:55] We don't do that with npm. [21:08:06] Krinkle [21:08:08] This is disruptive and slow and spreads a bad habit into recommendations. I've reverted the change on mw.org. [21:08:09] it is possible [21:08:16] See https://integration.wikimedia.org/ci/job/npm-node-4.3/705/console [21:08:20] please [21:08:22] It installs something, but is pointless. [21:08:41] paladox: What is the reason for installing 3.x inside the job? [21:08:55] If there is a problem, we can jus upgrade npm in general on the slaves. [21:09:03] Krinkle see https://phabricator.wikimedia.org/T124474 [21:09:08] It is blocked [21:09:15] or stalled [21:10:09] !log updating scap on deployment-tin to test D133 [21:10:09] D133: Rewrite refreshCdbJsonFiles in python - https://phabricator.wikimedia.org/D133 [21:10:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:11:04] paladox: I know it is blocked, but this solution is not acceptable and does not make sense. [21:11:21] Krinkle ok [21:18:23] PROBLEM - Free space - all mounts on integration-slave-trusty-1016 is CRITICAL: CRITICAL: integration.integration-slave-trusty-1016.diskspace._mnt.byte_percentfree (<10.00%) [21:20:44] Krinkle but it allows users to install newer npm version until that task can be changed from stalled to open and npm upgraded. [21:21:23] paladox: Nobody of our users is interested in using a newer npm version [21:21:30] The problem is not that the npm version is old [21:21:49] The problem is that our Nodepool images currently have a broken combination of new Node.js with old NPM [21:21:57] paladox: Node 0.10 goes with Npm 1.4 [21:22:04] Krinkle oh [21:22:06] Node 4 goes with Npm 2 [21:22:10] We upgraded Node.js [21:22:14] but forgot to upgrade npm [21:22:23] We should *not* specify npm versions in package.json [21:22:25] please never do that [21:22:37] Oh but that task is stalled because it could break other repos using unless i read it wrong [21:22:44] Krinkle sorryy [21:22:46] sorry [21:22:55] I was only trying it out [21:23:39] Nah, none of our projects rely on a specific version of npm explicitly [21:24:03] And we really dont want to pollute our code basis with hardcoded npm versions. Things will work naturally based on the supported version of Node. [21:24:09] Krinkle yeh i read it wrong it was because jessie repo has to been updated to use npm 2.x [21:24:11] See https://packages.debian.org/jessie/npm [21:24:31] Jessie repo has not been updated. [21:24:40] Jessie is old and contains Node 0.10 and Npm 1.4 [21:24:45] both are old, but they are compatibel with each other [21:24:52] Node 4 is compatibel with npm 2.x [21:25:09] Jessie does not have Node 4, so Wikimedia APT created their own package for Node 4 [21:25:15] but we forgot to also create a newer package for npm [21:25:32] That's why the task is titled "Package npm" (meaning: Create a Debian package for npm) [21:25:44] Krinkle, Oh, could we repopen the task since we wont be getting the update directly from jessie instead be doing it from the npm repo. [21:25:58] Krinkle oh [21:26:36] Krinkle it looks like npm is newer on trusty then on jessie [21:26:49] See https://integration.wikimedia.org/ci/job/npm/59453/console [21:26:54] Yes, they are unrelated (One is Debian, the other is Ubuntu) [21:27:03] Oh ok [21:27:12] Debian: Jessie, Strech, Sid [21:27:16] Ubuntu: Precise, Trusty [21:27:44] http://packages.ubuntu.com/trusty/npm [21:27:47] Krinkle oh ok, I know it isent important to update so i was just testing out a few things. [21:27:49] Thanks [21:28:05] hashar: http://packages.ubuntu.com/xenial/npm [21:28:06] 3.x [21:28:08] could be interesting [21:29:42] Yes [21:51:14] 6Release-Engineering-Team: MW-1.27.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T128218#2132302 (10greg) 5Open>3Resolved [21:55:18] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Generate code coverage reports for extensions - https://phabricator.wikimedia.org/T71685#2132319 (10Jdlrobson) Does that mean no? [21:55:49] 6Release-Engineering-Team, 6Operations: setup automatic deletion of old l10nupdate - https://phabricator.wikimedia.org/T130317#2132321 (10Dzahn) [21:56:30] 6Release-Engineering-Team, 6Operations: setup automatic deletion of old l10nupdate - https://phabricator.wikimedia.org/T130317#2132333 (10Dzahn) [21:58:45] Project beta-scap-eqiad build #94208: 04FAILURE in 3 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/94208/ [22:03:18] 10Deployment-Systems, 6Release-Engineering-Team, 6Operations: setup automatic deletion of old l10nupdate - https://phabricator.wikimedia.org/T130317#2132339 (10greg) [22:04:16] 10Deployment-Systems, 6Release-Engineering-Team, 6Operations: setup automatic deletion of old l10nupdate - https://phabricator.wikimedia.org/T130317#2132321 (10Reedy) Could we just purge those when we purge the usual localisation caches in /srv/mediawiki-staging ? [22:11:46] 10Continuous-Integration-Infrastructure: npm jobs sometimes fail with Cannot find module '/usr/local/bin/npm' - https://phabricator.wikimedia.org/T130320#2132392 (10Catrope) [22:15:53] 10Continuous-Integration-Infrastructure: npm jobs sometimes fail with Cannot find module '/usr/local/bin/npm' - https://phabricator.wikimedia.org/T130320#2132392 (10Paladox) This is a duplicate of another task. Might be resolved by upgrading to npm-node-4.3 which haser is planning to migrate all the extensions t... [22:16:05] Krinkle ^^ isent there a similar task i think one you created [22:19:17] 10Deployment-Systems, 10Architecture, 7Availability: WikiDev 16 working area: Software engineering - https://phabricator.wikimedia.org/T119032#2132481 (10RobLa-WMF) I hope that people interested in this area could help with {T129651}; in particular, this part of the conversation: >>! In T129651#2132418, @Ro... [22:21:57] Yippee, build fixed! [22:21:57] Project beta-scap-eqiad build #94210: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/94210/ [22:23:52] 10Continuous-Integration-Infrastructure, 6Release-Engineering-Team, 13Patch-For-Review: npm tests constantly failing for oojs/ui changesets - https://phabricator.wikimedia.org/T130063#2132603 (10Paladox) Can this be closed as fixed now. [22:25:28] 10Continuous-Integration-Infrastructure: npm jobs sometimes fail with Cannot find module '/usr/local/bin/npm' - https://phabricator.wikimedia.org/T130320#2132605 (10Paladox) https://phabricator.wikimedia.org/T129617 [22:29:45] hashar: Im not sure what the process is but could we whitelist Robert Vogel since he is a member of BlueSpiceFoundation which also has BlueSpiceExtensions which is frequently updated. See https://gerrit.wikimedia.org/r/#/q/owner:%22Robert+Vogel%22+status:open,n,z [22:30:37] He also looks like a long term user see his merged patches. [22:30:37] https://gerrit.wikimedia.org/r/#/q/owner:%22Robert+Vogel%22+status:merged,n,z [22:33:28] hashar: Would we annouce on the emails about mediawiki repos have been switched to use npm-node-4.3 and may notice speed differences as we are migrating [22:37:01] zuul really should run release branch gate-and-submit in a separate queue [22:37:11] * bd808 twiddles thumbs [22:37:52] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseMediaInfo, 10Wikidata: PHP Fatal error: Class 'PHPUnit_Util_Filter' not found in /mnt/jenkins-workspace/workspace/mwext-testextension-php55-composer/src/extensions/WikibaseMediaInfo/src/View/Med... - https://phabricator.wikimedia.org/T130327#2132725 [22:40:20] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseMediaInfo, 10Wikidata: PHP Fatal error: Class 'PHPUnit_Util_Filter' not found in /mnt/jenkins-workspace/workspace/mwext-testextension-php55-composer/src/extensions/WikibaseMediaInfo/src/View/Med... - https://phabricator.wikimedia.org/T130327#2132742 [22:43:44] (03PS1) 10Paladox: Whitelist user Robert Vogel [integration/config] - 10https://gerrit.wikimedia.org/r/278185 [22:43:55] hashar ^^ [22:59:30] 6Release-Engineering-Team, 6Operations, 6Performance-Team, 7Availability, and 2 others: Dig through logs from 15 Mar 2016 read-only test and file bugs - https://phabricator.wikimedia.org/T129973#2132873 (10aaron) p:5Normal>3High [23:53:54] PROBLEM - Puppet run on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]