[00:01:37] <greg-g>	 twentyafterfour: what sorcery is this? "((PhabUpgrade))"
[00:13:37] <wikibugs>	 10Continuous-Integration-Infrastructure, 10CirrusSearch, 6Discovery: ElasticSearch taking 17% of RAM on integration slaves - https://phabricator.wikimedia.org/T89083#1851038 (10Deskana) This task is quite old now. Is this still an issue?
[00:25:48] <wikibugs>	 5Release-Engineering-Epics, 10Browser-Tests-Infrastructure, 7Epic, 7Tracking: [EPIC] trigger browser tests from Gerrit (tracking) - https://phabricator.wikimedia.org/T55697#1851086 (10Deskana)
[00:25:49] <wikibugs>	 10Continuous-Integration-Config, 10Browser-Tests-Infrastructure, 10CirrusSearch, 6Discovery: Make browsertests for CirrusSearch run on every submitted patch - https://phabricator.wikimedia.org/T98374#1851083 (10Deskana) 5Open>3Resolved a:3Deskana This was resolved with the creation of Cindy: https://...
[00:58:26] <twentyafterfour>	 Greg: it's Phurl
[01:01:22] <greg-g>	 of course
[01:03:04] <greg-g>	 stupid old php url shortening project clogging up search results
[01:03:47] <greg-g>	 :( You do not have permission to view this object.
[01:03:50] <greg-g>	 https://phabricator.wikimedia.org/phurl/
[01:04:04] <greg-g>	 I wanna be an admin </whines>
[01:40:29] <wikibugs>	 10Continuous-Integration-Config, 10Browser-Tests-Infrastructure, 10CirrusSearch, 6Discovery: Make browsertests for CirrusSearch run on every submitted patch - https://phabricator.wikimedia.org/T98374#1851228 (10Legoktm) 5Resolved>3Open The point of this task is to get *rid* of Cindy and use the proper...
[01:40:30] <wikibugs>	 5Release-Engineering-Epics, 10Browser-Tests-Infrastructure, 7Epic, 7Tracking: [EPIC] trigger browser tests from Gerrit (tracking) - https://phabricator.wikimedia.org/T55697#1851230 (10Legoktm)
[01:40:39] <wikibugs>	 10Continuous-Integration-Config, 10Browser-Tests-Infrastructure, 10CirrusSearch, 6Discovery: Make browsertests for CirrusSearch run on every submitted patch - https://phabricator.wikimedia.org/T98374#1851231 (10Legoktm) a:5Deskana>3None
[01:42:06] <wikibugs>	 10Continuous-Integration-Config, 10Browser-Tests-Infrastructure, 10CirrusSearch, 6Discovery: Make browsertests for CirrusSearch run on every submitted patch - https://phabricator.wikimedia.org/T98374#1851236 (10Deskana) p:5Normal>3Lowest >>! In T98374#1851228, @Legoktm wrote: > The point of this task i...
[01:45:43] <wikibugs>	 10Continuous-Integration-Config, 10Browser-Tests-Infrastructure, 10CirrusSearch, 6Discovery: Make browsertests for CirrusSearch run on every submitted patch with proper CI infrastructure rather than a bot - https://phabricator.wikimedia.org/T98374#1851243 (10Deskana)
[03:18:25] <wmf-insecte>	 Project beta-scap-eqiad build #81098: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/81098/
[03:21:17] <Krenair>	 returned [255]: Warning: Permanently added 'deployment-mediawiki01.deployment-prep.eqiad.wmflabs,10.68.17.170' (ECDSA) to the list of known hosts.
[03:21:19] <Krenair>	 that's an error?
[03:22:34] <bd808>	 the add wouldn't be
[03:23:03] <bd808>	 but a 255 exit means something else went boom
[03:23:40] <bd808>	 the warning happens because we don't have puppet collecting host certs in beta cluster
[03:23:55] <bd808>	 scap has a local hack to not validate the host certs
[03:23:55] <Krenair>	 do we in prod?
[03:24:07] <bd808>	 in prod we do, yes
[03:24:18] <bd808>	 it's a bug in the self-hosted puppet setup
[03:24:33] <bd808>	 that keep it from collecting the resources during the puppet run
[03:25:04] <bd808>	 resource collection needs puppetdb and self-hosted doesn't have that
[03:25:54] <wmf-insecte>	 Yippee, build fixed!
[03:25:54] <wmf-insecte>	 Project beta-scap-eqiad build #81099: 09FIXED in 5 min 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/81099/
[03:52:48] <yuvipanda>	 hello releng
[03:52:53] <yuvipanda>	 wanted to point out that pupet's broken in all of beta cluster
[03:52:58] <yuvipanda>	 for a while at least, it looks like
[03:53:06] <yuvipanda>	 with a role::ntp issue?!
[03:53:16] <yuvipanda>	 one of the many cherry-picked patches, I suspect
[03:59:02] <Krenair>	 root@deployment-puppetmaster:/var/lib/git/operations/puppet# git diff origin/production..HEAD | grep ntp
[03:59:03] <Krenair>	 root@deployment-puppetmaster:/var/lib/git/operations/puppet# 
[04:00:44] <yuvipanda>	 indeed
[04:00:56] <yuvipanda>	 so this is a combination of a puppet-is-terrible and subtle-issue-somewhere else
[04:05:46] <Krenair>	 do you know when it started failing?
[04:08:15] <yuvipanda>	 no idea
[04:08:22] <yuvipanda>	 I just popped in there to check something else
[04:08:25] <yuvipanda>	 Krenair: can find out though
[04:09:52] <yuvipanda>	 http://graphite.wmflabs.org/render/?width=588&height=310&_salt=1449202179.685&target=deployment-prep.deployment-puppetmaster.puppetagent.failed_events
[04:09:54] <yuvipanda>	 not very long ago
[06:09:08] <wikibugs>	 10Continuous-Integration-Config, 10Browser-Tests-Infrastructure, 10CirrusSearch, 6Discovery: Make browsertests for CirrusSearch run on every submitted patch with proper CI infrastructure rather than a bot - https://phabricator.wikimedia.org/T98374#1851591 (10Legoktm) p:5Lowest>3Normal This isn't lowest...
[06:09:50] <wikibugs>	 10Continuous-Integration-Config, 10Browser-Tests-Infrastructure, 10CirrusSearch: Make browsertests for CirrusSearch run on every submitted patch with proper CI infrastructure rather than a bot - https://phabricator.wikimedia.org/T98374#1851593 (10Deskana)
[06:10:16] <wikibugs>	 10Continuous-Integration-Config, 10Browser-Tests-Infrastructure, 10CirrusSearch: Make browsertests for CirrusSearch run on every submitted patch with proper CI infrastructure rather than a bot - https://phabricator.wikimedia.org/T98374#1265741 (10Deskana) >>! In T98374#1851591, @Legoktm wrote: > This isn't l...
[06:12:04] <wikibugs>	 10Beta-Cluster-Infrastructure, 6Labs, 10Labs-Infrastructure, 6operations: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#1851595 (10Chmarkine) Let's Encrypt is in Public Beta now. Everyone can get free certificates from them now.  [1] https://letsenc...
[06:50:46] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikidata: Wikibase QUnit failure blocks Content translation CI - https://phabricator.wikimedia.org/T120349#1851613 (10santhosh) 3NEW
[06:51:23] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikidata: Wikibase QUnit failure blocks Content translation CI - https://phabricator.wikimedia.org/T120349#1851620 (10santhosh)
[07:30:05] <wikibugs>	 10Deployment-Systems, 3Scap3: create an environment object that centralizes the file and directory lookup logic for scap3 - https://phabricator.wikimedia.org/T119643#1851650 (10mmodell) So I wrote D67, @dduvall wrote D70, gonna probably abandon D67
[07:42:30] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikidata: Wikibase QUnit failure blocks Content translation CI - https://phabricator.wikimedia.org/T120349#1851659 (10KartikMistry) @JanZerebecki Can you take a look at this?
[07:43:37] <kart_>	 jzerebecki: https://phabricator.wikimedia.org/T120349 :)
[08:58:14] <kart_>	 hashar: morning
[08:59:09] <kart_>	 hashar: Can you look at T120349?
[08:59:19] <kart_>	 Blocks merging CX code.
[08:59:29] <hashar>	 kart_: morning
[08:59:42] <hashar>	 !phab 
[08:59:47] <hashar>	 qa-morebots: ping
[08:59:48] <qa-morebots>	 I am a logbot running on tools-exec-1211.
[08:59:48] <qa-morebots>	 Messages are logged to https://tools.wmflabs.org/sal/releng.
[08:59:48] <qa-morebots>	 To log a message, type !log <msg>.
[08:59:50] <hashar>	 wm-bot: pong
[08:59:52] <hashar>	 wm-bot: ping
[08:59:56] <hashar>	 !help log
[09:00:00] <wm-bot>	 Unknown command type @commands for a list of all commands I know
[09:00:00] <hashar>	 @help log
[09:00:29] <hashar>	 !phab is https://phabricator.wikimedia.org/$1
[09:00:29] <wm-bot>	 Key was added
[09:00:33] <hashar>	 !phab T120349
[09:00:33] <wm-bot>	 https://phabricator.wikimedia.org/T120349
[09:00:42] <kart_>	 nice :)
[09:00:48] <hashar>	 kart_: will do in a few minutes
[09:00:55] <kart_>	 Thanks!
[09:01:10] <hashar>	 reading the drama thread about php version bump at https://phabricator.wikimedia.org/T118932 :D
[09:01:45] <kart_>	 hashar: :)
[09:03:52] <hashar>	 kart_: ahaha
[09:03:59] <hashar>	 Deutsch versus German
[09:04:17] <hashar>	 I had that exact same discussion yesterday evening at my local hacker group meeting
[09:10:09] <hashar>	 kart_: there are bunch of logs attached to the build at https://integration.wikimedia.org/ci/job/mwext-qunit/8665/
[09:10:17] <hashar>	 https://integration.wikimedia.org/ci/job/mwext-qunit/8665/artifact/log/mw-error.log/*view*/  
[09:13:12] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikidata: Wikibase QUnit failure blocks Content translation CI - https://phabricator.wikimedia.org/T120349#1851758 (10hashar) Looking at the mwext-qunit console log for change https://gerrit.wikimedia.org/r/#/c/256895/ , we do query Special:BlankPage for debugging pur...
[09:14:18] <kart_>	 what BlueSkins :/
[09:14:36] <kart_>	 Sky*
[09:15:45] <hashar>	 !log salt --show-timeout '*' cmd.run 'rm -fR /mnt/jenkins-workspace/workspace/mwext-qunit/src/skins/*'  ( https://phabricator.wikimedia.org/T120349 )
[09:15:48] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[09:15:49] <hashar>	 kart_: that is a skin
[09:16:01] <hashar>	 somehow mwext-qunit has been run against mediawiki/skins/BlueSky
[09:16:07] <hashar>	 and the repo has been cloned in the workspace
[09:16:16] <hashar>	 but it is not deleted when another job is running
[09:16:23] <hashar>	 and MediaWiki autoload skins it find
[09:17:25] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikidata: Wikibase QUnit failure blocks Content translation CI - https://phabricator.wikimedia.org/T120349#1851774 (10aude) I could be wrong, but these are all appear to be notices and warnings and I think are probably not the problem.  Of course, it would be nice to...
[09:20:13] <hashar>	 aude: kart_ : I retriggered a build and there are a bunch of other issues when calling Special:BlankPage https://integration.wikimedia.org/ci/job/mwext-qunit/8672/artifact/log/mw-debug-www.log/*view*/  :(
[09:23:21] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikidata: Wikibase QUnit failure blocks Content translation CI - https://phabricator.wikimedia.org/T120349#1851776 (10hashar) I have cleaned the workspaces with: ``` salt --show-timeout '*' cmd.run 'rm -fR /mnt/jenkins-workspace/workspace/mwext-qunit/src/skins/*' ```...
[09:26:19] <hashar>	 kart_: and the skin stuff is tracked by https://phabricator.wikimedia.org/T113841 :D
[09:28:38] <kart_>	 hashar: :/
[09:28:49] <kart_>	 hashar: any easier way we can fix? :)
[09:28:52] <hashar>	 kart_: gave it a try again with a dummy change for ContentTranslation https://gerrit.wikimedia.org/r/#/c/248855/
[09:29:02] <hashar>	 qunit runs at https://integration.wikimedia.org/ci/job/mwext-qunit/8673/consoleFull
[09:29:14] <hashar>	 and https://integration.wikimedia.org/ci/job/mwext-qunit/8673/ has the debug logs
[09:31:29] <hashar>	 kart_: gotta dig in https://integration.wikimedia.org/ci/job/mwext-qunit/8673/artifact/log/mw-debug-www.log/*view*/
[09:31:48] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikidata: Wikibase QUnit failure blocks Content translation CI - https://phabricator.wikimedia.org/T120349#1851780 (10hashar) I reused a [[ https://gerrit.wikimedia.org/r/#/c/248855/ dummy change against ContentTranslation ]] to retry the build. Failed build is https:...
[09:32:08] <hashar>	 Dependency triggered: /mnt/jenkins-workspace/workspace/mwext-qunit/src/skins/BlueSky/i18n/en.json deleted.
[09:32:09] <hashar>	 hmm
[09:32:26] <hashar>	 looks like maybe the localisationCache is not properly cleared between runs :(
[09:32:56] <kart_>	 oh
[09:33:12] <kart_>	 hashar: last main build is success.
[09:33:21] <hashar>	 though we do call /srv/deployment/integration/slave-scripts/bin/mw-teardown-mysql.sh
[09:33:31] <hashar>	 but I am not sure where the l10n cache is written to
[09:36:51] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikidata: Wikibase QUnit failure blocks Content translation CI - https://phabricator.wikimedia.org/T120349#1851781 (10hashar) So we have: ``` Dependency triggered: /mnt/jenkins-workspace/workspace/mwext-qunit/src/skins/BlueSky/i18n/en.json deleted. ```  Which suggests...
[09:37:03] <hashar>	 kart_: any idea where the l10ncache .cdb files are written too ?
[09:37:47] <hashar>	 oh man
[09:37:49] <hashar>	 under /tmp
[09:43:07] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikidata: Wikibase QUnit failure blocks Content translation CI - https://phabricator.wikimedia.org/T120349#1851782 (10hashar) Seems the qunit job has the localisation cache written to /tmp, files belonging to www-data: ``` /tmp/l10n_cache-ar.cdb /tmp/l10n_cache-en.cdb...
[09:44:27] <kart_>	 hashar: sorry, was bit afk.
[09:44:55] <hashar>	 so one issue was BlueSkin skin being still around in the workspace
[09:44:57] <hashar>	 cleaned it up
[09:45:10] <hashar>	 and another potential issue is the l10ncache is kept between run
[09:48:13] <kart_>	 ah :)
[09:51:08] <hashar>	 ah
[09:51:12] <hashar>	 got it
[09:51:49] <kart_>	 cool.
[09:52:22] <mobrovac>	 hashar: which repo is cxserver's beta build job using?
[09:54:34] <wikibugs>	 10Continuous-Integration-Config: qunit jobs have Localisation cache under /tmp causing cache pollution between runs - https://phabricator.wikimedia.org/T120356#1851789 (10hashar) 3NEW
[10:02:13] <hashar>	 mobrovac: that is defined in integration/config.git  /jjb/beta.yaml 
[10:02:14] <hashar>	 the job is beta-cxserver-update-eqiad
[10:02:21] <hashar>	 it clones both mediawiki/services/cxserver mediawiki/services/cxserver/deploy
[10:02:37] <hashar>	 and use the config file ln -s /srv/deployment/cxserver/config.js "$CXSERVER_TARGET/cxserver/config.js"
[10:02:43] <hashar>	 where CXSERVER_TARGET is /srv/deployment/cxserver
[10:02:55] <hashar>	 for where the service points to, I guess it is provisioned by puppet
[10:03:51] <aude>	 fwiw, the qunit tests pass for me
[10:04:21] <hashar>	 aude: thanks!!! I think the issue might have been because of the l10ncache
[10:04:35] <aude>	 probably
[10:04:37] <hashar>	 mobrovac: modules/cxserver/manifests/init.pp:    $base_path = '/srv/deployment/cxserver/deploy',  ?
[10:04:58] * aude struggles to get karma to work for me
[10:05:05] <aude>	 but Special:JavaScriptTest works
[10:05:23] <hashar>	 mobrovac: so I would say cxserver on beta uses the deploy repo based on $base_path = '/srv/deployment/cxserver/deploy'
[10:05:45] <hashar>	 aude: the tmpdir is not properly set  under apache :(
[10:06:07] <aude>	 on jenkins?
[10:06:20] * aude using nginx and maybe i don't have stuff set correctly
[10:06:50] <aude>	 (for karma, but i'm not going to worry about it now)
[10:07:26] <hashar>	 what we do is load a bunch of shell script to set env variables  ( from integration/jenkins.git   /bin/mw-*env*.sh )
[10:08:07] <hashar>	 then run a shell that has those env and inject in LocalSettings.php a bunch of snippets from /mediawiki/conf.d/
[10:08:17] <hashar>	 and the php snippets there rely on env variables to set various $wg settings
[10:08:18] <hashar>	 BUT
[10:08:27] <hashar>	 on apache, the env settings do not exists
[10:08:47] <hashar>	 so mediawiki fallback to the default, turns out to point caches to /tmp/  which is shared between jobs / builds grrr
[10:09:14] <aude>	 :(
[10:09:39] <hashar>	 so when running the installer under cli (which has the proper env)  I need to find out a way to eval
[10:09:50] <hashar>	 and write / hardcode the output in LocalSettings.php
[10:09:53] <hashar>	 but eval() is evil :D
[10:19:12] <mobrovac>	 kk merci hashar
[10:19:23] <hashar>	 echo -en "<?php\n\$wgTmpDirectory = '$MW_TMPDIR';\n?>" >> LocalSettings.php
[10:19:25] <hashar>	 dirty hack
[10:20:12] <Nikerabbit>	 why php tags?
[10:27:37] <hashar>	 Nikerabbit: that is how we do it
[10:27:51] <hashar>	 the mw-apply-settings.sh loads a bunch of php files from  /mediawiki/conf.d/
[10:27:55] <hashar>	 which have the opening tag
[10:28:15] <grrrit-wm>	 (03PS1) 10Hashar: mw-apply-settings: set $wgTmpDirectory explicitly [integration/jenkins] - 10https://gerrit.wikimedia.org/r/256912 (https://phabricator.wikimedia.org/T120356) 
[10:28:26] <Nikerabbit>	 right
[10:28:44] <aude>	 hashar: does cindy the browser test bot run on jenkins (for cirrus) ? do you know?
[10:28:53] <hashar>	 aude: no idea
[10:28:55] * aude looking at https://github.com/wikimedia/mediawiki-extensions-CirrusSearch/blob/master/CirrusSearch.php#L902-L906 
[10:29:11] <hashar>	 aude: Cindy was written by Jon from mobile to run the browser tests.  No clue where it is running honestly
[10:29:15] <aude>	 now that we have a (soft) dependency on cirrus, this gets loaded for wikibase browser tests
[10:29:18] <aude>	 hashar: ok
[10:29:28] <hashar>	 aude: that eventually prompted Dan to write the mwext-selenium job which run browser tests
[10:29:40] <hashar>	 but repos using Cindy have not been migrated to the Jenkins job mwext-selenium
[10:29:50] <hashar>	 most probably, wherever Cindy runs, elasticsearch is not available
[10:29:51] <aude>	 i'm just thinking we could make the check for 'browsertests' in the jobname mor especific
[10:30:08] <aude>	 ok
[10:30:27] <wikibugs>	 10Continuous-Integration-Config, 5Patch-For-Review: qunit jobs have Localisation cache under /tmp causing cache pollution between runs - https://phabricator.wikimedia.org/T120356#1851857 (10hashar)
[10:30:42] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] mw-apply-settings: set $wgTmpDirectory explicitly [integration/jenkins] - 10https://gerrit.wikimedia.org/r/256912 (https://phabricator.wikimedia.org/T120356) (owner: 10Hashar)
[10:30:45] <hashar>	 lets be brave
[10:31:33] <grrrit-wm>	 (03Merged) 10jenkins-bot: mw-apply-settings: set $wgTmpDirectory explicitly [integration/jenkins] - 10https://gerrit.wikimedia.org/r/256912 (https://phabricator.wikimedia.org/T120356) (owner: 10Hashar)
[10:32:34] <grrrit-wm>	 (03CR) 10Hashar: "Deployed with:" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/256912 (https://phabricator.wikimedia.org/T120356) (owner: 10Hashar)
[10:33:07] <hashar>	 bah
[10:34:43] <grrrit-wm>	 (03PS1) 10Hashar: mw-apply-settings: fix duplicate ?> php closing tag [integration/jenkins] - 10https://gerrit.wikimedia.org/r/256913 
[10:34:51] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] mw-apply-settings: fix duplicate ?> php closing tag [integration/jenkins] - 10https://gerrit.wikimedia.org/r/256913 (owner: 10Hashar)
[10:35:01] <hashar>	 :(
[10:35:07] <hashar>	 my fault for not writing tests
[10:35:42] <grrrit-wm>	 (03Merged) 10jenkins-bot: mw-apply-settings: fix duplicate ?> php closing tag [integration/jenkins] - 10https://gerrit.wikimedia.org/r/256913 (owner: 10Hashar)
[10:38:40] <grrrit-wm>	 (03PS1) 10Hashar: mw-apply-settings: no PHP opening tag either [integration/jenkins] - 10https://gerrit.wikimedia.org/r/256914 
[10:38:49] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] mw-apply-settings: no PHP opening tag either [integration/jenkins] - 10https://gerrit.wikimedia.org/r/256914 (owner: 10Hashar)
[10:39:09] <hashar>	 Nikerabbit: I should have paid more attention. The php tags are not needed :D
[10:39:39] <grrrit-wm>	 (03Merged) 10jenkins-bot: mw-apply-settings: no PHP opening tag either [integration/jenkins] - 10https://gerrit.wikimedia.org/r/256914 (owner: 10Hashar)
[10:44:11] <wikibugs>	 10Continuous-Integration-Config, 5Patch-For-Review: qunit jobs have Localisation cache under /tmp causing cache pollution between runs - https://phabricator.wikimedia.org/T120356#1851872 (10hashar) Had to fix the script a bit more:  | https://gerrit.wikimedia.org/r/256913 | mw-apply-settings: fix duplicate ?>...
[10:47:08] <Nikerabbit>	 hashar: that's whay I thought :)
[10:52:31] <hashar>	 hmm
[10:52:37] <hashar>	 wikibase merges at least
[10:52:39] <hashar>	 https://gerrit.wikimedia.org/r/#/c/256711/
[10:52:39] <hashar>	 \O/
[10:52:57] <kart_>	 hashar: success?
[10:53:10] <hashar>	 kart_: no clue yet
[10:53:22] <hashar>	 doing a recheck on my dummy change https://gerrit.wikimedia.org/r/#/c/248855/
[10:53:58] <hashar>	 at least now we set Setting $wgTmpDirectory = '/mnt/home/jenkins-deploy/tmpfs/jenkins-0';
[10:54:26] <hashar>	 so when MW is hit via Apache it at least has the proper $wgTmpDirectory
[10:54:30] <hashar>	 instead of /tmp
[10:54:37] <kart_>	 That's nice.
[10:54:42] <Nikerabbit>	 this is so exciting
[10:54:47] <hashar>	 https://integration.wikimedia.org/ci/job/mwext-qunit/8683/console
[10:54:53] <hashar>	 it is failing still :(
[10:54:57] <hashar>	 00:01:17.591 	getLanguageNameByCode() returns language name.
[10:54:57] <hashar>	 00:01:17.591 	Expected: Deutsch
[10:54:59] <hashar>	 00:01:17.591 	Actual: German
[10:55:00] <Nikerabbit>	 yeah
[10:56:09] <Nikerabbit>	 could it be caused by ULS?
[10:56:33] <Nikerabbit>	 https://gerrit.wikimedia.org/r/#/c/243130/1/extensions/Wikibase/view/tests/qunit/wikibase/wikibase.getLanguageNameByCode.tests.js this is the test I believe
[10:57:14] <Nikerabbit>	 https://gerrit.wikimedia.org/r/#/c/243130/1/extensions/Wikibase/view/resources/wikibase/wikibase.getLanguageNameByCode.js and this is the code I believe
[10:57:26] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikidata: Wikibase QUnit failure blocks Content translation CI - https://phabricator.wikimedia.org/T120349#1851880 (10hashar)
[10:57:28] <wikibugs>	 10Continuous-Integration-Config, 5Patch-For-Review: qunit jobs have Localisation cache under /tmp causing cache pollution between runs - https://phabricator.wikimedia.org/T120356#1851877 (10hashar) 5Open>3Resolved a:3hashar ``` Setting $wgTmpDirectory = '/mnt/home/jenkins-deploy/tmpfs/jenkins-0'; ```
[10:58:22] <hashar>	 Nikerabbit: potentially
[10:58:26] <hashar>	 I am not sure how to reproduce though
[10:58:52] <hashar>	 we probably want ULS and ContentTranslation to be tested together
[11:03:18] <hashar>	 kart_: Nikerabbit  I am out of ideas :(
[11:03:45] <wikibugs>	 10Beta-Cluster-Infrastructure, 6Labs, 10Labs-Infrastructure, 6operations: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#1851883 (10Krenair) Yeah, beta.wmflabs.org was in the private beta. Don't know if it can actually work with our setup though.
[11:06:14] <kart_>	 hashar: Re: ULS, CX jenkins jobs depends on ULS just like wikibase AFAIK.
[11:06:26] <kart_>	 Correct me.
[11:06:49] <hashar>	 ah
[11:06:53] <hashar>	 so on https://gerrit.wikimedia.org/r/#/c/248855/
[11:06:58] <hashar>	 mwext-qunit fails
[11:07:08] <hashar>	 but mediawiki-extensions-qunit  pass
[11:07:14] <hashar>	 that later one has more extensions defined
[11:07:31] <hashar>	 namely all of AbuseFilter Babel Cards CheckUser Cite cldr ConfirmEdit Echo Elastica EventLogging Flow Gather GlobalCssJs GuidedTour JsonConfig MobileApp MobileFrontend MwEmbedSupport ParserFunctions SandboxLink SpamBlacklist Thanks TimedMediaHandler UniversalLanguageSelector VisualEditor ZeroBanner ZeroPortal
[11:08:07] <hashar>	 i.e. there is no Wikidata
[11:08:36] <hashar>	 while mwext-qunit has ContentTranslation Echo Elastica EventLogging GeoData GuidedTour  Scribunto UniversalLanguageSelector Wikidata cldr
[11:08:57] <kart_>	 hmm
[11:09:00] <Nikerabbit>	 I did not see explicit loading of ULS in the Wikidata code
[11:11:59] <Nemo_bis>	 Don't they use some ULS something to select the language of statements or similar?
[11:12:40] <hashar>	 * 2eb335f - (HEAD -> master, origin/master, origin/HEAD) New Wikidata Build - 2015-12-04T10:00:01+0000 (72 minutes ago) <WikidataBuilder>
[11:12:40] <hashar>	 * 63f1a6f - New Wikidata Build - 2015-12-03T10:00:01+0000 (25 hours ago) <WikidataBuilder>
[11:12:43] <hashar>	 that is the last two builds
[11:14:50] <hashar>	 going to try to run against a previous wikidata build
[11:19:45] <aude>	 hashar: i think possibly having qunit tests for our build is a new thing
[11:19:56] <hashar>	 possibly
[11:20:10] <hashar>	 giving a try with the Wikidata build from 2 days ago https://integration.wikimedia.org/ci/job/mwext-qunit-T120349/1/consoleFull
[11:20:10] <aude>	 now that i think about it..
[11:20:39] <hashar>	 there is a lot of dispredancy anyway
[11:20:43] <aude>	 https://phabricator.wikimedia.org/T117886 which provides no useful info for me
[11:21:01] <hashar>	 since we do not gate wikidata and contentranslation together, one can break the other :/
[11:21:32] <aude>	 but apparently they pass for our build now (alone)
[11:21:55] <aude>	 https://integration.wikimedia.org/ci/job/mwext-qunit/8676/console
[11:22:14] <aude>	 but i don't see ULS enabled there
[11:22:37] <hashar>	 whereas ContentTranslation qunit job has ULS enabled
[11:22:40] <hashar>	 so the side effect is that it breaks Wikidata 
[11:22:44] <hashar>	 looks like the proper bug
[11:22:49] <wikibugs>	 6Release-Engineering-Team, 5Testing-Initiative-2015, 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-Examples, 7Documentation: Improve documentation around running/writing (with lots of examples) browser tests - https://phabricator.wikimedia.org/T108108#1851911 (10zeljkofilipin) @aklapper: @janzer...
[11:22:50] <aude>	 and we should have it on https://integration.wikimedia.org/ci/job/mwext-qunit/8676/console
[11:33:13] <aude>	 i am looknig more
[11:33:22] <aude>	 apparently i did run into this long ago https://phabricator.wikimedia.org/T92532
[11:33:37] <aude>	 if i am logged in and have uls enabled (and probably ime)
[11:33:53] <aude>	 i'm told it's an issue with animations in uls
[11:35:37] <hashar>	 argrhbbm
[11:36:19] <hashar>	 i tried a run with contentranslation / ULS and karma fails with   Script error.
[11:36:21] <hashar>	 ....
[11:36:23] <aude>	 :(
[11:36:29] <hashar>	 https://integration.wikimedia.org/ci/job/mwext-qunit-T120349/3/consoleFull
[11:36:37] <hashar>	 how useful is that really ...
[11:40:27] <aude>	 i don't get the qunit failure now
[12:02:21] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikidata: Wikibase QUnit failure blocks Content translation CI - https://phabricator.wikimedia.org/T120349#1851975 (10hashar) The l10n cache issue being solved, I gave a try again on a dummy change for ContentTranslation https://gerrit.wikimedia.org/r/#/c/248855/  The...
[12:02:55] <hashar>	 kart_: Nikerabbit aude: I don't know what to do
[12:03:07] <hashar>	 maybe ContentTranslation should no more depends on Wikidata ?
[12:03:11] <hashar>	 until Wikidata/ULS works well together
[12:04:05] <kart_>	 hmm.
[12:04:07] <hashar>	 though Wikidata has been added ages ago
[12:04:23] <hashar>	 and ULS has been added to ContentTranslation back in August ( I5c1425c8e08c356dcbd8468656276dc6eafd30a0 )
[12:04:39] <hashar>	 so it seems something in Wikidata is regressing whenever ULS is around
[12:04:39] <kart_>	 Yes. both breaks each other time to time.
[12:05:19] <kart_>	 Nikerabbit: any thoughts?
[12:05:34] <kart_>	 I'll be AFK for sometime.
[12:05:53] <hashar>	 and we might want to add ULS as a dep of Wikidata :D
[12:06:43] <kart_>	 :D
[12:06:50] <kart_>	 aude: ^
[12:07:19] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikidata: Wikibase QUnit failure blocks Content translation CI - https://phabricator.wikimedia.org/T120349#1851982 (10hashar) And there is also {T92532} which has the same "Deutsch" vs "German"
[12:08:58] <aude>	 could be some issue that happens with particular settings
[12:12:33] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikidata: Wikibase QUnit failure blocks Content translation CI - https://phabricator.wikimedia.org/T120349#1851986 (10hashar)
[12:27:37] <aude>	 kart_: hashar apparently i didn't have cldr enabled for myself (but had it on my old laptop)
[12:27:44] <aude>	 enabling it again, i can reproduce
[12:28:20] <aude>	 disabling uls + keeping cldr then it passes again
[13:34:45] <aude>	 kart_: hashar think i have a not so nice workaround that maybe is ok for now (and have ideas on a more proper fix)
[13:35:05] <aude>	 i have no idea why this issue is just appearing now on jenkins though
[13:47:25] <hashar>	 aude: if you have some workaround  I guess that will be welcomed
[13:47:48] <hashar>	 maybe follow up in #wikidata ? I am there :}
[14:45:10] <wikibugs>	 6Release-Engineering-Team: Reduce production log errors to zero* - https://phabricator.wikimedia.org/T115630#1852273 (10hashar)
[14:46:20] <wikibugs>	 6Release-Engineering-Team: Reduce production log errors to zero* - https://phabricator.wikimedia.org/T115630#1852276 (10mmodell) @greg: there will be a whole lot of holding up the train if that's going to happen.
[14:47:27] <wikibugs>	 6Release-Engineering-Team: [EPIC] Enforce "no increase in log errors" during deployments - https://phabricator.wikimedia.org/T115629#1852278 (10mmodell) @greg: The problem is the new branch cut on tuesday. Since it incorporates a lot of new code it's difficult to avoid new errors sneaking in.
[14:47:34] <wikibugs>	 7Browser-Tests, 10Continuous-Integration-Config, 7Ruby: Cucumber linter should run for all repositories that contain Cucumber code - https://phabricator.wikimedia.org/T58251#1852280 (10hashar)
[14:47:35] <wikibugs>	 7Browser-Tests, 10Continuous-Integration-Config, 5MW-1.25-release, 5MW-1.26-release, and 6 others: Add Rakefile to repositories with Ruby code - https://phabricator.wikimedia.org/T117993#1852279 (10hashar)
[14:47:53] <wikibugs>	 7Browser-Tests, 10Continuous-Integration-Config, 5MW-1.25-release, 5MW-1.26-release, and 6 others: Add Rakefile to repositories with Ruby code - https://phabricator.wikimedia.org/T117993#1795365 (10hashar)
[14:47:54] <wikibugs>	 7Browser-Tests, 10Continuous-Integration-Config, 7Ruby: Cucumber linter should run for all repositories that contain Cucumber code - https://phabricator.wikimedia.org/T58251#592722 (10hashar)
[14:48:14] <wikibugs>	 7Browser-Tests, 10Continuous-Integration-Config, 7Ruby: Cucumber linter should run for all repositories that contain Cucumber code - https://phabricator.wikimedia.org/T58251#592722 (10hashar) We have a Rakefile entry point now ( T117993 ). So the cucumber linter can be added as needed.
[15:00:03] <shinken-wm>	 PROBLEM - Puppet failure on deployment-tin is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[15:12:43] <wikibugs>	 6Release-Engineering-Team: [EPIC] Enforce "no increase in log errors" during deployments - https://phabricator.wikimedia.org/T115629#1852343 (10MaxSem) That's why we need to shoot these errors while they're still on beta.
[15:45:14] <wikibugs>	 6Release-Engineering-Team: [EPIC] Enforce "no increase in log errors" during deployments - https://phabricator.wikimedia.org/T115629#1852432 (10mmodell) The errors need to be a lot more visible, honestly.  If mediawiki-vagrant, and the beta cluster, would surface the errors in a way that's not easily ignored, th...
[15:47:51] <wikibugs>	 6Release-Engineering-Team: [EPIC] Enforce "no increase in log errors" during deployments - https://phabricator.wikimedia.org/T115629#1852435 (10mmodell) I'd like to see something like http://phpdebugbar.com/ enabled by-default on vagrant and beta cluster. Perhaps it could even be offered as a per-user preference...
[15:49:12] <ottomata>	 hey, is deployment-prep's quota all full up?
[15:49:15] <ottomata>	 i can't create new instances
[15:54:43] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikidata: Wikibase QUnit failure blocks Content translation CI - https://phabricator.wikimedia.org/T120349#1852469 (10hashar) a:3aude Unrelated issues were:  * BlueSkins being left in the workspace (manually cleaned up) * localisation cache being shared between all...
[15:57:18] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikidata: Wikibase QUnit failure blocks Content translation CI - https://phabricator.wikimedia.org/T120349#1852488 (10hashar) 5Open>3Resolved
[15:57:25] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Wikidata, 7WorkType-Maintenance: Wikibase QUnit failure blocks Content translation CI - https://phabricator.wikimedia.org/T120349#1851613 (10hashar)
[15:58:19] <Luke081515>	 twentyafterfour: Habormaster at phab has a bug at your current config
[15:58:44] <Luke081515>	 for example: https://phabricator.wikimedia.org/harbormaster/build/176/ <-- build since 28 octobre
[15:58:48] <Luke081515>	 *october
[15:58:57] <Luke081515>	 that's not good... :-/
[15:59:05] <thcipriani>	 ottomata: yeah, the amount of memory allocated for the project is filled (as well as number of instances).
[15:59:22] <twentyafterfour>	 Luke081515: that's experimental, it's being worked on
[15:59:30] <Luke081515>	 ah, ok
[15:59:30] <ottomata>	 rats
[16:00:32] <Luke081515>	 twentyafterfour: A pricipial question: Is it allowed to use integration for own php unit tests/lints?
[16:01:00] <twentyafterfour>	 Luke081515: I don't know
[16:01:09] <wikibugs>	 10MediaWiki-Releasing, 6Developer-Relations, 10Wikimedia-Blog-Content, 3DevRel-December-2015, 5MW-1.26-release: Write blog post announcing MW 1.26 - https://phabricator.wikimedia.org/T112842#1852502 (10Aklapper) >>! In T112842#1850907, @jrbs wrote: > I guess it would be great to get an overview of the ne...
[16:01:32] <Luke081515>	 Do you know someone who could know that?
[16:04:07] <twentyafterfour>	 Luke081515: can you elaborate about what you would like to do?
[16:04:13] <twentyafterfour>	 what sort of tests? on what code?
[16:06:16] <Luke081515>	 I think a lint of my PHP Bot-Skripts would be enough, but at the moment I got no lint :-/
[16:07:03] <Luke081515>	 So I have to search for mistakes at syntax...
[16:07:37] <Luke081515>	 (scripts for wikimedia projects, the bot runs at dewiki)
[16:09:11] <twentyafterfour>	 is it hosted in a wikimedia repo or on github?
[16:10:05] <Luke081515>	 nothing of both.. at the moment at the phabricator repo of my instance...
[16:10:50] <twentyafterfour>	 ah, well you could set up arcanist linters, or we could create a repo for it on production phabricator and set up some sort of ci for it
[16:10:59] <twentyafterfour>	 arcanist runs lint locally on your machine
[16:11:24] <Luke081515>	 ah, ok
[16:36:08] <hashar>	 I am off, be back later tonight
[16:57:39] <wikibugs>	 10Continuous-Integration-Config, 10Wikidata, 3Wikidata-Sprint-2015-12-01: [Task] Add CirrusSearch and Elastica as dependencies of Wikibase on Jenkins - https://phabricator.wikimedia.org/T120217#1852629 (10aude) 5Open>3Resolved
[17:07:49] <wikibugs>	 10MediaWiki-Releasing, 6Developer-Relations, 10Wikimedia-Blog-Content, 3DevRel-December-2015, 5MW-1.26-release: Write blog post announcing MW 1.26 - https://phabricator.wikimedia.org/T112842#1852646 (10Qgil) https://meta.wikimedia.org/wiki/Wikimedia_Blog/Drafts/MediaWiki_1.26  I need help writing details...
[17:50:47] <twentyafterfour>	 anyone seen this error before? (on vagrant)  : Fatal error: Call to undefined method ObjectCache::getLocalClusterInstance() in /vagrant/mediawiki/includes/GlobalFunctions.php on line 3469
[18:03:42] <legoktm>	 uh
[18:03:59] <legoktm>	 git pull everything?
[18:05:02] <twentyafterfour>	 yeah ... did that
[18:05:08] <twentyafterfour>	 my vagrant vm is totally broken it seems
[18:45:11] <wikibugs>	 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Create CI slaves using Debian Jessie (tracking) - https://phabricator.wikimedia.org/T94836#1852922 (10hashar)
[18:45:14] <wikibugs>	 10Continuous-Integration-Infrastructure, 7Blocked-on-Operations, 5Patch-For-Review: Use systemd for xvfb service on Debian/Jessie - https://phabricator.wikimedia.org/T95003#1852917 (10hashar) 5Open>3Resolved a:3hashar And we now have xvfb on Jessie!!!
[19:24:15] <MaxSem>	 !log bumped portals
[19:24:18] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[20:05:28] <wikibugs>	 10Deployment-Systems, 3Scap3: scap3 configuration selection is confusing - https://phabricator.wikimedia.org/T120410#1853180 (10thcipriani) 3NEW a:3thcipriani
[20:06:16] <wikibugs>	 10Deployment-Systems, 3Scap3: scap3 configuration selection is confusing - https://phabricator.wikimedia.org/T120410#1853190 (10thcipriani) p:5Triage>3Normal
[20:20:12] <grrrit-wm>	 (03PS1) 10EBernhardson: Enable submodules for operations/mediawiki-config phpunit tests [integration/config] - 10https://gerrit.wikimedia.org/r/256979 
[20:21:10] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Enable submodules for operations/mediawiki-config phpunit tests [integration/config] - 10https://gerrit.wikimedia.org/r/256979 (owner: 10EBernhardson)
[20:26:52] <grrrit-wm>	 (03PS2) 10EBernhardson: Enable submodules for operations/mediawiki-config phpunit tests [integration/config] - 10https://gerrit.wikimedia.org/r/256979 
[20:28:36] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Enable submodules for operations/mediawiki-config phpunit tests [integration/config] - 10https://gerrit.wikimedia.org/r/256979 (owner: 10EBernhardson)
[20:35:23] <grrrit-wm>	 (03PS3) 10EBernhardson: Enable submodules for operations/mediawiki-config phpunit tests [integration/config] - 10https://gerrit.wikimedia.org/r/256979 
[20:36:17] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Enable submodules for operations/mediawiki-config phpunit tests [integration/config] - 10https://gerrit.wikimedia.org/r/256979 (owner: 10EBernhardson)
[20:36:55] <grrrit-wm>	 (03PS4) 10EBernhardson: Enable submodules for operations/mediawiki-config phpunit tests [integration/config] - 10https://gerrit.wikimedia.org/r/256979 
[21:15:43] <icinga-wm>	 PROBLEM - git_daemon_running on gallium is CRITICAL: PROCS CRITICAL: 2 processes with regex args ^/usr/lib/git-core/git-daemon
[21:21:33] <icinga-wm>	 RECOVERY - git_daemon_running on gallium is OK: PROCS OK: 1 process with regex args ^/usr/lib/git-core/git-daemon
[21:33:57] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Wikimedia-Site-Requests: Check status of beta interwiki json/cdb files - https://phabricator.wikimedia.org/T120427#1853735 (10Krenair) 3NEW
[22:25:15] <wikibugs>	 10Beta-Cluster-Infrastructure, 6Editing-Department, 10Math, 10RESTBase, 5WMF-deploy-2015-12-08_(1.27.0-wmf.8): [Regression pre-wmf.8] Rendering of new formulæ is instead replaced by UNIQ--math-00000014-QINU on Beta Cluster due to old version of RESTbase - https://phabricator.wikimedia.org/T120318#1854114...
[23:28:31] <wikibugs>	 10Continuous-Integration-Config, 10Browser-Tests-Infrastructure, 10CirrusSearch: Make browsertests for CirrusSearch run on every submitted patch with proper CI infrastructure rather than a bot - https://phabricator.wikimedia.org/T98374#1854353 (10JanZerebecki) Currently Cindy has special setup code for these...