[00:20:23] <wikibugs__>	 06Release-Engineering-Team (Long-Lived-Branches), 06Performance-Team: Don't trash cache for front-end resources - https://phabricator.wikimedia.org/T102578#3170208 (10Krinkle)
[01:48:01] <wikibugs>	 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T160552#3170291 (10Legoktm)
[04:07:03] <wmf-insecte>	 Project selenium-MultimediaViewer » safari,beta,OS X 10.9,BrowserTests build #358: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/358/
[04:23:31] <shinken-wm>	 RECOVERY - Puppet run on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0]
[05:18:08] <shinken-wm>	 PROBLEM - Puppet run on deployment-restbase02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[05:33:09] <shinken-wm>	 RECOVERY - Puppet run on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0]
[06:21:01] <wmf-insecte>	 Yippee, build fixed!
[06:21:02] <wmf-insecte>	 Project selenium-Wikibase » chrome,test,Linux,BrowserTests build #327: 09FIXED in 1 hr 41 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=BrowserTests/327/
[08:55:48] <hashar>	 !log Retriggering MediaWiki doxygen publishing job for 1.26.0 - T162506 : zuul enqueue-ref --trigger gerrit --pipeline publish --project mediawiki/core --ref refs/tags/1.26.0 --newrev 981ec62244e4806a16bda804dda1d14cb5d7f193
[08:55:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[08:55:53] <stashbot>	 T162506: doc.wikimedia.org docs for old releases is actually master - https://phabricator.wikimedia.org/T162506
[09:03:39] <wikibugs__>	 10Continuous-Integration-Config, 06Release-Engineering-Team, 07Regression: doc.wikimedia.org docs for old releases is actually master - https://phabricator.wikimedia.org/T162506#3170688 (10hashar) ``` mediawiki/core$ git rev-parse 1.26.0 981ec62244e4806a16bda804dda1d14cb5d7f193 ```  In the [[ https://integra...
[09:13:27] <wikibugs>	 10Continuous-Integration-Config, 06Release-Engineering-Team, 07Regression: doc.wikimedia.org docs for old releases is actually master - https://phabricator.wikimedia.org/T162506#3170694 (10hashar) https://doc.wikimedia.org/mediawiki-core/1.26.0/php/DefaultSettings_8php.html now shows `$wgVersion = '1.26.0'`....
[09:34:17] <wikibugs>	 10Continuous-Integration-Config, 06Release-Engineering-Team, 07Regression: doc.wikimedia.org docs for old releases is actually master - https://phabricator.wikimedia.org/T162506#3170718 (10hashar) I am regenerating the documentation for all 1.23.x releases (based on `git show-ref --tags|grep tags/1.23|grep -...
[09:50:24] <hashar>	 !log Regenerating MediaWiki doxygen documentations for all 1.23.x releases.
[09:50:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[10:02:48] <wikibugs__>	 (03PS1) 10Hashar: Decouple mw doxygen doc jobs for branches and tags [integration/config] - 10https://gerrit.wikimedia.org/r/347579
[10:03:10] <wikibugs>	 (03PS2) 10Hashar: Decouple mw doxygen doc jobs for branches and tags [integration/config] - 10https://gerrit.wikimedia.org/r/347579 (https://phabricator.wikimedia.org/T115755)
[10:04:25] <wikibugs>	 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Little Steps Sprint), 13Patch-For-Review: Rewrite mediawiki-core-doxygen-publish Jenkins job to poll scm instead of being triggered by Zuul - https://phabricator.wikimedia.org/T115755#3170755 (10hashar)
[10:06:00] <wikibugs__>	 (03CR) 10Hashar: [C: 032] Decouple mw doxygen doc jobs for branches and tags [integration/config] - 10https://gerrit.wikimedia.org/r/347579 (https://phabricator.wikimedia.org/T115755) (owner: 10Hashar)
[10:10:49] <wikibugs__>	 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream, 07WorkType-NewFunctionality: Jenkins trilead-ssh2 doesn't support our MAC/KEX algorithms - https://phabricator.wikimedia.org/T103351#3170758 (10Paladox) @hashar Great news. The change has been merged now :). He is planning to see weather he ca...
[10:15:12] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Zuul: Find a way to deduplicate post-merge builds like mediawiki-core-doxygen-publish - https://phabricator.wikimedia.org/T94715#3170761 (10hashar) The jobs are run via postmerge which is an independent pipeline. The best way to handle the deduplicate would be to poll...
[10:17:05] <wikibugs__>	 06Release-Engineering-Team, 05MW-1.23-release: MW 1.23.16 is not compatible with PHP 5.3 - https://phabricator.wikimedia.org/T162572#3170766 (10TTO) @demon, do you anticipate you'll be able to quickly push out a 1.23.17 update to address this for PHP 5.3 users?
[10:32:44] <wikibugs>	 (03PS11) 10Hashar: Run webdriverio tests for MediaWiki core [integration/config] - 10https://gerrit.wikimedia.org/r/324719 (https://phabricator.wikimedia.org/T139740) (owner: 10Zfilipin)
[10:36:12] <wikibugs__>	 (03PS12) 10Hashar: Run webdriverio tests for MediaWiki core [integration/config] - 10https://gerrit.wikimedia.org/r/324719 (https://phabricator.wikimedia.org/T139740) (owner: 10Zfilipin)
[10:40:32] <wikibugs>	 (03CR) 10Hashar: [C: 032] "That merges skins/Vector in the main mediawiki queue which is a good thing :-}" [integration/config] - 10https://gerrit.wikimedia.org/r/324719 (https://phabricator.wikimedia.org/T139740) (owner: 10Zfilipin)
[10:41:02] <hashar>	 !log Enable webdriver.io  browser tests for MediaWiki core - https://gerrit.wikimedia.org/r/#/c/324719/ - T139740
[10:41:04] <hashar>	 zeljkof: ^^^
[10:41:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[10:41:06] <stashbot>	 T139740: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740
[10:41:51] <wikibugs>	 (03Merged) 10jenkins-bot: Run webdriverio tests for MediaWiki core [integration/config] - 10https://gerrit.wikimedia.org/r/324719 (https://phabricator.wikimedia.org/T139740) (owner: 10Zfilipin)
[10:42:38] <paladox>	 hashar you can try http://gerrit-icinga.wmflabs.org/icingaweb2/dashboard with the login being guest and password being guest
[10:42:42] <paladox>	 :)
[10:42:56] <paladox>	 hashar it looks really nice and easy to move around too.
[10:43:03] <paladox>	 Not to mention its now in php
[10:43:22] <hashar>	 zeljkof: and I am trying it out on a dummy mw/core change https://gerrit.wikimedia.org/r/#/c/92621/ 
[10:43:35] <hashar>	 paladox: the challenge is to generate the configuration file :D
[10:43:45] <paladox>	 hashar actually thats easy
[10:43:49] <hashar>	 paladox: I have no idea how Shinken config is generated. It is probably somewhere in puppet
[10:43:50] <paladox>	 one one command
[10:43:52] <paladox>	 enter hostname
[10:43:59] <hashar>	 zeljkof: https://integration.wikimedia.org/ci/job/mediawiki-core-qunit-selenium-jessie/1/console 
[10:44:14] <paladox>	 then it starts its thing
[10:44:53] <paladox>	 icinga2 node wizard
[10:45:51] <paladox>	 hashar icinga 2.x is a rewrite of icinga 1.x
[10:47:33] <paladox>	 hashar do we use this https://wiki.jenkins-ci.org/display/JENKINS/PostBuildScript+Plugin jenkins plugin?
[10:48:22] <zeljkof>	 hashar: it failed?
[10:48:53] <zeljkof>	 00:03:15.666 ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
[10:49:01] <hashar>	 00:03:12.147 /tmp/hudson2527915239228643910.sh: line 2: syntax error near unexpected token `then'
[10:49:05] <hashar>	 I CANT EVEN WRITE BASH
[10:49:30] <zeljkof>	 hashar: don't we have a bash linter?
[10:50:42] <zeljkof>	 ah, but bash is inside yaml file o.O so that would not work
[10:50:50] <hashar>	 zeljkof: https://integration.wikimedia.org/ci/job/mediawiki-core-qunit-selenium-jessie/jobConfigHistory/showDiffFiles?timestamp1=2017-04-11_10-37-22&timestamp2=2017-04-11_10-50-34 :D
[10:51:35] <zeljkof>	 hashar: that looks better
[10:52:05] <wikibugs__>	 (03PS1) 10Hashar: Fix syntax glitch in mediawiki-core-qunit-selenium-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/347587 (https://phabricator.wikimedia.org/T139740)
[10:52:20] <hashar>	 updated 
[10:52:21] <hashar>	 rebuilding https://integration.wikimedia.org/ci/job/mediawiki-core-qunit-selenium-jessie/2/console
[10:55:10] <zeljkof>	 hashar: still failed
[10:55:36] <zeljkof>	 looks like this did not find anything
[10:55:37] <zeljkof>	 00:02:12.177 + '[' -f ./tests/selenium/wdio.conf.js ']'
[10:55:42] <hashar>	 and MW_INSTALL_PATH is not set bah
[10:55:48] <hashar>	 I should have done an experimental job
[10:58:54] <wikibugs>	 (03PS2) 10Hashar: Polish up mediawiki-core-qunit-selenium-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/347587 (https://phabricator.wikimedia.org/T139740)
[10:59:22] <hashar>	 MW_INSTALL_PATH is defined via . /srv/deployment/integration/slave-scripts/bin/mw-set-env-localhost.sh
[10:59:24] <hashar>	 bah
[11:00:58] <hashar>	 and the jjb version we have does not support the junit "allow-empty-results" setting bah
[11:07:24] <zeljkof>	 ouch
[11:07:38] <zeljkof>	 reverting? or do you have an idea on how to solve it? upgrade jjb?
[11:08:04] <shinken-wm>	 PROBLEM - Puppet run on integration-c1 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[11:10:13] <hashar>	 zeljkof: yeah will have to upgrade JJB
[11:11:28] <hashar>	 and wdio writes the junit report  to ../log
[11:11:32] <hashar>	 which is not  $WORKSPACE/log/  :/
[11:12:56] * hashar crhttps://integration.wikimedia.org/ci/job/mediawiki-core-qunit-selenium-jessie/6/console
[11:12:58] <hashar>	 https://integration.wikimedia.org/ci/job/mediawiki-core-qunit-selenium-jessie/6/console
[11:14:23] <hashar>	 zeljkof: so I guess revert and redo
[11:15:04] <zeljkof>	 hashar: ouch
[11:15:15] <zeljkof>	 looks like we have messed it up this time
[11:15:24] <wikibugs>	 (03PS1) 10Hashar: Revert "Run webdriverio tests for MediaWiki core" [integration/config] - 10https://gerrit.wikimedia.org/r/347590 (https://phabricator.wikimedia.org/T139740)
[11:15:32] <zeljkof>	 do you want to finish it up, or should I create the experimental job?
[11:15:45] <hashar>	 reverted
[11:17:42] <wikibugs__>	 (03PS3) 10Hashar: Run webdriverio tests for MediaWiki core [integration/config] - 10https://gerrit.wikimedia.org/r/347587 (https://phabricator.wikimedia.org/T139740)
[11:19:28] <wikibugs>	 (03CR) 10Hashar: [C: 04-1] "Redone with:" [integration/config] - 10https://gerrit.wikimedia.org/r/347587 (https://phabricator.wikimedia.org/T139740) (owner: 10Hashar)
[11:19:37] <hashar>	 https://gerrit.wikimedia.org/r/#/c/347587/
[11:19:45] <hashar>	 will have to polish up that one
[11:21:57] <wikibugs>	 10Continuous-Integration-Infrastructure: Upgrade  our JJB fork to support Junit `allow-empty-results` - https://phabricator.wikimedia.org/T162674#3170872 (10hashar)
[11:22:34] <hashar>	 zeljkof: I am going to grab some food
[11:22:38] <hashar>	 and will bump JJB
[11:59:42] <wikibugs>	 (03PS1) 10Jakob: Enable QUnit tests for WikibaseLexeme [integration/config] - 10https://gerrit.wikimedia.org/r/347596 (https://phabricator.wikimedia.org/T161287)
[12:20:50] <wmf-insecte>	 Project language-screenshots-VisualEditor-347595 » chrome,Windows 10,ci-jessie-wikimedia build #1: 09SUCCESS in 14 min: https://integration.wikimedia.org/ci/job/language-screenshots-VisualEditor-347595/BROWSER=chrome,PLATFORM=Windows%2010,label=ci-jessie-wikimedia/1/
[12:21:27] <wikibugs>	 (03CR) 10Hashar: [C: 032] "Looks like there is some magic happening! Congratulations on figuring out the QUnit failure!" [integration/config] - 10https://gerrit.wikimedia.org/r/347596 (https://phabricator.wikimedia.org/T161287) (owner: 10Jakob)
[12:22:36] <wikibugs__>	 (03Merged) 10jenkins-bot: Enable QUnit tests for WikibaseLexeme [integration/config] - 10https://gerrit.wikimedia.org/r/347596 (https://phabricator.wikimedia.org/T161287) (owner: 10Jakob)
[12:35:08] <hashar>	 !log Force updated jenkins-job-builder from 1.5.0 to 1.6.0 and bumped python-jenkins to 0.4.14.  6fcaf39b...1639a86e
[12:35:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[12:35:40] <hashar>	 !log Force updated jenkins-job-builder from 1.5.0 to 1.6.0 and bumped python-jenkins to 0.4.14.  6fcaf39b...1639a86e  - T162674
[12:35:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[12:35:44] <stashbot>	 T162674: Upgrade  our JJB fork to support Junit `allow-empty-results` - https://phabricator.wikimedia.org/T162674
[12:36:36] <hashar>	 bah 236 new commits to review/test
[12:42:34] <wikibugs__>	 10Scap (Scap3-Adoption-Phase1), 10scap2, 07WorkType-NewFunctionality: Need a way to see config diffs in Scap - https://phabricator.wikimedia.org/T118206#3171171 (10mmodell) p:05Normal>03High During this week's deployment cabal meeting, @mobrovac expressed a need for this to happen sooner rather than later.
[13:33:10] <wikibugs__>	 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: OpenStack instances stuck in deletion state - https://phabricator.wikimedia.org/T162529#3171346 (10chasemp) a:05chasemp>03Andrew @andrew was looking into this.  Another instance was reported on IRC in {T162691}...
[13:35:52] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: OpenStack instances stuck in deletion state - https://phabricator.wikimedia.org/T162529#3171358 (10chasemp)
[13:44:28] <hashar>	 !log Updating all Jenkins jobs using the git plugin due to JJB change  cdfeb7bf66b0eacfed3eaf2a77813d65ab0e29f2 -  T162674
[13:44:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[13:44:31] <stashbot>	 T162674: Upgrade  our JJB fork to support Junit `allow-empty-results` - https://phabricator.wikimedia.org/T162674
[13:44:43] <hashar>	 !log Forced updated jenkins-job-builder 1639a86e...86478421 - T162674
[13:44:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[13:49:09] <shinken-wm>	 PROBLEM - Puppet run on buildlog is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[13:53:56] <wikibugs__>	 10Continuous-Integration-Infrastructure, 06Wikipedia-Android-App-Backlog: [Dev] Stand up a secure production build server - https://phabricator.wikimedia.org/T136662#3171419 (10Mholloway) p:05Low>03Normal
[13:54:11] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Wikipedia-Android-App-Backlog, 07Technical-Debt: [Dev] Stand up a secure production build server - https://phabricator.wikimedia.org/T136662#2343451 (10Mholloway)
[13:55:34] <wikibugs__>	 10Continuous-Integration-Infrastructure, 06Wikipedia-Android-App-Backlog, 07Technical-Debt: [Dev] Stand up a secure production build server - https://phabricator.wikimedia.org/T136662#2343451 (10Mholloway) Our [[ https://www.mediawiki.org/wiki/Wikimedia_Apps/Team/Release_process#Beta_release | release proces...
[13:57:40] <elukey>	 hashar: o/
[13:57:57] <elukey>	 wondering if you and your team were able to discuss the redis timeout issue
[14:02:07] <hashar>	 elukey: yeah shortly
[14:02:36] <hashar>	 so in short we should pair adding some debugging log to a jobrunner on production
[14:02:50] <hashar>	 I missed poking aaron about it yesterday evening :(
[14:03:19] <Zppix>	 im around if theres absolutely anything i can do to assist.
[14:03:56] <hashar>	 elukey: have you looked at how the jobrunner debug()  calls are logged?
[14:04:33] <elukey>	 hashar: didn't find the time, I thought that there was a flag somewhere to enable
[14:05:13] <hashar>	     public function debug( $s ) {
[14:05:13] <hashar>	         if ( $this->verbose ) {
[14:05:14] <hashar>	             print date( DATE_ISO8601 ) . " DEBUG: $s\n";
[14:05:17] <hashar>	 I guess that ends on stdout :D
[14:07:53] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: OpenStack instances stuck in deletion state - https://phabricator.wikimedia.org/T162529#3171454 (10Andrew)
[14:13:32] <elukey>	 ah lol 
[14:14:47] <hashar>	 elukey: so yeah I guess we want to hack something in puppet so that the service that runs redisJobRunnerService  is invoked with   --verbose
[14:14:56] <hashar>	 though maybe we can just run it from the command line
[14:16:15] <hashar>	 on deployment-jobrunner02 sudo -H -u  www-data /usr/bin/php /srv/deployment/jobrunner/jobrunner/redisJobRunnerService --config-file=/etc/jobrunner/jobrunner.conf --verbose
[14:16:26] <hashar>	 which is quite spammy ( eg: DEBUG: No jobs available... )
[14:18:11] <elukey>	 yeah meant for development probably
[14:18:46] <elukey>	 anyhow, I think that Aaron might be able to tune the jobrunner logging to be less spammy
[14:20:35] <hashar>	 elukey: https://phabricator.wikimedia.org/P5240
[14:20:38] <hashar>	 that is an example output
[14:20:46] <hashar>	 I crafted a custom jobrunner config to only use 1 runner
[14:21:49] <elukey>	 interesting..
[14:22:30] <hashar>	 brb
[14:22:32] <hashar>	 nature's call
[14:34:04] <hashar>	 o/
[14:34:13] <wmf-insecte>	 Project selenium-WikiLove » firefox,beta,Linux,BrowserTests build #359: 04FAILURE in 2 min 13 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/359/
[14:40:18] <hashar>	 bah and  print date( DATE_ISO8601 )   only has second resolution ...
[14:41:27] <hashar>	 "2017-04-11T14:41:23+0000"
[14:41:28] <hashar>	 bah
[14:41:35] <hashar>	 I am confused the +0000 is the timezone :D
[14:46:28] <hashar>	 elukey: i looked at the code and already found a bug :-}
[14:46:52] <hashar>	 we have a loop that retrieve jobs but the result is cached for 1 second
[14:47:06] <hashar>	 then the loop sleep for 100ms
[14:47:45] <hashar>	 so it loops ten times against a cached result
[14:47:56] <elukey>	 nice 
[14:48:02] <hashar>	 that is why on https://phabricator.wikimedia.org/P5240
[14:48:10] <hashar>	 one can see a hGetAll every second
[14:48:15] <hashar>	 and ten "No jobs available"
[14:48:28] <hashar>	 that is the loop checking whether the cached return has any job :-}
[14:49:03] <elukey>	 is there anybody that owns the job queues?
[14:49:45] <hashar>	 iirc that was a sprint by ori/aaron years ago
[14:49:59] <hashar>	 to prevent the whole cluster from going down and the madness the crontab runJobs.php was doing
[14:50:29] <hashar>	 and to migrate the job queue from MySQL to something else
[14:50:40] <hashar>	 it is still maintained, but on a best effort / when something goes really wrong
[15:07:26] <hashar>	 filled the loop bug https://phabricator.wikimedia.org/T162697
[15:13:08] <wikibugs__>	 06Release-Engineering-Team, 10Page-Previews, 06Reading-Web-Backlog: Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980#3171717 (10Jhernandez)
[15:13:48] <hashar>	 !log Forced updated jenkins-job-builder 86478421...022738f8 - T162674
[15:13:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:13:53] <stashbot>	 T162674: Upgrade  our JJB fork to support Junit `allow-empty-results` - https://phabricator.wikimedia.org/T162674
[15:17:57] <wikibugs>	 06Release-Engineering-Team, 10Page-Previews, 06Reading-Web-Backlog: Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980#3171761 (10Jhernandez) p:05Triage>03Normal We have a dependency on release engineering here. It would be great to coordinate to come up with...
[15:20:13] <elukey>	 thanks for the bug hashar! 
[15:20:31] <elukey>	 in the future maybe ChangePropagation + Kafka could be a good replacement for Redis?
[15:20:33] <elukey>	 who knows :)
[15:21:04] <hashar>	 yeah there is a task about replacing our current jobrunner
[15:21:12] <hashar>	 though changepropagation/kafka might be overkill
[15:22:36] <hashar>	 elukey: so most probably we can drop that useless message about lack of jobs
[15:22:45] <hashar>	 though on prod that might not be an issue
[15:22:58] <hashar>	 then try to run the obrunner directly from the command line with --verbose
[15:23:11] <hashar>	 and hope to catch a sequence that shows a server has been marked as down
[15:23:27] <hashar>	 but really, there is a glitch in the code that can be found by actually reading it
[15:23:59] <hashar>	 !log Adding parameter ZUUL_VOTING to all Jenkins jobs due to JJB upgrade T162674
[15:24:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:24:03] <stashbot>	 T162674: Upgrade  our JJB fork to support Junit `allow-empty-results` - https://phabricator.wikimedia.org/T162674
[15:27:53] <wikibugs__>	 10Deployment-Systems, 10Scap, 13Patch-For-Review: Update Debian Package for Scap3 - https://phabricator.wikimedia.org/T127762#3171881 (10fgiunchedi) 05Open>03Resolved
[15:36:45] <hashar>	 !log Updating selenium-* jobs configuration for the performance plugin due to JJB upgrade T162674
[15:36:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:36:49] <stashbot>	 T162674: Upgrade  our JJB fork to support Junit `allow-empty-results` - https://phabricator.wikimedia.org/T162674
[15:40:49] <hashar>	 !log Upgraded JJB to latest master 4f77324f  with a couple cherrypicks on top of that. 022738f8...edebce7f  T162674
[15:40:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:41:50] <wikibugs>	 10Browser-Tests-Infrastructure, 10MediaWiki-General-or-Unknown, 07JavaScript, 05MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)), and 4 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3171944 (10hashar)
[15:41:51] <wikibugs__>	 10Continuous-Integration-Infrastructure: Upgrade  our JJB fork to support Junit `allow-empty-results` - https://phabricator.wikimedia.org/T162674#3171940 (10hashar) 05Open>03Resolved a:03hashar That is it. JJB is updated to latest master 4f77324feae72a0663441a6b47591409b4fb0396  With a couple cherry picks:...
[15:46:43] <wikibugs>	 (03CR) 10Hashar: [C: 04-1] "recheck" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/347587 (https://phabricator.wikimedia.org/T139740) (owner: 10Hashar)
[15:47:22] <wikibugs>	 (03CR) 10Hashar: [C: 04-1] "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/347587 (https://phabricator.wikimedia.org/T139740) (owner: 10Hashar)
[15:49:24] <wikibugs__>	 (03CR) 10jerkins-bot: [V: 04-1] Run webdriverio tests for MediaWiki core [integration/config] - 10https://gerrit.wikimedia.org/r/347587 (https://phabricator.wikimedia.org/T139740) (owner: 10Hashar)
[15:51:18] <wmf-insecte>	 Project selenium-MobileFrontend » firefox,beta,Linux,BrowserTests build #387: 04FAILURE in 29 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/387/
[15:51:56] <hashar>	 zeljkof: so at least jjb is updated
[15:52:22] <hashar>	 and I mailed QA list about it
[15:53:32] <zeljkof>	 hashar: nice
[15:53:40] <zeljkof>	 the job still does not work?
[15:55:14] <wikibugs__>	 06Release-Engineering-Team, 05MW-1.23-release: MW 1.23.16 is not compatible with PHP 5.3 - https://phabricator.wikimedia.org/T162572#3171999 (10demon) >>! In T162572#3170766, @TTO wrote: > @demon, do you anticipate you'll be able to quickly push out a 1.23.17 update to address this for PHP 5.3 users?  Already...
[15:55:58] <hashar>	 zeljkof: the new parameter is a bit misleading
[15:56:10] <zeljkof>	 which one?
[15:56:45] <hashar>	 the Junit publisher now supports allow-empty-results
[15:56:49] <hashar>	 in case a junit.xml is empty
[15:56:59] <hashar>	 but it will still fail if there are no files generated 
[15:57:22] <hashar>	 and there is no unit xml file when we skip the wdio part
[15:57:30] <hashar>	 qunit does not generate junit output
[15:57:35] <zeljkof>	 hm
[15:57:49] <zeljkof>	 can the file be empty?
[15:58:03] <zeljkof>	 eg: touch log/junit.xml
[15:58:15] <hashar>	 yeah probably
[15:58:17] <zeljkof>	 in case it does not exist
[15:58:27] <zeljkof>	 or even some simple xml if needed
[15:58:42] <hashar>	 and I will probably read the code for the junit plugin
[15:58:46] <zeljkof>	 echo '<xml here>'
[15:58:47] <hashar>	 or maybe switch to the xunit one
[15:58:49] <hashar>	 ;D
[15:59:05] <hashar>	 !log integration-config-tox-jessie job is broken due to the JJB upgrade
[15:59:06] <zeljkof>	 can qunit generate xml?
[15:59:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:59:30] <zeljkof>	 we could generate the files from qunit probably
[16:00:26] <etonkovidova>	 hashar: I saw  'Fatal exception of type "DBqueryError" on ruwiki production  - and I cannot find the details of it nor in kibana (do not know the link) nor in logs. Fluorine does not work for me...
[16:00:35] <etonkovidova>	 hashar: what to do? :(
[16:00:59] <hashar>	 etonkovidova: I am entering a meeting sorry :(
[16:01:20] <etonkovidova>	 hashar: ah, ok - not urgent 
[16:01:26] <hashar>	 and fluorine has been removed
[16:01:45] <hashar>	 etonkovidova: try mwlog1001.eqiad.wmnet  and  /srv/mw-log
[16:01:46] <etonkovidova>	 hashar I know that 
[16:01:50] <hashar>	 but those logs should end up in logstash
[16:02:20] <etonkovidova>	 hashar:  sorry - I meant that I knew about fluorine :)) Will try your link - thanks! 
[16:03:50] <hashar>	 etonkovidova: https://logstash.wikimedia.org/goto/37ba8f721daf247e92f001d619e871a1
[16:03:58] <hashar>	 that is DBQueryError for ruwiki over the last 12 hours
[16:04:13] <etonkovidova>	 hashar: ok...
[16:04:19] <hashar>	 looks like some database is exploding.  Probably want to poke jynus / marostegui in #wikimedia-databases
[16:05:05] <hashar>	 etonkovidova: with nice things such as : Error: 1176 Key 'pl_from' doesn't exist in table 'pagelinks' (10.64.48.152)
[16:05:06] <etonkovidova>	 hashar: I see it - thx a lot! 
[16:05:10] <hashar>	 \O/
[16:05:19] <hashar>	 so some db is in bad shape I guess
[16:05:26] * hashar escapes to meeting
[16:06:29] <wikibugs>	 10Scap, 07Beta-Cluster-reproducible: [scap] Recompute and sync git version cache when sync-* are used - https://phabricator.wikimedia.org/T38271#3172047 (10demon) 05Open>03Resolved a:03demon This is live everywhere now (scap 3.5.5), feel free to reopen if it doesn't work.
[16:10:11] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-slave-jessie-1002 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1002.diskspace.root.byte_percentfree (<11.11%)
[16:12:59] <wikibugs>	 (03PS1) 10Hashar: Update test suite for new JJB [integration/config] - 10https://gerrit.wikimedia.org/r/347628 (https://phabricator.wikimedia.org/T162674)
[16:16:32] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Update test suite for new JJB [integration/config] - 10https://gerrit.wikimedia.org/r/347628 (https://phabricator.wikimedia.org/T162674) (owner: 10Hashar)
[16:17:36] <wikibugs__>	 (03PS2) 10Hashar: Update test suite for new JJB [integration/config] - 10https://gerrit.wikimedia.org/r/347628 (https://phabricator.wikimedia.org/T162674)
[16:31:18] <wikibugs>	 (03CR) 10Hashar: [C: 032] "Tiny adjustment to JJB api :-}" [integration/config] - 10https://gerrit.wikimedia.org/r/347628 (https://phabricator.wikimedia.org/T162674) (owner: 10Hashar)
[16:35:05] <wikibugs>	 (03Merged) 10jenkins-bot: Update test suite for new JJB [integration/config] - 10https://gerrit.wikimedia.org/r/347628 (https://phabricator.wikimedia.org/T162674) (owner: 10Hashar)
[16:47:33] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: OpenStack instances stuck in deletion state - https://phabricator.wikimedia.org/T162529#3172219 (10hashar) Well I guess this issue is fixed since labvirt1002 is out of the pool and that is being debugged via T162640...
[16:53:16] <Zppix>	 in regards to setting up a job for a repo all i have to do is add the correct config params to ci-config repo correct?
[16:53:25] <Zppix>	 its been a while
[17:04:39] <wikibugs__>	 06Release-Engineering-Team, 05MW-1.23-release: MW 1.23.16 is not compatible with PHP 5.3 - https://phabricator.wikimedia.org/T162572#3172315 (10demon) Files uploaded to releases.wm.o, just need to prep-send announcement and do the on-wiki updates.
[17:11:56] <wikibugs__>	 06Release-Engineering-Team, 10Page-Previews, 06Reading-Web-Backlog: Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980#3172362 (10Jdlrobson) Sure. My ask is that we talk to @greg and see if we can schedule some time in their busy schedule to work on this since th...
[17:13:18] <wikibugs__>	 06Release-Engineering-Team, 10Fundraising-Backlog, 03Fundraising Sprint Gondwanaland Reunification Engine: Fix raw HTML tags breaking 1.29.0-wmf19 security release - https://phabricator.wikimedia.org/T162716#3172381 (10awight) p:05Triage>03Unbreak!
[17:16:10] <wikibugs__>	 06Release-Engineering-Team, 10Fundraising-Backlog, 03Fundraising Sprint Gondwanaland Reunification Engine: Fix raw HTML tags breaking 1.29.0-wmf19+ - https://phabricator.wikimedia.org/T162716#3172390 (10demon)
[17:16:30] <wikibugs>	 06Release-Engineering-Team, 10Fundraising-Backlog, 03Fundraising Sprint Gondwanaland Reunification Engine: Fix raw HTML tags breaking 1.29.0-wmf19+ - https://phabricator.wikimedia.org/T162716#3172368 (10demon) (minor tweak to title, it's not a security release, and donatewiki would be affected on all release...
[17:28:34] <wikibugs>	 10Scap, 07Beta-Cluster-reproducible: [scap] Recompute and sync git version cache when sync-* are used - https://phabricator.wikimedia.org/T38271#3172474 (10Jdforrester-WMF) Wonderful. Thank you!
[17:52:04] <wikibugs__>	 10Scap (Scap3-Adoption-Phase1), 06Operations, 10RESTBase, 13Patch-For-Review, and 2 others: Deploy RESTBase with scap3 - https://phabricator.wikimedia.org/T116335#3172576 (10mobrovac)
[17:52:44] <wikibugs>	 10Scap (Scap3-Adoption-Phase1), 10releng-201516-q2, 10releng-201516-q3, 10releng-201516-q4, 10scap2: [keyresult]  Migrate all Service team owned services and MW to scap - https://phabricator.wikimedia.org/T109926#3172580 (10mobrovac)
[17:52:47] <wikibugs__>	 10Scap (Scap3-Adoption-Phase1), 06Operations, 10RESTBase, 06Services (done), 15User-mobrovac: Deploy RESTBase with scap3 - https://phabricator.wikimedia.org/T116335#1746874 (10mobrovac) 05Open>03Resolved The switch has been fully completed!
[17:55:44] <wikibugs__>	 10Scap (Scap3-Adoption-Phase1), 10releng-201516-q2, 10releng-201516-q3, 10releng-201516-q4, 10scap2: [keyresult]  Migrate all Service team owned services and MW to scap - https://phabricator.wikimedia.org/T109926#3172588 (10greg)
[17:55:47] <wikibugs>	 10Scap (Scap3-Adoption-Phase1), 10scap2, 10Cassandra, 10RESTBase-Cassandra, 06Services: Deploy logstash logback encoder with scap3 - https://phabricator.wikimedia.org/T116340#3172587 (10greg)
[18:03:33] <wikibugs>	 06Release-Engineering-Team, 10Fundraising-Backlog, 03Fundraising Sprint Gondwanaland Reunification Engine: Fix raw HTML tags breaking 1.29.0-wmf19+ - https://phabricator.wikimedia.org/T162716#3172647 (10awight) See https://phabricator.wikimedia.org/T156184
[18:39:48] <wikibugs__>	 06Release-Engineering-Team, 05MW-1.23-release: MW 1.23.16 is not compatible with PHP 5.3 - https://phabricator.wikimedia.org/T162572#3172732 (10demon) I take that back, had to regenerate the tag and tarball, I hadn't bumped the version number yet.
[18:54:20] <wikibugs>	 06Release-Engineering-Team, 05MW-1.23-release: MW 1.23.16 is not compatible with PHP 5.3 - https://phabricator.wikimedia.org/T162572#3172810 (10demon) [[ https://lists.wikimedia.org/pipermail/mediawiki-announce/2017-April/000208.html | 1.23.17 released ]]
[18:54:24] <wikibugs__>	 06Release-Engineering-Team, 05MW-1.23-release: MW 1.23.16 is not compatible with PHP 5.3 - https://phabricator.wikimedia.org/T162572#3172813 (10demon) 05Open>03Resolved a:03demon
[19:20:11] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-slave-jessie-1002 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1002.diskspace.root.byte_percentfree (<11.11%)
[20:45:11] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-slave-jessie-1002 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1002.diskspace.root.byte_percentfree (<11.11%)
[20:45:59] <paladox>	 hashar ^^
[20:46:58] <Zppix>	 paladox:  see T162635
[20:46:59] <stashbot>	 T162635: Investigate disk usage of  integration-slave-jessie-1002 - https://phabricator.wikimedia.org/T162635
[20:47:24] <paladox>	 thanks
[21:03:23] <wikibugs__>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Jenkins Web UI error - https://phabricator.wikimedia.org/T162505#3173353 (10greg)
[21:03:47] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Jenkins Web UI error - Backend fetch failed  - https://phabricator.wikimedia.org/T162505#3165426 (10greg)
[21:15:02] <wikibugs>	 (03CR) 10BryanDavis: "> That looks like a good first step for a simple jobs.  Eventually we" [integration/config] - 10https://gerrit.wikimedia.org/r/347130 (owner: 10Thcipriani)
[21:45:21] <wikibugs__>	 (03PS1) 10Hashar: dib: point imagecache to local directory [integration/config] - 10https://gerrit.wikimedia.org/r/347769
[21:45:24] <wikibugs>	 (03PS1) 10Hashar: dib: stop hardcoding image type [integration/config] - 10https://gerrit.wikimedia.org/r/347770
[21:47:51] <wikibugs>	 (03CR) 10Hashar: "DIB_IMAGE_CACHE is where diskimage-builder keeps various caches such as .deb packages, debootstrap result...  I typically do:" [integration/config] - 10https://gerrit.wikimedia.org/r/347769 (owner: 10Hashar)
[21:48:34] <wikibugs__>	 (03CR) 10Hashar: "diskimage-create defaults to qcow2 and apparently supports generating a docker image." [integration/config] - 10https://gerrit.wikimedia.org/r/347770 (owner: 10Hashar)
[22:15:43] <wikibugs__>	 (03CR) 10Hashar: [C: 04-1] "The image type must be passed as -t , not via env variable." [integration/config] - 10https://gerrit.wikimedia.org/r/347770 (owner: 10Hashar)
[22:20:09] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-slave-jessie-1002 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1002.diskspace.root.byte_percentfree (<11.11%)
[22:43:29] <shinken-wm>	 PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[22:48:19] <wikibugs__>	 10Continuous-Integration-Config, 06Release-Engineering-Team, 07Regression: doc.wikimedia.org docs for old releases is actually master - https://phabricator.wikimedia.org/T162506#3173692 (10Krinkle) >>! In T162506#3170718, @hashar wrote: > @Krinkle I guess we can delete the documentation for the legacy releas...