[00:36:43] greg-g: How would you feel about doing a deploy for https://phabricator.wikimedia.org/T164049#3223733 tomorrow or some time this week? [00:37:04] I'm not really familiar with the switchover schedule beyond "I need to be on IRC at 7am on Wednesday because that's when the main show is" [01:47:34] RoanKattouw: jfdi [01:48:03] Cool, thanks [01:48:20] We've added a fix for a 503 in the meantime too :) [02:04:06] (03CR) 10Kaldari: [C: 031] "Sounds reasonable. Don't have +2 rights though." [integration/config] - 10https://gerrit.wikimedia.org/r/337225 (https://phabricator.wikimedia.org/T157750) (owner: 10Paladox) [04:07:58] Yippee, build fixed! [04:07:59] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,BrowserTests build #379: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/379/ [04:17:02] Yippee, build fixed! [04:17:02] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #379: 09FIXED in 21 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/379/ [04:17:59] Yippee, build fixed! [04:17:59] Project selenium-MultimediaViewer » chrome,beta,OS X 10.9,BrowserTests build #379: 09FIXED in 21 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/379/ [07:32:09] RECOVERY - Free space - all mounts on castor is OK: OK: integration.castor.diskspace._mnt.byte_percentfree (No valid datapoints found) [07:53:20] Project beta-code-update-eqiad build #153913: 15ABORTED in 19 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/153913/ [08:17:21] !log Reconfigured all Jenkins jobs via jjb [08:17:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:33:11] Project beta-code-update-eqiad build #153918: 15ABORTED in 10 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/153918/ [09:00:03] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Install the blue ocean plugin alongside jenkins 2.x upgrade - https://phabricator.wikimedia.org/T155840#3227290 (10hashar) [09:00:06] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream, 07WorkType-NewFunctionality: Jenkins trilead-ssh2 doesn't support our MAC/KEX algorithms - https://phabricator.wikimedia.org/T103351#3227291 (10hashar) [09:00:08] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Upgrade Jenkins from 1.x to latest 2.x - https://phabricator.wikimedia.org/T144106#3227287 (10hashar) 05Open>03Resolved a:03hashar We are now running Jenkins LTS 2.46.2 I have also upgraded all plugins and installed the Pi... [09:00:44] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Upgrade Jenkins to 1.651.3 - https://phabricator.wikimedia.org/T144105#3227297 (10hashar) 05Open>03declined We went with some next LTS instead and are now at Jenkins 2.46.2 ( T144106 ). [09:00:46] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Upgrade Jenkins from 1.x to latest 2.x - https://phabricator.wikimedia.org/T144106#3227299 (10hashar) [09:04:20] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 07Jenkins: Upload Jenkins LTS v2.46.2 to jessie-wikimedia/third-party - https://phabricator.wikimedia.org/T157429#3227306 (10hashar) [09:19:30] PROBLEM - nodepoold running on labnodepool1001 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 113 (nodepool), regex args ^/usr/bin/python /usr/bin/nodepoold -d [09:21:30] RECOVERY - nodepoold running on labnodepool1001 is OK: PROCS OK: 1 process with UID = 113 (nodepool), regex args ^/usr/bin/python /usr/bin/nodepoold -d [09:28:53] https://integration.wikimedia.org/zuul/ something is stuck [09:41:37] !log flip deployment-cache-upload04 to deployment-ms-fe02 - T162247 [09:41:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:41:41] T162247: Migrate beta cluster Swift cluster from Trusty to Jessie - https://phabricator.wikimedia.org/T162247 [09:42:38] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream, 07WorkType-NewFunctionality: Jenkins trilead-ssh2 doesn't support our MAC/KEX algorithms - https://phabricator.wikimedia.org/T103351#3227378 (10hashar) [09:42:40] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Install the blue ocean plugin alongside jenkins 2.x upgrade - https://phabricator.wikimedia.org/T155840#3227377 (10hashar) [09:42:43] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Upgrade Jenkins from 1.x to latest 2.x - https://phabricator.wikimedia.org/T144106#3227375 (10hashar) 05Resolved>03Open I had to rollback to 2.32.3 , Nodepool was no more able to add instances as Jenkins slaves. [09:55:34] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Upgrade Jenkins from 1.x to latest 2.x - https://phabricator.wikimedia.org/T144106#3227387 (10hashar) That is due to 2.46.2 to prevent cross script forgery ( https://jenkins.io/security/advisory/2017-04-26/ ). Looks like Nodepoo... [10:01:25] upload is temporarily broken on beta btw, I'm looking [10:09:18] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Install the blue ocean plugin alongside jenkins 2.x upgrade - https://phabricator.wikimedia.org/T155840#3227422 (10Paladox) 05stalled>03Open This can now be done as we have upgraded to Jenkins 2.x [10:16:39] !log upgrade scap on deployment-tin to overcome AttributeError: Lock instance has no attribute 'get_lock_excuse' [10:16:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:18:23] ok that didn't work either, I wanted to sync-file private/PrivateSettings.php onto beta to point to deployment-ms-fe02, how do I do that? [10:21:54] !log bounce varnish and varnish-frontend on deployment-cache-upload04 [10:21:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:24:24] 10Beta-Cluster-Infrastructure, 10media-storage, 13Patch-For-Review, 15User-fgiunchedi: Migrate beta cluster Swift cluster from Trusty to Jessie - https://phabricator.wikimedia.org/T162247#3227444 (10fgiunchedi) Also `private/PrivateSettings.php` needed a change on deployment-tin to pick up deployment-ms02 [10:26:10] 06Release-Engineering-Team, 06Operations, 05Goal, 13Patch-For-Review, and 3 others: Prepare and maintain base container images - https://phabricator.wikimedia.org/T162042#3227446 (10akosiaris) >>! In T162042#3221344, @mobrovac wrote: >>>! In T162042#3220775, @MoritzMuehlenhoff wrote: >> I don't think we ne... [10:27:31] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:32:56] 06Release-Engineering-Team, 06Operations, 05Goal, 13Patch-For-Review, and 3 others: Prepare and maintain base container images - https://phabricator.wikimedia.org/T162042#3227451 (10MoritzMuehlenhoff) Definitely, zotero can reside in sca* until trusty support runs out in two years. [10:59:32] PROBLEM - Puppet errors on deployment-pdf01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:05:20] 11:18 ok that didn't work either, I wanted to sync-file private/PrivateSettings.php onto beta to point to deployment-ms-fe02, how do I do that? [11:05:24] hashar ^ perhaps? [11:10:14] godog: I have no clue. Sounds to me privateSettings.php would be deployed via scap [11:10:19] which is driven by Jenkins on beta cluster [11:10:30] and would require the Swift machine to be a dsh/scap target [11:15:50] hashar: ah ok, does jenkins auto-deploy on a schedule or I need to manually scap even in the private/ case? [11:16:05] I've already committed my change on deployment-tin's private/ [11:22:09] godog: scap is triggered after a change to operations/mediawiki-config has merged [11:22:48] so probably you want to scap manually. I guess https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/ would do it it [11:23:11] and apparently it ends up running every 10 minutes. Then I dont think it targets the Swift machines [11:23:43] hashar: ack, thanks for the pointer! yeah swift doesn't care about PrivateSettings.php, only mediawiki [11:33:19] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins, 13Patch-For-Review: Upgrade Jenkins from 1.x to latest 2.x - https://phabricator.wikimedia.org/T144106#3227542 (10hashar) So that progressed nicely, thanks to Moritz for the support. We are now at Jenkins 2.46.1. Further upgr... [12:04:22] (03PS2) 10Gergő Tisza: Add TK-999 to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/350593 [12:09:08] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins, 13Patch-For-Review: Upgrade Jenkins from 1.x to latest 2.x - https://phabricator.wikimedia.org/T144106#3227604 (10Paladox) Should we file a bug upstream at Jenkins or on openstack? [12:11:15] 10Deployment-Systems, 10MediaWiki-Debug-Logger, 05MW-1.29-release (WMF-deploy-2017-02-21_(1.29.0-wmf.13)), 05MW-1.29-release-notes, and 2 others: Capture PHP warnings with stacktraces in MediaWiki and save to logstash - https://phabricator.wikimedia.org/T45086#3227608 (10Tgr) [12:22:40] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins, 13Patch-For-Review: Upgrade Jenkins from 1.x to latest 2.x - https://phabricator.wikimedia.org/T144106#3227637 (10Paladox) @hashar you probably want to adjust your bug report to say 0.4.14 instead of 0.4.12. This https://github... [12:50:30] 10Beta-Cluster-Infrastructure, 10media-storage, 13Patch-For-Review, 15User-fgiunchedi: Migrate beta cluster Swift cluster from Trusty to Jessie - https://phabricator.wikimedia.org/T162247#3227662 (10fgiunchedi) a:03fgiunchedi I'm going to pause (but not terminate) old deployment-ms trusty hosts since the... [12:52:54] PROBLEM - Host deployment-ms-be02 is DOWN: CRITICAL - Host Unreachable (10.68.21.135) [12:53:26] PROBLEM - Host deployment-ms-fe01 is DOWN: CRITICAL - Host Unreachable (10.68.16.96) [12:53:28] PROBLEM - Host deployment-ms-be01 is DOWN: CRITICAL - Host Unreachable (10.68.16.24) [13:01:45] (03PS5) 10Rammanojpotla: mediawiki_selenium should document SauceLabs usage [selenium] - 10https://gerrit.wikimedia.org/r/348935 [13:11:21] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'https://en.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 1989 bytes in 0.031 second response time [13:11:53] 10Browser-Tests-Infrastructure, 07Documentation, 07Easy, 13Patch-For-Review, 07Software-Licensing: Ruby gem documentation should state license - https://phabricator.wikimedia.org/T94001#3227731 (10Rammanojpotla) [13:16:21] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 47023 bytes in 1.046 second response time [13:42:09] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins, 13Patch-For-Review: Upgrade Jenkins from 1.x to latest 2.x - https://phabricator.wikimedia.org/T144106#3227804 (10MoritzMuehlenhoff) I've built a custom package of python-jenkins 0.4.14 for jessie-wikimedia, I'll deploy it tomo... [14:43:53] (03PS5) 10Paladox: Revert "Rely on `php` in assert-phpflavor macro" [integration/config] - 10https://gerrit.wikimedia.org/r/337225 (https://phabricator.wikimedia.org/T157750) [14:49:55] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins, 13Patch-For-Review: Upgrade Jenkins from 1.x to latest 2.x - https://phabricator.wikimedia.org/T144106#3228003 (10Paladox) Any future upgrades to any more jenkins its releases will be blocked by T162828 (i mean a new upcoming l... [15:29:37] (03PS6) 10Rammanojpotla: mediawiki_selenium should document SauceLabs usage [selenium] - 10https://gerrit.wikimedia.org/r/348935 [15:46:30] Anyone know if you can make phabricator query and return only tasks that are in a specific column on a workboard? (I don't see a way in the advanced query but figured I'd ask.) [15:46:43] mainly since ops clinic duty is only concerned with 'shell' items in #wikimedia-mailing-lists [15:47:54] bleh, nm, found the open bug on phabricator site and they dont support it [15:47:56] https://secure.phabricator.com/T7907 [16:05:59] hasharAway looks like the trilead update broke ubuntu 12.04 support and other os'es see https://issues.jenkins-ci.org/browse/JENKINS-43979 [16:14:27] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:25:22] 10Beta-Cluster-Infrastructure, 06Operations, 07HHVM: Move the MW Beta appservers to Debian - https://phabricator.wikimedia.org/T144006#3228284 (10elukey) [16:26:08] Yippee, build fixed! [16:26:08] Project selenium-RelatedArticles » chrome,beta-desktop,Linux,BrowserTests build #394: 09FIXED in 18 sec: https://integration.wikimedia.org/ci/job/selenium-RelatedArticles/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta-desktop,PLATFORM=Linux,label=BrowserTests/394/ [16:26:28] Yippee, build fixed! [16:26:29] Project selenium-RelatedArticles » chrome,beta-mobile,Linux,BrowserTests build #394: 09FIXED in 38 sec: https://integration.wikimedia.org/ci/job/selenium-RelatedArticles/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta-mobile,PLATFORM=Linux,label=BrowserTests/394/ [16:28:45] hey all.. any idea why https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-RelatedArticles/ is actually reporting failure? [16:28:56] for above builds ^ ? [16:30:01] 10Continuous-Integration-Config, 06Reading-Web-Backlog: Proof of concept to generate coverage report of some Wikimedia mediawiki extensions - https://phabricator.wikimedia.org/T151333#3228313 (10Addshore) >>! In T151333#2976089, @hashar wrote: > I believe one can run solely RelatedArticles tests with `php phpu... [16:32:05] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team: Browser tests report as failures despite passing - https://phabricator.wikimedia.org/T164276#3228314 (10Jdlrobson) [16:32:53] thcipriani: marxarelli: you probably want Pipeline: Stage View https://plugins.jenkins.io/pipeline-stage-view [16:34:37] hasharAway: nice, we put that on ci-staging, looks nice, makes logs easy to find: http://ci-staging-jenkins.wmflabs.org/ci/job/Jenkinsfile.registry/ [16:34:50] excellent [16:35:08] as I got it with the pipeline plugin, a job is made of several steps [16:35:15] that each have they own logs/console etc [16:35:21] and eventually can run in parallel [16:35:36] neat :) [16:35:55] but for the current CI we gotta switch to pipeline syntax [16:37:04] I had some questions about that. Does that supplant jjb stuff? Or are we just going to create a pipelineJob in jjb and then put groovy in our yaml? [16:41:42] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 06Reading-Web-Backlog: Browser tests report as failures despite passing - https://phabricator.wikimedia.org/T164276#3228387 (10Jdlrobson) [16:47:36] thcipriani: I have no idea :( [16:47:53] thcipriani: I havent even looked at how JJB supports the pipeline/groovy syntax [16:48:23] anyway lunch time [16:48:44] thcipriani: and I will try to look at your Zuul config patch for swat gate ( https://gerrit.wikimedia.org/r/#/c/351131/ ) [16:48:47] thanks for that!! [16:51:17] hasharAway: awesome :) I spent some time lost in zuul source last week :P [16:52:48] Yippee, build fixed! [16:52:48] Project selenium-MobileFrontend » chrome,beta,Linux,BrowserTests build #411: 09FIXED in 27 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/411/ [17:01:16] Yippee, build fixed! [17:01:16] Project selenium-MobileFrontend » firefox,beta,Linux,BrowserTests build #411: 09FIXED in 36 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/411/ [17:05:40] PROBLEM - Puppet errors on buildlog is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:05:44] PROBLEM - Puppet errors on swift is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:06:44] PROBLEM - Puppet errors on deployment-urldownloader is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:07:18] PROBLEM - Puppet errors on swift-storage-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:07:22] PROBLEM - Puppet errors on deployment-ircd is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:08:47] PROBLEM - Puppet errors on deployment-phab01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:49:30] 10MediaWiki-Codesniffer: Release Codesniffer v0.8.0 - https://phabricator.wikimedia.org/T154715#2921575 (10kaldari) Yes, it would be awesome to have this ready before May 12th (MediaWiki Documentation Day). Who can actually accomplish this? [19:02:35] !log Added multichill ( https://github.com/multichill ) to the Wikimedia Github organization [19:02:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:03:03] Yippee, build fixed! [19:03:03] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #386: 09FIXED in 51 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/386/ [19:10:07] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-GettingStarted: Missing XML files cause "Publish Performance test result report" - https://phabricator.wikimedia.org/T164296#3228948 (10Mattflaschen-WMF) [19:11:19] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-GettingStarted: Missing XML files cause "Publish Performance test result report" - https://phabricator.wikimedia.org/T164296#3228965 (10Mattflaschen-WMF) [19:14:13] PROBLEM - Puppet errors on deployment-aqs03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [19:17:38] 10Beta-Cluster-Infrastructure, 06Labs, 10media-storage: Rebalance deployment-ms-be01 and deployment-ms-be02 so they run on different labvirt - https://phabricator.wikimedia.org/T161083#3229015 (10hashar) T162247 is creating new instances deployment-ms-be03 and deployment-ms-be04. Though I have no way to find... [19:18:44] 10Beta-Cluster-Infrastructure, 06Labs, 10media-storage: Confirm deployment-ms-be03 and deployment-ms-be04 so they run on different labvirt - https://phabricator.wikimedia.org/T161083#3229029 (10hashar) [19:20:25] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10media-storage, 13Patch-For-Review: deployment-ms-be01.deployment-prep and deployment-ms-be02.deployment-prep have high load / system CPU - https://phabricator.wikimedia.org/T160990#3229034 (10hashar) [19:21:45] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10media-storage, 13Patch-For-Review: deployment-ms-be01.deployment-prep and deployment-ms-be02.deployment-prep have high load / system CPU - https://phabricator.wikimedia.org/T160990#3117645 (10hashar) T162247 cre... [19:46:47] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-GettingStarted: Missing XML files cause "Publish Performance test result report" - https://phabricator.wikimedia.org/T164296#3229104 (10hashar) [19:49:11] RECOVERY - Puppet errors on deployment-aqs03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:51:15] !log Jenkins: rolling back Performance plugin from 2.2 to 2.0 due to an exception / failure to find a junit xml file. T164296 [19:51:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:51:20] T164296: Missing XML files cause "Publish Performance test result report" - https://phabricator.wikimedia.org/T164296 [19:52:08] Yippee, build fixed! [19:52:09] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #387: 09FIXED in 48 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/387/ [19:57:10] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-GettingStarted: Missing XML files cause "Publish Performance test result report" - https://phabricator.wikimedia.org/T164296#3229141 (10hashar) I thought maybe the Performance plugin would be to blame but apparent... [19:59:36] !log Regenerate jobs selenium-GettingStarted from JJB - T164296 [19:59:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:59:40] T164296: Missing XML files cause "Publish Performance test result report" - https://phabricator.wikimedia.org/T164296 [20:00:30] Yippee, build fixed! [20:00:30] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #388: 09FIXED in 49 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/388/ [20:06:45] Yippee, build fixed! [20:06:46] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #389: 09FIXED in 57 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/389/ [20:17:11] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-GettingStarted, 07Jenkins, 07Upstream: Missing XML files cause "Publish Performance test result report" - https://phabricator.wikimedia.org/T164296#3229187 (10hashar) That is a regression in the [[ https://wik... [20:17:35] Yippee, build fixed! [20:17:36] Project beta-scap-eqiad build #153594: 09FIXED in 3 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/153594/ [20:17:48] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-GettingStarted, 07Jenkins, 07Upstream: Missing XML files cause "Publish Performance test result report" - https://phabricator.wikimedia.org/T164296#3229191 (10hashar) p:05Triage>03Normal [20:18:56] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream: [upstream] Jenkins should send cookies with secure flag - https://phabricator.wikimedia.org/T36379#3229196 (10hashar) [20:21:05] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream: [upstream] Jenkins should send cookies with secure flag - https://phabricator.wikimedia.org/T36379#3229201 (10hashar) [20:24:30] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream: [upstream] Jenkins should send cookies with secure flag - https://phabricator.wikimedia.org/T36379#3229213 (10hashar) 05stalled>03Resolved a:03hashar Fixed in Jenkins 1.604 [20:32:13] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream: Jenkins: Builds (for beta cluster and browser tests) are stuck forever if IRC notification failed - https://phabricator.wikimedia.org/T96183#3229222 (10hashar) 05Open>03Resolved a:03hashar Might have been solved via IRC plugin 2.27. [20:32:17] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream: Jenkins does not shutdown due to a deadlock in the IRC plugin - https://phabricator.wikimedia.org/T98976#3229225 (10hashar) 05Open>03Resolved a:03hashar Might have been solved via IRC plugin 2.27. [20:32:19] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream: /etc/init.d/jenkins script provided by Debian doesn't work properly - https://phabricator.wikimedia.org/T53817#3229228 (10hashar) [20:35:44] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Install the blue ocean plugin alongside jenkins 2.x upgrade - https://phabricator.wikimedia.org/T155840#3229233 (10hashar) https://wiki.jenkins-ci.org/display/JENKINS/Running+Jenkins+behind+Apache states we should use `nocanon` i... [20:36:23] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Install the blue ocean plugin alongside jenkins 2.x upgrade - https://phabricator.wikimedia.org/T155840#3229236 (10Paladox) Yep. We do that for gerrit too. [20:39:44] 10Continuous-Integration-Infrastructure, 10ContentTranslation, 10Wikidata, 07WorkType-Maintenance: Wikibase Qunit test fails with CX + ULS - https://phabricator.wikimedia.org/T120907#3229251 (10hashar) [20:41:43] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Implement "Jenkins uptime" KPI - https://phabricator.wikimedia.org/T108769#3229254 (10hashar) Maybe later? I dont think Jenkins uptime itself is much important, it has been pretty robust over the years beside a few incident incl... [20:43:07] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Wikimedia-Hackathon-2017, 10Wikimedia-Logstash, and 2 others: Send Jenkins build log and results to ElasticSearch - https://phabricator.wikimedia.org/T78705#3229256 (10hashar) That is worth discussion / proof of concept during the hacka... [21:13:51] RainbowSprinkles theres an update for libmysql updating from .39 to .41 [21:14:23] !log Manually cancelled a few mediawiki-core-jsduck-publish and mediawiki-core-doxygen-publish job in Jenkins build queue. They seems to deadlock Jenkins somehow :( [21:14:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:20:07] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Labs instance ci-jessie-wikimedia-631424 can not be deleted - https://phabricator.wikimedia.org/T164305#3229375 (10hashar) [21:48:35] 10Gerrit, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: setup/install gerrit2001/WMF6408 - https://phabricator.wikimedia.org/T152525#3229484 (10Dzahn) Gerrit: **Finish replication prep** - https://gerrit.wikimedia.org/r/#/c/351520/ has been deployed. [22:13:22] Question about DB replication: about how quickly do the replicas catch up to the master DB, in most circumstances? I ask because https://en.wikipedia.beta.wmflabs.org/wiki/Wikipedia:WikiProject_Beta_2/Members does automatic sorting of active and inactive members, and the two people sorted as Inactive should never have been sorted as such to begin with. What [22:13:22] I am wondering is if DB replication has anything to do with it. [22:19:36] (03CR) 10Legoktm: [C: 04-2] "We need to fix the mw-tools-codesniffer-mwcore-testrun job, not this. All this will do is mask the error." [integration/config] - 10https://gerrit.wikimedia.org/r/337225 (https://phabricator.wikimedia.org/T157750) (owner: 10Paladox) [22:23:18] 10Gerrit, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: setup/install gerrit2001/WMF6408 - https://phabricator.wikimedia.org/T152525#3229622 (10Dzahn) We need to allow SSH between both servers for clustering, just like for Phabricator in T137928#2565556. [https://gerrit.wikimedia.org/r/#/c/... [22:24:32] (03PS1) 10Legoktm: Release 0.8.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/351528 [22:25:00] hare: few seconds, less than 5 [22:25:26] legoktm: so the fact that those users are inexplicably treated as inactive is more a bug with the extension than anything else [22:26:53] probably? [22:27:46] legoktm we update to phpcs 2.8.1 [22:27:49] * paladox submits patch [22:27:51] if you want to take a look: https://github.com/wikimedia/mediawiki-extensions-CollaborationKit/blob/master/includes/content/CollaborationListContent.php#L1118 [22:28:00] (03PS2) 10Legoktm: Release 0.8.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/351528 (https://phabricator.wikimedia.org/T154715) [22:28:02] paladox: huh? [22:28:23] we can update to https://github.com/squizlabs/PHP_CodeSniffer/releases/tag/2.8.1 before you release 0.8.0 :) [22:28:32] includes security fixes [22:29:03] ok [22:29:13] (03Draft1) 10Paladox: Update php code sniffer to 2.8.1 (also security updates) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/351532 [22:29:15] (03PS2) 10Paladox: Update php code sniffer to 2.8.1 (also security updates) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/351532 [22:29:18] legoktm ^^ [22:56:07] (03PS4) 10Mholloway: Android: Set -no-boot-anim command line flag for emulator [integration/config] - 10https://gerrit.wikimedia.org/r/349630 [23:42:12] (03CR) 10Niedzielski: [C: 032] Android: Set -no-boot-anim command line flag for emulator [integration/config] - 10https://gerrit.wikimedia.org/r/349630 (owner: 10Mholloway) [23:54:50] 10Gerrit, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: setup/install gerrit2001/WMF6408 - https://phabricator.wikimedia.org/T152525#3230090 (10Dzahn) after some debug: We need to move the ssh public key from gerrit2's home dir to /etc/ssh/userkeys/ to make ssh work for replication, we shou...