[00:24:14] legoktm: Indeed. It is created at run time and re-created if needed. [00:25:22] (03PS3) 10Krinkle: Simplify deploying zuul with fabric [integration/config] - 10https://gerrit.wikimedia.org/r/196002 (owner: 10Legoktm) [00:55:50] 10Continuous-Integration, 5Patch-For-Review: Figure out paths that needs to be backed up on gallium - https://phabricator.wikimedia.org/T65938#1124033 (10Dzahn) on gallium, bacula got installed Notice: /Stage[main]/Bacula::Client/Package[bacula-fd]/ensure: ensure changed 'purged' to 'present' Notice: /Stage[m... [00:59:19] (03PS1) 10Tim Starling: Run rebuildLocalisationCache.php as www-data [tools/scap] - 10https://gerrit.wikimedia.org/r/197262 [00:59:35] (03CR) 10jenkins-bot: [V: 04-1] Run rebuildLocalisationCache.php as www-data [tools/scap] - 10https://gerrit.wikimedia.org/r/197262 (owner: 10Tim Starling) [01:02:25] (03CR) 10Krinkle: [C: 032] Simplify deploying zuul with fabric [integration/config] - 10https://gerrit.wikimedia.org/r/196002 (owner: 10Legoktm) [01:02:39] (03CR) 10Krinkle: "Thanks! Cool stuff." [integration/config] - 10https://gerrit.wikimedia.org/r/196002 (owner: 10Legoktm) [01:03:32] (03Merged) 10jenkins-bot: Simplify deploying zuul with fabric [integration/config] - 10https://gerrit.wikimedia.org/r/196002 (owner: 10Legoktm) [01:12:57] (03PS2) 10Tim Starling: Run rebuildLocalisationCache.php as www-data [tools/scap] - 10https://gerrit.wikimedia.org/r/197262 [01:13:10] (03CR) 10jenkins-bot: [V: 04-1] Run rebuildLocalisationCache.php as www-data [tools/scap] - 10https://gerrit.wikimedia.org/r/197262 (owner: 10Tim Starling) [01:15:13] (03PS3) 10Tim Starling: Run rebuildLocalisationCache.php as www-data [tools/scap] - 10https://gerrit.wikimedia.org/r/197262 [01:15:26] (03CR) 10jenkins-bot: [V: 04-1] Run rebuildLocalisationCache.php as www-data [tools/scap] - 10https://gerrit.wikimedia.org/r/197262 (owner: 10Tim Starling) [01:17:48] (03PS4) 10Tim Starling: Run rebuildLocalisationCache.php as www-data [tools/scap] - 10https://gerrit.wikimedia.org/r/197262 [01:32:58] (03CR) 10Tim Starling: "Tested on tin, seems to work." [tools/scap] - 10https://gerrit.wikimedia.org/r/197262 (owner: 10Tim Starling) [01:39:32] (03PS5) 10Ori.livneh: Run rebuildLocalisationCache.php as www-data [tools/scap] - 10https://gerrit.wikimedia.org/r/197262 (owner: 10Tim Starling) [01:39:44] (03CR) 10jenkins-bot: [V: 04-1] Run rebuildLocalisationCache.php as www-data [tools/scap] - 10https://gerrit.wikimedia.org/r/197262 (owner: 10Tim Starling) [01:41:01] (03PS6) 10Ori.livneh: Run rebuildLocalisationCache.php as www-data [tools/scap] - 10https://gerrit.wikimedia.org/r/197262 (owner: 10Tim Starling) [01:41:16] (03CR) 10jenkins-bot: [V: 04-1] Run rebuildLocalisationCache.php as www-data [tools/scap] - 10https://gerrit.wikimedia.org/r/197262 (owner: 10Tim Starling) [01:41:44] (03PS7) 10Ori.livneh: Run rebuildLocalisationCache.php as www-data [tools/scap] - 10https://gerrit.wikimedia.org/r/197262 (owner: 10Tim Starling) [01:49:25] (03PS8) 10Ori.livneh: Run rebuildLocalisationCache.php as www-data [tools/scap] - 10https://gerrit.wikimedia.org/r/197262 (owner: 10Tim Starling) [01:49:53] (03CR) 10Ori.livneh: [C: 032] Run rebuildLocalisationCache.php as www-data [tools/scap] - 10https://gerrit.wikimedia.org/r/197262 (owner: 10Tim Starling) [01:50:08] (03Merged) 10jenkins-bot: Run rebuildLocalisationCache.php as www-data [tools/scap] - 10https://gerrit.wikimedia.org/r/197262 (owner: 10Tim Starling) [02:21:08] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<88.89%) [03:10:55] 10Continuous-Integration, 10MediaWiki-Codesniffer, 10Possible-Tech-Projects, 3Google-Summer-of-Code-2015, 3Outreachy-Round-10: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1124133 (10Kingstuffy) Hi all, I am interested in this project for the 2015 GSOC. I am mo... [04:49:02] 10Continuous-Integration, 10MediaWiki-Codesniffer, 10Possible-Tech-Projects, 3Google-Summer-of-Code-2015, 3Outreachy-Round-10: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1124204 (10devunt) [05:27:53] Yippee, build fixed! [05:27:53] Project browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #32: FIXED in 4 min 28 sec: https://integration.wikimedia.org/ci/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/32/ [06:16:12] 10Deployment-Systems, 7I18n: the message Helppage-top-gethelp doesn't appear deployed to the Hebrew Wikipedia - https://phabricator.wikimedia.org/T92823#1124254 (10Nemo_bis) [06:25:01] 10Deployment-Systems, 7I18n: the message Helppage-top-gethelp doesn't appear deployed to the Hebrew Wikipedia - https://phabricator.wikimedia.org/T92823#1124257 (10Nikerabbit) So because these messages were exported correctly, the cause is likely in LocalisationUpdate. Something happened about a month ago: `... [06:25:03] 10Deployment-Systems, 7I18n: Localisation updates from translatewiki.net not updated for Telugu for more than 10 days - https://phabricator.wikimedia.org/T92721#1124258 (10Nemo_bis) [06:28:21] 10Deployment-Systems, 7I18n: Localisation updates from translatewiki.net not updated for Telugu for more than 10 days - https://phabricator.wikimedia.org/T92721#1124270 (10Nemo_bis) 5Open>3stalled The last export from translatewiki.net was 631186747a916454c71ee49c7d078c5faa1a009b (you now have to look for... [06:32:24] 10Deployment-Systems, 7I18n: the message Helppage-top-gethelp doesn't appear deployed to the Hebrew Wikipedia - https://phabricator.wikimedia.org/T92823#1124279 (10Nikerabbit) That time frame matches 1.21wmf16 and contains a change to LocalisationUpdate: https://www.mediawiki.org/wiki/MediaWiki_1.25/wmf16#Loca... [06:32:31] 10Deployment-Systems, 7I18n: the message Helppage-top-gethelp doesn't appear deployed to the Hebrew Wikipedia - https://phabricator.wikimedia.org/T92823#1124281 (10Nemo_bis) Better wait next run, there were further changes this morning: https://gerrit.wikimedia.org/r/#/c/197262/8 [06:36:15] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [06:57:26] 10Deployment-Systems, 7I18n: Localisation updates from translatewiki.net not updated for Telugu for more than 10 days - https://phabricator.wikimedia.org/T92721#1124306 (10Nikerabbit) [06:57:26] 10Deployment-Systems, 7I18n, 5Patch-For-Review: the message Helppage-top-gethelp doesn't appear deployed to the Hebrew Wikipedia - https://phabricator.wikimedia.org/T92823#1124307 (10Nikerabbit) [06:57:52] 10Deployment-Systems, 7I18n, 5Patch-For-Review: the message Helppage-top-gethelp doesn't appear deployed to the Hebrew Wikipedia - https://phabricator.wikimedia.org/T92823#1124309 (10Nikerabbit) a:3Nikerabbit [07:02:02] 10Deployment-Systems, 10MediaWiki-extensions-LocalisationUpdate, 7I18n, 5Patch-For-Review: the message Helppage-top-gethelp doesn't appear deployed to the Hebrew Wikipedia - https://phabricator.wikimedia.org/T92823#1124310 (10Nemo_bis) [08:22:58] (03PS1) 10Adrian Lang: Make mwext-Wikibase-qunit voting [integration/config] - 10https://gerrit.wikimedia.org/r/197283 [08:48:59] 10Continuous-Integration, 5Patch-For-Review: Remove integration/kss.git - https://phabricator.wikimedia.org/T92482#1124386 (10hashar) a:3hashar [08:49:14] 10Continuous-Integration, 5Patch-For-Review: Remove integration/kss.git - https://phabricator.wikimedia.org/T92482#1124387 (10hashar) 5Open>3Resolved Deleted on all slaves. [08:52:28] (03PS2) 10Adrian Lang: Disable gzip in PhantomJS calls [integration/jenkins] - 10https://gerrit.wikimedia.org/r/197014 [08:52:44] (03CR) 10Adrian Lang: Disable gzip in PhantomJS calls (031 comment) [integration/jenkins] - 10https://gerrit.wikimedia.org/r/197014 (owner: 10Adrian Lang) [08:55:41] (03CR) 10Adrian Lang: "This failed for me locally for Wikibase for the resource loader module mw.config.values.wbSiteDetails. I suppose PhantomJS tripped over th" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/197014 (owner: 10Adrian Lang) [09:12:10] (03CR) 10Zfilipin: Set timeout to 45 minutes for db update (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/197226 (https://phabricator.wikimedia.org/T92906) (owner: 10Greg Grossmeier) [09:14:09] 10Continuous-Integration, 5Patch-For-Review: Zuul: run 'test' jobs on jenkins when trusted user votes +1 and only 'check' jobs was ran - https://phabricator.wikimedia.org/T64429#1124421 (10hashar) a:5hashar>3None Although there is a patch pending, I don't have spare bandwith to test/deploy it :( Moving to... [09:15:33] 10Continuous-Integration, 6operations: gallium.wikimedia.org disk space running low - https://phabricator.wikimedia.org/T91211#1124425 (10hashar) 5Open>3Resolved Resolved for now. Work is in progress to reduce the number of jobs being run that will help keep disk usage at a sane level. [09:17:09] (03PS1) 10Gilles: Add multimedia alerts list to UW tests [integration/config] - 10https://gerrit.wikimedia.org/r/197291 [09:21:27] !log deleted mwext-Wikibase-lint job, not triggered anymore [09:21:33] Logged the message, Master [09:24:44] !log deleted operations-puppet-validate [09:24:46] Logged the message, Master [09:32:53] (03PS1) 10Gilles: Switch Media Viewer Chrome browser test to OS X [integration/config] - 10https://gerrit.wikimedia.org/r/197293 (https://phabricator.wikimedia.org/T92810) [09:35:09] 10Continuous-Integration, 7Tracking: Zuul: scale merge operations (tracking) - https://phabricator.wikimedia.org/T70480#1124486 (10hashar) [09:35:10] 10Continuous-Integration: Zuul: setup a second merger on lanthanum - https://phabricator.wikimedia.org/T70482#1124484 (10hashar) 5Open>3declined We will get additional mergers via the #Continuous-Integration-Isolation project. [09:35:27] 10Continuous-Integration, 7Upstream: Zuul: Implement support for customizing status_url to include the change.id - https://phabricator.wikimedia.org/T65744#1124491 (10hashar) a:5hashar>3None [09:38:28] (03PS1) 10Hashar: Remove experiment-gating-dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/197294 [09:38:44] (03CR) 10Hashar: [C: 032] Remove experiment-gating-dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/197294 (owner: 10Hashar) [09:50:28] PROBLEM - Puppet staleness on deployment-cache-bits01 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [43200.0] [09:54:15] 10Continuous-Integration, 5Patch-For-Review: Have jenkins jobs logrotate their build history - https://phabricator.wikimedia.org/T91396#1124540 (10hashar) p:5High>3Normal I have cleaned up a few more jobs. Still have to finish up the whole cleanup though. Maybe we can get a test to ensure all jobs have so... [09:56:42] 10Continuous-Integration, 10Wikidata: Make mwext-Wikibase-qunit voting - https://phabricator.wikimedia.org/T92946#1124552 (10Tobi_WMDE_SW) 3NEW [09:57:00] 10Continuous-Integration, 10Wikidata: Make mwext-Wikibase-qunit voting - https://phabricator.wikimedia.org/T92946#1124559 (10Tobi_WMDE_SW) [09:57:02] 10Continuous-Integration, 10Wikidata, 3§ Wikidata-Sprint-2015-02-25, 3§ Wikidata-Sprint-2015-03-11: fix the qunit tests for wikidata: mwext-Wikibase-qunit - https://phabricator.wikimedia.org/T74184#1124560 (10Tobi_WMDE_SW) [09:57:32] 10Continuous-Integration, 10Wikidata, 3§ Wikidata-Sprint-2015-02-25, 3§ Wikidata-Sprint-2015-03-11: fix the qunit tests for wikidata: mwext-Wikibase-qunit - https://phabricator.wikimedia.org/T74184#750168 (10Tobi_WMDE_SW) Passing, all changes merged. Next step is to make the job voting: T92946 [09:57:42] 10Continuous-Integration, 10Wikidata: Make mwext-Wikibase-qunit voting - https://phabricator.wikimedia.org/T92946#1124552 (10Tobi_WMDE_SW) [09:57:43] 10Continuous-Integration, 10Wikidata, 3§ Wikidata-Sprint-2015-02-25, 3§ Wikidata-Sprint-2015-03-11: fix the qunit tests for wikidata: mwext-Wikibase-qunit - https://phabricator.wikimedia.org/T74184#1124564 (10Tobi_WMDE_SW) 5Open>3Resolved a:3Tobi_WMDE_SW [10:00:48] 10Continuous-Integration, 10MediaWiki-ResourceLoader, 10MediaWiki-Vagrant, 10Wikidata, and 3 others: qunit test broken without explicitly setting $wgResourceLoaderMaxQueryLength - https://phabricator.wikimedia.org/T90453#1124578 (10Tobi_WMDE_SW) For Wikidata this has been resolved so far by https://gerrit.... [10:00:51] 6Release-Engineering, 10MediaWiki-General-or-Unknown, 5MW-1.23-release, 15User-Bd808-Test: Create a minimal backport of PSR-3 logging to MediaWiki 1.23 LTS - https://phabricator.wikimedia.org/T91653#1124579 (10Aklapper) [10:13:58] 10Beta-Cluster, 10Continuous-Integration, 10Math: beta-recompile-math-texvc-eqiad job fails with "/usr/local/bin/scap-recompile: No such file or directory" - https://phabricator.wikimedia.org/T91191#1124628 (10fgiunchedi) FWIW the original context for this is T47076, I tend to agree with the rationale there.... [10:17:24] (03CR) 10Hashar: [C: 031] Switch Media Viewer Chrome browser test to OS X (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/197293 (https://phabricator.wikimedia.org/T92810) (owner: 10Gilles) [10:24:18] (03PS2) 10Hashar: Make mwext-Wikibase-qunit voting [integration/config] - 10https://gerrit.wikimedia.org/r/197283 (https://phabricator.wikimedia.org/T92946) (owner: 10Adrian Lang) [10:24:35] (03PS3) 10Hashar: Make mwext-Wikibase-qunit voting [integration/config] - 10https://gerrit.wikimedia.org/r/197283 (https://phabricator.wikimedia.org/T92946) (owner: 10Adrian Lang) [10:25:26] (03CR) 10Hashar: Make mwext-Wikibase-qunit voting (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/197283 (https://phabricator.wikimedia.org/T92946) (owner: 10Adrian Lang) [10:25:47] (03PS4) 10Hashar: Make mwext-Wikibase-qunit voting [integration/config] - 10https://gerrit.wikimedia.org/r/197283 (https://phabricator.wikimedia.org/T92946) (owner: 10Adrian Lang) [10:26:00] (03CR) 10Gilles: Switch Media Viewer Chrome browser test to OS X (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/197293 (https://phabricator.wikimedia.org/T92810) (owner: 10Gilles) [10:27:20] (03CR) 10Hashar: [C: 032] "Congratulations!" [integration/config] - 10https://gerrit.wikimedia.org/r/197283 (https://phabricator.wikimedia.org/T92946) (owner: 10Adrian Lang) [10:28:05] (03PS2) 10Zfilipin: Switch Media Viewer Chrome browser test to OS X [integration/config] - 10https://gerrit.wikimedia.org/r/197293 (https://phabricator.wikimedia.org/T92810) (owner: 10Gilles) [10:28:25] (03Merged) 10jenkins-bot: Make mwext-Wikibase-qunit voting [integration/config] - 10https://gerrit.wikimedia.org/r/197283 (https://phabricator.wikimedia.org/T92946) (owner: 10Adrian Lang) [10:29:14] 10Continuous-Integration, 10Wikidata, 5Patch-For-Review: Make mwext-Wikibase-qunit voting - https://phabricator.wikimedia.org/T92946#1124670 (10hashar) 5Open>3Resolved a:3hashar I have deployed the Zuul configuration change, thus the job should be voting now. Congratulations! [10:29:28] (03PS3) 10Zfilipin: Switch Media Viewer Chrome browser test to OS X [integration/config] - 10https://gerrit.wikimedia.org/r/197293 (https://phabricator.wikimedia.org/T92810) (owner: 10Gilles) [10:29:41] (03PS4) 10Zfilipin: Switch Media Viewer Chrome browser test to OS X [integration/config] - 10https://gerrit.wikimedia.org/r/197293 (https://phabricator.wikimedia.org/T92810) (owner: 10Gilles) [10:29:50] (03CR) 10Zfilipin: [C: 032] Switch Media Viewer Chrome browser test to OS X [integration/config] - 10https://gerrit.wikimedia.org/r/197293 (https://phabricator.wikimedia.org/T92810) (owner: 10Gilles) [10:32:11] (03CR) 10Zfilipin: "I have +2d the commit, please delete the old job and deploy the new one. Let us know if you need help." [integration/config] - 10https://gerrit.wikimedia.org/r/197293 (https://phabricator.wikimedia.org/T92810) (owner: 10Gilles) [10:34:20] (03Merged) 10jenkins-bot: Switch Media Viewer Chrome browser test to OS X [integration/config] - 10https://gerrit.wikimedia.org/r/197293 (https://phabricator.wikimedia.org/T92810) (owner: 10Gilles) [11:05:43] (03CR) 10Gilles: "All done, thanks for the pointers." [integration/config] - 10https://gerrit.wikimedia.org/r/197293 (https://phabricator.wikimedia.org/T92810) (owner: 10Gilles) [11:18:36] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce build #1: FAILURE in 29 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce/1/ [12:00:25] YuviPanda: is there a task I can work on to help out with staging stuff? not sure which one to do next [12:01:30] aharoni: looks like the sbu meeting is now [12:01:50] I think I have messed up the time in the original e-mail [12:10:35] mgooley: hi! [12:10:40] great to see you here on IRC [12:20:45] mgooley: you are welcome to join my team's channel: #mediawiki-i18n [12:37:35] PROBLEM - SSH on deployment-lucid-salt is CRITICAL: Connection refused [13:01:04] (03CR) 10Polybuildr: "... okay, I might have had some extra files in that directory. 66 matches, not 81." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/196872 (https://phabricator.wikimedia.org/T92749) (owner: 10Polybuildr) [13:18:49] 10Staging, 5Patch-For-Review: Create staging-mx (Mail server, pollonium replacement) - https://phabricator.wikimedia.org/T91562#1125135 (10thcipriani) Parametrization is done in the gerrit patch [[ https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep | deployment-prep hiera ]] has been updated. Prod and... [13:28:36] (03PS1) 10Hashar: Remove DataTypes mw extension [integration/config] - 10https://gerrit.wikimedia.org/r/197317 (https://phabricator.wikimedia.org/T63601) [14:39:20] !log me versus debian packaging tool chain http://xkcd.com/1168/ [14:39:22] Logged the message, Master [14:40:01] hashar: https://github.com/jordansissel/fpm ;-) [14:40:25] werdna: fpm generated packages are not really suitable for Debian uploading :D [14:40:46] hashar: that's because Debian packaging people suck :p [15:03:22] (03CR) 10Hashar: [C: 032] Remove DataTypes mw extension [integration/config] - 10https://gerrit.wikimedia.org/r/197317 (https://phabricator.wikimedia.org/T63601) (owner: 10Hashar) [15:08:01] (03Merged) 10jenkins-bot: Remove DataTypes mw extension [integration/config] - 10https://gerrit.wikimedia.org/r/197317 (https://phabricator.wikimedia.org/T63601) (owner: 10Hashar) [15:18:46] hmm, this is blank for me: https://wikitech.wikimedia.org/w/index.php?title=Special:NovaProject&action=displayquotas&projectname=staging [15:18:58] (looking at the labs quota for staging) [15:20:00] twentyafterfour: moving all role defs to staging.yaml (the ENC got merged, woo!) maybe? [15:20:16] (03PS1) 10Hashar: Package python deps with dh-virtualenv [integration/zuul] (debian/trusty-wikimedia) - 10https://gerrit.wikimedia.org/r/197328 (https://phabricator.wikimedia.org/T48552) [15:20:18] (03PS1) 10Hashar: Forward port precise dh-virtualenv to trusty [integration/zuul] (debian/trusty-wikimedia) - 10https://gerrit.wikimedia.org/r/197329 (https://phabricator.wikimedia.org/T48552) [15:20:40] aharoni: around? [15:21:44] (03CR) 10Hashar: [C: 04-2] "That is a dupe of a change on the reference branch debian/precise-wikimedia which is pending merge. https://gerrit.wikimedia.org/r/195272" [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/195541 (https://phabricator.wikimedia.org/T48552) (owner: 10Hashar) [15:21:51] (03CR) 10Hashar: [C: 04-2] "That is a dupe of a change on the reference branch debian/precise-wikimedia which is pending merge. https://gerrit.wikimedia.org/r/195272" [integration/zuul] (debian/trusty-wikimedia) - 10https://gerrit.wikimedia.org/r/197328 (https://phabricator.wikimedia.org/T48552) (owner: 10Hashar) [15:22:00] greg-g: that's weird, that page works for me. And we seem to have the same role in the staging project. [15:22:15] (03CR) 10Hashar: "check experimental" [integration/zuul] (debian/trusty-wikimedia) - 10https://gerrit.wikimedia.org/r/197329 (https://phabricator.wikimedia.org/T48552) (owner: 10Hashar) [15:25:10] thcipriani: greg-g maybe greg-g has to log out and back in? [15:25:12] ala wikitecccch [15:27:14] YuviPanda: really? /me tries [15:27:54] YuviPanda: you were right cc thcipriani [15:28:14] :) does the quota need to be kicked up? [15:28:59] probably [15:29:10] YuviPanda: I'm looking at  [15:29:12] https://phabricator.wikimedia.org/T73886 [15:29:23] YuviPanda: where are the roles defined currently? just in wikitech ui? [15:29:30] twentyafterfour: yup [15:31:57] 6Release-Engineering, 10Wikimedia-Labs-General: Increase quota for deployment-prep (beta) project - https://phabricator.wikimedia.org/T73886#1125449 (10greg) Currently.... #staging is: Cores: 29/30 RAM: 59392/102400 Floating IPs: 0/0 Instances: 12/15 Security Groups: 13/20 and deployment-... [15:34:51] twentyafterfour: I've also been keeping some roles on palladium under /etc/puppet/hieradata/labs/staging/host/*.yaml just so they're easy to destroy/spin-up. [15:35:02] 10Beta-Cluster, 10RESTBase: Update / maintain Beta Cluster restbase cluster - https://phabricator.wikimedia.org/T91102#1125454 (10greg) [15:35:23] 10Beta-Cluster, 10RESTBase: Update / maintain Beta Cluster restbase cluster - https://phabricator.wikimedia.org/T91102#1074329 (10greg) What's the status of this? This was considered a blocker for the initial rollout. [15:36:12] greg-g: yeah, on it. [15:36:46] YuviPanda: thanks, I'm just going through old bugmail :) [15:36:58] greg-g: :) cool. [15:37:18] thcipriani: I’m going to get rid of hiera_include(‘classes’) now. [15:37:22] since we have this instead…. [15:37:42] <^d> Where are we on trebuchet for tin? [15:37:47] YuviPanda: yup, do iiit. [15:37:48] <^d> (and autosigner?) [15:38:00] autosigner is done [15:38:12] (and works) [15:38:29] <^d> Ah ok, I thought I saw something in scrollback about it yesterday [15:39:07] ^d: YuviPanda staging-tin looks like it needs some manual stuff done as of late yesterday. git deploy start/sync...maybe...wasn't sure. [15:39:35] ^d: yup, it does. I am not sure how to automate that, but I’ve been swamped with ops clinic duty stuff yesterday, and just woke up :) [15:40:52] 10Beta-Cluster, 10RESTBase: Update / maintain Beta Cluster restbase cluster - https://phabricator.wikimedia.org/T91102#1125471 (10GWicke) @eevans is looking into setting up a cluster in labs. In the meantime, labs testing can directly use the prod cluster. [15:42:49] 10Beta-Cluster, 10RESTBase: Update / maintain Beta Cluster restbase cluster - https://phabricator.wikimedia.org/T91102#1125476 (10greg) Just to be pedantic: "Beta Cluster". "Labs" is too generic of a word in our world :) Beta Cluster is lagging production (which shouldn't happen too often, if ever) since VE i... [15:43:28] ^d: thcipriani hmm, actually, we should be able to set this up automatically... [15:43:36] let me try setup sca* and see what happens. [15:45:34] 10Beta-Cluster, 10RESTBase: Update / maintain Beta Cluster restbase cluster - https://phabricator.wikimedia.org/T91102#1125493 (10GWicke) @greg, beta labs VE can use restbase if configured to do so. Prod is mostly not using RB yet. We realized pretty late that the VE wmf20 code in prod doesn't have the restbas... [15:46:27] (03CR) 10Hashar: [C: 04-1] "check-only pipeline should still ignore l10n-bot." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/194990 (https://phabricator.wikimedia.org/T91707) (owner: 10Legoktm) [15:49:27] 10Continuous-Integration, 6translatewiki.net, 5Patch-For-Review: l10n-bot self-force-merging sometimes breaks mediawiki/core master - https://phabricator.wikimedia.org/T91707#1125504 (10hashar) I am fine having the l10n update changes be pushed between 19:00 and 22:00 UTC, though I am not sure why they need... [15:51:37] (03CR) 10Hashar: Set timeout to 45 minutes for db update (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/197226 (https://phabricator.wikimedia.org/T92906) (owner: 10Greg Grossmeier) [15:51:50] 10Beta-Cluster, 10RESTBase: Update / maintain Beta Cluster restbase cluster - https://phabricator.wikimedia.org/T91102#1125510 (10greg) >>! In T91102#1125493, @GWicke wrote: > @greg, beta labs VE can use restbase if configured to do so. Prod is mostly not using RB yet. We realized pretty late that the VE wmf20... [15:54:35] 6Release-Engineering, 10Wikimedia-Labs-General: Increase quota for deployment-prep (beta) project - https://phabricator.wikimedia.org/T73886#1125542 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Set staging's quota to match deployment-prep's. [15:56:01] (03PS2) 10Hashar: Beta timing out jobs now abort + 45 mins for db update [integration/config] - 10https://gerrit.wikimedia.org/r/197226 (https://phabricator.wikimedia.org/T92906) (owner: 10Greg Grossmeier) [15:56:32] 10Beta-Cluster, 10RESTBase: Update / maintain Beta Cluster restbase cluster - https://phabricator.wikimedia.org/T91102#1125556 (10greg) a:5GWicke>3Eevans >>! In T91102#1125471, @GWicke wrote: > @eevans is looking into setting up a cluster in beta [cluster]. In the meantime, labs testing can directly use th... [15:56:41] 10Beta-Cluster, 10RESTBase: Update / maintain Beta Cluster restbase cluster - https://phabricator.wikimedia.org/T91102#1125558 (10GWicke) RB and cassandra are both fully puppetized. The difficult bits are dealing with trebuchet, and possibly updating things from master. [16:00:56] ^d: twentyafterfour hashar meeting ping :) [16:01:07] <^d> Yes moment [16:01:26] give me time to click all those buttons ! [16:16:22] !log created staging-sca01 [16:16:25] Logged the message, Master [16:18:05] What runs the l10nupdate for Beta Cluster? beta-scap-eqiad? 'Cos it doesn't appear to be working… [16:18:29] James_F: yeah, scap does it [16:18:53] there was a change last night for prod problems, it may have beta blowback [16:18:54] bd808: Hmm. Might the fixes for real-scap last night have broken it? (Seems unlikely.) [16:18:58] * James_F nods. [16:19:24] * bd808 ignored personas talk and debugs [16:20:24] Warning: dba_open(/srv/mediawiki-staging/php-master/cache/l10n/l10n_cache-ab.cdb.tmp.1846872619): failed to open stream: Permission denied in /mnt/srv/mediawiki-staging/multiversion/vendor/wikimedia/cdb/src/Writer/DBA.php on line 38 [16:20:38] looks like the new change isn't there maybe [16:21:32] That'd not help, I imagine. [16:21:37] Does scap scap scap? [16:21:53] * bd808 pokes more [16:25:28] !log chown -R trebuchet:wikidev && chmod -R g+rwX deployment-bastion:/srv/deployment/scap/scap [16:25:30] Logged the message, Master [16:27:03] James_F: heh. no Trebuchet flings scap [16:27:12] Aha. [16:27:32] And git-deploy gushes Trebuchet and puppet strings along git-deploy? ;-) [16:27:52] something like that [16:28:37] !log Updated scap to include I61dcf7ae6d52a93afc6e88d3481068f09a45736d (Run rebuildLocalisationCache.php as www-data) [16:28:40] Logged the message, Master [16:29:08] bd808: Should I manually trigger a beta-scap-eqiad run? [16:29:26] I'm running manually to see if it blows up again [16:29:50] looks like it is working this time [16:30:21] Kk. Nice. [16:31:05] 10Deployment-Systems, 10Staging: Trebuchet doesn't work until manual 'git deploy start' on deployment-server - https://phabricator.wikimedia.org/T92978#1125632 (10yuvipanda) 3NEW [16:31:39] 10Deployment-Systems, 10Staging, 6operations, 7Puppet: provider => trebuchet doesn't work until manual 'git deploy start' on deployment-server - https://phabricator.wikimedia.org/T92978#1125640 (10yuvipanda) [16:34:13] Project beta-scap-eqiad build #45583: FAILURE in 0.87 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/45583/ [16:34:15] bd808: do you know whom I should poke to learn aboug git-deploy? [16:34:35] Well... [16:34:45] Ryan [16:35:08] but I think apergos knows quite a bit [16:35:22] cough [16:35:25] and ErikB I think has poked at the guts [16:35:26] hi apergos [16:35:34] Ori read all the code at some point [16:35:36] not so much but if I poke around I can remember [16:35:46] alright, I’ll try them in order :) [16:35:55] feel free to add me on that last task [16:36:03] and I have some idea of how the UI/UX ends up being a pain [16:36:04] 92978 I mean [16:37:43] 10Deployment-Systems, 10Staging, 6operations, 7Puppet: provider => trebuchet doesn't work until manual 'git deploy start' on deployment-server - https://phabricator.wikimedia.org/T92978#1125647 (10yuvipanda) [16:38:10] bd808: https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/45583/console :-( [16:38:19] apergos: added you [16:38:35] James_F: manual run is still going [16:38:45] l10nupdate is sloooooow in beta [16:39:33] greg-g, hey, I noticed the operations/mediawiki-config queue has a bunch of things in it that should probably just happen [16:39:38] I think Reedy used to go through these [16:39:38] thanks [16:40:10] Krenair: oh, open patches/ [16:40:11] ? [16:40:14] yeah [16:40:22] bd808: Ah. [16:40:30] they are SWATed nowadays arent they? [16:40:46] hashar: if they get on the list, sometimes they don't [16:40:47] right but they need to actually be put up for swat [16:40:47] hashar: Some of them. [16:40:53] otherwise they just sit there [16:40:57] Which needs a community member to know that. [16:41:07] marxarelli: zeljkof_ and I found a nice delay in browser tests yesterday. apparently caused by PageObject doing very lame iterations :D https://phabricator.wikimedia.org/T92613 [16:41:15] Often these patches are written by non-staff. [16:41:20] marxarelli: that should cut the run time by a lot! [16:41:20] ^d, Glaisher: What happened to https://gerrit.wikimedia.org/r/#/c/196779/ ? [16:41:24] bd808: apergos if I can find out which code sets up /srv/deployment on the *deployment host* I’ll be good. I can only trace it down to a repo_config.sls file set, and then I can’t find out what uses it... [16:41:29] one or two of them are by staff, James_F :) [16:41:41] hashar, marxarelli: I plan to take a look this week [16:41:47] anyway I was looking at https://phabricator.wikimedia.org/P405 [16:41:47] YuviPanda: eh... I'll look [16:41:48] Krenair: Sure. :-) [16:41:51] Krenair: see the phab ticket [16:41:56] bd808: \o/ <3 [16:42:02] zeljkof: do we have any way to enable some debug logs / function calls to page object? [16:42:25] hashar: I will create a small script that does the same thing with pure selenium and with page object [16:42:36] we will have all selenium logs in sauce [16:42:39] and we can compare [16:42:51] zeljkof: oky doky :] [16:42:55] YuviPanda: I *think* this does it -- https://github.com/wikimedia/operations-puppet/blob/production/modules/deployment/manifests/deployment_server.pp#L61 [16:43:05] zeljkof, hashar: oh wow. is it something we can patch? [16:43:14] Glaisher, okay. looked weird because it was approved but jenkins did not merge [16:43:21] have left a -1 saying to see the ticket, thanks [16:43:29] bd808: oh, hmm. where’s the code for that salt module going to be? [16:43:53] marxarelli, hashar: I have to investigate if the problem is on our side of page-object, but feel free to take a loo [16:43:54] look [16:43:54] zeljkof, hashar: yeah, try to repro it with a smaller script and then we can profile it [16:44:11] YuviPanda: https://github.com/wikimedia/operations-puppet/blob/production/modules/deployment/files/modules/deploy.py#L257 [16:44:18] marxarelli: no idea, pointed it to you for info :) [16:44:31] YuviPanda: oooh... https://github.com/wikimedia/operations-puppet/blob/production/modules/deployment/files/modules/deploy.py#L155 [16:44:32] I am sure zeljkof will figure it out :] [16:44:38] cool :) [16:44:38] twentyafterfour: Krenair (above) has a point re the mediawiki-config backlog. Can you take a look at that P405 and then https://gerrit.wikimedia.org/r/#/q/project:operations/mediawiki-config+status:open,n,z for anything simple, obviously OK, and/or with a phab ticket that explains rationale and looks like there's concensus? [16:44:45] zeljkof: happy hunting :) [16:44:59] hashar, marxarelli: thanks, oiling my gun [16:45:03] spelling is hard [16:46:02] YuviPanda: I really don't see obviously how any of that creates /srv/deployment [16:46:27] 6Release-Engineering, 10Wikimedia-Hackathon-2015: Release/QA tasks at the Wikimedia Hackathon 2015 - https://phabricator.wikimedia.org/T92565#1125683 (10zeljkofilipin) [16:47:26] James_F: 16:45:31 Finished mw-update-l10n (duration: 16m 27s) [16:47:31] Whee. [16:47:33] almost there [16:47:46] bd808: I guess’ config[‘location’] is set to that path somehow [16:48:20] bd808: /srv/deployment itself is created in puppet, role/deployment.pp, line 188 [16:48:32] ah [16:48:42] I was looking the ::deployment class [16:49:06] bd808: hmm, now even doing a ‘git deploy start’ tells me ‘Failed to create lockfile, failed to start deployment' [16:50:53] I remember having to chown and chmod after the first setup on deployment-bastion [16:51:14] chown -R trebuchet:wikidev [16:51:23] and chmod -R g+rwX [16:51:58] Which I actually just had to do for scap's deploy dir 30 minutes ago [16:52:07] James_F: 16:51:35 Finished scap: testing l10nupdate #2 (duration: 22m 39s [16:52:36] bd808: Does that mean it should now be pushed? I'm not very familiar with the internals of scap. :-) [16:52:52] l10n should be up to date [16:53:00] if it's not there is another problem [16:53:13] bd808: yup, I did the chown (to trebuchet:project-staging) but not the chmod [16:54:51] * James_F refreshes. [16:55:32] twentyafterfour/^d/all: just FYI: I have a drs appt in a bit and then a DMV appt, so I'll be afk much of the middle of the day. [16:56:01] * ^d puts on his crazy party hat [16:56:38] ^d: I presume it is green [16:58:12] bd808: Think it's just caching in RL. Thank you for everything. [16:58:20] (03CR) 10Legoktm: "@hashar: the TWN change is Idaf1fc15f1a52d377ee6fb2f29889d789aa07883" [integration/config] - 10https://gerrit.wikimedia.org/r/194990 (https://phabricator.wikimedia.org/T91707) (owner: 10Legoktm) [16:58:28] James_F: yw [16:58:32] * bd808 was never here [17:01:16] (03PS2) 10Legoktm: Don't ignore l10n-bot in gate-and-submit pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/194990 (https://phabricator.wikimedia.org/T91707) [17:01:25] (03CR) 10Legoktm: Don't ignore l10n-bot in gate-and-submit pipeline (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/194990 (https://phabricator.wikimedia.org/T91707) (owner: 10Legoktm) [17:01:34] (03PS3) 10Legoktm: Don't ignore l10n-bot in gate-and-submit pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/194990 (https://phabricator.wikimedia.org/T91707) [17:08:37] Yippee, build fixed! [17:08:38] Project beta-scap-eqiad build #45585: FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/45585/ [17:49:35] (03PS1) 10Krinkle: Remove non-voting property of mwext-Collection-jslint [integration/config] - 10https://gerrit.wikimedia.org/r/197371 (https://phabricator.wikimedia.org/T63594) [17:49:49] (03CR) 10Krinkle: [C: 032] Remove non-voting property of mwext-Collection-jslint [integration/config] - 10https://gerrit.wikimedia.org/r/197371 (https://phabricator.wikimedia.org/T63594) (owner: 10Krinkle) [17:50:57] (03Merged) 10jenkins-bot: Remove non-voting property of mwext-Collection-jslint [integration/config] - 10https://gerrit.wikimedia.org/r/197371 (https://phabricator.wikimedia.org/T63594) (owner: 10Krinkle) [17:51:57] !log Reloading Zuul to deploy I206c81fe9bb88feda6 [17:52:01] Logged the message, Master [17:55:17] 10Continuous-Integration, 7Technical-Debt, 7Tracking: All repositories should pass jshint test (tracking) - https://phabricator.wikimedia.org/T62619#1125993 (10Krinkle) [18:30:13] (03PS1) 10Krinkle: fab: Pause to let user "!log" the Zuul reload [integration/config] - 10https://gerrit.wikimedia.org/r/197380 [18:31:20] (03CR) 10jenkins-bot: [V: 04-1] fab: Pause to let user "!log" the Zuul reload [integration/config] - 10https://gerrit.wikimedia.org/r/197380 (owner: 10Krinkle) [18:31:32] Krinkle: hmm, my future plan was to have it auto !log it with something like scap+logmsgbot [18:32:00] Well, I'm working with the present situation [18:32:09] And can see how this will make people forget logging [18:32:20] (03CR) 10Legoktm: fab: Pause to let user "!log" the Zuul reload (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/197380 (owner: 10Krinkle) [18:32:34] mhm [18:32:56] (03PS2) 10Krinkle: fab: Pause to let user "!log" the Zuul reload [integration/config] - 10https://gerrit.wikimedia.org/r/197380 [18:33:11] (03CR) 10Krinkle: "I blame Sublime for failing to detect the spaces." [integration/config] - 10https://gerrit.wikimedia.org/r/197380 (owner: 10Krinkle) [18:33:34] legoktm: I wouldn't know how to break the line in that case. [18:33:40] Python skils [18:33:54] do you want me to amend? [18:33:57] But yeah, will fix [18:34:02] Nah, I'll figure it out [18:34:05] (03CR) 10jenkins-bot: [V: 04-1] fab: Pause to let user "!log" the Zuul reload [integration/config] - 10https://gerrit.wikimedia.org/r/197380 (owner: 10Krinkle) [18:35:17] (03PS3) 10Krinkle: fab: Pause to let user "!log" the Zuul reload [integration/config] - 10https://gerrit.wikimedia.org/r/197380 [18:35:43] YuviPanda: how hard would it be to set up a version of https://github.com/wikimedia/operations-puppet/blob/21c72942dd7bf25dbe0759d2f867082e966bfb45/manifests/role/tcpircbot.pp for -releng? [18:35:59] YuviPanda: that listens to gallium [18:36:27] legoktm: shouldn’t be too hard. a ferm rule + tcpircbot::instance call seems enough [18:36:28] (03CR) 10jenkins-bot: [V: 04-1] fab: Pause to let user "!log" the Zuul reload [integration/config] - 10https://gerrit.wikimedia.org/r/197380 (owner: 10Krinkle) [18:36:40] * Krinkle runs tox locally now [18:41:13] legoktm: What? Continuation has to align with the opening parenthesis at the end of the previous line? ... [18:41:23] Insert 50 spaces [18:41:38] and of course hit the line limit [18:41:57] https://www.python.org/dev/peps/pep-0008/#indentation [18:42:13] (03PS4) 10Krinkle: fab: Pause to let user "!log" the Zuul reload [integration/config] - 10https://gerrit.wikimedia.org/r/197380 [18:43:21] Yeah, starting the string on the next line fixes it [18:43:36] thcipriani: twentyafterfour so I’m lost now, with staging-tin [18:43:46] A copy of your installation's LocalSettings.php [18:43:46] must exist and be readable in the source directory. [18:43:46] Use --conf to specify it. [18:44:38] YuviPanda: where are you reading that? [18:44:47] thcipriani: if I try running scap from staging-tin [18:44:48] YuviPanda: does gallium not have IPv6? [18:44:53] it should try to scap to itself [18:44:59] legoktm: uh, I’m not sure? [18:45:07] it might not [18:45:39] thcipriani: there also needs to be a PrivateSettings.php setup... [18:45:42] I’ve created an empty file [18:45:52] I guess PrivateSettings.php has to be manual for now [18:45:57] copy-pasta the one from deployment-prep [18:45:59] I’m trying to automate the other stuff though. [18:46:01] it is manual [18:46:10] yeah [18:46:20] mediawiki-staging cloning made automatic seems ok. I’m not sure about php-master cloning [18:46:49] it would be beta specific [18:46:57] and you hate that :) [18:47:03] legoktm: there is no map for it in puppet so probably not [18:47:33] bd808: :D I guess I’ll have to write a small ‘scap bootstrap’ script... [18:47:38] that does things like these. [18:47:53] please name it "robo-reedy" [18:48:00] sounds appropriate [18:48:18] bd808: and re: source ordering, I don’t think it was needed... [18:48:32] cool. I didn't read to see just wondered [18:49:12] YuviPanda: https://gerrit.wikimedia.org/r/#/c/197386/ I have no idea if that's right... [18:51:01] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #376: FAILURE in 43 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/376/ [18:51:26] bd808: :) did you see the new ENC? [18:51:59] https://github.com/wikimedia/operations-puppet/blob/production/nodes/labs/staging.yaml [18:52:03] BeCaaS! [18:53:40] 12 Invalid argument: function: not string, closure, or array in /srv/mediawiki/php-1.25wmf21/includes/TemplateParser.php on line 203 [18:53:40] 9 error: syntax error, unexpected T_STRING in /srv/mediawiki/php-1.25wmf21/includes/TemplateParser.php(136) : eval()'d code on line [18:53:49] these are still happening :/ [18:53:53] thcipriani: so I suspect mwscript to not work... [18:54:09] * YuviPanda copies PrivateSettings from deployment-bastion [18:55:22] legoktm: if you want, gallium can get IPv6 :) [18:56:02] JohnFLewis: nah, I was just trying to figure out if I needed to whitelist it [18:56:13] Ah, alright [19:11:56] bd808: Roan says someone "needs to run clearMessageBlobs.php on Beta Cluster". [19:11:58] (And has gone to lunch.) [19:12:00] bd808: (To fix the i18n issue.) [19:13:56] This sounds like it's one of those little scripts that can deeply damage things. :-) [19:14:20] What could possibly go wrong [19:14:25] er [19:14:31] Indeed. [19:14:39] Where is that file anyway? [19:14:42] legoktm: Feel like a challenge? [19:14:49] Krenair: WikimediaMaintenance. [19:14:55] we don't run that script [19:15:03] "We"? [19:15:05] use refreshMessageBlobs.php [19:15:08] we == us [19:15:32] clear will just empty the cache causing a stampede [19:15:35] OK, can you run that one? [19:15:41] * James_F nods. [19:15:59] what is this i18n issue? [19:16:18] probably shouldn't be me, I'm already running SULF scripts [19:16:41] * James_F nods. [19:17:11] Krenair: http://en.wikipedia.beta.wmflabs.org/wiki/MediaWiki:Citoid-citeFromIDDialog-use-general-dialog-message exists but Varnish has cached that it doesn't. [19:17:28] Krenair: (And several other i18n strings from Citoid, and probably others.) [19:17:37] Multiple scaps haven't fixed it. [19:17:43] I browsed to that page and it showed [19:18:19] but it's still not there... [19:18:20] hm [19:18:34] (in the VE UI I mean) [19:18:49] Yeah. [19:19:38] hope this isn't "dependencies " all over again :D [19:19:49] :<<<<<<<< [19:20:19] Krenair: I'm sure legoktm won't ever make that mistake again after all that grief. :-) [19:20:58] I ran it but nothing happened [19:21:43] Hmm. [19:22:55] James_F, now? [19:23:03] Krenair: http://bits.beta.wmflabs.org/en.wikipedia.beta.wmflabs.org/load.php?modules=ext.citoid.visualEditor&debug=true is looking good. [19:23:23] Jenkins is scapping so it probably just needed that to fix it [19:23:31] It's fixed. [19:23:32] Thanks! [19:23:45] that was just on enwiki [19:23:53] let me know if we need it for other wikis as well [19:24:09] We've not configured it for other wikis. [19:24:19] By the time we do, it'll likely be fixed elsewise. [19:28:35] 10Staging: Create staging-elastic* (ElasticSearch machines) - https://phabricator.wikimedia.org/T91552#1126161 (10demon) `staging-elastic0[1-4]` are setup now, based on the work in [[ https://gerrit.wikimedia.org/r/#/c/196640/ | gerrit 196640 ]]. Waiting to resolve until that's merged but this is basically done.... [19:28:59] <^d> thcipriani, YuviPanda: ^^ [19:29:18] legoktm: do you remember what we did to fix your mw-vagrant setup issue the other day? [19:29:27] ^d: nice. [19:29:50] marxarelli: use the rpm from vagrantup.com instead of fedora's (fedora since fixed their rpm so I'm using it now) [19:29:55] legoktm: something was wrong with the ubuntu package iirc, but i don't remember how you resolved it [19:30:45] legoktm: ah, that's right. cool, thanks [19:31:06] i think i'm having the same issue with the ubuntu 14.10 package [19:37:03] ^d: sweeet:) [19:37:20] ^d: want to take a look at mwscript on staging-tin? :) run scap and see it error out... [19:38:24] <^d> Lemme finish lunch and then yeah [20:05:57] (03CR) 10Hashar: "I myself never bother logging Zuul reloads. I assume that as soon as a patch impacting Zuul is merged it gets deployed :)" [integration/config] - 10https://gerrit.wikimedia.org/r/197380 (owner: 10Krinkle) [20:11:34] (03CR) 10Krinkle: "And it's more about the reload than the deployment of a change. It communicates to team members that the queue will be stalled for a while" [integration/config] - 10https://gerrit.wikimedia.org/r/197380 (owner: 10Krinkle) [20:21:04] 10Continuous-Integration, 10Wikimedia-Hackathon-2015: All new extensions should be setup automatically with Zuul - https://phabricator.wikimedia.org/T92909#1126388 (10hashar) For the hackathon I I will probably spend most of the week-end helping / training volunteers + catching up with people. I am usually no... [20:21:26] 10Continuous-Integration, 10Wikimedia-Hackathon-2015: All new extensions should be setup automatically with Zuul - https://phabricator.wikimedia.org/T92909#1126390 (10hashar) p:5Triage>3Low [20:24:26] (03CR) 10Legoktm: [C: 031] fab: Pause to let user "!log" the Zuul reload [integration/config] - 10https://gerrit.wikimedia.org/r/197380 (owner: 10Krinkle) [20:35:03] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-salt is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:40:29] RECOVERY - Puppet staleness on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:43:52] hashar: https://gerrit.wikimedia.org/r/#/c/197405/ [20:44:10] any idea why that isn't passing jenkins ? [20:44:29] * hashar look for the standard text to copy paste [20:44:45] Please rebase your change and upload a new patchset. [20:44:51] but it's on the head of the remote repo [20:45:06] ah that is annoying. Well Jenkins/Zuul seems to be working properly as far as I know. It seems the patch has some bad code that cause a random test to fail it would need a bit more investigation on your side. [20:45:09] :D [20:45:12] looking [20:45:31] thedj: seems something is wrong on zuul/gerrit side [20:47:21] GitCommandError: 'git remote update origin' returned exit status 1: fatal: internal server error [20:47:21] remote: internal server error [20:47:21] fatal: protocol error: bad pack header [20:47:22] error: Could not fetch origin [20:47:26] thedj: poor Gerrit [20:47:34] yikes [20:48:23] thedj: explained it on https://gerrit.wikimedia.org/r/#/c/197405/ [20:48:23] omg they killed Gerrit [20:48:31] in short: recheck [20:49:03] thedj: when a patch is received, Zuul attempts to merge it against the tip of the branch. Thus it needs to do a git remote update && git fetch && git merge [20:49:09] then the result is being tested by Jenkins [20:49:27] physikerwelt: got that ? [20:49:40] it seems to happen from time to time [20:50:07] what does Commenting 'recheck' mean? [20:50:21] do I have to put recheck in the commit message?\ [20:50:45] hashar: what’s the name of the integration puppetmaster? [20:51:44] guess ? [20:51:50] integration-puppetmaster.eqiad.wmflabs [20:51:54] !!!! [20:52:05] hashar: :D ok [20:52:54] hashar: I just landed a patch that should make life of people hosting their own puppetmasters much easier. [20:53:01] thedj: I see https://phabricator.wikimedia.org/T66015 [20:53:11] YuviPanda: self signing + autoupdate ? [20:53:11] hashar: autosigning for puppet / salt ceritificates, and default setting of puppetmasters for all new instances in a project... [20:54:02] hashar: recheck lead to the same problem [20:54:12] hashar: yeah, autoupdate as well. also, you can now specify roles for individual nodes b ased on regex checks of the hostnames, no need to use wikitech [20:54:32] hashar: see https://github.com/wikimedia/operations-puppet/blob/production/nodes/labs/staging.yaml for example [21:03:01] physikerwelt: hmm, seems it's truly down for some reason [21:04:21] strange, since there has been a VE merge in this timeframe [21:06:25] physikerwelt: can you 'git remote update origin' yourself ? [21:08:21] yes [21:09:12] ^d: thcipriani so I’m trying to beat some sense into sca01, and fix git-deploy on the way. I am writing a ‘bootstrap git-deploy’ script as well [21:09:34] I even my local copy and created a new one [21:09:58] then i'm out of ideas... [21:10:15] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #371: FAILURE in 31 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/371/ [21:11:42] thedj: Don't worry. I can just ignore Jenkins for now [21:11:54] YuviPanda: beauty. I started digging into app servers: I'm way down a rabbit hole there, but that's just me getting more base understanding rather than there actually being any kind of show-stoppers there. I think -mx is done after this merges: https://gerrit.wikimedia.org/r/#/c/196658/ [21:13:18] thcipriani: sweet :) I think appservers will be intricately tied up between appservers, tin and db, and memcached, and redis... [21:13:23] andd vvvaarnish…. [21:13:27] maybe not varnish [21:13:28] but still [21:15:00] yup, appservers seem to be thoroughly intertwined with the other machines :) [21:15:20] yeah [21:15:39] YuviPanda: if there's anything _less_ intertwined that we should get to first, I'd gladly take a look at that [21:15:56] bbaaaam, I think sca01 is done now. [21:16:10] thcipriani: nope, I think dbs, redis, memcached all done... [21:16:45] 10Continuous-Integration, 6operations, 7Blocked-on-Operations, 3Continuous-Integration-Isolation, and 2 others: Create a Debian package for Zuul - https://phabricator.wikimedia.org/T48552#1126582 (10hashar) Zuul packaging as been discussed during the weekly ops meeting on 03/16. Andrew B. relayed the info... [21:16:52] YuviPanda: kk, I'll keep on keepin' on [21:17:31] thcipriani: cool. scap is still nonfunctional, though. git-deploy is fine. [21:18:52] * bd808 should remove his scap ping some day [21:18:53] 10Staging, 5Patch-For-Review: Setup staging-tin as deployment host - https://phabricator.wikimedia.org/T88442#1126586 (10yuvipanda) P409 is the 'bootstrap' script for git-deploy, should be run manually once on the deployment server (tin) [21:19:10] * YuviPanda gives bd808 hugs and <3 [21:19:16] but what will the rest of us dooo? :) [21:19:39] the "vagrant" ping is much much noisier [21:20:00] 10Staging, 5Patch-For-Review: Setup staging-tin as deployment host - https://phabricator.wikimedia.org/T88442#1126588 (10yuvipanda) Note that ^ is a total hack :D should be fixed in git-deploy itself... at some point... by someone... [21:20:48] YuviPanda: omg. that script is the worst! ;) [21:21:03] bd808: yup. [21:21:34] very ‘let us turn this off and on and put some pink paint in there and see if it works' [21:25:36] bd808: your reminder a week ago about some yaks not needing shaving was very useful... [21:26:10] It's a horrible temptation [21:26:28] But if you want to become the new knower of all things trebuchet... [21:27:08] yeah, I figured that if I start down that path, I’ll be there for quite a while... [21:28:02] I don't even know if we are tracking Ryan's upstream at this point :/ [21:28:16] is upstream being maintained? [21:28:43] ... not very active [21:28:55] but maybe it's perfect! [21:29:31] YuviPanda: thanks for reminding me i need to shave.. [21:29:47] * YuviPanda moves bd808 under a bridge [21:30:25] thcipriani: I’m going to get rid of hiera_include now [21:30:47] kk [21:32:35] thcipriani: re: bootstrap scripts, I wonder if we should just put them in the puppet repo as well, and install them in /usr/local/sbin? [21:33:06] YuviPanda: they should definitely be put in _a_ repo [21:33:10] yeah [21:37:17] thcipriani: we should also just stop using wikitech and use the yaml file to specify roles :) [21:37:28] thcipriani: also, thoughts on ‘mediawiki01’ vs ‘mediawiki1’ as hostnames? [21:37:33] with staging- prefix, of course [21:37:47] ^d: ^ as well [21:38:37] mw1001.eqiad.wmstage ! [21:38:57] YuviPanda: I noticed you've been doing \d\d in t he regex. I'm fine with that, I can re-spin-up db1 as db01. I usually end up regretting not using 01 when it comes time to sort stuffs. [21:39:07] thcipriani: yeah, I agree. [21:39:38] thcipriani: plus, ideally, for things that don’t have any data, it should be as trivial as deleting + recreating and then just… waiting [21:39:55] thcipriani: and it will also help us test our bootstrap scripts :D [21:40:47] !log deleted staging-sca01 because why not :) [21:40:50] Logged the message, Master [21:41:30] YuviPanda: that's definitely the positive way of looking at it :) [21:42:43] !log recreated staging-sca01, let’s wait and see if it just automagically configures itself :) [21:42:45] Logged the message, Master [21:50:49] thcipriani: you should commit your additions to staging.yaml :) having them be there makes updating puppet code tough... [21:52:16] heh, sorry. Just made those when you said you were ditching hiera_include. Doing now. [21:52:26] thcipriani: :) np. [21:52:41] 10Deployment-Systems, 6Release-Engineering: Clean up erroneously created wmf/1.20wmf21 branches - https://phabricator.wikimedia.org/T92501#1126676 (10Krinkle) [21:52:59] thcipriani: we should probably also turn on auto updating for puppet there. except that if we do, we can’t make local uncommited changes there - update will clean those up. [21:55:26] YuviPanda: staging.yaml updated [21:55:39] sweet [21:56:00] YuviPanda: I say leave off autoupdating until we're less in flux [21:56:19] yeah, makes sense [21:56:28] I will undoubtedly forget about it, be confused, and be mad :) [21:58:20] :D [22:09:20] thcipriani|afk: but… I just recreated sca01, and everything just magically works :D \o/ [22:12:13] YuviPanda: awesome. [22:17:11] thcipriani|afk: merged your mail::mx role :) am off to sleep now. night [22:17:30] YuviPanda: saw that, thanks! [22:17:37] * YuviPanda waves [22:29:31] 10Continuous-Integration, 6translatewiki.net, 5Patch-For-Review: l10n-bot self-force-merging sometimes breaks mediawiki/core master - https://phabricator.wikimedia.org/T91707#1126749 (10Raymond) @hashar I will prepare tomorrow (Wednesday) such a patch set. [22:43:21] Request URL: http://bits.beta.wmflabs.org/static-master/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.js [22:43:22] Request Method: GET [22:43:22] Status Code: 403 HTTP method not allowed. [22:43:26] Why does this keep happening? [22:52:21] 10Beta-Cluster: Occasionally getting 403 HTTP Method not allowed from bits - https://phabricator.wikimedia.org/T93021#1126789 (10Krenair) 3NEW [23:51:28] (03CR) 10Alex Monk: "This broke things during the deployment today." [tools/scap] - 10https://gerrit.wikimedia.org/r/196306 (https://phabricator.wikimedia.org/T92534) (owner: 10Legoktm) [23:56:11] PROBLEM - SSH on deployment-salt is CRITICAL: CRITICAL - Socket timeout after 10 seconds