[04:45:59] 10Continuous-Integration-Config, 10Pywikibot-i18n, 7I18n, 5Patch-For-Review: Jenkins job to validate JSON files submitted to Gerrit repo pywikibot/i18n - https://phabricator.wikimedia.org/T85335#1588381 (10jayvdb) 5Open>3Resolved Thanks @Legoktm for helping finish this task! [05:35:23] Yippee, build fixed! [05:35:24] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #527: FIXED in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/527/ [06:59:59] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #555: FAILURE in 34 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/555/ [07:57:07] 10Continuous-Integration-Config, 10Pywikibot-i18n, 7I18n, 5Patch-For-Review: Jenkins job to validate JSON files submitted to Gerrit repo pywikibot/i18n - https://phabricator.wikimedia.org/T85335#1588497 (10hashar) Summary GruntJS support globbing patterns http://gruntjs.com/configuring-tasks#globbing-patt... [08:04:31] 10Deployment-Systems, 6Release-Engineering, 6operations: SCAP fails with Permission denied (publickey) - https://phabricator.wikimedia.org/T110794#1588501 (10hashar) Seems a root has to arm it with: ``` sudo -u keyholder env SSH_AUTH_SOCK=/run/keyholder/agent.sock ssh-add /etc/keyholder.d/mwdeploy_rsa ``` [08:15:22] 6Release-Engineering, 10Gitblit-Deprecate: Update mediawiki.org templates to link to Diffusion, not gitblit - https://phabricator.wikimedia.org/T108864#1588528 (10Spage) [08:16:47] 10Deployment-Systems, 6Release-Engineering, 6operations: SCAP fails with Permission denied (publickey) - https://phabricator.wikimedia.org/T110794#1588537 (10jcrespo) If that is true, it should be documented at https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment#In_your_own_repo_via_gerrit and https... [08:22:39] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Isolation, 7Tracking: tracking: Cleanup contint puppet manifests so they are easier to reuse / split slaves by functions - https://phabricator.wikimedia.org/T110864#1588551 (10hashar) 3NEW [08:23:20] 10Deployment-Systems, 6Release-Engineering, 6operations, 5Patch-For-Review: SCAP fails with Permission denied (publickey) - https://phabricator.wikimedia.org/T110794#1588558 (10ori) 5Open>3Resolved a:3ori @jcrespo, I'll update the docs. [08:25:57] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Isolation: Factor out ruby stuff from contint puppet manifests - https://phabricator.wikimedia.org/T110865#1588563 (10hashar) 3NEW a:3zeljkofilipin [08:26:08] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Isolation: Factor out ruby stuff from contint puppet manifests - https://phabricator.wikimedia.org/T110865#1588563 (10hashar) p:5Triage>3Normal [08:26:30] zeljkof-meeting: I created https://phabricator.wikimedia.org/T110865 "Factor out ruby stuff from contint puppet manifests". Assigned it to you and marked it as in progress [08:26:39] zeljkof-meeting: we can talk about it this afternoon :-} [08:28:41] hashar: thanks, will take a look in a few minutes [08:43:50] Project beta-scap-eqiad build #67973: FAILURE in 28 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/67973/ [08:51:04] hashar: sorry about running away from the meeting, had to do some emergency babysitting :) I am back now [08:51:25] I understand :-D [08:51:40] zeljkof: I filled the task about refactoring the contint puppet class to extract ruby bits [08:51:59] hashar: thanks, saw it, will start working on it today [08:52:24] just have to check first if I need to finish stuff from last week, I think I had some minor things to finish up [09:12:25] (03PS8) 10Hashar: Disk image building [integration/config] - 10https://gerrit.wikimedia.org/r/234300 (https://phabricator.wikimedia.org/T110733) [09:12:39] (03CR) 10Hashar: "Rewritten to shell" [integration/config] - 10https://gerrit.wikimedia.org/r/234300 (https://phabricator.wikimedia.org/T110733) (owner: 10Hashar) [09:24:04] (03PS9) 10Hashar: Disk image building [integration/config] - 10https://gerrit.wikimedia.org/r/234300 (https://phabricator.wikimedia.org/T110733) [09:25:19] (03PS10) 10Hashar: Disk image building [integration/config] - 10https://gerrit.wikimedia.org/r/234300 (https://phabricator.wikimedia.org/T110733) [09:43:15] Yippee, build fixed! [09:43:15] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #596: FIXED in 6 min 14 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/596/ [09:57:26] 10Deployment-Systems, 6operations, 5Patch-For-Review: [Trebuchet] Salt times out on parsoid restarts - https://phabricator.wikimedia.org/T63882#1588760 (10mark) a:3ArielGlenn [10:01:10] 10Deployment-Systems, 6operations, 5Patch-For-Review: [Trebuchet] Salt times out on parsoid restarts - https://phabricator.wikimedia.org/T63882#1588768 (10ArielGlenn) I chatted with akosiaris about this, I had not been able to get anything useful after digging around but I will have another go at it this week. [11:00:38] (03PS1) 10Hashar: (WIP) dib: wikimedia-puppet element (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/234975 [11:17:29] 5Continuous-Integration-Isolation: Migrate dib packages installation to package-installs.sh - https://phabricator.wikimedia.org/T110880#1588926 (10hashar) 3NEW [11:59:16] Yippee, build fixed! [11:59:16] Project browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #219: FIXED in 2 min 15 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/219/ [12:01:14] (03PS2) 10Hashar: (WIP) dib: wikimedia-puppet element (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/234975 [12:01:16] (03PS11) 10Hashar: Disk image building [integration/config] - 10https://gerrit.wikimedia.org/r/234300 (https://phabricator.wikimedia.org/T110733) [12:06:26] (03PS3) 10Hashar: (WIP) dib: wikimedia-puppet element (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/234975 [12:06:28] (03PS12) 10Hashar: Disk image building [integration/config] - 10https://gerrit.wikimedia.org/r/234300 (https://phabricator.wikimedia.org/T110733) [12:12:19] yeah puppet running!! [12:12:22] Notice: /Stage[main]/Packages::Python3_tk/Package[python3-tk]/ensure: ensure changed 'purged' to 'present' [13:41:15] (03PS4) 10Hashar: (WIP) dib: wikimedia-puppet element (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/234975 [13:41:17] (03PS13) 10Hashar: Disk image building [integration/config] - 10https://gerrit.wikimedia.org/r/234300 (https://phabricator.wikimedia.org/T110733) [13:41:19] (03PS1) 10Hashar: dib: bump to 1.0.0 and remove wikimedia-networking [integration/config] - 10https://gerrit.wikimedia.org/r/234994 (https://phabricator.wikimedia.org/T105152) [13:41:36] (03CR) 10Hashar: [C: 032] "Upstream merged my patch https://review.openstack.org/#/c/200030/ :-}" [integration/config] - 10https://gerrit.wikimedia.org/r/234994 (https://phabricator.wikimedia.org/T105152) (owner: 10Hashar) [14:17:47] 10Continuous-Integration-Config, 5Patch-For-Review, 7Puppet: Setup rubocop for operations/puppet ruby code lints - https://phabricator.wikimedia.org/T102020#1589214 (10zeljkofilipin) ``` $ git submodule status -76a5e8627659197f81a4e5240c3eb4fe01cb9888 modules/cdh -e4c66b04b3f4df82569f4d14d0cf74b6fb79e57d mod... [14:24:34] (03Abandoned) 10Hashar: dib: bump to 1.0.0 and remove wikimedia-networking [integration/config] - 10https://gerrit.wikimedia.org/r/234994 (https://phabricator.wikimedia.org/T105152) (owner: 10Hashar) [14:29:39] 10Continuous-Integration-Config, 5Patch-For-Review, 7Puppet: Setup rubocop for operations/puppet ruby code lints - https://phabricator.wikimedia.org/T102020#1589266 (10hashar) >>! In T102020#1589214, @zeljkofilipin wrote: > ``` > $ git submodule status > -76a5e8627659197f81a4e5240c3eb4fe01cb9888 modules/cdh... [15:08:12] (03PS1) 10Krinkle: Fix "curl: Failed writing body" [integration/config] - 10https://gerrit.wikimedia.org/r/235013 [15:10:08] 10Continuous-Integration-Config, 5Patch-For-Review, 7Puppet: Setup rubocop for operations/puppet ruby code lints - https://phabricator.wikimedia.org/T102020#1589373 (10greg) >>! In T102020#1589266, @hashar wrote: > Maybe the job should be made to not process submodules? Logically that makes sense, and shoul... [15:12:21] blugh https://integration.wikimedia.org/ci/job/beta-scap-eqiad/ [15:13:04] !log did https://phabricator.wikimedia.org/T109007#1537572 [15:13:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:13:20] lets see if that helped... [15:13:50] (03CR) 10Jforrester: "Neat hack." [integration/config] - 10https://gerrit.wikimedia.org/r/235013 (owner: 10Krinkle) [15:14:08] (03PS1) 10Krinkle: localhost: Fix double slash in 'curl' test run [integration/config] - 10https://gerrit.wikimedia.org/r/235014 [15:15:47] jzerebecki: ty, it probably will, see also the same issue in prod: https://phabricator.wikimedia.org/T110794 [15:15:50] :) [15:16:20] (03PS2) 10Krinkle: localhost: Fix double slash in 'curl' test run [integration/config] - 10https://gerrit.wikimedia.org/r/235014 [15:18:12] (03PS3) 10Krinkle: localhost: Fix double slash in 'curl' test run [integration/config] - 10https://gerrit.wikimedia.org/r/235014 [15:18:14] (03PS1) 10Krinkle: localhost: Resolve redirect for Special:BlankPage [integration/config] - 10https://gerrit.wikimedia.org/r/235015 [15:25:33] 10Beta-Cluster: New Accounts (Confirmed) Not Allowed to Use Twinkle - https://phabricator.wikimedia.org/T110897#1589421 (10Skyllfully) 3NEW [15:26:57] Yippee, build fixed! [15:26:58] Project beta-scap-eqiad build #68010: FIXED in 22 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/68010/ [15:27:24] jzerebecki: ^ :) :) [15:28:08] (03CR) 10Krinkle: [C: 032] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/235013 (owner: 10Krinkle) [15:28:16] (03CR) 10Krinkle: [C: 032] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/235014 (owner: 10Krinkle) [15:28:24] (03CR) 10Krinkle: [C: 032] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/235015 (owner: 10Krinkle) [15:29:18] greg-g: any idea what needs to be done so we don't need to do that every day? [15:29:49] * greg-g sighs [15:29:53] (03Merged) 10jenkins-bot: Fix "curl: Failed writing body" [integration/config] - 10https://gerrit.wikimedia.org/r/235013 (owner: 10Krinkle) [15:30:07] (03Merged) 10jenkins-bot: localhost: Fix double slash in 'curl' test run [integration/config] - 10https://gerrit.wikimedia.org/r/235014 (owner: 10Krinkle) [15:30:12] (03Merged) 10jenkins-bot: localhost: Resolve redirect for Special:BlankPage [integration/config] - 10https://gerrit.wikimedia.org/r/235015 (owner: 10Krinkle) [15:35:01] 10Continuous-Integration-Config: JJB qunit macro ignores curl error exit code - https://phabricator.wikimedia.org/T99854#1589468 (10JanZerebecki) Might have been fixed by: https://gerrit.wikimedia.org/r/#/c/235013/ [15:40:21] jzerebecki: should only be needed when that server restarts or when keyholder-agent restarts [15:48:15] 10Browser-Tests: Empty failing tests - not sure on action - https://phabricator.wikimedia.org/T110898#1589495 (10Jdlrobson) 3NEW [16:08:47] (03PS5) 10Hashar: (WIP) dib: wikimedia-puppet element (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/234975 [16:08:49] (03PS14) 10Hashar: Disk image building [integration/config] - 10https://gerrit.wikimedia.org/r/234300 (https://phabricator.wikimedia.org/T110733) [16:10:01] 10Deployment-Systems, 6Release-Engineering: Don't continue scap if sync to all proxies failed - https://phabricator.wikimedia.org/T110791#1589568 (10greg) Or at any stage, really? (sync-proxies, sync-apaches, scap-rebuild-cdbs, and sync_wikiversions) [16:11:02] 10Beta-Cluster, 10ContentTranslation-Deployments, 10Wikimedia-Site-Requests, 5Patch-For-Review: Put beta eswiki to read-only mode - https://phabricator.wikimedia.org/T109157#1589570 (10Krenair) Doesn't seem to have taken effect. The closed tag is applied, but it doesn't get the right settings for closed wi... [16:57:33] so...why are slaves randomly going down? [16:57:39] Host DOWN alert for integration-slave-trusty-1017! [16:58:04] I don't see it in jenkins either... [17:04:28] 10Beta-Cluster, 10ContentTranslation-Deployments, 10Wikimedia-Site-Requests, 5Patch-For-Review: Put beta eswiki to read-only mode - https://phabricator.wikimedia.org/T109157#1589694 (10Luke081515) p:5Normal>3Low I added so .* to the titleblacklist, so it is not important to do this lock fast. [17:08:34] 10Beta-Cluster, 5Patch-For-Review: Beta cluster not updating MW code - https://phabricator.wikimedia.org/T109007#1589708 (10JanZerebecki) Added the location of the passphrase to the keyholder docs: https://wikitech.wikimedia.org/w/index.php?title=Keyholder&type=revision&diff=175639&oldid=175543 [17:31:01] 10Beta-Cluster, 10MediaWiki-extensions-TimedMediaHandler, 6Multimedia, 10Wikimedia-Video: Videoscalers (tmh*) not working in Beta Cluster - https://phabricator.wikimedia.org/T90332#1589795 (10brion) 5Open>3Resolved a:3brion We got this working on Friday. \o/ [17:32:40] 10Beta-Cluster, 10MediaWiki-extensions-TimedMediaHandler, 6Multimedia, 10Wikimedia-Video: Videoscalers (tmh*) not working in Beta Cluster - https://phabricator.wikimedia.org/T90332#1589806 (10Jdforrester-WMF) Woo. [17:35:44] 10Beta-Cluster, 7HHVM, 5Patch-For-Review: Upgrade Beta Cluster tmh* host(s) to HHVM and Trusty - https://phabricator.wikimedia.org/T110707#1589827 (10greg) Just to confirm, the upgrade to HHVM and Trusty is done, yes? If so, let's close this task and you can work out your config/code tweaking as you like :) [17:48:32] (03PS8) 10Paladox: Update ConfirmAccount tests [integration/config] - 10https://gerrit.wikimedia.org/r/225311 [17:53:21] 10Beta-Cluster, 7HHVM, 5Patch-For-Review: Upgrade Beta Cluster tmh* host(s) to HHVM and Trusty - https://phabricator.wikimedia.org/T110707#1589898 (10brion) 5Open>3Resolved Yep! I'll file a separate task for the job runner count issue. [17:56:18] 10Beta-Cluster, 10MediaWiki-JobRunner: Video transcode job runner on beta cluster runs 5 jobs even though configured for 2 - https://phabricator.wikimedia.org/T110916#1589911 (10brion) 3NEW [18:41:54] 10Beta-Cluster, 7HHVM, 5Patch-For-Review: Upgrade Beta Cluster tmh* host(s) to HHVM and Trusty - https://phabricator.wikimedia.org/T110707#1590205 (10bd808) 5Resolved>3Open I'd like to keep this open until the ops/puppet patches are merged: * https://gerrit.wikimedia.org/r/#/c/234599/ * https://gerrit.wi... [18:58:58] 10Beta-Cluster, 7HHVM, 5Patch-For-Review: Upgrade Beta Cluster tmh* host(s) to HHVM and Trusty - https://phabricator.wikimedia.org/T110707#1590273 (10greg) word, good point. [19:11:38] thcipriani: can't make it for tonight triage :/ [19:12:04] hashar: okie doke, no problem [19:12:23] I have just finished with kids :\ [19:43:36] 5Continuous-Integration-Isolation, 6Labs, 10Labs-Infrastructure: Labs project admin can not create per project image on Horizon - https://phabricator.wikimedia.org/T110936#1590423 (10hashar) 3NEW [20:20:36] 10Continuous-Integration-Config, 6Labs, 10Tool-Labs: Job labs-toollabs-debian-glue is failing for labs/toollabs repository - https://phabricator.wikimedia.org/T110939#1590522 (10scfc) 3NEW [20:21:35] 10Continuous-Integration-Config, 6Labs, 10Tool-Labs: Job labs-toollabs-debian-glue is failing for labs/toollabs repository - https://phabricator.wikimedia.org/T110939#1590536 (10scfc) [20:52:26] 10Continuous-Integration-Infrastructure, 10Ops-Access-Requests, 6operations: Let contint-admins force run puppet with /usr/local/sbin/puppet-run - https://phabricator.wikimedia.org/T110943#1590605 (10hashar) 3NEW [20:55:46] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #751: FAILURE in 29 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/751/ [22:34:13] (03PS6) 10Hashar: (WIP) dib: wikimedia-puppet element (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/234975