[00:01:40] legoktm: good! [00:27:59] (03PS1) 10Dduvall: Support links to Sauce jobs in Raita logging [selenium] - 10https://gerrit.wikimedia.org/r/214274 [00:28:19] (03CR) 10jenkins-bot: [V: 04-1] Support links to Sauce jobs in Raita logging [selenium] - 10https://gerrit.wikimedia.org/r/214274 (owner: 10Dduvall) [00:30:17] (03PS2) 10Dduvall: Support links to Sauce jobs in Raita logging [selenium] - 10https://gerrit.wikimedia.org/r/214274 [01:43:40] so it looks like we repeat the deployment train tomorrow. our tuesday-wednesday-thursday cadence gets implemented ahead of schedule [02:04:45] 6Release-Engineering: make-release should run a syntax checker over the generated tarball - https://phabricator.wikimedia.org/T100599#1316420 (10Legoktm) 3NEW [02:07:06] 10Deployment-Systems, 6Release-Engineering: scap should lint JSON files - https://phabricator.wikimedia.org/T100600#1316432 (10Legoktm) 3NEW [02:09:30] 6Release-Engineering, 10MediaWiki-extensions-ConfirmEdit-(CAPTCHA-extension), 5MW-1.25-release, 5Patch-For-Review: [QuestyCaptcha+ReCaptcha] typo in extension.json files - https://phabricator.wikimedia.org/T100351#1316451 (10Legoktm) Follow up bugs to prevent this issue from happening again: * {T100599} *... [02:11:47] ^ never again [02:20:37] 10Deployment-Systems, 6Release-Engineering: scap should lint JSON files - https://phabricator.wikimedia.org/T100600#1316458 (10Legoktm) a:3Legoktm [02:25:20] Yippee, build fixed! [02:25:20] Project browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-linux-android-sauce build #114: FIXED in 3 min 19 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-linux-android-sauce/114/ [02:26:46] 10Deployment-Systems, 6Release-Engineering: scap should lint JSON files - https://phabricator.wikimedia.org/T100600#1316459 (10Legoktm) Hmm, I just remembered that we only lint wmf-config and multiversion on scap, and individual files or sync-dirs. Any reason not to lint everything? [02:36:36] (03PS1) 10Legoktm: Lint JSON files [tools/scap] - 10https://gerrit.wikimedia.org/r/214288 (https://phabricator.wikimedia.org/T100600) [03:15:11] 10Continuous-Integration-Infrastructure, 7Technical-Debt, 7Tracking: All repositories should pass jshint test (tracking) - https://phabricator.wikimedia.org/T62619#1316476 (10Legoktm) [03:17:06] (03PS1) 10Legoktm: Make jshint and jsonlint voting for ConfirmEdit [integration/config] - 10https://gerrit.wikimedia.org/r/214292 (https://phabricator.wikimedia.org/T63597) [03:17:20] (03CR) 10Legoktm: [C: 032] Make jshint and jsonlint voting for ConfirmEdit [integration/config] - 10https://gerrit.wikimedia.org/r/214292 (https://phabricator.wikimedia.org/T63597) (owner: 10Legoktm) [03:18:57] (03CR) 10jenkins-bot: [V: 04-1] Make jshint and jsonlint voting for ConfirmEdit [integration/config] - 10https://gerrit.wikimedia.org/r/214292 (https://phabricator.wikimedia.org/T63597) (owner: 10Legoktm) [03:19:25] erg wtf [03:25:27] (03PS2) 10Legoktm: Make jshint and jsonlint voting for ConfirmEdit [integration/config] - 10https://gerrit.wikimedia.org/r/214292 (https://phabricator.wikimedia.org/T63597) [03:25:29] (03PS1) 10Legoktm: Remove operations/puppet/varnish, deleted from Gerrit [integration/config] - 10https://gerrit.wikimedia.org/r/214293 (https://phabricator.wikimedia.org/T100602) [03:28:41] (03CR) 10Legoktm: [C: 032] Make jshint and jsonlint voting for ConfirmEdit [integration/config] - 10https://gerrit.wikimedia.org/r/214292 (https://phabricator.wikimedia.org/T63597) (owner: 10Legoktm) [03:28:46] (03CR) 10Legoktm: [C: 032] Remove operations/puppet/varnish, deleted from Gerrit [integration/config] - 10https://gerrit.wikimedia.org/r/214293 (https://phabricator.wikimedia.org/T100602) (owner: 10Legoktm) [03:30:34] (03Merged) 10jenkins-bot: Remove operations/puppet/varnish, deleted from Gerrit [integration/config] - 10https://gerrit.wikimedia.org/r/214293 (https://phabricator.wikimedia.org/T100602) (owner: 10Legoktm) [03:30:36] (03Merged) 10jenkins-bot: Make jshint and jsonlint voting for ConfirmEdit [integration/config] - 10https://gerrit.wikimedia.org/r/214292 (https://phabricator.wikimedia.org/T63597) (owner: 10Legoktm) [03:54:40] (03CR) 1020after4: [C: 031] Lint JSON files [tools/scap] - 10https://gerrit.wikimedia.org/r/214288 (https://phabricator.wikimedia.org/T100600) (owner: 10Legoktm) [03:58:03] 6Release-Engineering, 10Continuous-Integration-Config, 10Wikidata, 7Composer, 5Patch-For-Review: enable use of a composer created autoloader in extensions deployed to production - https://phabricator.wikimedia.org/T97560#1316536 (10Legoktm) 5Open>3Resolved a:3JeroenDeDauw [03:59:32] 10Deployment-Systems, 6Release-Engineering, 5Patch-For-Review: scap should lint JSON files - https://phabricator.wikimedia.org/T100600#1316557 (10bd808) >>! In T100600#1316459, @Legoktm wrote: > Hmm, I just remembered that we only lint wmf-config and multiversion on scap, and individual files or sync-dirs. A... [04:29:45] PROBLEM - Free space - all mounts on deployment-videoscaler01 is CRITICAL deployment-prep.deployment-videoscaler01.diskspace._var.byte_percentfree (<30.00%) [04:33:14] 6Release-Engineering, 7user-notice: Shorten/Simplify MW train deploy cadence to Tu->W->Th - https://phabricator.wikimedia.org/T97553#1316611 (10Glaisher) [06:30:50] Yippee, build fixed! [06:30:51] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #626: FIXED in 11 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/626/ [06:46:01] (03CR) 10Pastakhov: "Hashar: please, merge it." [integration/config] - 10https://gerrit.wikimedia.org/r/207754 (owner: 10Pastakhov) [06:54:43] RECOVERY - Free space - all mounts on deployment-videoscaler01 is OK All targets OK [07:19:58] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #456: FAILURE in 54 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/456/ [07:42:25] (03PS1) 10Polybuildr: Add tests for mediawiki/extensions/SmiteSpam [integration/config] - 10https://gerrit.wikimedia.org/r/214304 [07:42:51] Can someone please review that patch ^ and tell me if anything is missing. [07:43:32] The documentation (https://www.mediawiki.org/wiki/Continuous_integration/Tutorials/Adding_a_MediaWiki_extension) was difficult to understand and probably outdated, so I just guessed from other extensions' entries in layout.yaml [07:55:54] 6Release-Engineering, 6Project-Creators: SWAT Project (Tag) - https://phabricator.wikimedia.org/T99411#1316936 (10mmodell) ok this really seems quite usable. https://phabricator.wikimedia.org/releeph/branch/1/ [08:10:54] 10Browser-Tests: Enable Elena and Rummana to run Jenkins jobs - https://phabricator.wikimedia.org/T100172#1316961 (10hashar) 5Open>3Resolved Should be good now :-} [08:19:50] 6Release-Engineering, 10Wikimedia-Git-or-Gerrit, 5Patch-For-Review: https://github.com/wikimedia/mediawiki/ release tags vanished - https://phabricator.wikimedia.org/T100409#1316971 (10Foxtrott) > Also https://github.com/wikimedia/mediawiki/releases/tag/1.24.2 is working for me and showing the links to the a... [08:26:17] (03PS2) 10Polybuildr: Add tests for mediawiki/extensions/SmiteSpam [integration/config] - 10https://gerrit.wikimedia.org/r/214304 [08:26:53] 10Deployment-Systems, 6Release-Engineering, 5Patch-For-Review: scap should lint JSON files - https://phabricator.wikimedia.org/T100600#1316976 (10hashar) That last time check is a good idea but I would prefer CI to be enhanced to catch json issues on patch proposal and prevent them from being merged in. Sin... [08:27:58] (03CR) 10Polybuildr: "Not sure if I've done this right. What I want is for phplint and phpcs to run as `check`." [integration/config] - 10https://gerrit.wikimedia.org/r/214304 (owner: 10Polybuildr) [09:22:24] PROBLEM - Puppet failure on deployment-mx is CRITICAL 100.00% of data above the critical threshold [0.0] [09:43:52] 6Release-Engineering, 10Wikimedia-Git-or-Gerrit, 5Patch-For-Review: https://github.com/wikimedia/mediawiki/ release tags vanished - https://phabricator.wikimedia.org/T100409#1317059 (10hashar) Danke @qchris ! [09:44:40] Yippee, build fixed! [09:44:41] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #498: FIXED in 7 min 39 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/498/ [09:45:14] 6Release-Engineering, 10Continuous-Integration-Config, 10Wikidata, 7Composer, and 3 others: enable use of a composer created autoloader in extensions deployed to production - https://phabricator.wikimedia.org/T97560#1317062 (10hashar) @JanZerebecki seems that fix would unlock a lot of issue for Wikibase an... [09:58:21] (03CR) 10Hashar: "The root cause is the extensions are cloned by zuul-cloner using the 'master' branch since there is no 'fundraising/REL1_25' branch availa" [integration/config] - 10https://gerrit.wikimedia.org/r/212739 (https://phabricator.wikimedia.org/T94528) (owner: 10Awight) [10:01:13] 10Continuous-Integration-Infrastructure, 10MediaWiki-Codesniffer, 7HHVM: Upgrade Jenkins slave nodes to HHVM 3.5.0 or above - https://phabricator.wikimedia.org/T100544#1317094 (10Addshore) [10:04:18] (03CR) 10Addshore: [V: 031] "This is failing due to running on an old version of HHVM" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/196880 (owner: 10Polybuildr) [10:06:06] hashar: ^^ any idea of a timescale for the above update to HHVM? If its more than a week I will probably go and make the test non voting on the codesniffer repo [10:06:37] 10Continuous-Integration-Infrastructure, 10MediaWiki-Codesniffer, 7HHVM: Upgrade Jenkins slave nodes to HHVM 3.5.0 or above - https://phabricator.wikimedia.org/T100544#1317106 (10hashar) mediawiki/core tests broke when upgrading hhvm from `3.3.1+dfsg1-1+wm3.1` to `3.6.1+dfsg1-1+wm2` which is T98876. [10:06:44] 10Continuous-Integration-Infrastructure, 7HHVM, 5Patch-For-Review: "mediawiki-phpunit-hhvm" failures on all changes in mediawiki/core due to hhvm upgrade from 3.3.1+dfsg1-1+wm3.1 to 3.6.1+dfsg1-1+wm2 - https://phabricator.wikimedia.org/T98876#1279657 (10hashar) [10:06:45] 10Continuous-Integration-Infrastructure, 10MediaWiki-Codesniffer, 7HHVM: Upgrade Jenkins slave nodes to HHVM 3.5.0 or above - https://phabricator.wikimedia.org/T100544#1317108 (10hashar) [10:07:18] ahhh, lovely :P [10:08:19] 10Continuous-Integration-Infrastructure, 7HHVM, 5Patch-For-Review: "mediawiki-phpunit-hhvm" failures on all changes in mediawiki/core due to hhvm upgrade from 3.3.1+dfsg1-1+wm3.1 to 3.6.1+dfsg1-1+wm2 - https://phabricator.wikimedia.org/T98876#1279657 (10hashar) This is blocked on T98882 [10:11:18] (03PS1) 10Hashar: Use zuul source @debian/precise-wikimedia [integration/config] - 10https://gerrit.wikimedia.org/r/214319 [10:13:45] (03CR) 10Hashar: [C: 032] Use zuul source @debian/precise-wikimedia [integration/config] - 10https://gerrit.wikimedia.org/r/214319 (owner: 10Hashar) [10:15:30] (03Merged) 10jenkins-bot: Use zuul source @debian/precise-wikimedia [integration/config] - 10https://gerrit.wikimedia.org/r/214319 (owner: 10Hashar) [10:55:54] addshore: yeah hhvm is breaking :/ [11:32:27] (03PS3) 10Hashar: Can't run the new tests on the fundraising/REL1_25 branch, yet [integration/config] - 10https://gerrit.wikimedia.org/r/212739 (https://phabricator.wikimedia.org/T94528) (owner: 10Awight) [11:34:06] (03CR) 10Hashar: [C: 032] "I have enhanced the wildcard to ignore all fundraising/REL.* branches. We should use git-submodule instead to test whatever is tagged." [integration/config] - 10https://gerrit.wikimedia.org/r/212739 (https://phabricator.wikimedia.org/T94528) (owner: 10Awight) [11:36:02] (03PS4) 10Hashar: Can't run the new tests on the fundraising/REL1_25 branch, yet [integration/config] - 10https://gerrit.wikimedia.org/r/212739 (https://phabricator.wikimedia.org/T94528) (owner: 10Awight) [11:36:09] (03CR) 10Hashar: [C: 032] Can't run the new tests on the fundraising/REL1_25 branch, yet [integration/config] - 10https://gerrit.wikimedia.org/r/212739 (https://phabricator.wikimedia.org/T94528) (owner: 10Awight) [11:37:46] (03Merged) 10jenkins-bot: Can't run the new tests on the fundraising/REL1_25 branch, yet [integration/config] - 10https://gerrit.wikimedia.org/r/212739 (https://phabricator.wikimedia.org/T94528) (owner: 10Awight) [11:45:42] PROBLEM - Free space - all mounts on deployment-videoscaler01 is CRITICAL deployment-prep.deployment-videoscaler01.diskspace._var.byte_percentfree (<50.00%) [12:03:02] 10Continuous-Integration-Infrastructure, 6Collaboration-Team, 10Flow, 6Mobile-Web, 10VisualEditor: Create Jenkins builds for Editing across repositories (MobileFrontend, VisualEditor etc) - https://phabricator.wikimedia.org/T90647#1317335 (10hashar) [12:31:59] 10Continuous-Integration-Infrastructure, 7Regression: ERROR: Failed to notify endpoint 'HTTP:http://127.0.0.1:8001/jenkins_endpoint' - https://phabricator.wikimedia.org/T93321#1317362 (10hashar) I did a first upgrade from 4bf020e..c492c0b The next commit is 09bd39f which is https://review.openstack.org/#/c/1... [13:06:45] 6Release-Engineering, 6Project-Creators: SWAT Project (Tag) - https://phabricator.wikimedia.org/T99411#1317435 (10Krenair) (No one else can actually see that, application is restricted to administrators) [13:11:20] 10Continuous-Integration-Infrastructure, 7Regression: ERROR: Failed to notify endpoint 'HTTP:http://127.0.0.1:8001/jenkins_endpoint' - https://phabricator.wikimedia.org/T93321#1317457 (10hashar) There are a bunch of change that require us to bump our git plugin version. It is fairly updated and we should do it... [13:22:06] 6Release-Engineering, 10Hackathon-Mexico-City-2015: Hackathon Proposal: Wikimedia Site Requests Sprint - https://phabricator.wikimedia.org/T90468#1317493 (10Krenair) a:5greg>3Krenair After speaking to @greg and @qgil, I've decided to do this [13:23:42] 10Continuous-Integration-Infrastructure: Upgrade Jenkins git related plugins - https://phabricator.wikimedia.org/T100655#1317499 (10hashar) 3NEW [13:35:12] 10Continuous-Integration-Infrastructure: Jenkins: The GIT Plugin is broken for submodules ("git-submodule: git reset: not found") - https://phabricator.wikimedia.org/T73533#1317524 (10hashar) Apparently this is fixed upstream with: https://github.com/jenkinsci/git-client-plugin/commit/f75720d5de9d79ab4be2633a21d... [13:36:29] 10Continuous-Integration-Infrastructure: Upgrade Jenkins git related plugins - https://phabricator.wikimedia.org/T100655#1317525 (10hashar) [13:37:01] 6Release-Engineering, 10Wikimedia-Git-or-Gerrit, 5Patch-For-Review: https://github.com/wikimedia/mediawiki/ release tags vanished - https://phabricator.wikimedia.org/T100409#1317527 (10QChris) >>! In T100409#1316971, @Foxtrott wrote: > [...], but it does not show up in the selector on the github webpage. Pro... [13:40:00] hashar: around? [13:40:18] or anyone that is a github owner of wikimedia [13:40:45] 10Deployment-Systems, 6Release-Engineering: Make sync-wikiverisons check that a valid localisation cache exists when syncing new versions - https://phabricator.wikimedia.org/T100573#1317539 (10Reedy) I guess a file existence check of php-$version/cache/l10n/en.cdb (path from memory) is probably enough of a san... [13:40:56] * aude needs wikidata added as collaborator to https://github.com/wikimedia/mediawiki-extensions-WikibaseJavaScriptApi [13:41:46] aude: can you fill a task in phabricator against Wikimedia-Git-or-Gerrit please ? [13:41:48] updating it [13:41:52] ok [13:43:20] aude: so add wikidata team as owner of the github repo right ? [13:43:37] yes or collaborator [13:43:44] https://phabricator.wikimedia.org/T100658 [13:43:55] we have the rights for wikibase [13:43:56] aude: should be good now [13:43:59] thanks [13:44:12] looks good [13:44:52] thanks ! [13:45:35] 6Release-Engineering, 10Wikimedia-Git-or-Gerrit, 5Patch-For-Review: https://github.com/wikimedia/mediawiki/ release tags vanished - https://phabricator.wikimedia.org/T100409#1317571 (10JeroenDeDauw) Thanks for fixing this issue! [13:45:59] !log upgrading Jenkins git plugin from 1.4.6+wmf1 to 1.7.1 {{bug|T100655}} and restarting Jenkins [13:46:02] Logged the message, Master [13:46:03] which me luck [13:46:38] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #632: ABORTED in 15 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/632/ [14:02:10] 10Continuous-Integration-Infrastructure: Upgrade Jenkins git related plugins - https://phabricator.wikimedia.org/T100655#1317607 (10hashar) I have upgraded the git client from 1.4.6-wmf1 to 1.17.1 . mediawiki/core and VE jobs pass :-} [14:14:59] 10Continuous-Integration-Infrastructure: Jenkins: The GIT Plugin is broken for submodules ("git-submodule: git reset: not found") - https://phabricator.wikimedia.org/T73533#1317640 (10hashar) Plugin bumped to upstream 1.7.1 [14:32:35] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #633: STILL FAILING in 31 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/633/ [14:39:22] Project browsertests-Wikidata-WikidataTests-linux-chrome-sauce build #36: ABORTED in 10 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-chrome-sauce/36/ [14:39:26] Project browsertests-Wikidata-WikidataTests-linux-firefox-sauce build #239: ABORTED in 17 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-firefox-sauce/239/ [14:39:34] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #266: ABORTED in 22 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/266/ [14:39:42] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce build #623: ABORTED in 20 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce/623/ [15:24:45] Project beta-scap-eqiad build #54772: FAILURE in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/54772/ [15:27:46] Yippee, build fixed! [15:27:47] Project beta-scap-eqiad build #54773: FIXED in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/54773/ [16:08:35] (03PS3) 10Polybuildr: Add tests for mediawiki/extensions/SmiteSpam [integration/config] - 10https://gerrit.wikimedia.org/r/214304 [16:11:25] Can someone please review https://gerrit.wikimedia.org/r/#/c/214304/? Patch for a new extension's tests in integration/config. [16:15:16] 10Continuous-Integration-Infrastructure, 10Librarization, 10MediaWiki-Vendor: Unable to update libraries in MediaWiki core and vendor due to version mismatch and circular dependencies - https://phabricator.wikimedia.org/T88211#1317971 (10JanZerebecki) The --skip-external-dependencies won't fix the circular d... [16:17:56] (03CR) 10JanZerebecki: [C: 04-1] "As explained in T88211#1317971 for the short term I'd suggest to use --skip-external-dependencies only when testing mediawiki/vendor, i.e." (031 comment) [integration/jenkins] - 10https://gerrit.wikimedia.org/r/187933 (https://phabricator.wikimedia.org/T88211) (owner: 10Legoktm) [16:37:54] (03CR) 10Zfilipin: [C: 032] Support links to Sauce jobs in Raita logging [selenium] - 10https://gerrit.wikimedia.org/r/214274 (owner: 10Dduvall) [16:38:13] (03Merged) 10jenkins-bot: Support links to Sauce jobs in Raita logging [selenium] - 10https://gerrit.wikimedia.org/r/214274 (owner: 10Dduvall) [16:41:13] 10Deployment-Systems, 6Release-Engineering: scap only one version - https://phabricator.wikimedia.org/T100575#1318034 (10greg) p:5Triage>3Normal [16:46:27] 10Continuous-Integration-Infrastructure, 6Labs: Designate should support split horizon resolution to yield private IP of instances behind a public DNS entry - https://phabricator.wikimedia.org/T95288#1318052 (10yuvipanda) [16:46:50] 10Continuous-Integration-Infrastructure, 6Labs: Designate should support split horizon resolution to yield private IP of instances behind a public DNS entry - https://phabricator.wikimedia.org/T95288#1185706 (10yuvipanda) [16:51:51] (03PS1) 10Dduvall: Releasing minor version 1.2.0 [selenium] - 10https://gerrit.wikimedia.org/r/214379 [16:53:09] (03CR) 10Zfilipin: [C: 032] Releasing minor version 1.2.0 [selenium] - 10https://gerrit.wikimedia.org/r/214379 (owner: 10Dduvall) [16:53:23] (03Merged) 10jenkins-bot: Releasing minor version 1.2.0 [selenium] - 10https://gerrit.wikimedia.org/r/214379 (owner: 10Dduvall) [16:58:45] 6Release-Engineering, 10Ops-Access-Requests, 10Wikimedia-Git-or-Gerrit, 6operations, 5Patch-For-Review: Add all Release-Engineering team as Gerrit admins - https://phabricator.wikimedia.org/T100565#1318095 (10RobH) @Greg, can we get you to approve this as the manager of release engineering? [17:01:56] 6Release-Engineering, 10Wikidata, 10Wikidata-Page-Banner: Setup Jenkins - https://phabricator.wikimedia.org/T100495#1318110 (10greg) I think I'm missing a lot of context here. What's the goal/purpose of this task? What's needed? We already have those running on our Jenkins. [17:02:42] 6Release-Engineering, 10Ops-Access-Requests, 10Wikimedia-Git-or-Gerrit, 6operations, 5Patch-For-Review: Add all Release-Engineering team as Gerrit admins - https://phabricator.wikimedia.org/T100565#1318113 (10greg) {F100025} [17:04:23] 6Release-Engineering: Changes to ForrestBot for change to MW deploy cadence - https://phabricator.wikimedia.org/T100567#1318125 (10greg) a:5Legoktm>3None [17:05:32] 6Release-Engineering: Changes to ForrestBot for change to MW deploy cadence - https://phabricator.wikimedia.org/T100567#1315804 (10greg) @valhallasw Any changes need to be made to ForrestBot (the code) in response to https://lists.wikimedia.org/pipermail/wikitech-l/2015-May/081863.html ? [17:09:27] 6Release-Engineering, 6operations: Try out hack ( 10Deployment-Systems, 6Release-Engineering: make-release should run a syntax checker over the generated tarball - https://phabricator.wikimedia.org/T100599#1318148 (10greg) [17:10:20] 10Browser-Tests, 10Continuous-Integration-Infrastructure, 10Wikimedia-Fundraising: Create unit and integration tests for Fundraising extensions to identify breaking MediaWiki changes - https://phabricator.wikimedia.org/T89404#1318153 (10awight) [17:14:34] 5Continuous-Integration-Isolation, 10hardware-requests, 6operations: eqiad: 2 hardware access request for CI isolation on labsnet - https://phabricator.wikimedia.org/T93076#1318159 (10RobH) a:3hashar Since the second server is on hold, rather than keep this unassigned (and end up with me checking it every... [17:14:49] (03CR) 10Jdlrobson: "Please someone merge... :)" [integration/config] - 10https://gerrit.wikimedia.org/r/191046 (https://phabricator.wikimedia.org/T74794) (owner: 10Hashar) [17:15:04] (03Abandoned) 10Jdlrobson: Publish documentation for MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/181693 (https://phabricator.wikimedia.org/T74794) (owner: 10Jdlrobson) [17:25:45] 10Browser-Tests, 6Release-Engineering, 10MediaWiki-Vagrant, 5Patch-For-Review: Vagrant command for running browser tests - https://phabricator.wikimedia.org/T96283#1318169 (10dduvall) The second patch (https://gerrit.wikimedia.org/r/#/c/206172/) still needs some cleanup. [17:27:03] 10Browser-Tests, 10MediaWiki-Vagrant, 5Patch-For-Review: Vagrant command for running browser tests - https://phabricator.wikimedia.org/T96283#1318171 (10dduvall) [17:30:19] 10Browser-Tests, 5Patch-For-Review: Upgrade CentralNotice browser tests to use mediawiki_selenium 1.x - https://phabricator.wikimedia.org/T99652#1318195 (10dduvall) a:3zeljkofilipin [18:02:24] 6Release-Engineering, 7user-notice: Shorten/Simplify MW train deploy cadence to Tu->W->Th - https://phabricator.wikimedia.org/T97553#1318302 (10valhallasw) [18:02:25] 6Release-Engineering: Changes to ForrestBot for change to MW deploy cadence - https://phabricator.wikimedia.org/T100567#1318299 (10valhallasw) 5Open>3Resolved a:3valhallasw No, I don't think so. It just adds #wm1.27 or #wm1.27wmf6 to a task. Those are aliases for the project which is then actually named wi... [18:21:32] PROBLEM - Puppet failure on integration-zuul-packaged is CRITICAL 100.00% of data above the critical threshold [0.0] [19:14:20] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL 22.22% of data above the critical threshold [0.0] [19:14:32] 6Release-Engineering, 10Ops-Access-Requests, 10Wikimedia-Git-or-Gerrit, 6operations, 5Patch-For-Review: Add all Release-Engineering team as Gerrit admins - https://phabricator.wikimedia.org/T100565#1318550 (10hashar) Some more discussion happening on https://gerrit.wikimedia.org/r/#/c/214255/ . I have po... [19:17:47] PROBLEM - Puppet failure on deployment-salt is CRITICAL 50.00% of data above the critical threshold [0.0] [19:17:59] PROBLEM - Puppet failure on deployment-db2 is CRITICAL 30.00% of data above the critical threshold [0.0] [19:18:15] PROBLEM - Puppet failure on deployment-db1 is CRITICAL 44.44% of data above the critical threshold [0.0] [19:18:37] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL 30.00% of data above the critical threshold [0.0] [19:18:39] PROBLEM - Puppet failure on deployment-kafka02 is CRITICAL 40.00% of data above the critical threshold [0.0] [19:19:05] PROBLEM - Puppet failure on deployment-fluorine is CRITICAL 33.33% of data above the critical threshold [0.0] [19:22:50] 10Beta-Cluster, 10MediaWiki-JobRunner, 10MediaWiki-extensions-TimedMediaHandler, 6Multimedia: deployment-videoscaler01 PHP Fatal error: /srv/mediawiki/wikiversions-labs.cdb has no version entry for `webVideoTranscode%2Fdewiki/webVideoTranscode%2Fdewiki`.#0... - https://phabricator.wikimedia.org/T100694#1318564 [19:23:27] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL 70.00% of data above the critical threshold [0.0] [19:24:10] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL 44.44% of data above the critical threshold [0.0] [19:24:48] 10Beta-Cluster, 10MediaWiki-JobRunner, 10MediaWiki-extensions-TimedMediaHandler, 6Multimedia: deployment-videoscaler01 PHP Fatal error: /srv/mediawiki/wikiversions-labs.cdb has no version entry for `webVideoTranscode%2Fdewiki/webVideoTranscode%2Fdewiki`.#0... - https://phabricator.wikimedia.org/T100694#1318579 [19:29:14] 10Beta-Cluster, 10MediaWiki-JobRunner, 10MediaWiki-extensions-TimedMediaHandler, 6Multimedia: deployment-videoscaler01 PHP Fatal error: /srv/mediawiki/wikiversions-labs.cdb has no version entry for `webVideoTranscode%2Fdewiki/webVideoTranscode%2Fdewiki`.#0... - https://phabricator.wikimedia.org/T100694#1318590 [19:31:29] 10Beta-Cluster, 10MediaWiki-JobRunner, 10MediaWiki-extensions-TimedMediaHandler, 6Multimedia: deployment-videoscaler01 PHP Fatal error: /srv/mediawiki/wikiversions-labs.cdb has no version entry for `webVideoTranscode%2Fdewiki/webVideoTranscode%2Fdewiki`.#0... - https://phabricator.wikimedia.org/T100694#1318592 [19:37:21] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL 28.57% of data above the critical threshold [0.0] [19:42:22] RECOVERY - Puppet failure on deployment-videoscaler01 is OK Less than 1.00% above the threshold [0.0] [19:42:48] RECOVERY - Puppet failure on deployment-salt is OK Less than 1.00% above the threshold [0.0] [19:43:16] RECOVERY - Puppet failure on deployment-db1 is OK Less than 1.00% above the threshold [0.0] [19:44:22] RECOVERY - Puppet failure on deployment-elastic05 is OK Less than 1.00% above the threshold [0.0] [19:47:56] RECOVERY - Puppet failure on deployment-db2 is OK Less than 1.00% above the threshold [0.0] [19:48:26] RECOVERY - Puppet failure on deployment-cxserver03 is OK Less than 1.00% above the threshold [0.0] [19:48:36] RECOVERY - Puppet failure on deployment-logstash1 is OK Less than 1.00% above the threshold [0.0] [19:48:44] RECOVERY - Puppet failure on deployment-kafka02 is OK Less than 1.00% above the threshold [0.0] [19:49:04] RECOVERY - Puppet failure on deployment-fluorine is OK Less than 1.00% above the threshold [0.0] [19:49:10] RECOVERY - Puppet failure on deployment-elastic06 is OK Less than 1.00% above the threshold [0.0] [19:53:21] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL 11.11% of data above the critical threshold [0.0] [19:54:49] bah [19:54:56] videoscaler is out of disk [19:55:05] jobchron / jobservice is spamming fatal errors [19:57:35] Yippee, build fixed! [19:57:36] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » vi,contintLabsSlave && UbuntuTrusty build #55: FIXED in 18 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=vi,label=contintLabsSlave%20&&%20UbuntuTrusty/55/ [20:03:26] RECOVERY - Puppet failure on deployment-videoscaler01 is OK Less than 1.00% above the threshold [0.0] [20:05:42] PROBLEM - Free space - all mounts on deployment-videoscaler01 is CRITICAL deployment-prep.deployment-videoscaler01.diskspace._var.byte_percentfree (<30.00%) [20:16:38] Yippee, build fixed! [20:16:38] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » pt,contintLabsSlave && UbuntuTrusty build #55: FIXED in 37 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=pt,label=contintLabsSlave%20&&%20UbuntuTrusty/55/ [20:25:37] hashar: I think aaron was poking at that last night (videoscaler disk space) [20:28:27] bd808: I cced him. Apparently puppet was broken and the jobchron/jobservice were not running [20:29:09] PROBLEM - Puppet staleness on deployment-restbase02 is CRITICAL 100.00% of data above the critical threshold [43200.0] [20:29:50] that box should be rebuilt. It's got the old 2G /var [20:31:13] "Fatal error: /srv/mediawiki/wikiversions-labs.cdb has no version entry for `webVideoTranscode%2Fenwiki/webVideoTranscode%2Fwikidatawiki`" [20:31:20] junk in the job queue looks like [20:31:36] yeah filled that [20:32:00] ori and I figured out how to clean up things like that once ... [20:34:03] User::incEditCountImmediate holding locks yeahhh [20:34:20] PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL 100.00% of data above the critical threshold [43200.0] [20:35:01] found it -- https://phabricator.wikimedia.org/T87360 [20:35:21] Yippee, build fixed! [20:35:21] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » et,contintLabsSlave && UbuntuTrusty build #55: FIXED in 56 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=et,label=contintLabsSlave%20&&%20UbuntuTrusty/55/ [20:41:07] something is not right looking about the h-ready-queues names [20:41:12] "webVideoTranscode%252Fwikidatawiki%2FwebVideoTranscode%252Fwikidatawiki/1429745518%2FwebVideoTranscode%252Fdewiki" [20:41:23] that looks like garbage [20:41:29] bd808: the bug is https://phabricator.wikimedia.org/T100694 [20:42:20] %252F is %2F which is / [20:42:27] looks like double encoding :} [20:43:14] I which we had a stacktrace [20:43:55] 10Beta-Cluster, 10MediaWiki-JobRunner, 10MediaWiki-extensions-TimedMediaHandler, 6Multimedia: deployment-videoscaler01 PHP Fatal error: /srv/mediawiki/wikiversions-labs.cdb has no version entry for `webVideoTranscode%2Fdewiki/webVideoTranscode%2Fdewiki`.#0... - https://phabricator.wikimedia.org/T100694#1318762 [20:44:31] hashar: I'm tempted to just purge that ready queue list [20:44:38] it all looks like junk [20:45:24] one looks like a timestamp ( 1429745518 ) [20:45:30] which is April 23 [20:45:41] things in that list should look like "/" [20:47:00] 10Beta-Cluster, 10MediaWiki-JobRunner, 10MediaWiki-extensions-TimedMediaHandler, 6Multimedia: deployment-videoscaler01 PHP Fatal error: /srv/mediawiki/wikiversions-labs.cdb has no version entry for `webVideoTranscode%2Fdewiki/webVideoTranscode%2Fdewiki`.#0... - https://phabricator.wikimedia.org/T100694#1318770 [20:47:27] +1 on deleting the old entries [20:47:40] and keep it open a bit to verify it is no more happening [20:48:10] now I just need to find the "empty a hash" redis magic [20:48:56] goodluck :} [20:49:00] I am off myself [20:50:47] !log Ran "del jobqueue:aggregator:h-ready-queues:v2" on deployment-redis01 [20:50:50] Logged the message, Master [20:52:01] 10Beta-Cluster, 10MediaWiki-JobRunner, 10MediaWiki-extensions-TimedMediaHandler, 6Multimedia: deployment-videoscaler01 PHP Fatal error: /srv/mediawiki/wikiversions-labs.cdb has no version entry for `webVideoTranscode%2Fdewiki/webVideoTranscode%2Fdewiki`.#0... - https://phabricator.wikimedia.org/T100694#1318781 [20:53:18] 10Beta-Cluster, 10MediaWiki-extensions-GettingStarted: GettingStarted on Beta Cluster periodically loses its Redis index - https://phabricator.wikimedia.org/T100515#1318782 (10hashar) > $sessionRedis from session-labs.php Seems that redis server should only be for session storages isn't it ? I guess on prod... [20:53:30] gotta sleep [20:53:33] thanks bd808 ! [20:56:02] Yippee, build fixed! [20:56:02] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » el,contintLabsSlave && UbuntuTrusty build #55: FIXED in 1 hr 17 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=el,label=contintLabsSlave%20&&%20UbuntuTrusty/55/ [21:12:49] greg-g: If I were to make a list of all extensions in production, and which team "owns" them for the purposes of e.g. fixing #Wikimedia-log-errors tasks, would I be duplicating work? [21:16:00] Yippee, build fixed! [21:16:01] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » ar,contintLabsSlave && UbuntuTrusty build #55: FIXED in 1 hr 37 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=ar,label=contintLabsSlave%20&&%20UbuntuTrusty/55/ [21:17:24] PROBLEM - Puppet staleness on deployment-eventlogging02 is CRITICAL 100.00% of data above the critical threshold [43200.0] [21:21:26] James_F: in what format? But the answer is no, you wouldn't be, and you'd be helping :) [21:24:28] greg-g: I was thinking of a spreadsheet at first, then a wiki page. [21:25:31] James_F: it would be great if we could take the list in make-wmf-branches and put it in yaml or something, we need that for forrestbot at least... [21:26:12] 10Beta-Cluster, 10MediaWiki-ResourceLoader, 3Mobile-Web-Sprint-48-Voyage-of-the-Damned: ResourceLoader urls with certain version serving different content on beta cluster - https://phabricator.wikimedia.org/T100542#1318887 (10hashar) On beta the the bits cache is not invalidated and I have no idea how it is... [21:26:51] 10Beta-Cluster: Caching makes it impossible to test JS changes when logged out - https://phabricator.wikimedia.org/T65034#663605 (10hashar) I pasted some headers responses on T100542#1318887 (most probably be a duplicate). [21:28:15] legoktm++ [21:36:03] Yippee, build fixed! [21:36:03] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » kn,contintLabsSlave && UbuntuTrusty build #55: FIXED in 1 hr 57 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=kn,label=contintLabsSlave%20&&%20UbuntuTrusty/55/ [21:38:22] I've been wanting to move the list out of make-wmf-branch as well [21:39:32] legoktm: I was just using https://noc.wikimedia.org/conf/extension-list [21:40:24] James_F: the extensions in conf and make-wmf-branch are essentially in sync, though only through manual effort [21:46:54] twentyafterfour: Yeah. :-( [21:50:03] 10Beta-Cluster, 10MediaWiki-ResourceLoader, 3Mobile-Web-Sprint-48-Voyage-of-the-Damned: ResourceLoader urls with certain version serving different content on beta cluster - https://phabricator.wikimedia.org/T100542#1318999 (10thcipriani) More header fun **Here is the backend from which Varnish caches, the s... [21:50:23] James_F: that doesn't have all extensions. Only those that have localized messages. [21:50:37] James_F: make-wmf-branches is the authoratative list. [21:50:37] legoktm: Eww. [21:51:17] the name extension-list is just misleading [21:55:36] * James_F grumbles. [21:56:17] Yippee, build fixed! [21:56:17] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » ce,contintLabsSlave && UbuntuTrusty build #55: FIXED in 2 hr 17 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=ce,label=contintLabsSlave%20&&%20UbuntuTrusty/55/ [22:00:59] legoktm: There are a few extensions missing from make-wmf-branches. [22:01:25] legoktm: Oh, wait, no, just… messy. [22:01:39] yeah....... [22:02:00] Ick. [22:07:12] Project beta-parsoid-update-eqiad build #1030: FAILURE in 0.83 sec: https://integration.wikimedia.org/ci/job/beta-parsoid-update-eqiad/1030/ [22:17:07] Yippee, build fixed! [22:17:08] Project beta-parsoid-update-eqiad build #1033: FIXED in 3.3 sec: https://integration.wikimedia.org/ci/job/beta-parsoid-update-eqiad/1033/ [22:27:37] (03PS1) 10Ori.livneh: Make SemanticForms unit tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/214501 [23:36:02] 10Beta-Cluster: Caching makes it impossible to test JS changes when logged out - https://phabricator.wikimedia.org/T65034#1319220 (10BBlack) This is all related to the same basic issue in production: T99096 [23:36:04] 10Beta-Cluster, 10MediaWiki-ResourceLoader, 3Mobile-Web-Sprint-48-Voyage-of-the-Damned: ResourceLoader urls with certain version serving different content on beta cluster - https://phabricator.wikimedia.org/T100542#1319224 (10BBlack) This is all related to the same basic issue in production: T99096 [23:41:52] (03CR) 10Ori.livneh: "The dependent change has been merge, so this is good to go, provided Yaron is interested in making the tests voting." [integration/config] - 10https://gerrit.wikimedia.org/r/214501 (owner: 10Ori.livneh)