[00:05:43] 3Continuous-Integration: MediaWiki installs in Jenkins frequently fail to access their sqlite database due to locks - https://phabricator.wikimedia.org/T89180#1045226 (10tstarling) I attempted to file a bug upstream, which involves subscribing to the sqlite-users mailing list and posting a message to it. Subscri... [00:13:14] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<12.50%) [00:16:52] Krinkle: is it possible for jjb templates to have optional parameters? [00:17:00] marxarelli: No [00:17:12] There is a workaround. Two of them. [00:17:31] marxarelli: We tend to abstract it with an in-between template. One for the default params [00:17:41] e.g. npm-conf used to take workspace as parameter [00:17:48] and npm calls npm-conf with $WORKSPACE as parameter [00:17:59] then use either npm or npm-conf accordingly [00:18:10] marxarelli: I've eradicated most optional parameters though [00:18:15] Krinkle: ah, ok [00:18:15] marxarelli: Where are you looking to use it? [00:18:27] i'm running into an issue with mw-vagrant and its bundle jobs [00:18:41] vagrant itself requires bundler < 1.8 [00:18:58] but our bundle job installs the newest version from rubygems [00:19:46] Krinkle: https://integration.wikimedia.org/ci/job/mediawiki-vagrant-bundle-rspec/4/console [00:21:39] we could just modify the bundle builder to use a specific version, but i'm a hesitant to introduce the maintenance burden of having to periodically increment it [00:22:53] my thought was to add an optional parameter to the builder for the version of bundler to be installed and used to exec the given command [00:23:28] unless you have a better idea, of course [00:24:48] it's pretty rare that a gem will depend on bundler directly, but vagrant is one such occurrence [00:27:50] Krinkle: given it's an edge case, should i just create an addition template to be used mw-vagrant? [00:28:04] *by* mw-vagrant [00:33:49] 3Continuous-Integration, VisualEditor: Jenkins: Convert mwext qunit from grunt-contrib-qunit (PhantomJS) to grunt-karma (Chromium) - https://phabricator.wikimedia.org/T74063#765899 (10Krinkle) [00:34:47] marxarelli: Hm.. How recent is the version vagrant wants compared to the version we're currently using everywhere else? [00:34:52] You reckon it's backwards compatible? [00:35:04] e.g. could we consistently use < 1.8? [00:35:13] Hm.. that sounds terrible to downgrade. [00:35:27] Krinkle: most likely [00:35:28] ah, only 1.8.2 [00:35:37] yeah, 1.8 is pretty recent [00:35:41] cool [00:35:54] marxarelli: how much duplication would there be? [00:36:30] Krinkle: one extra template in jjb/ruby-jobs.yaml [00:36:30] marxarelli: Is it a nested macro or is macro:bundler used directly? [00:36:34] https://github.com/wikimedia/integration-config/blob/da2f4ef8/jjb/ruby-jobs.yaml#L2 [00:37:16] marxarelli: Yeah, keeping bundler fresh seems nice at the moment. Though in general I'd advise against silence upgrades like that [00:37:41] But fixing to an older version is tricky. I'd duplicate bunlder into a suffixed variant for vagrant [00:37:48] bundler17 or some such [00:37:52] bundle* [00:38:14] Krinkle: so, an additional template that calls the same builder/macro? [00:39:30] marxarelli: Hm.. ah, it has a job-template as well [00:39:39] marxarelli: You can create the job directly [00:39:58] in ruby-jobs use job: instead of job-template and name it e.g. mediawiki-vagrant-bundle-rspec [00:40:05] overloading that name from template pattern [00:40:09] and use bundle17 macro there [00:41:42] Krinkle: i see... [00:42:25] marxarelli: I don't know this area too well though. Feel free to use a different approach as you see fit. [00:42:26] Krinkle: maybe i can adapt the bundle macro with a conditional on a specified or empty {version}? [00:43:01] marxarelli: The conditional could work in bash at run-time. [00:43:20] Krinkle: cool. i'll give it a shot and have you and hashar review it for ickiness [00:46:34] Yippee, build fixed! [00:46:34] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #1495: FIXED in 33 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/1495/ [00:46:54] Krinkle: thanks for the help! [01:21:32] 3Release-Engineering: Release MW 1.24.2 and 1.23.9 tarballs - https://phabricator.wikimedia.org/T88120#1045440 (10GPHemsley) >>! In T88120#1044063, @csteipp wrote: >>>! In T88120#1041713, @GPHemsley wrote: >> We're now beginning the second week since the week of February 2nd, and there still has not been a relea... [01:22:25] 3Deployment-Systems: l10nupdate user can't access scap shared ssh key causing nightly l10nupdate sync process to fail - https://phabricator.wikimedia.org/T76061#1045443 (10bd808) There are two possible ways to fix this problem for the l10nupdate user invoking scap: # Make a way (probably via a new command line o... [01:23:21] 3Deployment-Systems: l10nupdate user can't access scap shared ssh key causing nightly l10nupdate sync process to fail - https://phabricator.wikimedia.org/T76061#1045444 (10bd808) >>! In T76061#1045443, @bd808 wrote: > A possible third option that would be a bit of a combination of the other two but more involved... [01:34:58] Project beta-scap-eqiad build #42190: FAILURE in 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/42190/ [01:41:01] 3Release-Engineering: Release MW 1.24.2 and 1.23.9 tarballs - https://phabricator.wikimedia.org/T88120#1045485 (10greg) >>! In T88120#1045440, @GPHemsley wrote: >>>! In T88120#1044063, @csteipp wrote: >>>>! In T88120#1041713, @GPHemsley wrote: >>> We're now beginning the second week since the week of February 2n... [01:41:31] !log fixed git rebase conflict on deployment-salt caused by outdated cherry-pick; cherry-picks are merged now so reset to tracking origin/production [01:41:37] Logged the message, Master [01:41:57] Yuvi|Vacation will be proud of me for getting back to 0 cherry-picks for beta [01:54:59] Yippee, build fixed! [01:55:00] Project beta-scap-eqiad build #42192: FIXED in 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/42192/ [01:56:03] 3Deployment-Systems: l10nupdate user can't access scap shared ssh key causing nightly l10nupdate sync process to fail - https://phabricator.wikimedia.org/T76061#1045516 (10bd808) >>! In T76061#1045444, @bd808 wrote: >>>! In T76061#1045443, @bd808 wrote: >> A possible third option that would be a bit of a combina... [02:18:14] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<37.50%) [02:35:03] RECOVERY - Long lived cherry-picks on puppetmaster on deployment-salt is OK: OK: Less than 100.00% above the threshold [1.0] [02:52:34] PROBLEM - Puppet staleness on deployment-urldownloader is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [43200.0] [03:06:46] Yippee, build fixed! [03:06:47] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #419: FIXED in 46 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/419/ [03:18:49] Yippee, build fixed! [03:18:50] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #129: FIXED in 49 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/129/ [03:39:13] Yippee, build fixed! [03:39:14] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #321: FIXED in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/321/ [03:40:41] Yippee, build fixed! [03:40:42] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #459: FIXED in 1 min 27 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/459/ [03:49:09] Yippee, build fixed! [03:49:09] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #547: FIXED in 38 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/547/ [04:03:41] greg-g: omg :/ [04:04:02] sorry. It looks I messed up with timezone. [04:08:27] Yippee, build fixed! [04:08:28] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #304: FIXED in 29 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/304/ [04:09:17] Yippee, build fixed! [04:09:17] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #411: FIXED in 49 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/411/ [04:10:07] Yippee, build fixed! [04:10:07] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #193: FIXED in 49 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/193/ [04:18:17] Yippee, build fixed! [04:18:18] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #320: FIXED in 37 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/320/ [04:21:52] Yippee, build fixed! [04:21:53] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #514: FIXED in 28 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/514/ [04:24:26] Yippee, build fixed! [04:24:27] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #450: FIXED in 2 min 33 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/450/ [04:25:14] Yippee, build fixed! [04:25:14] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #415: FIXED in 46 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/415/ [04:43:15] Yippee, build fixed! [04:43:15] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #128: FIXED in 49 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/128/ [04:53:56] Yippee, build fixed! [04:53:57] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce build #128: FIXED in 1 min 22 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce/128/ [05:01:10] Yippee, build fixed! [05:01:11] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #523: FIXED in 17 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/523/ [05:27:21] Yippee, build fixed! [05:27:22] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #193: FIXED in 42 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/193/ [05:28:10] Yippee, build fixed! [05:28:10] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #346: FIXED in 48 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/346/ [05:40:35] If I can log on to deployment-bastion (I can), am I supposed to be able to run mwscript? [05:41:03] I'm trying to test a Flow maint script (which we want to run on production in the future) on Beta's enwiki. [05:41:27] yes [05:41:27] does it not work for you? [05:41:43] sudo is prompting me for my password (which I don't know, since I log in via ssh). [05:41:49] can you sudo? [05:42:06] Krenair, how do I learn my password? When I run passwd, it outputs: [05:42:12] passwd: Authentication token manipulation error [05:42:16] passwd: password unchanged [05:42:26] I was running into this issue earlier actually [05:42:33] sudo works for me [05:42:46] I am in wikidev [05:42:48] yeah you can't pull your password out of these systems :) [05:43:10] Krenair, well, I was more trying to change it then have it output it in plaintext. (hopefully it can't do that!). [05:43:16] But yeah [05:43:42] Krenair, it looks like the magic groups are sudo, wikidev, and root. [05:43:44] I am in only wikidev. [05:43:46] How about you? [05:44:14] according to the groups command I am in only wikidev. [05:44:18] (of that group) [05:44:57] Hmm, strange. [05:45:17] I don't need sudo to run it in production :) [05:46:37] Yippee, build fixed! [05:46:37] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #319: FIXED in 37 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/319/ [05:46:43] I don't remember needing it either, though I normally just run foreachwikiindblist or eval.php (but I think both use mwscript directly, and the latter even in the command line). [05:46:46] Yippee, build fixed! [05:46:46] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #468: FIXED in 13 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/468/ [05:47:33] Yippee, build fixed! [05:47:33] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #418: FIXED in 55 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/418/ [05:53:08] so it's not set up to allow people in wikidev to sudo as apache passwordless [05:58:28] superm401, could you sudo? [05:58:29] or not? [05:58:58] Krenair, when I run mwscript, it prompts me for sudo (I didn't explicitly typo sudo, it's part of the script). [05:59:16] And like you said, I don't know my password, and can't do it passwordless, so I'm not sure how to proceed. [05:59:32] actually prefix your command with sudo [05:59:54] Just tried it after I posted that, didn't help. [06:00:05] sudo -u apache mwscript extensions/Flow/maintenance/FlowFixEditCount.php --wiki=enwiki [06:00:14] [sudo] password for mattflaschen: [06:00:53] without the -u apache [06:01:34] Krenair, thanks. [06:01:42] :) [06:01:44] So I can sudo as root passwordless, but not some random user. :) [06:01:52] I wonder why it's requiring that though [06:01:57] Yeah, I'll file. [06:02:15] you should be able to do it as apache [06:02:48] but it's broken obviously [06:03:03] Yeah [06:05:24] 3Beta-Cluster: Can't run mwscript without explicit sudo on Beta Labs - https://phabricator.wikimedia.org/T89802#1045790 (10Mattflaschen) [06:07:33] I think that's set up in modules/mediawiki/manifests/users.pp [06:08:03] in that sudo::group block [06:12:13] 3Beta-Cluster: Can't run mwscript without explicit sudo on Beta Labs - https://phabricator.wikimedia.org/T89802#1045791 (10Krenair) (obviously the workaround is for projectadmins only, and shouldn't really be encouraged... although this is beta so...) [06:29:56] (03PS1) 10Mattflaschen: VectorBeta depends on EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/191262 [06:30:52] (03CR) 10jenkins-bot: [V: 04-1] VectorBeta depends on EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/191262 (owner: 10Mattflaschen) [06:33:14] (03PS2) 10Mattflaschen: VectorBeta depends on EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/191262 [06:38:09] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [07:32:52] Project beta-update-databases-eqiad build #7683: FAILURE in 1 hr 12 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/7683/ [07:33:42] Yippee, build fixed! [07:33:43] Project beta-update-databases-eqiad build #7684: FIXED in 45 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/7684/ [07:54:38] Project beta-scap-eqiad build #42222: FAILURE in 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/42222/ [08:04:19] 3Deployment-Systems: [scap] Syncing wmf-config/PrivateSettings.php syncs symlink and not file contents - https://phabricator.wikimedia.org/T72054#1045909 (10mmodell) 5Open>3Resolved [08:14:46] Yippee, build fixed! [08:14:46] Project beta-scap-eqiad build #42224: FIXED in 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/42224/ [08:15:16] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [08:25:19] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:05:16] hashar: I am in the hangout [09:05:17] zeljkof: almost here [09:26:40] 3Release-Engineering, Echo: Add credentials for Echo users to browser test builds in Jenkins - https://phabricator.wikimedia.org/T89497#1045969 (10hashar) On beta cluster, the Echo browser test is configured to use Selenium_user and the password in MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG . When changing to... [09:31:04] 3Beta-Cluster: Can't run mwscript without explicit sudo on Beta Labs - https://phabricator.wikimedia.org/T89802#1045982 (10Aklapper) p:5Triage>3Volunteer? [09:40:34] Yippee, build fixed! [09:40:34] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #376: FIXED in 7 min 46 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/376/ [09:41:47] 3Release-Engineering, Echo: Add credentials for Echo users to browser test builds in Jenkins - https://phabricator.wikimedia.org/T89497#1045993 (10hashar) 5Open>3Resolved The password update fixed both jobs: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrom... [09:42:30] current run is https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/356/console [09:47:48] Yippee, build fixed! [09:47:49] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #356: FIXED in 6 min 51 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/356/ [10:04:54] zeljkof: the other Echo job is fixed :D [10:17:26] 3Quality-Assurance, MediaWiki-extensions-UploadWizard, Multimedia: UploadWizard browser test for chunked upload - https://phabricator.wikimedia.org/T89289#1046043 (10Gilles) [10:22:47] 3Quality-Assurance, MediaWiki-extensions-UploadWizard, Multimedia: UploadWizard browser test for chunked upload - https://phabricator.wikimedia.org/T89289#1046063 (10Gilles) [10:30:58] (03CR) 10Hashar: [C: 032] "hoo: I have updated the jobs" [integration/config] - 10https://gerrit.wikimedia.org/r/190690 (owner: 10Hoo man) [10:34:17] (03PS2) 10Jhernandez: Add Gather extension conf [integration/config] - 10https://gerrit.wikimedia.org/r/191009 [10:35:31] (03CR) 10Jhernandez: "Added MF's dependencies to Gather's dependencies on PS2" [integration/config] - 10https://gerrit.wikimedia.org/r/191009 (owner: 10Jhernandez) [10:38:01] (03Merged) 10jenkins-bot: Add "cldr" to all Wikibase/Wikidata php tests [integration/config] - 10https://gerrit.wikimedia.org/r/190690 (owner: 10Hoo man) [10:39:09] (03PS3) 10Hashar: Add Gather extension conf [integration/config] - 10https://gerrit.wikimedia.org/r/191009 (owner: 10Jhernandez) [10:42:50] (03CR) 10Hashar: "I have adjusted the Zuul configuration to prevent the test extension job from running on master branch since we already run the shared job" [integration/config] - 10https://gerrit.wikimedia.org/r/191009 (owner: 10Jhernandez) [10:43:47] (03PS1) 10Hoo man: Also load cldr for Wikibase repo API tests [integration/config] - 10https://gerrit.wikimedia.org/r/191301 [10:49:38] (03CR) 10Hashar: [C: 032] "hoo: Job updated" [integration/config] - 10https://gerrit.wikimedia.org/r/191301 (owner: 10Hoo man) [10:56:09] (03Merged) 10jenkins-bot: Also load cldr for Wikibase repo API tests [integration/config] - 10https://gerrit.wikimedia.org/r/191301 (owner: 10Hoo man) [11:16:54] hashar: there is simple patch for you :) [11:21:56] PROBLEM - Content Translation Server on deployment-cxserver03 is CRITICAL: Connection refused [11:22:19] what! [11:36:56] RECOVERY - Content Translation Server on deployment-cxserver03 is OK: HTTP OK: HTTP/1.1 200 OK - 1103 bytes in 0.018 second response time [11:49:16] (03CR) 10Jhernandez: [C: 031] "I've tested the jobs with master, and after merging a patch for the hhvm/zend patches all the jobs seem to work fine with master." [integration/config] - 10https://gerrit.wikimedia.org/r/191009 (owner: 10Jhernandez) [12:09:35] 3Beta-Cluster: Can't run mwscript without explicit sudo on Beta Labs - https://phabricator.wikimedia.org/T89802#1046165 (10Krenair) p:5Volunteer?>3Triage [12:43:59] 3Deployment-Systems: l10nupdate user can't access scap shared ssh key causing nightly l10nupdate sync process to fail - https://phabricator.wikimedia.org/T76061#1046186 (10mmodell) scap change +2'd but I don't have +2 on ops/puppet so that one isn't merged. [12:47:05] 3Continuous-Integration: MediaWiki installs in Jenkins frequently fail to access their sqlite database due to locks - https://phabricator.wikimedia.org/T89180#1046190 (10Krinkle) >>! In T89180#1045226, @tstarling wrote: > Workaround ideas: > * Downgrade to SQLite 3.6 > * Reduce apache concurrency > * Create tabl... [14:14:39] Project beta-scap-eqiad build #42259: FAILURE in 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/42259/ [14:18:00] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:27:56] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:28:21] <^d> What a weird error in prod. [14:28:23] <^d> "parent, LightProcess exiting" [14:28:26] <^d> (that's it) [14:34:53] Yippee, build fixed! [14:34:53] Project beta-scap-eqiad build #42261: FIXED in 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/42261/ [15:10:43] (03PS4) 10Hashar: Add Gather extension conf [integration/config] - 10https://gerrit.wikimedia.org/r/191009 (owner: 10Jhernandez) [15:15:59] (03CR) 10Hashar: [C: 032] "PS4: I have added Gather to mediawiki-extensions-hhvm and mediawiki-extensions-zend shared jobs. \O/" [integration/config] - 10https://gerrit.wikimedia.org/r/191009 (owner: 10Jhernandez) [15:16:08] (03PS5) 10Hashar: Add Gather extension conf [integration/config] - 10https://gerrit.wikimedia.org/r/191009 (owner: 10Jhernandez) [15:16:16] (03CR) 10Hashar: [C: 032] Add Gather extension conf [integration/config] - 10https://gerrit.wikimedia.org/r/191009 (owner: 10Jhernandez) [15:17:00] (03CR) 10jenkins-bot: [V: 04-1] Add Gather extension conf [integration/config] - 10https://gerrit.wikimedia.org/r/191009 (owner: 10Jhernandez) [15:23:28] (03Merged) 10jenkins-bot: Add Gather extension conf [integration/config] - 10https://gerrit.wikimedia.org/r/191009 (owner: 10Jhernandez) [15:33:54] 3Continuous-Integration, VisualEditor: Investigate Chrome disconnect failures when running MediaWiki tests on labs slaves - https://phabricator.wikimedia.org/T89075#1046426 (10Jdforrester-WMF) [15:34:06] (03CR) 10Hashar: "`jenkins-jobs test` just generate the XML configuration files but that does not check consistency with Zuul configuration :-)" [integration/config] - 10https://gerrit.wikimedia.org/r/191119 (https://phabricator.wikimedia.org/T89489) (owner: 10Dduvall) [15:35:31] 3Continuous-Integration, MediaWiki-extensions-WikibaseRepository, § Wikidata-Sprint-2015-02-03, Wikidata: Enable the CLDR extension for Wikibase Repo unit tests - https://phabricator.wikimedia.org/T87242#1046436 (10hoo) 5Open>3Resolved Also https://gerrit.wikimedia.org/r/190690 [15:36:45] (03CR) 10Hashar: [C: 031] "All good to me. You can get the job generated as described on https://www.mediawiki.org/wiki/CI/JJB" [integration/config] - 10https://gerrit.wikimedia.org/r/191119 (https://phabricator.wikimedia.org/T89489) (owner: 10Dduvall) [15:56:40] (03CR) 10Jhernandez: "Thanks hashar! :D" [integration/config] - 10https://gerrit.wikimedia.org/r/191009 (owner: 10Jhernandez) [16:07:03] <^d> twentyafterfour: fyi, that (increasingly noisy) /\$(left|right)_langcode/ bug in DoubleWiki is already fixed in master. It should go away with the next branch. [16:09:44] 3MediaWiki-extensions-GWToolset, Multimedia, Beta-Cluster: Creating directory with special characters - https://phabricator.wikimedia.org/T75725#1046604 (10Lena) Hi everyone. I'm sorry if this sounds harsh but I really need to say it. This bug has been opened three months ago, and prevents me from uploading hi... [16:14:00] !log scap clone on deployment-mediawiki02 corrupt; git fsck did not fix; will delete and refetch [16:14:04] Logged the message, Master [16:14:54] Project beta-scap-eqiad build #42273: FAILURE in 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/42273/ [16:15:59] !log updated scap to 7c64584 (Add universal argument to ignore ssh_auth_sock) [16:16:03] Logged the message, Master [16:16:49] !log 5 deleted instances in trebuchet redis cache for salt/salt repo [16:16:52] Logged the message, Master [16:16:58] <^d> bd808: `git fsck --full`? [16:17:22] <^d> Oh, "this is now default" [16:17:22] <^d> nvm [16:17:26] I just moved it to the side and refetched with `sudo salt-call deploy.fetch 'scap/scap'` [16:17:32] worked like a charm [16:17:44] not sure what corrupted the clone [16:17:57] fatal: loose object 44c6d2e770eaf5ccb3b1e24f877a00004186f0be (stored in .git/objects/44/c6d2e770eaf5ccb3b1e24f877a00004186f0be) is corrupt [16:18:07] all better now [16:18:16] <^d> Boo [16:18:19] <^d> But yay [16:18:40] If you want to poke, /srv/deployment/scap/scap-corrupt on mw02 [16:19:07] * ^d avoids the nerd snipe [16:19:19] but git! [16:19:44] <^d> ugh, and I'll end up spending half my morning learning something mildly interesting but not terribly applicable to the future :p [16:19:47] 3Continuous-Integration: MediaWiki installs in Jenkins frequently fail to access their sqlite database due to locks - https://phabricator.wikimedia.org/T89180#1046656 (10Aklapper) [16:20:19] *nod* that's why I just moved it over and started again [16:20:32] * bd808 does not have the "ops mindset" [16:20:45] I was wondering if one of you guys would be able to investigate https://phabricator.wikimedia.org/T89802 [16:20:56] I only want to know the root cause the 3rd time something breaks in the same way [16:21:02] <^d> Krenair: Oh yeah, that annoys me [16:21:13] I thought this was working okay just a few days ago [16:21:20] but it seems broken at the moment [16:21:23] <^d> It's always been like that for me [16:21:31] I've seen that on labs-vagrant instances too :( [16:21:34] 3Phabricator-Sprint-Extension, Continuous-Integration, Phabricator: Integrate Jenkins with Phabricator with Harbormaster - https://phabricator.wikimedia.org/T89714#1046668 (10Christopher) [16:21:34] modules/mediawiki/manifests/users.pp - that sudo::group block [16:22:41] did the sudo grants get updated for the change from the apache user to the www-data user? [16:23:54] 3Beta-Cluster: Can't run mwscript without explicit sudo on Beta Cluster - https://phabricator.wikimedia.org/T89802#1046690 (10greg) [16:23:55] !log redis-cli srem 'deploy:scap/scap:minions' i-0000059b.eqiad.wmflabs i-000007f8.eqiad.wmflabs i-0000022e.eqiad.wmflabs i-0000044e.eqiad.wmflabs i-000004ba.eqiad.wmflabs [16:23:57] Logged the message, Master [16:25:34] !log scap failing with "ImportError: cannot import name cli" after latest update; investigating [16:25:38] Logged the message, Master [16:26:36] krenair@deployment-bastion:~$ sudo -u www-data mwscript eval.php enwiki [16:26:36] > [16:26:38] ok so that works [16:26:51] !log scap failures only from deployment-videoscaler01 and deployment-rsync01 [16:26:55] Logged the message, Master [16:26:56] but it tries to default to apache, I guess [16:27:06] ah. that needs fixing [16:27:10] 3MediaWiki-extensions-GWToolset, Multimedia, Beta-Cluster: Creating directory with special characters - https://phabricator.wikimedia.org/T75725#1046712 (10Gilles) Just to give you a bit of perspective, #multimedia is staffed by 3 engineers. We have 20+ extensions to maintain and currently a backlog of approxim... [16:27:48] the runtime user was changed. Maybe beta does something weird with that script or maybe it's broken in prod too [16:28:34] I can run mwscript without sudo in production :) [16:29:08] needs fixing in ops/puppet -- https://github.com/wikimedia/operations-puppet/blob/4a547bc88ec83febc2d8a19f932c37881f0b4ed9/modules/scap/files/mwscript -- I bet terbium and tin didn't get the apache -> www-data change yet [16:29:20] ah. [16:29:24] but in beta it has been changed everwhere [16:29:27] probably [16:30:23] Krenair: Care to reopen and comment on https://phabricator.wikimedia.org/T78076 ? [16:30:50] * bd808 is fighting with scap [16:32:21] bd808, I think we can just update modules/scap/files/mwscript [16:32:36] 3MediaWiki-extensions-GWToolset, Multimedia, Beta-Cluster: Creating directory with special characters - https://phabricator.wikimedia.org/T75725#1046748 (10Lena) Gilles it's really ok that this bug will not be corrected any time soon. (Well no it's not but it's not your fault :) I understand the burden of unders... [16:32:41] to use www-data [16:32:43] instead of apache [16:35:25] *nod* but tin and terbium may need some fixes for that to work correctly [16:35:59] It works fine for me on tin [16:36:03] !log rebuilt corrupt deployment-rsync01:/srv/deployment/scap/scap [16:36:06] Logged the message, Master [16:36:29] Krenair: that implies that apache is still the correct user on tin [16:37:03] would using www-data cause issues on tin/terbium then? [16:38:01] probably. I bet the problem is that nobody in Ops thought about the mwscript use-case on misc servers and only fixed the mw hosts [16:38:52] 3Beta-Cluster: Can't run mwscript without explicit sudo on Beta Cluster - https://phabricator.wikimedia.org/T89802#1046823 (10Krenair) Might have been caused by T78076 Current workaround is to "sudo -u www-data mwscript" [16:39:19] kart_: I assume I should ping you now and that it's topical for this channel? [16:39:22] !log rebuilt corrupt deployment-videoscaler01:/srv/deployment/scap/scap [16:39:25] Logged the message, Master [16:39:41] * bd808 shakes fist at trebuchet [16:40:11] 3operations, Beta-Cluster: Make www-data the web-serving user (is currently apache) - https://phabricator.wikimedia.org/T78076#835083 (10Krenair) Some things like mwscript still use apache, and this is now broken on deployment-prep. Please see T89802 [16:40:19] thcipriani: ^^ see, trebuchet deploying scap on Beta Cluster (what bd808's doing) [16:40:39] * bd808 isn't here. nobody saw a PM doing this work [16:40:48] Yippee, build fixed! [16:40:48] Project beta-scap-eqiad build #42277: FIXED in 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/42277/ [16:40:55] I mean, that ghost of bd808 past [16:41:38] !log beta-scap-eqiad job fixed after manually rebuilding git clones of scap/scap on rsync01 and videoscaler01 [16:41:41] Logged the message, Master [16:41:51] that was the suck [16:42:16] 3 of 6 trebuchet target hosts ended up with corrupt checkouts [16:42:25] :( [16:42:38] * thcipriani is looking, reading [16:42:39] * bd808 kicks trebuchet in the creek [16:43:15] ref: http://www.youtube.com/watch?v=S0p9Q6nthzE [16:48:26] 3Beta-Cluster: Can't run mwscript without explicit sudo on Beta Cluster - https://phabricator.wikimedia.org/T89802#1046867 (10bd808) The current mwscript still tries to `sudo apache` and normal users no longer have that right in beta. The script should be changed in operations/puppet.git to `sudo www-data` inste... [16:55:50] greg-g: there's a risk that I might get called away at the last minute and miss Scrum-of-scrums. I think it's OK, we had a pretty quiet week. I'll leave an update on the etherpad ahead of time just in case. [16:58:02] chrismcmahon: kk [17:22:28] ^d: you've mail with question :) [17:22:40] but I've to go now. [17:22:44] night. [17:22:45] <^d> Yes, I haven't figured out answers to all of it yet. [17:22:52] <^d> You'll have a response today at some point :) [17:22:56] ^d: no worries. [17:23:04] ^d: Thanks! [17:23:10] <^d> yw. have a good night [17:23:18] :) [17:24:07] Yippee, build fixed! [17:24:07] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #491: FIXED in 16 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/491/ [17:29:06] 3Continuous-Integration, VisualEditor: Investigate Chrome disconnect failures when running MediaWiki tests on labs slaves - https://phabricator.wikimedia.org/T89075#1047036 (10Jdforrester-WMF) [17:30:07] do we have a staging project in labs? apparently I'm not added to it if it exists at all. [17:30:35] can someone add me, and thcipriani as well? [17:30:59] deployment-prep? [17:31:07] Krenair: different one :) [17:31:24] new-deployment-prep [17:31:31] different-deployment-prep [17:31:31] :) [17:31:35] or did we decide to just keep it in the existing project [17:31:37] https://wikitech.wikimedia.org/wiki/Nova_Resource:Staging [17:31:38] https://wikitech.wikimedia.org/wiki/Nova_Resource:Staging [17:31:41] bah [17:31:42] :P [17:31:56] yuuuuuviiiii! [17:32:01] yeah nobody added me [17:32:27] nor chad it looks like [17:32:51] yeah :-/ [17:33:55] <^d> I can add you [17:34:05] <^d> Oh wait [17:34:07] <^d> nvm [17:34:10] <^d> Yuvi rm me? [17:34:45] <^d> Oh I am, Nova resource page is just wrong [17:35:38] <^d> Added [17:38:09] if I want to update a wmf branch to current master (like 4 commits+TWN), should I just cherry-pick each one individually or is there a better way to do this? [17:38:27] <^d> You can just point the sha1 at master [17:38:40] <^d> That's what I do if I really do want master [17:39:35] <^d> twentyafterfour: I want to see if I can get our puppetmaster up in staging [17:40:15] ok [17:40:23] that's deployment-bastion currently, right? or no [17:40:31] <^d> deployment-salt [17:40:33] deployment-salt, i think [17:40:43] ah yes [17:40:54] brb... [17:43:37] <^d> Actually we probably need security groups first :p [17:44:21] greg-g: https://wikitech.wikimedia.org/w/index.php?title=MediaWiki:Sidebar&diff=143793&oldid=136410 -- could we rename it 'Server admin log (RelEng)' to make it, neater? :) [17:46:49] more consistent [17:50:12] (03CR) 10Dduvall: [C: 032] Additional documentation on upgrading to 1.0.0 [selenium] - 10https://gerrit.wikimedia.org/r/189832 (owner: 10Dduvall) [17:50:31] (03Merged) 10jenkins-bot: Additional documentation on upgrading to 1.0.0 [selenium] - 10https://gerrit.wikimedia.org/r/189832 (owner: 10Dduvall) [17:54:17] (03PS1) 10Dduvall: Release pre-release patch version 1.0.0.pre.2 [selenium] - 10https://gerrit.wikimedia.org/r/191365 [17:56:45] JohnLewis: :) sure [17:56:56] chrismcmahon: ^ (if you have a moment) [17:57:30] JohnLewis: {{done}} [17:57:41] greg-g: thanks :) [18:00:20] marxarelli: 35 rubocop offenses? [18:00:34] * chrismcmahon looks at the details... [18:03:13] chrismcmahon: yeah, we have to redo the rubocop fixes at some point [18:03:33] marxarelli: I'm happy merging that if you want [18:03:34] maybe before the actual 1.0.0 release [18:03:51] chrismcmahon: that would be rad [18:04:18] (03CR) 10Cmcmahon: [C: 032] Release pre-release patch version 1.0.0.pre.2 [selenium] - 10https://gerrit.wikimedia.org/r/191365 (owner: 10Dduvall) [18:04:32] (03Merged) 10jenkins-bot: Release pre-release patch version 1.0.0.pre.2 [selenium] - 10https://gerrit.wikimedia.org/r/191365 (owner: 10Dduvall) [18:06:04] chrismcmahon: thanks! [18:06:17] that was pretty easy :-) [18:14:40] anyone know how to escape braces in jjb? i tried '{{' but that seems to lead to infinite recursion... [18:15:32] twentyafterfour: so for the globaluserpage deploy today, https://gerrit.wikimedia.org/r/#/c/191360/ is a config change, then I'll have a 1.25wmf17 submodule bump which needs to happen before you scap, and then I'll have another config change afterwards to enable it everywhere [18:21:30] twentyafterfour: https://gerrit.wikimedia.org/r/191370 is the submodule bump [18:33:09] Yippee, build fixed! [18:33:09] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #489: FIXED in 12 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/489/ [18:41:13] (03PS1) 10Dduvall: Template for running `bundle exec` using Bundler 1.7 [integration/config] - 10https://gerrit.wikimedia.org/r/191375 [18:45:39] (03PS2) 10Dduvall: Template for running `bundle exec` using Bundler 1.7 [integration/config] - 10https://gerrit.wikimedia.org/r/191375 [18:46:09] 3Engineering-Community, Release-Engineering, Team-Practices: RelEng team offsite - https://phabricator.wikimedia.org/T89036#1047280 (10Rfarrand) a:3Rfarrand [18:47:00] (03PS4) 10Dduvall: Experimental cucumber job for mediawiki-vagrant [integration/config] - 10https://gerrit.wikimedia.org/r/191119 (https://phabricator.wikimedia.org/T89489) [18:47:46] (03CR) 10Dduvall: "Sorry, I had to refactor this to use the new bundle17 job (that avoids a gem version conflict on bundler)." [integration/config] - 10https://gerrit.wikimedia.org/r/191119 (https://phabricator.wikimedia.org/T89489) (owner: 10Dduvall) [18:50:44] (03CR) 10Dduvall: [C: 04-1] Template for running `bundle exec` using Bundler 1.7 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/191375 (owner: 10Dduvall) [19:02:57] <^d> Krenair: Did you have any thoughts on a fix for the RL image bug? [19:11:11] 3Continuous-Integration, Wikimedia-Fundraising-CiviCRM, § Fundraising Tech Backlog, § Fundraising Sprint E, Release-Engineering: Deploy CiviCRM integration job to WMF integration server - https://phabricator.wikimedia.org/T86374#1047381 (10atgo) Cool. Sounds good. I was going to say we should get all the blockin... [19:11:36] ^d, I wonder if it shouldn't be being executed [19:11:40] per matmarex [19:11:50] <^d> Oh duh, Krenair != MatmaRex [19:11:53] <^d> lol. [19:12:43] hm [19:13:00] it's checking the value of wgSVGConverter [19:13:04] for 'rsvg' [19:13:11] ^d: um, well, isn't the immediate fix not to try deleting the file if it doesn't exist? [19:13:15] but in prod we have 'rsvg-secure' [19:13:34] ^d: the second-level fix is to track down the extension which has images that don't convert to PNG [19:13:46] ^d: and the third-level is to rename this stupid rsvg-secure to just rsvg [19:14:20] but yes the obvious fix is to check the result of ->rasterize for whether it returned exactly true or not [19:14:54] ^d: and the fourth-level fix is to rewrite SvgHandler from ground up, so that we don't need this special case [19:15:08] (i filed a bug for that one some time ago) [19:15:15] if it's not true, return false [19:16:22] <^d> Ok, I've got an idea for a patch [19:17:33] 3Multimedia, MediaWiki-extensions-GWToolset, Beta-Cluster: Creating directory with special characters - https://phabricator.wikimedia.org/T75725#1047421 (10Tgr) https://phabricator.wikimedia.org/project/sprint/profile/1015/ has a nice priority/expected arrival chart, I wonder if it would make sense to do somethi... [19:18:41] 3Multimedia, MediaWiki-extensions-GWToolset, Beta-Cluster: Creating directory with special characters - https://phabricator.wikimedia.org/T75725#1047428 (10Tgr) As a workaround, you can try to register another user for this, although as Jean-Fred said, GWT does not seem to have any problem with `é` in usernames. [19:19:10] (03CR) 10Dduvall: Template for running `bundle exec` using Bundler 1.7 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/191375 (owner: 10Dduvall) [19:22:15] <^d> Krenair, MatmaRex: https://gerrit.wikimedia.org/r/191383 [19:29:09] ^d: can we put in some logging for when the conversion fails? [19:29:26] (in both the rsvg and non-rsvg branch) [19:29:38] Krinkle: when you have a moment https://gerrit.wikimedia.org/r/#/c/191375/ [19:29:52] i hope it's not just failing spuriously, and that there are some broken files somewhere that we can fix [19:30:14] <^d> Likely [19:31:07] ^d: can *you* put in some ligging? :D i never knew exactly how to do it right, and now i probably need to re-learn it [19:31:17] <^d> I can [19:31:22] <^d> Sec. [19:34:21] <^d> Amended [19:40:22] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:44:47] ^d: thanks, bonk me if anything gets logged, please [19:44:48] <^d> If we get that in soon enough we'll probably make the new branch point :) [19:45:48] bah, i might as well merge it, probably [19:49:28] <^d> Nope, we missed it [19:49:30] <^d> lol [19:51:01] 3Beta-Cluster: Don't throttle WMF office IP(s) for account creation - https://phabricator.wikimedia.org/T87841#1047531 (10Dzahn) Hi @cajoel, do you have the IP ranges for office handy? @JohnLewis would like to use them to implement this throttle. Thanks! [19:53:29] 3Beta-Cluster: Don't throttle WMF office IP(s) for account creation - https://phabricator.wikimedia.org/T87841#1047539 (10Dzahn) got it. hmm. @jkrauska < guillom> mutante: https://office.wikimedia.org/wiki/Office_IT/Network/IP_Addresses is this current and can i release it to John (NDAed volunteer)? [19:56:58] ugh... cloning all these submodules is slow and stupid :-/ [19:57:26] 3Beta-Cluster: Don't throttle WMF office IP(s) for account creation - https://phabricator.wikimedia.org/T87841#1047571 (10Dzahn) a:3JKrauska [19:57:54] twentyafterfour: {{sofixit}} that's the job of releng :p [19:59:23] ^d, are you checking that these new wikimedia-log-errors tasks have occurred in production recently? [19:59:30] <^d> No [19:59:52] <^d> I'm just tagging them all as that's what they were reported as. [19:59:59] 3Beta-Cluster: Don't throttle WMF office IP(s) for account creation - https://phabricator.wikimedia.org/T87841#1047616 (10Dzahn) https://wikitech.wikimedia.org/wiki/Tan.corp.wikimedia.org [20:00:03] <^d> Then I was planning on making a second pass and getting a status on each [20:00:13] <^d> (some are likely close-able) [20:03:19] 3Beta-Cluster: Don't throttle WMF office IP(s) for account creation - https://phabricator.wikimedia.org/T87841#1047663 (10JohnLewis) >>! In T87841#1047616, @Dzahn wrote: > https://wikitech.wikimedia.org/wiki/Tan.corp.wikimedia.org Yeah that I believe is what @greg wants to be throttled. [20:06:25] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #524: FAILURE in 18 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/524/ [20:07:55] JohnLewis: that's what I'm trying to do [20:08:48] 3Beta-Cluster: Don't throttle WMF office IP(s) for account creation - https://phabricator.wikimedia.org/T87841#1047742 (10Krenair) These all have to go in the public throttle.php file to avoid throttling anyway, so an NDA won't help here. [20:14:00] 3Engineering-Community, Release-Engineering, Team-Practices: RelEng team offsite - https://phabricator.wikimedia.org/T89036#1047819 (10Rfarrand) p:5Triage>3Normal [20:15:57] RIP Reedy's inbox. [20:16:42] <^d> All done [20:24:08] <^d> twentyafterfour: All those tasks are now tagged with log-errors. Now we won't file dupes :) [20:34:38] Project beta-scap-eqiad build #42303: FAILURE in 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/42303/ [20:40:02] <^d> !logerrors [20:40:11] <^d> !logerrors is https://phabricator.wikimedia.org/maniphest/query/3nBU5bwR5HqG/#R [20:40:11] Key was added [20:55:56] Yippee, build fixed! [20:55:57] Project beta-scap-eqiad build #42305: FIXED in 1 min 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/42305/ [20:59:24] mw1235: #012Notice: Undefined property: stdClass::$ipb_timestamp in /srv/mediawiki/php-1.25wmf16/includes/api/ApiQueryBlocks.php on line 188 [20:59:26] I wonder what that one is [21:03:01] oh right [21:03:02] $this->addFieldsIf( 'ipb_timestamp', $fld_timestamp ); [21:03:57] if you don't set the timestamp parameter, you get this notice instead of valid continuation data? [21:12:48] hm nope [21:12:52] it's not that simple [21:23:21] ah timestamp prop is one of the ones set by default [21:23:28] default: id|user|by|timestamp|expiry|reason|flags [21:24:19] so set a bkprop that doesn't include it and you get that notice [21:29:33] filed https://phabricator.wikimedia.org/T89893#1048100 [21:35:52] 3Continuous-Integration, Release-Engineering, § Fundraising Tech Backlog, Wikimedia-Fundraising-CiviCRM, § Fundraising Sprint E: Configure Jenkins to run CiviCRM builds only on Fundraising CI slaves - https://phabricator.wikimedia.org/T89895#1048130 (10awight) 3NEW a:3awight [21:37:07] (03CR) 10Hashar: [C: 04-1] "The JJB diff shows that bundler-version ends up being 'None' instead of '' see inline diff." (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/191375 (owner: 10Dduvall) [21:37:57] (03CR) 10Hashar: [C: 031] "Excellent rebase :}" [integration/config] - 10https://gerrit.wikimedia.org/r/191119 (https://phabricator.wikimedia.org/T89489) (owner: 10Dduvall) [21:38:31] marxarelli: quite happy to see you handle JJB conf quite well :] [21:41:54] (03CR) 10Dduvall: "> I guess we can just send a pull request to bump the requirement upper limit for bundler from < 1.8.0 to < 1.9.0 ? If it passes their tes" [integration/config] - 10https://gerrit.wikimedia.org/r/191375 (owner: 10Dduvall) [21:43:05] hashar: oh, thanks. Krinkle pointed me in the right direction and now i'm starting to understand the different components [21:43:29] 3Continuous-Integration, Release-Engineering, § Fundraising Tech Backlog, Wikimedia-Fundraising-CiviCRM, § Fundraising Sprint E: Run CiviCRM testing scripts during CI - https://phabricator.wikimedia.org/T89896#1048160 (10awight) 3NEW a:3awight [21:43:45] (03CR) 10Hashar: "I have filled a pull request to bump bundler: https://github.com/mitchellh/vagrant/pull/5363" [integration/config] - 10https://gerrit.wikimedia.org/r/191375 (owner: 10Dduvall) [21:43:57] marxarelli: yeah it takes a while to get used to [21:44:08] it is not that hard overall though [21:44:17] marxarelli: Yep, takes ~ 6 minutes to run for me locally. Mostly because of mediawiki-extensions.yaml inflating the number of jobs. [21:44:26] :-/ [21:44:29] You can local git rm that file to speed it up if you don't needit. [21:44:47] RE: "I guess `jenkins-jobs test` just takes longer than I expected." [21:45:05] Krinkle: right :) [21:45:18] hashar: What's our status of things? [21:45:32] blockers / currently in the works from your end [21:46:08] in the middle of unpacking my stuff [21:46:11] I relocated over the week-end [21:46:40] on a more serious note: get zuul / nodepool packaged [21:46:50] and set up a meeting with some ops folks to present the ci thing [21:46:57] the ci isolation thing [21:47:04] marxarelli: I am not sure why JJB yields 'None' [21:47:18] marxarelli: it is supposed to complain it is missing a parameter to format the string [21:47:35] or sometime just carbon copy {bundler-version} without expanding it. :-/ [21:47:40] hashar: perhaps it's because i did `bundler-version: ~` [21:47:45] which is yaml for nil/null [21:47:59] OH [21:48:11] yaml keeps surprising me :-] [21:48:23] hashar: i can just change it to an empty string [21:48:37] I though the tilde was some gem syntax to say "take whatever version" [21:48:58] yeah an empty string is probably safer [21:49:15] s/safer/less confusing to the newbie yamler I am/ [21:49:42] hashar: probably less confusing all around given the script checks for an empty string :) [21:50:06] Krinkle: will try to get Zuul packaged this week [21:51:53] marxarelli: and for the context, timo has as much knowledge as me regarding CI. We are twins as far as CI is concerned [21:52:09] 3Continuous-Integration, § Fundraising Sprint E, § Fundraising Tech Backlog, Wikimedia-Fundraising-CiviCRM, Release-Engineering: Create CI slave instance for CiviCRM testing - https://phabricator.wikimedia.org/T89894#1048189 (10atgo) Hey @awight why is this in all of the past sprints? [21:52:24] Krinkle: Ping re https://gerrit.wikimedia.org/r/#/c/191063/ BTW. :-) [21:52:41] and I am pairing with Zeljkof 3 times per week to bring up on par [21:53:32] James_F: OK. Looks good now. Why jslint and npm both in test/gate pipeline though? [21:53:41] copy paste! [21:53:42] 3Continuous-Integration, § Fundraising Sprint E, § Fundraising Tech Backlog, Wikimedia-Fundraising-CiviCRM, Release-Engineering: Create CI slave instance for CiviCRM testing - https://phabricator.wikimedia.org/T89894#1048191 (10awight) [21:53:57] you can apply the template extension-unittests then add the jobs you need [21:54:25] Krinkle: Because you told me to copy-paste the template and then went away. [21:54:51] James_F: Right :) [21:55:08] Krinkle: And I always faithfully do what you say. :-) [21:55:14] hashar: Krinkle said not. [21:55:30] James_F: Yeah, it's come full circle [21:55:40] :-P [21:56:06] Krinkle: Also https://gerrit.wikimedia.org/r/#/c/191064/ whilst you're at it. [21:56:07] ah [21:56:07] (03CR) 10Hashar: Add npm test for Citoid extension (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/191063 (owner: 10Jforrester) [21:56:09] James_F: You wanted to add npm and have it replace jslint, so in order to undo jslint, one has to subsitute the template as you can't exclude afterwards. [21:56:18] James_F: yeah follow whatever Krinkle says :-] [21:56:25] 3Continuous-Integration, § Fundraising Sprint E, § Fundraising Tech Backlog, Wikimedia-Fundraising-CiviCRM, Release-Engineering: Configure Jenkins to run CiviCRM builds only on Fundraising CI slaves - https://phabricator.wikimedia.org/T89895#1048206 (10awight) [21:56:26] hashar: Lies! He lies! ;-) [21:56:28] James_F: but then including everything makes the template subtitutation no longer useful [21:56:34] though I prefer reusing the templates [21:56:48] that is less maintenance whenever we want to refactor what is being run on all extensions [21:56:54] hashar: templates need updating for that to work [21:56:56] I commented on it [21:56:59] Krinkle: What would be left on check if I remove the lints? [21:57:04] Krinkle: Nothing at all? [21:57:11] James_F: Don't remove the lint from check [21:57:17] (or anywhere) [21:57:30] Krinkle: Oh, just from test? So a job that runs for one pipeline but not the other. Eww. [21:57:31] ahhh yeah the lint checks are superseded by the npm command [21:57:34] hashar: I have a WIP to fix the template to be more flexible [21:57:39] we had that conversation earlier. So yeah follow Krinkle advice ::] [21:57:44] Krinkle: awesome [21:57:45] Kk. [21:58:09] James_F: jslint is a quick job to use a global jshint as an approximation for quick feedback [21:58:14] to new users [21:58:23] since we can't execute package.json there [21:58:37] (03PS3) 10Dduvall: Template for running `bundle exec` using Bundler 1.7 [integration/config] - 10https://gerrit.wikimedia.org/r/191375 [21:58:57] Krinkle: So remove mwext-Citoid-jslint but not from check? Remove mwext-Citoid-lint too? [21:59:05] James_F: Nope, lint=phplint [21:59:10] Krinkle: If only hashar's work on VMs was done. ;-) [21:59:14] unless we use composer, keep that one :) [21:59:16] Oh. Cool naming. [21:59:28] James_F: It's renamed elsewhere. Not for mwext. [21:59:41] James_F: If you have 6 hours to recompile and push all extension jobs, it's only one line of code to change it. [21:59:46] Krinkle: :-P [21:59:53] Krinkle: That's what I have you for. [21:59:56] Usually, something breaks at least once so I can't run it under screen. [22:00:20] There. [22:00:22] (03PS4) 10Jforrester: Add npm test for Citoid extension [integration/config] - 10https://gerrit.wikimedia.org/r/191063 [22:00:33] jslint removed. No other changes. [22:00:44] hashar: Maybe you can do a direct change on gallium once to do the lint/phplint rename? [22:00:57] Krinkle: can we repool integration-slave1009 ? [22:01:14] hashar: I'll look into it later tonight. I left some stuff in a rough state [22:01:19] hashar: See the phab ticket [22:01:37] Krinkle: yeah the underlying hardware died yesterday. Maybe it is easier to recreate it [22:01:48] for rename hmm yeah [22:01:51] hashar: No, I took it down manually [22:02:01] okk [22:02:03] hashar: see log :) [22:02:08] !sal [22:02:08] https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:02:54] Krinkle: ah yeah all the sqlite craziness. I am happy to see Tim helping on that front [22:03:08] hashar: short story: sqlite for trusty is shit. We can't do any concurrent requests. Immediate error for db is lock, no waiting. [22:03:19] :-( [22:03:41] which means unit tests often fail to populate resourceloader tables [22:03:46] in load.php requests [22:03:48] and random other tuff [22:03:51] stuff* [22:03:59] * hashar blames RL [22:04:02] :-D [22:04:30] poor JJB is so slow [22:04:45] almost 7 minutes to generate two versions of our jobs [22:04:52] hashar: I'm literally 24 layers deep into this. It started with: I think it's caused by karma or chrome (instead of qunit/phantomjs). But it's really caused by the difference between prod precise and labs trusty. [22:05:08] And now in cpp code from sqlite [22:05:16] Which is where the culprit was [22:05:26] maybe it is time to dish out sqlite and use mysql as a backend ? [22:05:37] we should be able to setup a user/db for test run [22:06:08] and tear the mysql user/db down on job completion [22:06:35] hashar: Let's evaluate our options. Getting unit tests from phantomjs to chrome is a high priority because it is a fundamental blocker for a lot of infrastructure (e.g. karma as wrapper, but also to use newer browser features, which means half of VE can't be tested right now). [22:06:51] hashar: we can downgrade sqlite, we can change mediawiki core sqlite, we can switch to mysql [22:07:10] hashar: or 4) switch to sqlite memory tables, or 5) reduce apache concurrency [22:07:19] What is most feasible for short term? [22:07:29] actually sqlite downgrade won't work probably [22:07:38] taht leaves 4 options [22:08:01] hashar: yeah, postbuildscript can do mw-teardown and mysqladmin to drop it [22:08:48] hashar: by the way (stack.pop.branch.previous), I've cleaned up phpcs and removed gzip/trap logic as it was making it hard to migrate existing jobs. [22:10:55] https://github.com/wikimedia/integration-config/commit/a477214fc1 https://github.com/wikimedia/integration-jenkins/commit/5a157a4 [22:11:49] bah [22:11:53] jenkins is in czech [22:12:36] (03CR) 10Hashar: [C: 031] "So that looks fine to me. Can you try refreshing one of the job having no bundler version specified and check it still works? If so good" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/191375 (owner: 10Dduvall) [22:12:54] MatmaRex: blame andre__ :-D [22:13:02] 3Deployment-Systems: l10nupdate user can't access scap shared ssh key causing nightly l10nupdate sync process to fail - https://phabricator.wikimedia.org/T76061#1048238 (10bd808) Both the scap and l10nupdate changes are deployed in production now (and we didn't break scap for normal deployers this time!). Lets s... [22:13:21] !log saving Jenkins configuration at https://integration.wikimedia.org/ci/configure to reset the locale [22:13:26] Logged the message, Master [22:13:32] hashar: i actually blame petr and his wikitech thread. :P [22:13:38] MatmaRex: fixed! [22:14:40] Krinkle: phpcs never really received much attention :/ [22:14:54] Krinkle: I was hoping for folks to start fixing errors first then warnings hence the two jobs [22:25:28] hashar: Yeah, but I think that works counter productive. People either fix it or they don't. Whatever makes people not interested is not caused by the few extra warnings. I've done this plenty of times before. [22:27:25] Krinkle: yeah that is fair. I am still naive [22:27:25] 3Beta-Cluster: Don't throttle WMF office IP(s) for account creation - https://phabricator.wikimedia.org/T87841#1048292 (10Dzahn) @JohnLewis: quote "As of summer 2014, the office network has been migrated to a new /24 block. 198.73.209.0/24 ^ there, that /24 it is [22:27:39] (03CR) 10Dduvall: [C: 032] Template for running `bundle exec` using Bundler 1.7 [integration/config] - 10https://gerrit.wikimedia.org/r/191375 (owner: 10Dduvall) [22:28:33] 3Beta-Cluster: Don't throttle WMF office IP(s) for account creation - https://phabricator.wikimedia.org/T87841#1048294 (10Dzahn) a:5JKrauska>3None [22:30:08] 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Make labs/private really private - https://phabricator.wikimedia.org/T89642#1048301 (10Krenair) [22:33:28] bed time. Sleep well [22:33:57] (03CR) 10jenkins-bot: [V: 04-1] Template for running `bundle exec` using Bundler 1.7 [integration/config] - 10https://gerrit.wikimedia.org/r/191375 (owner: 10Dduvall) [22:34:08] agrhg [22:34:33] marxarelli: the job integration-zuul-layoutvalidation-gate only run on +2 [22:34:44] it verify whether the jobs defined are actually deployed on Jenkins [22:34:52] and looking at https://integration.wikimedia.org/ci/job/integration-zuul-layoutvalidation-gate/225/console [22:34:58] 00:00:07.223 Job mediawiki-vagrant-bundle17-rspec not defined [22:34:59] 00:00:07.224 Job mediawiki-vagrant-bundle17-rubocop not defined [22:34:59] :D [22:35:10] you want to jenkins-jobs update [22:35:33] hashar: ah, crap. ok [22:35:43] such a pity it takes so long [22:36:34] for the jjb diff [22:36:43] we can probably get the two run to run in parallel [22:37:13] parent & ; proposed &; wait [22:37:17] might just work [22:39:14] hashar: Addshore and I proposed https://phabricator.wikimedia.org/T89682 as a GCI project [22:39:26] hashar: so `jenkins-jobs update` should be done before +2? [22:41:48] marxarelli: tyeah [22:42:07] marxarelli: and since it is super long, pass the jobs you want to updated as parameters. wildcard accepted [22:42:22] ie: jenkins-jobs update '*-bundle*' [22:42:45] !log running `jenkins-jobs update` to create mediawiki-vagrant-bundle17 jobs [22:42:46] until one figure out how to craft a job that will refresh the jobs on +2 [22:42:50] Logged the message, Master [22:43:29] i always look at the integration-jjb-config-diff output [22:43:34] ie https://integration.wikimedia.org/ci/job/integration-jjb-config-diff/2185/consoleFull [22:43:44] somewhere at the top is the list of modified jobs [22:43:59] here all *-bundle* jobs [22:44:14] + the bundle17 ones createn [22:44:17] created [22:44:51] i checked the diff, albeit with an untrained eye :) [22:45:47] (03CR) 10Dduvall: Template for running `bundle exec` using Bundler 1.7 [integration/config] - 10https://gerrit.wikimedia.org/r/191375 (owner: 10Dduvall) [22:45:57] (03CR) 10Dduvall: [C: 032] Template for running `bundle exec` using Bundler 1.7 [integration/config] - 10https://gerrit.wikimedia.org/r/191375 (owner: 10Dduvall) [22:46:27] the diff is a bit messy [22:46:39] should work out something nice to show the jobs being added / the one deleted and the one moddified [22:46:45] possibly with different colors [22:52:32] (03Merged) 10jenkins-bot: Template for running `bundle exec` using Bundler 1.7 [integration/config] - 10https://gerrit.wikimedia.org/r/191375 (owner: 10Dduvall) [22:55:00] greg-g: https://gerrit.wikimedia.org/r/#/c/191489/ -- take a look at that please [22:56:03] !log Reloading Zuul to deploy I3b71f4dc484d5f9ac034dc1050faf3ba6f321752 [22:56:08] Logged the message, Master [22:57:35] \O/ [22:57:36] (03PS3) 10Mattflaschen: VectorBeta depends on EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/191262 [22:57:48] marxarelli: https://integration.wikimedia.org/zuul/ is happy :-] [22:58:03] hashar: booya [22:58:18] * hashar awards marxarelli the "CI deployer badge" [22:59:06] so now you should 'recheck' a mediawiki/vagrant change and it should triggers the bundle17 jobs [22:59:37] poor bundle17 [22:59:53] Illformed requirement [">= 1.7.13, < 1.8.0"] [23:00:00] yeah hmm I haven't actually reviewed that part [23:00:36] argh. [23:00:44] i tested that locally [23:01:02] i'll have to try with an older version of `gem` [23:01:03] note how the requirement is passed as a single arg [23:01:04] ">= 1.7.13, < 1.8.0" [23:01:17] maybe it needs to be two args? [23:01:30] ">= 1.7.13" "< 1.8.0" [23:02:01] the single arg worked with rubygems 2.4.3 (ruby 2.1) [23:02:39] you can probably give it a try on the instance that ran the job [23:02:39] integration-slave1005.eqiad.wmflabs [23:03:23] hashar@integration-slave1005:/tmp$ gem2.0 install --env-shebang -i vendor --no-document -v '>= 1.7.13, <1.8.0' bundler [23:03:24] ERROR: While executing gem ... (Gem::Requirement::BadRequirementError) [23:03:24] Illformed requirement [">= 1.7.13, <1.8.0"] [23:03:33] we have gem 2.0.14 [23:04:57] looks like the single '< 1.8.0' argument works. that should be sufficient [23:05:59] yup [23:06:22] you will just have to refresh the bundler17 jobs [23:06:31] i.e. jenkins-jobs update '*bundler17*' [23:07:23] 3Beta-Cluster: Don't throttle WMF office IP(s) for account creation - https://phabricator.wikimedia.org/T87841#1048373 (10JohnLewis) a:3JohnLewis [23:07:42] marxarelli: and usually you can +2 your own changes if you know what you are doing :] [23:08:10] most of the trivial changes are not worth waiting a review [23:08:22] though we can change that once we have enough trained people [23:11:34] (03PS1) 10Dduvall: Fix Bundler version in bundle17 job template [integration/config] - 10https://gerrit.wikimedia.org/r/191492 [23:12:43] (03CR) 10Hashar: [C: 031] Fix Bundler version in bundle17 job template [integration/config] - 10https://gerrit.wikimedia.org/r/191492 (owner: 10Dduvall) [23:12:58] marxarelli: having that patch locally, you can refresh the jobs and retrigger them [23:13:14] amend patch locally / refresh jobs until it works as expected [23:13:25] hashar: right. i'm running `jenkins-jobs test` now [23:13:41] well you can just run update :] [23:13:46] then `jenkins-jobs update`, then rebuild the failed job [23:13:48] then head to the Jenkins job page at https://integration.wikimedia.org/ci/job/mediawiki-vagrant-bundle17-rspec/1/console [23:14:01] and hit [Rebuild] [23:14:02] hashar: oh, i thought it was required to run test before update [23:14:10] well in this case [23:14:14] to compile everything into output/ [23:14:14] the impact is low [23:14:30] update first compile the configuration and will thus bail out if something is wrong [23:14:38] oh, ok [23:14:40] good to know [23:14:52] so you can push your experimental job directly [23:15:04] each job has a configuration history [23:15:04] from https://integration.wikimedia.org/ci/job/mediawiki-vagrant-bundle17-rspec/ [23:15:12] on the left side is [Job Config History] [23:15:28] which currently show that you have created the job [23:15:40] once you update it with your local patch, there will be a new entry and [23:15:44] *rolls the drums* [23:15:48] you can diff! [23:15:50] !log Running `jenkins-jobs update` to update mediawiki-vagrant-bundle17 jobs [23:15:55] Logged the message, Master [23:16:13] just make sure that whenever you update jobs, you do not forget to send the patch to gerrit and have it merged in [23:16:33] another trick [23:16:47] you can delete locally jjb/mediawiki-extensions.yaml and jjb/wikidata.yaml [23:17:04] this way JJB will process the files way faster since that cut 80% or so of the jobs to generate [23:17:14] makes the test or update runs wayyyy faster [23:17:24] just have to remember to not commit -a the deletions :] [23:19:26] * hashar waits [23:19:49] "Your bundle is complete!" [23:19:56] SUCCESS [23:20:07] \o/ https://integration.wikimedia.org/ci/job/mediawiki-vagrant-bundle17-rspec/3/console [23:20:20] you want to rebuild the other one to be sure [23:20:20] https://integration.wikimedia.org/ci/job/mediawiki-vagrant-bundle17-rubocop/ [23:20:39] hashar: since there was no change to the layout.yaml, do i still have to deploy it on gallium? [23:20:39] or just "check experimental" :-] [23:20:48] when testing/experimenting I use the Jenkins web interface directly [23:20:58] na don't bother deploying on gallium [23:21:00] I never do it [23:21:17] on gallium I do a git pull [23:21:20] err [23:21:24] a git remote update [23:21:32] then : git log HEAD..origin/master -- zuul [23:21:42] which dumps the log of changes in the /zuul/ subdir [23:21:50] i.e. ignoring any changes that only impacted JJB [23:22:10] feel free to pull on gallium though [23:22:24] rubocop will fail for mw-vagrant [23:22:40] i'll make sure it fails correctly though [23:22:49] at least all the other steps pass [23:22:57] 3Continuous-Integration, Wikimedia-Fundraising-CiviCRM, § Fundraising Tech Backlog, Release-Engineering: Deploy CiviCRM integration job to WMF integration server - https://phabricator.wikimedia.org/T86374#1048438 (10atgo) [23:24:08] (03CR) 10Dduvall: [C: 032] Fix Bundler version in bundle17 job template [integration/config] - 10https://gerrit.wikimedia.org/r/191492 (owner: 10Dduvall) [23:25:00] so now [23:25:16] given rspec pass [23:25:22] you can probably get the patch merged https://gerrit.wikimedia.org/r/#/c/190586/ [23:25:42] and promote the rspec job from 'experimental' to the 'test' and 'gate-and-submit' pipelines [23:25:50] (or promote first, merge the patch after) [23:26:16] hurray! thanks for the help hashar [23:26:17] 3Continuous-Integration, § Fundraising Sprint E, § Fundraising Tech Backlog, Wikimedia-Fundraising-CiviCRM, Release-Engineering: Run CiviCRM testing scripts during CI - https://phabricator.wikimedia.org/T89896#1048160 (10awight) [23:26:24] you are all set and cover 95% of the requests we have to handle for CI configuration :]]] [23:26:32] 3Continuous-Integration, Wikimedia-Fundraising-CiviCRM, § Fundraising Tech Backlog, § Fundraising Sprint E, Release-Engineering: Configure Jenkins to run CiviCRM builds only on Fundraising CI slaves - https://phabricator.wikimedia.org/T89895#1048130 (10awight) [23:26:41] 3Continuous-Integration, Wikimedia-Fundraising-CiviCRM, § Fundraising Tech Backlog, § Fundraising Sprint E, Release-Engineering: Create CI slave instance for CiviCRM testing - https://phabricator.wikimedia.org/T89894#1048122 (10awight) [23:26:56] it is a bit tedious though. Any suggestion for improvement is most welcome [23:27:34] when promoting the rspec job, you will only change the zuul/layout.yaml config file [23:27:42] hashar: waiting on jenkins-jobs is the most painful part [23:27:44] so the jenkins-job diff job is not going to run [23:28:03] yeah jjb is slow :-/ Need to get less jobs [23:28:08] or figuring out a speed up [23:30:26] (03Merged) 10jenkins-bot: Fix Bundler version in bundle17 job template [integration/config] - 10https://gerrit.wikimedia.org/r/191492 (owner: 10Dduvall) [23:31:52] congratulations :] [23:33:27] (03PS5) 10Dduvall: Experimental cucumber job for mediawiki-vagrant [integration/config] - 10https://gerrit.wikimedia.org/r/191119 (https://phabricator.wikimedia.org/T89489) [23:34:28] midnight permission is overdue. Cinderella becomes a country girl again [23:36:08] (03CR) 10Hashar: "Since the bundle17 macro got changed between PS4 and PS5 parent commits, you want to refresh the mediawiki-vagrant-bundle17-cucumber job :" [integration/config] - 10https://gerrit.wikimedia.org/r/191119 (https://phabricator.wikimedia.org/T89489) (owner: 10Dduvall) [23:36:52] * hashar waves [23:37:33] !log Running `jenkins-jobs update` to create mediawiki-vagrant-bundle17-cucumber job [23:37:36] Logged the message, Master [23:41:11] (03CR) 10Dduvall: [C: 032] Experimental cucumber job for mediawiki-vagrant [integration/config] - 10https://gerrit.wikimedia.org/r/191119 (https://phabricator.wikimedia.org/T89489) (owner: 10Dduvall) [23:47:43] (03Merged) 10jenkins-bot: Experimental cucumber job for mediawiki-vagrant [integration/config] - 10https://gerrit.wikimedia.org/r/191119 (https://phabricator.wikimedia.org/T89489) (owner: 10Dduvall) [23:50:09] !log Reloading Zuul to deploy Id311d632e5032ed153277ccc9575773c0c8f30f1 [23:50:13] Logged the message, Master