[00:03:36] (03PS1) 10Legoktm: Use --no-check-publish in php-composer-validate, create php-composer-validate-package [integration/config] - 10https://gerrit.wikimedia.org/r/194421 (https://phabricator.wikimedia.org/T91176) [00:03:59] (03PS2) 10Legoktm: Use --no-check-publish in php-composer-validate, create php-composer-package-validate [integration/config] - 10https://gerrit.wikimedia.org/r/194421 (https://phabricator.wikimedia.org/T91176) [00:05:49] (03PS3) 10Legoktm: Use --no-check-publish in php-composer-validate, create php-composer-package-validate [integration/config] - 10https://gerrit.wikimedia.org/r/194421 (https://phabricator.wikimedia.org/T91176) [00:06:05] 10Beta-Cluster, 7Jenkins, 5Patch-For-Review: Announce beta-scap-eqiad failures in -qa every time - https://phabricator.wikimedia.org/T84947#1090775 (10greg) 5Open>3Resolved [00:06:13] 10Beta-Cluster, 7Jenkins: Announce beta-scap-eqiad failures in -qa every time - https://phabricator.wikimedia.org/T84947#934379 (10greg) [00:11:07] (03PS4) 10Legoktm: Use --no-check-publish in php-composer-validate, create php-composer-package-validate [integration/config] - 10https://gerrit.wikimedia.org/r/194421 (https://phabricator.wikimedia.org/T91176) [00:13:37] (03PS5) 10Legoktm: Use --no-check-publish in php-composer-validate, create php-composer-package-validate [integration/config] - 10https://gerrit.wikimedia.org/r/194421 (https://phabricator.wikimedia.org/T91176) [00:13:42] really? [00:13:48] 79 character max length?? [00:15:39] (03PS1) 10Legoktm: Have "tox" run flake8 by default [integration/config] - 10https://gerrit.wikimedia.org/r/194424 [00:15:54] legoktm: PEP-8 RULEZ!!!! [00:15:55] 6Release-Engineering, 10Staging, 3releng-201415-Q3: [Quarterly Success Metric] By team test history - https://phabricator.wikimedia.org/T88706#1090811 (10dduvall) [00:16:39] bd808: but PEP8 says 80! [00:17:09] also GvR has said he's fine with 100! [00:17:13] Limit all lines to a maximum of 79 characters. [00:17:17] https://www.python.org/dev/peps/pep-0008/#maximum-line-length [00:17:35] greg-g: let me know if there's anything missing from what we discussed earlier https://phabricator.wikimedia.org/T88706 [00:18:00] o.O [00:18:02] I thought it was 80 [00:18:15] "it is okay to increase the nominal line length from 80 to 100 characters" [00:18:20] i think it's 80 includes the newline [00:18:27] *including* [00:18:39] * bd808 disagrees with >80 [00:18:58] I live in an 80 column shell [00:19:07] and I will continue to do so [00:19:49] I live in an 80 column shell so it's obvious when I go over 80 and other people will yell at me. [00:20:25] * ^d set his editor to show rulers at 80 and 100 columns [00:20:33] <^d> 80 is what I shoot for, 100 still passes phpcs [00:23:51] marxarelli: looks great [00:24:57] greg-g: i kind of made some assumptions about our logging setup in the Standalone Dashboard option. who can i ask about logstash/graphite logging/querying? [00:25:04] (03CR) 10Legoktm: [C: 032] Use --no-check-publish in php-composer-validate, create php-composer-package-validate [integration/config] - 10https://gerrit.wikimedia.org/r/194421 (https://phabricator.wikimedia.org/T91176) (owner: 10Legoktm) [00:25:04] * marxarelli looks at bd808 [00:25:32] * marxarelli can hardly see him through his 80 column window [00:25:42] * bd808 looks for someone assigned to logging infrastructure [00:25:49] * legoktm hands bd808 a mirror [00:25:54] zero boogs found [00:26:15] marxarelli: what do you need to know more about? [00:26:39] graphite I know little about here [00:26:49] bd808: would it be possible to submit structured cucumber logs (json) to logstash and query it later for presentation on a dashboard? [00:27:12] as long as you only want 30 days of history, yes [00:27:25] doh! [00:27:30] we only keep a 30 day rolling window of logstash data [00:27:42] based on our log retention policy [00:27:55] the policy would actually allow 90 days [00:27:57] hmm, did my +2 on https://gerrit.wikimedia.org/r/#/c/194424/ not show up in here? [00:28:12] but operational logs are generally stale after 14 days anyway [00:28:31] 90 days might be ok [00:28:40] legoktm: [16:22] (CR) Legoktm: [V: 2] Update to dev-master#05e196893b1225898de280ef8f97d5f2be684e8f [00:28:47] different patch [00:29:21] marxarelli: I don't really want to up the retention in beta for a random reporting use case [00:29:35] but I can't stop it I suppose [00:29:47] it wouldn't be high volume [00:29:48] hmm, zuul didn't pick it up either? I think gerrit barfed [00:30:07] (03CR) 10Legoktm: [C: 032] Have "tox" run flake8 by default [integration/config] - 10https://gerrit.wikimedia.org/r/194424 (owner: 10Legoktm) [00:30:08] well it would be all or nothing if import via logstash [00:30:15] better [00:30:20] i see [00:30:31] we don't create separate indexes by log event type [00:30:59] Now if what you are really asking is if you can store things in elasticsearch... that's a different question [00:31:33] (03Merged) 10jenkins-bot: Use --no-check-publish in php-composer-validate, create php-composer-package-validate [integration/config] - 10https://gerrit.wikimedia.org/r/194421 (https://phabricator.wikimedia.org/T91176) (owner: 10Legoktm) [00:31:35] (03Merged) 10jenkins-bot: Have "tox" run flake8 by default [integration/config] - 10https://gerrit.wikimedia.org/r/194424 (owner: 10Legoktm) [00:32:01] bd808: oh, maybe. basically, i want to store browser test metrics and results outside of jenkins so as to build a browser-test dashboard [00:32:11] *nod* [00:32:24] That sounds different than operational logging [00:32:40] bd808: you think elasticsearch would be more appropriate? [00:32:52] setting up a single node elasticsearch for something like that would be trivial [00:33:19] there's a puppet class, apply it, profit! [00:33:30] ooooh, i like profit [00:33:53] are example you can think of that i might check out? [00:33:57] *any* example [00:34:11] !log deployed https://gerrit.wikimedia.org/r/194421 [00:34:12] of loading data into elasticsearch? [00:34:15] Logged the message, Master [00:34:48] bd808: a puppet role that sets it up, yeah [00:35:14] or i can ack and get off your back [00:35:20] :) [00:35:52] https://github.com/wikimedia/operations-puppet/blob/production/manifests/role/logstash.pp#L30-L41 [00:36:38] There is some cruft there with the trebuchet provider too (17-19) [00:37:00] <^d> Also the phab role [00:37:08] sending data in is just a PUT over HTTP. [00:37:09] <^d> (for labs) [00:37:55] awesome. this helps a lot! [00:37:59] bd808: thanks! [00:38:28] <^d> bd808: What if you try to POST a document and give it an ID but it clashes with an existing document? [00:38:35] <^d> Blow up? Or just ignore your ID? [00:39:25] 10Continuous-Integration, 10MediaWiki-Codesniffer, 5Patch-For-Review: Convert existing legacy phpcs jobs to use composer entry point + versioning - https://phabricator.wikimedia.org/T90943#1090860 (10Legoktm) [00:39:25] 10Continuous-Integration, 5Patch-For-Review: Create composer validate --no-check-publish job for MediaWiki extensions - https://phabricator.wikimedia.org/T91176#1090858 (10Legoktm) 5Open>3Resolved a:3Legoktm [00:46:00] 6Release-Engineering, 10Staging, 3releng-201415-Q3: [Quarterly Success Metric] By team test history - https://phabricator.wikimedia.org/T88706#1090866 (10dduvall) [00:50:01] ^d: hmm? [00:59:46] I don't remember if it overwrites by default or not [01:04:19] looks like the easiest way to deal with that is to not specify an id at all and let one be auto generated. If you specify ids then you need to use version numbers to decide what happens [01:21:38] <^d> bd808: We use page id as the doc id :) [01:22:15] Oh yeah for content like that it's the only way to go [01:22:42] Do you send indexing time as the version number? [01:22:44] <^d> It works for that because it lets us do partial updates without having to track the id or look it up [01:22:59] <^d> I think we let _version_ default [01:23:37] <^d> But rev_timestamp is a good idea there, hmm [01:23:45] external version is kind of neat. You can use it as a vector clock for eventual consistency [01:24:36] as long as time never runs backwards anyway [01:25:08] we had a fun bug with that at $DAYJOB-1 where the server time experienced DST [01:45:39] fortunately "since the UTC standard was established, negative leap seconds have never been needed." [03:44:26] 10Continuous-Integration, 6operations, 5Patch-For-Review: invalid byte sequence in US-ASCII - puppet issues with UTF-8 - https://phabricator.wikimedia.org/T91453#1091126 (10Dzahn) Thank you! that confirms we are past the UTF-8 related bugs and just see the next ones to fix now that we didn't see before. I s... [03:44:59] Yippee, build fixed! [03:44:59] Project browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-linux-android-sauce build #4: FIXED in 2 min 30 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-linux-android-sauce/4/ [03:48:10] 10Continuous-Integration, 6operations: Could not find class role::ci::website::labs on integration puppetmaster - https://phabricator.wikimedia.org/T91613#1091130 (10Dzahn) 3NEW [03:48:50] 10Continuous-Integration, 6operations: Could not find class role::ci::website::labs on integration puppetmaster - https://phabricator.wikimedia.org/T91613#1091140 (10Dzahn) p:5Triage>3Normal [03:50:22] 10Continuous-Integration, 6Labs, 6operations: Could not find class role::ci::website::labs on integration puppetmaster - https://phabricator.wikimedia.org/T91613#1091130 (10Dzahn) [03:51:46] 10Continuous-Integration, 6operations, 5Patch-For-Review: invalid byte sequence in US-ASCII - puppet issues with UTF-8 - https://phabricator.wikimedia.org/T91453#1091143 (10Dzahn) p:5Triage>3Normal [03:53:26] 10Continuous-Integration, 6operations, 5Patch-For-Review: invalid byte sequence in US-ASCII - puppet issues with UTF-8 - https://phabricator.wikimedia.org/T91453#1084541 (10Dzahn) >>! In T91453#1089996, @Krinkle wrote: > In the last run, only the following remained: let's fix that over here T91613 because i... [03:58:55] 10Continuous-Integration, 6operations, 5Patch-For-Review: invalid byte sequence in US-ASCII - puppet issues with UTF-8 - https://phabricator.wikimedia.org/T91453#1091151 (10Dzahn) 5Open>3Resolved I'll say it's resolved because the few non-ASCII chars we had in .pp files have been removed and in .erb file... [04:26:51] Yippee, build fixed! [04:26:52] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #348: FIXED in 41 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/348/ [04:32:40] Yippee, build fixed! [04:32:41] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #521: FIXED in 34 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/521/ [04:35:11] Project beta-scap-eqiad build #43961: FAILURE in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/43961/ [04:55:52] Yippee, build fixed! [04:55:52] Project beta-scap-eqiad build #43963: FIXED in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/43963/ [05:08:22] 10Continuous-Integration, 6Labs, 6operations: Could not find class role::ci::website::labs on integration puppetmaster - https://phabricator.wikimedia.org/T91613#1091217 (10Dzahn) role::ci::website exists in production, role::ci::website::**labs** does not. That's the class setting up doc.wm and integration... [05:25:38] Yippee, build fixed! [05:25:39] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #505: FIXED in 31 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/505/ [05:37:14] Yippee, build fixed! [05:37:14] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #498: FIXED in 14 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/498/ [05:55:04] Yippee, build fixed! [05:55:04] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #349: FIXED in 42 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/349/ [06:38:10] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [06:51:41] https://integration.wikimedia.org/zuul/ the graphs here seem to be broken for last few hours [07:10:42] (03PS1) 10Legoktm: Add experimental {name}-{ext-name}-composer-{phpflavor} jobs [integration/config] - 10https://gerrit.wikimedia.org/r/194461 (https://phabricator.wikimedia.org/T90943) [07:56:32] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [08:30:36] (03PS8) 10Zfilipin: Created the first Android CentralNotice Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/193361 (https://phabricator.wikimedia.org/T86092) [08:30:51] (03CR) 10Zfilipin: [C: 032] "The job is green! :)" [integration/config] - 10https://gerrit.wikimedia.org/r/193361 (https://phabricator.wikimedia.org/T86092) (owner: 10Zfilipin) [08:37:13] (03Merged) 10jenkins-bot: Created the first Android CentralNotice Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/193361 (https://phabricator.wikimedia.org/T86092) (owner: 10Zfilipin) [08:40:44] 10Continuous-Integration, 10MediaWiki-ResourceLoader, 10MediaWiki-Unit-tests: Fix "DatabaseSqlite::replace/single-row NOT NULL constraint failed" for md_module table - https://phabricator.wikimedia.org/T91567#1091539 (10hashar) [08:44:06] 6Release-Engineering, 7Browser-Tests, 5Patch-For-Review: Move the list of repositories with Selenium tests from mediawiki-selenium readme file to mediawiki.org - https://phabricator.wikimedia.org/T91486#1091545 (10zeljkofilipin) 5Open>3Resolved [08:45:59] 10Continuous-Integration, 10MediaWiki-ResourceLoader, 10MediaWiki-Unit-tests: Fix "DatabaseSqlite::replace/single-row NOT NULL constraint failed" for md_module table - https://phabricator.wikimedia.org/T91567#1091546 (10hashar) [09:29:00] (03PS5) 10Amire80: Move phpcs in ContentTranslation to composer [integration/config] - 10https://gerrit.wikimedia.org/r/194340 (https://phabricator.wikimedia.org/T90943) [09:29:37] 10Continuous-Integration: Zuul repositories have too many refs causing slow updates - https://phabricator.wikimedia.org/T70481#1091634 (10hashar) [09:42:14] legoktm: should the warning at https://integration.wikimedia.org/ci/job/php-composer-validate/756/console be resolved? [09:42:20] For example, by ~? [09:42:22] https://getcomposer.org/doc/faqs/why-are-unbound-version-constraints-a-bad-idea.md [10:07:01] aharoni: having hangout problems [10:07:06] I see :) [11:00:24] Project beta-scap-eqiad build #43998: FAILURE in 16 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/43998/ [11:17:01] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-salt is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:22:06] Yippee, build fixed! [11:22:06] Project beta-scap-eqiad build #44001: FIXED in 8 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/44001/ [11:24:11] release folks, why I can't see 1.25wmf20 branch in https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/ContentTranslation,branches [11:24:20] (ie for Content Translation) [11:26:59] twentyafterfour: ^ [11:40:42] created myself now. [12:32:26] (03PS6) 10Amire80: Move phpcs in ContentTranslation to composer [integration/config] - 10https://gerrit.wikimedia.org/r/194340 (https://phabricator.wikimedia.org/T90943) [12:33:21] legoktm: Can https://gerrit.wikimedia.org/r/#/c/194331/ and https://gerrit.wikimedia.org/r/#/c/194340/ be merged now? [12:37:34] PROBLEM - SSH on deployment-lucid-salt is CRITICAL: Connection refused [13:05:39] 10Beta-Cluster, 7Tracking: Setup monitoring for Beta cluster (tracking) - https://phabricator.wikimedia.org/T53497#1091913 (10yuvipanda) [13:05:41] 10Beta-Cluster, 7Easy: monitor unsigned salt keys - https://phabricator.wikimedia.org/T72862#1091909 (10yuvipanda) 5Open>3declined a:3yuvipanda Should just be fixed with the next version upgrade that will have auto accepting keys. [13:06:16] 10Beta-Cluster: Setup puppet exported resources to collect ssh host keys for beta - https://phabricator.wikimedia.org/T72792#1091914 (10yuvipanda) Right. Perhaps we could manually set the keys in hiera or someplace? [13:07:01] 10Beta-Cluster: Give all members of the deployment-prep project sudo - https://phabricator.wikimedia.org/T71269#1091916 (10yuvipanda) Alright, doing this now... [13:07:17] greg-g: breathing room acquired on toollabs, am doing beta stuff today and tomorrow [13:08:14] 6Release-Engineering: Rework beta apache config - https://phabricator.wikimedia.org/T1256#1091917 (10yuvipanda) This is actually... a fair bit harder than I expected, partially because it could also break prod's apaches, and I'm not very familiar with apache config language... [13:09:13] 10Beta-Cluster: Configure all deployment-prep instances to use local salt and puppet master by default - https://phabricator.wikimedia.org/T64795#1091918 (10yuvipanda) This can be done now easily thanks to thcipriani's patches. [13:11:15] 10Beta-Cluster: Give all members of the deployment-prep project sudo - https://phabricator.wikimedia.org/T71269#1091919 (10yuvipanda) Done. Everyone, NDA or not, can sudo on deployment-prep now. @greg I'll email releng list, am unsure where else this should be communicated... [13:14:22] 10Beta-Cluster, 6Labs: Setup multimaster salt for large projects using salt-syndic - https://phabricator.wikimedia.org/T78466#1091928 (10yuvipanda) [13:14:23] 10Beta-Cluster, 6Labs, 6operations: Backport new salt-syndic packages - https://phabricator.wikimedia.org/T85442#1091926 (10yuvipanda) 5Open>3Resolved Installed fine, will re-open if it doesn't actually work :) [13:16:01] 10Beta-Cluster: Give all members of the deployment-prep project sudo - https://phabricator.wikimedia.org/T71269#1091931 (10yuvipanda) 5Open>3Resolved a:3yuvipanda [13:19:04] 10Beta-Cluster, 7HHVM: hhvm creates core file in /tmp/ filling mediawiki02 labs instance root partition - https://phabricator.wikimedia.org/T71979#1091936 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Beta mw instances now have a saner disk partition set up, and this hasn't happened in a while because of th... [13:19:04] 10Beta-Cluster, 10Wikimedia-Labs-Infrastructure, 7Tracking: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#1091939 (10yuvipanda) [13:20:47] 10Beta-Cluster: Can't run mwscript without explicit sudo on Beta Cluster - https://phabricator.wikimedia.org/T89802#1091944 (10yuvipanda) 5Open>3Resolved a:3yuvipanda mwscript and other scripts now use www-data. [13:30:59] 10Beta-Cluster: Diamond logstash monitor fills /var/log/apache2 access log - https://phabricator.wikimedia.org/T74175#1091961 (10yuvipanda) 5Open>3Resolved a:3yuvipanda No such instances are found atm, and monitor_fatals.rb is dead. [13:33:24] 10Beta-Cluster, 6operations: File upload area resorts to 0777 permissions to for uploaded content - https://phabricator.wikimedia.org/T75206#1091974 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Fixed now with all the www-data work. [13:35:05] 10Beta-Cluster: Send MediaWiki profiling on beta cluster to graphite.wmflabs.org - https://phabricator.wikimedia.org/T75881#1091979 (10yuvipanda) >>! In T75881#790730, @hashar wrote: > Can you confirm labmon1001 is an OK receiver for the Beta cluster MediaWiki profiling? Yup. Please prefix all stats with projec... [13:37:05] RECOVERY - Long lived cherry-picks on puppetmaster on deployment-salt is OK: OK: Less than 100.00% above the threshold [0.0] [13:51:55] 10Beta-Cluster, 5Patch-For-Review: Send MediaWiki profiling on beta cluster to graphite.wmflabs.org - https://phabricator.wikimedia.org/T75881#1092006 (10yuvipanda) ^ might or might not work, I don't actually know enough about how / what statsd stats come from in MW to be sure. [14:17:02] legoktm: any idea about failures in, https://gerrit.wikimedia.org/r/#/c/194493/ [14:19:14] 10Beta-Cluster, 10Continuous-Integration, 10Math: beta-recompile-math-texvc-eqiad job fails with "/usr/local/bin/scap-recompile: No such file or directory" - https://phabricator.wikimedia.org/T91191#1092058 (10Physikerwelt) The deb package has diverged from texvc in master. @cscott did some improvements to t... [14:27:32] ^d: Please check, https://gerrit.wikimedia.org/r/#/c/194493 when you're around (ie test failures, have I done something wrong? :)) [14:30:59] 10Beta-Cluster, 5Patch-For-Review: Send MediaWiki profiling on beta cluster to graphite.wmflabs.org - https://phabricator.wikimedia.org/T75881#1092086 (10yuvipanda) Alright, that didn't actually change / do anything. [14:32:09] 10Beta-Cluster, 5Patch-For-Review: Send MediaWiki profiling on beta cluster to graphite.wmflabs.org - https://phabricator.wikimedia.org/T75881#1092096 (10yuvipanda) [14:36:57] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #36: FAILURE in 8 min 57 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/36/ [14:49:22] (03PS1) 10Hashar: Package python deps with dh-virtualenv [integration/zuul] (debian-precise-venv) - 10https://gerrit.wikimedia.org/r/194520 [14:57:06] 10Continuous-Integration, 6operations, 3Continuous-Integration-Isolation, 5Patch-For-Review, 7Upstream: [upstream] Create a Debian package for Zuul - https://phabricator.wikimedia.org/T48552#1092134 (10hashar) I am now hitting a wall because dh-virtualenv pip doesn't have network access: ``` I: dh_virtu... [15:13:22] zull borked? [15:13:40] ^d: ^^ [15:13:47] anyway, off for dinner. [15:18:38] define "borked" :) [15:48:36] greg-g: anything I ought to be mindful of with this travel booking request form? I know there was talk about possibly doing the offsite outside of Lyon. Currently I just have 17th–26th Denver → Lyon → Denver. [15:50:00] thcipriani: see the email from Rachel F "Release Engineering Team Offsite" [15:50:17] thcipriani: there's yet-another-google-spreadsheet to fill out (this one specific to us) [15:50:34] * chrismcmahon filled it out saying "sign me up for the default" [15:50:51] got it, thanks. [15:56:43] hey, is CI down? [15:56:53] https://integration.wikimedia.org/zuul/ has jobs queued "1 hr 49 min ago" [15:57:09] i +2'd https://gerrit.wikimedia.org/r/#/c/194155/ twenty minutes ago and it's not even queued [15:59:55] thcipriani: yay [15:59:59] (On the patches) [16:00:39] YuviPanda|food: yeah, a few blocking thing out of the way. Plus I learned a ton about the setup from doing them :) [16:01:24] thcipriani: yup :D [16:02:17] !log about to disconnect/reconnect gearman per https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Known_issues [16:02:23] Logged the message, Master [16:03:26] zeljkof: Chrome is showing me an error... [16:03:57] chrismcmahon: maybe because you have changed the time of the current meeting :) [16:04:02] !log not sure it helped [16:04:06] Logged the message, Master [16:04:09] chrismcmahon: try leaving and joining again [16:06:03] !log jenkins doesn't have anything queued and is processing jobs apparently, not sure why zuul is showing two jobs queued for almost 2 hours (one with all tests passing, the other with nothing tested yet) [16:06:07] Logged the message, Master [16:07:04] zeljkof: it looks like your in a meeting, but eyes on jenkins/zuul would be appreciated if you have any other ideas [16:07:38] greg-g: isn't it stuck? [16:08:00] kart_: yes, but, how to fix? [16:08:19] greg-g: I will ask in -releng :) [16:08:49] greg-g: in a meeting with chrismcmahon [16:09:06] also see -operations right now [16:09:15] 10Continuous-Integration, 6Release-Engineering, 10Fundraising Tech Backlog, 10Wikimedia-Fundraising-CiviCRM, and 3 others: Create CI slave instance for CiviCRM testing - https://phabricator.wikimedia.org/T89894#1092315 (10awight) [16:09:33] 10Continuous-Integration, 6Release-Engineering, 10Fundraising Tech Backlog, 10Wikimedia-Fundraising-CiviCRM, and 3 others: Configure Jenkins to run CiviCRM builds only on Fundraising CI slaves - https://phabricator.wikimedia.org/T89895#1092316 (10awight) [16:09:44] 10Continuous-Integration, 6Release-Engineering, 10Fundraising Tech Backlog, 10Wikimedia-Fundraising-CiviCRM, and 3 others: Run CiviCRM testing scripts during CI - https://phabricator.wikimedia.org/T89896#1092317 (10awight) [16:12:35] <^d> !log restarted zuul [16:12:39] Logged the message, Master [16:13:40] kart_: 8c40c7a is not found in gerrit... that seems to be why https://gerrit.wikimedia.org/r/#/c/194493 failed [16:14:44] twentyafterfour: blah. [16:14:46] fix? [16:14:57] wmf19 works fine [16:16:18] !log getting "/zuul/status.json: Service Temporarily Unavailable" after the zuul restart [16:16:22] Logged the message, Master [16:18:06] this job sure does take a while: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/1137/console [16:18:41] <^d> I have a status.json now [16:18:49] !log now there are jobs running on the zuul status page [16:18:53] Logged the message, Master [16:19:22] kart_: I'm working on it [16:19:25] g'morning CI :) [16:19:50] twentyafterfour: I took ContentTranslation HEAD as usually I do. [16:24:14] thcipriani: hey [16:24:28] YuviPanda|food: yo, so how do we verify T88304? [16:24:39] So to verify the salt and puppet master [16:24:40] We have to try setting up some other thing [16:24:42] Maybe the dB's? [16:25:16] sure. I also started some work on tin, but there are some roles still that need combining :\ [16:25:47] Right. File dependent bugs! [16:26:17] thcipriani: I think the dbs might be a good place to start... [16:26:17] gotcha. [16:26:17] Or redis or memc [16:26:17] You pick [16:26:30] dbs work fine for me. [16:26:35] Cool [16:26:45] twentyafterfour: any luck? [16:26:55] Look at what roles are on prod and see if we can use the same code with hiera fixes? [16:27:22] kart_: can't figure out why gerrit doesn't know about 8c40c7a147f003393a0f5e8a699920103e793612 [16:27:38] twentyafterfour: it is 'Merge' [16:27:42] is that useful? [16:29:16] YuviPanda|food: yeah role::coredb's seem pretty well paramaterized. I can spin one up after adding some things to Staging:hiera. [16:29:24] kart_: I'm not sure, that should be ok [16:30:05] twentyafterfour: strange. [16:30:26] twentyafterfour: if you can help further, I've deployment in 30 minutes ;) [16:31:29] kart_: I set up the deployed branch on tin to point to 1.25wmf20 [16:32:27] YuviPanda|food: actually do I want role::mariadb? [16:32:28] I'm not sure how to get the patch to verify though [16:32:55] thcipriani: not sure. I can look in an hour. Out eating food [16:32:56] greg-g: code coverage takes hours, it's super slow [16:33:08] fair enough. [16:33:22] legoktm: gotcha [16:36:00] twentyafterfour: so that should be fine, right? [16:36:49] I'm not sure it's "fine" but... it should deploy 1.25wmf20 next time someone scaps [16:37:03] ok. no worry. [16:37:13] I'm doing scapping today. [16:37:29] twentyafterfour: will poke you if needed :) [16:40:40] 10Quality-Assurance, 10MediaWiki-extensions-UploadWizard, 6Multimedia, 5Patch-For-Review: UploadWizard browser test for chunked upload - https://phabricator.wikimedia.org/T89289#1092443 (10Gilles) [16:41:48] 6Release-Engineering: Rework beta apache config - https://phabricator.wikimedia.org/T1256#1092469 (10greg) a:5Reedy>3yuvipanda Who's help do you need? [16:45:39] twentyafterfour: well, it still fails. [16:46:10] that's weird [16:49:26] 10Staging, 5Patch-For-Review: Setup staging-palladium as puppetmaster and saltmaster - https://phabricator.wikimedia.org/T88304#1092526 (10thcipriani) a:3thcipriani [16:55:50] twentyafterfour: ok. wmf19 change also fails :/ [16:56:38] Error: your composer.lock file is not up to date, run "composer update" to install newer dependencies [16:56:52] mmmm [16:56:52] twentyafterfour: and we don't have it. [16:57:05] is it checking out master of mediawiki/vendor (or core) instead of the proper wmf branch? [16:57:06] legoktm: that's why I poke you ;) [16:57:32] legoktm: my changes are in wmf19/wmf20 [16:58:12] well https://gerrit.wikimedia.org/r/#/c/194492/ is passing [16:58:38] 20 is failing :/ [16:58:53] make-wmf-branch must have screwed up more than I thought yesterday [16:59:12] no [16:59:15] this is zuul-cloner [16:59:16] 16:50:50 DEBUG:zuul.Cloner:Project mediawiki/vendor in Zuul does not have ref refs/zuul/wmf/1.25wmf20/Z762e4fe25d1448d291e72146f567f3ab [16:59:16] 16:50:51 DEBUG:zuul.Cloner:Project mediawiki/vendor in Zuul does not have ref refs/zuul/master/Z762e4fe25d1448d291e72146f567f3ab [16:59:16] 16:50:51 DEBUG:zuul.Cloner:Falling back to branch master [16:59:19] it shouldn't do that [16:59:26] it should fallback to wmf/1.25wmf20 [16:59:51] what if wmf/1.25wmf20 doesn't exist? [17:00:20] then we have a bigger problem? :P [17:00:31] make-wmf-branch didn't push the branches [17:00:33] kart_: if you're in the middle of a deployment I'd just override jenkins... [17:00:37] oh? [17:00:38] I thought it did but I was wrong apparently [17:00:50] :o [17:00:51] you're right [17:01:05] all the submodule refs got set up but they didn't get pushed back to gerrit [17:01:27] should we just manually create wmf20 branches then? [17:01:42] legoktm: I did for CX atleast [17:02:20] I'm gonna try to hack up make-wmf-branch to make branches where they don't exist [17:02:25] that script is a mess [17:04:57] twentyafterfour: should I wait? [17:05:33] kart_: if you can give me a minute, yeah. [17:05:59] twentyafterfour: sure! [17:06:23] 1.25wmf20 is only deployed to group0, if you need to push changes to wmf19 then go ahead with that for now [17:07:37] twentyafterfour: okay! [17:07:49] I'll go for wmf19 now [17:14:42] Project beta-scap-eqiad build #44038: FAILURE in 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/44038/ [17:31:05] phabspam from the greg-g :-) [17:35:03] Yippee, build fixed! [17:35:04] Project beta-scap-eqiad build #44041: FIXED in 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/44041/ [17:40:13] PROBLEM - Puppet failure on deployment-zotero01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:47:14] 10Continuous-Integration, 6Labs, 6operations: Could not find class role::ci::website::labs on integration puppetmaster - https://phabricator.wikimedia.org/T91613#1092736 (10Krinkle) As far as I know a "role::ci::website::labs" role never existed, nor would we want one. There is no website in labs...? I'm cu... [17:51:39] 10Continuous-Integration, 6Labs, 6operations: Could not find class role::ci::website::labs on integration puppetmaster - https://phabricator.wikimedia.org/T91613#1092748 (10Dzahn) Yea, that's what i was wondering too. Where does it even come from, i also couldn't find it in the puppet repo. [17:51:51] hello wikimedia-qa folks that presumably play in deployment-prep a lot [17:52:15] !log pushed wmf/1.25wmf20 branch to submodule repos [17:52:17] I had been planning to upgrade salt to 2014.7 today but I'm just now to the point where I'm ready to proceed, and feeling tired [17:52:19] Logged the message, Master [17:52:40] so this will happen tomorrow morning my time (it's evening here now for me) [17:52:52] * ^d will be around tomorrow evening your time [17:53:07] <^d> afternoon probably, I can check in during my AM [17:53:09] this should not affect anyone's services in any way, but if you were going to use trebuchet to deploy something, there could be a short-term hiccup [17:53:34] 10Continuous-Integration, 6Labs, 6operations: Could not find class role::ci::website::labs on integration puppetmaster - https://phabricator.wikimedia.org/T91613#1092760 (10scfc) The class is referenced by the instance's configuration (cf. https://wikitech.wikimedia.org/wiki/Nova_Resource:I-00000474.eqiad.wm... [17:53:49] this will oly b in deployment-prep for a first step; if this succeeds fine, as I expect it to, then I would move to do the whole cluster next week, plus labs [17:54:03] *only be [17:55:18] I'll repeat this announcement in here tomorrow morning a little while before I start. [17:55:51] thanks ^d that will be awesome. [17:56:11] <^d> no worries, enjoy your evening :) [17:57:04] kart_: I'm attempting to recheck your patch now for 1.25wmf20 [17:57:10] 10Continuous-Integration: "/usr/local/bin/tox: Permission denied" on integration-slave14xx instances - https://phabricator.wikimedia.org/T91525#1092762 (10Krinkle) Looks like this broke when the umask for puppet changed to be more restrictive (only readabel by root). ``` # contint/packages/labs.pp package {... [17:57:33] ^d: do I have recheck privileges? :-/ [17:57:43] <^d> Should... [17:58:58] yep, ok cool [17:59:31] greg-g: when we register for the hackathon we don't need to include accommodation, right? [17:59:58] marxarelli: everyone has that question, including me, I pinged quim about it [18:00:47] greg-g: cool, np [18:09:44] 10Continuous-Integration, 6Labs, 6operations: Could not find class role::ci::website::labs on integration puppetmaster - https://phabricator.wikimedia.org/T91613#1092796 (10Dzahn) >>! In T91613#1092760, @scfc wrote: > Someone needs to uncheck the corresponding marker on the configuration page. Logged in to... [18:11:24] 10Continuous-Integration, 6Labs, 6operations: Could not find class role::ci::website::labs on integration puppetmaster - https://phabricator.wikimedia.org/T91613#1092799 (10Dzahn) It does exist as a "puppet group" in wikitech (https://wikitech.wikimedia.org/wiki/Special:NovaPuppetGroup depending on your proj... [18:19:59] 10Continuous-Integration: Make browser tests voting for all repos of WMF deployed code - https://phabricator.wikimedia.org/T91669#1092830 (10greg) 3NEW [18:20:15] 10Continuous-Integration: Make browser tests voting for all repos of WMF deployed code - https://phabricator.wikimedia.org/T91669#1092839 (10greg) [18:20:16] 10Continuous-Integration, 7Browser-Tests, 7Tracking: [project] trigger browser tests from Gerrit (tracking) - https://phabricator.wikimedia.org/T55697#541910 (10greg) [18:21:19] 10Continuous-Integration, 7Browser-Tests: Make browser tests voting for all repos of WMF deployed code - https://phabricator.wikimedia.org/T91669#1092830 (10greg) [18:23:43] 10Continuous-Integration: "/usr/local/bin/tox: Permission denied" on integration-slave14xx instances - https://phabricator.wikimedia.org/T91525#1092859 (10Krinkle) Pending a workaround or a reevaluation of the restrictive umask, added the following patch to your setup script. The only reason I'm doing this now i... [18:25:28] 10Continuous-Integration: Pool new integration-slave12xx and integration-slave14xx instances and delete old ones - https://phabricator.wikimedia.org/T91524#1092874 (10akosiaris) [18:25:30] 10Continuous-Integration: "/usr/local/bin/tox: Permission denied" on integration-slave14xx instances - https://phabricator.wikimedia.org/T91525#1092871 (10akosiaris) 5Open>3Resolved a:3akosiaris Fixed in https://gerrit.wikimedia.org/r/#/c/194557/ . You will probably need to remove the package/recreate the... [18:27:09] 10Continuous-Integration, 7Browser-Tests: Make browser tests voting for all repos of WMF deployed code - https://phabricator.wikimedia.org/T91669#1092886 (10greg) [18:28:02] 10Continuous-Integration, 7Browser-Tests, 7Epic: Make browser tests voting for all repos of WMF deployed code - https://phabricator.wikimedia.org/T91669#1092830 (10greg) [18:28:10] (sorry for the spam) [18:29:07] 10Beta-Cluster, 6Release-Engineering: Process accounting + deployments routinely fill up /var on deployment-bastion - https://phabricator.wikimedia.org/T91354#1092912 (10coren) [18:29:31] 10Beta-Cluster, 6Release-Engineering: Process accounting + deployments routinely fill up /var on deployment-bastion - https://phabricator.wikimedia.org/T91354#1080743 (10coren) That patch was mistakenly assigned to the wrong task -- please ignore. [18:31:23] 10Quality-Assurance: Given I am logged in step not generic enough - https://phabricator.wikimedia.org/T91671#1092944 (10Jdlrobson) 3NEW [18:36:55] marxarelli: bd808 rachel answered our question re registration and rooms etc [18:37:05] 10Continuous-Integration, 7Browser-Tests, 7Epic: Make browser tests voting for all repos of WMF deployed code - https://phabricator.wikimedia.org/T91669#1092979 (10dduvall) Test suites need to be faster as well. Once we work out ways to increase test isolation (for instance T90964) we should also work toward... [18:39:38] greg-g: uh, is that an if (A || B) or an if (A && B)? [18:39:51] a || b it seems [18:39:58] || [18:40:53] right. a && b would be pretty silly [18:45:31] || [18:49:39] 10Continuous-Integration, 6Labs, 6operations: Could not find class role::ci::website::labs on integration puppetmaster - https://phabricator.wikimedia.org/T91613#1093037 (10Krinkle) >>! In T91613#1092760, @scfc wrote: > The class is referenced by the instance's configuration (cf. https://wikitech.wikimedia.o... [18:49:47] 10Continuous-Integration, 6Labs, 6operations: Could not find class role::ci::website::labs on integration puppetmaster - https://phabricator.wikimedia.org/T91613#1093038 (10Krinkle) 5Open>3Resolved a:3Krinkle [18:50:54] !log Re-creating integration-slave1405 [18:50:58] Logged the message, Master [18:51:57] thcipriani: looks like coredb::s1, s2, s3, etc are the roles used... [18:52:06] in prod [18:52:27] Krinkle: btw, thcipriani just added a neat way to make every instance in a project point to a local puppetmaster... [18:52:41] YuviPanda: Tell me more :) [18:52:49] thcipriani: can you document it at https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster [18:52:50] ? [18:53:04] Krinkle: look at http://wikitech.wikimedia.org/wiki/Hiera:staging [18:53:13] Krinkle: all classes under the ‘classes’ key are applied to all instances [18:53:19] so it applies role::puppet::self to them all [18:53:22] YuviPanda: Last I checked Hiera does not work [18:53:22] YuviPanda: huh, I've been working on the mariadb::core roles, and yes, I can document [18:53:27] At least not for integration. [18:53:31] Dunno anything else. [18:53:34] Krinkle: yup, he fixed that also [18:53:35] Krinkle: also, fixed hiera for labs :) [18:53:58] thcipriani: I’m looking at site.pp, under ‘# eqiad dbs' [18:54:06] thcipriani: OK :) [18:54:58] thcipriani: YuviPanda: How does this interact with existing instances? Can I add role::puppet::self/role::puppet::self-master to Hiera:Integration now and then create an instance? [18:55:06] Here's our current set up steps [18:55:07] https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/Setup [18:55:14] Krinkle: yup, you should be able to. [18:55:16] I'm about to create one in 1 minute [18:55:26] you’ll still have to accept the keys manually though. [18:55:32] but no need to futz around in wikitech interface [18:55:33] yeah, I have it setup on Staging here: https://wikitech.wikimedia.org/wiki/Hiera:Staging [18:56:32] thcipriani: It's refreshed every puppet run? How does it end up from wikitech into puppet. [18:56:42] Since we already have a puppetmater [18:56:44] master* [18:56:51] Hiera. [18:56:56] puppetmaster hits wikitech [18:57:09] and picks these values up [18:57:13] http://wikitech.wikimedia.org/wiki/Hiera [18:57:21] And our puppetmaster will do that too? [18:57:28] Sounds like a bootstrapping issue [18:57:34] how does the first run know to use ours from the start? [18:57:54] Krinkle: it won’t, but that’s the same case right now, and it shouldn’t matter... [18:57:59] Right [18:58:01] so first one will run global puppetmaster [18:58:09] and that’ll change it to point to project puppetmaster [18:58:15] and then you have to do the cert signing thing... [18:58:24] Right, so as long as I use it only to set puppetmaster and not to apply roles, it'll be fine. [18:58:42] right. so you should use it for roles that apply to *all* of the hosts. [18:58:49] puppetmaster is the only one that comes to mind atm [18:58:52] I was going to ask if there's any way to accept those certs and salt keys automagically. Would be nice. [18:59:00] YuviPanda: Well, no, because that would use the global puppetmaster for those roles in the first run. [18:59:12] Which can be incompatible or break stuff that isn't forward-compat withours. [18:59:18] Krinkle: ah, right. yup, yup [18:59:31] Krinkle: but yeah, as of now I can’t htink of anything other than the puppetmaster role that would benefit from this [18:59:37] Cool [18:59:54] thcipriani: ah, so salt keys would auto accept from next version of salt, if we wish them to. apergos is looking at doing the upgrade soon... [18:59:59] maybe in a couple of weeks. [19:00:10] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #406: FAILURE in 12 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/406/ [19:00:15] Does this look right? https://wikitech.wikimedia.org/wiki/Hiera:Integration [19:00:16] deployment-prep tmorrow morning [19:00:21] if all goes well, the rest next week [19:01:02] apergos: wheeee [19:01:12] thcipriani: although, autosigning has security implications... [19:01:25] Krinkle: looks ok [19:01:26] you just gave everyone sudo YuviPanda :P [19:01:36] dont' talk to me about security implications [19:01:38] greg-g: only with manager approval :P [19:01:41] :P [19:02:51] greg-g: I did a bunch of beta stuff today :) and thcipriani is doing all the work I wish I had done, so all yays :) [19:03:24] autosigning is largely insecure, sure, but if ever there was a good usecase... :) Also, a whitelist and firewall rules could, I think, mitigate a lot of risks. [19:03:51] thcipriani just got the "new employees" shoutout in the metrics meeting [19:03:54] https://www.youtube.com/watch?v=kIguVBIfnCY [19:04:33] hawt diggity. I still don't have flash installed on this machine :( [19:04:39] thcipriani: yup, and for autosigning we can do https://docs.puppetlabs.com/puppet/latest/reference/ssl_autosign.html#policy-based-autosigning [19:04:44] thcipriani: aw yeah. you. have. arrived! [19:04:48] oh, hangouts on air require flash not just webm? [19:05:02] thcipriani: so it will give us the name of the node, and we can check with the wikitech / nova api to see if that node belongs to our project... [19:05:11] although, that’s almost as much security as not giving a fuck.. [19:05:19] because you’re trusting the nodename from them... [19:05:27] greg-g: I guess. I don't have chrome or flash and it tells me "your browser does not recognize...etc" [19:05:32] :( [19:06:19] what did jenkins-bot just do??? [19:19:12] YuviPanda: so, looking at coredb, the problem is it extends role::coredb::config which means $topology is a non-hiera-able variable and kinda holds all the info. This may take a bit of work... [19:21:08] (03PS2) 10Legoktm: Add experimental {name}-{ext-name}-composer-{phpflavor} jobs [integration/config] - 10https://gerrit.wikimedia.org/r/194461 (https://phabricator.wikimedia.org/T90943) [19:33:01] (03PS3) 10Legoktm: Add experimental {name}-{ext-name}-composer-{phpflavor} jobs [integration/config] - 10https://gerrit.wikimedia.org/r/194461 (https://phabricator.wikimedia.org/T90943) [19:34:29] !log deploying https://gerrit.wikimedia.org/r/194461, lots of new jobs [19:34:35] Logged the message, Master [19:37:57] this is going to take a while... [19:39:56] 10Continuous-Integration: Continuous deployment for CI - https://phabricator.wikimedia.org/T91684#1093242 (10greg) 3NEW [19:40:01] zeljkof: ^ [19:40:47] greg-g: thanks! :) [19:41:30] eh, that's a dupe [19:42:13] legoktm: bah [19:42:13] 10Continuous-Integration: Continuous deployment for CI - https://phabricator.wikimedia.org/T91684#1093257 (10Legoktm) Partial dupe of T49056, which just covers jenkins. [19:42:54] 10Continuous-Integration: Jenkins: Set up postmerge job to auto-deploy jenkins-job-builder configuration - https://phabricator.wikimedia.org/T49056#1093261 (10greg) [19:42:54] 10Continuous-Integration: Continuous deployment for CI - https://phabricator.wikimedia.org/T91684#1093260 (10greg) [19:43:00] 10Continuous-Integration: Jenkins: Set up postmerge job to auto-deploy jenkins-job-builder configuration - https://phabricator.wikimedia.org/T49056#1093264 (10zeljkofilipin) [19:43:01] legoktm: there [19:43:03] :P [19:43:04] I'm not too sure about jenkins automatically deploying itself. [19:43:09] :) [19:43:29] it's not on the short-term list, just an idea discussed in a 1:1 we wanted to record [19:43:44] * legoktm nods [19:44:00] PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:44:28] 10Continuous-Integration, 6operations, 7Puppet: Puppet class Mediawiki::Packages::Fonts fails to install various fonts - https://phabricator.wikimedia.org/T91685#1093266 (10Krinkle) 3NEW [19:59:02] It's still deploying, but I'm going to have lunch in the meantime [20:18:07] !log Finished creation and provisioning of integration-slave1405 [20:18:11] Logged the message, Master [20:44:35] !log Deleting integration-slave1201-integration-slave1404, and integration-slave1401-integration-slave1404. [20:44:39] Logged the message, Master [20:47:03] Krinkle: it seems zuul is blocked. gearman deadlock? [20:47:14] * jzerebecki has no permission to do the reconnect dance [20:47:32] mobilefrontend job is stuck [20:48:21] !log Re-establishing Gearman connection from Jenkins [20:48:24] Logged the message, Master [20:51:16] hashar: See https://integration.wikimedia.org/zuul/ - any idea why the MobileFrontend job is not progressing? [20:51:29] It's got two sub jobs queued but nothing is happening [20:52:43] It seems jobs on gallium are still being run such as mwcore-whitespaces just now [20:57:50] Weird. It seems certain types of jobs (trusty labs?) just won't run [20:57:52] others are fine [20:57:59] perhaps there are no more eligible slaves for those jobs? [20:58:12] There's plenty of slaves. All idling [20:58:16] at least 25 slots [20:58:32] See left sidebar on https://integration.wikimedia.org/ci/ [20:59:41] goood Krinkle day [21:00:04] I;m sick of this stuff. Why is it so damn hard to just run a job. [21:00:06] let see what happen [21:00:22] there is a mediawiki-core-phplint job that is not being run [21:00:31] wanna google hangout the debug session? [21:00:56] The one that is longest in the queue was MobileFrontend (1h and 20min). [21:01:04] I rebased that change to trigger it to cancel it [21:01:10] The last one is now DonationInterface [21:01:10] the Zuul scheduler is either stuck [21:01:18] or the jobs are not registered in gearman [21:01:20] but I don't think the issue is an individual change or merge [21:01:32] It's running things on precise slaves and prod slaves finethough [21:01:52] gallium:~$ zuul-gearman.py status|fgrep mediawiki-core-phplint [21:01:54] build:mediawiki-core-phplint:contintLabsSlave 0 0 18 [21:01:54] build:mediawiki-core-phplint 3 0 18 [21:01:54] build:mediawiki-core-phplint:UbuntuPrecise 0 0 19 [21:02:00] 3 jobs in the queue [21:02:03] 18 workers [21:02:06] 0 running :( [21:02:30] that jub has phpflavor-hhvm && phpflavor-hhvm [21:02:41] which afaik no slave has [21:02:55] They do [21:03:01] hashar: It seems there's a channel terminiation [21:03:05] java.io.IOException: Unexpected termination of the channel [21:03:07] https://integration.wikimedia.org/ci/computer/integration-slave1001/log [21:03:19] It happens to almost every slave I disconnect/relaunch [21:03:53] that is zuul related I think [21:04:34] Jenkins doesn't communicate with Zuul when managing or launching slaves [21:04:35] right? [21:04:41] I took a stacktrace of Zuul scheduler [21:04:56] with: kill -SIGUSR2 [21:05:04] that spurts the python trace to /var/log/zuul/debug.log [21:05:11] jzerebecki: https://integration.wikimedia.org/ci/label/phpflavor-hhvm/ https://integration.wikimedia.org/ci/label/phpflavor-zend/ all slaves have their appropriate label [21:05:52] yea was confused by phpflavor-zend && phpflavor-hhvm but that is only one part of the ||-condition [21:06:20] Yeah, but this is also affecting -npm , -qunit and -doxygen-publish jobs [21:06:28] https://phabricator.wikimedia.org/P364 [21:06:33] looking at that stack trace [21:07:12] hashar: Is there anything we can capture from before that happened? [21:07:23] hashar: I'd like to do a force (stop/start) restart but first capture logs [21:07:34] We're not gonna be able to solve this while keeping the queue I think [21:07:45] So might as well restart and look at logs after and get stuff back up [21:08:16] Krinkle: hold on [21:08:22] holding on.. [21:08:33] !log killing browser tests running [21:08:35] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #588: ABORTED in 20 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/588/ [21:08:37] Logged the message, Master [21:08:37] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #494: ABORTED in 15 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/494/ [21:08:38] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #554: ABORTED in 16 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/554/ [21:08:59] lets see what happens now [21:09:05] I already did that 30 minutes ago [21:09:20] browsertests are not related [21:09:53] did you try reconnecting a slave? [21:09:56] I thought they were holding up the queue, but it didn't matter [21:10:11] the browser tests are using a plugin that mess with Jenkins build schedulr [21:10:19] I've relaunched half of each pool. Some came back up, others triggered a channel termination error as mentioned earlier [21:10:40] relaunching them again worked without the error. Probably race condition [21:10:53] I would just kill Jenkins [21:12:10] pff. Doesn [21:12:31] pff. Doesn't make any sense. Entire sidebar is empty. Nothing gets inserted by gearman [21:12:38] No error anywhere in the interface [21:13:21] hashar: beta-code-eqiad-update and browser tests jobs are running fine, even now [21:13:25] So it's not completely broken [21:14:45] This one also crashes: https://integration.wikimedia.org/ci/computer/lanthanum/log [21:14:47] Krinkle: unless you object, I am going to restart Jenkins [21:14:50] Yeah, let's restart. [21:15:13] !log stopping Jenkins [21:15:17] Logged the message, Master [21:15:29] Jenkins is the worst software ever conceived by man. [21:16:08] No doubt the error is caused by something else. As always. But it sure has no concept of error handling or recovery. [21:17:18] 10Continuous-Integration, 7Jenkins: Launching Jenkins slave agent fails with "java.io.IOException: Unexpected termination of the channel" - https://phabricator.wikimedia.org/T91697#1093554 (10Krinkle) 3NEW [21:17:25] 10Continuous-Integration, 7Jenkins: Launching Jenkins slave agent fails with "java.io.IOException: Unexpected termination of the channel" - https://phabricator.wikimedia.org/T91697#1093561 (10Krinkle) [21:18:13] 10Continuous-Integration, 7Jenkins: Launching Jenkins slave agent fails with "java.io.IOException: Unexpected termination of the channel" - https://phabricator.wikimedia.org/T91697#1093554 (10Krinkle) [21:18:28] (Why do we always pick software that we end up hating? Like, you know, PHP, Gerrit, Bugzilla, Mingle, Mailman, OTRS, and pretty much everything else.) [21:19:24] Because we dont like admitting that the rest of the world is incompetent. :P [21:20:22] we're pushing the boundaries of what Jenkins is designed to do [21:20:25] in some ways [21:20:29] what you use doesn't matter, people will hate it because that makes people feel better [21:20:42] also, open source [21:20:44] o.O [21:20:46] Yeah, we're not even using Jenkins for most of its purposes. [21:20:57] any chance the jobs I was deploying could have caused issues? there were a lot of them [21:21:09] We should just kick it out and use a static file server to store the build logs. That's what OpenStack do already. We can do everything we want with just zuul/gearman. [21:21:27] mutante: best quote 2015 :p [21:21:57] Jenkins is fine. I've used it elsewhere and also at jQuery Foundation. Never had any issues. [21:22:26] aharoni: On the other hand, these seem to be mostly positive in the public eye: Phabricator, HHVM, OpenStack, Nginx, Node.js; [21:25:47] Krinkle: have you done anything or did every pending jobs managed to get triggered properly ? [21:25:58] I did nothing [21:26:01] I"m seeing it too [21:26:08] Which is odd since Jenkins is still busy restarting [21:26:13] yeah [21:26:19] well the web interface is available later [21:26:23] jenkins load the plugins [21:26:30] the jobs list [21:26:35] hashar: We should clean up some of those plugins [21:26:39] We've got too much crap in it [21:26:51] then the Jenkins gearman plugin connects to Zuul Gearman server and start registering all the Jenkins jobs as Gearman function [21:26:56] thus the jobs start being triggered [21:27:05] the web UI come up much later [21:27:09] hashar: Should we create a task to migrate all jobs to use Zuul cloner instead of git-scm? [21:27:39] I would leave the plugins around unless we are absolutely sure they are of no use [21:27:47] and yeah we can use a task to migrate out of git-scm [21:27:56] The main blocker for that is workspace cleaning. [21:27:59] though that would be blocked by zuul cloner not cleaning up workspace / not processing submodules [21:28:09] YEah and submodules [21:28:16] hashar: And submodules is blocked on local git cache [21:28:25] is it ok for me to start re-deploying jobs or should I wait? [21:28:36] so for the outage: Jenkins internal scheduler or Gearman plugin managed to be screwed up for some unknown reason [21:28:40] legoktm: It's still restarting, wait like 30minutes [21:28:46] killing Jenkins make it start with a new fresh state [21:29:00] ok [21:29:03] also [21:29:10] when deploying a largeeeee amount of jobs [21:29:42] Zuul might not know about jobs anymore, cause the whole list keep being unregistered / registed on each job configuration :( [21:30:49] oh? [21:30:59] I haven't deployed any changes to zuul yet though [21:31:09] legoktm: Hold on, why do we need composer-zend/composer-hhvm at the extension level? Don't we use the main phpunit job and trigger it for different extensions cia ZUUL_PROJECT env? [21:31:25] Then the local composer would only run non-phpunit like line and phpcs, which doens't need both flavours afiak [21:32:12] Krinkle: no, it uses '{name}-{ext-name}-testextension-{phpflavor}' for each extension [21:32:21] Keep in mind that the end goal for isolation is 1 vm for a repo/pipeline event. maybe 2 (one for zend compat), though that could be done in a different way as well. [21:32:37] legoktm: Now yes. But not always? [21:33:00] legoktm: But point stands, does it need both? It won't include phpunit, right? [21:33:14] MediaWiki extensions can't run phpunit from within composer because it needs mediawiki-core. [21:33:28] does it need both what? [21:33:44] legoktm: Does mwext-name-composer need both zend and hhvm [21:33:44] We still need the testextension job yes [21:33:50] Umm, I think so [21:33:52] Why [21:34:07] well we need zend for 5.3 syntax checking [21:34:41] I guess hhvm isn't absolutely required if it's just doing lint + phpcs for now [21:34:53] But not setting up hhvm seems backwards. [21:36:06] I'd say pick one and remove it from the job name so we can change it without inflating 2x the workspaces. [21:36:23] every job name creates a clone on the slave that persists [21:36:43] even if the job is never run? [21:36:55] It won't create zend on trusty slaves. [21:37:15] What I mean is, when we consolidate these it'll be a different name and thus keep the old one [21:37:32] And also in the future: Every job is a vm. [21:37:41] Just be conservative, that's all :) [21:38:06] well I made it all the way to INFO:jenkins_jobs.builder:Creating jenkins job mwext-SubPageList3-composer-zend [21:38:12] hashar: Is it correct that you intend to use a shared job for phpunit? It makes a lot of sense, but I'm curious how that works with the workspace. [21:38:15] should I just drop the -hhvm jobs for now and only set up zend? [21:39:04] Yeah, I'd start with that. And use zend as an internal implementation detail, not the job name. We could even in the future compile php53 on trusty and use it for php linting. [21:39:32] well if I remove zend from the name I have to re-deploy them all again... :/ [21:39:43] Or we could consolidate all things (lint/phpcs via composer-test, phpunit, qunit) and fragment that between precise/trusty as two jobs. [21:39:49] That'd get better coverage and still run faster. [21:40:18] legoktm: If my suspicion is correct you won't need to compile one for each repo. [21:40:33] legoktm: for mass renaming, we can always rename them on disk :=) [21:41:13] legoktm: From where do you run the script? I usually run it from a labs slave (integration-dev-precise) home directory in a screen to reduce network latency [21:41:23] o.O [21:41:28] I've been running it from my laptop [21:41:35] which is quite significant for jenkins [21:42:16] I just clone the repo there in my home dir, install jenkins-job-builder with sudo (it's a dev instance, feel free) and run it :) [21:42:34] use the non-gitreview command from gerrit to fetch the change [21:42:52] so, are you suggesting that we should have a generic job like the php-composer-validate one which automatically figures out which project its for and runs the commands? [21:44:38] Im trying to think of a reason why that wouldn't work [21:45:02] Ah, it uses git-remote-zuul-shallow-clone [21:45:11] Which wipes the workspace and re-clones each run [21:45:16] Meh, yeah, that's fine [21:45:19] And actually even better [21:45:56] it can't use workspace as cache naturally. But that's fine. [21:46:01] We're working on making those clones fater [21:46:02] faster [21:46:10] as long as its not for mediawiki-core, it should be fast enough [21:46:45] hashar: I didn't realise php-composer-validate worked that way. Is that your invention? Nice! I thought it would be too slow, quite nice. [21:47:30] Ah "shallow-clone" [21:47:35] Awesome [21:49:49] ok, I'll re-work my patch to use that approach then [21:50:21] I guess I should delete the jobs that aren't going to be used? How do I do that? [21:51:17] legoktm: Jenkins UI is the most reliable [21:51:38] Alternatively, there is a way to do it with jenkins-jobs cli util, but that will delete all jobs not managed by jjb [21:51:55] which might delete too much. I filed a tech debt task for that to figure that out [21:52:05] :/ that's a lot of jobs to do manually... [21:52:12] Just let them be for now [21:52:18] don't hurt much [21:52:23] ok [21:52:34] https://phabricator.wikimedia.org/T91410 [21:53:18] legoktm: http://ci.openstack.org/jenkins-job-builder/installation.html#deleting-jobs [21:53:35] legoktm: There might be a glob pattern you can use. Youll have to checkout the change that defines them first and compile that. [21:54:23] the glob would be 'mwext-*-composer-*' [21:56:13] legoktm: Hi! Are https://gerrit.wikimedia.org/r/#/c/194340/ and https://gerrit.wikimedia.org/r/#/c/194331/ ready? [21:58:43] aharoni: the ContentTranslation patch looks fine, Krinkle and I discussed the jenkins changes and we're going to do something different, I'll set up the config for that in a few hours [21:59:13] (03PS1) 10Krinkle: Re-enable mwext-MultimediaViewer-jsduck-publish in postmerge [integration/config] - 10https://gerrit.wikimedia.org/r/194704 [21:59:34] legoktm: Thanks. It's never the focus of my work, but occasionally I'm curious about how this tooling works. [21:59:49] (03CR) 10Krinkle: [C: 032] Re-enable mwext-MultimediaViewer-jsduck-publish in postmerge [integration/config] - 10https://gerrit.wikimedia.org/r/194704 (owner: 10Krinkle) [22:00:58] (03Merged) 10jenkins-bot: Re-enable mwext-MultimediaViewer-jsduck-publish in postmerge [integration/config] - 10https://gerrit.wikimedia.org/r/194704 (owner: 10Krinkle) [22:01:38] !log Reloading Zuul to deploy I97c1d639313b [22:01:39] I can probably explain it in a bit when I'm not trying to race against an upcoming SWAT window :) [22:01:41] Logged the message, Master [22:01:50] greg-g: Sanity check? AFTv5 is so not-enabled that we've actually archived the project. However, 53 tickets are sitting around still open. Should I close them? [22:02:26] James_F: Just don't be too giddy while doing so. [22:02:30] * James_F grins. [22:04:38] 10Continuous-Integration, 7Technical-Debt, 7Tracking: All repositories should pass jshint test (tracking) - https://phabricator.wikimedia.org/T62619#1093911 (10Jdforrester-WMF) [22:05:58] Krinkle: sorry still busy packaging [22:06:13] Krinkle: there is a more or less shared jobs for some extensions [22:06:32] Krinkle: at least for the mobile related extensions and their PHPUnit tests. [22:06:44] greg-g: Is https://phabricator.wikimedia.org/T40516 on the roadmap? [22:06:52] hashar: Could we have a generic job (similar to php-composer-validate) for mwcore+ 1 extension? [22:06:57] E.g. for the majority [22:07:08] yup [22:07:19] hashar: And I suppose I could use that for npm-install and jsduck as well? [22:07:22] hey releng! I'm not able to run mwscript on deployment-bastion. whats up? [22:07:27] e.g. jsduck instead of name-jsduck [22:07:31] manybubbles: there is a bug for it [22:07:44] manybubbles: because mwscript uses sudo -u apache, instead of www-data [22:07:45] ah - is that, like, important? [22:07:49] manybubbles: pending prod upgrade [22:07:52] ah [22:08:12] manybubbles: workaround is to sudo as root [22:08:54] so I sudo mwscript? [22:09:14] manybubbles: I cant find the task :( [22:09:28] its ok! I can wait [22:09:32] I mean, wait days [22:09:34] its fine [22:09:44] I'll just do it in a few days when presumably it'll be all fixed [22:10:00] Krinkle: I think that is https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/ job [22:10:43] Krinkle: it is triggered by a bunch of repositories hardcoded in the job. Zuul cloner takes care of checking out the appropriate branch / zuul refs then run the PHPUnit extensions testsuite [22:10:46] hashar: That's the one for testing together. [22:11:02] (03PS1) 10Dduvall: Fix regex pattern in shared "I am logged in" step [selenium] - 10https://gerrit.wikimedia.org/r/194710 [22:11:04] James_F: PM incoming [22:11:15] Krinkle: and that uses an extension loader which list the extensions to load [22:11:21] hashar: That won't work for testing 1 extension. E.g. non-wmf stuff, or stuff that needs better tests before we can include it [22:11:34] Krinkle: so in theory we could have a workspace with a bunch of extension and ask the extension loader to only load one [22:11:40] Ah, righ [22:11:41] t [22:11:58] that would be nice for tests of core + one extension [22:12:01] (03CR) 10Cmcmahon: [C: 032] "paired with Dan. shared Login step was not working" [selenium] - 10https://gerrit.wikimedia.org/r/194710 (owner: 10Dduvall) [22:12:17] I would also like to have unit tests for extension that do not require mw/core [22:12:22] (only PM'ing because I'm paste bombing 20 lines of text) [22:12:26] hashar: Yeah, but that's gonna take another year. [22:12:27] ie just: git clone extensions/Foobar && phpunit [22:12:39] yeah I am getting old [22:12:44] Although individuadl extensions can start doing that [22:12:50] We can do it bottom up [22:12:51] now I might well die before that actually happen [22:12:55] Just like I did for qunit [22:13:00] hashar, manybubbles: that was https://phabricator.wikimedia.org/T89802 ... [22:13:01] yup [22:13:07] (03PS2) 10Dduvall: Fix regex pattern in shared "I am logged in" step [selenium] - 10https://gerrit.wikimedia.org/r/194710 [22:13:18] Krenair: oh solved already. Thank you :) [22:13:28] hashar: Ssss, you'll become millionaire and king of France before you die. [22:13:35] Wait, is it https://phabricator.wikimedia.org/T89165 ? [22:13:55] Krinkle: first is done. Second I am missing the blood lineage :D [22:14:02] Hehe [22:14:24] (03CR) 10Cmcmahon: "recheck" [selenium] - 10https://gerrit.wikimedia.org/r/194710 (owner: 10Dduvall) [22:14:28] pff [22:14:29] I hear the second one is for sale. [22:14:37] It's not like royal families follow rules. [22:14:41] Revolution! [22:14:52] I would not mind having a king in France [22:14:56] (03CR) 10Cmcmahon: Fix regex pattern in shared "I am logged in" step [selenium] - 10https://gerrit.wikimedia.org/r/194710 (owner: 10Dduvall) [22:14:59] but with autonomous regions :) [22:15:04] Benevolent dictator [22:15:07] (03CR) 10Cmcmahon: [C: 032] Fix regex pattern in shared "I am logged in" step [selenium] - 10https://gerrit.wikimedia.org/r/194710 (owner: 10Dduvall) [22:15:27] (03Merged) 10jenkins-bot: Fix regex pattern in shared "I am logged in" step [selenium] - 10https://gerrit.wikimedia.org/r/194710 (owner: 10Dduvall) [22:16:41] so [22:16:52] today I learned that when building a package there is no network access :( [22:17:00] took me the whole afternoon to figure out I need to pass: USENETWORK=yes [22:17:41] build command of doom just keeps growing [22:17:51] oh [22:17:58] I havent found out how to pass it in the build command :( [22:18:02] (03PS1) 10Dduvall: Releasing patch version 1.0.1 [selenium] - 10https://gerrit.wikimedia.org/r/194713 [22:18:11] heh [22:18:58] (03CR) 10Cmcmahon: [C: 032] "paired with Dan" [selenium] - 10https://gerrit.wikimedia.org/r/194713 (owner: 10Dduvall) [22:19:14] (03Merged) 10jenkins-bot: Releasing patch version 1.0.1 [selenium] - 10https://gerrit.wikimedia.org/r/194713 (owner: 10Dduvall) [22:20:09] greg-g: can we hire a team of volunteers to triage our email notifications ? :ÿ [22:20:12] I cant keep up [22:20:29] I received something like 150 mails from Phabricator over the last six hours [22:20:34] sorry for the BC workboard column moving spam from me :) [22:21:07] I do it like you do for CI, every now and then move ~20 old closed tickets from the backlog column to the Done column [22:21:10] I should unsubscribe a lot more [22:21:14] that too [22:21:40] <^demon|busy> James_F: I don't agree with closing AFTv5 bugs. We don't usually close bugs just because nobody's working on something [22:21:42] the default email settings are just too spammy, I removed a bunch of em [22:21:44] we should ask aklapper, there might be a way to have phabricator set the column for us automatically when the task is resolved [22:21:54] ^demon|busy: Then why did we close the project? [22:22:05] <^demon|busy> Don't ask me, I didn't say to do that either :) [22:22:16] ^demon|busy: Meh. Well, one or the other. [22:22:43] Eloquence: bugzilla was a bit nicer, you could change a lot of fields at once and it only caused a single notification [22:22:54] hashar: not yet :/ [22:22:54] Eloquence: you are right, I should look at the email settings [22:23:04] <^demon|busy> James_F: We have a lot of extensions that nobody is working on :) [22:23:19] what about dead projects? [22:23:28] "how do you define dead" "I dont' know..." [22:23:53] ^demon|busy: I leave this to greg-g to determine what makes a dead project. :-D [22:24:06] heh [22:24:15] I heard wikitext editor is dead, close it [22:24:39] <^demon|busy> I think as long as there's code the project should still exist. Sure, there may be no WMF group working on it (and such groups can be archived) [22:25:00] <^demon|busy> But if the code exists and has bugs and isn't superseded by something... [22:26:04] ^demon|busy: It's been superseded by not using it… ;-) [22:26:55] IMO we should only close bugs if the repo is archived and emptied [22:27:04] <^demon|busy> ^ that [22:27:26] <^demon|busy> Long as the repo's still there it's a legit extension, even if WMF doesn't use it anymore :) [22:28:01] fair [22:28:42] https://wikiapiary.com/wiki/Extension:Article_Feedback [22:29:24] ^demon|busy: So if I do a "delete all the things" commit to the repo and we merge it, I can not unmerge them? :-) [22:29:36] didn't we have a project to clear out / archive old extensions ? [22:29:53] I would love us to migrate some to an attic [22:30:26] * ^demon|busy thinks that extensions are harmless and unless they're actively broken we shouldn't remove them just because they don't have a maintainer [22:31:25] * James_F thinks that known-broken extensions are harmful and leaving them around as rakes in the long grass is disrespectful to unwary sysadmins who don't know the back-story. [22:31:36] :-) [22:32:28] there is some maintenance overhead, Jenkins comes to mind [22:32:36] or l10n peopl [22:32:37] e [22:32:52] I am sure we have people investing effort in translating messages which are not used anywhere :( [22:32:52] <^demon|busy> If they're not being updated often the burden should be low [22:33:28] meanwhile, I am screwed with packaging [22:33:42] <^demon|busy> Anyway, I don't want to have this conversation right now :p [22:33:46] * ^demon|busy goes back to being |busy [22:33:49] neither do i [22:33:56] * hashar heads to bed [22:39:41] (03PS2) 10Hashar: Package python deps with dh-virtualenv [integration/zuul] (debian-precise-venv) - 10https://gerrit.wikimedia.org/r/194520 [22:42:30] (03PS3) 10Hashar: Package python deps with dh-virtualenv [integration/zuul] (debian-precise-venv) - 10https://gerrit.wikimedia.org/r/194520 [22:42:42] src/MD2.c:31:20: fatal error: Python.h: No such file or directory [22:42:44] yeah obviously [22:45:03] see you tomorrow [22:56:23] 10Continuous-Integration, 6translatewiki.net: l10n-bot self-force-merging sometimes breaks mediawiki/core master - https://phabricator.wikimedia.org/T91707#1094080 (10matmarex) 3NEW [22:56:44] 10Continuous-Integration, 6Labs, 10Wikimedia-Labs-Infrastructure: Diamond collected metrics about memory usage inaccurate until third reboot - https://phabricator.wikimedia.org/T91351#1094087 (10Krinkle) [23:08:53] greg-g: I know what I forgot to bring up, I sent mail last week: are you up to speed on this Gather extension? [23:09:13] I saw it is on the way to prod now [23:09:28] Yippee, build fixed! [23:09:29] Project browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #10: FIXED in 3 min 50 sec: https://integration.wikimedia.org/ci/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/10/ [23:12:22] timing is everything [23:12:37] (03CR) 10Awight: [C: 031] CentralNotice browser tests: even more platforms and browsers [integration/config] - 10https://gerrit.wikimedia.org/r/193556 (https://phabricator.wikimedia.org/T86092) (owner: 10AndyRussG) [23:14:28] chrismcmahon: yeah, as much as I can be I think [23:17:55] I'm not, except in the vaguest way. [23:18:03] it's on my list [23:18:21] yeah, that's where I am [23:19:28] I am a little cranky about how it got onto beta labs, it seems to be moving very quickly without much of a mandate [23:34:52] Project beta-scap-eqiad build #44075: FAILURE in 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/44075/ [23:41:58] (03PS1) 10Mattflaschen: Pass MEDIAWIKI_CAPTCHA_BYPASS_PASSWORD to GettingStarted browser test [integration/config] - 10https://gerrit.wikimedia.org/r/194749 (https://phabricator.wikimedia.org/T91220) [23:53:04] 10Continuous-Integration, 10Wikimedia-Hackathon-2015: Jenkins: Set up PHPUnit testing on MySQL backend - https://phabricator.wikimedia.org/T37912#1094261 (10Mattflaschen)