[00:15:17] Project beta-scap-eqiad build #18066: FAILURE in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/18066/ [00:17:18] wmf-insecte: are you a bot? [00:17:18] spagewmf you may not issue bot commands in this chat! [00:17:47] wmf-insecte: there oughtta be a law that bots have bot in their nickname [00:17:47] spagewmf you may not issue bot commands in this chat! [00:18:32] I recall getting build failure messages from a bot named "...bot" [00:20:10] its the jenkins bot [00:20:23] nick is set in https://integration.wikimedia.org/ci/configure [00:20:35] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #571: SUCCESS in 33 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/571/ [00:23:12] Apparently insecte is french for insect so hashar was being cute that it bugs us about bugs [00:25:20] Yippee, build fixed! [00:25:21] Project beta-scap-eqiad build #18067: FIXED in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/18067/ [00:26:48] yippee, build fixed, when it fails it should go 'Yippee, a bug for you to fix!' :p [00:33:02] All the scap failures today have been transient network & dns errors in labs, so I think it should say "labs network is flakey again" [00:33:39] I wonder if I can set it to only alert on consecutive failures... [00:35:01] nope :( [00:46:23] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #572: SUCCESS in 23 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/572/ [00:53:34] Project UploadWizard-api-commons.wikimedia.org build #457: SUCCESS in 34 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.org/457/ [01:05:13] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #168: STILL FAILING in 46 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/168/ [03:05:08] Project browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs build #89: STILL FAILING in 8 sec: https://integration.wikimedia.org/ci/job/browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs/89/ [03:06:44] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #56: SUCCESS in 43 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/56/ [03:11:48] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #113: SUCCESS in 1 min 47 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/113/ [03:14:04] Project browsertests-TwnMainPage-sandbox.translatewiki.net-linux-firefox-sauce build #85: STILL FAILING in 10 min: https://integration.wikimedia.org/ci/job/browsertests-TwnMainPage-sandbox.translatewiki.net-linux-firefox-sauce/85/ [03:19:27] Project browsertests-UniversalLanguageSelector-language-browsertests.wmflabs.org-linux-firefox-sauce build #114: STILL FAILING in 1 min 27 sec: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-language-browsertests.wmflabs.org-linux-firefox-sauce/114/ [03:21:09] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #90: SUCCESS in 1 min 9 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/90/ [03:22:15] Project browsertests-UniversalLanguageSelector-sandbox.translatewiki.net-linux-firefox-sauce build #85: SUCCESS in 1 min 14 sec: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-sandbox.translatewiki.net-linux-firefox-sauce/85/ [03:28:25] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce build #124: FAILURE in 9 min 24 sec: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce/124/ [03:29:06] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #45: SUCCESS in 1 min 5 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/45/ [03:31:20] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #119: FAILURE in 12 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/119/ [03:34:29] Yippee, build fixed! [03:34:29] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #5: FIXED in 8 min 28 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/5/ [03:35:15] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #88: SUCCESS in 2 min 15 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/88/ [03:38:34] Project browsertests-ContentTranslation-language-stage.wmflabs.org-linux-firefox-sauce build #75: STILL FAILING in 2 min 34 sec: https://integration.wikimedia.org/ci/job/browsertests-ContentTranslation-language-stage.wmflabs.org-linux-firefox-sauce/75/ [03:39:45] Project browsertests-Translate-meta.wikimedia.org-linux-firefox-sauce build #113: SUCCESS in 44 sec: https://integration.wikimedia.org/ci/job/browsertests-Translate-meta.wikimedia.org-linux-firefox-sauce/113/ [03:39:54] Project browsertests-Flow-test2.wikipedia.org-windows_8-internet_explorer-sauce build #99: STILL FAILING in 38 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-windows_8-internet_explorer-sauce/99/ [03:41:20] Project browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce build #89: SUCCESS in 1 min 19 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce/89/ [03:41:42] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #56: SUCCESS in 42 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/56/ [03:46:39] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #149: STILL FAILING in 20 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/149/ [03:46:42] Project browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce build #29: SUCCESS in 41 sec: https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce/29/ [03:48:27] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #131: STILL FAILING in 17 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/131/ [03:50:14] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #113: SUCCESS in 2 min 14 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/113/ [03:51:44] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #114: SUCCESS in 1 min 44 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/114/ [03:54:18] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #169: STILL FAILING in 43 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/169/ [03:59:23] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #114: SUCCESS in 2 min 23 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/114/ [04:01:09] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #57: SUCCESS in 1 min 45 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/57/ [04:03:54] Project browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce build #115: SUCCESS in 8 min 54 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/115/ [04:19:36] Project browsertests-VisualEditor-test2.wikipedia.org-linux-chrome-sauce build #135: STILL FAILING in 59 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-chrome-sauce/135/ [04:19:56] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-windows_xp-firefox-sauce build #130: STILL FAILING in 59 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-windows_xp-firefox-sauce/130/ [04:20:33] Project browsertests-Flow-test2.wikipedia.org-linux-firefox-sauce build #101: STILL FAILING in 32 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-firefox-sauce/101/ [04:20:49] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #197: STILL FAILING in 53 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/197/ [04:22:28] Yippee, build fixed! [04:22:29] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #117: FIXED in 25 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/117/ [04:27:48] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #104: STILL FAILING in 39 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/104/ [04:29:33] Project browsertests-Translate-sandbox.translatewiki.net-linux-firefox-sauce build #85: STILL FAILING in 32 min: https://integration.wikimedia.org/ci/job/browsertests-Translate-sandbox.translatewiki.net-linux-firefox-sauce/85/ [04:34:30] Project browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce build #101: STILL FAILING in 42 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce/101/ [04:35:05] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #157: STILL FAILING in 48 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/157/ [04:37:13] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #154: STILL FAILING in 1 hr 5 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/154/ [04:44:49] Project browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce build #119: STILL FAILING in 57 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce/119/ [04:46:02] Project browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce build #132: STILL FAILING in 54 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce/132/ [04:47:03] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #173: STILL FAILING in 1 hr 5 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/173/ [05:45:16] Project beta-scap-eqiad build #18095: FAILURE in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/18095/ [05:45:25] (03PS9) 10Phuedx: Enable browser tests for the GettingStarted extension [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/150172 [05:54:03] Project beta-code-update-eqiad build #20888: FAILURE in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/20888/ [06:03:53] Yippee, build fixed! [06:03:53] Project beta-code-update-eqiad build #20889: FIXED in 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/20889/ [06:04:59] Yippee, build fixed! [06:04:59] Project beta-scap-eqiad build #18096: FIXED in 1 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/18096/ [06:46:25] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #573: SUCCESS in 25 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/573/ [06:53:28] Project UploadWizard-api-commons.wikimedia.org build #458: SUCCESS in 27 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.org/458/ [08:22:07] (03PS1) 10Hashar: Drop vim modeline from mobile.yaml [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155502 [08:22:10] (03PS1) 10Hashar: js/php linter for apps/ios/wikipedia.git [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155503 [08:22:22] (03CR) 10Hashar: [C: 032] Drop vim modeline from mobile.yaml [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155502 (owner: 10Hashar) [08:22:27] (03CR) 10Hashar: [C: 032] js/php linter for apps/ios/wikipedia.git [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155503 (owner: 10Hashar) [08:23:42] (03Merged) 10jenkins-bot: Drop vim modeline from mobile.yaml [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155502 (owner: 10Hashar) [08:24:16] (03Merged) 10jenkins-bot: js/php linter for apps/ios/wikipedia.git [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155503 (owner: 10Hashar) [08:26:37] 3Wikimedia / 3Continuous integration: All repositories should pass jshint test (tracking) - 10https://bugzilla.wikimedia.org/60619 (10Antoine "hashar" Musso) [08:29:53] (03PS1) 10Hashar: Triggers for apps/ios/wikipedia [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/155504 [08:30:31] (03CR) 10Hashar: [C: 032] Triggers for apps/ios/wikipedia [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/155504 (owner: 10Hashar) [08:31:06] (03PS2) 10Hashar: Triggers for apps/ios/wikipedia [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/155504 [08:31:49] (03CR) 10Hashar: [C: 032] Triggers for apps/ios/wikipedia [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/155504 (owner: 10Hashar) [08:31:58] (03Merged) 10jenkins-bot: Triggers for apps/ios/wikipedia [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/155504 (owner: 10Hashar) [09:14:51] (03PS10) 10Phuedx: Enable browser tests for the GettingStarted extension [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/150172 [09:15:01] (03CR) 10Phuedx: "PS10 is a rebase." [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/150172 (owner: 10Phuedx) [09:33:20] (03PS1) 10Hashar: Remove integration-jjb-config-test [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155508 [09:42:39] (03PS1) 10Hashar: Remove integration-jjb-config-test [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/155509 [09:42:43] (03CR) 10Hashar: [C: 032] Remove integration-jjb-config-test [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/155509 (owner: 10Hashar) [09:42:46] (03Merged) 10jenkins-bot: Remove integration-jjb-config-test [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/155509 (owner: 10Hashar) [09:43:01] (03CR) 10Hashar: "recheck" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155508 (owner: 10Hashar) [09:46:51] (03CR) 10Hashar: [C: 032] Remove integration-jjb-config-test [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155508 (owner: 10Hashar) [09:48:53] (03Merged) 10jenkins-bot: Remove integration-jjb-config-test [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155508 (owner: 10Hashar) [10:01:11] (03CR) 10Hashar: "recheck" [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/155337 (owner: 10Hashar) [10:11:54] (03CR) 10Hashar: "I have installed the plugin, seems to work as expected." [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/155337 (owner: 10Hashar) [10:12:08] (03CR) 10Hashar: "recheck" [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/155337 (owner: 10Hashar) [12:46:25] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #574: SUCCESS in 25 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/574/ [12:53:34] Project UploadWizard-api-commons.wikimedia.org build #459: SUCCESS in 33 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.org/459/ [13:03:07] 3Wikimedia / 3Continuous integration: Jenkins: shared job php-composer-validate should use a shallow git clone - 10https://bugzilla.wikimedia.org/69433 (10Antoine "hashar" Musso) p:5Unprio>3Normal s:5normal>3enhanc a:3Antoine "hashar" Musso [13:10:19] (03PS1) 10Hashar: php-composer-validate now use a shallow clone [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155534 (https://bugzilla.wikimedia.org/69433) [13:10:39] (03CR) 10Hashar: [C: 032] php-composer-validate now use a shallow clone [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155534 (https://bugzilla.wikimedia.org/69433) (owner: 10Hashar) [13:12:42] (03Merged) 10jenkins-bot: php-composer-validate now use a shallow clone [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155534 (https://bugzilla.wikimedia.org/69433) (owner: 10Hashar) [13:13:20] 3Wikimedia / 3Continuous integration: Jenkins: shared job php-composer-validate should use a shallow git clone - 10https://bugzilla.wikimedia.org/69433 (10Antoine "hashar" Musso) 5PATC>3RESO/FIX [13:26:05] 3Wikimedia / 3Continuous integration: Need a way to simulate replication lag to test replag issues - 10https://bugzilla.wikimedia.org/38945#c11 (10Antoine "hashar" Musso) When we migrated the beta cluster from pmtpa to eqiad, Sean Pringle added a master / slave setup on beta. Apparently the slave is usuall... [13:28:06] 3Wikimedia / 3Continuous integration: Install PHP xdiff extension on gallium (PHPUnit skipping tests) - 10https://bugzilla.wikimedia.org/41683#c10 (10Antoine "hashar" Musso) 5NEW>3RESO/WON Probably nobody is going to package libxdiff / php-xdiff so we can get installed so I am closing this bug. If there... [13:32:05] 3Wikimedia / 3Continuous integration: Jenkins: Create PHP_CodeSniffer‎ sniff to ensure test functions have the proper signature - 10https://bugzilla.wikimedia.org/46483 (10Antoine "hashar" Musso) s:5normal>3enhanc [13:34:06] 3Wikimedia / 3Continuous integration: Jenkins: Fail on BOM in submitted files - 10https://bugzilla.wikimedia.org/38233 (10Antoine "hashar" Musso) [13:34:07] 3Wikimedia / 3Continuous integration: Jenkins: Set up job to detect bad patterns (tracking) - 10https://bugzilla.wikimedia.org/47054 (10Antoine "hashar" Musso) [13:36:35] 3Wikimedia / 3Continuous integration: Zuul: project-templates override defined project jobs - 10https://bugzilla.wikimedia.org/50112#c4 (10Antoine "hashar" Musso) 5NEW>3RESO/FIX The upstream change has been merged in and our Zuul version now includes it. We already have a few project using that feature... [13:40:51] 3Wikimedia / 3Continuous integration: remove all tab characters from DNS repository, then enable jenkins check - 10https://bugzilla.wikimedia.org/68769#c15 (10Antoine "hashar" Musso) 5NEW>3RESO/FIX This is now completed. Another bug tracked the addition of the Jenkins job. [13:42:50] 3Wikimedia / 3Continuous integration: Enhance tutorial to trigger a job manually - 10https://bugzilla.wikimedia.org/61322#c2 (10Antoine "hashar" Musso) 5NEW>3RESO/WON This is better done by commenting 'recheck' in Gerrit. [14:39:35] (03PS1) 10Hashar: Point phpcs mw standard to git-deploy version [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155547 (https://bugzilla.wikimedia.org/64858) [14:46:21] 3Wikimedia / 3Continuous integration: Need a way to simulate replication lag to test replag issues - 10https://bugzilla.wikimedia.org/38945#c12 (10Niklas Laxström) According to my latest knowledge the replication delay setting only exists in recent MySQL [1] and not in MariaDB - unless the feature has been a... [14:50:44] (03CR) 10Hashar: [C: 032] Point phpcs mw standard to git-deploy version [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155547 (https://bugzilla.wikimedia.org/64858) (owner: 10Hashar) [14:52:46] (03Merged) 10jenkins-bot: Point phpcs mw standard to git-deploy version [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155547 (https://bugzilla.wikimedia.org/64858) (owner: 10Hashar) [14:53:07] 3Wikimedia / 3Continuous integration: Need a way to simulate replication lag to test replag issues - 10https://bugzilla.wikimedia.org/38945#c13 (10Antoine "hashar" Musso) That might depends on the percona toolkit + some custom setup. I think production has slaves which have a 24 hours delay. Someone should... [14:58:30] (03PS1) 10Hashar: Remove the codesniffer submodule [integration/jenkins] - 10https://gerrit.wikimedia.org/r/155552 (https://bugzilla.wikimedia.org/64858) [15:06:38] (03CR) 10Hashar: [C: 032] Remove the codesniffer submodule [integration/jenkins] - 10https://gerrit.wikimedia.org/r/155552 (https://bugzilla.wikimedia.org/64858) (owner: 10Hashar) [15:06:53] (03Merged) 10jenkins-bot: Remove the codesniffer submodule [integration/jenkins] - 10https://gerrit.wikimedia.org/r/155552 (https://bugzilla.wikimedia.org/64858) (owner: 10Hashar) [15:12:50] 3Wikimedia / 3Continuous integration: Jenkins: Remove "tools/mwcodesniffer" submodule from integration/jenkins.git - 10https://bugzilla.wikimedia.org/64858#c7 (10Antoine "hashar" Musso) 5PATC>3RESO/FIX Now handled via git-deploy on prod and git::clone on labs. The submodule has been removed and cleaned... [15:24:09] Project beta-scap-eqiad build #18152: FAILURE in 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/18152/ [15:31:34] Yippee, build fixed! [15:31:34] Project beta-scap-eqiad build #18153: FIXED in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/18153/ [15:46:19] (03PS2) 10Hashar: Throttle browser tests at 1 per node max [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/155337 [15:46:48] (03CR) 10jenkins-bot: [V: 04-1] Throttle browser tests at 1 per node max [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/155337 (owner: 10Hashar) [15:48:40] (03PS3) 10Hashar: Throttle browser tests at 1 per node max [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/155337 [15:50:45] (03PS4) 10Hashar: Throttle browser tests at 1 per node max [integration/jenkins-job-builder-config] (cloudbees) - 10https://gerrit.wikimedia.org/r/155337 [15:56:08] (03PS1) 10Hashar: MediaWiki skins: +jslint job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155571 [15:56:51] (03PS2) 10Hashar: MediaWiki skins: +jslint job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155571 [15:59:50] (03PS1) 10Hashar: Trigger mw-skins-Nimbus-jslint [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/155573 (https://bugzilla.wikimedia.org/68335) [16:00:03] (03CR) 10Hashar: [C: 032] MediaWiki skins: +jslint job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155571 (owner: 10Hashar) [16:00:42] (03CR) 10Hashar: [C: 032] Trigger mw-skins-Nimbus-jslint [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/155573 (https://bugzilla.wikimedia.org/68335) (owner: 10Hashar) [16:01:06] (03Merged) 10jenkins-bot: Trigger mw-skins-Nimbus-jslint [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/155573 (https://bugzilla.wikimedia.org/68335) (owner: 10Hashar) [16:02:07] (03Merged) 10jenkins-bot: MediaWiki skins: +jslint job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155571 (owner: 10Hashar) [16:02:31] and off [16:11:45] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #198: ABORTED in 13 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/198/ [16:50:04] wmf-insecte has gotten noisy [16:59:32] Yippee, build fixed! [16:59:49] Doesn't sound very French [17:00:00] bd808: heh. it didn't used to report ABORTED and stuff [18:05:18] Project browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs build #90: STILL FAILING in 18 sec: https://integration.wikimedia.org/ci/job/browsertests-ZeroBanner-en.m.wikipedia.org-linux-phantomjs/90/ [18:06:48] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #57: SUCCESS in 47 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/57/ [18:11:38] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #114: SUCCESS in 1 min 38 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/114/ [18:14:44] Project browsertests-TwnMainPage-sandbox.translatewiki.net-linux-firefox-sauce build #86: STILL FAILING in 10 min: https://integration.wikimedia.org/ci/job/browsertests-TwnMainPage-sandbox.translatewiki.net-linux-firefox-sauce/86/ [18:19:30] Project browsertests-UniversalLanguageSelector-language-browsertests.wmflabs.org-linux-firefox-sauce build #115: STILL FAILING in 1 min 30 sec: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-language-browsertests.wmflabs.org-linux-firefox-sauce/115/ [18:21:33] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #91: SUCCESS in 1 min 32 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/91/ [18:21:53] Project browsertests-Flow-test2.wikipedia.org-windows_8-internet_explorer-sauce build #100: STILL FAILING in 20 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-windows_8-internet_explorer-sauce/100/ [18:22:24] Project browsertests-UniversalLanguageSelector-sandbox.translatewiki.net-linux-firefox-sauce build #86: SUCCESS in 1 min 23 sec: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-sandbox.translatewiki.net-linux-firefox-sauce/86/ [18:29:16] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #46: SUCCESS in 1 min 16 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/46/ [18:32:31] Yippee, build fixed! [18:32:31] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #120: FIXED in 13 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/120/ [18:32:38] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce build #125: STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce/125/ [18:35:05] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #6: FAILURE in 9 min 4 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/6/ [18:35:15] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #89: SUCCESS in 2 min 15 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/89/ [18:36:02] Project browsertests-ContentTranslation-language-stage.wmflabs.org-linux-firefox-sauce build #76: STILL FAILING in 1.7 sec: https://integration.wikimedia.org/ci/job/browsertests-ContentTranslation-language-stage.wmflabs.org-linux-firefox-sauce/76/ [18:39:54] Project browsertests-Translate-meta.wikimedia.org-linux-firefox-sauce build #114: SUCCESS in 54 sec: https://integration.wikimedia.org/ci/job/browsertests-Translate-meta.wikimedia.org-linux-firefox-sauce/114/ [18:40:56] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #170: STILL FAILING in 29 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/170/ [18:41:19] Project browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce build #90: SUCCESS in 1 min 19 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce/90/ [18:41:50] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #57: SUCCESS in 49 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/57/ [18:42:52] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #132: STILL FAILING in 11 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/132/ [18:46:43] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #575: SUCCESS in 43 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/575/ [18:47:00] Project browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce build #30: FAILURE in 1 min 0 sec: https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce/30/ [18:50:46] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #114: SUCCESS in 2 min 45 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/114/ [18:51:47] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #115: SUCCESS in 1 min 47 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/115/ [18:53:30] Project UploadWizard-api-commons.wikimedia.org build #460: SUCCESS in 29 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.org/460/ [18:59:12] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #115: SUCCESS in 2 min 11 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/115/ [19:00:37] 3Wikimedia / 3Continuous integration: jenkins on release branch has problem when patch set contains js files - 10https://bugzilla.wikimedia.org/65085#c3 (10Krinkle) Indeed. For pre-1.23 builds, resort to the global install of jshint (the same job we use for the 'check' pipeline of mediawiki-core, mediawiki-c... [19:05:18] Project browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce build #116: SUCCESS in 10 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/116/ [19:08:50] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #58: SUCCESS in 2 min 0 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/58/ [19:10:31] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #150: STILL FAILING in 44 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/150/ [19:14:58] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-windows_xp-firefox-sauce build #131: STILL FAILING in 54 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-windows_xp-firefox-sauce/131/ [19:20:04] (03PS2) 10Hashar: Add new extension repo: CodeMirror [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155290 (owner: 10Pastakhov) [19:22:11] (03CR) 10Hashar: [C: 032] "Jobs deployed:" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155290 (owner: 10Pastakhov) [19:22:16] Project browsertests-Flow-test2.wikipedia.org-linux-firefox-sauce build #102: STILL FAILING in 34 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-firefox-sauce/102/ [19:24:05] (03Merged) 10jenkins-bot: Add new extension repo: CodeMirror [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155290 (owner: 10Pastakhov) [19:25:37] 3Wikimedia / 3Continuous integration: All repositories should pass jshint test (tracking) - 10https://bugzilla.wikimedia.org/60619 (10Antoine "hashar" Musso) [19:26:04] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #199: STILL FAILING in 59 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/199/ [19:26:08] (03PS1) 10Hashar: Triggers for mw/ext/CodeMirror [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/155607 (https://bugzilla.wikimedia.org/69869) [19:26:27] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #118: SUCCESS in 27 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/118/ [19:26:46] (03CR) 10Hashar: [C: 032] Triggers for mw/ext/CodeMirror [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/155607 (https://bugzilla.wikimedia.org/69869) (owner: 10Hashar) [19:26:47] Project browsertests-VisualEditor-test2.wikipedia.org-linux-chrome-sauce build #136: STILL FAILING in 1 hr 6 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-chrome-sauce/136/ [19:26:58] (03Merged) 10jenkins-bot: Triggers for mw/ext/CodeMirror [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/155607 (https://bugzilla.wikimedia.org/69869) (owner: 10Hashar) [19:27:05] 3Wikimedia / 3Continuous integration: Create generic banana jenkins job template for comparing en.json and qqq.json entries - 10https://bugzilla.wikimedia.org/64045#c10 (10Krinkle) (In reply to Siebrand Mazeland from comment #9) > (In reply to James Forrester from comment #8) > > Krinkle wants essentially to... [19:27:22] 3Wikimedia / 3Continuous integration: Create generic banana jenkins job template for comparing en.json and qqq.json entries - 10https://bugzilla.wikimedia.org/64045 (10Krinkle) 5PATC>3NEW p:5Normal>3Low [19:28:16] (03CR) 10Hashar: "Zuul is configured with https://gerrit.wikimedia.org/r/#/c/155607/" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/155290 (owner: 10Pastakhov) [19:34:00] Project browsertests-Translate-sandbox.translatewiki.net-linux-firefox-sauce build #86: STILL FAILING in 36 min: https://integration.wikimedia.org/ci/job/browsertests-Translate-sandbox.translatewiki.net-linux-firefox-sauce/86/ [19:34:14] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #105: STILL FAILING in 46 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/105/ [19:36:24] !log upgrading elastic1007 to 1.3.2. this upgrade is pain due to https://github.com/elasticsearch/elasticsearch/issues/7386 . we'll have to be super careful [19:37:22] Project browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce build #102: STILL FAILING in 45 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce/102/ [19:41:59] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #155: STILL FAILING in 1 hr 9 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/155/ [19:46:44] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #158: STILL FAILING in 59 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/158/ [19:50:16] Project browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce build #120: STILL FAILING in 1 hr 3 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce/120/ [19:54:48] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #174: STILL FAILING in 1 hr 12 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/174/ [19:57:29] Project browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce build #133: STILL FAILING in 1 hr 5 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce/133/ [19:58:02] :-/ [20:02:19] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #200: ABORTED in 14 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/200/ [20:16:05] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #201: ABORTED in 13 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/201/ [20:27:47] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #202: ABORTED in 7 min 18 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/202/ [20:46:47] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #203: ABORTED in 18 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/203/ [21:29:56] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #175: ABORTED in 22 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/175/ [21:43:14] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #204: STILL FAILING in 56 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/204/ [22:24:12] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #176: STILL FAILING in 54 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/176/ [23:20:01] Yippee, build fixed! [23:20:01] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #205: FIXED in 55 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/205/ [23:23:51] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #177: STILL FAILING in 56 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/177/ [23:31:42] chrismcmahon: ^^^ Wow, a passing test suite for VE! [23:32:13] One out of eight, but hey. [23:33:29] James_F: I did a lot of updating, and y'all did a lot of bug fixing. The biggest failure mode right now is that Jenkins needs more horsepower. Antoine is going to throttle test jobs, but that's not the ultimate solution. [23:33:37] * James_F nods. [23:33:46] Do we need to throw some more boxes at the problem? [23:34:32] James_F: that's my theory. Jenkins just gets bogged down, sends no instructions for 90+ seconds, and after 90 seconds wait Sauce tears down the VM [23:34:46] Right. [23:35:51] James_F: so we can throttle jobs down to a small number and/or we can stagger the builds across all the projects, but bottom line is that Jenkins is way underpowered for how we are using it [23:38:08] James_F: btw, I added a bunch of tests for the new Cite UI, and looking forward to seeing Citoid, that would be nifty [23:38:17] Yeah. [23:38:30] Though it'll break the browser tests when we bring it in, I imagine. [23:38:45] oh sure, but that's normal [23:39:46] Sorry for not fixing the browser tests in the "insert xxx" to "insert" change. [23:39:57] I remembered for "reference list" to "references list"… [23:40:12] probably will anyway. I'm not sure that re-using all those generic textareas in Cite is a great idea design-wise, but it serves nicely right now [23:40:28] * James_F nods. [23:41:45] I test that not only by checking that the Cite diff is correct, but also that the descriptions of all the textareas are both correct and in the correct order. it's not perfect,but it's OK