[00:19:06] 10Beta-Cluster-Infrastructure, 10VisualEditor, 10VisualEditor-MediaWiki: Beta Cluster threw a 'internal_api_error_DBConnectionError' at me - https://phabricator.wikimedia.org/T129192#2097660 (10Jdforrester-WMF) [00:34:05] 10Beta-Cluster-Infrastructure, 10VisualEditor, 10VisualEditor-MediaWiki: Beta Cluster threw a 'internal_api_error_DBConnectionError' at me - https://phabricator.wikimedia.org/T129192#2097758 (10Jdforrester-WMF) (Seems to be generally working.) [00:35:20] 10Beta-Cluster-Infrastructure, 10VisualEditor, 10VisualEditor-MediaWiki: Beta Cluster threw a 'internal_api_error_DBConnectionError' at me - https://phabricator.wikimedia.org/T129192#2097660 (10AlexMonk-WMF) Did you get an exception ID? [00:36:19] 10Beta-Cluster-Infrastructure, 10VisualEditor, 10VisualEditor-MediaWiki: Beta Cluster threw a 'internal_api_error_DBConnectionError' at me - https://phabricator.wikimedia.org/T129192#2097763 (10Jdforrester-WMF) It does help if I remember to attach the screenshot, doesn't it? :-) {F3553891} [01:05:22] (03CR) 10Chad: [C: 032] Add "MF cache pollution debug log" live hack [tools/release] - 10https://gerrit.wikimedia.org/r/275701 (https://phabricator.wikimedia.org/T124356) (owner: 10Legoktm) [01:06:00] (03Merged) 10jenkins-bot: Add "MF cache pollution debug log" live hack [tools/release] - 10https://gerrit.wikimedia.org/r/275701 (https://phabricator.wikimedia.org/T124356) (owner: 10Legoktm) [01:50:26] !log integration-saltmater: salt -v '*slave-trusty*' cmd.run 'rm -rf /mnt/jenkins-workspace/workspace/mwext-testextension-hhvm-composer/src/skins/BlueSky' [01:50:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [01:50:35] !log integration-saltmater: salt -v '*slave-trusty*' cmd.run 'rm -rf /mnt/jenkins-workspace/workspace/mwext-testextension-hhvm/src/skins/BlueSky' (T117710) [01:50:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [01:54:40] 10Deployment-Systems, 10scap, 10Citoid, 10VisualEditor, 10Scap3 (scap3-adoption): Deploy zotero/translators with scap3 - https://phabricator.wikimedia.org/T129143#2096492 (10Jdforrester-WMF) p:5Triage>3Normal [02:14:34] (03PS1) 10Lethexie: add the spacebeforecontrolbrace sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/275747 [02:24:30] 10MediaWiki-Codesniffer, 10Possible-Tech-Projects: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#2098034 (10Lethexie) @Aklapper : Thank you, I have submitted my changes to Gerrit. [02:27:04] !log deployment-prep Updating HHVM on deployment-mediawiki02 [02:27:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:34:37] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #873: 04FAILURE in 1 min 36 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/873/ [02:36:31] ^ that specific failure was probably caused by me updating hhvm on deployment-mediawiki02, should be ok on next run [02:51:03] !log deployment-prep Updating HHVM on deployment-mediawiki01 [02:51:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:52:06] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #901: 04FAILURE in 2 min 6 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/901/ [02:54:30] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 50768 bytes in 0.010 second response time [02:55:58] that's me too [02:57:54] should be better now [02:59:30] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 29641 bytes in 0.703 second response time [03:12:03] PROBLEM - Free space - all mounts on deployment-jobrunner01 is CRITICAL: CRITICAL: deployment-prep.deployment-jobrunner01.diskspace.root.byte_percentfree (<50.00%) [03:13:35] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #1001: 04FAILURE in 31 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/1001/ [06:16:40] 10Beta-Cluster-Infrastructure: Could not find local user data for Riley Huntley@jawiki - https://phabricator.wikimedia.org/T129214#2098210 (10Riley_Huntley) [06:34:05] Yippee, build fixed! [06:34:06] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #39: 09FIXED in 7 min 5 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/39/ [06:39:52] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 6Reading-Infrastructure-Team: Could not find local user data for Riley Huntley@jawiki - https://phabricator.wikimedia.org/T129214#2098222 (10ori) [06:54:17] Yippee, build fixed! [06:54:17] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #751: 09FIXED in 29 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/751/ [07:47:06] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 6Reading-Infrastructure-Team, 7Wikimedia-log-errors: Could not find local user data for Riley Huntley@jawiki - https://phabricator.wikimedia.org/T129214#2098235 (10hashar) [08:17:33] Yippee, build fixed! [08:17:33] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #902: 09FIXED in 22 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/902/ [08:20:56] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #882: 04FAILURE in 55 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/882/ [08:30:13] Yippee, build fixed! [08:30:14] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #1002: 09FIXED in 35 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/1002/ [09:14:06] (03CR) 10Hashar: [C: 032] [DebugMode] Add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/275682 (owner: 10Paladox) [09:14:56] (03Merged) 10jenkins-bot: [DebugMode] Add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/275682 (owner: 10Paladox) [09:15:35] (03PS3) 10Hashar: [DiscussionThreading] Add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/275697 (owner: 10Paladox) [09:15:37] (03PS2) 10Hashar: [DeleteBatch] Add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/275686 (owner: 10Paladox) [09:15:39] (03PS2) 10Hashar: [Drafts] Add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/275704 (owner: 10Paladox) [09:15:41] (03PS2) 10Hashar: [Disambiguator] Add jshint to check: [integration/config] - 10https://gerrit.wikimedia.org/r/275691 (owner: 10Paladox) [09:17:11] (03PS3) 10Siebrand: Update WebChat settings [integration/config] - 10https://gerrit.wikimedia.org/r/275520 [09:23:21] (03PS4) 10Hashar: Update WebChat settings [integration/config] - 10https://gerrit.wikimedia.org/r/275520 (owner: 10Siebrand) [09:23:28] (03CR) 10Hashar: [C: 032] Update WebChat settings [integration/config] - 10https://gerrit.wikimedia.org/r/275520 (owner: 10Siebrand) [09:24:41] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/275704 (owner: 10Paladox) [09:26:10] (03Merged) 10jenkins-bot: Update WebChat settings [integration/config] - 10https://gerrit.wikimedia.org/r/275520 (owner: 10Siebrand) [09:39:18] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 6Reading-Infrastructure-Team, 7Wikimedia-log-errors: Could not find local user data for Riley Huntley@jawiki - https://phabricator.wikimedia.org/T129214#2098307 (10Glaisher) [09:39:20] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 7Beta-Cluster-reproducible: Could not find local user data for - https://phabricator.wikimedia.org/T115198#2098308 (10Glaisher) [09:41:14] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 7Beta-Cluster-reproducible: Could not find local user data for - https://phabricator.wikimedia.org/T115198#1717768 (10Glaisher) This was fixed for production at {T119736} [10:51:54] RECOVERY - Free space - all mounts on deployment-jobrunner01 is OK: OK: All targets OK [10:52:53] (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/275041 (owner: 10Paladox) [10:53:26] 10Browser-Tests-Infrastructure, 5Release-Engineering-Epics, 7Epic, 13Patch-For-Review, 7Tracking: Fix scenarios that fail at en.wikipedia.beta.wmflabs.org or do not run them daily - https://phabricator.wikimedia.org/T94150#2098436 (10zeljkofilipin) [10:53:39] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/275697 (owner: 10Paladox) [10:53:44] Yippee, build fixed! [10:53:44] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #874: 09FIXED in 1 min 49 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/874/ [10:53:54] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/275691 (owner: 10Paladox) [10:56:43] (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/270938 (https://phabricator.wikimedia.org/T97889) (owner: 10Paladox) [10:57:21] (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/262432 (https://phabricator.wikimedia.org/T121479) (owner: 10Paladox) [10:58:15] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/275686 (owner: 10Paladox) [10:58:35] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/275682 (owner: 10Paladox) [10:58:59] (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/274432 (owner: 10Paladox) [11:02:38] Yippee, build fixed! [11:02:39] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #884: 09FIXED in 43 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/884/ [11:13:04] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review: Merge tests/browser/environments.yml and tests/browser/config/config.yml in WikidataBrowserTests - https://phabricator.wikimedia.org/T128097#2098480 (10zeljkofilipin) See also https://github.com/wmde/WikidataBrowserTests/pull/107 [11:16:37] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review: Merge tests/browser/environments.yml and tests/browser/config/config.yml in WikidataBrowserTests - https://phabricator.wikimedia.org/T128097#2098484 (10zeljkofilipin) See also https://github.com/wmde/WikidataBrowserTests/compare/T127042 [11:17:08] 10Browser-Tests-Infrastructure, 5MW-1.27-release-notes, 13Patch-For-Review, 5WMF-deploy-2016-03-01_(1.27.0-wmf.15): Remove LoginPage from mediawiki_selenium Ruby gem - https://phabricator.wikimedia.org/T127042#2098486 (10zeljkofilipin) See https://github.com/wmde/WikidataBrowserTests/compare/T127042 [11:19:58] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review: Merge tests/browser/environments.yml and tests/browser/config/config.yml in WikidataBrowserTests - https://phabricator.wikimedia.org/T128097#2098487 (10zeljkofilipin) See also https://github.com/wmde/WikidataBrowserTests/compare/T128097 [11:20:55] (03PS1) 10Hashar: dib: set $labsproject [integration/config] - 10https://gerrit.wikimedia.org/r/275786 (https://phabricator.wikimedia.org/T129092) [11:21:44] (03CR) 10Hashar: [C: 032] dib: set $labsproject [integration/config] - 10https://gerrit.wikimedia.org/r/275786 (https://phabricator.wikimedia.org/T129092) (owner: 10Hashar) [11:22:44] (03Merged) 10jenkins-bot: dib: set $labsproject [integration/config] - 10https://gerrit.wikimedia.org/r/275786 (https://phabricator.wikimedia.org/T129092) (owner: 10Hashar) [11:26:42] Nodepool: created new snapshot to set puppet $::labsproject : ci-jessie-wikimedia-1457436175 hoping to fix hiera lookup T129092 [11:26:48] !log Nodepool: created new snapshot to set puppet $::labsproject : ci-jessie-wikimedia-1457436175 hoping to fix hiera lookup T129092 [11:26:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:29:08] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Wikidata, 13Patch-For-Review: Wikidata sauce browser tests time out - https://phabricator.wikimedia.org/T128065#2098521 (10zeljkofilipin) Submitted pull request https://github.com/wmde/WikidataBrowserTests/pull/108 [11:35:27] !sal [11:35:27] https://tools.wmflabs.org/sal/releng [11:36:18] Project browsertests-Wikidata-WikidataTests-linux-chrome-sauce-T128065 build #1: 15ABORTED in 5 min 13 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-chrome-sauce-T128065/1/ [11:36:48] (03PS1) 10Aude: Update Wikidata branch [tools/release] - 10https://gerrit.wikimedia.org/r/275789 [11:38:05] (03CR) 10Aude: [C: 032] Update Wikidata branch [tools/release] - 10https://gerrit.wikimedia.org/r/275789 (owner: 10Aude) [11:50:35] 5Continuous-Integration-Scaling, 7Nodepool, 13Patch-For-Review, 7Upstream: Nodepool delay instance deletions by one minute - https://phabricator.wikimedia.org/T113359#2098573 (10hashar) I have also found out we were rate limiting delete requests to at most one every ten seconds or six per minutes. https://... [11:51:24] (03Merged) 10jenkins-bot: Update Wikidata branch [tools/release] - 10https://gerrit.wikimedia.org/r/275789 (owner: 10Aude) [11:54:59] 5Continuous-Integration-Scaling, 7Nodepool, 13Patch-For-Review, 7Upstream: Nodepool delay instance deletions by one minute - https://phabricator.wikimedia.org/T113359#2098576 (10hashar) 5Resolved>3Open [11:56:16] 5Continuous-Integration-Scaling, 13Patch-For-Review, 7Puppet: Hiera is not properly configured on Nodepool instances - https://phabricator.wikimedia.org/T129092#2094783 (10hashar) p:5Triage>3High [12:22:54] (03PS1) 10Lethexie: fixed the bug of SpaceyParenthesisSniff cann't fix $a->foo( $b ) and $a->foo( $b ) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/275796 [13:07:42] icinga-wm: ping [13:07:48] shinken-wm: ping [13:35:03] 10Browser-Tests-Infrastructure, 5MW-1.27-release-notes, 13Patch-For-Review, 5WMF-deploy-2016-03-01_(1.27.0-wmf.15): Remove LoginPage from mediawiki_selenium Ruby gem - https://phabricator.wikimedia.org/T127042#2098688 (10zeljkofilipin) [13:38:23] !log Rebased integration puppet master. Dropped a make-wmf-branch patch and the one for raita role [13:38:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:53:11] 5Continuous-Integration-Scaling, 13Patch-For-Review, 7Puppet: Hiera is not properly configured on Nodepool instances - https://phabricator.wikimedia.org/T129092#2098713 (10hashar) a:3hashar [13:54:57] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 10VisualEditor, 7WorkType-NewFunctionality: Migrate VisualEditor core npm CI job to Nodepool - https://phabricator.wikimedia.org/T128093#2098719 (10hashar) [13:54:59] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 10OOjs-UI, 13Patch-For-Review, 7WorkType-NewFunctionality: Migrate OOjs UI npm CI job to Nodepool - https://phabricator.wikimedia.org/T128091#2098720 (10hashar) [13:55:01] 10Continuous-Integration-Infrastructure: Run 'npm' job with Node 4 (instead of Node 0.10) - https://phabricator.wikimedia.org/T126774#2098722 (10hashar) [13:55:03] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 10OOjs, 13Patch-For-Review, 7WorkType-NewFunctionality: Migrate OOjs npm CI job to Nodepool - https://phabricator.wikimedia.org/T128089#2098721 (10hashar) [13:55:06] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Jessie should have the ability to run several tests at once like UbuntuTrusty can - https://phabricator.wikimedia.org/T128135#2098715 (10hashar) 5Open>3Invalid The jobs can run concurrently. The only reason I can think of them being... [13:55:19] Project browsertests-Wikidata-WikidataTests-linux-chrome-sauce-T128065 build #2: 04FAILURE in 2 hr 17 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-chrome-sauce-T128065/2/ [13:56:12] 10Continuous-Integration-Config, 10Mathoid, 15User-mobrovac: Enable node4 for mathoid tests - https://phabricator.wikimedia.org/T124447#2098724 (10hashar) a:5hashar>3None [14:01:00] (03PS2) 10Hashar: Update test configuration for CharInsert [integration/config] - 10https://gerrit.wikimedia.org/r/275542 (owner: 10Siebrand) [14:01:21] (03CR) 10Hashar: [C: 032] Update test configuration for CharInsert [integration/config] - 10https://gerrit.wikimedia.org/r/275542 (owner: 10Siebrand) [14:02:27] (03Abandoned) 10Hashar: Allow us to recheck gate and submit [integration/config] - 10https://gerrit.wikimedia.org/r/275062 (owner: 10Paladox) [14:04:08] (03Abandoned) 10Hashar: WIP More tests for SmashPig [integration/config] - 10https://gerrit.wikimedia.org/r/228580 (owner: 10Awight) [14:04:43] (03Merged) 10jenkins-bot: Update test configuration for CharInsert [integration/config] - 10https://gerrit.wikimedia.org/r/275542 (owner: 10Siebrand) [14:05:11] (03Abandoned) 10Hashar: [Vector] Add extension-qunit-generic test [integration/config] - 10https://gerrit.wikimedia.org/r/266604 (owner: 10Paladox) [14:05:15] (03Abandoned) 10Hashar: [Metrolook] Add extension-qunit-generic test [integration/config] - 10https://gerrit.wikimedia.org/r/266605 (owner: 10Paladox) [14:09:05] (03PS3) 10Hashar: [Arrays] Add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/274801 (https://phabricator.wikimedia.org/T119973) (owner: 10Paladox) [14:09:34] (03CR) 10Hashar: [C: 032] [Arrays] Add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/274801 (https://phabricator.wikimedia.org/T119973) (owner: 10Paladox) [14:10:42] (03Merged) 10jenkins-bot: [Arrays] Add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/274801 (https://phabricator.wikimedia.org/T119973) (owner: 10Paladox) [14:12:38] (03Abandoned) 10Hashar: Job templates for `make docs` [integration/config] - 10https://gerrit.wikimedia.org/r/191045 (https://phabricator.wikimedia.org/T88999) (owner: 10Hashar) [14:13:55] (03Abandoned) 10Hashar: nodepool: contint::packages::ops in images [integration/config] - 10https://gerrit.wikimedia.org/r/240660 (owner: 10Hashar) [14:17:07] (03PS2) 10Hashar: bin/php: replace bash with lighter sh [integration/jenkins] - 10https://gerrit.wikimedia.org/r/269384 [14:23:47] (03PS3) 10Hashar: bin/php: replace bash with lighter sh [integration/jenkins] - 10https://gerrit.wikimedia.org/r/269384 [14:24:05] (03CR) 10Hashar: [C: 032] bin/php: replace bash with lighter sh [integration/jenkins] - 10https://gerrit.wikimedia.org/r/269384 (owner: 10Hashar) [14:24:53] 6Release-Engineering-Team, 10DBA, 10MediaWiki-Configuration, 6Operations, and 3 others: codfw is in read only according to mediawiki - https://phabricator.wikimedia.org/T124795#2098800 (10jcrespo) [14:24:58] (03Merged) 10jenkins-bot: bin/php: replace bash with lighter sh [integration/jenkins] - 10https://gerrit.wikimedia.org/r/269384 (owner: 10Hashar) [14:25:43] (03PS2) 10Hashar: bin/php: use exec to replace shell with php command [integration/jenkins] - 10https://gerrit.wikimedia.org/r/269385 [14:25:53] (03CR) 10Hashar: [C: 032] bin/php: use exec to replace shell with php command [integration/jenkins] - 10https://gerrit.wikimedia.org/r/269385 (owner: 10Hashar) [14:27:02] (03Merged) 10jenkins-bot: bin/php: use exec to replace shell with php command [integration/jenkins] - 10https://gerrit.wikimedia.org/r/269385 (owner: 10Hashar) [14:27:42] !log Mass refreshed CI slave-scripts 1d2c60d..e27c292 [14:27:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:46:48] Project browsertests-Wikidata-WikidataTests-linux-firefox build #129: 04STILL FAILING in 1 hr 15 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-firefox/129/ [14:52:10] (03PS1) 10Zfilipin: WIP Simplify creating of Jenkins jobs for running browser tests daily [selenium] - 10https://gerrit.wikimedia.org/r/275820 (https://phabricator.wikimedia.org/T128190) [14:53:00] 10Deployment-Systems, 10scap, 10Analytics, 10Analytics-Cluster, 10Scap3 (scap3-adoption): Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2098872 (10Ottomata) Not sure if this blocks T109926; analytics/refinery is not a Services team owned thang. But, we should still do... [14:57:41] 10MediaWiki-Codesniffer: Codesniffer allows missing space before opening brace - https://phabricator.wikimedia.org/T126976#2028658 (10Lethexie) Hi there,I have submitted my changes to Gerrit about this, I hope it may useful :) [15:01:35] 10MediaWiki-Codesniffer: SpaceyParenthesis bug - https://phabricator.wikimedia.org/T116519#1751426 (10Lethexie) Hi, actually I meet this situation also so that I have submitted my changes about it just now. [15:05:40] 10Continuous-Integration-Config: Doint use zuul to clone mediawiki/core and mediawiki/vendor under non generic tests including composer - https://phabricator.wikimedia.org/T126519#2098945 (10hashar) [15:06:17] 10Continuous-Integration-Config: Doint use zuul to clone mediawiki/core and mediawiki/vendor under non generic tests including composer - https://phabricator.wikimedia.org/T126519#2016378 (10hashar) 5Open>3declined That will be solved when the PHP jobs are moved to disposable instances and a git clone cache [15:08:02] (03CR) 10Hashar: "check experimental" [integration/raita] - 10https://gerrit.wikimedia.org/r/216348 (owner: 10Dduvall) [15:29:34] (03PS1) 10Hashar: Migrate some npm jobs to Nodepool/NodeJs 4.3 [integration/config] - 10https://gerrit.wikimedia.org/r/275828 (https://phabricator.wikimedia.org/T119143) [15:31:18] (03CR) 10Hashar: [C: 032] Migrate some npm jobs to Nodepool/NodeJs 4.3 [integration/config] - 10https://gerrit.wikimedia.org/r/275828 (https://phabricator.wikimedia.org/T119143) (owner: 10Hashar) [15:32:38] (03Merged) 10jenkins-bot: Migrate some npm jobs to Nodepool/NodeJs 4.3 [integration/config] - 10https://gerrit.wikimedia.org/r/275828 (https://phabricator.wikimedia.org/T119143) (owner: 10Hashar) [15:36:37] (03CR) 10Aklapper: [C: 04-1] "Thanks for the patch! Please fix the commit message (by using an infinite verb, removing typos, and potentially linking to a corresponding" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/275796 (owner: 10Lethexie) [15:37:54] 10MediaWiki-Codesniffer, 10Possible-Tech-Projects: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#2099052 (10Aklapper) @Lethexie: Please follow https://www.mediawiki.org/wiki/Gerrit/Commit_message_guidelines and add a "Bug: T89682" to your commit message, so there will... [15:45:10] 5Gerrit-Migration, 10Analytics-Tech-community-metrics, 6Developer-Relations, 3DevRel-March-2016: Make MetricsGrimoire/korma support gathering Code Review statistics from Phabricator's Differential - https://phabricator.wikimedia.org/T118753#2099080 (10Aklapper) @Lcanasdiaz: Not yet. :( I'm waiting for an u... [15:45:16] Project browsertests-Wikidata-WikidataTests-linux-chrome-sauce build #327: 04STILL FAILING in 2 hr 13 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-chrome-sauce/327/ [15:47:29] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 10VisualEditor, 7WorkType-NewFunctionality: Migrate VisualEditor core npm CI job to Nodepool - https://phabricator.wikimedia.org/T128093#2099109 (10hashar) ``` 00:03:42.305 Running "karma:main" (karma) task 00:03:47.188 08 03 2016 15... [15:48:03] thcipriani: i have another idea related to event-schemas deployment whenever you are around and have a sec [15:48:35] 10MediaWiki-Codesniffer: Fix the SpaceyParenthesisSniff cann't fix wffoo( $a, $b ) and wffoo( $a ); - https://phabricator.wikimedia.org/T129231#2099119 (10Lethexie) [15:48:36] ottomata: I'm around. Probably have to do swat in a few minutes, but what's up? [15:49:17] 5Gerrit-Migration, 10Analytics-Tech-community-metrics, 6Developer-Relations, 3DevRel-March-2016: Make MetricsGrimoire/korma support gathering Code Review statistics from Phabricator's Differential - https://phabricator.wikimedia.org/T118753#2099132 (10Qgil) @Lcanasdiaz could you estimate the cost of this p... [15:49:17] would it be possible to have multiple scap/ dirs in an event-schemas repo? [15:49:22] event-schema/deploy repo [15:49:23] like [15:49:48] event-schemas/deploy/eventbus/{scap,event-schema[submodule]} [15:49:56] and whatever else is necessary [15:50:11] event-schemas/deploy/other thing1/{scap/event-schemas submoulde} [15:50:11] etc. [15:50:21] sorry /{scap, event-schemas submodule} [15:50:54] then to deploy an new event schemas for your service, you update the submodule in the deploy repo for the scap target you want [15:51:04] and cd into that dir and deploy from there [15:51:09] hmmm [15:51:16] hmMMm [15:51:28] i guess not, because the .git dir is still global for all of those scap/ subdirs? [15:51:33] and there's no way to sync just the subdir? [15:52:05] trying to think if there's a way to do it with the configuration options... [15:53:04] I wonder if there's a way to do this by setting git_repo to a deeper subdirectory and leaving git_deploy_dir as /srv/deployment [15:53:22] yeah that's what I was looking at too. what is git_repo used for? just finding the path to rsync from on tin? [15:53:49] yeah, path to pull down via git from tin. [15:54:02] doc root on tin is at /srv/deployment so it's the path from there [15:54:03] oh right it git pulls [15:54:08] duh [15:54:20] Hm uh, somehow i doubt it could pull a subdir [15:55:26] it does seem tricky. definitely wasn't something we designed for. [15:56:27] (03PS1) 10Hashar: dib: browsers + Xvfb [integration/config] - 10https://gerrit.wikimedia.org/r/275832 (https://phabricator.wikimedia.org/T128090) [15:56:32] so on tin you want: /srv/deployment/event-schemas/deploy/one/{scap,event-schema-sub} /srv/deployment/event-schemas/deploy/two/{scap,event-schema-sub} and you want them on the other end to be...? [15:56:35] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 10OOjs, 13Patch-For-Review, 7WorkType-NewFunctionality: Provide the Chrome binary on the nodepool servers so OOjs, VE core can use it in testing - https://phabricator.wikimedia.org/T128090#2099177 (10hashar) a:3hashar [15:56:35] on the targets. [15:56:56] thcipriani: on the other end i don't really care, same path is fine [15:57:06] use case is: single repo, multiple services [15:57:30] (03CR) 10Hashar: [C: 032] dib: browsers + Xvfb [integration/config] - 10https://gerrit.wikimedia.org/r/275832 (https://phabricator.wikimedia.org/T128090) (owner: 10Hashar) [15:58:05] ottomata: that should be workable as long as the deploy user is the same, I think. [15:58:17] oh the same for each one? [15:58:18] hm [15:58:20] that might be harder [15:58:23] could make it work buuut [15:58:24] hm [15:58:26] hmmm [15:58:27] actually, [15:58:28] no [15:58:29] that should be fine [15:58:33] we can use deploy-service, i don't mind [15:58:38] it isn't running anything [15:58:45] (03Merged) 10jenkins-bot: dib: browsers + Xvfb [integration/config] - 10https://gerrit.wikimedia.org/r/275832 (https://phabricator.wikimedia.org/T128090) (owner: 10Hashar) [15:59:13] !log Nodepool: refreshing snapshot image to ship browsers+Xvfb for T128090 [15:59:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:59:46] ottomata: then yeah, should just be like setting up two repos that have totally different paths. [16:00:11] how would the pull work though, won't it try to pull the whole thing on all the hosts? [16:00:14] all the targets? [16:00:15] CI / browser test meeting starting now in https://plus.google.com/hangouts/_/wikimedia.org/btest-ci-weekly?authuser=0 [16:00:50] zeljkof: ^^^ ? :D [16:00:50] ottomata: gotta swat. I'll brb. [16:01:01] k [16:01:06] hashar: joining [16:03:19] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 10OOjs, 13Patch-For-Review, 7WorkType-NewFunctionality: Provide the Chrome binary on the nodepool servers so OOjs, VE core can use it in testing - https://phabricator.wikimedia.org/T128090#2099201 (10hashar) Stuff is being provisio... [16:03:34] !log Image ci-jessie-wikimedia-1457452766 is ready T128090 [16:03:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:05:22] (thcipriani i know you are busy, just brain dumping...) [16:05:33] hmm, maybe this is getting too crazy [16:05:46] maybe submodules of event-schemas inside their service repos is fine [16:06:13] one of the reasons i'm looking into this, is because i want to to a SIGHUP when event-schemas change, but a restart when code changes [16:06:41] maybe I can just make the systemd stuff smarter? [16:06:41] hm. [16:07:04] not sure how though [16:07:20] i guess i could make an environment override and deploy a different env when i know its just an event schcema change [16:07:21] hm [16:10:10] Project browsertests-Wikidata-WikidataTests-linux-firefox-sauce build #550: 04STILL FAILING in 2 hr 38 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-firefox-sauce/550/ [16:12:59] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 10OOjs, 13Patch-For-Review, 7WorkType-NewFunctionality: Provide the Chrome binary on the nodepool servers so OOjs, VE core can use it in testing - https://phabricator.wikimedia.org/T128090#2099241 (10hashar) With chromium package i... [16:14:08] Four times out of five, when gerrit complains that a patch can’t be rebased, I just run “git rebase origin” on the commandline and it works fine. And yet it seems unlikely that gerrit is using a different merge algorithm than my local git... [16:14:25] (this just happened to me in OpenStack gerrit, so it’s clearly nothing specific to our install) [16:16:04] andrewbogott: it is [16:16:09] jgit is a different implementation [16:16:14] So you get shit like that [16:16:25] oh, sure, that makes sense [16:16:45] Although it’s weird that ‘how to merge’ isn’t a well-defined property of git [16:16:52] heh [16:16:58] I think that will start to upset me if I think about it too much [16:17:05] One of those things you just get used to having to work around [16:21:16] andrewbogott: and there is also zuul-merger process using git CLI to merge the patchset against tip of branch [16:21:23] might use a slightly different merge process [16:22:53] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 10OOjs, 13Patch-For-Review, 7WorkType-NewFunctionality: Provide the Chrome binary on the nodepool servers so OOjs, VE core can use it in testing - https://phabricator.wikimedia.org/T128090#2099252 (10Paladox) But doint we need disp... [16:22:55] man, the fast track to a +2 in openstack is fixing a README file that no one has looked at in two years. [16:23:07] * James_F laughs. [16:23:10] git rm [16:23:11] FIXED [16:23:46] https://review.openstack.org/#/c/289701/ \O/ [16:25:33] That was one of those times when you read a doc file that’s so wrong you have to take a 20 minute break to calm down. [16:26:39] Has .16 been cut yet? [16:28:26] James_F: twentyafterfour is on train this week. AbuseFilter doesn't have a new branch so not yet (that's the first thing to get branched). [16:28:36] thcipriani: Kk. Thanks! [16:35:10] hashar: Thoughts; Should APC be installed on Jenkins slaves? [16:35:28] We have two important libs that use it (MemoizerCallable and ApcBagOStuff) [16:35:34] neither can be tested properly at the moment on php55 [16:35:44] See https://gerrit.wikimedia.org/r/#/c/274322/ for example [16:41:18] Actually, better to mock it [16:41:19] nvm [16:51:40] 10MediaWiki-Codesniffer: Fix the SpaceyParenthesisSniff cann't fix wffoo( $a, $b ) and wffoo( $a ); - https://phabricator.wikimedia.org/T129231#2099356 (10Aklapper) Hi! I have problems to parse/understand this task's summary... What is the problem this task is actually about? Or is the task summary simply mis... [17:21:48] (03CR) 10Krinkle: "Is there a task for those failures?" [integration/config] - 10https://gerrit.wikimedia.org/r/272662 (owner: 10Krinkle) [17:21:52] (03Abandoned) 10Krinkle: Don't ignore exit code in mediawik-core-code-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/272662 (owner: 10Krinkle) [17:24:28] 10Continuous-Integration-Infrastructure, 7Jenkins: Jenkins: Consistently getting 503 Varnish in response to succesful login - https://phabricator.wikimedia.org/T63710#2099509 (10Krinkle) Still happening. Makes logging into Jenkins unnecessarily difficult. [17:25:11] 10Continuous-Integration-Infrastructure: Jenkins: Assert no PHP errors (notices, warnings) were raised or exceptions were thrown - https://phabricator.wikimedia.org/T50002#2099514 (10Krinkle) [17:25:14] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 3Wikidata-Sprint-2016-03-01: Cannot find site jenkins_u3_mw - https://phabricator.wikimedia.org/T121083#2099512 (10Krinkle) 5Open>3Resolved a:3Krinkle [17:41:37] 7Browser-Tests, 10Continuous-Integration-Config, 10Wikidata, 3Wikidata-Sprint-2016-03-15: [Task] Move Wikidata browsertests into Wikibase repository - https://phabricator.wikimedia.org/T118727#2099579 (10Lydia_Pintscher) [17:41:49] 7Browser-Tests, 10Continuous-Integration-Config, 10Wikidata, 7Story, 3Wikidata-Sprint-2016-03-15: [Story] Run browsertests regularly on test.wikidata.org via Jenkins - https://phabricator.wikimedia.org/T101497#2099580 (10Lydia_Pintscher) [17:56:43] thcipriani: James_F sorry alarm fail. I'm just starting the branching now [17:59:56] twentyafterfour: No worries. :-) [18:57:45] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer, 5WMF-deploy-2016-03-01_(1.27.0-wmf.15), 5WMF-deploy-2016-03-08_(1.27.0-wmf.16): browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce fails with "757: unexpected token" - https://phabricator.wikimedia.org/T128079#2100003 (10J... [18:57:54] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer, 5WMF-deploy-2016-03-01_(1.27.0-wmf.15), 5WMF-deploy-2016-03-08_(1.27.0-wmf.16): browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce fails with "757: unexpected token" - https://phabricator.wikimedia.org/T128079#2063388 (10J... [19:04:41] PROBLEM - Puppet failure on deployment-tmh01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:07:13] PROBLEM - Puppet failure on deployment-eventlogging04 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:09:17] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:09:30] (03CR) 10Krinkle: [C: 04-1] "See inline comment. may wanna update ve-submodules-update as well so that they are the same." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/262432 (https://phabricator.wikimedia.org/T121479) (owner: 10Paladox) [19:15:52] (03PS7) 10Hashar: Sets DISPLAY and CHROME_BIN variables in parameter_functions.py [integration/config] - 10https://gerrit.wikimedia.org/r/273269 (https://phabricator.wikimedia.org/T128090) (owner: 10Paladox) [19:16:20] PROBLEM - Puppet failure on mira is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:18:56] (03PS8) 10Hashar: Sets DISPLAY in parameter_functions.py [integration/config] - 10https://gerrit.wikimedia.org/r/273269 (https://phabricator.wikimedia.org/T128090) (owner: 10Paladox) [19:19:47] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 10OOjs, 13Patch-For-Review, 7WorkType-NewFunctionality: Provide the Chrome binary on the nodepool servers so OOjs, VE core can use it in testing - https://phabricator.wikimedia.org/T128090#2100100 (10hashar) @Paladox good catch! I... [19:21:34] (03CR) 10Hashar: [C: 032] Sets DISPLAY in parameter_functions.py [integration/config] - 10https://gerrit.wikimedia.org/r/273269 (https://phabricator.wikimedia.org/T128090) (owner: 10Paladox) [19:22:54] (03Merged) 10jenkins-bot: Sets DISPLAY in parameter_functions.py [integration/config] - 10https://gerrit.wikimedia.org/r/273269 (https://phabricator.wikimedia.org/T128090) (owner: 10Paladox) [19:23:00] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 10OOjs, 13Patch-For-Review, 7WorkType-NewFunctionality: Provide the Chrome binary on the nodepool servers so OOjs, VE core can use it in testing - https://phabricator.wikimedia.org/T128090#2100124 (10Paladox) @Hashar thanks. [19:23:12] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/273269 (https://phabricator.wikimedia.org/T128090) (owner: 10Paladox) [19:23:44] !log Zuul inject DISPLAY https://gerrit.wikimedia.org/r/#/c/273269/ [19:23:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [19:24:33] greg-g How can I know if a mw commit was deployed ? [19:27:34] (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/274441 (owner: 10Paladox) [19:27:37] hashar: Where is the Xvfb deamon set up for nodepool images? [19:29:49] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 10OOjs, 13Patch-For-Review, 7WorkType-NewFunctionality: Provide the Chrome binary on the nodepool servers so OOjs, VE core can use it in testing - https://phabricator.wikimedia.org/T128090#2100192 (10hashar) My VisualEditor sandbox... [19:30:04] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 10OOjs, 13Patch-For-Review, 7WorkType-NewFunctionality: Provide the Chrome binary on the nodepool servers so OOjs, VE core can use it in testing - https://phabricator.wikimedia.org/T128090#2100194 (10Paladox) @Hashar still doesn't... [19:31:08] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 10OOjs, 13Patch-For-Review, 7WorkType-NewFunctionality: Provide the Chrome binary on the nodepool servers so OOjs, VE core can use it in testing - https://phabricator.wikimedia.org/T128090#2100203 (10Paladox) Is there anyway to deb... [19:38:22] nuria: Gerrit displays an "Included in" dropdown thingy for a merged commit. That will show "master" and often also something like "wmf/1.27.0-wmf.15" [19:38:40] RoanKattouw: oohhh, thnak youu [19:38:40] 1) If it doesn't list any wmf/* branches, it's not deployed yet [19:38:43] *thank you [19:38:51] 2) If it does, refer to https://wikitech.wikimedia.org/wiki/Deployments to see when that wmfN version will be deployed to which wiki [19:39:09] * hashar approves whatever RoanKattouw can say [19:42:42] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:45:26] Krinkle: you poked me a few hours ago. Sorry I was in a meeting :(- [19:47:29] hashar: I asked where Xvfb is done in nodepool image. Regarding DISPLAY, it seems like env DISPLAY=:94 isn't set, regardless of that patch [19:47:35] https://integration.wikimedia.org/ci/job/npm-node-4.3/224/consoleFull doesn't show it [19:50:58] EEkk [19:51:33] Krinkle: at least the parameters list for that build shows it: https://integration.wikimedia.org/ci/job/npm-node-4.3/224/parameters/ [19:51:52] I have added contint::browser to the nodepool images [19:52:09] so they ""should""™© have chromium [19:52:31] k [19:53:20] May wanna verify via ssh on the base image [19:53:46] and also to make sure the old CHROME_BIN override we have doesn't actively break it by setting it to the wrong value (if the bin has a differnet name on Jessie) [19:54:06] Right now we still have it because some older branches use older karma that doesn't have my patch yet [19:54:18] which is been blocked because Wikibase keeps breaking REL1_26 in new and mysterious ways [19:54:52] Krinkle: the npm-node-4.3 job does not load the slave scripts at all [19:54:58] but yeah you are right, we try to reproduce locally [19:55:12] if only karma could drop a more useful message and some kind of logs .. [19:55:31] hashar: They considered piping chrome process output, but it's incredibly verbose [19:55:43] and always has a million errors because.. Chromium [19:55:48] but none of them are "real". [19:55:53] can't you add to karma an option for that ?? [19:56:02] The syslog on my macbook is a stinking mess full of Chrome shit [19:56:02] --timo-need-that-log [19:56:06] grr [19:56:06] needs [19:56:17] as you can see, my english is still crappy [19:56:36] will try to diagnose tomorrow on an instance [19:56:44] I patched my local Karma install at some point to get it, but didn't help much. [19:58:49] 10Beta-Cluster-Infrastructure, 6Commons, 10MediaWiki-File-management, 6Multimedia: Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking) - https://phabricator.wikimedia.org/T84950#2100346 (10Tgr) [20:11:21] (03CR) 1020after4: "why was this added to make-wmf-branch? is it still needed?" [tools/release] - 10https://gerrit.wikimedia.org/r/275701 (https://phabricator.wikimedia.org/T124356) (owner: 10Legoktm) [20:12:13] RECOVERY - Puppet failure on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [20:12:44] TimStarling: legoktm ostriches: is https://gerrit.wikimedia.org/r/#/c/275701/ still needed? [20:12:57] (03CR) 10Legoktm: "It's a live hack that isn't being merged into master because it's a wmf logging hack, but I don't want to have to keep cherry-picking it f" [tools/release] - 10https://gerrit.wikimedia.org/r/275701 (https://phabricator.wikimedia.org/T124356) (owner: 10Legoktm) [20:13:02] twentyafterfour: yes. [20:13:30] https://phabricator.wikimedia.org/T124356#2098016 makes it seem like maybe not.. but ok I'll keep the cherry pick on wmf.16 [20:15:06] I don't like make-wmf-branch auto-cherry-picking patches. Why not merge into master if it needs to be in the next branch? [20:15:24] Uh, did you read my comment? ;) [20:15:52] I didn't see it ;) reading [20:16:14] this would be best solved by long-lived production branches :) [20:16:39] * twentyafterfour doesn't like branching from master every week but it's still The Way Things Are Done...for noe [20:16:41] now [20:22:32] PROBLEM - Host make-wmf-branch is DOWN: CRITICAL - Host Unreachable (10.68.21.34) [20:23:38] PROBLEM - Host integration-make-wmf-branch is DOWN: CRITICAL - Host Unreachable (10.68.19.152) [20:26:52] shinken-wm: I deleted those hosts :P [20:27:45] hashar: Hi would you know why chromium is failing on nodepool. Is there a way of debugging like you did on mediawiki/core. [20:38:26] paladox: yes we can find out the issue. How? I have no idea. [20:38:54] hashar: Ok, im going to try debugging here https://integration.wikimedia.org/ci/job/npm-node-4.3/225/console [20:39:01] maybe craft a patch that solely runs karma and has a bunch of debugging enabled [20:39:02] https://integration.wikimedia.org/ci/job/npm-node-4.3/225/console [20:39:17] https://gerrit.wikimedia.org/r/#/c/275903/ [20:39:22] yup [20:39:56] hashar: [20:39:57] 20:39:24 08 03 2016 20:39:23.266:DEBUG [temp-dir]: Creating temp dir at /tmp/karma-45068492 [20:39:57] 20:39:24 08 03 2016 20:39:23.266:DEBUG [launcher]: chromium --user-data-dir=/tmp/karma-45068492 --no-default-browser-check --no-first-run --disable-default-apps [20:41:04] hashar: Shoulden chromium be /usr/bin/chromium-browser [20:41:23] not on jessie afaik [20:41:28] So this bit 20:39:24 08 03 2016 20:39:23.266:DEBUG [launcher]: chromium should be 20:39:24 08 03 2016 20:39:23.266:DEBUG [launcher]: /usr/bin/chromium-browser [20:41:32] oh [20:41:36] let me confirm [20:41:42] hashar: Ok. [20:42:37] jenkins@ci-jessie-wikimedia-46734:~$ which chromium [20:42:38] jenkins@ci-jessie-wikimedia-46734:~$ which chromium-browser [20:42:38] jenkins@ci-jessie-wikimedia-46734:~$ [20:43:03] grr [20:43:23] /usr/bin/chromium [20:43:30] hashar: Oh so does that mean it carnt find chromium. [20:43:41] wrong paste in my IRC client [20:43:44] :-) [20:43:48] hashar: Oh. [20:44:53] hashar: Does that mean we found the problem. Because it found the chromium build but coulden find it to star. [20:45:30] na nodepool instances definitely have chromium as /usr/bin/chromium [20:45:35] but it fails to start for whatever reason [20:45:44] hashar: Oh. [20:46:09] will try to reproduce [20:46:31] hashar: Ok thanks. [20:47:24] hashar: It looks like it runs additional things on trusty, see 01:38:33 DEBUG [temp-dir]: Creating temp dir at /mnt/home/jenkins-deploy/tmpfs/jenkins-2/karma-65986548 [20:47:24] 01:38:33 DEBUG [launcher]: /usr/bin/chromium-browser --user-data-dir=/mnt/home/jenkins-deploy/tmpfs/jenkins-2/karma-65986548 --no-default-browser-check --no-first-run --disable-default-apps --disable-popup-blocking --disable-translate http://localhost:9876/?id=65986548 [20:47:31] https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/32771/console [20:47:33] please. [20:47:41] there is no xvfb running [20:48:03] hashar: Oh so that was why it wasent working. [20:50:10] hashar: it still fails but it is now running these [20:50:11] 20:49:27 08 03 2016 20:49:26.676:DEBUG [launcher]: chromium --user-data-dir=/tmp/karma-63781814 --no-default-browser-check --no-first-run --disable-default-apps --disable-popup-blocking --disable-translate --disable-background-timer-throttling http://localhost:9876/?id=63781814 [20:51:38] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 10OOjs, 13Patch-For-Review, 7WorkType-NewFunctionality: Provide the Chrome binary on the nodepool servers so OOjs, VE core can use it in testing - https://phabricator.wikimedia.org/T128090#2100584 (10hashar) The instances definitel... [20:51:59] Notice: /Stage[main]/Xvfb/Base::Service_unit[xvfb]/Service[xvfb]/ensure: ensure changed 'stopped' to 'running' [20:52:04] it should be there grrrrrr [20:57:19] F***************** SYSTEMD [21:01:47] hashar: Oh, so that means [21:01:58] it stopes running [21:01:59] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 10OOjs, 13Patch-For-Review, 7WorkType-NewFunctionality: Provide the Chrome binary on the nodepool servers so OOjs, VE core can use it in testing - https://phabricator.wikimedia.org/T128090#2100629 (10hashar) The puppet run mention:... [21:03:59] paladox: more like it never starts [21:04:11] hashar: Oh. [21:07:31] hashar: How is Xvf installed on trusty. [21:07:39] using upstart [21:07:44] jessie uses systemd [21:07:56] hashar: oh. thanks for the reply. [21:08:11] hashar: So Xvf is not found on jessie. [21:08:52] hashar: Is there anyway on installing that for jessie through puppet as a workaround. [21:09:35] systemctl list-units [21:09:35] Running in chroot, ignoring request. [21:09:37] seriously [21:09:56] hashar: Oh so it is installing it but ignoring it. [21:14:40] paladox: I have no clue / nothing obvious [21:14:47] hashar: Oh [21:15:59] 10Beta-Cluster-Infrastructure, 10scap, 10Analytics, 6Services, and 3 others: Set up AQS in Beta - https://phabricator.wikimedia.org/T116206#2100686 (10Nuria) [21:16:58] hashar: Maybe http://stackoverflow.com/questions/16726227/xvfb-failed-start-error [21:17:52] hashar: Also maybe display 99 is not working either, even though we set it. But i think we have to do $DISPLAY since with PHP_BIN we do $PHP_BIN. [21:20:38] 2016-03-08 21:20:31,513 INFO nodepool.image.build.wmflabs-eqiad.ci-jessie-wikimedia: Info: /Stage[main]/Xvfb/Base::Service_unit[xvfb]/File[/lib/systemd/system/xvfb.service]: Scheduling refresh of Exec[systemd reload for xvfb] [21:20:42] paladox: it should start [21:21:03] hashar: oh. [21:21:34] systemd is borked out of repair really [21:21:45] what a shitty script kiddies soft it is [21:21:49] hashar: Should we try doing it through the script to see if that works. [21:21:53] hashar: Yes. [21:22:11] mind you systemd status uses a pager [21:22:15] hashar: So systemd is broke. [21:22:17] even when you run it from a non TTY .... [21:22:20] hashar: Oh. [21:22:22] end result [21:22:30] it is blocked waiting for the non TTY to press SPACE [21:22:31] .. [21:22:33] or whatever rkey [21:22:38] hashar: Oh. [21:23:02] pid 1 is /sbin/init nofb nofb [21:25:04] (03PS1) 10Paladox: Add global-set-env-npm script [integration/jenkins] - 10https://gerrit.wikimedia.org/r/275932 [21:25:06] so my assumption is somehow the image/instances are not booting with systemd [21:25:23] hashar: https://gerrit.wikimedia.org/r/#/c/275932/ just to test. [21:25:30] hashar: Yes. [21:26:15] --no-pager [21:26:16] really [21:26:52] hashar: Oh so --no-pager could be stoping it. [21:27:08] na it just stop my debug command :D [21:27:17] digging more [21:27:27] hashar: Oh thanks. [21:28:12] hashar: Im searching through google. Seems to bring up the type of error we get but not in the context, they are old github tickets. [21:28:46] ah [21:28:48] it is running [21:28:54] at least when creating the snapshot [21:30:18] hashar: Ah. [21:33:57] !log Nodepool Image ci-jessie-wikimedia-1457472606 in wmflabs-eqiad is ready [21:34:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:42:27] hashar: Would the above try to fix the problem. [21:42:40] The rebuild of the image. [21:44:06] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 10OOjs, 13Patch-For-Review, 7WorkType-NewFunctionality: Provide the Chrome binary on the nodepool servers so OOjs, VE core can use it in testing - https://phabricator.wikimedia.org/T128090#2100791 (10hashar) So somehow the xvfb ser... [21:44:06] Unit display-manager.service failed to load [21:44:15] that is what I got to [21:44:33] hashar: Ah so we now need to figure out how to fix that. [21:44:59] I dont even know whether it is related [21:45:36] hashar: Oh ok. [21:45:57] Lets hope that was the problem so that if we fix it we know what was causing it. [21:57:32] giving up for tonight [21:58:04] hashar: Ok. [21:59:16] PROBLEM - Host cache-rsync is DOWN: CRITICAL - Host Unreachable (10.68.23.165) [22:04:44] hashar: ^^ [22:21:32] hashar: Maybe something like sh -e /etc/init.d/xvfb start [22:21:56] paladox: I will figure it out tomorrow :) [22:22:15] hashar: Ok. It seems cache-rsyn is down. [22:22:44] monitoring is broken, i have deleted that instance [22:22:56] feel free to poke #wikimedia-labs / fill a bug [22:23:29] hashar: Ok. [22:26:24] 6Release-Engineering-Team, 6Developer-Relations, 6Team-Practices, 15User-greg: Set up Code Review office hours - https://phabricator.wikimedia.org/T128371#2101035 (10greg) BTW, I completely forgot to bring this up during my team's meeting yesterday (we were talking about Q4 goals instead), so I'm going to... [22:30:33] 6Release-Engineering-Team: investigate carbon aggregation of stats >1 month old behavior - https://phabricator.wikimedia.org/T129285#2101044 (10greg) [22:55:58] (03PS1) 10Paladox: [DumpHTML] Add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/276029 [22:58:34] (03PS1) 10Paladox: [E3Experiments] Archive extension [integration/config] - 10https://gerrit.wikimedia.org/r/276031 [23:04:57] (03PS1) 10Paladox: [EditAccount] Add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/276033 [23:08:43] 10releng-201516-q2, 3releng-201516-q3, 10scap, 10Scap3 (scap3-adoption): [keyresult] Migrate all Service team owned services to scap - https://phabricator.wikimedia.org/T109926#2101234 (10greg) [23:08:53] 3releng-201516-q3, 10scap, 10Scap3 (scap3-adoption), 7WorkType-NewFunctionality: [keyresult] Migrate the MW weekly train deploy to scap3 - https://phabricator.wikimedia.org/T114313#2101251 (10greg) [23:08:55] 10releng-201516-q2, 3releng-201516-q3, 10scap, 10Scap3 (scap3-adoption): [keyresult] Migrate all Service team owned services to scap - https://phabricator.wikimedia.org/T109926#1562968 (10greg) [23:09:15] 3releng-201516-q4, 10Scap3 (scap3-adoption): Migrate remaining trebuchet deployed services - https://phabricator.wikimedia.org/T129290#2101252 (10greg) [23:09:28] 3releng-201516-q4, 10Scap3 (scap3-adoption): [keyresult] Migrate remaining trebuchet deployed services - https://phabricator.wikimedia.org/T129290#2101265 (10greg) p:5Triage>3Normal [23:11:01] (03PS1) 10Paladox: [Editcount] Add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/276037 [23:13:59] 10Deployment-Systems, 10scap, 10Wikimedia-IEG-grant-review, 10Scap3 (scap3-adoption): Deploy iegreview with scap3 - https://phabricator.wikimedia.org/T129154#2101293 (10greg) [23:14:01] 10Deployment-Systems, 10scap, 10Wikimedia-Stream, 10Scap3 (scap3-adoption): Deploy rcstream with scap3 - https://phabricator.wikimedia.org/T129153#2101294 (10greg) [23:14:02] sorry [23:14:03] 10Deployment-Systems, 10scap, 10Monitoring, 6Operations, 10Scap3 (scap3-adoption): Deploy servermon with scap3 - https://phabricator.wikimedia.org/T129152#2101295 (10greg) [23:14:05] 10Deployment-Systems, 10scap, 10MediaWiki-JobRunner, 10Scap3 (scap3-adoption): Deploy jobrunner with scap3 - https://phabricator.wikimedia.org/T129148#2101299 (10greg) [23:14:07] 10Deployment-Systems, 10scap, 6Discovery, 10Scap3 (scap3-adoption): Deploy discovery-analytics with scap3 - https://phabricator.wikimedia.org/T129149#2101298 (10greg) [23:14:08] lots o spam coming [23:14:09] 10Deployment-Systems, 10scap, 10Tilerator, 10Scap3 (scap3-adoption): Deploy tilerator with scap3 - https://phabricator.wikimedia.org/T129146#2101301 (10greg) [23:14:10] sorry sorry [23:14:11] 10Deployment-Systems, 10scap, 6Mobile-Apps, 10Scap3 (scap3-adoption): Deploy mobileapps/deploy with scap3 - https://phabricator.wikimedia.org/T129147#2101300 (10greg) [23:14:13] had to do it though [23:14:13] 10Deployment-Systems, 10scap, 10Scap3 (scap3-adoption): remove fluoride from the deployments directory - https://phabricator.wikimedia.org/T129145#2101302 (10greg) [23:14:16] 10Deployment-Systems, 10scap, 6Discovery, 10Wikidata, and 2 others: Deploy wdqs with scap3 - https://phabricator.wikimedia.org/T129144#2101303 (10greg) [23:14:19] 10Deployment-Systems, 10scap, 10Citoid, 10VisualEditor, 10Scap3 (scap3-adoption): Deploy zotero/translators with scap3 - https://phabricator.wikimedia.org/T129143#2101304 (10greg) [23:14:21] 10Deployment-Systems, 10scap, 10OCG-PDF-renderer, 10Scap3 (scap3-adoption): Deploy ocg with scap3 - https://phabricator.wikimedia.org/T129142#2101305 (10greg) [23:14:23] 10Deployment-Systems, 10scap, 10Wikimedia-Logstash, 10Scap3 (scap3-adoption): Deploy kibana with scap3 - https://phabricator.wikimedia.org/T129138#2101308 (10greg) [23:14:25] 10Deployment-Systems, 10scap, 10Monitoring, 10Scap3 (scap3-adoption): Deploy statsv with scap3 - https://phabricator.wikimedia.org/T129139#2101307 (10greg) [23:14:27] 10Deployment-Systems, 10scap, 10Scap3 (scap3-adoption): Deploy zotero/translation-server with scap3 - https://phabricator.wikimedia.org/T129140#2101306 (10greg) [23:14:31] 10releng-201516-q2, 3releng-201516-q3, 10scap, 10Scap3 (scap3-adoption): [keyresult] Migrate all Service team owned services to scap - https://phabricator.wikimedia.org/T109926#2101291 (10greg) [23:14:34] 3Scap3, 10scap, 10Phabricator, 13Patch-For-Review, 7WorkType-NewFunctionality: Deploy Phabricator with scap3 - https://phabricator.wikimedia.org/T114363#2101311 (10greg) [23:14:38] 10Deployment-Systems, 10scap, 10Monitoring, 6Operations, 10Scap3 (scap3-adoption): Deploy servermon with scap3 - https://phabricator.wikimedia.org/T129152#2096629 (10greg) [23:14:40] 10Deployment-Systems, 10scap, 10Sentry, 10Scap3 (scap3-adoption): Deploy sentry with scap3 - https://phabricator.wikimedia.org/T129155#2096676 (10greg) [23:14:42] 10Deployment-Systems, 10scap, 10Wikimedia-Stream, 10Scap3 (scap3-adoption): Deploy rcstream with scap3 - https://phabricator.wikimedia.org/T129153#2096650 (10greg) [23:14:44] 10Deployment-Systems, 10scap, 10Wikimedia-IEG-grant-review, 10Scap3 (scap3-adoption): Deploy iegreview with scap3 - https://phabricator.wikimedia.org/T129154#2096664 (10greg) [23:14:46] 10Deployment-Systems, 10scap, 10Kartotherian, 10Scap3 (scap3-adoption): Deploy kartotherian with scap3 - https://phabricator.wikimedia.org/T129150#2096601 (10greg) [23:14:49] 10Deployment-Systems, 10scap, 10MediaWiki-JobRunner, 10Scap3 (scap3-adoption): Deploy jobrunner with scap3 - https://phabricator.wikimedia.org/T129148#2096570 (10greg) [23:14:51] 10Deployment-Systems, 10scap, 6Discovery, 10Scap3 (scap3-adoption): Deploy discovery-analytics with scap3 - https://phabricator.wikimedia.org/T129149#2096588 (10greg) [23:14:53] 10Deployment-Systems, 10scap, 10Tilerator, 10Scap3 (scap3-adoption): Deploy tilerator with scap3 - https://phabricator.wikimedia.org/T129146#2096538 (10greg) [23:14:55] 10Deployment-Systems, 10scap, 10Analytics, 10Analytics-Cluster, 10Scap3 (scap3-adoption): Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2096617 (10greg) [23:14:58] 10Deployment-Systems, 10scap, 6Mobile-Apps, 10Scap3 (scap3-adoption): Deploy mobileapps/deploy with scap3 - https://phabricator.wikimedia.org/T129147#2096552 (10greg) [23:15:01] 10Deployment-Systems, 10scap, 10Citoid, 10VisualEditor, 10Scap3 (scap3-adoption): Deploy zotero/translators with scap3 - https://phabricator.wikimedia.org/T129143#2096492 (10greg) [23:15:03] 10Deployment-Systems, 10scap, 10Scap3 (scap3-adoption): remove fluoride from the deployments directory - https://phabricator.wikimedia.org/T129145#2096525 (10greg) [23:15:05] 10Deployment-Systems, 10scap, 10OCG-PDF-renderer, 10Scap3 (scap3-adoption): Deploy ocg with scap3 - https://phabricator.wikimedia.org/T129142#2096453 (10greg) [23:15:07] 10Deployment-Systems, 10scap, 10Scap3 (scap3-adoption): Deploy zotero/translation-server with scap3 - https://phabricator.wikimedia.org/T129140#2096415 (10greg) [23:15:18] 3Scap3, 10scap, 10Phabricator, 13Patch-For-Review, 7WorkType-NewFunctionality: Deploy Phabricator with scap3 - https://phabricator.wikimedia.org/T114363#1693175 (10greg) [23:15:21] 10Deployment-Systems, 10scap, 10Analytics, 10Analytics-Cluster, 10Scap3 (scap3-adoption): Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2096617 (10greg) @Ottomata yeah, that task has morphed into something it wasn't, fixed. [23:15:43] 3releng-201516-q1, 15User-greg: [keyresult] Implement the needed features in scap for RESTBase deploys - https://phabricator.wikimedia.org/T113119#2101368 (10greg) 5Open>3Resolved a:3greg [23:18:08] 10releng-201516-q2, 3releng-201516-q3, 10scap, 10Scap3 (scap3-adoption): [keyresult] Migrate all Service team owned services and MW to scap - https://phabricator.wikimedia.org/T109926#2101379 (10greg) [23:18:38] 3releng-201516-q3, 10scap, 10Scap3 (scap3-adoption), 7WorkType-NewFunctionality: [keyresult] Migrate the MW weekly train deploy to scap3 - https://phabricator.wikimedia.org/T114313#2101381 (10greg) [23:18:40] 10releng-201516-q2, 3releng-201516-q3, 10scap, 10Scap3 (scap3-adoption): [keyresult] Migrate all Service team owned services and MW to scap - https://phabricator.wikimedia.org/T109926#1562968 (10greg) [23:18:45] thanks greg-g :) [23:20:03] thcipriani: sorry for the spam, but I'm a stickler for not changing the past :) [23:20:46] I setup these bugs weirdly, then didn't realize until after I made a ton of spam :\ [23:21:13] it's ok, we broke the spamming up into two separate days, so people get to enjoy it twice! [23:21:31] "look, we care!" is the message we're sending, all customers want that [23:25:19] RECOVERY - Puppet failure on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [23:25:44] (03PS1) 10Paladox: [EditPageTracking] Add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/276040 [23:28:26] 10Continuous-Integration-Config, 13Patch-For-Review, 5WMF-deploy-2016-03-08_(1.27.0-wmf.16): Convert all repos to use npm with fallback to jshint and jsonlint for non-whitelisted users - https://phabricator.wikimedia.org/T119973#2101423 (10greg) a:3Paladox You're making good progress on this, so setting th... [23:29:52] 10Continuous-Integration-Config, 13Patch-For-Review, 5WMF-deploy-2016-03-08_(1.27.0-wmf.16): Convert all repos to use npm with fallback to jshint and jsonlint for non-whitelisted users - https://phabricator.wikimedia.org/T119973#2101428 (10Paladox) @greg thanks yes, I'm going to work on it. It will be hard f... [23:31:06] 6Release-Engineering-Team: Reassess idea of CI point person role - https://phabricator.wikimedia.org/T129295#2101429 (10greg) [23:32:09] thcipriani: greg-g: extra bonus, you got wikibugs kicked for spamming too ;) [23:34:28] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 7Beta-Cluster-reproducible: Could not find local user data for - https://phabricator.wikimedia.org/T115198#1717768 (10greg) >>! In T115198#2043804, @Glaisher wrote: > @legoktm Should the script be run on beta cluster too? The test case (http... [23:35:16] twentyafterfour: yeah, I like to kill wikibugs via spam at least once every two weeks [23:38:11] wikibugs shouldn't be so gullible [23:38:58] * bd808 tries to find his desktop under all the ping notifications [23:40:32] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 7Beta-Cluster-reproducible: Could not find local user data for - https://phabricator.wikimedia.org/T115198#2101453 (10Legoktm) a:3Legoktm Oh yeah, sure. This wasn't on my radar since I wasn't assigned to it :) [23:42:08] !log running CentralAuth's checkLocalUser.php --verbose=1 --delete=1 on deployment-tin for T115198 [23:42:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [23:44:03] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 7Beta-Cluster-reproducible: Could not find local user data for - https://phabricator.wikimedia.org/T115198#2101468 (10Legoktm) 5Open>3Resolved {P2724} [23:44:50] legoktm: thanks man [23:45:10] bd808: was that the quite-large straw? [23:45:21] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10Unplanned-Sprint-Work, 3Fundraising Sprint Elevator Maintenance 2016, 13Patch-For-Review: CI issues on fundraising/REL1_25 - https://phabricator.wikimedia.org/T128883#2089135 (10cwdent) 5Open>3Resolved Call this done as the described problems... [23:45:39] greg-g: naw. I just like to whine [23:46:54] typical [23:50:51] do you think unit tests should depend on configuration? [23:51:40] I mean, obviously they do, and sometimes it is necessary, e.g. the DB password needs to be correct [23:52:10] Ideally the test would setup anything it depends on [23:52:44] otherwise we're stearing into integration test realm [23:53:01] steering* bah [23:53:27] yeah, we don't really make that distinction though [23:53:40] I was going to ask whether we're talking about true unit tests or MW "unit tests" [23:53:51] yeah I mean tests that are run by phpunit [23:54:03] which are often pretty heavily integrated [23:54:30] https://phabricator.wikimedia.org/T84948 [23:54:45] better: https://phabricator.wikimedia.org/T87781 "Split mediawiki tests into unit and integration tests" [23:54:53] mmm [23:55:23] https://lists.wikimedia.org/pipermail/mediawiki-core/2015-February/000147.html [23:55:27] (sorry, will stop link spamming) [23:55:46] in writing new code, I think we should be working on making config settings be set in class constructors and stuff and not via globals so you can test different options without relying upon setMwGlobals() and stuff [23:56:36] I like the idea of splitting off an integration test directory, since I think there is some value in integration testing [23:58:05] integration testing definitely has value [23:58:07] then when a dev does a stupid thing like sets $wgAPIMaxResultSize = 200 to test some failure mode and then forgets to set it back and then can't figure out why all the FauxRequest API tests are failing, then it'll be a little bit more obvious what is going on [23:58:42] integration tests could be run against some other LocalSettings.php [23:59:15] oh and btw I just did that, set $wgAPIMaxResultSize = 200