[00:14:40] Project beta-scap-eqiad build #42030: FAILURE in 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/42030/ [00:34:53] Yippee, build fixed! [00:34:53] Project beta-scap-eqiad build #42032: FIXED in 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/42032/ [01:19:06] (03CR) 10Krinkle: Fix WikibaseJavaScriptApi tests (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (https://phabricator.wikimedia.org/T86176) (owner: 10Adrian Lang) [01:34:10] 3Continuous-Integration, Services, Parsoid, RESTBase: Move Parsoid and RESTBase testing from Travis CI to our Jenkins - https://phabricator.wikimedia.org/T78410#1042734 (10GWicke) @hashar, this still sounds fairly complex and labor intense. How many vms would we need to maintain? One per repository? Would we hav... [01:36:09] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<12.50%) [01:47:11] 3Labs, Labs-Vagrant, Wikimedia-Labs-Other, Wikimedia-Labs-wikistats, Wikimedia-Labs-wikitech-interface, Wikimedia-Labs-General, Beta-Cluster, Project-Creators, Wikimedia-Labs-Infrastructure, Tool-Labs-tools-Article-request, Tool-Labs, Wikimedia-Labs-extdist, MediaW... - https://phabricator.wikimedia.org/T89270#1042736 [03:38:05] !log Depooled integration-slave1009 [03:38:09] Logged the message, Master [03:38:41] Yippee, build fixed! [03:38:41] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #319: FIXED in 31 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/319/ [03:39:13] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #545: FAILURE in 28 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/545/ [03:42:51] I'm going to start with an strace and go from there, which I think will be easiest if I'm actually typing the commands [03:42:58] !log Depooled integration-slave1009 (Debugging T89180) [03:43:04] Logged the message, Master [03:43:09] TimStarling: Yeah [03:43:23] TimStarling: Just a minute, current job almost finished [03:43:57] cd /mnt/jenkins-workspace/workspace/mwext-VisualEditor-qunit-karma/src [03:45:06] as user www-data? [03:45:09] Aye, will need a few one-time commands to undo the teardown [03:45:10] Nope [03:45:37] jenkins-deploy [03:46:09] I don't have phpunit tests running there so not sure if they trigger it. Let's trigger the qunit tests just to match the jenkins build [03:46:09] I'll try to be patient ;) [03:46:44] I'll do the one time commands to get that build running into a position where you can run it multiple times [03:46:50] npm install, and symlinking to docroot basically [03:47:09] sounds good, thanks [03:47:42] Oh, and a crapload of env variables. Errr.. [03:48:54] ln -s /mnt/jenkins-workspace/workspace/mwext-VisualEditor-qunit-karma/src /srv/localhost/qunit/jenkins-mwext-VisualEditor-qunit-karma-259 [03:48:56] npm install [03:49:02] export MW_SERVER=http://localhost:9412 export MW_SCRIPT_PATH=/jenkins-mwext-VisualEditor-qunit-karma-259 [03:49:15] export CHROME_BIN=/usr/bin/chromium-browser export DISPLAY=:94 [03:49:22] almost there [03:50:00] Oh right, need to install mw db. Teardown takes that way too. [03:50:22] OK. I'll do this another way. [03:55:12] TimStarling: I've created a duplicate job with the actual test and cleanup steps omitted. It's ready in /mnt/jenkins-workspace/workspace/mwext-VisualEditor-qunit-karma-debug/src [03:55:38] To run the tests, these commands: [03:56:37] export MW_SERVER=http://localhost:9412 export MW_SCRIPT_PATH=/jenkins-mwext-VisualEditor-qunit-karma-debug-2 export CHROME_BIN=/usr/bin/chromium-browser export DISPLAY=:94 export CHROME_BIN=/usr/bin/chromium-browser export DISPLAY=:94 [03:56:47] then $ grunt karma:main [03:56:48] that's it [03:57:59] in ../log/mw-dberror.log are some database lock entries from the one I just ran the same way [03:58:26] All yours :) [03:59:49] thanks [04:01:04] I happen to pick an unmerged patch that actually fails, so it doesn't run very long. But long enough to trigger the lock. [04:03:27] MW runs under apache? [04:05:10] TimStarling: Yep [04:05:26] Let me double check [04:06:07] TimStarling: Yeah. [04:06:48] TimStarling: Unless our puppet has a standard way of using hhvm with apache, I suspect it might be using hhvm for cli and zend for apache. Not sure how that's handled at the moment. [04:08:50] I found the relevant access logs, I'm seeing if I can reproduce it just by doing a few requests with curl [04:10:04] cool [04:10:38] Yippee, build fixed! [04:10:39] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #512: FIXED in 29 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/512/ [04:10:48] TimStarling: load.php would likely hit it, though it's cache is primed now. [04:10:55] its [04:11:23] a quick update.php will clear that [04:11:41] mediawiki.language.data is an expensive one if you're going that route. [04:12:28] yeah, ok, there was no DB error when I ran it at 04:10, I will try cache clear [04:12:49] Even l10n cache on a plain blankpage could hit it I suppose [04:12:53] Yeah [04:22:06] it's enough to truncate msg_resource/msg_resource_links [04:23:53] TimStarling: what curl request are you using to have it try to populate again? [04:24:43] there's 26 requests altogether, I'm just doing some text processing to make them into a URL list [04:27:14] Right. A plain curl to e.g. blankpage shouldn't make resourceloader populate msg_resource. Though I wouldn't be surprised if we're instantiating ResourceLoader to some extend on non-load.php requests. Though ideally not enough ot make it start fetching messages [04:27:55] Yippee, build fixed! [04:27:55] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #521: FIXED in 17 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/521/ [04:28:00] Yeah, access logs are still shared. Though prefixed. Needs sudo, but you're there already. [04:29:31] TimStarling: I gotta go soon. [04:29:42] TimStarling: Leaving that slave offline for now, should be fine. [04:30:03] We're tight resourced, but not that tight. [04:30:13] ok [04:30:29] 5 slaves for trusty atm. [04:30:31] I've reproduced it with curl btw, with ~tstarling/run-test [04:30:31] k, o/ [04:30:46] bye [04:31:01] Ah, OK. I'll snapshot that. [04:31:10] later :) [05:15:03] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #473: FAILURE in 31 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/473/ [05:34:28] Yippee, build fixed! [05:34:28] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #472: FIXED in 24 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/472/ [05:39:56] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #317: FAILURE in 46 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/317/ [06:08:47] 3Continuous-Integration: MediaWiki installs in Jenkins frequently fail to access their sqlite database due to locks - https://phabricator.wikimedia.org/T89180#1042914 (10tstarling) So it looks like ATTR_TIMEOUT doesn't help. Krinkle helped me to reproduce the issue on integration-slave1009. My slightly reduced t... [06:24:44] Who can review: https://gerrit.wikimedia.org/r/#/c/190776 [06:24:47] :) [06:34:38] Project beta-scap-eqiad build #42071: FAILURE in 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/42071/ [06:35:10] 3Continuous-Integration: MediaWiki installs in Jenkins frequently fail to access their sqlite database due to locks - https://phabricator.wikimedia.org/T89180#1042941 (10tstarling) SQLite is very pleasant to read, with lots of comments. This is a possible location for the issue, in sqlite3PagerBegin(): ```... [06:36:08] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [06:53:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:54:55] Yippee, build fixed! [06:54:55] Project beta-scap-eqiad build #42073: FIXED in 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/42073/ [06:58:25] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [07:23:25] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:23:35] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [08:06:24] PROBLEM - Puppet failure on deployment-urldownloader is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:21:29] RECOVERY - Puppet failure on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [08:24:01] 3Continuous-Integration: hhvm-build fails on php-luasandbox and on wikidiff2 PHP extensions - https://phabricator.wikimedia.org/T89649#1043017 (10Joe) 5Open>3Resolved [08:24:21] 3Continuous-Integration: hhvm-build fails on php-luasandbox and on wikidiff2 PHP extensions - https://phabricator.wikimedia.org/T89649#1041419 (10Joe) Thanks @hashar <3 [08:29:53] 3Release-Engineering, Beta-Cluster: Reduce [LOCAL HACK] changes on Beta Cluster to zero - https://phabricator.wikimedia.org/T76392#1043025 (10Joe) [08:29:56] 3operations, Beta-Cluster: Make www-data the web-serving user (is currently apache) - https://phabricator.wikimedia.org/T78076#1043023 (10Joe) 5stalled>3Open a:5yuvipanda>3Joe [09:05:07] aharoni: I am in the hangout [09:07:17] PROBLEM - Puppet staleness on deployment-eventlogging02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [09:08:13] aharoni: can you hear me? [09:17:29] PROBLEM - Puppet failure on deployment-urldownloader is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:22:33] 3Release-Engineering, operations: /usr/local/bin/deploy2graphite broken on tin due to nc command syntax - https://phabricator.wikimedia.org/T1387#1043064 (10fgiunchedi) a:3fgiunchedi ping? @ori @bd808 there was a question in https://gerrit.wikimedia.org/r/#/c/183568/ re: the history of this file move which I'm... [09:57:30] RECOVERY - Puppet failure on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [10:09:57] PROBLEM - Content Translation Server on deployment-cxserver03 is CRITICAL: Connection refused [10:19:56] RECOVERY - Content Translation Server on deployment-cxserver03 is OK: HTTP OK: HTTP/1.1 200 OK - 1103 bytes in 0.023 second response time [10:46:08] (03PS1) 10Jhernandez: Add Gather extension conf [integration/config] - 10https://gerrit.wikimedia.org/r/191009 [11:09:11] PROBLEM - Puppet failure on deployment-test is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:12:55] 3Quality-Assurance, MediaWiki-extensions-GettingStarted: GettingStarted browser tests are broken - https://phabricator.wikimedia.org/T89539#1043252 (10Mattflaschen) 5Open>3Resolved [11:31:20] hi? [11:53:41] 3Deployment-Systems: LocalisationUpdate broken since 2014-12-16 - https://phabricator.wikimedia.org/T85790#1043342 (10Aklapper) Where are we here? Is this still a problem, or has the underlying issue been sorted out? [11:53:46] 3Deployment-Systems: LocalisationUpdate is not updating messages - https://phabricator.wikimedia.org/T76061#1043344 (10Aklapper) Where are we here? Is this still a problem, or has the underlying issue been sorted out? [11:54:12] RECOVERY - Puppet failure on deployment-test is OK: OK: Less than 1.00% above the threshold [0.0] [11:56:22] 3Deployment-Systems: LocalisationUpdate broken since 2014-12-16 - https://phabricator.wikimedia.org/T85790#1043350 (10Raymond) Per https://wikitech.wikimedia.org/wiki/Server_Admin_Log it works well since since January 9th again. [12:29:27] 3Continuous-Integration, Mobile-Web: Publish MobileFrontend JS Documentation - https://phabricator.wikimedia.org/T74794#1043382 (10phuedx) @hashar Did you settle on `make docs`? [12:37:35] PROBLEM - SSH on deployment-lucid-salt is CRITICAL: Connection refused [12:48:45] 3Continuous-Integration, Possible-Tech-Projects: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1042512 (10Nemo_bis) [12:48:55] (03PS21) 10Adrian Lang: Fix WikibaseJavaScriptApi tests [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (https://phabricator.wikimedia.org/T86176) [12:49:05] (03CR) 10Adrian Lang: Fix WikibaseJavaScriptApi tests (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (https://phabricator.wikimedia.org/T86176) (owner: 10Adrian Lang) [12:54:41] Project beta-scap-eqiad build #42108: FAILURE in 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/42108/ [13:14:55] Yippee, build fixed! [13:14:56] Project beta-scap-eqiad build #42110: FIXED in 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/42110/ [13:36:31] 3Release-Engineering, § Phabricator-Sprint-Extension, Phabricator: Integrate Jenkins with Phabricator with Harbormaster - https://phabricator.wikimedia.org/T89714#1043458 (10Christopher) 3NEW [13:59:22] 3Continuous-Integration, Mobile-Web: Publish MobileFrontend JS Documentation - https://phabricator.wikimedia.org/T74794#1043483 (10hashar) [13:59:23] 3Continuous-Integration, RESTBase: Publish RESTBase documentation on doc.wikimedia.org - https://phabricator.wikimedia.org/T87702#1043484 (10hashar) [13:59:23] 3Continuous-Integration: Define entry point to generate documentation - https://phabricator.wikimedia.org/T88999#1043481 (10hashar) 5Open>3Resolved After a week on wikitech-l, `make docs` is the documentation entry point from now on. I have [[ https://www.mediawiki.org/w/index.php?title=Continuous_integrat... [14:01:19] 3Continuous-Integration: Define entry point to generate documentation - https://phabricator.wikimedia.org/T88999#1043489 (10valhallasw) What should the output directory be? [14:02:49] 3Continuous-Integration, Mobile-Web: Publish MobileFrontend JS Documentation - https://phabricator.wikimedia.org/T74794#1043490 (10hashar) After a week on wikitech-l and barely any comment, `make docs` is adopted as an entry point to generate documentation. Could you provide a `/makefile` with a `docs` target p... [14:04:17] 3Release-Engineering, § Phabricator-Sprint-Extension, Phabricator: Integrate Jenkins with Phabricator with Harbormaster - https://phabricator.wikimedia.org/T89714#1043493 (10Qgil) [14:04:34] 3Continuous-Integration, Code-Review: Connect Differential code review with continuous integration - https://phabricator.wikimedia.org/T31#477 (10Qgil) [14:12:44] 3operations, Beta-Cluster: Make www-data the web-serving user (is currently apache) - https://phabricator.wikimedia.org/T78076#1043515 (10Joe) Production is being finally converted to www-data today; we will need to deploy the scap fix before we deploy today. I'll ping @bd808 [14:22:19] zeljkof: y [14:22:20] yo [14:22:24] thanks for merging. [14:22:30] I made a bunch of other rubocop commits. [14:25:31] aharoni: did you add me to reviewers? [14:25:36] yes [14:25:44] aharoni: will take a look [14:27:16] thanks [14:27:20] in GitHub, too :) [14:28:13] zeljkof: https://github.com/amire80/screenshot/pull/3 [14:35:06] 3Continuous-Integration, OOjs-UI: PHP docs should be auto-generated - https://phabricator.wikimedia.org/T74454#1043547 (10matmarex) p:5Normal>3High [14:35:30] 3Continuous-Integration, Mobile-Web: Publish MobileFrontend JS Documentation - https://phabricator.wikimedia.org/T74794#1043550 (10Florian) @hashar: We already have a Makefile with a "docs" target :) https://github.com/wikimedia/mediawiki-extensions-MobileFrontend/blob/master/Makefile#L44 [14:39:14] (03PS1) 10Hashar: Job templates for `make docs` [integration/config] - 10https://gerrit.wikimedia.org/r/191045 (https://phabricator.wikimedia.org/T88999) [14:39:16] (03PS1) 10Hashar: Docs jobs for MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/191046 (https://phabricator.wikimedia.org/T74794) [14:41:19] 3Continuous-Integration, RESTBase: Publish RESTBase documentation on doc.wikimedia.org - https://phabricator.wikimedia.org/T87702#1043584 (10hashar) After a week on wikitech-l and barely any comment, make docs is adopted as an entry point to generate documentation. Could you provide a /makefile with a docs targe... [15:06:56] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [15:11:19] aharoni: merged all the commits :) [15:11:41] (03PS2) 10Hashar: Job templates for `make docs` [integration/config] - 10https://gerrit.wikimedia.org/r/191045 (https://phabricator.wikimedia.org/T88999) [15:11:42] (03PS2) 10Hashar: Docs jobs for MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/191046 (https://phabricator.wikimedia.org/T74794) [15:12:46] zeljkof: yay [15:12:47] thank you [15:12:59] aharoni: thank _you_ :) [15:17:00] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:19:21] (03PS3) 10Hashar: Job templates for `make docs` [integration/config] - 10https://gerrit.wikimedia.org/r/191045 (https://phabricator.wikimedia.org/T88999) [15:19:23] (03PS3) 10Hashar: Docs jobs for MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/191046 (https://phabricator.wikimedia.org/T74794) [15:22:54] 3Continuous-Integration, Mobile-Web: Publish MobileFrontend JS Documentation - https://phabricator.wikimedia.org/T74794#1043629 (10hashar) I made a first pass job that clones mediawiki/core and MobileFrontend under /src/ then do: ``` export MW_INSTALL_PATH=/src cd /src/extensions/MobileFrontend make docs ```... [15:51:03] (03PS1) 10Jforrester: Add npm test for Citoid extension [integration/config] - 10https://gerrit.wikimedia.org/r/191063 [15:51:05] (03PS1) 10Jforrester: Make Citoid service npm job voting [integration/config] - 10https://gerrit.wikimedia.org/r/191064 [15:52:00] (03CR) 10jenkins-bot: [V: 04-1] Make Citoid service npm job voting [integration/config] - 10https://gerrit.wikimedia.org/r/191064 (owner: 10Jforrester) [15:52:02] (03CR) 10jenkins-bot: [V: 04-1] Add npm test for Citoid extension [integration/config] - 10https://gerrit.wikimedia.org/r/191063 (owner: 10Jforrester) [15:52:54] Bah. [15:54:13] (03CR) 10Jforrester: "I'm sure that error made sense to someone…" [integration/config] - 10https://gerrit.wikimedia.org/r/191063 (owner: 10Jforrester) [16:15:39] 3Release-Engineering: Describe input and actions that trigger work by Release Engineering people - https://phabricator.wikimedia.org/T89720#1043686 (10Cmcmahon) 3NEW a:3Cmcmahon [16:16:36] 3Release-Engineering, § Phabricator-Sprint-Extension, Phabricator: Integrate Jenkins with Phabricator with Harbormaster - https://phabricator.wikimedia.org/T89714#1043694 (10chasemp) There is a big leap here in that Harbormaster is a prototype application for which upstream will not take feature or //bug// repor... [16:18:26] 3Release-Engineering: Describe process for setting, reporting, and addressing priority of Release Engineering work - https://phabricator.wikimedia.org/T89722#1043704 (10Cmcmahon) 3NEW a:3Cmcmahon [16:25:43] legoktm: oh god, definition of bikeshed :) [16:40:12] 3Continuous-Integration, § Phabricator-Sprint-Extension, Phabricator: Integrate Jenkins with Phabricator with Harbormaster - https://phabricator.wikimedia.org/T89714#1043783 (10greg) [16:41:11] (03CR) 10KartikMistry: "Ping!" [integration/config] - 10https://gerrit.wikimedia.org/r/190708 (https://phabricator.wikimedia.org/T87607) (owner: 10KartikMistry) [16:41:22] ^demon|sick: you too?! [16:41:34] ^demon|sick: I was sick all weekend, great way to "enjoy" a 3 day weekend :/ [16:42:06] <^demon|sick> I at least got through Sat/Sun ok. Something I ate yesterday did not sit well over night. [16:42:47] so, I've infiltrated the office. After only a brief period standing in the atrium in front of the elevators. [16:42:58] ahh, mine is your typical "sickness you get from taking your toddler to gymnastics and jumping in that foam pit to play with him/get him out" [16:43:07] ewwww [16:43:20] <^demon|sick> greg-g: That foam pit is a pit of malaria. [16:44:14] thcipriani: where are you? [16:44:22] I have enormous respect for the parents of the world, but I am very very glad not to be one. [16:44:32] <^demon|sick> That ^^^^ [16:44:44] you're welcome [16:44:48] :) [16:45:03] thcipriani: are you on the 6th floor? [16:45:22] greg-g: yessur. I'm sitting at a little table in a very open area. [16:45:52] I feel like I'm sitting somewhere fairly conspicuous...but that could just be my imagination. [16:46:09] thcipriani: cool, Sarah is coming to get you [16:46:53] greg-g: cool, thanks :) [16:55:22] PROBLEM - Host deployment-sca01 is DOWN: CRITICAL - Host Unreachable (10.68.17.54) [16:56:08] oh good [16:58:58] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki exception - 1613 bytes in 0.553 second response time [16:59:08] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki exception - 1613 bytes in 0.538 second response time [16:59:36] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki exception - 1874 bytes in 2.104 second response time [17:03:00] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki exception - 1869 bytes in 0.549 second response time [17:06:38] 3operations, Beta-Cluster: File upload area resorts to 0777 permissions to for uploaded content - https://phabricator.wikimedia.org/T75206#1043885 (10Joe) [17:06:39] 3Release-Engineering, Beta-Cluster: Reduce [LOCAL HACK] changes on Beta Cluster to zero - https://phabricator.wikimedia.org/T76392#1043886 (10Joe) [17:06:40] 3operations, Beta-Cluster: Make www-data the web-serving user (is currently apache) - https://phabricator.wikimedia.org/T78076#1043884 (10Joe) 5Open>3Resolved [17:06:52] (03CR) 10Krinkle: [C: 04-1] "Issues:" [integration/config] - 10https://gerrit.wikimedia.org/r/191063 (owner: 10Jforrester) [17:07:30] 3operations, Beta-Cluster: Minimize differences between beta and production (Tracking) - https://phabricator.wikimedia.org/T87220#1043889 (10Joe) [17:07:31] 3Release-Engineering, Beta-Cluster: Reduce [LOCAL HACK] changes on Beta Cluster to zero - https://phabricator.wikimedia.org/T76392#1043887 (10Joe) 5Open>3Resolved a:3Joe [17:07:51] Krinkle: Oh, right, the stupid test/check naming problem. [17:08:28] (03CR) 10Krinkle: Make Citoid service npm job voting (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/191064 (owner: 10Jforrester) [17:09:56] Krinkle: Can't we add -npm to the extension-unittests template and have it pass if there's no Gruntfile.js in the repo? [17:10:12] Krinkle: It'd be a lot quicker than individually enabling it… [17:10:26] James_F: No, that'd be a ton of work. The infrastructure isn't built for that [17:10:48] account half a day [17:11:05] 1x 1/2 a day vs. 200x 1/8 a day. [17:11:24] Rather, simplifying jobs in general. Test entry points are reducing. [17:11:28] We wn't need any templates soon [17:12:30] (03PS2) 10Jforrester: Add npm test for Citoid extension [integration/config] - 10https://gerrit.wikimedia.org/r/191063 [17:12:45] <^demon|sick> twentyafterfour: Top 2 offenders at the moment are Cirrus. That seems....like something I have to fix :) [17:13:10] <^demon|sick> 173 Use of undefined constant CURLE_OPERATION_TIMEDOUT - assumed 'CURLE_OPERATION_TIMEDOUT' in /srv/mediawiki/php-1.25wmf16/extensions/Elastica/ElasticaConnection.php on line 100 [17:13:21] <^demon|sick> 170 Unexpected connection error communicating with Elasticsearch. Curl code: 28 [Called from {closure} in /srv/mediawiki/php-1.25wmf16/extensions/Elastica/ElasticaConnection.php at line 109] in /srv/mediawiki/php-1.25wmf16/includes/debug/MWDebug.php on line 300 [17:13:27] (03CR) 10jenkins-bot: [V: 04-1] Add npm test for Citoid extension [integration/config] - 10https://gerrit.wikimedia.org/r/191063 (owner: 10Jforrester) [17:13:40] (03PS2) 10Jforrester: Make Citoid service npm job voting [integration/config] - 10https://gerrit.wikimedia.org/r/191064 [17:14:01] * James_F sighs. [17:14:43] <^demon|sick> How is CURLE_OPERATION_TIMEDOUT undefined? [17:14:50] * ^demon|sick sighs a little [17:15:37] curl extension not loaded? [17:15:52] <^demon|sick> That seems wrong... [17:17:43] <^demon|sick> Multiple servers, not just one [17:18:55] that's weird [17:20:09] Project beta-update-databases-eqiad build #7670: FAILURE in 9.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/7670/ [17:20:51] 3Release-Engineering: Investigate production and/or beta requirements for Sentry - https://phabricator.wikimedia.org/T89732#1043962 (10dduvall) 3NEW [17:21:18] 3Release-Engineering: Add credentials for Echo users to browser test builds in Jenkins - https://phabricator.wikimedia.org/T89497#1043971 (10hashar) We rewetted the passwords last week, seems I have missed the EchoUser :-( The old and new passwords are at https://office.wikimedia.org/wiki/Selenium_passwords and... [17:24:23] 3Release-Engineering: Add credentials for Echo users to browser test builds in Jenkins - https://phabricator.wikimedia.org/T89497#1043983 (10zeljkofilipin) a:5hashar>3zeljkofilipin [17:25:19] http://www.php.net/manual/en/curl.constants.php#93950 [17:25:41] "Beware of CURLE_* constants! [17:25:43] some constants are different, some missing compared to the PHP implementation." [17:26:11] it's called "CURLE_OPERATION_TIMEOUTED" in php [17:26:14] <^demon|sick> Depends on libcurl version, right? [17:26:20] I guess so [17:26:31] notice the ED on the end? [17:26:32] PROBLEM - Host deployment-logstash1 is DOWN: CRITICAL - Host Unreachable (10.68.16.134) [17:27:06] depends on php version I suppose. not sure how hhvm does it [17:27:56] PROBLEM - Host deployment-cxserver03 is DOWN: CRITICAL - Host Unreachable (10.68.16.150) [17:28:00] <^demon|sick> Yeah, seems like CURLE_OPERATION_TIMEOUTED is what we want [17:29:42] PROBLEM - Host deployment-restbase02 is DOWN: CRITICAL - Host Unreachable (10.68.16.234) [17:29:47] yep [17:29:53] <^demon|sick> iirc, there's a hack in core for this too [17:30:24] might be better to define CURLE_OPERATION_TIMEOUT to match _TIMEOUTED's value [17:30:26] PROBLEM - Host deployment-db1 is DOWN: CRITICAL - Host Unreachable (10.68.16.193) [17:30:36] since that TIMEOUTED constant is batshit [17:32:49] ^demon: want me to submit a patch for that or you want to do it? anyone have an opinion on the constant naming? [17:34:21] <^demon|sick> https://gerrit.wikimedia.org/r/#/c/191077/ done [17:36:27] +2'd [17:36:49] 3Release-Engineering: Add credentials for Echo users to browser test builds in Jenkins - https://phabricator.wikimedia.org/T89497#1044025 (10zeljkofilipin) Jobs: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/ https://integration.wikimedia.org/ci/job/... [17:37:00] <^demon|sick> My editor recognized the new constant and highlighted it properly :) [17:37:04] <^demon|sick> So it must really be right [17:37:22] 3Deployment-Systems: LocalisationUpdate broken since 2014-12-16 - https://phabricator.wikimedia.org/T85790#1044026 (10bd808) >>! In T85790#1043350, @Raymond wrote: > Per https://wikitech.wikimedia.org/wiki/Server_Admin_Log it works well since since January 9th again. The SAL entries are a lie unfortunately. ``... [17:37:34] 3Continuous-Integration, § Phabricator-Sprint-Extension, Phabricator: Integrate Jenkins with Phabricator with Harbormaster - https://phabricator.wikimedia.org/T89714#1044027 (10hashar) I am not sure what Harbormaster provide and it probably does not match Zuul behavior ( I commented about it on T31#6408). Imho... [17:38:18] 3Deployment-Systems: l10nupdate user can't access scap shared ssh key causing nightly l10nupdate sync process to fail - https://phabricator.wikimedia.org/T76061#1044028 (10bd808) [17:38:24] <^demon|sick> Ok, survived the meeting. I'm going back to bed. [17:39:25] 3Release-Engineering: Add credentials for Echo users to browser test builds in Jenkins - https://phabricator.wikimedia.org/T89497#1044029 (10zeljkofilipin) Error: WrongPass (MediawikiApi::LoginError) From: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sau... [17:40:26] 3Deployment-Systems: l10nupdate user can't access scap shared ssh key causing nightly l10nupdate sync process to fail - https://phabricator.wikimedia.org/T76061#1044032 (10bd808) Still happening as of 2015-02-16: ``` 02:14:23 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'php-1.... [17:40:43] do we not have CI on mediawiki extensions? https://gerrit.wikimedia.org/r/#/c/191077/ didn't get verified +2 ... zuul queue backed up? [17:41:53] twentyafterfour: it's in the merge queue -- https://integration.wikimedia.org/zuul/ [17:42:15] the gate-and-submit goes one at a time across all projects [17:42:26] 3Continuous-Integration: Define entry point to generate documentation - https://phabricator.wikimedia.org/T88999#1044039 (10hashar) >>! In T88999#1043489, @valhallasw wrote: > What should the output directory be? We can configure the output dir to publish in the Jenkins job. It is the `docsrc` JJB variable whic... [17:42:27] have been wondering about https://phabricator.wikimedia.org/T75467#1039746 [17:42:29] twentyafterfour: it's in the queue https://integration.wikimedia.org/zuul/ [17:42:29] because reasons [17:43:09] I assume we can't just live hack to override the error handler in MW's memcached get function [17:43:17] Krenair: it's a bit of a mystery. We thought we turned off the use of igbinary compression in prod months and months ago [17:44:09] If we can be sure we can tell the difference between "no key in memcached" and "corrupt key in memcached" we could log [17:44:11] since then all the old keys holding igbinary compressed things would have expired etc.? [17:44:20] I don't know if we can tell that or not [17:44:50] *nod* they should have all expired several months ago (we think) [17:45:00] <^demon|sick> igbinary got cut off on zend, long before the hhvm rollout started [17:45:08] <^demon|sick> (so hopefully everything had been evicted already) [17:45:45] but hhvm thinks it's still getting igbinary results [17:45:58] 3Continuous-Integration, § Phabricator-Sprint-Extension, Phabricator: Integrate Jenkins with Phabricator with Harbormaster - https://phabricator.wikimedia.org/T89714#1044045 (10chasemp) >>! In T89714#1044027, @hashar wrote: > I am not sure what Harbormaster provide and it probably does not match Zuul behavior (... [17:46:00] https://github.com/facebook/hhvm/blob/HEAD/hphp/runtime/ext/memcached/ext_memcached.cpp#L406 [17:46:38] should be expired and yet we still see these messages in the logs today :( [17:46:58] memcached doesn't provide any way to get a list of all of these, afaik [17:47:42] !log beta cluster is mostly down because the instance supporting the main database (deployment-db1) is down. The root cause is an outage on the labs infra [17:47:47] Logged the message, Master [17:47:50] <^demon|sick> Ok, to bed for real. [17:47:51] krenair@fluorine:/a/mw-log$ grep igbinary hhvm.log -c [17:47:52] 25 [17:49:05] ^demon|sick: feel better [17:50:01] Krenair: https://phabricator.wikimedia.org/T75467 :-D [17:50:23] Krenair: maybe the error message can be hacked to list the memcached key? [17:50:58] live hack to an hhvm extension? :P [17:51:19] or overriding the error handler on mediawiki's side? [17:52:48] no clue :-D [17:52:55] maybe the error handler can catch it somehow? [17:53:31] rushing back home [17:53:42] set_error_handler( function ( $errno, $errstr, $errfile, $errline, $errcontext ) { [17:53:42] wfOneOfTheLoggingFunctions( "memc->get($key) caused warning in Memcached::get: $errstr" ); [17:53:42] }, E_WARNING ); [17:53:58] then client->get( ... ), restore_error_handler() [17:54:11] might work# [17:55:19] 3Continuous-Integration, § Phabricator-Sprint-Extension, Phabricator: Integrate Jenkins with Phabricator with Harbormaster - https://phabricator.wikimedia.org/T89714#1044053 (10mmodell) I have suggested several times that we could keep gerrit as an upstream to differential: do primary code review in differential... [17:57:55] hi! who can i ask about cucumber browser tests? I'm trying to set them up for a "new" extension and I'm having some troubles [17:58:05] 3Release-Engineering: Release MW 1.24.2 and 1.23.9 tarballs - https://phabricator.wikimedia.org/T88120#1044063 (10csteipp) >>! In T88120#1041713, @GPHemsley wrote: > We're now beginning the second week since the week of February 2nd, and there still has not been a release. > > What's the status of this? The is... [17:59:22] joakino: see chrismcmahonafk and zeljkof and marxarelli :) [17:59:42] on the other hand, it might break/slow down everything. [18:00:13] thanks greg-g [18:01:10] joakino: what's the problem? [18:01:26] terminal output or it did not happen ;) [18:02:04] chrismcmahonafk: zeljkof: I'm trying to set up browser testing for the extension Gather, but my little example fails with this error: https://gist.github.com/joakin/294d95d0fae06abe216c . Here is the wip patch https://gerrit.wikimedia.org/r/#/c/191083/ . If you could help me out it would be great :) [18:02:57] (03PS2) 1020after4: Author: Antoine Musso phabricator job to run arc lint on all repo [integration/config] - 10https://gerrit.wikimedia.org/r/183094 (https://phabricator.wikimedia.org/T85123) (owner: 10Hashar) [18:03:08] joakino: I have to go now, 7pm here, but I will take a look tomorrow [18:03:27] I have added chrismcmahonafk and marxarelli to reviewers [18:03:29] zeljkof: 7pm here too :p, will talk tomorrow then, thank you [18:03:32] (and myself) [18:03:37] thanks! [18:03:41] joakino: where are you? [18:03:50] zeljkof: spain, and you? [18:03:52] (03CR) 10jenkins-bot: [V: 04-1] Author: Antoine Musso phabricator job to run arc lint on all repo [integration/config] - 10https://gerrit.wikimedia.org/r/183094 (https://phabricator.wikimedia.org/T85123) (owner: 10Hashar) [18:03:59] joakino: croatia [18:04:08] (03CR) 1020after4: [C: 031] "^ This essentially just rebased @hashar's patch" [integration/config] - 10https://gerrit.wikimedia.org/r/183094 (https://phabricator.wikimedia.org/T85123) (owner: 10Hashar) [18:04:11] ping me tomorrow, I will be online all day [18:04:13] we can pair [18:04:22] night everybody [18:04:23] oki zeljkof, awesome, thanks! [18:07:06] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #418: FAILURE in 1 min 5 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/418/ [18:08:03] (03PS3) 1020after4: Author: Antoine Musso phabricator job to run arc lint on all repo [integration/config] - 10https://gerrit.wikimedia.org/r/183094 (https://phabricator.wikimedia.org/T85123) (owner: 10Hashar) [18:16:33] (03PS1) 10EBernhardson: Add ParserFunctions to Flow phpunit dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/191086 [18:18:59] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #128: FAILURE in 58 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/128/ [18:19:34] https://wikitech.wikimedia.org/wiki/Deployments/One_week and https://wikitech.wikimedia.org/wiki/Deployments [18:20:58] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #458: FAILURE in 58 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/458/ [18:21:36] chrismcmahonafk: I will work on this(https://phabricator.wikimedia.org/T89370) tomorrow [18:24:04] 3Continuous-Integration, Possible-Tech-Projects: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1044105 (10Addshore) [18:24:28] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #303: FAILURE in 3 min 28 sec: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/303/ [18:26:14] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #487: FAILURE in 7 min 14 sec: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/487/ [18:26:49] PROBLEM - Puppet failure on tyler-dev is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:29:59] 3Deployment-Systems: l10nupdate user can't access scap shared ssh key causing nightly l10nupdate sync process to fail - https://phabricator.wikimedia.org/T76061#1044124 (10greg) @mmodell: help here please. [18:30:28] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #319: FAILURE in 5 min 28 sec: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/319/ [18:31:31] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #513: FAILURE in 3 min 29 sec: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/513/ [18:31:56] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #410: FAILURE in 24 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/410/ [18:32:49] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #192: FAILURE in 52 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/192/ [18:33:42] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #489: FAILURE in 51 sec: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/489/ [18:36:38] I assume the brwoser test failure spam is due to the partial Labs (and partial Beta Cluster) outage? [18:39:12] 3Continuous-Integration, Possible-Tech-Projects: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1044181 (10Legoktm) [18:39:33] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #320: FAILURE in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/320/ [18:40:08] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #449: FAILURE in 33 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/449/ [18:41:48] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #414: FAILURE in 47 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/414/ [18:46:31] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #1493: FAILURE in 30 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/1493/ [18:47:31] (03CR) 1020after4: [C: 031] Author: Antoine Musso phabricator job to run arc lint on all repo [integration/config] - 10https://gerrit.wikimedia.org/r/183094 (https://phabricator.wikimedia.org/T85123) (owner: 10Hashar) [18:47:49] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #127: FAILURE in 48 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/127/ [18:51:37] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #522: FAILURE in 3 min 47 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/522/ [18:52:48] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce build #127: FAILURE in 1 min 10 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce/127/ [18:54:19] kart_: ping :) [18:56:41] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #462: FAILURE in 4 min 21 sec: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/462/ [18:59:34] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #192: FAILURE in 52 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/192/ [19:00:02] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #345: FAILURE in 27 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/345/ [19:03:36] marxarelli: https://phabricator.wikimedia.org/T89727 cold use a look from you. It may be that some of your new ruby is too fancy for older Vagrant installs [19:03:45] *could use [19:06:05] bd808: oh gnards. i should have bumped the minor version [19:07:08] oh why didn't that dawn on me either? [19:07:23] if that's all it is it's a relatively easy fix [19:08:54] (03PS2) 10EBernhardson: Add ParserFunctions to Echo phpunit dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/191086 [19:11:08] RECOVERY - Host deployment-cxserver03 is UP: PING OK - Packet loss = 0%, RTA = 0.61 ms [19:14:35] Project beta-scap-eqiad build #42151: FAILURE in 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/42151/ [19:18:12] PROBLEM - Host tyler-dev is DOWN: CRITICAL - Host Unreachable (10.68.16.215) [19:23:26] (03PS3) 10EBernhardson: Add ParserFunctions to Echo phpunit dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/191086 [19:25:15] Yippee, build fixed! [19:25:15] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #474: FIXED in 29 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/474/ [19:25:32] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #467: FAILURE in 3 min 17 sec: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/467/ [19:25:54] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #417: FAILURE in 38 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/417/ [19:26:45] (03CR) 10EBernhardson: "change pushed to jenkins and tested at https://gerrit.wikimedia.org/r/#/c/191011/" [integration/config] - 10https://gerrit.wikimedia.org/r/191086 (owner: 10EBernhardson) [19:26:58] (03CR) 10EBernhardson: [C: 031] Add ParserFunctions to Echo phpunit dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/191086 (owner: 10EBernhardson) [19:32:30] greg-g: he went sleeping alredy I think [19:34:55] Yippee, build fixed! [19:34:55] Project beta-scap-eqiad build #42153: FIXED in 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/42153/ [19:42:56] 3Continuous-Integration, Possible-Tech-Projects: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1044516 (10Qgil) [19:43:22] 3Continuous-Integration, Possible-Tech-Projects: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1042512 (10Qgil) [19:44:03] (03PS1) 10Dduvall: Experimental cucumber job for mediawiki-vagrant [integration/config] - 10https://gerrit.wikimedia.org/r/191119 [19:47:20] (03PS2) 10Dduvall: Experimental cucumber job for mediawiki-vagrant [integration/config] - 10https://gerrit.wikimedia.org/r/191119 (https://phabricator.wikimedia.org/T89489) [19:51:08] RECOVERY - Host deployment-db1 is UP: PING OK - Packet loss = 0%, RTA = 0.76 ms [20:02:32] RECOVERY - Host deployment-logstash1 is UP: PING OK - Packet loss = 0%, RTA = 0.63 ms [20:05:49] RECOVERY - Host deployment-restbase02 is UP: PING OK - Packet loss = 0%, RTA = 0.60 ms [20:07:27] c'mon beta dbs... [20:08:39] RECOVERY - Host deployment-sca01 is UP: PING OK - Packet loss = 0%, RTA = 0.69 ms [20:27:18] (03CR) 10Krinkle: "While this would be a good exception, tox (https://integration.wikimedia.org/ci/job/integration-config-tox-py27/975/consoleFull) currently" [integration/config] - 10https://gerrit.wikimedia.org/r/191063 (owner: 10Jforrester) [20:28:08] (03CR) 10Jforrester: "FFS." [integration/config] - 10https://gerrit.wikimedia.org/r/191063 (owner: 10Jforrester) [20:29:18] Krinkle: Can I make a CI job to instantly V-2 if a composer.json file is created in some repos? :-) [20:31:05] James_F: My desktop isl iterally filled with 35 screenshot of VE, MW and Chrome bugs from when I spent 10 minutes reviewing a patch this morning. So far only filed 2 of them. [20:33:29] (03PS3) 10Jforrester: Add npm test for Citoid extension [integration/config] - 10https://gerrit.wikimedia.org/r/191063 [20:33:32] Krinkle: :-( [20:34:39] (03CR) 10Jforrester: "Now a carbon-copy of extension-unittests with the -hhvm jobs enabled (clearly this is needed as the code has only been run in production o" [integration/config] - 10https://gerrit.wikimedia.org/r/191063 (owner: 10Jforrester) [20:39:25] Krinkle: Is there a sane way we could split up layout.yaml into different files for different purposes? [20:40:58] James_F: I don't know if Zuul supports that [20:41:08] James_F: It should drastically reduce over time soon though [20:42:09] Krinkle: You keep saying that. ;-) [20:42:22] Krinkle: Can you deploy/+2 https://gerrit.wikimedia.org/r/191063 then? [21:18:15] 3Continuous-Integration, § Phabricator-Sprint-Extension, Phabricator: Integrate Jenkins with Phabricator with Harbormaster - https://phabricator.wikimedia.org/T89714#1044801 (10hashar) I like the idea of keeping the Gerrit as a middle war to smooth the transition. That is less disruption for the CI workflow and... [21:25:02] 3Continuous-Integration, § Phabricator-Sprint-Extension, Phabricator: Integrate Jenkins with Phabricator with Harbormaster - https://phabricator.wikimedia.org/T89714#1044829 (10Qgil) >>! In T89714#1044801, @hashar wrote: > If we could get a proof of concept by Lyon hackathon in May that would give us a great mat... [21:52:32] (03CR) 10Hashar: "You made the author field the first line of the patch message:" [integration/config] - 10https://gerrit.wikimedia.org/r/183094 (https://phabricator.wikimedia.org/T85123) (owner: 10Hashar) [21:53:18] twentyafterfour: you almost made me an author of https://gerrit.wikimedia.org/r/#/c/183094/ :-D [21:53:35] need to git commit --amend --author "John Doe heh [21:53:55] hashar: ok I'll fix it [21:54:34] I am not sure that arc lint is any useful [21:55:07] that is was merely a proof of concept of a single job cloning all repos together [21:55:19] it would be good to run arc unit as well [21:55:41] I guess you can amend the patch https://gerrit.wikimedia.org/r/#/c/183094/3/jjb/phabricator.yaml [21:55:48] and take it over :-] [21:55:51] the proof of concept is a good starting point so that we can add some tests on top of that [21:56:08] I don't mind the authorship to be stripped, at worth just add a Signed-Off-By header [21:56:40] so `arc unit` in each directory ? [21:56:48] not sure we need to clone them altogether though [21:57:40] heck [21:57:58] twentyafterfour: we can even have the tests to be in a new repo phabricator/integrationtests or something like that [21:57:58] have jenkins clone that repo [21:58:10] and then execute whatever you have in phabricator/integrationtests :] [21:58:34] phabricator/testing.git \O/ [21:58:36] (03PS4) 1020after4: phabricator job to run arc lint on all repo [integration/config] - 10https://gerrit.wikimedia.org/r/183094 (https://phabricator.wikimedia.org/T85123) (owner: 10Hashar) [21:59:06] hehe [21:59:37] twentyafterfour: have you ever pushed jobs on Jenkins using the JJB script ? [22:00:11] hashar: no [22:00:37] https://www.mediawiki.org/wiki/CI/JJB should have everything needed [22:01:13] that is the python script which read the yaml definition, generate some XML config file for Jenkins and push it to the Jenkins API [22:01:34] * hashar end result is https://integration.wikimedia.org/ci/job/phabricator-wmf-integration/ [22:08:45] (03CR) 10Hashar: [C: 04-1] "Now we arc unit which is great." (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/183094 (https://phabricator.wikimedia.org/T85123) (owner: 10Hashar) [22:09:06] twentyafterfour: I have left some more instructions on the change. Crashing to bed now sorry :-( [22:09:18] (03CR) 10Christopher Johnson (WMDE): phabricator job to run arc lint on all repo (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/183094 (https://phabricator.wikimedia.org/T85123) (owner: 10Hashar) [22:13:11] (03PS2) 10Dduvall: Additional documentation on upgrading to 1.0.0 [selenium] - 10https://gerrit.wikimedia.org/r/189832 [22:18:21] (03CR) 10Hashar: [C: 04-1] "The Zuul definition is fine, kudos. But you now need to define a job in Jenkins Job configuration files :-]" [integration/config] - 10https://gerrit.wikimedia.org/r/191119 (https://phabricator.wikimedia.org/T89489) (owner: 10Dduvall) [22:27:00] (03CR) 10Hashar: [C: 04-1] "That fix the dependency issue for Echo. Now ParserFunctions could receive a change which pass its tests but could have some side effects " [integration/config] - 10https://gerrit.wikimedia.org/r/191086 (owner: 10EBernhardson) [22:31:30] (03PS3) 10Dduvall: Experimental cucumber job for mediawiki-vagrant [integration/config] - 10https://gerrit.wikimedia.org/r/191119 (https://phabricator.wikimedia.org/T89489) [22:31:53] (03CR) 10Dduvall: "Thanks, Antoine! Fixed it up." [integration/config] - 10https://gerrit.wikimedia.org/r/191119 (https://phabricator.wikimedia.org/T89489) (owner: 10Dduvall) [22:33:19] (03CR) 10Dduvall: "Also, when I ran `jenkins-jobs test` against the previous patch set, I didn't get an error. Should I have seen one?" [integration/config] - 10https://gerrit.wikimedia.org/r/191119 (https://phabricator.wikimedia.org/T89489) (owner: 10Dduvall) [23:18:02] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 48450 bytes in 2.343 second response time [23:18:58] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 48269 bytes in 0.609 second response time [23:19:06] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 48259 bytes in 0.594 second response time [23:19:17] !log deployment-prep Started mysql on deployment-db1; beta now appears much less broken than before [23:19:17] Logged the message, Master [23:19:20] From -labs [23:19:36] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 29084 bytes in 0.566 second response time [23:21:04] Yippee, build fixed! [23:21:04] Project beta-update-databases-eqiad build #7676: FIXED in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/7676/ [23:23:28] 3Continuous-Integration, Release-Engineering, § Fundraising Tech Backlog, Wikimedia-Fundraising-CiviCRM, § Fundraising Sprint E: Deploy CiviCRM integration job to WMF integration server - https://phabricator.wikimedia.org/T86374#967336 (10atgo) @awight is this a blocker for upgrading to 4.4? [23:32:15] 3Continuous-Integration, Release-Engineering, § Fundraising Tech Backlog, Wikimedia-Fundraising-CiviCRM, § Fundraising Sprint E: Deploy CiviCRM integration job to WMF integration server - https://phabricator.wikimedia.org/T86374#1045130 (10awight) I'd really like it to be. It will make the upgrade much less risk... [23:34:59] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-salt is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [1.0]