[02:39:11] RECOVERY - Host cache-rsync is UP: PING OK - Packet loss = 0%, RTA = 1.28 ms [03:12:32] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #1051: 04FAILURE in 30 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/1051/ [03:21:40] PROBLEM - Host cache-rsync is DOWN: CRITICAL - Host Unreachable (10.68.23.165) [05:25:38] Yippee, build fixed! [05:25:39] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #763: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/763/ [06:27:42] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #968: 04FAILURE in 8 min 41 sec: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/968/ [07:39:58] 10Continuous-Integration-Config, 10Wikidata, 07Documentation, 03Wikidata-Sprint-2016-04-12: [Story] Wikibase documentation - https://phabricator.wikimedia.org/T133028#2217487 (10hashar) jsduck is a ruby gems, we had a recentish version provided as a Debian package for Ubuntu Trusty but it turned out to be... [07:42:44] (03Abandoned) 10Hashar: [BlueSpiceExtensions] Add extension-unittests-composer and jsonlint tests [integration/config] - 10https://gerrit.wikimedia.org/r/263409 (owner: 10Paladox) [07:42:59] (03Restored) 10Hashar: [BlueSpiceExtensions] Add extension-unittests-composer and jsonlint tests [integration/config] - 10https://gerrit.wikimedia.org/r/263409 (owner: 10Paladox) [07:46:54] (03PS2) 10Hashar: [BlueSpiceExtensions] add extension-unittests-composer [integration/config] - 10https://gerrit.wikimedia.org/r/263409 (https://phabricator.wikimedia.org/T130811) (owner: 10Paladox) [07:47:30] 10Continuous-Integration-Config, 10BlueSpice, 13Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#2147279 (10hashar) @Paladox has a couple patches, I am rebasing and deploying them. [07:48:27] (03PS3) 10Hashar: [BlueSpiceFoundation] Update generic unittest test to composer unittest test [integration/config] - 10https://gerrit.wikimedia.org/r/282377 (https://phabricator.wikimedia.org/T130811) (owner: 10Paladox) [07:50:42] (03CR) 10Hashar: [C: 032] [BlueSpiceExtensions] add extension-unittests-composer [integration/config] - 10https://gerrit.wikimedia.org/r/263409 (https://phabricator.wikimedia.org/T130811) (owner: 10Paladox) [07:51:30] (03Merged) 10jenkins-bot: [BlueSpiceExtensions] add extension-unittests-composer [integration/config] - 10https://gerrit.wikimedia.org/r/263409 (https://phabricator.wikimedia.org/T130811) (owner: 10Paladox) [08:09:14] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 50703 bytes in 0.812 second response time [08:11:00] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 50704 bytes in 0.608 second response time [08:21:02] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #927: 04FAILURE in 1 min 2 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/927/ [08:25:15] 10Continuous-Integration-Config, 10BlueSpice, 13Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#2221879 (10hashar) The tests for BlueSpiceExtensions fails due to a method definition not being compatible with BlueSpiceFoundation. Should be an easy fix. I no... [08:34:00] Yippee, build fixed! [08:34:00] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #946: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/946/ [08:34:40] (03PS4) 10Phuedx: Run Popups browser tests per patch [integration/config] - 10https://gerrit.wikimedia.org/r/283568 (https://phabricator.wikimedia.org/T133018) (owner: 10Bmansurov) [08:35:01] (03CR) 10Phuedx: "PS4 is a rebase." [integration/config] - 10https://gerrit.wikimedia.org/r/283568 (https://phabricator.wikimedia.org/T133018) (owner: 10Bmansurov) [08:35:33] (03CR) 10Phuedx: [C: 031] Run Popups browser tests per patch [integration/config] - 10https://gerrit.wikimedia.org/r/283568 (https://phabricator.wikimedia.org/T133018) (owner: 10Bmansurov) [08:37:29] anyone got a minute or five to review that change ^ ? [08:41:32] (03PS3) 10Hashar: Add a non-voting generic composer phpunit job for extensions [integration/config] - 10https://gerrit.wikimedia.org/r/270737 (owner: 10Paladox) [08:48:36] (03PS4) 10Hashar: Add non-voting generic composer phpunit job for extensions [integration/config] - 10https://gerrit.wikimedia.org/r/270737 (owner: 10Paladox) [08:51:18] (03CR) 10Hashar: "Had the use case for BlueSpiceExtensions repository T130811" [integration/config] - 10https://gerrit.wikimedia.org/r/270737 (owner: 10Paladox) [08:52:40] (03PS1) 10Hashar: [BlueSpiceExtensions] make tests non voting [integration/config] - 10https://gerrit.wikimedia.org/r/284423 [08:52:59] (03PS2) 10Hashar: [BlueSpiceExtensions] make tests non voting [integration/config] - 10https://gerrit.wikimedia.org/r/284423 [08:53:10] (03CR) 10Hashar: [C: 032] Add non-voting generic composer phpunit job for extensions [integration/config] - 10https://gerrit.wikimedia.org/r/270737 (owner: 10Paladox) [08:53:57] (03CR) 10Hashar: [C: 032] [BlueSpiceExtensions] make tests non voting [integration/config] - 10https://gerrit.wikimedia.org/r/284423 (owner: 10Hashar) [08:54:00] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/270737 (owner: 10Paladox) [08:54:50] (03Merged) 10jenkins-bot: Add non-voting generic composer phpunit job for extensions [integration/config] - 10https://gerrit.wikimedia.org/r/270737 (owner: 10Paladox) [08:55:27] (03Merged) 10jenkins-bot: [BlueSpiceExtensions] make tests non voting [integration/config] - 10https://gerrit.wikimedia.org/r/284423 (owner: 10Hashar) [08:58:40] 10Deployment-Systems, 06Release-Engineering-Team, 06Operations, 06Services: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#2221923 (10akosiaris) Unsure, why this is #blocked-on-operations. I 'll remove the tag for now, feel free to readd. [09:04:38] 10Continuous-Integration-Infrastructure, 06Operations, 07Blocked-on-Operations, 07HHVM: /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so no such file or directory - https://phabricator.wikimedia.org/T126658#2020104 (10akosiaris) I see rOPUPf6c222149 has been merged since which might have fix... [09:05:16] 10Continuous-Integration-Config, 06TCB-Team, 13Patch-For-Review: Turn on CI for the RevisionSlider extension - https://phabricator.wikimedia.org/T133037#2221962 (10Addshore) Many thanks! [09:14:31] 10Continuous-Integration-Config, 10BlueSpice, 13Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#2222020 (10hashar) I have made the BlueSpiceExtensions jobs non voting (verified on https://gerrit.wikimedia.org/r/#/c/284299/ ) until the tests failure are sor... [09:22:05] 10Deployment-Systems, 10RESTBase, 06Services: RESTBase deployment process - https://phabricator.wikimedia.org/T103344#2222097 (10akosiaris) I assume this is now waiting on scap3 migration of restbase. Removing the #blocked-on-operations for now. [09:29:48] (03CR) 10Hashar: [C: 032] [BlueSpiceFoundation] Update generic unittest test to composer unittest test [integration/config] - 10https://gerrit.wikimedia.org/r/282377 (https://phabricator.wikimedia.org/T130811) (owner: 10Paladox) [09:29:53] (03PS4) 10Hashar: [BlueSpiceFoundation] Update generic unittest test to composer unittest test [integration/config] - 10https://gerrit.wikimedia.org/r/282377 (https://phabricator.wikimedia.org/T130811) (owner: 10Paladox) [09:30:05] (03CR) 10Hashar: [C: 032] [BlueSpiceFoundation] Update generic unittest test to composer unittest test [integration/config] - 10https://gerrit.wikimedia.org/r/282377 (https://phabricator.wikimedia.org/T130811) (owner: 10Paladox) [09:30:56] (03Merged) 10jenkins-bot: [BlueSpiceFoundation] Update generic unittest test to composer unittest test [integration/config] - 10https://gerrit.wikimedia.org/r/282377 (https://phabricator.wikimedia.org/T130811) (owner: 10Paladox) [09:35:15] (03CR) 10Hashar: "Tested on https://gerrit.wikimedia.org/r/#/c/284300/ and pass" [integration/config] - 10https://gerrit.wikimedia.org/r/282377 (https://phabricator.wikimedia.org/T130811) (owner: 10Paladox) [09:36:02] 10Continuous-Integration-Config, 10BlueSpice, 13Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#2222192 (10hashar) BlueSpiceFoundation has passing jobs. Tested on https://gerrit.wikimedia.org/r/#/c/284300/ Have to sort out BlueSpiceExtensions now :-} [09:37:15] 10Continuous-Integration-Config, 10BlueSpice, 13Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#2222193 (10Paladox) @hashar thanks. [09:38:25] (03CR) 10Paladox: "@hashar thanks we can now apply this to SemanticMediaWiki which needs composer jobs and for the extensions that require SemanticMediaWiki" [integration/config] - 10https://gerrit.wikimedia.org/r/270737 (owner: 10Paladox) [09:38:32] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #785: 04FAILURE in 17 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/785/ [09:57:42] (03PS1) 10Hashar: [labs/tools/stewardbots] experimental tox job [integration/config] - 10https://gerrit.wikimedia.org/r/284430 (https://phabricator.wikimedia.org/T128503) [09:58:39] (03CR) 10Hashar: [C: 032] [labs/tools/stewardbots] experimental tox job [integration/config] - 10https://gerrit.wikimedia.org/r/284430 (https://phabricator.wikimedia.org/T128503) (owner: 10Hashar) [09:59:33] (03Merged) 10jenkins-bot: [labs/tools/stewardbots] experimental tox job [integration/config] - 10https://gerrit.wikimedia.org/r/284430 (https://phabricator.wikimedia.org/T128503) (owner: 10Hashar) [10:00:04] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 07Documentation: [Story] Automatically create Doxygen documentation of Wikibase PHP code - https://phabricator.wikimedia.org/T66996#2222337 (10Jonas) [10:00:07] 10Continuous-Integration-Config, 10Wikidata, 07Documentation, 03Wikidata-Sprint-2016-04-12: [Story] Wikibase documentation - https://phabricator.wikimedia.org/T133028#2222336 (10Jonas) [10:00:56] 10Deployment-Systems, 10scap, 10Monitoring, 06Operations, 10Scap3 (Scap3-Adoption-Phase1): Deploy servermon with scap3 - https://phabricator.wikimedia.org/T129152#2222351 (10akosiaris) p:05Triage>03Normal [10:01:47] 10Continuous-Integration-Config, 10Tool-Labs-tools-stewardbots, 13Patch-For-Review: Implement jenkins tests on labs/tools/stewardbots - https://phabricator.wikimedia.org/T128503#2222358 (10hashar) a:03MarcoAurelio [10:50:54] (03CR) 10Hashar: "Paladox: please do :-) A single change that update all Semantic extensions will do I guess :-}" [integration/config] - 10https://gerrit.wikimedia.org/r/270737 (owner: 10Paladox) [10:51:27] (03PS5) 10Hashar: Run Popups browser tests per patch [integration/config] - 10https://gerrit.wikimedia.org/r/283568 (https://phabricator.wikimedia.org/T133018) (owner: 10Bmansurov) [10:51:40] (03CR) 10Hashar: [C: 032] "Hopefully they pass :-}" [integration/config] - 10https://gerrit.wikimedia.org/r/283568 (https://phabricator.wikimedia.org/T133018) (owner: 10Bmansurov) [10:52:41] (03Merged) 10jenkins-bot: Run Popups browser tests per patch [integration/config] - 10https://gerrit.wikimedia.org/r/283568 (https://phabricator.wikimedia.org/T133018) (owner: 10Bmansurov) [10:54:41] (03CR) 10Paladox: "@hashar Ok I will do that now." [integration/config] - 10https://gerrit.wikimedia.org/r/270737 (owner: 10Paladox) [10:58:07] (03PS1) 10Hashar: [Popups] depends on BetaFeatures [integration/config] - 10https://gerrit.wikimedia.org/r/284442 (https://phabricator.wikimedia.org/T133018) [10:58:31] (03CR) 10Hashar: [C: 032] [Popups] depends on BetaFeatures [integration/config] - 10https://gerrit.wikimedia.org/r/284442 (https://phabricator.wikimedia.org/T133018) (owner: 10Hashar) [10:59:25] (03Merged) 10jenkins-bot: [Popups] depends on BetaFeatures [integration/config] - 10https://gerrit.wikimedia.org/r/284442 (https://phabricator.wikimedia.org/T133018) (owner: 10Hashar) [11:15:42] 10Beta-Cluster-Infrastructure, 06Commons, 10MediaWiki-File-management, 06Multimedia: Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking) - https://phabricator.wikimedia.org/T84950#2222708 (10akosiaris) [11:15:46] 10Beta-Cluster-Infrastructure, 06Operations, 13Patch-For-Review: /mnt/upload7 does not exist anywhere, yet it is referenced in multiple places in wmf-config - https://phabricator.wikimedia.org/T129586#2222707 (10akosiaris) [11:16:02] 10Beta-Cluster-Infrastructure, 06Operations, 13Patch-For-Review: /mnt/upload7 does not exist anywhere, yet it is referenced in multiple places in wmf-config - https://phabricator.wikimedia.org/T129586#2110079 (10akosiaris) 05Open>03stalled p:05Triage>03Normal [11:57:03] 10Continuous-Integration-Infrastructure, 06Operations: Investigate usage of ttf-ubuntu-font-family which is not available on Jessie - https://phabricator.wikimedia.org/T103325#1387187 (10MoritzMuehlenhoff) According to T32288 it was added for SVG conversion: Various SVG diagrams use fonts from the 'Ubuntu' typ... [11:58:36] Project browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #458: 04FAILURE in 1 min 35 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/458/ [12:54:14] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #828: 04FAILURE in 13 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/828/ [13:06:37] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #888: 04FAILURE in 36 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/888/ [13:18:11] 10Beta-Cluster-Infrastructure, 07Puppet: Puppet failure on deployment-cache-parsoid05 due to removal of role::cache::parsoid - https://phabricator.wikimedia.org/T132260#2223293 (10BBlack) There is no parsoid varnish cache in production anymore, and therefore there shouldn't be one in beta anymore either. Use... [13:19:13] 10Beta-Cluster-Infrastructure, 07Puppet: Puppet failure on deployment-cache-parsoid05 due to removal of role::cache::parsoid - https://phabricator.wikimedia.org/T132260#2223302 (10BBlack) p:05Triage>03Low [13:19:21] 10Beta-Cluster-Infrastructure, 07Puppet: Puppet failure on deployment-cache-parsoid05 due to removal of role::cache::parsoid - https://phabricator.wikimedia.org/T132260#2192879 (10BBlack) a:05BBlack>03None [13:27:14] !log Restarted integration puppet master service (out of memory / mem leak) [13:27:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:30:09] PROBLEM - Puppet run on integration-slave-trusty-1018 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:30:52] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Disable HHVM fcgi server on CI slaves - https://phabricator.wikimedia.org/T126594#2223347 (10hashar) [13:30:54] 10Continuous-Integration-Infrastructure, 06Operations, 07Blocked-on-Operations, 07HHVM: /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so no such file or directory - https://phabricator.wikimedia.org/T126658#2223343 (10hashar) 05Open>03Resolved a:03ori Indeed @ori fixed it. On a CI slav... [13:31:50] PROBLEM - Puppet run on integration-slave-trusty-1023 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [13:33:02] !log salt -v '*trusty*' cmd.run 'rm /usr/lib/x86_64-linux-gnu/hhvm/extensions/current' # Cleanup on CI slaves for T126658 [13:33:03] T126658: /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so no such file or directory - https://phabricator.wikimedia.org/T126658 [13:33:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:39:15] hashar: Maybe you can take a look at beta? the abusefilter is dead [13:39:35] for example go to: http://deployment.wikimedia.beta.wmflabs.org/wiki/Special:AbuseLog [13:39:49] shows only a blank page for me, the rest except abusefilter works [13:39:59] hm, seems like it was only temporary [13:40:10] RECOVERY - Puppet run on integration-slave-trusty-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [13:40:16] strange [13:43:41] 10Beta-Cluster-Infrastructure: Beta-Cluster shows randomly white pages - https://phabricator.wikimedia.org/T133148#2223449 (10Luke081515) [13:43:49] 10Beta-Cluster-Infrastructure: Beta-Cluster shows randomly white pages - https://phabricator.wikimedia.org/T133148#2223461 (10Luke081515) p:05Triage>03High [13:47:14] 10Beta-Cluster-Infrastructure: Beta-Cluster shows randomly white pages - https://phabricator.wikimedia.org/T133148#2223486 (10Luke081515) p:05High>03Unbreak! This happens very often, round about 2 of 3 pages you load are white. You can't do any actions there without realoding the pages very often. [13:54:52] 10Beta-Cluster-Infrastructure: Beta-Cluster shows randomly white pages - https://phabricator.wikimedia.org/T133148#2223521 (10Luke081515) [13:57:07] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Disable HHVM fcgi server on CI slaves - https://phabricator.wikimedia.org/T126594#2223532 (10hashar) 05stalled>03Open HHVM configuration files pointed to `/usr/lib/x86_64-linux-gnu/hhvm/extensions/current` which was created by the HHVM Upstart se... [13:57:09] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: CI trusty slaves running out of memory - https://phabricator.wikimedia.org/T126545#2223536 (10hashar) [13:58:43] !log Stopping HHVM on CI slaves by cherry picking a couple puppet patches | T126594 [13:58:44] T126594: Disable HHVM fcgi server on CI slaves - https://phabricator.wikimedia.org/T126594 [13:58:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:59:58] 10Continuous-Integration-Infrastructure, 06Operations, 10Traffic: Move gallium to an internal host? - https://phabricator.wikimedia.org/T133150#2223557 (10BBlack) [14:02:29] hashar: Hi, thanks for merging my patches today and also i will do SemanticMediaWiki now, i was looking for a way to fix color in grunt. [14:03:27] hashar: Maybe if you got a bit time, you can take a look at T133148? I guess this would fix the two failing scenrios at 14:54 and 15:06 CEST too [14:03:28] T133148: Beta-Cluster shows randomly white pages - https://phabricator.wikimedia.org/T133148 [14:11:41] RECOVERY - Puppet run on integration-slave-trusty-1023 is OK: OK: Less than 1.00% above the threshold [0.0] [14:15:57] 10Continuous-Integration-Infrastructure, 06Operations: Investigate usage of ttf-ubuntu-font-family which is not available on Jessie - https://phabricator.wikimedia.org/T103325#2223600 (10MoritzMuehlenhoff) [14:19:00] 10Beta-Cluster-Infrastructure, 10Monitoring, 13Patch-For-Review, 07Shinken: Monitor keyholder on deployment-prep deployment servers - https://phabricator.wikimedia.org/T111064#2223603 (10Krenair) I think it's working now [14:24:58] James_F|Away: Where did you find .22?: ReleaseTaggerBot added projects: MW-1.27-release-notes, WMF-deploy-2016-04-26_(1.27.0-wmf.22). [14:27:36] 10Beta-Cluster-Infrastructure: Beta-Cluster shows randomly white pages - https://phabricator.wikimedia.org/T133148#2223449 (10Se4598) A quick test by repeating the request suggests that **deployment-mediawiki01.deployment-prep.eqiad.wmflabs** delivers for Special:Radom a blank page and deployment-mediawiki02.dep... [14:54:17] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 40318 bytes in 2.056 second response time [14:55:18] 10Beta-Cluster-Infrastructure: Beta-Cluster shows randomly white pages - https://phabricator.wikimedia.org/T133148#2223711 (10Krenair) a:03Krenair -mediawiki01 and -mediawiki03 broke around 9AM BST [14:55:59] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 40326 bytes in 2.500 second response time [14:56:23] jzerebecki: Hmm? [14:57:39] 10Beta-Cluster-Infrastructure: Beta-Cluster shows randomly white pages - https://phabricator.wikimedia.org/T133148#2223716 (10Krenair) 05Open>03Resolved ```lang=irc RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 40318 bytes in 2.056 second re... [14:57:47] 10Beta-Cluster-Infrastructure: Beta-Cluster shows randomly white pages - https://phabricator.wikimedia.org/T133148#2223718 (10Krenair) (I restarted HHVM) [14:58:17] jzerebecki: Do you mean, how does ReleaseTaggerBot magically know about wmf.22 when it hasn't been cut yet? It adds one to the latest one when things are merged to master. [15:06:38] 10Continuous-Integration-Infrastructure, 06Operations, 10Traffic: Move gallium to an internal host? - https://phabricator.wikimedia.org/T133150#2223557 (10Dzahn) it's also running Jenkins. added Hashar to answer if it need the public IP. also see T95757 [15:31:28] 10Beta-Cluster-Infrastructure, 07Puppet: Puppet failure on deployment-cache-parsoid05 due to removal of role::cache::parsoid - https://phabricator.wikimedia.org/T132260#2223861 (10Krenair) thanks @bblack. I haven't been able to find anything referencing this host (MW is set up to go to parsoid itself) or any n... [15:35:04] 10Beta-Cluster-Infrastructure, 07Puppet: Puppet failure on deployment-cache-parsoid05 due to removal of role::cache::parsoid - https://phabricator.wikimedia.org/T132260#2192879 (10hashar) Parsoid settings for beta are in mediawiki/services/parsoid/deploy and @bblack removed the bit referencing the cache-parsoi... [15:36:02] 10Beta-Cluster-Infrastructure, 07Puppet: Puppet failure on deployment-cache-parsoid05 due to removal of role::cache::parsoid - https://phabricator.wikimedia.org/T132260#2223892 (10Krenair) a:03Krenair It's shut off now, I will delete it on wednesday 27th [15:38:49] PROBLEM - Host deployment-cache-parsoid05 is DOWN: CRITICAL - Host Unreachable (10.68.20.102) [15:43:01] (marked it for downtime in shinken) [15:53:22] Krenair: thanks for all the maintenance work on beta cluster :-} [16:00:24] 07Browser-Tests, 10Reading-Web, 03Reading-Web-Sprint-72-N: Run browser tests on beta cluster from desktop domain - https://phabricator.wikimedia.org/T130429#2223971 (10dr0ptp4kt) [16:03:09] 10Continuous-Integration-Infrastructure, 06Operations, 10Traffic: Move gallium to an internal host? - https://phabricator.wikimedia.org/T133150#2223981 (10hashar) gallium has been setup in 2011 and is still on Precise. It received a public IP to serves the Jenkins web interface. With time, all the http entry... [16:04:01] 10Continuous-Integration-Infrastructure, 06Operations, 10Traffic: Move gallium to an internal host? - https://phabricator.wikimedia.org/T133150#2223557 (10hashar) [16:04:04] 10Continuous-Integration-Infrastructure: Phase out gallium.wikimedia.org - https://phabricator.wikimedia.org/T95757#2223985 (10hashar) [16:22:05] 10Continuous-Integration-Config, 10Wikidata, 07Documentation, 13Patch-For-Review, 03Wikidata-Sprint-2016-04-12: [Story] Wikibase documentation - https://phabricator.wikimedia.org/T133028#2224051 (10Jonas) I added the jsduck.json files to the relevant repos could someone please configure (or tell me how t... [16:27:43] James_F: thx [16:30:36] Krenair: thx for fixing the beta hosts :) [16:59:26] twentyafterfour: It is fully imported now see http://www.test-random-wikisaur.tk/R2:b060a0cb15231fb792a0e87f3c4eda5eaf6b929b please [17:31:19] (03PS1) 10Paladox: [SemanticMediaWiki] Switch unit tests to composer non voting unit tests [integration/config] - 10https://gerrit.wikimedia.org/r/284490 [17:37:50] (03PS1) 10Paladox: [SolrStore] Switch unit tests to composer non-voting unit tests [integration/config] - 10https://gerrit.wikimedia.org/r/284492 [17:44:17] hey hashar [17:44:39] (03PS1) 10Phuedx: [Popups] Depend on TextExtracts and PageImages [integration/config] - 10https://gerrit.wikimedia.org/r/284494 [17:44:58] looks like phuedx is already on it :) Is anyone here able to merge the above? [17:45:22] thcipriani: twentyafterfour mind deploying https://gerrit.wikimedia.org/r/#/c/284494/ for mobile folks? :D [17:45:26] https://gerrit.wikimedia.org/r/#/c/284494/1/zuul/parameter_functions.py,cm [17:45:37] it is injecting a couple dependencies to the CI job for the Popups extension [17:45:46] hashar: I have https://gerrit.wikimedia.org/r/284490 for SemanticMediaWiki [17:45:50] (yeah that dependency system is terrible) [17:46:16] It tests now but fails with api doc stuff [17:46:52] hashar: I could take a crack at it, in SoS at the moment. [17:48:12] thcipriani: once merged `fab deploy_zuul` would do the deploy [17:48:28] it basically ssh to gallium , sudo as zuul and then git pull in /etc/zuul/wikimedia then reload [17:49:20] okie doke. [17:49:54] (03CR) 10Thcipriani: [C: 032] [Popups] Depend on TextExtracts and PageImages [integration/config] - 10https://gerrit.wikimedia.org/r/284494 (owner: 10Phuedx) [17:50:51] (03Merged) 10jenkins-bot: [Popups] Depend on TextExtracts and PageImages [integration/config] - 10https://gerrit.wikimedia.org/r/284494 (owner: 10Phuedx) [17:54:14] !log Reloading Zuul to deploy [[gerrit:284494]] [17:54:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:54:30] !!!!!!!!!!!! [17:54:40] done! [17:54:49] <3 thcipriani hashar [18:15:04] anybody here familiar with phabricator/arcanist? [18:35:21] hashar: Looks like edge crashing because it is loading diffs such as those in gerrit will be fixed soon [18:35:23] https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/7134764/ [18:36:09] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 2.57 ms [18:46:10] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [19:08:05] 06Release-Engineering-Team, 15User-greg: Determine parental leave and delegation plan - https://phabricator.wikimedia.org/T131198#2224910 (10greg) p:05Normal>03Unbreak! [19:30:13] 05Continuous-Integration-Scaling, 07Upstream: Nodepool instances have duplicate entry for jessie-backports/main - https://phabricator.wikimedia.org/T120963#2224962 (10hashar) https://review.openstack.org/#/c/279821/ got merged and is included in diskimage-builder as of `1.12.0`. Have to rebuild an image with... [19:34:57] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 03releng-201516-q4, 07WorkType-NewFunctionality: [keyresult] Migrate php composer (Zend and HHVM) CI jobs to Nodepool - https://phabricator.wikimedia.org/T119139#2224969 (10hashar) [19:34:59] 05Continuous-Integration-Scaling, 06Operations, 07HHVM: Provide a HHVM package for jessie-wikimedia matching version of trusty-wikimedia - https://phabricator.wikimedia.org/T125821#2224967 (10hashar) 05Open>03stalled Whoa thank you @Joe ! I am not sure what ICU involves when it comes to testing but it se... [19:35:04] 05Continuous-Integration-Scaling, 06Operations, 07HHVM: Provide a HHVM package for jessie-wikimedia matching version of trusty-wikimedia - https://phabricator.wikimedia.org/T125821#2224970 (10hashar) [19:37:16] 05Continuous-Integration-Scaling, 03releng-201516-q4, 07WorkType-NewFunctionality: Attempt to provide a Trusty image for Nodepool - https://phabricator.wikimedia.org/T133203#2224981 (10hashar) [19:49:08] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: integration-dev instance changed of IP address - https://phabricator.wikimedia.org/T133207#2225061 (10hashar) [20:04:29] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: integration-dev instance changed of IP address - https://phabricator.wikimedia.org/T133207#2225160 (10hashar) DNS: ``` $ dig +short A integration-dev.integration.eqiad.wmflabs $ dig +short -x 10.68.23.123 $ dig +short -x 10.68.17.81 $ ``... [20:09:22] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 03Scap3, 06Operations: Automate the generation deployment keys (keyholder-managed ssh keys) - https://phabricator.wikimedia.org/T133211#2225168 (10mmodell) [20:12:22] Is zuul taking a nap, or am I just impatient? -- https://gerrit.wikimedia.org/r/#/c/283772/ [20:13:32] bd808: looks like it has a dependency that hasn't merged. [20:13:39] I jsut noticed that [20:13:41] thanks [20:14:01] yup :) [21:52:01] !log testing new scap version in beta on deployment-tin [21:52:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:57:59] Project beta-scap-eqiad build #99161: 04FAILURE in 3 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99161/ [21:58:24] welp. shoot. [22:05:57] Yippee, build fixed! [22:05:58] Project beta-scap-eqiad build #99162: 09FIXED in 1 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99162/ [22:09:05] thcipriani: Reverted back, or does the new version worked at the second time? [22:10:15] Luke081515: reverted on tin for the moment. I realized I have to roll forward all the targets as well in this instance. [22:10:55] I will roll forward again momentarily. [22:14:41] ah, ok :) [22:25:57] Project beta-scap-eqiad build #99164: 04FAILURE in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99164/ [22:28:12] !log rolling back scap version in beta, legit failure :( [22:28:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:28:27] * bd808 didn't do ti [22:29:22] * thcipriani did it :) [22:35:55] Yippee, build fixed! [22:35:55] Project beta-scap-eqiad build #99165: 09FIXED in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/99165/ [22:39:20] thcipriani: do you know the preferred way for a new extension to run unit tests? SHould `composer test` handle that these days? [22:39:54] mwext-testextension-* only tests a whitelisted collection of extenstions [22:40:13] btw, there is a typo at the "welcome message" at mira [22:40:15] it says: [22:40:26] if [22:40:26] will probably work, but the next deployer could lose track of any of [22:40:28] your canges. [22:40:28] bd808: hmm, I don't know the answer to that without some spelunking into jjb [22:40:48] thcipriani: :) np. I'll find somebody who knows [22:41:16] Luke081515: https://github.com/wikimedia/operations-puppet/blob/3218df65dcc4c9d42ce6deef0e130db817613f58/modules/role/files/deployment/inactive.motd [22:41:22] give us a patch! :) [22:41:38] bd808: I guess a patch for the normal puppet repo? :) [22:41:46] yeah. ops/puppet [22:41:59] ok [22:42:28] bd808: is there a simple way tp create this big letters, or is someone just experimenting with pipes till that looks good? :D [22:43:10] FIGlet [22:43:24] http://www.figlet.org/ [22:43:29] thx [22:57:50] bd808: Patch is available: https://gerrit.wikimedia.org/r/#/c/284605/ [22:57:51] :) [22:58:46] Luke081515: awesome. I'll put some ops folks on it to get it merged [22:58:57] thx [22:59:06] my first patch for this repo :) [23:29:54] PROBLEM - Host integration-trusty-1026 is DOWN: CRITICAL - Host Unreachable (10.68.17.98)