[00:30:49] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.62 ms [00:38:41] brion: I thought of you when writing this up: https://phabricator.wikimedia.org/D84 <3 [00:43:55] hehe [00:44:07] * brion doesn't miss running the cluster himself [00:44:12] of course back then it was like 5 machines [00:45:09] I feel like my career history here has been "do the things that brion used to do" :p [00:45:19] :D [00:49:03] brion: 5? 5?! I remember when it was one! ;-) [00:51:08] James_F: do you remember en2.wikipedia.org? [00:51:23] I remember it in apache templates [00:51:32] that was our first load-balancing attempt before we figured out how to use proxies [01:28:20] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [01:35:19] (03CR) 10Legoktm: [C: 032] Added CodeEditor as dependency to Graph [integration/config] - 10https://gerrit.wikimedia.org/r/262363 (owner: 10Yurik) [01:39:34] (03Merged) 10jenkins-bot: Added CodeEditor as dependency to Graph [integration/config] - 10https://gerrit.wikimedia.org/r/262363 (owner: 10Yurik) [01:40:50] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 1.13 ms [01:45:16] (03PS1) 10Legoktm: Also add WikiEditor as a dependency for Graph [integration/config] - 10https://gerrit.wikimedia.org/r/262419 [01:46:46] (03CR) 10Legoktm: [C: 032] "Already deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/262419 (owner: 10Legoktm) [01:48:47] (03Merged) 10jenkins-bot: Also add WikiEditor as a dependency for Graph [integration/config] - 10https://gerrit.wikimedia.org/r/262419 (owner: 10Legoktm) [03:07:03] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:11:54] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 39349 bytes in 0.877 second response time [03:29:20] Yippee, build fixed! [03:29:21] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #933: 09FIXED in 47 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/933/ [03:45:06] !log integration-slave-trusty-1015: rm -rf /mnt/home/jenkins-deploy/.npm per https://integration.wikimedia.org/ci/job/mediawiki-core-qunit/56577/console [03:45:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [03:51:37] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [03:55:51] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.71 ms [03:58:51] 10Deployment-Systems, 6Release-Engineering-Team, 6Performance-Team, 6operations, 7HHVM: Translation cache exhaustion caused by changes to PHP code in file scope - https://phabricator.wikimedia.org/T103886#1915627 (10Krinkle) [03:58:55] 10Deployment-Systems, 10MediaWiki-Internationalization, 6Performance-Team, 7Performance: Experiment with plain .php files for l10n cache instead of CDB - https://phabricator.wikimedia.org/T99740#1915625 (10Krinkle) [04:51:32] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [05:15:00] lol @ D85 [05:28:35] 6Release-Engineering-Team, 10DBA, 7user-notice: Requests to globally reset a user's skin preferences - https://phabricator.wikimedia.org/T119206#1915691 (10demon) >>! In T119206#1873407, @demon wrote: >>>! In T119206#1870832, @NickK wrote: >>>>! In T119206#1838886, @demon wrote: >>> No that isn't possible I... [05:28:42] 6Release-Engineering-Team, 10DBA, 7user-notice: Requests to globally reset a user's skin preferences - https://phabricator.wikimedia.org/T119206#1915692 (10demon) 5Open>3Resolved [06:14:30] 10Continuous-Integration-Config, 7Regression: Graph extension is not running structure tests - https://phabricator.wikimedia.org/T122872#1915716 (10Krinkle) 3NEW [08:32:58] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #835: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/835/ [09:10:51] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 1.11 ms [09:41:45] 10Continuous-Integration-Config, 10Graph, 7Regression: Graph extension is not running structure tests - https://phabricator.wikimedia.org/T122872#1915816 (10hashar) Seems the issue has been introduced by https://gerrit.wikimedia.org/r/#/c/262344/ which moved three files from the root under `/includes`. The... [09:56:33] 10Continuous-Integration-Infrastructure: Merge tests even though a repo is being tested before it - https://phabricator.wikimedia.org/T122602#1915825 (10Paladox) [09:56:58] (03PS1) 10Hashar: [Graph] add VisualEditor as a dependency [integration/config] - 10https://gerrit.wikimedia.org/r/262427 (https://phabricator.wikimedia.org/T122872) [09:58:26] 10Continuous-Integration-Config, 10Graph, 5Patch-For-Review, 7Regression: Graph extension is not running structure tests - https://phabricator.wikimedia.org/T122872#1915830 (10hashar) https://gerrit.wikimedia.org/r/262427 adds VisualEditor as a dependency. I gave it a try on the broken Graph change but the... [09:59:04] 10Continuous-Integration-Infrastructure: Merge tests even though a repo is being tested before it - https://phabricator.wikimedia.org/T122602#1915831 (10Paladox) It may reduce high loads if we allow patches to be merged without waiting for the one in front to be merged. when lots of patches are being merged sinc... [10:04:11] 10Continuous-Integration-Infrastructure, 6operations: Test mwext-qunit-composer database disk image is malformed - https://phabricator.wikimedia.org/T122599#1915834 (10Paladox) 5Open>3declined a:3Paladox Declining since the problem seems to not happen any more. [10:04:16] (03CR) 10Hashar: [C: 032] [Graph] add VisualEditor as a dependency [integration/config] - 10https://gerrit.wikimedia.org/r/262427 (https://phabricator.wikimedia.org/T122872) (owner: 10Hashar) [10:06:23] (03Merged) 10jenkins-bot: [Graph] add VisualEditor as a dependency [integration/config] - 10https://gerrit.wikimedia.org/r/262427 (https://phabricator.wikimedia.org/T122872) (owner: 10Hashar) [10:11:04] 10MediaWiki-Releasing, 6Release-Engineering-Team, 5Patch-For-Review: Resolve/remove symlink of README.mediawiki in tarball - https://phabricator.wikimedia.org/T122502#1915841 (10hashar) @demon I guess we will want to add an entry to release notes for the future releases. [10:23:20] 10Continuous-Integration-Config, 10Graph, 5Patch-For-Review, 7Regression: Graph extension is not running structure tests - https://phabricator.wikimedia.org/T122872#1915852 (10hashar) 5Open>3Resolved a:3hashar The dummy test change failed properly. Issue solved by injecting VisualEditor as a depende... [10:25:07] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:29:59] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 39317 bytes in 0.587 second response time [10:35:07] 10Continuous-Integration-Infrastructure: Move commit-message-validator.py tool out of integration/jenkins repository - https://phabricator.wikimedia.org/T121609#1915883 (10Paladox) We can request a repo be created and have those two users as authors so they can have +2 rights in the new repo. What should we name... [10:39:32] 10Deployment-Systems, 6Release-Engineering-Team: scap-purge-l10n-cache hanging - https://phabricator.wikimedia.org/T122008#1915884 (10hashar) a:5hashar>3None [10:46:05] (03PS1) 10Paladox: Fix dirty VisualEditor submodule [integration/config] - 10https://gerrit.wikimedia.org/r/262432 (https://phabricator.wikimedia.org/T121479) [10:51:35] (03PS2) 10Paladox: Fix dirty VisualEditor submodule [integration/config] - 10https://gerrit.wikimedia.org/r/262432 (https://phabricator.wikimedia.org/T121479) [10:56:44] hashar: Could you review https://gerrit.wikimedia.org/r/#/c/262432/ please. [10:56:59] paladox: sure [10:57:08] hashar: Thanks. [10:57:19] oh [10:58:22] paladox: funny that is what the ve-submodules-update macro is doing [10:58:33] looks like there is some code duplication :D [10:59:06] hashar: Yes. I made sure that before the update that it cleans the folder and then after the update. SO that way less likly it will cause it to fail. [10:59:16] (03CR) 10Hashar: "Ah that is what the JJB ve-submodules-update macro is doing. Probably want to refactor the code a bit to reuse the macro." [integration/config] - 10https://gerrit.wikimedia.org/r/262432 (https://phabricator.wikimedia.org/T121479) (owner: 10Paladox) [10:59:42] paladox: will look at it later on [10:59:57] those two usages should reuse the ve-submodules-update macro instead [11:01:08] hashar: Ok. How would i reuse it meaning how would the code look like if i use ve-submodules-update in those two builders. [11:01:51] paladox: instead of using an inline shell command [11:01:55] just invoke the macro: [11:01:57] - ve-submodules-update [11:02:01] but it slightly different [11:02:12] the macro looks for VE in /deps.txt [11:02:27] while the inlined shells scripts you edited look for it in src/extensions_load.txt [11:02:34] I think we generate /deps.txt first [11:02:40] then copy it to src/extensions_load.txt [11:03:10] that is a huge mess to be honest :( [11:03:48] so it would look like this - shell: - ve-submodules-update [11:04:31] hashar: [11:09:33] hashar: Or is it builders: - ve-submodules-update then we do shell. [11:13:57] paladox: - shell is inlined shell [11:14:07] "shell" is a builder [11:14:20] ve-submodules-update is a macro builder as well that invokes a "shell" builder [11:14:22] so you can just: [11:14:26] - ve-submodules-update [11:14:34] that will be expended by JJB as something like: [11:14:37] - shell : | [11:14:39] whatever [11:15:06] hashar: Ok but we woulden be able to use the deps.txt file since it is used by $ZUUL_PROJECT [11:15:25] So we woulden be able to copy it. [11:15:28] yup [11:15:32] that is what needs to be figured out [11:16:00] maybe the macro can be made to look in extension_load.txt instead of deps.txt [11:16:44] Ok. Maybe if we do - ve-submodules-update before anything else then copy it to the other file. [11:17:13] (03Abandoned) 10Hashar: Add experimental tests to npm test [integration/config] - 10https://gerrit.wikimedia.org/r/261094 (owner: 10Paladox) [11:21:14] hashar: But also the shell in - ve-submodules-update looks the wrong file in the wrong folder. We might as well just use the dupicated code until we figure out how we can reuse macro. [11:31:55] (03PS2) 10Hashar: [timeline] Add experimental tests to jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/261093 (owner: 10Paladox) [11:32:10] (03CR) 10Paladox: "I'm not sure how to reuse macro since visualeditor is put into the deps folder in macro and the extension_loads file in mediawiki." [integration/config] - 10https://gerrit.wikimedia.org/r/262432 (https://phabricator.wikimedia.org/T121479) (owner: 10Paladox) [11:34:09] (03CR) 10Paladox: "Even though it is duplicated code this fixed the bug as linked and we can always at a later point refractor and reuse macro." [integration/config] - 10https://gerrit.wikimedia.org/r/262432 (https://phabricator.wikimedia.org/T121479) (owner: 10Paladox) [11:46:13] 6Release-Engineering-Team, 10DBA, 7user-notice: Requests to globally reset a user's skin preferences - https://phabricator.wikimedia.org/T119206#1915916 (10Nemo_bis) 5Resolved>3Open I still need my preference to be changed back too. Was this request page ever announced to people? [11:54:29] 10Deployment-Systems, 10MediaWiki-Internationalization, 6Performance-Team, 7Performance: Experiment with plain .php files for l10n cache instead of CDB - https://phabricator.wikimedia.org/T99740#1915928 (10Nemo_bis) [11:56:55] (03CR) 10Hashar: [C: 032] [timeline] Add experimental tests to jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/261093 (owner: 10Paladox) [11:58:05] (03Merged) 10jenkins-bot: [timeline] Add experimental tests to jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/261093 (owner: 10Paladox) [12:16:54] (03Abandoned) 10Hashar: Remove VisualEditor from extension-gate [integration/config] - 10https://gerrit.wikimedia.org/r/259959 (https://phabricator.wikimedia.org/T116258) (owner: 10Legoktm) [12:18:56] (03PS3) 10Hashar: Assert gate jobs are properly configured [integration/config] - 10https://gerrit.wikimedia.org/r/259711 [12:29:40] 10Continuous-Integration-Config: Jenkins MediaWiki installer should use --scriptpath - https://phabricator.wikimedia.org/T116608#1915966 (10hashar) p:5Triage>3Low [12:29:55] 10Continuous-Integration-Config: Rewrite mediawiki-core-doxygen-publish Jenkins job to poll scm instead of being triggered by Zuul - https://phabricator.wikimedia.org/T115755#1915967 (10hashar) p:5Triage>3Normal [12:30:26] 10Continuous-Integration-Config: Convert all repos to use npm instead of jslint which is deprecated - https://phabricator.wikimedia.org/T119973#1915969 (10hashar) [12:30:34] 10Continuous-Integration-Config: Convert all repos to use npm instead of jslint which is deprecated - https://phabricator.wikimedia.org/T119973#1915972 (10hashar) p:5Triage>3Normal [12:32:55] 6Release-Engineering-Team, 10DBA, 7user-notice: Requests to globally reset a user's skin preferences - https://phabricator.wikimedia.org/T119206#1915976 (10Edgars2007) >>! In T119206#1915916, @Nemo_bis wrote: > Was this request page ever announced to people? Yes, it was at [[https://meta.wikimedia.org/wiki/... [12:35:00] (03PS1) 10Hashar: Remove pywikibot-core-tox-flake8-docstrings [integration/config] - 10https://gerrit.wikimedia.org/r/262440 (https://phabricator.wikimedia.org/T119910) [12:36:05] (03CR) 10Hashar: [C: 032] Remove pywikibot-core-tox-flake8-docstrings [integration/config] - 10https://gerrit.wikimedia.org/r/262440 (https://phabricator.wikimedia.org/T119910) (owner: 10Hashar) [12:36:12] 10Continuous-Integration-Config, 10pywikibot-core, 5Patch-For-Review: jenkins task pywikibot-core-tox-flake8-docstrings no longer used - https://phabricator.wikimedia.org/T119910#1915985 (10hashar) 5Open>3Resolved a:3hashar It is no more triggered and I have deleted the job in Jenkins. [12:37:49] (03Merged) 10jenkins-bot: Remove pywikibot-core-tox-flake8-docstrings [integration/config] - 10https://gerrit.wikimedia.org/r/262440 (https://phabricator.wikimedia.org/T119910) (owner: 10Hashar) [12:40:25] 10Continuous-Integration-Config, 10Analytics-EventLogging: EventLogging CI jobs hit meta.wikimedia.org - https://phabricator.wikimedia.org/T122463#1915989 (10hashar) a:5hashar>3None [12:41:43] 10Continuous-Integration-Config: CI generate docs for Flow and MobileFrontend wmf branches - https://phabricator.wikimedia.org/T118599#1915990 (10hashar) p:5Triage>3Low [12:48:01] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:52:53] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 39344 bytes in 0.583 second response time [13:00:50] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [13:02:04] (03PS1) 10Hashar: Drop /logs/ dir [integration/docroot] - 10https://gerrit.wikimedia.org/r/262441 [13:02:13] (03CR) 10Hashar: [C: 032] Drop /logs/ dir [integration/docroot] - 10https://gerrit.wikimedia.org/r/262441 (owner: 10Hashar) [13:02:42] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #900: 04FAILURE in 30 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/900/ [13:02:43] (03Merged) 10jenkins-bot: Drop /logs/ dir [integration/docroot] - 10https://gerrit.wikimedia.org/r/262441 (owner: 10Hashar) [13:17:25] 10Continuous-Integration-Infrastructure, 7Documentation, 7Zuul: Zuul can't merge two patches at once on integration/config.git because of fast-forward only Gerrit strategy - https://phabricator.wikimedia.org/T111426#1916026 (10hashar) 5Open>3declined a:3hashar That is surely annoying, happens infrequen... [13:18:14] 10Continuous-Integration-Infrastructure, 6Release-Engineering-Team: Upgrade PHP_CodeCoverage - https://phabricator.wikimedia.org/T122530#1916029 (10hashar) p:5Triage>3Low [13:21:14] 10Continuous-Integration-Infrastructure: All new extensions should be setup automatically with basic test support - https://phabricator.wikimedia.org/T92909#1916038 (10hashar) 5Open>3declined a:3hashar For MediaWiki extensions some might just want to use composer to bring in dependencies whereas others are... [13:21:55] 10Continuous-Integration-Infrastructure: Move commit-message-validator.py tool out of integration/jenkins repository - https://phabricator.wikimedia.org/T121609#1916041 (10hashar) p:5Triage>3Normal [13:45:14] (03PS4) 10Hashar: Assert gate jobs are properly configured [integration/config] - 10https://gerrit.wikimedia.org/r/259711 [13:45:24] (03PS5) 10Hashar: Assert gate jobs are properly configured [integration/config] - 10https://gerrit.wikimedia.org/r/259711 [13:45:54] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.86 ms [13:51:09] (03CR) 10Hashar: [C: 032] Assert gate jobs are properly configured [integration/config] - 10https://gerrit.wikimedia.org/r/259711 (owner: 10Hashar) [13:51:24] (03CR) 10Hashar: "It just does a set comparaison now :-)" [integration/config] - 10https://gerrit.wikimedia.org/r/259711 (owner: 10Hashar) [13:52:14] (03Merged) 10jenkins-bot: Assert gate jobs are properly configured [integration/config] - 10https://gerrit.wikimedia.org/r/259711 (owner: 10Hashar) [14:12:36] (03Abandoned) 10Hashar: Rerun failed browsertests once [integration/config] - 10https://gerrit.wikimedia.org/r/230715 (https://phabricator.wikimedia.org/T67773) (owner: 10Dduvall) [14:14:26] (03Abandoned) 10Hashar: tests: factor out code to find defined project [integration/config] - 10https://gerrit.wikimedia.org/r/173830 (owner: 10Hashar) [14:15:04] (03Abandoned) 10Hashar: Generic test runner entry point [integration/config] - 10https://gerrit.wikimedia.org/r/242177 (https://phabricator.wikimedia.org/T111181) (owner: 10Hashar) [14:17:22] (03Abandoned) 10Hashar: [LightweightRDFa] Add dependance on WikiEditor [integration/config] - 10https://gerrit.wikimedia.org/r/239189 (https://phabricator.wikimedia.org/T93727) (owner: 10Paladox) [14:18:43] (03PS4) 10Hashar: Update mw-checks and mw-checks-test templates [integration/config] - 10https://gerrit.wikimedia.org/r/260743 (owner: 10Paladox) [14:19:09] (03CR) 10Hashar: [C: 032] Update mw-checks and mw-checks-test templates [integration/config] - 10https://gerrit.wikimedia.org/r/260743 (owner: 10Paladox) [14:20:30] (03Abandoned) 10Hashar: [ConventionExtension] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/253287 (owner: 10Paladox) [14:22:26] (03Abandoned) 10Hashar: [WIP] Add generic 'test' job to run npm and composer [integration/config] - 10https://gerrit.wikimedia.org/r/220689 (owner: 10Legoktm) [14:23:15] (03Merged) 10jenkins-bot: Update mw-checks and mw-checks-test templates [integration/config] - 10https://gerrit.wikimedia.org/r/260743 (owner: 10Paladox) [14:24:45] (03CR) 10Hashar: [C: 04-1] "Apparently this is now just about using composer / extension-unittests-composer" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/191888 (owner: 10Werdna) [14:26:26] (03Abandoned) 10Hashar: Add OOUIPlayground extension to Jenkins config [integration/config] - 10https://gerrit.wikimedia.org/r/191888 (owner: 10Werdna) [14:26:32] (03PS1) 10Hashar: [OOUIPlayground] mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/262446 [14:26:48] 10Continuous-Integration-Infrastructure: Move commit-message-validator.py tool out of integration/jenkins repository - https://phabricator.wikimedia.org/T121609#1916116 (10jayvdb) Currently it only has a __dependency__ on Git, but it requires a Gerrit `Change-Id` line and implements https://www.mediawiki.org/wik... [14:26:49] (03CR) 10Hashar: [C: 032] [OOUIPlayground] mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/262446 (owner: 10Hashar) [14:27:56] (03Merged) 10jenkins-bot: [OOUIPlayground] mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/262446 (owner: 10Hashar) [14:28:57] (03PS3) 10Hashar: Add test to verify every extension+skin has an entry in zuul/layout.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/198185 (owner: 10Legoktm) [14:29:48] (03Abandoned) 10Hashar: [OOUIPlayground] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/248899 (owner: 10Paladox) [14:30:06] (03CR) 10jenkins-bot: [V: 04-1] Add test to verify every extension+skin has an entry in zuul/layout.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/198185 (owner: 10Legoktm) [14:36:45] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:37:53] (03PS6) 10Hashar: (WIP) test Zuul gate configuration (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/173831 [14:38:15] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:38:47] (03CR) 10jenkins-bot: [V: 04-1] (WIP) test Zuul gate configuration (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/173831 (owner: 10Hashar) [14:40:09] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #375: 04FAILURE in 12 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/375/ [14:41:36] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 39658 bytes in 0.808 second response time [14:42:46] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:47:38] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 31492 bytes in 0.531 second response time [14:48:04] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 39325 bytes in 0.520 second response time [14:56:21] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [15:05:51] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 1.03 ms [15:18:13] (03PS1) 10Hashar: [Kartographer] Add PHPUnit tests and npm [integration/config] - 10https://gerrit.wikimedia.org/r/262451 [15:21:28] (03CR) 10Hashar: [C: 032] [Kartographer] Add PHPUnit tests and npm [integration/config] - 10https://gerrit.wikimedia.org/r/262451 (owner: 10Hashar) [15:22:22] (03Merged) 10jenkins-bot: [Kartographer] Add PHPUnit tests and npm [integration/config] - 10https://gerrit.wikimedia.org/r/262451 (owner: 10Hashar) [16:05:53] (03PS1) 10Hashar: [ConventionExtension] mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/262458 [16:06:05] (03CR) 10Hashar: [C: 032] [ConventionExtension] mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/262458 (owner: 10Hashar) [16:07:18] (03Merged) 10jenkins-bot: [ConventionExtension] mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/262458 (owner: 10Hashar) [16:37:00] 10Continuous-Integration-Infrastructure: All new extensions should be setup automatically with basic test support - https://phabricator.wikimedia.org/T92909#1916234 (10Jdlrobson) I still think a lot of this should be automated. for instance if I have package.json in my repository why can't zuul see that and run... [16:41:33] (03PS4) 10Hashar: [ZeroPortal] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/256276 (owner: 10Paladox) [16:43:20] (03CR) 10Hashar: [C: 032] [ZeroPortal] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/256276 (owner: 10Paladox) [16:44:56] (03Merged) 10jenkins-bot: [ZeroPortal] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/256276 (owner: 10Paladox) [16:45:59] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [16:50:52] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.78 ms [16:53:44] 10Continuous-Integration-Infrastructure, 10CirrusSearch, 6Discovery, 5Patch-For-Review: ElasticSearch taking 17% of RAM on integration slaves - https://phabricator.wikimedia.org/T89083#1916247 (10hashar) 5Open>3Resolved a:3hashar I have cherry picked the puppet patch https://gerrit.wikimedia.org/r/#/... [16:54:00] !log Removed elastic search from CI slaves https://phabricator.wikimedia.org/T89083 https://gerrit.wikimedia.org/r/#/c/259301/ [16:54:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:16:18] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-ContentTranslation, 7WorkType-Maintenance: ContentTranslation phpunit run very slow due to inclusion of Scribunto and Wikibase - https://phabricator.wikimedia.org/T121595#1916272 (10hashar) We do inject extensions as dependencies, and they all e... [17:22:56] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/256276 (owner: 10Paladox) [17:44:57] (03PS2) 10Hashar: Dependencies install notes for Mac/Homebrew [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/253598 [18:20:59] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [18:40:51] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.80 ms [18:50:15] "Reflection is bad, don't use it" -- Krinkle [19:31:37] ostriches: Is there a task or label for blocking deployment-next [19:32:58] I don't think so. twentyafterfour greg-g? [19:35:55] Krinkle: Mark it as UBN and throw the usual suspects for deployment on it [19:44:19] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [19:55:23] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.61 ms [19:59:14] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:04:04] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 39346 bytes in 0.670 second response time [20:06:27] 3Scap3: Allow scap3 to read target host list from stdin - https://phabricator.wikimedia.org/T122913#1916643 (10ArielGlenn) 3NEW [20:08:39] 3Scap3: scap3 host restart batching should allow for delay between batches - https://phabricator.wikimedia.org/T122914#1916651 (10ArielGlenn) 3NEW [20:09:34] 10Deployment-Systems, 3Scap3: Allow scap3 to read target host list from stdin - https://phabricator.wikimedia.org/T122913#1916659 (10thcipriani) [20:10:16] 10Deployment-Systems, 3Scap3: scap3 host restart batching should allow for delay between batches - https://phabricator.wikimedia.org/T122914#1916661 (10greg) [20:10:53] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [20:11:10] 10Deployment-Systems, 3Scap3: Bug in scap3 git submodule url rewriting - https://phabricator.wikimedia.org/T121884#1916665 (10greg) [20:11:13] 10Deployment-Systems, 3Scap3, 5Patch-For-Review, 7Puppet: Move scap.cfg things out of scap and into puppet - https://phabricator.wikimedia.org/T121435#1916666 (10greg) [20:11:15] 10Deployment-Systems, 3Scap3: Bring co-master / fanout capabilities to `deploy` and friends - https://phabricator.wikimedia.org/T121276#1916667 (10greg) [20:11:19] 10Deployment-Systems, 3Scap3: Build a dependency graph resolver for deployment stages and tasks - https://phabricator.wikimedia.org/T120684#1916674 (10greg) [20:11:21] 10Deployment-Systems, 3Scap3: create a scap3 command to bootstrap a new deployment repo - https://phabricator.wikimedia.org/T118760#1916676 (10greg) [20:11:23] 10Deployment-Systems, 6Release-Engineering-Team, 3Scap3, 7Security-General: Scap should apply security patches - https://phabricator.wikimedia.org/T118478#1916677 (10greg) [20:11:26] 10Deployment-Systems, 3Scap3: Remove apache dependency from scap3 deployment host - https://phabricator.wikimedia.org/T116630#1916680 (10greg) [20:11:28] 10Deployment-Systems, 6Release-Engineering-Team, 3Scap3, 7Security-General: Scap should be aware of security patches - https://phabricator.wikimedia.org/T118477#1916681 (10greg) [20:11:31] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 3Scap3, 6Analytics-Backlog, and 3 others: Set up AQS in Beta - https://phabricator.wikimedia.org/T116206#1916682 (10greg) [20:11:34] 10Deployment-Systems, 3Scap3, 3releng-201516-q2, 3releng-201516-q3: [keyresult] Migrate all Service team owned services and MW deploys to scap3 - https://phabricator.wikimedia.org/T109926#1916684 (10greg) [20:11:36] 10Deployment-Systems, 6Release-Engineering-Team, 3Scap3, 10Wikimedia-Developer-Summit-2016: Scap3: updates, upgrades, and challenges - https://phabricator.wikimedia.org/T114045#1916683 (10greg) [20:11:38] 10Deployment-Systems, 6Release-Engineering-Team, 3Scap3: Instrument scap for "scap duration" KPI - https://phabricator.wikimedia.org/T108743#1916685 (10greg) [20:19:17] (03PS3) 10Physikerwelt: Add dependency from Math tests to Wikidata [integration/config] - 10https://gerrit.wikimedia.org/r/262374 (https://phabricator.wikimedia.org/T122656) [20:26:09] there is a flag feature in scap now for blocking deployments, but nothing specific in phabricator. It might be nice to have a way to do it from phabricator UI - we did something like that at deviantArt, with a custom phabricator plugin to surface deployment flags in the phabricator top-bar [20:26:39] ugh double-post. fail. [20:27:11] nvm [20:28:11] double post? I don't see it [20:28:38] but yeah, we used to have a tracking bug in BZ "next deployment" or somesuch, that had bugs that were blockers of the next deploy [20:29:27] We should definitely have something similar, but I'd love for the solution to come from mukunda/chad/tyler/dan/someone (iow: not me) [20:29:57] greg-g: that's how I've been doing phabricator deployments (with a tracking bug and blockers.) that works well enough but if we had something to surface any un-break-now issues in the global phab UI it'd be more difficult to miss it [20:30:51] RECOVERY - Host integration-t102459 is UP: PING OK - Packet loss = 0%, RTA = 0.74 ms [20:32:29] unfortunately the "all UBN!" tasks query is too busy: https://phabricator.wikimedia.org/maniphest/query/E3qGuXy8ug.S/#R [20:32:55] 10Deployment-Systems, 3Scap3: Remove apache dependency from scap3 deployment host - https://phabricator.wikimedia.org/T116630#1916734 (10mmodell) 5declined>3Open Ok after discussing this with the team, it's still something we want to support, it just won't be the default mode of operation. I didn't really... [20:33:23] 10Deployment-Systems, 3Scap3: Remove apache dependency from scap3 deployment host - https://phabricator.wikimedia.org/T116630#1916740 (10mmodell) p:5High>3Normal [20:35:25] man, you can't bulk add a flag :/ [20:38:32] anywho, I think we should probably look through https://phabricator.wikimedia.org/maniphest/query/q_UpV85fcnnE/#R manually every monday [20:40:59] s/manually/and have it automatized asap/ [20:41:19] greg-g: if it is manual it is prone to failure [20:41:26] PROBLEM - Host integration-t102459 is DOWN: CRITICAL - Host Unreachable (10.68.16.67) [20:42:11] hashar: totally, i think that's what twentyafterfour was thinking [20:42:32] hashar: also, hello! [20:42:39] yeah yeah good morning :-} [20:42:40] we all miss you [20:42:45] how is the dev summit going on ? [20:42:54] ahah [20:42:57] the scap3 and differential discussions went really well! [20:43:20] got a ton of folks poking me wondering where I am because they wanted to chat / talk about such tech topic / have a beer / hangout in the city :—) [20:43:37] yeah, you're popular ;) [20:44:28] among a cluster of specific folks, maybe [20:45:18] that's fandom [20:45:31] https://en.wikipedia.org/wiki/Fandom [20:46:16] looks like :-} [20:46:31] tldr: have drinks on roof top! [20:47:18] I haven't been up there yet! It was too rainy, we all hung out in the lounge in the lobby [20:47:51] are folks going to adopt scap3? :) [20:48:51] I have suggested to thcipriani to have devs handle the migration to scap3 so they learn about it and become responsible [20:49:06] instead of a couple of us migrating all the repos and having to handle all the support [20:49:14] My impression is we probably got some takers from the session. [20:49:17] yeah, I think the model will be "we'll help you migrate" [20:50:52] I made sure to mention that we're available to help in this channel, there are docs, and we're ready to have folks use it. I had folks mention that they were aware it was the next tooling, but weren't sure about timelines. [20:51:32] well [20:51:36] time to advertise / market it :-} [20:52:08] yeah, that was a large part of the scap3 session. [20:52:23] something like https://upload.wikimedia.org/wikipedia/commons/e/e9/Human_Billboard.jpg [20:52:35] with "SCAP3 available TODAY" "30% discount" [20:52:53] so we should have made sandwich boards for the summit :) [20:53:31] looks like lunch time! [20:53:35] ;-} [20:53:44] would you like some scap3 with your appetizer ? [20:53:54] good to know folks are interested at least [22:06:27] 10Continuous-Integration-Config, 10Kartographer, 10VisualEditor: Add VisualEditor as a CI dependency of the Kartographer extension - https://phabricator.wikimedia.org/T122926#1916879 (10Jdforrester-WMF) 3NEW [22:10:58] (03PS1) 10Hashar: [Kartographer] VE as a dependency [integration/config] - 10https://gerrit.wikimedia.org/r/262671 (https://phabricator.wikimedia.org/T122926) [22:11:36] (03CR) 10Paladox: "@Hashar could we merge this and do the refactoring in a separate patch please." [integration/config] - 10https://gerrit.wikimedia.org/r/262432 (https://phabricator.wikimedia.org/T121479) (owner: 10Paladox) [22:12:32] (03CR) 10Hashar: [C: 032] [Kartographer] VE as a dependency [integration/config] - 10https://gerrit.wikimedia.org/r/262671 (https://phabricator.wikimedia.org/T122926) (owner: 10Hashar) [22:13:50] (03Merged) 10jenkins-bot: [Kartographer] VE as a dependency [integration/config] - 10https://gerrit.wikimedia.org/r/262671 (https://phabricator.wikimedia.org/T122926) (owner: 10Hashar) [22:14:20] (03PS1) 10Jforrester: [Kartographer] Add VisualEditor as a dependency [integration/config] - 10https://gerrit.wikimedia.org/r/262672 (https://phabricator.wikimedia.org/T122926) [22:15:13] (03Abandoned) 10Jforrester: [Kartographer] Add VisualEditor as a dependency [integration/config] - 10https://gerrit.wikimedia.org/r/262672 (https://phabricator.wikimedia.org/T122926) (owner: 10Jforrester) [22:16:47] (03CR) 10Hashar: "A common source of confusion. Kartographer uses the generic job mwext-testextension-zend which has dependencies injected by Zuul. See htt" [integration/config] - 10https://gerrit.wikimedia.org/r/262672 (https://phabricator.wikimedia.org/T122926) (owner: 10Jforrester) [22:17:01] (03CR) 10Hashar: "I have rechecked Ed patch on https://gerrit.wikimedia.org/r/#/c/262448/4 and it passes now :-}" [integration/config] - 10https://gerrit.wikimedia.org/r/262671 (https://phabricator.wikimedia.org/T122926) (owner: 10Hashar) [22:17:08] (03CR) 10Jforrester: "Thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/262672 (https://phabricator.wikimedia.org/T122926) (owner: 10Jforrester) [22:17:59] 10Continuous-Integration-Config, 10Kartographer, 10VisualEditor, 5Patch-For-Review: Add VisualEditor as a CI dependency of the Kartographer extension - https://phabricator.wikimedia.org/T122926#1916936 (10hashar) 5Open>3Resolved a:3hashar Verified to pass on https://gerrit.wikimedia.org/r/#/c/262448/... [22:50:41] 10Continuous-Integration-Config, 10Kartographer, 10VisualEditor: Add VisualEditor as a CI dependency of the Kartographer extension - https://phabricator.wikimedia.org/T122926#1917016 (10Jdforrester-WMF) [23:59:36] (03PS27) 10Paladox: Enable zuul/jenkins to run unit tests in skin projects [integration/config] - 10https://gerrit.wikimedia.org/r/228470 (https://phabricator.wikimedia.org/T68926)