[00:16:49] Project selenium-Flow » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #165: 04FAILURE in 48 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/165/ [00:27:52] 06Release-Engineering-Team, 10Wikimedia-Developer-Summit, 06Developer-Relations (Oct-Dec-2016): Developer Summit 2017: Work with TPG and RelEng on solution to event documenting - https://phabricator.wikimedia.org/T132400#2196957 (10RobLa-WMF) @Rfarrand here's what we advised "scribes" to do in [last year's c... [00:46:35] PROBLEM - Puppet run on deployment-parsoid09 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [00:50:27] PROBLEM - Puppet run on deployment-mediawiki05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [01:25:28] RECOVERY - Puppet run on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [01:26:34] RECOVERY - Puppet run on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [01:28:53] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Build an apt repository on deployment-prep (for testing packages from jenkins) - https://phabricator.wikimedia.org/T146497#2695797 (10mmodell) 05Open>03Resolved [01:29:15] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Build an apt repository on deployment-prep (for testing packages from jenkins) - https://phabricator.wikimedia.org/T146497#2663278 (10mmodell) [01:29:17] 10Beta-Cluster-Infrastructure, 10scap, 13Patch-For-Review: On beta, puppet keeps downgrading scap ! - https://phabricator.wikimedia.org/T146618#2695798 (10mmodell) 05Open>03Resolved a:03mmodell [03:43:18] Project mediawiki-core-code-coverage build #2306: 04FAILURE in 43 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2306/ [04:17:47] Yippee, build fixed! [04:17:48] Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #163: 09FIXED in 21 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/163/ [05:01:55] (03PS1) 10Awight: Enable Python unit tests for the wikimedia/fundraising/tools repo [integration/config] - 10https://gerrit.wikimedia.org/r/314481 (https://phabricator.wikimedia.org/T145012) [07:02:05] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [07:24:42] 03Scap3, 07Documentation: Write "how to build scap plugins" developer documentation - https://phabricator.wikimedia.org/T147546#2696010 (10mmodell) [07:25:05] 03Scap3, 07Documentation: Write "how to build scap plugins" developer documentation - https://phabricator.wikimedia.org/T147546#2696023 (10mmodell) [07:26:20] 03Scap3, 07Documentation: Write "how to build scap plugins" developer documentation - https://phabricator.wikimedia.org/T147546#2696010 (10mmodell) p:05Triage>03Normal [08:55:08] PROBLEM - Puppet run on deployment-tin is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:04:49] PROBLEM - Puppet run on integration-slave-trusty-1014 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:18:32] PROBLEM - Puppet run on deployment-mira is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:39:50] RECOVERY - Puppet run on integration-slave-trusty-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [10:10:50] (03PS7) 10Hashar: Move composer-hhvm/php5 jobs back to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/306727 (https://phabricator.wikimedia.org/T143938) [10:10:52] (03PS1) 10Hashar: [SmashPig] move composer job to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/314523 [10:10:54] (03PS1) 10Hashar: [fundraising/php-queue] Move composer job to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/314524 [10:10:54] spaaaaam [10:10:56] (03PS1) 10Hashar: heritage + guc: move composer job to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/314525 [10:10:58] (03PS1) 10Hashar: Composer jobs of repo not supporting php5.3 to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/314526 [10:11:00] (03PS1) 10Hashar: Migrate composer jobs in 'experimental to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/314527 [10:15:37] (03CR) 10Hashar: [C: 04-1] "I have split this patch and send a bunch of atomic ones that migrate small chunks of repos to Nodepool. They are all very low traffic. I h" [integration/config] - 10https://gerrit.wikimedia.org/r/306727 (https://phabricator.wikimedia.org/T143938) (owner: 10Hashar) [10:19:03] (03CR) 10Hashar: [C: 032] "Skipif is properly applied:" [integration/config] - 10https://gerrit.wikimedia.org/r/314523 (owner: 10Hashar) [10:19:29] (03CR) 10Hashar: [C: 032] [fundraising/php-queue] Move composer job to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/314524 (owner: 10Hashar) [10:20:04] (03Merged) 10jenkins-bot: [SmashPig] move composer job to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/314523 (owner: 10Hashar) [10:20:28] (03Merged) 10jenkins-bot: [fundraising/php-queue] Move composer job to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/314524 (owner: 10Hashar) [10:25:34] brion: it is great to see you baby sitting TimedMediaHandler patches :] [10:27:49] (03CR) 10Hashar: [C: 032] heritage + guc: move composer job to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/314525 (owner: 10Hashar) [10:28:46] (03Merged) 10jenkins-bot: heritage + guc: move composer job to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/314525 (owner: 10Hashar) [10:31:04] (03CR) 10Hashar: [C: 032] Composer jobs of repo not supporting php5.3 to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/314526 (owner: 10Hashar) [10:31:48] (03CR) 10Hashar: [C: 032] Migrate composer jobs in 'experimental to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/314527 (owner: 10Hashar) [10:32:01] (03Merged) 10jenkins-bot: Composer jobs of repo not supporting php5.3 to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/314526 (owner: 10Hashar) [10:32:46] :D [10:32:49] it needs so much love [10:32:50] (03Merged) 10jenkins-bot: Migrate composer jobs in 'experimental to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/314527 (owner: 10Hashar) [10:45:05] 10Beta-Cluster-Infrastructure, 03Scap3, 06Services, 15User-mobrovac: The deploy-service can't SSH into target nodes in Beta - https://phabricator.wikimedia.org/T147551#2696211 (10mobrovac) [10:48:13] 10Beta-Cluster-Infrastructure, 03Scap3, 06Services, 15User-mobrovac: The deploy-service can't SSH into target nodes in Beta - https://phabricator.wikimedia.org/T147551#2696223 (10mobrovac) `deployment-mira` is of no use either: ```lines=15 mobrovac@deployment-mira:/srv/deployment/changeprop/deploy$ SSH_AU... [11:01:26] hashar: o/ [11:01:46] (03PS3) 10Jean-Frédéric: Publish code coverage post-merge in labs/tools/heritage [integration/config] - 10https://gerrit.wikimedia.org/r/314171 [11:02:06] I am working on a refactoring of the memcached puppet role (https://gerrit.wikimedia.org/r/#/c/314260), the goal would be to test more easily new versions in deploymnet-prep [11:02:29] is it ok if I install and test a new version of memcached during the next days? [11:02:47] elukey: good morning :) [11:03:30] elukey: oh on beta. Yeah go ahead at your convenience. You might want to !log it in #wikimedia-releng [11:03:39] also iirc we have different memcached instances [11:03:46] and I am not sure all have been migrated to Jessie yet [11:04:00] ahhh snap I didn't check, good point [11:04:11] nor how you can add a more specific .deb package on beta [11:04:26] we use the same components as production (eg: jessie-wikimedia/main ) [11:04:44] so potentially if you push a new memcached.deb on apt.wm.o for beta, that might well end up landing on prod :( [11:07:38] (03CR) 10Hashar: "The Zuul config is all fine. The wikimedia/fundraising/tools branches master and deploy do not pass flake8 yet though, there are pending " [integration/config] - 10https://gerrit.wikimedia.org/r/314481 (https://phabricator.wikimedia.org/T145012) (owner: 10Awight) [11:08:00] (03CR) 10Hashar: [C: 031] Enable Python unit tests for the wikimedia/fundraising/tools repo [integration/config] - 10https://gerrit.wikimedia.org/r/314481 (https://phabricator.wikimedia.org/T145012) (owner: 10Awight) [11:10:20] (03PS4) 10Hashar: [ReplaceText] Add composer-test test [integration/config] - 10https://gerrit.wikimedia.org/r/291838 (owner: 10Paladox) [11:11:06] (03CR) 10Hashar: [C: 032] [ReplaceText] Add composer-test test [integration/config] - 10https://gerrit.wikimedia.org/r/291838 (owner: 10Paladox) [11:12:03] (03Merged) 10jenkins-bot: [ReplaceText] Add composer-test test [integration/config] - 10https://gerrit.wikimedia.org/r/291838 (owner: 10Paladox) [11:16:10] (03Abandoned) 10Hashar: Migrate composer-validate to trusty nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/291505 (owner: 10Paladox) [11:16:41] (03Abandoned) 10Hashar: Migrate composer-package-validate to trusty nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/291506 (owner: 10Paladox) [11:19:03] (03PS3) 10Hashar: [mediawiki/core] experimental composer-php70-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/309556 (https://phabricator.wikimedia.org/T144961) (owner: 10Paladox) [11:19:35] (03CR) 10Hashar: "Rebased and switched to Nodepool:" [integration/config] - 10https://gerrit.wikimedia.org/r/309556 (https://phabricator.wikimedia.org/T144961) (owner: 10Paladox) [11:19:41] (03CR) 10Hashar: [C: 032] [mediawiki/core] experimental composer-php70-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/309556 (https://phabricator.wikimedia.org/T144961) (owner: 10Paladox) [11:47:38] (03PS4) 10Hashar: [mediawiki/core] experimental composer-php70-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/309556 (https://phabricator.wikimedia.org/T144961) (owner: 10Paladox) [11:48:01] (03CR) 10Hashar: [mediawiki/core] experimental composer-php70-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/309556 (https://phabricator.wikimedia.org/T144961) (owner: 10Paladox) [11:48:03] (03CR) 10Hashar: [C: 032] [mediawiki/core] experimental composer-php70-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/309556 (https://phabricator.wikimedia.org/T144961) (owner: 10Paladox) [11:49:01] (03Merged) 10jenkins-bot: [mediawiki/core] experimental composer-php70-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/309556 (https://phabricator.wikimedia.org/T144961) (owner: 10Paladox) [11:52:47] 03Scap3, 10scap, 06Operations, 13Patch-For-Review, and 2 others: Scap::server::sources is out of sync with the repositories actually present on tin/mira - https://phabricator.wikimedia.org/T143692#2696285 (10Joe) So, apart from servermon, which points to a (still) inexistent url, all other services that us... [11:52:58] 03Scap3, 10scap, 06Operations, 13Patch-For-Review, and 2 others: Scap::server::sources is out of sync with the repositories actually present on tin/mira - https://phabricator.wikimedia.org/T143692#2696286 (10Joe) 05Open>03Resolved [11:59:13] hashar: sorry got distracted by other things. The deb package is not an issue since I am going to upload and install manually [12:00:34] also just checked deployment-memc04.deployment-prep.eqiad.wmflabs and it is running debian [12:00:46] with memcached 1.4.21 that is the right version [12:08:03] elukey: neat :) [12:10:54] I am trying to install 1.4.28 that will give us commands like 'watch fetchers' or 'watch evictions' [12:11:21] but there are new parameters to add so I can just simply install the package :) [12:11:39] the code review that I listed above should give us the possibility to configure them via hiera [12:16:35] !log Jenkins slave deployment-tin.eqiad , removing label "deployment-tin.eqiad" it has "BetaClusterBastion" and all jobs are bound to it already [12:16:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:35:31] grocery shopping brb [12:43:58] hashar: any idea why https://gerrit.wikimedia.org/r/#/c/313979/ isnt running the tests! (for when you get back from tomato shopping) :D [13:14:53] addshore: looking [13:15:31] addshore: I guess that repo hasn't been added to the zuul layout file yet ? [13:15:44] bah, I always forget! [13:16:03] zuul/layout.yaml: - name: mediawiki/extensions/InterwikiSorting [13:16:04] na it is there [13:17:44] oh, yeh, maybe that is the place I added it too! I just remember the old days of having to add it to 2 different places! [13:18:05] addshore: ah [13:18:11] so Jenkins cant comment [13:18:27] oooh [13:18:29] https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/InterwikiSorting,access [13:18:40] though it inherits from mediawiki/extensions [13:18:43] but Rights Inherit From: mediawiki/extensions ? [13:19:05] hmm [13:19:09] Exception: Gerrit error executing gerrit query --format json --all-approvals --comments --commit-message --current-patch-set --dependencies --files --patch-sets --submit-records 313979 [13:19:12] that is not comment [13:19:18] it cant query somehow [13:19:30] mind you, I reproduce locally [13:19:34] fatal: internal server error [13:19:46] you cant comment either? [13:20:18] com.google.gwtorm.server.OrmException: No ChangeDataSource: ownerin:"ldap%3Acn%3Dwmde%2Cou%3Dgroups%2Cdc%3Dwikimedia%2Cdc%3Dorg" [13:20:29] there is java stacktrace server side [13:21:19] hashar: I just pushed a second change to see if it is just something odd with that change https://gerrit.wikimedia.org/r/#/c/314543/ [13:21:43] but also nothing for that one in zuul! [13:21:44] ah [13:22:00] so the previous error was different [13:22:02] java.lang.IllegalArgumentException: numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count [13:22:05] that is for your patch [13:22:10] and it is a known issue fixed in Gerrit [13:22:36] https://phabricator.wikimedia.org/T146260 :D [13:23:00] >> I'm fairly sure it only starts working on the third commit to a repo. ? [13:23:03] will want ostriches to upgrade our Gerrit from 2.12.3 to 2.12.5 [13:23:14] so I need to make 1` more commit? [13:23:37] maybe ? [13:23:51] * addshore makes a 3rd and 4th as drafts [13:26:07] maybe that needs at least 2 commits merged in [13:26:30] hmm, maybe! [13:26:30] well, I'll just go ahead and merge this one then... [13:26:38] yup :( [13:26:39] infact, let me split the .gitreview out as a first commit.... [13:27:26] and the readme maybe... [13:27:57] :D [13:32:35] well, it looks like that worked.... :P [13:34:06] 10Beta-Cluster-Infrastructure, 06Labs, 13Patch-For-Review: Replace all class imports on Labs with role imports - https://phabricator.wikimedia.org/T147233#2696393 (10Andrew) [13:34:48] (03PS4) 10Hashar: [timeline] Add test extensions-unittests-generic [integration/config] - 10https://gerrit.wikimedia.org/r/303305 (owner: 10Paladox) [13:35:06] addshore: yeah but tests are failling :( [13:35:15] https://integration.wikimedia.org/ci/job/mwext-testextension-php55/23995/console [13:35:23] yep, but let me look into that! [13:35:24] 00:00:11.701 Could not load some extensions because they are missing [13:35:25] 00:00:11.701 the expected entry point: [13:35:25] 00:00:11.701 [13:35:25] 00:00:11.701 InterwikiSorting [13:35:26] 00:00:11.701 [13:35:37] needs extension.json and or InterwikiSorting.php [13:35:42] oh, this should be using extension.json [13:35:52] zuul hasn't reported back yet [13:36:05] because it suspect the mediawiki/core change that got +2ed before might be the source of breakage [13:36:07] bad zuul :D [13:36:14] I can confirm tests all pass lcoally [13:36:22] haha! [13:36:54] (03CR) 10Hashar: [C: 032] "Thanks" [integration/config] - 10https://gerrit.wikimedia.org/r/303305 (owner: 10Paladox) [13:37:05] hashar: it did report back https://integration.wikimedia.org/ci/job/mwext-testextension-php55/23996/console [13:37:24] the error you posted was for the .gitreview and the readme (when there was no entry file) [13:37:50] (03Merged) 10jenkins-bot: [timeline] Add test extensions-unittests-generic [integration/config] - 10https://gerrit.wikimedia.org/r/303305 (owner: 10Paladox) [13:37:58] addshore: you can probably just push the change [13:38:27] $files = array_merge( $files, __DIR__ . '/tests/phpunit' ); haha lol [13:38:35] :D [13:38:58] im slightly confused as I defintly copied that from somewhere... [13:39:34] addshore: I think nowadays mediawiki auto detect tests in extension [13:39:41] I think I have seen such a patch floating around [13:39:55] *looks on phab* [13:40:18] (03CR) 10Hashar: "Works. Tested on https://gerrit.wikimedia.org/r/#/c/313328/" [integration/config] - 10https://gerrit.wikimedia.org/r/303305 (owner: 10Paladox) [13:40:38] Since MediaWiki 1.28 test case files with the suffix "Test.php" in the extension subdirectory "tests/phpunit" are automatically appended [13:40:40] epic! [13:43:22] magic [13:45:20] now I just have a php7 vs php55 issue to resolve! [13:46:40] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #170: 04FAILURE in 2 min 39 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/170/ [13:46:47] addshore: php7 ? [13:47:08] yeh, I develop on php7 :P [13:47:21] well, I have 54, 56 and 7 installed... [13:47:38] hmm let me check [13:47:42] bash: php: command not found [13:47:49] yeah does not surprise me [13:52:57] remembers me I wanted to port the mw code coverage job to php7 [13:55:40] !job mediawiki-core-code-coverage-php7 [13:55:43] !jenkins mediawiki-core-code-coverage-php7 [13:55:43] https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7 [13:58:13] (03PS1) 10Hashar: Try MediaWiki code coverage on Jessie/php7 [integration/config] - 10https://gerrit.wikimedia.org/r/314559 [14:22:57] (03PS2) 10Hashar: Try MediaWiki code coverage on Jessie/php7 [integration/config] - 10https://gerrit.wikimedia.org/r/314559 [14:23:29] (03CR) 10Hashar: "The job fail very early on and I have no idea why :( https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/" [integration/config] - 10https://gerrit.wikimedia.org/r/314559 (owner: 10Hashar) [14:27:50] looks like master is broken [14:29:52] ? [14:36:30] Reedy: some random database issue when running unit tests under phpdbg7.0 https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/3/console :D [14:36:47] REL1_27 does not have that issue though https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/4/console [14:36:50] no clue what changed [14:36:50] php sucks [14:36:55] it is just an experimental job anyway [14:45:43] !log deployment-mira disarmed/rearmed keyholder in an attempt to clear a Shinken alarm [14:45:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:46:49] RECOVERY - Keyholder status on deployment-mira is OK: OK: Less than 100.00% above the threshold [0.0] [14:46:53] magic [14:50:07] hashar hi, thanks for merging my patches today :) [15:08:13] 06Release-Engineering-Team, 03Scap3 (Scap3-MediaWiki-MVP), 07Technical-Debt: What is /fonts/ in wmf-config and is it important? - https://phabricator.wikimedia.org/T147481#2696872 (10hashar) [15:09:24] Debug: hiera(): Fetching http://labs-puppetmaster-eqiad.wikimedia.org:8100/v1/contintcloud/node/hasharcomputer [15:09:25] yeah [15:09:30] that is not going to work :] [15:10:04] :) [15:10:21] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 10MobileFrontend, 06Reading-Web-Backlog, 13Patch-For-Review: beta cluster: Notice: Undefined variable: wmgMFUseCentralAuthToken in /srv/mediawiki-staging/wmf-config/mobile-labs.php on line... - https://phabricator.wikimedia.org/T146945#2696880 [15:11:17] 10Continuous-Integration-Config, 10MediaWiki-Codesniffer: Add cache support to PHPCode_Sniffer jobs - https://phabricator.wikimedia.org/T146644#2696882 (10hashar) p:05Triage>03Normal [15:11:30] 06Release-Engineering-Team, 07Wikimedia-log-errors: Error: Couldn't find trailer dictionary - https://phabricator.wikimedia.org/T145772#2696883 (10hashar) p:05Triage>03Low [15:11:57] 10Gerrit: In Gerrit, clear watchlist of several accounts that are no more used (to avoid mail server errors) - https://phabricator.wikimedia.org/T113926#2696884 (10hashar) p:05Triage>03Low [15:30:41] (03PS1) 10Hashar: dib: craft our own hiera lookup data [integration/config] - 10https://gerrit.wikimedia.org/r/314570 [15:30:43] (03PS1) 10Hashar: dib: run puppet with --debug [integration/config] - 10https://gerrit.wikimedia.org/r/314571 [15:31:25] (03CR) 10Hashar: [C: 04-1] "That is just to remember about setting --debug. Might want to rely on DIB_DEBUG_TRACE to set --debug." [integration/config] - 10https://gerrit.wikimedia.org/r/314571 (owner: 10Hashar) [15:31:44] (03CR) 10Hashar: [C: 04-2] "Pending image rebuild on my machine. Jessie going on, will check Ubuntu later on." [integration/config] - 10https://gerrit.wikimedia.org/r/314570 (owner: 10Hashar) [15:32:29] done for today *wave* [15:33:39] * greg-g waves [15:35:25] 10Beta-Cluster-Infrastructure, 03Scap3, 06Services, 15User-mobrovac: The deploy-service can't SSH into target nodes in Beta - https://phabricator.wikimedia.org/T147551#2696919 (10thcipriani) 05Open>03Resolved a:03thcipriani @mobrovac I got `deployment-tin` up-to-date. It should be good to use. The ro... [15:39:48] PROBLEM - Puppet run on deployment-phab02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:46:04] Project mediawiki-core-code-coverage build #2307: 04STILL FAILING in 46 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2307/ [15:51:04] PROBLEM - Puppet run on deployment-phab01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:52:59] hasharAway ostriches hi, i found this https://gerrit.googlesource.com/plugins/zuul/ plugin for gerrit for zuul today [15:53:01] :) [16:14:38] 10Beta-Cluster-Infrastructure, 06Labs, 13Patch-For-Review: Replace all class imports on Labs with role imports - https://phabricator.wikimedia.org/T147233#2697077 (10Andrew) [16:27:32] (03CR) 10Hashar: [C: 032] "That fixed the Jessie image. Made progress on the Trusty image as well though it is now blocked on another issue." [integration/config] - 10https://gerrit.wikimedia.org/r/314570 (owner: 10Hashar) [16:28:29] (03Merged) 10jenkins-bot: dib: craft our own hiera lookup data [integration/config] - 10https://gerrit.wikimedia.org/r/314570 (owner: 10Hashar) [16:29:21] PROBLEM - Puppet run on deployment-restbase02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:43:32] PROBLEM - Puppet run on deployment-restbase01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:45:14] 10Beta-Cluster-Infrastructure, 03Scap3, 06Services, 15User-mobrovac: The deploy-service can't SSH into target nodes in Beta - https://phabricator.wikimedia.org/T147551#2697151 (10mobrovac) All good. I was able to deploy Citoid and Change-Prop. Thnx @thcipriani ! [17:04:53] hasharAway: Thanks for all the review! [17:25:47] A couple of gerrits ago, every patch had a 'this depends on' and 'what depends on this' link so that you could step forward and backwards in a patchset. [17:25:52] Is there something like that, still, buried in the new UI? [17:26:04] I find that i've stopped reviewing patchsets because I can't tell what the hell is going on [17:26:37] same here, can't seem to find the dependencies between patches in the gui any more [17:26:57] Yes [17:26:59] there [17:27:00] is [17:27:14] It is in the side of the path [17:27:15] patch [17:27:38] Patches are either above or below the patch to indicate weather it depends on that patch or another [17:28:52] Related changes [17:29:02] andrewbogott mobrovac ^^ [17:29:29] https://gerrit-review.googlesource.com/Documentation/user-review-ui.html#related-changes [17:29:35] 10Beta-Cluster-Infrastructure, 06Labs, 13Patch-For-Review: Replace all class imports on Labs with role imports - https://phabricator.wikimedia.org/T147233#2697241 (10Andrew) [17:29:41] the same topic column? [17:29:52] that's not it [17:30:02] because same topic extends over repos [17:30:07] Nope [17:30:13] this is related changes [17:30:31] That is the new ui for this depends on and what depends on this [17:30:39] It's a little confussing [17:30:51] it sucks [17:30:52] but that's what upstream directed me too [17:31:06] ok, so 'related changes' shows the whole patchset, in (reverse) order? [17:31:21] Nope, it shows the patch in the middle [17:31:23] then around it [17:31:51] hm... [17:31:51] at the top and below, it will show weather the patch depends on this patch or weather the current patch depends on another patch [17:32:05] that section of my window is a catastrophe [17:32:13] yep [17:34:22] http://bogott.net/misc/gerrit.png [17:34:47] it says there are 5 related changes, then it has like 20 lines below…? [17:35:43] Oh! If I actually click on the related changes link I get something semi-reasonable [17:36:06] so I guess that gets me what I need, mostly [17:36:07] thanks [17:36:11] oh [17:36:13] your welcome [17:54:58] andrewbogott: and you can search in Gerrit for changes having the same topic with topic:foo [17:55:07] andrewbogott: then for all your changes: git-review --topic foo [17:55:14] and you "easily" find them all [17:55:25] its not user friednly though :( [17:57:37] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 06Services, 05services-tooling: RFC: Streamline NodeJS testing+deployment - https://phabricator.wikimedia.org/T147581#2697296 (10Yurik) [17:58:06] 10Beta-Cluster-Infrastructure, 10Graphoid, 06Services: Automate Graphoid deployment to beta cluster (and auto-rebuild?) - https://phabricator.wikimedia.org/T146810#2697309 (10Yurik) [17:58:08] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 06Services, 05services-tooling: RFC: Streamline NodeJS testing+deployment - https://phabricator.wikimedia.org/T147581#2697308 (10Yurik) [17:59:04] also, most of the time that i'm on a patch (in the gerrit gui), the screen is furiously flickering [17:59:19] i'll have to make a video once of that [17:59:26] it's a real WTF [18:05:21] 10Gerrit: Rename the Semantic Forms extension to "Page Forms" - https://phabricator.wikimedia.org/T147582#2697328 (10Yaron_Koren) [18:05:28] Project beta-code-update-eqiad build #124601: 04FAILURE in 2 min 27 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/124601/ [18:11:39] PROBLEM - Puppet run on integration-slave-precise-1012 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:12:31] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:12:43] PROBLEM - Puppet run on integration-slave-precise-1011 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:14:23] PROBLEM - Puppet run on integration-slave-precise-1002 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:18:24] Yippee, build fixed! [18:18:25] Project beta-code-update-eqiad build #124602: 09FIXED in 5 min 24 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/124602/ [18:25:36] Project beta-code-update-eqiad build #124603: 04FAILURE in 2 min 35 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/124603/ [18:27:45] PROBLEM - Puppet run on integration-slave-jessie-1003 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:30:21] PROBLEM - Puppet run on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:32:01] PROBLEM - Puppet run on deployment-sentry01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:32:41] PROBLEM - Puppet run on integration-slave-jessie-1004 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:34:57] Project beta-code-update-eqiad build #124604: 04STILL FAILING in 1 min 56 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/124604/ [18:35:48] PROBLEM - Puppet run on integration-slave-trusty-1014 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:36:08] PROBLEM - Puppet run on integration-slave-trusty-1011 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:37:20] PROBLEM - Puppet run on deployment-eventlogging04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:40:08] PROBLEM - Puppet run on integration-slave-trusty-1018 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:43:00] PROBLEM - Puppet run on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:44:55] Project beta-code-update-eqiad build #124605: 04STILL FAILING in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/124605/ [18:46:53] PROBLEM - Puppet run on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:49:31] PROBLEM - Puppet run on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:51:13] PROBLEM - Puppet run on integration-slave-trusty-1016 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:53:54] PROBLEM - Puppet run on integration-slave-trusty-1013 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:54:10] PROBLEM - Puppet run on integration-slave-jessie-1005 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:54:54] PROBLEM - Puppet run on integration-slave-trusty-1017 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:54:54] Project beta-code-update-eqiad build #124606: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/124606/ [18:57:14] PROBLEM - Puppet run on integration-slave-trusty-1012 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:58:50] PROBLEM - Puppet run on integration-slave-trusty-1004 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [19:04:56] Project beta-code-update-eqiad build #124607: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/124607/ [19:09:26] gerrit: down for everyone? [19:09:41] andrewbogott: Yeah, trying to debug in -operations [19:09:54] Started flipping out over an hour ago [19:14:53] Project beta-code-update-eqiad build #124608: 04STILL FAILING in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/124608/ [19:17:31] 10Beta-Cluster-Infrastructure, 06Labs, 13Patch-For-Review: Replace all class imports on Labs with role imports - https://phabricator.wikimedia.org/T147233#2697541 (10Andrew) [19:24:55] Project beta-code-update-eqiad build #124609: 04STILL FAILING in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/124609/ [19:34:58] Project beta-code-update-eqiad build #124610: 04STILL FAILING in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/124610/ [19:35:15] what's going on with beta? [19:35:30] greg-g: gerrit is down [19:35:35] it can't pull [19:35:39] and turtles [19:35:53] it pulls over HTTPS [19:36:10] durrr [19:36:23] Moar coffee [19:37:47] I quit the stuff :/ [19:44:54] Project beta-code-update-eqiad build #124611: 04STILL FAILING in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/124611/ [19:55:03] Project beta-code-update-eqiad build #124612: 04STILL FAILING in 2 min 2 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/124612/ [20:04:56] Project beta-code-update-eqiad build #124613: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/124613/ [20:14:53] Project beta-code-update-eqiad build #124614: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/124614/ [20:21:40] who needs this whole gerrit thing anyway... [20:23:27] isn't that the point of a non centralised version control? ;) [20:24:55] Project beta-code-update-eqiad build #124615: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/124615/ [20:34:56] Project beta-code-update-eqiad build #124616: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/124616/ [20:35:50] again no train this week? :( [20:36:44] aharoni hi, see #wikimedia-operations [20:36:47] gerrit is down [20:36:54] so without that there can be no train [20:37:25] bum bum bummer [20:37:39] aharoni: we got wmf.21 on group0, had to rollback yesterday. Was hoping to get caught up today, but...yeah [20:37:53] *rollback group1 yesterday [20:44:54] Project beta-code-update-eqiad build #124617: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/124617/ [20:54:54] Project beta-code-update-eqiad build #124618: 04STILL FAILING in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/124618/ [21:11:16] Project beta-code-update-eqiad build #124619: 04STILL FAILING in 8 min 14 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/124619/ [21:16:00] Yippee, build fixed! [21:16:00] Project beta-code-update-eqiad build #124620: 09FIXED in 2 min 59 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/124620/ [21:26:52] RECOVERY - Puppet run on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [21:29:10] RECOVERY - Puppet run on integration-slave-jessie-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [21:29:30] RECOVERY - Puppet run on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [21:31:12] RECOVERY - Puppet run on integration-slave-trusty-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [21:32:12] RECOVERY - Puppet run on integration-slave-trusty-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [21:33:51] RECOVERY - Puppet run on integration-slave-trusty-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [21:33:55] RECOVERY - Puppet run on integration-slave-trusty-1013 is OK: OK: Less than 1.00% above the threshold [0.0] [21:34:55] RECOVERY - Puppet run on integration-slave-trusty-1017 is OK: OK: Less than 1.00% above the threshold [0.0] [21:37:03] RECOVERY - Puppet run on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:37:41] RECOVERY - Puppet run on integration-slave-jessie-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [21:37:43] RECOVERY - Puppet run on integration-slave-jessie-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [21:40:19] RECOVERY - Puppet run on integration-slave-trusty-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [21:40:49] RECOVERY - Puppet run on integration-slave-trusty-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [21:42:22] RECOVERY - Puppet run on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [21:45:10] RECOVERY - Puppet run on integration-slave-trusty-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [21:46:10] RECOVERY - Puppet run on integration-slave-trusty-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [21:47:44] RECOVERY - Puppet run on integration-slave-precise-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [21:49:26] RECOVERY - Puppet run on integration-slave-precise-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [21:51:39] RECOVERY - Puppet run on integration-slave-precise-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [21:52:35] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [21:53:01] RECOVERY - Puppet run on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [22:09:47] 10Gerrit, 06Operations, 10hardware-requests: Allocate spare misc box in eqiad for gerrit replacement - https://phabricator.wikimedia.org/T147596#2697736 (10demon) [22:09:48] 10Gerrit, 06Operations, 10hardware-requests: Allocate spare misc box in eqiad for gerrit replacement - https://phabricator.wikimedia.org/T147596#2697752 (10demon) [22:10:01] 10Gerrit, 06Operations, 13Patch-For-Review: setup/deploy cobalt as gerrit warm standby/replacement - https://phabricator.wikimedia.org/T147597#2697753 (10RobH) [22:10:11] 10Gerrit, 06Operations, 13Patch-For-Review: setup/deploy cobalt as gerrit warm standby/replacement - https://phabricator.wikimedia.org/T147597#2697796 (10RobH) [22:10:22] 10Gerrit, 06Operations, 13Patch-For-Review: setup/deploy cobalt as gerrit warm standby/replacement - https://phabricator.wikimedia.org/T147597#2697753 (10RobH) [22:10:33] 10Gerrit, 06Operations, 10hardware-requests: Allocate spare misc box in eqiad for gerrit replacement - https://phabricator.wikimedia.org/T147596#2697814 (10RobH) a:03mark Since this was potentially an outage condition, I recommended wmf4725 (hostname to be cobalt) be allocated for this. We've already star... [22:10:37] 10Gerrit, 06Operations, 10hardware-requests: Allocate spare misc box in eqiad for gerrit replacement - https://phabricator.wikimedia.org/T147596#2697736 (10RobH) [22:10:53] 10Gerrit, 06Operations, 13Patch-For-Review: setup/deploy cobalt as gerrit warm standby/replacement - https://phabricator.wikimedia.org/T147597#2697832 (10Dzahn) a:03Dzahn [22:10:55] 10Gerrit, 06Operations, 13Patch-For-Review: setup/deploy cobalt as gerrit warm standby/replacement - https://phabricator.wikimedia.org/T147597#2697753 (10Dzahn) [radon:~] $ host cobalt.mgmt.eqiad.wmnet cobalt.mgmt.eqiad.wmnet has address 10.65.2.127 [radon:~] $ host cobalt.wikimedia.org cobalt.wikimedia.or... [22:11:05] 10Gerrit, 06Operations, 13Patch-For-Review: setup/deploy cobalt as gerrit warm standby/replacement - https://phabricator.wikimedia.org/T147597#2697753 (10Dzahn) [22:27:36] PROBLEM - Puppet run on deployment-kafka05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:49:35] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T145220#2697975 (10thcipriani) [23:02:37] RECOVERY - Puppet run on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [23:04:25] 10Gerrit, 06Operations, 13Patch-For-Review: setup/deploy cobalt as gerrit warm standby/replacement - https://phabricator.wikimedia.org/T147597#2698059 (10Dzahn) [23:04:39] 10Gerrit, 06Operations: setup/deploy cobalt as gerrit warm standby/replacement - https://phabricator.wikimedia.org/T147597#2697753 (10Dzahn) [23:33:48] Why does editing throught he gerrit interface prefer leading spaces? :( [23:35:14] Reedy i think that may be fixed in gerrit 2.13