[00:08:17] twentyafterfour: Zuul queues are emtpy now, guess we can start with phabricator? ;) [00:10:51] Actually, that's a bug [00:10:56] Zuul appears to not be picking up events [00:11:11] I uploaded and +2ed https://gerrit.wikimedia.org/r/#/c/285886/ and Zuul noticed neither [00:12:42] meh [00:13:17] RoanKattouw i think zuul needs restarting [00:13:29] legoktm or jzerebecki ^^ [00:14:57] * RoanKattouw does not have ssh on gallium [00:23:15] looks like it works again [00:23:44] meh, not fully [00:25:39] Yay it's working now [00:27:41] hmm, just checked zuul's log on gallium, it seems to be putting along [00:38:49] Project selenium-Flow » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #2: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/2/ [00:41:50] Project selenium-Flow » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #2: 04FAILURE in 25 min: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/2/ [01:06:02] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Replicate open patchsets to diffusion - https://phabricator.wikimedia.org/T89940#2245718 (10mmodell) 05Open>03Resolved a:03mmodell [01:06:05] 05Gitblit-Deprecate, 06Release-Engineering-Team, 10Diffusion, 07WorkType-NewFunctionality: Use Diffusion as canonical location for browsing code repos (not gitblit) - https://phabricator.wikimedia.org/T752#2245720 (10mmodell) [01:07:33] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10Unplanned-Sprint-Work, 07FR-ActiveMQ, and 3 others: Run PHPUnit on PHP-Queue repo - https://phabricator.wikimedia.org/T133574#2245726 (10awight) [01:07:42] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10Unplanned-Sprint-Work, 07FR-ActiveMQ, and 3 others: Run PHPUnit on PHP-Queue repo - https://phabricator.wikimedia.org/T133574#2236211 (10awight) 05Open>03Resolved [01:08:18] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Replicate open patchsets to diffusion - https://phabricator.wikimedia.org/T89940#2245737 (10mmodell) [01:21:34] 10Beta-Cluster-Infrastructure, 10Math, 10VisualEditor, 10VisualEditor-MediaWiki, 07Beta-Cluster-reproducible: Math nodes are not getting rendered for the first time in a session on Beta Cluster - https://phabricator.wikimedia.org/T132620#2204608 (10Jdforrester-WMF) No issue in production, just in Beta Cl... [01:30:04] 06Release-Engineering-Team, 06Operations, 10Phabricator, 10Traffic, 13Patch-For-Review: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#2245804 (10chasemp) [01:30:10] 06Release-Engineering-Team, 06Operations, 10Phabricator, 10Traffic, 13Patch-For-Review: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1528619 (10chasemp) [01:34:26] Yippee, build fixed! [01:34:27] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #609: 09FIXED in 17 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/609/ [01:57:36] Yippee, build fixed! [01:57:37] Project browsertests-Wikidata-WikidataTests-Group0-SmokeTests-linux-firefox-sauce build #31: 09FIXED in 17 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-Group0-SmokeTests-linux-firefox-sauce/31/ [02:12:47] 10Deployment-Systems, 10MediaWiki-Debug-Logger: Capture PHP warnings with stacktraces in MediaWiki and save to logstash - https://phabricator.wikimedia.org/T45086#2245932 (10Reedy) [02:12:50] 10Deployment-Systems, 10MediaWiki-Debug-Logger: Capture PHP warnings with stacktraces in MediaWiki and save to logstash - https://phabricator.wikimedia.org/T45086#2245933 (10Krinkle) [02:12:56] 10Deployment-Systems, 10MediaWiki-Debug-Logger: Capture PHP warnings with stacktraces in MediaWiki and save to logstash - https://phabricator.wikimedia.org/T45086#479093 (10Krinkle) [02:35:33] 10Continuous-Integration-Infrastructure: Jenkins: integration-zuul-layoutdiff job says "No layout changes" when there are - https://phabricator.wikimedia.org/T73740#2246000 (10Krinkle) [02:40:48] 10Continuous-Integration-Infrastructure, 06Collaboration-Team-Interested, 10Flow, 10VisualEditor: Flow tests fails to run with VisualEditor installed - https://phabricator.wikimedia.org/T86920#2246076 (10matthiasmullie) [03:02:07] 10Deployment-Systems, 10MediaWiki-Configuration, 05MW-1.27-release-notes, 13Patch-For-Review: extension-list should live in the mediawiki branch rather than mediawiki-config - https://phabricator.wikimedia.org/T125678#2246173 (10mmodell) [03:02:10] 10Deployment-Systems, 10MediaWiki-Configuration, 05MW-1.27-release-notes, 13Patch-For-Review: extension-list should live in the mediawiki branch rather than mediawiki-config - https://phabricator.wikimedia.org/T125678#1994561 (10mmodell) [03:03:07] 03Scap3, 10scap, 13Patch-For-Review: scap::target shouldn't allow users to redefine the user's key - https://phabricator.wikimedia.org/T132747#2246177 (10mmodell) 05Open>03Resolved [03:03:09] 10Beta-Cluster-Infrastructure, 03Scap3, 10Citoid, 06Services, 10VisualEditor: Can't deploy Citoid in Beta - https://phabricator.wikimedia.org/T132666#2246179 (10mmodell) [03:03:12] 03Scap3, 10scap, 13Patch-For-Review: scap::target shouldn't allow users to redefine the user's key - https://phabricator.wikimedia.org/T132747#2209132 (10mmodell) 05Open>03Resolved [03:03:17] 03Scap3, 10scap, 13Patch-For-Review: scap::target shouldn't allow users to redefine the user's key - https://phabricator.wikimedia.org/T132747#2209132 (10mmodell) [03:03:19] 03Scap3, 10scap, 13Patch-For-Review: scap::target shouldn't allow users to redefine the user's key - https://phabricator.wikimedia.org/T132747#2209132 (10mmodell) [03:03:21] 03Scap3, 10scap, 13Patch-For-Review: scap::target shouldn't allow users to redefine the user's key - https://phabricator.wikimedia.org/T132747#2209132 (10mmodell) [03:03:23] 03Scap3, 10scap, 13Patch-For-Review: scap::target shouldn't allow users to redefine the user's key - https://phabricator.wikimedia.org/T132747#2209132 (10mmodell) [03:03:25] 03Scap3, 10scap, 13Patch-For-Review: scap::target shouldn't allow users to redefine the user's key - https://phabricator.wikimedia.org/T132747#2209132 (10mmodell) [03:06:40] 03Scap3, 10Phabricator, 07WorkType-Maintenance: Move /srv/phab/repos to /srv/repos - https://phabricator.wikimedia.org/T125853#2246226 (10mmodell) [03:06:50] 03Scap3, 10Phabricator, 07WorkType-Maintenance: Move /srv/phab/repos to /srv/repos - https://phabricator.wikimedia.org/T125853#2246230 (10mmodell) [03:07:45] 03Scap3, 10scap, 13Patch-For-Review: Make puppet provider for scap3 - https://phabricator.wikimedia.org/T113072#2246235 (10mmodell) [03:07:47] 03Scap3, 10scap, 13Patch-For-Review: Make puppet provider for scap3 - https://phabricator.wikimedia.org/T113072#1654085 (10mmodell) [03:07:49] 03Scap3, 10scap, 13Patch-For-Review: Make puppet provider for scap3 - https://phabricator.wikimedia.org/T113072#1654085 (10mmodell) [03:07:54] 03Scap3, 10scap, 13Patch-For-Review: Make puppet provider for scap3 - https://phabricator.wikimedia.org/T113072#1654085 (10mmodell) [03:07:56] 03Scap3, 10scap, 13Patch-For-Review: Make puppet provider for scap3 - https://phabricator.wikimedia.org/T113072#1654085 (10mmodell) [03:12:48] 06Release-Engineering-Team, 06Operations, 10Phabricator, 10Traffic, 13Patch-For-Review: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#2246257 (10chasemp) [03:12:54] 06Release-Engineering-Team, 06Operations, 10Phabricator, 10Traffic, 13Patch-For-Review: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1529683 (10chasemp) [03:13:00] 06Release-Engineering-Team, 06Operations, 10Phabricator, 10Traffic, 13Patch-For-Review: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1529683 (10chasemp) [03:13:06] 06Release-Engineering-Team, 06Operations, 10Phabricator, 10Traffic, 13Patch-For-Review: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1529708 (10chasemp) [03:13:30] 06Release-Engineering-Team, 06Operations, 10Phabricator, 10Traffic, 13Patch-For-Review: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1620392 (10chasemp) [03:13:36] 06Release-Engineering-Team, 06Operations, 10Phabricator, 10Traffic, 13Patch-For-Review: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1620392 (10chasemp) [03:15:51] 10Beta-Cluster-Infrastructure, 03Scap3, 10Citoid, 06Services, 10VisualEditor: Can't deploy Citoid in Beta - https://phabricator.wikimedia.org/T132666#2246304 (10mmodell) [03:15:54] 03Scap3, 10scap, 13Patch-For-Review: scap::target shouldn't allow users to redefine the user's key - https://phabricator.wikimedia.org/T132747#2246303 (10mmodell) 05Resolved>03Open [03:18:26] 05Continuous-Integration-Scaling, 06Operations, 13Patch-For-Review: Remove hashar and dduvall root access on to be installed labnodepool1001 - https://phabricator.wikimedia.org/T95303#2246309 (10chasemp) [03:20:22] 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: Move sudo permissions for deployment from modules/mediawiki/manifests/users.pp to data.yaml - https://phabricator.wikimedia.org/T97678#2246320 (10chasemp) [03:20:52] 06Release-Engineering-Team, 06Operations, 10Ops-Access-Requests, 10Phabricator: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#1216676 (10chasemp) [03:20:57] 06Release-Engineering-Team, 06Operations, 10Ops-Access-Requests, 10Phabricator: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#2246324 (10chasemp) [03:21:02] 06Release-Engineering-Team, 06Operations, 10Ops-Access-Requests, 10Phabricator: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#2246327 (10chasemp) [03:21:39] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: /var/lib/l10nupdate fills up deployment-bastion /var partition - https://phabricator.wikimedia.org/T95564#2246328 (10mmodell) [03:22:33] 06Release-Engineering-Team, 06Operations, 10Ops-Access-Requests, 10Phabricator, 13Patch-For-Review: Chad H. needs access to iridium (Phabricator host) to manage repos - https://phabricator.wikimedia.org/T92564#2246334 (10chasemp) [04:51:18] 10MediaWiki-Codesniffer, 03Google-Summer-of-Code-2016: Community bonding evaluation for Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T133829#2246443 (10Lethexie) [05:09:19] 10MediaWiki-Codesniffer, 03Google-Summer-of-Code-2016: [GSoC 2016 Proposal] Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T130574#2246489 (10Lethexie) @Legoktm @Addshore @EBernhardson Hi, could you give me your opinions about the detailed plan of GSoC. Also, please tell me... [06:33:38] PROBLEM - Puppet run on deployment-tmh01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:13:41] RECOVERY - Puppet run on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:02:12] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Replicate open patchsets to diffusion - https://phabricator.wikimedia.org/T89940#1049136 (10Ricordisamoa) What have you done?!? I got dozens of notifications about "commits" for Gerrit changes I never merged. [08:04:36] PROBLEM - Puppet run on deployment-tmh01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:06:50] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Replicate open patchsets to diffusion - https://phabricator.wikimedia.org/T89940#2247108 (10Paladox) >>! In T89940#2247090, @Ricordisamoa wrote: > What have you done?!? I got dozens of notifications about "commits" for Gerrit changes I never merged. We a... [08:08:04] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Replicate open patchsets to diffusion - https://phabricator.wikimedia.org/T89940#2247109 (10Paladox) @mmodell is there a way we can disable notifications for refs/changes/ please. [08:29:43] 05Continuous-Integration-Scaling, 10Parsoid, 06Services, 13Patch-For-Review, 07WorkType-NewFunctionality: Migrate Parsoid CI jobs from node 0.8/0.10 to 4.3 - https://phabricator.wikimedia.org/T126992#2247232 (10hashar) I am not finishing up the Parsoid migration yet. Been switching priority to migrate PH... [08:31:37] Yippee, build fixed! [08:31:38] Project selenium-MultimediaViewer-2016-04-26 » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #4: 09FIXED in 22 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer-2016-04-26/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/4/ [08:31:57] zeljkof: MultimediaViewer fixed some how :-}}} [08:32:01] flappy test [09:12:11] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [09:16:07] PROBLEM - Host integration-trusty-1026 is DOWN: CRITICAL - Host Unreachable (10.68.17.98) [09:41:48] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #794: 04FAILURE in 20 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/794/ [10:13:05] 06Release-Engineering-Team, 10scap, 10MediaWiki-Database, 10MediaWiki-JobRunner: Scap should restart job runners to pick up new config - https://phabricator.wikimedia.org/T126632#2019230 (10Joe) This is not what happens. As the jobrunner itself lives in https://phabricator.wikimedia.org/diffusion/GJOB/ and... [10:13:23] 06Release-Engineering-Team, 10scap, 10MediaWiki-Database, 10MediaWiki-JobRunner: Scap should restart job runners to pick up new config - https://phabricator.wikimedia.org/T126632#2247648 (10Joe) 05Open>03Invalid [10:14:51] 05Continuous-Integration-Scaling, 03releng-201516-q4, 07Blocked-on-Operations, 13Patch-For-Review, 07WorkType-NewFunctionality: Attempt to provide a Trusty image for Nodepool - https://phabricator.wikimedia.org/T133203#2247667 (10MoritzMuehlenhoff) [10:45:51] PROBLEM - Puppet run on deployment-mediawiki01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [11:20:50] RECOVERY - Puppet run on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:00:42] (03PS1) 10Hashar: dib: add /usr/bin/php wrapper [integration/config] - 10https://gerrit.wikimedia.org/r/285934 (https://phabricator.wikimedia.org/T126211) [12:02:00] (03PS2) 10Hashar: dib: add /usr/bin/php wrapper [integration/config] - 10https://gerrit.wikimedia.org/r/285934 (https://phabricator.wikimedia.org/T126211) [12:02:27] (03CR) 10Hashar: [C: 032] dib: add /usr/bin/php wrapper [integration/config] - 10https://gerrit.wikimedia.org/r/285934 (https://phabricator.wikimedia.org/T126211) (owner: 10Hashar) [12:03:14] (03Merged) 10jenkins-bot: dib: add /usr/bin/php wrapper [integration/config] - 10https://gerrit.wikimedia.org/r/285934 (https://phabricator.wikimedia.org/T126211) (owner: 10Hashar) [12:09:22] 10Continuous-Integration-Infrastructure: Make /usr/bin/php a wrapper that picks the right PHP version on CI slaves - https://phabricator.wikimedia.org/T126211#2248047 (10hashar) a:03hashar [12:09:33] 10Continuous-Integration-Infrastructure: Make /usr/bin/php a wrapper that picks the right PHP version on CI slaves - https://phabricator.wikimedia.org/T126211#2007557 (10hashar) [12:14:12] 10Continuous-Integration-Infrastructure: Make /usr/bin/php a wrapper that picks the right PHP version on CI slaves - https://phabricator.wikimedia.org/T126211#2248050 (10hashar) `contint::php` requires `contint::slave_scripts` and that ends up causing: Error: Duplicate declaration: Git::Clone[jenkins CI slave s... [12:16:13] (03PS1) 10Hashar: dib: we can now use contint::slave_scripts [integration/config] - 10https://gerrit.wikimedia.org/r/285935 (https://phabricator.wikimedia.org/T126211) [12:16:59] (03CR) 10Hashar: [C: 032] dib: we can now use contint::slave_scripts [integration/config] - 10https://gerrit.wikimedia.org/r/285935 (https://phabricator.wikimedia.org/T126211) (owner: 10Hashar) [12:17:43] (03Merged) 10jenkins-bot: dib: we can now use contint::slave_scripts [integration/config] - 10https://gerrit.wikimedia.org/r/285935 (https://phabricator.wikimedia.org/T126211) (owner: 10Hashar) [12:23:24] (03PS1) 10Hashar: dib: dropped 'git' package by mistake [integration/config] - 10https://gerrit.wikimedia.org/r/285937 [12:23:51] (03CR) 10Hashar: [C: 032] dib: dropped 'git' package by mistake [integration/config] - 10https://gerrit.wikimedia.org/r/285937 (owner: 10Hashar) [12:24:32] (03Merged) 10jenkins-bot: dib: dropped 'git' package by mistake [integration/config] - 10https://gerrit.wikimedia.org/r/285937 (owner: 10Hashar) [12:34:57] 10Continuous-Integration-Infrastructure: Investigate installing php5.3 on trusty and/or debian instance - https://phabricator.wikimedia.org/T103786#2248174 (10hashar) [12:35:44] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Make /usr/bin/php a wrapper that picks the right PHP version on CI slaves - https://phabricator.wikimedia.org/T126211#2248175 (10hashar) a:05hashar>03Legoktm @legoktm did the bulk of the work. [12:38:39] PROBLEM - Puppet run on deployment-changeprop is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:42:17] !log Rebuild Nodepool Trusty instance to include the PHP wrapper script T126211 [12:42:18] T126211: Make /usr/bin/php a wrapper that picks the right PHP version on CI slaves - https://phabricator.wikimedia.org/T126211 [12:42:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:47:27] !log apt-get upgrade deployment-changeprop (outdated exim package) [12:47:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:52:27] !log Puppet is happy on deployment-changeprop [12:52:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:53:01] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #1033: 04FAILURE in 21 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/1033/ [13:03:38] RECOVERY - Puppet run on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [13:05:21] (03PS1) 10Hashar: dib: provision rsyslog package [integration/config] - 10https://gerrit.wikimedia.org/r/285941 [13:06:55] (03CR) 10Hashar: [C: 032] dib: provision rsyslog package [integration/config] - 10https://gerrit.wikimedia.org/r/285941 (owner: 10Hashar) [13:07:44] (03Merged) 10jenkins-bot: dib: provision rsyslog package [integration/config] - 10https://gerrit.wikimedia.org/r/285941 (owner: 10Hashar) [13:12:33] (03PS1) 10Hashar: dib: require_package -> ensure_packages [integration/config] - 10https://gerrit.wikimedia.org/r/285942 [13:12:43] (03CR) 10Hashar: [C: 032] dib: require_package -> ensure_packages [integration/config] - 10https://gerrit.wikimedia.org/r/285942 (owner: 10Hashar) [13:13:13] Yippee, build fixed! [13:13:14] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » en,contintLabsSlave && UbuntuTrusty build #97: 09FIXED in 8 min 30 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=en,label=contintLabsSlave%20&&%20UbuntuTrusty/97/ [13:13:25] (03Merged) 10jenkins-bot: dib: require_package -> ensure_packages [integration/config] - 10https://gerrit.wikimedia.org/r/285942 (owner: 10Hashar) [13:17:58] (03PS1) 10Hashar: dib: try require ::rsyslog [integration/config] - 10https://gerrit.wikimedia.org/r/285943 [13:18:14] (03CR) 10Hashar: [C: 032] dib: try require ::rsyslog [integration/config] - 10https://gerrit.wikimedia.org/r/285943 (owner: 10Hashar) [13:18:39] Project selenium-MobileFrontend-279364 » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #6: 15ABORTED in 5 min 43 sec: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend-279364/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/6/ [13:18:40] Project selenium-MobileFrontend-279364 » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #6: 15ABORTED in 5 min 43 sec: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend-279364/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/6/ [13:19:09] (03Merged) 10jenkins-bot: dib: try require ::rsyslog [integration/config] - 10https://gerrit.wikimedia.org/r/285943 (owner: 10Hashar) [13:19:39] PROBLEM - Puppet run on deployment-changeprop is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [13:21:14] 06Release-Engineering-Team, 10MediaWiki-General-or-Unknown, 06Operations: Intermittent read-only errors on s3 wikis on March 14th - https://phabricator.wikimedia.org/T129947#2248437 (10fgiunchedi) p:05Triage>03Normal [13:33:25] 06Release-Engineering-Team, 10MediaWiki-General-or-Unknown, 06Operations: Intermittent read-only errors on s3 wikis on March 14th - https://phabricator.wikimedia.org/T129947#2248480 (10jcrespo) 05Open>03Resolved a:03jcrespo I would close this, AFAIK this didn't repeat, and after failover, state is comp... [13:33:54] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Replicate open patchsets to diffusion - https://phabricator.wikimedia.org/T89940#2248488 (10demon) I've long recommended that people turn off "[[ /settings/panel/emailpreferences/ | a commit is created ]]" in their settings anyway. It's a mostly useless n... [13:40:30] Project selenium-MultimediaViewer-master » internet_explorer 11.0,beta,Windows 7,contintLabsSlave && UbuntuTrusty build #5: 04FAILURE in 14 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer-master/BROWSER=internet_explorer%2011.0,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Windows%207,label=contintLabsSlave%20&&%20UbuntuTrusty/5/ [13:40:50] Project selenium-MultimediaViewer-master » internet_explorer 11.0,beta,Windows 8.1,contintLabsSlave && UbuntuTrusty build #5: 04FAILURE in 14 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer-master/BROWSER=internet_explorer%2011.0,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Windows%208.1,label=contintLabsSlave%20&&%20UbuntuTrusty/5/ [13:43:25] Project selenium-MultimediaViewer-master » safari,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #5: 04FAILURE in 17 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer-master/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/5/ [13:45:02] 06Release-Engineering-Team, 10scap, 10MediaWiki-Database, 10MediaWiki-JobRunner: Scap should restart job runners to pick up new config - https://phabricator.wikimedia.org/T126632#2248548 (10demon) Yeah this task is bogus, thx for tidying up. [13:47:50] Project selenium-MultimediaViewer-master » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #5: 04FAILURE in 21 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer-master/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/5/ [13:49:11] Project selenium-MultimediaViewer-master » internet_explorer 10.0,beta,Windows 8,contintLabsSlave && UbuntuTrusty build #5: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer-master/BROWSER=internet_explorer%2010.0,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Windows%208,label=contintLabsSlave%20&&%20UbuntuTrusty/5/ [13:52:18] 10Beta-Cluster-Infrastructure, 10EventBus, 06Services: Set up change-propagation in BetaCluster - https://phabricator.wikimedia.org/T133908#2248554 (10mobrovac) [13:54:54] 10Beta-Cluster-Infrastructure, 10EventBus, 06Services: Set up change-propagation in BetaCluster - https://phabricator.wikimedia.org/T133908#2248571 (10mobrovac) p:05Triage>03High I created [deployment-changeprop.deployment-prep.eqiad.wmflabs](https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-c... [14:04:45] (03PS1) 10Hashar: dib: drop rsyslog [integration/config] - 10https://gerrit.wikimedia.org/r/285952 [14:05:02] (03CR) 10Hashar: [C: 032] dib: drop rsyslog [integration/config] - 10https://gerrit.wikimedia.org/r/285952 (owner: 10Hashar) [14:05:43] (03Merged) 10jenkins-bot: dib: drop rsyslog [integration/config] - 10https://gerrit.wikimedia.org/r/285952 (owner: 10Hashar) [14:07:18] 10Beta-Cluster-Infrastructure, 10EventBus, 06Services, 15User-mobrovac: Set up change-propagation in BetaCluster - https://phabricator.wikimedia.org/T133908#2248602 (10mobrovac) [14:10:05] 10Beta-Cluster-Infrastructure, 10EventBus, 06Services, 15User-mobrovac: Set up change-propagation in BetaCluster - https://phabricator.wikimedia.org/T133908#2248603 (10hashar) [14:18:03] Project browsertests-Wikidata-SmokeTests-linux-firefox build #183: 04STILL FAILING in 20 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox/183/ [14:21:15] Hi [14:22:15] Hello [14:23:08] I would like to run a Puppet catalog test compilation, but I don't have access to the operations-puppet-catalog-compiler job, and so I can't use (puppet) utils/pcc to start such Jenkins task. [14:23:36] hmm [14:23:48] Could someone start a job @ https://integration.wikimedia.org/ci/job//buildWithParameters with 285932 as gerrit change and mw1017.eqiad.wmnet as target? [14:26:11] Er correct URL is https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/buildWithParameters [14:26:29] https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/2620/ [14:26:45] Thanks. [14:27:39] I'd give you permission to run it yourself but I don't know how [14:28:07] https://puppet-compiler.wmflabs.org/2620/ [14:29:39] 05Continuous-Integration-Scaling, 06Labs, 10Labs-Infrastructure: Bump quota of Nodepool instances (contintcloud tenant) - https://phabricator.wikimedia.org/T133911#2248624 (10hashar) [14:32:40] I imagine hashar knows how. [14:32:53] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Replicate open patchsets to diffusion - https://phabricator.wikimedia.org/T89940#2248643 (10mmodell) The more annoying thing is 'added a commit to a task' notification which doesn't have it's own separate setting, so you have to disable 'other maniphest t... [14:33:17] 05Continuous-Integration-Scaling, 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Bump quota of Nodepool instances (contintcloud tenant) - https://phabricator.wikimedia.org/T133911#2248644 (10hashar) Note, as we migrate jobs to run on `contintcloud` we will delete some instances from `integration`tenant... [14:37:54] (03PS1) 10Hashar: dib: create a 'syslog' user on Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/285960 [14:38:04] (03CR) 10Hashar: [C: 032] dib: create a 'syslog' user on Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/285960 (owner: 10Hashar) [14:38:52] (03Merged) 10jenkins-bot: dib: create a 'syslog' user on Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/285960 (owner: 10Hashar) [14:40:50] Project browsertests-Wikidata-WikidataTests-linux-firefox build #182: 04STILL FAILING in 42 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-firefox/182/ [14:42:32] 10Deployment-Systems, 06Release-Engineering-Team, 06Operations, 03Scap3 (Scap3-MediaWiki-MVP): Completely port l10nupdate to scap - https://phabricator.wikimedia.org/T133913#2248683 (10mmodell) [14:42:42] (03CR) 10Hashar: "/Stage[main]/Main/User[syslog]/ensure: created" [integration/config] - 10https://gerrit.wikimedia.org/r/285960 (owner: 10Hashar) [14:43:00] !log Rebuild Nodepool Jessie image. Comes with hhvm [14:43:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:45:10] hashar: :) [14:50:41] (03PS1) 10Hashar: dib: provide hhvm on Trusty [integration/config] - 10https://gerrit.wikimedia.org/r/285964 [14:51:16] (03CR) 10Hashar: [C: 032] dib: provide hhvm on Trusty [integration/config] - 10https://gerrit.wikimedia.org/r/285964 (owner: 10Hashar) [14:52:10] (03Merged) 10jenkins-bot: dib: provide hhvm on Trusty [integration/config] - 10https://gerrit.wikimedia.org/r/285964 (owner: 10Hashar) [15:09:44] 10Deployment-Systems, 05codfw-rollout, 03codfw-rollout-Apr-Jun-2015: Selecting configuration files depending on the realm of the current (bastion) server isn't always sensible - https://phabricator.wikimedia.org/T46889#2248765 (10faidon) 05Open>03Invalid This hasn't been an issue since. [15:15:15] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 03releng-201516-q4, and 2 others: [keyresult] Migrate php composer (Zend and HHVM) CI jobs to Nodepool - https://phabricator.wikimedia.org/T119139#2248786 (10hashar) Did progress on provisioning PHP: | Distro | Zend | H... [15:15:30] 05Continuous-Integration-Scaling, 10OOjs-UI, 07Blocked-on-Operations, 13Patch-For-Review, 07WorkType-NewFunctionality: Provide composer on the nodepool servers so OOjs UI can use it in the npm job - https://phabricator.wikimedia.org/T128092#2248787 (10hashar) Did progress on provisioning PHP: | Distro |... [15:15:54] 05Continuous-Integration-Scaling, 10OOjs-UI, 07Blocked-on-Operations, 13Patch-For-Review, 07WorkType-NewFunctionality: Provide composer on the nodepool servers so OOjs UI can use it in the npm job - https://phabricator.wikimedia.org/T128092#2248788 (10hashar) So in theory if we set `PHP_BIN=hhvm` for the... [15:21:06] 10Deployment-Systems, 06Release-Engineering-Team, 06Operations, 03Scap3 (Scap3-MediaWiki-MVP): Completely port l10nupdate to scap - https://phabricator.wikimedia.org/T133913#2248812 (10Reedy) https://github.com/wikimedia/operations-puppet/blob/production/modules/scap/files/l10nupdate-1 Will still need to... [15:23:50] 05Continuous-Integration-Scaling, 10OOjs-UI, 07Blocked-on-Operations, 13Patch-For-Review, 07WorkType-NewFunctionality: Provide composer on the nodepool servers so OOjs UI can use it in the npm job - https://phabricator.wikimedia.org/T128092#2248816 (10hashar) And composer is available. Will follow up on... [15:23:55] 05Continuous-Integration-Scaling, 10OOjs-UI, 07Blocked-on-Operations, 13Patch-For-Review, 07WorkType-NewFunctionality: Provide composer on the nodepool servers so OOjs UI can use it in the npm job - https://phabricator.wikimedia.org/T128092#2248818 (10hashar) 05Open>03Resolved [15:23:58] 05Continuous-Integration-Scaling, 10OOjs-UI, 13Patch-For-Review, 07WorkType-NewFunctionality: Migrate OOjs UI npm CI job to Nodepool - https://phabricator.wikimedia.org/T128091#2063634 (10hashar) [15:24:00] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 03releng-201516-q4, and 2 others: [keyresult] Migrate php composer (Zend and HHVM) CI jobs to Nodepool - https://phabricator.wikimedia.org/T119139#2248820 (10hashar) [15:24:27] 05Continuous-Integration-Scaling, 10OOjs-UI, 13Patch-For-Review, 07WorkType-NewFunctionality: Migrate OOjs UI npm CI job to Nodepool - https://phabricator.wikimedia.org/T128091#2063634 (10hashar) Composer is available. The npm job runs on Jessie which lacks Zend for now but has HHVM. So for oojs/ui we can... [15:25:53] (03PS1) 10Hashar: oojs/ui npm jobs needs HHVM [integration/config] - 10https://gerrit.wikimedia.org/r/285974 (https://phabricator.wikimedia.org/T128091) [15:26:08] (03CR) 10Hashar: [C: 032] oojs/ui npm jobs needs HHVM [integration/config] - 10https://gerrit.wikimedia.org/r/285974 (https://phabricator.wikimedia.org/T128091) (owner: 10Hashar) [15:27:14] (03Merged) 10jenkins-bot: oojs/ui npm jobs needs HHVM [integration/config] - 10https://gerrit.wikimedia.org/r/285974 (https://phabricator.wikimedia.org/T128091) (owner: 10Hashar) [15:29:05] 05Continuous-Integration-Scaling, 10OOjs-UI, 13Patch-For-Review, 07WorkType-NewFunctionality: Migrate OOjs UI npm CI job to Nodepool - https://phabricator.wikimedia.org/T128091#2248827 (10hashar) Play test area is https://gerrit.wikimedia.org/r/#/c/285972/ Got an experimental job running at https://integr... [15:29:19] I am off [15:30:36] thcipriani, twentyafterfour - there is a swat window coming up, and my deploy-service right has been merged. Plus my deploy window is coming right afterwards. If you have time, lets do the scap3 switch :) [15:39:23] 05Continuous-Integration-Scaling, 10OOjs-UI, 07Blocked-on-Operations, 13Patch-For-Review, 07WorkType-NewFunctionality: Provide composer on the nodepool servers so OOjs UI can use it in the npm job - https://phabricator.wikimedia.org/T128092#2248872 (10Jdforrester-WMF) \o/ [15:46:08] yurik: have you looked at https://doc.wikimedia.org/mw-tools-scap/scap3/quickstart/deployer.html [15:53:35] twentyafterfour, yes, reading through it... painful :) [15:54:00] actually no, sorry, i was looking at the wiki [15:54:25] why not put it on the wiki? [15:54:41] * yurik is not a big fan of multiple doc locations :) [15:54:55] yurik: just got out of a meeting and heading into a new one. The main 2 things left to switching your service to scap3 will be (1) in puppet to use 'deployment' => 'scap3' in the service::node definition and (2) the scap/scap.cfg file inside your repo. But there is a good example here: https://wikitech.wikimedia.org/wiki/Services/Scap_Migration [15:55:29] thcipriani, yeah, that's what i'm writing. Btw, we don't have canary servers for this :( [15:55:37] we should, but its a work in progress [15:55:55] eh, it's not a requirement for scap, it's just a nice-to-have thing [15:56:02] yurik: because it's python generated docs, not wikitext [15:56:11] twentyafterfour: could you run again https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/buildWithParameters 285932 mw1017.eqiad.wmnet please? I've sent to Gerrit a new patchset. [15:56:18] 10MediaWiki-Codesniffer, 03Google-Summer-of-Code-2016: [GSoC 2016 Proposal] Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T130574#2248915 (10EBernhardson) In terms of contact, IRC (freenode) is typically the most direct method. Phabricator also works plenty well for more asy... [15:56:25] twentyafterfour, python should create a huge link at the top that said - "go here" :) [15:56:41] instead it creates awesome animations with the shell :) [15:56:46] you can get rid of the 'server_groups' line in the .cfg as well as the 'canary_dsh_targets' line if you don't have any canary targets. [15:57:36] (server_groups is just 'default' by default, so no need to have the line if it only contains 'default') [15:58:05] tried to do a good "quick"start on it here: https://doc.wikimedia.org/mw-tools-scap/scap3/quickstart/setup.html#scap-cfg [16:00:04] although mobrovac 's documentation on it for services seems like a quicker quickstart :) [16:05:06] thcipriani, - https://gerrit.wikimedia.org/r/285979 - https://gerrit.wikimedia.org/r/285980 ? [16:05:22] * thcipriani looks [16:06:08] thcipriani, there is a weirdness that i'm not sure scap3 will handle ok - tilerator is actually two services - one is regular, and one (called tileratorui) is the same service but with different configuration [16:06:17] they both run from the same dir [16:06:21] on the same servers [16:07:17] ah, you'll probably want to use command checks to do the acutal service restart then: https://doc.wikimedia.org/mw-tools-scap/scap3/quickstart/setup.html#command-checks [16:07:36] we're working on making multiple service restarts easier. [16:08:20] you'll also have to ensure there is a sudoers rule to allow the deploy-service user to restart *both* services (IIRC service::node handles the setup for one service) [16:08:37] thcipriani, at this point i can do manual restart - not a biggie [16:08:55] i would actually prefer to do manual until the automated testing is in place [16:09:15] gotcha. [16:09:54] I'm in a meeting now. I can give those patches some review after this meeting. [16:10:05] or if twentyafterfour is available ^ [16:10:06] sounds good, thanks! [16:11:06] either way - i need to deploy them in an hour (found some bugs in maps), so with some hand holding i could try scap3 [16:15:02] * twentyafterfour takes a look [16:19:18] phab isn't happy this morning, gotten a few Attempt to connect to phuser@m3-master.eqiad.wmnet failed with error #2003: Can't connect to MySQL server on 'm3-master.eqiad.wmnet' (99). [16:19:19] yurik: looks ok to me but I haven't actually done a scap deploy in production so thcipriani is the real expert [16:19:33] ebernhardson: known issue. it should be mostly under control [16:19:46] ebernhardson: we're importing all of the refs/changes/* from gerrit [16:19:49] twentyafterfour, ok, lets do it later today then, after service depl [16:20:58] twentyafterfour: ahh ok, thanks! [16:59:38] RECOVERY - Puppet run on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [17:11:12] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Replicate open patchsets to diffusion - https://phabricator.wikimedia.org/T89940#2249071 (10Ricordisamoa) Also when I list my commits all sorts of intermediate patch sets appear. [17:20:00] General question: Whats the difference, if jenkins marks a build with a green or a blue symbol? [17:23:17] Hi hi - is there a way to go from a Jenkins job configured fully in the UI to it's XML config form? [17:25:46] ooh i might have found it [17:32:37] PROBLEM - Puppet run on deployment-tmh01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:32:39] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Replicate open patchsets to diffusion - https://phabricator.wikimedia.org/T89940#2249117 (10mmodell) >>! In T89940#2249071, @Ricordisamoa wrote: > Also when I list my commits all sorts of intermediate patch sets appear. This was intended. [17:33:58] 10Beta-Cluster-Infrastructure, 06Labs, 10Labs-Infrastructure, 06Operations, and 2 others: Clean up labs graphite datapoints - https://phabricator.wikimedia.org/T111540#2249119 (10Krenair) [17:53:41] 05Continuous-Integration-Scaling, 10OOjs-UI, 07WorkType-NewFunctionality: Migrate OOjs UI npm CI job to Nodepool - https://phabricator.wikimedia.org/T128091#2249195 (10hashar) a:03hashar The job for OOjs UI passed on the Jessie Nodepool instance. It ran `exec:phpGenerateJSPHPForKarma` properly under HHVM. [17:55:55] (03PS1) 10Hashar: [OOJS/ui] Migrate to Node 4.3 / Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/286008 (https://phabricator.wikimedia.org/T128091) [17:56:49] 05Continuous-Integration-Scaling, 10OOjs-UI, 13Patch-For-Review, 07WorkType-NewFunctionality: Migrate OOjs UI npm, npm-run-doc and npm-run-demos CI jobs to Nodepool - https://phabricator.wikimedia.org/T128091#2249207 (10hashar) [18:05:38] yurik: left a comment on the tilerator patch, same comments apply to the other patch. Both of the patches look good to me. Caveat emptor I am unfamiliar with the layout of those services. After that merges, should just be a matter of adding: `deployment => 'scap3',` to service::node inside the kartotherian/tilerator init.pp and running puppet on the target nodes. From there you should be able to [18:05:40] run: deploy from /srv/deployment/{kartotherian,tilerator}/deploy to deploy the respective service to the targets. [18:12:49] RECOVERY - Puppet run on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:13:05] 10Beta-Cluster-Infrastructure, 10EventBus, 06Services, 13Patch-For-Review, 15User-mobrovac: Set up change-propagation in BetaCluster - https://phabricator.wikimedia.org/T133908#2249258 (10mobrovac) Again SSH keys problems in beta. From `deploy-log` on `deployment-tin`: ``` -- Opening log file: '/mnt/srv... [18:14:01] 03Scap3, 10scap, 13Patch-For-Review: scap::target shouldn't allow users to redefine the user's key - https://phabricator.wikimedia.org/T132747#2249260 (10mobrovac) [18:14:03] 10Beta-Cluster-Infrastructure, 10EventBus, 06Services, 13Patch-For-Review, 15User-mobrovac: Set up change-propagation in BetaCluster - https://phabricator.wikimedia.org/T133908#2249259 (10mobrovac) [18:14:30] 10Beta-Cluster-Infrastructure, 03Scap3, 10EventBus, 06Services, and 2 others: Set up change-propagation in BetaCluster - https://phabricator.wikimedia.org/T133908#2248554 (10mobrovac) [18:24:32] (03CR) 10Hashar: [C: 032] [OOJS/ui] Migrate to Node 4.3 / Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/286008 (https://phabricator.wikimedia.org/T128091) (owner: 10Hashar) [18:25:31] (03Merged) 10jenkins-bot: [OOJS/ui] Migrate to Node 4.3 / Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/286008 (https://phabricator.wikimedia.org/T128091) (owner: 10Hashar) [18:41:09] 05Continuous-Integration-Scaling, 10OOjs-UI, 13Patch-For-Review, 07WorkType-NewFunctionality: Migrate OOjs UI npm, npm-run-doc and npm-run-demos CI jobs to Nodepool - https://phabricator.wikimedia.org/T128091#2249413 (10hashar) OOJSnow runs `npm-node-4.3`. Still have to migrate `npm-run-doc` and `npm-run-... [18:49:02] 10Beta-Cluster-Infrastructure, 03Scap3, 10EventBus, 06Services, and 2 others: Set up change-propagation in BetaCluster - https://phabricator.wikimedia.org/T133908#2249420 (10mmodell) @mobrovac: The host key verification thing is a known issue with beta cluster - it doesn't have exported resources so you ha... [18:50:26] mobrovac: ^ if you verify the host key for change-propogation, does that solve it? [18:50:27] 10Beta-Cluster-Infrastructure, 10Staging, 10DBA, 03Collab-Archive-2015-2016, and 2 others: Use External Store on Beta Cluster - https://phabricator.wikimedia.org/T95871#1202118 (10Etonkovidova) - Re-run the @Mattflaschen queries on betalabs for newly created flow-board and wikitext pages - all refer to... [18:50:51] I can't test properly because I'm not a member of service-deploy group [18:53:51] 05Gerrit-Migration, 10Differential: Create useful `.arcconfig`s for migrated repos - https://phabricator.wikimedia.org/T130787#2249423 (10Paladox) @greg and @mmodell according to https://secure.phabricator.com/T10366 they now support setting this globaly by setting it in /etc/arcconfig but it does not support... [18:54:31] twentyafterfour: on beta I setup that group manually, you can just add yourself. [18:54:38] 05Gerrit-Migration, 10Differential: Create useful `.arcconfig`s for migrated repos - https://phabricator.wikimedia.org/T130787#2249425 (10mmodell) I think we still want to have it defined in the repo. [18:58:02] 10Beta-Cluster-Infrastructure, 10Staging, 10DBA, 03Collab-Archive-2015-2016, and 2 others: Use External Store on Beta Cluster - https://phabricator.wikimedia.org/T95871#2249433 (10Etonkovidova) [18:59:06] hashar: Hi! Can I bother you a little? I'm trying to figure out what pages like this are in JJB world - https://integration.wikimedia.org/ci/job/analytics-release-test/m2release/ - Are they just builders? It's defined here - https://github.com/jenkinsci/m2release-plugin/blob/master/src/main/java/org/jvnet/hudson/plugins/m2release/M2ReleaseAction.java [18:59:54] madhuvishy: I am busy deploying mediawiki sorry :/ [19:00:21] hashar: okay no problem - i'll leave comments in the task and ping you - you can look at it whenever! thanks! [19:00:35] which is done now :D [19:00:37] but gotta monitor [19:01:18] madhuvishy: people (including myself) sometime create jobs directly via the web UI and thus they are not in JJB [19:01:30] hashar: Im getting this error [19:01:31] https://integration.wikimedia.org/ci/job/parsoidsvc-source-parse-tool-check/6080/console [19:01:34] oh sorry [19:01:38] 18:35:02 npm ERR! tar.unpack untar error /mnt/home/jenkins-deploy/tmpfs/jenkins-1/npm-12402-69c9156a/registry.npmjs.org/npm/-/npm-2.14.13.tgz [19:02:39] hashar: this is not exactly job creation no? Its kinda like build but a special build that does a release [19:02:47] madhuvishy: yup seems like [19:02:58] I translated the job creation config to jjb [19:03:09] madhuvishy: looks like a different kind of job. I have really zero idea how that works -:-/ [19:03:19] Right [19:03:44] Yeah I don't know either. I don't know where to find the XML version of the config too [19:03:59] paladox: well it is filling the small tmpfs ... [19:04:04] paladox: just recheck I guess [19:04:04] 10Beta-Cluster-Infrastructure, 03Scap3, 10EventBus, 06Services, and 2 others: Set up change-propagation in BetaCluster - https://phabricator.wikimedia.org/T133908#2249453 (10mobrovac) @mmodell I followed @thcipriani's steps outlined in T132666#2207090 but no luck: ``` mobrovac@deployment-tin:/srv/deployme... [19:04:09] Ok [19:04:19] paladox: would be nice to get why it is install npm 2.14.13 ... [19:04:23] I doint know why it is using npm 2.14 when we have npm 2.15 [19:04:34] magic? :( [19:07:04] hashar: Found where it is comming from. [19:07:07] Its comming from npm-shrinkwrap [19:12:09] 06Release-Engineering-Team, 13Patch-For-Review, 05Release: MW-1.27.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T131556#2249496 (10hashar) 05Open>03Resolved Done. Nothing more to say \O/ [19:13:36] 06Release-Engineering-Team, 05Release: MW-1.27.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T133934#2249499 (10Jdforrester-WMF) 05Open>03Resolved a:03hashar [19:15:12] 06Release-Engineering-Team, 05Release: MW-1.27.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T133934#2249502 (10Dereckson) [19:15:14] 06Release-Engineering-Team, 13Patch-For-Review, 05Release: MW-1.27.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T131556#2249504 (10Dereckson) [19:16:35] James_F: I pressed enter on the wrong link, was an invalid duplicate (I wanted to create the .23, but it already exists) [19:16:47] Dereckson: Oh, OK. [19:17:54] 10Beta-Cluster-Infrastructure, 03Scap3, 10EventBus, 06Services, and 2 others: Set up change-propagation in BetaCluster - https://phabricator.wikimedia.org/T133908#2249511 (10thcipriani) So the message: 'Agent admitted failure to sign using the key' is a problem with keyholder. The server is asking to verif... [19:18:21] By the way, if you can check if https://wikitech.wikimedia.org/wiki/Deployments#Week_of_May_2nd looks good, I would be grateful. I added the row adding 7 days to the previous one, and incrementing from 1 the wmfxx in wikitrain but I've no idea of rotation rules or anything else there. [19:25:08] 10Beta-Cluster-Infrastructure, 07Varnish: Beta cluster varnish sets overly broad domain on GeoIP cookie - https://phabricator.wikimedia.org/T133936#2249530 (10bd808) [19:27:48] that one is nice [19:28:08] that is how I discovered templates/varnish/geoip.inc.vcl.erb which is an erb template for VCL that has inlined C ... [19:29:33] 10Beta-Cluster-Infrastructure, 07Varnish: Beta cluster varnish sets overly broad domain on GeoIP cookie - https://phabricator.wikimedia.org/T133936#2249530 (10hashar) Looks like templates/varnish/geoip.inc.vcl.erb  function `geo_get_top_cookie_domain()` which does not take in account the additional level of su... [19:36:46] hashar, we used to have ruby generating a python script that would write out lua code [19:37:07] for the labs dns aliaser [19:38:53] Krenair: that sounds robust as well :) [19:39:13] now we have puppet generating the json which the python script reads to write out lua code [19:40:32] hashar: I was hoping you'd seen something of the sort before! Now I'm concerned :D [19:41:24] 10Beta-Cluster-Infrastructure, 10Staging, 10DBA, 03Collab-Archive-2015-2016, and 2 others: Use External Store on Beta Cluster - https://phabricator.wikimedia.org/T95871#2249594 (10Mattflaschen) The separate blobs1 tables (e.g. testwiki.blobs1, enwiki.blobs1) (which is the actual External Store) are also ge... [19:53:05] 10Beta-Cluster-Infrastructure, 10ContentTranslation-cxserver: Shinken is warning about deployment-cxserver03 refusing connections on port 8080 - https://phabricator.wikimedia.org/T133939#2249611 (10Krenair) [20:06:20] !log Disabling puppet on deployment-restbase0[1-2].deployment-prep : T126629 [20:06:21] T126629: Cassandra 2.1.13 and/or 2.2.5 - https://phabricator.wikimedia.org/T126629 [20:06:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:11:50] PROBLEM - Puppet run on deployment-mediawiki01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:12:51] !log cherry-picking https://gerrit.wikimedia.org/r/#/c/284078/ to deployment-puppetmaster : T126629 [20:12:52] T126629: Cassandra 2.1.13 and/or 2.2.5 - https://phabricator.wikimedia.org/T126629 [20:12:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:14:36] !log Re-enable puppet on deployment-restbase01.deployment-prep, and force a run : T126629 [20:14:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:15:11] T126629: Cassandra 2.1.13 and/or 2.2.5 - https://phabricator.wikimedia.org/T126629 [20:20:00] 10Continuous-Integration-Config, 06Operations, 13Patch-For-Review: Switch CI from jsduck deb package to a gemfile/bundler system - https://phabricator.wikimedia.org/T109005#2249695 (10cscott) @Krinkle +1 Adding `bundle install jsduck` to node's `predoc` target is emphatically *not* the right way to do this. [20:25:12] !log Restarting Cassandra on deployment-restbase01.deployment-prep : T126629 [20:25:13] T126629: Cassandra 2.1.13 and/or 2.2.5 - https://phabricator.wikimedia.org/T126629 [20:25:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:33:31] !log Cassandra on deployment-restbase01.deployment-prep started : T126629 [20:33:32] T126629: Cassandra 2.1.13 and/or 2.2.5 - https://phabricator.wikimedia.org/T126629 [20:33:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:37:06] !log Snapshotting Cassandra tables on deployment-restbase02 : T126629 [20:37:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:37:38] !log Snapshotting Cassandra tables on deployment-restbase02 (snapshot name = 1461875833996) : T126629 [20:37:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:37:49] T126629: Cassandra 2.1.13 and/or 2.2.5 - https://phabricator.wikimedia.org/T126629 [20:38:27] !log Halting Cassandra on deployment-restbase02, masking systemd unit, and upgrading package(s) to 2.2.6 : T126629 [20:38:28] T126629: Cassandra 2.1.13 and/or 2.2.5 - https://phabricator.wikimedia.org/T126629 [20:38:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:41:01] !log Re-enable puppet and force run on deployment-restbase02 : T126629 [20:41:02] T126629: Cassandra 2.1.13 and/or 2.2.5 - https://phabricator.wikimedia.org/T126629 [20:41:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:46:28] !log Starting Cassandra on deployment-restbase02 (now v2.2.6) : T126629 [20:46:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:46:51] T126629: Cassandra 2.1.13 and/or 2.2.5 - https://phabricator.wikimedia.org/T126629 [20:58:44] (03PS1) 10BryanDavis: [labs/toollabs] Add experimental tox-jessie tests [integration/config] - 10https://gerrit.wikimedia.org/r/286032 (https://phabricator.wikimedia.org/T132475) [21:03:08] (03CR) 10Hashar: [C: 032] [labs/toollabs] Add experimental tox-jessie tests [integration/config] - 10https://gerrit.wikimedia.org/r/286032 (https://phabricator.wikimedia.org/T132475) (owner: 10BryanDavis) [21:04:14] (03Merged) 10jenkins-bot: [labs/toollabs] Add experimental tox-jessie tests [integration/config] - 10https://gerrit.wikimedia.org/r/286032 (https://phabricator.wikimedia.org/T132475) (owner: 10BryanDavis) [21:04:35] bd808: done ! [21:04:44] oh nice [21:04:47] bd808: we have the idea to migrate CI deployment to scap3 [21:05:02] bd808: so folks can +2 then scap3 deploy from tin/mira ;-} [21:05:27] that would be neat [21:05:48] I've done a few zuul deploys but its bee a while [21:05:53] my first target will be to migrate zuul/nodepool deployment to scap3 [21:05:58] and use wheels to ship dependencies [21:06:05] cause .deb packaging is no fun : -D [21:06:23] yeah. wheels seem like a neat compromise [21:07:25] bd808: that is what ORES folks did and apparently with success [21:07:35] so gotta look at how they managed wheels / scap3 integration then copy paste [21:07:51] have fun with tox! [21:08:06] it worked -- https://integration.wikimedia.org/ci/job/tox-jessie/7450/console [21:08:41] will the tox job freak out if there is no tox.ini or will it just skip running? [21:09:10] ERROR: toxini file 'tox.ini' not found [21:09:12] freak out ;-:) [21:09:23] k. it stays experimental for now then [21:09:40] so usually we introduce a very basic boilerplate [21:09:43] get it working [21:09:49] merge the CI conf and deploy it [21:09:58] then devs can iterate from there [21:10:14] yeah. I have tox.ini in my feature patch -- https://gerrit.wikimedia.org/r/#/c/285435/ [21:10:27] so when that one is ready to merge [21:10:35] we can craft a CI conf / deploy it [21:10:35] right [21:10:35] then +2 your patch [21:10:38] and it should be merged [21:10:40] or [21:10:51] just merge your patch, and catch up with CI later [21:11:03] yeah, either way will work [21:11:22] I did ugly things there to get tox to work :/ [21:11:22] that becomes more complicated when multiple branches are involved [21:11:50] I trust Merlijn :} [21:12:40] it wasn't tox so much as running doctest and flake8 on scripts with no extension and no module [21:13:36] splitting that repo up into separate things and getting rid of deb for most of it is on my longer term wishlist [21:14:24] oh changedir [21:14:39] well at least it works and tests stuff [21:14:52] yeah. it's a hack that will work for now [21:19:40] bd808: that is good enough. At least you took care of adding tox + the CI part. That is a good step forward. Thanks! [21:19:49] I am heading to bed *wave* [21:21:49] o/ [21:25:52] do we have graphite/grafana in labs? [21:29:47] urandom: yes [21:30:08] urandom: emit to labmon1001.eqiad.wmnet [21:30:19] which is available via hiera('statsd') [21:30:23] hieradata/labs.yaml:statsd: labmon1001.eqiad.wmnet:8125 [21:30:35] 8125? [21:30:39] the port [21:30:39] is that statsd? [21:30:45] i need carbon [21:30:50] usually port 2003 [21:30:56] oh that I have no clue [21:31:08] maybe it is on labmon1001 as well [21:31:13] k [21:31:20] no idea whether it is reachable / open though. Gotta ask #wikimedia-labs [21:31:29] but once you get your bits on that machine [21:31:50] you can reach set it as the datasource from the production grafana ( admin interface is http://grafana-admin.wikimedia.org ) [21:33:05] oh my english is crap [21:33:17] you can reach the labmon1001 source from the production grafana ... [21:33:21] urandom: ^^:} [21:33:29] heading to sleep for real now [21:33:32] hashar: something is listening on labmon1001:2003 [21:33:35] hashar: thanks! [21:33:40] gotta try ;-} [21:33:58] 10Beta-Cluster-Infrastructure, 13Patch-For-Review, 07Varnish: Beta cluster varnish sets overly broad domain on GeoIP cookie - https://phabricator.wikimedia.org/T133936#2249982 (10bd808) 05Open>03Resolved a:03BBlack Verified new header: `Set-Cookie: GeoIP=COUNTRY:REGION:CITY:LAT:LON:v4; Path=/; secure;... [21:38:40] PROBLEM - Host cache-rsync is DOWN: CRITICAL - Host Unreachable (10.68.23.165) [21:51:18] !log Cherry picking operations/puppet refs/changes/78/284078/10 to puppmaster : T126629 [21:51:19] T126629: Cassandra 2.1.13 and/or 2.2.5 - https://phabricator.wikimedia.org/T126629 [21:51:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:52:20] !log Forcing puppet run on deployment-restbase02 : T126629 [21:52:21] T126629: Cassandra 2.1.13 and/or 2.2.5 - https://phabricator.wikimedia.org/T126629 [21:52:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:55:11] !log Snapshotting Cassandra tables on deployment-restbase01 : T126629 [21:55:13] T126629: Cassandra 2.1.13 and/or 2.2.5 - https://phabricator.wikimedia.org/T126629 [21:55:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:55:32] !log Snapshotting Cassandra tables on deployment-restbase01 (name = 1461880519833) : T126629 [21:55:33] T126629: Cassandra 2.1.13 and/or 2.2.5 - https://phabricator.wikimedia.org/T126629 [21:55:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:56:25] !log Stopping Cassandra on deployment-restbase01, upgrading package to 2.2.6, and forcing puppet run : T126629 [21:56:26] T126629: Cassandra 2.1.13 and/or 2.2.5 - https://phabricator.wikimedia.org/T126629 [21:56:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:03:50] !log deployment-restbase01 upgrade to 2.2.6 complete : T126629 [22:03:50] T126629: Cassandra 2.1.13 and/or 2.2.5 - https://phabricator.wikimedia.org/T126629 [22:03:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:43:59] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10Unplanned-Sprint-Work, 07FR-ActiveMQ, and 3 others: Run PHPUnit on PHP-Queue repo - https://phabricator.wikimedia.org/T133574#2250199 (10DStrine) [22:46:02] 10Continuous-Integration-Config, 10Fundraising-Backlog, 07FR-ActiveMQ, 03Fundraising Sprint Hermit Crab Husbandry, and 2 others: Run PHPUnit on PHP-Queue repo - https://phabricator.wikimedia.org/T133574#2250201 (10DStrine)