[00:18:08] 10Beta-Cluster, 5Patch-For-Review: deployment-cache-text02 puppet ganglia-monitor error - https://phabricator.wikimedia.org/T103278#1386395 (10BBlack) a:3thcipriani [00:18:13] 10Beta-Cluster, 5Patch-For-Review: deployment-cache-text02 puppet ganglia-monitor error - https://phabricator.wikimedia.org/T103278#1386396 (10BBlack) 5Open>3Resolved [03:41:37] (03PS1) 10Mattflaschen: PronunciationRecording depends on UploadWizard [integration/config] - 10https://gerrit.wikimedia.org/r/219778 [03:42:29] (03PS2) 10Mattflaschen: PronunciationRecording depends on UploadWizard [integration/config] - 10https://gerrit.wikimedia.org/r/219778 [05:16:30] 10Beta-Cluster, 10MediaWiki-extensions-GettingStarted, 6operations: GettingStarted on Beta Cluster periodically loses its Redis index - https://phabricator.wikimedia.org/T100515#1386438 (10Mattflaschen) [05:36:12] Yippee, build fixed! [05:36:12] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #457: FIXED in 34 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/457/ [05:49:56] Hm, why do I have glusterfs installed on my desktop, was it a requirement of vagrant at some point perhaps? [08:05:21] Project beta-scap-eqiad build #58279: FAILURE in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/58279/ [08:15:32] Yippee, build fixed! [08:15:32] Project beta-scap-eqiad build #58280: FIXED in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/58280/ [08:35:35] (03CR) 10JanZerebecki: "Wikidata is a build that contains Wikibase." [integration/config] - 10https://gerrit.wikimedia.org/r/216630 (owner: 10Paladox) [08:39:08] (03CR) 10Paladox: "Ok should we be testing both wikidata and Wikibase or just wikidata." [integration/config] - 10https://gerrit.wikimedia.org/r/216630 (owner: 10Paladox) [09:22:02] !log upgrading Jenkins gearman plugin from 0.1.1 to latest master (f2024bd). [09:22:05] Logged the message, Master [09:22:18] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #471: ABORTED in 1 min 17 sec: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/471/ [09:31:35] !log cant reach integration-lightslave-jessie-1002 , probably NFS related [09:31:38] Logged the message, Master [09:40:05] !log fixed puppet certificates on integration-lightslave-jessie-1002 by deleting the SSL certs [09:40:08] Logged the message, Master [09:49:35] 10Beta-Cluster, 5Patch-For-Review: deployment-cache-text02 varnish puppet error (Numerical result out of range - log10) - https://phabricator.wikimedia.org/T102570#1386712 (10hashar) 5Open>3Resolved a:3hashar That specific error is now gone :) [09:53:50] 10Beta-Cluster, 10Analytics-EventLogging: puppet agent disabled on beta cluster deployment-eventlogging02.eqiad.wmflabs instance - https://phabricator.wikimedia.org/T96921#1386731 (10hashar) a:3yuvipanda [09:54:14] 10Beta-Cluster, 10Analytics-EventLogging: puppet agent disabled on beta cluster deployment-eventlogging02.eqiad.wmflabs instance - https://phabricator.wikimedia.org/T96921#1386732 (10hashar) 5Open>3Resolved Puppet has been reenabled by @Yuvipanda and it is passing. [10:02:02] 10Beta-Cluster, 6Labs: deployment-bastion: Cannot create /home/l10nupdate/.ssh; parent directory /home/l10nupdate does not exist - https://phabricator.wikimedia.org/T103300#1386751 (10hashar) 3NEW [10:05:53] !log removing puppet lock on deployment-elastic07 ( rm /var/lib/puppet/state/agent_catalog_run.lock ) [10:05:56] Logged the message, Master [10:07:22] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #472: STILL FAILING in 39 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/472/ [10:07:32] 10Beta-Cluster, 6Labs: Things broken by betacluster suddenly being moved off NFS - https://phabricator.wikimedia.org/T102953#1386763 (10hashar) [10:07:35] 10Beta-Cluster, 6Labs: deployment-bastion: Cannot create /home/l10nupdate/.ssh; parent directory /home/l10nupdate does not exist - https://phabricator.wikimedia.org/T103300#1386760 (10hashar) 5Open>3Resolved a:3hashar On deployment-bastion I created the dir: ``` # mkdir /home/l10nupdate # chown l10nupdat... [10:10:40] 10Beta-Cluster, 6Labs: Disable NFS home directories on deployment-prep - https://phabricator.wikimedia.org/T102169#1386764 (10hashar) 5Resolved>3Open Reopening, some instances apparently still rely on NFS :( [10:12:33] 10Beta-Cluster, 10Analytics: Puppet does not pass on beta cluster instance deployment-zookeeper01 - https://phabricator.wikimedia.org/T103301#1386770 (10hashar) 3NEW [10:13:16] 10Beta-Cluster, 10Analytics: Puppet does not pass on beta cluster instance deployment-zookeeper01 - https://phabricator.wikimedia.org/T103301#1386787 (10hashar) ``` # cat /etc/resolv.conf ## THIS FILE IS MANAGED BY PUPPET ## ## source: modules/base/resolv.conf.labs.erb ## from: base::resolving domain eqiad... [10:14:13] 10Beta-Cluster, 10Analytics: Puppet does not pass on beta cluster instance deployment-zookeeper01 - https://phabricator.wikimedia.org/T103301#1386788 (10hashar) a:3hashar And puppet.conf ``` # cat /etc//puppet/puppet.conf # This file is managed by Puppet! [main] logdir = /var/log/puppet vardir = /var/lib/p... [10:17:08] 10Beta-Cluster, 10Analytics: Puppet does not pass on beta cluster instance deployment-zookeeper01 - https://phabricator.wikimedia.org/T103301#1386792 (10hashar) That fixed the original issue: ``` Warning: Unable to fetch my node definition, but the agent run will continue: Warning: getaddrinfo: Name or service... [10:17:47] 10Beta-Cluster, 10Analytics: Puppet does not pass on beta cluster instance deployment-zookeeper01: Could not find class role::analytics::zookeeper::server - https://phabricator.wikimedia.org/T103301#1386801 (10hashar) [10:17:56] 10Beta-Cluster, 10Analytics: Puppet does not pass on beta cluster instance deployment-zookeeper01: Could not find class role::analytics::zookeeper::server - https://phabricator.wikimedia.org/T103301#1386770 (10hashar) a:5hashar>3None [10:20:03] !log enabled puppet agent on deployment-urldownloader [10:20:06] Logged the message, Master [10:25:18] !log fixed puppet.conf on deployment-urldownloader [10:25:22] Logged the message, Master [10:26:46] what a mess [10:28:39] 10Beta-Cluster, 10Analytics: deployment-kafka02 does not pass puppet: Error 400 on SERVER: $brokers[$::fqdn] is :undef, not a hash or array at /etc/puppet/modules/kafka/manifests/server.pp:194 - https://phabricator.wikimedia.org/T103304#1386829 (10hashar) 3NEW [10:28:59] 10Beta-Cluster, 10Analytics: deployment-kafka02 does not pass puppet: Error 400 on SERVER: $brokers[$::fqdn] is :undef, not a hash or array at /etc/puppet/modules/kafka/manifests/server.pp:194 - https://phabricator.wikimedia.org/T103304#1386836 (10hashar) And I rebooted the instance to get rid of the NFS shares. [10:29:09] !log rebooted deployment-kafka02 to get rid of /home NFS share [10:29:12] Logged the message, Master [10:36:43] 10Beta-Cluster, 10Analytics: deployment-kafka02 does not pass puppet: Error 400 on SERVER: $brokers[$::fqdn] is :undef, not a hash or array at /etc/puppet/modules/kafka/manifests/server.pp:194 - https://phabricator.wikimedia.org/T103304#1386847 (10hashar) Removed the class `role::analytics::kafka::server` to b... [10:48:03] 10Continuous-Integration-Infrastructure, 10Gerrit-Migration, 3releng-201516-q1: Prototype CI integration with Differential - https://phabricator.wikimedia.org/T103127#1386874 (10Qgil) [10:48:05] 10Continuous-Integration-Infrastructure, 10Gerrit-Migration: Connect Differential code review with continuous integration - https://phabricator.wikimedia.org/T31#1386873 (10Qgil) [10:52:48] 10Beta-Cluster: en.wikipedia.beta.wmflabs.org create account contains - https://phabricator.wikimedia.org/T100800#1386887 (10hashar) 5Open>3Resolved a:3hashar Resolved somehow. I guess the l10n cache was stalled. [11:07:21] !log fixing puppet on integration-zuul-server [11:07:24] Logged the message, Master [11:13:28] 10Continuous-Integration-Infrastructure, 6Labs: Continuous integration should not depend on labs NFS - https://phabricator.wikimedia.org/T90610#1386948 (10hashar) `salt '*' cmd.run 'grep labstore /etc/fstab'` yields: ``` integration-slave-jessie-1001.integration.eqiad.wmflabs: labstore.svc.eqiad.wmnet:/pro... [11:30:17] 10Continuous-Integration-Infrastructure, 6Labs: Cant ssh to integration-slave-jessie-1001.integration.eqiad.wmflabs - https://phabricator.wikimedia.org/T103312#1387030 (10hashar) 3NEW a:3hashar [11:36:12] 10Continuous-Integration-Infrastructure, 6Labs: Cant ssh to integration-slave-jessie-1001.integration.eqiad.wmflabs - https://phabricator.wikimedia.org/T103312#1387049 (10hashar) Remove the puppet class `role::ci::slave::labs` which prevents puppet from completing. Under /home/ only /home/admin/ exists :( [11:40:48] 10Browser-Tests, 10Continuous-Integration-Infrastructure, 6Release-Engineering, 5Patch-For-Review: Experiment with JJB builder for running a subset of integration MW-Selenium tests - https://phabricator.wikimedia.org/T103039#1387053 (10hashar) [11:50:45] 10Continuous-Integration-Infrastructure, 6operations: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#1387086 (10hashar) [11:57:31] 10Continuous-Integration-Infrastructure, 6operations: Jessie does not have libmemcached10 - https://phabricator.wikimedia.org/T103315#1387102 (10hashar) 3NEW [11:58:58] 10Continuous-Integration-Infrastructure, 6operations: Jessie does not have libmemcached10 - https://phabricator.wikimedia.org/T103315#1387102 (10hashar) [12:01:33] 10Continuous-Integration-Infrastructure, 6operations: Jessie does not have libvips15 - https://phabricator.wikimedia.org/T103322#1387163 (10hashar) 3NEW [12:03:08] 10Continuous-Integration-Infrastructure, 6operations: Investigate usage of ttf-ubuntu-font-familly which is not available on Jessie - https://phabricator.wikimedia.org/T103325#1387187 (10hashar) 3NEW [12:06:24] 10Continuous-Integration-Infrastructure, 6operations: Investigate usage of ttf-ubuntu-font-familly which is not available on Jessie - https://phabricator.wikimedia.org/T103325#1387251 (10hashar) [12:06:42] 10Continuous-Integration-Infrastructure, 6operations: Investigate usage of ttf-ubuntu-font-familly which is not available on Jessie - https://phabricator.wikimedia.org/T103325#1387187 (10hashar) [12:06:44] 10Continuous-Integration-Infrastructure, 6operations: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#1177707 (10hashar) [12:09:04] 10Continuous-Integration-Infrastructure, 6operations: Investigate impact of switching from ffmpeg to libav (ffmpeg is not in Jessie) - https://phabricator.wikimedia.org/T103335#1387291 (10hashar) 3NEW [12:09:32] 10Continuous-Integration-Infrastructure, 6operations: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#1387311 (10faidon) [12:09:34] 10Continuous-Integration-Infrastructure, 6operations, 5Patch-For-Review: Jessie does not have libvips15 - https://phabricator.wikimedia.org/T103322#1387308 (10faidon) 5Open>3Resolved a:3faidon The manifests were wrong to hardcode specific package names for libraries. [12:09:41] 10Continuous-Integration-Infrastructure, 6operations: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#1177707 (10faidon) [12:09:43] 10Continuous-Integration-Infrastructure, 6operations, 5Patch-For-Review: Jessie does not have libmemcached10 - https://phabricator.wikimedia.org/T103315#1387312 (10faidon) 5Open>3Resolved a:3faidon The manifests were wrong to hardcode specific package names for libraries. [12:09:56] 6Release-Engineering, 10Gather, 6Mobile-Web, 10MobileFrontend, 7Epic: [EPIC] Create a formal release process for MobileFrontend/Gather - https://phabricator.wikimedia.org/T100296#1387323 (10Jhernandez) Thanks for keeping the conversation going. That seems like a nice summary @greg. If everything goes we... [12:10:12] 10Continuous-Integration-Infrastructure, 6operations: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#1387325 (10hashar) I have created sub tasks, the fonts related ones being under T102623. Result is: * {T103315} * {T103322} * {T102623} ** {T103325... [12:15:28] 6Release-Engineering, 10Wikimedia-Git-or-Gerrit: Create a Gerrit group to easily add reviews to CI related changes - https://phabricator.wikimedia.org/T100319#1387369 (10hashar) 5Open>3Resolved a:3hashar The Gerrit group that yields permission for CI changes is `integration` https://gerrit.wikimedia.org/... [12:39:40] 10Continuous-Integration-Infrastructure, 6operations, 7Blocked-on-Operations: Backport libjsch-java to Precise - https://phabricator.wikimedia.org/T103342#1387394 (10hashar) 3NEW a:3hashar [12:40:31] 10Beta-Cluster, 10Continuous-Integration-Infrastructure: Reenable ssh MAC/KEX hardening on beta cluster and integration labs project - https://phabricator.wikimedia.org/T100518#1387411 (10hashar) [12:40:34] 10Continuous-Integration-Infrastructure, 6operations, 5Patch-For-Review: Jenkins master / client ssh connection fails due to missing ssh algorithm - https://phabricator.wikimedia.org/T100509#1314411 (10hashar) [12:40:36] 10Continuous-Integration-Infrastructure, 7Jenkins, 7Upstream: Jenkins jar should ship with a more recent jsch java lib version to support hardened algorithm - https://phabricator.wikimedia.org/T100517#1387406 (10hashar) 5Open>3Invalid a:3hashar From the github pull request, jsch is an external dependen... [12:40:51] 10Beta-Cluster, 10Continuous-Integration-Infrastructure: Reenable ssh MAC/KEX hardening on beta cluster and integration labs project - https://phabricator.wikimedia.org/T100518#1314596 (10hashar) [12:40:52] 10Continuous-Integration-Infrastructure, 6operations, 7Blocked-on-Operations: Backport libjsch-java to Precise - https://phabricator.wikimedia.org/T103342#1387394 (10hashar) [12:41:13] 10Beta-Cluster, 10Continuous-Integration-Infrastructure: Reenable ssh MAC/KEX hardening on beta cluster and integration labs project - https://phabricator.wikimedia.org/T100518#1314596 (10hashar) Blocked on {T103342} [12:42:34] 10Continuous-Integration-Infrastructure, 6operations, 7Blocked-on-Operations: Backport libjsch-java to Precise - https://phabricator.wikimedia.org/T103342#1387418 (10hashar) a:5hashar>3None [12:44:28] 6Release-Engineering, 7Epic, 7Tracking: Provide pre-merge reports on patchsets (tracking) - https://phabricator.wikimedia.org/T101542#1387427 (10hashar) [12:44:52] 6Release-Engineering, 7Epic, 7Tracking: Provide pre-merge reports on patchsets (tracking) - https://phabricator.wikimedia.org/T101542#1342089 (10hashar) Moved it to #releng workboard since that is epic. The sub tasks are on the CI boards though. [12:47:50] 10Continuous-Integration-Infrastructure, 6operations, 7Jenkins: Please refresh Jenkins package on apt.wikimedia.org to 1.609.1 - https://phabricator.wikimedia.org/T103343#1387431 (10hashar) 3NEW [12:48:20] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Create CI slaves using Debian Jessie (tracking) - https://phabricator.wikimedia.org/T94836#1387440 (10hashar) p:5High>3Normal [13:04:00] Yippee, build fixed! [13:04:00] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #692: FIXED in 31 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/692/ [13:08:30] 10Continuous-Integration-Infrastructure, 6operations, 7Blocked-on-Operations: Backport libjsch-java to Precise - https://phabricator.wikimedia.org/T103342#1387481 (10hashar) @MoritzMuehlenhoff Thanks! lanthanum is the other CI Precise slave so we can get the package upgraded there. I am happy to see the o... [13:09:19] 10Continuous-Integration-Infrastructure, 6operations, 7Blocked-on-Operations: Backport libjsch-java to Precise - https://phabricator.wikimedia.org/T103342#1387482 (10hashar) [13:17:27] !log activated firejail service containment for graphoid, citoid and mathoid in deployment-sca [13:17:28] 10Continuous-Integration-Infrastructure, 6Release-Engineering, 7Jenkins, 7Upstream: [upstream] Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597#1387499 (10hashar) I have up... [13:17:30] Logged the message, Master [13:18:10] 10Continuous-Integration-Infrastructure, 5Patch-For-Review, 7Upstream, 7Zuul: zuul-cloner does not support Ref events - https://phabricator.wikimedia.org/T76003#1387508 (10hashar) a:5hashar>3None I am not working this anymore. [13:19:06] 5Continuous-Integration-Isolation: Create a Jessie image with diskimage-builder suitable for nodepool - https://phabricator.wikimedia.org/T102878#1387510 (10hashar) I had an instance created for Jessie using diskimage-builder. It even booted in labs! Now I have to puppetize it the magic recipe. [13:23:00] 5Continuous-Integration-Isolation, 7Jenkins, 7Upstream: Nodepool can't create slaves on Jenkins - https://phabricator.wikimedia.org/T103120#1387512 (10hashar) 5Open>3declined a:3hashar It works just fine when given a credentials-id which is our use case. Leaving upstream bug open but there is no need... [13:28:29] (03CR) 10JanZerebecki: [C: 032] Add Wikidata dependance on scribunto and cldr [integration/config] - 10https://gerrit.wikimedia.org/r/216630 (owner: 10Paladox) [13:29:08] (03CR) 10JanZerebecki: "Deployed to Jenkins." [integration/config] - 10https://gerrit.wikimedia.org/r/216630 (owner: 10Paladox) [13:30:47] (03Merged) 10jenkins-bot: Add Wikidata dependance on scribunto and cldr [integration/config] - 10https://gerrit.wikimedia.org/r/216630 (owner: 10Paladox) [13:32:12] (03CR) 10JanZerebecki: "Both Wikidata https://integration.wikimedia.org/ci/job/mwext-Wikidata-testextension-zend/ and Wikibase https://integration.wikimedia.org/c" [integration/config] - 10https://gerrit.wikimedia.org/r/216630 (owner: 10Paladox) [13:38:18] 10Deployment-Systems, 10RESTBase: RESTBase deployment process - https://phabricator.wikimedia.org/T103344#1387528 (10Krenair) [13:40:11] Krenair: hehe, i'm still editing the ticket ^^ (pressed enter by mistake) [13:45:47] (03CR) 10Paladox: "Ok thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/216630 (owner: 10Paladox) [13:45:57] 10Deployment-Systems, 10RESTBase: RESTBase deployment process - https://phabricator.wikimedia.org/T103344#1387535 (10mobrovac) [13:52:22] 10Beta-Cluster, 10Analytics: Puppet does not pass on beta cluster instance deployment-zookeeper01: Could not find class role::analytics::zookeeper::server - https://phabricator.wikimedia.org/T103301#1387537 (10Ottomata) 5Open>3Resolved a:3Ottomata Zookeeper classes have moved out of analytics:: context.... [13:55:58] 10Deployment-Systems, 10RESTBase: RESTBase deployment process - https://phabricator.wikimedia.org/T103344#1387540 (10mobrovac) [13:58:31] (03CR) 10JanZerebecki: [C: 04-1] "You probably want to change this in zuul/ext_dependencies.py instead. Otherwise it creates new additional jobs with {name}-{ext-name}-test" [integration/config] - 10https://gerrit.wikimedia.org/r/219778 (owner: 10Mattflaschen) [13:58:52] 10Beta-Cluster, 10Analytics: deployment-kafka02 does not pass puppet: Error 400 on SERVER: $brokers[$::fqdn] is :undef, not a hash or array at /etc/puppet/modules/kafka/manifests/server.pp:194 - https://phabricator.wikimedia.org/T103304#1387541 (10Ottomata) 5Open>3Resolved a:3Ottomata Edited hieradata to... [14:10:48] 6Release-Engineering, 10Continuous-Integration-Config, 7HHVM, 5Patch-For-Review: Jenkins: Implement hhvm based voting jobs for mediawiki and extensions (tracking) - https://phabricator.wikimedia.org/T75521#1387568 (10JanZerebecki) [14:13:08] 10Deployment-Systems, 10RESTBase: RESTBase deployment process - https://phabricator.wikimedia.org/T103344#1387573 (10GWicke) [14:19:54] 6Release-Engineering, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, 6Multimedia, and 6 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1387583 (10demon) >>! In T102566#1385611, @Nemo_bis wrote: >> Surely we have to draw the line of where the oldest... [14:20:16] 10Deployment-Systems, 10RESTBase: RESTBase deployment process - https://phabricator.wikimedia.org/T103344#1387586 (10GWicke) [14:20:50] 10Deployment-Systems, 10RESTBase: RESTBase deployment process - https://phabricator.wikimedia.org/T103344#1387474 (10GWicke) [14:21:36] 10Deployment-Systems, 10RESTBase: RESTBase deployment process - https://phabricator.wikimedia.org/T103344#1387474 (10GWicke) [14:30:54] !log Reenable sshd MAC/KEX hardening on beta by cherry picking https://gerrit.wikimedia.org/r/#/c/219828/ [14:30:58] Logged the message, Master [14:31:47] Project browsertests-Wikidata-WikidataTests-linux-firefox-sauce build #264: ABORTED in 9 min 46 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-firefox-sauce/264/ [14:32:31] !log restarting Jenkins [14:32:34] Logged the message, Master [14:33:39] Project browsertests-Wikidata-WikidataTests-linux-chrome-sauce build #61: ABORTED in 4 min 38 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-chrome-sauce/61/ [14:38:30] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce build #649: ABORTED in 19 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce/649/ [14:38:31] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #291: ABORTED in 21 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/291/ [14:49:02] 10Deployment-Systems, 10RESTBase, 6Services: RESTBase deployment process - https://phabricator.wikimedia.org/T103344#1387666 (10mobrovac) [14:52:27] 10Deployment-Systems, 10RESTBase, 6Services: RESTBase deployment process - https://phabricator.wikimedia.org/T103344#1387681 (10mobrovac) [14:52:29] 10Deployment-Systems, 6Release-Engineering, 10RESTBase, 6Services, 3releng-201516-q1: Create new RESTBase deploy method (tracking) - https://phabricator.wikimedia.org/T102667#1387680 (10mobrovac) [14:56:17] 10Beta-Cluster, 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Reenable ssh MAC/KEX hardening on beta cluster and integration labs project - https://phabricator.wikimedia.org/T100518#1387694 (10hashar) [14:56:20] 10Continuous-Integration-Infrastructure, 6operations, 5Patch-For-Review: Jenkins master / client ssh connection fails due to missing ssh algorithm - https://phabricator.wikimedia.org/T100509#1387695 (10hashar) [14:56:21] 10Continuous-Integration-Infrastructure, 6operations, 7Blocked-on-Operations: Backport libjsch-java to Precise - https://phabricator.wikimedia.org/T103342#1387691 (10hashar) 5Open>3Resolved a:3hashar I have reenable the MAC/KEX on beta cluster but then: ``` fatal: no matching mac found: client: hmac-sh... [14:58:13] !log disabled sshd MAC/KEX hardening on beta (was https://gerrit.wikimedia.org/r/#/c/219828/ ) [14:58:16] Logged the message, Master [15:01:03] 10Beta-Cluster, 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Jenkins trilead-ssh2 doesn't support our MAC/KEX algorithms - https://phabricator.wikimedia.org/T103351#1387714 (10hashar) 3NEW [15:01:14] 10Beta-Cluster, 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Jenkins trilead-ssh2 doesn't support our MAC/KEX algorithms - https://phabricator.wikimedia.org/T103351#1387714 (10hashar) [15:01:16] 10Continuous-Integration-Infrastructure, 6operations, 5Patch-For-Review: Jenkins master / client ssh connection fails due to missing ssh algorithm - https://phabricator.wikimedia.org/T100509#1387723 (10hashar) [15:04:18] 10Deployment-Systems, 10RESTBase, 6Services: RESTBase deployment process - https://phabricator.wikimedia.org/T103344#1387738 (10GWicke) Our [current Ansible-based solution](https://wikitech.wikimedia.org/wiki/RESTBase) handles the most important parts of this (rolling deploys, health checks, automatic aborts... [15:06:46] 10Deployment-Systems, 10RESTBase, 6Services: RESTBase deployment process - https://phabricator.wikimedia.org/T103344#1387745 (10GWicke) [15:06:56] 10Continuous-Integration-Infrastructure, 6operations, 7Blocked-on-Operations: Backport libjsch-java to Precise - https://phabricator.wikimedia.org/T103342#1387748 (10hashar) The trielad-ssh2 version is not the Debian package: ``` $ apt-cache search trilead libjenkins-trilead-ssh2-java - Trilead SSH2 implemen... [15:07:11] 10Continuous-Integration-Infrastructure: Backport libjsch-java to Precise - https://phabricator.wikimedia.org/T103342#1387758 (10hashar) [15:07:46] 10Beta-Cluster, 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Jenkins trilead-ssh2 doesn't support our MAC/KEX algorithms - https://phabricator.wikimedia.org/T103351#1387714 (10hashar) The trielad-ssh2 version is not the Debian package: ``` $ apt-cache search trilead libjenkins-trilead-ssh2-jav... [15:07:57] 10Continuous-Integration-Infrastructure, 6operations: Backport libjsch-java to Precise - https://phabricator.wikimedia.org/T103342#1387394 (10hashar) [15:09:10] 10Beta-Cluster, 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Jenkins trilead-ssh2 doesn't support our MAC/KEX algorithms - https://phabricator.wikimedia.org/T103351#1387775 (10hashar) [15:10:00] 10Beta-Cluster, 6Mobile-Web, 5Patch-For-Review, 5WMF-deploy-2015-06-16_(1.26wmf10), 7Wikimedia-log-errors: Visiting sign up form shows 500 - https://phabricator.wikimedia.org/T103107#1387798 (10Krenair) (You can probably reverse your QA skip test patch now, btw. NFS is back up with old files, so the capt... [15:18:31] legoktm: Did you merge Roan's fix for ForrestBot but not deploy it or something? [15:22:56] 10Deployment-Systems: trebuchet should expect salt APIs to be asynchronous and poll for status updates from all minions - https://phabricator.wikimedia.org/T103013#1387867 (10ArielGlenn) https://gerrit.wikimedia.org/r/#/c/219841/ this would make the report output a little more comprehensible. More to be done. [15:26:19] 10Beta-Cluster, 10Continuous-Integration-Infrastructure, 7Jenkins, 5Patch-For-Review, 7Upstream: Jenkins trilead-ssh2 doesn't support our MAC/KEX algorithms - https://phabricator.wikimedia.org/T103351#1387872 (10hashar) So Jenkins fork/port is hosted at https://github.com/jenkinsci/trilead-ssh2 Upstream... [15:29:00] Yippee, build fixed! [15:29:01] Project browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #165: FIXED in 9 min 6 sec: https://integration.wikimedia.org/ci/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/165/ [15:37:30] 10Deployment-Systems: Provide mechanism to add/remove minions from git-deploy - https://phabricator.wikimedia.org/T74319#1387912 (10ArielGlenn) https://gerrit.wikimedia.org/r/#/c/219845/ to see what's in redis (untested). more to do. [15:59:50] 6Release-Engineering, 10Wikidata, 10Wikimedia-General-or-Unknown, 6operations: Wikidata and Wikiversity logo 404ing on wikimedia.org - https://phabricator.wikimedia.org/T103296#1388011 (10Joe) [16:32:27] (03PS2) 10BryanDavis: Add HHVM restart support [tools/scap] - 10https://gerrit.wikimedia.org/r/219751 (https://phabricator.wikimedia.org/T103008) [16:32:44] (03CR) 10jenkins-bot: [V: 04-1] Add HHVM restart support [tools/scap] - 10https://gerrit.wikimedia.org/r/219751 (https://phabricator.wikimedia.org/T103008) (owner: 10BryanDavis) [16:34:34] 10Deployment-Systems: Provide mechanism to add/remove minions from git-deploy - https://phabricator.wikimedia.org/T74319#1388213 (10ArielGlenn) https://gerrit.wikimedia.org/r/#/c/219852/ to remove a minion. note none of this is tested yet. [16:40:12] James_F: uh, I deployed it...I think. [16:45:53] (03PS3) 10BryanDavis: Add HHVM restart support [tools/scap] - 10https://gerrit.wikimedia.org/r/219751 (https://phabricator.wikimedia.org/T103008) [16:46:01] 10Deployment-Systems, 10RESTBase, 6Services: RESTBase deployment process - https://phabricator.wikimedia.org/T103344#1388291 (10mobrovac) [16:47:09] (03CR) 10BryanDavis: Add HHVM restart support (034 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/219751 (https://phabricator.wikimedia.org/T103008) (owner: 10BryanDavis) [16:55:02] ostriches: is there any way to view the history for the Www.wikimedia.org_template template on www.wikimedia.org? Either that template changed or...I have no idea. See https://phabricator.wikimedia.org/T103296 [16:57:29] git log has no idea about docroot/wwwportal/static which makes it seem like something else must've changed. [16:57:46] thcipriani: it's on metawiki [16:57:59] legoktm: thanks [16:58:09] thcipriani: https://meta.wikimedia.org/w/index.php?title=Www.wikimedia.org_template&action=history [16:59:18] hmm this might be our issue: https://meta.wikimedia.org/w/index.php?title=Www.wikimedia.org_template&diff=12374381&oldid=12369391 [16:59:25] legoktm: thanks again! [16:59:31] np [16:59:46] what's wrong? [17:00:14] weird [17:00:17] * legoktm looks at bug [17:01:29] 6Release-Engineering, 10Wikidata, 10Wikimedia-General-or-Unknown, 6operations: Wikidata and Wikiversity logo 404ing on wikimedia.org - https://phabricator.wikimedia.org/T103296#1388338 (10thcipriani) It doesn't look like the a static link directory existed in mediawiki-config at `docroot/wwwportal` before,... [17:04:05] 10Continuous-Integration-Infrastructure: Request Jenkins shell access for account "sniedzielski" - https://phabricator.wikimedia.org/T103192#1384173 (10Niedzielski) This doesn't seem to quite be working yet: ssh sniedzielski@integration-slave-trusty-1015.eqiad.wmflabs Permission denied (publickey). I also... [17:04:22] 10Continuous-Integration-Infrastructure: Request Jenkins shell access for account "sniedzielski" - https://phabricator.wikimedia.org/T103192#1388345 (10Niedzielski) 5Resolved>3Open [17:59:00] Yippee, build fixed! [17:59:01] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #578: FIXED in 1 min 0 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/578/ [18:13:37] 5Continuous-Integration-Isolation, 6operations: Figure out fine sudo rules for the nodepool service - https://phabricator.wikimedia.org/T102281#1388700 (10chasemp) [18:15:41] 5Continuous-Integration-Isolation, 6operations: Figure out fine sudo rules for the nodepool service - https://phabricator.wikimedia.org/T102281#1388702 (10chasemp) Let's not add #Ops-Access-Requests here as it flags this as a real needs review access request. Post this ticket can make some with whatever the o... [18:20:31] legoktm: Well, it's not working… [18:23:37] 10Browser-Tests: Wikidata browser test jobs fail since upgrading to mediawiki-selenium 1.2.1 - https://phabricator.wikimedia.org/T102458#1388732 (10dduvall) The PR has been merged. I'll upgrade our Cucumber dependency for the next release of MW-Selenium. https://github.com/cucumber/cucumber-ruby/pull/872 [18:26:06] (03PS1) 10Dduvall: Upgrade cucumber dependency for fix to JUnit logger [selenium] - 10https://gerrit.wikimedia.org/r/219882 (https://phabricator.wikimedia.org/T102458) [18:28:37] Looking for some gerrit help--our master branch contains two patches that weren't reviewed or merged: https://git.wikimedia.org/log/wikimedia%2Ffundraising%2Fcrm/refs%2Fheads%2Fmaster [18:28:58] The last two commits are the bad ones [18:30:21] awight: can you file a task, plz? [18:30:29] sure! [18:31:08] ty! [18:33:36] (03CR) 10Krinkle: [C: 032] Add MobileFrontend js docs to doc.wikimedia.org [integration/docroot] - 10https://gerrit.wikimedia.org/r/219591 (https://phabricator.wikimedia.org/T74794) (owner: 10Florianschmidtwelzow) [18:33:43] awight: are you referring to the merge commits ? [18:33:50] (03Merged) 10jenkins-bot: Add MobileFrontend js docs to doc.wikimedia.org [integration/docroot] - 10https://gerrit.wikimedia.org/r/219591 (https://phabricator.wikimedia.org/T74794) (owner: 10Florianschmidtwelzow) [18:34:51] 6Release-Engineering: Unreviewed commits merged in gerrit - https://phabricator.wikimedia.org/T103396#1388774 (10awight) 3NEW [18:35:15] 10Continuous-Integration-Infrastructure, 3Mobile-Web: Jenkins: Set up jsduck test and publish jobs for MobileFrontend - https://phabricator.wikimedia.org/T66374#1388784 (10Krinkle) [18:35:22] 6Release-Engineering, 10Wikimedia-Git-or-Gerrit: Unreviewed commits merged in gerrit - https://phabricator.wikimedia.org/T103396#1388787 (10greg) [18:35:29] 10Continuous-Integration-Infrastructure, 3Mobile-Web: Jenkins: Set up jsduck test and publish jobs for MobileFrontend - https://phabricator.wikimedia.org/T66374#698417 (10Krinkle) [18:36:17] hasharAway: No, these are regular commits. There's a little more info in that card. [18:38:17] 10Continuous-Integration-Infrastructure, 6Release-Engineering, 10Wikimedia-Git-or-Gerrit: Unreviewed commits merged in gerrit - https://phabricator.wikimedia.org/T103396#1388804 (10hashar) The "merge failed" message reported back in Gerrit is issued when zuul-merger on gallium.wikimedia.org is unable to merg... [18:38:27] awight: might have a look at that after a meeting :D [18:38:37] for now, need to finish to repair my wife bike [18:40:56] Ah, if only relationships could be fixed as easily as a bike :p [18:49:56] Who runs meetbot? I want to configure it to point to a different channel… [18:50:06] (I know it's down now, but…) [18:52:28] James_F: according to tools, hasharAway (in this channel anyway) [18:53:14] JohnFLewis: Aha, nice. Thanks. [18:53:23] * James_F will make it hasharAway's problem, then. [18:54:26] :( [19:02:33] (03CR) 10Ori.livneh: [C: 04-1] "Thanks a ton for this. Looks good, couple of minor points." (035 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/219751 (https://phabricator.wikimedia.org/T103008) (owner: 10BryanDavis) [19:20:53] 10Continuous-Integration-Infrastructure, 7Jenkins, 5Patch-For-Review, 7Upstream: Jenkins trilead-ssh2 doesn't support our MAC/KEX algorithms - https://phabricator.wikimedia.org/T103351#1389003 (10hashar) [19:22:46] 10Beta-Cluster: [[wikitech:]] in Beta should not link to non-existant wikitech.wikimedia.deployment.wmflabs.org - https://phabricator.wikimedia.org/T103248#1389012 (10hashar) [19:23:17] 10Beta-Cluster: [[wikitech:]] in Beta should not link to non-existant wikitech.wikimedia.deployment.wmflabs.org - https://phabricator.wikimedia.org/T103248#1389019 (10hashar) p:5Triage>3Low [19:34:19] 10Beta-Cluster: Enable the possibility to block users by the AbuseFilter at the deployment wiki at the beta cluster - https://phabricator.wikimedia.org/T103060#1389046 (10hashar) Do you mean the AbuseFilter is not enabled on beta deploymentwiki? deploymentwiki is the AbuseFilter central wiki, having the global... [19:35:47] 10Beta-Cluster, 6Labs: Things broken by betacluster suddenly being moved off NFS - https://phabricator.wikimedia.org/T102953#1389055 (10thcipriani) [19:35:50] 10Beta-Cluster, 6Labs: Beta Cluster uploads (new and viewing existing files/thumbnails, including captchas) broken due to WMF Labs NFS outage - https://phabricator.wikimedia.org/T102963#1389052 (10thcipriani) 5Open>3Resolved a:3thcipriani All tests passing now [19:40:42] 10Beta-Cluster, 6Labs: Things broken by betacluster suddenly being moved off NFS - https://phabricator.wikimedia.org/T102953#1389060 (10hashar) 5Open>3Resolved a:3hashar Seems all NFS related breakages have been fixed now. [19:43:01] 10Beta-Cluster, 6Labs: Things broken by betacluster suddenly being moved off NFS - https://phabricator.wikimedia.org/T102953#1389076 (10Krenair) By NFS being up, sure... Shouldn't we be trying to make it not depend on NFS? [19:43:47] 10Beta-Cluster, 6Labs, 6operations, 7Monitoring: Setup (simple) catchpoint monitoring for betacluster - https://phabricator.wikimedia.org/T97865#1389081 (10hashar) @yuvipanda can you handle replicating one of the catchpoint probe to hit en.wikipedia.beta.wmflabs.org ? Whatever is done for the production e... [19:44:16] 10Beta-Cluster, 6Labs, 6operations, 7Monitoring: Setup (simple) catchpoint monitoring for enwiki betacluster just like production - https://phabricator.wikimedia.org/T97865#1389084 (10hashar) [19:47:09] 10Beta-Cluster, 6Labs: Things broken by betacluster suddenly being moved off NFS - https://phabricator.wikimedia.org/T102953#1389089 (10Krenair) 5Resolved>3Open This is not resolved unless nothing in beta is expected to break next time NFS does. [19:49:27] 5Continuous-Integration-Isolation, 10Ops-Access-Requests, 6operations: Get Dan Duvall TEMP root to labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T102133#1389095 (10chasemp) We talked about this in last weeks ops meeting. We are fine with Mr. Duvall in this context. [19:49:48] 10Beta-Cluster, 10Continuous-Integration-Infrastructure: Ensure /srv/deployment/integration/slave-scripts is latest master on deployment-bastion - https://phabricator.wikimedia.org/T97324#1389097 (10hashar) The git::clone() under contint::slave::labs::common has been setup originally because we could not use T... [19:50:10] 10Beta-Cluster, 10Continuous-Integration-Infrastructure: Ensure /srv/deployment/integration/slave-scripts is latest master on deployment-bastion - https://phabricator.wikimedia.org/T97324#1389099 (10hashar) p:5Triage>3Normal [19:52:42] 10Beta-Cluster, 6Labs: Things broken by betacluster suddenly being moved off NFS - https://phabricator.wikimedia.org/T102953#1389115 (10hashar) Thanks @krenair. Well NFS is gone from the deployment-prep. Its only use now are the upload/thumbnails. We need to migrate to swift which is T64835 blocking {T84950}. [19:53:30] 10Beta-Cluster, 10MediaWiki-File-management, 6Multimedia: Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking) - https://phabricator.wikimedia.org/T84950#934455 (10hashar) [19:53:33] 10Beta-Cluster, 6Labs: Things broken by betacluster suddenly being moved off NFS - https://phabricator.wikimedia.org/T102953#1389119 (10hashar) [19:53:37] (03PS4) 10BryanDavis: Add HHVM restart support [tools/scap] - 10https://gerrit.wikimedia.org/r/219751 (https://phabricator.wikimedia.org/T103008) [19:53:51] 10Beta-Cluster, 6Labs: Beta Cluster uploads (new and viewing existing files/thumbnails, including captchas) broken due to WMF Labs NFS outage - https://phabricator.wikimedia.org/T102963#1389122 (10Krenair) By NFS being up, yes... This task should probably remain open until it no longer depends on NFS or is clo... [19:55:10] (03CR) 10BryanDavis: Add HHVM restart support (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/219751 (https://phabricator.wikimedia.org/T103008) (owner: 10BryanDavis) [19:56:45] 10Beta-Cluster, 6Labs: Things broken by betacluster suddenly being moved off NFS - https://phabricator.wikimedia.org/T102953#1389137 (10hashar) a:5hashar>3None [19:59:37] 5Continuous-Integration-Isolation, 6operations: Remove hashar and dduvall root access on to be installed labnodepool1001 - https://phabricator.wikimedia.org/T95303#1389157 (10chasemp) [20:06:55] 5Continuous-Integration-Isolation, 10Ops-Access-Requests, 6operations: Get Dan Duvall TEMP root to labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T102133#1389184 (10chasemp) 5Open>3Resolved a:3chasemp {F687} [20:08:14] 5Continuous-Integration-Isolation, 10Ops-Access-Requests, 6operations: Get Dan Duvall TEMP root to labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T102133#1389198 (10chasemp) https://gerrit.wikimedia.org/r/#/c/219959/ [20:17:20] 6Release-Engineering, 10Wikidata, 10Wikimedia-General-or-Unknown, 6operations: Wikidata and Wikiversity logo 404ing on wikimedia.org - https://phabricator.wikimedia.org/T103296#1389234 (10thcipriani) so it looks like if you revert to this revision: https://meta.wikimedia.org/w/index.php?title=Www.wikimedia... [20:25:13] 6Release-Engineering, 10Wikidata, 10Wikimedia-General-or-Unknown, 6operations: Wikidata and Wikiversity logo 404ing on wikimedia.org - https://phabricator.wikimedia.org/T103296#1389268 (10Krenair) >>! In T103296#1389234, @thcipriani wrote: > so it looks like if you revert to this revision: > https://meta.w... [20:26:52] 10Continuous-Integration-Infrastructure, 6Release-Engineering, 10Wikimedia-Git-or-Gerrit: Unreviewed commits merged in gerrit - https://phabricator.wikimedia.org/T103396#1389274 (10hashar) The git log --oneline --graph looks like: ``` * 15c080c - (HEAD, gerrit/master, gerrit/HEAD, master) Write test for con... [20:36:24] (03CR) 10Ori.livneh: "Tiny tiny point, +2 otherwise" (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/219751 (https://phabricator.wikimedia.org/T103008) (owner: 10BryanDavis) [20:38:08] 10Beta-Cluster: Enable the possibility to block users by the AbuseFilter at the deployment wiki at the beta cluster - https://phabricator.wikimedia.org/T103060#1389333 (10Luke081515) @hashar I mean, that the possibility "Block User/IP" is not enabled at deployment wiki, just on meta, where I can't use it as an g... [20:42:11] 5Continuous-Integration-Isolation, 10Ops-Access-Requests, 6operations: Get Dan Duvall TEMP root to labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T102133#1389388 (10dduvall) Thanks! [20:42:49] (03PS5) 10BryanDavis: Add HHVM restart support [tools/scap] - 10https://gerrit.wikimedia.org/r/219751 (https://phabricator.wikimedia.org/T103008) [20:43:36] (03CR) 10BryanDavis: Add HHVM restart support (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/219751 (https://phabricator.wikimedia.org/T103008) (owner: 10BryanDavis) [20:45:57] (03CR) 10Ori.livneh: [C: 032] Add HHVM restart support [tools/scap] - 10https://gerrit.wikimedia.org/r/219751 (https://phabricator.wikimedia.org/T103008) (owner: 10BryanDavis) [20:46:17] (03Merged) 10jenkins-bot: Add HHVM restart support [tools/scap] - 10https://gerrit.wikimedia.org/r/219751 (https://phabricator.wikimedia.org/T103008) (owner: 10BryanDavis) [20:47:27] 10Deployment-Systems, 6operations, 7HHVM, 5Patch-For-Review, 15User-Bd808-Test: Scap should restart HHVM - https://phabricator.wikimedia.org/T103008#1389415 (10chasemp) thanks @bd808 [20:48:42] (03PS3) 10BryanDavis: Move dsh group file names to config [tools/scap] - 10https://gerrit.wikimedia.org/r/219752 [20:49:04] (03CR) 10BryanDavis: "PS3 was a manual rebase" [tools/scap] - 10https://gerrit.wikimedia.org/r/219752 (owner: 10BryanDavis) [20:56:02] 10Beta-Cluster, 3Mobile-Web, 5Patch-For-Review, 5WMF-deploy-2015-06-23_(1.26wmf11), 7Wikimedia-log-errors: Visiting sign up form shows 500 - https://phabricator.wikimedia.org/T103107#1389447 (10Jdforrester-WMF) [20:56:04] (03CR) 10Ori.livneh: [C: 032] Move dsh group file names to config [tools/scap] - 10https://gerrit.wikimedia.org/r/219752 (owner: 10BryanDavis) [20:56:24] (03Merged) 10jenkins-bot: Move dsh group file names to config [tools/scap] - 10https://gerrit.wikimedia.org/r/219752 (owner: 10BryanDavis) [21:00:10] twentyafterfour: is https://secure.phabricator.com/D13098 live on our phabricator instance? [21:02:04] legoktm: he's on vacation but I think yes, you can verify by looking at the current tag tho if you need [21:02:10] Can anyone help me understand this? [21:02:11] Error: /Stage[main]/Mediawiki::Scap/Package[scap]/ensure: change from 62d5cb2b0185fba2f35bd631e2bc57cf7a78d978 to latest failed: Could not get latest version: 403 Forbidden [21:02:25] Apparently ‘scap’ is a package that is deployed via puppet with trebuchet? [21:02:34] /me ’s head spins [21:02:45] it is indeed [21:02:48] is that the silver puppet error? [21:02:56] where is it blowing up? [21:03:20] bd808: methinks wikitech [21:03:22] silver aka wikitech [21:03:27] Krenair: yes [21:03:40] the 403 from tin's trebuchet http server has been seen before I think [21:03:46] and has been transient [21:03:55] chasemp: ah. well based on https://phabricator.wikimedia.org/conduit/method/project.create/ it looks like it has been deployed \o/ [21:03:55] This one has been happening all day [21:04:26] bit more than a day: https://phabricator.wikimedia.org/T103138 [21:04:54] legoktm: mukunda is officially off today, btw [21:04:57] andrewbogott: is silver trying to hit tin via ipv6 maybe? I think we saw that a few weeks ago somewhere else [21:05:06] I heard ;) [21:05:09] legoktm: oh, you were already told that :) [21:05:39] bd808: I can’t think of why that would’ve started breaking yesterday [21:05:53] or, whatever, Friday [21:05:56] (wikitech-static looks better, btw) [21:06:09] icinga says it broke 12 days ago [21:06:17] 12d 2h 35m 33s [21:06:27] Krenair: oh... [21:06:32] :P [21:06:33] well, that could be when ipv6 was set up then :( [21:06:58] 2015-06-10? [21:07:53] when tin got ipv6? [21:07:58] Or wikitech [21:08:19] https://github.com/wikimedia/operations-puppet/commit/650d1f75410f59cd7a8ce454e29db4559dcb2b20 etc. [21:08:38] I think that's it. Looking at tin:/etc/apache2/sites-enabled/50-deployment.conf and the allow there [21:09:40] bblack added a bunch of v6 subnets -- https://github.com/wikimedia/operations-puppet/commit/f6f2b47f70883b36f21a19807e790139da071b9e [21:09:53] but possibly not the one that silver is on [21:10:22] which looks to be 2620:0:861:2::/64 ? [21:11:32] how did you get that? dig -6 silver.wikimedia.org hangs for me [21:14:36] I'm on silver and looking at `ip addr` [21:14:48] but now I wonder how it ever worked... [21:14:56] yeah [21:15:04] public1-b-eqiad isn't in the allowed subnets list [21:15:04] maybe that restriction on tin was recently added [21:15:40] ah... '208.80.152.0/22' is [21:16:58] so yeah you need to add the ipv6 range to $::mw_appserver_networks [21:17:45] Yeah, I’ll do that as soon as this eternal fetch finishes [21:18:04] * bd808 shakes fist in general direction of gerrit [21:32:45] 6Release-Engineering, 6Labs, 6operations, 10wikitech.wikimedia.org, 5Patch-For-Review: silver / scap - Could not get latest version: 403 Forbidden - https://phabricator.wikimedia.org/T103138#1389685 (10Andrew) 5Open>3Resolved a:3Andrew Fixed by attached patch. [21:37:38] !log Updated scap to 81b7c14 (Move dsh group file names to config) [21:37:41] Logged the message, Master [21:44:18] is deployment-videoscaler01.eqiad.wmflabs being busted a known thing? Looks like maybe it has puppet problems? [21:44:28] I can't ssh in (permission denied) [21:44:45] and a new python package for scap is missing there [21:47:53] !log scap emitting soft failures due to missing python-netifaces on deployment-videoscaler01; should be fixed by a current puppet run [21:47:56] Logged the message, Master [21:52:24] andrewbogott, ^ [21:52:56] deployment-videoscaler01 is news to me, probably something that got disconnected during NFS failures and didn't come back because of a broken puppet. [21:53:15] I’ll look… [21:56:01] bd808: Notice: Skipping run of Puppet configuration client; administratively disabled (Reason: 'reason not specified'); [21:56:13] So, that would do it [21:56:16] sigh. [21:56:16] Shall I re-enable? [21:56:23] I think we had another case of this with another host [21:56:51] A case of ‘when you disable or break puppet the instance rots and loses contact with things’? [21:57:02] yes. sadness [21:57:06] e.g. https://phabricator.wikimedia.org/T96921 [21:57:26] Since we don't know why I'd vote to see it re-enabled and puppet forced [21:59:32] done, it seems to be shaping up [22:03:20] bd808: can you log in now? [22:04:04] I could log in [22:04:08] Although I didn't test beforehand, so.. [22:04:48] (03PS3) 10Mattflaschen: PronunciationRecording depends on UploadWizard [integration/config] - 10https://gerrit.wikimedia.org/r/219778 [22:05:15] 6Release-Engineering: Update mediawiki-tools-release to use new API continuation - https://phabricator.wikimedia.org/T102866#1389885 (10Legoktm) https://github.com/legoktm/harej-bots/commit/b557511ecddc78a9fc056dad52fab99c52513325 is my patch to botclasses.php to add &rawcontinue everywhere. [22:05:34] (03CR) 10Mattflaschen: "Thanks, done." [integration/config] - 10https://gerrit.wikimedia.org/r/219778 (owner: 10Mattflaschen) [22:05:58] andrewbogott: yes, I can log in now. Thanks! [22:06:20] cool [22:14:38] (03PS4) 10Legoktm: Add 'npm' for more extensions for banana-checker and jsonlint [integration/config] - 10https://gerrit.wikimedia.org/r/219603 [22:16:09] (03CR) 10Legoktm: [C: 032] Add 'npm' for more extensions for banana-checker and jsonlint [integration/config] - 10https://gerrit.wikimedia.org/r/219603 (owner: 10Legoktm) [22:23:33] (03Merged) 10jenkins-bot: Add 'npm' for more extensions for banana-checker and jsonlint [integration/config] - 10https://gerrit.wikimedia.org/r/219603 (owner: 10Legoktm) [22:23:53] !log deploying https://gerrit.wikimedia.org/r/219603 [22:23:56] Logged the message, Master [22:26:07] 10Continuous-Integration-Infrastructure, 10pywikibot-core: Travis-CI access for pywikibot project - https://phabricator.wikimedia.org/T103434#1390010 (10jayvdb) 3NEW [22:28:48] 10Deployment-Systems, 6Release-Engineering: Update mediawiki-tools-release to use new API continuation - https://phabricator.wikimedia.org/T102866#1390020 (10greg) [22:51:10] 10Browser-Tests, 6Collaboration-Team, 10Echo: 503 on Echo tests - https://phabricator.wikimedia.org/T103437#1390077 (10Mattflaschen) 3NEW [22:53:22] 10Browser-Tests, 6Collaboration-Team, 10Echo: 503 on Echo tests - https://phabricator.wikimedia.org/T103437#1390090 (10dduvall) p:5Triage>3High [23:13:58] 6Release-Engineering, 10Wikidata, 10Wikimedia-General-or-Unknown, 6operations: Wikidata and Wikiversity logo 404ing on wikimedia.org - https://phabricator.wikimedia.org/T103296#1390148 (10Addshore) Well... https://meta.wikimedia.org/w/index.php?title=Www.wikimedia.org_template&diff=next&oldid=12369391 The... [23:31:20] (03PS1) 10Legoktm: Add 'npm' for more extensions to run banana-checker & jsonlint [integration/config] - 10https://gerrit.wikimedia.org/r/220020 [23:31:40] (03PS2) 10Legoktm: Add 'npm' for more extensions to run banana-checker & jsonlint [integration/config] - 10https://gerrit.wikimedia.org/r/220020 [23:49:44] 10Browser-Tests, 6Collaboration-Team, 10Echo: 503 on Echo tests - https://phabricator.wikimedia.org/T103437#1390276 (10dduvall) The 503 errors seem intermittent so I'd chalk this up to Beta Labs misbehaving. @demon, @thcipriani, do you see any upticks in 503s on beta around the time of the failed build? [23:50:53] 10Deployment-Systems, 6operations: Corrupt /srv/deployment/scap/scap checkouts on WMF prod cluster - https://phabricator.wikimedia.org/T103441#1390287 (10bd808) 3NEW [23:51:05] 10Deployment-Systems, 6operations: Corrupt /srv/deployment/scap/scap checkouts on WMF prod cluster - https://phabricator.wikimedia.org/T103441#1390294 (10bd808) p:5Triage>3High [23:55:00] 10Deployment-Systems, 6Release-Engineering, 6operations: Corrupt /srv/deployment/scap/scap checkouts on WMF prod cluster - https://phabricator.wikimedia.org/T103441#1390295 (10chasemp) [23:59:48] 6Release-Engineering, 10Gather, 10MobileFrontend, 7Epic, and 2 others: [EPIC] Encourage developers to increase code coverage - https://phabricator.wikimedia.org/T100294#1390310 (10Jdlrobson) It seems a cheap way to do this would be to use the existing `grunt qunit:cov` command and then reject patches that...