[00:08:58] 10Continuous-Integration, 6Labs, 10OOjs, 10Wikimedia-Labs-Infrastructure, 6operations: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118444 (10Krinkle) ``` [16:03 CET] krinkle at KrinkleMac in ~ $ host saucelabs.com saucel... [00:22:49] 10Continuous-Integration, 6Labs, 10OOjs, 10Wikimedia-Labs-Infrastructure, 6operations: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118458 (10Dzahn) I found the root cause to be this option in /etc/resolv.conf ``` option... [00:28:59] 10Continuous-Integration, 6Labs, 10OOjs, 10Wikimedia-Labs-Infrastructure, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118464 (10coren) ndots:2 is necessary for something else, the actual bug is that the dnsma... [00:49:38] 10Continuous-Integration, 6Labs, 10OOjs, 10Wikimedia-Labs-Infrastructure, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118500 (10scfc) The error doesn't seem to lie with dnsmasq. On `tools-login`, the look-up... [00:55:56] 10Continuous-Integration, 6Labs, 10OOjs, 10Wikimedia-Labs-Infrastructure, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118537 (10coren) No, it's just that the Precise libresolv seems to be a little more forgiv... [01:03:15] 10Continuous-Integration, 6Labs, 10OOjs, 10Wikimedia-Labs-Infrastructure, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118547 (10coren) To wit: ``` marc@tools-trusty:~$ host notexist Host notexist.eqiad.wmflab... [01:03:43] 10Continuous-Integration, 6Labs, 10OOjs, 10Wikimedia-Labs-Infrastructure, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118555 (10Krinkle) I suspect this error got introduced when I switched over the CI pool fr... [01:10:00] 10Continuous-Integration, 6Labs, 10OOjs, 10Wikimedia-Labs-Infrastructure, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118565 (10Krinkle) [01:10:01] 10Continuous-Integration: Pool new integration-slave14xx instances and delete old ones - https://phabricator.wikimedia.org/T91524#1118564 (10Krinkle) [01:10:02] 10Continuous-Integration, 6operations, 7Puppet: Puppet (silently) fails to setup apache on some integration-slave14xx instances - https://phabricator.wikimedia.org/T91832#1118566 (10Krinkle) [01:10:44] 24 Missing "texvccheck" executable. Please see math/README to configure. in /srv/mediawiki/php-1.25wmf20/extensions/Math/MathInputCheckTexvc.php on line 64 [01:10:44] sigh [01:18:16] it's present on mw1001 but not various others (mw1114, mw1018, mw1019) [01:22:43] apparently these are "canary appservers" [01:36:27] 10Continuous-Integration, 6Labs, 10OOjs, 10Wikimedia-Labs-Infrastructure, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118579 (10scfc) @Coren: But there are you querying the Labs server, and (I think) dnsmasq... [01:39:24] 10Continuous-Integration, 6Labs, 10OOjs, 10Wikimedia-Labs-Infrastructure, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118580 (10scfc) And: ``` scfc@tools-login:~$ dig @10.68.16.1 tools-login.eqiad.wmflabs ;... [01:41:03] 10Continuous-Integration, 6Labs, 10OOjs, 10Wikimedia-Labs-Infrastructure, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118589 (10scfc) (Or a host name that does not exist.) [01:41:38] 10Continuous-Integration, 6Labs, 10OOjs, 10Wikimedia-Labs-Infrastructure, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118590 (10coren) >>! In T92351#1118579, @scfc wrote: > So (from a distance) it appears as... [01:42:00] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:44:18] krenair@mw1018:~$ which texvc [01:44:18] /usr/bin/texvc [01:44:18] krenair@mw1018:~$ which texvccheck [01:44:18] krenair@mw1018:~$ [01:44:19] weird [01:45:01] Ohh... [01:45:12] That's not good. [01:45:13] krenair@mw1001:~$ dpkg -s mediawiki-math-texvc | grep Version [01:45:13] Version: 2:1.0+git20140526-1 [01:45:20] krenair@mw1018:~$ dpkg -s mediawiki-math-texvc | grep Version [01:45:20] Version: 2:1.0+git20120528-8 [01:51:14] https://phabricator.wikimedia.org/T92707 [01:58:29] 3 timed out after 0.10000000000000001 seconds when connecting to fluorine.eqiad.wmnet [110]: Connection timed out [01:58:33] from the log... on fluorine :/ [02:00:17] 10Continuous-Integration, 6Labs, 10OOjs, 10Wikimedia-Labs-Infrastructure, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118610 (10scfc) http://www.linuxquestions.org/questions/linux-networking-3/powerdns-servfa... [02:00:31] 10Continuous-Integration, 5Patch-For-Review: Remove integration/kss.git - https://phabricator.wikimedia.org/T92482#1118612 (10Krinkle) ``` Mar 14 01:56:08 integration-slave1401 puppet-agent[18663]: (/Stage[main]/Contint::Slave-scripts/Git::Clone[jenkins CI kss]/Exec[git_pull_jenkins CI kss]/returns) fatal: rem... [02:03:46] 10Continuous-Integration, 5Patch-For-Review: Remove integration/kss.git - https://phabricator.wikimedia.org/T92482#1118618 (10Krinkle) >>! In T92482#1118612, @Krinkle wrote: > ``` > Mar 14 01:56:08 integration-slave1401 puppet-agent[18663]: (/Stage[main]/Contint::Slave-scripts/Git::Clone[jenkins CI kss]/Exec[g... [02:14:29] 10Continuous-Integration, 6Labs, 6operations: Evaluate options to make puppet errors more visible - https://phabricator.wikimedia.org/T92710#1118636 (10Krinkle) 3NEW [02:19:45] 10Continuous-Integration, 6Labs, 6operations: Evaluate options to make puppet errors more visible - https://phabricator.wikimedia.org/T92710#1118645 (10scfc) What do you mean by "puppet failures and random regressions" in this case? [02:25:16] 6Release-Engineering, 10MediaWiki-General-or-Unknown, 15User-Bd808-Test: Create a minimal backport of PSR-3 logging to MediaWiki 1.23 LTS - https://phabricator.wikimedia.org/T91653#1118659 (10Krenair) [02:38:39] 10Continuous-Integration, 6Labs, 6operations: Evaluate options to make puppet errors more visible - https://phabricator.wikimedia.org/T92710#1118670 (10Krinkle) >>! In T92710#1118645, @scfc wrote: > What do you mean by "puppet failures and random regressions" in this case? Integrity errors or quality issues... [02:43:43] (03CR) 10Krinkle: [C: 031] "Nice" [integration/config] - 10https://gerrit.wikimedia.org/r/196540 (owner: 10Legoktm) [03:42:16] (03CR) 10Legoktm: [C: 032] Create generic 'npm' job [integration/config] - 10https://gerrit.wikimedia.org/r/196540 (owner: 10Legoktm) [03:45:24] Yippee, build fixed! [03:45:24] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #369: FIXED in 38 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/369/ [03:48:00] (03Merged) 10jenkins-bot: Create generic 'npm' job [integration/config] - 10https://gerrit.wikimedia.org/r/196540 (owner: 10Legoktm) [03:51:59] !log deployed https://gerrit.wikimedia.org/r/196540 [03:52:05] Logged the message, Master [04:14:54] Project beta-scap-eqiad build #45068: FAILURE in 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/45068/ [04:21:15] 10Continuous-Integration, 6Labs, 10OOjs, 10Wikimedia-Labs-Infrastructure, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118720 (10Dzahn) about workarounds: /etc/nsswitch says: hosts: files dns so to... [04:24:42] Yippee, build fixed! [04:24:42] Project beta-scap-eqiad build #45069: FIXED in 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/45069/ [05:02:11] Yippee, build fixed! [05:02:12] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #175: FIXED in 55 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/175/ [05:23:38] (03PS1) 10Legoktm: Replace 'mwext-{name}-npm' jobs with generic 'npm' job [integration/config] - 10https://gerrit.wikimedia.org/r/196743 [05:40:50] (03PS1) 10Legoktm: Use tox-flake8 for UploadWizard [integration/config] - 10https://gerrit.wikimedia.org/r/196745 [05:51:08] (03PS1) 10Legoktm: zuul: Create "npm" template and use it [integration/config] - 10https://gerrit.wikimedia.org/r/196746 [06:27:53] Yippee, build fixed! [06:27:53] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #364: FIXED in 42 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/364/ [06:31:03] 10Beta-Cluster, 10Staging, 6Labs, 5Patch-For-Review: Provide option to autosign puppet certs for self hosted puppetmasters - https://phabricator.wikimedia.org/T92606#1118769 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Bam, done! [06:31:04] 10Beta-Cluster: Make beta cluster puppet master to auto sign client keys - https://phabricator.wikimedia.org/T75767#1118772 (10yuvipanda) [06:31:58] 10Beta-Cluster: Configure all deployment-prep instances to use local salt and puppet master by default - https://phabricator.wikimedia.org/T64795#1118773 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Done in https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep now [06:34:39] 10Beta-Cluster: Configure all deployment-prep instances to use local salt and puppet master by default - https://phabricator.wikimedia.org/T64795#1118780 (10yuvipanda) [06:34:40] 10Beta-Cluster: Make beta cluster puppet master to auto sign client keys - https://phabricator.wikimedia.org/T75767#1118777 (10yuvipanda) 5Open>3Resolved a:3yuvipanda I've enabled it. [06:35:05] 10Beta-Cluster: Make beta cluster salt master to auto accept minions - https://phabricator.wikimedia.org/T75766#1118782 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Done now via the puppetmaster::autosigner class [06:35:06] 10Beta-Cluster: Configure all deployment-prep instances to use local salt and puppet master by default - https://phabricator.wikimedia.org/T64795#665949 (10yuvipanda) [06:52:45] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:57:33] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 29355 bytes in 0.562 second response time [07:21:35] 10Beta-Cluster, 6operations, 7Puppet: Minimize differences between beta and production (Tracking) - https://phabricator.wikimedia.org/T87220#1118800 (10yuvipanda) [07:21:35] 10Beta-Cluster, 6operations, 5Patch-For-Review, 7Puppet: Use keyholder for deploy key management - https://phabricator.wikimedia.org/T92367#1118798 (10yuvipanda) 5Open>3Resolved Done now. Anyone who is a member of the deployment-prep project can now run scap without having to sudo to anything. [07:22:39] 10Beta-Cluster, 5Patch-For-Review, 7Puppet: Unify labs and prod roles for role::deployment::deployment_servers - https://phabricator.wikimedia.org/T86885#1118801 (10yuvipanda) 5Open>3Resolved a:3yuvipanda DDDONE. That was painful :) See I3e947637b49ce2a94128e21db35798a49e8d45e8 [07:22:40] 10Beta-Cluster, 5Patch-For-Review, 7Puppet, 7Tracking: Remove all ::beta roles in puppet - https://phabricator.wikimedia.org/T86644#1118804 (10yuvipanda) [07:25:03] 10Beta-Cluster, 10Staging, 6operations, 7Puppet: Move scap puppet code into a module - https://phabricator.wikimedia.org/T87221#1118807 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Done. beta/scap is gone. [07:25:04] 10Beta-Cluster, 6operations, 7Puppet: Minimize differences between beta and production (Tracking) - https://phabricator.wikimedia.org/T87220#1118810 (10yuvipanda) [07:30:18] 10Beta-Cluster, 6Release-Engineering: Beta cluster unable to create thumbnail for WebM video - https://phabricator.wikimedia.org/T90332#1118812 (10yuvipanda) I don't think the videoscaler ever worked, tbh :) [07:36:16] 10Beta-Cluster, 6Release-Engineering: Convert Beta Cluster specific puppet configs to use Hiera (tracking) - https://phabricator.wikimedia.org/T451#1118814 (10yuvipanda) 5Open>3Invalid a:3yuvipanda Closing since we don't actually seem to be using this for tracking... [09:54:33] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:13:42] 10Continuous-Integration, 10MediaWiki-Codesniffer, 10Possible-Tech-Projects, 3Google-Summer-of-Code-2015, 3Outreachy-Round-10: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1118876 (10Albertcoder) Hi! I am also interested in this project and I willing to take th... [10:14:05] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:18:57] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 48438 bytes in 0.909 second response time [10:59:36] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [12:20:14] PROBLEM - App Server bits response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:02] RECOVERY - App Server bits response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 3895 bytes in 0.002 second response time [12:53:39] 10Beta-Cluster, 6Release-Engineering: Beta cluster unable to create thumbnail for WebM video - https://phabricator.wikimedia.org/T90332#1118920 (10hashar) >>! In T90332#1118812, @yuvipanda wrote: > I don't think the videoscaler ever worked, tbh :) It definitely did. When some people worked on TimedMediaHandle... [13:24:03] 10Continuous-Integration, 10MediaWiki-Codesniffer, 10Possible-Tech-Projects, 3Google-Summer-of-Code-2015, 3Outreachy-Round-10: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1118948 (10devunt) [13:24:04] 10Continuous-Integration, 10Incident-20150312-whitespace, 6MediaWiki-Core-Team, 6operations: add a check for whitespace before leading 10Continuous-Integration, 6Labs, 10OOjs, 10Wikimedia-Labs-Infrastructure, and 2 others: Jenkins failing with "Error: GET https://saucelabs.com: Couldn't resolve host name." - https://phabricator.wikimedia.org/T92351#1118965 (10scfc) Has someone looked at whether there is an SOA record in LDAP? If that is... [14:01:57] 10Continuous-Integration, 6Labs, 6operations: Evaluate options to make puppet errors more visible - https://phabricator.wikimedia.org/T92710#1118969 (10scfc) There are two aspects to this: # Whether the Git failure caused Puppet to fail. This seems to have been the case. # Whether the Puppet failure trigge... [17:51:55] (03CR) 10Jforrester: [C: 031] zuul: Create "npm" template and use it [integration/config] - 10https://gerrit.wikimedia.org/r/196746 (owner: 10Legoktm) [19:12:35] 10Continuous-Integration: Migrate Jenkins slaves from Ubuntu Trusty to Debian Jessie - https://phabricator.wikimedia.org/T86728#1119208 (10coren) [20:08:01] 15 error: syntax error, unexpected T_STRING in /srv/mediawiki/php-1.25wmf21/includes/TemplateParser.php(136) : eval()'d code on line 1 [22:44:25] 10Continuous-Integration, 10MediaWiki-Codesniffer, 10Possible-Tech-Projects, 3Google-Summer-of-Code-2015, 3Outreachy-Round-10: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1119322 (10Legoktm) [23:05:55] 10Continuous-Integration, 10MediaWiki-Codesniffer: Consider disabling empty catch body sniff - https://phabricator.wikimedia.org/T54413#1119347 (10Legoktm) 5Open>3Resolved a:3Legoktm @hashar disabled the rule in 9a574f68dde8093f9fc29aca128addf27b561b6d. [23:06:04] 10Continuous-Integration, 10MediaWiki-Codesniffer: Consider disabling empty catch body sniff - https://phabricator.wikimedia.org/T54413#1119351 (10Legoktm) a:5Legoktm>3hashar [23:16:39] 10Continuous-Integration, 10MediaWiki-Codesniffer, 10Possible-Tech-Projects, 3Google-Summer-of-Code-2015, 3Outreachy-Round-10: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1119361 (10Legoktm)