[00:08:20] 10Continuous-Integration-Infrastructure: Grant Zuul deploy rights to Jan Zerebecki - https://phabricator.wikimedia.org/T98865#1300556 (10JanZerebecki) 5Open>3Resolved Yes. Thx. [00:09:46] PROBLEM - Puppet staleness on integration-slave-jessie-1001 is CRITICAL 100.00% of data above the critical threshold [43200.0] [00:13:40] 10Continuous-Integration-Infrastructure, 10MediaWiki-JobQueue, 10MediaWiki-Unit-tests, 10Wikidata, and 2 others: Qunit test always fail in Wikidata Jenkins jobs. - https://phabricator.wikimedia.org/T99775#1300584 (10JanZerebecki) 5Open>3Resolved a:3JanZerebecki That made it work, example: https://int... [02:07:14] PROBLEM - Puppet staleness on deployment-elastic07 is CRITICAL 100.00% of data above the critical threshold [43200.0] [05:24:17] PROBLEM - Content Translation Server on deployment-cxserver03 is CRITICAL: Connection refused [05:39:16] RECOVERY - Content Translation Server on deployment-cxserver03 is OK: HTTP OK: HTTP/1.1 200 OK - 1103 bytes in 0.020 second response time [05:47:08] PROBLEM - Puppet staleness on deployment-restbase02 is CRITICAL 100.00% of data above the critical threshold [43200.0] [05:52:21] PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL 100.00% of data above the critical threshold [43200.0] [06:20:22] 10Continuous-Integration-Infrastructure, 6Mobile-Web, 7Epic: [EPIC] Run browser tests against every patch - https://phabricator.wikimedia.org/T99792#1300787 (10hashar) [06:20:23] 10Browser-Tests, 10Continuous-Integration-Infrastructure, 7Tracking: [project] trigger browser tests from Gerrit (tracking) - https://phabricator.wikimedia.org/T55697#1300788 (10hashar) [06:25:57] 10Continuous-Integration-Infrastructure, 6Mobile-Web, 7Epic: [EPIC] Run browser tests against every patch - https://phabricator.wikimedia.org/T99792#1300798 (10hashar) We experimented them during winter 2013-2014 and were not very happy with the outcome. In short they take a while to run and most tests ends... [06:28:40] 10Browser-Tests, 10Continuous-Integration-Infrastructure, 7Tracking: [project] trigger browser tests from Gerrit (tracking) - https://phabricator.wikimedia.org/T55697#1300800 (10hashar) [06:36:48] RECOVERY - Free space - all mounts on deployment-bastion is OK All targets OK [08:20:49] 10Continuous-Integration-Infrastructure, 6Scrum-of-Scrums, 6operations, 7Blocked-on-Operations, 5Patch-For-Review: Jenkins is using php-luasandbox 1.9-1 for zend unit tests; precise should be upgraded to 2.0-8 or equivalent - https://phabricator.wikimedia.org/T88798#1300839 (10akosiaris) 5Open>3Resolv... [08:34:17] 10Continuous-Integration-Infrastructure, 10MediaWiki-JobQueue, 10MediaWiki-Unit-tests, 10Wikidata, and 2 others: Qunit test always fail in Wikidata Jenkins jobs. - https://phabricator.wikimedia.org/T99775#1300857 (10daniel) That did it, thanks a lot, guys! [08:44:46] 6Release-Engineering: MobileFrontend should run Gather tests as well - https://phabricator.wikimedia.org/T99877#1300859 (10Jdlrobson) 3NEW a:3Jdlrobson [08:49:02] 6Release-Engineering, 10Gather, 10Gather Sprint Help!, 7Jenkins: Gather can't merge code due to issue with karma (jenkins mwext-qunit job) - https://phabricator.wikimedia.org/T99686#1300866 (10Jdlrobson) 5Open>3Resolved This is now fixed. Summary: a seemingly harmless change in MobileFrontend broke Gat... [09:16:15] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL 33.33% of data above the critical threshold [0.0] [09:22:21] RECOVERY - Puppet staleness on deployment-restbase01 is OK Less than 1.00% above the threshold [3600.0] [09:27:18] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL 33.33% of data above the critical threshold [0.0] [09:32:06] RECOVERY - Puppet staleness on deployment-restbase02 is OK Less than 1.00% above the threshold [3600.0] [09:37:16] RECOVERY - Puppet failure on deployment-restbase02 is OK Less than 1.00% above the threshold [0.0] [09:41:12] RECOVERY - Puppet failure on deployment-restbase01 is OK Less than 1.00% above the threshold [0.0] [09:44:47] PROBLEM - Puppet failure on deployment-salt is CRITICAL 50.00% of data above the critical threshold [0.0] [09:57:53] 10Continuous-Integration-Infrastructure, 6Mobile-Web, 7Epic: [EPIC] Run browser tests against every patch - https://phabricator.wikimedia.org/T99792#1300980 (10Jdlrobson) In mobile we could explore categorising browser tests by ResourceLoader modules. If we were to generate dependency trees for RL modules wh... [10:04:46] RECOVERY - Puppet failure on deployment-salt is OK Less than 1.00% above the threshold [0.0] [10:05:18] PROBLEM - Content Translation Server on deployment-cxserver03 is CRITICAL: Connection refused [10:20:17] RECOVERY - Content Translation Server on deployment-cxserver03 is OK: HTTP OK: HTTP/1.1 200 OK - 1103 bytes in 0.018 second response time [10:23:39] 6Release-Engineering: No /static/1.26wmf7 tree -- mediawiki.org missing "Powered by MediaWiki" logo and ResourceLoader ?debug=1 mode broken - https://phabricator.wikimedia.org/T99886#1301033 (10Spage) 3NEW a:3mmodell [10:23:49] 10Continuous-Integration-Infrastructure, 6Scrum-of-Scrums, 6operations, 7Blocked-on-Operations, 5Patch-For-Review: Jenkins is using php-luasandbox 1.9-1 for zend unit tests; precise should be upgraded to 2.0-8 or equivalent - https://phabricator.wikimedia.org/T88798#1301041 (10Anomie) 5Resolved>3Open... [10:39:01] hey release engineers, you should fix https://phabricator.wikimedia.org/T99886 . Looking at https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys it seems 1.26wmf6 doesn't match the instructions (but it works) [10:41:30] I wonder if this is related to "bits-related changes" mentioned in operations/mediawiki-config log [10:43:32] beta labs has resources like "Powered by MediaWiki" coming from /static/master/resources..., not /static/1.26wmf7/master/resources [10:45:11] scratch that, of course beta runs master, not a specific branch. Hmm [11:02:18] 6Release-Engineering: No /static/1.26wmf7 tree -- mediawiki.org missing "Powered by MediaWiki" logo and ResourceLoader ?debug=1 mode broken - https://phabricator.wikimedia.org/T99886#1301086 (10mmodell) @spage: ok I'm on it [11:11:43] 6Release-Engineering: No /static/1.26wmf7 tree -- mediawiki.org missing "Powered by MediaWiki" logo and ResourceLoader ?debug=1 mode broken - https://phabricator.wikimedia.org/T99886#1301093 (10mmodell) Ok I just noticed something else that looks out of order: ``` /srv/mediawiki-staging$ ls docroot/bits 404.htm... [11:16:06] 6Release-Engineering: No /static/1.26wmf7 tree -- mediawiki.org missing "Powered by MediaWiki" logo and ResourceLoader ?debug=1 mode broken - https://phabricator.wikimedia.org/T99886#1301118 (10Spage) I wonder if this is related to "bits-related changes" mentioned in operations/mediawiki-config log, and/or {T954... [11:44:02] 6Release-Engineering: No /static/1.26wmf7 tree -- mediawiki.org missing "Powered by MediaWiki" logo and ResourceLoader ?debug=1 mode broken - https://phabricator.wikimedia.org/T99886#1301156 (10mmodell) Lovely ... [11:47:38] :f [12:06:17] PROBLEM - Content Translation Server on deployment-cxserver03 is CRITICAL: Connection refused [12:07:39] 6Release-Engineering: No /static/1.26wmf7 tree -- mediawiki.org missing "Powered by MediaWiki" logo and ResourceLoader ?debug=1 mode broken - https://phabricator.wikimedia.org/T99886#1301178 (10mmodell) p:5High>3Normal https://gerrit.wikimedia.org/r/#/c/212535/ is the result of re-running checkoutMediawiki f... [12:16:16] RECOVERY - Content Translation Server on deployment-cxserver03 is OK: HTTP OK: HTTP/1.1 200 OK - 1103 bytes in 0.018 second response time [16:17:55] 10Deployment-Systems, 6operations: need package 'trebuchet-trigger' for trusty - https://phabricator.wikimedia.org/T99919#1301717 (10Dzahn) 3NEW [16:19:23] 10Deployment-Systems, 6operations: need package 'trebuchet-trigger' for trusty - https://phabricator.wikimedia.org/T99919#1301728 (10Dzahn) [16:20:50] 10Deployment-Systems, 6operations: need package 'trebuchet-trigger' for trusty - https://phabricator.wikimedia.org/T99919#1301738 (10cscott) Oh, hey, I was just talking about trebuchet. There were some quoting fixes I contributed upstream, which I don't think we've deployed yet: https://github.com/trebuchet-d... [17:20:54] 10Deployment-Systems, 6operations: errors reported by "eventual_consistency_deployment_server_init" on new deploy server - https://phabricator.wikimedia.org/T99928#1301923 (10Dzahn) 3NEW [17:22:07] 10Deployment-Systems, 6operations: errors reported by "eventual_consistency_deployment_server_init" on new deploy server - https://phabricator.wikimedia.org/T99928#1301931 (10Dzahn) [17:59:02] 10Continuous-Integration-Infrastructure, 10Wikimedia-Hackathon-2015: All new extensions should be setup automatically with Zuul - https://phabricator.wikimedia.org/T92909#1302132 (10Jdlrobson) I'm assuming this is not going to happen without help from @krinkle or @hashar ...if so I guess I should remove the ha... [19:21:15] 6Release-Engineering: No /static/1.26wmf7 tree -- mediawiki.org missing "Powered by MediaWiki" logo and ResourceLoader ?debug=1 mode broken - https://phabricator.wikimedia.org/T99886#1302360 (10Spage) @ori might know about the status and intent of the static cleanup. I updated https://wikitech.wikimedia.org/wiki... [19:33:26] (03CR) 10Jforrester: "What's this waiting on?" [integration/composer] - 10https://gerrit.wikimedia.org/r/212324 (https://phabricator.wikimedia.org/T99819) (owner: 10Legoktm) [19:35:10] (03CR) 10Legoktm: "Confirmation from either Aude or Jan that Wikidata is compatible with alpha10 and then I'll deploy it sometime after the hackathon." [integration/composer] - 10https://gerrit.wikimedia.org/r/212324 (https://phabricator.wikimedia.org/T99819) (owner: 10Legoktm) [20:00:25] 10Continuous-Integration-Infrastructure, 6Scrum-of-Scrums, 6operations, 7Blocked-on-Operations, 5Patch-For-Review: Jenkins is using php-luasandbox 1.9-1 for zend unit tests; precise should be upgraded to 2.0-8 or equivalent - https://phabricator.wikimedia.org/T88798#1302458 (10akosiaris) 5Open>3Resolv... [21:29:45] 10Browser-Tests, 10Wikimedia-Hackathon-2015: Write browser tests for DonationInterface - https://phabricator.wikimedia.org/T99955#1302602 (10awight) 3NEW [21:30:38] 10Browser-Tests, 10Wikimedia-Hackathon-2015: Write browser tests for DonationInterface - https://phabricator.wikimedia.org/T99955#1302617 (10awight) [21:43:18] PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL 22.22% of data above the critical threshold [43200.0] [21:48:06] PROBLEM - Puppet staleness on deployment-restbase02 is CRITICAL 33.33% of data above the critical threshold [43200.0] [22:20:26] 10Deployment-Systems, 6operations, 5Patch-For-Review: need package 'trebuchet-trigger' for trusty - https://phabricator.wikimedia.org/T99919#1302773 (10Dzahn) I have @carbon:~# ls /home/dzahn/trebuchet* /home/dzahn/trebuchet-trigger_0.5.6-1_all.deb /home/dzahn/trebuchet-trigger_0.5.6-1_i386.build /home... [22:23:17] 10Deployment-Systems, 6operations, 5Patch-For-Review: need package 'trebuchet-trigger' for trusty - https://phabricator.wikimedia.org/T99919#1302787 (10Dzahn) >>! In T99919#1301738, @cscott wrote: > So it would be helpful to me to know the mechanism of getting latest trebuchet packaged and deployed on our in... [22:52:48] greg-g: can you un-archive projects in phab ? [23:31:20] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL 100.00% of data above the critical threshold [0.0] [23:31:24] PROBLEM - Puppet staleness on deployment-urldownloader is CRITICAL 100.00% of data above the critical threshold [43200.0] [23:31:44] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL 100.00% of data above the critical threshold [0.0]