[00:29:15] !log Restarted grrrit-wm for I48ed549dc2b. [00:29:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:32:41] !log Didn't work, r [00:32:43] Bah. [00:32:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:32:58] !log Didn't work, rolled back grrrit-wm to 2f5de55ff75c3c268decfda7442dcdd62df0a42d. [00:33:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:54:01] !log Restarted grrrit-wm with I7eb67e3482 as well as I48ed549dc2b. [00:54:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [01:01:31] James_F: can you log in -labs? !log tools.lolrrit-wm [msg] [01:04:11] legoktm: Oh, sure. [01:04:30] legoktm: But this is a RelEng tool… [01:04:34] legoktm: Actually, can you help? Seems to be broken, but I can't find [01:04:36] Err. [01:04:44] Can't find an error log with anything useful. [01:05:32] James_F: it's definitely not maintained by releng nor does anything else related to it appear in here [01:05:36] how is it broken? [01:06:02] legoktm: It's depended on by them, but whatever. [01:06:17] legoktm: Well, it's not connecting to e.g. this channel. [01:06:34] legoktm: And it's not emitting information. [01:06:37] 2015-09-01T00:54:38.070Z - info: joining channels 0=#mediawiki-i18n, 1=#mediawiki-parsoid, 2=#mediawiki-visualeditor, 3=#wikimedia-editing, 4=#pywikibot, 5=#semantic-mediawiki, 6=#wikimedia-analytics, 7=#wikimedia-perf, 8=#wikimedia-dev, 9=#wikimedia-design, 10=#wikimedia-fundraising, 11=#wikimedia-collaboration, 12=#wikimedia-labs, 13=#wikimedia-operations, 14=#wikimedia-releng, 15=#wikidata-feed, 16=#wikimedia-multimedia, 17=#wikipedia-en- [01:06:37] ambassadors, 18=##wmt, 19=#brickimedia, 20=#wikimedia-services, 21=#mediawiki-feed [01:06:47] Yeah. [01:06:50] And nothing since then. [01:07:05] Despite "grrrit-wm1 has left IRC (Remote host closed the connection)" [01:07:37] hm [01:07:43] are you using the fabric script? [01:07:47] fabric? [01:08:03] I used `qmod -rj lolrrit-wm` per https://wikitech.wikimedia.org/wiki/Grrrit-wm [01:08:12] Is that wrong? [01:08:20] no, that's fine [01:09:15] legoktm: It seemed to work without the last two patches to master, but I can't work out why which is unsatisfying. [01:09:25] I'm not really sure... [01:09:33] I try to avoid touching that unless valhallasw is around [01:09:38] * James_F nods. [01:09:55] OK, I'll rollback. [01:10:41] Is grrrit-wm completely broken? [01:12:18] Krenair: Yup. Restarting. [01:12:39] !log Re-restarting grrrit-wm rolled back to 2f5de55ff75c3c268decfda7442dcdd62df0a42d [01:12:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [01:17:25] legoktm: Well, it's back up… [01:17:37] * James_F will seek valhallasw's advice in the morning. [02:55:05] Project browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce build #703: FAILURE in 4.7 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/703/ [03:16:38] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #803: FAILURE in 34 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/803/ [03:25:33] Project beta-scap-eqiad build #68085: FAILURE in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/68085/ [03:35:22] Yippee, build fixed! [03:35:22] Project beta-scap-eqiad build #68086: FIXED in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/68086/ [05:54:21] (03PS2) 10Polybuildr: Automatically fix warnings caught by SpaceBeforeSingleLineCommentSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/228993 (owner: 10Legoktm) [09:03:13] 6Release-Engineering: Admins for QA-alerts mailing list need to be updated - https://phabricator.wikimedia.org/T111013#1591984 (10hashar) 3NEW [09:05:25] 6Release-Engineering: Admins for QA-alerts mailing list need to be updated - https://phabricator.wikimedia.org/T111013#1591994 (10hashar) I have removed @cmcmahon and added @dduvall. Will have to send / store credentials somewhere. [09:05:34] 6Release-Engineering: Admins for QA-alerts mailing list need to be updated - https://phabricator.wikimedia.org/T111013#1591996 (10hashar) 5Open>3Resolved a:3hashar [09:07:07] 6Release-Engineering: QA-alerts mailing list moderates messages larger than 100KBytes - https://phabricator.wikimedia.org/T111014#1592001 (10hashar) 3NEW [09:07:43] 6Release-Engineering: QA-alerts mailing list moderates messages larger than 100KBytes - https://phabricator.wikimedia.org/T111014#1592011 (10hashar) 5Open>3Resolved a:3hashar Raised to 250KBytes [09:25:41] (03PS7) 10Hashar: (WIP) dib: wikimedia-puppet element (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/234975 [09:26:40] 5Continuous-Integration-Isolation: Write a diskimage-builder element to run puppet - https://phabricator.wikimedia.org/T110735#1592044 (10hashar) p:5Triage>3High a:3hashar [09:27:08] (03PS8) 10Hashar: (WIP) dib: wikimedia-puppet element (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/234975 (https://phabricator.wikimedia.org/T110735) [09:27:56] 5Continuous-Integration-Isolation, 5Patch-For-Review: Write a diskimage-builder element to run puppet - https://phabricator.wikimedia.org/T110735#1592048 (10hashar) I have a working dib element to run puppet out of operations/puppet.git though it doesn't install much it at least brings us the apt configuratio... [09:28:17] 5Continuous-Integration-Isolation: Figure out how to inject facts in the diskimage-builder chroot - https://phabricator.wikimedia.org/T110737#1592050 (10hashar) [09:28:17] 5Continuous-Integration-Isolation, 5Patch-For-Review: Write a diskimage-builder element to run puppet - https://phabricator.wikimedia.org/T110735#1592049 (10hashar) [09:28:38] 5Continuous-Integration-Isolation, 5Patch-For-Review: Write a diskimage-builder element to run puppet - https://phabricator.wikimedia.org/T110735#1585509 (10hashar) I did not depend on any fact, so {T110737} is no more a blocker. [09:28:59] 5Continuous-Integration-Isolation: Figure out how to inject facts in the diskimage-builder chroot - https://phabricator.wikimedia.org/T110737#1592054 (10hashar) p:5Triage>3Low No more blocks {T110735} [09:30:11] 5Continuous-Integration-Isolation, 6Labs, 10Labs-Infrastructure: Include Base::Standard-packages in labs images - https://phabricator.wikimedia.org/T94995#1592058 (10hashar) 5Open>3declined a:3hashar From T110735 , we now only apply a subset of `operations/puppet` since lot of parts are not easily appl... [09:30:25] Yippee, build fixed! [09:30:25] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #709: FIXED in 1 hr 20 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/709/ [09:35:45] 5Continuous-Integration-Isolation, 6Labs, 10Labs-Infrastructure: Investigate non blocking fs resizing when instance is booted - https://phabricator.wikimedia.org/T104974#1592076 (10hashar) 5Open>3Resolved a:3hashar I have filled this tasks for instances booted from #labs images. The dib images using Je... [09:38:00] 5Continuous-Integration-Isolation, 6operations, 7Database: MySQL database for Nodepool - https://phabricator.wikimedia.org/T110693#1592079 (10hashar) [09:50:08] Yippee, build fixed! [09:50:08] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #545: FIXED in 29 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/545/ [10:04:00] 5Continuous-Integration-Isolation, 6operations, 7Database: MySQL database for Nodepool - https://phabricator.wikimedia.org/T110693#1592130 (10jcrespo) a:3jcrespo @Hashar is this related to the other OpenStack-related databases that normally @Andrew works with? [10:04:20] 5Continuous-Integration-Isolation, 6operations, 7Database: MySQL database for Nodepool - https://phabricator.wikimedia.org/T110693#1592132 (10jcrespo) p:5Triage>3Normal [10:06:52] 10Beta-Cluster, 10QuickSurveys, 3Reading-Web-Next-Sprint-55: Get QuickSurveys enabled on beta cluster - https://phabricator.wikimedia.org/T110199#1592141 (10phuedx) [11:44:24] 5Continuous-Integration-Isolation, 6operations, 7Database: MySQL database for Nodepool - https://phabricator.wikimedia.org/T110693#1592390 (10hashar) //This task is to pick a database for Nodepool// For continuous integration purposes, we are setting up a python based daemon named Nodepool. It maintains a... [11:45:59] 5Continuous-Integration-Isolation, 6operations, 7Database: MySQL database for Nodepool - https://phabricator.wikimedia.org/T110693#1592393 (10hashar) >>! In T110693#1592130, @jcrespo wrote: > @Hashar is this related to the other OpenStack-related databases that normally @Andrew works with? Unrelated. It is... [12:16:57] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Wikidata-Gadgets: Browsertests for CommonsMedia gadget on beta - https://phabricator.wikimedia.org/T68253#1592444 (10Lydia_Pintscher) p:5Normal>3Low [12:30:41] bd808: addshore: could one of you +2 this please? https://gerrit.wikimedia.org/r/#/c/233836/ [12:31:00] or anybody else, for that matter. :P [12:31:20] Um [12:31:24] Messages*.php? [12:31:48] oh, namespacey stuff left [12:33:11] Reedy: sorry? didn't get you [12:33:26] I thought those files had died with the switch to json [12:33:50] I forgot about all the other weird and wonderful language things mw does [12:34:37] ha :P okay. [12:34:50] I thought you were talking about the "s", which, now that you mention it [12:34:52] should probably be there. [12:35:30] haha, yup it should [12:54:49] Yippee, build fixed! [12:54:49] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #582: FIXED in 48 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/582/ [12:58:33] Yippee, build fixed! [12:58:33] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #771: FIXED in 26 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/771/ [13:49:59] 10Continuous-Integration-Config, 5Patch-For-Review, 7Puppet: Setup rubocop for operations/puppet ruby code lints - https://phabricator.wikimedia.org/T102020#1592844 (10zeljkofilipin) [[ https://gerrit.wikimedia.org/r/#/c/225238/ | 225238 ]] was reverted by [[ https://gerrit.wikimedia.org/r/#/c/226898/ | 2268... [13:54:17] 10Continuous-Integration-Infrastructure, 6Release-Engineering, 5Patch-For-Review: Repositories with Ruby code should be documented and appropriate Jenkins jobs should be running - https://phabricator.wikimedia.org/T1361#1592853 (10zeljkofilipin) [13:54:19] 10Continuous-Integration-Config, 5Patch-For-Review, 7Puppet: Setup rubocop for operations/puppet ruby code lints - https://phabricator.wikimedia.org/T102020#1592852 (10zeljkofilipin) 5Open>3stalled [14:07:25] 10Browser-Tests, 10Continuous-Integration-Config, 10Wikidata: fix `negative argument (ArgumentError) in browsertests - https://phabricator.wikimedia.org/T110510#1592905 (10JanZerebecki) Diff that made the crome job also fail: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-... [14:08:02] 10Browser-Tests, 10Continuous-Integration-Config, 10Wikidata: fix `negative argument (ArgumentError) in browsertests - https://phabricator.wikimedia.org/T110510#1592906 (10zeljkofilipin) This problem is introduced by {T106839}. [14:10:40] 10Browser-Tests, 10Continuous-Integration-Config, 10Wikidata: fix `negative argument (ArgumentError) in browsertests - https://phabricator.wikimedia.org/T110510#1592919 (10hashar) Since JJB and Jenkins generated configurations are different, if you want to compare the current JJB version you can configure th... [14:12:09] 10Browser-Tests, 10Continuous-Integration-Config, 10Wikidata: fix `negative argument (ArgumentError) in browsertests - https://phabricator.wikimedia.org/T110510#1592921 (10zeljkofilipin) The actual job diff: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-chrome-sauce/jobCo... [14:14:44] 10Browser-Tests, 10Continuous-Integration-Config, 10Wikidata: fix `negative argument (ArgumentError) in browsertests - https://phabricator.wikimedia.org/T110510#1592941 (10hashar) Maybe you can regenerate the job entirely and see what happens? Make sure your JJB is the current version from integration/jenkin... [14:20:58] 10Browser-Tests, 10Continuous-Integration-Config, 10Wikidata: fix `negative argument (ArgumentError) in browsertests - https://phabricator.wikimedia.org/T110510#1592971 (10zeljkofilipin) a:3zeljkofilipin [14:22:15] 10Browser-Tests, 10Continuous-Integration-Config, 10Wikidata: fix `negative argument (ArgumentError) in browsertests - https://phabricator.wikimedia.org/T110510#1592973 (10JanZerebecki) There is an earlier error message in the log: no implicit conversion of String into Integer (TypeError) [14:25:25] 10Browser-Tests, 10Continuous-Integration-Config, 10Wikidata: fix `negative argument (ArgumentError) in browsertests - https://phabricator.wikimedia.org/T110510#1592995 (10JanZerebecki) I reverted to the old config, triggered a run and reverted to the jjb generated one again: https://integration.wikimedia.or... [14:54:54] 10Browser-Tests, 10Continuous-Integration-Config, 10Wikidata: fix `negative argument (ArgumentError) in browsertests - https://phabricator.wikimedia.org/T110510#1593122 (10JanZerebecki) So it seems "no implicit conversion of String into Integer (TypeError)" also happens with the old config. But "negative arg... [15:01:29] zeljkof-meeting: I am publishing minutes from last meeting before I forget [15:01:38] hashar: yeah! :) [15:01:56] that's the spirit, the sooner you do it, the easier it is [15:23:22] 10Beta-Cluster, 10Continuous-Integration-Config: Jenkins job beta-scap-eqiad should abort early when Keyholder is not armed - https://phabricator.wikimedia.org/T111062#1593252 (10hashar) 3NEW a:3bd808 [15:24:17] 10Beta-Cluster, 10Continuous-Integration-Config: Jenkins job beta-scap-eqiad should abort early when Keyholder is not armed - https://phabricator.wikimedia.org/T111062#1593252 (10hashar) a:5bd808>3None [15:25:40] 10Beta-Cluster, 7Monitoring, 7Shinken: Monitor keyholder on deployment-bastion - https://phabricator.wikimedia.org/T111064#1593275 (10hashar) 3NEW [15:26:00] 10Beta-Cluster, 10Continuous-Integration-Config: Jenkins job beta-scap-eqiad should abort early when Keyholder is not armed - https://phabricator.wikimedia.org/T111062#1593284 (10Reedy) See also T110794 for production, and also T110791 T110793 for related error messes [15:28:53] 10Beta-Cluster, 10Continuous-Integration-Config: Jenkins job beta-scap-eqiad should abort early when Keyholder is not armed - https://phabricator.wikimedia.org/T111062#1593303 (10hashar) [15:29:50] 6Release-Engineering, 10Wikimedia-Mailing-lists: QA-alerts mailing list moderates messages larger than 100KBytes - https://phabricator.wikimedia.org/T111014#1593318 (10hashar) [15:29:57] 6Release-Engineering, 10Wikimedia-Mailing-lists: Admins for QA-alerts mailing list need to be updated - https://phabricator.wikimedia.org/T111013#1593319 (10hashar) [15:30:03] hashar: non-lowercase hash tags don't work (aren't parsed in markup and don't create a permalink url either) [15:30:05] Iv'e removed the uppercase variants. [15:30:17] Krinkle: thanks! [15:30:22] The parser is case-insensitive so #vArIaTiOns are always linked anyway [15:30:41] (ciscaling) [15:30:49] yw [15:30:55] hashar: How's things in CI? [15:31:27] jan / zeljko have just finished the weekly meeting. Minutes at https://www.mediawiki.org/wiki/Continuous_integration/Meetings/2015-09-01/Minutes [15:31:44] got a few things to polish up with nodepool and I will enable it soonish [15:32:11] we had some discussion about the npm/composer job that fails on REL branches [15:32:21] though we don't backport much often [15:32:32] and https://www.mediawiki.org/wiki/User:Legoktm/ci !! [15:33:14] hashar: What about git-cache? [15:34:11] hashar: btw, a yaml or json file to "enable" entry points sounds good. That'll also take care of the branching problem since adding it will naturally not affect old branches. [15:36:43] it will also make sure that developers don't need to work with yet-another entry point. I really strongly recommend against adding Makefile ontop of npm/composer. It's inappropriate and foreign to the environment. We already have an entry point. [15:37:20] Doesn't make sense to expect a javascript browser package, node.js program, php C-library and mediawiki core to have the same development practice. [15:37:30] Let each have their own optimised environment and use a single entry point. [15:37:40] We can maybe merge a few of them, e.g. run 'npm test' from 'composer test' for php projects. [15:37:51] so that there's only one. If we really want that. [15:38:30] and yeah, would love an update on git-cache so that we can fix the growing number of inconsistency problems we have. [15:38:35] Eveyrthing depends on that [15:40:43] Krinkle: yeah that sounds wize [15:40:46] wise [15:40:47] bah [15:41:08] one problem with a file configuration entry point that would state composer=no, is we still have to trigger the 'npm' job :-( [15:41:51] hashar: The file would only list what we run, not what we don't run. e.g. composer=True, and/or npm=True [15:42:04] or entry: - npm - composer [15:59:42] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: fix no implicit conversion of String into Integer (TypeError) in browsertests - https://phabricator.wikimedia.org/T111069#1593421 (10JanZerebecki) 3NEW [16:00:31] 10Browser-Tests, 10Continuous-Integration-Config, 10Wikidata: fix `negative argument (ArgumentError) in browsertests - https://phabricator.wikimedia.org/T110510#1593428 (10JanZerebecki) Created {T111069} for the other bug. [16:03:15] PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:03:16] PROBLEM - Puppet failure on deployment-mx is CRITICAL 100.00% of data above the critical threshold [0.0] [16:03:16] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:06:32] (03PS1) 10Zfilipin: Add documentation for release process [selenium] - 10https://gerrit.wikimedia.org/r/235259 (https://phabricator.wikimedia.org/T108873) [16:08:10] (03PS2) 10Zfilipin: Add documentation for release process [selenium] - 10https://gerrit.wikimedia.org/r/235259 (https://phabricator.wikimedia.org/T108873) [16:44:29] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL - Socket timeout after 10 seconds [16:49:06] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 44526 bytes in 0.621 second response time [17:01:09] 10Continuous-Integration-Config, 6Scrum-of-Scrums, 5Patch-For-Review, 7Puppet: Setup rubocop for operations/puppet ruby code lints - https://phabricator.wikimedia.org/T102020#1593617 (10zeljkofilipin) [17:24:43] (03CR) 10Dduvall: [C: 04-1] "Yay for better documentation! I left a few suggestions but it looks pretty good overall." (034 comments) [selenium] - 10https://gerrit.wikimedia.org/r/235259 (https://phabricator.wikimedia.org/T108873) (owner: 10Zfilipin) [17:31:29] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL - Socket timeout after 10 seconds [17:40:35] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL - Socket timeout after 10 seconds [17:45:01] ostriches: are we branching REL1_26 next week after wmf22? [17:45:05] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 44530 bytes in 0.731 second response time [17:51:13] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL - Socket timeout after 10 seconds [17:55:42] 10Beta-Cluster, 10Continuous-Integration-Config, 10Scap3: Jenkins job beta-scap-eqiad should abort early when Keyholder is not armed - https://phabricator.wikimedia.org/T111062#1593798 (10JanZerebecki) That is a good idea, also for production, so perhaps scap should do that? [17:56:20] 10Beta-Cluster, 10Continuous-Integration-Config, 10Deployment-Systems: Jenkins job beta-scap-eqiad should abort early when Keyholder is not armed - https://phabricator.wikimedia.org/T111062#1593818 (10JanZerebecki) [17:57:25] 10Beta-Cluster, 10Continuous-Integration-Config, 10Deployment-Systems, 6Release-Engineering: Jenkins job beta-scap-eqiad should abort early when Keyholder is not armed - https://phabricator.wikimedia.org/T111062#1593252 (10JanZerebecki) [18:05:25] Krinkle: do i still need to update the config of zuul now for dismissable sitenotice ? or is it voting automaticly now because it uses npm test ? [18:05:27] thedj: Hm...? [18:06:17] jenkins jshint that is [18:08:47] ah i just rm [18:08:47] - name: mwext-DismissableSiteNotice-jslint # bug 61602 [18:08:47] voting: false [18:18:53] (03PS1) 10TheDJ: Configure Dismissable Sitenotice with npm [integration/config] - 10https://gerrit.wikimedia.org/r/235280 (https://phabricator.wikimedia.org/T63602) [18:52:15] (03PS1) 10Florianschmidtwelzow: Change link href for PHP docs of OOJs UI [integration/docroot] - 10https://gerrit.wikimedia.org/r/235287 (https://phabricator.wikimedia.org/T111090) [19:00:59] (03CR) 10Legoktm: [C: 04-1] "We should fix the CI job to make it output to /php/ again" [integration/docroot] - 10https://gerrit.wikimedia.org/r/235287 (https://phabricator.wikimedia.org/T111090) (owner: 10Florianschmidtwelzow) [19:26:48] 10Continuous-Integration-Config, 6Labs, 10Tool-Labs: Job labs-toollabs-debian-glue is failing for labs/toollabs repository - https://phabricator.wikimedia.org/T110939#1594306 (10hashar) That is an issue in jenkins-debian-glue on Trusty: ``` ... 00:00:01.212 Checking out Revision f275d97d7010b3bb2709d4a5211e2... [19:35:08] 10Continuous-Integration-Config, 6Labs, 10Tool-Labs: Job labs-toollabs-debian-glue is failing for labs/toollabs repository - https://phabricator.wikimedia.org/T110939#1594360 (10hashar) I manually triggered the debian-glue job. It uses the `jessie` distribution as provided by the Debian project (no apt.wikim... [19:35:19] RECOVERY - Host integration-slave-trusty-1017 is UPING OK - Packet loss = 0%, RTA = 0.71 ms [19:48:00] 10Continuous-Integration-Config, 6Labs, 10Tool-Labs: Job labs-toollabs-debian-glue is failing for labs/toollabs repository - https://phabricator.wikimedia.org/T110939#1594380 (10hashar) We could get the target distribution from the `debian/changelog` file using: export distribution=$(dpkg-parsechangelog --s... [19:50:45] 10Continuous-Integration-Config, 6Labs, 10Tool-Labs: Change sid pbuilder image name to 'unstable' - https://phabricator.wikimedia.org/T111097#1594382 (10hashar) 3NEW a:3akosiaris [19:53:28] 10Continuous-Integration-Config, 6Labs, 10Tool-Labs: Job labs-toollabs-debian-glue is failing for labs/toollabs repository - https://phabricator.wikimedia.org/T110939#1594400 (10hashar) p:5Triage>3Low [20:00:39] 6RelEng-Admin, 15User-greg: Get RelEng team members greater access - https://phabricator.wikimedia.org/T107926#1594415 (10Dzahn) >no one with root is actively watching the puppet/dns repositories for trivial patches from people who can't just +2. Ok, let's check this assumption and the real numbers. I regula... [20:01:37] !log Starting scans/spidering on integration-mediawiki03 [20:01:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:06:04] dapatrick: fun times :) [20:06:29] 6RelEng-Admin, 15User-greg: Get RelEng team members greater access - https://phabricator.wikimedia.org/T107926#1594449 (10Krenair) Yes, I don't really agree with wiki creation being used as an example of this problem. My original comment expressed this better but it got changed around as I edited it before sub... [20:06:33] \m/ [20:07:15] greg-g: I just learned about logging to the this channel from marxarelli. I like this idea. [20:08:14] :) [20:08:17] it's a good place [20:10:18] PROBLEM - Host integration-slave-trusty-1017 is DOWN: CRITICAL - Host Unreachable (10.68.17.136) [20:11:15] eh? [20:18:45] uh, there is no trusty-1017 [20:25:02] good thing it's down then [20:26:49] hashar: having some trouble running dib on integration-dev [20:27:01] ahh [20:27:07] marxarelli: shoot ! :} [20:27:08] i keep getting "install: missing destination file operand after '/tmp/image.ILFTBYVy/mnt/usr/local/bin/dib-run-parts'" [20:27:32] i'm using a different cache directory if that could have something to do with it [20:27:39] might be [20:27:47] oh [20:27:47] since /srv/dib/cache is owned by you [20:28:06] i'm doing `DIB_IMAGE_CACHE=/srv/home/dduvall/dib/cache DIB_DEBIAN_USE_DEBOOTSTRAP_CACHE=1 ./build_image.sh` [20:28:19] maybe it is dig-run-parts being the wrong version [20:29:04] "diskimage-builder (1.1.1)" [20:29:46] /srv/dib/cache can probably be opened a bit more :-} [20:30:01] can you paste the console output around the error ? [20:31:01] 10Deployment-Systems, 3releng-201516-q2: [keyresult] Migrate all Service team owned services and MW deploys to scap3 - https://phabricator.wikimedia.org/T109926#1594582 (10greg) [20:32:45] hashar: https://phabricator.wikimedia.org/P1961 [20:34:04] * hashar reads [20:34:50] marxarelli: which dib-run-parts ; dib-run-parts --version [20:34:57] dib-run-parts Tue Sep 1 20:34:43 UTC 2015 Scripts directory [--version] must exist and be a directory [20:35:04] I got it installed via pip apparently [20:35:36] mine is installed via pip to ~/.local [20:35:42] 1.1.1 [20:35:46] so same as me :/ [20:36:20] mm [20:36:22] + exec sudo install -m 0755 -o root -g root -D /tmp/image.ILFTBYVy/mnt/usr/local/bin/dib-run-parts [20:36:52] it misses a destination :/ [20:37:14] seems to be the script 90-base-dib-run-parts [20:38:21] which does: [20:38:22] exec sudo install -m 0755 -o root -g root -D \ [20:38:22] $(which dib-run-parts) \ [20:38:23] $TARGET_ROOT/usr/local/bin/dib-run-parts [20:38:24] is anyone doing anything with elastic search in beta where I would be messing them up if I test a bit of failover [20:38:28] so no dig-run-parts found :/ [20:38:42] i.e. elastic search instability in beta...you like it? [20:38:51] chasemp: ^d is probably the guy for it [20:39:01] ^d loves instability [20:39:32] chasemp: I don't think we have any instability policy beside making our best to restore the service once testing is complete :-} so I would said go ahead [20:39:36] chasemp: and maybe !log it :} [20:39:49] definitely !log it :) [20:39:52] ok will do thanks and ^d if you see soemthing and it's causing issue let me know [20:40:02] so I can point blame, er, causation when someone complains [20:40:03] <^d> Herp derp. [20:40:15] marxarelli: so for some reason $(which dib-run-parts) returns empty. Maybe your PATH is not exported :/ [20:40:18] hashar: ah, so which dib-run-parts is returning nothing likely [20:40:18] Perp. [20:40:27] i'll try it [20:40:56] marxarelli: all / most shell script can have set -x set by exporting DIB_DEBUG_TRACE=1 [20:41:40] marxarelli: I have refreshed my notes on https://wikitech.wikimedia.org/wiki/Nodepool#Diskimage an hour or so ago [20:41:53] you can also do [20:41:55] I am not sure why it doesn't abort though since there is set -e [20:41:55] bash -x foo.sh [20:43:09] hashar: uh, where does dib-run-parts live? [20:43:21] ohh [20:43:34] marxarelli: it is part of diskimage-builder so should be in ~/.local/bin [20:43:44] oh, it's there [20:44:14] I should poke the .deb maintainer to bump the debian sid package to 1.1.1 [20:44:27] and the inject the package to apt.wikimedia.org [20:44:38] ah ha~ [20:44:40] ! [20:44:43] :) [20:44:45] so [20:44:46] set -e [20:44:57] exec /bin/ls $(which aozieaozeioaze) [20:44:59] that does not abort [20:45:01] i prepending my PATH with an unexpanded path [20:45:01] thank you exec [20:45:13] `which` does not like that [20:45:32] i had "~/.local/bin:$PATH" [20:46:16] ok, trying again [20:46:49] I am preparing a patch for upstream [20:47:55] good OSS citizen, hashar [20:49:09] one day I will count the number of patches I have sent upstream vs patches for wmf :D [20:50:27] wee! made it to puppet apply so far [20:52:37] 10Beta-Cluster, 10Continuous-Integration-Config, 10Deployment-Systems, 6Release-Engineering: Jenkins job beta-scap-eqiad should abort early when Keyholder is not armed - https://phabricator.wikimedia.org/T111062#1594693 (10greg) >>! In T111062#1593798, @JanZerebecki wrote: > That is a good idea, also for p... [20:54:31] 10Deployment-Systems, 6Release-Engineering: Jenkins job beta-scap-eqiad should abort early when Keyholder is not armed - https://phabricator.wikimedia.org/T111062#1594704 (10greg) [20:54:51] Yippee, build fixed! [20:54:51] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #752: FIXED in 28 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/752/ [20:55:57] hashar: "Disk image successful image-jessie-20150901T204832Z.qcow2" [20:56:01] \o/ [20:58:36] marxarelli: https://review.openstack.org/219453 :D [20:58:48] marxarelli: congratulations! [20:59:12] so in theory you can boot the image on integration-dev [20:59:38] I added some commands to inspect the image at https://wikitech.wikimedia.org/wiki/Nodepool#Mount_a_qcow2_image [20:59:57] but I have no idea how to boot the image in an instance using qemu. Haven't looked up [20:59:58] 10Deployment-Systems, 6Release-Engineering: Scap should abort early when Keyholder is not armed - https://phabricator.wikimedia.org/T111062#1594706 (10greg) [21:00:14] then you will have to retrieve the image on labnodepool1001.eqiad.wmnet [21:00:37] since that is where we have the OpenStack API credentials ( as user nodepool source /var/lib/nodepool/.profile ) [21:00:54] from there it is "all about" ™ running openstack image create --file .qcow2 ci-debian-jessie --disk-format qcow2 --property show=true [21:01:06] and blame the "ci-debian-jessie" image is updated [21:03:18] wee! [21:03:39] hashar: cool. should i try applying more of ops/puppet packages to the build? [21:04:00] seems like the more contint packages we can get on it the better [21:04:11] sure! [21:04:27] what I did is hack the wikimedia-puppet element on my machine [21:04:39] then rsync that to integration-dev:/home/hashar/dib [21:04:50] and from the dev machine: rebuild the image, publish it under /var/www/html [21:04:57] retrieve it on labnodepool1001.eqiad.wmnet [21:05:00] publish it to openstack [21:05:02] boot an instance [21:05:05] see what happens [21:05:07] rinse and repeat [21:05:19] would be smarter to boot it directly on integration-dev though [21:05:33] ar [21:05:37] i'll try out some different scenarios [21:05:50] zeljkof is willing to refactor our contint puppet manifests that are related to ruby/bundler [21:05:53] i can probably get it building within my vagrant env too [21:05:59] so we can include it easily in the image [21:06:05] yeah probably [21:06:10] it is "just" a qcow2 image :-} [21:06:53] great. i'll play around with it [21:06:56] right after lunch! [21:08:21] !log marxarelli properly build a CI image using diskimage-builder \O/ [21:08:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:08:25] for historical purposes [21:08:59] :) :) [21:17:11] !sal [21:17:11] https://tools.wmflabs.org/sal/releng [21:17:23] that one is completely awesome [21:17:37] or bd808 refactoring decade old process and code [21:19:02] oh and dapatrick does some scanning on beta cluster. I assume that is some security audit / perf testing [21:19:40] hashar: Yep, exactly. We're working on getting automated scanning deployed. [21:19:56] pleased to meet you :-} [21:19:58] I am https://wikimediafoundation.org/wiki/Staff_and_contractors#/media/File:Antoine_Musso-3500.jpg [21:20:06] or https://www.mediawiki.org/wiki/User:Hashar [21:20:27] hashar: We've met (I think)! In Lyon. [21:20:34] potentially :} [21:20:40] But yes, good to meet you again. [21:21:03] ahhh yeah [21:21:15] the only unsmart thing I found out to say was asking whether you were french [21:25:57] 6RelEng-Admin, 10Gerrit-Migration: Outline work (outcomes and outputs) of RelEng's Q2 Gerrit migration work - https://phabricator.wikimedia.org/T110623#1594803 (10greg) Meeting to discuss what the roadmap looks like between "here" and "code review happens in Phab) is scheduled for Thursday morning Pacific with... [21:41:47] 6RelEng-Admin, 10Gerrit-Migration: Outline work (outcomes and outputs) of RelEng's Q2 Gerrit migration work - https://phabricator.wikimedia.org/T110623#1594876 (10Qgil) I'll be next week in San Francisco. If you are there as well ;) then I'll just go hunt you early some morning for a Differential chat. [22:09:49] 10Deployment-Systems, 10ReleaseTaggerBot: Update ReleaseTaggerBot to deal with SemVer for WMF deployed branches (eg 1.23.0-wmf.6) - https://phabricator.wikimedia.org/T107192#1595004 (10Jdforrester-WMF) [22:22:36] 10Continuous-Integration-Infrastructure, 10Ops-Access-Requests, 6operations, 5Patch-For-Review: Let contint-admins force run puppet with /usr/local/sbin/puppet-run - https://phabricator.wikimedia.org/T110943#1595045 (10RobH) a:3RobH I'll claim this to my assignment until our Operations meeting next week.... [22:42:35] 10Continuous-Integration-Config, 10VisualEditor, 7Documentation, 5Patch-For-Review: VisualEditor documentation examples on doc.wikimedia.org not working - https://phabricator.wikimedia.org/T109170#1595133 (10Krenair) a:3Esanders [22:53:47] 6Release-Engineering: Remove EOL MediaWiki release branches - https://phabricator.wikimedia.org/T92503#1595216 (10demon) Here's REL1_1/1.1.0: ``` lines=10 chad@notsexy /a/vag/mediawiki (master)$ git log origin/REL1_1 ^1.1.0 --no-merges --oneline aa3b6f6 Hide upload link if uploads are disabled. 3108d98 URL-encod... [22:54:38] Krinkle: Heh, ^ [22:55:02] Digging old branch history is fun. [22:59:07] ea5c70a Added wfAbruptExit() function, to replace exit() calls with. [22:59:47] ironically, that function is less abrupt than exit() [22:59:54] Lol [23:05:35] function wfErrorExit() { [23:05:35] wfAbruptExit( true ); [23:05:35] } [23:22:13] ostriches: Do you know why group 0 is still on wmf20, not wmf21? [23:22:27] ostriches: The SAL claims it went there three hours ago but Special:Version disagrees? [23:22:42] Nope I don't offhand [23:22:45] Kk. [23:31:45] 10Beta-Cluster: Thumbnail scaling broken on beta - https://phabricator.wikimedia.org/T111132#1595389 (10Tgr) 3NEW [23:33:07] 10Beta-Cluster: Thumbnail scaling broken on beta - https://phabricator.wikimedia.org/T111132#1595389 (10Tgr) [23:37:30] 10Browser-Tests: mediawiki_api gem recursion on log_in - https://phabricator.wikimedia.org/T111133#1595404 (10rmoen) 3NEW [23:38:49] 10Browser-Tests: mediawiki_api gem recursion on log_in - https://phabricator.wikimedia.org/T111133#1595412 (10rmoen) [23:40:12] 10Browser-Tests: mediawiki_api gem recursion on log_in - https://phabricator.wikimedia.org/T111133#1595419 (10dduvall) a:3dduvall [23:51:03] 10Beta-Cluster: Thumbnail scaling broken on beta - https://phabricator.wikimedia.org/T111132#1595449 (10greg) @tgr: do you know where this might be breaking down? We recently rebuilt the tmh host in Beta (deleted the deployment-videoscaler01 instance, rebuilt as deployment-tmh01 on Trusty/HHVM, see: T110707), if... [23:52:25] 10Browser-Tests: mediawiki_api gem recursion on log_in - https://phabricator.wikimedia.org/T111133#1595459 (10dduvall) p:5Triage>3Normal [23:55:30] 10Browser-Tests: mediawiki_api gem recursion on log_in - https://phabricator.wikimedia.org/T111133#1595468 (10rmoen) @dduvall in efforts to provide more data i've ran client.log_in( un, pw ) and quickly aborted to retrieve the first few results. ``` irb(main):004:0> client.log_in( 'Rob', 'asdfasdf' ) I, [2015-...