[00:53:35] Yippee, build fixed! [00:53:35] Project UploadWizard-api-commons.wikimedia.org build #1783: FIXED in 33 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.org/1783/ [05:35:54] Yippee, build fixed! [05:35:55] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #482: FIXED in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/482/ [06:27:37] Yippee, build fixed! [06:27:37] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #682: FIXED in 8 min 35 sec: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/682/ [06:34:58] 10Beta-Cluster, 10Compact-Personal-Bar-(Beta), 10MediaWiki-extensions-VectorBeta, 7Beta-Feature, 7user-notice: Remove Compact Personal Bar from Beta Cluster and Production - https://phabricator.wikimedia.org/T104659#1459717 (10Bugreporter) [06:42:28] RECOVERY - Free space - all mounts on deployment-videoscaler01 is OK All targets OK [07:30:01] 10Beta-Cluster, 10MediaWiki-General-or-Unknown, 7Wikimedia-log-errors: Viewing beta's en_rtlwiki just throws a fatal exception - https://phabricator.wikimedia.org/T106125#1459770 (10Krenair) 3NEW [07:41:54] (03CR) 10Paladox: "@Hashar please merge I am the author of this extension." [integration/config] - 10https://gerrit.wikimedia.org/r/224819 (owner: 10Paladox) [07:44:26] 10Browser-Tests, 10Wikimania-Hackathon-2015, 7I18n: Hacking: Load i18n messages from MediaWiki to browser tests - https://phabricator.wikimedia.org/T90577#1459808 (10Vikassy) @zeljkofilipin: Here it is: https://gerrit.wikimedia.org/r/#/c/225288/ [07:56:16] 6Release-Engineering, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1459837 (10Tau) >>! In T102566#1439571, @Tgr wrote: >>>! In T102566#1434294, @Tau wrote: >> Still nothing ... Any fu... [08:08:44] (03PS1) 10Paladox: Update AjaxLogin tests [integration/config] - 10https://gerrit.wikimedia.org/r/225289 [08:09:30] (03PS2) 10Paladox: Update AjaxLogin tests [integration/config] - 10https://gerrit.wikimedia.org/r/225289 [08:15:19] (03PS3) 10Paladox: Update AjaxLogin tests [integration/config] - 10https://gerrit.wikimedia.org/r/225289 [08:20:43] (03PS1) 10Paladox: Check for json in BreadCrumbs [integration/config] - 10https://gerrit.wikimedia.org/r/225291 [08:22:13] (03PS2) 10Paladox: Check for json in BreadCrumbs [integration/config] - 10https://gerrit.wikimedia.org/r/225291 [09:22:25] PROBLEM - Puppet failure on deployment-mx is CRITICAL 100.00% of data above the critical threshold [0.0] [09:39:01] 6Release-Engineering, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1459959 (10Tgr) [[ https://www.mediawiki.org/wiki/Manual:$wgDebugLogFile | Manual:$wgDebugLogFile ]] recommends chec... [09:43:54] (03PS1) 10Paladox: Remove unused mwext-MaintenanceShell [integration/config] - 10https://gerrit.wikimedia.org/r/225293 [09:44:28] (03PS2) 10Paladox: Remove unused mwext-MaintenanceShell [integration/config] - 10https://gerrit.wikimedia.org/r/225293 [09:50:28] (03CR) 10Filippo Giunchedi: "looks good, I've prefixed pedantic/nitpick comments with ~ so feel free to ignore or postpone those!" (0311 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/224374 (owner: 10Thcipriani) [10:02:43] 10Beta-Cluster, 6operations, 5Patch-For-Review: deployment-bastion fails puppet because some classes were moved from nodes to role class - https://phabricator.wikimedia.org/T106003#1459979 (10hashar) 5Open>3Resolved Beta cluster puppetmaster rebased just fine and puppet pass on deployment-bastion. Danke... [10:23:44] (03PS1) 10Paladox: Update ShoutWikiAds tests [integration/config] - 10https://gerrit.wikimedia.org/r/225298 [10:24:03] (03PS2) 10Paladox: Update ShoutWikiAds tests [integration/config] - 10https://gerrit.wikimedia.org/r/225298 [11:29:33] 6Release-Engineering, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1460075 (10Tau) In both php.ini files the ";open_basedir= " (blank). Is it okay? Is this string set anywhere else to... [11:43:30] PROBLEM - Free space - all mounts on deployment-videoscaler01 is CRITICAL deployment-prep.deployment-videoscaler01.diskspace._var.byte_percentfree (<40.00%) [12:14:22] (03PS3) 10Paladox: Update ShoutWikiAds tests [integration/config] - 10https://gerrit.wikimedia.org/r/225298 [13:13:40] Yippee, build fixed! [13:13:40] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #719: FIXED in 41 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/719/ [13:22:37] (03PS1) 10Paladox: Update ConfirmAccount tests [integration/config] - 10https://gerrit.wikimedia.org/r/225311 [13:22:55] (03PS2) 10Paladox: Update ConfirmAccount tests [integration/config] - 10https://gerrit.wikimedia.org/r/225311 [13:23:49] (03PS3) 10Paladox: Update ConfirmAccount tests [integration/config] - 10https://gerrit.wikimedia.org/r/225311 [13:37:28] (03PS1) 10Paladox: Update interwiki tests [integration/config] - 10https://gerrit.wikimedia.org/r/225313 [13:37:48] 10Deployment-Systems, 7I18n, 7Wikimedia-log-errors: l10n-update not updating Vector - https://phabricator.wikimedia.org/T103879#1460227 (10Aklapper) Do you know if this has happened again since then? [13:39:29] (03PS2) 10Paladox: Update interwiki tests [integration/config] - 10https://gerrit.wikimedia.org/r/225313 [13:39:57] (03PS3) 10Paladox: Update interwiki tests [integration/config] - 10https://gerrit.wikimedia.org/r/225313 [13:46:12] (03PS4) 10Paladox: Update interwiki tests [integration/config] - 10https://gerrit.wikimedia.org/r/225313 [14:34:37] Yippee, build fixed! [14:34:37] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #567: FIXED in 8 min 35 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/567/ [14:37:39] Yippee, build fixed! [14:37:39] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #194: FIXED in 9 min 38 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/194/ [14:41:13] hashar: I’m gone for a week starting tomorrow. If you have any nodepool stuff sorted out before then (e.g. https://gerrit.wikimedia.org/r/#/c/224102/) , now is the time :) [14:41:35] andrewbogott: I am swamped in systemd config :( [14:41:53] I should really have used a simple init script cause that is a huge horse :D [14:42:02] ok. I don't know much about systemd but I'm here to review/merge when you have something working :) [14:42:20] andrewbogott: beside that, got some puppet craziness to populate the image. and have to write something about diskimage requiring root [14:42:38] we will probably build the images ourselves and upload them manually [14:42:47] and thus get rid of the automatic disk image building process [14:42:56] beside that, should be fine :) [14:43:06] hm, that's too bad... but may be a better option [14:43:46] I looked a bit at fakeroot fakechroot etc [14:44:00] but that does not play well. diskimage uses a bunch of sudo commands intenrally [14:44:29] is the main issue getting apt to point at a wmf repo? [14:55:42] andrewbogott: na fixed it up by injecting the related files directly [14:56:44] andrewbogott: the drama Dan and I surfaced is attempting to reuse the contint puppet module from operations/puppet [14:56:58] oh yeah, that :( [14:57:10] doesn't work well in a bootstrapping image. Turns out Bryan Davis had a similar issue while attempting to make a Vagrant image [14:57:36] so maybe we will end up with our own puppet repo :/ [14:57:56] I have some ideas for how to fix bootstrapping for use of ops/puppet [14:58:39] I talked to Faidon and got a few pointers. Basically there are some things that are done to the base images for prod/labs that we need to expose [14:59:29] now I just need to figure out how to sleep less and code more :) [15:01:47] a [15:01:54] here is the VagrantFile magic patch https://gerrit.wikimedia.org/r/#/c/212294/ :D [15:23:28] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:14:16] 6Release-Engineering, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1460542 (10Tgr) >>! In T102566#1460075, @Tau wrote: > In both php.ini files the ";open_basedir= " (blank). Is it oka... [16:32:43] 10Deployment-Systems, 7I18n, 7Wikimedia-log-errors: l10n-update not updating Vector - https://phabricator.wikimedia.org/T103879#1460613 (10Nemo_bis) `git log i18n` assists us; looks like the error keeps occurring: https://ky.wikipedia.org/wiki/MediaWiki:Vector-more-actions is still in English, AKA 800b50ffbd... [16:43:30] 10Beta-Cluster, 10Deployment-Systems, 7I18n: Beta dewiki's strings are in English when logged in (does not happen in debug mode or other languages) - https://phabricator.wikimedia.org/T106019#1460655 (10Luke081515) 5Open>3Resolved a:3Luke081515 Should be resolved now, since my last comment it was and i... [16:47:36] 10Continuous-Integration-Infrastructure: Request Jenkins shell access for account "sniedzielski" - https://phabricator.wikimedia.org/T103192#1460671 (10hashar) >>! In T103192#1457734, @Dzahn wrote: > @hashar For some reason he doesn't have a home directory yet. How did you grant him access? > > root@integrati... [17:08:43] 6Release-Engineering, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1460783 (10Tau) Should the open_basedir be enabled then? What directory I should set to open_basedir? If I change t... [17:14:05] (03CR) 10Paladox: [C: 031] Update MsUpload tests [integration/config] - 10https://gerrit.wikimedia.org/r/225231 (owner: 10Paladox) [18:07:49] PROBLEM - Free space - all mounts on deployment-fluorine is CRITICAL deployment-prep.deployment-fluorine.diskspace.root.byte_percentfree (<100.00%) [18:50:28] 6Release-Engineering, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1460950 (10Tgr) If it is not a large production website, I would just do `chmod -R a+rw /var/log/mediawiki`. [20:03:28] !log stopping Zuul to get rid of a faulty registered function "build:Global-Dev Dashboard Data". Job is gone already. [20:03:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:18:27] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL 100.00% of data above the critical threshold [0.0] [21:26:16] PROBLEM - Puppet failure on deployment-cache-text03 is CRITICAL 100.00% of data above the critical threshold [0.0] [21:45:00] !log upgraded nodepool to 0.0.1-104-gddd6003-wmf4 . That fix graceful stop via SIGUSR1 and let me complete the systemd integration [21:45:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:11:28] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Isolation, 6operations, 7Nodepool, 5Patch-For-Review: Use systemd for Nodepool - https://phabricator.wikimedia.org/T96867#1461538 (10hashar) Status update ------ I have finished the systemd patch: https://gerrit.wikimedia.org/r/#/c/224102/... [22:18:14] oh [22:18:20] {Day changed to Saturday, 18 July 2015} [22:18:26] so that is technically the end of the week [22:18:31] just in time [22:18:42] *wave* [22:21:18] so long hashar [22:21:38] andrewbogott: yeah had to fix some magic in nodepool itself :) [22:21:56] andrewbogott: will get the patches reviewed / merged with european folks [22:22:07] and sprint the image creation with puppet [22:22:16] going to be busy next week [23:35:50] 10Beta-Cluster, 10MediaWiki-General-or-Unknown, 7Wikimedia-log-errors: Viewing beta's en_rtlwiki just throws a fatal exception - https://phabricator.wikimedia.org/T106125#1461632 (10demon) Does enrtl even work lol?