[00:23:56] 10Gerrit, 10Patch-For-Review, 10User-Ladsgroup: Make gerrit use the new WMF logo - https://phabricator.wikimedia.org/T174576#3566488 (10Legoktm) I agree with @demon's sentiments. As an alternative proposal, could we use a custom logo for Gerrit based off of the Wikimedia Community Logo? https://commons.wikim... [00:33:50] RainbowSprinkles: I discovered that gitlab actually properly supports mirrors (you give them the URL and they regularly pull it), would you have any objections to me setting up mirrors on gitlab a script that just syncs the repo list with Gerrit? [00:34:24] I'd rather not add more mirrors of code nobody cares is mirrored :( [00:34:34] "Mirror everything" was a bad decision [00:35:12] I'm also curious what "regularly" means [00:35:15] For all repos [00:35:19] but! driveby contributions! [00:35:38] * legoktm finds docs [00:35:46] * RainbowSprinkles gives greg-g an evil look [00:36:32] Once you activate the pull mirroring feature, the mirror will be inserted into a queue. A scheduler will start every minute and schedule a fixed amount of mirrors for update, based on the configured maximum capacity. [00:36:32] If the mirror successfully updates it will be enqueued once again with a small backoff period. [00:36:53] ...that sounds like it could DOS gerrit [00:38:17] I read it as it gets put into a giant queue for all of gitlab.com and only a few get pulled at a time, but it's not very clear, I can ask. [00:38:48] My main motivation for this was to have an alternative repo viewer that is closer to free software (aka not Github) [00:40:21] If only we had one.... [00:40:58] I've been tempted to write a bot that anytime someone puts a github link on IRC it responds with the appropriate Phab link :p [00:41:19] RainbowSprinkles: oo, that'd be a fun addition to stashbot [00:42:15] greg-g: gitlab lets you disable PRs and issues, so I was thinking it could just be a pure repo viewer and not allow people to leave PRs that don't get looked at ever [00:42:45] I prefer to just pretend the PRs don't exist at all so I won't lose sleep over them :P [00:42:54] (also I'd enjoy not replicating repos nobody cares about replicating) [00:43:59] RIP my gitlab resume /s [00:59:02] If phab's git browser was remotely close to as usable as github is, I'd use it more. I often use github search to find things and then go through the pain of using diffusion to find the same file to link in a task. It's really annoying. [01:12:41] We should totally go back to gitblit [01:12:56] Actually, I considered just installing cgit on the gerrit machine and doing it within gerrit itself [01:13:09] It'd be easier than constantly playing whack a mole when we pick a new favorite git viewer [01:32:12] I like cgit [01:32:45] But I don't think that addresses the search feature? [01:42:08] 10Scap (Scap3-Adoption-Phase1), 10releng-201516-q4, 10releng-201718-q1, 10Trebuchet: [keyresult] Migrate remaining trebuchet deployed services - https://phabricator.wikimedia.org/T129290#3568749 (10Krinkle) [02:09:06] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Beta should not use productions interwiki.cdb - https://phabricator.wikimedia.org/T69931#3568784 (10Mattflaschen-WMF) 05Resolved>03Open a:05Krenair>03Mattflaschen-WMF This was reverted (and thus re-opened) on Sep 14, 2015: {0feb37f45a89dfb4273ba09f907e... [02:25:35] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Beta should not use productions interwiki.php - https://phabricator.wikimedia.org/T69931#3568797 (10Mattflaschen-WMF) [02:26:25] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Beta should not use productions interwiki.php - https://phabricator.wikimedia.org/T69931#3568799 (10Mattflaschen-WMF) Now that interwiki.php is committed and the script works fine, this should be fine. [03:13:48] 10Scap (Scap3-Adoption-Phase1), 10Trebuchet: Cleanup /srv/deployment - https://phabricator.wikimedia.org/T170881#3568834 (10demon) [03:14:41] 10Scap (Scap3-Adoption-Phase1), 10Trebuchet: Cleanup /srv/deployment - https://phabricator.wikimedia.org/T170881#3446038 (10demon) [03:17:08] 10Scap (Scap3-Adoption-Phase1), 10Trebuchet: Cleanup /srv/deployment - https://phabricator.wikimedia.org/T170881#3446038 (10demon) Actually, most of these had wikidev group ownership so dropped them myself. * `grafana/*` needs a root, got some files in `.git/` owned by them * `kibana/*` I just want confirmati... [03:18:04] 10Scap (Scap3-Adoption-Phase1), 10Trebuchet: Cleanup /srv/deployment - https://phabricator.wikimedia.org/T170881#3568838 (10demon) [03:23:46] Project beta-scap-eqiad build #171035: 04FAILURE in 0.45 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/171035/ [03:36:25] Yippee, build fixed! [03:36:26] Project beta-scap-eqiad build #171036: 09FIXED in 2 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/171036/ [04:17:04] Yippee, build fixed! [04:17:05] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #502: 09FIXED in 21 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/502/ [04:46:49] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:41:53] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [05:56:52] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [10.0] [06:17:03] (03PS1) 10Legoktm: Whitelist libraryupgrader [integration/config] - 10https://gerrit.wikimedia.org/r/374937 [06:17:18] (03CR) 10Legoktm: [C: 032] "The bots are coming" [integration/config] - 10https://gerrit.wikimedia.org/r/374937 (owner: 10Legoktm) [06:18:16] (03Merged) 10jenkins-bot: Whitelist libraryupgrader [integration/config] - 10https://gerrit.wikimedia.org/r/374937 (owner: 10Legoktm) [06:19:00] legoktm: I just saw that upload from that bot, is there a wiki page about it? [06:19:05] * greg-g is just curious [06:19:08] yes! [06:19:16] https://www.mediawiki.org/wiki/User:Legoktm/library_upgrader [06:19:33] bd [06:20:34] ? [06:21:15] two thumbs up [06:21:32] hah :D [06:21:37] (I'm trying to socialize that emoticon) [06:21:57] * greg-g goes to sleep [06:25:36] !log deploying https://gerrit.wikimedia.org/r/374937 [06:25:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:48:01] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<30.00%) [07:03:00] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:18:08] (03CR) 10WMDE-leszek: "Thanks Hashar, I missed the other list!" [integration/config] - 10https://gerrit.wikimedia.org/r/373077 (https://phabricator.wikimedia.org/T173831) (owner: 10WMDE-leszek) [08:01:11] Is there any explanation for why jenkins didn't test the build at all and isn't even a subscriber? [08:01:19] *a build [08:01:30] patch here FYI https://gerrit.wikimedia.org/r/#/c/374682/ [08:10:03] PROBLEM - Free space - all mounts on deployment-kafka01 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<100.00%) [08:12:29] Reception123: Probably the extension isn't setup in CI at all [08:20:00] oh, that would probably explain it :P [08:21:06] shouldn't all extensions inherit mediawiki-extensions-* therefore be setup, or is the setup something that has to be done manually by a maintainer? [08:21:21] The latter [08:22:46] oh, well then it is most likely the case [08:25:42] I found another case with https://gerrit.wikimedia.org/r/#/c/372489/ . For another patch, it is clear that the CI works, but for this one it seems to not be doing anything [08:26:34] Whitelisted user? [08:26:53] I don't think so, but the basic test build should work nonetheless? [08:35:30] Depends what tests are defined for a repo AFAIK [08:36:33] 10Scap (Scap3-Adoption-Phase1), 10Trebuchet: Cleanup /srv/deployment - https://phabricator.wikimedia.org/T170881#3569148 (10hashar) [08:37:08] hmm, ok [08:41:02] 10Scap (Scap3-Adoption-Phase1), 10Trebuchet: Cleanup /srv/deployment - https://phabricator.wikimedia.org/T170881#3569179 (10hashar) [08:46:22] 10Scap (Scap3-Adoption-Phase1), 10Trebuchet: Cleanup /srv/deployment - https://phabricator.wikimedia.org/T170881#3569218 (10hashar) I have cleared out all of/srv/deployment/integration/ @fgiunchedi kindly deleted /srv/deployment/grafana [08:46:29] 10Scap (Scap3-Adoption-Phase1), 10Trebuchet: Cleanup /srv/deployment - https://phabricator.wikimedia.org/T170881#3569219 (10hashar) [08:46:51] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [08:48:23] 10Scap (Scap3-Adoption-Phase1), 10Trebuchet: Cleanup /srv/deployment - https://phabricator.wikimedia.org/T170881#3446038 (10hashar) From the original list that task has, everything got cleaned up. Might want to revisit /srv/deployment and see whether anything still need to be deleted. /srv/deployment/STALE c... [08:59:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Upgrade git package on zuul-merger instances contint1001 / contint2001 to benefit git-daemon - https://phabricator.wikimedia.org/T161086#3569233 (10hashar) Upgraded on contint2001 ``` # apt-get install -t jessie... [09:04:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Upgrade git package on zuul-merger instances contint1001 / contint2001 to benefit git-daemon - https://phabricator.wikimedia.org/T161086#3569238 (10hashar) 05Open>03Resolved [09:13:04] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Update gerrit to 2.14.3 - https://phabricator.wikimedia.org/T156120#3569248 (10Paladox) [09:13:06] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3569247 (10Paladox) [09:13:12] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3030929 (10Paladox) [09:17:23] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-Newsletter, 10Patch-For-Review, 10Regression: HTTP Error 500 when trying to publish a Newsletter issue in test.wikipedia.org - https://phabricator.wikimedia.org/T174605#3569262 (10Qgil) Confirmed, it works now. Thank you very much! [09:17:31] (03CR) 10Alexandros Kosiaris: [C: 04-1] "I this this is no longer relevant. Isn't operations-puppet-rake-jessie deprecated ?" [integration/config] - 10https://gerrit.wikimedia.org/r/331861 (https://phabricator.wikimedia.org/T78342) (owner: 10Hashar) [09:47:16] (03PS1) 10Zfilipin: Set up skin dependencies for mwext-mw-rspec [integration/config] - 10https://gerrit.wikimedia.org/r/374962 (https://phabricator.wikimedia.org/T164479) [09:47:54] hashar: please take a look if you have a minute [09:47:55] https://gerrit.wikimedia.org/r/#/c/374962/ [10:00:03] (03CR) 10Zfilipin: [C: 032] Set up skin dependencies for mwext-mw-rspec [integration/config] - 10https://gerrit.wikimedia.org/r/374962 (https://phabricator.wikimedia.org/T164479) (owner: 10Zfilipin) [10:02:59] (03Merged) 10jenkins-bot: Set up skin dependencies for mwext-mw-rspec [integration/config] - 10https://gerrit.wikimedia.org/r/374962 (https://phabricator.wikimedia.org/T164479) (owner: 10Zfilipin) [10:06:00] !log Reloading Zuul to deploy e91b30ff6df435a6a3c55fe978be6bbf1f1cbde4 [10:06:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:09:26] (03PS1) 10Hashar: debian-glue jobs with BACKPORTS=yes [integration/config] - 10https://gerrit.wikimedia.org/r/374970 (https://phabricator.wikimedia.org/T173999) [10:13:36] (03PS2) 10Hashar: debian-glue jobs with BACKPORTS=yes [integration/config] - 10https://gerrit.wikimedia.org/r/374970 (https://phabricator.wikimedia.org/T173999) [10:13:52] (03CR) 10Hashar: "PS2 actually switch cumin to use backports." [integration/config] - 10https://gerrit.wikimedia.org/r/374970 (https://phabricator.wikimedia.org/T173999) (owner: 10Hashar) [10:15:03] 10Continuous-Integration-Config, 10Patch-For-Review: CI job debian-glue-non-voting: add support for BACKPORTS=yes - https://phabricator.wikimedia.org/T173999#3547757 (10hashar) Volans pinged me about it and I went with a basic implementation based on Tyler comment above. [10:17:49] (03CR) 10jerkins-bot: [V: 04-1] debian-glue jobs with BACKPORTS=yes [integration/config] - 10https://gerrit.wikimedia.org/r/374970 (https://phabricator.wikimedia.org/T173999) (owner: 10Hashar) [11:13:40] (03CR) 10Volans: "thanks for the implementation, see a comment inline." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/374970 (https://phabricator.wikimedia.org/T173999) (owner: 10Hashar) [11:26:33] (03CR) 10Hashar: debian-glue jobs with BACKPORTS=yes (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/374970 (https://phabricator.wikimedia.org/T173999) (owner: 10Hashar) [11:27:52] (03PS3) 10Hashar: debian-glue jobs with BACKPORTS=yes [integration/config] - 10https://gerrit.wikimedia.org/r/374970 (https://phabricator.wikimedia.org/T173999) [11:54:31] PROBLEM - Puppet errors on deployment-imagescaler02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:22:46] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #511: 04FAILURE in 45 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/511/ [12:29:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Wiki-Loves-Monuments-Database, 10User-JeanFred: Automate deployment of heritage on Gerrit post-merge - https://phabricator.wikimedia.org/T157893#3569685 (10hashar) @Legoktm on T157893#3054609 commented how the CI Jenkins update wikibugs... [13:37:59] (03PS1) 10Zfilipin: Deleted RelatedArticles Selenium Ruby Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/374991 (https://phabricator.wikimedia.org/T139740) [13:39:20] (03CR) 10Jdlrobson: [C: 031] Deleted RelatedArticles Selenium Ruby Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/374991 (https://phabricator.wikimedia.org/T139740) (owner: 10Zfilipin) [13:39:23] (03CR) 10Jdlrobson: [C: 031] "thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/374991 (https://phabricator.wikimedia.org/T139740) (owner: 10Zfilipin) [13:46:10] (03CR) 10Zfilipin: [C: 032] Deleted RelatedArticles Selenium Ruby Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/374991 (https://phabricator.wikimedia.org/T139740) (owner: 10Zfilipin) [13:46:28] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #508: 04FAILURE in 2 min 28 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/508/ [13:48:31] (03Merged) 10jenkins-bot: Deleted RelatedArticles Selenium Ruby Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/374991 (https://phabricator.wikimedia.org/T139740) (owner: 10Zfilipin) [13:56:17] 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: For MediaWiki extensions, merge rubocop inside mwext-mw-selenium-jessie - https://phabricator.wikimedia.org/T164479#3569907 (10zeljkofilipin) mwext-mw-rspec-jessie n... [13:58:00] 10Release-Engineering-Team (Kanban), 10Phabricator (Upstream), 10Upstream: Phabricator project page should not default to workboard - https://phabricator.wikimedia.org/T89865#3569913 (10MarthaSimons) Actually T127903 ...[[ https://www.essayhave.com | . ]]. [14:01:03] 10Release-Engineering-Team, 10Page-Previews, 10Readers-Web-Backlog, 10Epic: [EPIC] Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980#3569962 (10Jdlrobson) [14:08:06] (03PS1) 10Zfilipin: RSpec job is not longer RSpec specific, it is a generic Ruby job [integration/config] - 10https://gerrit.wikimedia.org/r/374996 (https://phabricator.wikimedia.org/T164479) [14:10:57] (03CR) 10Zfilipin: [C: 032] RSpec job is not longer RSpec specific, it is a generic Ruby job [integration/config] - 10https://gerrit.wikimedia.org/r/374996 (https://phabricator.wikimedia.org/T164479) (owner: 10Zfilipin) [14:13:39] 10Gerrit, 10Patch-For-Review, 10User-Ladsgroup: Make gerrit use the new WMF logo - https://phabricator.wikimedia.org/T174576#3570017 (10Ladsgroup) [14:16:07] (03CR) 10jerkins-bot: [V: 04-1] RSpec job is not longer RSpec specific, it is a generic Ruby job [integration/config] - 10https://gerrit.wikimedia.org/r/374996 (https://phabricator.wikimedia.org/T164479) (owner: 10Zfilipin) [14:29:48] (03PS1) 10Hashar: dib: mediawiki::packages::php5 is already included [integration/config] - 10https://gerrit.wikimedia.org/r/375001 [14:30:54] (03CR) 10Zfilipin: RSpec job is not longer RSpec specific, it is a generic Ruby job [integration/config] - 10https://gerrit.wikimedia.org/r/374996 (https://phabricator.wikimedia.org/T164479) (owner: 10Zfilipin) [14:31:04] (03CR) 10Hashar: "Upstreamed as https://gerrit.wikimedia.org/r/#/c/374999/" [integration/config] - 10https://gerrit.wikimedia.org/r/375001 (owner: 10Hashar) [14:31:11] (03CR) 10Zfilipin: [C: 032] RSpec job is not longer RSpec specific, it is a generic Ruby job [integration/config] - 10https://gerrit.wikimedia.org/r/374996 (https://phabricator.wikimedia.org/T164479) (owner: 10Zfilipin) [14:34:37] (03Merged) 10jenkins-bot: RSpec job is not longer RSpec specific, it is a generic Ruby job [integration/config] - 10https://gerrit.wikimedia.org/r/374996 (https://phabricator.wikimedia.org/T164479) (owner: 10Zfilipin) [14:50:05] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10JavaScript, 10Patch-For-Review, and 2 others: Port Echo Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T171848#3570138 (10zeljkofilipin) [14:52:18] !log Reloading Zuul to deploy dd504839b3a48f431199785421751a5e3e38af57 [14:52:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:53:59] 10Release-Engineering-Team (Kanban), 10ContentTranslation, 10Browser-Tests, 10User-zeljkofilipin: ContentTranslation: Migrate browser tests from Ruby to node.js - https://phabricator.wikimedia.org/T174595#3570147 (10zeljkofilipin) a:03zeljkofilipin [14:59:35] 10Release-Engineering-Team (Kanban), 10ContentTranslation, 10Browser-Tests, 10User-zeljkofilipin: ContentTranslation: Migrate browser tests from Ruby to node.js - https://phabricator.wikimedia.org/T174595#3570154 (10zeljkofilipin) p:05Triage>03Low [15:22:40] (03PS1) 10Zfilipin: Do not run Ruby job for ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/375013 [15:24:52] (03PS2) 10Zfilipin: Do not run Ruby job for ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/375013 (https://phabricator.wikimedia.org/T174595) [15:37:01] (03CR) 10Thcipriani: [C: 032] debian-glue jobs with BACKPORTS=yes [integration/config] - 10https://gerrit.wikimedia.org/r/374970 (https://phabricator.wikimedia.org/T173999) (owner: 10Hashar) [15:39:32] (03CR) 10jerkins-bot: [V: 04-1] debian-glue jobs with BACKPORTS=yes [integration/config] - 10https://gerrit.wikimedia.org/r/374970 (https://phabricator.wikimedia.org/T173999) (owner: 10Hashar) [15:39:45] thcipriani: jobs are not created :D [15:40:27] hashar: what do you mean? I just made these: https://integration.wikimedia.org/ci/view/Ops-DebGlue/job/debian-glue-backports/ https://integration.wikimedia.org/ci/view/Ops-DebGlue/job/debian-glue-backports-non-voting/ ? [15:40:56] 00:00:40.590 FAILURE: Job debian-glue-backports-non-voting not defined [15:41:04] so I guess you created them after the gate job already ran [15:41:20] race condition! [15:41:50] ah... [15:41:53] * thcipriani tries again [15:42:09] meanwhile I am doing: disk-image-create debian vm -t docker --docker-target dibdebian [15:42:17] (03CR) 10Thcipriani: debian-glue jobs with BACKPORTS=yes [integration/config] - 10https://gerrit.wikimedia.org/r/374970 (https://phabricator.wikimedia.org/T173999) (owner: 10Hashar) [15:42:23] 10Gerrit, 10Patch-For-Review, 10User-Ladsgroup: Make gerrit use the new WMF logo - https://phabricator.wikimedia.org/T174576#3570298 (10demon) >>! In T174576#3568659, @Legoktm wrote: > I agree with @demon's sentiments. As an alternative proposal, could we use a custom logo for Gerrit based off of the Wikimed... [15:42:27] and maybe I can manage to craft a docker image that identical to the ones we have on nodepool [15:42:30] (03CR) 10Thcipriani: [C: 032] "Jobs created now" [integration/config] - 10https://gerrit.wikimedia.org/r/374970 (https://phabricator.wikimedia.org/T173999) (owner: 10Hashar) [15:44:09] (03Merged) 10jenkins-bot: debian-glue jobs with BACKPORTS=yes [integration/config] - 10https://gerrit.wikimedia.org/r/374970 (https://phabricator.wikimedia.org/T173999) (owner: 10Hashar) [15:44:43] there we go [15:44:47] * thcipriani deploys [15:44:51] OH MAN [15:44:54] I got a docker image [15:45:30] how huge is it? :P [15:46:03] 540MB [15:46:07] but that is a basic one [15:47:15] that's not too terrible for being an image that could run all tests, really [15:48:40] na that one is just debian + some cruft [15:48:45] that was to test out docker [15:48:50] and i am now running the real one [15:49:41] (03PS1) 10Hashar: dib: produce Docker image [integration/config] - 10https://gerrit.wikimedia.org/r/375018 [15:49:57] !log reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/374970/ [15:50:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:50:06] (03CR) 10Hashar: "And I am running it locally with:" [integration/config] - 10https://gerrit.wikimedia.org/r/375018 (owner: 10Hashar) [15:51:03] I also managed to free up ~30G of SSD disk space on my machine [15:51:09] after deleting bunch of docker containers and images [15:51:11] they pill up fast [15:54:09] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: CI job debian-glue-non-voting: add support for BACKPORTS=yes - https://phabricator.wikimedia.org/T173999#3570345 (10thcipriani) 05Open>03Resolved a:03hashar >>! In T173999#3569473, @hashar wrote: > Volans pinged m... [15:54:29] docker is definitely a sneaky space user [15:55:17] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: CI job debian-glue-non-voting: add support for BACKPORTS=yes - https://phabricator.wikimedia.org/T173999#3570349 (10hashar) I have done a recheck of the latest change that happened in the debian branch: https://gerrit.w... [15:58:50] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: CI job debian-glue-non-voting: add support for BACKPORTS=yes - https://phabricator.wikimedia.org/T173999#3570369 (10hashar) The D02backports uses: 00:00:30.551 Get:6 http://apt.wikimedia.org jessie-wikimedia/backport... [16:07:56] 10Scap (Scap3-Adoption-Phase1), 10Trebuchet: Cleanup /srv/deployment - https://phabricator.wikimedia.org/T170881#3570423 (10demon) Dropped most of STALE, just needs a root to hit the remaining few files. Then we can resolve this. [16:13:05] Error: Duplicate declaration: Class[Apt] is already declared; cannot redeclare at /tmp/in_target.d/wikimedia-puppet/ciimage.pp:39 on node pico [16:13:09] bah something changed [16:20:40] (03PS2) 10Hashar: dib: produce Docker image [integration/config] - 10https://gerrit.wikimedia.org/r/375018 [16:20:42] (03PS1) 10Hashar: dib: invoke ::apt before ::zuul [integration/config] - 10https://gerrit.wikimedia.org/r/375025 [16:20:51] bah [16:20:53] include ::zuul [16:20:58] causes apt to be included [16:21:14] with apt $use_proxy=true when I want it false [16:21:32] (03CR) 10Hashar: [C: 032] "That fixed it" [integration/config] - 10https://gerrit.wikimedia.org/r/375025 (owner: 10Hashar) [16:22:56] (03Merged) 10jenkins-bot: dib: invoke ::apt before ::zuul [integration/config] - 10https://gerrit.wikimedia.org/r/375025 (owner: 10Hashar) [17:13:45] 10Deployment-Systems, 10Scap, 10Patch-For-Review: Update Debian Package for Scap3 - https://phabricator.wikimedia.org/T127762#3570738 (10thcipriani) 05Resolved>03Open Pong :) Hi @fgiunchedi just tagged debian/3.7.0-1 on scap's release branch. Could you update scap on carbon please? Puppet patch on the w... [17:21:45] thcipriani: that scap 3.7.0 would fix up jobrunner isn't it ? [17:26:06] hashar: yep, among other fixes [17:26:42] after that is deployed, should be able to deploy jobrunner and have it restart both jobrunner and jobchron but only in the active datacenter. [18:10:39] 10Gerrit, 10Cleanup, 10utfnormal: php-normal unused? - https://phabricator.wikimedia.org/T174583#3570954 (10Aklapper) [18:21:12] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T170634#3571002 (10Jdlrobson) [18:38:36] the poor image is still being build :/ [18:40:22] wowzers [18:40:59] it timed out cloning some repo [18:52:18] 10Gerrit, 10Cleanup, 10utfnormal: php-normal unused? - https://phabricator.wikimedia.org/T174583#3571131 (10demon) Marked all Phab mirrors as archived, deleted github mirrors, marked all gerrit repos as read-only. [18:53:32] (03PS3) 10Hashar: dib: produce Docker image [integration/config] - 10https://gerrit.wikimedia.org/r/375018 [18:53:34] (03PS1) 10Hashar: dib: allow tmpfs [integration/config] - 10https://gerrit.wikimedia.org/r/375054 [18:56:22] ah [18:56:36] with tmpfs it is slightly faster [19:02:32] PROBLEM - Puppet errors on deployment-memc05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:07:26] all of that to slow down while doing the packages installs [19:25:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Cloud-VPS, 10Nodepool, and 2 others: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3571225 (10hashar) I have noticed: ``` # Security groups for big projects... [19:33:30] (03CR) 10Hashar: [C: 04-2] "I have found out why: the image is too big to fit in the default 4GB :D" [integration/config] - 10https://gerrit.wikimedia.org/r/375054 (owner: 10Hashar) [19:37:31] RECOVERY - Puppet errors on deployment-memc05 is OK: OK: Less than 1.00% above the threshold [0.0] [20:33:00] 51-puppet 2745.820 seconds [20:46:24] thcipriani: so I got a docker image :] 4.275 GB [20:47:36] and probably we can get a .qcow2 image and convert it to a docker image [20:55:26] that's....a big one :) [21:18:29] thcipriani: I blame texlive :( [21:19:17] at least I think I managed to validate that the php5.5 package on jessie does not seem to have side effects [21:19:26] so I will most probably include it on the nodepool images on monday [21:19:30] and add some test jobs for it [21:19:37] then I guess start moving some packages to it [21:19:43] PHP_BIN=php5.5 ! [21:21:12] hrm, yeah, texlive is going to take some room regardless. How big are the qcow2 images? [21:21:41] awesome that php5.5 on jessie works :) [21:31:18] thcipriani: and the ubuntu.qcow2 one is just 1.3GB [21:31:55] the jessie.qcow2 is 2.3GB [21:32:35] maybe they are compressed compared to what Docker reports [21:32:36] 0% 12MB 156.9KB/s 2:14:14 ETA [21:32:36] :((( [21:32:36] 2 hours.. [21:33:16] will uploaded them later I guess [21:33:24] it is bed time!! [22:29:07] 10Release-Engineering-Team (Kanban), 10User-greg: Distill OCG background into retrospective etherpad - https://phabricator.wikimedia.org/T174735#3571710 (10greg) [22:39:40] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Pywikibot-core, 10Pywikibot-tests: Run pywikibot test suite regularly on beta cluster as part of MediaWiki/Wikimedia CI - https://phabricator.wikimedia.org/T100903#3571765 (10Mattflaschen-WMF) [22:39:44] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Beta should not use productions interwiki.php - https://phabricator.wikimedia.org/T69931#3571764 (10Mattflaschen-WMF) 05Open>03Resolved [22:50:23] !log fixed deployment-ms-be0[34] puppet by removing cherry-pick of https://gerrit.wikimedia.org/r/#/c/371582/1 - details in a comment there [22:50:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:57:31] RECOVERY - Puppet errors on deployment-ms-be03 is OK: OK: Less than 1.00% above the threshold [0.0] [22:57:43] 10Beta-Cluster-Infrastructure, 10Analytics: deployment-kafka01 - disk is full - https://phabricator.wikimedia.org/T174742#3571849 (10Krenair) [22:59:05] 10Beta-Cluster-Infrastructure, 10Analytics: deployment-kafka01 - disk is full - https://phabricator.wikimedia.org/T174742#3571866 (10Krenair) [23:00:53] RECOVERY - Puppet errors on deployment-ms-be04 is OK: OK: Less than 1.00% above the threshold [0.0] [23:04:31] !log that also did deployment-cache-(upload|text)04 [23:04:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:14:07] RECOVERY - Puppet errors on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0] [23:15:47] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [23:17:20] RECOVERY - Puppet errors on deployment-ms-fe02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:20:24] 10Beta-Cluster-Infrastructure: deployment-imagescaler0[12] puppet broken due to missing nginx lua support - https://phabricator.wikimedia.org/T174746#3571937 (10Krenair) [23:23:01] 10Beta-Cluster-Infrastructure: deployment-imagescaler0[12] puppet broken due to missing nginx lua support - https://phabricator.wikimedia.org/T174746#3571955 (10Krenair) [23:31:49] 10Beta-Cluster-Infrastructure: deployment-imagescaler0[12] puppet broken due to missing nginx lua support - https://phabricator.wikimedia.org/T174746#3571977 (10Krenair) * Installed libnginx-mod-http-lua * Manually installed /etc/nginx/prometheus.lua file to avoid puppet catch-22, need to make the nginx service... [23:32:12] !log fixed deployment-imagescaler0[12] puppet by installing a package and file manually, some puppetisation still needed - https://phabricator.wikimedia.org/T174746 [23:32:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:40:38] RECOVERY - Puppet errors on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:44:30] RECOVERY - Puppet errors on deployment-imagescaler02 is OK: OK: Less than 1.00% above the threshold [0.0]