[00:05:36] (03CR) 10Dzahn: "final attempt to ping - any opinion? i think otherwise we should give up on it" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/316230 (https://phabricator.wikimedia.org/T22343) (owner: 10Paladox) [00:06:41] (03CR) 10Dzahn: "@Paladox please see new comments. Do you think you will amend to it or abandon?" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/296060 (https://phabricator.wikimedia.org/T124572) (owner: 10Paladox) [00:10:32] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Scap, 10Patch-For-Review: Deploy gerrit with scap3 - https://phabricator.wikimedia.org/T157414#3004394 (10Dzahn) I was gonna merge https://gerrit.wikimedia.org/r/#/c/374667/3 but when i looked at /var/lib/gerrit2/review_site/plugins vs. /srv/deployment/ger... [00:18:43] Project selenium-Flow » chrome,beta,Linux,BrowserTests build #514: 04FAILURE in 2 min 42 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/514/ [01:10:18] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [01:45:18] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [02:02:17] PROBLEM - Puppet staleness on deployment-kafka-jumbo-1 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [43200.0] [02:06:18] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [03:16:19] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [04:37:17] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [05:03:01] 10Release-Engineering-Team (Kanban), 10Phabricator: Custom task form for #WMF-CTO-Team-Backlog - https://phabricator.wikimedia.org/T175869#3609722 (10mmodell) https://phabricator.wikimedia.org/maniphest/task/edit/form/41/ [05:17:18] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [05:33:50] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [05:44:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:08:17] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:14:51] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:43:17] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [06:46:27] 10Gerrit, 10Repository-Ownership-Requests, 10VPS-project-libraryupgrader: Grant libraryupgrader +2 rights for some library bumps - https://phabricator.wikimedia.org/T174760#3609753 (10Legoktm) So are there any remaining issues with giving libraryupgrader +2? [06:57:19] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [07:09:19] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:40:00] 10Beta-Cluster-Infrastructure: Beta cluster rights clarification - https://phabricator.wikimedia.org/T175917#3609779 (10Sau226) https://meta.wikimedia.beta.wmflabs.org/wiki/Special:ListGroupRights @Aklapper @Samtar I mean WMF office IT and WMF Trust and Safety. https://meta.wikimedia.org/wiki/Meta:WMF_Support_a... [07:40:07] Project selenium-Wikibase » chrome,test,Linux,BrowserTests build #484: 15ABORTED in 3 hr 0 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=BrowserTests/484/ [07:44:18] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [07:45:59] 10Deployment-Systems, 10Release-Engineering-Team (Backlog), 10MediaWiki-Special-pages: All extension should link to the deployed Git revision from Special:Version - https://phabricator.wikimedia.org/T155747#2953229 (10Seb35) In my understanding, there are at least two different cases about why some extension... [08:05:16] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:26:11] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): integration-slave-jessie-1003 (and others?) missing jsduck executable - https://phabricator.wikimedia.org/T175764#3609823 (10hashar) [09:13:23] 10Beta-Cluster-Infrastructure: Beta protection error - https://phabricator.wikimedia.org/T175981#3609899 (10Sau226) [09:39:05] addshore: yesterday the kabal has settled a name for the project to dockerify the CI stack #ci-shipyard https://phabricator.wikimedia.org/project/board/3008/ :D [09:39:17] 10Release-Engineering-Team (Kanban): wmf.14 Blocker - Post Mortem - Cannot flush pre-lock snapshot because writes are pending - https://phabricator.wikimedia.org/T173477#3609935 (10aaron) * PROBLEM: in LinksUpdate, runForTitle() starting off with acquirePageLock(), then calling doUpdate() for the secondary updat... [09:39:23] since you proposed to fill tasks, you can use that. Keywords being #ci-docker and #ci-shipyard [09:39:38] the project picture comes from my home town :D [09:43:34] Woohoo! [09:44:04] 10Continuous-Integration-Infrastructure (shipyard): CI docker build should use an apt cache - https://phabricator.wikimedia.org/T175966#3609967 (10hashar) The `wikimedia-jessie` base image is generated via bootstrap-vz with: ``` plugins: minimize_size: apt: autoclean: true languages: [none]... [09:46:49] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [10:05:30] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10TCB-Team, and 2 others: Deploy new Wikidiff2 version on beta-cluster - https://phabricator.wikimedia.org/T175818#3610032 (10MoritzMuehlenhoff) I cherrypicked the two commits into a 1.4.1+wmf2 package, uploaded... [10:29:04] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10TCB-Team, and 2 others: Deploy new Wikidiff2 version on beta-cluster - https://phabricator.wikimedia.org/T175818#3610041 (10Tobi_WMDE_SW) @MoritzMuehlenhoff great, thanks! I think what's still missing is settin... [10:30:13] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10TCB-Team, and 2 others: Deploy new Wikidiff2 version on beta-cluster - https://phabricator.wikimedia.org/T175818#3610042 (10Addshore) >>! In T175818#3610041, @Tobi_WMDE_SW wrote: > @MoritzMuehlenhoff great, tha... [10:31:13] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10TCB-Team, and 2 others: Deploy new Wikidiff2 version on beta-cluster - https://phabricator.wikimedia.org/T175818#3610043 (10Tobi_WMDE_SW) @addshore yay, great! [10:45:19] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [10:54:33] 10Deployment-Systems, 10Scap (Scap3-Adoption-Phase1), 10scap2, 10OCG-PDFRenderer, 10Services (watching): Deploy ocg with scap3 - https://phabricator.wikimedia.org/T129142#2096453 (10MoritzMuehlenhoff) Salt (on which Trebuchet is build on) is being removed by end of September. in TechOps we'll need a few... [11:02:53] AJJHHMHHM [11:03:07] my whole .plan vanished. docker build doesn't seem to support mounting a volume [11:05:03] 10Beta-Cluster-Infrastructure, 10Salt: deployment-imagescaler02 is not responding to salt - https://phabricator.wikimedia.org/T173628#3610078 (10MoritzMuehlenhoff) 05Open>03declined Salt is being removed. [11:06:16] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:18:23] https://github.com/moby/moby/issues/14080 "build time only -v option" [12:13:36] 10Beta-Cluster-Infrastructure: Beta cluster rights clarification - https://phabricator.wikimedia.org/T175917#3610251 (10Sau226) [12:22:58] Yippee, build fixed! [12:22:59] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #526: 09FIXED in 57 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/526/ [12:53:08] ok [12:53:19] I am fed up with Docker teenagers already [12:54:31] 10Scap (Scap3-Adoption-Phase1), 10releng-201516-q4, 10releng-201718-q1, 10Trebuchet: [keyresult] Migrate remaining trebuchet deployed services - https://phabricator.wikimedia.org/T129290#3610305 (10thcipriani) [12:54:34] 10Deployment-Systems, 10Scap (Scap3-Adoption-Phase1), 10scap2, 10OCG-PDFRenderer, and 2 others: Deploy ocg with scap3 - https://phabricator.wikimedia.org/T129142#3610303 (10thcipriani) 05Open>03declined >>! In T129142#3610052, @MoritzMuehlenhoff wrote: > Is the plan to still move ocg to scap3 despite i... [13:00:47] 10Beta-Cluster-Infrastructure: Beta cluster rights clarification - https://phabricator.wikimedia.org/T175917#3610332 (10Samtar) @Sau226 it would be highly unlikely these groups would be granted to non-WMF accounts, even on the beta cluster, given their intended use. Did you need a specific right granted by one... [13:08:24] 10Beta-Cluster-Infrastructure: Beta protection error - https://phabricator.wikimedia.org/T175981#3609899 (10Samtar) > a user Which? "User:Sau226"? > a spam page Which page was it you opened? Roughly what time did this occur? Please have a look [[ https://www.mediawiki.org/wiki/How_to_report_a_bug | at this p... [13:11:14] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10TCB-Team, and 3 others: Deploy new Wikidiff2 version on beta-cluster - https://phabricator.wikimedia.org/T175818#3610346 (10Addshore) Now set on beta! [13:11:38] hashar: docker teenagers? :P [13:16:16] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [13:23:58] addshore: I am just ranting/trolling. It is friday after all :] [13:24:00] but in short [13:24:05] I am not super fan of Dockerfile [13:24:37] really I am just thinking of doing something based on run-parts :D [13:24:47] COPY . /stage0 [13:24:52] RUN run-parts /stage0 [13:24:55] COPY . /stage1 [13:25:00] RUN run-parts /stage1 [13:25:01] etc [13:25:03] ;D [13:25:35] though I havent played yet with the multi stage thingie which might solve some of ideas I have in mind [13:27:55] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10TCB-Team, and 3 others: Deploy new Wikidiff2 version on beta-cluster - https://phabricator.wikimedia.org/T175818#3610358 (10Tobi_WMDE_SW) Confirming that it seems to work: https://en.wikipedia.beta.wmflabs.org/... [13:30:15] multi stage is nice [13:35:38] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10TCB-Team, and 3 others: Deploy new Wikidiff2 version on beta-cluster - https://phabricator.wikimedia.org/T175818#3610361 (10Tobi_WMDE_SW) 05Open>03Resolved [13:37:17] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:45:39] addshore: pip3 install --user urwid && pip3 install --user sen [13:45:52] "sen" being a ncurse like interface to itneract with the docker-engine :] [13:46:30] shows up all the images and by pressing "enter" all the layers :D [13:51:08] ooooh [14:05:27] hashar: No module named 'docker.errors' [14:05:31] serisouly i hate python [14:05:32] just fucking work [14:05:44] haha [14:05:45] ;( [14:06:12] everytime i try to use any python package it just doesnt work [14:06:30] unlike php which always work (tm) [14:07:00] I have no clue what could be wrong sorry :( [14:07:05] surprisingly it does for me :P [14:09:52] hashar: also need "docker-py" [14:10:35] NameError: name 'fcntl' is not defined [14:11:06] Urwid doesn't have windows support at the moment. Maybe consider python prompt toolkit instead? [14:11:30] thats what really pisses me off about python, just fucking work cross platform.... please.... [14:17:16] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [14:41:58] addshore: RUN IT IN A CONTAINER PLEASE ! ;:]]]]]]]]]]]]]]]]] [14:43:12] addshore: or install bash from Microsoft/Ubuntu. That might give you an env sane enough for urwid [14:43:18] but yeah, I feel your pain :( [14:43:47] anyway [14:44:01] I have read too much doc this afternoon. Will check in later tonight [14:48:53] hasharAway: docker pull tomastomecek/sen [14:48:59] why didnt you just do that ';) [15:08:16] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:24:43] Project selenium-MobileFrontend » chrome,beta,Linux,BrowserTests build #558: 04FAILURE in 2 min 42 sec: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/558/ [15:43:17] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [15:49:59] 10Beta-Cluster-Infrastructure: Beta protection error - https://phabricator.wikimedia.org/T175981#3610791 (10Aklapper) [ @Sau226: Please do not remove me as a subscriber from tasks. Thanks. ] [15:51:11] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:00:36] 10Beta-Cluster-Infrastructure, 10Multimedia, 10Thumbor, 10Multimedia-Team-Working-Board: On beta commons, thumbnailing of 3D files is broken still - https://phabricator.wikimedia.org/T170444#3610855 (10MBinder_WMF) [16:03:39] 10Release-Engineering-Team (Kanban), 10Phabricator: Custom task form for #WMF-CTO-Team-Backlog - https://phabricator.wikimedia.org/T175869#3610896 (10ksmith) @mmodell : With that form, would we leave the other dropdown set to "Public", even for private tasks? [16:09:18] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:34:07] 10Release-Engineering-Team (Kanban), 10Phabricator: Custom task form for #WMF-CTO-Team-Backlog - https://phabricator.wikimedia.org/T175869#3610955 (10mmodell) @ksmith Yeah. the space is applied first then the view policy so a space is all that's required I believe. [17:21:00] 10Deployment-Systems, 10Release-Engineering-Team (Kanban), 10Scap (Scap3-Adoption-Phase1), 10scap2, and 3 others: Deploy jobrunner with scap3 (Trebuchet jobrunner/jobrunner) - https://phabricator.wikimedia.org/T129148#3611066 (10greg) >>! In T129148#3582888, @thcipriani wrote: > @Krinkle is there jobrunner... [17:22:54] 10Release-Engineering-Team (Kanban), 10Phabricator, 10User-Zppix: Unhandled Exception ("AphrontParameterQueryException") when viewing T174362 - https://phabricator.wikimedia.org/T175942#3611067 (10greg) 05Open>03Resolved [19:05:20] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Scap, 10Patch-For-Review: Deploy gerrit with scap3 - https://phabricator.wikimedia.org/T157414#3611246 (10Paladox) @Dzahn @demon has created https://gerrit.wikimedia.org/r/#/c/378192/ which includes its-phabricator :). Not sure if he built it around https:... [19:07:12] drafts are now disabled in gerrit :). [19:13:04] (03CR) 10Legoktm: "Another one is @cover -> @covers (c.f. Change-Id: I2d5f876c9945ba21f8748562230170e3c7ea2ff2)" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/377028 (owner: 10Umherirrender) [19:14:18] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [19:33:01] (03PS1) 10Jforrester: Add sniff for @cover instead of @covers [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/378268 [19:33:13] (03CR) 10Jforrester: "Done in Ief227924d36a739674549df0937439f1a68af8df." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/377028 (owner: 10Umherirrender) [19:40:16] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [20:41:40] 10Release-Engineering-Team (Kanban), 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499#3611316 (10MBinder_WMF) @mmodell I think having "Bug" as a type is really useful. We have avoided the tag in the past because we didn't want Phab tasks submitted as just "Bug" without a pro... [20:42:32] Yippee, build fixed! [20:42:32] Project selenium-Echo » firefox,beta,Linux,BrowserTests build #518: 09FIXED in 1 min 30 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/518/ [20:42:33] Yippee, build fixed! [20:42:33] Project selenium-Echo » chrome,beta,Linux,BrowserTests build #518: 09FIXED in 1 min 31 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/518/ [20:49:59] 10Release-Engineering-Team (Kanban), 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499#3611322 (10mmodell) @MBinder_WMF: I've made a form for submitting a bug report, it should be essentially the same as a regular task but it will be labeled as a bug report: [[ /transaction... [20:54:08] 10Release-Engineering-Team (Kanban), 10Trash: Example Bug Report - https://phabricator.wikimedia.org/T176020#3611324 (10mmodell) [20:54:35] 10Release-Engineering-Team (Kanban), 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499#3611340 (10mmodell) {T176020} [21:15:44] (03PS1) 10Hashar: dib: jsduck is now in operations/puppet [integration/config] - 10https://gerrit.wikimedia.org/r/378341 (https://phabricator.wikimedia.org/T175764) [21:45:17] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [21:49:59] 10Gerrit, 10Repository-Ownership-Requests, 10VPS-project-libraryupgrader: Grant libraryupgrader +2 rights for some library bumps - https://phabricator.wikimedia.org/T174760#3611453 (10demon) I think we'll be fine. [21:52:14] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Scap, 10Patch-For-Review: Deploy gerrit with scap3 - https://phabricator.wikimedia.org/T157414#3611454 (10demon) its-phabricator included and upstream #92830 included. [21:53:04] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Scap, 10Patch-For-Review: Deploy gerrit with scap3 - https://phabricator.wikimedia.org/T157414#3611455 (10Paladox) thanks :). I guess we are now ready for scap deploys :). [22:06:16] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:57:02] 10Release-Engineering-Team (Kanban), 10Operations, 10Release Pipeline, 10Patch-For-Review: Provision Docker >= 17.05 on contint1001 - https://phabricator.wikimedia.org/T175293#3611511 (10thcipriani) >>! In T175293#3599497, @hashar wrote: > Potentially the required upstream package could be added to a new c... [23:16:17] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [23:22:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Zuul refused to start from contint1001 - https://phabricator.wikimedia.org/T167833#3611549 (10Dzahn) first: 21:50 mutante: contint2001 - systemctl start zuul 21:47 mutante: contint1001 - tmp d... [23:27:48] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Zuul: Migrate zuul-server behind systemd service - https://phabricator.wikimedia.org/T167845#3611554 (10Dzahn) 05Open>03Resolved 17:17 <+wikibugs> (PS24) Paladox: Zuul: Add s... [23:37:16] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]