[00:50:29] 10Continuous-Integration-Infrastructure: Spike: Evaluate experimental Docker based CI w/ scap builds - https://phabricator.wikimedia.org/T150501#2788024 (10dduvall) [00:53:47] 10Continuous-Integration-Infrastructure: Set up experimental Docker CI slave - https://phabricator.wikimedia.org/T150502#2788048 (10dduvall) [00:54:14] \o/ [01:01:25] 10Continuous-Integration-Infrastructure: Define scap/tox job that runs unit tests within a Docker container - https://phabricator.wikimedia.org/T150504#2788099 (10dduvall) [01:07:00] 10Continuous-Integration-Infrastructure: Install and configure Jenkins Docker plugin - https://phabricator.wikimedia.org/T150505#2788120 (10dduvall) [01:07:21] 10Continuous-Integration-Infrastructure: Define scap/tox job that runs unit tests within a Docker container - https://phabricator.wikimedia.org/T150504#2788133 (10dduvall) a:05dduvall>03None [01:41:46] merged https://gerrit.wikimedia.org/r/#/c/317322/ [01:41:49] restarted gerrit [01:41:59] " Up the size for packedGitLimit to 2gb" [01:42:36] we are hoping it's good for performance [01:42:42] as suggested by 20after4 [01:43:01] :) [01:43:03] and patch by paladox [01:43:06] PROBLEM - Puppet run on integration-slave-precise-1002 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:43:10] looks fine so far [01:45:20] 10Continuous-Integration-Infrastructure: Define scap/tox job that runs unit tests within a Docker container - https://phabricator.wikimedia.org/T150504#2788099 (10mmodell) We could use harbormaster directly and bypass jenkins? [01:46:01] mutante: I finally updated https://gerrit.wikimedia.org/r/#/c/318662/ (Move config for git-ssh(phabricator) to hiera) [01:47:30] twentyafterfour: :) cool, i'll look for sure [01:49:41] no rush :) [01:52:03] 10Gerrit, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: Investigate why gerrit slowed down on 17/10/2016 / 18/10/2016 / 21/10/2016 - https://phabricator.wikimedia.org/T148478#2788190 (10Dzahn) We have now increased the packedGitLimit setting to 2g. Like @20after4 originally said on [1] "2... [01:54:15] twentyafterfour: fwiw, that link you originally used as reference [01:54:20] https://git.help.collab.net/entries/24136688-Memory-settings-in-Gerrit-configuration [01:54:29] asks for a login [01:55:08] expected? [02:23:05] RECOVERY - Puppet run on integration-slave-precise-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [04:03:51] mutante: no, not expected. weird [04:05:25] mutante: google cache: https://webcache.googleusercontent.com/search?q=cache:40pSyHHGscUJ:https://git.help.collab.net/entries/24136688-Memory-settings-in-Gerrit-configuration+&cd=1&hl=en&ct=clnk&gl=us [04:11:10] 03Scap3, 10scap: create an app to audibilize logstash events - https://phabricator.wikimedia.org/T123419#2788248 (10mmodell) 05stalled>03declined This is a silly idea (even if it is really cool) and realistically there will never be time to work on it. [04:11:54] 03Scap3, 10scap: Investigate parallel-ssh library once paramiko supports hmac-256/hmac-512 - https://phabricator.wikimedia.org/T114110#2788250 (10mmodell) 05stalled>03declined If we make a change we will almost certainly go with clustershell instead. [06:26:21] twentyafterfour: perfect, thanks, totally supports what we changed [06:48:50] Yippee, build fixed! [06:48:50] Project selenium-Wikibase » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #171: 09FIXED in 2 hr 8 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/171/ [08:15:05] https://phabricator.wikimedia.org/T150512 [08:45:55] 10Browser-Tests-Infrastructure, 07Ruby, 15User-zeljkofilipin: Mediawiki Ruby gem incorrectly assumes path to index.php - https://phabricator.wikimedia.org/T149169#2788355 (10zeljkofilipin) Thanks, I will take a look. [08:46:04] 10Browser-Tests-Infrastructure, 07Ruby, 15User-zeljkofilipin: Mediawiki Ruby gem incorrectly assumes path to index.php - https://phabricator.wikimedia.org/T149169#2788356 (10zeljkofilipin) a:03zeljkofilipin [08:46:22] 10Browser-Tests-Infrastructure, 07Ruby, 15User-zeljkofilipin: Mediawiki Ruby gem incorrectly assumes path to index.php - https://phabricator.wikimedia.org/T149169#2744054 (10zeljkofilipin) p:05Triage>03Normal [09:51:35] PROBLEM - Puppet run on deployment-apertium01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:32:05] 10Gerrit, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: Investigate why gerrit slowed down on 17/10/2016 / 18/10/2016 / 21/10/2016 - https://phabricator.wikimedia.org/T148478#2788464 (10ArielGlenn) This setting change means that we'll have more things in memory and that (logically) GC pause... [10:56:52] 10Gerrit, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: Investigate why gerrit slowed down on 17/10/2016 / 18/10/2016 / 21/10/2016 - https://phabricator.wikimedia.org/T148478#2788495 (10Paladox) @ArielGlenn so should we revert? We should try CMS? [10:57:21] 06Release-Engineering-Team, 10Wikimedia-Developer-Summit, 06Developer-Relations (Oct-Dec-2016), 07Documentation: Developer Summit 2017: Work with TPG and RelEng on solution to event documenting - https://phabricator.wikimedia.org/T132400#2788497 (10Qgil) I could not attend the meeting, but I had shared som... [10:58:45] 10Gerrit, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: Investigate why gerrit slowed down on 17/10/2016 / 18/10/2016 / 21/10/2016 - https://phabricator.wikimedia.org/T148478#2788499 (10ArielGlenn) Just leave it for now. If the logs show a sharp enough increase in pause times, I'll report... [11:24:23] 10Gerrit, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: Investigate why gerrit slowed down on 17/10/2016 / 18/10/2016 / 21/10/2016 - https://phabricator.wikimedia.org/T148478#2724169 (10MoritzMuehlenhoff) Now we have gerrit running on Debian we also have the option to use openjdk-8 instead... [11:44:11] 03Scap3 (Scap3-MediaWiki-MVP), 10scap, 13Patch-For-Review, 07Security-General: Scap should apply security patches - https://phabricator.wikimedia.org/T118478#2788577 (10mmodell) [11:44:23] 03Scap3 (Scap3-MediaWiki-MVP), 10scap, 13Patch-For-Review, 07Security-General: Scap should apply security patches - https://phabricator.wikimedia.org/T118478#1801285 (10mmodell) 325b5f52fff3 should be ready to land. [11:46:59] 03Scap3: scap version flag - https://phabricator.wikimedia.org/T147155#2788584 (10mmodell) [12:49:22] 10Gerrit, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: Investigate why gerrit slowed down on 17/10/2016 / 18/10/2016 / 21/10/2016 - https://phabricator.wikimedia.org/T148478#2788712 (10ArielGlenn) >>! In T148478#2788533, @MoritzMuehlenhoff wrote: > Now we have gerrit running on Debian we a... [12:53:56] 10Gerrit, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: Investigate why gerrit slowed down on 17/10/2016 / 18/10/2016 / 21/10/2016 - https://phabricator.wikimedia.org/T148478#2788735 (10Paladox) I could do this on the test instance I am using, but it may not work with gerrit 2.12 but may wi... [13:46:24] Yippee, build fixed! [13:46:25] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #209: 09FIXED in 2 min 24 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/209/ [13:50:52] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10MediaWiki-Unit-tests, 13Patch-For-Review, 07Technical-Debt: Clone mediawiki into mediawiki-config when running test's via jenkins - https://phabricator.wikimedia.org/T115713#2788852 (10dcausse) @hashar thanks! I tested a patch where I remove... [14:53:10] PROBLEM - Host deployment-pdf02 is DOWN: CRITICAL - Host Unreachable (10.68.16.129) [14:54:29] PROBLEM - Host deployment-conftool is DOWN: CRITICAL - Host Unreachable (10.68.20.30) [15:39:49] PROBLEM - Puppet run on deployment-phab02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:44:25] 10Gerrit, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: Investigate why gerrit slowed down on 17/10/2016 / 18/10/2016 / 21/10/2016 - https://phabricator.wikimedia.org/T148478#2788931 (10Dzahn) Since the original now asks for a login, here's the Google cache version to why this was done: ht... [15:51:04] PROBLEM - Puppet run on deployment-phab01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:52:30] PROBLEM - Puppet run on deployment-pdfrender02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:27:32] RECOVERY - Puppet run on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:18:30] PROBLEM - Puppet run on deployment-pdfrender02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:28:30] RECOVERY - Puppet run on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:37:25] PROBLEM - Host deployment-puppetmaster is DOWN: CRITICAL - Host Unreachable (10.68.16.63) [18:59:30] PROBLEM - Puppet run on deployment-pdfrender02 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [0.0] [19:04:31] RECOVERY - Puppet run on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:06:20] I found the error [20:06:21] TypeError: Cannot read property 'match' of undefined [20:18:12] Hey Everyone, I have a question about beta cluster configuration - anyone here to help ? [20:18:24] Question - PHP reads host name from config - key `Server` [20:18:32] I just want to check what's under that key for `wikipedia.beta.wmflabs.org` [20:24:57] 05Continuous-Integration-Scaling, 06Labs, 10Labs-Infrastructure: Bump quota of Nodepool instances (contintcloud tenant) - https://phabricator.wikimedia.org/T133911#2789326 (10hashar) [20:28:55] It should now be fixed hopefully with https://gerrit.wikimedia.org/r/321020 [20:29:00] which i just deployed. [21:05:35] Project beta-scap-eqiad build #128439: 15ABORTED in 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/128439/ [21:08:15] PROBLEM - jenkins_zmq_publisher on contint1001 is CRITICAL: connect to address 127.0.0.1 and port 8888: Connection refused [21:11:15] RECOVERY - jenkins_zmq_publisher on contint1001 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 8888 [21:12:26] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:24:54] Project beta-scap-eqiad build #128440: 04FAILURE in 0.33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/128440/ [21:26:29] Project beta-scap-eqiad build #128441: 04STILL FAILING in 0.32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/128441/ [21:26:40] ... [21:26:44] 00:00:00.228 OSError: [Errno 17] File exists: '/var/lock/scap' [21:27:33] !log deployment-tin deleted /var/lock/scap . Was left over after beta-scap-eqiad job got abruptly aborted [21:27:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:32:11] Yippee, build fixed! [21:32:12] Project beta-scap-eqiad build #128442: 09FIXED in 5 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/128442/ [21:34:39] !!! [22:46:46] (03PS1) 10Florianschmidtwelzow: Re-Apply "Add ContentTranslation as dependency to ArticlePlaceholder" [integration/config] - 10https://gerrit.wikimedia.org/r/321083 [22:46:54] (03CR) 10jenkins-bot: [V: 04-1] Re-Apply "Add ContentTranslation as dependency to ArticlePlaceholder" [integration/config] - 10https://gerrit.wikimedia.org/r/321083 (owner: 10Florianschmidtwelzow) [22:48:54] (03PS2) 10Florianschmidtwelzow: Re-Apply "Add ContentTranslation as dependency to ArticlePlaceholder" [integration/config] - 10https://gerrit.wikimedia.org/r/321083 [23:24:20] grrrit-wm: restart [23:24:22] re-connecting to gerrit [23:24:23] reconnected to gerrit [23:44:57] !log Cherry-picked https://gerrit.wikimedia.org/r/#/c/320441/ for testing on deployment-logstash2 [23:45:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:48:37] !log Updated _template/logstash on deployment-logstash2 to include change from https://gerrit.wikimedia.org/r/#/c/320441/ [23:48:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:48:53] * bd808 waits until 00:01Z to find out if this works