[00:46:18] Yippee, build fixed! [00:46:19] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #3018: 09FIXED in 17 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/3018/ [02:38:48] PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [03:22:21] Project beta-scap-eqiad build #79555: 04FAILURE in 3 min 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/79555/ [03:26:24] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<11.11%) [03:26:34] PROBLEM - Puppet failure on deployment-eventlogging03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [03:26:55] Yippee, build fixed! [03:26:56] Project beta-scap-eqiad build #79556: 09FIXED in 2 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/79556/ [03:28:47] !log Freed 800M on deployment-bastion by running /home/bd808/cleanup-var-crap.sh [03:28:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [03:41:27] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [03:51:39] krenair@deployment-poolcounter01:~$ df -h [03:51:39] df: `/sys/kernel/debug': Function not implemented [03:51:45] Filesystem Size Used Avail Use% Mounted on [03:51:46] what [03:53:14] Krenair: sounds like -- https://bugs.launchpad.net/ubuntu/+source/coreutils/+bug/1465180 [04:01:31] RECOVERY - Puppet failure on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0] [04:21:39] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [05:24:26] Project beta-scap-eqiad build #79568: 04FAILURE in 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/79568/ [05:30:58] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #611: 04FAILURE in 28 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/611/ [05:35:42] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:36:17] Yippee, build fixed! [05:36:18] Project beta-scap-eqiad build #79569: 09FIXED in 1 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/79569/ [05:36:18] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 28.57% of data above the critical threshold [0.0] [05:36:19] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [05:36:59] PROBLEM - Puppet failure on integration-puppetmaster is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [05:37:11] PROBLEM - Puppet failure on deployment-kafka02 is CRITICAL: CRITICAL: 71.43% of data above the critical threshold [0.0] [05:37:11] PROBLEM - Puppet failure on deployment-sca02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [05:37:11] PROBLEM - Puppet failure on deployment-tin is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [05:37:11] PROBLEM - Puppet failure on integration-slave-trusty-1013 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:37:53] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL: CRITICAL: 71.43% of data above the critical threshold [0.0] [05:37:53] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [05:38:03] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [05:39:36] PROBLEM - Puppet failure on integration-slave-trusty-1023 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:40:43] PROBLEM - Puppet failure on integration-publisher is CRITICAL: CRITICAL: 90.00% of data above the critical threshold [0.0] [05:40:55] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 88.89% of data above the critical threshold [0.0] [05:40:57] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [0.0] [05:41:09] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:42:15] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce build #262: 04FAILURE in 26 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce/262/ [06:02:11] RECOVERY - Puppet failure on integration-slave-trusty-1013 is OK: OK: Less than 1.00% above the threshold [0.0] [06:05:46] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [06:06:11] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [06:06:25] RECOVERY - Puppet failure on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:06:57] RECOVERY - Puppet failure on integration-puppetmaster is OK: OK: Less than 1.00% above the threshold [0.0] [06:07:05] RECOVERY - Puppet failure on deployment-kafka02 is OK: OK: Less than 1.00% above the threshold [0.0] [06:07:05] RECOVERY - Puppet failure on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0] [06:07:05] RECOVERY - Puppet failure on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [06:07:59] RECOVERY - Puppet failure on deployment-pdf01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:07:59] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [06:09:35] RECOVERY - Puppet failure on integration-slave-trusty-1023 is OK: OK: Less than 1.00% above the threshold [0.0] [06:10:45] RECOVERY - Puppet failure on integration-publisher is OK: OK: Less than 1.00% above the threshold [0.0] [06:10:53] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [06:10:55] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [06:11:17] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [06:13:02] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [06:40:51] 10Deployment-Systems, 3Scap3: scap3 dsh_target should check the scap directory of a repo as well as `/etc/dsh/groups` - https://phabricator.wikimedia.org/T119200#1824365 (10mmodell) [06:52:56] 10Deployment-Systems, 6operations: l10nupdate user uid mismatch between tin and mira - https://phabricator.wikimedia.org/T119165#1824367 (10mmodell) @bd808: so, reading the manpage for rsync, it seems that you have to specify --numeric-ids for this to even matter? Otherwise rsync should be smart enough to rem... [07:52:21] 10Deployment-Systems, 3Scap3, 7Documentation, 5Patch-For-Review: Add documentation of the new scap3 features to the scap docs - https://phabricator.wikimedia.org/T112554#1824417 (10mmodell) 5Open>3Resolved https://doc.wikimedia.org/mw-tools-scap/scap3/index.html [08:46:02] 10MediaWiki-Codesniffer, 3Outreachy-Round-11: Outreachy proposal for : Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T115585#1824501 (1001tonythomas) 5Open>3declined Thank you for your proposal. Sadly, the Outreachy administration team made it strict that candidates with... [08:46:04] 10MediaWiki-Codesniffer, 10Possible-Tech-Projects, 3Outreachy-Round-11: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1824505 (1001tonythomas) [09:07:22] (03PS1) 10Hashar: Add rake-jessie experimental to MediaWiki core [integration/config] - 10https://gerrit.wikimedia.org/r/254816 [09:07:32] (03CR) 10Hashar: [C: 032] Add rake-jessie experimental to MediaWiki core [integration/config] - 10https://gerrit.wikimedia.org/r/254816 (owner: 10Hashar) [09:08:49] (03Merged) 10jenkins-bot: Add rake-jessie experimental to MediaWiki core [integration/config] - 10https://gerrit.wikimedia.org/r/254816 (owner: 10Hashar) [09:22:22] 7Browser-Tests, 10Continuous-Integration-Config, 10MediaWiki-extensions-ContentTranslation, 5Patch-For-Review, and 3 others: Add Rakefile to repositories with Ruby code - https://phabricator.wikimedia.org/T117993#1824528 (10Nemo_bis) [09:24:13] (03CR) 10Hashar: "To skip the rake-jessie branch on a specific repo/branch we should now be able to do:" [integration/config] - 10https://gerrit.wikimedia.org/r/253343 (https://phabricator.wikimedia.org/T114860) (owner: 10Zfilipin) [09:38:32] Yippee, build fixed! [09:38:32] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #682: 09FIXED in 1 min 30 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/682/ [10:17:43] !log added Jcrespo (Jaime) to the beta cluster project as an admin + sudo rights [10:17:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:41:41] 10Beta-Cluster-Infrastructure, 7Database, 7WorkType-NewFunctionality: Send deployment-db1 deployment-db2 syslog to beta cluster logstash - https://phabricator.wikimedia.org/T119370#1824659 (10hashar) 3NEW [10:46:08] 10Beta-Cluster-Infrastructure, 7Database, 7WorkType-NewFunctionality: Send deployment-db1 deployment-db2 syslog to beta cluster logstash - https://phabricator.wikimedia.org/T119370#1824686 (10hashar) The configuration is around: ``` deployment-db1:~$ cat /etc/rsyslog.d/30-remote-syslog.conf *.info;mail.none... [10:49:11] 10Beta-Cluster-Infrastructure, 7Database, 7WorkType-NewFunctionality: Send deployment-db1 deployment-db2 syslog to beta cluster logstash - https://phabricator.wikimedia.org/T119370#1824692 (10hashar) p:5Normal>3Low wfLogDBError is already sent to logstash beta, which comes with stracktraces. So syslog is... [10:56:50] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Database, 7Database, 7WorkType-NewFunctionality: Enable MariaDB/MySQL strict mode on CI slaves - https://phabricator.wikimedia.org/T119371#1824701 (10hashar) 3NEW [11:24:10] 10Beta-Cluster-Infrastructure, 7Database: Investigate slow query logging/digest for Beta Cluster - https://phabricator.wikimedia.org/T116793#1824748 (10hashar) Clarified with @jcrespo. We can just enable `performance_schema` just like for production (T99485). The informations will then be available in the be... [11:27:15] (03CR) 10Zfilipin: "https://gerrit.wikimedia.org/r/#/c/252686/ is merged and rake-jessie job runs fine for operations/puppet:" [integration/config] - 10https://gerrit.wikimedia.org/r/252689 (https://phabricator.wikimedia.org/T110019) (owner: 10Zfilipin) [13:58:56] (03PS1) 10Hashar: dib: make git mirrors belong to jenkins:jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/254851 [13:59:15] zeljkof: I am bumping the nodepool images to have /srv/git to belong to jenkins user :-D [13:59:21] zeljkof: something I noticed this morning [13:59:32] hashar: +1 :) [14:00:01] (03CR) 10jenkins-bot: [V: 04-1] dib: make git mirrors belong to jenkins:jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/254851 (owner: 10Hashar) [14:01:36] I should inject mediawiki/core as well [14:02:08] 10MediaWiki-Releasing, 6Developer-Relations, 10Wikimedia-Blog-Content, 3DevRel-November-2015, 5MW-1.26-release: Write blog post announcing MW 1.26 - https://phabricator.wikimedia.org/T112842#1824970 (10Qgil) Thank you @greg! Quick question: are you planning to release on 2015-11-25 -- next Wednesday? In... [14:02:23] (03PS2) 10Hashar: dib: make git mirrors belong to jenkins:jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/254851 [14:07:07] (03PS1) 10Hashar: dib: mirror mediawiki/core inside images [integration/config] - 10https://gerrit.wikimedia.org/r/254852 [14:07:09] (03PS1) 10Hashar: dib: tweak 01-mirror-gerrit-repos feedback [integration/config] - 10https://gerrit.wikimedia.org/r/254853 [14:07:15] Creating host cache /srv/dib/cache/git-repos/mediawiki/core.git [14:07:15] mkdir: created directory '/srv/dib/cache/git-repos/mediawiki' [14:07:16] Cloning into bare repository '/srv/dib/cache/git-repos/mediawiki/core.git'... [14:08:54] hacking while listening to french music https://www.youtube.com/watch?v=NwVA5zYfNWw [14:17:25] in live, they are literally surfing on top of the concert crowd https://youtu.be/B81iYZUov7Q?t=2m18s :D [14:21:26] (03CR) 10Hashar: [C: 032] dib: mirror mediawiki/core inside images [integration/config] - 10https://gerrit.wikimedia.org/r/254852 (owner: 10Hashar) [14:21:37] (03CR) 10Hashar: [C: 032] dib: tweak 01-mirror-gerrit-repos feedback [integration/config] - 10https://gerrit.wikimedia.org/r/254853 (owner: 10Hashar) [14:21:45] (03CR) 10Hashar: [C: 032] dib: make git mirrors belong to jenkins:jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/254851 (owner: 10Hashar) [14:23:10] !log pushing new disk image to labs for Nodepool [14:23:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:23:20] (03Merged) 10jenkins-bot: dib: make git mirrors belong to jenkins:jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/254851 (owner: 10Hashar) [14:23:22] (03Merged) 10jenkins-bot: dib: mirror mediawiki/core inside images [integration/config] - 10https://gerrit.wikimedia.org/r/254852 (owner: 10Hashar) [14:23:31] 6Release-Engineering-Team, 5Testing-Initiative-2015, 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-Examples, 7Documentation: Improve documentation around running/writing (with lots of examples) browser tests - https://phabricator.wikimedia.org/T108108#1825017 (10zeljkofilipin) a:5dduvall>3zel... [14:23:48] (03Merged) 10jenkins-bot: dib: tweak 01-mirror-gerrit-repos feedback [integration/config] - 10https://gerrit.wikimedia.org/r/254853 (owner: 10Hashar) [14:24:38] !log refreshing nodepool snapshot [14:24:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:27:49] !log Image ci-jessie-wikimedia-1448288646 in wmflabs-eqiad is ready [14:27:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:31:46] !log deleted obsolete nodepool instances so nodepool replenish the pool with new image [14:31:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:37:13] (03PS3) 10Zfilipin: Run Ruby jobs using Rake [integration/config] - 10https://gerrit.wikimedia.org/r/253343 (https://phabricator.wikimedia.org/T114860) [14:38:05] (03CR) 10jenkins-bot: [V: 04-1] Run Ruby jobs using Rake [integration/config] - 10https://gerrit.wikimedia.org/r/253343 (https://phabricator.wikimedia.org/T114860) (owner: 10Zfilipin) [14:39:27] bah [14:42:44] !log regenerating the nodepool images and snapshot. 51-git-mirror-ownership did not run because of missing executable bit [14:42:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:02:16] !log updating rake-jessie job to use cached repos under /srv/git (for nodepool) [15:02:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:02:25] zeljkof: ^^^ [15:02:26] some mess [15:02:52] I am update rake-jessie to clone the repo using /srv/git/$ZUUL_PROJECT.git as a reference [15:02:55] might speed it up [15:04:20] doesn't seem to show up https://integration.wikimedia.org/ci/job/rake-jessie/1252/consoleFull [15:04:21] :( [15:04:24] stupid git [15:09:44] time to migrate to zuul-cloner! [15:14:04] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:14:04] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:14:09] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:17:17] (03PS1) 10Hashar: dib: add zuul to nodepool instances [integration/config] - 10https://gerrit.wikimedia.org/r/254871 (https://phabricator.wikimedia.org/T117223) [15:17:31] (03CR) 10Hashar: [C: 032] dib: add zuul to nodepool instances [integration/config] - 10https://gerrit.wikimedia.org/r/254871 (https://phabricator.wikimedia.org/T117223) (owner: 10Hashar) [15:18:33] (03Merged) 10jenkins-bot: dib: add zuul to nodepool instances [integration/config] - 10https://gerrit.wikimedia.org/r/254871 (https://phabricator.wikimedia.org/T117223) (owner: 10Hashar) [15:18:35] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 30650 bytes in 0.642 second response time [15:18:35] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 39322 bytes in 0.612 second response time [15:18:59] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 38981 bytes in 0.493 second response time [15:20:34] (03CR) 10Hashar: "Refreshing snapshot with:" [integration/config] - 10https://gerrit.wikimedia.org/r/254871 (https://phabricator.wikimedia.org/T117223) (owner: 10Hashar) [15:25:21] Image ci-jessie-wikimedia-1448292050 in wmflabs-eqiad is ready [15:25:23] !log Image ci-jessie-wikimedia-1448292050 in wmflabs-eqiad is ready [15:25:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:49:35] hashar: sorry, family emergency, have to go [16:35:01] 10Deployment-Systems, 6operations: l10nupdate user uid mismatch between tin and mira - https://phabricator.wikimedia.org/T119165#1825437 (10bd808) I can assert that we are currently seeing uid preservation when rsycning from tin to mira. Checking the ownership of `/srv/mediawiki-staging/php-1.27.0-wmf.7/cache/... [17:13:22] !log deleting old Nodepool snapshots. Current one is ci-jessie-wikimedia-1448296278 [17:13:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:28:58] PROBLEM - Puppet failure on deployment-eventlogging03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:46:09] 10Deployment-Systems, 6operations: l10nupdate user uid mismatch between tin and mira - https://phabricator.wikimedia.org/T119165#1825834 (10mmodell) @bd808: So it sounds like having consistent UIDs is the better / easier solution, based on the complexity of getting name mapping to work? [17:47:01] 10Deployment-Systems, 3Scap3: Need a way to see config diffs in Scap - https://phabricator.wikimedia.org/T118206#1825842 (10mmodell) [17:51:10] 10Deployment-Systems, 10MediaWiki-extensions-LocalisationUpdate, 7I18n, 7Wikimedia-log-errors: l10n-update not updating Vector and extensions - https://phabricator.wikimedia.org/T103879#1825877 (10Elitre) >>! In T103879#1663136, @Nemo_bis wrote: > @supernino reports about 5ccf6668d4e0c17c not being sync'ed... [17:58:20] I'm thinking for the ERROR/WARNING graph we should just use linear scaling instead of log. [17:58:35] Since we actually care about the absolute numbers for these 2 metrics and not just the trend. [17:58:59] That has it looking like https://grafana.wikimedia.org/dashboard/db/production-logging?panelId=14&fullscreen [17:59:26] Nice thing about linear is if one metric goes kaboom, it's really really freaking obvious. [18:02:28] 10Continuous-Integration-Config, 5MW-1.26-release, 5Patch-For-Review: MediaWiki 1.26 bundled repo should be state of the art - https://phabricator.wikimedia.org/T115392#1723180 (10hashar) Refreshed table. ConfirmEdit got composer/npm added. [18:02:33] RECOVERY - Puppet failure on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0] [18:06:33] https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/21216/console uhhh what? [18:37:23] 10Continuous-Integration-Config, 5MW-1.26-release, 5Patch-For-Review: MediaWiki 1.26 bundled repo should be state of the art - https://phabricator.wikimedia.org/T115392#1826091 (10demon) I don't think we'll get composer.json created & merged w/ tests and backported to REL1_26 by Wednesday. Suggest just tryin... [19:05:26] (03PS1) 10Niedzielski: Move long running Android tests to periodic job [integration/config] - 10https://gerrit.wikimedia.org/r/254905 (https://phabricator.wikimedia.org/T118098) [19:09:21] (03CR) 10Niedzielski: "@hashar, hello! Two questions:" [integration/config] - 10https://gerrit.wikimedia.org/r/254905 (https://phabricator.wikimedia.org/T118098) (owner: 10Niedzielski) [19:30:13] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:05] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 39005 bytes in 0.614 second response time [19:47:19] twentyafterfour: git-lfs only speaks http(s). I think that makes it a non-contender. [19:48:41] ostriches: what are you guys looking for? (just curious) [19:49:06] Replacement for git-fat support for large binaries (mostly jars) [19:49:27] I'll probably go with git-annex since it supports rsync already. [19:49:39] And it's already got a debian package :) [19:51:28] cool [19:53:57] ostriches: https isn't necessarily a bad thing... but I'm also not against git-annex [20:01:01] twentyafterfour: The problem with git-lfs too is it requires a server implementation. The only canonical one is Githubs, which is closed source. There's a couple others...none in Python [20:01:46] (The repos on tin from which you fetch would have to be special) [20:01:55] yeah it doesn't really seem too mature yet [20:11:58] (03PS1) 10Hashar: [Cite] add composer tests [integration/config] - 10https://gerrit.wikimedia.org/r/254910 [20:24:32] ostriches: I started a bunch of composer test entry point for REL1_26 bundled stuff [20:24:34] else Jenkins will faill [20:24:49] we already trigger composer test on most of those repositories master branches apparently [20:35:04] hashar: Sounds good :) [20:38:02] ostriches: should I just bump some REL1_26 branches ? [20:38:07] not sure whether I should do a merge commit [20:38:10] or force push [20:38:16] an example is Interwiki [20:38:45] it received l10n spam and basic tests/formatting changes [20:39:27] lame merge example: https://gerrit.wikimedia.org/r/254925 [20:45:31] ostriches: any clue ? :D [20:45:50] Why not pull --rebase? [20:46:16] Oh dur. [20:46:17] Meh [20:46:28] I am merging master into REL1_26 [20:46:39] but I can well just push an update of REL1_26 [20:46:52] I hate merge commits in gerrit. [20:46:58] who care about Gerrit :D [20:47:16] the question is more: should I silently update the REL1_26 [20:47:28] or make it clear it has been manually updated since we have cut the branch by crafting a merge commit [20:47:36] Yeah do the merge commit. [20:47:40] Force pushing is even worse. [20:47:48] I believe [20:48:02] wanna review them or should I just +2? [20:56:12] go ahead, making lunch [21:06:03] (03PS2) 10Hashar: composer tests for a few extensions [integration/config] - 10https://gerrit.wikimedia.org/r/254910 [21:06:25] adding composer [21:07:41] (03CR) 10Hashar: [C: 032] composer tests for a few extensions [integration/config] - 10https://gerrit.wikimedia.org/r/254910 (owner: 10Hashar) [21:08:44] (03Merged) 10jenkins-bot: composer tests for a few extensions [integration/config] - 10https://gerrit.wikimedia.org/r/254910 (owner: 10Hashar) [21:14:44] (03PS1) 10Arlolra: Increase parsoid npm test timeout [integration/config] - 10https://gerrit.wikimedia.org/r/254946 [21:15:21] 10Deployment-Systems, 3Scap3: Scap3 needs a way to handle large binary file transport - https://phabricator.wikimedia.org/T119443#1826547 (10thcipriani) 3NEW [21:21:05] 10Deployment-Systems, 3Scap3: Scap3 needs a way to handle large binary file transport - https://phabricator.wikimedia.org/T119443#1826575 (10thcipriani) From our meeting, @demon specifically called out `.jar` files being important in this workflow. Contenders are: * git-fat * git-lfs * git-annex There have b... [21:23:16] 10Deployment-Systems, 3Scap3: Scap3 needs a way to handle large binary file transport - https://phabricator.wikimedia.org/T119443#1826579 (10demon) >>! In T119443#1826575, @thcipriani wrote: > @demon mentioned in IRC that `git-lfs` is https-only and requires a special server implementation—which may not jive w... [21:24:29] thcipriani: The only two implementations that seem general enough for us are in Java and Node. [21:24:35] The rest seem rather use-case-specific. [21:25:12] Requiring either as a dependency for scap rubs me real icky like [21:25:26] heh, agreed. [21:26:07] git-annex on the other hand is really just client-side. You can fetch the objects from basically anywhere [21:26:43] (on disk, rsync, etc etc etc) [21:26:43] In that regards, it's much more like git-fat. [21:26:43] I use git-annex for all my raw photo files and it works really well for my personal use-case I've had no real problems with it + the s3-gpg-backing. Works gerat. [21:27:10] Plus it has debian packages. [21:28:38] So just have to add it to our package deps and call it a day [21:28:43] 10Continuous-Integration-Config, 5MW-1.26-release, 5Patch-For-Review: MediaWiki 1.26 bundled repo should be state of the art - https://phabricator.wikimedia.org/T115392#1826587 (10hashar) [21:28:43] The hardest part I think is figuring out the workflow. Do you do the initial local/Gerrit/Phab work with git-annex too? Or is it something you handle on the deploy master on the fly? [21:29:04] I think the latter. [21:30:05] Also: do we make it "smart" and have it automatically annex binary files over N size? [21:30:14] Or do we require explicit registration of binaries to track [21:30:48] well if it's going to be explicit then we'd have to make it part of the phab/gerrit/local workflow mostly, I'd think. [21:31:02] Hmm true [21:31:06] developers would have to select which files to add to annex then. [21:31:31] also, what is the default transport mechanism for annex? [21:31:40] I mean for our use. [21:31:46] I'd say rsync for us. [21:31:57] We already have it everywhere, it's a known beast to deal with atm. [21:32:30] sure, have to expand the rsync server modules to cover /srv/deployment in that case. [21:34:30] hmm, this is a big ball of work :P [21:34:53] Heh, now that I poke git-annex more, I wonder if `git annex sync` to keep co-masters sane would be better than rsync [21:37:32] sigh. To paraphrase master p: mo' masters mo' problems. [21:37:47] (03PS1) 10Hashar: Add composer test to three REL1_26 skins [integration/config] - 10https://gerrit.wikimedia.org/r/255016 [21:39:39] (03CR) 10Cscott: [C: 031] Increase parsoid npm test timeout [integration/config] - 10https://gerrit.wikimedia.org/r/254946 (owner: 10Arlolra) [21:39:41] (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/255016 (owner: 10Hashar) [21:40:07] ostriches: have time to take on T119443? You probably seem like you've got a good start with it :P [21:40:20] Not this week [21:41:12] kk. It's in the to triage column for now, we'll just leave it there until some one has time to circle back. [21:42:57] (03CR) 10Hashar: [C: 032] Add composer test to three REL1_26 skins [integration/config] - 10https://gerrit.wikimedia.org/r/255016 (owner: 10Hashar) [21:45:20] (03Merged) 10jenkins-bot: Add composer test to three REL1_26 skins [integration/config] - 10https://gerrit.wikimedia.org/r/255016 (owner: 10Hashar) [21:53:29] 10Continuous-Integration-Config, 5MW-1.26-release, 5Patch-For-Review: MediaWiki 1.26 bundled repo should be state of the art - https://phabricator.wikimedia.org/T115392#1826668 (10hashar) [21:54:15] 10Continuous-Integration-Config, 5MW-1.26-release, 5Patch-For-Review: MediaWiki 1.26 bundled repo should be state of the art - https://phabricator.wikimedia.org/T115392#1723180 (10hashar) I have added a bunch of composer entry points. Some are falling such as LocalizationUpdate https://gerrit.wikimedia.org/r... [22:02:39] 10Deployment-Systems, 3Scap3: Need a way to restart services without deploying via scap - https://phabricator.wikimedia.org/T119449#1826691 (10thcipriani) 3NEW [22:05:23] marxarelli: oh, drive by link drop again: https://office.wikimedia.org/wiki/Systems_guide_for_new_hires [22:05:38] marxarelli: https://office.wikimedia.org/wiki/New_tech_employee_orientation [22:06:33] greg-g: danke! [22:07:10] np :) [22:07:42] i don't think i've ever seen this page [22:07:45] :) [22:09:14] that happened for me too don't feel bad :) [22:10:56] marxarelli: wait, really? I could've swore I sent that to you and mukunda and tyler.... maybe I didn't for you since officially robla was the hiring manager [22:10:59] * greg-g shrugs [22:11:43] greg-g: yeah, you weren't my manager yet [22:11:47] ? [22:11:52] can someone review https://gerrit.wikimedia.org/r/#/c/254946/ and enable it .. we are getting a bit too many false failures because of the tight timeout after we merged some code that slows down our test runs. [22:12:05] it's entirely possible that robla sent it to me, but also likely that i dropped it due to overload [22:12:15] * robla tries to figure out what he's being thrown under the bus for :-P [22:12:29] it's robla's fault [22:12:31] get him! [22:12:35] lol [22:12:43] greg-g: you might have sent that out, but it looks like something I might have skipped in the information overload of my first few days. [22:14:37] I definitely got the new tech employee thing. [22:15:33] * greg-g forgot robla was in here [22:15:35] * greg-g denies everything [22:20:15] first few days are a blur :) [22:21:55] 10Deployment-Systems, 3Scap3: Need a way to restart services without deploying via scap - https://phabricator.wikimedia.org/T119449#1826747 (10thcipriani) Unlike a normal `deploy` there is no recourse for deployers that do a restart via Scap3 that fails subsequent checks. If a service restart fails or a servic... [22:25:28] hashar: btw, subbu's request up there, if you have time ^^ [22:25:49] hashar said earlier that he as "sleeping" :) [22:26:14] "I am busy/ sleeping " [22:26:37] nvm :) [22:26:58] but, don't let it stop anyone else who can do it. ;) [22:29:28] subbu: well actually you should be able to refresh the jobs using JJB [22:29:36] I would, but too late to monitor them :( [22:29:44] hashar, we do refresh them via recheck. [22:29:46] ultimately, we will want to speed the test! [22:30:08] but, that is not a good solution to keep rechecking whenever the tests take a bit too long to run. [22:30:16] hence the increased timeout. [22:30:23] (03PS2) 10Hashar: Increase parsoid npm test timeout [integration/config] - 10https://gerrit.wikimedia.org/r/254946 (owner: 10Arlolra) [22:30:44] subbu: do the tests can take advantage of parallelization? [22:30:48] just wondering [22:31:00] no, they run one after another. [22:31:25] and our parser tests just slowed down by 1 minute .. so, that adds 2 minutes to our previous run time .. which cuts it quite closed with the 10 min. limit we have. [22:31:49] !log Updating parsoidsvc-deploy-npm-* jobs https://gerrit.wikimedia.org/r/254946 [22:31:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:31:59] legoktm, ^ [22:32:02] thanks. [22:32:10] !log Updating parsoidsvc-source-npm-* jobs https://gerrit.wikimedia.org/r/254946 [22:32:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:32:25] (03CR) 10Hashar: [C: 032] "Updated all four jobs with JJB :-)" [integration/config] - 10https://gerrit.wikimedia.org/r/254946 (owner: 10Arlolra) [22:32:43] and I'm still slower than sleeping hashar [22:32:55] ha ha .. [22:32:58] subbu: how and the jenkins job update doc is at https://www.mediawiki.org/wiki/CI/JJB :-D [22:33:16] legoktm: that is because you are doing some other things while I am only starring at this channel ! :D [22:33:23] thanks. will add a link to that on our wiki page. [22:33:33] subbu: and we will want to run them in parallel one day [22:33:34] (03Merged) 10jenkins-bot: Increase parsoid npm test timeout [integration/config] - 10https://gerrit.wikimedia.org/r/254946 (owner: 10Arlolra) [22:33:52] subbu: assuming they don't have side effect, we could split them in four groups and consume four cpus :D [22:34:16] subbu: the ones from mediawiki/core are slow as hell, takes up to 6 minutes under Zend :-((( [22:35:26] hashar, yes, in parallel would be good. [22:37:04] 10Continuous-Integration-Config, 5MW-1.26-release, 5Patch-For-Review: MediaWiki 1.26 bundled repo should be state of the art - https://phabricator.wikimedia.org/T115392#1826794 (10hashar) ConfirmEdit registration was added via T88047 https://gerrit.wikimedia.org/r/#/c/250277/ is the backport for REL1_26. W... [22:37:48] subbu: and maybe some profiling can show low hanging fruit to speed them up :D [22:37:50] but anyway [22:37:56] I am supposed to sleep by now [22:38:24] hashar: go sleep! [22:38:27] g'night :) [22:39:20] thx *wave* [22:44:06] 10Deployment-Systems, 6Performance-Team, 6operations, 7HHVM: Make scap able to depool/repool servers via the conftool API - https://phabricator.wikimedia.org/T104352#1826816 (10thcipriani) Started talking about this at the deployment [[https://www.mediawiki.org/wiki/Deployment_tooling/Cabal/2015-11-23#etcd... [22:44:19] 10Deployment-Systems, 3Scap3, 6Performance-Team, 6operations, 7HHVM: Make scap able to depool/repool servers via the conftool API - https://phabricator.wikimedia.org/T104352#1826818 (10thcipriani) [22:46:43] anyone mind if i take the 3-4 (starts in 15min) deploy window to try again at enabling the ES labs replica writes? [22:47:25] last time around the write load was too heavy and the disks couldn't keep up, this time enabling just enwiki and dewiki (whereas last time around enabled everything but enwiki and dewiki) [22:48:32] I'm guessing it's open between now-swat? [22:52:55] yeah, I said yes in -operations, but then spam [22:59:18] ostriches: still wanna land https://phabricator.wikimedia.org/D51 ? [23:02:42] accepted. [23:08:19] 10Deployment-Systems, 3Scap3: Need a way to restart services without deploying via scap - https://phabricator.wikimedia.org/T119449#1826945 (10dduvall) In thinking about this problem a bit more, I think it may be problematic to entirely split promote into promote/restart, mainly for the pooling reasons you men... [23:43:46] 10Beta-Cluster-Infrastructure, 10netops, 6operations, 7Database: Evaluate security concerns of logging beta cluster db queries on tendril - https://phabricator.wikimedia.org/T119461#1827042 (10jcrespo) 3NEW [23:54:53] PROBLEM - Puppet failure on pmcache is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]