[02:24:59] Yippee, build fixed! [02:24:59] Project browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-linux-android-sauce build #62: FIXED in 2 min 58 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-linux-android-sauce/62/ [02:32:26] RECOVERY - Puppet failure on integration-publisher is OK: OK: Less than 1.00% above the threshold [0.0] [02:36:43] Yippee, build fixed! [02:36:44] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #535: FIXED in 3 min 42 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/535/ [04:31:47] Yippee, build fixed! [04:31:48] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #402: FIXED in 39 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/402/ [04:41:39] Yippee, build fixed! [04:41:40] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #407: FIXED in 34 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/407/ [05:42:45] Yippee, build fixed! [05:42:45] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce build #33: FIXED in 26 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce/33/ [06:20:26] (03PS1) 10TTO: Fix MediaWiki core PHP documentation link [integration/docroot] - 10https://gerrit.wikimedia.org/r/202683 [06:31:24] Yippee, build fixed! [06:31:24] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #576: FIXED in 12 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/576/ [06:43:16] 6Release-Engineering: Do not say "< wmf-insecte> Yippee, build fixed!" - https://phabricator.wikimedia.org/T95395#1188546 (10awight) 3NEW [06:43:42] wmf-insecte: help [06:43:42] awight you may not issue bot commands in this chat! [06:46:41] 6Release-Engineering: Do not say "< wmf-insecte> Yippee, build fixed!" - https://phabricator.wikimedia.org/T95395#1188554 (10awight) Also: respond helpfully to a CTCP source request. ``` /ctcp wmf-insecte source 23:45 [freenode] [ctcp(wmf-insecte)] SOURCE 23:45 [freenode] -!- wmf-insecte is away: Working: ``` [07:19:19] 10Beta-Cluster, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 5ContentTranslation-Release5, 3LE-Sprint-85: Setup new wikis in Beta Cluster for Content Translation - https://phabricator.wikimedia.org/T90683#1188766 (10Arrbee) See also T93213 [08:17:39] 10Browser-Tests, 10Continuous-Integration, 6Release-Engineering, 5Patch-For-Review: It takes about 20 seconds just to start a Sauce Labs browser - https://phabricator.wikimedia.org/T92613#1188819 (10hashar) a:5hashar>3None [08:17:57] 10Browser-Tests, 10Continuous-Integration, 6Release-Engineering, 5Patch-For-Review: It takes about 20 seconds just to start a Sauce Labs browser - https://phabricator.wikimedia.org/T92613#1116195 (10hashar) I am not actively working on this since I lack even the most basic ruby skills :( [08:30:00] hashar: https://gerrit.wikimedia.org/r/#/c/202689/ - here. [08:30:25] hashar: Please comment, I'll go via usual 'Add beta wiki' page on wikkitech [08:45:48] 10Browser-Tests: IE Browser tests job have no test being run due to a mistake in cucumber tag - https://phabricator.wikimedia.org/T95398#1188856 (10hashar) 3NEW a:3zeljkofilipin [08:49:08] 10Browser-Tests: IE Browser tests job have no test being run due to a mistake in cucumber tag - https://phabricator.wikimedia.org/T95398#1188866 (10hashar) From jjb/macro-browsertests.yaml if [ $BROWSER == "internet_explorer" ] then BROWSER_TAG=${{BROWSER}}_$VERSION els... [09:23:41] (03PS4) 10Hashar: Forward port precise dh-virtualenv to trusty [integration/zuul] (debian/trusty-wikimedia) - 10https://gerrit.wikimedia.org/r/197329 (https://phabricator.wikimedia.org/T48552) [09:23:43] (03PS2) 10Hashar: Package python deps with dh-virtualenv [integration/zuul] (debian/trusty-wikimedia) - 10https://gerrit.wikimedia.org/r/197328 (https://phabricator.wikimedia.org/T48552) [09:31:23] (03PS5) 10Hashar: Forward port precise dh-virtualenv to trusty [integration/zuul] (debian/trusty-wikimedia) - 10https://gerrit.wikimedia.org/r/197329 (https://phabricator.wikimedia.org/T48552) [09:32:30] (03CR) 10Hashar: [C: 031] "Rebased and further tweaked the dependencies." [integration/zuul] (debian/trusty-wikimedia) - 10https://gerrit.wikimedia.org/r/197329 (https://phabricator.wikimedia.org/T48552) (owner: 10Hashar) [09:45:29] Yippee, build fixed! [09:45:29] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #448: FIXED in 8 min 28 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/448/ [09:51:24] 6Release-Engineering: Read "Vagrant: Up and Running" book - https://phabricator.wikimedia.org/T95401#1188929 (10zeljkofilipin) 3NEW a:3zeljkofilipin [09:53:53] 6Release-Engineering: Read "Vagrant: Up and Running" book - https://phabricator.wikimedia.org/T95401#1188937 (10zeljkofilipin) At 30%. [09:54:59] 6Release-Engineering: Read "Vagrant: Up and Running" book - https://phabricator.wikimedia.org/T95401#1188929 (10zeljkofilipin) The files created while reading the book: https://github.com/zeljkofilipin/vagrant_book_example [09:57:22] 10Browser-Tests: IE Browser tests job have no test being run due to a mistake in cucumber tag - https://phabricator.wikimedia.org/T95398#1188941 (10zeljkofilipin) p:5Triage>3Normal [09:58:28] 10Continuous-Integration, 6Release-Engineering, 5Patch-For-Review: Repositories with Ruby code should be documented and appropriate Jenkins jobs should be running - https://phabricator.wikimedia.org/T1361#1188943 (10zeljkofilipin) a:5zeljkofilipin>3None [09:59:15] 10Continuous-Integration, 6Release-Engineering, 5Patch-For-Review: Repositories with Ruby code should be documented and appropriate Jenkins jobs should be running - https://phabricator.wikimedia.org/T1361#23935 (10zeljkofilipin) Not working on this. I plan to continue with this later, but feel free to take t... [10:00:16] 10Continuous-Integration, 6Release-Engineering, 7Documentation: Document RuboCop workflow - https://phabricator.wikimedia.org/T1368#1188946 (10zeljkofilipin) Not working on this. I plan to continue with this later, but feel free to take the task. [10:00:22] 10Continuous-Integration, 6Release-Engineering, 7Documentation: Document RuboCop workflow - https://phabricator.wikimedia.org/T1368#1188947 (10zeljkofilipin) a:5zeljkofilipin>3None [10:01:59] 10Browser-Tests, 6Release-Engineering, 7Tracking: Fix easy problems reported by RuboCop (tracking) - https://phabricator.wikimedia.org/T91485#1188949 (10zeljkofilipin) a:5zeljkofilipin>3None [10:02:05] 10Browser-Tests, 6Release-Engineering, 7Tracking: Fix easy problems reported by RuboCop (tracking) - https://phabricator.wikimedia.org/T91485#1087225 (10zeljkofilipin) Not working on this. I plan to continue with this later, but feel free to take the task. [10:02:35] 10Browser-Tests: Create Sauce Labs account for Andrew Russell Green - https://phabricator.wikimedia.org/T94192#1188951 (10zeljkofilipin) a:5zeljkofilipin>3None [10:03:01] 10Browser-Tests: Transfer the main Sauce Labs account to zeljkofilipin - https://phabricator.wikimedia.org/T94191#1188954 (10zeljkofilipin) [10:03:02] 10Browser-Tests: Create Sauce Labs account for Andrew Russell Green - https://phabricator.wikimedia.org/T94192#1157154 (10zeljkofilipin) [10:03:21] 10Browser-Tests: Create Sauce Labs account for Andrew Russell Green - https://phabricator.wikimedia.org/T94192#1157154 (10zeljkofilipin) Nothing to do until the blocking task is resolved. [10:04:21] 10Browser-Tests, 10Continuous-Integration, 7Tracking: Fix or delete browsertests* Jenkins jobs that are failing for more than a week (tracking) - https://phabricator.wikimedia.org/T94150#1188956 (10zeljkofilipin) 5Open>3stalled a:5zeljkofilipin>3None [10:05:11] 10Browser-Tests, 10Continuous-Integration, 7Tracking: Fix or delete browsertests* Jenkins jobs that are failing for more than a week (tracking) - https://phabricator.wikimedia.org/T94150#1156290 (10zeljkofilipin) Contacted relevant teams, see tasks in "blocked by". Nothing to do here until the blocking tasks... [10:54:55] PROBLEM - Content Translation Server on deployment-cxserver03 is CRITICAL: Connection refused [10:59:54] RECOVERY - Content Translation Server on deployment-cxserver03 is OK: HTTP OK: HTTP/1.1 200 OK - 1103 bytes in 0.044 second response time [11:22:52] 10Continuous-Integration, 10OOjs-UI, 10VisualEditor: Ignore abstract methods in code coverage reports - https://phabricator.wikimedia.org/T95413#1189212 (10Jdforrester-WMF) p:5Triage>3Normal [11:29:14] hasharLunch: https://integration.wikimedia.org/ci/job/cxserver-deploy-npm/55/console - how can we fix this. Seems blocker now. [12:06:27] hasharLunch: ping me once back :) [12:14:57] (03PS17) 10Hashar: Package python deps with dh-virtualenv [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/195272 (https://phabricator.wikimedia.org/T48552) [12:15:40] (03CR) 10Hashar: "Added /var/run/zuul and /var/run/zuul-merger to debian/dirs" [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/195272 (https://phabricator.wikimedia.org/T48552) (owner: 10Hashar) [12:24:29] !log killed zuul on gallium :/ [12:24:35] Logged the message, Master [12:30:08] hasharLunch: Zuul-killer! [12:30:58] now i was wondering why it didn't react to me anymore [12:35:29] andre__: is this a known issue? it.wikipedia.org is down [12:35:36] andre__: looking in phab... [12:36:00] zeljkof, uh, I don't know - #wikimedia-operations ? [12:36:10] andre__: thanks, will ask [12:36:14] yes, known there [12:36:18] zeljkof, all s2 wikis [12:36:32] andre__: somebody asked at wikiteck [12:36:38] ah, wikitech [12:37:35] PROBLEM - SSH on deployment-lucid-salt is CRITICAL: Connection refused [12:43:10] !log Zuul is back and it is nasty [12:43:13] Logged the message, Master [12:43:35] kart_: hey! Sorry no bandwith to handle anything this week :( [12:49:06] hashar: just have a look? [12:49:14] hashar: blocks deployment :/ [12:49:40] kart_: :- [12:49:45] I made a hack for it iirc [12:49:58] ah yeah https://gerrit.wikimedia.org/r/#/c/189473/ [12:50:05] (03PS6) 10Hashar: WIP: Hack for npm oid jobs [integration/config] - 10https://gerrit.wikimedia.org/r/189473 (https://phabricator.wikimedia.org/T92369) [12:50:32] kart_: so in theory we can deploy that change for your cxserver job [12:52:20] (03CR) 10Hashar: "I have manually redeployed the jobs cxserver-deploy-npm cxserver-source-npm" [integration/config] - 10https://gerrit.wikimedia.org/r/189473 (https://phabricator.wikimedia.org/T92369) (owner: 10Hashar) [12:56:18] hashar: thanks [12:57:03] kart_: seems it pass now https://gerrit.wikimedia.org/r/#/c/202703/ [12:57:20] cool [12:57:33] kart_: I think the issue is for the deploy repository we set NPM_SET_PATH to /src/ [12:57:40] and grunt does not recognize that env var :( [13:02:49] (03PS6) 10Hashar: Forward port precise dh-virtualenv to trusty [integration/zuul] (debian/trusty-wikimedia) - 10https://gerrit.wikimedia.org/r/197329 (https://phabricator.wikimedia.org/T48552) [13:05:07] (03CR) 10JanZerebecki: [C: 031] "Thanks! Deployed to Jenkins. Works: https://integration.wikimedia.org/ci/job/mwext-WikibaseJavaScriptApi-qunit/50/" [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (https://phabricator.wikimedia.org/T86176) (owner: 10Adrian Lang) [13:28:21] PROBLEM - Puppet failure on integration-slave1405 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [13:30:09] PROBLEM - Puppet failure on integration-slave1403 is CRITICAL: CRITICAL: 71.43% of data above the critical threshold [0.0] [13:30:12] !log integration: upgrading python-gear and python-six on Trusty slaves [13:30:15] Logged the message, Master [13:31:04] !log integration: running apt-get upgrade on Trusty slaves [13:31:07] Logged the message, Master [13:31:47] PROBLEM - Puppet failure on integration-slave-trusty-1002 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [13:32:16] !log Disabled Zuul install based on git clone / setup.py by cherry picking https://gerrit.wikimedia.org/r/#/c/202714/ . Installed the Zuul debian package on all slaves [13:32:19] Logged the message, Master [13:33:03] PROBLEM - Puppet failure on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [13:42:12] PROBLEM - Puppet failure on integration-slave-trusty-1005 is CRITICAL: CRITICAL: 28.57% of data above the critical threshold [0.0] [13:42:52] PROBLEM - Puppet failure on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:45:28] PROBLEM - Puppet failure on integration-slave-trusty-1004 is CRITICAL: CRITICAL: 42.86% of data above the critical threshold [0.0] [13:52:58] why isn't https://gerrit.wikimedia.org/r/#/c/202710/ merging? [13:53:19] hashar, kart_: any idea? ^ [13:53:21] RECOVERY - Puppet failure on integration-slave1405 is OK: OK: Less than 1.00% above the threshold [0.0] [13:54:12] aharoni: the jobs are still running maybe? [13:54:23] or havent triggered caused I killed zuul when it has been +2ed [13:54:26] so remove the +2 vote [13:54:29] and vote +2 again [13:54:34] that should trigger the jobs and merge it [13:55:08] RECOVERY - Puppet failure on integration-slave1403 is OK: OK: Less than 1.00% above the threshold [0.0] [14:02:52] RECOVERY - Puppet failure on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [14:03:03] RECOVERY - Puppet failure on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [14:05:37] RECOVERY - Puppet failure on integration-slave-trusty-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [14:07:16] RECOVERY - Puppet failure on integration-slave-trusty-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [14:11:42] RECOVERY - Puppet failure on integration-slave-trusty-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [14:14:08] Yippee, build fixed! [14:14:08] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #582: FIXED in 43 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/582/ [14:15:03] 10Continuous-Integration, 10Tool-Labs: labs-toollabs-debian-glue fails apparently with a timeout - https://phabricator.wikimedia.org/T91247#1189504 (10hashar) 5Open>3Resolved a:3hashar Maybe some transient issue ? We might had an issue on the slaves when the job ran. From the build history at https://in... [14:18:37] having a break with kids [14:18:41] be back later on [14:35:35] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #466: FAILURE in 9 min 34 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/466/ [14:50:16] Yippee, build fixed! [14:50:17] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce build #573: FIXED in 31 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce/573/ [15:00:47] 3Continuous-Integration-Isolation, 6operations: install/deploy labnodepool1001 - https://phabricator.wikimedia.org/T95045#1189604 (10Cmjohnson) [15:10:55] PROBLEM - Content Translation Server on deployment-cxserver03 is CRITICAL: Connection refused [15:11:47] 6Release-Engineering, 6MediaWiki-Core-Team, 10MediaWiki-Debug-Logging, 10Wikimedia-Logstash, and 2 others: Log php fatals with full backtraces again (fatal.log on fluorine) - https://phabricator.wikimedia.org/T89169#1189640 (10Legoktm) Progress! We now have logs that look like: ``` 2015-04-08 15:08:56 mw12... [15:15:56] RECOVERY - Content Translation Server on deployment-cxserver03 is OK: HTTP OK: HTTP/1.1 200 OK - 1103 bytes in 0.041 second response time [15:23:51] 10Deployment-Systems, 6Release-Engineering, 7Documentation: document trebuchet - https://phabricator.wikimedia.org/T94619#1189677 (10thcipriani) 5Open>3Resolved [15:32:51] 10Browser-Tests, 6Release-Engineering: Things to do after Chris leaves - https://phabricator.wikimedia.org/T94032#1189737 (10zeljkofilipin) 5Open>3stalled a:5zeljkofilipin>3None [15:33:19] 10Browser-Tests, 6Release-Engineering: Things to do after Chris leaves - https://phabricator.wikimedia.org/T94032#1153118 (10zeljkofilipin) Nothing to do here, there is one blocking task that needs to be resolved. [15:36:17] 10Browser-Tests: Transfer the main Sauce Labs account to zeljkofilipin - https://phabricator.wikimedia.org/T94191#1189757 (10zeljkofilipin) > Renata Santillan > Sauce Labs > > Hi Zeljiko, > > There's already an account linked with that email: zfilipin@wikimedia.org. You can go ahead and create a new account u... [15:36:23] greg-g: around? [15:36:23] zeljkof: You sent me a contentless ping. This is a contentless pong. Please provide a bit of information about what you want and I will respond when I am around. [15:37:09] greg-g: regarding https://phabricator.wikimedia.org/T94191, did you ping OIT and get releng@wikimedia.org e-mail address created? [16:01:20] 10Browser-Tests: Transfer the main Sauce Labs account to a generic WMF account - https://phabricator.wikimedia.org/T94191#1189909 (10zeljkofilipin) [16:17:56] git HEAD in deployment-salt is point to March 31. [16:17:56] What's up with it? [16:17:57] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:24:17] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:24:20] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:24:33] 10Browser-Tests, 6Release-Engineering: Do not say "< wmf-insecte> Yippee, build fixed!" - https://phabricator.wikimedia.org/T95395#1190063 (10greg) p:5Triage>3Low Since wmf-insecte talks mostly about #browser-tests I'm associating that project. @zeljkofilipin do you know where this bot is managed (where i... [16:25:57] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 47068 bytes in 0.575 second response time [16:25:57] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:58] Project beta-scap-eqiad build #48244: FAILURE in 2 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/48244/ [16:26:14] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 47190 bytes in 5.195 second response time [16:31:35] kart_: not sure. aper :( [16:31:36] greg-g: should I reset to HEAD? [16:31:37] PROBLEM - App Server bits response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:31:37] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:31:38] PROBLEM - App Server bits response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:31:38] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:31:41] 10Browser-Tests, 6Release-Engineering: Do not say "< wmf-insecte> Yippee, build fixed!" - https://phabricator.wikimedia.org/T95395#1190068 (10greg) [16:32:36] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 27930 bytes in 0.602 second response time [16:42:38] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:43:35] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 27930 bytes in 0.665 second response time [16:44:57] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 46869 bytes in 0.638 second response time [16:45:03] RECOVERY - App Server bits response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 3895 bytes in 0.002 second response time [16:45:50] Yippee, build fixed! [16:45:51] Project beta-scap-eqiad build #48246: FIXED in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/48246/ [16:45:59] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 47058 bytes in 0.970 second response time [17:17:37] (03PS29) 10Legoktm: Fix WikibaseJavaScriptApi tests [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (https://phabricator.wikimedia.org/T86176) (owner: 10Adrian Lang) [17:17:46] (03CR) 10Legoktm: [C: 032] Fix WikibaseJavaScriptApi tests [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (https://phabricator.wikimedia.org/T86176) (owner: 10Adrian Lang) [17:21:22] (03PS3) 10Krinkle: Delete old jshint jobs from repos without any *.js files (A-B) [integration/config] - 10https://gerrit.wikimedia.org/r/202445 [17:22:27] (03CR) 10Krinkle: [C: 032] Delete old jshint jobs from repos without any *.js files (A-B) [integration/config] - 10https://gerrit.wikimedia.org/r/202445 (owner: 10Krinkle) [17:23:36] (03PS1) 10Dduvall: [WIP] Fallback on base password configuration [selenium] - 10https://gerrit.wikimedia.org/r/202777 [17:31:14] (03Merged) 10jenkins-bot: Fix WikibaseJavaScriptApi tests [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (https://phabricator.wikimedia.org/T86176) (owner: 10Adrian Lang) [17:33:08] legoktm@gallium:~$ tail -f -n100 /var/log/zuul/zuul.log [17:33:08] tail: cannot open `/var/log/zuul/zuul.log' for reading: Permission denied [17:33:13] 10Deployment-Systems, 6Release-Engineering, 6operations: Determine Trebuchet/git-deploy maintenance plan - https://phabricator.wikimedia.org/T85008#1190603 (10demon) >>! In T85008#964243, @Ryan_Lane wrote: > I'm more than happy to add WMF as maintainers. It's not unmaintained, but no one has been bugging me... [17:33:26] <^d> greg-g: ^^ [17:33:35] drwxr-x--- 2 zuul adm 32768 Apr 8 00:01 zuul [17:33:46] why does root own that? [17:34:03] Krinkle: ^ ? [17:34:18] ^d: So I need to branch from REL1_25 today and create 1.26wmf1? [17:34:37] <^d> Branch from master like usual [17:34:39] legoktm: it doesn't say root there? [17:34:45] it says zuul, no? [17:34:48] <^d> twentyafterfour: Master is 1.26alpha [17:34:54] <^d> REL1_25 is 1.25beta [17:35:03] ok [17:35:12] (03PS4) 10Krinkle: Delete old jshint jobs from repos without any *.js files (A-B) [integration/config] - 10https://gerrit.wikimedia.org/r/202445 [17:35:16] Krinkle: that's the /var/log/zuul directory...I can't even look at it [17:35:21] (03CR) 10Krinkle: [C: 032] Delete old jshint jobs from repos without any *.js files (A-B) [integration/config] - 10https://gerrit.wikimedia.org/r/202445 (owner: 10Krinkle) [17:35:36] legoktm: sudo -su zuul [17:35:40] cd /var/log/zuul [17:35:50] 10Deployment-Systems, 6Release-Engineering, 6operations: Determine Trebuchet/git-deploy maintenance plan - https://phabricator.wikimedia.org/T85008#1190612 (10greg) 5Open>3Resolved >>! In T85008#1190603, @demon wrote: >>>! In T85008#964243, @Ryan_Lane wrote: >> I'm more than happy to add WMF as maintaine... [17:35:53] oh ugh [17:36:05] I used to be able to do that as legoktm [17:36:09] me too [17:36:12] ^d: ty! [17:36:19] !log deployed https://gerrit.wikimedia.org/r/180418 [17:36:23] Logged the message, Master [17:36:34] <^d> twentyafterfour: For deployments you can generally ignore the REL1_* branches. They're for the stable releases that we don't run [17:37:41] (03PS2) 10Krinkle: doc: Add redirect for old mediawiki-core/master/php/html/ [integration/docroot] - 10https://gerrit.wikimedia.org/r/202413 (https://phabricator.wikimedia.org/T73060) [17:38:29] <^d> thcipriani, marxarelli, twentyafterfour: Also, this. https://phabricator.wikimedia.org/T85008#1190603 [17:38:37] <^d> (re: what we talked about monday) [17:38:52] (03Merged) 10jenkins-bot: Delete old jshint jobs from repos without any *.js files (A-B) [integration/config] - 10https://gerrit.wikimedia.org/r/202445 (owner: 10Krinkle) [17:41:38] ^d: nice! Trebuchet certainly does look flexible. [17:41:42] (03PS2) 10Legoktm: Run mwext-Wikibase phpunit jobs on HHVM too [integration/config] - 10https://gerrit.wikimedia.org/r/202289 (https://phabricator.wikimedia.org/T95230) [17:45:37] (03CR) 10Legoktm: [C: 032] Run mwext-Wikibase phpunit jobs on HHVM too [integration/config] - 10https://gerrit.wikimedia.org/r/202289 (https://phabricator.wikimedia.org/T95230) (owner: 10Legoktm) [17:50:05] (03PS1) 10Krinkle: doc: Update link to MediaWiki core to avoid redirect. [integration/docroot] - 10https://gerrit.wikimedia.org/r/202783 [17:50:28] (03CR) 10Krinkle: [C: 032] doc: Update link to MediaWiki core to avoid redirect. [integration/docroot] - 10https://gerrit.wikimedia.org/r/202783 (owner: 10Krinkle) [17:51:16] (03CR) 10Krinkle: [C: 032] Fix MediaWiki core PHP documentation link [integration/docroot] - 10https://gerrit.wikimedia.org/r/202683 (owner: 10TTO) [17:51:44] (03Abandoned) 10Krinkle: doc: Update link to MediaWiki core to avoid redirect. [integration/docroot] - 10https://gerrit.wikimedia.org/r/202783 (owner: 10Krinkle) [17:51:56] (03PS2) 10Krinkle: Fix MediaWiki core PHP documentation link [integration/docroot] - 10https://gerrit.wikimedia.org/r/202683 (owner: 10TTO) [17:52:15] (03PS3) 10Krinkle: Fix MediaWiki core PHP documentation link [integration/docroot] - 10https://gerrit.wikimedia.org/r/202683 (owner: 10TTO) [17:52:21] (03CR) 10Krinkle: [C: 032] Fix MediaWiki core PHP documentation link [integration/docroot] - 10https://gerrit.wikimedia.org/r/202683 (owner: 10TTO) [18:01:25] !log Jobs for Precise slaves are not starting. Stuck in Zuul as 'queued'. Disconnected and restarted slave agent on them. Queue is back up now. [18:01:28] Logged the message, Master [18:04:47] (03Merged) 10jenkins-bot: Run mwext-Wikibase phpunit jobs on HHVM too [integration/config] - 10https://gerrit.wikimedia.org/r/202289 (https://phabricator.wikimedia.org/T95230) (owner: 10Legoktm) [18:06:23] Krinkle: you didn't deploy your zuul change yet? [18:06:59] legoktm: Indeed. [18:07:10] legoktm: Had to get my power [18:07:24] legoktm: Feel free to let it tag along [18:07:27] ok [18:07:40] !log deploying https://gerrit.wikimedia.org/r/202289 and https://gerrit.wikimedia.org/r/202445 [18:07:43] Logged the message, Master [18:10:39] legoktm: Wow, integration.wikimedia.org/ci/ [18:10:43] all slots of all slaves are in use [18:10:59] Try not to do too much right now. :-/ [18:11:07] no job reconfigures [18:11:10] yeah I think duplicating all the Wikibase jobs made it worse :/ [18:11:12] or deletions [18:11:20] I'm about to delete the workspaces of the old Wikibase jobs [18:11:25] ok [18:19:31] PROBLEM - Host integration-slave-trusty-1005 is DOWN: CRITICAL - Host Unreachable (10.68.18.2) [18:19:59] PROBLEM - Host integration-slave-trusty-1004 is DOWN: CRITICAL - Host Unreachable (10.68.17.244) [18:20:27] PROBLEM - Host integration-slave-trusty-1001 is DOWN: CRITICAL - Host Unreachable (10.68.17.130) [18:20:39] PROBLEM - Host integration-slave-trusty-1003 is DOWN: CRITICAL - Host Unreachable (10.68.17.239) [18:21:57] PROBLEM - Host integration-slave-trusty-1002 is DOWN: CRITICAL - Host Unreachable (10.68.17.209) [18:24:33] (03CR) 10Aude: "@legoktm ok with the change generally." [integration/config] - 10https://gerrit.wikimedia.org/r/202289 (https://phabricator.wikimedia.org/T95230) (owner: 10Legoktm) [18:24:50] (03CR) 10Aude: "and thanks for making the patch so quick :)" [integration/config] - 10https://gerrit.wikimedia.org/r/202289 (https://phabricator.wikimedia.org/T95230) (owner: 10Legoktm) [18:29:01] !log Another attempt at re-creating the Trusty slave pool (T94916) [18:29:03] Logged the message, Master [18:44:03] https://integration.wikimedia.org/zuul/ huge queue on gate-and-submit [18:44:43] I was just about to say ... it's backed up almost an hour [18:45:28] [21:09:02] Krinkle> all slots of all slaves are in use [18:45:56] still, 60 mins seems excessive [18:46:08] Nikerabbit: See log [18:46:13] that was not the reason for delay [18:46:26] https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:46:30] 18:01 Krinkle: Jobs for Precise slaves are not starting. Stuck in Zuul as 'queued'. Disconnected and restarted slave agent on them. Queue is back up now. [18:46:49] Trusty jobs were ahead and Precise was holding back the queue from merging in Gerrit [18:46:57] They were stuck for about 50 minutes before we noticed. [18:49:37] PROBLEM - Puppet failure on integration-slave-trusty-1010 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:50:11] uhm ... [18:50:40] soo, previous phpunit-zend finished in 10 minutes, the current one blocking everything is 15 minutes and at 50% progress :( [18:54:30] Nikerabbit: Our load is increasing a lot lately, and we're understaffed, doing everything we can while also fulfilling our quarterly goals in other teams :-/ [18:54:33] RECOVERY - Puppet failure on integration-slave-trusty-1010 is OK: OK: Less than 1.00% above the threshold [0.0] [18:54:48] I'm provisioning 2 extra slaves at the moment to provide some extra power [18:55:07] And I'm hoping mediawiki core will continue to separate libraries into standalone libs so our main unit test runs faster [18:56:17] it sucks that a simple symlink change in mediawiki-staging triggers a whole slew of test runs (initial test, gate and submit, post build) on mediawiki-core ... [18:57:22] twentyafterfour: Due to our job shortcircuiting jobs have to wait for somewahat unrelated jobs [18:58:06] I think it actually triggers mediawiki core tests, not just waiting for unrelated ones [18:58:14] And thanks to someone force merging (not sure what/where), I now have to restart the queue [18:58:22] twentyafterfour: No, it does not and it never did [18:58:34] mwcore-config takes 2 seconds to run [18:58:41] hmm [18:58:42] it's waiting for other changes that are ahead in the queue [19:00:13] !log Zuul queue is not being distributed properly. Many slaves are idling waiting to receive builds but not getting any. [19:00:15] Logged the message, Master [19:00:34] !log Jenkins Master unable to re-establish Gearman connection [19:00:38] Logged the message, Master [19:00:43] fuck stuff double roasted duck shit [19:00:49] It lost the queue [19:00:57] Krinkle: sounds like something to bubble up for management to get more resources [19:00:58] !log Restarting Jenkins [19:01:00] Logged the message, Master [19:02:12] Nikerabbit: As much as we have resources to improve features and frameworks, we're held back by instabilities in Gerrit, Jenkins, Gearman, and Zuul. [19:02:19] Jenkins itself is fine most of the time. [19:02:28] We've had one maybe 2 issues with Jenkins over the past year. [19:02:41] But everybody just calls it "Jenkins" [19:04:26] I wouldn't be surprised if gerrit was a big part of the problem. ..it sure is slow [19:04:45] anything i can do to help? [19:05:07] Try not to force merge anything. Zuul is known to get confused by that. [19:05:14] Maybe work on something else and let it merge later. [19:05:22] Though for deployment that is hard :D [19:06:56] yes it's very difficult for outsiders to differentiate different reasons for "omg my patches are not getting merged" [19:09:33] !log Re-establishing Gearman-Jenkins connection [19:09:35] Logged the message, Master [19:09:51] OK. It should be flushing now [19:11:38] (03PS1) 10Ori.livneh: Stop running Zend tests [integration/config] - 10https://gerrit.wikimedia.org/r/202804 [19:12:35] (03CR) 10Greg Grossmeier: "Why not just move them to post-merge?" [integration/config] - 10https://gerrit.wikimedia.org/r/202804 (owner: 10Ori.livneh) [19:13:03] ahoy [19:13:10] ahoy hoy [19:13:16] ori: The slow down of zend jobs just now is unrelated to them being zend. zend tests tend to take 10 minutes compared to 4 minutes for hhvm. [19:13:28] i know, but it's not just now [19:13:43] The slow down just now was that Gearman locked up, and zend jobs where on top of the queue because we have less precise slaves in general. Things are generally not blocked on opt. [19:13:47] on it(. [19:14:48] not in my (admittedly anecdotal) experience. but really, if travis ci is free / open source, why not use it for zend tests? [19:14:51] (03CR) 10Krinkle: "(also in zuul/layout.yaml to avoid job-not-exist errors)" [integration/config] - 10https://gerrit.wikimedia.org/r/202804 (owner: 10Ori.livneh) [19:15:09] ori: I;ve proposed that [19:15:11] but.. [19:15:16] (it's not fully open source) [19:15:17] We're still using zend in prod, and nobody looks at Travis. [19:15:23] Not in time for deployment anyway. [19:15:34] We may be able to socially enforce making Travis CI pass before releases. [19:15:36] but not deployments. [19:16:15] are the zend tests voting? [19:16:18] Yes [19:16:37] voting is not related to speed though voting or not is blocking either way. [19:16:46] we already removed zend jobs from test pipeline, only on postmerge [19:16:48] when was the last time they caught a bug? [19:17:32] ori: a week ago [19:17:43] link? [19:18:00] I don't want to answer that question. I refuse principally. Risk is too great. [19:18:11] on https://gerrit.wikimedia.org/r/#/c/201495/ [19:18:16] ori: ^ [19:19:17] last year I would've said, let's add Travis CI badge to integration.wikimedia.org and set up .travis.yaml irc bot in #wikimedia-dev [19:19:23] well, okay [19:19:25] the problem is, it's no longer just one repository [19:19:27] legoktm's example is pretty good [19:20:10] I think having jenkins do the 5.3 tests is crucial because no one does it locally [19:20:28] (03Merged) 10jenkins-bot: Fix MediaWiki core PHP documentation link [integration/docroot] - 10https://gerrit.wikimedia.org/r/202683 (owner: 10TTO) [19:20:37] there's also tons of room for improving the tests in general to make them faster [19:20:46] ori: Yeah, our tests are the worst. [19:20:50] LIttle to no mocking. [19:20:50] (03CR) 10Ori.livneh: "legoktm has a good example of the zend tests making a difference, which is a point against this patch: https://gerrit.wikimedia.org/r/#/c/" [integration/config] - 10https://gerrit.wikimedia.org/r/202804 (owner: 10Ori.livneh) [19:20:54] legoktm: see also: https://phabricator.wikimedia.org/T95282 :) [19:21:04] (i have written some code recently that would have worked incorrectly in 5.3, jenkins caught it) [19:21:25] ok, good arguments! [19:21:30] * ori concedes. [19:21:35] ori: I love you still [19:21:36] (oh look, that's the example you have) [19:21:49] but, what about post-merge? [19:21:51] https://gerrit.wikimedia.org/r/#/c/195026/ removed a full minute of tests on zend, https://gerrit.wikimedia.org/r/#/c/200284/ found random tests that were hitting the DB when they didn't need to [19:21:52] greg-g: what about your suggestion to move it to post-merge? [19:21:53] samesies [19:21:54] (03CR) 10Krinkle: "We removed zend tests from mediawiki-core test pipeline to speed up the most common build. They still run on gate-and-submit, however." [integration/config] - 10https://gerrit.wikimedia.org/r/202804 (owner: 10Ori.livneh) [19:21:55] is that the level of support we care about? [19:22:07] no one ever looks at post-merge [19:22:18] that might be a "bigger than this group of people on this irc channel over lunch SF time" question [19:22:37] post-merge means monitoring indivudual Jenkins jobs manually. [19:22:42] And there are too many to easily draw up a dashboard. [19:22:42] :/ yeah, fair [19:22:46] Though we can try [19:22:56] what about doing it on submission rather than gate-and-submit? [19:22:57] when can we deprecate zend support? :P [19:22:59] * greg-g kids [19:23:00] that way if you cherry-pick and +2 [19:23:06] the submit isn't blocked on zend [19:23:10] ori: define submission [19:23:17] patch upload i mean [19:23:22] sorry, i see the ambiguity [19:23:24] ori: That's what we used to do. [19:23:26] it already doesn't run on patch upload [19:23:30] Which slows down everything [19:23:34] you have to comment "check zend" explicitly [19:23:38] oh yeah, because there are many more [19:23:43] also a good point [19:23:52] https://phabricator.wikimedia.org/T94322 would probably make the most noticable impact for people [19:23:59] god damn it [19:24:00] We removed them form the test pipeline (=on patch creation), and instead only on gate (=CR+2) [19:24:02] you guys have all the good arguments [19:24:21] We've been optimising it a lot. Squeezing what we can. [19:24:42] We're down to just Zend being slow and mediawiki-core unit tests being largely Shit (TM) [19:24:59] (03Abandoned) 10Ori.livneh: Stop running Zend tests [integration/config] - 10https://gerrit.wikimedia.org/r/202804 (owner: 10Ori.livneh) [19:25:16] you win this time, releng [19:25:20] ;) [19:25:22] but i'll be back! [19:25:37] The main three things to speed that up. 1) Use fs mock, 2) Use class object mocks, 3) separate more libraries so that we run tests in general. [19:25:40] And we'll be waiting! [19:25:59] Yes ori please :) We definitely miss things sometimes. Stay tuned :) [19:26:09] changes merged. yay I can go back to deploying now ;) [19:26:11] ori: btw, got a few questions for you later about perf. [19:26:23] taking bus now first though. [19:26:24] * twentyafterfour is still following this discussion, good stuff! [19:26:25] cya ltr [19:31:29] PROBLEM - Puppet failure on integration-slave-trusty-1010 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [19:38:16] 10Browser-Tests, 10VisualEditor, 10VisualEditor-MediaWiki-Mobile: VisualEditor: Create browser tests for using VE via Mobile UI - https://phabricator.wikimedia.org/T62290#1191459 (10Jdforrester-WMF) [19:38:35] 10Browser-Tests, 10VisualEditor, 10VisualEditor-MediaWiki-Mobile: VisualEditor: Create browser tests for using VE via Mobile UI - https://phabricator.wikimedia.org/T62290#651148 (10Jdforrester-WMF) In the Triage meeting we felt that this wasn't a clear enough ask for Q4. [19:43:25] 10Continuous-Integration: Re-create ci slaves (April 2015) - https://phabricator.wikimedia.org/T94916#1191472 (10Krinkle) In creating `integration-slave-trusty-1010` I ran into the following issues: After the first two puppet runs, the base class (before applying `role::ci::slave::labs`) was still failing two i... [20:32:26] looks like there are 3 post-merge builds stuck in the queue (elapsed time > 1 hour) [20:35:29] twentyafterfour: I think maybe the doxygen job is pinned to slave1401, which is full right now [20:35:59] I don't even see them now .. hmm [20:36:48] oh it's because I refreshed the status page. I think it was just a ui glitch [20:37:11] now if this test would just finish on mediawiki-core ;) [21:10:39] 10Continuous-Integration: Re-create ci slaves (April 2015) - https://phabricator.wikimedia.org/T94916#1191787 (10hashar) > (/Stage[main]/Zuul/Package[zuul]/ensure) E: Unable to locate package zuul I have switched the instances to the Zuul Debian package via cherry picked patch https://gerrit.wikimedia.org/r/#/... [21:15:14] !log deleting non-existent jobs' workspaces on labs slaves [21:15:16] Logged the message, Master [21:28:33] (03CR) 10Hashar: "The MediaWiki core tests being crazy slow has a few recent causes:" [integration/config] - 10https://gerrit.wikimedia.org/r/202804 (owner: 10Ori.livneh) [21:37:06] (03CR) 10Hashar: [C: 04-1] "mediawiki/tools/codesniffer holds rules meant to be consumed by PHP CodeSniffer." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/201956 (owner: 10MarkAHershberger) [21:47:56] 10Continuous-Integration, 10Incident-20150312-whitespace: add a check for whitespace before leading (03PS1) 10Legoktm: Run tox-flake8-bin for operations/software/tools-manifest too [integration/config] - 10https://gerrit.wikimedia.org/r/202930 [21:53:18] 6Release-Engineering, 3releng-201415-Q3, 3releng-201415-q4: [Quarterly Success Metric] RelEng+TPG process discussion and improvements (tracking) - https://phabricator.wikimedia.org/T88708#1191994 (10greg) [21:54:20] (03CR) 10Legoktm: [C: 032] Run tox-flake8-bin for operations/software/tools-manifest too [integration/config] - 10https://gerrit.wikimedia.org/r/202930 (owner: 10Legoktm) [21:55:03] 6Release-Engineering, 6Engineering-Community, 6Team-Practices, 10Wikimedia-Hackathon-2015: RelEng team offsite - May 2015 - Pre Wikimedia Hackathon - https://phabricator.wikimedia.org/T89036#1191998 (10greg) [21:55:06] 6Release-Engineering, 3releng-201415-Q3, 3releng-201415-Q4: [Quarterly Success Metric] RelEng+TPG process discussion and improvements (tracking) - https://phabricator.wikimedia.org/T88708#1018568 (10greg) [21:55:36] (03Merged) 10jenkins-bot: Run tox-flake8-bin for operations/software/tools-manifest too [integration/config] - 10https://gerrit.wikimedia.org/r/202930 (owner: 10Legoktm) [21:55:36] 6Release-Engineering, 3releng-201415-Q3, 3releng-201415-Q4: [Quarterly Success Metric] RelEng+TPG process discussion and improvements (tracking) - https://phabricator.wikimedia.org/T88708#1192000 (10greg) 5Open>3stalled Stalling for now until we do the team offsite in France. [21:56:08] !log deploying https://gerrit.wikimedia.org/r/202930 [21:56:12] Logged the message, Master [22:00:33] 6Release-Engineering, 3releng-201415-Q4: RelEng Roadmap April - June 2015 (Q4 2014/2015) - https://phabricator.wikimedia.org/T93955#1192066 (10greg) [22:02:48] (03CR) 10Legoktm: "We could do alternate stuff like that, except it feels like you're just asking for subtle bugs that way :P" [integration/config] - 10https://gerrit.wikimedia.org/r/202289 (https://phabricator.wikimedia.org/T95230) (owner: 10Legoktm) [22:03:55] 6Release-Engineering, 5MW-1.25-release, 3releng-201415-Q3, 3releng-201415-Q4: [Quarterly Success Metric] Release MediaWiki 1.25 - https://phabricator.wikimedia.org/T88709#1192089 (10greg) [22:06:33] 6Release-Engineering, 5MW-1.25-release, 3releng-201415-Q3, 3releng-201415-Q4: [Quarterly Success Metric] Release MediaWiki 1.25 - https://phabricator.wikimedia.org/T88709#1192105 (10greg) a:3demon [22:09:03] (03PS1) 10Legoktm: Merge mwext-Wikibase-* repo and repo-api jobs [integration/config] - 10https://gerrit.wikimedia.org/r/202932 [22:09:58] (03CR) 10Legoktm: "I7ac55f413abf9879d8ccf1a4fb7186d4e0636e09" [integration/config] - 10https://gerrit.wikimedia.org/r/202289 (https://phabricator.wikimedia.org/T95230) (owner: 10Legoktm) [22:13:52] 10Deployment-Systems, 5Patch-For-Review: LocalisationUpdate needs to support updating skins/ as well as extensions/ - https://phabricator.wikimedia.org/T69154#1192156 (10greg) >>! In T69154#1158234, @greg wrote: > This can be merged and deployed at any time that someone is willing to either run l10nupdate manu... [22:14:02] 10Deployment-Systems, 5Patch-For-Review: LocalisationUpdate needs to support updating skins/ as well as extensions/ - https://phabricator.wikimedia.org/T69154#1192158 (10greg) 5Open>3Resolved [22:14:57] 6Release-Engineering, 6MediaWiki-API-Team, 10MediaWiki-Debug-Logging, 10Wikimedia-Logstash, and 2 others: Log php fatals with full backtraces again (fatal.log on fluorine) - https://phabricator.wikimedia.org/T89169#1192161 (10ksmith) [22:17:00] (03CR) 10MarkAHershberger: "I didn't know where else to put this. If you point me somewhere appropriate, I'll put it there." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/201956 (owner: 10MarkAHershberger) [22:19:22] 6Release-Engineering, 10MediaWiki-Maintenance-scripts, 10MediaWiki-Redirects, 5Patch-For-Review: namespaceDupes not handling deleted namespace redirects as desired - https://phabricator.wikimedia.org/T91401#1192194 (10He7d3r) Any progress since last week? This regression is still causing confusion: https:/... [22:19:48] 6Release-Engineering, 10MediaWiki-Debug-Logging, 6Security-Team, 6operations, 5Patch-For-Review: Store unsampled API and XFF logs - https://phabricator.wikimedia.org/T88393#1192197 (10RobLa-WMF) [22:25:08] 10Deployment-Systems, 5Patch-For-Review: [l10n] Use Scap in Localisation Update - https://phabricator.wikimedia.org/T72443#1192230 (10greg) What's the status of this @mmodell / @bd808 ? [22:27:09] 10Deployment-Systems: [scap] Add a log appender to log to a local file - https://phabricator.wikimedia.org/T68857#1192249 (10greg) a:5mmodell>3None resetting assignee for now [22:27:57] 10Deployment-Systems: [scap] Make the hostname of a failing host more prominent in the error messages - https://phabricator.wikimedia.org/T68302#1192251 (10greg) a:5mmodell>3None [22:28:23] 10Deployment-Systems: [scap] Consolidate scripts as sub-commands of `scap` - https://phabricator.wikimedia.org/T67827#1192254 (10greg) a:5mmodell>3None [22:29:10] 10Deployment-Systems: Make make-wmf-branch able to branch extensions with replaced substring of the version of mediawiki being branched - https://phabricator.wikimedia.org/T51392#1192259 (10greg) a:5Reedy>3None [22:30:01] 10Deployment-Systems, 6WMF-Legal, 7Documentation: mediawiki/tools/scap is lacking a license - https://phabricator.wikimedia.org/T94239#1192261 (10greg) p:5Triage>3Low [22:31:44] 10Deployment-Systems, 7HHVM: HHVM lock-ups - https://phabricator.wikimedia.org/T89912#1192271 (10greg) Quoting the description from @ori: > HHVM has been locking up in production, typically right after a big deployment which touches lots of file. Typically only one or two app servers are affected. I don't have... [22:33:54] 10Deployment-Systems, 6Release-Engineering: [scap] Suppress/de-emphasize errors from hosts marked for maintenance in icinga - https://phabricator.wikimedia.org/T78319#1192281 (10greg) p:5High>3Normal [22:34:34] (03CR) 10Legoktm: [C: 04-1] Test that all mediawiki repos have phplint jobs (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/202452 (owner: 10Legoktm) [22:35:28] 10Deployment-Systems, 7I18n: i18n cache vs resourceloader race condition (RL message key empty) - https://phabricator.wikimedia.org/T68543#1192294 (10greg) p:5High>3Normal [22:39:06] (03PS2) 10Legoktm: Generalize test_mw_repos_have_composer_validate_job for any job [integration/config] - 10https://gerrit.wikimedia.org/r/202451 [22:39:08] (03PS2) 10Legoktm: Test that all mediawiki repos have phplint jobs [integration/config] - 10https://gerrit.wikimedia.org/r/202452 [22:39:24] (03PS3) 10Legoktm: Test that all mediawiki repos have phplint jobs [integration/config] - 10https://gerrit.wikimedia.org/r/202452 [22:41:54] (03CR) 10Legoktm: [C: 032] Generalize test_mw_repos_have_composer_validate_job for any job [integration/config] - 10https://gerrit.wikimedia.org/r/202451 (owner: 10Legoktm) [22:42:40] (03PS4) 10Legoktm: Test that all mediawiki repos have phplint jobs [integration/config] - 10https://gerrit.wikimedia.org/r/202452 [22:42:52] (03CR) 10Legoktm: [C: 032] Test that all mediawiki repos have phplint jobs [integration/config] - 10https://gerrit.wikimedia.org/r/202452 (owner: 10Legoktm) [22:44:26] 10Beta-Cluster, 6operations, 7HHVM: Convert work machines (tin, terbium) to Trusty and hhvm usage - https://phabricator.wikimedia.org/T87036#1192308 (10bd808) [22:44:29] 6Release-Engineering, 10MediaWiki-Debug-Logging, 10MediaWiki-General-or-Unknown, 5MW-1.23-release, 15User-Bd808-Test: Create a minimal backport of PSR-3 logging to MediaWiki 1.23 LTS - https://phabricator.wikimedia.org/T91653#1192311 (10bd808) [22:45:06] (03PS1) 10Legoktm: Add phplint job for mediawiki/vendor [integration/config] - 10https://gerrit.wikimedia.org/r/202938 [22:49:04] (03Merged) 10jenkins-bot: Generalize test_mw_repos_have_composer_validate_job for any job [integration/config] - 10https://gerrit.wikimedia.org/r/202451 (owner: 10Legoktm) [22:49:06] (03Merged) 10jenkins-bot: Test that all mediawiki repos have phplint jobs [integration/config] - 10https://gerrit.wikimedia.org/r/202452 (owner: 10Legoktm) [22:51:24] it looks like most jobs are grouping on slave1401 and 1404 [22:53:52] https://phabricator.wikimedia.org/T84911 [23:19:33] 10Deployment-Systems: [l10n] l10nupdate process should respect the scap lock file - https://phabricator.wikimedia.org/T72752#1192482 (10bd808) [23:19:34] 10Deployment-Systems, 5Patch-For-Review: [l10n] Use Scap in Localisation Update - https://phabricator.wikimedia.org/T72443#1192479 (10bd808) 5Open>3Resolved a:3bd808 {{done}} [23:25:11] 10Deployment-Systems, 10MediaWiki-extensions-LocalisationUpdate, 7Epic, 7I18n: Localization Cache Redo - https://phabricator.wikimedia.org/T78802#1192507 (10bd808) [23:42:23] 10Continuous-Integration: Re-evaluate use of "Dependent Pipeline" in Zuul for gate-and-submit in the short term - https://phabricator.wikimedia.org/T94322#1192568 (10Legoktm) @hashar: What you described is an ideal situation, but the reality is that any project that uses a general job like `phplint` or `tox-flak... [23:42:34] (03PS1) 10Legoktm: Make gate-and-submit an independent pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/202958 (https://phabricator.wikimedia.org/T94322)