[00:08:09] hashar: at this point it seems way more complicated to use jenkins to automatically do this stuff...why don't we just have a cron or something do it every 5 minutes? [00:10:14] legoktm: cause nobody looks at cron / mail spam :D [00:10:31] legoktm: we tried early on :/ [00:11:07] well, having it break jenkins seems like a terrible way to get visibility :/ [00:11:09] legoktm: also Jenkins provides some job history / trend / dashboard etc. That being said, we should probably set up a Jenkins dedicated to beta cluster [00:12:38] 10Beta-Cluster, 10Deployment-Systems, 7Regression: All http://bits.beta.wmflabs.org/static/master/* file urls return HTTP 403 Forbidden - https://phabricator.wikimedia.org/T98046#1259486 (10hashar) [00:12:38] 10Beta-Cluster: Occasionally getting 403 HTTP Method not allowed from bits - https://phabricator.wikimedia.org/T93021#1259485 (10hashar) [00:13:05] 10Beta-Cluster: Occasionally getting 403 HTTP Method not allowed from bits - https://phabricator.wikimedia.org/T93021#1126789 (10hashar) See T98046 for a more documented bug report :) [00:15:53] RECOVERY - Host integration-slave-trusty-1017 is UP: PING OK - Packet loss = 0%, RTA = 0.80 ms [00:17:29] PROBLEM - Host integration-zuul-packaged is DOWN: CRITICAL - Host Unreachable (10.68.16.200) [00:20:05] RECOVERY - Host integration-zuul-packaged is UP: PING OK - Packet loss = 0%, RTA = 0.62 ms [00:20:08] 10Deployment-Systems: Come up with an abstract deployment model that roughly addresses the needs of existing projects - https://phabricator.wikimedia.org/T97068#1259502 (10mmodell) p:5Normal>3High [00:21:05] Yippee, build fixed! [00:21:06] Project beta-update-databases-eqiad build #9382: FIXED in 1 min 5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9382/ [00:21:13] 10Staging: Create staging-eventlogging - https://phabricator.wikimedia.org/T91561#1259504 (10mmodell) a:5mmodell>3None [00:30:06] ah [00:30:10] good night folks [00:30:50] hashar: goodnight! [01:20:18] Project beta-update-databases-eqiad build #9383: FAILURE in 17 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9383/ [01:38:55] 10Beta-Cluster, 10Deployment-Systems, 7Regression: All http://bits.beta.wmflabs.org/static/master/* file urls return HTTP 403 Forbidden - https://phabricator.wikimedia.org/T98046#1259661 (10Mattflaschen) p:5Triage>3Unbreak! Note this has nothing to do with CORS AFAICT. Just visiting the file directly or... [01:58:47] 10Beta-Cluster, 10Deployment-Systems, 7Regression: All http://bits.beta.wmflabs.org/static/master/* file urls return HTTP 403 Forbidden - https://phabricator.wikimedia.org/T98046#1259676 (10ori) Fixed (indirectly) in https://gerrit.wikimedia.org/r/#/c/208878/ . [01:59:07] 10Beta-Cluster, 10Deployment-Systems, 7Regression: All http://bits.beta.wmflabs.org/static/master/* file urls return HTTP 403 Forbidden - https://phabricator.wikimedia.org/T98046#1259677 (10ori) 5Open>3Resolved a:3ori [02:20:30] Yippee, build fixed! [02:20:31] Project beta-update-databases-eqiad build #9384: FIXED in 30 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9384/ [02:25:16] Yippee, build fixed! [02:25:17] Project browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-linux-android-sauce build #90: FIXED in 3 min 15 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-linux-android-sauce/90/ [02:35:55] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #562: FAILURE in 2 min 54 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/562/ [03:29:19] PROBLEM - Puppet staleness on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [03:50:13] (03PS1) 10Legoktm: Use generic phpunit job for ApiFeatureUsage [integration/config] - 10https://gerrit.wikimedia.org/r/208889 [03:50:15] (03PS1) 10Legoktm: Use generic phpunit job for Capiunto [integration/config] - 10https://gerrit.wikimedia.org/r/208890 [03:50:17] (03PS1) 10Legoktm: Use generic phpunit job for Citoid [integration/config] - 10https://gerrit.wikimedia.org/r/208891 [03:50:19] (03PS1) 10Legoktm: Use generic phpunit job for CodeEditor [integration/config] - 10https://gerrit.wikimedia.org/r/208892 [03:50:21] (03PS1) 10Legoktm: Use generic qunit job for ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/208893 [04:00:01] (03CR) 10Legoktm: [C: 032] Use generic phpunit job for ApiFeatureUsage [integration/config] - 10https://gerrit.wikimedia.org/r/208889 (owner: 10Legoktm) [04:00:26] (03CR) 10Legoktm: [C: 032] Use generic phpunit job for Capiunto [integration/config] - 10https://gerrit.wikimedia.org/r/208890 (owner: 10Legoktm) [04:00:34] (03CR) 10Legoktm: [C: 032] Use generic phpunit job for Citoid [integration/config] - 10https://gerrit.wikimedia.org/r/208891 (owner: 10Legoktm) [04:00:51] (03CR) 10Legoktm: [C: 032] Use generic phpunit job for CodeEditor [integration/config] - 10https://gerrit.wikimedia.org/r/208892 (owner: 10Legoktm) [04:02:01] (03Merged) 10jenkins-bot: Use generic phpunit job for ApiFeatureUsage [integration/config] - 10https://gerrit.wikimedia.org/r/208889 (owner: 10Legoktm) [04:02:22] (03Merged) 10jenkins-bot: Use generic phpunit job for Capiunto [integration/config] - 10https://gerrit.wikimedia.org/r/208890 (owner: 10Legoktm) [04:03:24] (03Merged) 10jenkins-bot: Use generic phpunit job for Citoid [integration/config] - 10https://gerrit.wikimedia.org/r/208891 (owner: 10Legoktm) [04:03:26] (03Merged) 10jenkins-bot: Use generic phpunit job for CodeEditor [integration/config] - 10https://gerrit.wikimedia.org/r/208892 (owner: 10Legoktm) [04:04:57] !log deploying https://gerrit.wikimedia.org/r/208889,90,91,92 [04:05:00] Logged the message, Master [04:25:13] (03PS1) 10Legoktm: Use generic phpunit/qunit jobs for extensions with dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/208899 [04:27:00] (03CR) 10jenkins-bot: [V: 04-1] Use generic phpunit/qunit jobs for extensions with dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/208899 (owner: 10Legoktm) [04:28:13] (03PS2) 10Legoktm: Use generic phpunit/qunit jobs for extensions with dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/208899 [04:34:13] (03CR) 10Legoktm: [C: 032] Use generic phpunit/qunit jobs for extensions with dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/208899 (owner: 10Legoktm) [04:36:03] (03Merged) 10jenkins-bot: Use generic phpunit/qunit jobs for extensions with dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/208899 (owner: 10Legoktm) [04:36:50] !log deploying https://gerrit.wikimedia.org/r/208899 [04:36:53] Logged the message, Master [06:25:49] Project browsertests-VisualEditor-production-linux-firefox-sauce build #60: FAILURE in 1 hr 25 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-production-linux-firefox-sauce/60/ [06:36:20] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [07:06:23] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [07:45:27] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce build #27: FAILURE in 36 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce/27/ [08:56:54] Yippee, build fixed! [08:56:54] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #590: FIXED in 46 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/590/ [09:14:39] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:15:37] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:16:00] Project beta-scap-eqiad build #51642: FAILURE in 2 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/51642/ [09:19:33] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 46896 bytes in 0.732 second response time [09:20:29] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 47405 bytes in 0.616 second response time [09:25:31] Yippee, build fixed! [09:25:32] Project beta-scap-eqiad build #51643: FIXED in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/51643/ [09:48:51] 5Continuous-Integration-Isolation, 6operations: Disable diamond collector on contintcloud labs project - https://phabricator.wikimedia.org/T98121#1259982 (10hashar) 3NEW a:3hashar [10:09:05] 10Beta-Cluster, 6operations, 7Puppet: Trebuchet on deployment-bastion: wrong group owner - https://phabricator.wikimedia.org/T97775#1260027 (10mobrovac) >>! In T97775#1252075, @chasemp wrote: > sure, I mean all of those should be owned by trebuchet and deployment since deployment is the group for deployers.... [10:09:54] PROBLEM - Puppet failure on integration-vmbuilder-trusty is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:34:54] RECOVERY - Puppet failure on integration-vmbuilder-trusty is OK: OK: Less than 1.00% above the threshold [0.0] [10:48:37] (03PS1) 10Zfilipin: WIP Password fallback [selenium] - 10https://gerrit.wikimedia.org/r/208931 [10:48:39] (03CR) 10jenkins-bot: [V: 04-1] WIP Password fallback [selenium] - 10https://gerrit.wikimedia.org/r/208931 (owner: 10Zfilipin) [11:16:22] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-UploadWizard, 6Multimedia: UploadWizard mwext-qunit job bogus failures: "mw.fileApi isPreviewableFile FAILED: afterEach failed on isPreviewableFile: Unfinished AJAX requests: 1" - https://phabricator.wikimedia.org/T98130#1260165 (10matmarex) 3N... [11:52:50] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-UploadWizard, 6Multimedia: UploadWizard mwext-qunit job bogus failures: "mw.fileApi isPreviewableFile FAILED: afterEach failed on isPreviewableFile: Unfinished AJAX requests: 1" - https://phabricator.wikimedia.org/T98130#1260242 (10matmarex) I t... [12:00:57] PROBLEM - Puppet failure on integration-vmbuilder-trusty is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:08:42] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-UploadWizard, 6Multimedia, 5Patch-For-Review: UploadWizard mwext-qunit job bogus failures: "mw.fileApi isPreviewableFile FAILED: afterEach failed on isPreviewableFile: Unfinished AJAX requests: ... - https://phabricator.wikimedia.org/T98130#1260263 [12:20:54] RECOVERY - Puppet failure on integration-vmbuilder-trusty is OK: OK: Less than 1.00% above the threshold [0.0] [12:39:55] 10Continuous-Integration-Infrastructure, 10Wikimedia-Fundraising-CiviCRM: Disable job on CRM deployment branch - https://phabricator.wikimedia.org/T94586#1260306 (10hashar) Zuul can filter out triggered jobs using filters and filter on branches is supported. In integration/config.git zuul/layout.yaml ``` jobs... [12:48:02] Yippee, build fixed! [12:48:02] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #245: FIXED in 1 min 1 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/245/ [12:49:00] Project browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce build #503: FAILURE in 2 min 59 sec: https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce/503/ [13:08:37] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #535: FAILURE in 2 min 37 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/535/ [13:12:26] hasharAway: I'm heading out in a few minutes so won't be attending the CI triage this week. [13:16:38] hasharAway: The only thing from me this week is that it'd be cool to see Zuul or JJB upgraded (and the various issues that would be solved by that). [14:12:05] hi there [14:13:07] I'm troubleshooting Flow's browser tests. I would like to be able to run them from my local machine against http://en.wikipedia.beta.wmflabs.org, just like Jenkins is doing. [14:13:52] I think I need the "Selenium user" credentials to do that [14:14:21] Where can I find it? Or is there another (a better) way? [14:16:43] stephanebisson: isn't the username a parameter, so that you can just create your own user for that? [14:18:02] jzerebecki: yes for some tests but the moderation features require an admin. Selenium user is setup with the right permissions. [14:19:29] stephanebisson: the permission usually is handed out on request on beta [14:21:47] jzerebecki: I have created a user, who do I ask for specific permissions? [14:28:35] stephanebisson: all passwords are at office wiki [14:28:41] * zeljkof is looking for the link... [14:28:56] stephanebisson: https://office.wikimedia.org/wiki/Selenium_passwords [14:29:04] let me know if you need help [14:29:59] zeljkof: thanks! [14:32:40] * zeljkof feels the selenium force getting stronger in this channel [14:34:51] err, that was yesterday :P [14:46:15] Yippee, build fixed! [14:46:16] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #242: FIXED in 29 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/242/ [14:49:02] zeljkof: we can surely mute some of the bots [14:49:09] or at least make them less verbose [14:57:54] hashar: move all browser test notifs to the channels of teams who should own them! :) [15:03:47] 10Beta-Cluster, 6Release-Engineering, 10Continuous-Integration-Config, 10Parsoid: Parsoid patches don't update Beta Cluster automatically -- only deploy repo patches seem to update that code - https://phabricator.wikimedia.org/T92871#1260721 (10hashar) Potentially we were originally running the parsoid mas... [15:04:09] greg-g: yeah some requested that already :] [15:04:18] we should probably stop spamming the qa-alerts list as well [15:12:05] hashar: is anybody even reading that list? [15:23:20] zeljkof: no clue [15:24:59] (not me) [15:34:49] (03PS1) 10Legoktm: Thanks depends on MF, not Flow [integration/config] - 10https://gerrit.wikimedia.org/r/208975 [15:34:51] (03PS1) 10Legoktm: Use non-generic job for Flow, generic is causing Echo failures [integration/config] - 10https://gerrit.wikimedia.org/r/208976 [15:38:26] (03CR) 10Legoktm: [C: 032] Use non-generic job for Flow, generic is causing Echo failures [integration/config] - 10https://gerrit.wikimedia.org/r/208976 (owner: 10Legoktm) [15:38:30] (03CR) 10Legoktm: [C: 032] Thanks depends on MF, not Flow [integration/config] - 10https://gerrit.wikimedia.org/r/208975 (owner: 10Legoktm) [15:40:06] (03Merged) 10jenkins-bot: Thanks depends on MF, not Flow [integration/config] - 10https://gerrit.wikimedia.org/r/208975 (owner: 10Legoktm) [15:41:57] (03Merged) 10jenkins-bot: Use non-generic job for Flow, generic is causing Echo failures [integration/config] - 10https://gerrit.wikimedia.org/r/208976 (owner: 10Legoktm) [15:42:25] !log deploying https://gerrit.wikimedia.org/r/208975 & https://gerrit.wikimedia.org/r/208976 [15:42:30] Logged the message, Master [16:19:36] (03PS1) 10BryanDavis: Update statsd events [tools/scap] - 10https://gerrit.wikimedia.org/r/208987 (https://phabricator.wikimedia.org/T64667) [16:23:36] (03CR) 10BryanDavis: Update statsd events (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/208987 (https://phabricator.wikimedia.org/T64667) (owner: 10BryanDavis) [16:39:16] (03CR) 10Filippo Giunchedi: [C: 031] Update statsd events [tools/scap] - 10https://gerrit.wikimedia.org/r/208987 (https://phabricator.wikimedia.org/T64667) (owner: 10BryanDavis) [17:04:39] 6Release-Engineering, 10Wikidata, 7Composer: enable use of production deployed autoloader for extensions that is created by composer - https://phabricator.wikimedia.org/T97560#1261011 (10JanZerebecki) [17:06:15] 6Release-Engineering, 10Wikidata, 7Composer: enable use of a composer created autoloader in extensions deployed to production - https://phabricator.wikimedia.org/T97560#1261023 (10JanZerebecki) [17:12:21] greg-g: is https://phabricator.wikimedia.org/T97560 in the area of responsibility of the rel-eng team? can you comment on if the aproach is fine? [18:26:29] 10Beta-Cluster, 10Graphoid: Deploy Graphoid on Beta Cluster - https://phabricator.wikimedia.org/T97606#1261858 (10Yurik) 5Open>3Resolved a:3Yurik Yes, thanks. At what point will the betalabs be auto-updated - when I merge the graphoid repo or graphoid/deploy repo? [18:32:30] 10Beta-Cluster, 10Graphoid: Deploy Graphoid on Beta Cluster - https://phabricator.wikimedia.org/T97606#1261913 (10mobrovac) >>! In T97606#1257255, @akosiaris wrote: > @mobrovac, deployment-prep's parsoidcache would be autoupdated but https://gerrit.wikimedia.org/r/#/c/208644/ was missing. Done and parsoidcache... [18:35:39] PROBLEM - Host integration-slave-jessie-1001 is DOWN: CRITICAL - Host Unreachable (10.68.16.72) [18:49:57] RECOVERY - Host integration-slave-jessie-1001 is UP: PING OK - Packet loss = 0%, RTA = 0.75 ms [19:00:27] PROBLEM - Free space - all mounts on deployment-eventlogging02 is CRITICAL: CRITICAL: deployment-prep.deployment-eventlogging02.diskspace._var.byte_percentfree (<100.00%) [19:06:02] !log integration-slave-trusty-1015:~$ sudo -u jenkins-deploy rm -rf /mnt/jenkins-workspace/workspace/mwext-Wikibase-qunit/src/node_modules [19:06:09] Logged the message, Master [19:26:13] PROBLEM - Puppet staleness on integration-saltmaster is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [19:29:48] 10Beta-Cluster, 10Analytics-EventLogging, 10VisualEditor: Beta cluster is sending VisualEditor events to production bits.wikimedia.org/statsv - https://phabricator.wikimedia.org/T98196#1262125 (10Krinkle) 3NEW [19:36:29] 10Beta-Cluster: Can't connect to Beta Cluster database deployment-db1 or deployment-db2 (MariaDB down) - https://phabricator.wikimedia.org/T96905#1262137 (10hashar) 5Open>3Resolved The databases are up and running since the last manual intervention back in Apr 24th. So that is fixed :-) [19:39:51] 6Release-Engineering, 10MediaWiki-Vagrant, 7Documentation: Document RSpec workflow on MediaWiki-Vagrant - https://phabricator.wikimedia.org/T97464#1262152 (10hashar) a:3dduvall Seems @dduvall is going to take care of the documentation. [19:40:09] 6Release-Engineering, 10MediaWiki-Vagrant, 7Documentation: Document RSpec workflow on MediaWiki-Vagrant - https://phabricator.wikimedia.org/T97464#1262154 (10hashar) p:5Triage>3Normal [19:46:31] 6Release-Engineering, 10Wikimedia-Git-or-Gerrit, 7Documentation: Document how to tag extensions in git - https://phabricator.wikimedia.org/T94412#1262186 (10hashar) 5Open>3Resolved a:3hashar Seems the issue is solved (push tags to the ssh based remote) [19:48:19] 10Beta-Cluster, 6operations, 5Patch-For-Review, 7Puppet: Trebuchet on deployment-bastion: wrong group owner - https://phabricator.wikimedia.org/T97775#1262210 (10thcipriani) Pushed my patch up and attached to this bug. As I was reviewing this patch, I actually think it may be a better idea to have the dep... [19:48:51] 10Beta-Cluster, 6operations, 5Patch-For-Review, 7Puppet: Trebuchet on deployment-bastion: wrong group owner - https://phabricator.wikimedia.org/T97775#1262216 (10thcipriani) [19:49:24] 6Release-Engineering: Move mediawiki_selenium Gemfile metadata to .ruby-version and .ruby-gemset - https://phabricator.wikimedia.org/T75727#1262219 (10hashar) 5Open>3declined a:3hashar Per @zeljkofilipin [19:49:25] 6Release-Engineering: Remove lines from Gemfile that are used by RVM - https://phabricator.wikimedia.org/T1331#1262222 (10hashar) [19:49:38] 6Release-Engineering: Remove lines from Gemfile that are used by RVM - https://phabricator.wikimedia.org/T1331#23427 (10hashar) [19:49:39] 6Release-Engineering: Move mediawiki_api Gemfile metadata to .ruby-version and .ruby-gemset - https://phabricator.wikimedia.org/T75728#1262223 (10hashar) 5Open>3declined a:3hashar Per @zeljkofilipin [19:50:02] 6Release-Engineering: Remove lines from Gemfile that are used by RVM - https://phabricator.wikimedia.org/T1331#1262227 (10hashar) 5Open>3declined a:3hashar Per @zeljkofilipin [19:51:21] 10Beta-Cluster, 10Analytics-EventLogging, 10VisualEditor: Beta cluster is sending VisualEditor events to production bits.wikimedia.org/statsv - https://phabricator.wikimedia.org/T98196#1262230 (10greg) @ori: just pinging you because of the bits change you pushed yesterday re beta cluster, related? [19:51:52] 6Release-Engineering, 7Performance: Performance Testing Cluster - https://phabricator.wikimedia.org/T282#1262233 (10hashar) @greg I don't think we are in a position to create and maintain a performance testing cluster. I am not sure what was the original discussion back in September 2014, but potentially if on... [19:54:12] 6Release-Engineering, 7Performance: Performance Testing Cluster - https://phabricator.wikimedia.org/T282#1262238 (10greg) 5Open>3stalled Setting to Stalled, it's probably something that will come up again, but you're right, not on the plan for now. [19:54:35] 6Release-Engineering, 10REFLEX: UX Test Env Test MVP: create 1-5 test instances on WMF Labs - https://phabricator.wikimedia.org/T970#1262241 (10hashar) Since designer / UX folks are now embedded in vertical teams, I guess it is up to the teams to create their instances to demo/test things. [19:57:24] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:01:08] 5Continuous-Integration-Isolation, 6Labs, 10Labs-Infrastructure: Include Base::Standard-packages in labs images - https://phabricator.wikimedia.org/T94995#1262253 (10hashar) [20:01:27] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 47405 bytes in 0.674 second response time [20:02:46] 10Continuous-Integration-Infrastructure: [OPS] Jenkins: Package for mobile jobs (androidsdk, libdclass) missing in Trusty - https://phabricator.wikimedia.org/T70259#1262259 (10hashar) [20:03:39] 10Beta-Cluster, 10Ops-Access-Requests, 6operations: Add niedzielski release-mobile and deployment-prep project - https://phabricator.wikimedia.org/T98179#1262264 (10Krenair) [20:04:17] 10Continuous-Integration-Infrastructure: [OPS] Jenkins: Package for mobile jobs (androidsdk, libdclass) missing in Trusty - https://phabricator.wikimedia.org/T70259#1262266 (10hashar) 5Open>3declined a:3hashar I am declining it. There is no more any jobs running AndroidSDK and the analytics libs jobs have... [20:09:43] 10Beta-Cluster, 6Release-Engineering: Determine weekly triage meeting for Beta Cluster - https://phabricator.wikimedia.org/T98204#1262310 (10greg) 3NEW [20:11:34] (03PS2) 10Krinkle: Add test to verify every extension+skin has an entry in zuul/layout.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/198185 (owner: 10Legoktm) [20:11:40] 10Deployment-Systems, 6Release-Engineering: Determine weekly triage meeting for Deployment Systems - https://phabricator.wikimedia.org/T98206#1262327 (10greg) 3NEW [20:12:54] 10Browser-Tests, 6Release-Engineering: Determine weekly triage meeting for Browser Tests - https://phabricator.wikimedia.org/T98207#1262335 (10greg) 3NEW [20:13:14] (03CR) 10jenkins-bot: [V: 04-1] Add test to verify every extension+skin has an entry in zuul/layout.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/198185 (owner: 10Legoktm) [20:30:41] 10Beta-Cluster, 10Graphoid: Deploy Graphoid on Beta Cluster - https://phabricator.wikimedia.org/T97606#1262386 (10Yurik) @mobrovac, what are the steps for the manual deployment from deployment-bastion? How are they different from production? Thx! (P.S. its alive!!!) [21:41:26] 10Beta-Cluster, 10Ops-Access-Requests, 6operations: Add niedzielski release-mobile and deployment-prep project - https://phabricator.wikimedia.org/T98179#1262691 (10Dzahn) yea, but it got already separated from yet another thing: T97866, so that's good [21:53:00] 10Beta-Cluster, 10Ops-Access-Requests, 6operations: Add niedzielski release-mobile and deployment-prep project - https://phabricator.wikimedia.org/T98179#1262713 (10JohnLewis) I've added "niedzielski" to deployment-prep as a member. [21:54:29] 10Beta-Cluster, 10Ops-Access-Requests, 6operations: Add niedzielski release-mobile - https://phabricator.wikimedia.org/T98179#1262714 (10coren) [21:54:41] 10Beta-Cluster, 10Ops-Access-Requests, 6operations: Add niedzielski release-mobile - https://phabricator.wikimedia.org/T98179#1261519 (10coren) I've retitled the task accordingly. [21:55:52] 10Beta-Cluster, 10Ops-Access-Requests, 6operations: Add niedzielski release-mobile - https://phabricator.wikimedia.org/T98179#1262717 (10coren) p:5Triage>3Normal [21:57:00] 10Beta-Cluster, 10Ops-Access-Requests, 6operations: Add niedzielski releasers-mobile in production and deployment-prep in labs - https://phabricator.wikimedia.org/T98179#1262720 (10Krenair) [22:07:39] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<33.33%) [22:22:40] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [23:31:21] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:31:23] PROBLEM - Puppet staleness on deployment-urldownloader is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [23:31:43] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]