[00:13:50] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:15:39] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [00:23:49] Reedy: ^d am heading to the office [00:23:55] I probably should [00:23:58] puppet runs now [00:27:20] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:41:15] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:00:53] * ^d is home, reattached [01:04:05] <^d> I see a working beta. Go team [01:12:11] ^d: it works? [01:12:21] <^d> en.wp.beta.wmflabs.o works [01:12:25] <^d> At least homepage does [01:12:40] <^d> Oh shit. [01:12:44] <^d> clicking login blew up [01:12:47] <^d> (Cannot access the database: Can't connect to MySQL server on '10.68.16.193' (111) (10.68.16.193)) [01:12:54] ^d: ?debug=true bro :) [01:12:59] <^d> pfft. [01:17:07] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 49109 bytes in 1.211 second response time [01:18:06] Who needs a database anyway [01:20:42] <^d> Reedy: lets just use sqlite [01:20:44] <^d> on nfs [01:20:47] lol [01:21:10] on deployment-db1 [01:21:10] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Error from DataBinding 'hiera' while looking up 'apt::unattendedupgrades::ensure': Reading data from Deployment-prep failed: Errno::ECONNREFUSED: Connection refused - connect(2) (https://wikitech.wikimedia.org:443) on node i-00000220.eqiad.wmflabs [01:21:10] Warning: Not using cache on failed catalog [01:21:10] Error: Could not retrieve catalog; skipping run [01:21:11] ffs [01:21:42] <^d> dear puppet, [01:21:49] <^d> go screw yourself [01:21:50] <^d> xoxo, [01:21:51] <^d> chad [01:22:43] Yippee, build fixed! [01:22:44] Project beta-update-databases-eqiad build #7184: FIXED in 2 min 43 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/7184/ [01:22:53] oh, wikitech is dead [01:23:36] <^d> yep [01:39:38] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:41:57] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [02:59:39] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [03:01:57] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [03:04:37] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [03:18:33] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [03:18:57] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [03:29:21] 3Beta-Cluster: Beta Cluster is down due to restart of WMF Labs servers - https://phabricator.wikimedia.org/T87678#997484 (10greg) Things seem to be working again, no? Thanks @dzahn, @chad, @reedy, etc. Fun day with yet another exploit announced during a group meeting. [03:38:12] PROBLEM - Host deployment-sca-cache01 is DOWN: PING CRITICAL - Packet loss = 100% [03:51:43] Yippee, build fixed! [03:51:44] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #443: FIXED in 11 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/443/ [04:11:10] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #263: FAILURE in 25 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/263/ [04:13:00] 3Beta-Cluster: Beta Cluster is down due to restart of WMF Labs servers - https://phabricator.wikimedia.org/T87678#997523 (10Ryasmeen) Its not working yet though, I am still getting a different kind of error: Error loading data from server:0: parsoidserver-http:HTTP:0, would you like to retry when I try to load V... [04:20:16] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #406: FAILURE in 2 min 52 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/406/ [04:22:39] (03PS17) 10Addshore: Create phpcs standard for MW core compatibility [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/153399 [04:28:07] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #471: FAILURE in 26 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/471/ [04:32:38] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #482: FAILURE in 4 min 29 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/482/ [05:13:34] Project browsertests-Flow-test2.wikipedia.org-linux-firefox-sauce build #419: STILL FAILING in 40 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-firefox-sauce/419/ [05:18:27] Yippee, build fixed! [05:18:27] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #151: FIXED in 45 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/151/ [05:19:22] Yippee, build fixed! [05:19:23] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #305: FIXED in 55 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/305/ [05:27:57] (03CR) 10Krinkle: [C: 031] "Untested, but verified that AntiSpoof can and should be installed alongside AbuseFilter." [integration/config] - 10https://gerrit.wikimedia.org/r/185634 (https://phabricator.wikimedia.org/T84859) (owner: 1001tonythomas) [05:32:19] (03CR) 10Krinkle: [C: 04-1] Create phpcs standard for MW core compatibility (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/153399 (owner: 10Addshore) [05:33:46] (03CR) 10Krinkle: Create phpcs standard for MW core compatibility (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/153399 (owner: 10Addshore) [05:35:01] Project browsertests-VisualEditor-test2.wikipedia.org-windows_8-internet_explorer-sauce build #275: STILL FAILING in 38 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-windows_8-internet_explorer-sauce/275/ [05:35:50] Project browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce build #421: STILL FAILING in 32 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce/421/ [05:37:47] Yippee, build fixed! [05:37:48] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #377: FIXED in 1 min 3 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/377/ [05:49:48] Yippee, build fixed! [05:49:49] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #427: FIXED in 13 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/427/ [06:04:36] Project browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce build #451: STILL FAILING in 51 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce/451/ [06:25:26] RECOVERY - Host deployment-elastic05 is UP: PING OK - Packet loss = 0%, RTA = 182.47 ms [06:34:32] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 28.57% of data above the critical threshold [0.0] [06:36:20] PROBLEM - Host deployment-elastic05 is DOWN: CRITICAL - Host Unreachable (10.68.16.38) [06:39:40] RECOVERY - Host deployment-mx is UP: PING OK - Packet loss = 0%, RTA = 0.94 ms [06:40:32] RECOVERY - Host deployment-restbase03 is UP: PING OK - Packet loss = 0%, RTA = 1.04 ms [06:40:34] RECOVERY - Host deployment-mediawiki03 is UP: PING OK - Packet loss = 0%, RTA = 0.85 ms [06:40:56] RECOVERY - Host deployment-pdf02 is UP: PING OK - Packet loss = 0%, RTA = 0.63 ms [06:41:20] RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 0.79 ms [06:41:30] RECOVERY - Host deployment-elastic05 is UP: PING OK - Packet loss = 0%, RTA = 0.63 ms [06:46:33] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [06:46:33] Yippee, build fixed! [06:46:34] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #1381: FIXED in 32 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/1381/ [06:52:03] RECOVERY - App Server bits response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 3895 bytes in 0.238 second response time [06:57:28] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 49119 bytes in 1.260 second response time [07:04:18] 3Beta-Cluster: Beta Cluster is down due to restart of WMF Labs servers - https://phabricator.wikimedia.org/T87678#997856 (10yuvipanda) @Ryasmeen the parsoid cache machine was in virt1009 which was down. Is back up now, can you check / verify if it still works? [07:04:32] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [07:07:25] 3Phabricator: Fatal upon maniphest search in a component - https://phabricator.wikimedia.org/T87739#997861 (10Nemo_bis) 3NEW [07:12:12] 3Phabricator, operations: merge tickets in project "ops-core" into project "operations" - https://phabricator.wikimedia.org/T87291#988096 (10Dzahn) [08:31:31] (03CR) 10Adrian Lang: "Currently, npm only runs jscs. I can add jshint to npm, though, and then put up another change removing jslint, if you prefer that." [integration/config] - 10https://gerrit.wikimedia.org/r/184592 (owner: 10Adrian Lang) [08:57:45] (03PS20) 10Adrian Lang: Fix WikibaseJavaScriptApi tests [integration/config] - 10https://gerrit.wikimedia.org/r/180418 (https://phabricator.wikimedia.org/T86176) [09:07:18] PROBLEM - Puppet staleness on deployment-eventlogging02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [43200.0] [10:51:46] PROBLEM - Free space - all mounts on deployment-elastic07 is CRITICAL: CRITICAL: deployment-prep.deployment-elastic07.diskspace._var_log.byte_percentfree.value (<44.44%) [11:42:52] PROBLEM - Free space - all mounts on deployment-elastic05 is CRITICAL: CRITICAL: deployment-prep.deployment-elastic05.diskspace._var_log.byte_percentfree.value (<44.44%) [12:37:36] PROBLEM - SSH on deployment-lucid-salt is CRITICAL: Connection refused [12:57:33] 3MediaWiki-extensions-MathSearch, Beta-Cluster: Broken submodule - https://phabricator.wikimedia.org/T87643#998126 (10Physikerwelt) Any updates here? by the way I fixed the gitsubmodules problem in my git repo. And at least the demo of the ui for math input works ok http://math-min.wmflabs.org/w/extensions/MathS... [13:08:24] 3Phabricator: MediaWiki user page url should be once encoded. - https://phabricator.wikimedia.org/T87758#998149 (10devunt) 3NEW [13:09:23] 3Phabricator: MediaWiki user page url should be once encoded. - https://phabricator.wikimedia.org/T87758#998157 (10devunt) [13:11:22] 3Phabricator: MediaWiki user page url should be once encoded. - https://phabricator.wikimedia.org/T87758#998161 (10valhallasw) Where is this? The 'MediaWiki User' link at https://phabricator.wikimedia.org/p/devunt/ just shows https://www.mediawiki.org/w/index.php?title=User:%2Adevunt . [13:13:19] 3Phabricator: MediaWiki user page url should be once encoded. - https://phabricator.wikimedia.org/T87758#998168 (10devunt) >>! In T87758#998161, @valhallasw wrote: > Where is this? The 'MediaWiki User' link at https://phabricator.wikimedia.org/p/devunt/ just shows https://www.mediawiki.org/w/index.php?title=User... [13:42:33] PROBLEM - App Server bits response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:42:51] PROBLEM - App Server bits response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:43:20] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:43:24] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:45] PROBLEM - Free space - all mounts on deployment-elastic06 is CRITICAL: CRITICAL: deployment-prep.deployment-elastic06.diskspace._var_log.byte_percentfree.value (<11.11%) [13:57:59] 3Phabricator: Decide whether project reporting should be moved to Phabricator as well - https://phabricator.wikimedia.org/T24#998216 (10Kelson) I can only speak for myself. This kind of report needs anyway rephrasing/polishing work and this is not something I like to do nor automated. Being near or not of the "r... [13:58:04] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:03] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 49298 bytes in 8.785 second response time [14:33:23] PROBLEM - HHVM Queue Size on deployment-mediawiki01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [80.0] [15:17:24] RECOVERY - App Server bits response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 3895 bytes in 0.003 second response time [15:18:10] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 49100 bytes in 0.553 second response time [15:18:14] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 49108 bytes in 0.562 second response time [15:23:24] RECOVERY - HHVM Queue Size on deployment-mediawiki01 is OK: OK: Less than 30.00% above the threshold [10.0] [16:50:37] Party people, there appears to be an issue on deployment-upload an betalabs [16:50:50] Not sure why or how to fix, James_F says he doesn't get any images on betawiki [16:51:05] I appear to agree [16:51:11] :-) [16:52:26] I guess if I poked nginx with a big and pointy enough stick, it might start doing its bloody job [16:52:48] !log restarting nginx on deployment-upload so beta images might work again [16:52:54] Logged the message, Master [16:53:07] No dice, James_F [16:55:24] James_F: Gotta meeting, we can look at it again in an hour or so [16:55:31] marktraceur: Ah well. [16:55:47] It's beta, it breaks sometimes [16:55:56] #shithappens [17:03:38] YuviPanda: You might want to know about this. [17:04:30] Reedy: ^d twentyafterfour ^ [17:05:29] * ^d hides [17:05:48] ^d: How's that working out for you. [17:12:23] <^d> James_F: I need irc camo, obviously. [17:12:45] ^d: /nick ObliviousBystander [17:15:54] g'morn [17:20:09] <^lurker> James_F: How's that? [17:22:10] 11:55 < marktrace> It's beta, it breaks sometimes [17:22:11] 11:55 < marktrace> #shithappens [17:22:23] just for the record, it hasn't lately, not until this "restart the world" yesterday [17:30:59] 3Triagers, Phabricator, operations, Project-Creators: Broaden the group of users that can create projects in Phabricator - https://phabricator.wikimedia.org/T706#998482 (10RobLa-WMF) Please add @Gilles to Project-Creators. He is managing sprints for the Multimedia team. Thanks! [17:31:08] <^lurker> `mv home office` [17:36:51] 3RESTBase, Continuous-Integration: Publish RESTBase documentation on doc.wikimedia.org - https://phabricator.wikimedia.org/T87702#998503 (10Jdouglas) What's the process for pushing documentation artifacts, once they're ready for publication? [17:38:36] ^lurker: Totally not obvious. :-) [17:38:38] greg-g: Indeed. [17:44:46] 3Triagers, Phabricator, operations, Project-Creators: Broaden the group of users that can create projects in Phabricator - https://phabricator.wikimedia.org/T706#998551 (10Legoktm) [17:44:58] 3Triagers, Phabricator, operations, Project-Creators: Broaden the group of users that can create projects in Phabricator - https://phabricator.wikimedia.org/T706#998557 (10Qgil) {{done}} [17:50:30] 3Release-Engineering: Evaluate and decide on a distribution strategy targeted at VMs - https://phabricator.wikimedia.org/T87774#998575 (10GWicke) [17:51:18] 3operations, Beta-Cluster: Minimize differences between beta and production (Tracking) - https://phabricator.wikimedia.org/T87220#998581 (10Dzahn) p:5Triage>3Normal [17:51:51] 3Phabricator, operations: Add @emailbot to #wmf-nda - https://phabricator.wikimedia.org/T87611#998583 (10Dzahn) p:5Triage>3Normal [17:54:08] 3Release-Engineering: Evaluate and decide on a distribution strategy targeted at VMs - https://phabricator.wikimedia.org/T87774#998596 (10GWicke) [17:59:38] 3Phabricator, operations: Add @emailbot to #wmf-nda - https://phabricator.wikimedia.org/T87611#998617 (10csteipp) a:5csteipp>3None >>! In T87611#995412, @RobH wrote: > I've assigned this to Chris for his commentary. > > Chris: Please provide feedback and then feel free to unassign yourself as owner (or assi... [18:04:45] 3Release-Engineering: Evaluate and decide on a distribution strategy targeted at VMs - https://phabricator.wikimedia.org/T87774#998636 (10GWicke) [18:08:36] 3Phabricator, operations: Add @emailbot to #wmf-nda - https://phabricator.wikimedia.org/T87611#998655 (10Dzahn) a:3RobH [18:09:02] 3Code-Review, Multimedia: Add Multimedia team members as reviewers to multimedia-related Gerrit patches - https://phabricator.wikimedia.org/T87776#998660 (10Tgr) 3NEW [18:09:14] 3Release-Engineering: Evaluate and decide on a distribution strategy targeted at VMs - https://phabricator.wikimedia.org/T87774#998668 (10GWicke) [18:12:30] 3operations, Beta-Cluster: Set up an alert for unmerged changes in deployment-prep - https://phabricator.wikimedia.org/T87616#998675 (10Dzahn) production has a check for this, should be among this: modules/monitoring/manifests/icinga/git_merge.pp: description => "Unmerged changes on repository ${title}"... [18:36:32] 3Beta-Cluster: Beta Cluster is down due to restart of WMF Labs servers - https://phabricator.wikimedia.org/T87678#998756 (10Ryasmeen) @Yuvipanda: Yup, it is loading now,Thanks so much! [18:36:46] 3Beta-Cluster: Beta Cluster is down due to restart of WMF Labs servers - https://phabricator.wikimedia.org/T87678#998759 (10Ryasmeen) 5Open>3Resolved [18:42:51] 3Architecture, Release-Engineering: Evaluate and decide on a distribution strategy targeted at VMs - https://phabricator.wikimedia.org/T87774#998772 (10GWicke) [18:43:34] 3Services, Parsoid, Architecture, Release-Engineering: Evaluate and decide on a distribution strategy targeted at VMs - https://phabricator.wikimedia.org/T87774#998566 (10GWicke) [18:44:01] Yippee, build fixed! [18:44:01] Project beta-scap-eqiad build #39326: FIXED in 22 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39326/ [18:44:47] 3Services, Parsoid, Architecture, Release-Engineering: Evaluate and decide on a distribution strategy targeted at VMs - https://phabricator.wikimedia.org/T87774#998566 (10GWicke) [18:53:33] Yippee, build fixed! [18:53:33] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #506: FIXED in 42 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/506/ [18:58:33] RECOVERY - Content Translation Server on deployment-cxserver03 is OK: HTTP OK: HTTP/1.1 200 OK - 1103 bytes in 0.033 second response time [18:59:58] 3Beta-Cluster: Broken image is appearing in the media search in Betalabs - https://phabricator.wikimedia.org/T87785#998838 (10Ryasmeen) 3NEW [19:03:24] ryasmeen: "Beta Cluster" :) [19:10:07] (03Abandoned) 10EBernhardson: Update jenkins phpunit to 4.1.4 [integration/phpunit] - 10https://gerrit.wikimedia.org/r/151252 (owner: 10EBernhardson) [19:15:20] greg-g: I think the images being broken is probably the same issue that marktraceur reported earlier on [19:15:32] someone who isn’t me should look into that :) [19:16:40] YuviPanda: yeppers [19:16:57] Reedy: wheree aaaareeee you [19:17:12] Reedy: do you have a moment to look into broken deployment-upload? [19:17:15] You might have mistaken me for greg-g [19:17:16] ;) [19:17:31] Yippee, build fixed! [19:17:32] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #264: FIXED in 33 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/264/ [19:27:33] Yippee, build fixed! [19:27:33] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #472: FIXED in 33 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/472/ [19:31:51] Yippee, build fixed! [19:31:52] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #280: FIXED in 44 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/280/ [19:34:17] might get a flood of fixed builds today :) [19:35:50] 3Code-Review, Multimedia: Add Multimedia team members as reviewers to multimedia-related Gerrit patches - https://phabricator.wikimedia.org/T87776#999043 (10Aklapper) Is this some technical request or a social one? For the latter and our current infrastructure: https://gerrit.wikimedia.org/r/#/settings/projects... [19:37:40] 3Phabricator: Fatal upon maniphest search in a component - https://phabricator.wikimedia.org/T87739#999048 (10Aklapper) p:5Triage>3Low It loads for me (tried twice) but took ~22 seconds. [19:38:16] 3Phabricator: Fatal error (30 seconds timeout) upon certain maniphest search in a component - https://phabricator.wikimedia.org/T87739#999053 (10Aklapper) [19:40:58] 3Continuous-Integration: Have unit tests of all wmf deployed extensions pass when installed together, in both PHP-Zend and HHVM (tracking) - https://phabricator.wikimedia.org/T69216#999097 (10Quiddity) [19:44:59] 3Code-Review, Multimedia: Add Multimedia team members as reviewers to multimedia-related Gerrit patches - https://phabricator.wikimedia.org/T87776#999138 (10Tgr) Mostly a reminder for myself to figure out how Reviewer-bot works / where the code is stored. Gerrit does not allow file name filtering AFAIK (T63463). [19:52:36] 3Phabricator: Fix search in Wikimedia Phabricator - https://phabricator.wikimedia.org/T75854#999165 (10Aklapper) @Chad: Does anybody have capacity to investigate and fix this in the next one or two weeks? Currently it is impossible to search for words at all (e.g. have to fall back to using old-bugzilla for sea... [19:57:27] 3Code-Review, Multimedia: Add Multimedia team members as reviewers to multimedia-related Gerrit patches - https://phabricator.wikimedia.org/T87776#999181 (10valhallasw) * Configuration is as https://www.mediawiki.org/wiki/Git/Reviewers * Source code & bug tracker at https://github.com/valhallasw/gerrit-reviewer... [20:00:44] 3Services, Parsoid, Architecture, Release-Engineering: Evaluate and decide on a distribution strategy targeted at VMs - https://phabricator.wikimedia.org/T87774#999203 (10Worden.lee) Request: make it straightforward for extension developers to customize it and provide it to users with special extensions and othe... [20:04:33] 3Phabricator: MediaWiki user page url should be once encoded. - https://phabricator.wikimedia.org/T87758#999207 (10Aklapper) p:5Normal>3Volunteer? Confirming for https://phabricator.wikimedia.org/settings/panel/external/ [20:05:19] 3Phabricator: MediaWiki user page url under "External accounts" settings should be once encoded - https://phabricator.wikimedia.org/T87758#999209 (10Aklapper) [20:07:17] 3Phabricator: bzimport comment assignment regressions - https://phabricator.wikimedia.org/T75761#999223 (10Aklapper) p:5Normal>3Low [20:08:26] Project browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce build #437: FAILURE in 40 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce/437/ [20:14:01] so deployment-cache-upload02 seems to not be running the web service in the way I would expect [20:15:00] 3Services, Parsoid, Architecture, Release-Engineering: Evaluate and decide on a distribution strategy targeted at VMs - https://phabricator.wikimedia.org/T87774#999238 (10GWicke) @Worden.lee: Do you think we could get away with a conf.d style solution where each optional extension can drop in a config fragment?... [20:15:05] yeah, known issue [20:19:36] Krenair: how about now? [20:19:40] marktraceur: upload should be fixed [20:20:09] root@deployment-cache-upload02:/home/reedy# service varnish status [20:20:09] * varnishd is running [20:20:12] root@deployment-cache-upload02:/home/reedy# service varnish-frontend status [20:20:12] * varnishd-frontend is running [20:20:38] greg-g: ^ [20:20:56] what'd you do? [20:21:01] start varnish-frontend? [20:21:16] No [20:21:19] Fixed the bad permissions [20:21:20] root@deployment-cache-upload02:/home/reedy# ls -l /var/lib/varnish/ [20:21:20] drwxr-xr-x 2 root root 100 Jan 27 23:52 deployment-cache-upload02 [20:21:20] drwx------ 2 root root 40 Jan 28 20:16 frontend [20:21:23] Then ran puppet [20:21:32] chmod +rx /var/lib/varnish/* [20:37:11] 3Services, Parsoid, Architecture, Release-Engineering: Evaluate and decide on a distribution strategy targeted at VMs - https://phabricator.wikimedia.org/T87774#999281 (10brion) For extensions it may be worth looking at how extensions are installed via roles in MediaWiki-Vagrant -- a lot of them come with a drop... [20:43:11] 3Services, Parsoid, Architecture, Release-Engineering: Evaluate and decide on a distribution strategy targeted at VMs - https://phabricator.wikimedia.org/T87774#999285 (10Worden.lee) @GWicke I'm not sure what the conf.d style would and wouldn't allow. I can see wanting to run some bash commands while setting up... [20:48:32] 3Thanks, Mobile-Web, Continuous-Integration: Thanks is broken again (Mobile Thanks needs qunit tests) - https://phabricator.wikimedia.org/T86687#999289 (10Jdlrobson) Patch has update. [21:00:35] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #428: FAILURE in 15 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/428/ [21:05:32] 3Phabricator: Fix search in Wikimedia Phabricator - https://phabricator.wikimedia.org/T75854#999306 (10Chad) It's just a matter of backing out [[ https://secure.phabricator.com/D11011 | upstream D11011 ]]. We'll lose the fix for T75743 but I think that's better than the status quo. [21:18:39] Reedy: I would test it, but T-Mobile is too busy BEING AN ASSHOLE. [21:19:10] Oh, there we go. [21:19:11] Thanks [21:19:22] James_F: Upload is fixed on ze beta [21:19:30] marktraceur: Yeah. Thanks. [21:21:59] (03PS3) 10Krinkle: De-duplicate LOG_DIR logic [integration/jenkins] - 10https://gerrit.wikimedia.org/r/186976 [21:31:08] 3Beta-Cluster: Broken image is appearing in the media search in Beta Cluster - https://phabricator.wikimedia.org/T87785#999408 (10greg) [21:32:02] 3Phabricator, WMF-Legal, Legalpad: remove legalpad.wm.org - https://phabricator.wikimedia.org/T87688#999411 (10Aklapper) There is #wmf-legal with three members in this instance and I wonder what their plans are, and especially what they do NOT plan to do. [21:33:53] 3Beta-Cluster: Broken image is appearing in the media search in Beta Cluster - https://phabricator.wikimedia.org/T87785#999425 (10greg) Things should be fixed with the upload server. Can you verify Rummana? ``` 15:19 <+ Reedy> marktraceur: upload should be fixed 15:20 <+ Reedy> root@deployment-cache-uploa... [21:35:30] 3Beta-Cluster: Broken image is appearing in the media search in Beta Cluster - https://phabricator.wikimedia.org/T87785#999438 (10greg) (This was related to/caused by the great restart of MediaWiki Dev Summit 2015, see other fallout in T87678) [21:40:54] (03PS4) 10Krinkle: De-duplicate LOG_DIR logic [integration/jenkins] - 10https://gerrit.wikimedia.org/r/186976 [21:42:52] (03CR) 10Krinkle: [C: 032] De-duplicate LOG_DIR logic [integration/jenkins] - 10https://gerrit.wikimedia.org/r/186976 (owner: 10Krinkle) [21:43:24] (03Merged) 10jenkins-bot: De-duplicate LOG_DIR logic [integration/jenkins] - 10https://gerrit.wikimedia.org/r/186976 (owner: 10Krinkle) [21:59:50] (03PS1) 10Brion VIBBER: Add testing dep for CodeEditor (needs WikiEditor) [integration/config] - 10https://gerrit.wikimedia.org/r/187236 (https://phabricator.wikimedia.org/T87806) [22:00:59] (03CR) 10jenkins-bot: [V: 04-1] Add testing dep for CodeEditor (needs WikiEditor) [integration/config] - 10https://gerrit.wikimedia.org/r/187236 (https://phabricator.wikimedia.org/T87806) (owner: 10Brion VIBBER) [22:07:57] wtf is up with Jenkins [22:08:42] marktraceur: James_F was after you in person [22:11:05] Well...I'm at the airport [22:11:10] So it might not work very well [22:11:26] marktraceur: waaaaat. you left without a hug [22:12:00] YuviPanda: I was wandering around the 3rd floor but everyone was busy or not there [22:12:58] marktraceur: aww man [22:14:12] 3Beta-Cluster: Setup multiversion on Beta Cluster for nightly build browser testing support - https://phabricator.wikimedia.org/T67127#999592 (10greg) [22:14:32] PROBLEM - Content Translation Server on deployment-cxserver03 is CRITICAL: Connection refused [22:21:08] marktraceur: I did tell him that [22:27:55] having an odd jenkins error, the qunit job is failing with a permission denied error at : https://integration.wikimedia.org/ci/job/mwext-Flow-qunit/3924/consoleFull [22:28:16] 22:19:01 /srv/deployment/integration/slave-scripts/bin/mw-install-sqlite.sh: line 3: /srv/deployment/integration/slave-scripts/bin/mw-set-env.sh: Permission denied [22:28:56] (03CR) 10Paladox: [C: 031] Add testing dep for CodeEditor (needs WikiEditor) [integration/config] - 10https://gerrit.wikimedia.org/r/187236 (https://phabricator.wikimedia.org/T87806) (owner: 10Brion VIBBER) [22:29:17] it seems unlikely though, because line 3 should just be sourcing the mw-set-env script [22:30:23] ebernhardson: most of my team is traveling :/ [22:33:35] 3Phabricator, Community-Engagement: Experiment with a Volunteer team tag - https://phabricator.wikimedia.org/T87808#999637 (10TheDJ) 3NEW [22:40:34] a bunch of tests are failing with that now [22:41:15] muther [22:41:48] Yeah. :-( [22:42:25] greg-g: Krinkle is looking at it now. [22:43:14] !log /srv/deployment/integration/slave-scripts got corrupted by puppet on labs slaves. No longer has the appropriate permission flags. [22:43:19] Logged the message, Master [22:52:02] 3Quality-Assurance: Update QA/testing documentation - https://phabricator.wikimedia.org/T59841#999744 (10Spage) 5Open>3Resolved a:3Spage I'm declaring victory on this. The pages are all under https://www.mediawiki.org/wiki/Quality_Assurance , you get a reasonable set of pages searching for "New contributor... [22:53:47] !log rm -rf integration-slave1007 rm -rf /mnt/jenkins-workspace/workspace/mwext-DonationInterface-np* [22:53:50] Logged the message, Master [22:54:34] 3Phabricator, WMF-Legal, Legalpad: remove legalpad.wm.org - https://phabricator.wikimedia.org/T87688#999759 (10Qgil) The legal team plans to use https://phabricator.wikimedia.org/legalpad/ only. https://legalpad.wikimedia.org/ can be safely removed. [22:59:53] Is there a Phabricator ticket for https://integration.wikimedia.org/ci/view/Beta/job/beta-code-update-eqiad/ failing for the past 4 days? [23:01:35] Unable to checkout '54fc46411406dc3c17d8fff2ac844ebe8be637fb' in submodule path 'MathSearch' [23:01:38] no [23:08:56] 3Continuous-Integration: Set up salt for integration slaves in labs - https://phabricator.wikimedia.org/T87819#999799 (10Krinkle) 3NEW [23:16:36] 3Code-Review, Multimedia: Add Multimedia team members as reviewers to multimedia-related Gerrit patches - https://phabricator.wikimedia.org/T87776#999828 (10Tgr) From T86318, the repos we should watch are: * `mediawiki/extensions/CommonsMetadata` * `mediawiki/extensions/GlobalUsage` * `mediawiki/extensions/GWToo... [23:17:45] 3Code-Review, Multimedia: Add Multimedia team members as reviewers to multimedia-related Gerrit patches - https://phabricator.wikimedia.org/T87776#999832 (10Tgr) The multimedia-related core files are * `mediawiki/core` ** `img_auth.php`, `thumb.php`, `thumb_handler.php` ** `images/*` ** `includes/MimeMagic.php`,... [23:19:21] greg-g: OK, will make one. [23:22:07] 3MediaWiki-extensions-MathSearch, Release-Engineering: beta-code-update-eqiad has been failing since 24 January - https://phabricator.wikimedia.org/T87820#999860 (10Jdforrester-WMF) 3NEW [23:22:13] {{done}} [23:25:40] greg-g: James_F: It should be fixed for now [23:26:36] Krinkle: Thanks! [23:27:04] 3Beta-Cluster: Broken image is appearing in the media search in Beta Cluster - https://phabricator.wikimedia.org/T87785#999876 (10Ryasmeen) Yeah , the image search results are appearing properly now. [23:27:14] 3Beta-Cluster: Broken image is appearing in the media search in Beta Cluster - https://phabricator.wikimedia.org/T87785#999877 (10Ryasmeen) 5Open>3Resolved [23:32:23] 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#999884 (10yuvipanda) I'll note that this thing is finally going to be fixed shortly by T87003 [23:32:29] 3Beta-Cluster, Release-Engineering: deployment-prep mobile sites are down - https://phabricator.wikimedia.org/T87821#999886 (10Krenair) 3NEW [23:32:38] greg-g, by the way, I was wondering if I could have admin on deployment-prep? [23:33:32] I asked YuviPanda and he said ask you [23:37:28] Krenair: "Successfully added Alex Monk to projectadmin. " [23:39:55] 3Beta-Cluster: deployment-mx is its own puppetmaster - https://phabricator.wikimedia.org/T86575#999948 (10greg) p:5Triage>3Normal [23:40:14] 3Beta-Cluster, Release-Engineering: deployment-prep mobile sites are down - https://phabricator.wikimedia.org/T87821#999956 (10greg) p:5Triage>3High [23:40:22] 3operations, Beta-Cluster: Move scap puppet code into a module - https://phabricator.wikimedia.org/T87221#999959 (10greg) p:5Triage>3Normal [23:40:28] 3operations, Beta-Cluster: Set up an alert for unmerged changes in deployment-prep - https://phabricator.wikimedia.org/T87616#999963 (10greg) p:5Triage>3Normal [23:43:21] thanks [23:44:43] 3Beta-Cluster: Account creation throttling too restrictive on Beta Labs - https://phabricator.wikimedia.org/T87704#999991 (10greg) In this case I would like to increase this number to the lowest useful spot. Could you describe the use case a bit more? I can think of a couple: * one or a few testers creating ne... [23:46:11] 3Beta-Cluster: deployment-mx is its own puppetmaster - https://phabricator.wikimedia.org/T86575#999999 (10greg) >>! In T86575#999914, @gerritbot wrote: > Change 186891 merged by Yuvipanda: > Make standard class's exim including behavior configurable > > [[https://gerrit.wikimedia.org/r/186891]] is that all tha... [23:47:04] 3Beta-Cluster: deployment-mx is its own puppetmaster - https://phabricator.wikimedia.org/T86575#1000006 (10yuvipanda) That didn't actually work yet :) Me and Joe are looking into it. After that I'll need to set the param in hiera for that instance and switch it to deployment-salt, and then hope that it doesn't f... [23:48:05] 3operations, Beta-Cluster: Move deployment-prep hiera data values into ops/puppet.git repo - https://phabricator.wikimedia.org/T87223#1000016 (10greg) p:5Triage>3Normal [23:55:30] Krinkle: I was thinking about it and I kind of fear that when puppet runs from cron instead of sudo in your shell that the umask will go back to messing up the permissions. I'll poke at a patch for git::clone that sets the umask on the various exec commands. [23:56:51] Reedy, shouldn't it be possible to sudo if you are in projectadmin? [23:57:03] yeah [23:57:08] it might not have propogated [23:57:41] I see I'm not listed under admins on the nova resource page [23:58:15] It's possible to set a sudo policy that is more restrictive too I think [23:58:18] Krenair: which one? [23:58:23] which page, that is [23:58:25] https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep [23:59:03] Krenair: Reedy bd808 greg-g you need to be added to an NDA group I think on deployment-prep [23:59:16] sigh [23:59:24] yeah, sounds right [23:59:34] ah, I added him here (select deployment-pre): https://wikitech.wikimedia.org/wiki/Special:NovaProject [23:59:38] +p [23:59:52] where's the NDA group? [23:59:55] stupid privacy [23:59:58] bd808: Krenair but of course, if you are projectadmin you can add yourself to the NDA group