[00:19:28] 3VisualEditor, Beta-Cluster: Search on the Beta Cluster is broken, and thus VE no longer shows suggestion list for Templates upon typing into Add template selector - https://phabricator.wikimedia.org/T89007#1029601 (10Krenair) a:3Chad [00:22:25] 3VisualEditor-MediaWiki, VisualEditor, Beta-Cluster: On Beta Cluster, switching from VisualEditor to edit source mode intermittently loads the wikitext editor without any CSS - https://phabricator.wikimedia.org/T86624#1029605 (10Ryasmeen) This happened consistently for me for last few days, when I save my first... [00:25:40] 3Release-Engineering, MediaWiki-Core-Team, Wikimedia-Logstash: Log php fatals with full backtraces again (fatal.log on fluorine) - https://phabricator.wikimedia.org/T89169#1029608 (10bd808) ``` [17:15] < MaxSem> bd808, MaxSem: we'll raise a normal php fatal for the oom too; so you should get one a... [00:53:24] 3Phabricator: "Security" values no longer updates visibility/editability policies when changed on existing bugs? - https://phabricator.wikimedia.org/T88681#1029652 (10mmodell) p:5Unbreak!>3Normal [01:14:00] 3Phabricator: Fatal error (30 seconds timeout) for some maniphest search queries - https://phabricator.wikimedia.org/T87739#1029677 (10mmodell) Here's my analysis of what's happening with this particular search query: 1. Because of some sillyness in phabricator's elasticsearch indexing logic, almost every task... [01:15:52] 3Phabricator: Fatal error (30 seconds timeout) for some maniphest search queries - https://phabricator.wikimedia.org/T87739#1029678 (10mmodell) a:5mmodell>3None [01:37:26] 3Phabricator: Fatal error (30 seconds timeout) for some maniphest search queries - https://phabricator.wikimedia.org/T87739#1029706 (10chasemp) Thanks for the input man. I think this may be another indicator that without a concerted effort on our behalf to support ES we are better off with the stock mysql search. [01:43:29] 3Phabricator, Phabricator.org: Fatal error (30 seconds timeout) for some maniphest search queries - https://phabricator.wikimedia.org/T87739#1029708 (10Tgr) Searching for #phabricator OR #phabricator.org, it returns 7 results in 500ms. If I add "workboard" to the freetext field, it takes 20s to return the same 7... [01:43:57] 3Phabricator: Fatal error (30 seconds timeout) for some maniphest search queries - https://phabricator.wikimedia.org/T87739#1029710 (10Tgr) [01:45:52] 3Phabricator: Fatal error (30 seconds timeout) for some maniphest search queries - https://phabricator.wikimedia.org/T87739#997861 (10Tgr) Also, is there a theoretical problem with doing pagination and sorting in ES, is that just something that's not implmeneted in the ES adapter in Phabricator? It sounds exactl... [01:53:55] 3Release-Engineering, Engineering-Community, Wikibugs: Only use -devtools irc channel for phab-related ticket announcements - https://phabricator.wikimedia.org/T89153#1029727 (10Aklapper) [01:54:03] 3Release-Engineering, Engineering-Community, Wikibugs: Only use -devtools irc channel for phab-related ticket announcements - https://phabricator.wikimedia.org/T89153#1028550 (10Aklapper) sounds good to me [01:59:08] 3Phabricator: "Security" values no longer updates visibility/editability policies when changed on existing bugs? - https://phabricator.wikimedia.org/T88681#1029747 (10chasemp) Are we saying that there was no hard set to public/all users on setting the drop down to none? That AFAIK never happened at any point --... [01:59:42] 3Release-Engineering, Engineering-Community, Wikibugs: Only use -devtools irc channel for phab-related ticket announcements - https://phabricator.wikimedia.org/T89153#1029759 (10Legoktm) Why wasn't -devtools merged into -releng? > Now: The signal/noise ratio is too low in -releng and I think we can decide to ke... [03:10:57] PROBLEM - Content Translation Server on deployment-cxserver03 is CRITICAL: Connection refused [03:15:55] RECOVERY - Content Translation Server on deployment-cxserver03 is OK: HTTP OK: HTTP/1.1 200 OK - 1103 bytes in 0.021 second response time [03:48:52] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #533: FAILURE in 37 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/533/ [03:54:05] is anyone else having issues with gerrit giving authentication failures? [03:54:30] during a git pull (via ssh) [03:55:28] 3Release-Engineering, Engineering-Community, Wikibugs: Only use -devtools irc channel for phab-related ticket announcements - https://phabricator.wikimedia.org/T89153#1029868 (10greg) >>! In T89153#1029759, @Legoktm wrote: > Why wasn't -devtools merged into -releng? Because it was good to have a dedicated chann... [03:58:53] uhhhh [04:01:54] "Received disconnect from 208.80.154.81: 2: Too may authentication failures" [04:02:05] "Too may" [04:02:14] error message typo, which is surprising [04:02:57] * Fiona looks. [04:03:17] java sshd? http://mail-archives.apache.org/mod_mbox/mina-dev/201401.mbox/%3CJIRA.12691500.1390864543787.22428.1390864599697@arcas%3E [04:04:01] I'm trying with the wikibugs2 repo - https://gerrit.wikimedia.org/r/#/admin/projects/labs/tools/wikibugs2 [04:04:36] I'm updating mediawiki-core without issue. [04:05:07] Funny enough, I have that repo too. [04:05:20] "time git pull --rebase --prune" worked just fine. [04:05:37] url = ssh://mzmcbride@gerrit.wikimedia.org:29418/labs/tools/wikibugs2 [04:05:47] Sounds like maybe your SSH connection is wonky? [04:06:03] must be, same thing happening with other repos (well, at least opspuppet) [04:06:47] (I don't know how to make git more verbose in the ssh setup stage) [04:08:09] 3Release-Engineering, Engineering-Community, Wikibugs: Only use -devtools irc channel for phab-related ticket announcements - https://phabricator.wikimedia.org/T89153#1029877 (10Dzahn) >>! In T89153#1029868, @greg wrote: > Because it was good to have a dedicated channel for the bugzilla->phab migration I think... [04:08:11] You can do "ssh -p 29418 mzmcbride@gerrit.wikimedia.org" [04:08:32] greg-g: If you ssh to that host, you should get a "**** Welcome to Gerrit Code Review ****" message. [04:08:55] That will let you futz with -vvv or whatever. [04:09:23] yeah, it's offering the wrong key multiple times... stupid, now to figure out why this changed [04:10:12] luckily I keep .ssh/config in git, and.. well, nothing changed for any WMF host section since August.... [04:10:33] * greg-g ponders [04:15:44] Did you mess around with ssh-add? [04:16:18] no, but I explicitly set -i in ssh and it worked [04:16:49] so, trying to figure out why ssh and my interpretation of my .ssh/config aren't the same [04:17:03] Hmm. [04:18:08] tada [04:18:12] now, why'd that change.... [04:19:25] -Host gerrit [04:19:25] +Host gerrit.wikimedia.org Hostname gerrit.wikimedia.org [04:19:38] er, bad (lack of) line break [04:20:16] http://paste.debian.net/145363/ [04:20:20] greg-g, did you see my big wall of text ticket? [04:20:39] not yet [04:20:41] ok [04:21:17] ugh [04:21:19] just skimmed it [04:22:48] Krenair: looks like springle got it? [04:23:06] He got the data to re-produce the missing entries, yep [04:23:11] cool [04:23:20] The bit I wanted you to be aware of was my unscheduled deployment [04:24:09] heh, I skimmed right over that part. Good job though. [04:24:47] Okay. I don't really know what your policy is on these kinds of things. [04:25:01] Krenair: "if crap is broken, don't wait for greg" [04:25:14] Ok, good :) [04:25:20] I don't want to be a blocker for 99% of the things [04:54:24] 3Phabricator: Adapting Gadget-BugStatusUpdate.js to Phabricator - https://phabricator.wikimedia.org/T539#1029915 (10Ricordisamoa) [05:08:42] 3Release-Engineering, Engineering-Community, Wikibugs: Only use -devtools irc channel for phab-related ticket announcements - https://phabricator.wikimedia.org/T89153#1029923 (10MZMcBride) >>! In T89153#1029868, @greg wrote: >>>! In T89153#1029759, @Legoktm wrote: >> Why wasn't -devtools merged into -releng? >... [05:11:24] 3Phabricator: Fatal error (30 seconds timeout) for some maniphest search queries - https://phabricator.wikimedia.org/T87739#1029924 (10mmodell) ok on further investigation, I think that keyword search is simply broken. The search term isn't being passed on to elasticsearch so es is returning everything it's got.... [05:14:38] Project beta-scap-eqiad build #41193: FAILURE in 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41193/ [05:34:51] Yippee, build fixed! [05:34:52] Project beta-scap-eqiad build #41195: FIXED in 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41195/ [05:51:41] 3Phabricator: Adapting Gadget-BugStatusUpdate.js to Phabricator - https://phabricator.wikimedia.org/T539#1029945 (10Ricordisamoa) >>! In T539#1028871, @Mattflaschen wrote: I've made a canonical version at https://www.mediawiki.org/wiki/Extension:Gadgets/Scripts/BugStatusUpdate.js , linked from https://www.mediaw... [05:54:06] 3Phabricator: Project search issue - https://phabricator.wikimedia.org/T89209#1029946 (10Bugreporter) 3NEW [05:58:11] 3Phabricator: Unregistered users should be able to see subscribers of a task - https://phabricator.wikimedia.org/T89210#1029954 (10Bugreporter) 3NEW [06:06:29] 3Phabricator: Project search issue - https://phabricator.wikimedia.org/T89209#1029975 (10Krinkle) [06:38:07] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [08:01:38] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:10:19] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:12:15] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:14:05] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:17:05] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 49057 bytes in 0.709 second response time [08:17:56] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:18:58] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 49040 bytes in 0.617 second response time [08:22:00] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:24:26] PROBLEM - Puppet failure on deployment-redis02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:26:42] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [08:30:09] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:30:31] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [08:32:38] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:34:28] 3Phabricator: Process to request a private project - https://phabricator.wikimedia.org/T823#1030131 (10Qgil) Seeing the discussion at T88762 and also knowing that other teams are considering Phabricator if the problem of private tasks can be solved (Zero, some non-tech WMF teams, and even the [[ https://meta.wik... [08:34:42] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:36:15] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [08:37:34] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:37:50] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:40:20] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [08:40:50] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:41:58] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [08:47:18] PROBLEM - Puppet failure on deployment-test is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:47:38] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [08:49:31] RECOVERY - Puppet failure on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [08:50:30] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [08:53:41] 3operations, Project-Creators, Phabricator: Create projects for Ops goals - https://phabricator.wikimedia.org/T87262#1030149 (10Nemo_bis) Please don't add this discussion to #HTTPS-by-default. I watch that project to read about HTTPS by default, not about Phabricator processes, and there is no ignore flag in Pha... [08:55:10] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:56:09] 3Phabricator: Project removal fails without warning - https://phabricator.wikimedia.org/T89221#1030158 (10Nemo_bis) 3NEW [08:57:39] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [08:59:41] RECOVERY - Puppet failure on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:02:32] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [09:04:24] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:05:46] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [09:07:17] PROBLEM - Puppet staleness on deployment-eventlogging02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [09:07:41] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [09:08:43] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0] [09:09:41] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [09:10:07] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:11:06] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:11:34] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [09:13:26] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:13:36] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:18:24] 3Phabricator: "Security" values no longer updates visibility/editability policies when changed on existing bugs? - https://phabricator.wikimedia.org/T88681#1030177 (10matmarex) 5Open>3Invalid a:3matmarex Uh, that's the behavior I would expect... l keep discovering that the security setting are more messed... [09:21:22] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:21:32] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:22:10] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [09:23:33] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:23:52] bah puppet/beta going wild :/ [09:25:47] !log bunch of puppet failure since 8:00am UTC. Seems to be DNS timeouts. [09:25:49] Logged the message, Master [09:28:45] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:29:23] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:29:48] 3Phabricator: Project removal fails without warning - https://phabricator.wikimedia.org/T89221#1030217 (10Qgil) This happens because "#https-by-default" is mentioned in the task description. Upstream is discussing about changing this behavior. [09:30:48] 3operations, Project-Creators, Phabricator: Create projects for Ops goals - https://phabricator.wikimedia.org/T87262#1030225 (10Qgil) [09:31:04] 3operations, Project-Creators, Phabricator: Create projects for Ops goals - https://phabricator.wikimedia.org/T87262#987167 (10Qgil) [09:31:20] 3Phabricator, Phabricator.org: Switching back and forth between image in lightbox is slow (redirects, no cache) - https://phabricator.wikimedia.org/T89225#1030229 (10Krinkle) 3NEW [09:31:31] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:32:04] 3Phabricator: Adapting Gadget-BugStatusUpdate.js to Phabricator - https://phabricator.wikimedia.org/T539#1030241 (10Edokter) >>! In T539#1029945, @Ricordisamoa wrote: > ...and enwiki should load it dynamically. Please define 'dynamically'. If you mean load it raw from MediaWiki, I think that is a bad idea. How... [09:33:44] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:34:42] RECOVERY - Puppet failure on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:35:00] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [09:36:08] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:38:15] 3operations, Project-Creators, Phabricator: Create projects for Ops goals - https://phabricator.wikimedia.org/T87262#1030264 (10Qgil) >>! In T87262#1028378, @Krenair wrote: > This broke the policy at https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects#New_projects that all project creations... [09:38:25] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:38:33] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [09:38:33] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [09:41:32] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:42:06] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [09:45:06] Project beta-scap-eqiad build #41221: FAILURE in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41221/ [09:45:46] hashar: sorry for being late :( [09:46:00] I need 5-10 more minutes [09:46:47] zeljkof: take your time. I am filling a security bug :] [09:48:40] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [09:54:59] Yippee, build fixed! [09:55:00] Project beta-scap-eqiad build #41222: FIXED in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41222/ [09:57:03] !log restarting Jenkins to upgrade the Credentials plugin [09:57:08] Logged the message, Master [09:57:14] RECOVERY - Puppet failure on deployment-test is OK: OK: Less than 1.00% above the threshold [0.0] [09:58:55] nature's call [10:05:10] hashar: sorry, this is taking way longer than expected [10:05:13] :( [10:05:21] 3Phabricator: Unregistered users should be able to see subscribers of a task - https://phabricator.wikimedia.org/T89210#1030359 (10Qgil) ? As anonymous user, I can see the subscribers of this task: Subscribers Aklapper, Bugreporter Please share URLs where the problem you are reporting can be reproduced. [10:10:55] 3Phabricator: Fatal error (30 seconds timeout) for some maniphest search queries - https://phabricator.wikimedia.org/T87739#1030370 (10Qgil) [10:10:58] 3Phabricator: Fix provided search results in Wikimedia Phabricator - https://phabricator.wikimedia.org/T75854#1030369 (10Qgil) [10:11:27] hashar: sorry for the long delay, I am ready now [10:11:30] joining the hangout [10:11:40] zeljkof: hold on !D [10:11:59] hashar: holding on... :) [10:18:09] 3Phabricator: Please remove the two-factor authentication from my Phabricator account - https://phabricator.wikimedia.org/T87495#1030414 (10zhaofengli) @Aklapper Thanks, it's on P280. [10:35:27] Project beta-scap-eqiad build #41226: FAILURE in 1 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41226/ [10:39:25] 3operations, Project-Creators, Phabricator: Create projects for Ops goals - https://phabricator.wikimedia.org/T87262#1030470 (10faidon) >>! In T87262#1030264, @Qgil wrote: > Theoretically it is possible to have private projects, but there should be a reason for that. In this case, no reason has been presented so... [10:48:03] 3operations, Project-Creators, Phabricator: Create projects for Ops goals - https://phabricator.wikimedia.org/T87262#1030497 (10Krenair) >>! In T87262#1030470, @faidon wrote: >>>! In T87262#1030264, @Qgil wrote: >> Theoretically it is possible to have private projects, but there should be a reason for that. In t... [10:55:02] 3Phabricator: Adapting Gadget-BugStatusUpdate.js to Phabricator - https://phabricator.wikimedia.org/T539#1030505 (10Krenair) >>! In T539#1030241, @Edokter wrote: >>>! In T539#1029945, @Ricordisamoa wrote: >> ...and enwiki should load it dynamically. > > Please define 'dynamically'. If you mean load it raw from... [10:55:17] Yippee, build fixed! [10:55:18] Project beta-scap-eqiad build #41228: FIXED in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41228/ [10:55:52] 3Phabricator: Fatal error (30 seconds timeout) for some maniphest search queries - https://phabricator.wikimedia.org/T87739#1030509 (10Tgr) Still doesn't explain why adding a keyword search actually makes the query slower, though. Does it trigger something expensive on the SQL side? Is the ES query different in... [10:56:43] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0] [11:05:51] Project beta-scap-eqiad build #41229: FAILURE in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41229/ [11:06:49] 3Phabricator: Adapting Gadget-BugStatusUpdate.js to Phabricator - https://phabricator.wikimedia.org/T539#1030532 (10He7d3r) >>! In T539#1029945, @Ricordisamoa wrote: > It should be in the MediaWiki namespace, to avoid edits by untrusted people, +1, for many other reasons too (see e.g. {T86904}). > and enwiki sh... [11:08:42] 3operations, Project-Creators, Phabricator: Create projects for Ops goals - https://phabricator.wikimedia.org/T87262#1030535 (10Qgil) [11:08:54] PROBLEM - Puppet failure on deployment-restbase03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:12:28] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [11:14:10] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [11:15:07] Yippee, build fixed! [11:15:08] Project beta-scap-eqiad build #41230: FIXED in 1 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41230/ [11:16:04] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:16:57] 3Phabricator: Adapting Gadget-BugStatusUpdate.js to Phabricator - https://phabricator.wikimedia.org/T539#1030543 (10Arkanosis) >>! In T539#1030505, @Krenair wrote: >>>! In T539#1030241, @Edokter wrote: >>>>! In T539#1029945, @Ricordisamoa wrote: >>> ...and enwiki should load it dynamically. >> >> Please define... [11:17:31] 3operations, Project-Creators, Phabricator: Create projects for Ops goals - https://phabricator.wikimedia.org/T87262#1030544 (10faidon) [11:18:39] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:24:15] 3Engineering-Community, Code-Review, Phabricator, Wikimedia-Git-or-Gerrit: Migrate Gerrit project ownership request system (+2 rights) to Phabricator - https://phabricator.wikimedia.org/T86639#1030550 (10Qgil) If someone is watching regularly, then probably on-wiki and on-task are probably very similar. However,... [11:24:35] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:25:23] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:29:38] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [11:33:54] Project beta-code-update-eqiad build #44095: FAILURE in 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/44095/ [11:33:57] RECOVERY - Puppet failure on deployment-restbase03 is OK: OK: Less than 1.00% above the threshold [0.0] [11:36:06] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [11:39:13] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:39:35] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [11:39:45] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:40:25] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:41:45] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [11:43:41] RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:44:13] Yippee, build fixed! [11:44:14] Project beta-code-update-eqiad build #44096: FIXED in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/44096/ [11:45:29] Project beta-scap-eqiad build #41232: FAILURE in 1 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41232/ [11:46:33] PROBLEM - Puppet failure on deployment-redis02 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [11:49:41] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [11:52:31] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:55:37] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:57:18] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [11:58:28] 3Phabricator: Adapting Gadget-BugStatusUpdate.js to Phabricator - https://phabricator.wikimedia.org/T539#1030572 (10Ricordisamoa) >>! In T539#1030241, @Edokter wrote: >>>! In T539#1029945, @Ricordisamoa wrote: >> ...and enwiki should load it dynamically. > > Please define 'dynamically'. If you mean load it raw... [11:59:50] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:01:20] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [12:06:08] Yippee, build fixed! [12:06:08] Project beta-scap-eqiad build #41234: FIXED in 2 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41234/ [12:06:47] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [12:09:58] 3pywikibot-core, Continuous-Integration: pip install --process-dependency-links fails on build boxes - https://phabricator.wikimedia.org/T89231#1030578 (10jayvdb) p:5High>3Triage [12:10:27] 3pywikibot-core, Continuous-Integration: pip install --process-dependency-links fails on build boxes - https://phabricator.wikimedia.org/T89231#1030515 (10jayvdb) We can work around this by Verified+2 patches like I did on https://gerrit.wikimedia.org/r/#/c/189929/ [12:11:26] RECOVERY - Puppet failure on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:16:33] (03PS1) 10Hashar: browsertests: move user/pass to Credentials plugin [integration/config] - 10https://gerrit.wikimedia.org/r/189938 (https://phabricator.wikimedia.org/T89226) [12:17:25] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:20:47] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [12:21:23] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:21:25] off for lunch [12:23:25] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [12:24:05] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [12:24:24] 3Phabricator: Fatal error (30 seconds timeout) for some maniphest search queries - https://phabricator.wikimedia.org/T87739#1030621 (10Aklapper) Thanks a lot Mukunda for analyzing what's happening! >>! In T87739#1029677, @mmodell wrote: > Here's my analysis of what's happening with this particular search query:... [12:24:45] 3Phabricator: Fatal error (30 seconds timeout) for some maniphest search queries (with ElasticSearch backend) - https://phabricator.wikimedia.org/T87739#1030627 (10Aklapper) [12:25:43] 3Phabricator: Adapting Gadget-BugStatusUpdate.js to Phabricator - https://phabricator.wikimedia.org/T539#1030632 (10Ricordisamoa) >>! In T539#1030543, @Arkanosis wrote: >>>! In T539#1030505, @Krenair wrote: >>>>! In T539#1030241, @Edokter wrote: >>>>>! In T539#1029945, @Ricordisamoa wrote: >>>> ...and enwiki sho... [12:25:44] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0] [12:26:54] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [12:29:30] 3Phabricator, Phabricator.org: Switching back and forth between image in lightbox is slow (redirects, no cache) - https://phabricator.wikimedia.org/T89225#1030646 (10Aklapper) p:5Triage>3Low [12:37:35] PROBLEM - SSH on deployment-lucid-salt is CRITICAL: Connection refused [12:40:51] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:43:26] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:44:04] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [13:00:42] RECOVERY - Puppet failure on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:08:45] 3MediaWiki-extensions-MathSearch, Continuous-Integration: MathSearch tests fail - https://phabricator.wikimedia.org/T89237#1030774 (10Physikerwelt) 3NEW [13:42:41] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:44:02] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #1454: FAILURE in 58 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/1454/ [13:44:19] PROBLEM - Host Generic Beta Cluster is DOWN: check_ping: Invalid hostname/address - en.wikipedia.beta.wmflabs.org [13:44:35] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:45:35] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:49:22] RECOVERY - Host Generic Beta Cluster is UP: PING OK - Packet loss = 0%, RTA = 0.65 ms [13:49:54] PROBLEM - Puppet failure on deployment-restbase03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:51:40] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:53:02] PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [13:53:04] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [13:53:34] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [13:56:10] Project UploadWizard-api-commons.wikimedia.org build #1155: FAILURE in 20 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.org/1155/ [13:57:57] Yippee, build fixed! [13:57:57] Project UploadWizard-api-commons.wikimedia.org build #1156: FIXED in 1 min 30 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.org/1156/ [13:58:23] !log labs dns / network is fuzzy again [13:59:02] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #1458: ABORTED in 6 min 26 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/1458/ [14:00:37] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [14:02:07] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:02:41] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [14:03:26] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:03:46] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [14:03:56] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:04:28] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:05:53] Project beta-scap-eqiad build #41246: FAILURE in 1 min 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41246/ [14:06:59] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 49040 bytes in 0.735 second response time [14:08:20] 3Phabricator: Sending email to create tasks doesn't work for me - https://phabricator.wikimedia.org/T1077#1030966 (10MarkAHershberger) chasemp writes: > chasemp added a comment. > > In https://phabricator.wikimedia.org/T1077#940279, @Qgil wrote: > >> In https://phabricator.wikimedia.org/T85156 you created a tas... [14:10:38] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [14:11:45] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:13:33] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:13:34] (03PS2) 10Hashar: browsertests: move user/pass to Credentials plugin [integration/config] - 10https://gerrit.wikimedia.org/r/189938 (https://phabricator.wikimedia.org/T89226) [14:14:13] (03CR) 10Hashar: "PS2 make some fix to the UploadWizard-api jobs. I have deployed both of them and they pass fine. Will update the rest." [integration/config] - 10https://gerrit.wikimedia.org/r/189938 (https://phabricator.wikimedia.org/T89226) (owner: 10Hashar) [14:14:59] RECOVERY - Puppet failure on deployment-restbase03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:15:26] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [14:15:34] 3Phabricator: Sending email to create tasks doesn't work for me - https://phabricator.wikimedia.org/T1077#1030986 (10MarkAHershberger) 5Open>3Resolved a:3MarkAHershberger Appears to be working. T89241. Thanks for all your help! (One more reason never to leave emacs added.) [14:15:49] Yippee, build fixed! [14:15:50] Project beta-scap-eqiad build #41247: FIXED in 1 mn 50 s: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41247/ [14:16:37] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [14:17:01] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:18:05] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:18:23] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:19:14] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [14:22:26] PROBLEM - Puppet failure on deployment-redis02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:23:51] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [14:25:07] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:26:33] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:26:56] Project beta-scap-eqiad build #41248: FAILURE in 2 mn 42 s: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41248/ [14:27:47] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:28:07] RECOVERY - Puppet failure on deployment-cache-mobile03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:28:19] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [14:29:43] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [14:30:39] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [14:32:36] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 29872 bytes in 0.583 second response time [14:34:03] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [14:34:26] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #472: ABORTED in 2 mn 12 s: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/472/ [14:34:29] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:35:27] Yippee, build fixed! [14:35:27] Project beta-scap-eqiad build #41249: FIXED in 1 mn 12 s: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41249/ [14:35:29] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:36:17] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:36:47] RECOVERY - Puppet failure on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:38:45] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:40:55] (03CR) 10Hashar: "I am updating all the jobs. The MediaWiki core does use login and failed because MEDIAWIKI_PASSWORD was not exported. Fixed in next patch " [integration/config] - 10https://gerrit.wikimedia.org/r/189938 (https://phabricator.wikimedia.org/T89226) (owner: 10Hashar) [14:41:12] (03PS3) 10Hashar: browsertests: move user/pass to Credentials plugin [integration/config] - 10https://gerrit.wikimedia.org/r/189938 (https://phabricator.wikimedia.org/T89226) [14:42:04] RECOVERY - Puppet failure on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:42:18] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [14:44:11] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [14:44:51] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:45:11] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [14:45:11] Project beta-scap-eqiad build #41250: FAILURE in 1 mn 5 s: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41250/ [14:45:24] 3Release-Engineering, translatewiki.net: Browser test for sandbox.translatewiki.net fails because of a SSL host mismatch (cert has 'translatewiki.net') - https://phabricator.wikimedia.org/T89244#1031076 (10hashar) 3NEW [14:48:17] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [14:48:37] Yippee, build fixed! [14:48:38] Project browsertests-Wikidata-PerformanceTests-linux-firefox-sauce build #151: FIXED in 1 mn 11 s: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-PerformanceTests-linux-firefox-sauce/151/ [14:48:42] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 29873 bytes in 5.583 second response time [14:50:47] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #434: FAILURE in 4 mn 2 s: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/434/ [14:51:14] PROBLEM - Puppet failure on deployment-test is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [14:51:36] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [14:52:29] 3Phabricator: "Security" values no longer updates visibility/editability policies when changed on existing bugs? - https://phabricator.wikimedia.org/T88681#1031118 (10chasemp) >>! In T88681#1030177, @matmarex wrote: > Uh, that's the behavior I would expect... l keep discovering that the security setting are more... [14:52:34] RECOVERY - Puppet failure on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:54:14] zeljkof: around ? [14:54:22] hashar: yes [14:54:22] zeljkof: I got a browser tests question for you [14:54:29] hashar: go ahead [14:54:32] looking that the execution of https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/460/console [14:54:32] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:54:44] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:54:50] I sea that each scenario has a long delay before actually reporting the first output [14:55:03] 30s to 1 minute roughly [14:55:18] I am wondering what is happening behind the scene [14:55:35] that run above is mediawiki selenium 0.4.0 [14:55:43] Yippee, build fixed! [14:55:44] Project beta-scap-eqiad build #41251: FIXED in 1 mn 39 s: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41251/ [14:56:18] and on https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/436/console it took 3min40s to login [14:56:19] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [14:58:32] hashar: hm [14:58:47] I would need to take a closer look at how the tests are implemented [14:58:57] who knows what went wrong [14:59:19] chris was recently refactoring a lot of code, he might know more [14:59:26] Yippee, build fixed! [14:59:27] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #436: FIXED in 7 mn 33 s: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/436/ [14:59:59] hashar: jenkins is in french again :) [15:00:33] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #473: FAILURE in 25 mn: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/473/ [15:00:36] 3Phabricator: Fix provided search results in Wikimedia Phabricator - https://phabricator.wikimedia.org/T75854#1031138 (10chasemp) [15:01:08] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0] [15:01:54] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:02:22] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [15:02:51] hashar: I am in the hangout [15:04:51] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [15:07:34] zeljkof: coming [15:08:55] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:13:26] 3Release-Engineering, translatewiki.net: Browser test for sandbox.translatewiki.net fails because of a SSL host mismatch (cert has 'translatewiki.net') - https://phabricator.wikimedia.org/T89244#1031195 (10Nikerabbit) We might do a star cert next time we need to renew it. But I'd also like these tests migrated a... [15:14:06] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:16:17] RECOVERY - Puppet failure on deployment-test is OK: OK: Less than 1.00% above the threshold [0.0] [15:16:47] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:25:07] 3Code-Review: GitHub->Phabricator import system - https://phabricator.wikimedia.org/T715#1031233 (10Ricordisamoa) Even the [[ //tools.wmflabs.org/gerrit-patch-uploader/ | Gerrit Patch Uploader ]] by @valhallasw is hosted on GitHub... [15:26:05] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [15:29:13] hello zeljkof! [15:29:26] vikasyal_: sorry, in a meeting [15:29:36] zeljkof: ah ! OK :) [15:41:58] PROBLEM - Content Translation Server on deployment-cxserver03 is CRITICAL: Connection refused [15:47:02] 3MediaWiki-extensions-MathSearch, Continuous-Integration: MathSearch tests fail - https://phabricator.wikimedia.org/T89237#1031269 (10hoo) 5Open>3Resolved a:3hoo [15:54:14] Q: I need to do cxserver deployment today. Do I need to add it to slot or 'just do it'? [15:55:41] 3Phabricator: Fatal error (30 seconds timeout) for some maniphest search queries (with ElasticSearch backend) - https://phabricator.wikimedia.org/T87739#1031322 (10chasemp) 5Open>3Resolved a:3chasemp takes a minute but finishes successfully now for me logged in and out [15:55:43] 3Phabricator: Fix provided search results in Wikimedia Phabricator - https://phabricator.wikimedia.org/T75854#1031325 (10chasemp) [15:56:57] RECOVERY - Content Translation Server on deployment-cxserver03 is OK: HTTP OK: HTTP/1.1 200 OK - 1103 bytes in 0.016 second response time [15:57:46] greg-g: Let me know if it is okay to add slot for cxserver service deployment, but we don't have many changes always. [15:57:59] It is *as needed* [16:00:47] 3Phabricator: Fix provided search results in Wikimedia Phabricator - https://phabricator.wikimedia.org/T75854#1031345 (10chasemp) >>! In T75854#1024713, @chasemp wrote: > Very sick today but in general I can do this, probably wed as it will > create an unknown period of search outage. I wanted to give @springle... [16:01:12] kart_: add a slot, please. :) [16:02:43] greg-g: cool. We need it today then :) [16:05:54] 3Phabricator: Fix provided search results in Wikimedia Phabricator - https://phabricator.wikimedia.org/T75854#1031380 (10chasemp) [16:05:56] 3Phabricator, Phabricator.org: Improve default search to prefer tasks over changesets - https://phabricator.wikimedia.org/T76273#1031377 (10chasemp) 5Open>3declined a:3chasemp There is an implicit behavior that I think can solve this for individual users if they desire (wish is already noted somewhat above... [16:06:07] kart_: pick a time that's free on the calendar and add it. This is just the service right? any MW/Extension code updates needed? [16:06:16] 3Phabricator, Phabricator.org: Improve default search to prefer tasks over changesets - https://phabricator.wikimedia.org/T76273#1031382 (10chasemp) [16:06:17] greg-g: added today now. I'll wait till SWAT is done. [16:06:23] <^d> greg-g: Are you wfh today or in the office? [16:06:26] greg-g: yes. service. [16:06:43] greg-g: no extension/mw updates. [16:07:01] ^d: home [16:07:09] kart_: cool, thanks :) [16:07:44] 3Phabricator: Searchable "Reference" custom field - https://phabricator.wikimedia.org/T991#1031384 (10chasemp) This has been sitting for awhile now. Is anyone going to take this on? [16:07:49] <^d> greg-g: k, just wondering if I needed to hurry up and get in for our 1:1 in person or if we'd be hanging out anyway [16:08:53] greg-g: how do I know if SWAT is done? [16:09:29] <^d> Ask the person who's doing swat [16:09:56] ^d: thanks! [16:10:01] 3Phabricator, Phabricator.org: Can't search an exact phrase in Phabricator - https://phabricator.wikimedia.org/T75743#1031388 (10chasemp) 5Open>3Resolved >>! In T75743#978756, @Qgil wrote: > Unless there is some indexing to be done or some special syntax to be used other than quotes, searching for "whole blo... [16:10:03] 3Phabricator: Fix provided search results in Wikimedia Phabricator - https://phabricator.wikimedia.org/T75854#1031390 (10chasemp) [16:11:09] 3Phabricator: Fix provided search results in Wikimedia Phabricator - https://phabricator.wikimedia.org/T75854#783695 (10chasemp) [16:11:12] 3Phabricator, Phabricator.org: Phabricator search does not reliably match assigned projects - https://phabricator.wikimedia.org/T78219#1031392 (10chasemp) 5Open>3Resolved a:3chasemp Unless I'm mistaken the expected results are now there: https://phabricator.wikimedia.org/search/query/x.ZdhI1NTHPD/#R [16:12:00] 3Phabricator: Searchable "Reference" custom field - https://phabricator.wikimedia.org/T991#1031396 (10chasemp) >>! In T991#1031384, @chasemp wrote: > This has been sitting for awhile now. Is anyone going to take this on? Better question, is this a necessary feature at this point? [16:12:48] 3Phabricator: Fix provided search results in Wikimedia Phabricator - https://phabricator.wikimedia.org/T75854#1031399 (10mmodell) [16:12:49] ^d: :) [16:12:49] 3Phabricator: Fatal error (30 seconds timeout) for some maniphest search queries (with ElasticSearch backend) - https://phabricator.wikimedia.org/T87739#1031397 (10mmodell) 5Resolved>3Open >>! In T87739#1031322, @chasemp wrote: > takes a minute but finishes successfully now for me logged in and out The sear... [16:13:01] 3Phabricator: Fatal error (30 seconds timeout) for some maniphest search queries (with ElasticSearch backend) - https://phabricator.wikimedia.org/T87739#1031401 (10mmodell) a:5chasemp>3mmodell [16:17:31] 3Phabricator: Fatal error (30 seconds timeout) for some maniphest search queries (with ElasticSearch backend) - https://phabricator.wikimedia.org/T87739#1031406 (10chasemp) >>! In T87739#1031397, @mmodell wrote: >>>! In T87739#1031322, @chasemp wrote: >> takes a minute but finishes successfully now for me logged... [16:22:54] 3Phabricator: Lots of unrelated results when searching for specific string - https://phabricator.wikimedia.org/T86805#1031412 (10chasemp) I'm pretty sure this is a 'special characters' and 'mysql full text search' issue. How exactly to resolve that I am not sure. @springle any wisdom on this or what we could f... [16:25:24] 3Phabricator: Maniphest advanced search broken when keyword field is provided. - https://phabricator.wikimedia.org/T87739#1031421 (10mmodell) [16:27:26] 3Phabricator, Phabricator.org: Maniphest advanced search broken when keyword field is provided. - https://phabricator.wikimedia.org/T87739#1031423 (10mmodell) Elasticsearch backend is still broken completely, we need to at least upstream the bug. [16:28:39] hashar: did you change a global setting in Jenkins? my Jenkins UI is in French now. :-) [16:29:15] <^d> That's such a funny bug [16:29:19] <^d> It gets stuck [16:29:25] <^d> On someone else's language [16:29:41] <^d> I've seen Italian a few times [16:30:00] Yeah. And Polish and German. [16:30:07] Yay for Jenkins. :-) [16:30:27] Jenkins will be in French only. [16:30:28] :D [16:30:36] <^d> Until it picks a new language [16:32:42] chrismcmahon: yeah that happens from time to time [16:32:48] gotta restart jenkins to get rid of the weird locale [16:33:59] !log saved Jenkins global configuration at https://integration.wikimedia.org/ci/configure to hopefully reset the web interface default locale [16:34:05] chrismcmahon: might be using english now [16:34:08] qa-morebots: come on [16:34:10] !log foo [16:34:13] sucks [16:34:13] I am a logbot running on tools-exec-11. [16:34:13] Messages are logged to https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL. [16:34:13] To log a message, type !log . [16:34:20] !log saved Jenkins global configuration at https://integration.wikimedia.org/ci/configure to hopefully reset the web interface default locale [16:34:27] * hashar gives up [16:34:52] chrismcmahon: I have migrated all user/pass for the browser tests with https://gerrit.wikimedia.org/r/#/c/189938/ [16:34:54] could use a review [16:35:06] hashar: yeah, I saw that, good idea [16:35:12] rationale is in security task https://phabricator.wikimedia.org/T89226 which you should be able to read if you are in Phabricator WMF-NDA group [16:35:27] I have refreshed all the jobs already so the JJB change should be fine [16:35:50] but could use another pair of eyes, and most importantly verify the jobs are fine (they should I triggered a bunch of them earlier) [16:35:55] 3Phabricator: Fix provided search results in Wikimedia Phabricator - https://phabricator.wikimedia.org/T75854#1031440 (10chasemp) [16:36:06] I am off [16:36:16] have 1/1 with greg in 25 minutes and need to commute back sorry [16:36:20] hashar: fwiw, Zeljko and I are probably going to delete all the test2 browser test builds, we may be able to remove the production test user [16:36:26] this week [16:36:30] great! [16:36:43] they're just not useful any more [16:36:52] will probably get my JJB patch merged this evening (i.e. in roughly 4 hours from now) [16:37:05] rushing back home [16:45:15] 3Release-Engineering, Wikimedia-Labs-wikitech-interface: add [[wikitech:Release Engineering/SAL]] to [[wikitech:mediawiki:sidebar]] - https://phabricator.wikimedia.org/T73165#1031450 (10greg) 5Open>3Resolved Apparently I had/have the rights: https://wikitech.wikimedia.org/w/index.php?title=MediaWiki%3ASideba... [16:45:28] 3Release-Engineering, Wikimedia-Labs-wikitech-interface: add [[wikitech:Release Engineering/SAL]] to [[wikitech:mediawiki:sidebar]] - https://phabricator.wikimedia.org/T73165#1031452 (10greg) a:5coren>3greg [16:55:12] (03CR) 10Cmcmahon: [C: 031] "This is a good idea. Is it going to conflict with anything in Dans' EAL work?" [integration/config] - 10https://gerrit.wikimedia.org/r/189938 (https://phabricator.wikimedia.org/T89226) (owner: 10Hashar) [17:18:04] 3Phabricator: Task creation by email is unclear - https://phabricator.wikimedia.org/T865#1031600 (10valhallasw) 5Open>3Resolved a:3valhallasw [17:23:13] 3Continuous-Integration: php-composer-validate job should not be triggered if a composer.json file is removed from the repository - https://phabricator.wikimedia.org/T89263#1031627 (10Legoktm) 3NEW [17:42:16] 3operations, Phabricator: The options of the Security dropdown in Phabricator need to be clear and documented - https://phabricator.wikimedia.org/T76564#1031747 (10chasemp) [17:45:59] 3operations, Phabricator: The options of the Security dropdown in Phabricator need to be clear and documented - https://phabricator.wikimedia.org/T76564#1031766 (10faidon) So, where do e.g. Parsoid security bugs should be filled under? How about e.g. Heartbleed-type of issues? This "MediaWiki security bug" makes... [17:48:45] chrismcmahon: The Chrome bugs in our browser tests that require sleep 1 everywhere, is there any bug in upstream tracking that? [17:50:55] 3operations, Phabricator: The options of the Security dropdown in Phabricator need to be clear and documented - https://phabricator.wikimedia.org/T76564#1031780 (10Parent5446) I believe the security drop-down is an indication of the severity of the bug, not the project with which it is associated. So any bug tha... [17:51:09] marktraceur: afaik, the Chrome people consider it a feature not a bug [17:51:39] 3operations, Phabricator: The options of the Security dropdown in Phabricator need to be clear and documented - https://phabricator.wikimedia.org/T76564#1031785 (10chasemp) Updated the text and https://www.mediawiki.org/wiki/Phabricator/Security can someone verify and see if more is required here? [17:52:25] marktraceur: and (again afaik) because of the way RL works, it is fiendishly difficult to work around [17:52:56] *sigh* [17:53:06] chrismcmahon: Should we just bite the bullet and add the sleep 1 everywhere then? [17:53:26] marktraceur: I say yeah, sleep 1 is not that evil. sleep 10 would be a lot worse [17:53:38] OK [17:53:50] * marktraceur will !logs this conversation for the record [17:54:10] marktraceur: someday WebDriver will be a WW3 standard, but for now Mozilla and Google disagree on the definition of when an element can be interacted with [17:54:59] Project beta-scap-eqiad build #41268: FAILURE in 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41268/ [17:56:02] !log hashar saved Jenkins global configuration at https://integration.wikimedia.org/ci/configure to hopefully reset the web interface default locale [17:56:04] Logged the message, Master [17:57:43] <^d> Elasticsearch is back up in beta. [17:57:46] <^d> Rebuilding indexes now [17:57:51] RECOVERY - Host deployment-elastic07 is UP: PING OK - Packet loss = 0%, RTA = 0.98 ms [17:59:09] RECOVERY - Host deployment-elastic06 is UP: PING OK - Packet loss = 0%, RTA = 0.91 ms [17:59:36] RECOVERY - Host deployment-elastic05 is UP: PING OK - Packet loss = 0%, RTA = 0.72 ms [18:01:22] RECOVERY - Host deployment-elastic08 is UP: PING OK - Packet loss = 0%, RTA = 0.62 ms [18:01:24] greg-g: I'm going to bring up https://phabricator.wikimedia.org/T89049 in a general sort of way in SoS, basically http://memegenerator.net/instance/59063787 [18:14:55] Yippee, build fixed! [18:14:56] Project beta-scap-eqiad build #41270: FIXED in 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41270/ [18:18:32] chrismcmahon: :) kk [18:29:46] 3Release-Engineering, Engineering-Community, Wikibugs: Only use -devtools irc channel for phab-related ticket announcements - https://phabricator.wikimedia.org/T89153#1031940 (10greg) 5Open>3Resolved >>! In T89153#1031745, @gerritbot wrote: > Change 189901 merged by jenkins-bot: Thanks @legoktm. And your st... [18:30:00] 3Release-Engineering, Engineering-Community, Wikibugs: Only use -devtools irc channel for phab-related ticket announcements - https://phabricator.wikimedia.org/T89153#1031943 (10greg) p:5Triage>3Normal [18:30:06] 3Project-Creators, Wikimedia-Labs-wikistats, MediaWiki-extensions-OpenStackManager, Wikimedia-Labs-Infrastructure, Wikimedia-Labs-wikitech-interface, Labs, Labs-Vagrant, Tool-Labs-tools-Article-request, Tool-Labs, Wikimedia-Labs-Other, Beta-Cluster, Wikimedia-Labs-extdist, Wikimedia-Labs-General: Labs' Phabricator... [18:30:21] >.> [18:30:35] too many projects! [18:31:02] haha [18:37:11] 3Multimedia, MediaWiki-extensions-UploadWizard, Quality-Assurance: UploadWizard API tests failing on beta Commons due to login problem - https://phabricator.wikimedia.org/T89272#1031965 (10Tgr) 3NEW [18:37:11] legoktm: there's a patch in gerrit to fix that :P [18:37:35] https://gerrit.wikimedia.org/r/#/c/187469/ [18:37:45] there's a few more wikibugs patches that could use a review [18:39:23] oops [18:39:26] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #308: FAILURE in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/308/ [18:48:12] 3Project-Creators, Wikimedia-Labs-wikistats, MediaWiki-extensions-OpenStackManager, Wikimedia-Labs-Infrastructure, Wikimedia-Labs-wikitech-interface, Labs, Labs-Vagrant, Tool-Labs-tools-Article-request, Tool-Labs, Wikimedia-Labs-Other, Beta-Cluster, Wikimedia-Labs-extdist, Wikimedia-Labs-General: Labs' Phabricator... [18:49:56] Yippee, build fixed! [18:49:57] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #534: FIXED in 38 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/534/ [18:51:06] Yippee, build fixed! [18:51:06] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #474: FIXED in 11 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/474/ [19:12:42] Yippee, build fixed! [19:12:43] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #1463: FIXED in 26 min: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/1463/ [19:37:32] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #307: FAILURE in 47 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/307/ [19:39:34] !log Jenkins UI is stuck in French. Resetting.. [19:39:37] Logged the message, Master [19:44:08] <^d> Krinkle: If you force the language to en_US I haven't had that problem. [19:44:28] 3Project-Creators, Wikimedia-Labs-wikistats, MediaWiki-extensions-OpenStackManager, Wikimedia-Labs-Infrastructure, Wikimedia-Labs-wikitech-interface, Labs, Labs-Vagrant, Tool-Labs-tools-Article-request, Tool-Labs, Wikimedia-Labs-Other, Beta-Cluster, Wikimedia-Labs-extdist, Wikimedia-Labs-General: Labs' Phabricator... [19:47:32] ^d: When it happens (usually after a restart by someone with a French user preference), it becomes the default for all logged-out users. [19:47:48] ^d: Eventhough the settings are set to en-US and to ignore Accept-Language headers. [19:47:52] (because caching) [19:48:38] The fix is to set the configuration from en-US to en-us, or visa versa. Just a no-op change. [19:49:23] Annoyingly, however, in build logs, messages are substituted. Many older builds have random lines in Russion or French. [19:49:37] Russian* [19:51:08] * ^d nods [20:28:13] 3MediaWiki-extensions-MathSearch, Continuous-Integration: MathSearch tests fail - https://phabricator.wikimedia.org/T89237#1032435 (10hashar) hoo and aude added a new test for extensions with https://gerrit.wikimedia.org/r/#/c/189607/ which detects wrong user rights. I have [[ https://gerrit.wikimedia.org/r/#... [20:32:28] 3Release-Engineering, translatewiki.net: Browser test for sandbox.translatewiki.net fails because of a SSL host mismatch (cert has 'translatewiki.net') - https://phabricator.wikimedia.org/T89244#1032454 (10hashar) Should we get rid of the https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSel... [20:39:01] 3Multimedia, MediaWiki-extensions-UploadWizard, Quality-Assurance: UploadWizard browser test for chunked upload - https://phabricator.wikimedia.org/T89289#1032475 (10Tgr) 3NEW a:3Tgr [20:54:47] 3Release-Engineering, translatewiki.net: Browser test for sandbox.translatewiki.net fails because of a SSL host mismatch (cert has 'translatewiki.net') - https://phabricator.wikimedia.org/T89244#1032533 (10Nikerabbit) I would say drop the tests, but I hope Amir will comment as he has been the contact person for... [20:57:19] (03CR) 10Hashar: "Good catch Chris, I forgot about the EAL :-(" [integration/config] - 10https://gerrit.wikimedia.org/r/189938 (https://phabricator.wikimedia.org/T89226) (owner: 10Hashar) [20:59:59] (03CR) 10Hashar: "I proposed to make that job voting Monday on the OPS list (subject: puppet-lint errors free). Seems nobody complained so it can be deploy" [integration/config] - 10https://gerrit.wikimedia.org/r/189589 (https://phabricator.wikimedia.org/T87132) (owner: 10Matanya) [21:02:21] 3Project-Creators, Wikimedia-Labs-wikistats, MediaWiki-extensions-OpenStackManager, Wikimedia-Labs-Infrastructure, Wikimedia-Labs-wikitech-interface, Labs, Labs-Vagrant, Tool-Labs-tools-Article-request, Tool-Labs, Wikimedia-Labs-Other, Beta-Cluster, Wikimedia-Labs-extdist, Wikimedia-Labs-General: Labs' Phabricator... [21:09:39] 3Release-Engineering: Make log responsibilities changes - https://phabricator.wikimedia.org/T89049#1032549 (10Nemo_bis) From a user perspective, we also still need to fix T40095: namely, the fact that the user who gets "Fatal exception of type MWException" is unable to look for meaningful information, and when r... [21:12:07] 3Release-Engineering: Make log responsibilities changes - https://phabricator.wikimedia.org/T89049#1032557 (10greg) >>! In T89049#1032549, @Nemo_bis wrote: > From a user perspective, we also still need to fix T40095: namely, the fact that the user who gets "Fatal exception of type MWException" is unable to look... [21:18:58] 3Continuous-Integration: Design the Jenkins isolation architecture - https://phabricator.wikimedia.org/T86171#1032564 (10hashar) I have updated [[ https://www.mediawiki.org/wiki/Continuous_integration/Architecture/Isolation | the architecture document ]] this week. A concern I had was to have all Jenkins runner... [21:19:45] 3Release-Engineering: Make log responsibilities changes - https://phabricator.wikimedia.org/T89049#1032571 (10Nemo_bis) My point was: since 2012 we pretend that sysadmins are paying a lot of attention to logs, to the point we can afford not telling actual errors to the users. Since 2015, it would be nice for tha... [21:20:34] 3Project-Creators, Wikimedia-Labs-wikistats, MediaWiki-extensions-OpenStackManager, Wikimedia-Labs-Infrastructure, Wikimedia-Labs-wikitech-interface, Labs, Labs-Vagrant, Tool-Labs-tools-Article-request, Tool-Labs, Wikimedia-Labs-Other, Beta-Cluster, Wikimedia-Labs-extdist, Wikimedia-Labs-General: Labs' Phabricator... [21:21:26] Nemo_bis: :) [21:21:46] (instead of making a comment just for a smiley, you get it here) [21:22:35] 3operations, Continuous-Integration: Create a Debian package for NodePool - https://phabricator.wikimedia.org/T89142#1032598 (10hashar) I have poked the internal OPS list about creating Debian packages for python software that have conflicting or missing dependencies. Pasted at P284 follow up on OPS list. [21:22:38] 3operations, Continuous-Integration: [upstream] Create a Debian package for Zuul - https://phabricator.wikimedia.org/T48552#1032600 (10hashar) I have poked the internal OPS list about creating Debian packages for python software that have conflicting or missing dependencies. Pasted at P284 follow up on OPS list. [21:22:50] 3Beta-Cluster: /var/log full on deployment-elasticsearch* hosts - https://phabricator.wikimedia.org/T88280#1032602 (10Chad) 5Open>3Resolved a:3Chad Reimaged. [21:23:10] (much more efficient, I agree) [21:24:01] Nemo_bis greg-g apropos of ^^, in the Socialtext wiki, when a user gets a fatal exception, there is a hidden link (the period at the end of the error message) that points to a detailed stack trace. That was often very convenient. [21:24:17] greg-g: should we add a link to https://phabricator.wikimedia.org/maniphest/task/create/ from the "Fatal exception of type MWException" message in Wikimedia? [21:24:22] 3VisualEditor, Beta-Cluster: Search on the Beta Cluster is broken, and thus VE no longer shows suggestion list for Templates upon typing into Add template selector - https://phabricator.wikimedia.org/T89007#1032610 (10Chad) >>! In T89007#1029535, @Ryasmeen wrote: > I can now see the template suggestions now.Ther... [21:24:33] chrismcmahon: we used to have detailed stack trace till 2012 [21:24:47] That's what https://phabricator.wikimedia.org/T40095 is about [21:25:43] Since 2012, all the work is left to WMF employees ;) [21:27:22] 3Release-Engineering: Make log responsibilities changes - https://phabricator.wikimedia.org/T89049#1032620 (10Chad) >>! In T89049#1032571, @Nemo_bis wrote: > My point was: since 2012 we pretend that sysadmins are paying a lot of attention to logs, to the point we can afford not telling actual errors to the users... [21:31:08] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #461: FAILURE in 25 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/461/ [21:38:48] 3Multimedia, MediaWiki-extensions-Sentry, Beta-Cluster, Wikimedia-Logstash: Channel PHP errors from Logstash to Sentry on the beta cluster - https://phabricator.wikimedia.org/T85239#1032685 (10greg) [21:41:49] 3Multimedia, MediaWiki-extensions-Sentry, Beta-Cluster, Wikimedia-Logstash: Channel PHP errors from Logstash to Sentry on the beta cluster - https://phabricator.wikimedia.org/T85239#1032689 (10Tgr) [21:44:05] 3Release-Engineering: Make log responsibilities changes - https://phabricator.wikimedia.org/T89049#1032693 (10greg) (This is the email I just sent to the engineering@ list, posting here as well.) As the person who sits where the buck proverbially stops on this issue I need to figure out how to make the rubber... [21:45:46] 3Release-Engineering: Make log responsibilities changes - https://phabricator.wikimedia.org/T89049#1032699 (10greg) [21:49:16] 3Release-Engineering, Project-Creators: Create project/tag to collect fatal/exception log related bugs - https://phabricator.wikimedia.org/T89292#1032707 (10greg) 3NEW a:3greg [21:52:39] 3Release-Engineering, Project-Creators: Create project/tag to collect fatal/exception log related bugs - https://phabricator.wikimedia.org/T89292#1032732 (10greg) [21:53:10] 3Release-Engineering: Make log responsibilities changes - https://phabricator.wikimedia.org/T89049#1032736 (10Tgr) [21:53:50] 3Release-Engineering, Project-Creators: Create project/tag to collect fatal/exception log related bugs - https://phabricator.wikimedia.org/T89292#1032739 (10chasemp) Award for most forward slashes in a ticket creation 2015. :) [21:53:59] 3Release-Engineering: Make log responsibilities changes - https://phabricator.wikimedia.org/T89049#1026260 (10Tgr) T85188 is only useful for 3rd-party wikis, on the WMF clusters logstash will replace it. [21:59:30] 3Release-Engineering, Project-Creators: Create project/tag to collect fatal/exception log related bugs - https://phabricator.wikimedia.org/T89292#1032755 (10greg) >>! In T89292#1032739, @chasemp wrote: > Award for most forward slashes in a ticket creation 2015. :) I told you I wasn't sure on naming! ;) Honest... [22:00:44] <^d> greg-g: #WMF-fix-yo-stuff [22:00:49] <^d> #WMF-fix-yo-shit [22:00:52] <^d> #WMF-shit-code [22:01:35] 3Release-Engineering, Project-Creators: Create project/tag to collect fatal/exception log related bugs - https://phabricator.wikimedia.org/T89292#1032756 (10Legoktm) So the intent is that this would be something similar to the browser-test-bug tag? I like #Wikimedia-log-errors. [22:02:08] ^d: I made myself not curse :) [22:03:54] 3Release-Engineering, Project-Creators: Create project/tag to collect fatal/exception log related bugs - https://phabricator.wikimedia.org/T89292#1032757 (10chasemp) Honest question, why put in WMF? Whose errors would they be if that wasn't specified? We have (in new things) tried to stray from putting Wikimed... [22:05:38] 3Release-Engineering, Project-Creators: Create project/tag to collect fatal/exception log related bugs - https://phabricator.wikimedia.org/T89292#1032763 (10greg) >>! In T89292#1032757, @chasemp wrote: > Honest question, why put in WMF? Whose errors would they be if that wasn't specified? We have (in new thing... [22:13:09] 3Multimedia, Release-Engineering, MediaWiki-Uploading: Fatal error "Object does not implement ArrayAccess" in FileRepo::publishBatch() - https://phabricator.wikimedia.org/T89296#1032785 (10Tgr) 3NEW [22:13:36] 3Multimedia, Release-Engineering, MediaWiki-Uploading: Fatal error "Object does not implement ArrayAccess" in FileRepo::publishBatch() - https://phabricator.wikimedia.org/T89296#1032793 (10Tgr) [22:13:57] 3Multimedia, Release-Engineering, MediaWiki-Uploading: Fatal error "Object does not implement ArrayAccess" in FileRepo::publishBatch() - https://phabricator.wikimedia.org/T89296#1032785 (10Tgr) > Presumably related to I015635a9bf080ef6d98b2cff49b949c4378a859f. Mark verified via bisect. [22:14:51] 3Release-Engineering, Project-Creators: Create project/tag to collect fatal/exception log related bugs - https://phabricator.wikimedia.org/T89292#1032803 (10chasemp) If you are thinking of grouping //all// exceptions from prod (even infrastructure?) under one big umbrella I would be concerned that it would come... [22:16:38] 3Release-Engineering, Project-Creators: Create project/tag to collect fatal/exception log related bugs - https://phabricator.wikimedia.org/T89292#1032806 (10greg) Maybe it could be just MW/extension exceptions/fatals and leave the services to their own devices? prod vs staging-exception could be useful, but pro... [22:21:27] 3Release-Engineering, Project-Creators: Create project/tag to collect fatal/exception log related bugs - https://phabricator.wikimedia.org/T89292#1032823 (10chasemp) the staging suggestion was mostly illustrative yeah :) In theory we'll never see any prod-exceptions once we are using the heck out of staging-ex... [22:31:11] twentyafterfour: if you're still busy with deploy and fallout stuff, we can do our 1:1 tomorrow or Friday [22:31:32] just waiting for scap right now [22:31:48] greg-g: ^ ..be in hangout in < 1 minute [22:32:27] coolio [22:43:14] 3Multimedia, Release-Engineering, MediaWiki-Uploading: Fatal error "Object does not implement ArrayAccess" in FileRepo::publishBatch() - https://phabricator.wikimedia.org/T89296#1032892 (10Tgr) 5Open>3Resolved a:3Tgr Fix verified on beta. Thanks Aaron! [23:14:19] 3Release-Engineering, Wikimedia-General-or-Unknown, operations, WMF-Design: Better WMF error pages - https://phabricator.wikimedia.org/T76560#1032950 (10Jaredzimmerman-WMF) @technical13 your point is taken, as we refine the wording we can think about the relationship between the donation action and the page. I'm... [23:17:27] 3Release-Engineering, § Fundraising Tech Backlog, § Fundraising Sprint E, Continuous-Integration, Wikimedia-Fundraising-CiviCRM: Deploy CiviCRM integration job to WMF integration server - https://phabricator.wikimedia.org/T86374#1032964 (10atgo)