[00:50:35] Krinkle|detached: plz to open bz? [02:43:31] Betacommand: I am thankful for the fact that you did give me help [02:48:18] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597 (10Greg Grossmeier) 3NEW p:3Unprio s:3normal a:3None From James' email to the QA list: Beta Labs isn't synchronising; AFAICS it hasn't do... [02:48:29] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597 (10Greg Grossmeier) p:5Unprio>3Highes [03:00:18] Coren: English is my primary language (German's second). The fact that I'm constantly being asked to "clarify" what I wrote and that it takes an hour to do those simple replies is a clear indicator I'm no good at it. [03:01:23] !log integration Restarted agent on deployment-bastion (twice) [03:01:25] Logged the message, Master [03:05:15] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597#c1 (10Bryan Davis) 5NEW>3RESO/FIX This happens once in a while. It's some sort of deadlock in Jenkins itself. Here's how I generally try to re... [03:33:15] DispenserAFK: tbh, your English is much better than plenty of people I've interacted with in the Wikimedia-world, so don't let that ever stop you. [04:58:44] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597#c2 (10James Forrester) Thanks! Should we write these down somewhere for the next time it occurs? [05:01:14] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597#c3 (10Bryan Davis) (In reply to James Forrester from comment #2) > Thanks! Should we write these down somewhere for the next time it occurs? Greg... [05:03:58] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597#c4 (10James Forrester) Ha. Thanks, both! [05:29:59] 3Wikimedia Labs / 3deployment-prep (beta): Commons beta cannot resolve redirect URLs - 10https://bugzilla.wikimedia.org/70124#c2 (10dan) 5NEW>3RESO/FIX https://gerrit.wikimedia.org/r/159214 took care of the issue. tested it on commons beta successfully. [06:15:47] there is also a mail about it... traffic from Labs is not good [06:18:21] What is "traffic from Labs"? [06:25:08] things like Reasonator are slow and slower [06:26:44] I doubt that's a surprise, Labs has been overloaded since the day before Toolserver showdown https://ganglia.wikimedia.org/latest/?r=month&cs=&ce=&m=cpu_report&s=descending&c=Virtualization+cluster+eqiad&h=&host_regex=&max_graphs=0&tab=m&vn=&hide-hf=false&sh=1&z=small&hc=4 [06:32:56] ah ... for me it is. [06:33:04] What is done to remedy this ? [06:51:27] GerardM-: no idea; one additional server started working a few weeks ago [06:51:34] (it was there, but idling) [06:52:26] do you by any chance know the status of the dumps ? [06:52:53] it was also under construction new servers et al ... but it does not work yet as far as I know [07:14:54] I've not been following updates last month [08:39:55] @unjb ToAruShiroNeiko [08:40:06] omg [08:40:34] @unjb ToAruShiroiNeiko [08:40:58] why does someone has so idiotic nickname... [08:41:03] @unjb ToAruShiroiNeko [08:41:12] finally I wrote it correctly XD [10:43:25] Can anyone recommend ame profiling tool for PHP please? [11:08:43] I wish I could [12:16:17] Nemo_bis: I see no "overload"; while we have one server that's having issues (virt1006) the others are around 50%, which is pretty good. [13:40:17] Coren: managed to throw up a replacment tool : [13:40:19] http://tools.wmflabs.org/betacommand-dev/cgi-bin/backlinks [13:40:43] That was quick. [13:41:26] Coren: had I not made 6 idiot coding typos it would have been half the time [13:41:43] page_ns vs page_namespace [13:42:04] Heh. You should spin that off to its own tool and grab (more) maintainers. :-) [13:42:13] forgetting the .value for a cgi dict [13:44:21] Coren: its like 20 lines of code.... [13:45:00] If I cant maintain that we have some serious problems [13:46:01] And I hate forking, it leads to fork bombs :P [13:48:38] Coren: Im a coder, I can do most of this stuff in my sleep. Its not a matter of waiting for me to finish, its me waiting on a project. [16:19:31] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597#c5 (10Chris McMahon) Meta: one of the themes for RelEng coming up is improving Jenkins performance, this list of issues is a good place to start. [16:34:21] Hi, where can I find a list of currently running tools? [16:38:17] harej: What do you mean by "running"? Web applications are listed at http://tools.wmflabs.org/, and jobs run by bots & Co. can be seen at http://tools.wmflabs.org/?status (for the time they are actually running). [16:39:08] I mean more along the lines of web apps that run on Tool Labs [16:39:16] Edit counters, things like that [16:43:45] scfc_de ^ [16:44:45] harej: Web applications are listed [16:44:52] at http://tools.wmflabs.org/. [16:45:59] petan: wm-bot seems to have a hiccup: [16:46:03] @notify petan [16:46:03] You've already asked me to watch this user [16:46:18] @notify harej [16:46:18] This user is now online in #wikimedia-tech. I'll let you know when they show some activity (talk, etc.) [16:46:39] scfc_de: thank you [16:47:12] petan: Ha! "Vorführeffekt". In a /query, wm-bot adds "Hi, I am robot, this command was not understood. [...]" [17:02:39] harej: you want tools.w.o/directory/ I think [17:03:28] Not necessarily. What I would really like though would be analytics/metrics tools [17:03:33] In addition to Wikimetrics, of course [17:04:59] so directory? :) [17:08:05] Coren: you around? [17:08:14] What's up? [17:08:33] Im still seeing a lot of TweetmemeBot UA's in my access logs [17:09:04] For a dead business, they sure do a lot of spidering. [17:09:11] 2400 hits [17:10:03] There's probably a range of theirs that I missed. I'll take a look at it later today. [17:10:18] ~20% of my traffic [17:18:18] Coren: I guess Im one of the few who review my access logs :P [17:19:06] Betacommand: Or you're one of the few whose tool(s) have internal links from dynamic content to dynamic content that can form closed graphs. [17:21:59] closed graphs? [17:23:24] Coren: ^ [17:24:16] Betacommand: Where it is possible from following links to return to a page visited earlier. If the crawler is stupid (and there is every indication that one is) then it'll never be "finished" crawling the site because there are infinitely many links to follow. [17:27:41] Coren: I really dont have may if any of those [17:27:47] *many [17:28:36] Closed graphs would show up in access.log as well. [17:30:13] scfc_de: correct, they would also increase the log volume too [17:44:15] Hi all - any chance someone can take a look at https://bugzilla.wikimedia.org/show_bug.cgi?id=68387 for SSL on betalabs? [17:45:07] and add me to deployment-prep roots :) [17:45:17] (projectadmin) [17:46:28] Hey folks. I'm struggling to mount /data/project [17:46:40] I followed the instructions here: https://wikitech.wikimedia.org/wiki/Help:Shared_storage [17:46:49] Anyone interested in helping me troubleshoot? [17:46:57] scfc_de: Coren I should add UA based blocking to dynamicproxy at some point [17:46:59] shouldn't be too hard [17:47:10] halfak: Oy! That page is completely, entirely and irremediably out of hand. [17:47:18] YuviPanda|zzzz: That may be useful for some rogue bots. [17:47:20] Seems like it [17:47:24] indeed [17:47:29] shouldn't be too hard either [17:48:06] halfak: The correct thing to do to get /data/project is actually simple. (a) ignore that page, (b) make sure you have project storage turned on in your project's configuration and (c) let puppet do its thing; /data/project will appear automagically. :-) [17:48:08] halfak: just refer to fstab? [17:48:23] ohhh [17:48:27] Gotcha. Checking config on wikitech [17:48:36] halfak: Possibly undo the damage caused by that outdated help page with a 'apt-get purge autofs5' first. :-) [17:49:00] I was thinking he didn't get the new mount yet after the move [17:49:07] YuviPanda|zzzz: I had some ad-hoc stuff in Apache, so that's definitely useful (blocking at the proxy is cheap). [17:49:17] scfc_de: yeah [17:49:23] scfc_de: plus we can keep the list in puppet :) [17:50:19] YuviPanda|zzzz: Exactly. (Maybe even sync with the main cluster.) [17:50:32] Coren, do you know the name of the config I'm looking for off-hand? [17:50:53] scfc_de: ah, yes, if there's one [17:51:12] halfak: There are only four options or so?! :-) [17:51:14] halfak: https://wikitech.wikimedia.org/w/index.php?title=Special:NovaProject&action=configureproject [17:51:49] But you really want to follow the 'configure' link from https://wikitech.wikimedia.org/w/index.php?title=Special:NovaProject [17:52:01] (In the rightmost column, under 'Actions') [17:52:29] OH! I was looking within the instance. [17:52:47] "Create shared project storage"? [17:52:54] * Coren nods. [17:53:06] Nothing to do on the instance then? [17:53:16] You almost certainly want shared homes too. Honestly, those should both default to on now that we have gotten rid of gluster. [17:53:51] halfak: Nope; the presence of project-wide storage is... project-wide. :-) But make sure you purge all traces of autofs if you installed it at any point. [17:59:07] Thanks Coren. :) [18:15:10] (03CR) 10Jforrester: [C: 032] preprocess: Clean up arbitrary quoting of object key 'message' [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/156880 (owner: 10Krinkle) [18:15:13] (03Merged) 10jenkins-bot: preprocess: Clean up arbitrary quoting of object key 'message' [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/156880 (owner: 10Krinkle) [18:23:19] halfak: I also remove the old cruft from the help page. :-) [18:23:37] <3 I was going to leave a notice. [19:01:08] andrewbogott_afk: ping when you get back. [19:40:46] 3Wikimedia Labs / 3deployment-prep (beta): Add RecentActivityFeed extension to beta labs - 10https://bugzilla.wikimedia.org/69785 (10Sage Ross) [19:53:58] 3Wikimedia Labs / 3deployment-prep (beta): Add RecentActivityFeed extension to beta labs - 10https://bugzilla.wikimedia.org/69785#c11 (10Sage Ross) Thanks everyone. Complete checklist items: *MW extension page: https://www.mediawiki.org/wiki/Extension:RecentActivityFeed *Gerrit: https://gerrit.wikimedia.or... [20:08:46] !log deployment-prep updated OCG to version c9a2b4cf2502479eeabed07ab2de728695d96e46 [20:08:50] Logged the message, Master [20:15:46] 3Wikimedia Labs / 3tools: Migrate Tools access request process to Phabricator - 10https://bugzilla.wikimedia.org/70625 (10Tim Landscheidt) 3NEW p:3Unprio s:3enhanc a:3Marc A. Pelletier Quim demonstrated recently that Phabricator allows different bug forms for different use cases. If feasible, this w... [20:16:29] 3Wikimedia Labs / 3tools: Migrate Tools access request process to Phabricator - 10https://bugzilla.wikimedia.org/70625 (10Tim Landscheidt) p:5Unprio>3Low [20:19:16] 3Wikimedia Labs / 3wikitech-interface: Migrate new Labs projects request process to Phabricator - 10https://bugzilla.wikimedia.org/70626 (10Tim Landscheidt) 3NEW p:3Unprio s:3enhanc a:3None As per bug #70625, if possible, moving the process to Phabricator seems like a good idea. This would probably... [20:19:58] 3Wikimedia Labs / 3wikitech-interface: Migrate new Labs projects request process to Phabricator - 10https://bugzilla.wikimedia.org/70626 (10Tim Landscheidt) p:5Unprio>3Low [20:26:01] 3Wikimedia Labs / 3wikitech-interface: Migrate shells access request process to Phabricator - 10https://bugzilla.wikimedia.org/70627 (10Tim Landscheidt) 3NEW p:3Unprio s:3normal a:3None As per bug #70625, it would be nice to move the shell access request process to Phabricator. In the same manner as... [20:26:29] 3Wikimedia Labs / 3wikitech-interface: Migrate shell access request process to Phabricator - 10https://bugzilla.wikimedia.org/70627 (10Tim Landscheidt) p:5Unprio>3Low s:5normal>3enhanc [20:29:44] 3Wikimedia Labs / 3tools: tools: "git error: server certificate verification failed" for git.wikimedia.org on tools-login-eqiad - 10https://bugzilla.wikimedia.org/62432#c6 (10Tim Landscheidt) 5NEW>3RESO/WOR I can no longer reproduce this on tools-login or tools-dev. I'm not aware that someone consciousl... [20:31:14] 3Wikimedia Labs / 3tools: Migrate Tools access request process to Phabricator - 10https://bugzilla.wikimedia.org/70625#c1 (10Marc A. Pelletier) That seems to specific to be generally useful, but how about a "make a MW API call" extension? [20:35:34] andrewbogott_afk: Coren: wikitech is still broken from the upgrade a weekish ago [20:35:47] and morebots is still monkeypatched to compensate [20:44:31] 3Wikimedia Labs / 3wikitech-interface: wikitech strict warnings on API save - 10https://bugzilla.wikimedia.org/70628 (10jeremyb) 3NEW p:3Unprio s:3normal a:3None Created attachment 16420 --> https://bugzilla.wikimedia.org/attachment.cgi?id=16420&action=edit morebots-patch-wikitech-strict-warning-20... [20:45:34] ok, filed [20:55:45] 3Wikimedia Labs / 3wikitech-interface: wikitech strict warnings on API save - 10https://bugzilla.wikimedia.org/70628#c1 (10Tim Landscheidt) (I'm pretty sure this is the same issue, but am not confident.) [20:55:45] 3Wikimedia Labs / 3wikitech-interface: Wikitech: Performing content actions results in PHP strict warning by MWSearch outputted on the page - 10https://bugzilla.wikimedia.org/70436 (10Tim Landscheidt) [20:57:10] jeremyb: I assume your fix to morebots is future-compatible, i. e. if the bug is fixed, it's a no-op? [21:01:37] scfc_de: that was the idea. but no guarantees. the fix is deployed for days and no complaints yet [22:12:44] 3Tool Labs tools / 3Matthewrbowker's tools: MATTHEWRBOWKER-1 Add functionality for !link - 10https://bugzilla.wikimedia.org/59074#c2 (10Matthew Bowker) 5NEW>3RESO/WON Not going to fix this, as the bot itself is no longer active. [23:10:48] <^d> beta-mediawiki-config-update-eqiad jobs are stuck. [23:10:58] <^d> ETA: unknown, queued 4 hr 43 min ago [23:10:59] <^d> beta-mediawiki-config-update-eqiad [23:11:33] <^d> So I'm guessing something's screwy with beta. [23:12:42] <^d> beta-scap-eqiad is "pending—Waiting for next available executor on deployment-bastion.eqiad" [23:13:03] blerg [23:13:14] I fixed the same thing last nigt [23:14:09] ^d: Want to be the first tester of the instructions I wrote to fix that? https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code.2Fdb_update [23:14:48] <^d> Lemme finish swatting [23:15:15] It's that stupid db updater job that hung it again :( [23:15:37] ^d: I'll fix it. I'm sure there will be another chance to test the instructions :/ [23:24:40] !log integration Restarted jenkins slave on deploymnet-bastion multiple times to fix "waiting for executor" problem [23:24:43] Logged the message, Master [23:28:46] 3Wikimedia Labs / 3wikitech-interface: [Regression] WMFLabs: Nova project quota broken - 10https://bugzilla.wikimedia.org/70634 (10Krinkle) 3NEW p:3Unprio s:3normal a:3None I'm no longer able to see the available quota for my projects. https://wikitech.wikimedia.org/w/index.php?title=Special:NovaPro... [23:30:30] 3Wikimedia Labs / 3tools: Can't delete NovaProxy instance with malformed DNS hostname - 10https://bugzilla.wikimedia.org/67927#c1 (10Krinkle) Might be related to bug 62770. [23:33:56] Coren: andrewbogott_afk: I'm unable to delete any instance and can't see available quota. [23:34:16] will check back tomorrow, hopefully I can continue then. [23:34:18] calling it a day :) [23:34:20] o/ [23:38:17] !log integration Enabled "Throttle Concurrent Builds" for beta-update-databases-eqiad in an attempt to keep it from hanging all executors on deployment-bastion. Change only made via Jenkins interface, not JJB. [23:38:21] Logged the message, Master [23:46:15] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597#c6 (10Bryan Davis) 5RESO/FIX>3REOP p:5Highes>3Normal a:3Bryan Davis This happened again today. It seems like it always involves the data... [23:46:17] 3Wikimedia Labs / 3wikitech-interface: [Regression] WMFLabs: Unable to delete any instance - 10https://bugzilla.wikimedia.org/70636 (10Krinkle) 3NEW p:3Unprio s:3normal a:3None I'm no longer able to delete any instances. 1. Go to https://wikitech.wikimedia.org/wiki/Nova_Resource:I-000004b5.eqiad.wmf...