[00:13:12] Question: is it possible to send something to Gerrit (by way of git review) directly from Labs? [00:13:38] theoretically [00:13:56] When I do it, I keep getting authentication failures, even though I know I am typing my Gerrit password correctly. [00:14:16] you need to go to your gerrit settings and generate an HTTP password [00:14:30] also go and change your password now that you've exposed it to a labs instance [00:17:25] then when you're done uploading from the instance, clear or regenerate the HTTP pass [00:17:47] is exposing a password to a labs instance insecure? [00:18:29] exposing your real ldap password, yes [00:20:04] With that a person can access any instances you can access, if you're a project admin create new instances, in gerrit create and merge changes if you have that ability, etc [00:20:28] edit wikitech [00:20:38] use any of your other wikitech privileges [00:20:44] same on phab [00:21:03] in the case of some people it may be an NDA violation [00:21:07] Probably wouldn't take too long for that attacker to be found out if they went around destroying stuff, but they could still do *a lot* of damage [00:24:20] Probably wouldn't make much difference if you have 2FA enabled either. That only seems to affect some things [00:24:34] Like wikitech, and horizon. [00:28:59] I'd hope any ops or admins of project-proxy/bastion/etc. have 2FA enabled, though it's only really enforced for cloudadmins [00:34:46] 06Labs, 10Wikimedia-Site-requests, 10wikitech.wikimedia.org, 13Patch-For-Review: Allow wikitech to write files for Math - https://phabricator.wikimedia.org/T126628#2603926 (10Dereckson) [00:41:00] It's enforced for anyone that uses Horizon. [00:41:20] So most project admins I'd have thought [00:51:07] 06Labs, 10Wikimedia-Site-requests, 10wikitech.wikimedia.org, 13Patch-For-Review: Enable math extension on wikitech - https://phabricator.wikimedia.org/T126338#2603976 (10Dereckson) Extension works, but can't currently write files, as `wgMathDirectory` was forced to `/mnt/upload7/math`. A puppet and a conf... [01:17:00] 06Labs, 10Wikimedia-Site-requests, 10wikitech.wikimedia.org, 13Patch-For-Review: Allow wikitech to write files for Math - https://phabricator.wikimedia.org/T126628#2604022 (10Dereckson) Changes have been deployed. We've still a math_output_error error returned here: ```lang=php,name=MathTexvc.php... [01:21:31] PROBLEM - Free space - all mounts on tools-web-static-02 is CRITICAL: CRITICAL: tools.tools-web-static-02.diskspace._srv.byte_percentfree (<100.00%) [03:18:15] Hi [03:20:54] Anyone? [03:44:16] 06Labs, 10DBA, 06Editing-Analysis, 05Security: Replicate editor_month table from analytics-store to Labs - https://phabricator.wikimedia.org/T143955#2604140 (10Bawolff) >Interesting. I don't know about the original use case, but from what I've heard, the main use case now is gender-correct interfaces, whic... [05:17:31] Hey? [05:25:20] 06Labs, 10DBA, 06Editing-Analysis, 05Security: Replicate editor_month table from analytics-store to Labs - https://phabricator.wikimedia.org/T143955#2604220 (10jcrespo) > (1) they would either be much harder to use or much less discoverable Lots of people use the monuments database or the external links... [05:28:41] 06Labs, 10DBA, 06Editing-Analysis, 05Security: Replicate editor_month table from analytics-store to Labs - https://phabricator.wikimedia.org/T143955#2604221 (10jcrespo) To clarify the difference between redactatron (the replication filtering) and maint-replicas (what has issues) see: T138450#2402968 and ht... [05:36:14] RECOVERY - Puppet run on tools-static-11 is OK: OK: Less than 1.00% above the threshold [0.0] [07:01:26] Platonides: yeah there are multiple typos typical of german family language speakers :) [07:16:16] 10Striker, 06Project-Admins: Allow self-service creation of Maniphest projects for Tools - https://phabricator.wikimedia.org/T144111#2604286 (10Aklapper) >>! In T144111#2601969, @bd808 wrote: >>>! In T144111#2600712, @Peachey88 wrote: >> We should keep this naming scheme, It easy for people to detect that its... [07:35:49] PROBLEM - Free space - all mounts on tools-services-01 is CRITICAL: CRITICAL: tools.tools-services-01.diskspace.root.byte_percentfree (<11.11%) [07:37:15] 10Striker, 06Project-Admins: Allow self-service creation of Maniphest projects for Tools - https://phabricator.wikimedia.org/T144111#2588823 (10yuvipanda) Can confirm that the current naming convention is just an artifact of bugzilla migration, and I do favor a shorter name. Thanks for working on it! [07:48:52] 06Labs, 10Tool-Labs: BUB 503: AttributeError: 'module' object has no attribute 'python_2_unicode_compatible' - https://phabricator.wikimedia.org/T144554#2604324 (10Nemo_bis) [11:08:47] Nemo_bis i've hit a bit of a wall yesterday, while trying to figure out why one table isn't accessible in labs when it is through a special page ^^' [11:09:00] (global blocks) [11:09:21] (ok, technically the whole database is inaccessible :D ) [11:36:54] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:02:58] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: DB operations in Labs hitting sqlalchemy limits - https://phabricator.wikimedia.org/T144339#2604892 (10chasemp) 05Open>03Resolved p:05Triage>03High a:03chasemp Fixes merged let's see if we run into this again [14:44:51] chasemp: should we move some more jobs to Nodepool ? :] [14:45:18] hashar: if you want to do 1 revert today seems ok to me [14:45:23] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: DB operations in Labs hitting sqlalchemy limits - https://phabricator.wikimedia.org/T144339#2605028 (10hashar) [14:45:49] chasemp: with that SQLAlchemy pool starvation, I now better understand why you wanted to closely monitor the infra :] [14:46:11] that won't be the last thing that isn't setup for this load but yes [14:49:40] hashar: can you tell me what trusty jobs ran overnight that spiked wait times a bit? [14:49:47] it was max 600s I think and very short time but [14:49:52] what is the overnight load? [14:53:30] hmm [14:56:10] the wait time also take in account the time a change is idling in Zuul [14:57:01] https://grafana.wikimedia.org/dashboard/db/releng-gerrit sometime gives clue [14:57:10] it shows # of events in Gerrit [14:58:21] but really if I looked at https://grafana.wikimedia.org/dashboard/db/nodepool from 22:00 to 23:00 utc yesterday and after midnight the pool was quite busy [15:11:16] chasemp: I will move the tox / python jobs back [15:11:36] what is the load in relation to existing? [15:11:40] jobs per day or per hour or something [15:12:38] https://grafana.wikimedia.org/dashboard/db/zuul-job?var-pipeline=All&var-job=*tox*&var-status=All [15:12:46] peak of 25 over an hour [15:13:16] but typically less than 15 per hours [15:13:43] (the graph at bottom right is per week with one hour buckets) [15:13:46] err [15:13:51] (the graph at bottom LEFT is per week with one hour buckets) [15:13:57] bottom right is 2 months and daily buckets [15:14:09] the top graphs show the history of builds on permanent vs nodepool slaves [15:14:25] ok let's see how it goes, are you going to be around for a bit in case of revert? [15:16:24] patch is https://gerrit.wikimedia.org/r/#/c/306725/ [15:16:38] can be revert by pressing the revert, cr+2 then fab deploy_zuul [15:16:45] which anyone in #wikimedia-releng can do easily [15:16:52] the jobs are all up to date [15:16:58] I will leave a comment on the task [15:19:23] k [15:19:40] did [15:20:49] the way Kunal reverted it made it very easy to revert again [15:20:56] no jenkins job have been altered [15:21:15] k [15:23:38] I will leave soonish for dinners etc but will show up again in my evening [15:24:31] alright man later on [15:25:07] also [15:25:27] PROBLEM - Free space - all mounts on tools-web-static-01 is CRITICAL: CRITICAL: tools.tools-web-static-01.diskspace._srv.byte_percentfree (<100.00%) [15:25:46] I noticed the median time of OpenStack API response for server creations went from <200 ms to around 750ms [15:25:47] https://grafana.wikimedia.org/dashboard/db/nodepool?from=1470237865598&to=1472829865598&var-provider=All&var-task=CreateServerTask&var-task=DeleteKeypairTask&var-task=ListFloatingIPsTask&var-task=ListServersTask&panelId=4&fullscreen [15:26:02] not sure how it is relevant anyway [15:26:36] oh [15:26:38] i found out [15:26:53] that is because the instances could not spawn / the API would yield a 403 constantl [15:26:58] hence the faster response [15:27:42] that's a bit of a problem [15:28:14] hashar: is that a quota issue or what? [15:29:17] ah I see [15:29:19] yes [15:29:29] that was a red herring there in those days [15:29:35] it was fast to do nothing at all [15:40:05] yeah :] [15:40:22] meanwhile, I have added to https://grafana.wikimedia.org/dashboard/db/continuous-integration a pie chart of the nodepool capacity [15:40:34] showing the various status and the free space against the quota [15:41:07] there is a tiny box with an arrow that spawn a button which link to the history graph [15:44:49] be back later! [16:06:29] (03CR) 10Jean-Frédéric: [C: 032] Importing daily-uploads [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303933 (https://phabricator.wikimedia.org/T142570) (owner: 10EdouardHue) [16:07:09] (03Merged) 10jenkins-bot: Importing daily-uploads [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303933 (https://phabricator.wikimedia.org/T142570) (owner: 10EdouardHue) [16:24:19] !log tools.heritage Deployed latest from Git: 1603efb (T142570) [16:24:20] T142570: Import 'Daily_uploads' tool to heritage project - https://phabricator.wikimedia.org/T142570 [16:24:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [18:13:06] 06Labs: Request increased quota for reading-web-staging labs project - https://phabricator.wikimedia.org/T144623#2605517 (10Jdlrobson) [18:15:12] 06Labs, 10Labs-Infrastructure, 10Tool-Labs, 07Wikimedia-Incident: Write a simple script that handles failovering proxies - https://phabricator.wikimedia.org/T143639#2574425 (10madhuvishy) While switching over floating ips using openstack floating ip add, remember to prefix with OS_TENANT_NAME=... [18:22:23] 06Labs: Request increased quota for reading-web-staging labs project - https://phabricator.wikimedia.org/T144623#2605558 (10bd808) stack should really be moved to tool labs I think. I'd be glad to work with someone (@Jhernandez?) to get it running on Tool Labs. Is it taking up a whole VM at the moment or just co... [18:29:30] 06Labs: Request increased quota for reading-web-staging labs project - https://phabricator.wikimedia.org/T144623#2605587 (10Jdlrobson) @bd808 is pretty small. Also @phuedx can you confirm that we use reading-web-staging-2 for testing Central Notice? (I notice that's a large vm which is probably overkill but I'm... [19:10:55] jdlrobson: I am many things, but small is typically not one of them :) [19:11:14] lol [19:51:51] 06Labs, 06Operations: Set up monitoring for secondary labstore HA cluster - https://phabricator.wikimedia.org/T144633#2605766 (10madhuvishy) [20:06:10] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 0.57 ms [20:12:26] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 0.64 ms [20:13:08] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [20:19:35] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [20:25:07] !log ores deployed ores-wmflabs-deploy:751860f [20:25:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [20:37:25] 06Labs, 10DBA, 06Editing-Analysis, 05Security: Replicate editor_month table from analytics-store to Labs - https://phabricator.wikimedia.org/T143955#2605880 (10Neil_P._Quinn_WMF) >>! In T143955#2604220, @jcrespo wrote: > > Lots of people use the monuments database or the external links database. Those are... [21:11:19] Hello. Will we get any for of nagios or Icinga on tool labs? Alternatively, is it OK for me to set up a monitoring service to just ping my tools? [21:12:38] yuvipanda: -en is requesting deletion of https://quarry.wmflabs.org/query/11795 - do you have the powah? [21:13:38] I can use that list to nuke those accounts :P [21:14:29] that seems more fun :P [21:15:20] MarcoAurelio: would you be interested to get a ping on -stewards when a new account matching those names is registered? [21:15:50] Platonides: I'm not 24/7 connected [21:16:30] Platonides: Oshwah is requesting that now, as a matter of fact. Just as an enwiki admin, though. [21:16:38] * Platonides was assuming that some other steward would do it [21:16:56] Just started the discussion in #countervandalism [21:17:45] I made the bot to match a few vandal names [21:17:55] a couple more patterns would be no problem [21:20:13] All those accounts are globally locked [21:20:18] no fun for me :( [21:20:27] Platonides: it'd be good I think [21:22:33] ok [21:22:36] "detected signup of Incorrigible Troll sock «$username» @$wiki" [21:23:11] Platonides: we have #cvn-unifications though [21:23:30] added [21:35:36] Platonides : you're gonna be disappointed : global blocks can't be shown, the database they reside in is entirely inaccessible [21:35:54] Alphos ? [21:36:10] Why do you think I would need that? [21:36:49] you seemed to be happy i was working on blockfinder yesterday [21:37:22] ah [21:37:32] in the context of that [21:37:39] yep [21:42:13] Alphos, you can't get globalblocks data in labs replicas? [21:43:00] Krenair not yesterday anyway [21:43:07] not through mysql-client anyway [21:43:51] krenair@tools-bastion-03:~$ mysql -s -h labsdb1003.eqiad.wmnet centralauth_p -e 'select count(*) from globalblocks;' [21:43:51] count(*) [21:43:51] 5324 [21:45:30] hold on, i'll try on that host [21:46:12] it shouldn't matter which host you run it from [21:46:57] oh, i see the issue [21:47:04] ? [21:47:05] i tried in the db "globalblocks" [21:47:15] yeah that won't work [21:47:22] there goes my faith in the documentation ! :D [21:47:38] which docs? [21:47:51] for the global blocks extension [21:48:04] the global blocks extension doesn't enforce a particular database name [21:48:11] you can use any database name you want [21:48:15] we use centralauth [21:48:26] yah, i just assumed we were using the same ^^' [21:53:43] marktraceur hey! yes, I can delete it, I'll need some form of confirmation from someone who is steward or whatever [21:53:47] maybe write on my meta page? [22:08:01] Hi, are there known issues with phab-01.wmflabs.org? [22:08:18] I get an error trying to open an existing task [22:09:18] "Unhandled Exception ("InvalidArgumentException") [22:09:18] Argument 4 passed to ManiphestTaskDetailController::buildPropertyView() must be of the type array, object given, called in /srv/phab/phabricator/src/applications/maniphest/controller/ManiphestTaskDetailController.php on line 68 and defined" [22:09:20] ?? [22:09:21] volans what error? [22:09:46] Oh [22:09:47] wait [22:09:48] paladox: the same just pasted [22:09:50] :) [22:09:54] Ive reproduced [22:10:05] volans was related to the update im testing from upstream [22:10:11] before we deploy it to production [22:11:28] I guessed something like that [22:11:42] I will see if there is a fix upstream and hotfix it for now [22:16:58] in our current code at line 337 is just $handles or array $handles paladox ? [22:17:03] 25 [22:17:12] Not sure [22:17:19] maybe one of our custom projects [22:17:21] are doing that [22:17:23] ie extensions [22:17:45] yuvipanda: Okay, I think maybe Oshwah wants it more than I do. [22:17:49] looks like there is no requirement of being an array in the signature at least on the HEAD of github [22:17:54] https://github.com/phacility/phabricator/blob/master/src/applications/maniphest/controller/ManiphestTaskDetailController.php#L337 [22:18:39] volans his is what it looks for me [22:18:40] $details = $this->buildPropertyView($task, $field_list, $edges, $handles); [22:19:05] paladox: I was referring to line 337, where teh buildPropertyView() is defined [22:19:13] Oh [22:19:33] the error says the 4th argument must be array [22:19:43] but at least in the HEAD on github is not in the signature [22:19:49] of the function [22:19:54] Oh [22:19:59] we use stable branch [22:20:03] not master branch [22:20:38] Oh wait twentyafterfour do a fix for task graphs [22:20:48] maybe when merging it didnt correctly merge properly [22:21:03] Hola, I have been pinged/summoned? [22:21:11] * Oshwah wishes everyone here a Happy Friday!!! [22:21:17] Happy Friday, Oshwah [22:21:45] * Oshwah tips his hat towards Platonides [22:21:53] Thank you, good sir! [22:22:09] https://phabricator.wikimedia.org/rPHABcf12fdf248df82dc414d96bddd147c058bc3d636 [22:22:58] volans ^^ [22:22:58] ohhh the signature changed, now has 5 parameters [22:23:05] Oh? [22:23:07] you added array $edge_types, [22:23:07] 279 [22:23:35] go me, i will apply [22:23:40] 20after4 change [22:23:55] and line 68 was deleted [22:23:59] Yep [22:24:15] so either we have this version or the old one, a mix of both I guess doesn't work ;) [22:24:57] Yep [22:28:56] 10Tool-Labs-tools-stewardbots: General update of HTML and CSS for stewardbot tools and portals - https://phabricator.wikimedia.org/T128745#2606116 (10MarcoAurelio) Updated stewardbot. Next to update: SULWatcher. [22:30:22] volans ok applying it isent hard now [22:30:26] there are alot of changes [22:31:02] isent = is [22:31:10] lol :) [22:31:48] sure, no problem. I wanted to test something but I can do it on Monday [22:32:00] volans no i mean i can do [22:32:03] it but it is har [22:32:05] hard [22:32:27] volans it does [22:32:27] if (!$task_graph->isEmpty()) { [22:32:28] $parent_type = ManiphestTaskDependedOnByTaskEdgeType::EDGECONST; [22:32:28] $subtask_type = ManiphestTaskDependsOnTaskEdgeType::EDGECONST; [22:32:28] $parent_map = $task_graph->getEdges($parent_type); [22:32:28] $subtask_map = $task_graph->getEdges($subtask_type); [22:32:29] $parent_list = idx($parent_map, $task->getPHID(), array()); [22:32:31] $subtask_list = idx($subtask_map, $task->getPHID(), array()); [22:32:33] $has_parents = (bool)$parent_list; [22:32:35] $has_subtasks = (bool)$subtask_list; [22:32:37] $search_text = pht('Search...'); [22:32:42] now [22:32:44] $message = pht( [22:32:45] $edge_types += array( [22:32:47] ManiphestTaskDependedOnByTaskEdgeType::EDGECONST [22:32:50] => pht('Parent Tasks'), [22:32:52] ManiphestTaskDependsOnTaskEdgeType::EDGECONST [22:32:54] => pht('Subtasks'), [22:32:56] ); [22:32:58] not ^^ [22:33:06] woops i mean [22:33:08] I'm wondering why it's in a strange state [22:33:27] if (!$task_graph->isEmpty()) { [22:33:27] if ($task_graph->isOverLimit()) { [22:33:27] $edge_types += array( [22:33:28] ManiphestTaskDependedOnByTaskEdgeType::EDGECONST [22:33:28] => pht('Parent Tasks'), [22:33:28] ManiphestTaskDependsOnTaskEdgeType::EDGECONST [22:33:30] => pht('Subtasks'), [22:35:41] Platonides could I be unblocked please [22:35:43] ? [22:35:50] paladox: ? [22:35:52] IRC has blocked my home ip [22:36:02] that'll be FreeNode [22:36:05] nothing we can do [22:36:13] Oh [22:36:25] paladox: my point is it should work with bot standard code or patched one but I'm not getting in which state is now. A branch? a tag? randomly modified manually? [22:36:42] and yeah, freenode doesn't like much pasting ;) [22:36:44] O [22:36:47] Oh [22:36:50] paladox: you pasted too much [22:37:13] 38 lines… [22:37:38] Oh [22:37:39] given the topic you could have created a paste on phab-01.wmflabs.org :-P [22:37:49] Oh [22:37:59] Yep I didn't know I would be blocked for doing that [22:38:26] * volans thinks paladox was replaced by a bot that replies "Oh" to anyone :D [22:38:41] Lol [22:40:13] * Platonides opens paladox() and sees: $answers = array("Oh", "Lol"); echo $answers[rand() % count($answers)]; [22:40:40] lol lol [22:46:18] xD [22:46:32] return (0); [22:46:42] Im back [22:49:07] the real one or the bot? [22:49:09] Yay me. [22:49:13] Im not a bot [22:49:17] looks more real now :D [22:49:21] i doint know how to programe a bot. [22:51:21] * Luke081515 knows :P [22:51:37] write a framework => use it [22:51:39] sounds easy :D [22:51:52] LOL [22:52:43] 06Labs, 06Discovery, 06Maps: Maps-wma1 not responding please reboot - https://phabricator.wikimedia.org/T144646#2606183 (10MaxSem) 05Open>03Resolved a:03MaxSem Rebooted, can ssh into it now. [22:54:56] volans the problem was caused by https://github.com/phacility/phabricator/commit/15c7eb142570360df4d071dc716c2e70b0191886 [22:55:43] paladox: the performance issue? [22:56:03] volans Yes, it not dosent show if there are more then 100 tasks [22:56:17] which means we have to bring our customisation for parent / sub back [22:56:18] because I don't see in that patch any changes to the line we're seeing the error [22:56:31] since we doint want it being removed due to the amout of parent or sub tasks we have [22:56:43] just we doint want task graph past a certain amout of tasks [22:56:44] ahh we had a patch for the same thing before already [22:57:00] yeh [22:57:01] here [22:57:01] https://phabricator.wikimedia.org/rPHABcf12fdf248df82dc414d96bddd147c058bc3d636 [22:57:14] i just need to update it to suppor the new code added [22:57:32] ok now make sense why the upgrade left it mixed [22:57:54] git is not *that* smart unfortunately [22:58:13] volans lets try some new things [22:58:22] Im applying manually now [22:59:35] from the commit comment it looks like their patch is doing pretty much what our does (I didn't check the code though) [22:59:48] volans it actually isent [22:59:53] theres cuts it off complety [23:00:08] When a task graph has too much stuff, only show adjacent nodes (direc… [23:00:09] …t parents/children) [23:00:13] ours cuts task graphs and replace it with a basic parent / sub tasks like it was before. [23:00:17] the commit says :) [23:00:25] if it receaches a certain amout of tasks [23:00:37] ah ok [23:00:41] we ereach 100? [23:00:56] s/ereach/reach/ [23:04:16] (03PS1) 10MarcoAurelio: Update SULWatcher information page [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/308292 [23:05:42] volans it is 50 tasks for us due to performance and the size paste 50 [23:06:12] ok [23:06:23] it starts to be really late here [23:06:29] I'll probably go very soon [23:06:48] (03CR) 10MarcoAurelio: [C: 032] Update SULWatcher information page [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/308292 (owner: 10MarcoAurelio) [23:06:54] volans oh it is midnight here [23:07:02] (03Merged) 10jenkins-bot: Update SULWatcher information page [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/308292 (owner: 10MarcoAurelio) [23:09:59] !log tools.stewardbots [[gerrit:308292|Updated HTML information page for SULWatcher]] (T128745) [23:10:00] T128745: General update of HTML and CSS for stewardbot tools and portals - https://phabricator.wikimedia.org/T128745 [23:10:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [23:13:50] paladox: thanks for looking into it, let me know if you need anything else from me, I'll probably be around tomorrow at some point [23:14:17] ok [23:14:23] Guess it is late there too [23:14:29] 00:14am here [23:14:35] volans ^^ [23:15:03] add 1h for me ;) [23:15:28] Oh your 1am [23:15:34] Your in france? [23:15:55] volans ^^ [23:15:56] nope, Italy atm [23:16:04] Oh [23:16:12] Guess you speak italian and eat pizza [23:16:20] rotfl [23:16:25] how comes there is no pizza hut there but they are know for there pizza [23:16:52] volans ^^ [23:17:51] lol [23:17:55] (03PS1) 10BryanDavis: Set default STAFF_GROUP_DN == SUPERUSER_GROUP_DN [labs/striker] - 10https://gerrit.wikimedia.org/r/308295 [23:17:57] (03PS1) 10BryanDavis: Wrap MilestoneManager.recordMilestone in a transaction [labs/striker] - 10https://gerrit.wikimedia.org/r/308296 (https://phabricator.wikimedia.org/T144296) [23:17:59] (03PS1) 10BryanDavis: Register an admin model to view admin log entries [labs/striker] - 10https://gerrit.wikimedia.org/r/308297 [23:18:06] yep [23:18:11] time to go paladox, ttyl [23:18:15] ok [23:18:16] thanks again [23:18:26] your welcome [23:18:30] im still working on it [23:19:05] (03CR) 10Alex Monk: [C: 032] Set default STAFF_GROUP_DN == SUPERUSER_GROUP_DN [labs/striker] - 10https://gerrit.wikimedia.org/r/308295 (owner: 10BryanDavis) [23:19:36] volans [23:19:38] see https://phab-01.wmflabs.org/T75 [23:19:43] i think i may have fixed it [23:20:26] (03Merged) 10jenkins-bot: Set default STAFF_GROUP_DN == SUPERUSER_GROUP_DN [labs/striker] - 10https://gerrit.wikimedia.org/r/308295 (owner: 10BryanDavis) [23:21:47] Nope still broken [23:21:47] https://phab-01.wmflabs.org/T68 [23:25:40] (03CR) 10Alex Monk: [C: 032] Wrap MilestoneManager.recordMilestone in a transaction [labs/striker] - 10https://gerrit.wikimedia.org/r/308296 (https://phabricator.wikimedia.org/T144296) (owner: 10BryanDavis) [23:26:49] (03Merged) 10jenkins-bot: Wrap MilestoneManager.recordMilestone in a transaction [labs/striker] - 10https://gerrit.wikimedia.org/r/308296 (https://phabricator.wikimedia.org/T144296) (owner: 10BryanDavis) [23:30:46] (03CR) 10Alex Monk: [C: 032] Register an admin model to view admin log entries [labs/striker] - 10https://gerrit.wikimedia.org/r/308297 (owner: 10BryanDavis) [23:31:15] thanks Krenair :) [23:31:35] (03Merged) 10jenkins-bot: Register an admin model to view admin log entries [labs/striker] - 10https://gerrit.wikimedia.org/r/308297 (owner: 10BryanDavis) [23:53:33] Matthew_ hi phab-01 should now be fixed :) [23:55:13] paladox: Thanks :) [23:55:30] Your welcome :), twentyafterfour helped :) [23:57:15] PROBLEM - SSH on tools-exec-1219 is CRITICAL: Server answer