[01:01:02] <ryankemper>	 https://wikitech.wikimedia.org/wiki/Incident_documentation/20200723-wdqs-outage is the first draft of the writeup for today's (23/07/2020 DD/MM/YYYY) wdqs outage is ready for review
[01:29:12] <cdanis>	 thanks ryankemper I'll take a pass tomorrow
[01:29:30] <ryankemper>	 cdanis: thanks for all the help today!
[01:29:40] <cdanis>	 np :)
[01:30:16] <andrewbogott>	 XioNoX, when you come back  to work, this might be something that you immediately know the solution for:  https://phabricator.wikimedia.org/T258764
[01:36:28] <cdanis>	 andrewbogott: grepping around for a random switch name in the puppet repo, it comes from here: https://phabricator.wikimedia.org/source/operations-puppet/browse/production/hieradata/common/monitoring.yaml$302
[01:36:34] <cdanis>	 https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/profile/manifests/icinga.pp$9
[01:36:41] <cdanis>	 probably just add the switch hostname and a description to the yaml
[01:36:59] <cdanis>	 please try that, right now icinga isn't loading new configurations, per the alert in #-operations
[02:16:45] <andrewbogott>	 cdanis: no joy from https://gerrit.wikimedia.org/r/c/operations/puppet/+/615906.  Maybe just hostname and not fqdn?
[02:17:05] <andrewbogott>	 It's not clear to me, hostnames and fqdns both appear in that list and also in the icinga config
[02:25:49] <andrewbogott>	 nope, doesn't  help either way
[09:15:18] <kormat>	 are there any docs on 'sandbox' branches in gerrit?
[09:17:34] <kormat>	 or, generally, how to make a branch in gerrit that can be shared with other people?
[09:19:13] <godog>	 mmhh I thought I read about sandbox branches on wikitech but can't find documentation, then it must have been by looking at permissions on gerrit for sth that I could push-force too (JFTR)
[09:20:00] <kormat>	 ah hah - just found https://www.mediawiki.org/wiki/Gerrit/personal_sandbox
[09:30:46] <kormat>	 mmph. `git push gerrit HEAD:sandbox/kormat/pontoon-mariadb104-test`
[09:30:47] <kormat>	 er
[09:30:55] <kormat>	 `remote: You need 'Push' rights with 'Force' flag set to do a non-fastforward push.`
[12:55:50] <cdanis>	 kormat: seems unfortunate that you'd need that permission, I might bring that up in #-releng and/or with Paladox
[12:55:54] <cdanis>	 _joe_: did you have a chance to look at my patches?
[12:59:39] <cdanis>	 ah he's off today
[13:00:03] <cdanis>	 akosiaris: I added you as a reviewer on the very-rough https://gerrit.wikimedia.org/r/c/operations/puppet/+/615877 which is maybe almost correct
[13:27:56] <cdanis>	 wkandek: btw, you should join #wikimedia-sre-private
[17:56:37] <ryankemper>	 hey all, we've got a PR to add airflow privileges to the scap deploy user: https://gerrit.wikimedia.org/r/c/operations/puppet/+/615582/2/modules/profile/manifests/analytics/search/airflow.pp#52
[17:57:16] <ryankemper>	 since this is adding to sudoers, is review needed from the broader sre team? not sure on the process here
[18:17:03] <hauskatze>	 not-sre: I think Wikitech says all 'sudo'ers needs approval in weekly ops meeting?
[19:00:06] <cdanis>	 ryankemper: changes to the sudo permission of groups are generally expected to wait one SRE meeting, although, I think we set that policy before we moved it from weekly to bi-weekly
[19:00:29] <ryankemper>	 thanks. next meeting is this upcoming monday so waiting is no problem in this case
[19:00:35] <cdanis>	 yeah, was just about to say that
[19:00:45] <cdanis>	 can you add it to the meeting doc, and also, add j.bond + m.oritzm to reviewers?
[19:02:41] <ryankemper>	 done
[19:02:47] <ryankemper>	 oh wait
[19:02:56] <ryankemper>	 I thought you meant add them on the google doc but you definitely mean the gerrit patch huh
[19:02:57] <ryankemper>	 xD
[19:04:10] <cdanis>	 :D
[19:04:49] <ryankemper>	 okay done for real now
[19:05:32] <cdanis>	 thanks :)
[19:38:42] <mutante>	 twentyafterfour: aphlict1001 has the scap target now. puppet is happier but still missing some stuff. now that i look at /srv/deployment i already see phab files
[19:39:00] <twentyafterfour>	 cool
[19:39:19] <twentyafterfour>	 mutante: want me to take a look at the remaining stuff that's missing?
[19:39:51] <mutante>	 twentyafterfour: right now the issue is i used /srv/aphlict and let puppet create the symlink to /srv/deployment... but...
[19:40:07] <mutante>	 in some other place it still looks for /srv/phab/phabricator/support/aphlict/server
[19:40:21] <mutante>	 and the "/srv/phab" is the basedir
[19:40:40] <mutante>	 but basedir is already a variable and should not be hardcoded
[19:42:38] <mutante>	 also let me make sure phab-admins also get shell access to that instance
[19:42:54] <twentyafterfour>	 mutante: yeah I can't log in apparently
[19:44:14] <twentyafterfour>	 modules/phabricator/manifests/aphlict.pp  $aphlict_dir = "${phabdir}/support/aphlict/server"
[19:44:42] <mutante>	 yes, that
[19:45:31] <twentyafterfour>	 if we make aphlict_dir a parameter then it doesn't need to be based off of $phabdir  ...
[19:46:11] <mutante>	 yea, i already started that:
[19:46:12] <mutante>	 Stdlib::Unixpath $base_dir = lookup('aphlict_base_dir', { 'default_value' => '/srv/aphlict' }),
[19:46:39] <mutante>	 but this is another one in the phab module instead of the profile
[19:47:46] <mutante>	 or we just need to set $basedir when using phabricator::aphlict
[19:49:06] <mutante>	 yea, base_dir   => $base_dir,  should do it .. trying
[19:49:45] <twentyafterfour>	 well several pieces of aphlict are inside of phabricator's source directory
[19:50:52] <mutante>	 that should be ok since the symlink  aphlict -> /srv/deployment/phabricator/deployment
[19:51:05] <twentyafterfour>	 right
[20:34:34] <mutante>	 twentyafterfour: shell access - resolved,  base_dir issue - resolved.   new issue: we don't have PHP installed :)
[20:35:00] <mutante>	 aphlict[20819]: /usr/bin/env: ‘php’: No such file or directory
[20:35:04] <twentyafterfour>	 oh
[20:35:35] <twentyafterfour>	 that's probably just used in the start script.. could probably replace it with a shell script but I'm not sure if it's worth the trouble
[20:36:28] <mutante>	 i mean.. yea.. we have nodejs installed obviously. and per https://secure.phabricator.com/T6889  they won't replace that with PHP alternatives
[20:36:47] <mutante>	 so now it needs both :p
[20:53:52] <mutante>	 twentyafterfour: let puppet install php-cli. errors are gone. aphlict is now running :)
[20:54:51] <mutante>	 next for me: add envoy, cert
[21:00:08] <twentyafterfour>	 sweet thanks mutante
[21:02:00] <andrewbogott>	 Is anyone around who is comfortable making switch config changes?  I have the feeling I need that for https://gerrit.wikimedia.org/r/c/operations/puppet/+/616153 (since it's moving from a public IP to cloud-hosts1-b-eqiad)
[21:07:28] <andrewbogott>	 cmjohnson1 maybe?
[21:09:45] <andrewbogott>	 or… I guess bblack?
[21:11:36] * andrewbogott should probably back out that change and embrace the Friday