[04:57:12] <ryankemper>	 The good news is it turned out most of my area had an internet outage, so it wasn't anything specific to my house, which means no multi-week/month process to get the issue fixed :P Seems like service is back to normal now
[04:58:40] <ryankemper>	 @Erik I went ahead and shipped that patch, I love the use of the rewrite rule to allow specific microsites to include their own assets that differ from the build, super neat
[06:22:55] <dcausse>	 mpham: yes adding more traceability is 100% blazegraph
[06:37:37] <dcausse>	 oops T290545
[06:37:37] <stashbot>	 T290545: Wikidata Query Service inaccessible via GUI, only partially accessible via embedded queries - https://phabricator.wikimedia.org/T290545
[06:38:49] <dcausse>	 ryankemper (sorry for late ping): could this be related to https://gerrit.wikimedia.org/r/c/operations/puppet/+/714624 ?
[06:40:10] <ryankemper>	 dcausse: oof definitely is, looks related to the rootdir stuff, rolling back now
[06:40:26] <dcausse>	 thanks!
[06:50:11] <ryankemper>	 Puppet change is reverted and I ran the puppet-agent on `miscweb*`, still seeing the directory behavior though
[06:50:27] <ryankemper>	 Checking the hosts to make sure the new config has been loaded
[06:50:51] <ryankemper>	 The only effective change on the host was the config file so that should be all that's required to revert
[06:51:19] <dcausse>	 might be due to varnish now
[06:51:45] <dcausse>	 IIRC there are ways to purge the cache manually
[06:52:24] <dcausse>	 https://wikitech.wikimedia.org/wiki/Kafka_HTTP_purging
[06:53:24] <dcausse>	 perhaps something like: echo 'https://query.wikidata.org/' | mwscript purgeList.php
[06:53:52] <dcausse>	 done
[06:54:05] <ryankemper>	 dcausse: that did it
[06:54:38] <dcausse>	 cool let's close the ticket, thanks for the quick response!
[06:57:33] <ryankemper>	 dcausse: thanks for the heads up
[06:57:39] <dcausse>	 np!
[09:15:54] <dcausse>	 errand
[10:41:19] <dcausse>	 lunch
[13:07:09] <dcausse>	 errand
[14:23:37] <gehel>	 dcausse: is T289836 a blocker for going live with the streaming updater?
[14:23:37] <stashbot>	 T289836: Upgrade to latest flink (1.14) - https://phabricator.wikimedia.org/T289836
[14:24:06] <dcausse>	 gehel: no I think we can live 1.12 in the meantime
[14:24:11] <dcausse>	 s/live/live with/
[14:29:45] <gehel>	 cool, thanks!
[14:30:46] <gehel>	 do we have anything blocking us from starting the go live procedure? Beside communication?
[14:31:26] <dcausse>	 gehel: alerts
[14:31:48] <dcausse>	 and some cookbook adaptations
[14:31:55] <gehel>	 dcausse: and you were looking into alert manager?
[14:32:13] <dcausse>	 yes will work on this for the rest of the week
[14:32:43] <gehel>	 cool!
[14:33:11] <gehel>	 I'm not sure we have much to adapt in the cookbooks. I'll do a pass with ryankemper on T276469
[14:33:12] <stashbot>	 T276469: Cookbooks and / or operation procedures are adapted for the new Flink based WDQS Streaming Updater - https://phabricator.wikimedia.org/T276469
[14:33:51] <dcausse>	 gehel: this more less adaptation regarding https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/670242
[14:34:07] <dcausse>	 basically a new file to carry when doing data-transfer
[14:35:35] <gehel>	 quick errand, back in a few
[17:31:28] <dcausse>	 dinner
[18:49:41] <ryankemper>	 dcausse: Do you have a short chunk of time to go over any cookbook changes required for the streaming updater tmrw?
[18:51:19] <ryankemper>	 after retro would probably be a good time, I see you've got the `Wikidata & WMF Sync` for the 30 minutes following the retro so if you'll be at that meeting then maybe directly after? (that is getting kind of late your time tho)
[19:12:30] <dcausse>	 ryankemper: any time after the retro works for me and I do not attent the Wikidata & WMF Sync so just after the retro is fine too
[19:16:26] * ebernhardson writes a silly script to turn a puppet catalog from the compiler into files on disk so i can guess what miscweb1002 looks like
[19:51:04] <ryankemper>	 dcausse: cool let's do immediately after retro then
[21:21:55] <ebernhardson>	 ryankemper: was able to reproduce the httpd problem locally, https://gerrit.wikimedia.org/r/c/operations/puppet/+/719502 should work next go round, hopefully :)
[21:51:05] <ryankemper>	 ebernhardson: awesome, about to ship it
[21:51:22] <ryankemper>	 I rewrote the commit message to be the same as the first time around but with the note about the `REQUEST_FILENAME` btw
[21:56:07] <ryankemper>	 ebernhardson: merged, and purged the varnish cache for both `query.wikidata.org` and `query-preview.wikidata.org` for good measure, looks good
[21:56:45] <ryankemper>	 one small thing...`query.wikidata.org` doesn't seem to have a favicon. But I'm not sure if it had one before? `query-preview` does have one
[22:06:05] <Amir1>	 o/
[22:18:43] <ryankemper>	 Amir1: hey
[22:18:59] <Amir1>	 ryankemper: hiii
[22:19:17] <Amir1>	 I'm waiting for the puppet agent to trigger
[22:21:37] <ryankemper>	 Amir1: just saw your comment. puppet's been ran on miscweb but not the rest of the fleet
[22:21:48] <Amir1>	 after my merge?
[22:22:04] <Amir1>	 because puppet agent also triggers the "git pull"
[22:22:06] <ryankemper>	 Amir1: oh wait, thought you were referring to https://gerrit.wikimedia.org/r/c/operations/puppet/+/719502, one sec
[22:22:23] <Amir1>	 I'm referring to https://gerrit.wikimedia.org/r/c/wikidata/query/gui-deploy/+/714623
[22:22:33] * ryankemper missed the other comment of yours and the merge notification on the phab ticket :P
[22:22:42] <Amir1>	 :D
[22:23:12] <ryankemper>	 Amir1: let me check if miscweb already auto-ran puppet yet, if not I'll kick it off manually
[22:23:46] <Amir1>	 Thanks!
[22:24:25] <ryankemper>	 Amir1: done
[22:24:32] <ryankemper>	 `Notice: /Stage[main]/Profile::Microsites::Query_service/Git::Clone[wikidata/query/gui-deploy]/Exec[git_pull_wikidata/query/gui-deploy]/returns: executed successfully (corrective)`
[22:24:32] <Amir1>	 Awesome
[22:26:35] <ryankemper>	 Amir1: I sanity-checked `/srv/org/wikidata/query` manually and it's on commit SHA `502f70fd772be8d772dd523c28bd7e3b075c2cfb` as expected 
[22:27:12] <Amir1>	 Thanks. I checked the webservice and I confirm it's not broken
[22:27:20] <Amir1>	 https://query.wikidata.org/logo.svg?random
[22:27:24] <Amir1>	 has the logo
[22:27:42] <ebernhardson>	 yay!
[22:27:56] <ryankemper>	 Amir1: the `?random` was just to bypass the cache right?
[22:28:03] <ryankemper>	 and woot!
[22:28:13] <Amir1>	 yup, just in case it's in varnish/ATS
[22:28:25] <ryankemper>	 perfect
[22:28:28] <Amir1>	 (I think it gets cached there)
[22:28:48] <ryankemper>	 Yeah I think so as well, although like any good SRE I share your uncertainty :D
[22:29:52] <Amir1>	 well, I'll be SRE soon :D
[22:29:56] <Amir1>	 preparing :P
[22:30:15] <Amir1>	 ebernhardson: do you want me to merge https://gerrit.wikimedia.org/r/c/wikidata/query/gui-deploy/+/717649 as well, or later?
[22:32:32] <ebernhardson>	 Amir1: might as well go now
[22:32:52] <ebernhardson>	 thx
[22:33:29] <Amir1>	 done
[22:33:40] <Amir1>	 ryankemper: it would be great if you run puppet agent again :D
[22:34:12] <ebernhardson>	 getting closer :) We made a decision on how we will do traffic routing as well, patches soon and we should have a "live" microsite, although without a backend yet
[22:34:47] <ryankemper>	 Amir1: done
[22:34:52] <Amir1>	 Thanks
[22:35:14] <Amir1>	 ebernhardson: btw, do you want to keep showBanner to true there?
[22:35:58] <ebernhardson>	 Amir1: hmm, what does that do? :)
[22:36:10] <Amir1>	 https://query.wikidata.org
[22:36:16] <Amir1>	 The survey banner at the top
[22:36:34] <Amir1>	 your choice, feel free to change it later
[22:36:35] <ryankemper>	 Definitely not then since it had wdqs specific language
[22:36:38] <ryankemper>	 has*
[22:36:49] <ryankemper>	 question: if we want to see the internal site, do we have to tunnel to miscweb and visit localhost:80?
[22:36:52] <ebernhardson>	 ahh, I'm not sure.  indeed it has query specific language, but we will be finding wdqs specific things for awhile :)
[22:37:10] <ebernhardson>	 ryankemper: to test, basically yea tunnel and i edit /etc/hosts to assign the domain to 127.0.0.1
[22:38:32] <ryankemper>	 ebernhardson: and I imagine the `/etc/hosts` is necessary for the routing to work properly? i.e. you're not just doing that for the convenience of not typing out localhost
[22:38:42] <ebernhardson>	 ryankemper: also, i realized why i can't login to wcqs servers, they are missing some hiera. I compared wdqs and wcqs public and this is result: https://gerrit.wikimedia.org/r/c/operations/puppet/+/719643
[22:39:04] <ebernhardson>	 ryankemper: /etc/hosts just lets me type the domain into a web browser, instead of something like curl -H 'Host: query....' localhost:80
[22:39:55] <ebernhardson>	 I gotta run now though, dentist wants to see me in 20m. No rush on that puppet, i think i got all the differences but i could have missed something
[22:40:23] * ebernhardson mostly figures he could have missed something, because obviously i missed something last time :)
[22:40:40] <ryankemper>	 ack, will take a proper look later
[22:45:14] <ryankemper>	 re changing config: I think the banner should be false but it doesn't matter too much either way; we will also need a different URL for the copyright notice I imagine
[22:45:19] <ryankemper>	 Currently we have https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/Copyright which is wdqs-specific