[04:59:36] cdanis: thanks! I will use it today and will update the documentation too [05:13:22] <_joe_> is it making your daily life easier? [05:13:40] it is :) [05:14:05] <_joe_> not always a given at first [05:14:16] Yeah, there are stuff that we need to tweak [05:14:20] <_joe_> now the next step is allowing to use the libraries in cookbooks [05:14:27] And we have to adapt our workflows to it as well [05:14:30] <_joe_> like what? [05:14:45] Like the fact that we cannot add comments next to each host [05:15:05] Or that for now we cannot check the https://noc.wikimedia.org/conf/highlight.php?file=db-eqiad.php and we have to use the CLI to check the config [05:15:08] things like that [05:15:28] <_joe_> ok the latter is going to be solved pretty soon I think [05:33:07] wikibugs down again? [05:34:24] let me restart it [05:34:58] ok, that worked [05:43:36] Someone can double check this IP for me? https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/528313/ :) [05:45:49] nope... gerrit looks down from here :/ [05:45:57] woot [05:46:02] works for me [05:46:15] yeah yeah, being in Asia is a PITA sometimes to reach the US [05:46:57] a few retries.. and it's done :) [05:47:10] haha [05:48:03] thanks apergos! [05:48:20] and also of course thanks vgutierrez for trying! :) [05:48:22] heh [05:54:30] cdanis: dbctl config commit -m "xxxx" worked fine! Going to change the wikitech page [09:55:54] We are not syncing puppet repo to github anymore? [09:56:06] uh? [09:56:09] Right, looks like we aren't: https://github.com/wikimedia/puppet/commit/bdc6e091c22efe7fb24dbe12a7492ec2a211c6f2 [09:56:39] last commit in GH says "gerrit: do not treat github as a mirror" [09:56:54] https://lists.wikimedia.org/pipermail/wikitech-l/2019-August/092390.html [09:57:42] I've since removed GitHub as a "mirror" -- meaning Gerrit will not delete branches there. [09:58:04] hmmm IMHO it looks like a bug the fact than github is not longer updated [09:58:07] Yeah, but the patch thcipriani sends on that email https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/528276/ doesn't include our puppet repo [09:58:44] Maybe paladox knows [09:59:43] more repos than puppet are affected, the DNS one doesn't updates in github as well [10:00:10] thcipriani ^ [10:00:20] nice catch BTW marostegui :) [10:31:09] <_joe_> I would bet something around the upgrade process broke replication [10:31:12] <_joe_> or it was stopped [10:31:26] <_joe_> let's ask tyler or paladox [10:31:36] <_joe_> and yes nice catch indeed! [10:32:25] heh, I didn't realise cdanis emojis were making it to SAL too! [10:32:29] https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:40:06] <_joe_> ahahahha [10:40:28] <_joe_> so basically he wrote a wrapper around cumin and debdeploy to log to sal automatically [10:40:33] <_joe_> AIUI [10:41:05] that's his prompt [10:41:13] <_joe_> yeah [10:41:14] it's just copy/paste from his prompt with !log in front [10:41:46] <_joe_> yeah I thought he sent the log from cumin1001 via tcpircbot [10:42:06] <_joe_> that's actually a good idea for a few things [10:42:37] yeah we could add a sal-log wrapper binary [10:42:52] that !logs and execute whatever you pass to it [10:46:27] marostegui: I saw it on twitter yesterday [10:46:43] jijiki: what did you see? [10:46:48] the emojis :p [10:47:33] one could assume that he was drinking beer while deploying [11:30:53] Hi [11:32:53] We removed mirror option from the github remote because apparently the wmf gerrit bot was deleting branches and pull requests for the wikimedia iOS repo on github. We’ve since found that you can blacklist repos so I suppose we can switched the mirror function back on (though would like to ask Tyler first) [11:33:25] paladox: but what I saw is that for mediawiki repo, it is still enabled [11:33:26] _joe_: marostegui Vgutierrez ^ [11:33:39] paladox: or at least, it is updating github [11:33:56] I wonder why it’s failing for the puppet repo [11:35:48] Gerrit should have a replication log on cobalt (should be in /var/log/gerrit/replication.log) [11:36:01] That may give you the reason why the pushing is failing [11:37:57] I don't see any errors there [11:38:47] https://phabricator.wikimedia.org/P8870 [11:40:19] hmm [11:43:11] Appears running just "ssh -p 29418 paladox@gerrit.wikimedia.org replication start operations/puppet --wait" is showing only gerrit2001 replicating successfully [11:43:13] Replicate operations/puppet ref ..all.. to gerrit2001.wikimedia.org, Succeeded! (OK) [11:43:13] Replication of operations/puppet ref ..all.. completed to 1 nodes, [11:49:53] yeah, for puppet repo only gerrit2001 appears on the log [11:54:43] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/528433/ [11:55:17] Let's wait for thcipriani as well, maybe this was expected? [11:56:28] the breaking of replication for the puppet repo was not expected :) [11:56:34] though yeh [13:02:41] marostegui: it will also prompt you for a message if you don't provide one with -m, so not strictly needed to update the docs [13:02:59] yeah, but for automation and all that :) [13:03:04] aye :) [13:10:03] also marostegui re: needing to use cli to view sections, will be fixed this week [13:18:27] * paladox is back [13:41:16] cdanis: great, that will help a lot :) [17:09:18] * thcipriani reading scrollback after morning meetings [17:09:38] wasn't expected. I have some debug logging I want to deploy to figure out what happened. [17:10:34] awesome [17:10:40] I didn't create a task or anything [17:10:52] hope that's ok :) [17:10:53] let me know if I should [17:12:40] just did that: https://phabricator.wikimedia.org/T229945 [17:13:52] ah thanks :) [17:14:27] * marostegui subscribed [20:32:14] apergos: marostegui: thanks to the both of you for your work on dbctl's documentation! just added more myself :) [20:40:06] apergos: biggest change I made was removing the stuff that's more an artifact of the etcd API than of dbctl's design/implementation itself [20:40:17] I think it makes it clearer [21:20:16] fyi that replication is fixed (there's a bug in the replication plugin when it reloads the config)