[06:35:19] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+1/-0/±0] 13https://git.io/Jfd5A
[06:35:20] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 03c774b1d - RELEASE: bump dev to version 7
[06:35:27] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 opened pull request 03#200: RELEASE: bump dev to version 7 - 13https://git.io/Jfd5x
[06:36:15] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/Jfd5h
[06:36:17] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 0319659e5 - Update responses.py
[06:36:18] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 synchronize pull request 03#200: RELEASE: bump dev to version 7 - 13https://git.io/Jfd5x
[06:36:52] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/Jfd5j
[06:36:54] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 03d858b7c - Update SECURITY.md Only support latest version
[06:36:55] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 synchronize pull request 03#200: RELEASE: bump dev to version 7 - 13https://git.io/Jfd5x
[06:39:58] <wikibugs>	 (03PS4) 10RhinosF1: update for v7 release [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/606219 (owner: 10MacFan4000)
[06:40:15] <wikibugs>	 (03CR) 10RhinosF1: [C: 03+1] update for v7 release [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/606219 (owner: 10MacFan4000)
[06:41:00] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 edited pull request 03#200: RELEASE: bump dev to version 7 - 13https://git.io/Jfd5x
[06:42:17] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 edited pull request 03#199: Release Version 7 - 13https://git.io/Jfd0i
[06:43:13] <RhinosF1>	 Reception123: I'm glad travis has flood control, over 150 commits to merge.
[06:43:58] <Reception123>	 RhinosF1: morning :) heh, yeah
[07:33:55] <RhinosF1>	 Reception123: I wonder how long git will take
[07:49:44] <Reception123>	 probably quite a while
[08:14:48] <RhinosF1>	 Reception123: bumping dev now
[08:15:26] <Reception123>	 k
[08:15:58] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 closed pull request 03#200: RELEASE: bump dev to version 7 - 13https://git.io/Jfd5x
[08:15:59] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 pushed 034 commits to 03dev [+2/-0/±4] 13https://git.io/JfdAL
[08:16:01] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 036ba416d - Merge pull request #200 from Pix1234/RhinosF1-patch-1 RELEASE: bump dev to version 7
[08:17:03] <RhinosF1>	 .botversion
[08:17:03] <ZppixBot>	 The current version of this bot is 6.0 (v6)
[08:22:29] <wikibugs>	 10ZppixBot: Release ZppixBot 7 - https://phabricator.wikimedia.org/T253381 (10RhinosF1)
[08:23:05] <wikibugs>	 10ZppixBot: Release ZppixBot 7 - https://phabricator.wikimedia.org/T253381 (10RhinosF1) I've created the draft release and bumped the dev branch to version 7.
[08:24:22] <wikibugs>	 10ZppixBot: Release ZppixBot 7 - https://phabricator.wikimedia.org/T253381 (10RhinosF1)
[08:24:45] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 closed pull request 03#199: Release Version 7 - 13https://git.io/Jfd0i
[08:24:48] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 pushed 03160 commits to 03master [+18/-47/±178] 13https://git.io/JfdAV
[08:24:49] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 03127f662 - Merge pull request #199 from Pix1234/version-7 Release Version 7
[08:24:55] <RhinosF1>	 Reception123: ^
[08:25:10] <Reception123>	 :)
[08:25:27] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 tagged 0311397ca as 03v7 13https://git.io/JfdAo
[08:25:37] <RhinosF1>	 https://github.com/Pix1234/ZppixBot-Source/releases/tag/v7
[08:25:38] <ZppixBot>	 [ Release Version 7 · Pix1234/ZppixBot-Source · GitHub ] - github.com
[08:26:51] <wikibugs>	 10ZppixBot, 10User-RhinosF1: Release ZppixBot 7 - https://phabricator.wikimedia.org/T253381 (10RhinosF1) 05Open→03Resolved a:03RhinosF1 Tagged version 7.  264 commits
[08:27:02] <wikibugs>	 10ZppixBot, 10User-RhinosF1: Release ZppixBot 7 - https://phabricator.wikimedia.org/T253381 (10RhinosF1)
[08:27:18] <travis-ci>	 Pix1234/ZppixBot-Source/v7/11397ca - RhinosF1 The build passed. https://travis-ci.org/Pix1234/ZppixBot-Source/builds/699609962
[08:27:44] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 deleted branch 03version-7
[08:27:46] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 created branch 03version-7 13https://git.io/JfdAo
[08:27:47] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 deleted branch 03RhinosF1-patch-1
[08:31:19] <wikibugs>	 (03CR) 10RhinosF1: [C: 03+2] update for v7 release [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/606219 (owner: 10MacFan4000)
[08:32:00] <wikibugs>	 (03Merged) 10jenkins-bot: update for v7 release [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/606219 (owner: 10MacFan4000)
[08:33:34] <ZppixBot>	 [ANNOUNCEMENT] On behalf of all ZppixBot developers, I am proud to announce the release of v7. You can find more information at https://github.com/Pix1234/ZppixBot-Source/releases/tag/v7. Thank you for your continued support of this project! --RhinosF1, Developer
[08:34:13] <RhinosF1>	 Reception123: ^
[08:35:30] <Reception123>	 .gj RhinosF1 
[08:35:30] <ZppixBot>	 You're doing good work, RhinosF1 !
[08:35:42] <RhinosF1>	 Thanks Reception123 
[08:35:53] <ZppixBot>	 RhinosF1: Thank you for getting me a new version!
[08:36:09] <RhinosF1>	 No problem ZppixBot
[08:36:42] <ZppixBot>	 Reception123: I can see you!
[08:37:20] <Reception123>	 :P
[08:37:36] <RhinosF1>	 Reception123: I love it when the bots start talking
[08:37:44] <RhinosF1>	 They are a lot nicer than most people
[08:38:38] <Reception123>	 yup :D
[08:46:22] <RhinosF1>	 Reception123: you might have a fun surprise on Phab in ~30s
[08:47:53] <Reception123>	 RhinosF1: wow :)
[09:32:01] <RhinosF1>	 Reception123: erm, ZppixBot seems sad
[09:34:02] <RhinosF1>	 Reception123: We better not have to rollback
[09:34:34] * ZppixBot sad
[09:41:16] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JfFeI
[09:41:18] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 032cedc30 - use bot.config
[09:41:56] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 opened pull request 03#201: use bot.config - 13https://git.io/JfFeL
[09:42:48] <travis-ci>	 Pix1234/ZppixBot-Source/RhinosF1-patch-1/2cedc30 - RhinosF1 The build was broken. https://travis-ci.org/Pix1234/ZppixBot-Source/builds/699629933
[09:43:36] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JfFeO
[09:43:37] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 038cbd698 - Update mh_phab.py
[09:43:39] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 synchronize pull request 03#201: use bot.config - 13https://git.io/JfFeL
[09:43:55] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 ready_for_review pull request 03#201: use bot.config - 13https://git.io/JfFeL
[09:45:08] <travis-ci>	 Pix1234/ZppixBot-Source/RhinosF1-patch-1/8cbd698 - RhinosF1 The build was fixed. https://travis-ci.org/Pix1234/ZppixBot-Source/builds/699630540
[09:45:47] <RhinosF1>	 Reception123: If ZppixBot crashes much more, I'll rollback.
[09:46:07] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 edited pull request 03#201: use bot.settings - 13https://git.io/JfFeL
[09:46:46] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JfFeB
[09:46:47] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 03f30347c - Update mh_phab.py
[09:46:49] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 synchronize pull request 03#201: use bot.settings - 13https://git.io/JfFeL
[09:48:00] <Reception123>	 ok :(
[09:53:48] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 pushed 031 commit to 03revert-v7 [+6/-2/±5] 13https://git.io/JfFeD
[09:53:49] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 0310c2c86 - Revert "Merge pull request #199 from Pix1234/version-7" This reverts commit 127f6627f7eef75764adfec018ec3dca2892903f, reversing changes made to ed503109004039d5b462fa8b009656ca5e77a6a9. Rollback Version 7 Upgrade
[09:55:19] <travis-ci>	 Pix1234/ZppixBot-Source/revert-v7/10c2c86 - RhinosF1 The build passed. https://travis-ci.org/Pix1234/ZppixBot-Source/builds/699633020
[09:55:36] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 opened pull request 03#202: Revert "Merge pull request #199 from Pix1234/version-7" - 13https://git.io/JfFe9
[09:56:15] <RhinosF1>	 Reception123: if we need to revert, use ^
[10:00:58] <ZppixBot>	 MacFan4000: Release ZppixBot v7!
[10:01:00] <ZppixBot>	 MacFan4000: Release ZppixBot v7!
[10:02:35] <RhinosF1>	 done and close to rolling back
[10:04:26] <RhinosF1>	 MacFan4000, Texas: fyi ^
[10:09:16] * RhinosF1 is restarting everything
[10:29:06] <RhinosF1>	 Reception123: rolling back
[10:29:44] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 closed pull request 03#202: Revert "Merge pull request #199 from Pix1234/version-7" - 13https://git.io/JfFe9
[10:29:45] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 pushed 032 commits to 03master [+12/-4/±10] 13https://git.io/JfFfH
[10:29:47] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 03ca31e34 - Merge pull request #202 from Pix1234/revert-v7 Revert "Merge pull request #199 from Pix1234/version-7"
[10:30:52] <Reception123>	 ok :(
[10:31:55] <RhinosF1>	 Reception123: can you file a task with release version 7 as a subtask.
[10:32:05] <RhinosF1>	 Just say it keeps crashing as that's all we know
[10:34:59] <RhinosF1>	 Reception123: rollback complete
[10:41:09] <RhinosF1>	 Reception123: cleaned all caches as well
[10:43:22] <wikibugs>	 10ZppixBot: ZppixBot timing out repeatedly on ZppixBot version 7 - https://phabricator.wikimedia.org/T255763 (10RhinosF1)
[10:43:35] <wikibugs>	 10ZppixBot: ZppixBot timing out repeatedly on ZppixBot version 7 - https://phabricator.wikimedia.org/T255763 (10RhinosF1) p:05Triage→03High
[10:55:09] <wikibugs>	 10ZppixBot: ZppixBot timing out repeatedly on ZppixBot version 7 - https://phabricator.wikimedia.org/T255763 (10RhinosF1) Timeline (UTC): - update deployed 08:30 - disconnected 08:46 - disconnected 09:29 - rollback deployed 10:31
[11:24:09] <wikibugs>	 10ZppixBot: ZppixBot timing out repeatedly on ZppixBot version 7 - https://phabricator.wikimedia.org/T255763 (10RhinosF1) > 2020-06-18 08:46:12,191 << :orwell.freenode.net PONG orwell.freenode.net :chat.freenode.net > 2020-06-18 08:46:22,495 << :RhinosF1!uid339563@miraheze/RhinosF1 PRIVMSG #ZppixBot :Reception12...
[11:25:00] <RhinosF1>	 QuIRC: not a clue what happened yet
[11:30:12] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 deleted branch 03revert-v7
[11:34:53] <QuIRC>	 RhinosF1: did -test have any issues?
[11:35:31] <RhinosF1>	 QuIRC: it has timeouts but not to this degree
[11:42:39] <QuIRC>	 I'm putting together a sopel bot with the ZppixBot modules, and I'll try testing it to see if I get the same issues
[11:43:15] <RhinosF1>	 QuIRC: it's not a resource overload https://grafana-labs.wikimedia.org/d/toolforge-k8s-namespace-resources/kubernetes-namespace-resources?orgId=1&var-namespace=tool-zppixbot&from=1592469660000&to=1592472840000
[11:43:15] <ZppixBot>	 [ Grafana ] - grafana-labs.wikimedia.org
[11:44:57] <RhinosF1>	 QuIRC: all I can see is a surge in network traffic immediately after the crash
[11:49:27] <RhinosF1>	 https://kubernetes.io/blog/2019/03/29/kube-proxy-subtleties-debugging-an-intermittent-connection-reset/???
[11:49:28] <ZppixBot>	 [ kube-proxy Subtleties: Debugging an Intermittent Connection Reset | Kubernetes ] - kubernetes.io
[11:53:03] <QuIRC>	 I don't understand kubernetes enough to even read that article, but the title does sound promising
[11:54:36] <RhinosF1>	 QuIRC: it's something to do with the network setup but raw logs don't look like it's a proper timeout. It looks racey
[11:54:52] <RhinosF1>	 QuIRC: I'm going to try and trace the code
[11:55:06] <QuIRC>	 Let me know if there's anything I can do to help
[11:55:50] <RhinosF1>	 QuIRC: for now, ping us if it dies
[11:56:05] <QuIRC>	 I'll keep an eye on it
[11:56:54] <QuIRC>	 Also, your reversion announcement only went out on -test, so channels with the main instance only saw the update announcement
[11:57:59] <RhinosF1>	 QuIRC: I was getting access denied on the bot
[11:58:11] <RhinosF1>	 ^ me restarting
[11:58:46] <QuIRC>	 ack
[12:13:11] <MacFan4000>	 .accesslevel
[12:13:11] <ZppixBot>	 The access level for MacFan4000 is Admin.
[12:13:51] <ZppixBot>	 [ANNOUNCEMENT] The ZppixBot update earlier was rolled back due to a technical issue.
[12:14:07] <MacFan4000>	 There we go
[12:32:37] <RhinosF1>	 MacFan4000: ty
[12:33:38] <MacFan4000>	 I assume we had updated the mh_phab configuration in default.cfg?
[12:34:26] <RhinosF1>	 MacFan4000: it should have been mostly there but I tested and that doesnt break it
[12:34:28] <RhinosF1>	 No logs
[12:37:21] <RhinosF1>	 MacFan4000: paste me the kubectl logs
[12:38:17] <MacFan4000>	 They'll only be from when the bot last started up, since the old ones were wiped upon manual restart
[12:39:43] <RhinosF1>	 MacFan4000: I want to see if my trace is doing what I want to do
[12:40:48] <MacFan4000>	 ah
[12:42:31] <MacFan4000>	 RhinosF1: see -logs
[13:40:27] <RhinosF1>	 MacFan4000: rolling forward, hoping, I will mobile for a bit but feel free to paste any logs
[13:43:14] <RhinosF1>	 MacFan4000: ^
[13:43:21] <RhinosF1>	 .botversion
[13:43:29] <ZppixBot>	 The current version of this bot is 7.0 (v7)
[13:43:38] <RhinosF1>	 .ip 8.8.8.8
[13:43:39] <ZppixBot>	 Downloading GeoIP database, please wait...
[13:44:51] <ZppixBot>	 [IP/Host Lookup] Hostname: dns.google | Location: United States | ISP: AS15169 GOOGLE
[14:07:45] <MacFan4000>	 T12
[14:07:47] <ZppixBot>	 https://phabricator.miraheze.org/T12 - $wmgUseXXX should work and $wgConf->settings should override extensions' def. variables, authored by Southparkfan, assigned to github-migration, Priority: Normal, Status: Resolved
[14:08:07] <RhinosF1>	 MacFan4000: now to hope we stay online
[14:08:19] <MacFan4000>	 yup
[14:08:31] <RhinosF1>	 just switch straight back to master if it crashes and save kubectl logs
[16:13:41] <RhinosF1>	 hey Zppix
[16:13:48] <Zppix>	 hi
[16:18:39] <RhinosF1>	 Zppix: been stable since the re-roll
[16:18:40] <Texas>	 .mhca Sourav Halder
[16:18:41] <ZppixBot>	 https://meta.miraheze.org/wiki/Special:CentralAuth/Sourav_Halder
[16:18:44] <Texas>	 yayy
[16:18:48] <Texas>	 my patch is there
[16:18:51] <RhinosF1>	 Bot is currently tracking the v7 branch
[16:19:04] <RhinosF1>	 Texas: anything on dev is now on master
[16:19:11] <Texas>	 great!
[16:19:12] <Texas>	 :)
[16:19:28] <Zppix>	 ZppixBot: should always track master v7 should be anyone that wants to run the bots code but ok
[16:19:35] <RhinosF1>	 Texas: well on version-7 ehich we are tracking
[16:19:57] <RhinosF1>	 Zppix: only temp for a test, will be tracking master again from tonight assuming it's stable
[16:20:23] <RhinosF1>	 (Version 7 is reverted on master pending investigation of this morning)
[16:21:50] <RhinosF1>	 Zppix: if we get to 10pm my time and the issue hasn't happened, I'm going to roll the update back on master and switch it back.
[16:23:09] <Not-944e>	 [02ZppixBot-Source] 07MacFan4000 pushed 031 commit to 03revert-202-revert-v7 [+2/-6/±5] 13https://git.io/JfFZC
[16:23:10] <Not-944e>	 [02ZppixBot-Source] 07MacFan4000 03efd177c - Revert "Revert "Merge pull request #199 from Pix1234/version-7""
[16:23:19] <Not-944e>	 [02ZppixBot-Source] 07MacFan4000 opened pull request 03#203: Revert "Revert "Merge pull request #199 from Pix1234/version-7"" - 13https://git.io/JfFZW
[16:23:36] <MacFan4000>	 ^will be merged at 9 PM UTC assuming things go to plan
[16:24:31] <Not-944e>	 [02ZppixBot-Source] 07MacFan4000 labeled pull request 03#203: Revert "Revert "Merge pull request #199 from Pix1234/version-7"" - 13https://git.io/JfFZW
[16:24:45] <travis-ci>	 Pix1234/ZppixBot-Source/revert-202-revert-v7/efd177c - MacFan4000 The build passed. https://travis-ci.org/Pix1234/ZppixBot-Source/builds/699776158
[16:24:55] <RhinosF1>	 MacFan4000: correct
[16:25:11] <RhinosF1>	 I'll do the deploy as I don't get why it failed this morning
[16:25:18] <Not-944e>	 [02ZppixBot-Source] 07MacFan4000 assigned pull request 03#203: Revert "Revert "Merge pull request #199 from Pix1234/version-7"" - 13https://git.io/JfFZW
[16:25:20] <Not-944e>	 [02ZppixBot-Source] 07MacFan4000 assigned pull request 03#203: Revert "Revert "Merge pull request #199 from Pix1234/version-7"" - 13https://git.io/JfFZW
[16:25:40] <RhinosF1>	 And why making it track version-7 worked
[16:26:11] <Not-944e>	 [02ZppixBot-Source] 07RhinosF1 edited pull request 03#203: Re-deploy version7 - 13https://git.io/JfFZW
[16:29:03] <MacFan4000>	 Zppix: for some reason PR comments still aren't being reported by notifico
[16:29:43] <Zppix>	 ill look in a bit
[16:30:06] <RhinosF1>	 Zppix: any clue why this morning's deploy failed?
[16:31:24] <Zppix>	 RhinosF1:  nope
[16:32:27] <MacFan4000>	 Zppix: looks like you just need to login to notifico and click sync with GitHub
[16:32:37] <RhinosF1>	 Zppix: It doesn't make sense as being our code. This is what the python changes caused.
[16:32:46] * RhinosF1 does think this is toolforge
[16:32:58] <Zppix>	 I doubt it
[16:33:20] <RhinosF1>	 Zppix: every time we make major changes, we have a surge in unexplainable connection issues
[16:33:24] <MacFan4000>	 .at 5:00 re-deploy v7
[16:33:31] <ZppixBot>	 MacFan4000: Okay, will remind at 2020-06-19 - 04:59:59EDT
[16:34:00] <RhinosF1>	 It's either that or the setup of python/docker/kuberenetes
[16:34:04] <RhinosF1>	 I found https://kubernetes.io/blog/2019/03/29/kube-proxy-subtleties-debugging-an-intermittent-connection-reset/??
[16:34:05] <ZppixBot>	 [ kube-proxy Subtleties: Debugging an Intermittent Connection Reset | Kubernetes ] - kubernetes.io
[16:34:07] <Zppix>	 done
[16:34:44] <Zppix>	 RhinosF1 dk
[16:34:45] <Zppix>	 idk
[16:35:17] <MacFan4000>	 Zppix: Hmm my test comment didn't work (deleted now)
[16:35:44] <Zppix>	 MacFan4000:  wym?
[16:35:54] <Zppix>	 oh idk
[16:36:50] <MacFan4000>	 within the webhook settings on the notifico website there is box to select events
[16:37:28] <RhinosF1>	 Zppix: The only thing I changed is wiping .pyc files and .mmdb files
[16:37:31] <Zppix>	 MacFan4000:  its set to send whatever the github webhook sends
[16:37:53] <RhinosF1>	 Maybe when I do the switch I should wipe  the ones in the venv as well
[16:37:55] <Zppix>	 RhinosF1:  theres no need to do that
[16:38:28] <RhinosF1>	 Zppix: wiping pyc files is the only thing I did different during the deploy
[16:39:10] <MacFan4000>	 meh, must be something their end
[16:39:31] <MacFan4000>	 i guess we'll just wait and see if it eventually starts working
[16:41:29] <RhinosF1>	 MacFan4000: pyc is complied python files, it's as if they're not being purged properly
[16:41:54] <MacFan4000>	 i was referring to notifico
[16:43:33] <RhinosF1>	 Oh
[16:48:13] <MacFan4000>	 Zppix: I just figured it out, PR comments are sent as issue comments, so you'll have to add issue comments
[16:50:31] <MacFan4000>	 (I didn't even know that until I tested it on one of my repos)
[16:54:17] <RhinosF1>	 Zppix: I spoke to another dev and we do think it's the pyc files. I'm going to clean some stuff up when I redeploy from the master branch
[17:16:05] <wikibugs>	 10ZppixBot: ZppixBot timing out repeatedly on ZppixBot version 7 - https://phabricator.wikimedia.org/T255763 (10RhinosF1) After some testing, I think this is the pycs in our venv.  I'll deploy again and fix the venv tonight on both instances.  It will probably cause downtime of up to 15 mins per instance.
[17:16:16] <RhinosF1>	 Zppix, MacFan4000: ^
[17:30:20] <RhinosF1>	 I just had a thought
[17:30:42] <RhinosF1>	 If that takes longer than 15 mins on prod, I'll have to stop the auto deploy cron
[17:49:22] <RhinosF1>	 .in 2.5hours CRON CRON CRON
[17:49:23] <ZppixBot>	 RhinosF1: Okay, will remind at 2020-06-18 - 21:19:23BST
[20:19:24] <ZppixBot>	 RhinosF1: CRON CRON CRON
[20:49:53] <RhinosF1>	 MacFan4000, Texas: deploying at 22:15
[20:49:59] <RhinosF1>	 I will ping when I start
[21:02:39] <MacFan4000>	 oops, I set a reminder to go off 2 mins ago, but i actually set it to go off at 5 am it seems
[21:07:11] * ZppixBot is excited
[21:13:00] <ZppixBot>	 [ANNOUNCEMENT] Very very soon, ZppixBot v7 will be re-deployed - there may be a short downtime as there will be a bit of maintenance. The test instance (ZppixBot-test will receive the maintenance as well and thus may also have a short downtime. Thanks for your understanding! -The ZppixBot team
[21:14:28] <RhinosF1>	 MacFan4000: ack, It should be about 15 mins per instance. Will have to stop cron as well.
[21:15:09] <MacFan4000>	 RhinosF1: what does cron have to do with sopel?
[21:15:21] <RhinosF1>	 MacFan4000: same venv
[21:15:22] <MacFan4000>	 afaik its just for website auto-deploy
[21:15:34] <RhinosF1>	 I need to reset pip
[21:15:46] <RhinosF1>	 Which means the cron will fail
[21:15:52] <MacFan4000>	 website doesn't use python at all thus no venv for it
[21:17:26] <RhinosF1>	 MacFan4000: auto-deploy is a python script
[21:17:47] <RhinosF1>	 There's a font awesome update as well
[21:18:02] <MacFan4000>	 i don't think it uses pip though
[21:18:11] <RhinosF1>	 it does
[21:19:01] <MacFan4000>	 https://github.com/wikimedia/labs-tools-ZppixBot/blob/master/deploy.py
[21:19:02] <ZppixBot>	 [ labs-tools-ZppixBot/deploy.py at master · wikimedia/labs-tools-ZppixBot · GitHub ] - github.com
[21:19:10] <MacFan4000>	 i don't see pip mentioned anywhere there
[21:19:21] <RhinosF1>	 MacFan4000: from git import *
[21:19:30] <RhinosF1>	 Git is installed on pip
[21:20:02] <MacFan4000>	 ah
[21:22:33] <RhinosF1>	 One sec
[21:22:47] <RhinosF1>	 Stupid mum delayed me leaving the car unlocked and her keys in it
[21:29:05] <RhinosF1>	 MacFan4000: test stopped, feel free to merge the patch to master now
[21:29:21] <Not-944e>	 [02ZppixBot-Source] 07MacFan4000 closed pull request 03#203: Re-deploy version7 - 13https://git.io/JfFZW
[21:29:22] <Not-944e>	 [02ZppixBot-Source] 07MacFan4000 pushed 031 commit to 03master [+2/-6/±5] 13https://git.io/JfFEB
[21:29:24] <Not-944e>	 [02ZppixBot-Source] 07MacFan4000 03bc76cc0 - Revert "Revert "Merge pull request #199 from Pix1234/version-7"" (#203)
[21:29:25] <Not-944e>	 [02ZppixBot-Source] 07MacFan4000 deleted branch 03revert-202-revert-v7
[21:29:32] <MacFan4000>	 done
[21:31:43] <RhinosF1>	 MacFan4000: ty
[21:32:50] <RhinosF1>	 MacFan4000: pip resetting now
[21:32:54] <MacFan4000>	 k
[21:33:17] <MacFan4000>	 Zppix: Texas: its happening
[21:34:24] <RhinosF1>	 MacFan4000: clearing cache now
[21:34:32] <MacFan4000>	 k
[21:35:24] <RhinosF1>	 MacFan4000: {{done}}, now to git pull and restart
[21:35:24] <wm-bot3>	 https://meta.miraheze.org/wiki/Template:done 
[21:38:12] <RhinosF1>	 MacFan4000: complete, we're starting up
[21:38:17] <MacFan4000>	 k
[21:40:17] <RhinosF1>	 MacFan4000: AttributeError: module 'pkg_resources' has no attribute 'get_distribution'
[21:40:30] <MacFan4000>	 hmm
[21:41:32] <RhinosF1>	 MacFan4000: stackoverflow says setuptools so updating that
[21:41:49] <MacFan4000>	 k
[21:42:05] <RhinosF1>	 MacFan4000: looks like that was out of date
[21:42:11] <MacFan4000>	 k
[21:45:45] <MacFan4000>	 next is the main instance
[21:46:08] <RhinosF1>	 MacFan4000: stopped
[21:47:57] <RhinosF1>	 MacFan4000: resetting pip now
[21:48:01] <MacFan4000>	 k
[21:50:13] <RhinosF1>	 MacFan4000: clearing pyc
[21:50:19] <MacFan4000>	 k
[21:50:34] <wikibugs>	 10ZppixBot, 10User-RhinosF1: ZppixBot timing out repeatedly on ZppixBot version 7 - https://phabricator.wikimedia.org/T255763 (10RhinosF1) a:03RhinosF1
[21:51:48] <RhinosF1>	 MacFan4000: {{done}}, restarting the cron and then the bot
[21:51:48] <wm-bot3>	 https://meta.miraheze.org/wiki/Template:done 
[21:51:57] <MacFan4000>	 k
[21:56:03] <RhinosF1>	 MacFan4000: starting up
[21:56:12] <MacFan4000>	 k
[22:00:19] <RhinosF1>	 MacFan4000: it's being slow
[22:00:30] <MacFan4000>	 yep
[22:00:54] <RhinosF1>	 MacFan4000: unsurprisingly, resetting most of the venv means long time to boot
[22:01:14] <MacFan4000>	 yep
[22:01:59] <RhinosF1>	 MacFan4000: it's alive
[22:02:24] <RhinosF1>	 .op
[22:02:27] <ZppixBot>	 RhinosF1: Access Denied. If in error, please contact the channel founder.
[22:02:36] <RhinosF1>	 MacFan4000: set the topic pls
[22:03:22] <MacFan4000>	 we need to fix that
[22:03:54] <RhinosF1>	 MacFan4000: once it responds in -logs you are free to blow whatever up
[22:05:03] <MacFan4000>	 RhinosF1: its strange, because you are listed in https://github.com/Pix1234/ZppixBot-Source/blob/master/modules/channelmgnt.py#L53
[22:05:04] <ZppixBot>	 [ ZppixBot-Source/channelmgnt.py at master · Pix1234/ZppixBot-Source · GitHub ] - github.com
[22:05:36] <RhinosF1>	 .op
[22:05:36] <ZppixBot>	 Please wait...
[22:05:49] <RhinosF1>	 MacFan4000: maybe just a bot still loading?
[22:05:57] <ZppixBot>	 [ANNOUNCEMENT] My maintenance is done! I apologize for any inconvenience.
[22:06:08] <MacFan4000>	 .deop ZppixBot
[22:07:35] <MacFan4000>	 Zppix: Texas: its done!
[22:07:38] <wikibugs>	 10ZppixBot, 10User-RhinosF1: ZppixBot timing out repeatedly on ZppixBot version 7 - https://phabricator.wikimedia.org/T255763 (10RhinosF1) 05Open→03Resolved I believe this was a broken venv, many bugs maybe (we should retry the python 3.7 upgrade).  Boldy resolving this!
[22:07:43] * RhinosF1 hopes
[22:08:05] * RhinosF1 will cry if ZppixBot survives the night and I wasted hours on a dodgy venv
[22:09:31] <MacFan4000>	 .help
[22:09:31] <ZppixBot>	 I've posted a list of my commands at https://clbin.com/rXUyy - You can see more info about any of these commands by doing .help <command> (e.g. .help time)
[22:09:57] <MacFan4000>	 .happy
[22:09:57] <ZppixBot>	 ᕕ( ᐛ )ᕗ
[22:12:11] <RhinosF1>	 .in 21mins go to sleep
[22:12:53] <MacFan4000>	 that's been having a bit of a delay for some reason
[22:13:08] <MacFan4000>	 .in 1m test
[22:14:58] <RhinosF1>	 MacFan4000: I just nuked a lot, It'll probably be slow for a bit
[22:15:11] <MacFan4000>	 k
[22:15:41] <RhinosF1>	 .ping
[22:15:41] <ZppixBot>	 Pong!
[22:15:49] <RhinosF1>	 .in 21mins go to sleep
[22:26:37] <RhinosF1>	 .in 6mins go to sleep
[22:26:43] <ZppixBot>	 RhinosF1: Okay, will remind at 2020-06-18 - 23:32:43BST
[22:28:16] <wikibugs>	 10ZppixBot, 10User-RhinosF1: Extended Test Window: ZppixBot-Test - https://phabricator.wikimedia.org/T255261 (10RhinosF1) 05Open→03Declined
[22:28:19] <wikibugs>	 10ZppixBot, 10Upstream, 10User-RhinosF1: ZppixBot-test fails restarts correctly on python 3.7 after crash - https://phabricator.wikimedia.org/T254348 (10RhinosF1)
[22:28:21] <wikibugs>	 10ZppixBot, 10Upstream, 10User-RhinosF1: Sopel is not recording last event correctly causing server timeout - https://phabricator.wikimedia.org/T248960 (10RhinosF1)
[22:28:27] <wikibugs>	 10ZppixBot, 10User-RhinosF1: Extended Test Window: ZppixBot-Test - https://phabricator.wikimedia.org/T255261 (10RhinosF1) a:05RhinosF1→03None
[22:29:12] <wikibugs>	 10ZppixBot, 10Upstream, 10User-RhinosF1: Sopel is not recording last event correctly causing server timeout - https://phabricator.wikimedia.org/T248960 (10RhinosF1) I may have found the cause of this and fixed it. Monitoring for the next week.
[22:32:37] <wikibugs>	 10ZppixBot, 10Upstream, 10User-RhinosF1: ZppixBot-test fails restarts correctly on python 3.7 after crash - https://phabricator.wikimedia.org/T254348 (10RhinosF1) This may have been fixed, monitoring for the next week then will deploy & test.
[22:32:45] <ZppixBot>	 RhinosF1: go to sleep
[22:42:53] <RhinosF1>	 MacFan4000: do you want to deploy https://github.com/FortAwesome/Font-Awesome/releases/tag/5.13.1 to the website repo
[22:42:54] <ZppixBot>	 [ Release Release 5.13.1 · FortAwesome/Font-Awesome · GitHub ] - github.com
[22:45:04] <RhinosF1>	 don't forget to bump ?v=x on each html page
[23:15:40] <wikibugs>	 (03PS1) 10MacFan4000: Update FA [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/606544
[23:17:07] <wikibugs>	 (03CR) 10MacFan4000: [C: 03+2] Update FA [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/606544 (owner: 10MacFan4000)
[23:17:43] <wikibugs>	 (03Merged) 10jenkins-bot: Update FA [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/606544 (owner: 10MacFan4000)