Improve publish service (#382)

* Fix nostr publish * Use latest commit * Closes publish services * Use context from worker pool * Rename fn * Fix notion data struct (#383) * Fix notion block fails * Refine log output * Append large page * Media blocks * Fix event data * update changelog * Add shutdown testing * Change os.Setenv to testing.Setenv * Fix meilisearch testing * Make urlcheck exclude-patterns configurable
wabarc · Apr 29, 2023 · 4bbf9ea · 4bbf9ea
1 parent e4b7701
commit 4bbf9ea
Show file tree

Hide file tree

Showing 32 changed files with 628 additions and 266 deletions.
diff --git a/.github/workflows/linter.yml b/.github/workflows/linter.yml
@@ -41,7 +41,7 @@ jobs:
     name: URLCheck
     uses: wabarc/.github/.github/workflows/reusable-urlcheck.yml@main
     with:
-      exclude-patterns: '.onion,https://github.com/,https://repo.wabarc.eu.org/,twitter.com,example.com'
+      exclude-patterns: ${{ vars.URLCHECH_EXCLUDE }}
 
   goreportcard:
     name: Go Report Card

diff --git a/docs/changelog.md b/docs/changelog.md
@@ -15,6 +15,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Changed
 - Set NoWait to true for tor listen config ([#368](https://github.com/wabarc/wayback/pull/368))
 - Improve context deadline ([#381](https://github.com/wabarc/wayback/pull/381))
+- Improve publish services ([#382](https://github.com/wabarc/wayback/pull/382))
 
 ## [0.19.1] - 2023-03-21
 

diff --git a/docs/integrations/irc.md b/docs/integrations/irc.md
@@ -22,3 +22,5 @@ Note that some IRC servers may require additional configuration, such as TLS cer
 - [IRCv3 Specifications](https://ircv3.net/irc/)
 - [List of IRC networks](https://netsplit.de/networks/top100.php)
 - [Libera.chat documentation](https://libera.chat/guides)
+- [IRC Networks - Top 100](https://netsplit.de/networks/top100.php)
+- [BlockingIrc](https://gitlab.torproject.org/legacy/trac/-/wikis/doc/BlockingIrc)
diff --git a/docs/integrations/irc.zh.md b/docs/integrations/irc.zh.md
@@ -22,3 +22,5 @@ title: IRC
 - [IRCv3规范](https://ircv3.net/irc/)
 - [IRC网络列表](https://netsplit.de/networks/top100.php)
 - [Libera.chat文档](https://libera.chat/guides)
+- [IRC Networks - Top 100](https://netsplit.de/networks/top100.php)
+- [BlockingIrc](https://gitlab.torproject.org/legacy/trac/-/wikis/doc/BlockingIrc)
diff --git a/docs/integrations/notion.md b/docs/integrations/notion.md
@@ -2,19 +2,19 @@
 title: Publish to Notion
 ---
 
-## How to build a Notion Bot
+## How to build a Notion Integration
 
 1. Sign up for a Notion account, if you don't have one already.
-2. Create a new Integration by going to the [Notion API page](https://developers.notion.com/docs/getting-started) and clicking on "My integrations" in the top-right corner of the page.
-3. Give your integration a name and click on "Create Integration".
-4. On the integration page, click on "Add a new integration".
-5. Select "Internal Integration" and click "Submit".
-6. On the next page, you will see your "Integration Token". Copy this token as you will need it later.
-7. Grant your integration access to a database by sharing the database with your integration. To do this, go to the database you want to use with your bot, click on the three-dot menu, and select "Share".
-8. In the "Invite people" field, enter the name of your integration and select it from the list.
-9. Select the appropriate permissions for your integration and click on "Send".
+2. Create a new Integration by going to the [Notion API page](https://www.notion.so/my-integrations) and clicking on "**My integrations**" in the top-right corner of the page.
+3. Give your integration a name and click on "**Create Integration**".
+4. On the integration page, click on "**Add a new integration**".
+5. Select "**Internal Integration**" and click "**Submit**".
+6. On the next page, you will see your "**Integration Token**". Copy this token as you will need it later.
+7. Grant your integration access to a database by sharing the database with your integration. To do this, go to the database you want to use with your bot, click on the three-dot menu, and select "**Share**".
+8. In the "**Connections**" field, navigate to "**Add connections**", find your integration and select it from the list.
+9. Select the appropriate permissions for your integration and click on "**Confirm**".
 
-Note: Wayback currently only supports publishing to Nostr.
+Note: A Notion database must be created using the "**New page**" option, not the "**Add a page**" option.
 
 ## Configuration
 
@@ -24,3 +24,4 @@ Next, place these keys in the environment or configuration file:
 
 - `WAYBACK_NOTION_TOKEN`: Notion integration token.
 - `WAYBACK_NOTION_DATABASE_ID`: Notion database ID for archiving results.
+
diff --git a/docs/integrations/notion.zh.md b/docs/integrations/notion.zh.md
@@ -2,19 +2,19 @@
 title: 发布到Notion
 ---
 
-## 如何构建Notion机器人
+## 如何构建Notion集成
 
 1. 如果您没有Notion帐户，请注册一个Notion帐户。
-2. 创建一个新的集成，方法是转到[Notion API页面](https://developers.notion.com/docs/getting-started)，然后单击页面右上角的“我的集成”。
+2. 创建一个新的集成，方法是转到[Notion API页面](https://www.notion.so/my-integrations)，然后单击页面右上角的“我的集成”。
 3. 给您的集成命名，然后单击“创建集成”。
 4. 在集成页面上，单击“添加新集成”。
 5. 选择“内部集成”，然后单击“提交”。
 6. 在下一页上，您将看到您的“集成令牌”。复制此令牌，因为稍后您将需要它。
 7. 通过与您的集成共享数据库，授予您的集成对数据库的访问权限。要执行此操作，请转到要与您的机器人一起使用的数据库，单击三点菜单，然后选择“共享”。
-8. 在“邀请人员”字段中，输入您的集成名称，并从列表中选择它。
-9. 选择适当的权限，然后单击“发送”。
+8. 点击展开右上角菜单，在“Connections”选项中，输入您的集成名称，并从列表中选择它。
+9. 选择适当的权限，然后单击“确认”。
 
-注意：Wayback目前仅支持将归档结果发布到Nostr。
+注意：一个Notion数据库必须通过"New page"选项创建, 不是"Add a page"选项。
 
 ## 配置
 

diff --git a/go.mod b/go.mod
@@ -11,7 +11,7 @@ require (
 	github.com/davecgh/go-spew v1.1.1
 	github.com/dghubble/go-twitter v0.0.0-20201011215211-4b180d0cc78d
 	github.com/dghubble/oauth1 v0.7.1
-	github.com/dstotijn/go-notion v0.6.1
+	github.com/dstotijn/go-notion v0.11.0
 	github.com/dustin/go-humanize v1.0.0
 	github.com/fatih/color v1.15.0
 	github.com/gabriel-vasile/mimetype v1.4.2
@@ -25,7 +25,7 @@ require (
 	github.com/ipsn/go-libtor v1.0.380
 	github.com/jedib0t/go-pretty/v6 v6.4.0
 	github.com/mattn/go-mastodon v0.0.5-0.20210515144304-86627ec7d635
-	github.com/nbd-wtf/go-nostr v0.13.1
+	github.com/nbd-wtf/go-nostr v0.17.1-0.20230426111250-32ca737acf77
 	github.com/phf/go-queue v0.0.0-20170504031614-9abe38d0371d
 	github.com/pkg/errors v0.9.1
 	github.com/prometheus/client_golang v1.14.0
@@ -56,13 +56,14 @@ require (
 
 require (
 	github.com/MercuryEngineering/CookieMonster v0.0.0-20180304172713-1584578b3403 // indirect
-	github.com/SaveTheRbtz/generic-sync-map-go v0.0.0-20220414055132-a37292614db8 // indirect
+	github.com/SaveTheRbtz/generic-sync-map-go v0.0.0-20230201052002-6c5833b989be // indirect
 	github.com/VividCortex/ewma v1.2.0 // indirect
 	github.com/andybalholm/cascadia v1.3.1 // indirect
 	github.com/benbjohnson/clock v1.3.0 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/bitly/go-simplejson v0.5.0 // indirect
-	github.com/btcsuite/btcd/btcec/v2 v2.2.0 // indirect
+	github.com/btcsuite/btcd/btcec/v2 v2.3.2 // indirect
+	github.com/btcsuite/btcd/btcutil v1.1.3 // indirect
 	github.com/btcsuite/btcd/chaincfg/chainhash v1.0.2 // indirect
 	github.com/cenkalti/backoff v2.2.1+incompatible // indirect
 	github.com/cenkalti/backoff/v4 v4.2.0 // indirect
@@ -72,8 +73,8 @@ require (
 	github.com/chromedp/chromedp v0.9.1 // indirect
 	github.com/chromedp/sysutil v1.0.0 // indirect
 	github.com/crackcomm/go-gitignore v0.0.0-20170627025303-887ab5e44cc3 // indirect
-	github.com/decred/dcrd/crypto/blake256 v1.0.0 // indirect
-	github.com/decred/dcrd/dcrec/secp256k1/v4 v4.1.0 // indirect
+	github.com/decred/dcrd/crypto/blake256 v1.0.1 // indirect
+	github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0 // indirect
 	github.com/dghubble/sling v1.3.0 // indirect
 	github.com/dlclark/regexp2 v1.7.0 // indirect
 	github.com/dop251/goja v0.0.0-20221115122301-6c0d9883792e // indirect
@@ -142,12 +143,11 @@ require (
 	github.com/tidwall/pretty v1.2.0 // indirect
 	github.com/tidwall/sjson v1.2.4 // indirect
 	github.com/tomnomnom/linkheader v0.0.0-20180905144013-02ca5825eb80 // indirect
-	github.com/valyala/fastjson v1.6.3 // indirect
 	github.com/wabarc/memento v0.0.0-20210703205719-adc2f8ab8bae // indirect
 	github.com/whyrusleeping/tar-utils v0.0.0-20201201191210-20a61371de5b // indirect
 	github.com/ybbus/httpretry v1.0.1 // indirect
 	golang.org/x/crypto v0.4.0 // indirect
-	golang.org/x/exp v0.0.0-20221205204356-47842c84f3db // indirect
+	golang.org/x/exp v0.0.0-20230425010034-47ecfdc1ba53 // indirect
 	golang.org/x/sys v0.7.0 // indirect
 	golang.org/x/text v0.9.0 // indirect
 	google.golang.org/protobuf v1.28.1 // indirect