Skip to content

Latest commit

 

History

History
121 lines (110 loc) · 2.94 KB

README.md

File metadata and controls

121 lines (110 loc) · 2.94 KB

Feed Scraper

This is a simple project that can fetch feed's data from different source, supporting different protocols (Atom, RSS).

API

Subscribe Feed:

curl -X POST "http://127.0.0.1:8080/feed/subscribe" \
	--header "Content-Type: application/json" \
	--data '{"userId":"1", "url":"https://rss.art19.com/apology-line"}'

Unsubscription Feed:

curl -X PUT http://127.0.0.1:8080/feed/unSubscribe \
  	--header "Content-Type: application/json" \
	--data '{"userId":"1", "feedId":"1"}'

Feed Update :

curl -X PUT http://127.0.0.1:8080/feed/update \  
	--header "Content-Type: application/json"\
	--data '{"userId":"1", "feedId":"8"}'

Get Feeds From Specific User :

curl "http://127.0.0.1:8080/users/{userId}/feeds?fetchSize=15&toDate=2022-07-11%2012:00:00"

Get Specific Feed Data:

curl "http://127.0.0.1:8080/users/2/feeds/7/items?fetchSize=15&toDate=2022-07-11%2012:00:00"

Bookmark Specific FeedItem:

curl -X PUT "http://127.0.0.1:8080/feed-item/bookmark" \
	--data '{"userId":"2", "feedId":"7", "feedItemId":"14423"}' \
	--header "Content-Type: application/json"

UnBookmark Specific FeedItem:

curl -X PUT "http://127.0.0.1:8080/feed-item/unBookmark" \
	--data '{"userId":"2", "feedId":"7", "feedItemId":"14423"}' \
	--header "Content-Type: application/json"

Get Bookmark FeedItems:

curl "http://127.0.0.1:8080//users/1/bookmarked"

Set up

  1. Before running the project, it's necessary to create DB schema. I used JOOQ framework to work with the database, And it needs some classes to be generated at compile time. So for building the project, it is necessary to run 00_create_tables.sql in a PostgreSQL DB.
  2. Then change build.gradle by replacing your database date
jooq {
  version = '3.16.5'
  edition = nu.studer.gradle.jooq.JooqEdition.OSS
  configurations {
    main {
      generateSchemaSourceOnCompilation = true

      generationTool {
        jdbc {
          driver = 'org.postgresql.Driver'
          url = 'jdbc:postgresql://192.168.122.242:5432/feed'
          user = 'feed'
          password = 'feed'
        }
        ...
      }
      ...
    }
    ...
  }
  ...
}
  1. Build the project
./gradlew build
  1. [Just in case] Above command should create needed classes automatically. but if you have some issues, you can run these commands:
./gradlew cleanGenerateJooq
./gradlew generateJooq
  1. Create docker images and run them
docker build -t simple-feed-scraper-1.0.jar . 
docker-compose up -d

Scripts

To Generate JOOQ classes based on database schema

./gradlew generateJooq

To Clean JOOQ classes

./gradlew cleanGenerateJooq

To Compile

./gradlew compileJava

To Build

./gradlew build

To Unit Test

./gradlew test

To Integration Test

./gradlew integrationTest