feat: v4 (#9)

* wip: v4 * wip: v4 * feat: add byebug * feat: spec_data * feat: update rules * refactor: secret_\d -> secret\d * feat: move audio data parsing into utility callable * wip: spec files * refactor: fix rubocop warnings * wip: full rework * feat: move legacy code into ignored directory * fix: move utility callables into utility namespace * feat: audio class * fix: requires * feat: JSON dep * fix: wrong method call * feat: .env * feat: console * fix: requiring utility classes * feat: logger * feat: update rules * feat: request helper * feat: update rules * wip: logging in * feat: spec helpers * feat: login requests * feat: cache response * feat: login * feat: update rules * feat: log redirect path on login failure * feat: link decoder * feat: docs * wip: all main methods * feat: playlist * feat: before_call and after_call hooks * feat: remove empty spec files * feat: update rules * feat: web parsers * feat: playlist URL parser * feat: web parsers * refactor: use class in callables * wip: search request * feat: playlist node parser * fix: require new utilities * fix: require web parsers * feat: use web parsers * feat: search parser * feat: more tests * feat: method forwarding * feat: cookies storing * feat: allow disabling rubocop * fix: no results case * feat: find method * feat: audio items parser * feat: playlist web parser * feat: do not throw exceptions from URL parser * feat: allow storing client id in parsers * fix: my page parsing * feat: update rules * feat: section parser * feat: section request * wip: playlist parsing * feat: login_helper * feat: section requesting * feat: debug log all requests * feat: playlists loading * fix: storing mechanize cookies * fix: large playlist loading * test: large playlist test * feat: use vcr and webmock to improve requests testing * feat: raise error when captcha met * fix: user ID parsing * feat: profile ID resolver * fix: tests * refactor: satisfy rubocop * feat: move sections loading into separate loader * fix: full url case * fix: playlist gibberish and tests for #audios * feat: #audios * refactor: rename Section utilities into PlaylistSection * fix: remove doubled test * feat: return self in Request#call * feat: last post id from profile * fix: correctly handle nil case * feat: auto-load files * fix: base request tests * feat: wall * feat: url_*? methods * fix: parser * feat: Audio#like?(audio) * feat: Playlist#audios and delegations * feat: audio methods * feat: Client#post * test: also check 404 and feed * feat: simplecov * feat: additional tests * fix: require base class first * fix: audio full id * feat: id matcher * fix: IDs parsing * feat: improved data handling for WebParser::Base * fix: audio loading exceeds limits * feat: Client#get_urls * feat: Client#update_urls * test: use large data for URL updating test * feat: loading audio URL * feat: get rid of responses and move spec data * feat: update texts * refactor: namings and doubled code * feat: data type guesser * feat: expose encoded URL
fizvlad · Nov 26, 2020 · 2073128 · 2073128
1 parent 797ada9
commit 2073128
Show file tree

Hide file tree

Showing 90 changed files with 3,060 additions and 2,112 deletions.
diff --git a/.env.example b/.env.example
@@ -0,0 +1,3 @@
+# VkMusic spcification loads ENV variables from .env files
+VK_LOGIN=+79991234567
+VK_PASSWORD=password
diff --git a/.gitignore b/.gitignore
@@ -1,8 +1,14 @@
+/.env
+/.byebug_history
 /.bundle/
+/.legacy/
 /.yardoc
 /_yardoc/
 /coverage/
 /doc/
 /pkg/
 /spec/reports/
+/spec/.cookies
+/spec/cassetes/*
+!/spec/cassetes/README.md
 /tmp/
diff --git a/.rspec b/.rspec
@@ -0,0 +1 @@
+--require spec_helper
diff --git a/.rubocop.yml b/.rubocop.yml
@@ -0,0 +1,56 @@
+# The behavior of RuboCop can be controlled via the .rubocop.yml
+# configuration file. It makes it possible to enable/disable
+# certain cops (checks) and to alter their behavior if they accept
+# any parameters. The file can be placed either in your home
+# directory or in some project directory.
+#
+# RuboCop will start looking for the configuration file in the directory
+# where the inspected file is and continue its way up to the root directory.
+#
+# See https://docs.rubocop.org/rubocop/configuration
+
+AllCops:
+  TargetRubyVersion: 2.7
+  EnabledByDefault: true
+
+Layout/FirstMethodParameterLineBreak:
+  Enabled: false
+Layout/FirstMethodArgumentLineBreak:
+  Enabled: false
+Layout/MultilineMethodArgumentLineBreaks:
+  Enabled: false
+Layout/EndOfLine:
+  Enabled: false
+Layout/EndAlignment:
+  EnforcedStyleAlignWith: variable
+Layout/FirstHashElementIndentation:
+  Enabled: false
+Layout/MultilineAssignmentLayout:
+  EnforcedStyle: same_line
+Layout/CaseIndentation:
+  EnforcedStyle: end
+Lint/ConstantResolution:
+  Enabled: false
+Metrics/ParameterLists:
+  Enabled: false
+Metrics/BlockLength:
+  Exclude:
+    - 'spec/**/*_spec.rb'
+Style/MethodCallWithArgsParentheses:
+  Enabled: false
+Style/Copyright:
+  Enabled: false
+Style/ClassVars:
+  Enabled: false
+Style/MissingElse:
+  Enabled: false
+Style/StringHashKeys:
+  Enabled: false
+Style/ImplicitRuntimeError:
+  Enabled: false
+Style/NumericLiterals:
+  Enabled: false
+Style/DisableCopsWithinSourceCodeDirective:
+  Enabled: false
+Layout/MultilineHashKeyLineBreaks:
+  Enabled: false
diff --git a/.travis.yml b/.travis.yml
diff --git a/Gemfile b/Gemfile
@@ -1,10 +1,38 @@
-source "https://rubygems.org"
+# frozen_string_literal: true
+
+source 'https://rubygems.org'
 
 # Specify your gem's dependencies in vk_music.gemspec
 gemspec
 
-gem "logger", "~> 1.4"
+group :test, :development do
+  # Debugging console
+  gem 'pry'
+
+  # Another debugging console
+  gem 'byebug'
+
+  # The thing, forcing you to write good code
+  gem 'rubocop', require: false
+
+  # Rake tasks
+  gem 'rake', require: false
+
+  # Testing
+  gem 'rspec', require: false
+
+  # Docs
+  gem 'yard', require: false
+
+  # .env support
+  gem 'dotenv'
+
+  # Save web requests for fast and reliable testing
+  gem 'vcr'
 
-gem "mechanize", "~> 2.7"
+  # Stub web requests
+  gem 'webmock'
 
-gem "pry", "~> 0.12.2"
+  # Test coverage
+  gem 'simplecov', require: false, group: :test
+end
diff --git a/Gemfile.lock b/Gemfile.lock
@@ -1,20 +1,29 @@
 PATH
   remote: .
   specs:
-    vk_music (3.1.7)
+    vk_music (4.0.0)
       execjs (~> 2.7)
-      json (~> 2.0)
+      json (~> 2.3)
       logger (~> 1.4)
       mechanize (~> 2.7)
       net-http-persistent (= 2.9.4)
 
 GEM
   remote: https://rubygems.org/
   specs:
+    addressable (2.7.0)
+      public_suffix (>= 2.0.2, < 5.0)
+    ast (2.4.1)
+    byebug (11.1.3)
     coderay (1.1.3)
+    crack (0.4.4)
+    diff-lcs (1.4.4)
+    docile (1.3.2)
     domain_name (0.5.20190701)
       unf (>= 0.0.5, < 1.0.0)
+    dotenv (2.7.6)
     execjs (2.7.0)
+    hashdiff (1.0.1)
     http-cookie (1.0.3)
       domain_name (~> 0.5)
     json (2.3.1)
@@ -28,27 +37,68 @@ GEM
       nokogiri (~> 1.6)
       ntlm-http (~> 0.1, >= 0.1.1)
       webrobots (>= 0.0.9, < 0.2)
-    method_source (0.9.2)
+    method_source (1.0.0)
     mime-types (3.3.1)
       mime-types-data (~> 3.2015)
-    mime-types-data (3.2020.0512)
+    mime-types-data (3.2020.1104)
     mini_portile2 (2.4.0)
-    minitest (5.14.1)
     net-http-digest_auth (1.4.1)
     net-http-persistent (2.9.4)
     nokogiri (1.10.10)
       mini_portile2 (~> 2.4.0)
     nokogiri (1.10.10-x64-mingw32)
       mini_portile2 (~> 2.4.0)
     ntlm-http (0.1.1)
-    pry (0.12.2)
-      coderay (~> 1.1.0)
-      method_source (~> 0.9.0)
+    parallel (1.19.2)
+    parser (2.7.2.0)
+      ast (~> 2.4.1)
+    pry (0.13.1)
+      coderay (~> 1.1)
+      method_source (~> 1.0)
+    public_suffix (4.0.6)
+    rainbow (3.0.0)
     rake (13.0.1)
+    regexp_parser (1.8.2)
+    rexml (3.2.4)
+    rspec (3.10.0)
+      rspec-core (~> 3.10.0)
+      rspec-expectations (~> 3.10.0)
+      rspec-mocks (~> 3.10.0)
+    rspec-core (3.10.0)
+      rspec-support (~> 3.10.0)
+    rspec-expectations (3.10.0)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.10.0)
+    rspec-mocks (3.10.0)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.10.0)
+    rspec-support (3.10.0)
+    rubocop (1.2.0)
+      parallel (~> 1.10)
+      parser (>= 2.7.1.5)
+      rainbow (>= 2.2.2, < 4.0)
+      regexp_parser (>= 1.8)
+      rexml
+      rubocop-ast (>= 1.0.1)
+      ruby-progressbar (~> 1.7)
+      unicode-display_width (>= 1.4.0, < 2.0)
+    rubocop-ast (1.1.1)
+      parser (>= 2.7.1.5)
+    ruby-progressbar (1.10.1)
+    simplecov (0.19.1)
+      docile (~> 1.1)
+      simplecov-html (~> 0.11)
+    simplecov-html (0.12.3)
     unf (0.1.4)
       unf_ext
     unf_ext (0.0.7.7)
     unf_ext (0.0.7.7-x64-mingw32)
+    unicode-display_width (1.7.0)
+    vcr (6.0.0)
+    webmock (3.9.5)
+      addressable (>= 2.3.6)
+      crack (>= 0.3.2)
+      hashdiff (>= 0.4.0, < 2.0.0)
     webrobots (0.1.2)
     yard (0.9.25)
 
@@ -57,14 +107,17 @@ PLATFORMS
   x64-mingw32
 
 DEPENDENCIES
-  bundler (~> 2.0)
-  logger (~> 1.4)
-  mechanize (~> 2.7)
-  minitest (~> 5.0)
-  pry (~> 0.12.2)
-  rake (~> 13.0)
+  byebug
+  dotenv
+  pry
+  rake
+  rspec
+  rubocop
+  simplecov
+  vcr
   vk_music!
-  yard (~> 0.9)
+  webmock
+  yard
 
 BUNDLED WITH
    2.1.4
diff --git a/README.md b/README.md
@@ -1,6 +1,6 @@
 # VkMusic
 
-*vk_music* gem is a library to work with audios on popular Russian social network [vk.com](https://www.vk.com "vk.com"). VK disabled their public API for audios, so it is now necessary to use parsers instead.
+*vk_music* gem is a library to work with audios on popular Russian social network
 
 ## Installation
 
@@ -11,79 +11,96 @@ gem 'vk_music'
 ```
 
 And then execute:
+```
+$ bundle
+```
 
-    $ bundle
-
-Or install it yourself as:
+Or install it using `gem`:
 
-    $ gem install vk_music
+```
+$ gem install vk_music
+```
 
 ## Usage
 
-You can take a look on documentation [here](https://www.rubydoc.info/gems/vk_music/).
+You can take a look on documentation at [rubydoc](https://www.rubydoc.info/gems/vk_music/).
 
 ### Logging in
-Firstly, it is required to create new *VkMusic::Client* and provide login credentials:
+
+Firstly, it is required to create new `VkMusic::Client` instance and provide login credentials:
 
 ```ruby
-client = VkMusic::Client.new(username: "+71234567890", password: "password")
+client = VkMusic::Client.new(username: "+79991234567", password: "password")
 ```
 
-### Searching for audios
-You can search audios by name with following method:
+### Search
+
+You can search audios using `Client#find`:
 
 ```ruby
 audios = client.find("Acid Spit - Mega Drive")
-puts audios[0]     # Basic information about audio
-puts audios[0].url # URL to access audio. Notice that it is only accessible from your IP
 ```
 
-### Parsing playlists
-You can load all the audios from playlist using following method:
+You can also search for playlists using same method:
 
 ```ruby
-playlist = client.playlist(url: "https://vk.com/audio?z=audio_playlist-37661843_1/0e420c32c8b69e6637")
-last_audios = playlist.first(10) # => Array<Audio>
-client.update_urls(last_audios) # We have to manually retrieve URLs for playlists
-urls = last_audios.map(&:url) # URLs for every audio
+playlists = client.find("Jazz", type: :playlist)
+```
+
+### Playlists
+
+You can load playlist audios with `Client#playlist`
+
+```ruby
+playlist = client.playlist(url: "link")
 ```
 
 ### User or group audios
-You can load audios from user or group page. Those audios will be returned as playlist. To do it simply pass user or group id:
+
+You can load profile audios with `Client#audios`
 
 ```ruby
 playlist = client.audios(owner_id: 8024985)
-last_audios = playlist.first(10) # => Array<Audio>
-client.update_urls(last_audios) # We have to manually retrieve URLs for playlists
-urls = last_audios.map(&:url) # URLs for every audio
 ```
-You can set how many audios you actually need as well:
+
+### Wall audios
+
+You can load audios from profile wall with `Client#wall`
 
 ```ruby
-user_playlist = client.audios(url: "vk.com/id8024985", up_to: 10)
+playlist = client.wall(owner_id: 8024985)
 ```
 
 ### Audios from post
+
 You can load up to 10 audios attached to some post. Those audios will be returned as array:
 
 ```ruby
-audios = client.post(url: "https://vk.com/wall-4790861_5453")
-urls = audios.map(&:url) # URLs for every audio
+audios = client.post(url: "link")
 ```
 
-### Recommended audios and etc
-You can load audios from recommended sections, novices and VK charts:
+### Getting audio URL
+
+To get audio URL you should go through following chain:
+1. Get audio ID
+2. Get audio encrypted URL
+3. Get audio decrypted URL
+
+Usually most of audios already go with ID. Getting encrypted URL requires additional request to web, so it is performed with `Client#update_urls`, which will mutate provided array of audios:
 
 ```ruby
-audios = client.block(url: "https://m.vk.com/audio?act=block&block=PUlQVA8GR0R3W0tMF2teRGpJUVQPGVpfdF1YRwMGXUpkXktMF2tYUWRHS0IXDlpKZFpcVA8FFg")
+client.update_urls(audios_array)
 ```
 
+After this you can get decrypted URL using `Audio#url`
 
 ## Development
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+Feel free to add features. However, please make sure all your code is covered with tests.
+
+### Testing
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+This gem uses `rspec` and `vcr` for easy testing. *Be careful*, though, running `rspec` in clean repo may result in lot of requests to web and rate limiting or ban
 
 ## Contributing