Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Original Bovespa's API is dead #5

Open
nihey opened this issue Jul 7, 2016 · 53 comments
Open

Original Bovespa's API is dead #5

nihey opened this issue Jul 7, 2016 · 53 comments
Assignees

Comments

@nihey
Copy link
Owner

nihey commented Jul 7, 2016

I need to find another way to scrape the data.

@carlopires
Copy link

Do we have some way to get data from B3 already?

@nihey
Copy link
Owner Author

nihey commented Jul 28, 2017

As of now, there is no open current way to scrape it that I know of.

It seems there are however some paid alternatives, like: Bolsa Financeira

Upon doing some search, there seems to be way, this page gets it using a WebSocket protocol, the data seems to appear like this:
screenshot from 2017-07-28 19-55-36

There is also this API, it seems to return just stock codes, but I do not know the full extent of it:

$ curl "https://data-bovespa.tradingview.com/search/?text=ABEV3&exchange=BMFBOVESPA&type=&domain=bovespa" -H "Origin: https://s.tradingview.com"

Whenever I get some free time, I'll try to migrate node-bovespa to use this (it won't be as smooth wrapping a REST API, but it is doable).

@carlopires
Copy link

This is weird. TradingView uses data from ICE data services. My main concern is due to the reliability of data, so I wanted to have data as close as possible from bovespa's services.

@lenilsonjr
Copy link

lenilsonjr commented Oct 27, 2017

Do you guys got any progress on finding another source for the data?

Alpha Vantage seems to be a great alternative. It is free (although it needs registration to obtain a API Key), and have quotes from B3 (with some delay, of course). It is also worth noting that some of the assets listed on B3, like subscription rights, are not available in Alpha Vantage.

And of course, depending on your goal, it might not be a good idea to rely on a third party for stock quotes.

@nihey
Copy link
Owner Author

nihey commented Oct 28, 2017

@lenilsonjr Thanks for the information, Alpha Vantage might be the easiest way to got from now. I'll work on it as soon as possible.

@djalmaaraujo
Copy link

@lenilsonjr @nihey I could not find B3 quotes in Alpha. How you should pass in the URL? CSNA:B3? This is not working for me

@patrickbard
Copy link

@lenilsonjr @nihey I could not find B3 quotes in Alpha. How you should pass in the URL? CSNA:B3? This is not working for me

Don't know if it's still relevant to you, but I could get appending .SA to the stock symbol, similar how Yahoo Finance does.

Example:

https://www.alphavantage.co/query?...&symbol=PETR4.SA)

@nihey
Copy link
Owner Author

nihey commented Mar 30, 2019

Hey everyone, I'm planning to make this project alive once more.

I've been thinking about the plans on how to do it and found out that we can download historical series on these files.

My plan to is to make an API out of it for historical data querying. After doing it, the plan is to offer it publicly (or self hosted via the package).

Once the historical data API is done, I can add a realtime API via webscraping from various sources. It may be hard to maintain it, but we'll see the best way to handle it once we reach this point.

@nihey
Copy link
Owner Author

nihey commented Apr 1, 2019

I've managed to setup a new API:

https://bovespa.nihey.org/api/quote/ABEV3/2018-02-14

I'll rewrite the code for node-bovespa so that it uses this brand new API, until then, I recommend using this API directly for whoever needs consuming this data.

Note: it currently do not have realtime data, just historic ones.

@djalmaaraujo
Copy link

Hey, this is awesome. I just read your email and I am all up to make this project be big. How can I help? Do we have a roadmap?

@nihey
Copy link
Owner Author

nihey commented Apr 1, 2019

@djalmaaraujo Great to see you're interested 😄 !

I'll get everything organized this week and possibly create issues, but as a brief explanation on everything I wanted for this project, we should add:

# Something like:
$ bovespa ABEV3 -d 2018-01-08
<outputs the data>
  • an easy way to access the data
// Something like:
const bovespa = require('bovespa');
bovespa.get("ABEV3", "2018-01-08").then(...)

⬆️ That would be it for now, but I'm open for suggestions too.

@djalmaaraujo
Copy link

@nihey A few days ago I received an email for a legal department of XP and a while ago also from bank safra, because I had open source projects with their name or part of it. I am pretty sure this will be a near future for this project as-well, which sucks.

Also, I really love the idea but while I am very interested I am on zero time now, so I am sorry if I passed the wrong impression in my last comment.

I think the roadmap is great but I would ignore the CLI for now since this is not very used compared to an API. That's my opinion.

I will try to find time for this anytime soon. tks

@nihey
Copy link
Owner Author

nihey commented Apr 16, 2019

@djalmaaraujo

A few days ago I received an email for a legal department of XP and a while ago also from bank safra, because I had open source projects with their name or part of it. I am pretty sure this will be a near future for this project as-well, which sucks.

That sucks a lot, I really hope that if this happens here, it takes a long time to happen.

Also, I really love the idea but while I am very interested I am on zero time now, so I am sorry if I passed the wrong impression in my last comment.

No problems!

I think the roadmap is great but I would ignore the CLI for now since this is not very used compared to an API. That's my opinion.

I think you're right, although the CLI can be quite handy sometimes, it is not nearly as useful as the API
I should definitely add some documentation not only for the node API, but for the REST API too, and this should be the main focus for the project (ideally aiming into the real time API).

I will try to find time for this anytime soon. tks

Alright, thanks!

@krolow
Copy link

krolow commented May 7, 2019

I was searching for some solution to get real time and endup here...

Any thoughts in get real-time data?

I'm doing one dashboard for myself my current solution is using google sheet, is not the best one as it does not reflect the current stock price (they say it can be like 20min delay), but at least I get something close to the current stock price like using the formula =GOOGLEFINANCE("ITUB4"; "price")

@krolow
Copy link

krolow commented May 7, 2019

Maybe for historical data, you can use the same? I think it's going to be pretty easy to maintain...

@fernandoghisi
Copy link

@krolow but how do you get the data from google sheets? I mean, is there a way to get the sheet generated data from code running outside (not using google script)?

@fernandoghisi
Copy link

BTW, I was playing with Alpha Vantage (using Node), but they now have some limits ("our standard API call frequency is 5 calls per minute and 500 calls per day") and, in this case, it would work just to get historical data - for example, updating daily a database with 500 assets at most.

@krolow
Copy link

krolow commented Jul 13, 2019

so the way that i'm using right now I have a public sheet with all the stock codes and I put it to export as CSV and I consume that CSV url when I need to fetch it... not perfect but for my own purposes it's working well...

and what I did was to also have several columns so I can have the history of the stock in the last week, 30d ago, 60d ago etc...

@nihey
Copy link
Owner Author

nihey commented Jul 18, 2019

@krolow This might be useful too: https://sheetsu.com/ I have not used them yet, but its looks like a good alternative to extract the data from a spreadsheet via API.

@krolow
Copy link

krolow commented Jul 18, 2019

nice I did a small demo just to test the idea https://github.com/krolow/carteira/ it's basically consuming the CSV of a public sheet

@djalmaaraujo
Copy link

Hey guys, not sure if this helps at some point, but I am building a integrated database with all the companies/papers in b3.

https://www.tradertax.com.br/api/v1/b3_companies.json?query=xp

At some point this will be private, or not. I get info from a bunch of sources. Some companies don't have the CNPJ attached, I am working on that.

@nihey
Copy link
Owner Author

nihey commented Jul 18, 2019

@djalmaaraujo I'm not sure if node-bovespa could use it, but I can see some uses of it on some of my personal projects.

Seeing these last updates I think I've came up with a reliable solution to keep the real time data. I'll take some time to implement it, but it surely will be a major enhancement on this project.

@djalmaaraujo
Copy link

@nihey That's what I thought. Are you using the spreadsheet solution or alphavantage?

@nihey
Copy link
Owner Author

nihey commented Jul 22, 2019

@djalmaaraujo I'm using the SpreadSheet solution for now.

@djalmaaraujo
Copy link

@nihey I could not find in the GOOGLEFINANCE formula ETF's: SMLL11 for example. Can you?

@nihey
Copy link
Owner Author

nihey commented Jul 22, 2019

@djalmaaraujo I could not find SMLL11 indeed, even though some other ETF's could be found (like XBOV11). Seems like we may have to extract these data from AlphaVantage (or another source anyway).

@krolow
Copy link

krolow commented Jul 22, 2019

have u tried SMAL11? it worked for me here, also I have tried ETF IVVB11 works as well

@djalmaaraujo
Copy link

@krolow Interesting. The hard part is to find all possibilities, because it's not the ticker name.

@krolow
Copy link

krolow commented Jul 22, 2019 via email

@nihey
Copy link
Owner Author

nihey commented Aug 14, 2019

So, I'm finally coming up with a solution, I've Google Finance API Proxy, consuming data from spreadsheets, storing them in the database. If any of you want to try it out, I'm keeping the API at: https://bovespa.nihey.org/api/realtime (or https://bovespa.nihey.org/api/realtime/ABEV3).

I'll also add AlphaVantage API proxying at some point, but it seems like this should be enough for the next version.

So, finishing a couple of things will mark the next version of node-bovespa:

  • Add data for the current day back to the cli
  • Document and add these new methods to README.md

@nihey nihey self-assigned this Aug 14, 2019
@djalmaaraujo
Copy link

@nihey This is a great addition. Great job. Realtime is actually 15min, right? I saw google formulas uses 15min delay, I think

@nihey
Copy link
Owner Author

nihey commented Aug 14, 2019

@djalmaaraujo Yes, that's right! As Google Finance API actually has delayed data, we are limited by it, it is not actually "realtime".

@irae
Copy link

irae commented Aug 16, 2019

Hello,

I've been trying to solve the same issue, but I don't quite like the Google Finance solution. As mentioned before, there is no SMAL11 and other ETFs. It also does not have commodities, such as gold (OZ1D, OZ2D, OZ3D), and also no options.

Real time is an issue, because it appears there is no API at all for the assets I mentioned. Some APIs suck as Yahoo Finance and Alpha Vantage only have reliable delayed data for SMAL11 and other ETFs but not for commodities. But all of those break on the time series data beyond a couple of days or so. Looks like Alpha Vantage keeps reliable data in 5m intervals for everything it has for 2 days only.

For daily data I've been using B3 historical data, which is behind a captcha and available about 3 hours after marked closes. It has reliable data for most options, ETFs and BDRs as well as regular stocks. But being behind a captcha makes it ok for development but bad for automation.

http://www.b3.com.br/pt_br/market-data-e-indices/servicos-de-dados/market-data/historico/mercado-a-vista/series-historicas/

Another problem with B3 is that it does not contain splits and adjusted data points. And I was not able to find an open api for splits and other corporate events.

So recently I discovered BMF FTP service, which is 2 days behind, but it has all operations, not only open/close/min/max/vol data points. It is easy to crawl and automate. But I don't have any idea if it will be kept up indefinitely.

ftp://ftp.bmf.com.br/MarketData/NEG_LAYOUT_english.txt

The plan I am thinking for my project is as follows:

  1. When bootstrapping the DB, use BMF FTP data, easy to crawl and extract data with filesystem → gunzip → split → parse streams
  2. Crawl BMF FTP daily for 2 days ago quotes
  3. All BMF data stored in DB
  4. Use Alpha Vantage plus in memory cache for past 2 days

This looks like the least shortcoming possible. The only missing data within 2 days would be commodities and 5m and 1min data is restricted to 1 day only on all other assets.

I thought I should share the idea here, as I still didn't build an abstraction and DB schema for that and could potentially use and contribute to this project if you are also aiming to close the gaps on ETFs, BDRs and commodities as best as we can. Do you have any thoughts on that?

@irae
Copy link

irae commented Aug 16, 2019

Forgot to mention: Looks like splits and dividends are possible to crawl from Alpha Vantage's TIME_SERIES_DAILY_ADJUSTED . Not sure how it would fit in my idea yet 🤔

@nihey
Copy link
Owner Author

nihey commented Aug 19, 2019

I've been trying to solve the same issue, but I don't quite like the Google Finance solution. As mentioned before, there is no SMAL11 and other ETFs. It also does not have commodities, such as gold (OZ1D, OZ2D, OZ3D), and also no options.

Yeah, over the time, I've found google_finance solution to be quite limited. I'll make another version with it just so we can fill some gaps, but it surely won't be enough.

For daily data I've been using B3 historical data, which is behind a captcha and available about 3 hours after marked closes. It has reliable data for most options, ETFs and BDRs as well as regular stocks. But being behind a captcha makes it ok for development but bad for automation.

I'm using those to provide the historical API too. The funny thing is: their CAPTCHA is not actually very strong. If you get the download link for their files and look at its name, you'll find out you can download it straight from the website, without having to solve the CAPTCHA again. This allowed me to automate their data extraction entirely.

Example: You can download 2019 data directly from: http://bvmf.bmfbovespa.com.br/InstDados/SerHist/COTAHIST_A2019.zip . If you want further details, you can look on this file, where I do this.

When bootstrapping the DB, use BMF FTP data, easy to crawl and extract data with filesystem → gunzip → split → parse streams
Crawl BMF FTP daily for 2 days ago quotes
All BMF data stored in DB
Use Alpha Vantage plus in memory cache for past 2 days

I think this is the way to go. This is actually what I'm doing with this project too. The only things that could be improved are the data sources used to feed the database. As of now, I believe the best setting would be: B3's site historical data for past day data, Google Finance + AlphaVantage for realtime.

Forgot to mention: Looks like splits and dividends are possible to crawl from Alpha Vantage's TIME_SERIES_DAILY_ADJUSTED . Not sure how it would fit in my idea yet

That's a very good information to crawl, sounds like something that would be very hard to extract directly from B3's website. Thanks for the information!

@irae
Copy link

irae commented Aug 19, 2019

Here is something I found out during the weekend:

Closing price of COTAHIST_A2019.zip and similar files (don't know how long back it goes) is wrong! It is using the highest buy offer at closing time instead of last successful transaction. From BMF FTP not only I can get Options and Commodities historical data, but also information looks more reliable.

I didn't compare with Alpha Vantage or other free APIs. I did compare COTAHIST_A2019.zip to NEG_20190812.gz (after running reduce operations) and also with my homebroker COTAHIST_A2019.zip was the one that looked wrong to me.

I am not sure I want to report this to BMF, in case they discover this URLs are still up and decide to remove those. IDK if you notice, but they stopped linking this particular files from the official B3 website. The URL still exists, but the navigation won't let people without the bookmark find it again.

I plan to have this code extracted from my codebase as small npm modules. Maybe stuff like 'bmf-parser' and 'cotahist-parser'. Something like that, that we can all share and help maintain, improve etc. For now, each day takes about 7 seconds on my i7 computer with SSD. I am reading the compressed file and streaming results using es-stream internally. I'll try to have some time to accomplish this before end of week, but might take a bit of time, as I don't like to post npm packages without testing and I am still in prototype stage (no tests written yet).

@gfcantu
Copy link

gfcantu commented May 12, 2020

Hi, we have all this data and more, from bonds, funds to equities.
We want to make it avaliable for more people to use (for free), but we need some help.
If you guys want, we can talk to see what we can develop together: gabriel@carteiraglobal.com

@milesibastos
Copy link

👋 hey guys! It was a loooong thread... I just landed here after bootstrap another solution this weekend and I learned some things here in the hard way 😅!
Great job you are doing here @nihey... nice parser u wrote!
So! My 50 cents to this thread will be:

curl https://arquivos.b3.com.br/apinegocios/ticker/ITUB4/2020-06-08

@gfcantu I would like to help

@nihey
Copy link
Owner Author

nihey commented Jun 18, 2020

@milesibastos Hey! Its good to hear you've liked it!

My work has been a little rough on the last weeks/months and I haven't been able to give as much attention to the project as I would like to.

curl https://arquivos.b3.com.br/apinegocios/ticker/ITUB4/2020-06-08

This API looks very promising, I believe it will be the easiest one to integrate so far.

Just sharing on this thread some thoughts I shared with @gfcantu privately too:

It seems we can also integrate Yahoo Finance as another source, they have an API that exposes JSON Data. An example of their API is:

bash "https://query1.finance.yahoo.com/v8/finance/chart/ITSA4.SA?region=US&lang=en-US&includePrePost=false&interval=2m&range=1d&corsDomain=finance.yahoo.com&.tsrc=finance"

@irae That's good to know, and it's really bad 😞 . Unfortunately we still do not have other sources on the database to compare to COTAHIST_A2019.zip files, But after adding at least one or two, one key feature would be to compare the prices all sources provide and select the ones that are less conflicting between them.

@irae
Copy link

irae commented Jun 18, 2020

Wow! I was about to reply saying that Yahoo Finance API is not worth because of symbols ending in "11" were broken. But apparently...

Turns out Yahoo Finance API fixed their 4 year old bug! 🎉

In turn, this fixed both the graphs on their site and AlphaVantage now has full data for a bug report I've sent both Yahoo and AlphaVantage about 2 years ago.

Details collapsed because I don't want to rewrite everything, I'll leave in the original tone for historical sake 😄

What I wrote before I tested their APIs again

Yahoo Finance is the most used source for stock data worldwide. Even Alpha-vantage and Google spreadsheed APIs apparently crawl Yahoo APIs.

The major issue with Yahoo Finance APIs refuses to fix a longstanding bug in all the symbols ending in 11 and some are found nowhere. You can get accurate data for symbols ending in 11 only for the past week or so, in 30mins intervals. Or intra day 5min intervals IIRC.

So all "real state funds" (fundos imobiliários) and a lot of units like BPAC11, BIDI11 will be missing accurate data on Yahoo APIs. I used to work at Yahoo and even probing some people from the right org didn't help them to fix it.

It gets worst for special symbols like fixed income funds such as IMAB11, FIXA11 and others will not even show on COTAHIST* and the FTP source I mentioned above, making those impossible to get on free or cheap APIs globally.

@irae
Copy link

irae commented Jun 18, 2020

BTW, SMAL11 is fixed:

curl "https://query1.finance.yahoo.com/v8/finance/chart/SMAL11.SA?region=US&lang=en-US&includePrePost=false&interval=2m&range=1ain=finance.yahoo.com&.tsrc=finance"

IMAB11 (used to just 404) now exists, but data is broken like SMAL11 was before:

curl "https://query1.finance.yahoo.com/v8/finance/chart/IMAB11.SA?region=US&lang=en-US&includePrePost=false&interval=2m&range=1d&corsDomain=finance.yahoo.com&.tsrc=finance"

output as of today (null everywhere)

{"chart":{"result":[{"meta":{"currency":"BRL","symbol":"IMAB11.SA","exchangeName":"SAO","instrumentType":"EQUITY","firstTradeDate":1558098000,"regularMarketTime":1592507793,"gmtoffset":-10800,"timezone":"BRT","exchangeTimezoneName":"America/Sao_Paulo","regularMarketPrice":74.63,"chartPreviousClose":75.2,"previousClose":75.2,"scale":3,"priceHint":2,"currentTradingPeriod":{"pre":{"timezone":"BRT","start":1592484300,"end":1592485200,"gmtoffset":-10800},"regular":{"timezone":"BRT","start":1592485200,"end":1592510400,"gmtoffset":-10800},"post":{"timezone":"BRT","start":1592510400,"end":1592514000,"gmtoffset":-10800}},"tradingPeriods":[[{"timezone":"BRT","start":1592485200,"end":1592510400,"gmtoffset":-10800}]],"dataGranularity":"2m","range":"1d","validRanges":["1d","5d","1mo","3mo","6mo","1y","2y","ytd","max"]},"timestamp":[1592488080,1592488200,1592488320,1592488440,1592488560,1592488680,1592488800,1592488920,1592489040,1592489160,1592489280,1592489400,1592489520,1592489640,1592489760,1592489880,1592490000,1592490120,1592490240,1592490360,1592490480,1592490600,1592490720,1592490840,1592490960,1592491080,1592491200,1592491320,1592491440,1592491560,1592491680,1592491800,1592491920,1592492040,1592492160,1592492280,1592492400,1592492520,1592492640,1592492760,1592492880,1592493000,1592493120,1592493240,1592493360,1592493480,1592493600,1592493720,1592493840,1592493960,1592494080,1592494200,1592494320,1592494440,1592494560,1592494680,1592494800,1592494920,1592495040,1592495160,1592495280,1592495400,1592495520,1592495640,1592495760,1592495880,1592496000,1592496120,1592496240,1592496360,1592496480,1592496600,1592496720,1592496840,1592496960,1592497080,1592497200,1592497320,1592497440,1592497560,1592497680,1592497800,1592497920,1592498040,1592498160,1592498280,1592498400,1592498520,1592498640,1592498760,1592498880,1592499000,1592499120,1592499240,1592499360,1592499480,1592499600,1592499720,1592499840,1592499960,1592500080,1592500200,1592500320,1592500440,1592500560,1592500680,1592500800,1592500920,1592501040,1592501160,1592501280,1592501400,1592501520,1592501640,1592501760,1592501880,1592502000,1592502120,1592502240,1592502360,1592502480,1592502600,1592502720,1592502840,1592502960,1592503080,1592503200,1592503320,1592503440,1592503560,1592503680,1592503800,1592503920,1592504040,1592504160,1592504280,1592504400,1592504520,1592504640,1592504760,1592504880,1592505000,1592505120,1592505240,1592505360,1592505480,1592505600,1592505720,1592505840,1592505960,1592506080,1592506200,1592506320,1592506440,1592506560,1592506680,1592506800,1592506920,1592507040,1592507160,1592507280,1592507400,1592507520,1592507640,1592507760,1592507880,1592508000,1592508120,1592508240,1592508360,1592508480,1592508600,1592508720,1592508840,1592508960,1592509080,1592509200,1592509320,1592509440,1592509560,1592509680,1592509800,1592509920,1592510040,1592510160,1592510280],"indicators":{"quote":[{"low":[74.80000305175781,74.80000305175781,74.80000305175781,null,null,null,null,null,null,null,null,null,null,74.80999755859375,null,null,null,null,null,null,null,null,null,null,null,null,null,74.91000366210938,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,75.01000213623047,null,null,null,null,null,null,null,null,75.0199966430664,75.0199966430664,null,null,75.0199966430664,null,null,null,null,null,null,75.0999984741211,null,75.0999984741211,null,null,75.1500015258789,null,null,75.0199966430664,null,null,null,75.1500015258789,null,null,null,null,null,null,null,null,null,75.16000366210938,null,75.16000366210938,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,75.0199966430664,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,75.0,null,null,null,74.62999725341797,null,null,null,null,null,74.62999725341797,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],"volume":[0,0,313,null,null,null,null,null,null,null,null,null,null,300,null,null,null,null,null,null,null,null,null,null,null,null,null,70,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,112,null,null,null,null,null,null,null,null,150,690,null,null,0,null,null,null,null,null,null,10,null,6,null,null,129,null,null,0,null,null,null,1,null,null,null,null,null,null,null,null,null,7,null,133,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,4050,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,664,null,null,null,2670,null,null,null,null,null,14,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],"high":[75.19999694824219,74.80000305175781,74.80000305175781,null,null,null,null,null,null,null,null,null,null,74.80999755859375,null,null,null,null,null,null,null,null,null,null,null,null,null,74.91000366210938,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,75.01000213623047,null,null,null,null,null,null,null,null,75.0199966430664,75.0199966430664,null,null,75.0199966430664,null,null,null,null,null,null,75.0999984741211,null,75.0999984741211,null,null,75.1500015258789,null,null,75.0199966430664,null,null,null,75.1500015258789,null,null,null,null,null,null,null,null,null,75.16000366210938,null,75.16000366210938,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,75.0199966430664,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,75.0199966430664,null,null,null,75.0,null,null,null,null,null,74.62999725341797,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],"close":[74.80000305175781,74.80000305175781,74.80000305175781,null,null,null,null,null,null,null,null,null,null,74.80999755859375,null,null,null,null,null,null,null,null,null,null,null,null,null,74.91000366210938,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,75.01000213623047,null,null,null,null,null,null,null,null,75.0199966430664,75.0199966430664,null,null,75.0199966430664,null,null,null,null,null,null,75.0999984741211,null,75.0999984741211,null,null,75.1500015258789,null,null,75.0199966430664,null,null,null,75.1500015258789,null,null,null,null,null,null,null,null,null,75.16000366210938,null,75.16000366210938,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,75.0199966430664,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,75.0,null,null,null,74.62999725341797,null,null,null,null,null,74.62999725341797,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],"open":[75.19999694824219,74.80000305175781,74.80000305175781,null,null,null,null,null,null,null,null,null,null,74.80999755859375,null,null,null,null,null,null,null,null,null,null,null,null,null,74.91000366210938,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,75.01000213623047,null,null,null,null,null,null,null,null,75.0199966430664,75.0199966430664,null,null,75.0199966430664,null,null,null,null,null,null,75.0999984741211,null,75.0999984741211,null,null,75.1500015258789,null,null,75.0199966430664,null,null,null,75.1500015258789,null,null,null,null,null,null,null,null,null,75.16000366210938,null,75.16000366210938,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,75.0199966430664,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,75.0199966430664,null,null,null,75.0,null,null,null,null,null,74.62999725341797,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null]}]}}],"error":null}}

@GalaticStryder
Copy link

What a marvelous job over here @nihey looking forward to work with it on a new project I'm thinking about: using a Telegram bot account that sends notifications to the user according to the price of the desired stock, just to get a nice heads up when certain stock passes a resistance for example. 15min I think is pretty good enough for my purpose, I mean, realtime would be a killer feature to process on the long run considering the limited amount of calls the bot may have when ping the API.

The Yahoo Finance implementation seems feasible too, best wishes and good trades!

@Bordotti
Copy link

Guys i also need some help to gather bovespa stocks from Alpha Vantage, i tryed MTSA4, MTSA4.SAO, PETR4, PETR4.SAO, ^IBOV, any symbol give me a return. Can someone show me where i'm wrong?

@irae
Copy link

irae commented Sep 16, 2020

@Bordotti did you read the past comments? There is a lot of correct stock symbols for bovespa. I am pretty sure you can spot how it is done.

@Bordotti
Copy link

Sorry I wasn't clear, i need the fundamental data, the function TimeSeries it worked.

@lucasalmeida92
Copy link

Guys i also need some help to gather bovespa stocks from Alpha Vantage, i tryed MTSA4, MTSA4.SAO, PETR4, PETR4.SAO, ^IBOV, any symbol give me a return. Can someone show me where i'm wrong?

I don't know if this can help you man:
https://www.alphavantage.co/query?function=SYMBOL_SEARCH&keywords=IBOV&apikey=HBOWYYPHIYUXVK86

In my case, I'm trying to find a source to get 5min interval data of the "Mini Ibovespa" index (WIN) in intraday.

Does anyone know if I can get it in Yahoo Finance API ?

@nihey , congrats about the project! 😉
And everyone else, you are doing a great job so far! 🏆

@djalmaaraujo
Copy link

A hope: https://developers.b3.com.br/

@djalmaaraujo
Copy link

Ontem passei o dia trabalhando nesta api: https://b3-price.tradertax.com.br/tickers
Ela tem um número limitado de tickers, por motivos pessoais, mas caso queiram o codigo, só não vou deixar open por que já recebi cartinha de advogado da xp por causa de crawler, mas se quiserem me diz que eu adiciono no projeto

Mas resumindo, eu to usando esse link: https://arquivos.b3.com.br/apinegocios/ticker/ITUB4/2020-06-08

@djalmaaraujo
Copy link

djalmaaraujo commented Oct 22, 2020

Criei uma API atualizada a cada 24 horas com TODOS os tickers da bovespa: Inclui fundos, bdr, opções.

São 11.6mb de dados em um single request, não pus paginação ainda, então vou liberar caso alguém queira via request, sem custos. Manda email para mim em djalma [at] nossomos.cc.

@Bordotti
Copy link

@djalmaaraujo Eu tenho interesse em contribuir com o projeto.

@romeubertho
Copy link

@djalmaaraujo dei uma olhada no arquivos.b3.com.br, bem legal eles estarem disponibilizando o trade dos últimos dias, não sabia disso. Saudade do ftp deles (ftp.bvmf.com.br). Pra quem não sabe, eles forneciam dados para a reconstrução do book e os negócios eram bem mais detalhados que os dados públicos de hoje. Segue o header do negócios:

`Layout do arquivo NEG (Negócios)

Cabeçalho/Rodapé

Coluna Posição Inicial Tamanho Descrição

Identificação 1 2 RH Cabeçalho - RT Rodapé
Nome Arquivo 4 20 Nome do arquivo
Data Inicial 25 10 Data Inicial desse arquivo
Data Final 36 10 Data Final desse arquivo
Total de Linhas 47 9 No caso do rodapé apresenta o total de linhas do arquivo

Detalhes

Coluna Posição Inicial Tamanho Descrição

Data Sessão 1 10 Data de sessão
Símbolo do Instrumento 12 50 Símbolo do Instrumento
Nr.Negócio 63 10 Número do negócio
Preço Negócio 74 20 Preço do negócio
Quantidade 95 18 Quantidade negociada
Hora 114 15 Horário da negociação (formato HH:MM:SS.NNN)
Ind.Anulação 127 1 Indicador de Anulação: "1" - ativo / "2" - cancelado
Data Oferta Compra 129 10 Data da oferta de compra
Seq.Oferta Compra 140 15 Número sequencial da oferta de compra
GenerationID - Of.Compra 156 15 Número de geração (GenerationID) da Oferta de compra. Quando um negócio for gerado por 2 ofertas com quantidade escondida e isso gerar "n" linhas será gravado aqui a maior geração
Condição Oferta de Compra 172 1 Código que identifica a condição da oferta de compra. Pode ser: 0 - Oferta Neutra - é aquela que entra no mercado e não fecha com oferta existente. / 1 - Oferta Agressora - é aquela que ingressa no mercado para fechar com uma oferta existente. / 2 - Oferta Agredida - é a oferta (existente) que é fechada com uma oferta agressora.
Data Oferta Venda 174 10 Data da oferta de venda
Seq.Oferta Venda 185 15 Número sequencial da oferta de venda
GenerationID - Of.Venda 201 15 Número de geração (GenerationID) da Oferta de venda. Quando um negócio for gerado por 2 ofertas com quantidade escondida e isso gerar "n" linhas será gravado aqui a maior geração
Condição Oferta de Venda 217 1 Código que identifica a condição da oferta de venda. Pode ser: 0 - Oferta Neutra - é aquela que entra no mercado e não fecha com oferta existente. / 1 - Oferta Agressora - é aquela que ingressa no mercado para fechar com uma oferta existente. / 2 - Oferta Agredida - é a oferta (existente) que é fechada com uma oferta agressora.
Indicador de direto 219 1 Código que identifica se o negócio direto foi intencional: 1 - Intencional / 0 - Não Intencional
Corretora Compra 221 8 Código de identificação da corretora de compra - Disponível a partir de 03/2014
Corretora Venda 230 8 Código de identificação da corretora de venda - Disponível a partir de 03/2014

Obs: Delimitador das colunas de detalhes ';'
`

@henriquesss
Copy link

@djalmaaraujo também tenho interesse em contribuir

@irae
Copy link

irae commented Dec 21, 2020

Assim que os códigos forem open source, tiver repositórios pra testar, etc, eu me proponho a ajudar.
O maior problema, é que todos que querem fazer uma API subestimam a quantidade de recursos necessários pra manter no ar. Mesmo com a boa vontade, ou passa a monetizar, ou acaba fechando a API gratis.

Seria melhor ideia um repositório que tem vários adaptadores, um pra AlphaVantage, outro pra Yahoo APIs, e faz um DB de cache pra expor uma API padronizada, testada, etc com mais possibilidades do que somente a API free faria. Assim cada um roda a sua on damand, e escala seus próprios custos conforme o crescimento de seus produtos.

Se alguém começar um projeto assim (eu mesmo se arrumar tempo começaria), eu também gostaria de participar e contribuir.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests