Neo4j Twitter Import for Community Graph (and others)

Currently uses Python and iPython Notebook, the Twitter Search API via requests.

Run the script and notebook server with these environment variables:

nb.sh

cat ../nb.sh
export NEO4J_URL=bolt://localhost
export NEO4J_USER=neo4j
export NEO4J_PASSWORD=****
export TWITTER_BEARER='...'
# export TWITTER_SEARCH='#neo4j'

ipython notebook

Approach

• Use Twitter search API

• Control direction of ingest with catchUp: False → backward in history using max_id, catchUp: True → newer tweets using since_id

• Optionally provide twitter search via env-param

• Use idempotent Cypher statement to merge Tweets, Users, Tags

Data Model

Uses the Twitter part of this data model:

Queries

TODO

• store in json files and then import those

• save a "hash" of the query used with the tweet, so we can compute "maxId" for different queries

References

• https://dev.twitter.com/rest/public/search

• https://dev.twitter.com/rest/public/timelines (max_id and since_id explained)

  • Note watch out of maxId of retweeted / replied tweets, that can be much older

• https://dev.twitter.com/rest/public/rate-limiting

Neo4j & Twitter

• https://neo4j.com/blog/oscon-twitter-graph/

• https://github.com/neo4j-examples/twitter-graph-viz

• http://www.markhneedham.com/blog/2015/03/11/pythonneo4j-finding-interesting-computer-sciency-people-to-follow-on-twitter/

• http://opensourceconnections.com/blog/2013/11/27/quick-start-with-neo4j-using-your-twitter-data/

• http://www.lyonwj.com/twizzard-a-tweet-recommender-system-using-neo4j

• http://blog.bruggen.com/search/label/exporttweet