reddit-analytics.txt

stuff left to implement:


sentiment ana


scatter char for entities

---
1 hour ->

length limit ->  50000



reddit ->
subreddit =  5  [all,hot,politics,worldnews,videos]
type =  2 [rising,hot]
[#] of posts = 20 -> submission title + submission comments -> join -> send

1 submission row ->

[ 1 day graph ->  
for hot rising ->  24 times   [] ->  20 different post per hour though -> gotta combine them
]
  1st analysis gotten data example:
    categoies -> label & confidence
    sentiment -> label & confidence pos/new
        entities ->


database schema in one hour -- with 20 post:

worldnews-hot table [one-post] 
    reddit_submission_id: String  --> create post link via    ---> reddit.com/{{id}}
    reddit_submission_title: String -- not necessary ? 
    reddit_submission_url: String  -- not necessary ?
    sentiment: { label: string
                 & confidence: integer}
    catagories: { label: string
                 & confidence: integer}
    entities: [{

    }]

post * hour * month = 20 * 24 * 30 = 3360
       hour * month =  24 * 30  =  720

catagories:
----------
1. article
[2017-04-22 16:15:59,373] INFO: root: {'categories': [{'label': 'LAW_GOVERNMENT_AND_POLITICS', 'confidence': 0.12227408626371404}], 'responseHeaders': {'Strict-Transport-Security': 'max-age=63072000; includeSubdomains; preload', 'X-RosetteAPI-ProcessedLanguage': 'eng', 'Server': 'openresty', 'Connection': 'keep-alive', 'X-RosetteAPI-Request-Id': '81c7278674bb4acab491c9f93a053265', 'X-RosetteAPI-App-Id': '1409614621895', 'Content-Length': '89', 'X-RosetteAPI-Concurrency': '2', 'Date': 'Sat, 22 Apr 2017 20:15:59 GMT', 'Content-Type': 'application/json'}}
'categories':[

    {
        'label':'LAW_GOVERNMENT_AND_POLITICS',
        'confidence':0.12227408626371404
    }

]

2. reddit
[2017-04-22 17:45:22,799] INFO: root: {'responseHeaders': {'X-RosetteAPI-App-Id': '1409614621895', 'Server': 'openresty', 'Strict-Transport-Security': 'max-age=63072000; includeSubdomains; preload', 'Connection': 'keep-alive', 'X-RosetteAPI-Request-Id': 'ba86796c12445afafccf24620f0944e1', 'Content-Length': '85', 'Content-Type': 'application/json', 'X-RosetteAPI-ProcessedLanguage': 'eng', 'X-RosetteAPI-Concurrency': '2', 'Date': 'Sat, 22 Apr 2017 21:45:22 GMT'}, 'categories': [{'confidence': 0.5316236499823158, 'label': 'TECHNOLOGY_AND_COMPUTING'}]}


root.catagories -> array of json containing (usually 1) { label:TECHNOLOGY_AND_COMPUTING   & confidence:  }
    'confidence':0.5316236499823158,
    'label':'TECHNOLOGY_AND_COMPUTING'


sentiment:
----------
1. article
{

    'entities':[16],
    'responseHeaders':{},
    'document':{
        'label':'neu',
        'confidence':0.9719513972797291
    }

}
SyntaxError: JSON.parse: expected property name or '}' at line 1 column 2 of the JSON data

'entities':[16],

2. reddit
root.document -> confidence of label, label-> +/- neutral
{

    'responseHeaders':{},
    'document':{
        'confidence':0.9999999675996256,
        'label':'pos'
    },
    'entities':[171]

}

entities:
----------

SyntaxError: JSON.parse: expected property name or '}' at line 1 column 2 of the JSON data

'entities':[],

1. article
  'entities':[
    {
        'mention':'Reuters',
        'count':4,
        'type':'ORGANIZATION',
        'sentiment':{
            'label':'neu',
            'confidence':0.6934126616866995
        },
        'normalized':'Reuters',
        'entityId':'Q130879'
    }

2. reddit

---> more parsing





-----------------
reddit trends public api via rosetta? gather data evrer certain time and store it on db... use db to take data?

use d3 js visualizer



see hot post every hour?

tools for mod to track analytics

--

docker installer app node js
--
next step for tomorrow -> use same data  to create array of apis to go and log it out on file

look at log file, come up with a schema 
start  parsing functions to match schema/stopre

--

/sentiment [use]

-> document's label -> confidence 
-> entitiy's label -> condidence


/catagories [use]
-> categorization mapping -> 21 catagories with confidence mapping

Arts & Entertainment 	Travel
Business 	Automotive
Education 	Careers
Food & Drink 	Family & Parenting
Hobbies & Interests 	Health & Fitness
Law, Gov’t & Politics 	Home & Garden
Personal Finance 	Pets
Real Estate 	Religion & Spirituality
Science 	Sports
Society 	Technology & Computing
Style & Fashion 	


/relationships[nope]
relation -> org of, member of etc.


/entities [use]
-> entities -> detailed category
    LOCATION: A city, state, country, region, building, monument, body of water, park, or address.

    ORGANIZATION: A corporation, institution, agency, or other group defined by an organizational structure.

    PERSON: A human identified by name, nickname, or alias.

The following entities are available in select languages. Contact us for more information.

    PRODUCT: A brand name item produced by an organization.

    TITLE: Appellation associated with an occupation, office, or status.

    NATIONALITY: Reference to a country or region of origin.

    RELIGION: Reference to an organized religion or theology, as well as its followers.

    IDENTIFIER:CREDIT_CARD_NUM: Credit card numbers.

    IDENTIFIER:EMAIL: Email addresses.

    IDENTIFIER:MONEY: Currencies.

    IDENTIFIER:PERSONAL_ID_NUM: Personal identification numbers.

    IDENTIFIER:PHONE_NUMBER: Phone numbers.

    IDENTIFIER:URL: Web addresses.

    TEMPORAL:DATE: Date.

    TEMPORAL:TIME: Time.

    IDENTIFIER:DISTANCE: Distance.

    IDENTIFIER:LATITUDE_LONGITUDE: Geographic locations in latitude and longitude coordinates.


/syntactic-dependencies
-> future save those
/text-embedding

reddit trends public api via rosetta? gather data evrer certain time and store it on db... use db to take data?

use d3 js visualizer


--
try using memchache free open source on gitlab
--
see hot post every hour?

tools for mod to track analytics

--

docker installer app node js

-
i
node module to generate node js code from swagger docs for api?


===

riot api some data viz for each matches

===
data - viz ->
combine one sentiment with another sentiment to produce new sentiments.

next step for tomorrow -> use same data  to create array of apis to go and log it out on file

look at log file, come up with a schema 
start  parsing functions to match schema/stopre
--
one more end point to get data by date in the future
--

SELECT * FROM "all" WHERE DATE(process_datetime) >= DATE('now', 'weekday 0', '-7 days');


CREATE VIEW IF NOT EXISTS thisWeek AS 
SELECT "1" as id, process_datetime,reddit_id ,sentiment ,entities ,catagories FROM "all" WHERE DATE(process_datetime) >= DATE("now", "weekday 0", "-7 days")
UNION ALL
SELECT "2" as id, process_datetime,reddit_id ,sentiment ,entities ,catagories FROM "worldnews" WHERE DATE(process_datetime) >= DATE("now", "weekday 0", "-7 days")
UNION ALL
SELECT "3" as id,process_datetime,reddit_id ,sentiment ,entities ,catagories FROM "askreddit" WHERE DATE(process_datetime) >= DATE("now", "weekday 0", "-7 days")
UNION ALL
SELECT "4" as id,process_datetime,reddit_id ,sentiment ,entities ,catagories FROM "videos" WHERE DATE(process_datetime) >= DATE("now", "weekday 0", "-7 days")
UNION ALL
SELECT "5" as id,process_datetime,reddit_id ,sentiment ,entities ,catagories FROM "politics" WHERE DATE(process_datetime) >= DATE("now", "weekday 0", "-7 days")


select * from thisWeek


--
sunburst

-> center -> subreddits -> sentiment//entities//categories -> group by remaining

// entities
// catagories
// sentiment



->

catagories label -> sentiment (smoothen out color from red to white to green using d3 interpolater) -> entities ->


-> 
scope.data structure 

[
    { // REDDIT 
        "name":"Subreddits",
        "height":200,
        "color":"grey",
            "children":[ 4 objects => {   /R/ SUBREDDIT
                                    "name":"/r/worldnews",
                                    "color":"#FFwede",
                                    "children":[ 11 objects =>{ // CATEGORIES // 11 POST OBJECTS
                                                    "name":"3 : BUSINESS : 01:08:31 GMT-0400 (EDT) - reddit.com/685grs",
                                                    "color":"#4c005c",
                                                    "children":[ 1 object => { // SENTIMENT
                                                                    "name":"reddit.com/685grs00",
                                                                    "color":"red",
                                                                    "children":[ 6 objects => { // ENTITIES
                                                                            "name":"Kim Jong-un0000",
                                                                            "color":"red",
                                                                            "children":[ 0 children ]
                                                                    }]
                                                            }]
                                                     }]
                                                }
                       ]
    }
]


===


store home page on first api call -- use localstorage


==

red -> white -> green
99 -> 0 -> 99 -> 0 -> 99-> 0

99 -> 0
red // negative
red to approach black (grey ??? white black)----->

99 -> 0
white // neutral ->
white to black

99 -> 0
green // positive ->
green to black

Positive: Green.
Negative: Red. (though negative and positive are reversed in China)
Warning: Yellow.
Neutral: Gray.

label id -> 1 . .2. .20
label -> 1 2 3
confidence -> 1 to 99

sort ->
label id * 1005 + label * 105 + confidence

1111
12 1