# This is a combination of 6 commits.

# This is the 1st commit message:
Init

# This is the commit message paypal#2:

Init

# This is the commit message paypal#3:

[paypal#34] Updated documentation and added CONTRIBUTING.md (paypal#45)

# This is the commit message paypal#4:

Update scheduler.py

# This is the commit message paypal#5:

Initialize Github and Scheduler Extensions

# This is the commit message paypal#6:

g

build/extension_init.sh

@@ -0,0 +1,14 @@
+#!/bin/sh
+cd ppextensions/extensions/github
+jupyter nbextension install static
+cd ../scheduler
+jupyter nbextension install static
+
+jupyter nbextension enable static/github --section='tree'
+jupyter nbextension enable static/githubmain --section='tree'
+jupyter nbextension enable static/githubcommit --section='notebook'
+jupyter nbextension enable static/schedulermain --section='tree'
+jupyter nbextension enable static/scheduler --section='tree'
+
+jupyter serverextension enable --user ppextensions.extensions.github.github
+jupyter serverextension enable --user ppextensions.extensions.scheduler.scheduler

docs/CONTRIBUTING.md

@@ -0,0 +1,45 @@
+                                   
+# Contributing to PPExtensions
+
+## Discussions
+
+Our recommendation is to start a slack discussion as soon as you have an idea for contributing to PPExtensions.
+This will help you, other interested contributors & the committers get to common grounds early in time.
+Contact the PPExtenions community on [slack](https://join.slack.com/t/ppextensions/shared_invite/enQtNDIyODk5NzYzMzEyLTIwOGM3MWE0OGZlNjFkYTUxZTJiN2NjOWFlNmUxNDRiY2U3MzE0Nzg5NDRjZjE2M2VmZGI4NWJhOGVjYTRiMTk).
+
+--------------------------------------------------------------------------------------------------------------------
+
+## How can you help
+
+### Code
+Look for Issues that are open on below categories, pick an issue, start a slack discussion on slack channel, once we get to common grounds on the solution approach, open a PR with your implementation.
+* Bug Fixes
+* Enhancements & Improvements (Jira Details)
+* Add a new extension that you see are useful to Jupyter community
+
+### Documentation
+You can also improve our documentation
+* readme.md
+* docs/*.md
+* Any other valuable comment you'd like to add to the code that will simplify other developers' lives.
+
+--------------------------------------------------------------------------------------------------------------------
+
+
+## Contribution Process
+
+* Get your github account.
+* Fork the PPExtensions repo into your account.
+* Create an issue branch using the master branch.
+* Make modifications to the code.
+* Ensure code coverage by added test cases.
+* All commits must have the issue ID & summary. Say "[#32] Add Codacy Integration and Badge".
+* Ensure all your commits are squashed.
+* Make a Pull Request to develop branch.
+* If there are code review related changes - ensure those commits are also squashed.
+* DO NOT include changes that are not directly related to the issue you are trying to resolve.
+* Once PR is approved, code will be merged to Development branch.
+* Once all regression test cases have passed - changes will be merged to master branch.
+
+--------------------------------------------------------------------------------------------------------------------
+

docs/getting-started/build-ppextensions.md

@@ -1,15 +1,16 @@
 ## Install 
-```buildoutcfg
+```
 pip install ppextensions
 ```
+
 ## Try ppextensions
 ```
 %load_ext ppextensions.ppmagics
 ```
 
 Try help to see all the available options
-```buildoutcfg
-%%help?
+```
+%help
 ```
 
 | Magic               | Usage          | Explanation                                                                                                                    |
@@ -24,4 +25,4 @@ Try help to see all the available options
 
 
 For more info:
-[Github Link](git@github.com:paypal/PPExtensions.git)
+[Github Link](https://github.com/paypal/PPExtensions/)

docs/index.md

@@ -5,10 +5,10 @@ PPExtensions is a suite of ipython and jupyter extensions built to improve user
 
 # Features
 
-- **PPMagics** - Set of magics to simplify access to different storage systems and tableau.
-- **Github Integration** - A jupyter extension to integrate notebooks with github. This extension simplifies version controlling, sharing and resolving merge conflicts of notebooks.
-- **Notebooks Scheduling** - A jupyter extension to productionalize the notebooks development environment. This extension enables scheduling notebooks with help of [airflow](https://airflow.apache.org/).
-- **Config UI** - A simple UI built to change the configurations of different extensions like PPMagic, [sparkmagic](https://github.com/jupyter-incubator/sparkmagic) ..etc.
+- **[PPMagics](ppmagics/csv.md)** - Set of magics to simplify access to different storage systems and tableau.
+- **[Github Integration](ppextensions-github-integration/github-integration.md)** - A jupyter extension to integrate notebooks with github. This extension simplifies version controlling, sharing and resolving merge conflicts of notebooks.
+- **[Notebooks Scheduling](ppextensions-scheduler/scheduler.md)** - A jupyter extension to productionalize the notebooks development environment. This extension enables scheduling notebooks with help of [airflow](https://airflow.apache.org/).
+- **[Config UI](ppextensions-config-ui/config-ui.md)** - A simple UI built to change the configurations of different extensions like PPMagic, [sparkmagic](https://github.com/jupyter-incubator/sparkmagic) ..etc.
 
 
 # Installation
@@ -33,6 +33,6 @@ PPExtensions is a suite of ipython and jupyter extensions built to improve user
 
 # Questions
 
-* [Slack](https://ppextensions.slack.com)
+* [Slack](https://join.slack.com/t/ppextensions/shared_invite/enQtNDIyODk5NzYzMzEyLTIwOGM3MWE0OGZlNjFkYTUxZTJiN2NjOWFlNmUxNDRiY2U3MzE0Nzg5NDRjZjE2M2VmZGI4NWJhOGVjYTRiMTk)
 * [User Forum](https://groups.google.com/d/forum/ppextensions)
 * [Developer Forum](https://groups.google.com/d/forum/ppextensions)

docs/ppextensions-config-ui/config-ui.md

@@ -0,0 +1,3 @@
+## Config UI
+
+Coming Soon

docs/ppextensions-github-integration/github-integration.md

@@ -1,3 +1,102 @@
-# GitHub Integration
+# Github Integration
 
-Coming soon
+# About
+A Jupyter extension to integrate notebooks with Github. This extension simplifies version controlling, sharing and resolving merge conflicts of notebooks.
+
+# Getting Started <a id='getstart'></a>
+
+Install Github Extension
+---
+~~~
+cd PPExtension/ppextension/extensions/github
+jupyter nbextension install static
+jupyter nbextension enable static/github --user --section=tree
+jupyter nbextension enable static/githubmain --user --section=tree
+jupyter nbextension enable static/githubcommit --user --section=notebook
+jupyter serverextension enable --user ppextensions.extensions.github.github
+~~~
+
+Alternatively, if you want to install all extensions in ppextension module
+~~~
+bash PPExtension/build/extension_init.sh
+~~~
+
+This command will automatically install all frontend and backend Jupyter extensions we provide.
+
+Setup
+---
+
+**Register private Github token:** Go to Github website, click `Settings` --> `Developer settings` --> `Personal access tokens` --> `Generate new token`, copy the new token and export that as an environment variable.
+~~~
+export githubtoken=<Your token here>
+~~~
+
+Notice if the token is replaced, all local repo will be "unlinked" to remote. 
+**Enable git merge driver:**
+To show conflict in notebook, a nbmerge driver from nbdime module should be enabled as well.
+~~~
+git-nbmergedriver config --enable --global
+~~~
+
+**(Optional) Initialize a Github repo for notebooks**
+
+If you want to create a separate repo for sharing the notebooks, go to github website and create a new repo, be sure to create a README as well in order to initialize the master branch, otherwise when you pull the repo, there will be a "master branch not found" error. 
+
+**(Optional) Use an existing Github repo for sharing the notebooks**
+Either push to or pull from that repo will create a local workspace in Private Sharing folder in the notebook startup folder.
+
+Push to Github
+---
+**Push a single notebook to Github:** Select the notebook to be pushed, click `Sharing` --> `Push to Github`, select the repo, branch and type commit messages in the popup, and click on `Push`.
+
+When you push a notebook outside the `Sharing` folder, the notebook will be moved under `Sharing/<Repo Name>/<Notebook Name>` path, and the be pushed to Github.
+When you push a notebook inside the `Sharing` folder, only the "Linked" repo in the dropdown will display in the dropdown.
+
+In the following situation, the push command will fail. 
+
+***During a merge:*** Cannot do partial commit during a merge, please choose commit all notebooks option and push. Notice: this operation will push all other notebooks in this repo!
+
+***There is a conflict:*** Updates were rejected because the remote contains work that you do not have locally. Please do git pull and fix the possible conflicts before pushing again!
+
+**Push a folder to Github:** Select the folder, click on `Sharing` --> `Push to Github`, select the repo, branch and type commit messages in the popup, and click on `Push`. 
+
+When you push a folder outside the `Sharing` folder, that entire folder will be moved under "Sharing/<Repo Name>" path, and then be pushed to Github.
+
+
+Pull from Github
+---
+Click on `Sharing` --> `Pull from Github`, copy the Github repo url and paste that in the input area, then click on `Pull`. 
+
+In the following situations, the pull command will fail.
+
+***During a merge:*** You have not conclued your merge(MERGE_HEAD exists). Please, commit your changes before you can merge.
+
+***There is a conflict:*** Auto-mergeing **.ipynb. CONFLICT(content): Merge conflict in **.ipynb. Automatic merge failed; fix conflicts and then commit the result.
+
+***Untracked notebook in local:*** Your local changes to the following files would be overwritten by merge: xx.ipynb. Please, commit your changes or stash them before you can merge. Aborting.
+
+Commit
+---
+Open up a notebook, click on the Github icon in the tool bar. There are two types of commit:
+
+**Commit one notebook:** This option will be used in most cases. 
+In the following situations, this command will fail.
+
+***Worktree clean, nothing to commit***
+
+***There are other untracked/uncommitted notebooks:*** Nothing committed but untracked files presented. 
+
+***During a merge:***Cannot do partial commit during a merge, please choose commit all notebooks option and push. Notice: this operation will push all other notebooks in this repo!
+
+**Commit all notebooks in the same folder:** This option will only be used when a merge conflict is fixed. 
+
+
+Conflict Fix
+---
+When you pull from Github and you local commit is different from remote commit, a conflict will be generated, if the conflict cannot be automatically resolved, you should fix the conflicts.
+
+In the error message, the conflicting files will be displayed. 
+
+Notice: The merge-driver is depending on nbdime module,  while it is working well in identifying "cell level" conflicts, it doe not fully support "notebook level" merging. Therefore, it is not guaranteed that a "notebook level" conflict (such as a deleted cell/added cell) will be identified in 100 percent correctness. Before the improved ndime module is released, we would recommend the user to keep the number of cells unchanged in a collaborative circumstance. 
+
+To commit, first click on the Github icon in the notebook toolbar, choose either `Commit this notebook only` or `Commit all notebooks in this folder`, then click on `Commit`. 

docs/ppextensions-scheduler/scheduler.md

@@ -1,3 +1,102 @@
 # Scheduler 
 
-Coming soon
+# About
+A Jupyter extension to productionalize your notebooks by scheduling them to run in the background
+
+# Getting Started <a id='getstart'></a>
+
+Install Scheduler Extension
+---
+~~~
+cd PPExtension/ppextension/extensions/scheduler
+jupyter nbextension install static
+jupyter nbextension enable static/scheduler --user --section=tree
+jupyter nbextension enable static/schedulermain --user --section=tree
+jupyter serverextension enable --user ppextensions.extensions.scheduler.scheduler
+~~~
+
+Alternatively, if you want to install all extensions in ppextension module
+~~~
+bash PPExtension/build/extension_init.sh
+~~~
+
+This command will automatically install all frontend and backend Jupyter extensions we provide.
+
+Pre Requisites
+---
+**Configure Airflow**
+~~~
+export AIRFLOW_HOME=<path to airflow_home>
+~~~
+
+Run airflow in command line, a `airflow.cfg` file will be generated in airflow home. Here are a list of paramenter needs to be changed.
+
+~~~
+dags_folder = <path to airflow home>/dags
+executor = LocalExecutor
+sql_alchemy_conn = mysql+mysqlconnector:://<user name>:<password>@<host>:<port>/airflow
+dags_are_paused_at_creation = False (recommended)
+load_examples = False (recommended)
+~~~
+
+Create a `dags` and a `variables` folder in airflow home to store the dag files and their related vairable files.
+
+**Setup MySQL**
+
+Create a database `airflow` in mysql. This servers as the metadata db for airflow.
+
+Setup
+---
+Here are a few preparations to make scheduler extension work. The Pre-req steps can be skipped with those are already configured.
+
+**Export Path Variables**
+~~~
+export AIRFLOW_METADATA_CONNECTION_STRING='mysql+mysqlconnector://<user name>:<password>@<host>:<port>/airflow'
+~~~
+
+**Start Airflow Scheduler, Webserver**
+
+In this tutorial, we are using airflow LocalExecutor, hence airflow worker is not required. But if you are using some other executors like CeleryExecutor, then the airflow worker should also be started. 
+
+~~~
+airflow webserver
+airflow scheduler 
+~~~
+
+By default, the log files will be generated in airflow_home, you can configure that as well. Refer to https://airflow.apache.org/howto/write-logs.html.
+
+After everything is settled, source the profile and start Jupyter notebook. 
+
+Schedule Notebook
+---
+To schedule a notebook, first select a notebook, click on the `schedule` button apeared in the dynamic tool bar, a scheduler menu will pop up. 
+
+Currently scheduler extension provides the following configurable dag parameters:
+
+***Interval:*** Three different scales of frequency are provided: hourly, daily and weekly. 
+
+***Start Time/Date:*** The start time/date can not be ealier than current time.  
+
+***Number of Runs:*** The number of runs the job should be executed. For example, if a job is scheduled to at `12:00AM 11/11/2018` with an interval of `1 hour`, and the number of runs is set to 5 times, then the job will be ended at `5:00 AM 11/11/2018`. 
+
+***Emails:*** To receive failure email and success email, check the box and input the email address in the input area. 
+
+To receive the email alert, the STMP server should be setup in the host machine and corresponding parameters in `airflow.cfg`  `[smtp]` section need to be configured.
+
+Click on `Schedule` button, the job will be displayed in `Scheduled Jobs` tab, from which you can see the **Last Run Time**, **Last Run Time**, **Last Run Duration**, **Next Scheduled Run** of each job scheduled. Notice, there will be some delay in the airflow UI to show the job.
+
+Edit Job
+---
+To edit a job, go to the `Scheduled Jobs` tab, click on `Edit` button in `Action` column of the target job, the current configuration of that job except number of runs will be displayed in the configuration menu as default values.  Change the configuration and hit on `Confirm Edit` button, the changes will be applied to the job.
+
+
+Delete Job
+---
+To delete a job, go to the `Scheduled Jobs` tab, click on `Remove` button in `Action` column of the target job, the dag/vairable file of the related job as well as the records in the metadata db will be removed. 
+
+
+
+
+
+
+

ppextensions/extensions/extension_logger/extension_logger.py

@@ -0,0 +1,21 @@
+import logging
+from pathlib import Path
+
+logger_name = "extension_logger"
+logger = logging.getLogger(logger_name)
+logger.setLevel(logging.DEBUG)
+
+log_path = str(Path.home()) + "/.extensionlog.log"
+fh = logging.FileHandler(log_path)
+fh.setLevel(logging.DEBUG)
+
+sh = logging.StreamHandler()
+sh.setLevel(logging.DEBUG)
+
+fmt = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
+formatter = logging.Formatter(fmt)
+
+fh.setFormatter(formatter)
+sh.setFormatter(formatter)
+logger.addHandler(fh)
+logger.addHandler(sh)