Introduction

Over the past couple of months, there has been an ongoing discussion about Jupyter Notebooks affectionately called the Notebook Wars. The genesis of the discussion is Joel Grus presentation I Dont Like Notebooks and has been followed up with Tim Hoppers response, aptly titled I Like Notebooks. There have been several follow-on posts on this topic including thoughtful analysis from Yihui Xie.

The purpose of this post is to use some of the points brought up in these discussions as a background for describing my personal best practices for the analysis I frequently perform with notebooks. In addition, this approach can be tailored for your unique situation. I think many new python users do not take the time to think through some of these items I discuss. My hope is that this article will spark some discussion and provide a framework that others can build off for making repeatable and easy to understand data analysis pipelines that fit theirneeds.

Specific UseCases

My use case is much narrower than what Joel describes. As much as possible, I try to use a Jupyter Notebook as my go-to solution when I need to do moderately complex data analysis in a business setting. Instead of creating an Excel spreadsheet, I build a consistent set of notebook files to document my analysis journey. The key distinctions between my approach and the data science scenarios discussed in the presentations aboveare:

• This analysis is only used by me. I do not share actual python code with anyone. All results are shared by other means (email, presentations, Excel,etc).
• I do not build models that are put intoproduction.
• All analysis is internal, proprietary and not sharedpublicly.
• If a solution needs to be used by others, I will build a standalone python script for them touse.
• The vast majority of work I describe is data wrangling, EDA and simple statistical analysis. The work is the bread and butter work that Excel is used for in mostorganizations.

The rest of this article will outline the approach I use in the hopes it can be a framework for others and might help people develop their own repeatable and maintainable workflow.

Why HaveStandards?

I imagine that most people that have used Jupyter Notebooks for any significant time have ended up with a directory structure that looks likethis:

[Messy notebook layout]

At a quick glance, there are a lot of problems with thisstructure:

• Inconsistent or absent naming scheme fornotebooks
• Mixture of notebooks, scripts, Excel, CSV, images, etc all in onedirectory
• Vague directorynames
• Difficult to follow flow of the processingsteps

On top of the non-intuitive structure, each notebook has its own unique structure for analyzing data. Some are documented but many are not. None of these issues is a flaw with notebooks per se but is an example of a sloppy approach to solving a problem. You could just as easily end up with this situation with Excel files or stand alone pythonscripts.

Ive certainly done all of the things described above. It is incredibly frustrating when I know I did some really useful analysis but I cant find it 6 months after the fact. If only I had a little more discipline up front, it would have saved a lot of time over the longrun.

One of my biggest complaints about Excel is that it is really difficult to understand how the data was pulled together and how all the cells, formulas and VBA relate to each other. There are very limited options for documenting Excel data analysis flow. I believe that using a well-formed Jupyter Notebook structure can lead to a much more reusable set of data analysisartifacts.

DirectoryStructures

The first step in the process is creating a consistent directory structure. I have leveraged very heavily from the Cookiecutter Data Science project. If you are doing more complex modeling and sharing code with others, then I encourage you to use the above mentioned cookiecutterframework.

In general, I create a new directory for each analysis and take the time to give the directory a descriptive name. Then, I setup the following directorystructure:

FY_18_Sales_Comp/
   1-Data_Prep.ipynb
   2-EDA.ipynb
   data
    interim
    processed
    raw
   reports

I will cover the details of the notebooks in a bit but the important item to note is that I include a number followed by the stage in the analysis process. This convention helps me quickly figure out where I need to go to learn more. If Im just interested in the final analysis, I look in the 2-EDA notebook. If I need to see where the data comes from, I can jump into 1-Data_Prep . I will often create multiple EDA files as I am working through the analysis and try to be as careful as possible about the naming structure so I can see how items arerelated.

The other key structural issue is that the input and output files are stored in differentdirectories:

• raw - Contains the unedited csv and Excel files used as the source foranalysis.
• interim - Used if there is a multi-step manipulation. This is a scratch location and not always needed but helpful to have in place so directories do not get cluttered or as a temp location form troubleshootingissues.
• processed - In many cases, I read in multiple files, clean them up and save them to a new location in a binary format. This streamlined format makes it easier to read in larger files later in the processingpipeline.

Finally, any Excel, csv or image output files are stored in the reports directory.

Here is a simple diagram of how the data typically flows in these types ofscenarios:

[Data flow process]

NotebookStructure

Once I create each notebook, I try to follow consistent processes for describing the notebooks. The key point to keep in mind is that this header is the first thing you will see when you are trying to figure out how the notebook was used. Trust me, future you will be eternally thankful if you take the time to put some of these comments in thenotebook!

Heres an image of the top of an examplenotebook:

[Notebook header]

There are a couple of points I always try toinclude:

• A good name for the notebook (as describedabove)
• A summary header that describes theproject
• Free form description of the business reason for this notebook. I like to include names, dates and snippets of emails to make sure I remember thecontext.
• A list of people/systems where the dataoriginated.
• I include a simple change log. I find it helpful to record when I started and any major changes along the way. I do not update it with every single change but having some date history is verybeneficial.

I tend to include similar imports in most of mynotebooks:

import pandas as pd
from pathlib import Path
from datetime import datetime

Then I define all my input and output file paths and directories. It is very useful to do this all in one place at the top of the file. The other key thing I try to do is make all of my file path references relative to the notebook directory. By using Path.cwd() I can move notebook directories around and it will stillwork.

I also like to include date and time stamps in the file names. The new f-strings plus pathlib make thissimple:

today = datetime.today()
sales_file = Path.cwd() / "data" / "raw" / "Sales-History.csv"
pipeline_file = Path.cwd() / "data" / "raw" / "pipeline_data.xlsx"
summary_file = Path.cwd() / "data" / "processed" / f"summary_{today:%b-%d-%Y}.pkl"

If you are not familiar with the Path object, my previous article might beuseful.

The other important item to keep in mind is that raw files should NEVER bemodified.

The next section of most of my notebooks includes a section to clean up column names. My most common stepsare:

• Remove leading and trailing spaces in columnnames
• Align on a naming convention (dunder, CamelCase, etc.) and stick withit
• When renaming columns, do not include dashes or spaces innames
• Use a rename dictionary to put all the renaming options in oneplace
• Align on a name for the same value. Account Num, Num, Account ID might all be the same. Name them thatway!
• Abbreviations may be ok but make sure it is consistent (for example - always use num vsnumber)

After cleaning up the columns, I make sure all the data is in the type I expect/need. This previous article on data types should behelpful:

• If you have a need for a date column, make sure it is stored asone.
• Numbers should be int or float and not object
• Categorical types can be used based on yourdiscretion
• If it is a Yes/No, True/False or 1/0 field make sure it is a boolean
• Some data like US zip codes or customer numbers might come in with a leading 0. If you need to preserve the leading 0, then use an object type.

Once the column names are cleaned up and the data types are correct, I will do the manipulation of the data to get it in the format I need for furtheranalysis.

Here are a few other guidelines to keep inmind:

• If you find a particular tricky piece of code that you want to include, be sure to keep a link to where you found it in thenotebook.

• When saving files to Excel, I like to create an ExcelWriter object so I can easily save multiple sheets to the output file. Here is what it lookslike:

  writer = pd.ExcelWriter(report_file, engine='xlsxwriter')
  df.to_excel(writer, sheet_name='Report')
  writer.save()
  

Operationalizing & Customing ThisApproach

There are a lot of items highlighted here to keep in mind. I am hopeful that readers have thought of their own ideas as well. Fortunately, you can build a simple framework that is easy to replicate for your own analysis by using the cookiecutter project to build your own template. I have placed an example based on this project on github.

Once you install cookiecutter, you can replicate this structure for your ownprojects:

$ cookiecutter https://github.com/chris1610/pbp_cookiecutter
$ project_name [project_name]: Deep Dive On December Results
$ directory_name [deep_dive_on_december_results]:
$ description [More background on the project]: R&D is trying to understand what happened in December

After answering these questions, you will end up with the directory structure and a sample notebook that looks likethis:

[Final Notebook]

The nice result of this approach is that you only need to answer a couple of simple questions to get the template started and populate the notebook with some of the basic project description. My hope is that this lightweight approach will be easy to incorporate into your analysis. I feel this providew a framework for repeatable analysis but is not so burdensome that you do not want to use it because of the additional work in implementingit.

In addition, if you find this approach useful, you could tailor it even more for your own needs by adding conditional logic to the process or capturing additional information to include in the notebooks. One idea I have played around with is including a snippets.py file in the cookiecutter template where I save some of my random/useful code that I usefrequently.

Ill be curious what others think about this approach and any ideas you may have incorporated in your own workflow. Feel free to chime in below with your input in the commentsbelow.