Not all of us have the privilege of having access to the internet at all times of the day. Imagine someone who travels to another city to attend a coding bootcamp, where they learn code with the help of freecodecamp.org, but when they go home, they can’t continue to learn on their own because they don’t have access to the internet at their home. Imagine also another person who struggles to learn to code on their own during the 4 hours every day when the internet happens to be working. This is the reality for many who aspire to learn how to code.

In this guide, I will show you how to run freecodecamp.org offline, so you can complete coding certifications completely without internet access.

Before you get started

You will need:

• A computer
• Temporary internet access for downloading FreeCodeCamp and its dependencies

FreeCodeCamp takes approximately 45 minutes to set up and install.

I will be providing instructions for both Windows 10. At the beginning of each step, I link to the installation instructions on other systems. If you have any questions, let me know in the comments below this article.

Step 1: Installing MongoDB on Windows

Below are the instructions for installing MongoDB on Windows 10. For other systems, click here for all the tutorials on installing MongoDB on each platform. Otherwise, continue below.

1. Go to https://www.mongodb.com/try/download/community and select the platform “Windows” and the package-type “msi”. The version I installed was 4.0.24. Your screen should look something like this:

2. After clicking “Download”, locate the newly downloaded .msi file and double click it.

3. When you get to this page, ensure all the features are selected, and that “Server” is configured to install on the local hard drive.

4. After clicking Next >, you’ll be brought to this page. Make sure “Install MongoDB as a service” is unselected. You don’t need MongoDB running as a service all the time since it uses memory. Instead, we will only start it before running FreeCodeCamp.

5. Now click through until the end of the installation instructions, and wait for MongoDB to install.

Step 2: Installing NPM and Node.js on Windows

Below are instructions for installing NPM and Node.js on Windows. For other systems, click here for instructions on installing with a package manager.

To install NPM on windows, go to nodejs.org and click the download button to get the .msi. Double-click the downloaded .msi and step through the prompts on the Node.js setup wizard.

Once you get to this page, make sure “install the necessary tools” is checked:

This will make sure any dependancies you need are met, and also will provide you with some packages like Chocolately, C, C++, Python, enabling you to make your own projects based on things you learn in FreeCodeCamp.

choco install package_name

Step 3: Cloning FreeCodeCamp

In this step, for Windows use “Windows Powershell”, and for Linux use the terminal.

1. Navigate to the directory you want to install freeCodeCamp using ls and cd commands.

2. If you don’t have Git installed, install it using your computer’s package manager.

3. For windows users, since we installed “Chocolately” in the previous step, we can use that to install Git, like so:

1

choco install git

4. Now use Git to clone FreeCodeCamp. depth=1 is used to only download necessary commits which will speed up the download process.

1

git clone --depth=1 https://github.com/freeCodeCamp/freeCodeCamp

Step 4: Installing FreeCodeCamp

First, navigate into the newly created directory with cd freeCodeCamp, and then run this command to start the installation process:

1

npm ci

This part takes a while. While you wait for the dependencies to install, continue to Step 5.

Step 5: Starting the database

Open up a new powershell or terminal, and start the MongoDB database service like so:

For Linux

1

mongod --dbpath=/path/to/freeCodeCamp

For Windows

1

"C:\Program Files\MongoDB\Server\3.6\bin\mongod.exe" --dbpath=/path/to/freeCodeCamp

This service will need to be started each time before you can run FreeCodeCamp in the future, so take note of this command.

Step 6: Seed the database

After Step 4 completes, run this command from the powershell or terminal used in Step 4 to set up the data in the MongoDB database:

1

npm run seed

You only have to run this command once. The next time you start FreeCodeCamp, the database should already be initialized.

Step 7: Start FreeCodeCamp

In the same powershell or terminal from Step 6, run this command to start FreeCodeCamp:

1

npm run develop

The first time you run this command, it will take a while to complete due to building and installing FreeCodeCamp. Once you see this line, this means the command finished and FreeCodeCamp is ready for you to use:

1

success Building development bundle - 303.218s

As you see from the output above, it took 5 minutes to run this command for me.

Step 8: Open FreeCodeCamp in the browser

1. Open up your web browser and navigate to http://localhost:8000.

2. Click “Get started now, it’s free!”

3. Now you will see “Welcome back, Development User.”

4. You are now signed in as a development user, and can now access all the courses without the internet. Any progress you make will be saved in the MongoDB instance you started in Step 5.

Next time you want to start FreeCodeCamp, you can do so without internet by following both Step 5 and Step 8 in two different powershells or terminals.

Have fun learning!

https://follow.it/s/6VTEhttps://follow.it/s/6VTEFor more than a year this blog used a Google Form for managing subscriptions, and relied on me to manually copy all the emails and write a newsletter to keep you all updated.

There has been an upgrade…

As you might notice, the “Subscribe” button at the top of this screen looks a bit different. Not only has its color changed, but its behavior changed as well. If you click on that button you’ll be brought to Follow.it‘s website where you can enroll in automatic email subscriptions to my blog. In addition, the bottom of every post now also has an area for you to subscribe to the blog.

This service was easy to set up, with the only steps required for me after signing up was to modify my “Subscribe” button to link to their website like so:

1
2
3
4
5

<a style="color: green;"
   href="https://follow.it/coding-w-ricky1?action=followPub"
   title="Subscribe for email updates">
     <i class="fa fa-envelope"></i>Subscribe
</a>

And I also needed to include a file they provide called verification-follow_it-3QprZbtAaJCwAQG7GdKd.html at the base-level of my website that verifies I am the owner.

After this, I now have a subscription system set up for automatic email updates, with no limits to the number of subscribers, completely for free!

Let me know what you think about this new feature in the comments below.

The world of web development can be insanely complex and overwhelming, however building a simple website doesn’t have to be difficult. Many people’s needs can be fulfilled simply by taking 30 minutes to set up a website on Github pages completely for free.

Power of Github Pages

Although the original purpose of Github Pages is to showcase a programmer’s work on Github, it can also be used for many other purposes.

• Hosting a blog
• a portfolio
• a business landing page
• an open letter

All these, and more, are possible through Github Pages, which has a soft-cap of 100 GB of bandwidth per month. On my blog, the average user uses about 612.43 KB of bandwidth per month, which means if my blog were hosted on Github Pages this soft-cap won’t be met until I have 163,284 monthly users. (This blog is hosted on Netlify, which will be another tutorial).

With Github Pages you have the option of

1. simply write your content using Markdown files and use one of Github’s default themes, with the option of customization
2. use a static site generator along with Github Actions, which has more customization and many templates to get you started
3. or write HTML, CSS, and Javascript directly in Github

With this blog, I used a static site generator called “Hexo” which can also be used with Github Pages. However for this tutorial I will be showing you how to simply using one of Github’s provided themes:

• jekyll-theme-primer
• minima
• architect
• cayman
• jekyll-theme-dinky
• jekyll-theme-hacker
• jekyll-theme-leap-day
• jekyll-theme-merlot
• jekyll-theme-midnight
• jekyll-theme-minimal
• jekyll-theme-modernist
• jekyll-theme-slate
• jekyll-theme-tactile
• jekyll-theme-time-machine

Clicking through the links above, you might notice that many of the themes are designed to showcase a coding project, however with a bit of configuration, we can easily change them to fit many needs. Also, the first two themes, primer and minima, are a bit different from the other themes. Primer is the most simple theme and is basically just a single page, whereas Minima is a starter theme for building a blog. These two themes being a blank slate are a good place to start with if you want to build a website from the ground up.

Choosing a theme

The theme I will use for this tutorial is minimal because its two-column format is perfect for a personal portfolio website.

The [Logo] is a good spot for a picture of myself, and below that is space for links to my Github, my Linkedin, and my contact info. The column to the right is where I will list my skills, experience, projects, and resume.

Also, like all themes, this is totally customizable so I can add more things to the left column such as a graph showing which skills I’m best at.

Creating the website

Now let’s get started. First navigate to github.com and either log in or create an account if you don’t have one. Once you are logged in, click the “New” button to create a new repository:

Repository Name After this, you are brought to the “Create a new repository” page. If you wish your website to be hosted on the URL .github.io, then make this URL your repository (repo) name. Otherwise, the website will be hosted on .github.io/. (If you plan on purchasing your own domain, don’t worry too much about the name of your repo)

Description Some themes for Github Pages use the description, so fill in this field with a subtitle for your website. You can change this later.

Public / Private Select “Public” so that way you can host your website for free on Github Pages. This also means anyone can see the source code for your website, but this shouldn’t be too different from the content people can see on your website anyways.

The rest of the config on this page doesn’t matter too much, other than possibly choosing a license.

After clicking “Create repository” navigate to configure Github Pages with the instructions below:

1. Click on “Settings”
2. Click on “Pages”
3. Select “main” branch
4. Select / directory
5. Click “Save”
6. Choose theme (you can change this later)
7. Optionally choose a custom domain to serve your website (you will need to buy and configure this domain from a website like domains.google. I won’t be going through how to configure a custom domain in this tutorial.)

After following these steps, you should see the message “Your site is published at “. Click on this URL and admire the beauty of your live website.

Modifying the website

Your repository should look like this now, with just a README.md file and a README.md file and a _config.yml file:

Modifying website content

Since an index.md file doesn’t exist, Github Pages uses README.md as the default entry-point for the website. This is where you can modify the contents of your website.

Click on the file README.md and then click the icon to edit the file.

This file is in Markdown format. Here is a quick guide to markdown.

You can click “

Preview” to see how your markdown will render before you submit. Once you are ready to see your changes live on your website, scroll to the bottom of the page and click “Commit changes” with the bubble checked “

Commit directly to the main branch.”

One cool thing about Markdown - you can write HTML directly within the markdown file. This is how I’m able to render the Github icons in this tutorial
. I copied the HTML for the icons from inspecting github.com with my browser, and pasted it directly into the Markdown file that is rendering this tutorial. This feature of Markdown makes it super flexible at building very custom webpages.

Choosing a different theme

Click on _config.yml. In this file you will see something like:

1

theme: jekyll-theme-...

For this tutorial I am using the “minimal” theme, so this is what my _config.yml looks like:

1

theme: jekyll-theme-minimal

You can change the theme here by modifying the line after theme: to reflect the name of the new theme, and then clicking “Commit changes” at the bottom of the page.

Customizing the theme

You can add other configuration to _config.yml depending on what your theme supports. The “minimal” theme behaves different for a repo with the name <username>.github.io than it does for a regular repo, where in this case it doesn’t show links to download the TAR or ZIP file and instead just links to the Github profile.

The README.md for the “Minimal” theme which you can read at the bottom of this page shows all instructions for how to customize this theme.

This is what my _config.yml looks like:

1
2
3
4
5

theme: jekyll-theme-minimal
title: Richard Barella
description: Software Engineer at Intel
logo: https://media-exp1.licdn.com/dms/image/C5603AQEjpWRtfGTu9Q/profile-displayphoto-shrink_800_800/0/1593112650172?e=1626307200&v=beta&t=_8iQKxnbGrZ3UbyAC1tOFfIV193I99dadgJh3pVN9DA
show_downloads: false

For the logo: part of _config.yml I added a link to my profile picture from Linkedin. To get the URL for this picture I right-clicked on the picture in my Linkedin and clicked “open in new tab”, and then the picture’s URL is shown in the new tab.

In addition, this theme supports google_analytics: ..., which you can use to help Google index your site faster for others to find your website from a Google search. To set this up, you need to set up a google analytics tracking id on analytics.google.com and insert this into your _config.yml.

After committing my changes to _config.yml, my website now looks like this:

As you can see it’s a good start, however there is some small things I would like to change. For example, I want my picture to be a circle, and I would like more links under “View My Github Profile” that link to my Linkedin and also my blog. These things can be changed by using the template provided by the theme and also creating my own style.scss file.

Customizing the theme’s style

For the theme’s style, my goal is simply to make my picture rounded rather than square. To do this, add a file to the repo called /assets/css/style.scss. The theme will automatically use this file to style your website.

When creating a new file, you can create it within the /assets/css/ folders by adding assets/css/ to the beginning of your filename.

Within this file, add this to the top:

1
2
3

---
---
@import "{{ site.theme }}";

After inspecting the HTML of my website at ribab.github.io, I noticed that the <img> for my profile image is located within the <header> element in the HTML. This means I can apply SCSS to only images within the <header> element through something like this: header { img { ... } }.

This is what my final scss file looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

---
---
@import "{{ site.theme }}";
header {
  img {
    border-radius: 50%;
  }
  p {
    text-align: center;
  }
  p.view {
    text-align: left;
  }
}

And voilà! I now have a circular icon for my profile picture, and also a centered description with the links staying left-aligned.

Customizing the theme’s template

To add more links to the left column on my webpage, I will need to customize the template for this theme.

1. The first step is to navigate to the minimal theme’s _layouts/default.html file.
2. Then create a new file _layouts/default.html in your repo and paste the contents of the theme’s layout file into the new file.
3. Before the end of the <header> section (or before </header>), add links as you would like with the format: <p class="view"><a href="insert link URL here">Insert link text here</a></p>

This is what I added before </header>:

1
2
3
4
5
6
7
8

...

<p class="view"><a href="https://www.linkedin.com/in/richard-barella/">Visit my Linkedin Profile</a></p>

<p class="view"><a href="https://www.codingwithricky.com/">Checkout my blog</a></p>

</header>
...

Now we’re talking! My website (at ribab.github.io) now looks almost perfect. If I wanted to I could remove the Hosted on Github Pages – Theme by orderedlist text at the bottom by modifying the theme’s layout more, but this is good enough for me.

Conclusion

So this is how to use default theme’s in Github Pages to make your own websites completely for free. I plan to continue working on my own portfolio website and modify the README.md file to showcase all my work and experience. Now it’s time to get creative and make your very own website.

Happy coding!

I am not endorsed in any way by Plaid to share this info, just helping others get a head start on using Plaid with Django.

Plaid is a plaform-as-a-service API which you can use to build your own financial apps. It has features such as easily linking to existing bank accounts, and also auto-categorization of transactions. Plaid’s API brings all the data from different financial sources into one place and one format so you can build apps with a holistic view of a user’s finances.

Ok. Now all praise of Plaid aside, I’m sure you are here because you want to build a Django app that uses Plaid to obtain users’ financial data. In all of Plaid’s tutorials shown in their quickstart github page, they have a tutorial for Python, but it’s using Flask for the backend rather than Django.

I’ve taken the time to translate their tutorial from Flask to Django. The main difference was Django’s requirement of having CSRF protection.

Plaid Django Example Repo

Here is a link to my repo in case you want to clone or fork it and play around with Plaid: https://github.com/ribab/plaid_django_example

Use this template as a starting point to build amazing fintech apps using Plaid.

To try out the repo, follow these steps:

1. Clone the repo

   1	git clone https://github.com/ribab/plaid_django_example.git

2. Navigate to the repo

   1	cd plaid_django_example

3. Start venv for python3

   1 2	python3 -m venv env source env/bin/activate

4. Install required python packages into venv

   1	pip3 instal -r requirements.txt

5. Set up environment variables as shown on the sandbox page after logging into Plaid: https://dashboard.plaid.com/overview/sandbox

   1 2 3 4

   export PLAID_CLIENT_ID=??? export PLAID_SECRET=??? export PLAID_PUBLIC_KEY=??? export PLAID_ENV=???

6. Run the web server

   1	./manage.py runserver

7. Now you should be able to access the webpage at http://localhost:8000

8. Click on “Open Link”, and then click on “Bank Of America”. When it asks you for username and password, give it “user_good” and “pass_good”, which are the sandbox credentials.

9. After this, “Get Accounts”, “Get Item”, and “Get Transactions” should all work.

For the last few days I have been scratching my head. How do you plot all the data in your django website, without having to figure out D3 or any other complicated graphing libraries?

I've looked at several solutions, including Djangos Graphos, Chartit, NVD3… Everyone online seems to be recommending to use charting libraries off of this list: https://djangopackages.org/grids/g/charts/

It's frustrating to try many solutions to find out that they are all too complicated to understand, and the copy+paste approach doesn't work most of the time.

However, one good thing came out of this process. I discovered that Plotly is now open-source! Years ago I once played around with Plotly's graphing library, marveling at it's ease of use and admiring it's beauty, however to use it in the past we had to make our plots public for all to see and store them on Plotly's servers. This limited how I could use Plotly in my projects, but that limit isn't on Plotly anymore. Now we can use Plotly just like any other library in Python, Javascript, and other programming languages.

Today I will show you how absolutely easy it is to plot graphs in Django using Plotly's Python API. We are going to make a simple interactive plot that looks like this:

The Basics of plotting with Plotly

To get a simple graph up and rendering in plotly is easy, just 8 lines of code:

1
2
3
4
5
6
7
8
9

from plotly.offline import plot
import plotly.graph_objs as go

fig = go.Figure()
scatter = go.Scatter(x=[0,1,2,3], y=[0,1,2,3],
                     mode='lines', name='test',
                     opacity=0.8, marker_color='green')
fig.add_trace(scatter)
plt_div = plot(fig, output_type='div')

Now plt_div holds all the HTML needed to render the plot!

How easy was that?

If you want to quickly see your plot rendering in a web browser, paste the contents of plt_div into an HTML file like so, and then open your HTML file with your browser.

1
2
3
4
5
6
7

<html>
  <head>
  </head>
  <body>
    PLT_DIV_CONTENTS_GO_HERE
  </body>
</html>

Plotting with Plotly + Django

Here is an example Django view and template to display this plot:

views.py

1
2
3
4
5
6
7
8
9
10
11
12

from django.shortcuts import render
from plotly.offline import plot
from plotly.graph_objs import Scatter

def index(request):
    x_data = [0,1,2,3]
    y_data = [x**2 for x in x_data]
    plot_div = plot([Scatter(x=x_data, y=y_data,
                        mode='lines', name='test',
                        opacity=0.8, marker_color='green')],
               output_type='div')
    return render(request, "index.html", context={'plot_div': plot_div})

index.html

1
2
3
4
5
6
7
8
9
10
11
12
13

<!DOCTYPE HTML>
<html>
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>test</title>
</head>
<body>
  {% autoescape off %}
  {{ plot_div }}
  {% endautoescape %}
</body>
</html>

The above code renders this:

Go ahead and click around on the plot, it’s fully responsive.

I actually added that plot on this page by copying the javascript code that Plotly generated right into the markdown file of this blog post. I modified the code to point to a CDN for plotly.js instead of including the entire plotly.js in the source-code of my blog.

Making the pages load faster

With the above code, the entire plotly javascript library is inside of the generated HTML within plot_div. This makes it simple to get plotly working on the web, since there are no dependancies, however the plots will load faster if you instead manually include plotly.js.

This is the change to plot() that you will need to make:

1

plt_div = plot(fig, output_type='div', include_plotlyjs=False)

This is the change to the HTML you will need to make:

1
2
3
4

<head>
  ...
  <script src="https://cdn.plot.ly/plotly-latest.min.js"></script>
</head>

Another tip:

If you don’t want the link to the plotly website on your graph, you can turn it off with including these options:

1

plot_div = plot(... , show_link=False, link_text="")

Conclusion

Plotly is the only graphing library I know of that works the same in Python as it does on the web, meaning I don’t need to make any transformations of the data from a python format to generate the plots. This made it extremely easy for me to get started right away developing visuals for a project I'm working on.

There are no special django plugins to download. No special javascript libraries to learn, no special way to format the data between python and html… everything just works :)

Checkout https://plot.ly for more information on how to make different kinds of plots.

As always, any questions you have leave them in the comments below. I'd be glad to help, and maybe see what kind of projects you are working on.

Many people who use python are used to the scriptfile workflow, where you write a rigid script in python and run it, somewhat similar to how people use bash scripts.

1. You find an answer in a blog somewhere that describes how to do something in python
2. You draft up a python script for the task and get it to work
3. You hard-code all the parameters the script needs

Don't get me wrong – drafting up python scripts to prototype your idea is really useful. You get your idea working fast, even if your code becomes messy and unmaintainable.

However, wouldn't you like your python script to be as elegant and usable as all the command-line utilities you're used to?

1
2
3
4

ls -l
find -name *.py
grep -r "some text"
ssh --version

What if I told you it's SUPER easy to give your python script a full-featured Command Line Interface (CLI)

For kicks, lets take the python3 print() function, and give it a CLI. Kind of like the echo command in bash.

This is the definition of print(): https://docs.python.org/3/library/functions.html#print

print(*objects, sep=' ', end='\n', file=sys.stdout, flush=False)

So in python, you can print in all sorts of ways, more than just print('hello'). For example, you can add customized string to append to the end, change the seperator between words, and output to a file instead of stdout.

1
2
3

print('hello', end='\r') # brings cursor back to beginning of line
print('hello', 'hi', sep=',') # prints commas between each word
print('hello', file=open('out.txt', 'w')) # write output to file

The simple CLI

Take a look on how easy it is to turn the print statement above into a full-featured CLI.

print.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

#!/usr/bin/env python3
import click
import sys

@click.command()
@click.argument('args', nargs=-1)
@click.option('-s', '--sep', default=' ', help='String to seperate each argument (default: " ")]')
@click.option('-e', '--end', default='\n', help='String to append to end (default: "\\n")')
@click.option('-o', '--out', default=None, help='Output file (default: stdout)')
def print_prog(args, sep, end, out):
    '''Print the arguments you give this program to stdout or to a file. Similar to echo.
    '''
    print(*args, sep=sep, end=end, file=open(out, 'w') if out else sys.stdout)

if __name__ == '__main__':
    print_prog()

Essentially this script is taking all the command line arguments and giving them directly to the print() function. So we are in effect giving the bash-terminal all the capabilities of python's print(). Instead of print this CLI program could easily be changed to perform any scripting task with flexible and configurable input.

Now that we have our command-line-enabled python script, lets see how well it does…

echoing things

1

./print.py can it echo this?

can it echo this?

displaying help menu

1

./print.py --help

Usage: print.py [OPTIONS] [ARGS]...  Print the arguments you give this program to stdout or to a file. Similar  to echo.Options:  -s, --sep TEXT  String to seperate each argument (default: " ")]  -e, --end TEXT  String to append to end (default: "\n")  -o, --out TEXT  Output file (default: stdout)  --help          Show this message and exit.

Wow… we can already see this has more features than the ever-so-useful echo

For example, we can change the seperator between the args

1

./print.py these args are seperated by dashes --sep='-'

these-args-are-seperated-by-dashes

Also, not always ending with just a newline

1

./print.py maybe I want 2 newlines --end=$'\n\n' && print.py instead

maybe I want 2 newlinesinstead

You can write to a file too

1

./print.py write this to a file --out=file.txt

Then read the file

1

cat file.txt

write this to a file

Making this your own

This was a demonstration to show you how easy it is to create your own CLI. You don't need write plain python scriptfiles anymore.

Take this template and apply it to your own program.

Visit the Click website to reference the Click documentation while building your projects.

Also, for a slightly more complex example, you can check out my QuadArt project which uses Click CLI to be usable by anyone through the terminal. I wrote a blog post about it here. Here is an exerpt of the part of quadart.py that uses Click:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

@click.command()
@click.argument('filename')
@click.option('-l', '--left', default=None, help='left pixel of image')
@click.option('-r', '--right', default=None, help='right pixel of image')
@click.option('-u', '--up', default=None, help='top pixel of image')
@click.option('-d', '--down', default=None, help='bottom pixel of image')
@click.option('-o', '--output', default=None, help='name of file to save result to')
@click.option('-s', '--size', default=512, help='Output size')
@click.option('-t', '--type', 'draw_type', default='circle', help='Draw type')
@click.option('--thresh', default=10, help='Standard deviation threshold for color difference')
@click.option('-m', '--max-recurse', 'max_recurse', default=0, help='Maximum allowed recursion depth. Default is infinity.')
def main(filename, left, right, up, down, output, size, draw_type, thresh, max_recurse):
    quadart = QuadArt(std_thresh=thresh, draw_type=draw_type, max_recurse=max_recurse)
    quadart.generate(filename, left=left, right=right,
                               up=up, down=down,
                               output_size=size)
    if output is None:
        quadart.display()
    else:
        quadart.save(output)

if __name__ == '__main__':
    main()

This is the magic of combining python and the terminal with the glue of CLI. The possibilities are endless!

See you next post.

–Ricky

For a CLI program, or any program with text output, many programmers have been challenged with displaying data in the terminal in a readable format.

Either you are making a script to display data from a database via the python module sqlalchemy, or you are displaying the results of a RESTful API call via the python module requests, you likely have a handful of arbitrary complex python data structures to display. These might be a set of lists, dicts, ints, datetimes, and other objects.

Follow this guide and lets code together a way to display this data in a way you can easily read from the terminal.

In order to have data to display, I will use a complex data structure in the variable fake_data.

1

complex_data=(['hklmvKwrCboZpiwfUhoP', 'bass/explore/search/terms.php', 1678], {'party': 'utyUSpwEkRWicOcjDIeV', 'responsibility': 'williams/list/search.php', 'conference': 'TSrylvPlLNZifFJiqoFS'}, {'culture': {'apples': 'lcZORekRWSmDqfrQRPjc', 'oranges': ['hrobinsonhotmailcom', 6203562684090.42, 'tDNiboBzQtgWdFApUdYC'], 'pairs': {1: '2011-01-12 15:14:13', 2: 'DQsaoDQPYHolbVSmJqRGDQsaoDQPYHolbVSmJqRGDQsaoDQPYHolbVSmJqRGDQsaoDQPYHolbVSmJqRGDQsaoDQPYHolbVSmJqRGDQsaoDQPYHolbVSmJqRG', 3: ['2000-1-10 09:05:01', 'parker-poole/blog/posts/category/login.php']}}})

One way of displaying text is using the popular library pprint which outputs in a format like this:

1
2

import pprint
pprint.pprint(complex_data)

(['hklmvKwrCboZpiwfUhoP', 'bass/explore/search/terms.php', 1678], {'conference': 'TSrylvPlLNZifFJiqoFS',  'party': 'utyUSpwEkRWicOcjDIeV',  'responsibility': 'williams/list/search.php'}, {'culture': {'apples': 'lcZORekRWSmDqfrQRPjc',              'oranges': ['hrobinsonhotmailcom',                          6203562684090.42,                          'tDNiboBzQtgWdFApUdYC'],              'pairs': {1: '2011-01-12 15:14:13',                        2: 'DQsaoDQPYHolbVSmJqRGDQsaoDQPYHolbVSmJqRGDQsaoDQPYHolbVSmJqRGDQsaoDQPYHolbVSmJqRGDQsaoDQPYHolbVSmJqRGDQsaoDQPYHolbVSmJqRG',                        3: ['2000-1-10 09:05:01',                            'parker-poole/blog/posts/category/login.php']}}})

This is already a mostly readable way of displaying data, however this doesn’t work for user-facing output. All the python-specific syntax is left in the output, which end-users don’t care about. Also, the data isn’t all lined up which makes it slightly more difficult to read.

pprint just isn’t pretty enough. What we need is a way to display complex data in a neat and human-readable fashion.

Follow along as we code a function to display arbitrary data.

Defining the problem

In computer science, there is a concept called Dynamic Programming, where you solve a larger problem by solving the sub-problems first. In this case, the larger problem is displaying any arbitrary data, however the sub-problems are displaying units of data which python knows how to display through the built-in str conversion.

All python objects have a string representation through the __str__() function, so the smallest sub-problem to solve is to display any arbitrary string. After solving that problem, we can think about displaying lists of strings and dicts of strings. Even lists of dicts of strings.

Displaying a string

The first step to displaying any arbitrary data is to display a string. You may think it’s as easy as print(my_string), however that isn’t accounting whether the string is being displayed somewhere in the middle of the terminal window.

1
2

                                              We don't want the text to wra
p how we all dread

So how do we fix this?

Write a string-formatter function that knows where to wrap and how much to indent the next line.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

def format_string(string, term_width=80, indent=0):
    # Use the string representation of any object to display
    string = str(string)
    # We can't format a string when there is no room left, so just return it
    if indent >= term_width - 5:
        return string
    # Format each line at a time
    formatted_lines = []
    for line in string.split('\n'):
        # Wrap lines that are too long
        while len(line) > term_width - indent:
            # get line up to width of terminal
            subline = line[:term_width - indent]
            # don't wrap in the middle of a word, if the words are small enough
            if ' ' in subline:
                subline = ' '.join(subline.split(' ')[:-1]) + ' '
            # append the line to the list of formatted lines
            formatted_lines.append(subline)
            # set-up next line to be formatted
            line = line[len(subline):]
        formatted_lines.append(line)
    # Since we aren't indenting the first line,
    if not formatted_lines:
        return ''
    # Return the lines with an indent on all lines except the first one
    return '\n'.join([formatted_lines[0]] +
                     [' '*indent + l for l in formatted_lines[1:]])

This function displays a string without any weird line-wrapping issues. Also, when displaying in the center of the terminal window, the next lines in the string are indented via the indent function argument.

The term_width argument is set to 80 to simulate a 80-character width terminal, however in a real terminal this can be obtained by reading the value of shutil.get_terminal_size()[0].

Now onto the fun stuff. With the function we made above, extra-long strings of text are formatted very nicely, like so:

1
2
3

short_string = 'a short string'

print(format_string(short_string))

a short string

1
2
3

very_long_string = 'a ' + 'very '*20 + 'long string for displaying the format_string\'s text-wrapping feature. As you can see the text does not wrap in the middle of a word.'

print(format_string(very_long_string))

a very very very very very very very very very very very very very very veryvery very very very very long string for displaying the format_string'stext-wrapping feature. As you can see the text does not wrap in the middle of aword.

Also, lets say we want to print a paragraph of text with a label on front. This is possible through format_string‘s indenting of every line except the first line.

An example of this is below.

1
2
3

label = 'the label for my text:'

print(label, format_string(very_long_string, indent=len(label) + 1))

the label for my text: a very very very very very very very very very very very                       very very very very very very very very very long string                       for displaying the format_string's text-wrapping                       feature. As you can see the text does not wrap in the                       middle of a word.

Displaying a list

Now that we can display arbitrary strings, now is time for displaying lists of strings that call the format_string function with the correct indent data.

We will work up to displaying a list of any arbitrary data, however for now lets simply display a list of strings.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

def format_list(lst, term_width=80, indent=0):
    list_item_label = ' - '
    if not lst:
        return ''
    formatted_list = []
    indent_text = ''
    for item in lst:
        if isinstance(item, list):
            formatted_item = format_list(item, term_width,
                                         indent + len(list_item_label))
        else:
            formatted_item = format_string(item, term_width,
                                           indent + len(list_item_label))
        formatted_list.append(indent_text + list_item_label + formatted_item)
        indent_text = ' '*indent
    return '\n'.join(formatted_list)

This function returns the format_string representation of each item in the list, using ' - ' as the label for each item. For handling nested lists, format_list is called recursively when one of its elements is also a list.

Here are a couple examples of formatted lists:

1

print(format_list([1, 2, 3]))

- 1- 2- 3

1

print(format_list([short_string, very_long_string]))

- a short string- a very very very very very very very very very very very very very very very  very very very very very long string for displaying the format_string's  text-wrapping feature. As you can see the text does not wrap in the middle  of a word.

You can see that the text wrapping indents nicely for elements of the list. Lets see how the function does with nested lists.

1
2
3

list_of_lists = [['apples','oranges','pears'],'peaches',[['tangerines','bananas','avacados'],['carrots','oranges','beans']]]

print(format_list(list_of_lists))

-  - apples   - oranges   - pears- peaches-  -  - tangerines      - bananas      - avacados   -  - carrots      - oranges      - beans

Now we’re talking. This nested list of lists of lists printed in a nice readable format to the screen.

1

print(format_list(complex_data))

-  - hklmvKwrCboZpiwfUhoP   - bass/explore/search/terms.php   - 1678- {'party': 'utyUSpwEkRWicOcjDIeV', 'responsibility':  'williams/list/search.php', 'conference': 'TSrylvPlLNZifFJiqoFS'}- {'culture': {'apples': 'lcZORekRWSmDqfrQRPjc', 'oranges':  ['hrobinsonhotmailcom', 6203562684090.42, 'tDNiboBzQtgWdFApUdYC'], 'pairs':  {1: '2011-01-12 15:14:13', 2:  'DQsaoDQPYHolbVSmJqRGDQsaoDQPYHolbVSmJqRGDQsaoDQPYHolbVSmJqRGDQsaoDQPYHolbVSm  JqRGDQsaoDQPYHolbVSmJqRGDQsaoDQPYHolbVSmJqRG', 3: ['2000-1-10 09:05:01',  'parker-poole/blog/posts/category/login.php']}}}

This function’s attempt at printing dicts in a list however epicly fails. To fix this problem, we need a dict-formatter function that is called when formatting an element of a list containnig a dictionary datatype.

Displaying a dict

Now that we have a list-formatting function, we need to format dict-datatypes as well.

Formatting the dict is a bit different than formatting a list. The main difference is that every item in the dict has a key as a label on front of each value. This means that we can’t shortcut the problem with something like format_list(dic.items()).

Lets use the same strategy of populating a list of lines to return, and then combining the lines into one string.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

def format_dict(dic, term_width=80, indent=0):
    max_key_len = max([len(str(k)) for k in dic.keys()])
    formatted_list = []
    indent_text = ''
    for k,v in dic.items():
        if isinstance(v, dict):
            formatted_line = format_dict(v, term_width,
                                         indent + max_key_len + 2)
        elif isinstance(v, list):
            formatted_line = format_list(v, term_width,
                                         indent + max_key_len + 2)
        else:
            formatted_line = format_string(v, term_width,
                                           indent + max_key_len + 2)
        formatted_list.append(indent_text +
                              '{}:'.format(k).ljust(max_key_len + 2) +
                              formatted_line)
        indent_text = ' '*indent
    return '\n'.join(formatted_list)

This simple function for displaying dicts sets up a list of lines to display by calling corresponding format functions recursively for each element in the dict.

1
2
3

nested_dict = {1: 'apples', 123: {'oranges': 20, 'tangerines': 30, 'carrots': 0}, 'something': list(range(3))}

print(format_dict(nested_dict))

1:         apples123:       oranges:    20           tangerines: 30           carrots:    0something:  - 0            - 1            - 2

There is still one problem. The format_list function doesn’t call format_dict when it finds a dict-type object. To fix this issue lets move on to the next step, which is combining all 3 functions.

Combining all 3 string-formatting functions to display any arbitrary data in one function

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

def format_obj(obj, term_width=80, indent=0):
    # format a dict
    if isinstance(obj, dict):
        max_key_len = max([len(str(k)) for k in obj.keys()])
        return '\n'.join([(' '*indent if i > 0 else '') +
                          '{}:'.format(k).ljust(max_key_len + 2) +
                          format_obj(v, term_width,
                                     indent + max_key_len + 2)
                          for i,(k,v) in enumerate(obj.items())])
    # format a list
    elif isinstance(obj, list) or isinstance(obj, tuple):
        list_item_label = ' - '
        return '\n'.join([(' '*indent if i > 0 else '') +
                          list_item_label +
                          format_obj(item, term_width,
                                     indent + len(list_item_label))
                          for i,item in enumerate(obj)])
    # format anything else (with a __str__ representation)
    else:
        obj = str(obj)
        if indent >= term_width - 5:
            return obj
        formatted_lines = []
        for line in obj.split('\n'):
            while len(line) > term_width - indent:
                subline = line[:term_width - indent]
                if ' ' in subline:
                    subline = ' '.join(subline.split(' ')[:-1]) + ' '
                formatted_lines.append(subline)
                line = line[len(subline):]
            formatted_lines.append(line)
        return '\n'.join([(' '*indent if i > 0 else '') + line
                          for i,line in enumerate(formatted_lines)])

1

print(format_obj(complex_data))

-  - hklmvKwrCboZpiwfUhoP   - bass/explore/search/terms.php   - 1678- party:          utyUSpwEkRWicOcjDIeV  responsibility: williams/list/search.php  conference:     TSrylvPlLNZifFJiqoFS- culture: apples:  lcZORekRWSmDqfrQRPjc           oranges:  - hrobinsonhotmailcom                     - 6203562684090.42                     - tDNiboBzQtgWdFApUdYC           pairs:   1: 2011-01-12 15:14:13                    2: DQsaoDQPYHolbVSmJqRGDQsaoDQPYHolbVSmJqRGDQsaoDQPYHolbVSm                       JqRGDQsaoDQPYHolbVSmJqRGDQsaoDQPYHolbVSmJqRGDQsaoDQPYHol                       bVSmJqRG                    3:  - 2000-1-10 09:05:01                        - parker-poole/blog/posts/category/login.php

Now to display a random json file from the internet

1
2
3
4

import requests
import json
json_str = requests.get('https://my-json-server.typicode.com/typicode/demo/db').content.decode('utf-8')
dict_from_json = json.loads(json_str)

1

print(dict_from_json)

{'posts': [{'id': 1, 'title': 'Post 1'}, {'id': 2, 'title': 'Post 2'}, {'id': 3, 'title': 'Post 3'}], 'comments': [{'id': 1, 'body': 'some comment', 'postId': 1}, {'id': 2, 'body': 'some comment', 'postId': 1}], 'profile': {'name': 'typicode'}}

1

print(format_obj(dict_from_json))

posts:     - id:    1             title: Post 1           - id:    2             title: Post 2           - id:    3             title: Post 3comments:  - id:     1             body:   some comment             postId: 1           - id:     2             body:   some comment             postId: 1profile:  name: typicode

This neat representation of nested python lists and dicts is easy to read and flexible for displaying all types of data.

Feel free to use my code and apply it to your own projects.

Happy coding!