Data literacy with Python

Table of Contents

Introduction

What you will learn

Presented here is an example of a typical data analysis and visualisation workflow using Python. You will learn about Python, along with

• Accessing and storing data
• ‘Cleaning’ data
• Analysing data
• Visualising data
• Workflow automation
• Interactive visualisations and beyond

This workflow example is intended to give a coding experience using Python. It is not intended as a rigorous academic exercise in learning Python. There are already many wonderful resources available for free on the internet which serve this purpose.

Our recommendation is to use the self-paced online courses available from the Carpentries for further learning. These include

• Software Carpentry
• Data Carpentry
• Library Carpentry

This example does assume you have already Python installed. Follow the Carpentries instructions, however when asked download this environment file instead of the Carpentries version and use dataliteracywithpython_environment.yml in place of carpentries_environment.yml in the commands.

To run the Jupyter environment, open a terminal window (in windows use the Miniforge prompt), and run these two commands

conda activate dataliteracywithpython
jupyter lab

and at the end of the session

conda deactivate

Basics

So Python is…?

Briefly, Python is

• a programming language.
• thanks to many open-source extension packages is a very useful tool for working with data.
• rumoured to be named after the Monty Python’s Flying Circus comedy series.

Python is used by executing statements of the Python programming language, or code, commonly in a Jupyter Notebook.

Whilst Python is an ‘interpreted language’, many of the extension packages are ‘compiled’, which makes them seriously fast and powerful for processing data.

Preflight

The next steps in this workflow need the following libraries/packages to extend the capabilities of Python. The code can be copy and pasted! Python may return a number of messages related to loading these packages, they can be helpful when developing workflows and can be suppressed when no longer required.

from math import sqrt, sin, pi
import numpy as np
from statistics import mean
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
import sqlite3
import geopandas
import geojson
import json
plt.rcParams["axes.formatter.limits"] = (-99, 99) # avoid scientific notation on matplotlib plots

Let’s begin

Simple mathematics is possible with Python in a Jupyter Notebook. Much like a calculator.

Let’s start with the following code. Execute the code by using the keyboard shortcut Ctrl+Enter (Command+Enter on a Mac).

The output response from Python will appear below the executed code.

1 + 1 

2

Hopefully the answer 2 appeared below the code. The code can be typed over and re-executed again. (Try changing the code above, perhaps to 2 + 3 for example, and re-execute to observe the changed output).

All sorts of maths is possible, including a variety of functions similar to those available on a calculator.

Try the following

( 5 * 6 ) + 12

42

try also

6**2 + 6

42

and finally also try

sqrt(49) * mean(range(1,12)) * sin(pi/2)

42.0

Note. Trig functions are in radians. The range(1,12) command returns a number series, starting from 1 and finishing with 11.

Variables

Variables are at the heart of coding. Variables are placeholders for sets of data, and allow shorthand style code statements to powerfully manipulate data, repeatedly as required.

To see the value of a variable, simply execute the variable name, or use print().

For example, lets assign the variable integer1 with the integer value 42 and then print it.

integer1 = 42
integer1

42

Try the following one at a time to explore some common variable data types and structures.

• Strings

string1 = "forty two"
string1

'forty two'

• Lists

list_string1 = ["apples","oranges","lemons"]
list_string1

['apples', 'oranges', 'lemons']

• Dictionary Strings

dict_string1 = {"fruit1":"apples","fruit2":"oranges","fruit3":"lemons"} 
dict_string1

{'fruit1': 'apples', 'fruit2': 'oranges', 'fruit3': 'lemons'}

One of the packages we loaded earlier was Pandas, which allows for easy manipulation of the data frame type/structure in Python.

The data frame type/structure is very commonly used in data analysis.

Below is an example of creating a data frame manually, showing how it is a variable and how it is stored.

data_frame1 = pd.DataFrame({"fruit":["apples","oranges","lemons"],
                                   "quantity":[7,14,21]})
data_frame1

Sample Data in Python

Data in tabular format, or tables, is a very common starting point when working with data.

Python and the Pandas package don’t include sample datasets, but is possible to use the well known R sample datasets. One way is to load statsmodel package, as we have done previously in the preflight.

One such data set is called mtcars, which has various features for 32 now ancient cars from a 1974 survey for a US car magazine.

Running this with head() will give a stylised table consisting of all columns for the first five rows of data.

mtcars = sm.datasets.get_rdataset('mtcars')
pd.DataFrame(mtcars.data).head(5)

Data analysis and visualisation

Real World Datasets

There are many sources of data on the internet. Governments make public sector data available for activities such as Hackathons, allowing diverse groups of people to provide innovative solutions for communities.

For example, the West Australian State Government has the site Data WA, with over 2000 datasets.

The Australian Government has the site data.gov.au, with over 100,000 datasets.

Another good source is the Australian Bureau of Statistics (ABS)

Dataset 1 - Accessing and Cleaning

Our first dataset is Australian Taxation Statistics 2021-2022, in particular Table 6B which gives summary tax details for individual returns by postcode.

This dataset can be accessed directly from Python, it is an Excel xlsx file. It is published under a Creative Commons Attribution 2.5 Australia licence so is suitable for use here.

Previewing this file, the data really starts in row 2 with the column names. Let’s say we are only interested in some data, somewhat arbitrarily Taxable Income and Private Health cover status related data across States and Postcodes. So we only need to import Columns 1, 3, 4, 6 and 155 (in Python these are referenced as 0, 2, 3, 5 and 154 as the first column is numbered ‘0’).

tax2022_url = 'https://data.gov.au/data/dataset/4be150cc-8f84-46b8-8c61-55ff1d48a700/resource/43d41d1d-4e39-45df-b693-a06255779cff/download/ts22individual06taxablestatusstatesa4postcode.xlsx'

tax2022_raw = pd.read_excel(tax2022_url, sheet_name='Table 6B', skiprows=1, usecols=[0,2,3,5,154])

tax2022_raw["Postcode"] = tax2022_raw["Postcode"].astype('str').str.pad(width=4, side='left', fillchar='0')

tax2022_raw.head(5)

Python/Pandas provides the functionality to change the column names to something easier to work with.

tax2022_raw.columns = ['State', 'Postcode', 'Returns', 'TaxableIncome_dollars', 'PrivateHealth_returns']
tax2022_raw.head(5)

Using only code, Python/Pandas provides the ability to get structure information (info()) and summary descriptions (describe()) of the data.

• Structure/Info

tax2022_raw.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2639 entries, 0 to 2638
Data columns (total 5 columns):
 #   Column                 Non-Null Count  Dtype 
---  ------                 --------------  ----- 
 0   State                  2639 non-null   object
 1   Postcode               2639 non-null   object
 2   Returns                2639 non-null   int64 
 3   TaxableIncome_dollars  2639 non-null   int64 
 4   PrivateHealth_returns  2639 non-null   int64 
dtypes: int64(3), object(2)
memory usage: 103.2+ KB

• Summary/Describe

tax2022_raw.describe()

To filter rows, columns, or any combination of these, there are multiple ways of achieving this in Python/Pandas. Python uses 0-based indexing, the first index of a list is 0.

Filter by

• Columns/Fields, either by column number

tax2022_raw.iloc[:, 1].head(3) # 2nd column

0    2600
1    2601
2    2602
Name: Postcode, dtype: object

• or by Column ‘Name’ (with tail instead of head, to access the last rows of the data frame),

tax2022_raw.loc[:, "Postcode"].tail(4)

2635        6951
2636        6959
2637        6985
2638    WA other
Name: Postcode, dtype: object

• to retrieve the unique values from a column,

tax2022_raw["State"].unique()

array(['ACT', 'NSW', 'NT', 'Overseas', 'QLD', 'SA', 'TAS', 'VIC', 'WA'],
      dtype=object)

• Rows/Records, either by row number, returned a row as a series,

tax2022_raw.iloc[1] # 

State                          ACT
Postcode                      2601
Returns                       3614
TaxableIncome_dollars    265604097
PrivateHealth_returns         1965
Name: 1, dtype: object

• Rows/Records, either by row number, returned a row as a data frame,

tax2022_raw.iloc[[1]] # return a row as a data frame

• by rows which have particular values (or categories) in a column,

tax2022_raw[tax2022_raw["State"].isin(["Unknown","Overseas"])] # returns rows where State **is** Unknown or Overseas

• by Rows and Columns, either by row and column number,

tax2022_raw.iloc[1,1] # 2nd row, 2nd column

'2601'

• or by the row with the maximum value of a column, and returning only three columns of the dataframe.

tax2022_raw.loc[tax2022_raw["PrivateHealth_returns"] == tax2022_raw["PrivateHealth_returns"].max(), ["State","Postcode","PrivateHealth_returns"]]

There are two interesting aspects of the data above which demonstrate the need to ‘clean’ data.

• The tail command above reveals that the data is not exclusively ‘per postcode’; if the number of returns was small those postcodes are grouped into an ‘Other’ row. We will leave this for the moment and observe the impact later in this workflow.

• There are some State values for ‘Overseas’ and ‘Unknown’ which are not of interest. In Python we could create a new dataframe without these. We can also check the new dataframe with the filter to check that it returns no matching rows.

# Create a new, intermediate table without these rows
tax2022_raw_aus = tax2022_raw[~tax2022_raw["State"].isin(["Unknown","Overseas"])]

# Check the rows are now excluded 
tax2022_raw_aus[tax2022_raw_aus["State"].isin(["Unknown","Overseas"])].info()

<class 'pandas.core.frame.DataFrame'>
Index: 0 entries
Data columns (total 5 columns):
 #   Column                 Non-Null Count  Dtype 
---  ------                 --------------  ----- 
 0   State                  0 non-null      object
 1   Postcode               0 non-null      object
 2   Returns                0 non-null      int64 
 3   TaxableIncome_dollars  0 non-null      int64 
 4   PrivateHealth_returns  0 non-null      int64 
dtypes: int64(3), object(2)
memory usage: 0.0+ bytes

Method Chaining

In cleaning and subsetting the data above we now have two data frames, namely tax2022_raw and tax2022_raw_aus. In trying to keep the Python commands relatively short and understandable we can end up with a lot of intermediate or temporary data frames, which can be difficult to keep track of. Another alternative is to use method chaining, where one method is called on another method, and so on, forming a ‘chain’ of methods (or actions) performed on the data. Effectively the output of one command is ‘piped’ into the next command and so on. This strikes a great balance between command readability and minimising intermediate data frames.

Method chaining is available for a number of core and add-on packages, including Pandas. We have already been using method chaining in much of the above!

For example, to achieve removing the same rows in the previous step, we can apply the query() method directly to the tax2022_raw data and compare the total rows with the previous info() method. Methods are chained using the ‘dot’ operator/notation.

Note that the head command is part of method chaining too.

Note also that for the remainder of the workshop we have shown each new method on a new line, which requires enclosing the whole statement in outer brackets (). The indents here are not critical, as distinct from other commands in Python such as loops or if-the-else, where indentation is critical.

(tax2022_raw.query('~State.isin(["Unknown","Overseas"])')
            .info())

<class 'pandas.core.frame.DataFrame'>
Index: 2638 entries, 0 to 2638
Data columns (total 5 columns):
 #   Column                 Non-Null Count  Dtype 
---  ------                 --------------  ----- 
 0   State                  2638 non-null   object
 1   Postcode               2638 non-null   object
 2   Returns                2638 non-null   int64 
 3   TaxableIncome_dollars  2638 non-null   int64 
 4   PrivateHealth_returns  2638 non-null   int64 
dtypes: int64(3), object(2)
memory usage: 123.7+ KB

Key Learning

Key Learning #1 - Data is data, there is no need to constantly have a dedicated view for the raw, original data. These tools allow us to view it in any form needed so as to inform our analysis and visualisation.

Key Learning #2 - Cleaning the data involves investigating the original data, leaving it as it is and writing code to create a workable dataset, having removed unnecessary or incorrect data.

Key Learning #3 - Using method chaining makes it simpler to prepare, read and modify code and eliminates the need for the clutter of many intermediate or temporary data frames.

Further Learning

Further Learning #1 - The datasets in this workshop are quite ‘clean’ and complete. Then there are datasets which are incomplete with data that is not a number or NAN - for another time.

Dataset 1 - Analysis

As an example, let’s perform some aggregate functions, such as sums or totals of dollars and returns for each State. Whereas query() acts on rows, drop() can act on columns and rows (axis=). We will aggregate by State, so we remove the Postcode column using drop(). We could select the rows we require as per above, however here it is more convenient to drop the few column(s) we don’t require.

To calculate the sums we can pass the data to a groupby() command to group by State, and then pass that result to a sum() command to perform the aggregation on all columns.

# Totals by State
(tax2022_raw.query('~State.isin(["Unknown","Overseas"])')
            .drop("Postcode", axis=1)
            .groupby("State")
            .sum())

To calculate the sums for the whole of Australia for 2021-2022, let’s filter the State column too.

# Totals for Australia
(tax2022_raw.query('~State.isin(["Unknown","Overseas"])')
            .drop(["Postcode","State"], axis=1)
            .sum())

Returns                       15411738
TaxableIncome_dollars    1119737214609
PrivateHealth_returns          8722919
dtype: int64

As a further example, let’s calculate the

• Taxable Income per return per State, and
• Percent of Private Health Insurance per State

Here we are creating two new calculated columns based on the data for each row, and so will use assign() to create the new columns and mean() for the summary.

# Means per State
(tax2022_raw.query('~State.isin(["Unknown","Overseas"])')
            .drop("Postcode", axis=1)
            .assign(TaxableIncome_dollarspr = lambda x: x.TaxableIncome_dollars/x.Returns)
            .assign(PrivateHealth_percentpp = lambda x: x.PrivateHealth_returns/x.Returns*100)
            .loc[:, ["State", "TaxableIncome_dollarspr", "PrivateHealth_percentpp"]]
            .groupby("State")
            .mean())

Wide and Narrow Formats

The raw data is in summary form, or wide form. Easily read by people, not ideal for all the processing options available for machines eg AI.

Sometimes tasks in Python are more easily achieved with the data in narrow or long format, where each row essentially only has one item of data.

Fortunately Python/Pandas have tools which allow for easily swapping between formats, namely melt and pivot.

What is important is knowing which columns are to be kept, often called identifier variables ( Postcode and State ), and which columns are to be pivoted, often called measured variables ( Returns, Taxable Income and Private Health status ).

Let’s create a temporary dataframe tax2022_raw_long here just to show the effect of moving from wide to narrow/long formats and back again.

From Wide to Narrow/Long

tax2022_raw_long = (tax2022_raw.query('~State.isin(["Unknown","Overseas"])')
                               .melt(id_vars=["State","Postcode"],var_name="item"))
tax2022_raw_long.head(5)

and then back from Narrow/Long to Wide

(tax2022_raw_long.pivot(index=["State","Postcode"],columns="item",values="value")
                 .reset_index()
                 .head(5))

Visualisation in Python

Python also has functions to visualise data. One common package used for visualisations is matplotlib, which along with Pandas can create visualisations with a minimal effort.

The code defines the data to be used, and then we use code to generate the graphs and assign values to the different aspects of the graphs. Code generated visualisation can be very efficient compared to GUI based platforms.

For example, to quickly visualise (without much styling!) the summed totals of the raw data per State

• we take the code from earlier (#Totals by State),
• and simply use .plot.bar to create a barchart. Using subplots shows the different variables in separate subplots or facets, each with independent y axis ranges.

(tax2022_raw.query('~State.isin(["Unknown","Overseas"])')
            .drop("Postcode", axis=1)
            .groupby("State")
            .sum()
            .reset_index()
            .plot.bar(x="State",subplots=True, figsize=(12, 4), layout=(1,3), legend=False))
plt.tight_layout()

To quickly visualise the mean values per state

• we take the code from earlier (#Means per State),
• and similarly use .plot.bar to create a barchart.

(tax2022_raw.query('~State.isin(["Unknown","Overseas"])')
            .drop("Postcode", axis=1)
            .assign(TaxableIncome_dollarspr = lambda x: x.TaxableIncome_dollars/x.Returns)
            .assign(PrivateHealth_percentpp = lambda x: x.PrivateHealth_returns/x.Returns*100)
            .loc[:, ["State", "TaxableIncome_dollarspr", "PrivateHealth_percentpp"]]
            .groupby("State")
            .mean()
            .reset_index() 
            .plot.bar(x="State",subplots=True, figsize=(12, 4), layout=(1,2), legend=False))
plt.tight_layout()

Note: These plots are images; they can become blurry when enlarged and are not responsive on different devices such as mobiles or tablets. Other visualisation tools, such as Bokeh (presented later) are more versatile as they effectively recreate the plot to suit each device.

Automation

When the source data changes, for example more data samples are collected or updated, using code to manipulate the data brings a massive advantage - automation. The same code can be re-executed on the new data for updated analysis and visualisations.

As an example, here is the code used to sum the three Taxation parameters for Australia for 2021-2022, re-executed for the 2020-2021 dataset, also published under Creative Commons Attribution 2.5 Australia. The code required a small tweak, the columns are in a different order compared to 2021-2022.

Compare the results for 2020/21 and 2021/22.

• 2020/2021

tax2021_url = 'https://data.gov.au/data/dataset/07b51b39-254a-4177-8b4c-497f17eddb80/resource/fa05bac8-079d-4eba-bd4d-779466e45f02/download/ts21individual06taxablestatusstateterritorypostcode.xlsx'

tax2021_raw = pd.read_excel(tax2021_url, sheet_name='Table 6B', skiprows=1, usecols=[0,1,2,4,153])

tax2021_raw.columns = ['State', 'Postcode', 'Returns', 'TaxableIncome_dollars', 'PrivateHealth_returns']

(tax2021_raw.query('~State.isin(["Unknown","Overseas"])')
            .drop(["Postcode","State"], axis=1)
            .assign(TaxableIncome_dollarspr = lambda x: x.TaxableIncome_dollars/x.Returns)
            .assign(PrivateHealth_percentpp = lambda x: x.PrivateHealth_returns/x.Returns*100)
            .loc[:, ["TaxableIncome_dollarspr", "PrivateHealth_percentpp"]]
            .mean())

TaxableIncome_dollarspr    62881.598176
PrivateHealth_percentpp       55.713724
dtype: float64

• 2021/22

(tax2022_raw.query('~State.isin(["Unknown","Overseas"])')
            .drop(["Postcode","State"], axis=1)
            .assign(TaxableIncome_dollarspr = lambda x: x.TaxableIncome_dollars/x.Returns)
            .assign(PrivateHealth_percentpp = lambda x: x.PrivateHealth_returns/x.Returns*100)
            .loc[:, ["TaxableIncome_dollarspr", "PrivateHealth_percentpp"]]
            .mean())

TaxableIncome_dollarspr    67519.740410
PrivateHealth_percentpp       55.812433
dtype: float64

Also compare the plot of mean values for 2020/21.

(tax2021_raw.query('~State.isin(["Unknown","Overseas"])')
            .drop("Postcode", axis=1)
            .assign(TaxableIncome_dollarspr = lambda x: x.TaxableIncome_dollars/x.Returns)
            .assign(PrivateHealth_percentpp = lambda x: x.PrivateHealth_returns/x.Returns*100)
            .loc[:, ["State", "TaxableIncome_dollarspr", "PrivateHealth_percentpp"]]
            .groupby("State")
            .mean()
            .reset_index() 
            .plot.bar(x="State",subplots=True, figsize=(12, 4), layout=(1,2), legend=False))
plt.tight_layout()

Key Learning

Key Learning #4 - Data frame structures are easily transformed in Python, transforming to whatever form is convenient for a particular purpose.

Key Learning #5 - Using code to perform analysis and generate graphs and visualisations saves a lot of time and finessing, particularly when tasks need to be repeated regularly and often.

Further Learning

Further Learning #2 - The above visualisations were generated quickly, without too much concern for formatting. Every aspect of the graphs above can be controlled to give beautiful and purposeful visualisations.

Combining two datasets

Dataset 2 - Accessing and Cleaning

The ABS produces The Socio-Economic Indexes for Areas (SEIFA) based on the Census of Population and Housing. These indexes provide a measure of relative socio-economic advantage and disadvantage across different areas of Australia. It includes this dataset, which includes from Table 5 an Index of Education and Occupation, which reflects the education and occupational level per postcode. The dataset is also published under a suitable Creative Commons Licence.

Looking at the spreadsheet, the data we are interested in are the Postcode and Percentile/Ranking within Australia columns of the Table 5 sheet. The data starts on row 7. We can import this data directly and then discard all but columns 0 and 6 (Postcode and Percentile/Rank).

seifa2021_url = 'https://www.abs.gov.au/statistics/people/people-and-communities/socio-economic-indexes-areas-seifa-australia/2021/Postal%20Area%2C%20Indexes%2C%20SEIFA%202021.xlsx'

seifa2021_raw = pd.read_excel(seifa2021_url , sheet_name='Table 5', skiprows=5, nrows=2627, usecols=[0,6], dtype=object)

seifa2021_raw.columns = ['Postcode','ieo_percentile']

seifa2021_raw["ieo_percentile"] = pd.to_numeric(seifa2021_raw["ieo_percentile"]).astype('Int64')

seifa2021_raw.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2627 entries, 0 to 2626
Data columns (total 2 columns):
 #   Column          Non-Null Count  Dtype 
---  ------          --------------  ----- 
 0   Postcode        2627 non-null   object
 1   ieo_percentile  2627 non-null   Int64 
dtypes: Int64(1), object(1)
memory usage: 43.7+ KB

Combining datasets

Combining datasets offers great opportunities for data analysis and insights.

Our datasets both have postcode columns, so can be combined to compare education levels with income and private health status data.

Python/Pandas offer simple merge functions which can combine two datasets, based on common column(s) in both datasets.

We will use the merge method. We only have one column in common; if there are more than one columns in common, they would simply need adding to the by parameter as a list.

Finally a note about cleaning - there may be differences in the postcode allocations between the SEIFA and Taxation data, we will leave this discrepancy for now and see the impact later in this workflow.

(tax2022_raw.query('~State.isin(["Unknown","Overseas"])')
            .assign(TaxableIncome_dollarspr = lambda x: x.TaxableIncome_dollars/x.Returns)
            .assign(PrivateHealth_percentpp = lambda x: x.PrivateHealth_returns/x.Returns*100)
            .merge(seifa2021_raw, how="inner", on="Postcode")
            .loc[:, ["State", "Postcode", "ieo_percentile", "TaxableIncome_dollarspr", "PrivateHealth_percentpp"]]
            .query('Postcode=="6102"'))

A note about SQL

For those with a background in SQL, R interfaces with many different databases and queries can be executed as if using the native SQL interface.

In the example below, the package RSQLite is used to

• create a SQLite database,

conn = sqlite3.connect('tax_seifa.db', isolation_level = None)

• refresh our memories of the tax2022_raw data

tax2022_raw.head(5)

• refresh our memories of the seifa2021_raw data

seifa2021_raw.head(5)

• write the tax data from the above merge() method to a table,
• write the seifa data from the above merge() method to a table,

seifa2021_raw.to_sql("seifa2021_tbl",conn, if_exists='replace', method='multi')
tax2022_raw.to_sql("tax2022_tbl",conn, if_exists='replace', method='multi')

2639

• execute a query using SELECT on the two tables to achieve the same output as the above merge() command.

pd.read_sql_query('SELECT seifa2021_tbl.ieo_percentile, tax2022_tbl.* \
                   FROM seifa2021_tbl \
                   JOIN tax2022_tbl ON seifa2021_tbl."Postcode" = tax2022_tbl."Postcode" \
                   WHERE seifa2021_tbl."Postcode" = 6102', conn)

and finally close the connection to the database.

conn.close()

Map visualisations

GeoPandas and Map Shapefiles

Python and Pandas have some powerful packages to make use of map data. Map data can be imported for example as a shapefile and visualised along with data using GeoPandas.

These shapefiles are not images based on pixels, but use vectors to effectively redraw the postcode boundaries in whatever environment it is used. They provide much more definition than is needed to display on a screen on a web page, so there are tools in Python, namely topojson, which resample the vectors to suit a webpage, greatly simplifying the file and making it smaller and quicker to render.

Once again, the map/shapefile data for Postcodes in Australia is readily available from the ABS website, with a suitable Creative Commons Licence.

To reduce time and resources needed for this workflow, we have created a preprocessed shapefile named pc_sf_raw.shz, which is already a simplified version of the map file and can be loaded with the code below.

pc_sf_raw = geopandas.read_file('pc_sf_raw.shz')
pc_sf_raw.head(5)

See also the list of points to draw the postcode boundary/polygon for Bentley, Perth (6102).

pc_sf_raw[pc_sf_raw['POA_CODE21'] == '6102']['geometry'].get_coordinates()

Full citation for source of modified map/shapefile:

Australian Bureau of Statistics (2021) 'Non ABS Structures: Postal Areas - 2021 [https://www.abs.gov.au/statistics/standards/australian-statistical-geography-standard-asgs-edition-3/jul2021-jun2026/access-and-downloads/digital-boundary-files]' [Shapefile], Digital boundary files: Australian Statistical Geography Standard (ASGS) Edition 3, accessed 27th February 2024.

How was this file created?

The file was created using the code below, there is no need to execute the code in this workflow, it can take a while and depending on the prequantize/epsilon values chosen (which here determine the level of simplification for each postcode’s boundaries), can cause ‘out of memory’ type errors.

import topojson as tp
pc_sf_url = 'https://www.abs.gov.au/statistics/standards/australian-statistical-geography-standard-asgs-edition-3/jul2021-jun2026/access-and-downloads/digital-boundary-files/POA_2021_AUST_GDA2020_SHP.zip'
pc_sf_in = geopandas.read_file(pc_sf_url).to_crs(epsg=4283)
pc_sf_topo = tp.Topology(pc_sf_in.dropna(), prequantize=False, topology=True)
pc_sf_raw = pc_sf_topo.toposimplify(epsilon=0.006).to_gdf()
pc_sf_raw.to_file('pc_sf_raw.shz')

Demonstration 1 - GeoPandas Visualisation

Here we create a visualisation demonstration including a map of Australia with postcodes shown in colours reflecting the Index of Education and Occupation, and hovering over each postcode will display a label detailing the combined tax/seifa data from earlier in abbreviated form.

Though explaining the code for the visualisation is beyond the scope of this workflow, a powerful visualisation has been created with relatively little code. We can also see the effects of not ‘cleaning’ the data earlier. We can see areas missing data, where tax data was summarised into ‘Other’ categories, or there was a change in postcodes between the two data sources. The merge commands from earlier couldn’t find a match between the two datasets for these Postcodes and thus there is no corresponding data.

# Create a dataframe from earlier # Tax data combined workflow
tax_seifa = (tax2022_raw.query('~State.isin(["Unknown","Overseas"])')
            .assign(TaxableIncome_dollarspr = lambda x: round(x.TaxableIncome_dollars/x.Returns/1000,0))
            .assign(PrivateHealth_percentpp = lambda x: round(x.PrivateHealth_returns/x.Returns*100,0))
            .merge(seifa2021_raw, how="inner", on="Postcode"))

# Combine map shapefile and tax data
pc_sf = pc_sf_raw.merge(tax_seifa, left_on='POA_CODE21', right_on='Postcode', how="inner")

# Add a label to data which combines all of the tax data into a single abbreviated field
pc_sf["Details"] = "PCode:" + pc_sf["POA_CODE21"] + " Income:$" + pc_sf["TaxableIncome_dollarspr"].astype('str') + "K PrivHlth:" + pc_sf["PrivateHealth_percentpp"].astype('str') + "% IEO:" + pc_sf["ieo_percentile"].astype('str')

# Demo Visualisation 1
pc_sf.explore("ieo_percentile",
              cmap = "YlOrRd_r",
              scheme='quantiles',
              tiles = "cartodb voyager",
              tooltip = ["Details"],
              style_kwds = dict(color='black',weight=0.25),
              highlight_kwds = dict(color="black",weight=0.75,fillOpacity=0.75),
              legend_kwds = dict(caption="SEIFA Index of Education/Occupation percentile"))

REPLACEMEHERE

Interactive visualisations

Interactive Visualisations

In the previous visualisations the user can’t change the core information which is shown, in effect the visualisations are ‘static’.

The most useful visualisations don’t limit the user to a static analysis, they incorporate interactivity, where the visualisation can be changed by the user varying relevant parameters.

The user is not dictated to, instead they are guided and can choose their own analysis journey. Incorporating intuitive elements can allow even more insights.

There is always one key issue which needs to be addressed with data analysis and interactive visualisations - how to share them with those who will use them?

Essentially the humble web browser is the one application we know will most likely be on any device capable of using the visualisations.

Assuming we have a method of sharing a webpage (eg GitPages, static website instance), let’s explore some visualisations which function completely within a web browser without the need for a separate data server.

Bokeh and JS Callbacks

Bokeh is a powerful package for creating static and interactive visualisations. Whilst a little more complicated to setup than Matplotlib from earlier, Bokeh allows interactive visualisations which work in a web browser alone and are light on resource use. Interactivity is achieved using JavaScript callbacks, where Javascript ‘aware’ Bokeh functions and small amounts of JavaScript are included to update a visualisation when an interactive element is changed, such as a slider or a button.

Demonstration 2 - Bokeh Visualisation

Let’s create a visualisation which explores correlations between the two data sources earlier. We will plot SEIFA percentile/rank versus Private Health participation for each Postcode for a user chosen range of Taxable Income.

This is achieved by

• Recreating the previous dataframe combining tax and seifa data, with an additional column created ‘Returns_scaled’, to limit the variation in ‘bubble size’ in the plot.
• Defining a RangeSlider to allow the user to vary the range of Mean Taxable Income shown.
• In Bokeh 3.6.3, for the legend to function correctly it is necessary to plot the ‘bubbles’ separately for each State.
• Creating a ColumnDataSource for each State’s data.
• Creating BooleanFilters, also for each State, which are initialised to show all and then use JS Callbacks (CustomJS) to update the view (CDSView) for each State based on a change in the slider range (js_on_change).
• Adding Tooltips for the Hover function.

Can you draw any interesting conclusions from the visualisations?

from bokeh.plotting import figure, show
from bokeh.layouts import layout
from bokeh.io import output_notebook
from bokeh.models import ColumnDataSource, CDSView, BooleanFilter, RangeSlider, CustomJS, Range1d
from bokeh.transform import factor_cmap
from bokeh.palettes import Paired

output_notebook()

# Create a dataframe from earlier # Tax data combined workflow
# map the number of Returns for each Postcode into a range 0.5 thru 4, to limit range of circle sizes in plot
tax_seifa = (tax2022_raw.query('~State.isin(["Unknown","Overseas"])')
            .assign(TaxableIncome_dollarspr = lambda x: round(x.TaxableIncome_dollars/x.Returns/1000,0))
            .assign(PrivateHealth_percentpp = lambda x: round(x.PrivateHealth_returns/x.Returns*100,0))
            .assign(Returns_scaled = lambda x: np.interp(x.Returns,[tax_seifa['Returns'].min(),tax_seifa['Returns'].max()],[0.5,4]))
            .merge(seifa2021_raw, how="inner", on="Postcode"))

# The legend in Bokeh @ver:3.6.3 behaves incorrectly when using a single circle plot command for all States along with JS Callbacks
# Solution is to use separate circle plot commands maintained in arrays.
sources = []
tifilters = []
states = tax_seifa['State'].unique()
for state in states:
    sources.append(ColumnDataSource(tax_seifa[tax_seifa['State'] == state]))
    tifilters.append(BooleanFilter([True]*len(tax_seifa))) # initialise BooleanFilter to all True

range_sliderti = RangeSlider(
    title='Mean Taxable Income per return ($K)',
    start=tax_seifa['TaxableIncome_dollarspr'].min(),
    end=tax_seifa['TaxableIncome_dollarspr'].max(),
    step=1,
    value=(tax_seifa['TaxableIncome_dollarspr'].min(),tax_seifa['TaxableIncome_dollarspr'].max())
)

callback = CustomJS(args=dict(tifilters=tifilters, sources=sources), code="""
                                          const start = cb_obj.value[0];
                                          const end = cb_obj.value[1];
                                          for (var sourceno = 0; sourceno < sources.length; sourceno++) {
                                            const bools = []
                                            for (var i = 0; i < sources[sourceno].length; i++) {
                                              if (sources[sourceno].data['TaxableIncome_dollarspr'][i] >= start && sources[sourceno].data['TaxableIncome_dollarspr'][i] <= end) {
                                                bools.push(true);
                                              }
                                              else {
                                                bools.push(false);
                                              }
                                            }
                                            tifilters[sourceno].booleans = bools;
                                          }
                                          sources[sourceno].change.emit(); 
                                          """)
range_sliderti.js_on_change('value', callback)

TOOLTIPS = [
    ("ieo", "@ieo_percentile"),
    ("ph", "@PrivateHealth_percentpp"),
    ("rt", "@Returns"),
    ("ti", "@TaxableIncome_dollarspr"),
    ("pc", "@Postcode")    
]

p = figure(title='Demonstration 2 - Bokeh Visualisation with interactive slider',
           x_axis_label='ieo_percentile',
           y_axis_label='PrivateHealth_percentpp',
           tooltips=TOOLTIPS,           
           lod_threshold=None,
           match_aspect=False,
           width=1000)

for idx, state in enumerate(states):
    p.circle(x='ieo_percentile',
             y='PrivateHealth_percentpp',
             radius='Returns_scaled',
             fill_color=factor_cmap('State', palette=Paired[8], factors=states),
             fill_alpha=0.5,
             source=sources[idx],
             legend_label=state,
             view=CDSView(filter=tifilters[idx]))

p.legend.location = "top_left"
p.legend.click_policy="hide"
p.x_range = Range1d(-10,110)
p.y_range = Range1d(-10,110)

layout = layout(
    [
        [range_sliderti],        
        [p],
    ],
)

show(layout)

REPLACEMEHERE

Interactive map visualisations

Demonstration 3 - Bokeh Interactive Map Visualisation

In this visualisation, once again achieved with a relatively small amount of code, two sliders are used to select Postcodes based on SEIFA Index of Education and Occupation and also Private Health Cover Percentage.

What interesting correlations can you find?

from bokeh.plotting import figure, show
from bokeh.layouts import layout
from bokeh.io import output_notebook
from bokeh.models import GeoJSONDataSource, LinearColorMapper, ColumnDataSource, CDSView, BooleanFilter, RangeSlider, CustomJS

output_notebook()

# switch to mercator projection to match OpenStreetMap background
pc_sf_mercator = pc_sf.to_crs(epsg=3857)
pc_sf_mercator['long'] = pc_sf_mercator.centroid.x
pc_sf_mercator['lat'] = pc_sf_mercator.centroid.y
pc_sf_data = pc_sf_mercator[['Details','PrivateHealth_percentpp','ieo_percentile','Postcode','long','lat']]

source = ColumnDataSource(pc_sf_data)
geo_source = GeoJSONDataSource(geojson=pc_sf_mercator.to_json())

range_sliderieo = RangeSlider(
    title='SEIFA Index of Education and Occupation',
    start=0,
    end=100,
    step=1,
    value=(pc_sf_data['ieo_percentile'].min(), pc_sf_data['ieo_percentile'].max()),
)
range_sliderph = RangeSlider(
    title='Private Health Cover Percentage per Postcode',
    start=0,
    end=100,
    step=1,
    value=(pc_sf_data['PrivateHealth_percentpp'].min(), pc_sf_data['PrivateHealth_percentpp'].max())
)

ieofilter = BooleanFilter([True]*len(pc_sf_data))
phfilter = BooleanFilter([True]*len(pc_sf_data))

callback_ieo = CustomJS(args=dict(ieofilter=ieofilter, source=source), code="""
                                          const start = cb_obj.value[0];
                                          const end = cb_obj.value[1];
                                          const bools = []                                          
                                          for (var i = 0; i < source.length; i++) {
                                            if (source.data['ieo_percentile'][i] >= start && source.data['ieo_percentile'][i] <= end) {
                                              bools.push(true);
                                            }
                                            else {
                                              bools.push(false);
                                            }
                                          }
                                          ieofilter.booleans = bools;
                                          source.change.emit(); 
                                          """)

callback_ph = CustomJS(args=dict(phfilter=phfilter, source=source), code="""
                                          const start = cb_obj.value[0];
                                          const end = cb_obj.value[1];
                                          const bools = []                                          
                                          for (var i = 0; i < source.length; i++) {
                                            if (source.data['PrivateHealth_percentpp'][i] >= start && source.data['PrivateHealth_percentpp'][i] <= end) {
                                              bools.push(true);
                                            }
                                            else {
                                              bools.push(false);
                                            }
                                          }
                                          phfilter.booleans = bools;
                                          source.change.emit(); 
                                          """)

range_sliderieo.js_on_change('value', callback_ieo)
range_sliderph.js_on_change('value', callback_ph)

TOOLTIPS = [
    ("ieo", "@ieo_percentile"),
    ("ph", "@PrivateHealth_percentpp"),
    ("ti", "@TaxableIncome_dollarspr"),
    ("pc", "@Postcode")  
]

p = figure(title='Aus',
           height=600, 
           width=800, 
           tools=['pan','wheel_zoom','hover','reset'], 
           tooltips=TOOLTIPS,
           x_axis_type='mercator',
           y_axis_type='mercator',
           lod_threshold=None,
           match_aspect=True
          )

p.patches('xs', 'ys', 
          fill_alpha=0.0, 
          fill_color='white', 
          line_color='black', 
          line_width=0.5, 
          source=geo_source
         )

p.add_tile('CartoDB Voyager', 
           retina=True
          )

p.scatter(x="long", 
          y="lat", 
          size=8, 
          color="blue",          
          hover_color="red", 
          source=source, 
          view=CDSView(filter=ieofilter & phfilter)
         )

layout = layout(
    [
        [range_sliderieo,range_sliderph],        
        [p]
    ]
)

show(layout)

REPLACEMEHERE

Key Learning

Key Learning #6 - Datasets with common ‘keys’ can be combined to allow for more powerful analysis.

Key Learning #7 - Python with SQLite can perform SQL style queries with external SQL databases.

Key Learning #8 - Interactive visualisations are best; the user is not dictated to, instead they are guided and can choose their own analysis journey. Incorporating intuitive elements can allow even more insights.

Key Learning #9 - Web browsers are the ideal vehicle to use for sharing visualisations. Python has many tools which can present visualisations using only the web browser, no need for other software or databases/data servers.

Next steps

Looking beyond this introduction

GitHub and GitPages

Jupyter notebooks stored in GitHub will be shown formatted (but unrendered) in the code view, but with GitPages it is possible to display the complete rendered version of the notebook on your own site…for free!

JSON Data

JSON (JavaScript Object Notation) is a serial data format which is widely used, particularly on the internet when obtaining data from API’s (Application Programming Interface).

It allows nested storage of data not possible with a two-dimensional table alone.

{
    "title": "Fruits",
    "details": "JSON example for R workshop",
    "fruits": [
    {
        "fruit": "Apples",
        "quantity": 7,
        "varieties": [
		{
            "variety": "Royal Gala",
			"supplier": "Apples'R'Us"
		},
		{
            "variety": "Fuji",
			"supplier": "The Orchard down the road"
		}]
    },
    {
        "fruit": "Oranges",
        "quantity": 14,
        "varieties": [
		{
            "variety": "Navel",
			"supplier": "The other Orchard"
		}]
    },
    {
        "fruit": "Lemons",
        "quantity": 21
    }]
}

Licensing and copyright

Unless otherwise stated all content in this workflow is COPYRIGHT © Curtin University 2024. Curtin branding and trademarks are COPYRIGHT © Curtin University.

The text, images, data, documentation and general web content presented in this Jupyter notebook, unless stated otherwise, is licensed Creative Commons Attribution-ShareAlike 4.0 International. The Python code presented in this Jupyter notebook is licensed BSD-3-clause. The maps demonstration (Demonstration 1) is produced using Python’s Geopandas.explore and folium, which in turn uses the Leaflet.js library. Demonstrations 1 and 3 include content from OpenStreetMap and CartoDB.

As mentioned in the text, the data used here is licenced as follow.

• The dataset for Australian Taxation Statistics 2021-2022, which is an Excel formatted xlsx file, is published under a Creative Commons Attribution 2.5 Australia licence.
• The dataset for Australian Taxation Statistics 2020-2021, which is an Excel formatted xlsx file, is published under a Creative Commons Attribution 2.5 Australia licence.
• The specific dataset from the Socio-Economic Indexes for Areas (SEIFA), which is an Excel formatted xls file, is published under a Creative Commons Licence.
• The map data for Postcodes, which is an Shapefile, is published under a Creative Commons Licence.

The various software tools used here are licenced as follows.

• Jupyter Notebook is provided under the BSD-3-Clause license
• Python is provided under the PSF-2.0 licence.
• Python packages
  • Pandas, statsmodels, GeoPandas, geojson and topojson all use the BSD-3-Clause licence
  • Matplotlib uses this open source style licence

About UniSkills

UniSkills was created and is maintained by Curtin University Library. To report issues with UniSkills contact Library Help.

Twemoji icons by Twitter, Inc and other contributors, licensed under a CC-BY 4.0 licence.

Except where otherwise noted, UniSkills content in all it’s forms (website, PDFs etc) are licensed under a Creative Commons Attribution ShareAlike 4.0 International Licence. We ask that you attribute any use of the content as created by Curtin University Library with a link to the Library website.

This license does not extend to other Curtin University and Curtin University Library webpages, or to Curtin branding and trademarks. Curtin University’s copyright information is available on the Curtin website.