Python, handling dictionaries with dot notation

Logo

Introduction

A simple JSON file :

resources.json
{
	"id": 307
	,"date": "April 14, 2020"
	,"thumbnail": "python"
	,"langurls":[
		{"lang":"fr" , "file":"python-parsing-des-arguments-argparse-et-getopt.html"}
		,{"lang":"en", "file":"python-parsing-arguments-packages-argparse-getopt.html"}
	]
  ,"sitemap" : {  "title" : "Conception - Python", "url" : "/conception/python" }
}

With Javascript, after fetching a JSON file, properties or attributes are usually handled in dot notation :

props = {};

load_prop = async function() {
	const res = await fetch("https://www.sqlpac.ger/referentiel/docs/resources.json");
	if (res.status==200)  response = await res.json(); 
	props = response;
};

show_prop = function() {
	console.log(props.date);
};

load_prop().then(show_prop);
console> April 14, 2020

Switching to Python, bad news : dot notation is not directly available with the resulting dictionary.

import json

def main():
  with open('resources.json','rb') as f:
    props = json.load(f)
    print(props.date)

if (__name__=="__main__"):
  main()
    print(props.date)
AttributeError: 'dict' object has no attribute 'date'

We must use the usual syntax : props["attribute1"]["attribute2"]…

import json

def main():
  with open('resources.json','rb') as f:
    props = json.load(f)
    print(props["date"])

if (__name__=="__main__"):
  main()
April 14, 2020

Python is not Javascript. Habits, habits, hard to remove them. This example deals with loading JSON data, the same context occurs when loading YAML data…

How to use dot notation with Python for dictionaries ? It can be achieved natively but when dictionaries are nested, it becomes tedious without developing its own library. Maintained community packages exist to do the job : Prodict, python-box.

The native methods

Populating a dictionary class

A dictionary class is populated with the data : 

import json

class clsprops:
	def __init__(self, data):
		self.__dict__ = data
	
def main():
  with open('resources.json','rb') as f:
    props = clsprops(json.load(f))
    print(props.date)

if (__name__=="__main__"):
  main()
April 14, 2020

But what about nested dictionaries attributes ( { … ,"sitemap": { "url":"…", "title":"…" } … } ) ? Predictable, nested dictionaries attributes can not be used with dot notation

def main():
  with open('resources.json','rb') as f:
    props = clsprops(json.load(f))
    print(props.sitemap.url)
    print(props.sitemap.url)
AttributeError: 'dict' object has no attribute 'url'

props.sitemap["url"] must be used.

Using SimpleNamespace

A more elegant code using SimpleNamespace without declaring a custom class : 

import json
from types import SimpleNamespace

def main():
	with open('resources.json','rb') as f: 
		props = SimpleNamespace(** json.load(f))
		print(props.date)

if (__name__=="__main__"):
	main()
April 14, 2020

Same issue than using a custom class, nested dictionaries attributes are not accessible in dot notation :


def main():
	with open('resources.json','rb') as f: 
	props = SimpleNamespace(** json.load(f))
	print(props.sitemap.url)
AttributeError: 'dict' object has no attribute 'url'

Dot notation dictionary packages

Fortunately, there are so many packages in the Python community, so we should find some packages covering this need. The hardest is to find the ones providing the best functionalities, the less dependencies and above all a maintained package (look at the latest releases dates, revealing the package durability and support).

2 packages selected :

prodict

Prodict installation has no dependencies :

pip3 search prodict
prodict (0.8.3)  - Prodict = Pro Dictionary with IDE friendly(auto code completion), dot-accessible attributes and more.
pip3 install product
Successfully installed prodict-0.8.3

Dot notation usage is then very easy, undefined keys returns None instead of the default dict error AttributeError and keys can be dynamically added : 

import json
from prodict import Prodict

def main():
	with open('resources.json','rb') as f: 
		props = Prodict.from_dict(json.load(f))
		print(props.sitemap.url)
    
    if (props.sitemap.url2==None):
      props.sitemep.url2="/conception/python-3.8"
      print(props.sitemep.url2)

if (__name__=="__main__"):
	main()
/conception/python
/conception/python-3.8

Just one inconvenient, when nested dictionaries are defined in a list and only in this case :

"langurls":[
		{"lang":"fr" , "file":"python-parsing-des-arguments-argparse-et-getopt.html"}
		,{"lang":"en", "file":"python-parsing-arguments-packages-argparse-getopt.html"}
]

a new Prodict object is necessary to use dot notation, otherwise the error AttributeError is raised : 

import json
from prodict import Prodict

def main():
	with open('resources.json','rb') as f: 
    props = Prodict.from_dict(json.load(f))
		
    # print(props.langurls[0].lang)  Not possible with Prodict, Attribute Error raised
    print(Prodict.from_dict(props.langurls[0]).lang)

if (__name__=="__main__"):
	main()
fr

Prodict object is a dictionary, so it can be used as a dictionary and compared to regular dictionaries, no translation is necessary :

		with open('results.json','w') as r:
			json.dump(props, r, indent=4, ensure_ascii=False)
…, 
"sitemap": {
        "title": "Conception - Python",
        "url": "/conception/python",
        "url2": "/conception/python-3.8"
}

python-box

Python-box has dependencies (ruamel.yaml and toml parsers packages) :

pip3 search python-box
python-box (4.2.2)                       - Advanced Python dictionaries with dot notation access
pip3 install python-box
Successfully installed python-box-4.2.2 ruamel.yaml-0.16.10 ruamel.yaml.clib-0.2.0 toml-0.10.0

To use python-box

import json

from box import Box

def main():
	with open('resources.json','rb') as f: 
		props = Box(json.load(f))
		print(props.sitemap.url)

if (__name__=="__main__"):
	main()
/conception/python

About the inconvenient noticed with Prodict, dictionaries defined in a list are well managed by python-box : 

import json

from box import Box

def main():
	with open('resources.json','rb') as f: 
		props = Box(json.load(f))
		print(props.sitemap.url)
		print(props.langurls[0].lang)

if (__name__=="__main__"):
	main()

/conception/python
fr

Box objects can be compared to native dictionaries. The method to_dict translates a Box object to a native dictionary, but some tests performed on usual actions (json.dump for example) did not need to use the method to_dict.

The inconvenient :

  • Querying undefined keys raises an error (BoxKeyError)

Conclusion

In the author’s opinion of this paper, Prodict package is the most suitable : no dependencies, very light and above all, we can manage non existing keys as we would do in Javascript without any exception raised, that’s exactly the additional feature we were looking for in addition to the dot notation :

if (props.sitemap.url2 == undefined) { // Javascript code }
if (props.sitemap.url2 == None) :
  # Python code

In the case where dictionaries are defined in a list, workarounds can be easily found.