JSON workflow

As of Feb. 2021 the JSON tool sheet has been integrated into the Data-On-Disk toolbox with major revisions to pathlib Path, csv, json and sql. If you want to see the json sheet you will need only to look at the DoD toolbox. It is page 3.  In the process of putting together the new versions I discovered what I think is the real purpose of the pathlib Path one-line write command – it is optimized for saving json objects.  I could be wrong – but I don’t think so.

Understanding json’s functions can be aided by a moment to reflect on the workflow where it is used.  Assume you have Python software that manages a limited amount of data stored on disk in a json object in strings, numbers, lists, tuples and dictionaries all nested within a single large Python-like dictionary and this object is held inside a ”’string”’.  Actually, you can think of this as a kind of a small, poor man’s relational database. Ignore for the moment the chicken-or-the-egg issue of how the string/dictionary/object came to be.

Now here is a possible workflow scenario with json methods

  •  Create a very basic data structure of nested Python components in a standard Python dictionary, lets say you call it my_dict, then convert it using json.dumps(obj – in this case my_dict) to a ”’json string”’ in a try/except structure to eliminate obvious errors. For example, if you call your string data_string your command could be as smple as data_string = json.dumps(my_dict)
  •  You save that string to disk using json.dump(obj, fp – the python string you you just created, and a file on disk) knowing it has already passed most terminal error hurdles. The file you save has a .json extension, for example my_data.json.  Good practice is to do this in a with structure – see the demo.
  •  Later, You bring this data back in from disk using my_dict = json.load(fp the .json file you use for storing the data, my_data.json in the example) where it is retrieved and automatically converted (per the table in the toolbox) to nested dictionary and other data containers.
  • Your user programmatically adds, changes or deletes all the information data as needed with those changes being made to the data in my_dict.
  • In order to display a pretty-print of the data for the user, you create a json string of your dictionary using data_string = jsons.loads(obj -in this case my_dict) to use for this display.
  •  You once again use json.dumps(obj – in this case my_dict) and then json dump(obj, fp – in this case my_dict,data and my_data.json) to store the changes back to disk.  Your object could also be the data_string you created in the last step.

# json demo using all methods
# test json methods
import json
import time
import pprint
pp=pprint.PrettyPrinter(indent=3)
# set my_path to some directory on your system
my_path = "D:\Py Pi Code and Docs\json_test.json"
my_dict = {"cageA":{"monkey1":[{"type":"howler","age":"5","name":"Bigmouth"},{"shots":True}], "monkey2":[{"type":None,"age":4,"name":"Webster"},{"shots":False}]},
"cageB":{"ape1":[{"type":"gorilla","age":20,"name":"Mr. Big"},{"shots":True}],"ape2":[{"type":"orangutan","age":8,"name":"Longarm"},{"shots":True}]}}
# zoostr = "'''"+str(my_dict)+"'''" <-to create a json formatted string if you want to but True/False/None not corrected
print("This is our dictionary of nested objects")
print(type(my_dict))
print(my_dict)
print()
print("This is our converted json string - note True-true, None-null, False-false")
jz = json.dumps(my_dict)
print(type(jz))
print(jz)
#Now save it
print("\nSave to file\n")
with open(my_path,'w')as jobj:
    json.dump(my_dict, jobj)
print("Go to lunch, come back and get file again")
for i in range(1,4):
    print(i)
    time.sleep(1)
with open(my_path, 'r') as jobj:
    new_dict = json.load(jobj)
print("\nHere is our reloaded file\n")
print(new_dict)
print("\nConvert it to a string and pretty print it.\n")
pp.pprint(json.dumps(new_dict))
print("\nGet rid of cageB data.\n")
del new_dict["cageB"]
print("...and now we have ")
pp.pprint(json.dumps(new_dict))
print("\nfor comparison here is our original json string pretty printed:\n")
pp.pprint(json.loads(jz))

…and with a little luck our result looks like this – only if you run it on your system it will be nicely formatted:

This is our dictionary of nested objects
<class ‘dict’>
{‘cageA’: {‘monkey1’: [{‘type’: ‘howler’, ‘age’: ‘5’, ‘name’: ‘Bigmouth’}, {‘shots’: True}], ‘monkey2’: [{‘type’: None, ‘age’: 4, ‘name’: ‘Webster’}, {‘shots’: False}]}, ‘cageB’: {‘ape1’: [{‘type’: ‘gorilla’, ‘age’: 20, ‘name’: ‘Mr. Big’}, {‘shots’: True}], ‘ape2’: [{‘type’: ‘orangutan’, ‘age’: 8, ‘name’: ‘Longarm’}, {‘shots’: True}]}}

This is our converted json string – note True-true, None-null, False-false
<class ‘str’>
{“cageA”: {“monkey1”: [{“type”: “howler”, “age”: “5”, “name”: “Bigmouth”}, {“shots”: true}], “monkey2”: [{“type”: null, “age”: 4, “name”: “Webster”}, {“shots”: false}]}, “cageB”: {“ape1”: [{“type”: “gorilla”, “age”: 20, “name”: “Mr. Big”}, {“shots”: true}], “ape2”: [{“type”: “orangutan”, “age”: 8, “name”: “Longarm”}, {“shots”: true}]}}

Save to file

Go to lunch, come back and get file again
1
2
3

Here is our reloaded file

{‘cageA’: {‘monkey1’: [{‘type’: ‘howler’, ‘age’: ‘5’, ‘name’: ‘Bigmouth’}, {‘shots’: True}], ‘monkey2’: [{‘type’: None, ‘age’: 4, ‘name’: ‘Webster’}, {‘shots’: False}]}, ‘cageB’: {‘ape1’: [{‘type’: ‘gorilla’, ‘age’: 20, ‘name’: ‘Mr. Big’}, {‘shots’: True}], ‘ape2’: [{‘type’: ‘orangutan’, ‘age’: 8, ‘name’: ‘Longarm’}, {‘shots’: True}]}}

Convert it to a string and pretty print it.

(‘{“cageA”: {“monkey1”: [{“type”: “howler”, “age”: “5”, “name”: “Bigmouth”}, ‘
‘{“shots”: true}], “monkey2”: [{“type”: null, “age”: 4, “name”: “Webster”}, ‘
‘{“shots”: false}]}, “cageB”: {“ape1”: [{“type”: “gorilla”, “age”: 20, ‘
‘”name”: “Mr. Big”}, {“shots”: true}], “ape2”: [{“type”: “orangutan”, “age”: ‘
‘8, “name”: “Longarm”}, {“shots”: true}]}}’)

Get rid of cageB data.

…and now we have
(‘{“cageA”: {“monkey1”: [{“type”: “howler”, “age”: “5”, “name”: “Bigmouth”}, ‘
‘{“shots”: true}], “monkey2”: [{“type”: null, “age”: 4, “name”: “Webster”}, ‘
‘{“shots”: false}]}}’)

for comparison here is our original json string pretty printed:

{ ‘cageA’: { ‘monkey1’: [ {‘age’: ‘5’, ‘name’: ‘Bigmouth’, ‘type’: ‘howler’},
{‘shots’: True}],
‘monkey2’: [ {‘age’: 4, ‘name’: ‘Webster’, ‘type’: None},
{‘shots’: False}]},
‘cageB’: { ‘ape1’: [ {‘age’: 20, ‘name’: ‘Mr. Big’, ‘type’: ‘gorilla’},
{‘shots’: True}],
‘ape2’: [ {‘age’: 8, ‘name’: ‘Longarm’, ‘type’: ‘orangutan’},
{‘shots’: True}]}}