Hiding Python dictionaries in classes

I have been doing Python development again for the past two months as I’m working on the backends for some of Newfront’s AI services. I like to use Python’s type hinting system, and I have been using dataclasses for defining, validating and passing data around.

Frequently, our services will need to talk to external APIs to fetch data that often is provided as JSON. Whenever that’s the case, I see my beautiful, typed code, become contaminated with things like this:

import requests
headers = {
  'Accept': 'application/json'
}
 
r = requests.get(
    'https://api.carbonintensity.org.uk/intensity', 
    params={}, 
    headers = headers
)
 
response_data = r.json()

Assuming the response schema is something like that:

{
  "data":[
    {
    "from": "2018-01-20T12:00Z",
    "to": "2018-01-20T12:30Z",
    "intensity": {
      "forecast": 266,
      "actual": 263,
      "index": "moderate"
    }
  }]
}

Then, we might need to so something like this to get the forecast and actual values:

forecast = response_data["data"][0]["intensity"]["forecast"]
actual = response_data["data"][0]["intensity"]["actual"]

Which is AWFUL! First, it doesn’t express the schema very well as lists and dictionaries have items referenced using the same square brackets. Also, if you return this data to your application, every place needing some piece of this data will have to run it, see how the data looks like, and then do something like that. Lastly, that introduces a lot of opportunities for exceptions. Any changes to the schema have potential to break several places in the code. And then you’ll have to fix this leaked, buggy, bad code.

Recently, I started wrapping that into a class that, using properties, can help others consume the data more safely.

class CarbonIntensityAPIResponseItem:

    def __init__(self, raw_item) -> None:
        self.__item = raw_item

    @property
    def from(self) -> Optional[datetime]:
        raw_from = self.__item.get("from")
        if raw_from is not None:
            return datetime.strptime(raw_from, "%Y-%m-%dT%H:%MZ") 

    @property
    def to(self) -> Optional[datetime]:
        raw_to = self.__item.get("to")
        if raw_to is not None:
            return datetime.strptime(raw_to, "%Y-%m-%dT%H:%MZ") 

    @property
    def intensity_forecast(self) -> Optional[int]:
        raw_forecast = self.__item.get("intensity", {}).get("forecast")
        if raw_forecast is not None:
            return raw_forecast

    @property
    def intensity_actual(self) -> Optional[int]:
        raw_actual = self.__item.get("intensity", {}).get("actual")
        if raw_actual is not None:
            return raw_actual

    @property
    def intensity_index(self) -> Optional[str]:
        raw_index = self.__item.get("intensity", {}).get("index")
        if raw_index is not None:
            return raw_index

class CarbonIntensityAPIResponse:
    
    def __init__(self, raw_response) -> None:
        self.__response = raw_response

    def __get_data(self) -> list[dict]:
        self.__response.get("data", []) 

    @property
    def items(self) -> int:
       return len(self.__get_data())

    def get_item(self, idx: int) -> CarbonIntensityAPIResponseItem:
        raw_item = self.__get_data()[idx]
        return CarbonIntensityAPIResponseItem(raw_item)

If instead of returning that JSON object you return this class, now you have:

• Defined a typed interface to which other places in the code can depend on.
• Concentrated all the data extraction logic in one place. If the API changes, we have only one place to change.
• Much more expressive. As the return types are saying, some values are optional and code can deal with that without having to read the API docs or assume some contract, only to be proven wrong later by a bug when processing a null property.

When compared to the previous example, here’s how someone could use the same forecast and actual values from the class replacing the dictionary response:

item = response_data.get_item(0)
forecast = item.intensity_forecast
actual = item.intensity_actual

See? Much cleaner! Much safer!