Description
Subject of the issue
When using the json format on /v2/blocks it returns serialized lg data that is unusable / garbage in many cases.
In this case, the bad serialization is most easily identifiable by the visual presence of utf-8 replacement characters -> https://www.compart.com/en/unicode/U+FFFD
Steps to reproduce
Method One
Look at txn index 16 (assuming start index of 0) which is a Tinyman swap:
https://mainnet-api.algonode.cloud/v2/blocks/41335116
Fields in dt.ld.1 have garbage data along with lg, particularly logs like output_asset_id and output_amount
Method Two
Run this script to see a more thorough example of trying to access Tinyman log data, which is provided by v2/blocks in json format.
from algosdk.v2client import algod
from base64 import b64decode
import pprint
algod_address = "https://mainnet-api.algonode.cloud"
algod_token = ""
algod_client = algod.AlgodClient(algod_token, algod_address)
def decode_logs(logs: "list") -> dict:
decoded_logs = dict()
for log in logs:
print("Decoding:", log)
if type(log) == str:
log = b64decode(log.encode())
if b"%i" in log:
i = log.index(b"%i")
s = log[0:i].decode()
value = int.from_bytes(log[i + 2 :], "big")
decoded_logs[s] = value
else:
raise NotImplementedError()
return decoded_logs
block = algod_client.block_info(block=41335116)
pp = pprint.PrettyPrinter(indent=2)
tinyman_tx = block.get('block').get('txns')[16]
log = tinyman_tx.get('dt').get('lg')
pp.pprint(log)
result = decode_logs(log)
pp.pprint(result)Thoughts
There are a few issues here:
- JSON format is currently serializing data that may be unusable, wasting loads of bandwidth + cpu cycles
- Not a total waste in cases where the bytestream can be serialized
- Switching to
msgpackis replete with all sorts of quirks across various languages, the biggest being the main Javascript library can't even decode it without issues - @joe-p has mentioned:
We technically could "fix" the JSON response to use surrogate escaping like Python supports, but again not every client would support that and that could actually break code that is currently working.
Which really sounds like the ideal answer, and then using path-versioning like /v2.1/blocks to provide a new endpoint to maintain legacy endpoint functioning.
- @pbennett notes:
problem likely stems from the 'logs' being defined as string not []byte - so it should be base64 encoded by algod for eg and its not.