This content originally appeared on Level Up Coding - Medium and was authored by Adam Szpilewicz
Implementing Custom Serialization and Deserialization for Complex Data Types
In Go, the json.Marshal and json.Unmarshal functions are used for serializing and deserializing data to and from JSON format. By default, these functions use reflection to automatically convert the fields of a struct to JSON and vice versa. However, sometimes you may need custom behavior when serializing or deserializing specific types. That's where the MarshalJSON and UnmarshalJSON methods come in. In this post we will explore those features with some code exmaple.
MarshalJSON and UnmarshalJSON — overview
The MarshalJSON and UnmarshalJSON methods are part of the json.Marshaler and json.Unmarshaler interfaces, respectively. When you implement these methods for a type, you provide custom logic for converting the type to JSON and back. The json.Marshal and json.Unmarshal functions will automatically call these custom methods if they are defined for a type, instead of using the default behavior.
Here’s a brief overview of how these methods work:
- MarshalJSON: This method should be implemented on the type for which you want custom JSON serialization. It should return a byte slice containing the JSON representation of the value and an error, if any. When you call json.Marshal on a value of this type, it will invoke the MarshalJSON method instead of the default serialization behavior. You can implement any custom logic you need within this method, such as modifying field values, omitting fields, or encoding fields differently.
- UnmarshalJSON: This method should be implemented on the type for which you want custom JSON deserialization. It takes a byte slice containing the JSON data to be deserialized and should return an error, if any. When you call json.Unmarshal on a value of this type, it will invoke the UnmarshalJSON method instead of the default deserialization behavior. You can implement any custom logic you need within this method, such as validating field values, setting default values, or decoding fields differently.
By implementing the MarshalJSON and UnmarshalJSON methods on your types, you can have full control over the JSON serialization and deserialization process, allowing you to handle complex data structures, apply transformations or add validation as needed. These methods override the default behavior of the json package, enabling you to tailor the JSON handling to your specific use case.
MarshalJSON and UnmarshalJSON — under the hood
The Go encoding/json package uses the json.Marshaler and json.Unmarshaler interfaces to determine whether a custom implementation of MarshalJSON or UnmarshalJSON should be used. If the custom methods are not defined, it falls back to the default behavior. The encoding/json package will use then reflection to handle JSON serialization.
Below the interfaces definition from standard library json package that are required to implement to use custom logic for json handling.
// Marshaler is the interface implemented by types that
// can marshal themselves into valid JSON.
type Marshaler interface {
MarshalJSON() ([]byte, error)
}
// Unmarshaler is the interface implemented by types
// that can unmarshal a JSON description of themselves.
// The input can be assumed to be a valid encoding of
// a JSON value. UnmarshalJSON must copy the JSON data
// if it wishes to retain the data after returning.
//
// By convention, to approximate the behavior of Unmarshal itself,
// Unmarshalers implement UnmarshalJSON([]byte("null")) as a no-op.
type Unmarshaler interface {
UnmarshalJSON([]byte) error
}
The encoding/json package relies on Go's interfaces and type assertions to check if a type provides custom marshaling or unmarshaling methods. If the custom methods are not defined, the package will use reflection to handle JSON serialization and deserialization automatically.
Example in go
To apply in practice what we learnt about custom json handling we will show how to securely store and transmit JSON data in Go using custom serialization and deserialization methods. We will explore how to compress, encrypt, and encode the data before marshaling it into JSON format and then perform the reverse operations when unmarshaling the data.
Please note that the details of compression and encryption will not be covered in depth in this post; we will primarily focus on integrating these techniques with custom serialization and deserialization. In future posts, we may delve deeper into the specifics of compression and encryption algorithms.
Code Walkthrough
Firstly we define the Message struct and encryption key to resue it in our simple example.
type Message struct {
ID int `json:"id"`
Content string
}
const (
key = "a_secret_key_128" // 16, 24 or 32 bytes for AES-128, AES-192, or AES-256
)We define a simple Message struct with an ID field and a Content field. The encryption key is also defined as a constant.
Then we implement compression and decompression functions:
func compress(data []byte) ([]byte, error) { /*...*/ }
func decompress(data []byte) ([]byte, error) { /*...*/ }The compress function compresses the input data using the gzip compression algorithm, while the decompress function decompresses the gzip-compressed data. These functions will help us reduce the size of the data before encryption.
Implement encryption and decryption functions realying on the standard library packages.
func encrypt(plaintext []byte) ([]byte, error) { /*...*/ }
func decrypt(ciphertext []byte) ([]byte, error) { /*...*/ }The encrypt function encrypts the input data using AES-GCM encryption, and the decrypt function decrypts the encrypted data. AES-GCM provides both confidentiality and integrity for the data.
Implement custom MarshalJSON and UnmarshalJSON methods. The MarshalJSON and UnmarshalJSON methods are implemented for a custom Message type. These methods are part of the json.Marshaler and json.Unmarshaler interfaces, respectively. By implementing these interfaces, we can define custom JSON serialization and deserialization behavior for the Message type.
func (m *Message) MarshalJSON() ([]byte, error) { /*...*/ }
func (m *Message) UnmarshalJSON(data []byte) error { /*...*/ }In the MarshalJSON method, we first compress the Content field, then encrypt the compressed data, and finally encode the encrypted data using base64 encoding. This will ensure that the resulting JSON is both secure and text-based. In the UnmarshalJSON method, we reverse these operations: decode the base64-encoded data, decrypt the data, and decompress it to obtain the original Content field value.
Test the custom serialization and deserialization in main function.
func main() {
/*...*/
// Marshal message to JSON
jsonData, err := json.Marshal(&message)
/*...*/
// Unmarshal message to JSON
err = json.Unmarshal(jsonData, &decodedMessage)
/*...*/
}In the main function, we create a Message instance, marshal it to JSON using the custom MarshalJSON method, and then unmarshal the JSON back to a Message instance using the custom UnmarshalJSON method.
And the whole code snippet with additional comments.
package main
import (
"bytes"
"compress/gzip"
"crypto/aes"
"crypto/cipher"
"crypto/rand"
"encoding/base64"
"encoding/json"
"fmt"
"io"
)
// Message is a simple message struct that we want to
// encrypt and compress before marshalling to JSON
// and sending over the wire.
type Message struct {
ID int `json:"id"`
Content string
}
const (
key = "a_secret_key_128" // 16, 24 or 32 bytes for AES-128, AES-192, or AES-256
)
// compress compresses the given data using gzip compression
// and returns the compressed data.
func compress(data []byte) ([]byte, error) {
var buf bytes.Buffer
gw := gzip.NewWriter(&buf)
if _, err := gw.Write(data); err != nil {
return nil, err
}
if err := gw.Close(); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
// decompress decompresses the given gzip compressed data
// and returns the decompressed data.
func decompress(data []byte) ([]byte, error) {
buf := bytes.NewBuffer(data)
gr, err := gzip.NewReader(buf)
if err != nil {
return nil, err
}
defer gr.Close()
return io.ReadAll(gr)
}
// encrypt encrypts the given data using AES-GCM and returns
// the encrypted data.
func encrypt(plaintext []byte) ([]byte, error) {
// Generate a new AES cipher using our 16 byte long key
block, err := aes.NewCipher([]byte(key))
if err != nil {
return nil, err
}
// Generate a 12-byte nonce using a cryptographically
// secure random number generator
nonce := make([]byte, 12)
if _, err := io.ReadFull(rand.Reader, nonce); err != nil {
return nil, err
}
// Create a new GCM instance using the AES cipher
aesgcm, err := cipher.NewGCM(block)
if err != nil {
return nil, err
}
// Encrypt the data using the nonce and the GCM instance
// and return the encrypted data as a byte slice
ciphertext := aesgcm.Seal(
nonce, nonce, plaintext, nil)
return ciphertext, nil
}
// decrypt decrypts the given data using AES-GCM
// and returns the decrypted data.
func decrypt(ciphertext []byte) ([]byte, error) {
// Create a new AES cipher instance using the provided key
block, err := aes.NewCipher([]byte(key))
if err != nil {
return nil, err
}
// Create a new GCM instance using the AES cipher
aesgcm, err := cipher.NewGCM(block)
if err != nil {
return nil, err
}
// Decrypt the data using the nonce and the GCM instance
// and return the decrypted data as a byte slice
nonce, ciphertext := ciphertext[:12], ciphertext[12:]
plaintext, err := aesgcm.Open(
nil, nonce, ciphertext, nil)
if err != nil {
return nil, err
}
return plaintext, nil
}
// MarshalJSON implements the json.Marshaler interface.
func (m *Message) MarshalJSON() ([]byte, error) {
// Compress content before encrypting it.
compressedContent, err := compress([]byte(m.Content))
if err != nil {
return nil, err
}
// Encrypt compressed content using AES-GCM.
encryptedContent, err := encrypt(compressedContent)
if err != nil {
return nil, err
}
// Encode encrypted content as base64 string.
contentEncoded := base64.StdEncoding.EncodeToString(
encryptedContent)
return json.Marshal(&struct {
ID int `json:"id"`
Content string `json:"content"`
}{
ID: m.ID,
Content: contentEncoded,
})
}
// UnmarshalJSON implements the json.Unmarshaler interface.
func (m *Message) UnmarshalJSON(data []byte) error {
// Decode JSON into aux struct to get the base64-encoded content.
aux := &struct {
ID int `json:"id"`
Content string `json:"content"`
}{}
if err := json.Unmarshal(data, &aux); err != nil {
return err
}
m.ID = aux.ID
// Decode base64-encoded content to get encrypted content.
encryptedContent, err := base64.StdEncoding.DecodeString(
aux.Content)
if err != nil {
return err
}
// Decrypt content using AES-GCM.
decryptedContent, err := decrypt(encryptedContent)
if err != nil {
return err
}
// Decompress content using gzip.
decompressedContent, err := decompress(decryptedContent)
if err != nil {
return err
}
// Set content to decompressed content.
m.Content = string(decompressedContent)
return nil
}
func main() {
// Original message
message := Message{
ID: 1,
Content: "This is a secret message that needs " +
"to be encrypted and compressed.",
}
// Marshal message to JSON
jsonData, err := json.Marshal(&message)
if err != nil {
fmt.Println("Error marshalling message:", err)
return
}
fmt.Println("Encoded JSON:", string(jsonData))
// Unmarshal JSON back to message
var decodedMessage Message
err = json.Unmarshal(jsonData, &decodedMessage)
if err != nil {
fmt.Println("Error unmarshalling message:", err)
return
}
fmt.Printf("Decoded message: %+v\n", decodedMessage)
}
Conclusion
In this blog post, we’ve demonstrated how to securely store and transmit JSON data in Go by implementing custom serialization and deserialization methods. By combining compression, encryption, and base64 encoding, we’ve ensured that the data is transmitted securely.
Implementing custom serialization and deserialization methods, such as MarshalJSON and UnmarshalJSON, can provide numerous benefits for handling data in various scenarios. Some of the key advantages are as follows:
- Data Transformation: Custom serialization and deserialization methods allow you to transform data during the (de)serialization process. This can include compression, encryption, base64 encoding, or any other transformations required by your specific use case.
- Optimization: By tailoring the (de)serialization process to the specific requirements of your application, you can optimize storage, transmission and processing of data. This can lead to reduced bandwidth usage, lower storage costs and improved performance.
- Control over Data Representation: Custom (de)serialization methods give you complete control over how data is represented in serialized formats, like JSON. This can be useful when you need to comply with specific data formats or when you want to create a more human-readable or compact representation of your data.
- Validation: Implementing custom deserialization methods allows you to add validation checks during the deserialization process. This can help ensure that incoming data conforms to the required structure and constraints, improving data integrity and reducing the likelihood of errors.
- Backward Compatibility: Custom serialization and deserialization methods can be used to maintain backward compatibility with older data formats. This can help ensure that your application continues to work seamlessly as you evolve your data structures and formats over time.
In summary, custom serialization and deserialization methods provide you with increased flexibility and control over your data handling. By tailoring the (de)serialization process to the specific needs of your application, you can optimize performance, ensure data integrity, maintain compatibility and meet the requirements of various data formats and use cases.
Level Up Coding
Thanks for being a part of our community! Before you go:
- 👏 Clap for the story and follow the author 👉
- 📰 View more content in the Level Up Coding publication
- 💰 Free coding interview course ⇒ View Course
- 🔔 Follow us: Twitter | LinkedIn | Newsletter
🚀👉 Join the Level Up talent collective and find an amazing job
Advanced JSON Manipulation in Go was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Level Up Coding - Medium and was authored by Adam Szpilewicz
Adam Szpilewicz | Sciencx (2023-04-18T20:20:48+00:00) Advanced JSON Manipulation in Go. Retrieved from https://www.scien.cx/2023/04/18/advanced-json-manipulation-in-go/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.