Protocol Buffers: A Schema-Based Language for Structured Data

Protocol Buffers, often referred to as Protobuf, are a powerful, efficient, and language-agnostic method of serializing structured data, similar to XML or JSON but smaller, faster, and simpler. Designed by Google, Protobuf offers a unique approach to defining data structures and interfaces, allowing developers to specify a schema for their data once and then use auto-generated source code to easily write and read the structured data to and from multiple data streams. This mechanism not only streamlines data exchange across services and applications but also ensures backward compatibility and future-proofing of data models.

Documentation: https://protobuf.dev/

gRPC: Leveraging Protocol Buffers for High-Performance RPC APIs

gRPC is an open-source remote procedure call (RPC) system initially developed by Google. It uses Protocol Buffers as its interface definition language (IDL), enabling seamless, high-performance communication between systems, with support for multiple programming languages. gRPC is designed for low latency and high throughput communication, making it ideal for lightweight systems that require efficient inter-system calls. It supports both synchronous and asynchronous communication, featuring built-in load balancing, authentication, and advanced flow control. With gRPC, developers can create scalable and efficient APIs that are suited for distributed systems and cloud-native applications, leveraging the robust serialization capabilities of Protocol Buffers to ensure efficient data exchange.

Documentation: https://grpc.io/

Definition of ready

The “Definition of Ready” for prompts sent to GPT-4 involves preparing the prompt in such a manner that it maximizes the likelihood of a coherent, accurate, and relevant response. This preparation can be likened to the criteria used in Agile methodologies, such as Scrum, to ensure that a task or user story is ready for development. In the context of interacting with GPT-4, the Definition of Ready includes several crucial components:

1. Clear and Specific Objective: The prompt should have a well-defined goal. Whether it’s generating text, creating an image, or performing a calculation, the desired outcome should be explicitly stated. Ambiguity in the prompt can lead to irrelevant or generic responses.
2. Sufficient Context: Provide all necessary background information that GPT-4 needs to understand the task at hand. This includes explaining any specific context, setting, or constraints relevant to the request. Lack of context may result in answers that, while technically correct, are not useful in the given scenario.
3. Appropriate Complexity: The complexity of the prompt should match the capabilities of the model. While GPT-4 can handle a wide range of requests, overly complex or multi-layered prompts may need to be broken down into simpler, more manageable parts.
4. Compliance with Guidelines: The prompt must adhere to OpenAI’s use case policy, including avoiding requests for prohibited content, respecting privacy and copyright laws, and ensuring the prompt does not promote harmful or unsafe information.
5. Realistic Expectations: Users should have a realistic understanding of the model’s limitations. GPT-4 is a powerful tool but is not infallible. It can generate incorrect or nonsensical responses, especially in areas where it lacks sufficient training data or the task is highly specialized.
6. Preparation for Interaction: Be ready for a back-and-forth interaction with the model. Some tasks may require clarification, additional information, or refinement of the initial prompt. This iterative process can help hone in on the exact response desired.
7. Ethical Considerations: Ensure that the prompt does not ask the model to generate content that could be considered unethical, such as fabricating information, creating misleading content, or anything that could harm individuals or groups.

Creating the repository

I asked GPT-4 to come up with a directory structure for an program that uses Golang 1.22, protocol buffers and gRPC. It comes with the following structure:

Payments Schema

syntax = "proto3";

package internal.payments;

option go_package = "/internal/payments;payments";

import "google/protobuf/timestamp.proto";

// Payment creation request
message CreatePaymentRequest {
  string id = 1; // Unique identifier for the payment
  string user_id = 2; // ID of the user making the payment
  double amount = 3; // Payment amount
  string currency = 4; // Currency of the payment
  google.protobuf.Timestamp created_at = 5; // Timestamp of the payment creation
}

// Payment creation response
message CreatePaymentResponse {
  string id = 1; // Unique identifier for the payment
  string status = 2; // Status of the payment (e.g., "processed", "failed")
}

// Webhook event for payments
message PaymentWebhookEvent {
  string event_type = 1; // Type of the event (e.g., "payment_created", "payment_failed")
  CreatePaymentRequest payment = 2; // The payment related to the event
}

// Service for managing payments
service PaymentsService {
  // Create a new payment
  rpc CreatePayment(CreatePaymentRequest) returns (CreatePaymentResponse) {}
}

Audit logs schema

syntax = "proto3";

package internal.payments;

option go_package = "/internal/payments";

import "google/protobuf/timestamp.proto";

// Audit log creation request
message CreateAuditLogRequest {
  string id = 1; // Unique identifier for the audit log entry
  string event_type = 2; // Type of event being logged
  string description = 3; // Description of the event
  google.protobuf.Timestamp occurred_at = 4; // Timestamp of the event
}

// Audit log creation response
message CreateAuditLogResponse {
  string id = 1; // Unique identifier for the audit log entry
  bool success = 2; // Whether the log entry was successfully created
}

// Service for creating audit log entries
service AuditLogService {
  // Create a new audit log entry
  rpc CreateAuditLog(CreateAuditLogRequest) returns (CreateAuditLogResponse);
}

Building the server

Before we can start implementing the server we have to compile the proto definitions into go. This can be done with Protocol Buffer Compiler (https://grpc.io/docs/protoc-installation/).  

protoc --go_out=. \
       --go-grpc_out=. \
       api/proto/payments.proto

protoc --go_out=. \
       --go-grpc_out=. \
       api/proto/audit_logs.proto

After running protoc the following files are created:

Implementing business logic

In this phase, we have payment and audit log services that are yet to be implemented. These services will contain the business logics.

package payments

import (
	"context"
	"log/slog"
)

type PaymentsServer struct {
	UnimplementedPaymentsServiceServer
}

// CreatePayment creates a new payment
func (s *PaymentsServer) CreatePayment(ctx context.Context, in *CreatePaymentRequest) (*CreatePaymentResponse, error) {
	//todo: implement business logic.
	slog.Info("Received CreatePayment request", "request", in.String())
	return &CreatePaymentResponse{}, nil
}

package payments

import (
	"context"
	"log/slog"
)

type AuditLogsServer struct {
	UnimplementedAuditLogServiceServer
}

// CreateAuditLog creates a new audit log entry
func (s *AuditLogsServer) CreateAuditLog(ctx context.Context, in *CreateAuditLogRequest) (*CreateAuditLogResponse, error) {
	slog.Info("Received CreateAuditLog request", "request", in.String())
	return &CreateAuditLogResponse{}, nil
}

Starting the server

We’re now in a position to launch the server. This involves constructing a basic server application situated within the cmd/server directory. Implementation:

package main

import (
	"fmt"
	"google.golang.org/grpc"
	"grpc/internal/payments"
	"log"
	"net"
)

func main() {

	lis, err := net.Listen("tcp", fmt.Sprintf("localhost:%d", 50051))
	if err != nil {
		log.Fatalf("failed to listen: %v", err)
	}
	var opts []grpc.ServerOption

	grpcServer := grpc.NewServer(opts...)
	payments.RegisterPaymentsServiceServer(grpcServer, &payments.PaymentsServer{})
	payments.RegisterAuditLogServiceServer(grpcServer, &payments.AuditLogsServer{})

	grpcServer.Serve(lis)
}

Building the client

This article won’t delve into the construction of the client. Further details on constructing the client are available here.

Introducing new end-points (using AI)

I asked GPT-4 to introduce a status field on payments and a ChangeStatus method on the payments service. It comes with the following implemetation:

syntax = "proto3";

package internal.payments;

option go_package = "/internal/payments;payments";

import "google/protobuf/timestamp.proto";

// Payment creation request
message CreatePaymentRequest {
  string id = 1; // Unique identifier for the payment
  string user_id = 2; // ID of the user making the payment
  double amount = 3; // Payment amount
  string currency = 4; // Currency of the payment
  google.protobuf.Timestamp created_at = 5; // Timestamp of the payment creation
  string status = 6; // Status of the payment
}

// Payment creation response
message CreatePaymentResponse {
  string id = 1; // Unique identifier for the payment
  string status = 2; // Status of the payment (e.g., "processed", "failed")
}

// Request to change the status of a payment
message ChangeStatusRequest {
  string id = 1; // Unique identifier for the payment
  string status = 2; // New status for the payment
}

// Response after changing the status of a payment
message ChangeStatusResponse {
  string id = 1; // Unique identifier for the payment
  string status = 2; // Updated status of the payment
}

// Webhook event for payments
message PaymentWebhookEvent {
  string event_type = 1; // Type of the event (e.g., "payment_created", "payment_failed")
  CreatePaymentRequest payment = 2; // The payment related to the event
}

// Service for managing payments
service PaymentsService {
  // Create a new payment
  rpc CreatePayment(CreatePaymentRequest) returns (CreatePaymentResponse) {}
  // Change the status of a payment
  rpc ChangeStatus(ChangeStatusRequest) returns (ChangeStatusResponse) {}
}

To update the payments package we simply run:

protoc --go_out=. \
       --go-grpc_out=. \
       api/proto/payments.proto