The Network Is Not There

By the end of this session you will be able to:

• Name the three states a callee can be in when your call fails, and say which one your code silently assumes.
• Derive a per-call timeout from the caller's remaining budget instead of a constant, and read a Go error to tell "never arrived" apart from "no answer came back".
• Explain why the two generals problem means no acknowledgement scheme closes the gap.

A timeout is not an error code

You call a service. Two hundred milliseconds later your client gives up and hands you an error. What do you know?

One thing: no response arrived in time. That is the entire content of the message. Every other reading of it is something you invented. Three states the server could be in produce exactly that error.

1. The request never arrived. Connection refused, packet dropped, name did not resolve. Nothing ran.
2. The request arrived, ran, and the response was lost coming back. The card was charged, the row was written. You just do not have the receipt.
3. The request arrived and is still running. You stopped waiting. The server did not stop working. The write lands three seconds from now, long after you decided the call failed.

A local call has two outcomes, returned or panicked, and you can always tell which. A remote call has these three, and from the caller's side they are indistinguishable. That is not a gap in your instrumentation. It is the shape of the problem.

This is where real money goes missing. A payment client times out, the caller retries, and the retry is a second charge. A scheduler times out, marks the worker dead, and hands the job to a second worker while the first still holds the lock. Both read a timeout as "it did not happen". A timeout means "I do not know".

Two generals, and why acknowledgements do not help

The obvious fix is to make the callee confirm. Send the request, wait for an ack, and now both sides agree. This does not work, and the reason bounds everything you build later in this course.

Two armies sit on hills either side of a valley. They win only if they attack at the same time, and the only way to communicate is a messenger who crosses the valley and may be captured.

Suppose a protocol exists that gets both generals to attack together. Take the shortest one and look at its final message. Either the sender attacks whether or not that message arrives, so the message is not needed and you have a shorter protocol, contradiction; or the sender's action depends on it arriving, which it can never confirm without a further message. Either way, the protocol does not exist.

No finite exchange over a lossy channel produces certainty that both sides agree. An ack of an ack does not help. TCP does not help, because its ack says bytes reached a kernel buffer, not that your handler committed.

What you get instead is a choice of which failure you prefer. Retry and you may execute twice. Do not retry and you may execute zero times. There is no third option at the transport layer, so the fix moves up into the application, which phase 3 spends a session on. Here the point is narrower: your failure model has to state which of the two you chose, per call path, on purpose.

Deriving the timeout instead of guessing it

Most timeouts in production are constants somebody typed once. That is how you get a chain where the edge waits one second, the service behind it waits two, and the one behind that retries three times at a second each. The inner work runs for six seconds against a caller who left after one, and every one of those seconds is a request in state 3.

The rule: a timeout is not a property of the call you are making, it is a property of the deadline you were given. Take the caller's deadline, subtract a margin for your own reply to get home, and cap it locally.

var errBudget = errors.New("call budget exhausted")
 
func callDownstream(ctx context.Context, c *http.Client, url string) (*http.Response, error) {
	// Spend what the caller still allows, less a margin for our reply.
	budget := 250 * time.Millisecond
	if dl, ok := ctx.Deadline(); ok {
		if left := time.Until(dl) - 20*time.Millisecond; left < budget {
			budget = left
		}
	}
	if budget <= 0 {
		return nil, errBudget
	}
	ctx, cancel := context.WithTimeoutCause(ctx, budget, errBudget)
	defer cancel()
 
	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
	if err != nil {
		return nil, err
	}
	return c.Do(req)
}

context.WithTimeoutCause is there so context.Cause(ctx) tells you whose deadline fired: your budget, or the caller's propagated down. Without a cause, every layer logs the same context deadline exceeded.

Reading the error you actually got

Some failures do rule out execution. Connection refused means no TCP session was established, so no bytes reached the handler, and a DNS failure means the same. Those are safely retryable. Everything else is unknown.

In Go 1.25 the errors are wrapped consistently enough to test with errors.Is. http.Client returns a *url.Error, and both a context deadline and the older http.Client.Timeout unwrap to context.DeadlineExceeded, because the internal timeoutError type defines Is(err) bool { return err == context.DeadlineExceeded }.

type Outcome int
 
const (
	Unknown Outcome = iota // it may have run; retrying may run it twice
	NotSent                // nothing reached the handler; safe to retry
	Done                   // the server answered
)
 
func classify(err error) Outcome {
	if err == nil {
		return Done
	}
	var dns *net.DNSError
	if errors.As(err, &dns) || errors.Is(err, syscall.ECONNREFUSED) {
		return NotSent
	}
	return Unknown
}

Note what it does not do: it never returns NotSent for a timeout, and it has no default case that quietly guesses. Anything it cannot prove is Unknown, which is deliberately the zero value. Never match on err.Error() strings here, because those messages change between releases.

Try it

Make state 3 visible. Save this as main.go and run it with go run main.go on Go 1.25.

package main
 
import (
	"context"
	"errors"
	"fmt"
	"net/http"
	"net/http/httptest"
	"sync/atomic"
	"time"
)
 
func main() {
	var executed atomic.Int64
 
	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		time.Sleep(1 * time.Second) // pretend this is the write
		executed.Add(1)
		fmt.Fprintln(w, "ok")
	}))
	defer srv.Close()
 
	client := &http.Client{}
	for i := 1; i <= 3; i++ {
		ctx, cancel := context.WithTimeout(context.Background(), 200*time.Millisecond)
		req, _ := http.NewRequestWithContext(ctx, http.MethodPost, srv.URL, nil)
		_, err := client.Do(req)
		cancel()
		fmt.Printf("attempt %d failed: deadline=%v err=%v\n",
			i, errors.Is(err, context.DeadlineExceeded), err)
	}
 
	time.Sleep(2 * time.Second)
	fmt.Println("handler executions:", executed.Load())
}

Success condition: three lines reporting deadline=true, then handler executions: 3. The client believes nothing happened. The server did the work three times.

Now change the handler sleep to 100 * time.Millisecond and run it again. Same client, same retry loop, one execution. The only difference between the correct run and the triple charge is where a latency number landed relative to a constant you typed.

Common mistakes

• Reading a timeout as "it did not happen". It is the one thing a timeout never tells you. If your failure path treats the outcome as anything but unknown, write down what makes that safe.
• Retrying a non-idempotent write on Unknown. The retry is a second execution, not a second attempt. Either the write carries a key the callee dedupes on, or the timeout ends the call path and something asynchronous reconciles later.
• Constant timeouts that ignore the caller's deadline. Pass context.Context from the inbound request all the way down and derive from ctx.Deadline(). A downstream timeout longer than your remaining budget is dead work with a live side effect.
• Cancelling the client and assuming the server stopped. Cancellation cancels your wait. The handler goroutine runs on until it checks r.Context() itself, and most handlers never do.

Where this goes next

You have one source of uncertainty pinned down: you cannot tell whether a call ran. The next session, Clocks Lie, adds the second. Even for writes you know succeeded, the timestamps you would order them by come from clocks that drift, jump backwards, and disagree across machines, which is why last-write-wins on a wall-clock timestamp drops writes without telling anyone.