Why timeouts are essential
When an application communicates with another service, we often think about handling errors, but we sometimes forget an essential element: timeouts. This has already happened to me, at my own expense...
A network call may seem simple:
Application
↓
External API
↓
Response
But what happens if the API never responds? Without a timeout, the application can remain blocked indefinitely while waiting for a response that may never arrive.
In this article, we will look at the different issues related to timeouts.
Blocked HTTP calls
Let's take a simple example. An application displays user information. To do this, it calls an external service:
Application
↓
User service
↓
Response
Normally, the response arrives within a few tens or hundreds of milliseconds. But one day:
- the remote server is down;
- the network has problems;
- the service is overloaded;
- a server-side request takes too long (deadlock, slow SQL query, infinite loop, blocked external dependency...).
Without a timeout:
Request sent
↓
Waiting...
↓
Waiting...
↓
Waiting...
A thread, connection, or server resource can remain occupied unnecessarily. Resources are consumed for no reason. If several users encounter the same problem, the application can quickly run out of resources:
100 blocked requests
↓
100 occupied connections
↓
No resources available
↓
Application unavailable
A problem with an external service can therefore cause a complete outage of our own application. Not guilty but responsible...
Connection timeout vs read timeout
Not all timeouts control the same thing. Let's go through them.
Connection timeout
This timeout corresponds to the time required to establish a connection with a remote server.
For example:
Application
↓
TCP connection
↓
Remote server
If the server is unreachable, we should not wait several minutes before giving up.
Example:
connection timeout = 3 seconds
After three seconds without successfully establishing the connection, the application considers the call failed.
Good, we did not remain blocked forever 🙂
Read timeout
A connection can be established, but the server can then take a very long time to respond.
Example:
Connection established
↓
Request sent
↓
Server processing
↓
Response after 10 minutes
The read timeout limits how long we wait for the response.
Example:
read timeout = 5 seconds
After five seconds without a response, the call is interrupted.
The two are therefore complementary:
Connection timeout:
"Can I reach the server?"
Read timeout:
"Is the server responding within an acceptable time?"
Retries
When a call fails, a common reaction is to try again.
The idea seems logical:
First attempt
↓
Failure
↓
Second attempt
↓
Success
This can indeed improve reliability when an error is temporary.
However, retries can also make a situation worse. Let's imagine a service that is already overloaded:
Slow service
↓
Pending requests
↓
Timeout
↓
Clients retry
↓
More requests
↓
Even slower service
The system enters a vicious circle, with retries consuming resources that could have been used to process other requests that might have succeeded.
Retries must therefore be used carefully:
- limit the number of attempts;
- add a delay between attempts (backoff);
- avoid retries on permanent errors;
- use idempotency when necessary.
A retry on an order creation or a payment can, for example, create duplicates if the operation was not designed to be repeated.
The domino effect
In an architecture composed of multiple services, a simple timeout can cause a domino effect.
Let's imagine:
Application
↓
Service A
↓
Service B
↓
Database
If service B becomes slow:
Slow service B
↓
Service A waits
↓
Application waits
↓
Users impacted
Each layer accumulates pending requests.
This is why timeouts must be defined at every level.
For example:
Database: 2 seconds
↓
Service B: 5 seconds
↓
Service A: 8 seconds
↓
Application: 10 seconds
Generally, an upper layer should have a slightly higher timeout than the calls it makes, while still keeping enough margin to process the response and handle errors.
The circuit breaker
When a service becomes unavailable, continuing to call it often only makes the situation worse. Each attempt consumes resources, increases the load on the failing service, and can further delay its recovery.
A circuit breaker detects repeated failures and temporarily stops calls to a service that is no longer responding.
Its behavior is similar to an electrical circuit breaker:
Service available
↓
Normal calls
↓
Successful responses
If too many calls fail:
Too many failures
↓
Circuit open
↓
Calls temporarily blocked
Instead of systematically waiting for a timeout, the application can quickly return an error or use a degraded behavior.
After a certain amount of time, the circuit enters a test state:
Time elapsed
↓
Automatic test
↓
Service available?
↓
Circuit reopened
If the service responds correctly, normal calls resume. If failures continue, the circuit remains open for another period.
This approach prevents a failing service from causing a domino effect throughout the entire application. Circuit breakers are particularly useful in distributed architectures where many services depend on each other.
That being said, implementing this is not always simple or even possible.
Common mistakes
Some common mistakes:
Not defining timeouts
Many HTTP libraries have different default behaviors. Some do not even have a read timeout.
A simple configuration mistake can therefore leave an application waiting indefinitely.
Using timeouts that are too long
A timeout of several minutes may seem comfortable, but it can quickly exhaust available resources.
A user will often prefer receiving an error after a few seconds rather than waiting for a response that may never arrive. They may then start many requests in parallel.
Using timeouts that are too short
Conversely, a timeout that is too aggressive can cause unnecessary errors on a service that normally works but occasionally requires more time.
Timeout values must therefore be chosen according to the context.
Some best practices
Some rules help avoid many problems:
- always define timeouts for network calls;
- distinguish connection timeout and read timeout;
- limit the number of retries;
- use backoff between attempts;
- make sensitive operations idempotent;
- monitor response times of external services;
- provide a degraded behavior when a service is unavailable.
Conclusion
Timeouts are often considered a simple configuration detail. We usually realize their importance too late, during a production incident (and thanks for the pressure from management to fix it...).
With good practices, most problems can be limited 🙂
