How to handle config changes in consul

It is common to have dedicated systems to manage configurations for distributed applications. One of the options to do so is to have a consul running and serving key-value pairs to all the application instances interested in reading them. Usually the configuration is used during the application startup, but sometimes configurations change and applications need to obtain the latest data to act accordingly. In this article I will show how you can use consul API to listen to configuration changes.

I will rely on the consul KV API described here, the blocking queries mentioned here, the Consul library that conveniently does all the heavy lifting, and as bonus, I will use Polly to add some basic resiliency to the solution.

Let’s get straight to business and create a console application that does two simple things:

  1. Reads the config data from consul when it changes
  2. Writes the value read into the stdout every second.

The application entry point method is here is in full:

static void Main(string[] args)


It kicks of 2 Tasks, discussed below and then locks the main thread to prevent the application from successfully exiting.

 private static async Task MainLoop()
   while (true)
     await Task.Delay(TimeSpan.FromSeconds(1));
     Console.WriteLine($"Current value '{_someSetting}'");
private static string _someSetting = string.Empty;

The main loop outputs the value of the field every second.

private static Policy _retryPolicy = Policy.Handle<HttpRequestException>().RetryForeverAsync((e, c) => Console.WriteLine(e.Message));

private static async Task ConsumeConfig() {
 ulong waitIndex = 0;
 while (true)
   await _retryPolicy.ExecuteAsync(async () => {
     using (var consulClient = new ConsulClient(consulConfig => consulConfig.Address = new Uri("http://consul:8500"))) {
       QueryResult<KVPair> queryResult = await consulClient.KV.Get("someSetting", new QueryOptions() { WaitIndex = waitIndex, WaitTime = TimeSpan.FromMinutes(5) });
       byte[] responseValue = queryResult.Response.Value;
       string setting = Encoding.UTF8.GetString(responseValue);
       _someSetting = setting;
       waitIndex = queryResult.LastIndex; } // end polling consul
     }); // end policy execution
} // end method

I’ll explain the ConsumeConfig method line by line.

First we define a Polly policy, so if we get an HttpRequestException we should retry unconditionally. Obviously this should be changed into something more practical. Since my consul server starts at the same time the program does, usually the cluster is not ready to serve as fast as my program, so I retry until the consul starts serving.

The waitIndex is used to tell a long polling request from an ordinary key-value request, will get back to it later.

The only way to stop the program I designed is to kill it, thus while(true) will do the job of keeping the program listening to consul for ever.

I used ExecuteAsync version of Polly API since I planned to use async API of Consul library.

The next line issues a request to get the value of the key-value pair, which key is “someSetting”. I supply the value to the WaitIndex which is the heart of Consul blocking queries. To cut a long story short, if you pass the same value that was returned by the previous call to consul, the request will be blocked by the consul server until it either times out or the value for the key changes. The default WaitTime is 5 minutes, while you can set it to its maximum value of 10 minutes if you want.

The rest of the code gets the value from the response, applies it to the program by updating the field and updates the value of the waitIndex which will be reused immediately to make a new blocking call.

The code was tested on .NET Core 1.1 and Consul, both hosted in Docker and is free for copying and modification. If you run consul and this code, then go to your consul installation, create the setting and then keep updating it, then you will notice the appliation reacts to the cahnge immediately. In my case the url to edit the key-value pair is the following http://consul:8500/ui/#/dc1/kv/someSetting/edit

Good luck!



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s