Marathon, Mesos

Marathon Event Bus

It’s almost 3 months since i’ve started with Mesos + Marathon. Yesterday, i was going through the Marathon Doc Site, and found that Marathon has a cool internal Event Bus. Currently Marathon has an ibuilt HTTP callback subscriber that POSTs events in JSON format to one or more endpoints. So i decided to give it a try to my Mesos test environment. More documentations of the Event Bus are available in the Documentation page.

Currently, in my test environment all the Mesos and Marathon are installed from the Mesosphere Debian packages. Whether we use the packaged init or upstart script, both of them are directly calling the /usr/local/bin/marathon binary. For the Event Bus with http callback, we need to enable two flags (–event_subscriber http_callback –http_endpoints http://host1/foo,http://host2/bar) while starting the Marathon service ie, –event_subscriber, the type of subscriber that we are going to use and –http_endpoints, endpoints corresponding to the subscriber.

From the marathon binary, i found the it looks for files under the /etc/marathon/conf/ folder, where each file is the name of the flag to be enabled and the content of the file is the flag values. So in our case, we need two files inside the “/etc/marathon/conf/”, 1) event_subscriber and 2) http_endpoints

root@vagrant-ubuntu-trusty-64:/etc/marathon/conf# ls /etc/marathon/conf/
event_subscriber  http_endpoints

And the content of these files are,

root@vagrant-ubuntu-trusty-64:/etc/marathon/conf# cat /etc/marathon/conf/event_subscriber

root@vagrant-ubuntu-trusty-64:/etc/marathon/conf# cat /etc/marathon/conf/http_endpoints

Now for test purpose, lets start a minimal webserver using netcat listening to port 1234

$ nc -l -p 1234

Now netcat is listening on port 1234. Lets restart Marathon with our new flags.

$ service marathon restart

Now lets check if the flags are enabled properly,

root@vagrant-ubuntu-trusty-64:/etc/marathon/conf# ps axf | grep marathon | grep -v grep
2780 ?        Sl     0:36 java -Xmx512m -Djava.library.path=/usr/local/lib -Djava.util.logging.SimpleFormatter.format=%2$s %5$s%6$s%n -cp /usr/local/bin/marathon mesosphere.marathon.Main --zk zk://localhost:2181,localhost:2182,localhost:2183/marathon --master zk://localhost:2181,localhost:2182,localhost:2183/mesos --http_endpoints http://localhost:1234/ --event_subscriber http_callback
2791 ?        S      0:00  \_ logger -p -t marathon[2780]
2792 ?        S      0:00  \_ logger -p user.notice -t marathon[2780]

From the above ps command, it can be seen that the flags are enabled properly. Now i’m going to start a docker container. So as per the Event Bus documentation, Callbacks are Fired every time Marathon receives an API request that modifies an app (create, update, delete)

$ curl -X POST -H "Content-Type: application/json" localhost:8080/v2/apps -d@ubuntu.json

where ubuntu.json is,

    "container": {
    "image": "docker:///libmesos/ubuntu",
    "options" : []
  "id": "ubuntu2",
  "instances": "1",
  "cpus": ".3",
  "mem": "200",
  "uris": [ ],
  "ports": [9999],
  "cmd": "while sleep 10; do date -u +%T; done"

Once the Marathon API is fired, we will get a POST request on our netcat with the Event details. Below is the Event received on my netcat server.

root@vagrant-ubuntu-trusty-64:/var/tmp# nc -l -p 1234
Host: localhost:1234
Accept: application/json
User-Agent: spray-can/1.2.1
Content-Type: application/json; charset=UTF-8
Content-Length: 427

{"clientIp":"","uri":"/v2/apps","appDefinition":{"id":"ubuntu2","cmd":"while sleep 10; do date -u +%T; done","env":{},"instances":1,"cpus":0.3,"mem":200.0,"disk":0.0,"executor":"","constraints":[],"uris":[],"ports":[9999],"taskRateLimit":1.0,"container":{"image":"docker:///libmesos/ubuntu","options":[]},"healthChecks":[],"version":{"dateTime":{}}},"eventType":"api_post_event","timestamp":"2014-08-25T13:23:27.059Z"}

This Event bus is really usefull as it notifies us on all the Event changes happening on our Mesos cluster. We can also buit an Event notification system based on these callbacks. Though the current Events has very minimal details, i’m sure that more Event types will get added soon into Marathon.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s