New installation of Openshift Origin cannot create medium size gears

Question

I recently installed Openshift Origin on a server (clean installation, I am not using the VM). I can create small size NodeJS applications, but when I specify the gear size medium (either through rhc or the web administration interface) I get this error : Unable to complete the requested operation due to: No nodes available. The user I am doing this with as access to 100 nodes and all node sizes. Anybody knows what might be happening here?

Besides, the medium gear size is needed to accommodate a KeystoneJS app.

--- EDIT ---

I was unable to fix the server, so I started the Openshift installation from scratch on a new CentOS install. I configured an all-in-one server with only one medium node within the default district. But after installation, running oo-diagnostics gives me these errors:

WARN: test_node_profiles_districts_from_broker The following gear profile(s) are configured but not provided by any node hosts: medium Attempts to create apps using these gear profiles will fail. Please fix the settings in /etc/openshift/broker.conf or add node hosts accordingly.

WARN: block in test_node_profiles_districts_from_broker There are no node hosts in district 'Default' FAIL: test_node_profiles_districts_from_broker Default gear profile 'medium' has no active node hosts supplying it in any district. Attempts to create apps without specifying gear size may fail. Please add active node hosts to a district with profile 'medium' using oo-admin-ctl-district or fix the settings in /etc/openshift/broker.conf

FAIL: test_node_profiles_districts_from_broker The following gear profile(s) are configured: small, medium but not provided by any active district hosts. Attempts to create apps using these gears may fail. Please add districts / node hosts with oo-admin-ctl-district or fix the settings in /etc/openshift/broker.conf

WARN: block in test_node_profiles_districts_from_broker Node host master.os.bonqoeur.ca with profile 'small' is not a member of any district. Please add it to a district with oo-admin-ctl-district.

Also, on this install I cannot create medium sized apps, but small apps works fine (even though no node is supposed to be available)... Anyone has a clue?

Answer 1

This issue could go many different ways, so I'll try to cover as many as I can here and hope that it moves you in the right direction.

First, I would ensure that you have a medium node profile (district) with available nodes. You can verify your districts by running oo-admin-ctl-district on your broker system.

You will be presented with output like this, that contains a block for each district.

{"_id"=>"55131197ecdd5c0555000001",
 "active_servers_size"=>2,
 "available_capacity"=>6000,
 "available_uids"=>"<6000 uids hidden>",
 "created_at"=>2015-03-25 19:50:47 UTC,
 "gear_size"=>"medium",
 "max_capacity"=>6000,
 "max_uid"=>6999,
 "name"=>"medium",
 "platform"=>"linux",
 "servers"=>
  [{"_id"=>"551318bbecdd5c0555000001",
    "active"=>true,
    "name"=>"node0.example.com",
    "unresponsive"=>false},
   {"_id"=>"551318d3ecdd5cd555000001",
    "active"=>true,
    "name"=>"node1.example.com",
    "unresponsive"=>false}],
 "updated_at"=>2015-03-25 19:50:47 UTC,
 "uuid"=>"55131197ecdd5c0555000001"}

Notice in the output where it shows active=true and unresponsive=false for the nodes within the district. This indicates that a node is healthy and responsive.

---

Mcollective should show that all your nodes are pingable when you run oo-mco ping from your broker system.

node0.example.com    time=220.43 ms
node1.example.com    time=228.63 ms

If you can't ping your nodes, you may have an issue with iptables or possibly mcollective configuration. Make sure that your nodes can communicate with your msgserver on port 61613. Also ensure that your node systems are correctly configured to communicate with the msgserver as the correct user, etc.

On node hosts, this configuration lives in /opt/rh/ruby193/root/etc/mcollective/server.cfg.

---

Next, assuming that your entire installation is contained on one system, you may only have a small node profile available (the one system). If this is true, you can create an additional node as well as a district/node profile for that node. This will require separate infrastructure with different configuration in /etc/openshift/resource_limits.conf.

To create a new district,

oo-admin-ctl-district -c create -n medium -p medium
oo-admin-ctl-district -c add-node -n medium -i yournode1.example.com

More information on district creation and population for origin is available here: http://docs.openshift.org/origin-m4/oo_administration_guide.html#creating-and-populating-districts